Department of Homeland Security 

Office of Inspector General 

Review of Management Agreements 
 

Developed for DHS' Primary Data Center 
 

OIG-10-56 February 2010 
Office ofInspector General 

u.s. Department of Homeland Security 

Washington, DC 20528 

Homeland
 

Security
 FEB	 18 Z010 

MEMORANDUM FOR:	 Richard Spires 

Chie:( Information Officer 

FROM:	 Kichard 

~i(.~ L. Skinner 

Inspector General 

SUBJECT:	 Final Letter Report: Review ofManagement Agreements 

Developed for DHS' Primary Data Center (OIG-10-56) 

We audited the efficiency of the Department of Homeland Security's (DHS) management 

agreements to establish a primary data center at the Stennis Space Center in Mississippi. 

The department has agreements with the U.S. Navy to establish and maintain the DHS 

primary data center. We determined that the associated interagency agreements for this 

effort are missing key elements. Additionally, the department's technical representatives 

have not reviewed invoices associated with $160 million in expenditures. 

The audit included a review of DHS policies and procedures, interagency agreements, the 

memorandum of agreement, and prior audit reports. We are recommending that DHS: 

• 

•	 

Update its interagency agreements to include (a) the location of the building; 

(b) DHS' percentage of space, power, and utilities; and (c) the methodology for 

the allocation of costs. 

Review invoices and supporting documentation for fund transfers to ensure that 

funds were appropriately used. 

Background 

In April 2009, we reported on DHS' efforts to improve its disaster recovery planning for 

information systems.1 We identified technical and managerial concerns that might hinder 

the ability of DHS' primary data center to perform mission-essential functions. As a 

result of that audit, we initiated a review to determine whether the management 

agreements among the various stakeholders at the Stennis Space Center assist or hinder 

DRS' efforts to establish a primary data center. 

1 DHS' Progress in Disaster Recovery Planning for Information Systems, OIG-09-60, April 2009. 

Page 1 
We reviewed actions DHS had taken to select the location of its primary data center. In 

2004, DHS staff visited the Stennis Space Center in Mississippi several times to survey 

proposed buildings that could be used for the data center. In November 2004, the DHS 

Chief Information Officer committed $5 million to establish data center services at the 

Stennis Space Center. However, the buildings that DHS staff had surveyed were not 

available, and the U.S. Navy provided DHS another building at the Stennis Space Center. 

The new building was a former munitions manufacturing facility that needed extensive 

renovations. This building is on land permitted to the U.S. Army and operated by the 

U.S. Navy. DHS is one of several tenants in the U.S. Navy building. 

We reviewed the interagency agreements between DHS and the U.S. Navy. We also 

reviewed the process DHS uses to verify that goods and services received for the primary 

data center are approved before payments are made. 

Interagency Agreements Do Not Contain Key Elements 

Since 2005, the department has obligated more than $160 million in interagency 

agreements with the U.S. Navy to convert a former munitions manufacturing facility at 

the Stennis Space Center in Mississippi into a data center. However, the interagency 

agreements between DHS and the U.S. Navy do not contain key elements essential to 

ensure that the facility can continue to operate as DHS’ primary data center. 

According to Management Directive 125-02, Interagency Agreements, 

“Each Interagency Agreement identifies the supplies and services to be provided 

and indicates the fee charged by the Servicing Agency. The description of 

supplies or services must be specific, definite, and clear in order to support a 

binding agreement that will be recorded as an obligation. Further, the Interagency 

Agreement must establish a ceiling amount limiting the Requesting Agency's 

financial obligation.” 

Key elements that are missing from these agreements include DHS’ allotted floor space, 

the amount of power and utilities provided, and DHS’ percentage of shared costs. 

Without these items in the interagency agreements, DHS cannot be assured that it can 

continue to house its primary data center in the current location, or that DHS is 

reimbursing the U.S. Navy only for expenses directly related to the primary data center. 

In November 2004, DHS committed to placing its primary data center at the Stennis 

Space Center without establishing an agreement with the U.S. Navy about the specific 

building that DHS would occupy. After DHS committed $5 million to use the Stennis 

Space Center, DHS was provided another building that it had not previously reviewed. 

According to the U.S. Navy, DHS spent $64.1 million to renovate the replacement 

building. 

In September 2008, DHS and the U.S. Navy signed a formal memorandum of agreement 

for the operation and maintenance of this renovated facility. This document and the 

Page 2 
original interagency agreements specify what services the U.S. Navy will provide. 

However, these documents do not specify the amount of floor space that DHS will 

control or DHS’ percentage of shared operating costs. 

Our review of DHS’ payments indicates that the department is paying more than its fair 

share of costs for this facility. The DHS primary data center is located in a multitenant 

building. This building is operated by the U.S. Navy on land that has been permitted to 

the U.S. Army. The U.S. Army is charging the U.S. Navy $549,332 for the use of 

250,973 square feet of office space on 64 acres of land at the Stennis Space Center. 

According to the U.S. Navy, DHS is occupying approximately 81%, or 61,206 square 

feet, of the floor space in one of its shared buildings. However, the U.S. Navy is 

charging DHS $444,959, or approximately 81% of the U.S. Navy’s $549,332 total cost, 

instead of charging DHS based on the percentage of space DHS is using. We discussed 

this matter with DHS officials, who agreed that DHS may be paying the U.S. Navy too 

much for this space. 

Without documented agreements identifying the fees to be charged, as well as the 

specific supplies and services to be provided, DHS cannot be assured that it is operating 

its primary data center as cost-effectively as possible. 

DHS Is Not Reviewing Invoices for Its Primary Data Center 

DHS technical representatives have not reviewed invoices and other documents that 

detail the expenditures associated with the $160 million that DHS has paid the U.S. Navy 

for its primary data center since January 2005. Without reviews of supporting 

documentation, senior DHS management cannot be assured that department funds are 

being used to acquire the goods and services outlined in the interagency agreements in a 

cost-effective manner and for authorized purposes. 

The U.S. Navy is using the Intra-Governmental Payment and Collection (IPAC) System 

to obtain DHS funds designated for DHS’ primary data center.2 However, the U.S. Navy 

did not provide supporting documentation or invoices for these transfers before receiving 

the funds. In addition, DHS technical representatives did not request supporting 

documentation for the transactions and did not notify the DHS contracting officers, fund 

transfer officials, or budget officers that supporting documentation was not provided. 

According to an U.S. Navy official, the IPAC System was used to obtain DHS funds 

made available through the interagency agreements for capital improvements. For 

example, as discussed above, DHS funds were used for renovating a munitions 

manufacturing facility and for rent. Additionally, according to a U.S. Navy official, DHS 

funds were used for rent; salaries; and the purchase of nonexpendable property such as air 

conditioning units (chillers), uninterruptible power supplies, fuel tanks, electrical power 

supply elements, and five of six emergency generators. However, DHS was not provided 

the invoices or documentation supporting charges for these goods and services. 

2

 The IPAC System’s primary purpose is to provide a standardized interagency fund transfer mechanism for 

Federal Program Agencies. 

Page 3 
According to Management Directive 125-02, Interagency Agreements, 

“The Program Manager is ultimately responsible for ensuring that program funds 

are spent in a manner that results in planned outcomes (i.e., that all Interagency 

Agreements contain clearly defined requirements, and performance measures and 

monitoring plans to the maximum extent practicable). Further, DHS Program 

Offices must ensure that adequate documentation is maintained with respect to the 

results achieved under each Interagency Agreement.” 

According to DHS Directive 125-02-001, Instruction Guide on Interagency Agreements, 

the following activities are to occur before the close-out phase of an interagency 

agreement: 

“Prior to close-out, reconciliation of the funds and acceptance of performance 

must occur (Requesting Agency Program Manager in conjunction with the 

appropriate Servicing Agency Point of Contact).” 

Without supporting documentation, DHS cannot be assured that IPAC transfers to the 

U.S. Navy related to the DHS primary data center were for authorized goods or services 

and were appropriately used for capital improvements, rent, salaries, and nonexpendable 

property. 

Recommendations 

We recommend that the DHS Chief Information Officer (CIO) direct DHS technical 

representatives and contracting officers to: 

Recommendation #1: Update interagency agreements between the U.S. Navy and DHS 

to include (a) the location of DHS’ data center building; (b) DHS’ percentage of space, 

power, and utilities; and (c) the methodology for determining DHS’ allocation of costs 

for space at the Stennis Space Center. 

Recommendation #2: Obtain and review invoices and supporting documentation for 

U.S. Navy’s IPAC transfers related to the DHS primary data center to ensure that funds 

were appropriately used for designated goods and services. 

Page 4 
Management Comments and Analysis 

We obtained written comments on a draft of this report from the DHS CIO. In the 

comments, the CIO indicates that the report contains several inaccuracies. Specifically, 

the CIO states that two specific interagency agreements contained power and floor space 

requirements. We disagree. Neither agreement specifies the amount of floor space or 

power that will be dedicated for DHS’ use. Additionally, the other 29 agreements we 

reviewed did not contain the amount of floor space and power that would be dedicated to 

DHS’ use. 

Further, the CIO indicated that our report appeared to confuse raised floor space with 

other types of space. We modified the text in this section of the report to emphasize our 

concern that DHS was paying 81% of the U.S. Navy’s total costs, including costs for 

buildings not occupied by DHS. 

Finally, the CIO asserted that onsite surveys and visual inspections were sufficient 

actions to ensure that program funds were properly spent. However, it is our opinion that 

these actions alone did not provide sufficient oversight. These actions must be supported 

by verifying that adequate documentation exists for costs charged to DHS for 

construction, supplies, and personnel. 

The CIO also did not concur with our second recommendation. Specifically, the CIO 

noted that both DHS and the Office of Inspector General auditors sought invoices from 

the U.S. Navy without success. However, we disagree that DHS should have accepted 

this apparent lack of cooperation from the U.S. Navy. Staff from the DHS Office of the 

CIO should have notified senior DHS officials of this problem. When informed of the 

problem, DHS officials should have negotiated with the U.S. Navy and the General 

Services Administration for access to the relevant documents. 

Page 5 
Appendix A 

Purpose, Scope, and Methodology 

This audit is a follow-up audit to our report on DHS’ Progress in 

Disaster Recovery Planning for Information Systems (OIG-09-60). 

The previous report concerned DHS’ disaster recovery planning 

and its two new data centers. This report focuses on the DHS 

Management Directorate’s efforts to establish one of those data 

centers, DHS’ primary data center. 

The objective of our audit was to determine whether the 

management/ownership agreements among the various 

stakeholders assist or hinder DHS’ efforts to establish a primary 

data center. We reviewed DHS policies and procedures, 

interagency agreements, memorandum of agreement, statements of 

work, and prior audit reports. On-site inspections and interviews 

of key personnel performed for the previous report were also used 

to support this effort. 

We conducted fieldwork at DHS Management Directorate 

facilities and organizational elements in the Washington DC, 

metropolitan area. We also requested information from the 

U.S. Navy, the U.S. Army, and the National Aeronautics and 

Space Administration. While these organizations provided us with 

most of the data we requested, the U.S. Navy did not provide 

invoices related to work performed at DHS’ primary data center. 

We conducted this audit between April and October 2009. 

We provided DHS staff with briefings and presentations 

concerning the results of fieldwork and the information 

summarized in this report. We conducted the audit according to 

generally accepted government auditing standards. Those 

standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We 

believe that the evidence obtained provides a reasonable basis for 

our findings and conclusions based on our audit objectives. 

We appreciate the efforts of DHS management and staff to provide 

the information and access necessary to accomplish this audit. The 

principal Office of Inspector General points of contact for the audit 

are Frank Deffer, Assistant Inspector General for Information 

Technology Audits, (202) 254-4100, and Sharon Huiswoud, 

Director, Information Systems Division, (202) 254-5451. Major 

contributors to the audit are identified in Appendix C. 

Page 6 
I .s. I)rp.rlmrn' ofllomrlud S«urill

\\ """nglon. DC 2OS28

-7~'

\~'

Homeland

Security

MEMORANDUM FOR: Richard L. Skinner -,

JnspcctorGen~ra~v

FROM: Richard Spires

Chicf Infonnatio 0 ccr

SUBJECT: Filial Ldler Repor/: Rel'iew ofMWIUj.!emellf Agreellumls

Dewdopedfor DHS' Primary [)(I/(I Celller Re~"[JlJl1se

Please find our comments on the subject report. In addition to responding to thc

recommendations we have addressed severnl inaccuracies that \\ere noted in the report.

Page 2, fifth paragr:lph ("Key clements... ") - DHS' requirements for power and floor

space were specified in Interagency Agrecmcnts IISHQI'A-OS-X-OI066 and I-lSl-IQPAOS-X-Ol066,1'00002. The current Memorandum ofAgreement (MOA) with the

Commandcr Naval Metcorology and Oceanography Command (CNMOC) specifies that

annual budget for facility services will be proposed and mutually discussed. Interagency

agreements for recurring facility costs prior and subsequent to this MOA have specified

services and COSts.

()age 3, first full paragrallh ("Our rc\'iew.....) - The report appears to confuse raised

floor spacc with other types ofspace. DHS uses a number of buildings and types of

space within the fonner Mississippi Anny Ammunition Plant (MSAAP). Raised floor

space, as the premium facility commodity. is used by CNMOC as the dClemlining metric

for costs cited in this pllrllgraph. DHS currently has exclusive use of 81% ofthe raised

floor space withintht: MSSAP.

I'age 4, initial refcrence to Management Directivc 125-02 - DHS personnel undertook

repeated on-site surveys ofthe facility during the course construction, perfomling II

thorough visual inspection of assets such as chillers. fuel tanks. power supply clements

and emergency generators. Such inspection represents full altention to the MD 125-02

requirement to ensure program funds arc properly spent.

Recommendation #1: Updale illlerogellCY agreemenl.\· between Ihe U.S. Navy and D/-IS

10 il/dllde: (a) Ihe loca/ioll ofDHS d(l/a eenler (bui/dinJ.U. (b) DIfS' pl!rcenta?e 0/space,

Appendix B 

Management Comments to the Draft Report 

Page 7 
 
(lower, (lnd uli/hies, alld (c) Ih" melhodologyfor delerminin~DHS' allocation ofcosts

for spac/! alllll! SI/!lInis Space Cenler.

OCIO Concurs with caveat· Facility ownership is expected to transfer from CNMOC

to the NASA Stennis Space Center (SSC) in April 2010. DHS and SSC are presently

working to establish an agreement for specific terms and conditions. DI-IS plans to

incorporate this recommendation into the new agreclllclll with SSe.

Recommelldaliflll #2: Oblain (1/1d rn'iew invoices and supporling doel/mentationfor

u.s. Navy's IPAC lramfers related 10 the DNS primary dl/Ill cenler to ensure thatfimds

were appropriately usedfor de,rig/la/('llWwd\' (J/u! sen'ices.

OCIO Non-concurs - OilS has sought invoices from Navy without success. The Dl-IS

Office of Inspector General also sought invoices without success. It is not a common

practice for invoices to be provided in the kinds ofintra-govemmental relationships such

as that DHS has for the data center at sse. COnlfacting functions related to facility

services arc pcrfomlcd by the Navy and the General Services Administration (GSA). thus

access to invoices is controlled by CNMOC Contracting Officer Rcprcsclllativcs and

GSA Contracting Officers. These p<1rtics h(\ve direct responsibility for vendor oversight

and invoice review, Funher, these panics have treated invoice infomlation as subject to

privity. As DHS works with sse to establish a new agreement. eITons will be made to

impose a requirement lor access to vendor invoices, but results cannot be assured. DHS

has and will continue to mitigate this risk through direct on-site observation of rendered

services and capabilities.

Page 8 
 
Appendix C 

Major Contributors to This Report 

Sharon Huiswoud, Director, Department of Homeland Security, 

Information Technology Audits 

Kevin Burke, Audit Manager, Department of Homeland Security, 

Information Technology Audits 

Domingo Alvarez, Senior Auditor, Department of Homeland 

Security, Information Technology Audits 

Matthew Worner, Senior Auditor, Department of Homeland 

Security, Information Technology Audits 

Philip Greene, Referencer 

Page 9 
Appendix D 

Report Distribution 

Department of Homeland Security 

Secretary 

Deputy Secretary 

Chief of Staff for Policy 

Chief of Staff for Operations 

Deputy Chiefs of Staff 

General Counsel 

Executive Secretary 

Under Secretary for Management 

Director, GAO/OIG Liaison Office 

Assistant Secretary for Policy 

Assistant Secretary for Legislative Affairs 

Assistant Secretary for Public Affairs 

Office of Management, Audit Liaison 

Chief Information Officer, DHS 

Deputy Chief Information Officer, DHS 

Chief Information Security Officer, DHS 

Director, Compliance and Oversight 

Deputy Director, Compliance and Oversight 

DHS Chief Information Officer, Audit Liaison 

DHS Chief Information Security Officer, Audit Liaison 

Office of Management and Budget 

Chief, Homeland Security Branch DHS OIG Budget Examiner 

Congress 

Congressional Oversight and Appropriations Committees, as 

appropriate 

Page 10 
ADDITIONAL INFORMATION AND COPIES 

To obtain additional copies of this report, please call the Office of Inspector General (OIG) at (202) 254-4100, 

fax your request to (202) 254-4305, or visit the OIG web site at www.dhs.gov/oig. 

OIG HOTLINE 

To report alleged fraud, waste, abuse or mismanagement, or any other kind of criminal or noncriminal 

misconduct relative to department programs or operations: 

• Call our Hotline at 1-800-323-8603; 

• Fax the complaint directly to us at (202) 254-4292; 

• Email us at DHSOIGHOTLINE@dhs.gov; or 

• Write to us at: 

DHS Office of Inspector General/MAIL STOP 2600, 

Attention: Office of Investigations - Hotline, 

245 Murray Drive, SW, Building 410, 

Washington, DC 20528. 

The OIG seeks to protect the identity of each writer and caller.