**GAO-11-596**                      Published: Jun 15, 2011. Publicly Released: Jul 14, 2011.                  

-   
-   
-   
-   

    Jump To:  
-  Elder Abuse 
-  Services for Older Adults 
-  State and Local-Level Drivers and Trends 
-  State and Local Fiscal Projections 
-  Federal Borrowing 
-  Managing the Debt 
-  Summary of Funding Provisions 
-  Auditing the Government's Books 
-  Unpacking the Financial Report 
-  Highlights 
-  Recommendations 
-  Background 
-  About This Work 
-  About This Work 
-  Trends 
-  About the Center 
-  Why It's High Risk 
-  What Remains To Be Done 
-  Key Reports 
-  Newest COVID-Related Reports 
-  AI Use Cases 
-  Flood Insurance 
-  Past Pandemic-Related Reports 
-  CARES Act Oversight Reports 
-  All COVID-19 Reports 
-  Report Suspected Fraud 
-  Subjects 
-  How to Get Notified 
-  Past Reports 
-  Public Health Threats 
-  Explore the FY 2022 and 2023 Funds 
-  Lessons Learned 
-  Agency Reports 
-  Recent Reports 
-  Recommendations 
-  Supplemental Material 
-  Full Report 
-  Additional Data 
-  View Decision 
-  Downloads 
-  Previously Identified Actions 
-  Related Pages 
-  Associated Agencies 
-  Tough Choices and Opportunities Ahead 
-  Americas Fiscal Future Key Areas 
-  Retirement Security Key Areas 
-  Current List 
-  About The High Risk List 
-  Area Ratings 
-  Previous High Risk Products 
-  Search Federal Vacancies 
-  Past Federal Vacancies 
-  Violation Letters 
-  About the Yellow Book 
-  Advisory Council 
-  Related Resources 
-  Comment Letters 
-  Related Publications 
-  Contacts and Resources 
-  Major Issues Facing the Nation 
-  Meeting Strategic Challenges 
-  Additional Resources 
-  Resources 
-  GAO Contacts 
-  Mission Teams 
-  Operations and Staff Offices  
-  Organization Chart  
-  Why a Career at GAO? 
-  Am I Qualified? 
-  Company Culture 
-  Reasonable Accommodations 
-  Video Gallery 
-  Paid Internships 
-  Student Volunteers 
-  Professional Development Program 
-  Advancement 
-  Executive Candidate Program 
-  Summer Associates 
-  Externships 
-  Benefits 
-  Contacts 

Skip to Highlights

## Highlights

	In recent years, GAO has reported on challenges the Department of Homeland Security (DHS) has faced in effectively managing major acquisitions, including programs which were deployed before appropriate testing and evaluation (T&E) was completed. In 2009 and 2010 respectively, DHS issued new T&E and acquisition directives to address these challenges. Under these directives, DHS Science and Technology Directorate's (S&T) Test & Evaluation and Standards Office (TES) is responsible for overseeing T&E of DHS major acquisition programs--that is, those with over $300 million in life-cycle costs--to ensure that T&E and certain acquisitions requirements are met. GAO was asked to identify (1) the extent to which TES oversees T&E of major acquisitions; and (2) what challenges, if any, TES officials report facing in overseeing T&E across DHS components. GAO reviewed DHS directives and test plans, interviewed DHS officials, and reviewed T&E documentation from a sample of 11 major acquisition programs from each of 11 different DHS components. The results of the sample cannot be generalized to all DHS programs, but provided insights.

	TES met some of its oversight requirements for T&E of acquisition programs GAO reviewed, but additional steps are needed to ensure that all requirements are met. Specifically, since DHS issued the T&E directive in May 2009, TES has reviewed or approved T&E documents and plans for programs undergoing testing, and conducted independent assessments for the programs that completed operational testing during this time period. TES officials told GAO that they also provided input and reviewed other T&E documentation, such as components' documents describing the programs' performance requirements, as required by the T&E directive. DHS senior level officials considered TES's T&E assessments and input in deciding whether programs were ready to proceed to the next acquisition phase. However, TES did not consistently document its review and approval of components' test agents--a government entity or independent contractor carrying out independent operational testing for a major acquisition--or document its review of other component acquisition documents, such as those establishing programs' operational requirements, as required by the T&E directive. For example, 8 of the 11 acquisition programs GAO reviewed had hired test agents, but documentation of TES approval of these agents existed for only 3 of these 8 programs. Approving test agents is important to ensure that they are independent of the program and that they meet requirements of the T&E directive. TES officials agreed that they did not have a mechanism in place requiring a consistent method for documenting their review or approval and the extent to which the review or approval criteria were met. Without mechanisms in place for documenting its review or approval of acquisition documents and T&E requirements, such as approving test agents, it is difficult for DHS or a third party to review and validate TES's decision-making process and ensure that it is overseeing components' T&E efforts in accordance with acquisition and T&E directives and internal control standards for government entities. TES and DHS component officials stated that they face challenges in overseeing T&E across DHS components which fell into 4 categories: (1) ensuring that a program's operational requirements--the key performance requirements that must be met for a program to achieve its intended goals-- can be effectively tested; (2) working with DHS component program staff who have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time for T&E while remaining within program cost and schedule estimates. Both TES and DHS, more broadly, have begun initiatives to address some of these challenges, such as establishing a T&E council to disseminate best practices to component program managers, and developing specific guidance for testing and evaluating information technology acquisitions. In addition, S&T has reorganized to assist components in developing requirements that can be tested, among other things. However, since these efforts have only recently been initiated to address these DHS-wide challenges, it is too soon to determine their effectiveness. GAO recommends, among other things, that S&T develop mechanisms for TES to document its review or approval of component acquisition documentation and T&E requirements, such as approving operational test agents. DHS agreed with GAO's recommendations.

## Recommendations

## Recommendations for Executive Action

## Full Report

Highlights

Full Report (61 pages)

Accessible Text 

## GAO Contacts

Diana Maurer
maurerd@gao.gov

(202) 512-9627

## Office of Public Affairs

Sarah Kaczmarek
kaczmareks@gao.gov

(202) 512-4800

## Topics

Homeland Security

Agency evaluationDocumentationInformation technologyIT acquisitionsLife cycle costsOperational testingProcurementProgram managementSystems evaluationSystems managementSystems monitoringTechnology assessmentGovernment agency oversight