DHS Science and Technology: Additional Steps Needed to Ensure Test and Evaluation Requirements Are Met
Published: 2011-06-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In recent years, GAO has reported on challenges the Department of Homeland Security (DHS) has faced in effectively managing major acquisitions, including programs which were deployed before appropriate testing and evaluation (T&E) was completed. In 2009 and 2010 respectively, DHS issued new T&E and acquisition directives to address these challenges. Under these directives, DHS Science and Technology Directorate's (S&T) Test & Evaluation and Standards Office (TES) is responsible for overseeing T&E of DHS major acquisition programs--that is, those with over $300 million in life-cycle costs--to ensure that T&E and certain acquisitions requirements are met. GAO was asked to identify (1) the extent to which TES oversees T&E of major acquisitions; and (2) what challenges, if any, TES officials report facing in overseeing T&E across DHS components. GAO reviewed DHS directives and test plans, interviewed DHS officials, and reviewed T&E documentation from a sample of 11 major acquisition programs from each of 11 different DHS components. The results of the sample cannot be generalized to all DHS programs, but provided insights.
section_title: What GAO Found
paragraphs: TES met some of its oversight requirements for T&E of acquisition programs GAO reviewed, but additional steps are needed to ensure that all requirements are met. Specifically, since DHS issued the T&E directive in May 2009, TES has reviewed or approved T&E documents and plans for programs undergoing testing, and conducted independent assessments for the programs that completed operational testing during this time period. TES officials told GAO that they also provided input and reviewed other T&E documentation, such as components' documents describing the programs' performance requirements, as required by the T&E directive. DHS senior level officials considered TES's T&E assessments and input in deciding whether programs were ready to proceed to the next acquisition phase. However, TES did not consistently document its review and approval of components' test agents--a government entity or independent contractor carrying out independent operational testing for a major acquisition--or document its review of other component acquisition documents, such as those establishing programs' operational requirements, as required by the T&E directive. For example, 8 of the 11 acquisition programs GAO reviewed had hired test agents, but documentation of TES approval of these agents existed for only 3 of these 8 programs. Approving test agents is important to ensure that they are independent of the program and that they meet requirements of the T&E directive. TES officials agreed that they did not have a mechanism in place requiring a consistent method for documenting their review or approval and the extent to which the review or approval criteria were met. Without mechanisms in place for documenting its review or approval of acquisition documents and T&E requirements, such as approving test agents, it is difficult for DHS or a third party to review and validate TES's decision-making process and ensure that it is overseeing components' T&E efforts in accordance with acquisition and T&E directives and internal control standards for government entities. TES and DHS component officials stated that they face challenges in overseeing T&E across DHS components which fell into 4 categories: (1) ensuring that a program's operational requirements--the key performance requirements that must be met for a program to achieve its intended goals-- can be effectively tested; (2) working with DHS component program staff who have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time for T&E while remaining within program cost and schedule estimates. Both TES and DHS, more broadly, have begun initiatives to address some of these challenges, such as establishing a T&E council to disseminate best practices to component program managers, and developing specific guidance for testing and evaluating information technology acquisitions. In addition, S&T has reorganized to assist components in developing requirements that can be tested, among other things. However, since these efforts have only recently been initiated to address these DHS-wide challenges, it is too soon to determine their effectiveness.
section_title: What GAO Recommends
paragraphs: GAO recommends, among other things, that S&T develop mechanisms for TES to document its review or approval of component acquisition documentation and T&E requirements, such as approving operational test agents. DHS agreed with GAO's recommendations.

section_title: Letter
paragraphs: Department of Homeland Security (DHS) acquisitions represent hundreds  of billions of dollars in life-cycle costs to support a wide range of missions,  including securing the nation’s borders, mitigating natural disasters,  screening airline passengers and baggage, and investigating security  threats.  DHS acquisition spending has increased by 50 percent from $9.1  billion in fiscal year 2004 to $13.6 billion in fiscal year 2010. A key goal of  DHS’s acquisitions process is ensuring that programs and technologies  meet technical and performance specifications and are tested in DHS’s  operational environment, and that the results of these tests are evaluated  before these programs and technologies are allowed to progress toward  purchase and deployment. The Homeland Security Act of 2002 created  DHS and, within it, established the Science and Technology Directorate  (S&T). The act provided S&T with responsibility for conducting national  research, development, test and evaluation (T&E), and procurement of  technology and systems for, among other things, detecting, preventing,  protecting against, and responding to terrorist attacks. S&T’s Test &  Evaluation and Standards Office (TES) is responsible for, among other  things, conducting oversight over T&E activities across all of DHS’s  components to ensure that major acquisitions—those with life-cycle costs  of over $300 million dollars—being considered by DHS are appropriately  tested and evaluated by DHS component agencies prior to their purchase  and deployment.
We have previously reported on several major DHS acquisitions that were  deployed before appropriate T&E was successfully completed. For  example, in October 2009, we reported that the Transportation Security  Administration (TSA) procured and deployed explosives trace portal  equipment—a machine which detects traces of explosives on airline  passengers by using puffs of air to dislodge particles from the passengers’  body or clothing into an analyzer—even though TSA officials were aware  that earlier tests did not demonstrate reliable performance in an airport  environment. We recommended that TSA conduct an evaluation and  determine whether it was cost effective to continue to use these machines.  TSA concurred with this recommendation and later halted further  deployment of these machines due to performance, maintenance, and  installation problems. As of April 2011, TSA reported that it had removed  all 101 machines that it had deployed from airports. Furthermore, in  January 2010, we reported that DHS had not effectively managed key  aspects of the testing of Customs and Border Protection’s (CBP) Secure  Border Initiative Network (SBInet), a multibillion dollar program to  deliver surveillance and decision-support technologies along the U.S.  border with Mexico and Canada. Among other things, we reported that  test procedures were largely not executed as written and changes to these  procedures were not made according to a documented quality assurance  process which increases the risk that the procedures would not support  test objectives or reflect the system’s ability to perform as intended. We  made a number of recommendations related to the content, review, and  approval of test planning documentation and resolution of system  problems, with which DHS generally concurred. In January 2011, the  Secretary of Homeland Security directed CBP to end SBInet as originally  conceived and to develop a new border technology deployment plan.  These past problems highlight the importance that T&E plays in the  successful development of major acquisition programs and technologies,  as well as the importance of overseeing the T&E efforts of DHS’s  components. In May 2009, DHS issued a T&E directive requiring  components to, among other things, ensure adequate and timely T&E is  performed to support informed acquisition decision making and also  requires that S&T’s TES oversee these activities by reviewing component  T&E activities and documentation, and approving those T&E activities and  documentation related specifically to operational testing.
You asked us to evaluate S&T’s efforts to oversee T&E across DHS and  identify any challenges that it faces in performing this mission.  Specifically, this report addresses:  (1) the extent to which TES oversees T&E of selected DHS major  acquisition programs throughout the system acquisition process; and  (2) what challenges, if any, TES and component officials report facing in  coordinating and overseeing T&E across DHS acquisition programs.
To address our objectives, we reviewed DHS departmental and  component-level policies and guidance, such as DHS’s acquisition  directive and guidebook, T&E directive, and our past reports. We also  reviewed relevant program documentation, including memoranda of  acquisition decision events in the acquisition process.  Further, we  conducted interviews with relevant DHS and component officials involved  in the acquisition process including T&E of programs and technologies.  We focused our review generally on the period after May 2009 when DHS  issued its T&E directive since there were no specific T&E requirements  prior to its issuance. We focused our review on TES’s T&E staff and  activities and did not review the efforts of TES staff who were engaged in  developing national standards to meet homeland security mission needs or  TES staff in two testing facilities, since developing standards and  conducting tests did not relate to overseeing T&E across DHS components  and thus, were outside the scope of our review.
Regarding our objective to determine the extent to which TES oversees  T&E of selected DHS major acquisition programs throughout the system  acquisition process, we initially selected a nonprobability sample of 12  programs of the 86 on DHS’s major acquisitions list for fiscal year 2010  and, for 11 of the 12 programs, we analyzed related acquisition and T&E  documentation, such as the program’s test plans, as well as memoranda  documenting approval of operational test agents (test agents), and  interviewed DHS component officials to help determine the extent that  TES reviewed these plans and agents and documented its approval during  fiscal year 2010. Selections were based on three factors—programs  which had undergone a senior DHS management review in fiscal year  2010, programs representing different DHS components, and programs  which were overseen by all nine of TES’s Test Area Managers. For 11  programs, we conducted semistructured interviews and collected  information from component-level officials involved in the development of  T&E documentation. The results of these analyses cannot be generalized  to TES’s efforts to oversee all major acquisition programs, but they  provide informative examples of the extent to which TES carried out its  oversight responsibilities. We reviewed documentation of TES’s oversight  efforts and compared them to the requirements in the DHS acquisition and  T&E directive to determine the extent to which TES fulfilled its  responsibilities. We also compared TES efforts to document their  oversight to Standards for Internal Control in the Government. Further,  we conducted site visits to S&T’s Transportation Security Laboratory and  TSA’s Transportation Security Integration Facility to obtain an overview of  testing in general and to specifically observe testing of transportation  security technologies. See appendix I for a description of the 11 DHS  major acquisition programs selected for our analysis.
To determine what challenges, if any, TES and component officials report  facing in coordinating and overseeing T&E of major DHS acquisition  programs, we interviewed TES and component officials responsible for  T&E from the 11 selected programs and conducted semistructured  interviews with all nine TES Test Area Managers who were primarily  responsible for conducting T&E oversight in fiscal year 2010. We analyzed  and categorized the challenges that these officials said they faced in  conducting T&E of their programs. We discussed the major challenges we  identified with TES officials, who agreed with our evaluation. We also  reviewed various TES and DHS initiatives to address these challenges. For  example, we reviewed minutes from TES’s T&E Council, which was  formed to promote T&E best practices and lessons learned in establishing  consistent T&E policies and processes for use in acquisition programs  throughout DHS. In addition, we reviewed minutes of T&E Council  working groups, which were formed to address particular challenges, such  as testing and evaluating information technology acquisitions. We also  reviewed documents and interviewed TES and component officials with  regard to T&E training and certification efforts. Further, we interviewed  the Under Secretary for S&T and other S&T executives regarding S&T’s  T&E efforts and the challenges that S&T and TES face in overseeing T&E  across DHS. Finally, we reviewed our past work related to these  challenges.
We conducted this performance audit from May 2010 through June 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Overview of DHS Acquisition Process
paragraphs: DHS acquisitions support a wide range of missions and investments  including ships and aircraft, border surveillance and screening equipment,  nuclear detection equipment, and systems to track the department’s  financial and human resources. In support of these investments, DHS, in  2003, established an investment review process to help reduce risk and  increase the chances for successful acquisition outcomes by providing  departmental oversight of major investments throughout their life cycles  and to help ensure that funds allocated for investments through the budget  process are being spent wisely, efficiently, and effectively.
Our work over the past several years has consistently pointed to the  challenges DHS has faced in effectively managing and overseeing its  acquisition of programs and technologies.
In November 2008, we reported that DHS had not effectively  implemented its investment review process, and as a result, the  department had not provided the oversight needed to identify and  address cost, schedule, and performance problems for its major  acquisitions. Specifically, we reported that of the 48 major  investments reviewed requiring milestone or annual reviews, 45 were  not reviewed in accordance with the departments’ investment review  policy, and 18 were not reviewed at all. Four of these investments had  transitioned into a late acquisition phase—production and  deployment—without any required reviews. We recommended and  DHS concurred that DHS identify and align sufficient management  resources to implement oversight reviews in a timely manner  throughout the investment life cycle.
In June 2010, we reported that over half of the 15 DHS programs we  reviewed awarded contracts to initiate acquisition activities without  component or department approval of documents essential to planning  acquisitions, setting operational requirements, and establishing  acquisition program baselines. Our work noted that without the  development, review, and approval of these key acquisition documents,  agencies are at risk of having poorly defined requirements that can  negatively affect program performance and contribute to increased  costs. In January 2011, DHS reported that it has begun to implement an  initiative to assist programs with completing departmental approval of  acquisition program baselines.
In our February 2011 biennial update of the status of high-risk areas  needing attention by Congress and the executive branch, we continued  to designate DHS’s implementation and transformation, which includes  the department’s management functions, as a high-risk area. For  example, because of acquisition management weaknesses, major  programs, such as SBInet, have not met capability, benefit, cost, and  schedule expectations. Further, DHS had not fully planned for or  acquired the workforce needed to implement its acquisition oversight  policies as we previously recommended. As of January 2011, DHS  reported that it had increased its acquisitions management staffing and  planned to hire more staff to develop cost estimates.
DHS has taken several actions to address these recommendations and  implement more discipline and rigor in its acquisition processes.  Specifically, DHS created the Acquisition Program Management Division  in 2007 to develop and maintain acquisition policies, procedures, and  guidance as a part of the system acquisition process. DHS also issued an  interim acquisition directive and guidebook in November 2008 for  programs to use in preparing key documentation to support component  and departmental making. In January 2010, DHS finalized the acquisition  directive which established acquisition life-cycle phases and senior-level  approval of each major acquisition program at least three times at key  acquisition decision events during a program’s acquisition life-cycle. This  directive established the acquisition life-cycle framework with four  phases:   (1) identify a capability need (need phase);  (2) analyze and select the means to provide that capability (analyze/select  phase);  (3) obtain the capability (obtain phase); and   (4) produce, deploy, and support the capability (produce/deploy/support  phase).
Each acquisition phase culminates in a presentation to the Acquisition  Review Board (ARB), which is to review each major acquisition (that is,  those designated as level 1 or level 2 programs) at least three times at key  acquisition decision events during a program’s acquisition life cycle. The  acquisition decision authority—the Chief Acquisition Officer or other  designated senior-level official—is to chair the ARB and decide whether  the proposed acquisition meets certain requirements necessary to move on  to the next phase and eventually to full production. The directive outlines  the extent and scope of required program, project, and service  management; level of reporting requirement; and the acquisition decision  authority based on whether the acquisition is classified as level 1, 2, or 3.  The acquisition decision authority for major acquisitions—level 1 and level  2—is to be at the department or component level and the acquisition  decision authority for nonmajor acquisitions—level 3—is to be at the  component level.An acquisition may be raised to a higher level  acquisition level by the ARB. The ARB supports the acquisition decision  authority in determining the appropriate direction for an acquisition at key  Acquisition Decision Events.
Following an ARB meeting, the Acquisition Program Management Division  is to prepare an acquisition decision memorandum as the official record of  the meeting to be signed by the acquisition decision authority. This memo  is to describe the approval or other decisions made at the ARB and any  action items to be satisfied as conditions of the decision. The ARB reviews  are to provide an opportunity to determine a program’s readiness to  proceed to the following life-cycle phase. However, we reported in March  2011 that the ARB had not reviewed most of DHS’s major acquisition  programs by the end of fiscal year 2009 and programs that were reviewed  had not consistently implemented action items identified as part of the  review by established deadlines. Our prior work has shown that when  these types of reviews are skipped or not fully implemented, programs  move forward with little, if any, early department-level assessment of the  programs’ costs and feasibility, which contributes to poor cost, schedule,  and performance outcomes.
As a part of its responsibilities, the Acquisition Program Management  Division has identified major DHS acquisition programs, projects, or  services for oversight through the ARB process. According to Acquisition  Program Management Division officials, beginning in fiscal year 2009, the  list was to be updated on a yearly basis through interviews with and  documentation from component program offices. In May 2010, the  Undersecretary for Management identified 86 programs on DHS’s major  oversight list for fiscal year 2010, 62 of which TES and component officials  determined required T&E oversight—that is programs that were in an  acquisition phase where T&E was being planned or conducted. Several of  the 62 programs consisted of multiple subprojects, such as TSA’s  Passenger Screening Program. For more information on these 86 major  acquisition programs, see appendix II.
DHS’s 2010 acquisition directive also includes guidance for preparing  documentation to support component and departmental decision making  and specifies requirements for developmental and operational T&E as a  part of the acquisition review process. Developmental T&E may include  a variety of tests, such as system qualification testing, system acceptancetesting, and software testing. Developmental testing may be carried out by  the user and may be conducted in simulated environments, such as  laboratories, test facilities, or engineering centers that might or might not  be representative of the complex operational environment. Operational  T&E is a field test, performed under realistic conditions by actual users in  order to determine the operational effectiveness and suitability of a  system, and the corresponding evaluation of the data resulting from the  test.
section_title: TES’s Role in Overseeing Component Testing and Evaluation
paragraphs: To carry out its responsibilities for overseeing T&E, S&T established TES  in 2006 and created the position of Director of TES in June 2007. TES’s  mission is to establish and manage DHS T&E policies and procedures and  to oversee and coordinate T&E resources to verify attainment of technical  performance specifications and operational effectiveness and suitability.   To carry out its T&E oversight, in fiscal year 2010, TES had a budget of  about $23 million and as of February 2011 had a staff of 26, which includes  the TES Director, 19 staff dedicated to T&E activities, and 6 dedicated to  developing standards.
In May 2009, DHS issued a delegation which specified the responsibilities  and duties of the Director of Operational Test & Evaluation. The TES  Director and Director of Operational Test and Evaluation, while distinct  positions in the T&E directive, share some advisory, review, and oversight  responsibilities.  For example, both are responsible for advising program  managers in developing T&E documentation and approving test and  evaluation master plans. The TES Director is responsible for developing  DHS T&E policy and the Director of Operational Test and Evaluation is to  approve operational test plans and report to the ARB after assessing  operational test reports.  Since May 2009, the Director of Operational Test  and Evaluation position has not been continuously filled according to the  current TES Director. In a November 2010 memo, the Under Secretary  for Science and Technology designated one person as both the director of  TES and the Director of Operational Test and Evaluation until further  notice.
The T&E directive outlines the responsibilities of the TES Director and the  Director of Operational Test and Evaluation. According to the directive,  the TES Director is to establish the department’s testing and evaluation  policies and processes and the Director of Operational Test and  Evaluation is to administer those policies and processes. The directive  also outlines TES’s responsibilities in overseeing T&E across DHS  components and its role in the acquisition review process. Table 1  describes TES’s T&E responsibilities as outlined in the T&E directive for  all level 1, level 2, and special oversight acquisition programs.
The T&E directive requires TES to review and approve required  component acquisition documentation before an ARB meets for an  acquisition decision event.  These documents are meant to be reviewed  and, if required, approved in a sequential order associated with the  acquisition phase, because these documents build upon one another.  Figure 1 presents TES’s responsibilities throughout the four DHS  acquisition phases as defined in the acquisition directive.
To carry out these responsibilities for the 62 acquisition programs under  its oversight in fiscal year 2010, TES has test area managers who assist  component officials in fulfilling their T&E responsibilities and provide  guidance and clarification in regard to the requirements in the T&E  directive.  According to TES, each major acquisition program is assigned a  test area manager and as of February 2011, TES employed nine test area  managers.
section_title: TES Met Some Oversight Requirements for T&E of Acquisition Programs Reviewed; Additional Steps Needed to Ensure That All Requirements Are Met
paragraphs: TES met its oversight requirements when approving test plans and test  reports in accordance with DHS acquisition and T&E directives for the 11  major acquisition programs we selected for review.  However, TES did not  consistently document its review and approval of operational test agents  or its review of other required acquisition documentation, which could  provide more assurance that components were meeting T&E directives  when TES reviewed these documents. Further, TES does not plan an  independent assessment of TSA’s Advanced Spectroscopic Portal’s  operational test results, as required by the T&E directive.
subsections: section_title: TES Oversight of Components’ Test Plans and Test Reports
paragraphs: TES is to oversee T&E of major DHS acquisition programs by ensuring  that the requirements set forth in the T&E directive are met and by  working with component program officials to develop T&E  documentation, such as test and evaluation master plans, as required by  DHS’s acquisition directive. TES’s T&E oversight responsibilities set forth  in the T&E and acquisition directives pertain to programs primarily in the  analyze/select and obtain phases of the acquisition process because most  testing and evaluation efforts occur in these phases. As a result, the  requirements of the T&E directive and TES’s oversight vary depending on  when a program progresses through certain phases of the acquisition  process. For example, when a program is in the produce/deploy/support  phase there is usually little to no T&E activity, so TES’s involvement is  limited.
We reviewed TES’s T&E oversight efforts for 11 DHS programs and found  that TES had conducted oversight of components’ test plans and test  reports, as set forth in the acquisition and T&E directives, as it asserted.  The 11 programs, each managed by different DHS components, were in  one phase of the acquisition process or had two or more subprojects  simultaneously in different phases of the acquisition process. For example,  Coast Guard’s H-65 helicopter program has 6 discrete subprojects, each  with its own completion schedule, including 4 subprojects in the  Produce/Deploy/Support phase and 2 subprojects in the Obtain phase.  Acquisition Program Management Division, TES, and component officials  determine if subprojects need to develop separate sets of acquisition  documents as they progress through the acquisition process. Figure 2  provides an overview of these programs and their associated acquisition  phases. Additional details on these programs can be found in appendix I.
As shown in figure 3, for the 11 selected DHS programs, TES reviewed and  approved test and evaluation master plans for 6 of the 7 programs that  were required to develop such plans by the T&E and acquisition directives  and had documented their approval of these plans. For the one program  that was in the phase that required such a plan—ATLAS Tactical  Communications—the program had not yet drafted its test and evaluation  master plan. The remaining  4 programs had plans in draft form that had  not yet been submitted to TES for review. As a result, TES was not yet  required to review these plans.
Component officials from each of these six programs stated that TES  provided input to the development of the test and evaluation master plans.  For example, Office of Health Affairs officials stated that TES officials  suggested that the BioWatch Gen-3 program office incorporate an  additional test event to ensure that the program was tested under specific  environmental conditions described in the program’s operational  requirements document, which resulted in more tests.  In addition, U.S.  Customs and Border Protection (CBP) officials stated that TES  participated in a line-by-line review of the SBInet test plan and provided  detailed suggestions. Further, TES suggested that the criteria used for  operational testing in the test and evaluation master plan needed to be  expanded, and that an update may be required for SBInet to progress to  the next acquisition phase. All of the component program officials who  had undergone TES review or approval told us that TES test area  managers provided their input in a variety of ways, including participating  in T&E working groups, in specific meetings to discuss T&E issues, or by  providing written comments culminating in TES’s approval of the plan.
After the test and evaluation master plan is developed, the test agent is to  develop operational test plans, which detail field testing of the system  under realistic conditions for determining that the system’s overall  effectiveness and suitability for use before deployment of the system. As  shown in figure 4, of the 11 selected acquisition programs, TES reviewed  and approved operational test plans for the 4 programs that were required  to develop such plans by the acquisition directive and documented their  approval of these plans.
Component officials from these 4 programs said that TES provided input  into their test plans.  For example, National Protection and Programs  Directorate officials from the National Cybersecurity Protection System  program stated that TES had significant comments on their operational  test plan, such as including confidence levels associated with the syste key performance requirements and helping program officials select a  sample size necessary to measure statistically significant results. In  addition, TES officials requested that the plan include different testing  scenarios in order to demonstrate a varied use of the system. In a officials from the Transportation Security Administration’s (TSA)  Advanced Technology-2 program indicated that TES provided significan input to their plan through a working group. The remaining 7 of the 11  programs had not yet begun to develop their operational test plan.
At the conclusion of operational testing, the test agent is to write a re on the results of the test. The T&E directive specifies that TES is to  receive the operational test report, which is to address all the critic issues and provide an evaluation of the operational suitability and  operational effectiveness of the system. After reviewing the operatio test report, TES then is to write a letter of assessment—which is an  d  independent assessment of the adequacy of the operational test an provides TES’s concurrence or nonconcurrence on the test agent  evaluation of operational suitability and operational effectiveness. TES is  to provide the letter of assessment to the ARB as it is determining whe a program should progress to the production and deployment phase.
Of the 11 programs we selected to review, TES developed a letter of  assessment for the 1 program— TSA’s Advanced Technology 2 —that had completed operational testing and had a written operational T&E report  on the results. The assessment concluded that while the T&E activities  were adequate to inform the ARB as to system performance, TES did not  concur with TSA’s test agent’s assessment as to system effectiveness  because the system did not achieve a key performance parameter during  testing. The ARB considered the letter of assessment and TES’s input and  granted TSA permission to procure and deploy a limited number of  screening machines. TSA will have to go before the ARB again to  determine if full-scale production can proceed after TSA has provided the  ARB with a business case and risk mitigation plan related to testing issues.  The remaining 10 selected programs had not completed operational testing  and thus, were not ready for letters of assessment.
In addition to letters of assessment, TES officials told us that they  regularly discuss T&E issues and concerns either verbally or through e- mails with Acquisition Program Management Division officials, who are  responsible for organizing ARB meetings. For example, Acquisition  Program Management Division officials stated that they rely on TES to  provide candid information about the suitability of various programs’ T&E  and whether these issues impact their program’s readiness to go before  the ARB.  Further, the officials told us that TES’s input at the ARBs, if any,  is to be documented in acquisition decision memorandums.  Acquisition  Program Management Division officials also noted that TES’s input may be  used in making the decision about when to hold an ARB for a particular  program. T&E input from TES is one of many factors the ARB uses in  overseeing acquisitions. For example, according to S&T officials, the ARB  considers the current threat assessments and the extent to which the  program, if implemented sooner, would help to address that threat. The  ARB also considers factors such as the cost of the program and potential  costs of conducting more testing and whether the results of operational  testing were sufficient to achieve the intended benefits of the program. As  a result, the ARB may accept a higher level of risk and allow a program to  proceed even if testing concerns have been raised, if it determines that  other reasons for quicker implementation outweigh these concerns.
TES officials also stated that they work extensively with components prior  to ARB meetings to ensure that T&E issues are addressed, with the goal to  address these issues before going before the ARB. TES meets with  component officials during regular acquisition review team meetings to  resolve various issues before ARB meetings are convened. For example,  due to concerns about the results of system qualification tests, TES  recommended to SBInet program and ARB officials that the program  should not proceed to the next milestone—site preparation, tower  construction, and sensor and communication equipment installation at the  Ajo, Arizona test site—until after operational testing was completed at the  Tucson, Arizona test site. In May 2009, the ARB authorized SBInet to  proceed with plans for the Ajo, Arizona site despite TES’s advice to the  contrary, and directed TES to work with component officials to revise test  plans, among other things.
section_title: Additional Steps Needed to Ensure that T&E Requirements Are Met
paragraphs: While TES’s oversight of the test plans and reports for major acquisition  programs selected for review is in accordance with provisions in the T&E  directive, it did not consistently document its review and approval of  certain acquisition documentation or document the extent to which  certain requirements in the T&E directive were met.
subsections: section_title: TES Did Not Consistently Document the Extent to which Criteria Used in Its Approval of Operational Test Agents Were Met
paragraphs: The T&E directive requires that an operational test agent—a government  agency or independent contractor carrying out independent operational  testing for major acquisition programs—is to meet certain requirements to  be qualified and approved by TES, but does not specify how TES’s  approval is to be documented. According to the T&E directive, the test  agent may be within the same component, another government agency, or  a contractor, but is to be independent of the developer and the  development contractor. Because the responsibilities of a test agent are  significant throughout the T&E process, this independence is to allow the  agent to present objective and unbiased conclusions regarding the  system’s operational effectiveness and suitability to DHS decision makers,  such as the ARB. For example, some the test agent’s responsibilities in the  T&E directive include:     Being involved early in the acquisition cycle by reviewing draft  requirements documents to help ensure that requirements are testable  and measurable.
Assisting the component program manager in the preparation of the  test and evaluation master plan.
Planning, coordinating, and conducting operational tests, and  preparing the operational T&E report.
Reporting operational test results to the program manager and TES.
According to TES officials, the test agent is also to meet other  requirements in order to be approved by TES, such as having the expertise  or knowledge about the product being tested and having the capacity and  resources to execute the operational tests. To ensure that criteria for test  agents are met, the T&E directive requires TES to approve all agents for  major acquisition programs. As shown in figure 5, of the 11 programs we  reviewed, 8 programs had selected a test agent and the others were in the  process of selecting a test agent. TES provided documentation, such as  memoranda, of its approval for 3 of these 8 programs. For the remaining 5  programs, there was no documentation of the extent to which these test  agents had met the criteria and that TES had approved them. According to  TES officials, they did not have a mechanism in place requiring a  consistent method for documenting their review and approval of  component agents or the extent to which criteria used in reviewing these  agents were met.
In the absence of such a mechanism in fiscal year 2010, TES’s approval of  test agents was not consistently documented. TES and component officials  stated that the approval for the five programs was implicit or provided  verbally without documentation regarding whether the test agent met the  T&E directive requirements. The T&E directive states that the test agent is  to be identified and approved as early as possible in the acquisition  process to, among other things, assist the component program officials in  developing the test and evaluation master plan and review draft  requirements documents to provide feedback regarding the testability of  proposed requirements. TES and component officials stated that they  assumed that test agents were approved using various approaches.  Specifically, of the five programs that had test agents sign the test and  evaluation master plan, one program had documented approval from TES.  For example, Coast Guard and Office of Health Affairs officials stated that  they did not have explicit documentation of TES’s approval of their agents;  however, they believed that TES’s approval was implicit when TES  approved their test and evaluation master plan since the test agent and  TES are both signatories on the plan. CBP and National Protection and  Programs Directorate officials told us that TES provided verbal approval  for their test agents. Since there is no mechanism requiring TES to  document its approval of the agent, and approval was granted verbally,  there is no institutional record for DHS or an independent third party to  validate whether TES followed its criteria when approving these test  agents and whether the test agent was identified and approved before the  test and evaluation master plan and requirements documents were  finalized, as outlined in the T&E directive.
With regard to the three programs in which TES had documented its  approval in memoranda, these memoranda detailed TES’s agreement or  nonagreement with a particular agent and highlighted whether the agent  met the criteria outlined in the T&E directive. For example, TES provided  interim approval to all three of the programs with the conditions that the  programs prove at a later date that the test agents met all the  requirements.  For example:  In April 2010, TES wrote a memo and granted interim approval with  “serious reservations” for 1 year to TSA’s test agent for the Passenger  Screening program. In the memo, TES cited concerns about the  organizational structure and the lack of independence of the test agent  since the test agent was part of the same TSA office responsible for  managing the program.  The memo outlined several steps that TSA  should take, including the implementation of interim measures, such as  new procedures, to ensure the necessary independence critical to  testing and evaluation efforts as required by DHS directives. TES  officials told us that by documenting TES’s interim approval in a memo,  they were able to communicate their concerns about the test agent’s  independence to TSA and DHS decision makers and set forth interim  measures that TSA needed to address regarding their concerns.
In July 2010, TES granted conditional approval to the test agent for the  U.S. Citizenship and Immigration Services’ (USCIS) Transformation  program’s test agent. TES made its approval contingent on the program  developing a plan to ensure that the test agent was familiar with the  component’s business practices. According to TES officials, after  component officials gave a briefing to TES, they determined that the  test agent met the requirements and it was approved.
In January 2011, TES granted conditional approval for the U.S. Secret  Service’s Information Integration and Transformation program to bring  its selected test agent on board.  TES’s final approval will be given after  program officials brief TES on the test agent’s operational testing  approach, which is to demonstrate that the test agent has knowledge of  the product and has the capacity to execute the tests.
TES officials told us that they do not have approval memos for all of the  test agents that have been hired by program offices since the T&E  directive was implemented in May 2009.  Because TES did not consistently  document their approvals of test agents, it is unclear whether TES has  ever disapproved a test agent. TES officials acknowledged that they did  not consistently document that the test agents met T&E requirements and  did not document their approval of test agents. TES officials said that it  would be beneficial to do so to ensure that agents met the criteria required  in the T&E directive. In addition, Standards for Internal Control in the  Federal Government and associated guidance state that agencies should  document key decisions in a way that is complete and accurate, and that  allows decisions to be traced from initiation, through processing, to after  completion. These standards further state that documentation of key  decisions should be readily available for review. Without a mechanism for  documenting its review and approval of test agents for major acquisition  programs, it will be difficult for DHS or an independent third party to  validate TES’s decision-making process to ensure that it is effectively  overseeing component testing.  Moreover, it will be difficult for TES to  provide reasonable assurance that these agents met the criteria outlined in  the T&E directive, such as the requirement that they be independent of the  program being tested.
section_title: TES Did Not Consistently Document the Extent to Which Certain Acquisition Documents Met T&E Criteria
paragraphs: In addition to reviewing and approving test plans, under the T&E directive,  TES is required to review certain component acquisition documents,  including the mission need statements, operational requirements  document, concept of operations, and developmental test reports,  amongst others. These documents, which are required at the need,  Analyze/Select, and Obtain phases of the acquisition process, are to be  reviewed by TES to assist component program managers in identifying  and resolving technical, logistical, and operational issues early in the  acquisition process and to ensure that these documents meet relevant  criteria.
Specifically, as outlined in the T&E directive, TES is to review the mission  need statement to establish awareness of the program and help ensure  that the required standards are developed and that the component has  identified the appropriate resources and support needed to conduct  testing. TES is also to review the operational requirements document,  including the key performance parameters and critical operational issues  that specify the operational effectiveness and operational suitability issues  that the test agent is to examine in order to assess the system’s capability  to perform the mission. Further, TES is to review the concept of  operations, since this document describes how the technology or  equipment will be used in an operating environment. TES is to review the  developmental test reports to maintain knowledge of contractor testing  and to assist in its determination of the program’s readiness to progress to  operational testing. We have previously reported that inadequate attention  to developing requirements results in requirements instability, which can  ultimately cause cost escalation, schedule delays and fewer end items.  Further, we reported that without the required development and review of  key acquisition data, DHS cannot provide reasonable assurance that  programs have mitigated risks to better ensure program outcomes.
TES officials stated that they do not have a mechanism to document or  track those that they did review, what criteria they used when reviewing  these documents, and the extent to which the documents reviewed met  those criteria. For the 11 DHS programs that we reviewed, 8 programs had  component-approved mission need statements; 2 programs, Atlas Tactical  Communications and Transformation, had not yet completed such  statements; and 1 program, the initial SBInet program, had completed a  mission need statement in October 2006 before the T&E directive was  issued and did not develop a separate mission need statement for the  Block 1 increment of the program. Of the 8 programs that had mission  need statements, 6 components told us that they did not have evidence  that TES reviewed the mission need statement in accordance with the  T&E directive. Further, TES could not demonstrate that it had received or  reviewed these documents. Since TES did not have documentation of its  review, it is difficult to determine the extent to which the documents were  reviewed and the extent to which these documents met the review criteria.  TES officials told us that they do not usually provide substantial input into  the mission need statements and that they receive these documents to  establish awareness of a new program. Further, while one TES test area  manager told us that he reviews all developmental test reports, another  test area manger told us that some programs do not routinely send him  developmental test reports.
Also, for example, Secret Service officials said that for the Information  Integration and Transformation program they provided the operational  requirements document, concept of operations, and integrated logistic  support plan to TES. Specifically, the officials said that TES officials were  very helpful in providing input on draft documents and made  improvements to the documents by suggesting, for example, that the tests  be more realistic by including personnel from field offices, headquarters,  and external agencies in the live/production test environment. In contrast,  officials from TSA stated that while they provided their mission need  statement, concept of operations, integrated logistics support plan, and  acquisition program baseline documents for the Advanced Technology 2  (AT-2) program to TES, TES officials did not provide input or comments  on any of those documents.  TES officials told us that the AT-2 program  was initiated and developed some acquisition documentation prior to May  2009 when the T&E directive was issued. Specifically the operational  requirements document was approved and finalized by TSA in June 2008  prior to the T&E directive and provided later to TES in February 2010  when the program was being reviewed. When TES reviewed the  operational requirements document along with other documents such as  the test and evaluation master plan, TES wrote a memo to TSA in March  2010 requesting that detection performance requirements be clarified and  that users concur with the requirements. After several months of  discussion, TSA and TES agreed on an approach which was used as the  basis for initial operational T&E.
Standards for Internal Controls in the Federal Government, as outlined  earlier, state that agencies should document key decisions, and further  that documentation of key decisions should be readily available for  review. TES officials stated that they do not have a mechanism requiring  that they document their review of certain acquisition documentation or  the extent to which the document met the criteria used in reviewing these  documents, and recognized that doing so would be beneficial. Developing  a mechanism for TES to document its review of key acquisition documents  could better position TES to provide reasonable assurance that it is  reviewing key documentation and providing input that is important for  determining the outcome of future testing and evaluation efforts, as  required by the T&E directive. Moreover, such a policy could help to  ensure that an institutional record exists for DHS or an independent third  party to use in determining whether TES is effectively overseeing  component T&E efforts and assisting in managing DHS major acquisition  programs.
section_title: TES Does Not Plan an Independent Assessment of ASP’s Operational Test Results as Required by the T&E Directive
paragraphs: According to the T&E directive, TES is to conduct an independent  assessment of the adequacy of an operational test, provide a concurrence  or nonconcurrence on the test agent’s evaluation of operational suitability  and operational effectiveness, and provide any further independent  analysis it deems necessary for all major DHS acquisition programs. TES is  to document this independent assessment by writing a letter of assessment  within 30 days of receiving the operational test report from the  components’ test agent and provide the letter of assessment to the ARB,  who then uses the assessment in making its determination of whether the  program can proceed to purchase and implementation.
While TES has developed a letter of assessment for the two other  programs undergoing an ARB decision to enter into the production and  deployment phase since the T&E directive was issued in May 2009, TES  officials told us that they do not plan to write such an assessment for the  Advanced Spectroscopic Portal (ASP) program because they are the test  agent for ASP and thus, are not in a position to independently assess the  results of testing that they conducted.
In April 2008, over a year before the T&E directive was issued, senior level  executives from DHS, S&T, CBP, and the Domestic Nuclear Detection  Office (DNDO) signed a memorandum of understanding regarding  arrangements for ASP operational testing. The memo designated Pacific  Northwest National Lab, a U.S. Department of Energy laboratory, as the  test agent. However, the memo also outlined the roles and responsibilities  of TES, many of which reflected the duties of a test agent, such as  developing and approving all operational test plans, responsibility for the  management of testing and field validation, and developing and approving  operational test reports. TES officials told us that they were using Pacific  Northwest National Lab staff to carry out the operational tests, but are  acting, for all intents and purposes, as the test agent for ASP. TES and  DNDO officials told us that this arrangement was made after repeated  testing issues arose with the ASP program.
In September 2008, we reported that ASP Phase 3 testing by DNDO  provided little information about the actual performance capabilities of  ASP and that the resulting test report should not be used in determining  whether ASP was a significant improvement over currently deployed  equipment. Specifically, we found that the ASP Phase 3 test results did  not help determine an ASP’s “true” level of performance because DNDO  did not design the tests to assess ASP performance with a high degree of  statistical confidence.  In response to our report, DHS convened an  independent review team to assist the Secretary in determining whether he  should certify that there will be a significant increase in operational  effectiveness with the procurement of the ASP system. The independent  review team found that the test results and measures of effectiveness were  not properly linked to operational outcomes.
In May 2009, we reported that DHS had increased the rigor of ASP testing  in comparison with previous tests. For example, DNDO mitigated the  potential for bias in performance testing (a concern we raised about prior  testing) by stipulating that there would be no ASP contractor involvement  in test execution. However, the testing still had limitations, such as a  limited set of scenarios used in performance testing to conceal test objects  from detection. Moreover, we also reported that TES was to have the lead  role in the final phase of ASP testing. As of February 2011, TES officials  told us that the final phase of testing, consisting of 21 days of continuous  operation, had not yet been scheduled.
With TES acting as the test agent, it is not in a position to exercise its  responsibilities during the operational testing phase, such as approving the  operational test plan or writing a letter of assessment of the final results of  operational testing.  As it has done for two other recent DHS acquisition  programs, TES was able to confirm through its independent assessment  whether the test agent conducted operational testing as described in the  test and evaluation master plan and operational test plan. For example,  TES outlined concerns in its letter of assessment to the ARB that the AT-2  system did not meet a stated operational requirement key performance  parameter—a throughput measure of bags per hour—for the majority of  the time under test which resulted in a “not effective” determination by  TES.
TES officials recognized that, as the test agent, they are not in a position to  conduct an independent assessment of operational test results and write a  letter of assessment for ASP and that they are the highest level  organization within DHS for both T&E oversight and operational test  expertise. They further stated that the decision to have TES serve as the  test agent was made prior to the issuance of the T&E directive and that it  was too late in the program’s development to go back and select another  agent. Nevertheless, TES officials recognized that this one-time situation  would result in the lack of an independent assessment of ASP test results  and there were no plans to conduct or contract for such an independent  assessment. While we acknowledge that this decision was made prior to  the T&E directive and the requirement that TES write a letter of  assessment of all major acquisition programs, it is nonetheless important  that ASP undergo an independent assessment of its test results since its  operational test plan, which was developed by TES, was not subject to  oversight.  Because ASP has faced testing issues, many of which we have  reported on in past years, it is important that this program undergo  oversight to help avoid similar problems from reoccurring. Without an  independent assessment of ASP’s operational test results, it will be  difficult to ensure that operational testing was properly planned,  conducted, and that the performance results are useful.  In addition,  arranging for an independent assessment of operational tests results could  provide the ARB with critical information on testing and evaluation efforts  to help it determine whether ASP should be approved for purchase and  implementation.
section_title: TES and Component Officials Cited Challenges in Coordinating and Overseeing T&E across DHS; Efforts Are Underway to Address Some Challenges
paragraphs: TES and component officials reported challenges faced in coordinating  and overseeing T&E across DHS components that fell into four primary  categories: (1) ensuring that a program’s operational requirements—the  key requirements that must be met for a program to achieve its intended  goals—can be effectively tested; (2) working with DHS component  program staff that have limited T&E expertise and experience; (3) using  existing T&E directives and guidance to oversee complex information  technology acquisitions; and (4) ensuring that components allow sufficient  time and resources for T&E while remaining within program cost and  schedule estimates. Both TES and DHS, more broadly, have begun  initiatives to address some of these challenges, but it is too early to  determine their effectiveness.
subsections: section_title: Ensuring Operational Requirements Can Be Tested and Are Suitable to Meet Mission Needs
paragraphs: Both TES and component officials stated that one of their challenges is  developing requirements that are testable, consistent, accurate, and  complete. Specifically, six of the nine TES test area managers told us that  working with DHS components to ensure that operational requirements  can be tested and are suitable to meet mission needs is important because  requirements development is one of the biggest challenges facing DHS.  For example, one TES test area manager described the difficulty in  drafting a test and evaluation master plan if operational requirements are  not testable and measurable. Another TES test area manager indicated  that programs’ operational requirements documents often do not contain  user needs or operational requirements for system performance. This  leads to difficulties in testing those requirements later.  Further, six of the  nine TES test area managers cited that some components’ operational  requirements are difficult to test as written, which results in delays in  drafting T&E documents as well as impacting the program cost and  schedule parameters.
Our prior work has found that program performance cannot be accurately  assessed without valid baseline requirements established at the program  start. According to DHS guidance, the baseline requirements must include  a threshold value that is the minimum acceptable value which, in the  user’s judgment, is necessary to satisfy the need. In June 2010, we reported  that if threshold values are not achieved, program performance is  seriously degraded, the program may be too costly, or the program may no  longer be timely. In addition, we reported that inadequate knowledge of  program requirements is a key cause of poor acquisition outcomes, and as  programs move into the produce and deploy phase of the acquisition  process, problems become much more costly to fix. To help remedy these  issues, we have made a number of recommendations to address them.   DHS has generally agreed with these recommendations and, to varying  degrees, has taken actions to address them. For example:  In May 2010, we reported that not all of the SBInet operational  requirements that pertain to Block 1—a surveillance, command,  control, communications, and intelligence system being fielded in two  portions of the international border in Arizona—were achievable,  verifiable, unambiguous, and complete. For example, a November 2007  DHS assessment determined that 19 operational requirements, which  form the basis for the lower-level requirements used to design and  build the system, were not complete, achievable, verifiable, or  affordable. Further, the DHS assessment noted that a requirement that  the system should provide for complete coverage of the border was  determined to be unverifiable and unaffordable because defining what  complete coverage meant was too difficult and ensuring complete  coverage, given the varied and difficult terrain along the border, was  cost prohibitive. To address these issues, we recommended that the  currently defined Block 1 requirements, including key performance  parameters, are independently validated as complete, verifiable, and  affordable and any limitations found in the requirements are  addressed. Furthermore, CBP program officials told us that they  recognized the difficulties they experienced with requirements  development practices with the SBInet program. Within CBP, the  Office of Technology, Innovation, and Acquisition has responsibility for  managing the SBInet program. Office of Technology, Innovation, and  Acquisition officials told us that their office was created to strengthen  expertise in acquisition and program management of SBInet.
In May 2009, we reported that ASP testing uncovered multiple  problems in meeting the requirements for successful integration into  operations at ports of entry. As a result, we recommended that DHS  assess ASPs against the full potential of current equipment and revise  the program schedule to allow time to conduct computer simulations  of ASP’s capabilities and to uncover and resolve problems with ASPs  before full-scale deployment.
We also reported that other TSA technology projects were delayed  because TSA had not consistently communicated clear requirements in  order to test the technologies. We recommended that TSA evaluate  whether current passenger screening procedures should be revised to  require the use of appropriate screening procedures until it is  determined that existing emerging technologies meet their functional  requirements in an operational environment.
In March 2011 testimony, the Under Secretary for S&T stated that S&T had  begun working with the DHS Under Secretary for Management to use their  collective expertise and resources to better address the “front end” of the  acquisition cycle, namely, the translation of mission needs into testable  requirements. Further, in response to this challenge, S&T has reorganized  and established an Acquisition Support and Operations Analysis Group,  which is to provide a full range of coordinated operations analysis,  systems engineering, T&E, and standards development support for DHS  components. In addition, TES’s T&E Council is currently focusing on the  challenges related to requirements development. Specifically, TES test  area managers have presented specific briefings to component officials at  council meetings which provide information on how to better generate  requirements. Further, in response to our previously mentioned report  designating DHS on the high-risk list, DHS developed a strategy to, among  other things, strengthen its requirements development process. DHS’s  January 2011 strategy describes the establishment of a capabilities and  requirements council to evaluate and approve operational requirements  early in the acquisition process. Specifically, the capabilities and  requirements council is to, among other things, reconcile disagreements  across program offices and approve analyses of alternatives and  operational requirement documents. We stated in a March 2011 response  to DHS on its strategy that it was unclear how the introduction of new  governance groups will streamline the process and address previously  identified issues because it appeared that the governance groups are  chaired by the Deputy Secretary and have many of the same participants.   Since the S&T reorganization has only recently taken place and the T&E  Council and the department’s strategy have only recently begun to address  the challenge of requirements generation, it is too soon to determine the  effectiveness of these actions in addressing this challenge.
section_title: T&E Experience and Expertise within DHS Components Varies
paragraphs: TES officials told us that T&E experience and expertise within DHS  components varies, with some components possessing staff with extensive  T&E experience and expertise and others having relatively little. For  example, TES officials noted that the Coast Guard and TSA have T&E  policies and procedures in place, as well as staff with extensive T&E  experience, which limited their dependence on TES for T&E expertise.   Other components in DHS told us they rely more on TES or contractors  for T&E expertise. For the 11 DHS programs we reviewed, officials from  components which do not have many acquisition programs, such as the  Office of Intelligence and Analysis, reported needing more assistance from  TES in identifying and selecting appropriate and qualified test agents, for  example. Conversely, components with more acquisition programs, such  as the Coast Guard, told us that they have well-established test agents and  procedures in place, and require little guidance from TES. For example,  we reported in April 2011 that most Coast Guard major acquisition  programs leverage Navy expertise, in some way, to support a range of  testing, engineering, and other program activities.
Furthermore, CBP recently established a new office whose goal is to  strengthen expertise in acquisition and program management, including  T&E, and ensure that CBP’s technology efforts are focused on its mission  and integrated across the agency.
In response to this challenge, TES has worked with DHS’s Acquisition  Workforce Office to develop T&E certification requirements and training  for components. TES officials told us that they have worked with the  Acquisition Workforce Branch and developed pilot courses on T&E for  component T&E staff, including Fundamentals of Test and Evaluation,  Intermediate Test and Evaluation, and Advanced Test and Evaluation.  In  April 2010, DHS issued an acquisition workforce policy which establishes  the requirements and procedures for certification of DHS T&E managers.  The policy allows T&E managers to be certified at a level that is  commensurate with their education, training, and experience. Component  staff from 6 of the 11 programs we reviewed said they participated in  TES’s certification training program and believed that the training would  assist them in carrying out their T&E responsibilities. In addition, TES is in  the process of hiring four additional staff to assist the test area managers  in their T&E oversight responsibilities and hoped to have the additional  staff hired by the end of fiscal year 2011.
Lack of DHS staff to conduct acquisition oversight, including T&E, is a  departmentwide challenge. In our previous reports, DHS acquisition  oversight officials said that funding and staffing levels have limited the  number of programs they can review. We recommended that DHS  identify and align sufficient management resources to implement oversight  reviews in a timely manner. DHS generally concurred with the  recommendation and, as of January 2011, has reported taking action to  address it by identifying needed capabilities and hiring staff to fill  identified gaps.
Further, to address this challenge, in 2009 and 2010, T&E Council  representatives from the Acquisition Workforce Branch made  presentations at council meetings to update members on the status of  various acquisition workforce issues, including T&E certification.  For  example, presenters asked T&E Council members to inform their  respective components about new T&E certification courses and to  provide information on how to sign up for the courses.  In 2010, the  Acquisition Workforce Policy was implemented by DHS, which allowed  the department to begin to certify T&E acquisition personnel. While DHS  has undertaken efforts to help address these challenges, it is too soon to  evaluate the impact that these efforts will have in addressing them.
section_title: Using Existing T&E Directives and Guidance to Oversee Information Technology Acquisitions
paragraphs: Effectively managing IT acquisitions is a governmentwide challenge. TES  and component officials we interviewed told us that T&E guidance, such  as specific guidance for integrating developmental testing and operational  testing, may not be sufficient for the acquisition of complex IT systems.  Specifically, component officials stated that the assessment of risks and  environmental factors are different for IT programs than other acquisitions  and that conducting testing in an operational environment may not be  necessary for IT programs because the operational environment is no  different than the test environment.  In addition, four of the nine test area  managers told us that aspects of the existing T&E guidance may not  directly apply to IT acquisitions.
The department is in the process of making modifications to its  acquisitions process to better accommodate information technology  acquisitions. According to the previously mentioned January 2011 strategy  submitted to GAO, DHS is piloting a new model for IT acquisitions. This  model, which is to be consistent with the department’s overall acquisition  governance process, is to have many of the steps in the modified process  that are similar or the same as what currently exists but time frames for  different types of acquisitions would be instituted.  For example,  acquisition programs designated as IT programs may go through a more  streamlined acquisition process that may better fit the rapidly changing IT  environment, and the ARB would have the option to delegate oversight  responsibilities to an executive steering committee. In other cases, TES  and component officials are investigating the possibility of conducting  integrated testing—the combination of developmental and operational  testing—for some programs although this process may take longer to plan  and pose greater risks because testing is being done simultaneously.  Further, the T&E Best Practices Integrated Working Group, a subgroup of  the T&E Council, including TES, Acquisition Program Management  Division, and Office of Chief Information Officer officials, was working to  identify and promote T&E best practices for IT system acquisition. This  group drafted an operational test agent risk assessment process to validate  the streamlining process approach while adhering to acquisition and T&E  policy and directives, and as of March 2011, one component, USCIS, has  made use of this process. Additionally, three other programs are  investigating the possible use of this process and the possibility of  tailoring or eliminating T&E deliverables or operational T&E requirements  for IT programs, with the approval of TES. The group has identified three  IT acquisition programs to serve as a pilot for this effort.
As DHS considers modifications to its T&E process for IT programs, it also  must consider the effect such a change could have on determining a  system's technical performance and evaluating the system's operational  effectiveness and suitability. For example, we have previously reported on  testing problems with SBInet, a CBP program designated as an IT  program.  We found that SBInet testing was not performed in a manner  that would adequately ensure that the system would perform as intended.  Among the factors contributing to these problems was insufficient time for  reviewing and approving test documentation, which in part, led to test  plans and test cases not being well-defined. As a result, we recommended  that test schedules, plans, cases, and procedures are adequately reviewed  and approved consistent with the revised test and evaluation master plan.  Since the efforts DHS is taking to address this challenge have only recently  been initiated, it is too early to tell what impact they will have on the  overall challenges of T&E for IT programs.
section_title: Allowing Appropriate Time for T&E within Program Cost and Schedule
paragraphs: Both TES and component officials stated that balancing the need to  conduct adequate T&E within the confines of a program’s costs and  schedule is a recurring challenge, and a challenge that is difficult to solve.  We have previously reported on the challenges associated with balancing  the need to conduct testing within program cost and schedules. Our past  review of the Department of Defense’s  (DOD) Director of Operational  Test and Evaluation found that while the acquisition community has three  central objectives—performance, cost, and schedule—the Director of  Operational Test and Evaluation has but one--operational testing of  performance. We reported that these distinct priorities can lead to  testing disputes. We reported that these disputes encompassed issues such as (1) how many and what types of test to conduct; (2) when testing  should occur; (3) what data to collect, how to collect them, and how b to analyze them; and (4) what conclusions were supportable, given th e  analysis and limitations of the test program. The foundation of most of  these disputes laid in different notions of the costs and benefits of testing  and the levels of risk that were acceptable when making full-rate  production decisions. The DOD Director of Operational Test and  Evaluation consistently urged more testing (and consequently more time,  resources, and cost) to reduce the level of risk and number of unknowns  before the decision to proceed to full-rate production, while the services  consistently sought less testing and accepted more risk when making  production decisions. These divergent dispositions frequently led to  est  healthy debates about the optimal test program, and in a small number of  cases, the differences led to contentious working relations.
TES and DHS component officials expressed views similar to those  expressed in our past work at DOD. Of the nine TES test area managers  we talked with, four told us that allowing appropriate time and resources  for T&E within program cost and schedule is a challenge. According to the  test area manager’s, component program management officials often do  not incorporate sufficient time within their schedule for T&E or reduce the  time allowed for T&E to save time and money. In one test area managers’  view, doing so can reduce the effectiveness of testing or negatively impact  the results of the tests. However, TSA officials told us that TES wanted to  insert new test requirements for the AT-2 program—including the  involvement of more TSA staff in the tests—after the program schedule  was established and it was difficult to accommodate the changes and  resulted in some delays. TES officials told us that these test requirements  were in lieu of other planned field testing, which were not consistent with  the program’s concept of operations and that TSA officials agreed with the  new test requirements. According to TES and component officials we  spoke with, both the program officials and TES understand the views and  perspectives of one another and recognize that a balance must be struck  between effective T&E and managing programs within cost and schedule.  As a result, TES is working with program officials through the T&E  Council or T&E working groups to discuss these issues early in the  acquisition cycle (before it is too late), particularly while developing the  test and evaluation master plan, which outlines the time allowed for  testing and evaluation.
section_title: Conclusions
paragraphs: Timely and accurate information resulting from T&E of major acquisitions  early in the acquisition process can provide valuable information to DHS’s  senior level managers to make informed decisions about the development,  procurement, deployment, and operation of DHS’s multibillion dollar  portfolio of systems and services. Improving the oversight of component  T&E activities is but one part of the significant challenges DHS faces in  managing its acquisitions. Components themselves are ultimately  responsible for the management and implementation of their programs  and DHS senior level officials are responsible for making key acquisitions  decisions which lead to production and deployment. TES helps support  acquisition decisions by providing oversight over major acquisitions’ T&E,  which can help reduce, but not eliminate, the risk that new systems will  not be operationally effective and suitable.
Since the Homeland Security Act creating DHS was enacted in 2002, S&T  has had the responsibility for overseeing T&E activities across the  department. However, S&T did not have staff or the acquisition and T&E  directives in place to conduct such oversight across DHS components  until May 2009 when DHS issued its T&E directive. Since then, TES has  implemented some of the requirements and overseen T&E of major  acquisitions we reviewed, as well as provided independent assessments of  operational test results to the ARB. However, TES has not consistently  documented its compliance with the directives. Documenting that TES is  fulfilling the requirements within DHS acquisition and T&E directives and  the extent to which the criteria it is using to review and approve these  documents are met, including approving operational test agents and  reviewing key acquisition documentation, would assist TES in  demonstrating that it is conducting T&E oversight and meeting  requirements in these directives. Furthermore, without an independent  assessment of operational test results for the Advance Spectroscopic  Portal program, a key T&E oversight requirement in the T&E directive, the  ARB will lack T&E oversight and input it needs to determine whether ASP  is ready to progress toward production and deployment. This is especially  important, given that program’s troubled history, which we have  highlighted in a series of prior reports.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that testing and evaluation requirements are met, we  recommend that the Secretary of Homeland Security direct the Under  Secretary for Science & Technology to take the following two actions:    Develop a mechanism to ensure that TES documents its approval of  operational test agents and the extent that the test agents meet the  requirements in the T&E directive, and criteria that TES use in  reviewing these test agents for major acquisition programs.
Develop a mechanism to ensure that TES documents its required  review of component acquisition documents, including the mission  need statements, concept of operations, operational requirements  documents, developmental test reports, test plans, and other  documentation required by the T&E directive, the extent that these  documents meet the requirements in the T&E directive, and criteria  that TES uses in reviewing these documents.
To ensure that the ARB is provided with an independent assessment of the  operational test results of the Advanced Spectroscopic Portal program to  help determine whether the program should be approved for purchase and  implementation, we recommend that the Secretary of Homeland Security  take the following action:     Arrange for an independent assessment, as required by the T&E  directive, of ASP’s operational test results, to include an assessment of  the adequacy of the operational test and a concurrence or  nonconcurrence on the operational test agent’s evaluation of  operational suitability and operational effectiveness.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from DHS on June  10, 2011, which are reproduced in full in appendix III. DHS concurred with  all three of our recommendations.
DHS concurred with our first recommendation (1) that S&T develop a  mechanism to ensure that TES documents its approval of operational test  agents, (2) the extent that the test agents meet the requirements in the  T&E directive, and (3) the criteria that TES uses in reviewing these test  agents for major acquisition programs. Specifically, DHS stated that the  Director of TES issued a memorandum to test area managers and TES  staff regarding the operational test agent approval process which  describes the responsibilities, considerations for selection, and the  process necessary to select an operational test agent. In addition, DHS  stated that TES is drafting memos approving operational test agents using  the new test agent approval process.
DHS also concurred with our second recommendation that S&T develop a  mechanism to ensure that TES documents (1) its required review of  component acquisition documents required by the T&E directive, (2) the  extent that these documents meet the requirements in the T&E directive,  and (3) the criteria that TES uses in reviewing these documents. DHS  stated that the Director of TES issued a memorandum to test area  managers and TES staff detailing the role of TES in the document review  process and the process that TES staff should follow for submitting their  comments to these documents.
Finally, DHS concurred with our third recommendation that S&T arrange  for an independent assessment of ASP’s operational test results. DHS  stated that the ASP program is under review and does not have an  operational test scheduled. However, TES is investigating the option of  using a separate test agent to conduct operational testing of ASP, which  would allow TES to perform the independent assessment and fulfill its  independent oversight role as outlined in DHS policy. Such actions, if  taken, will fulfill the intent of this recommendation. DHS also provided  technical comments on the report, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to interested  congressional committees and the Secretary of Homeland Security. The  report will be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staff have questions regarding this report, please contact me  at (202) 512-9627 or at maurerd@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix IV.
section_title: Appendix I: Descriptions of Selected Major Acquisition Programs
paragraphs: An effort to develop nd deploy technologie to llow Custom nd Border  Protection to detect ncler or rdiologicl mteri from conveynce, such  as trck, entering the United Ste t lnd nd  port of entry.
U.S. Immigrtion  nd Custom  Enforcement  (ICE)
An effort to modernize ICE’scticl commniction tem nd equipment,  tht Ice gent nd officer use to support mission-criticl commniction from otdted log tem to modern nd ndrdized digittem. Project  25 pgrde will modernize tcticl commniction nd deploy ite  infrastrctre nd end-user subscrier rdio. Interoperable Rpid Deployment  Stem (IRDS) will otfit ICE with trportable commniction tem to  support rpid deployment requirement fro rotine, emergency nd disaster  repone, nd pecil opertion. The progrm i divided into ix egment,  inclding: (1)  P25 pgrde for the Atlnt Region, (2)  P25 pgrde for the Boton Region,  ()  P25 pgrde for the Denver Region,  (4)  P25 pgrde for the centrl hub infrastrctre,  (5)  IRDS moile rdio commniction kit thsupport disaster nd emergency   repone opertion, nd  (6)  IRDS Moile Commniction Stem (MCS) moile commniction   vehicle thsupport disaster nd emergency repone opertion. In Mrch 2011, the Component Acquition Exective determined tht the  TACCOM progrm wold e conolidted with other ICE infrastrctre progr  nd tht the progrm wold e required to submit quition docmenttion to  the ARB prior to Aust 2011.
Segment 1:  Prodce/ Deploy/ Support Segment 2-6:  In the  process of  eing pdted.
A ntionwide, interoperting network of detector/identifier tht i to provide  autonomous ir-sampling ly of the environment for iologicgent of  concern. The tem i to enable detection, identifiction, nd reporting of  recognized orgni within  6-hor period. (4)  Obsolete Component Moderniztion (OCM) – Replce obsolete   component nd substem;  (5)  Ship Helicopter Secre Trvere & Stem (SHSTS) – Provide the   ability to automticlly ecre the ircrft to the flight deck nd   trvere it into the hnger; nd,  (6)  Atomtic Flight Control Stem (AFCS/Avionic) Modernize digitl    Common Avionic Architectre Stem (CAAS) common with the    H-60T pgrde nd digitl Atomtic Flight Control Stem.
An effort to modernize Secret Service’s IT infrastrctre, commniction  tem, ppliction, nd process. The progrm i divided into for dicrete  egment:  (1)  Enabling Cabilitie: IT Infrastrctre Moderniztion/Cyer Secrity/  Dabase Architectre; (2)  Commniction Cabilitie; ()  Control Cabilitie; nd, (4)  Mission Support Cabilitie.  In Feuary 2011, the ARB grnted quition deciion event 2A nd  quition deciion event 2B deciion for the Enabling Cabilitie egment.  The remining three egment remined in the Anlyze/Select phase.
An integrted tem of intrusion detection, lyticl, intrusion prevention, nd  informtion-ring cabilitie thre to used to defend the federl civilin  government’s informtion technology infrastrctre from cyer thre. Inclde  the hrdwre, oftwre, supporting process, trining, nd ervice thre to  e developed nd quired to support the mission. The inititem, known as  Eintein, was renmed as Block 1.0 nd incldeabilitie such as  centrlized d torge.  (1)  Block 2.0 i to dd n Intrusion Detection Stem (IDS) which i to         assss network trffic for the preence of mlicious ctivity; (2)  Block 2.1 i to provide  Secrity Incident nd Event Mgement         which i to enable d ggregtion, correltion, nd visualiztion.  ()  Block .0 i to provide n intrusion prevention cability.
A joint inititive etween Intelligence nd Anly (I&A) nd the Office of the  Chief Informtion Officer which i to ring  nified, enterpripproch to the  mgement of ll classified informtion technology infrastrctre inclding: (1)  Homelnd Secre D Network (HSDN) for ecret level commniction   infrastrctre; (2)  Homelnd Top Secret Network (HTSN) for top ecret commniction   infrastrctre; nd ()  Homelnd Secre Commniction (HSC) for classified voice nd video   teleconference cabilitie.
A next genertion of x-ry technology tht i to complement the trditionl x-ry  technology nd provide new technicl cabilitie, such as automted detection  lgorithm, thret imge projection, lternte viewing tion, bulk exploive  lgorithm, nd expnded thret lit tht incorporte emerging thre to  vition ecrity. A tem which i to provide Trporttion Secrity Officerability to creen passenger' crry-on baggge irporttionwide.
A progrm which i to deliver surveillnce nd deciion-support technologie  tht crete  virtual fence nd ituationreness long the U.S. order with  Mexico nd C. The firSBInet deployment of the Block I tem took  plce in the Ton, Arizon tion. The econd deployment of the Block I  tem took plce in the Ajo, Arizon tion.  In Juary 2011, the SBInet  progrm ended as originlly conceived; however, limited deployment of  technology, inclding 15 enor tower nd 10 commniction tower, remined  deployed nd opertionl in Arizon. The T&E result on thee tower were to  e reported ometime in April 2011.
TASC i to develop nd field n integrted finncil mgement, asset  mgement, nd procrement mgement tem oltion. The progrm i  to usndrd business process nd  ingle line of cconting complint  with the common governmentwide cconting classifiction trctre. The TASC  Exective Steering Committee determined tht the Federl Emergency  Mgement Agency will e the firt DHS component to migrte to TASC.
U.S. Citizenhip  nd Immigrtion  Service  (USCIS)
An effort to move immigrtion ervice from per-based model to n  electronic environment. The progrm i to deliver  implified, We-based  tem for enefit eeker to submit nd trck their ppliction. The new,  ccont-based tem i to provide customer with improved ervice.
section_title: Appendix II: DHS’s Fiscal Year 2010 Major Acquisition Programs
paragraphs: In fiscal year 2010, there were 86 acquisition programs on the Acquisition  Program Management Division’s oversight list, which included the  acquisition level and designation as an information technology acquisition.  Table 2 lists information on these 86 acquisition programs, and in addition,  includes information on the acquisition phase for each program as of April  2011 and whether the program was subject to the test and evaluation  (T&E) directive. For example, some programs, such as Customs and  Border Protection’s acquisition of Border Patrol Facilities would not  involve any T&E activities and therefore would not be subject to the  requirements in the T&E directive or DHS Science and Technology  Directorate’s Test and Evaluation and Standards office (TES) oversight.
section_title: Appendix III: Comments from the Department of Homeland Security
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Christopher Currie (Assistant  Director), Nancy Kawahara, Bintou Njie, Melissa Bogar, Jessica Drucker,  Caitlin White, Richard Hung, Michele Fejfar, Labony Chakraborty, Tracey  King, Paula Moore, Dan Gordon, Michele Mackin, Molly Traci, and Sean  Seales made significant contributions to this report.