DHS Research and Development: Science and Technology Directorate's Test and Evaluation and Reorganization Efforts
Published: 2011-11-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: This testimony discusses our prior work examining the Department of Homeland Security's (DHS) Science and Technology Directorate (S&T) and Research and Development (R&D) efforts. The Homeland Security Act of 2002 created DHS and, within it, established S&T with the responsibility for conducting national research, development, test and evaluation (T&E) of technology and systems for, among other things, detecting, preventing, protecting against, and responding to terrorist attacks. Since its creation in 2003, DHS, through both S&T and its components, has spent billions of dollars researching and developing technologies used to support a wide range of missions including securing the border, detecting nuclear devices, and screening airline passengers and baggage for explosives, among others. S&T has a wide-ranging mission, which includes conducting basic and applied research of technologies, and overseeing the testing and evaluation of component acquisitions and technologies to ensure that they meet DHS acquisition requirements before implementation in the field. In recent years, we have reported that DHS has experienced challenges in managing its multibillion-dollar technology development and acquisition efforts, including implementing technologies that did not meet intended requirements and were not appropriately tested and evaluated. These problems highlight the important role that S&T plays in overseeing DHS testing and evaluation. S&T has reorganized to better achieve its goals and provide better assistance to DHS components in developing technologies. In addition to the challenge of implementing its varied mission, S&T is also managing a decline in available R&D resources. S&T's fiscal year 2011 appropriation decreased 20 percent from fiscal year 2010 and, while its fiscal year 2012 appropriation has not yet been enacted, both the House and Senate marks for the agency are lower than what was appropriated in fiscal year 2011. As a result, S&T has had to adjust resources and re-prioritize its efforts. In the past, we have reported on issues related to the transformation and reorganization of R&D efforts in the federal government, particularly related to shifting of priorities and managing a reduction in resources. In addition, we identified DHS R&D as an area for potential costs savings in our March 2011 report regarding opportunities to reduce potential duplication in government programs, save tax dollars, and enhance revenue. Specifically, we reported that DHS could take further actions to improve its management of R&D and reduce costs by ensuring that testing efforts are completed before making acquisition decisions and cost-benefit analyses are conducted to reduce R&D inefficiencies and costs. The testimony today focuses on the key findings from our prior work related to S&T's test and evaluation efforts, S&T's recent reorganization efforts, and key findings from our past work related to federal R&D. Specifically, this statement will address: (1) the extent to which S&T oversees T&E of major DHS acquisitions and what challenges, if any, S&T officials report facing in overseeing T&E across DHS; and (2) S&T's recent reorganization efforts and how key findings from our prior work on R&D in the federal government can inform how S&T moves forward. This statement is based on reports and testimonies we issued from March 1995 to July 2011 related to DHS's efforts to manage, test, and deploy various technology programs; transformation of federal R&D; and selected updates conducted from July 2011 to the present related to S&T's reorganization efforts..
section_title: What GAO Found
paragraphs: In June 2011, we reported that S&T met some of its oversight requirements for T&E of acquisition programs we reviewed, but additional steps were needed to ensure that all requirements were met. Specifically, since DHS issued the T&E directive in May 2009, S&T reviewed or approved T&E documents and plans for programs undergoing testing, and conducted independent assessments for the programs that completed operational testing during this time period. S&T officials told us that they also provided input and reviewed other T&E documentation, such as components' documents describing the programs' performance requirements, as required by the T&E directive. DHS senior level officials considered S&T's T&E assessments and input in deciding whether programs were ready to proceed to the next acquisition phase. However, S&T did not consistently document its review and approval of components' test agents--a government entity or independent contractor carrying out independent operational testing for a major acquisition--or document its review of other component acquisition documents, such as those establishing programs' operational requirements, as required by the T&E directive. We also reported in June 2011 that S&T and DHS component officials stated that they face challenges in overseeing T&E across DHS components which fell into 4 categories: (1) ensuring that a program's operational requirements--the key performance requirements that must be met for a program to achieve its intended goals--can be effectively tested; (2) working with DHS component program staff who have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time for T&E while remaining within program cost and schedule estimates. Since 2009, S&T has undertaken a series of efforts related to its organizational structure. S&T underwent a new strategic planning process, developed new strategic goals, and conducted a reorganization intended to better achieve its strategic goals. These efforts were implemented after a 2009 National Academy of Public Administration study found that S&T's organizational structure posed communication challenges across the agency and that the agency lacked a cohesive strategic plan and mechanisms to assess performance in a systematic way, among other things. In August 2010, S&T reorganized to align its structure with its top strategic goals, allow for easier interaction among senior leadership, and reduce the number of personnel directly reporting to the Under Secretary of S&T.

section_title: Letter
paragraphs: I am pleased to be here today to discuss our prior work examining the  Department of Homeland Security’s (DHS) Science and Technology  Directorate (S&T) and Research and Development (R&D) efforts. The  Homeland Security Act of 2002 created DHS and, within it, established  S&T with the responsibility for conducting national research,  development, test and evaluation (T&E) of technology and systems for,  among other things, detecting, preventing, protecting against, and  responding to terrorist attacks. Since its creation in 2003, DHS, through  both S&T and its components, has spent billions of dollars researching  and developing technologies used to support a wide range of missions  including securing the border, detecting nuclear devices, and screening  airline passengers and baggage for explosives, among others. S&T has a  wide-ranging mission, which includes conducting basic and applied  research of technologies, and overseeing the testing and evaluation of  component acquisitions and technologies to ensure that they meet DHS  acquisition requirements before implementation in the field. In recent  years, we have reported that DHS has experienced challenges in  managing its multibillion-dollar technology development and acquisition  efforts, including implementing technologies that did not meet intended  requirements and were not appropriately tested and evaluated. These  problems highlight the important role that S&T plays in overseeing DHS  testing and evaluation.
S&T has reorganized to better achieve its goals and provide better  assistance to DHS components in developing technologies. In addition to  the challenge of implementing its varied mission, S&T is also managing a  decline in available R&D resources. S&T’s fiscal year 2011 appropriation  decreased 20 percent from fiscal year 2010 and, while its fiscal year 2012  appropriation has not yet been enacted, both the House and Senate marks  for the agency are lower than what was appropriated in fiscal year 2011.  As a result, S&T has had to adjust resources and re-prioritize its efforts. In  the past, we have reported on issues related to the transformation and  reorganization of R&D efforts in the federal government, particularly related  to shifting of priorities and managing a reduction in resources. In addition,  we identified DHS R&D as an area for potential costs savings in our March  2011 report regarding opportunities to reduce potential duplication in  government programs, save tax dollars, and enhance revenue.  Specifically, we reported that DHS could take further actions to improve its  management of R&D and reduce costs by ensuring that testing efforts are  completed before making acquisition decisions and cost-benefit analyses  are conducted to reduce R&D inefficiencies and costs.
My testimony today focuses on the key findings from our prior work  related to S&T’s test and evaluation efforts, S&T’s recent reorganization  efforts, and key findings from our past work related to federal R&D.  Specifically, this statement will address:  the extent to which S&T oversees T&E of major DHS acquisitions and  what challenges, if any, S&T officials report facing in overseeing T&E  across DHS; and
S&T’s recent reorganization efforts and how key findings from our  prior work on R&D in the federal government can inform how S&T  moves forward.
This statement is based on reports and testimonies we issued from March  1995 to July 2011 related to DHS’s efforts to manage, test, and deploy  various technology programs; transformation of federal R&D; and  selected updates conducted from July 2011 to the present related to  S&T’s reorganization efforts. For the updates, we reviewed recent S&T  testimonies and documentation related to the reorganization as well as  information on annual S&T appropriations and budget requests from fiscal  years 2009 to 2012. For our past work, we reviewed DHS directives and  testing plans, interviewed DHS, Department of Energy, Department of  Defense, Environmental Protection Agency, and other agency officials,  reviewed documentation from these agencies, visited laboratory facilities,  and examined agency databases, among other things. We conducted this  work in accordance with generally accepted government auditing  standards. More detailed information on the scope and methodology from  our previous work can be found within each specific report.
subsections: section_title: S&T Could Take Additional Steps to Ensure that DHS T&E Requirements Are Met; Officials Cited Challenges to Overseeing T&E across DHS
subsections: section_title: S&T Oversight of DHS Testing and Evaluation
paragraphs: In June 2011, we reported that S&T met some of its oversight  requirements for T&E of acquisition programs we reviewed, but additional  steps were needed to ensure that all requirements were met.Specifically, since DHS issued the T&E directive in May 2009, S&T  reviewed or approved T&E documents and plans for programs  undergoing testing, and conducted independent assessments for the  programs that completed operational testing during this time period. S&T  officials told us that they also provided input and reviewed other T&E  documentation, such as components’ documents describing the  programs’ performance requirements, as required by the T&E directive.  DHS senior level officials considered S&T’s T&E assessments and input  in deciding whether programs were ready to proceed to the next  acquisition phase. However, S&T did not consistently document its review  and approval of components’ test agents—a government entity or  independent contractor carrying out independent operational testing for a  major acquisition—or document its review of other component acquisition  documents, such as those establishing programs’ operational  requirements, as required by the T&E directive. For example, 8 of the 11  acquisition programs we reviewed had hired test agents, but  documentation of S&T approval of these agents existed for only 3 of  these 8 programs. We reported that approving test agents is important to  ensure that they are independent of the program and that they meet  requirements of the T&E directive.
S&T officials agreed that they did not have a mechanism in place  requiring a consistent method for documenting their review or approval  and the extent to which the review or approval criteria were met. We  reported that without mechanisms in place for documenting its review or  approval of acquisition documents and T&E requirements, such as  approving test agents, it is difficult for DHS or a third party to review and  validate S&T’s decision-making process and ensure that it is overseeing  components’ T&E efforts in accordance with acquisition and T&E  directives and internal control standards for the federal government. As a  result, we recommended that S&T develop a mechanism to document  both its approval of operational test agents and component acquisitions  documentation to ensure that these meet the requirements of the DHS  T&E directive. S&T concurred and reported that the agency has since  developed internal procedures to ensure that the approval of test agents  and component acquisition documents are documented.
section_title: Challenges in Coordinating and Overseeing T&E across DHS
paragraphs: We also reported in June 2011 that S&T and DHS component officials  stated that they face challenges in overseeing T&E across DHS  components which fell into 4 categories: (1) ensuring that a program’s  operational requirements—the key performance requirements that must  be met for a program to achieve its intended goals—can be effectively  tested; (2) working with DHS component program staff who have limited  T&E expertise and experience; (3) using existing T&E directives and  guidance to oversee complex information technology acquisitions; and (4)  ensuring that components allow sufficient time for T&E while remaining  within program cost and schedule estimates.
Both S&T and DHS, more broadly, have begun initiatives to address  some of these challenges, such as establishing a T&E council to  disseminate best practices to component program managers, and  developing specific guidance for testing and evaluating information  technology acquisitions. In addition, as part of S&T’s recent  reorganization, the agency has developed a new division specifically  geared toward assisting components in developing requirements that can  be tested, among other things. However, since these efforts have only  recently been initiated to address these DHS-wide challenges, it is too  soon to determine their effectiveness.
section_title: S&T Recently Reorganized and Our Prior R&D Work Could Inform How S&T Moves Forward
paragraphs: Since 2009, S&T has undertaken a series of efforts related to its  organizational structure. S&T underwent a new strategic planning  process, developed new strategic goals, and conducted a reorganization  intended to better achieve its strategic goals. These efforts were  implemented after a 2009 National Academy of Public Administration  study found that S&T’s organizational structure posed communication  challenges across the agency and that the agency lacked a cohesive  strategic plan and mechanisms to assess performance in a systematic  way, among other things. In August 2010, S&T reorganized to align its  structure with its top strategic goals, allow for easier interaction among  senior leadership, and reduce the number of personnel directly reporting  to the Under Secretary of S&T. Additionally, after the Under Secretary  was confirmed in November 2009, S&T instituted a new strategic  planning process which helped inform the development of new strategic  goals. The new strategic goals announced in August 2010 include:  rapidly developing and delivering knowledge, analyses, and  innovative solutions that advance the mission of DHS;  leveraging its expertise to assist DHS components’ efforts to establish  operational requirements, and select and acquire needed  technologies;  strengthening the Homeland Security Enterprise and First  Responders’ capabilities to protect the homeland and respond to  disasters;  conducting, catalyzing, and surveying scientific discoveries and  inventions relevant to existing and emerging homeland security  challenges; and  fostering a culture of innovation and learning in S&T and across DHS  that addresses mission needs with scientific, analytic, and technical  rigor.
According to S&T, the agency has developed a draft strategic plan that  provides its overall approach to meeting these strategic goals, which is  currently in the process of being finalized.
Moreover, according to testimony by the Undersecretary of S&T in March  2011, to ensure that individual R&D projects are meeting their goals, S&T  has committed to an annual review of its portfolio of basic and applied  R&D and all proposed “new start” projects. According to S&T, the review  process uses metrics determined by S&T, with input from DHS  components, that are aligned with DHS priorities. These metrics consider:  the impact on the customer’s mission;  the ability to transition these products to the field; whether the investment positions S&T for the future; whether the projects are aligned with customer requirements; whether S&T has the appropriate level of customer interaction; and whether S&T is sufficiently innovative in the way it is approaching its  challenges.
We are currently reviewing DHS and S&T’s processes for prioritizing,  coordinating, and measuring the results of its R&D efforts for the Senate  Committee on Homeland Security and Governmental Affairs and we will  report on this issue next year.
Our prior work related to R&D at other federal agencies could provide  insight for S&T as it moves forward with new structures and processes  operating within potential fiscal constraints. During the 1990s, we issued  a series of reports on federal efforts to restructure R&D in the wake of  changing priorities and efforts to balance the federal budget. More  recently, we have issued reports on R&D issues at the Department of  Defense (DOD), Department of Energy (DOE), the Environmental  Protection Agency (EPA), and DHS. Although the specific  recommendations and issues vary from department to department, there  are key findings across this body of work that could potentially help inform  S&T’s efforts to meet DHS’s R&D needs, as well as Congressional  oversight of these activities. Since our assessment of R&D efforts at DHS  is currently under way, we have not determined the extent to which these  key findings from our prior work are applicable to DHS’s R&D efforts or  the extent to which DHS already has similar efforts under way. However,  our prior work could provide valuable insights into how DHS could  leverage the private sector to help conduct R&D, restructure R&D efforts  in response to fiscal constraints, and develop comprehensive strategies  to mitigate the risk of duplication and overlap. For example:
We reported on federal agencies that have restructured their research  and development efforts in response to fiscal constraints. For  example, in January 1998, we reported on efforts by federal agencies,  such as DOD, the DOE National Laboratories, and NASA, to  streamline their R&D activities and infrastructure. We reported that  restructuring research, development, testing and evaluation to meet  current and future needs required interagency agreements and cross- agency efforts, in addition to ongoing individual efforts. Additionally,  we reported on five elements that were useful in the successful  restructuring of R&D in corporate and foreign government  organizations. For example, we found that successful restructuring of  R&D activities included having a core mission that supports overall  goals and strategies, clear definitions of those responsible for  supporting that mission, and accurate data on total costs of the  organization’s activities.
In addition, we have reported that comprehensive strategies mitigate  risk of duplication and overlap. For example, we reported in March  2011 that DOD did not have a comprehensive approach to manage  and oversee the breadth of its activities for developing new  capabilities in response to urgent warfighter needs, including entities  engaged in experimentation and rapid prototyping to accelerate the  transition of technologies to the warfighter, and lacked visibility over  the full range of its efforts.issue guidance that defined roles, responsibilities, and authorities  across the department to lead its efforts. DOD agreed with this  recommendation.
As a result, we recommended that DOD
Within DHS itself, we reported in May 2004 that DHS did not have a  strategic plan to guide its R&D efforts. We recommended that DHS  complete a strategic R&D plan and ensure that the plan was  integrated with homeland security R&D conducted by other federal  agencies. We also recommended that DHS develop criteria for  distributing annual funding and for making long-term investments in  laboratory capabilities, as well as develop guidelines that detailed how  DOE’s laboratories would compete for funding with private sector and  academic entities. DHS agreed with our recommendations. While S&T  developed a 5-year R&D plan in 2008 to guide its efforts and is  currently finalizing a new strategic plan to align its own R&D  investments and goals, DHS has not yet completed a strategic plan to  align all R&D efforts across the department, as we previously  recommended.
Our work on DOE National Laboratories provides additional insights  related to oversight of R&D efforts that could be useful for DHS S&T.  In 1995, we reported that DOE’s national laboratories did not have  clearly defined missions focused on accomplishing DOE’s changing  objectives and national priorities. DOE, at that time, managed the  national laboratories on a program by program basis which inhibited  cooperation across programs and hindered DOE’s ability to use the  laboratories to meet departmental missions. We recommended,  among other things, that DOE develop a strategy that maximized the  laboratories’ resources. In responding, DOE said that it had  undertaken a new strategic planning process which resulted in a  strategic plan. Though DOE developed a strategic plan intended to  integrate its missions and programs, in 1998 we reported that the  laboratories did not function as an integrated national research and  development system and recommended that DOE develop a  comprehensive strategy to be used to assess success in meeting  objectives, monitor progress, and report on that progress. DOE  acknowledged that it needed to better focus the laboratories’ missions  and tie them to the annual budget process, but that it would take time  to accomplish.
More recently, we reported in June 2009 that DOE could not  determine the effectiveness of its laboratories' technology transfer  efforts because it has not yet defined its overarching strategic goals  for technology transfer and lacks reliable performance data.  Instead, individual DOE programs such as the National Nuclear  Security Administration and DOE's Office of Science articulated their  own goals for technology transfer at the national laboratories. We  recommended, among other things, that DOE articulate department  wide priorities and develop clear goals, objectives, and performance  measures. DOE generally agreed with our findings.
Lastly, our work on Environmental Protection Agency (EPA)  laboratory facilities also offers insights into the importance of planning  and coordination in managing R&D. Specifically, we reported in July  2011 that EPA has yet to fully address the findings of numerous past  studies that have examined EPA’s science activities. These past  evaluations noted the need for EPA to improve long-term planning,  priority setting, and coordination of laboratory activities, establish  leadership for agency wide scientific oversight and decision making,  and better manage the laboratories’ workforce and infrastructure.  We  recommended, among other things, that EPA develop a coordinated  planning process for its scientific activities and appoint a top-level  official with authority over all the laboratories, improve physical and  real property planning decisions, and develop a workforce planning  process for all laboratories that reflects current and future needs of  laboratory facilities. EPA generally agreed with our findings and  recommendations.
Chairman Lungren, Ranking Member Clarke, and Members of the  Subcommittee, this concludes my prepared statement. I would be  pleased to respond to any questions that you may have.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For questions about this statement, please contact David C. Maurer at  (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals making key contributions to this statement  include Chris Currie, Assistant Director; Emily Gunn and Margaret  McKenna. Key contributors for the previous work that this testimony is  based on are listed within each individual product.
section_title: Related GAO Products
paragraphs: Homeland Security: DHS Could Strengthen Acquisitions and  Development of New Technologies. GAO-11-829T. Washington, D.C.:  July 15, 2011.
Environmental Protection Agency: To Better Fulfill Its Mission, EPA  Needs a More Coordinated Approach to Managing Its Laboratories,   GAO-11-347. Washington, D.C.: July 25, 2011.
DHS Science and Technology: Additional Steps Needed to Ensure Test  and Evaluation Requirements Are Met. GAO-11-596. Washington, D.C.:  June 15, 2011.
Opportunities to Reduce Potential Duplication in Government Programs,  Save Tax Dollars, and Enhance Revenue, GAO-11-318SP. Washington,  D.C.: Mar. 2011.
Homeland Security: Improvements in Managing Research and  Development Could Help Reduce Inefficiencies and Costs,  GAO-11-464T. Washington D.C.: Mar. 15, 2011.
Warfighter Support: DOD’s Urgent Needs Processes Need a More  Comprehensive Approach and Evaluation for Potential Consolidation,  GAO-11-273. Washington, D.C.: Mar. 1, 2011.
Technology Transfer: Clearer Priorities and Greater Use of Innovative  Approaches Could Increase the Effectiveness of Technology Transfer at  Department of Energy Laboratories GAO-09-548. Washington, D.C.: June  16, 2009.
Homeland Security: DHS Needs a Strategy to Use DOE’s Laboratories  for Research on Nuclear, Biological, and Chemical Detection and  Response Technologies. GAO-04-653. Washington, D.C.: May 24, 2004.
Department of Energy: Uncertain Progress in Implementing National  Laboratory Reforms, GAO/RCED-98-197. Washington, D.C.: Sept. 10,  1998.
Best Practices: Elements Critical to Successfully Reducing Unneeded  RDT&E Infrastructure. GAO/NSIAD/RCED-98-23. Washington, D.C.:   Jan. 8, 1998.
Department of Energy: National Laboratories Need Clearer Missions and  Better Management, GAO/RCED-95-10. Washington, D.C.: Jan. 27,  1995.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.