DHS Research and Development: Science and Technology Directorate's Test and Evaluation and Reorganization Efforts
Released: 2011-11-17T00:00:00

Letter

S&T Could Take Additional Steps to Ensure that DHS T&E Requirements Are Met; Officials Cited Challenges to Overseeing T&E across DHS

S&T Oversight of DHS Testing and Evaluation

In June 2011, we reported that S&T met some of its oversight requirements for T&E of acquisition programs we reviewed, but additional steps were needed to ensure that all requirements were met.Specifically, since DHS issued the T&E directive in May 2009, S&T reviewed or approved T&E documents and plans for programs undergoing testing, and conducted independent assessments for the programs that completed operational testing during this time period. S&T officials told us that they also provided input and reviewed other T&E documentation, such as components’ documents describing the programs’ performance requirements, as required by the T&E directive. DHS senior level officials considered S&T’s T&E assessments and input in deciding whether programs were ready to proceed to the next acquisition phase. However, S&T did not consistently document its review and approval of components’ test agents—a government entity or independent contractor carrying out independent operational testing for a major acquisition—or document its review of other component acquisition documents, such as those establishing programs’ operational requirements, as required by the T&E directive. For example, 8 of the 11 acquisition programs we reviewed had hired test agents, but documentation of S&T approval of these agents existed for only 3 of these 8 programs. We reported that approving test agents is important to ensure that they are independent of the program and that they meet requirements of the T&E directive.

S&T officials agreed that they did not have a mechanism in place requiring a consistent method for documenting their review or approval and the extent to which the review or approval criteria were met. We reported that without mechanisms in place for documenting its review or approval of acquisition documents and T&E requirements, such as approving test agents, it is difficult for DHS or a third party to review and validate S&T’s decision-making process and ensure that it is overseeing components’ T&E efforts in accordance with acquisition and T&E directives and internal control standards for the federal government. As a result, we recommended that S&T develop a mechanism to document both its approval of operational test agents and component acquisitions documentation to ensure that these meet the requirements of the DHS T&E directive. S&T concurred and reported that the agency has since developed internal procedures to ensure that the approval of test agents and component acquisition documents are documented.

Challenges in Coordinating and Overseeing T&E across DHS

We also reported in June 2011 that S&T and DHS component officials stated that they face challenges in overseeing T&E across DHS components which fell into 4 categories: (1) ensuring that a program’s operational requirements—the key performance requirements that must be met for a program to achieve its intended goals—can be effectively tested; (2) working with DHS component program staff who have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time for T&E while remaining within program cost and schedule estimates.

Both S&T and DHS, more broadly, have begun initiatives to address some of these challenges, such as establishing a T&E council to disseminate best practices to component program managers, and developing specific guidance for testing and evaluating information technology acquisitions. In addition, as part of S&T’s recent reorganization, the agency has developed a new division specifically geared toward assisting components in developing requirements that can be tested, among other things. However, since these efforts have only recently been initiated to address these DHS-wide challenges, it is too soon to determine their effectiveness.

S&T Recently Reorganized and Our Prior R&D Work Could Inform How S&T Moves Forward

Since 2009, S&T has undertaken a series of efforts related to its organizational structure. S&T underwent a new strategic planning process, developed new strategic goals, and conducted a reorganization intended to better achieve its strategic goals. These efforts were implemented after a 2009 National Academy of Public Administration study found that S&T’s organizational structure posed communication challenges across the agency and that the agency lacked a cohesive strategic plan and mechanisms to assess performance in a systematic way, among other things. In August 2010, S&T reorganized to align its structure with its top strategic goals, allow for easier interaction among senior leadership, and reduce the number of personnel directly reporting to the Under Secretary of S&T. Additionally, after the Under Secretary was confirmed in November 2009, S&T instituted a new strategic planning process which helped inform the development of new strategic goals. The new strategic goals announced in August 2010 include: rapidly developing and delivering knowledge, analyses, and innovative solutions that advance the mission of DHS; leveraging its expertise to assist DHS components’ efforts to establish operational requirements, and select and acquire needed technologies; strengthening the Homeland Security Enterprise and First Responders’ capabilities to protect the homeland and respond to disasters; conducting, catalyzing, and surveying scientific discoveries and inventions relevant to existing and emerging homeland security challenges; and fostering a culture of innovation and learning in S&T and across DHS that addresses mission needs with scientific, analytic, and technical rigor.

According to S&T, the agency has developed a draft strategic plan that provides its overall approach to meeting these strategic goals, which is currently in the process of being finalized.

Moreover, according to testimony by the Undersecretary of S&T in March 2011, to ensure that individual R&D projects are meeting their goals, S&T has committed to an annual review of its portfolio of basic and applied R&D and all proposed “new start” projects. According to S&T, the review process uses metrics determined by S&T, with input from DHS components, that are aligned with DHS priorities. These metrics consider: the impact on the customer’s mission; the ability to transition these products to the field; whether the investment positions S&T for the future; whether the projects are aligned with customer requirements; whether S&T has the appropriate level of customer interaction; and whether S&T is sufficiently innovative in the way it is approaching its challenges.

We are currently reviewing DHS and S&T’s processes for prioritizing, coordinating, and measuring the results of its R&D efforts for the Senate Committee on Homeland Security and Governmental Affairs and we will report on this issue next year.

Our prior work related to R&D at other federal agencies could provide insight for S&T as it moves forward with new structures and processes operating within potential fiscal constraints. During the 1990s, we issued a series of reports on federal efforts to restructure R&D in the wake of changing priorities and efforts to balance the federal budget. More recently, we have issued reports on R&D issues at the Department of Defense (DOD), Department of Energy (DOE), the Environmental Protection Agency (EPA), and DHS. Although the specific recommendations and issues vary from department to department, there are key findings across this body of work that could potentially help inform S&T’s efforts to meet DHS’s R&D needs, as well as Congressional oversight of these activities. Since our assessment of R&D efforts at DHS is currently under way, we have not determined the extent to which these key findings from our prior work are applicable to DHS’s R&D efforts or the extent to which DHS already has similar efforts under way. However, our prior work could provide valuable insights into how DHS could leverage the private sector to help conduct R&D, restructure R&D efforts in response to fiscal constraints, and develop comprehensive strategies to mitigate the risk of duplication and overlap. For example:

We reported on federal agencies that have restructured their research and development efforts in response to fiscal constraints. For example, in January 1998, we reported on efforts by federal agencies, such as DOD, the DOE National Laboratories, and NASA, to streamline their R&D activities and infrastructure. We reported that restructuring research, development, testing and evaluation to meet current and future needs required interagency agreements and cross- agency efforts, in addition to ongoing individual efforts. Additionally, we reported on five elements that were useful in the successful restructuring of R&D in corporate and foreign government organizations. For example, we found that successful restructuring of R&D activities included having a core mission that supports overall goals and strategies, clear definitions of those responsible for supporting that mission, and accurate data on total costs of the organization’s activities.

In addition, we have reported that comprehensive strategies mitigate risk of duplication and overlap. For example, we reported in March 2011 that DOD did not have a comprehensive approach to manage and oversee the breadth of its activities for developing new capabilities in response to urgent warfighter needs, including entities engaged in experimentation and rapid prototyping to accelerate the transition of technologies to the warfighter, and lacked visibility over the full range of its efforts.issue guidance that defined roles, responsibilities, and authorities across the department to lead its efforts. DOD agreed with this recommendation.

As a result, we recommended that DOD

Within DHS itself, we reported in May 2004 that DHS did not have a strategic plan to guide its R&D efforts. We recommended that DHS complete a strategic R&D plan and ensure that the plan was integrated with homeland security R&D conducted by other federal agencies. We also recommended that DHS develop criteria for distributing annual funding and for making long-term investments in laboratory capabilities, as well as develop guidelines that detailed how DOE’s laboratories would compete for funding with private sector and academic entities. DHS agreed with our recommendations. While S&T developed a 5-year R&D plan in 2008 to guide its efforts and is currently finalizing a new strategic plan to align its own R&D investments and goals, DHS has not yet completed a strategic plan to align all R&D efforts across the department, as we previously recommended.

Our work on DOE National Laboratories provides additional insights related to oversight of R&D efforts that could be useful for DHS S&T. In 1995, we reported that DOE’s national laboratories did not have clearly defined missions focused on accomplishing DOE’s changing objectives and national priorities. DOE, at that time, managed the national laboratories on a program by program basis which inhibited cooperation across programs and hindered DOE’s ability to use the laboratories to meet departmental missions. We recommended, among other things, that DOE develop a strategy that maximized the laboratories’ resources. In responding, DOE said that it had undertaken a new strategic planning process which resulted in a strategic plan. Though DOE developed a strategic plan intended to integrate its missions and programs, in 1998 we reported that the laboratories did not function as an integrated national research and development system and recommended that DOE develop a comprehensive strategy to be used to assess success in meeting objectives, monitor progress, and report on that progress. DOE acknowledged that it needed to better focus the laboratories’ missions and tie them to the annual budget process, but that it would take time to accomplish.

More recently, we reported in June 2009 that DOE could not determine the effectiveness of its laboratories' technology transfer efforts because it has not yet defined its overarching strategic goals for technology transfer and lacks reliable performance data. Instead, individual DOE programs such as the National Nuclear Security Administration and DOE's Office of Science articulated their own goals for technology transfer at the national laboratories. We recommended, among other things, that DOE articulate department wide priorities and develop clear goals, objectives, and performance measures. DOE generally agreed with our findings.

Lastly, our work on Environmental Protection Agency (EPA) laboratory facilities also offers insights into the importance of planning and coordination in managing R&D. Specifically, we reported in July 2011 that EPA has yet to fully address the findings of numerous past studies that have examined EPA’s science activities. These past evaluations noted the need for EPA to improve long-term planning, priority setting, and coordination of laboratory activities, establish leadership for agency wide scientific oversight and decision making, and better manage the laboratories’ workforce and infrastructure. We recommended, among other things, that EPA develop a coordinated planning process for its scientific activities and appoint a top-level official with authority over all the laboratories, improve physical and real property planning decisions, and develop a workforce planning process for all laboratories that reflects current and future needs of laboratory facilities. EPA generally agreed with our findings and recommendations.

Chairman Lungren, Ranking Member Clarke, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have.

GAO Contact and Staff Acknowledgments

For questions about this statement, please contact David C. Maurer at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Chris Currie, Assistant Director; Emily Gunn and Margaret McKenna. Key contributors for the previous work that this testimony is based on are listed within each individual product.

Related GAO Products

Homeland Security: DHS Could Strengthen Acquisitions and Development of New Technologies. GAO-11-829T. Washington, D.C.: July 15, 2011.

Environmental Protection Agency: To Better Fulfill Its Mission, EPA Needs a More Coordinated Approach to Managing Its Laboratories, GAO-11-347. Washington, D.C.: July 25, 2011.

DHS Science and Technology: Additional Steps Needed to Ensure Test and Evaluation Requirements Are Met. GAO-11-596. Washington, D.C.: June 15, 2011.

Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue, GAO-11-318SP. Washington, D.C.: Mar. 2011.

Homeland Security: Improvements in Managing Research and Development Could Help Reduce Inefficiencies and Costs, GAO-11-464T. Washington D.C.: Mar. 15, 2011.

Warfighter Support: DOD’s Urgent Needs Processes Need a More Comprehensive Approach and Evaluation for Potential Consolidation, GAO-11-273. Washington, D.C.: Mar. 1, 2011.

Technology Transfer: Clearer Priorities and Greater Use of Innovative Approaches Could Increase the Effectiveness of Technology Transfer at Department of Energy Laboratories GAO-09-548. Washington, D.C.: June 16, 2009.

Homeland Security: DHS Needs a Strategy to Use DOE’s Laboratories for Research on Nuclear, Biological, and Chemical Detection and Response Technologies. GAO-04-653. Washington, D.C.: May 24, 2004.

Department of Energy: Uncertain Progress in Implementing National Laboratory Reforms, GAO/RCED-98-197. Washington, D.C.: Sept. 10, 1998.

Best Practices: Elements Critical to Successfully Reducing Unneeded RDT&E Infrastructure. GAO/NSIAD/RCED-98-23. Washington, D.C.: Jan. 8, 1998.

Department of Energy: National Laboratories Need Clearer Missions and Better Management, GAO/RCED-95-10. Washington, D.C.: Jan. 27, 1995.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.