Department of Homeland Security: Actions Needed to Strengthen Management of Research and Development
Released: 2014-09-09T00:00:00

Letter

DHS Does Not Know Its Total Investment in R&D, but Has Taken Some Steps to Update Guidance

In September 2012, we found that DHS did not know how much its components have invested in R&D, making it difficult to oversee R&D efforts across the department. According to DHS budget officials, S&T, DNDO, and the U.S. Coast Guard were the only components that conducted R&D, and we found that they were the only components that reported budget authority, obligations, or outlays for R&D activities to OMB as part of the budget process. However, we reported that the data DHS submitted to OMB underreported DHS’s R&D obligations because DHS components obligated money for R&D contracts that were not reported to OMB as R&D. Specifically, for fiscal year 2011, we identified an additional $255 million in R&D obligations by other DHS components. These obligations included DHS components providing S&T with funding to conduct R&D on their behalf and components obligating funds through contracts directly to industry, to universities, or with DOE’s national laboratories for R&D.

Further, we found that the data for fiscal years 2010 through 2013 DHS submitted to OMB also underreported DHS’s R&D budget authority and outlays because DNDO did not properly report at least $293 million in R&D budget authority and at least $282 million in R&D outlays. We reported that DHS budget officials agreed that DHS underreported its R&D spending and when asked, could not provide a reason why the omission was not flagged by DHS review.

In addition, in our 2012 report, we found that DHS’s R&D budget accounts included a mix of R&D and non-R&D spending. For fiscal year 2011, we estimated that 78 percent of S&T’s Research, Development, Acquisition, & Operations account; 51 percent of DNDO’s Research, Development, & Operations account; and 43 percent of the Coast Guard’s R&D budget account funded R&D activities. As a result, this further complicated DHS’s ability to identify its total investment in R&D.

We also reported in September 2012 that DHS did not have a department-wide policy defining R&D or guidance directing components how to report R&D activities. As a result, we concluded that it was difficult to identify the department’s total investment in R&D, a fact that limited DHS’s ability to oversee components’ R&D efforts and align them with agency-wide R&D goals and priorities, in accordance with Standards for Internal Control in the Federal Government.time that DHS used OMB’s definition of R&D, but the definition was broad and its application may not be uniform across components, and thus, R&D investments may not always be identified as R&D. We found that the variation in R&D definitions may contribute to the unreliability of the reporting mechanisms for R&D investments in budget development and execution, as discussed above.

DHS officials told us at the We recommended that DHS develop and implement policies and guidance for defining and overseeing R&D at the department that include, among other things, a well-understood definition of R&D that provides reasonable assurance that reliable accounting and reporting of R&D resources and activities for internal and external use are achieved. DHS agreed with our recommendation and stated that it planned to evaluate the most effective path forward to guide uniform treatment of R&D across the department in compliance with OMB rules and was considering a management directive, multicomponent steering committee, or new policy guidance to help better oversee and coordinate R&D. As of September 2014, DHS has updated its guidance to include a definition of R&D, but, as discussed in more detail below, efforts to develop a specific policy outlining R&D roles and responsibilities and a process for overseeing and coordinating R&D with other offices remain ongoing and have not yet been completed. We will continue to monitor DHS’s efforts to implement these recommendations.

S&T Has Taken Some Actions to Coordinate R&D across DHS, but R&D Activities Are Fragmented and Overlapping

We reported in September 2012 that the Homeland Security Act of 2002 provides S&T with the responsibility for, among other things, coordinating and integrating all research, development, demonstration, testing, and evaluation activities within DHS and establishing and administering the primary R&D activities of the department. S&T developed coordination practices that fall into four general categories: (1) S&T component liaisons, (2) R&D agreements between component heads and S&T, (3) joint R&D strategies between S&T and components, and (4) various R&D coordination teams made up of S&T and component project managers, which are discussed in detail in our 2012 report and 2013 testimony.

Despite S&T’s efforts to coordinate R&D activities, in September 2012, we reported that R&D at DHS was inherently fragmented because several components within DHS—S&T, the Coast Guard, and DNDO—were each given R&D responsibilities in law, and other DHS components may pursue and conduct their own R&D efforts as long as those activities are coordinated through S&T. Fragmentation among R&D efforts at DHS may be advantageous if the department determines that it could gain better or faster results by having multiple components engage in R&D activities toward a similar goal; however, it can be disadvantageous if those activities are uncoordinated or unintentionally overlapping or duplicative.

Specifically, we found at least six department components involved in R&D activities in our review of data on about 15,000 federal procurement contract actions coded as R&D taken by DHS components from fiscal years 2007 through 2012. We examined 47 R&D contracts awarded by these components—selected because they appeared to have activities similar to those of another contract—and found 35 instances among 29 contracts in which the contracts overlapped with activities conducted elsewhere in the department. Taken together, these 29 contracts were worth about $66 million. In one example of the overlap, we found that two DHS components awarded 5 separate contracts that each addressed detection of the same chemical.

While we did not identify instances of unnecessary duplication among these contracts, in September 2012, we found that DHS had not developed a policy defining who is responsible for coordinating R&D activities at DHS that could help prevent overlap, fragmentation, or unnecessary duplication and did not have tracking mechanisms or policies to help ensure that overlap is avoided and efforts are better coordinated consistent with Standards for Internal Control in the Federal Government. S&T officials told us at the time that a process did not exist at DHS or within S&T to prevent overlap or unnecessary duplication but that relationships with components mitigate that risk. They also stated that S&T has improved interactions with components over time. We concluded that the existence of overlapping R&D activities coupled with the lack of policies and guidance defining R&D and coordination processes was an indication that not all R&D activities at DHS were coordinated to ensure that R&D is not unnecessarily duplicative.

We also found in September 2012 that neither DHS nor S&T tracked all ongoing R&D projects across the department, including R&D activities contracted through the national laboratories. As part of our review, we identified 11 components that reimbursed the national laboratories for R&D from fiscal years 2010 through 2012, but S&T’s Office of National Laboratories could not provide us with any information on those activities and told us it did not track them. According to S&T, the Office of National Laboratories’ ability to provide information on activities across the department is limited by components inconsistently operating within the defined process for working with the national laboratories.

As a result, we recommended that DHS develop and implement policies and guidance for overseeing R&D that includes, among other things, a description of the department’s process and roles and responsibilities for overseeing and coordinating R&D investments and efforts, and a mechanism to track existing R&D projects and their associated costs across the department. DHS agreed with our recommendation and stated at the time that S&T was implementing a collaborative, end-user-focused strategy to coordinate and interact with components to better ensure S&T’s efforts aligned with components’ needs and that it was considering developing new policy guidance for R&D activities across the department. According to DHS officials, the department implemented an R&D portfolio review process, as directed by committee reports accompanying the fiscal year 2013 DHS appropriations act, which is aimed at better coordinating R&D activities by reviewing components’ individual R&D projects. In April 2014, DHS developed a definition for R&D and stated that S&T was responsible for coordinating and integrating R&D activities throughout the department. However, as of September 2014, not enough time has passed to determine whether this process and new memorandum have improved coordination. Furthermore, to better define and manage R&D across the department, DHS should also establish a mechanism to track R&D projects and costs, as we recommended. Fully implementing our recommendation to develop a policy that defines roles and responsibilities for coordinating R&D and coordination processes, as well as a mechanism that tracks all DHS R&D projects, could better position DHS to mitigate the risk of overlapping and unnecessarily duplicative R&D projects. We will continue to monitor DHS’s efforts to develop a policy to better coordinate and track R&D activities at the department.

S&T Has Taken Steps to Obtain Feedback and Evaluate the Impact of Its Border and Maritime R&D Efforts

Costs and Types of Completed Border and Maritime R&D Projects Varied

In September 2013, we reported that DHS S&T, Coast Guard, and DNDO reported producing 97 R&D deliverables at an estimated cost of $177 million between fiscal years 2010 and 2012. The type of border and maritime R&D deliverables produced by these R&D entities were wide ranging in their cost and scale, and included knowledge products and reports, technology prototypes, and software. For example:

Knowledge products or reports: One of the DHS Centers of Excellence developed formulas and models to assist in randomizing Coast Guard patrol routes and connecting networks together to assist in the detection of small vessels.

Technology prototypes: S&T BMD developed prototype radar and upgraded video systems for use by Border Patrol agents and a prototype scanner to screen interior areas of small aircraft without removing panels or the aircraft skin.

Software: DNDO developed software that extracts data from radiation portal monitors and uses the data to improve algorithms used in detecting radioactive material.

As we reported in September 2013, R&D customers we met with had mixed views on the impact of the R&D deliverables they received. For example, we reviewed the 20 S&T BMD deliverables produced between fiscal years 2010 and 2012 at a cost of $28.7 million. We found that the customers of 7 deliverables stated that the deliverables met their office’s needs, customers of 7 did not, customers of 4 did not know, and customers for 2 could not be identified. For example, customers within CBP’s Office of Technology Innovation and Acquisition reported that S&T’s analysis and test results on aircraft-based use of wide area surveillance technology helped CBP to make a decision on whether it should pursue acquiring such technology. In cases where customers said that the deliverables were not meeting their needs, the customers explained that budget changes, other ongoing testing efforts, or changes in mission priorities were the reasons deliverables had not met their needs, and customers pointed out that their relationship with S&T had been positive and highly collaborative. In other cases, customers pointed out that while the deliverable had not been used as intended, it informed their office’s decision making and helped to rule out certain technologies as possibilities. In this regard, the customers felt the R&D was successful, despite the fact that the deliverable had not or was not being used.

S&T BMD officials explained that some of its older projects did not have identifiable customers because its former process for selecting projects created the potential to engage in R&D without a clear commitment from the customer. In February 2012, S&T issued a new project management guide that requires project managers to specify the customer by office and name, and to describe customer support for the project, including how the customer has demonstrated commitment for and support of the project. S&T officials said they believed this new process would prevent future R&D funding from going toward projects without a clear customer.

Additionally, we reported that from fiscal year 2010 through fiscal year 2012, DNDO produced 42 deliverables at a cost of $115.9 million, which included 6 discontinued projects and 36 projects that were either transitioned to the next phase of R&D or were completed. DNDO R&D is different from the R&D of S&T for many reasons. For one, a DNDO project may start at a basic research level, and may end up being merged into other similar efforts in order to achieve a higher project goal. In these cases, the R&D customers are DNDO project managers rather than another DHS customer, such as CBP. We discussed 5 DNDO R&D deliverables at various R&D phases with DNDO officials—4 of which were deliverables from ongoing or completed projects and 1 of which was a discontinued project. These officials said that the early-stage R&D at DNDO feeds into the prioritized ranking of gaps in the global nuclear detection architecture, as well as into the analysis-of-alternatives phase of DNDO’s solutions development process.discussed had moved from early-stage R&D into other projects further along in DNDO’s project management process. Two of the 5 projects were completed, with 1 project that was reported to have provided information that further informed DNDO decision making and the other project resulting in a commercialized product. With regard to the 1 discontinued project, DNDO officials said that the particular project’s technology was determined to be too expensive to continue pursuing.

S&T Did Not Gather and Evaluate Feedback

We reported that although S&T project managers sought feedback from their customers during the execution of projects, S&T did not gather and evaluate feedback from its customers to determine the impact of its completed R&D efforts and deliverables, making it difficult to determine if the R&D met customer needs. Further, in some cases, the customer of S&T’s R&D was not clear or the results of the R&D were unknown. For example, a CBP customer identified by S&T was aware of 2 R&D deliverables that S&T said were transitioned to his office, but the official was unable to provide additional information on the project’s impact. According to S&T officials, since they deal with multiple DHS components and are not within the same agencies as its customers, it is sometimes difficult to identify who the customer of the R&D is and also difficult to determine what the impact of the R&D was. S&T officials also stated that in S&T’s 2012 update to its project management guide, in its project closeout process, S&T has included a step to collect feedback from all relevant customers and a template for collecting this feedback.

While we found in September 2013 that S&T had developed a process and template to collect feedback at the end of each project and incorporated this into its project management plan, we also found that it did not plan to survey customers each time it provides a deliverable to the customer. This is relevant because S&T projects are often conducted over several years before they are concluded and these projects also often produce multiple deliverables for a customer over many years that are designed to meet a specific operational need. For example, the Ground Based Technologies project began in fiscal year 2006 and is slated to continue through fiscal year 2018. During this period, S&T has provided multiple R&D deliverables to CBP—including test results comparing different ground-based radar systems. The National Academy of Sciences has stated that feedback from both R&D failures and successes may be communicated to stakeholders and used to modify future investments. At the time of our report, S&T had not established time frames and milestones for collecting and evaluating feedback from its customers on the extent to which the deliverables it provides were meeting its customers’ needs.

As a result, we recommended that S&T establish time frames and milestones for collecting and evaluating feedback from its customers to determine the usefulness and impact of both its R&D projects and project deliverables, and use it to make better-informed decisions regarding future work. S&T officials concurred with the recommendation at the time of our review, and reported that S&T was developing R&D strategies with DHS components that would include strategic assessments of components’ R&D needs and be updated annually on the basis of customer feedback. As of September 2014, S&T has completed strategic plans with Border Patrol, the Transportation Security Administration (TSA), and the Secret Service. Further, at the time of our review, S&T reported that it was developing a new project management guide to improve R&D management at all stages of development, and that the guide would include a template for project managers to use to gather customer feedback on a more consistent basis. In November 2013, S&T finalized its guide, which includes a customer survey template to obtain feedback on the quality and timeliness of a deliverable, as well as detailed descriptions of actions project managers should take throughout the project to ensure the R&D is aligned with customer needs. We will continue to review the implementation of these actions and determine whether they fully address our recommendation to S&T.

DHS Border and Maritime R&D Agencies Have Taken Action to Improve Internal and External R&D Coordination

In September 2013, we also reported that S&T’s BMD, the Coast Guard, and DNDO reported taking a range of actions to coordinate with one another and their customers to ensure that R&D is addressing high- priority needs. Officials from BMD identified several ways in which it coordinates R&D activities with its customers, which are primarily offices within CBP. For example, BMD officials reported having a person detailed to CBP’s Office of Technology Innovation and Acquisition and identified its integrated product teams, such as its cross-border tunnel threat team, and jointly funded projects as ways in which the division works to ensure its R&D efforts are coordinated with CBP. We found that opportunities exist for DHS to enhance coordination with universities conducting R&D on its behalf. Specifically, we reported that the S&T Office of University Programs could help ensure that the approximately $3 million to $4 million a year dedicated to each university center is used more effectively by more carefully considering data needs, potential access issues, and potential data limitations with its federal partners before approving projects. We recommended that S&T ensure design limitations with regard to data reliability, accessibility, and availability are reviewed and understood before approving Center of Excellence R&D projects. S&T Office of University Programs officials concurred with the recommendation and discussed the variety of ways in which centers and DHS components collaborate and share information. Office of University Programs officials stated that the office’s process for soliciting research topics and evaluating proposals is good and that it keeps the centers flexible. However, officials from DHS’s primary land border security Center of Excellence reported challenges with respect to a lack of clarity regarding protocols for access to DHS information when conducting R&D. Specifically, officials from this center reported that they have been regularly unable to obtain data from CBP to complete research it was conducting on CBP’s behalf, which resulted in delays and terminated R&D projects.

Given the challenges raised by officials from universities leading the R&D for land border security, we recommended that S&T conduct a more rigorous review of potential data-related challenges and limitations at the start of a project in order to help R&D customers (such as CBP) identify data requirements and potential limitations up front so that money is not allocated to projects that potentially cannot be completed. In concurring with our recommendation, S&T Office of University Programs officials agreed that making sure their clients take additional steps to identify data requirements up front could help address these challenges and following our review had started taking steps to address this. For instance, in September 2013, the Office of University Programs reported that it was working to develop standard guidelines and protocols that would apply to all of its centers of excellence. These protocols would describe how data sets must be modified to enable their use in open-source research formats. In March 2014, the Office of University Programs and the National Center for Border Security and Immigration, a DHS S&T Center of Excellence, cohosted a workshop to identify common problems the centers have in accessing data from DHS, understand DHS constraints in sharing data, and develop best practices for requesting and sharing data between the centers of excellence and DHS. We believe this is a step in the right direction and should move S&T closer toward meeting the intention of our recommendation. We will continue to monitor DHS’s efforts in this area.

Chairman Meehan, Chairman Buschon, Ranking Member Clarke, Ranking Member Lipinski, and members of the committees, this completes my prepared statement. I would be happy to respond to any questions you may have at this time.

GAO Contact and Staff Acknowledgments

If you or your staff members have any questions about this testimony, please contact me at (202) 512-9627 or Maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement include Adam Hoffman, Assistant Director, and Emily Gunn. Aditi Archer, Charlotte Gamble, and Gary Malavenda also made contributions to this testimony.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.