Homeland Security: Research & Development Coordination Has Improved, but Additional Actions Needed to Track and Evaluate Projects
Released: 2019-03-21T00:00:00

Letter

Background

DHS R&D Roles and Responsibilities

The Homeland Security Act of 2002, as amended, designates the Under Secretary for Science and Technology as responsible for coordinating all R&D activities of DHS. The Act also provides that nothing in it precludes other department components from carrying out R&D activities as long as the activities are coordinated through S&T. As of September 2018, seven DHS components have budget authority to conduct R&D activities—S&T, the Coast Guard, the CWMD, the Secret Service, the Cybersecurity and Infrastructure Security Agency, TSA, and the Office of the Chief Information Officer within the Office of the Undersecretary for Management. Figure 1 provides an organizational overview of DHS components and offices that are involved in the R&D process as of December 2018.

S&T reorganized its structure in September 2018 and currently has three technical divisions responsible for managing R&D programs related to improving border, immigration, and maritime security; supporting first responders; and countering physical and cybersecurity threats among others, as shown in figure 2.

Most of S&T’s R&D portfolio consists of applied and developmental R&D, which can be transitioned to use within 3 years, as opposed to longer- term basic research. In addition to conducting projects for its DHS customers, S&T conducts research for other federal agencies and first responders. S&T is also responsible for conducting basic and applied research, and collaborates with other government agencies, academia, the private sector, and others.

Overview of Past R&D Management Challenges

Questions have been raised about S&T’s ability to demonstrate the impact of its investments—in terms of value, tangible products, and advances toward the homeland security mission. Accordingly, for example, House appropriations committee report language has directed S&T to demonstrate how its R&D efforts are timely, with results relatively well defined and to make investment decisions based on clear and sensible priorities. In 2016, Congress passed the National Defense Authorization Act of 2017 (NDAA) which requires DHS to report annually to Congress on the department’s R&D projects including details such as the project name, the component carrying out the project, associated funding levels, and expected objectives and milestones for each project, among other items.

Since DHS began operations in 2003, we have made multiple recommendations designed to improve DHS efforts to manage and oversee R&D efforts, as described later in this report. In September 2012, we reported that DHS did not have a department-wide policy defining R&D or guidance directing its components how to report R&D activities. As a result, DHS did not know its total annual investment in R&D, which limited the department’s ability to oversee components’ R&D efforts and align them with agency-wide R&D goals and priorities. We also reported that DHS’s R&D efforts were fragmented and overlapping, which increased the risk of unnecessary duplication. We recommended that DHS develop policies and guidance for defining, reporting and coordinating R&D activities across the department, and that DHS establish a mechanism to track R&D projects. As of August 2017, DHS had implemented these recommendations by, among other things, issuing guidance defining research and development activities and establishing Integrated Product Teams (IPT) as a primary mechanism for coordinating R&D. These actions and others are described in more detail later in this report.

Figure 3 provides a summary of key events related to S&T since its inception.

Past DHS Accounting Challenges

After the consolidation in 2002 of 22 agencies into a single department, DHS had, until recently, different appropriation structures and budget management practices based on agencies’ funding structures prior to DHS consolidation. In 2018, we found that, with over 70 different appropriations and over 100 formal program, project, or activity accounts, DHS operated for over a decade with significant budget disparities and inconsistencies across its components. The lack of uniformity hindered visibility, inhibited comparisons between programs, and complicated spending decisions, including for R&D-related programs. For example, in 2012 we reported that S&T, the Domestic Nuclear Detection Office, and Coast Guard were the only three DHS components with budget authority to conduct R&D. However, in 2012, we identified an additional $255 million in R&D obligations by other DHS components at that time. Further, we found in 2012 that the Domestic Nuclear Detection Office did not report certain R&D budget data to OMB, and R&D budget accounts included a mix of R&D and non-R&D spending, which further complicated DHS’s ability to identify its total investment in R&D activities.

Role of the Integrated Product Teams in R&D

Within DHS, IPTs – established in 2015 – are to identify and prioritize technological capability gaps, and identify current or future R&D efforts or other solutions to close the gap, among other things. Specifically, the IPT process consists of three activities: 1) Identifying R&D activities in progress, funded, planned, or recently completed; 2) Prioritizing technological capability gaps and corresponding R&D efforts to address those gaps; and 3) Validating and reporting the gaps.

The DHS IPT effort is led by S&T, but the individual IPTs are composed of senior-level officials from across DHS. IPT members prioritize R&D gaps based on departmentwide needs and requirements, and align current and planned R&D efforts to the identified gaps. In prioritizing and evaluating the capability gaps, four pre-defined criteria and rating scales are used and are discussed below:

Strategic alignment: assesses the R&D gaps alignment with DHS- level and component-level strategic priorities.

Impact: assesses if addressing the R&D gap would result in enhanced risk or threat reduction capability, among other things.

Feasibility: assesses the feasibility of addressing the R&D gap, given its technical complexity. Considerations include feasibility related to technology, time, and transition.

R&D needs: assesses whether the R&D gap would provide a critical R&D solution in an otherwise unaddressed area.

The IPT’s role in coordinating R&D is discussed later in this report.

Overview of DHS’s Processes for Assessing R&D Performance

GPRA, as updated and expanded by GPRAMA, requires agencies to establish annual performance goals with target levels of performance against which to measure progress towards those goals. In addition, GPRA requires executive agencies to prepare an Annual Performance Report on program performance for the previous fiscal year. DHS has developed goals and targets to assess and communicate R&D performance. As shown in figure 4, DHS’s performance assessment process also includes identifying performance gaps and implementing corrective actions to address unmet performance goals.

DHS uses strategic and management performance goals and measures to assess and communicate on the performance of its R&D efforts. In addition, DHS uses milestones to track and communicate progress of its R&D project activities.

Milestones: A milestone is a scheduled event signifying the completion of a major deliverable or a phase of work. Milestones can help agencies demonstrate that they have clear and fully developed strategies and are tracking progress to accomplish their goals. Milestones are often used as the basis of an alternative form of performance goal. Milestones related to DHS R&D efforts are reported to Congress and publicly available through the DHS congressional budget justification.

Strategic Goals: A type of performance goal used to reflect achievement of missions that are publicly reported in the DHS Annual Performance Report. As part of DHS’s Annual Performance Report, these goals are subject to GPRA and GPRAMA requirements.

Management Goals: A type of performance goal used to gauge program results and tie to resource requests that are reported to Congress and publicly available through the DHS congressional budget justification along with the strategic goals.

As we previously reported in 1997, experts in research measurement have tried for years to develop indicators that would provide a measure of the results of R&D. However, the very nature of the innovative process makes measuring the performance of science-related projects difficult. For example, a wide range of factors determine if and when a particular R&D project will result in commercial or other benefits. It can also take many years for a research project to achieve results.

DHS Has Obligated More Than $1 Billion per Fiscal Year from 2010 through 2017 for Conducting R&D That Aligns with Department Mission Areas across Seven DHS Components

R&D Funding Amounts

DHS is required to report department-wide R&D-related funding to OMB on an annual basis. DHS uses several mechanisms to report the R&D- related funding, including budget authority (the legal authorization to obligate funds), obligations (binding agreements to make a payment for services), and outlays (payments to liquidate obligations representing amount expended). Further, OMB requires agencies to submit data on R&D programs as part of their annual budget submissions on investments for basic research, applied research, development, R&D facilities construction, and major equipment for R&D using OMB’s definition of R&D.

Based on our analysis of OMB’s federal obligations data, we identified R&D-related obligations data for DHS components for fiscal years 2010 through 2017. Figure 5 depicts the R&D related obligations that were reported for fiscal years 2010 through 2017, which, on average, were about $1.3 billion annually or more than $10 billion overall for that time frame. Additionally, S&T obligated nearly 80 percent of all DHS R&D funds for that time period.

Types of R&D Activities that S&T Conducts

S&T may conduct or fund R&D activities on its own or jointly with other entities. In addition to S&T, six other DHS components currently have budget authority to conduct R&D—the Coast Guard, CWMD, TSA, Secret Service, Cybersecurity and Infrastructure Security Agency, and the Undersecretary for Management. In August 2018, S&T reported that there were at least 132 ongoing R&D projects across the Department. Some R&D projects aim to produce a specific prototype or piece of technology for an end user, while others might be for developing IT systems, conducting specific training, or providing written reports, or knowledge products.

According to S&T officials, S&T generally leads or funds R&D projects by providing technology and knowledge products for four homeland security areas:

Disaster resilience. Improving community resilience to natural disasters through technology and tools that support planning, decision-making and mitigation efforts;

Critical incidents. Improving technological capabilities during all stages of critical incident response;

Border security. Improving the nation’s ability to detect, interdict and prosecute illegal activity across air, land and sea.

Cybersecurity. Developing technologies, tools and techniques to defend, mitigate, and secure current and future systems, networks and critical infrastructures against cyberattacks.

Figure 6 illustrates the types of R&D projects that are either led or funded by S&T for each category. For more in-depth examples and descriptions of S&T projects, please see appendix I.

In its efforts to determine how to best support the DHS components and first responders, S&T seeks first to identify the end user’s needs by discussing operational challenges with components and first responders; then develop prototypes or leverage existing technologies to find solutions; and finally to test and evaluate potential solutions to ensure that they meet the end user’s needs and ultimately deploy solutions to the field.

The other six DHS components with R&D budget authority typically lead and fund R&D projects tailored to support their specific operational requirements and respective missions. Examples of R&D projects conducted by DHS components other than S&T are listed below in table 1.

S&T Implemented R&D Coordination Mechanism and Directives to Strengthen Collaboration Across DHS, but Challenges Remain in Ensuring Participation

DHS Established a Process to Coordinate R&D Efforts

DHS established its IPT process in August 2015 as the central mechanism to coordinate R&D efforts across the department, in accordance with recommendations we made in 2012. The IPT process works to identify DHS technological capability gaps and coordinate R&D to close the gaps across DHS mission areas.

The IPTs consist of senior representatives from operational components. As of October 2018, IPTs are organized according to the department’s identified missions and include the following sub-IPTs, as shown in table 2.

Each IPT has an establishing charter document, which formally identifies the IPT component members and responsibilities and lists the corresponding sub-IPTs. IPTs and sub-IPTs are to meet multiple times throughout the year to support the process of identifying and prioritizing R&D capability gaps and R&D efforts. For example, the charter for the “Secure Borders” IPT states that they anticipate meeting at least 2 or 3 times per year, or more frequently to support the annual program planning and budgeting process.

Overall, components reported that the IPT process enhanced collaboration and improved visibility into R&D efforts across DHS. Officials from all 10 of the DHS components we interviewed reported the IPT process has been helpful in various ways, including identifying capability gaps, prioritizing and closing the gaps, and providing transparency and insight into other components’ R&D efforts. For example, CBP officials reported that, through the IPT process, they were able to identify R&D projects that the Coast Guard had been pursuing related to maritime security. The R&D projects that Coast Guard was pursuing were also of interest to CBP, and therefore CBP worked through the IPT process to prevent duplicative work and combine some of those efforts. In another example, TSA officials reported that they collaborated with the Secret Service to test explosive screening technologies, and that the IPT process facilitated their ability to collaborate and share information about the screening technologies.

In addition to enhancing collaboration, component officials provided their perspectives on how the IPT process prioritizes technology capability gaps that components have identified. For example, TSA officials reported that the gap identification and prioritization process works well, but that funding R&D activities to close the gaps is more challenging because it is influenced heavily by competing budget priorities, emerging threats, and other DHS senior leadership priorities. TSA officials further reported that departmental resource constraints limit the number of identified capability gaps that can be addressed. However, officials from CBP reported that several R&D projects were successfully implemented after CBP had worked with S&T to identify a capability gap and transition a solution to close the gap, such as certain upgrades needed on CBP trucks. S&T officials stated that they have also taken steps to integrate with the department’s Joint Requirements Council and utilize component requirement executives who work with component agencies to provide a basis for requirements and aid the components with the means to track the progress and disposition of each capability gap on a regular basis.

DHS Issued R&D Coordination Directives but Faces Challenges in Ensuring IPT Participation

In 2012, we found that, among other things, DHS had not developed a policy defining who was responsible for coordinating R&D within the department and what processes should be used to coordinate it. As a result, components did not consistently coordinate with S&T on what R&D was planned or underway, leading to increased risk of unnecessary duplication of R&D efforts. We recommended that DHS develop and implement policies and guidance for overseeing R&D that included, among other things, a description of the department’s process and roles and responsibilities for overseeing and coordinating R&D investments. DHS concurred with our recommendation, and, in response, the Secretary for Homeland Security delegated the authority to coordinate and integrate the department’s R&D, testing, evaluation efforts to the Under Secretary for Science and Technology in 2014. In 2015 and 2016, DHS issued two guidance documents regarding the establishment and progress of the IPT process. These documents specified how DHS, through the IPT process, is to implement processes and mechanisms to coordinate department-wide R&D efforts. Additionally, in January 2017, DHS issued an R&D directive and associated instruction to formalize R&D reporting and coordination among components, as shown in figure 7.

The 2017 directive and associated instruction identify the roles and responsibilities, including IPT participation requirements, for key entities involved in R&D across DHS. However, the directive and instruction do not specifically address steps to be taken if components do not adhere to the requirements. For example, the January 2017 DHS instruction states that “to effectively coordinate DHS R&D activities, DHS components are required to follow the DHS IPT process.” However, officials from CWMD stated that they do not participate in the S&T-led IPT sessions because they have their own internal process for identifying and prioritizing capability gaps. S&T officials stated that CWMD’s predecessor organization, the Domestic Nuclear Detection Office, participated in the IPT process until DHS initiated a reorganization of its weapons of mass destruction programs (resulting in the current CWMD).

Current non-participation by CWMD, which has the second-largest R&D budget within DHS and obligated approximately 17 percent of DHS R&D funds, or $176 million in fiscal year 2017, poses risk of R&D project information not being shared among components. In August 2018, we reported that DHS’s chemical defense programs and activities were fragmented and not well coordinated across the department, including R&D activities. We recommended that CWMD develop a strategy and implementation plan to help DHS integrate and coordinate its chemical defense programs and activities, among other things. Additionally, in its 2014-2018 Strategic Plan, DHS states that, to anticipate key threats, DHS should, among other things, prioritize R&D activities related to chemical, biological, radiological, and nuclear terrorism. Given these factors, CWMD’s participation in the IPT process is important to ensure that all R&D efforts are fully coordinated thereby mitigating the risk of potential duplication of other DHS R&D efforts.

S&T officials recognize that some components might not be complying fully with the departmental directives and associated guidance documents which require participating in the IPT process – the key R&D coordination mechanism within DHS. S&T officials stated that, despite these challenges, they have strong collaborative relationships with the components, and the existing collaboration mechanisms, such as the IPT process, continue to mature and facilitate R&D-related information sharing. However, DHS guidance documents require that components participate in the IPT process. By ensuring that all required components participate in the IPT process, DHS can help S&T maintain visibility of R&D projects in order to fulfill its statutory role of coordinating R&D.

S&T Has Taken Steps to Identify and Track DHS’s R&D Project Information, But Efforts Are Disjointed or Do Not Align with DHS Guidance

Since 2012, S&T has taken steps to identify and track information related to ongoing R&D projects across DHS, and in 2017, DHS developed a common appropriations structure that standardized R&D budgeting processes across the department. However, S&T’s efforts to identify and track R&D project information have limitations and can result in information that is not comprehensive. We also identified challenges in collecting information related to the achievement of R&D milestones.

DHS Developed the Common Appropriations Structure to Better Streamline and Standardize the Budgeting Process across Components

In 2017, DHS developed a common appropriations structure that allowed it to calculate and monitor its expenses, including R&D expenses, across the department. Officials from DHS’s OCFO reported that, prior to the new structure, some components categorized their R&D expenses as other types of expenses, such as “salaries and expenses.” These categorizations made it difficult to account for R&D expenses outside of an individual component’s budget management division. Furthermore, OCFO officials reported that components previously utilized inconsistent R&D definitions, which often led to discrepancies in how components would report R&D activities.

In our April 2018 report, we found that DHS had operated for over a decade with significant budget disparities and inconsistencies across its components. We found that the lack of uniformity hindered visibility, inhibited comparisons between programs, and complicated spending decisions. According to DHS OCFO officials, the introduction of the common appropriations structure, among other things, has helped improve transparency within DHS and among the components so that R&D can be more readily identified and tracked. DHS is also able to compare R&D funding amounts throughout DHS more easily than in previous years. In addition, of the seven components that have their own R&D funding to report, five indicated that the new structure has improved the department’s ability to identify and report R&D activities.

S&T’s Efforts to Maintain an Inventory of R&D Projects across DHS Have Resulted in Reporting That Is Not Comprehensive

We identified multiple sources of component R&D project information, each posing its own challenges or limitations. As described below, these challenges and limitations include difficulty in collecting and integrating R&D project information, and reporting that is not comprehensive.

DHS’s response to the National Defense Authorization Act of 2017: The NDAA, passed in December 2016, required DHS to provide a list of ongoing R&D projects and accompanying milestone information by January 2017, and annually thereafter, to specified congressional committees. In December 2017, DHS officials reported that they had not yet submitted the report, and anticipated that the response to the NDAA requirement would be completed by January 2018. In August 2018, DHS submitted its response to the committee, then 19 months late. S&T officials stated that the reporting delays were due to challenges in collecting and integrating the data. S&T officials also reported that it used the components’ congressional budget justification documents as a starting point to identify R&D projects to include in its report in response to the NDAA requirement. However, additional details about the R&D projects had to be collected via a “data call” process from the components. S&T officials told us that it was a challenge to have components report information about their R&D projects consistently and systematically. Furthermore, S&T officials identified terminology-related challenges in their R&D data call efforts, including making distinctions between R&D projects, efforts, and activities. S&T officials also reported that, in its current format, they would not be able to easily identify how many projects were added to the NDAA list across years, or if a given R&D project experienced a large increase or decrease in funding.

Annual Reports of Coordinated R&D: In response to a 2015 request from the Secretary of Homeland Security that the IPTs identify R&D work being performed across DHS, S&T issued a “Report of Coordinated R&D” in 2016 and 2017. The content for the reports was developed through a “data call” process, and the reports identified R&D activities and projects across DHS. The reports – for 2016 and 2017 – contain tables of R&D project names and the component leading the project, among other things. However, during the course of our review, S&T officials reported that these annual reports should not be considered authoritative lists of R&D projects due to inconsistencies in the project information that components reported which led to the reports not being comprehensive. For example, when we asked about some significant variations in the number of projects between 2016 and 2017, S&T officials told us that one DHS component responded to the data call with a list of R&D activities that included a “wish list” of R&D for their component, and not actual ongoing R&D activities. DHS officials acknowledged that they do not have a mechanism to ensure the comprehensiveness of information reported by the components through the data call process. In addition, two components did not respond to S&T’s request for R&D project information for the 2017 Annual Report of Coordinated R&D.

Congressional Budget Justifications: In May 2018, in the absence of a single, comprehensive list of R&D projects across DHS prior to the issuance of its report in response to the NDAA, S&T officials referred us to the R&D projects listed within the seven individual congressional budget justifications for the components that currently have budget authority to conduct R&D. Furthermore, as discussed earlier, S&T officials used the congressional budget justifications as their starting point in developing their response to the NDAA. However, S&T officials stated that there may be differences between the projects listed in the NDAA response and the projects listed in the congressional budget justifications. For example, S&T stated that the report in response to the NDAA includes all “ongoing” projects, regardless of the fiscal year in which they received funds; while the congressional budget justifications include R&D projects for which funding was requested for the given fiscal year. In other words, S&T officials clarified, they included all R&D projects in their response to the NDAA that had project activity, regardless of whether funding was requested in a particular congressional budget justification.

S&T’s Project Tracker Database and the S&T Analytical Tracking System: A 2014 House Appropriations Committee report noted that the committee had repeatedly raised questions about S&T’s prioritization of R&D projects and that, without the ability to easily review and compare detailed information on all S&T projects and activities, the Under Secretary for S&T could not effectively carry out S&T’s responsibilities. Accordingly, the Committee directed S&T to develop a method or system for tracking all S&T-funded R&D projects. A November 2016 DHS Directive reiterates this requirement, specifying that the list of projects should be updated on at least a quarterly basis throughout the duration of an R&D project. S&T officials told us that, in response to the committee report, they developed the Project Tracker Database, which was in use at the time of our review, but was transitioned to a new system, the S&T Analytical Tracking System, in September 2018. Neither the S&T Analytical Tracking System nor its predecessor system, the Project Tracker Database, is intended to comprehensively collect information on R&D projects across the department, only for R&D projects managed within S&T. Given the recent implementation of the S&T Analytical Tracking System, it is too soon to tell whether it will improve and streamline S&T’s efforts to collect and analyze R&D-related information within the directorate.

In addition, S&T officials stated that none of the above R&D information sources are suited to long-term trend analysis or data aggregation of department-wide R&D project information, and that these sources are disparate across DHS. S&T officials also acknowledged that better aligning R&D project information sources is an important aspect of improving how the department collects information DHS-wide.

Standards for Internal Control in the Federal Government call for agencies to maintain quality information that is, among other things, current, accurate, accessible, and provided on a timely basis. Furthermore, the standards call for an agency’s management team to process relevant data from reliable sources and utilize it to make informed decisions. The disparate R&D project information sources that S&T maintains, such as DHS’s response to the National Defense Authorization Act of 2017 and the Annual Reports of Coordinated R&D discussed above, and the manual data-call process it takes to update the sources limits departmental access to current and reliable R&D project information. For example, an internal DHS web-based portal with pre- defined fields could provide component officials with a means for reporting information more consistently and comprehensively. Without complete and readily accessible R&D information, DHS may not have the information it needs to make informed decisions about R&D investments, such as which projects are to be prioritized. By developing a mechanism to address challenges and limitations related to the collection, integration, and comprehensiveness of R&D data across the department, S&T can improve its visibility on R&D efforts across DHS in accordance with its role as DHS’s coordinator of R&D efforts.

DHS Has Not Effectively Developed Some R&D Performance Management Information, Posing Challenges to Assess and Report on the Progress of R&D Projects

DHS components have processes in place to collect certain indicators of R&D performance, but we found that these processes have limitations. The methods used to assess and report performance and progress of DHS R&D efforts we identified include:

Milestone information – used to assess and communicate progress to Congress and agency decision makers on individual R&D projects

Strategic and Management Performance Goals – milestone and other information is aggregated to provide summary information on R&D performance by mission area

Customer feedback – information gathered by component officials on R&D customer perspectives on the utility of ongoing or completed projects Below is our analysis of the three methods.

Milestones

Milestones are often used as the basis of an alternative form of performance goal. Performance goals specified in alternative form must be described in a way that makes it possible to discern if progress is being made toward the goal. Milestones related to DHS R&D efforts are reported to Congress and publicly available through the DHS congressional budget justification. A milestone is a scheduled event signifying the completion of a major deliverable or a phase of work, and can be described in a way that makes it possible to discern if progress is being made toward a goal. Milestones can also help agencies demonstrate that they have clear and fully developed strategies and are tracking progress to accomplish their goals. In our analysis of 14 milestones for seven S&T high-priority R&D projects identified in fiscal year 2018 DHS budget justification documents, we found that 3 of the 14 milestones fully adhered to DHS guidance for milestone descriptions. DHS budget development guidance suggests DHS components, which develop milestones for inclusion in congressional budget justification documents, utilize leading practices provided in the guidance. The leading practices state that successful milestones contain the following characteristics: 1. Specific - provide a clear understanding of expected results; 2. Measurable - the result can be reported in quantitative or qualitative 3. Results-Oriented/Relevant - milestone clearly links to results-oriented activities such as strategy, budget, and/or program/project plans; 4. Time-Bound - milestone specifies a beginning and end date for As shown in table 3, we identified that more than half of the milestones (8 of 14) were not specific and 10 of 14 were not results-oriented. Eleven of 14 milestones we analyzed were measurable and time-bound. While our analysis is not generalizable to all fiscal year 2018 R&D milestones, it illustrates areas where the selected milestones do not fully incorporate the DHS guidance.

Below is more detail on our assessment of the Specific and Results- Oriented guidance.

Specific. Of the 14 milestones we reviewed, eight did not contain specific information that would allow reviewers to have a clear understanding of the result expected in connection with the milestone. For example, one milestone for a cyber-related R&D project states that “testbeds and pilots would be conducted with at least one department or agency.” However, the milestone is not specific enough to ascertain what types of testbeds or pilots are being assessed and how the testbed effort would link to a possible end result.

Results-Oriented and Relevant. Ten of the 14 milestones that we reviewed did not clearly link the milestone back to results-oriented activities, such as strategy, budget, or project/program plans. For example, one milestone for a first responder program stated the following: “Transition, commercialize, or make available through open source platforms at least three technologies (e.g., Analyses, models, technology prototypes and/or knowledge prototypes).” It is unclear which technologies would be transitioned or how these technologies would be transitioned and made available.

According to DHS OCFO officials, DHS congressional budget justifications, which include milestones, serve to provide explanation and detail for why DHS believes Congress should support the department’s R&D projects. DHS components are instructed by DHS’s budget office to routinely submit their congressional budget justifications for internal DHS review, which is a process and mechanism that results in the supporting justifications for R&D funding requests. DHS OCFO officials also stated that they are not aware of a singular reason for why milestones do not consistently incorporate DHS’s guidance and stated that they have also identified instances in which milestones do not align with the guidance. As our analysis indicates, S&T’s milestones could better incorporate milestone criteria included in DHS’s budget preparation guidance. Without milestone information that more closely aligns with DHS guidance, Congress and DHS decision-makers may not be able to fully assess whether R&D projects are meeting specific goals within assigned time frames or identify what adjustments, if any, may be needed to facilitate the achievement of project goals and the R&D mission overall.

Strategic and Management Performance Goals

DHS has developed 12 performance goals to assess and report on its R&D efforts, DHS is required to identify department-wide goals in its strategic plan and annual performance report. For fiscal years 2016 through 2018, DHS’s Annual Performance Report included two strategic performance goals related to S&T’s R&D efforts. DHS’s congressional budget justification includes the two strategic performance goals as well as 10 related management performance goals. For a detailed listing of the 12 performance goals, see table 4.

Seven of the 10 management performance goals were for S&T R&D efforts and the remaining three were for Domestic Nuclear Detection Office’s R&D efforts, which cover the components that account for 96 percent of DHS’ fiscal year 2017 R&D obligations. DHS has performance goals for mission programs that produce operational results that link directly to the DHS Strategic Plan, according to officials from the OCFO’s Program Analysis and Evaluation division. DHS also uses milestones to track the progress of the other components’ R&D efforts.

R&D Customer Feedback

DHS components that conduct R&D use various methods to collect and analyze customer feedback to assess their R&D efforts, as shown in table 5. However, DHS is not well positioned to integrate the results because limited customer feedback information is collected and analyzed.

Six DHS components that have R&D-related responsibilities evaluate customers’ needs and improve customer satisfaction by listening to customers’ feedback about the quality of deliverables they receive—both good and bad— and making changes necessary to enhance that deliverable. Specifically, officials from S&T, the Coast Guard, CWMD, TSA, the Cybersecurity and Infrastructure Security Agency, and the Secret Service stated they have varying methods in place for gathering customer feedback regarding the progress and the results of R&D activities and deliverables. Below is a summary of these components’ efforts to consider customer feedback.

S&T. S&T’s project management guide outlines a process for ensuring customer requirements are being adequately met using a customer survey that can be modified and provided to the customer to complete at each major milestone. In addition, proceedings (e.g., minutes) from regularly scheduled meetings with customer and end user groups may be used to gather information regarding value and operational impact in lieu of a survey. The S&T survey asks customers to rate their overall satisfaction with S&T products and services, along with specific aspects of support, such as providing products in time to meet needs and effectively keeping customers informed. However, out of the 97 R&D activities that S&T reported in fiscal year 2017 and the 110 activities in fiscal year 2016, S&T collected one customer survey form.

Coast Guard. The Coast Guard also has a process in place for surveying and interviewing its customers following the completion of an R&D project and officials reported using this information for future R&D planning. The Coast Guard’s survey instrument seeks feedback on: customer satisfaction, timeliness, utility, and communications, among other things. The customer service survey is distributed for feedback on deliverables. At least 6 months after an R&D project is completed, Coast Guard also conducts an in-person interview with project sponsors to collect project transition performance success and feedback information. The surveys that Coast Guard uses to obtain feedback elicit a relatively low number of responses from customers, significantly limiting their usefulness in soliciting feedback data. Specifically, the response rates for fiscal years 2013-2017 were 16%, 17%, 27%, 13%, and 17%, respectively. Experts on customer satisfaction measurement have stated that although survey response rates are never 100 percent, an organization should strive to get its rate as close as possible to that number. They suggest that ideally, organizations can obtain response rates of over 70 percent.

CWMD. CWMD does not have a formal mechanism, such as standard processes and procedures, for collecting and analyzing customer feedback. However, CWMD officials stated that certain informal mechanisms are used to collect customer feedback. For example, CWMD officials reported that the CWMD Office of Policy, Plans, Analysis, and Requirements Directorate communicate with customers and gather customer needs and requirements. In addition, as part of these informal mechanisms, internal and external reviews feedback may be obtained from eventual end users of the R&D technology such as operators from CBP, the U.S. Coast Guard, and the TSA, according to CWMD officials.

Cybersecurity and Infrastructure Security Agency. The directorate does not have a formal mechanism for collecting and analyzing customer feedback. However, periodic control gates are used to gather customer feedback, according to directorate officials. The input received during these reviews is used to make corrective actions and manage R&D efforts as necessary. For example, according to directorate officials, they conduct a comprehensive review of R&D coordination efforts annually to determine what was effective and what can be improved.

TSA. TSA does not have a formal mechanism for collecting and analyzing customer feedback. However, according to TSA officials, informal feedback may be obtained through review of the weekly reports and meetings regarding recent developments and project milestones. In addition, feedback may be obtained during quarterly program management reviews, third party project development, and certification testing.

Secret Service. The Secret Service does not have a formal mechanism for collecting and analyzing customer feedback. However, according to Secret Service officials, informal feedback may be obtained in conjunction with other related internal review activities, including program management reviews.

To formalize and improve customer feedback processes for R&D efforts, the National Academy of Sciences has stated that feedback from both R&D failures and successes may be communicated to stakeholders and used to modify future investments. Research on leading practices in the area of customer satisfaction suggests that multiple approaches are needed to effectively listen to customers about their perceptions of quality service and needs. The research also points to a need for centrally integrating all customer feedback so that managers can achieve a better understanding of customers’ perceptions and needs. Also, we have previously reported that leading organizations combine quantitative and qualitative listening tools to obtain customer feedback and then centrally integrate the data in one location. Such approaches include the following:

Customer satisfaction surveys. We previously reported that most major organizations use tools such as surveys to periodically capture customers’ overall perceptions about their organization and to measure satisfaction with specific transactions soon after they occur. These surveys can be administered through the mail, by telephone, in person, or electronically.

Benchmark surveys. Benchmark surveys gather perceptions of performance from the entire market. These surveys usually gather customer perceptions of performance about top competitors in an industry. This allows the company to examine its customer-perceived strengths and weaknesses in the overall marketplace. While continuous improvement may be a result of this listening tool, the real value, according to the research in this area, comes from breakthrough thinking to gain a sustainable advantage.

Focus groups. Organizations use focus groups to get better information from customers than survey results provide. In these groups, customers are probed about why they answered survey questions the way they did.

Customer interviews. Conducting interviews with customers can provide a way to get very detailed information about their specific needs and problems. Like focus groups, this tool is used by leading customer service organizations to probe survey respondents as to why they answered survey questions a certain way.

The National Academy of Sciences have stated that evaluating the relevance and impact of R&D is a key stage of the R&D process and that measuring the impact of R&D activities requires looking to the end users and stakeholders for an evaluation of the impact of a research program, such as through polling or systematic outreach. In addition, Standards for Internal Control in the Federal Government calls for entities to determine an oversight structure to fulfill responsibilities that are set forth by feedback from key stakeholders, among other things. As a result of the limited customer feedback information that is collected and analyzed, DHS is unable to more fully understand its customers’ perceptions and experience to allow it to assess the benefits and performance of its R&D efforts. Moving forward, standard processes and procedures for collecting and analyzing R&D customer feedback would help in assessing R&D efforts.

Conclusions

Since 2010, DHS has obligated more than $10 billion dollars on R&D to develop technologies to support DHS’s efforts to prevent, mitigate, and recover from terrorist and natural threats. S&T officials indicated that they have strong collaborative relationships with components; however, it is important that required components fully participate in the IPT process in order for S&T to maintain visibility of R&D projects and successfully fulfill its statutory role of coordinating R&D and to help reduce the risk of potential duplication of R&D efforts across the department. Furthermore, S&T faces challenges and limitations related to the collection, integration, and comprehensiveness of information on R&D projects. Without a mechanism that aligns information sources and results in comprehensive and accurate data, among other things, DHS may not have the information it needs to make informed decisions about R&D investments. S&T also does not fully leverage existing guidance when developing milestones for R&D efforts. Without milestone information that more fully aligns with DHS criteria, Congress and DHS decision-makers may not have a full understanding of R&D progress and challenges. Finally, standard processes and procedures for collecting and analyzing R&D customer feedback would help to assess its R&D efforts.

Recommendations for Executive Action

We are making the following four recommendations to the Deputy Secretary of the Department of Homeland Security: The Deputy Secretary of the Department of Homeland Security should ensure that all components adhere to IPT participation requirements, in accordance with the DHS directives. (Recommendation 1)

The Deputy Secretary of the Department of Homeland Security should develop a mechanism that aligns processes and information sources for collecting R&D project data from DHS components to ensure that the information can be collected, integrated and result in a comprehensive accounting of R&D projects DHS-wide. (Recommendation 2)

The Deputy Secretary of the Department of Homeland Security should direct OCFO program officials to ensure that S&T take steps to more fully incorporate leading practices, such as those included in DHS’s budget preparation guidance, into R&D milestones. (Recommendation 3)

The Deputy Secretary of the Department of Homeland Security should develop standard processes and procedures for collecting and analyzing customer feedback, applicable to components conducting R&D, for improving the usefulness of existing customer feedback mechanisms to assess R&D efforts and for implementing such mechanisms where absent. (Recommendation 4)

Agency Comments and our Evaluation

We provided a draft of this report to DHS for review and comment. DHS provided written comments which are reproduced in appendix II. In its comments, DHS concurred with our recommendations and described actions planned to address them. S&T, OCFO, CBP, the Cybersecurity and Infrastructure Security Agency, and CWMD also provided technical comments, which we incorporated as appropriate.

With regard to our first recommendation, that the Deputy Secretary of the Department of Homeland Security should ensure that all components adhere to IPT participation requirements, in accordance with DHS directives, DHS stated that S&T’s Office of Science & Engineering will revise the relevant DHS directive to require participation in the IPT process by all components. DHS estimated that this effort would be completed by December 31, 2019. This action, if fully implemented, should address the intent of the recommendation.

With regard to our second recommendation, that the Deputy Secretary of the Department of Homeland Security should develop a mechanism for collecting R&D project data in order to complete a comprehensive accounting of R&D projects DHS-wide, DHS stated that S&T’s Office of Science & Engineering will revise the relevant DHS directive to include requirements for data collection on all R&D projects across DHS to ensure alignment of the appropriate data elements and existing guidance. DHS estimated that this effort would be completed by December 31, 2019. This action, if fully implemented, should address the intent of the recommendation.

With regard to our third recommendation, that the Deputy Secretary of the Department of Homeland Security should direct OCFO program officials to ensure that S&T take steps to more fully incorporate leading practices, such as those included in DHS’s budget preparation guidance, into R&D milestones, DHS stated that the OCFO will continue to work with S&T to incorporate the leading practices and that the OCFO will validate all S&T annual budget submissions and provide S&T feedback, as appropriate. DHS estimated that this effort would be completed by April 30, 2020. This action, if fully implemented, should address the intent of the recommendation.

With regard to our fourth recommendation, that the Deputy Secretary of the Department of Homeland Security should standardize processes for collecting and analyzing customer feedback to aid in assessing R&D efforts, DHS stated that S&T’s Office of Science & Engineering will revise the relevant DHS directive to incorporate customer feedback procedures into the IPT process for the recipients of R&D programs. DHS estimated that this effort would be completed by December 31, 2019. This action, if fully implemented, should address the intent of the recommendation.

We are sending copies of this report to the appropriate congressional committees and the Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact William Russell at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Overview of the Science and Technology Directorate’s Research and Development Projects

The following appendix provides a general overview of the Department of Homeland Security’s (DHS) Science and Technology Directorate (S&T) research and development (R&D) projects and programs that support the following homeland security mission areas: (1) border security; (2) disaster resilience; (3) critical incidents, and (4) cybersecurity. The examples provided below are illustrative and therefore not intended to provide a comprehensive list of DHS R&D programs or projects.

S&T and the U.S. Coast Guard conducted drone demonstrations and tests at two sites in Mississippi to test unmanned aerial systems before they are deployed to the field.

The Technical Assessment of Counter Unmanned Aerial Systems Technologies in Cities is designed to help public safety and industry officials identify potential methods for countering nefarious uses of small unmanned aerial systems.

The Wall System Design Support Tool Independent Verification and Validation seeks to strengthen the U.S. Border Patrol’s decision analysis model used to identify the areas of the border where a wall would be most beneficial.

The Border Research in Instrumented Construction Project is designed to identify cameras, sensors and other technology that can be applied on or near a smart wall via ground, surface/air, subsurface and water to enhance border security and agent safety.

The Apex Border Situational Awareness program aims to help U.S.

Customs and Border Protection access more data sources, develop decision support tools and share information with partner law enforcement agencies to improve situational awareness.

The Integrated Maritime Domain Enterprise is a platform that seeks to bridge disparate data systems to make it easier for DHS components to share information and collaborate.

The Adaptive Sensor Analytics Project aims to provide automated data analytics to process satellite imagery, identify patterns of nefarious activity and alert DHS officials.

Ground-Based Technologies Program seeks to improve the ability to detect illegal activity at the border through stronger situational awareness, automated detection and alerts, target classification and tools to promote agent safety

Air-Based Technologies Program is designed to identify, test and evaluate unmanned and manned aircraft platforms and sensors for law enforcement, search and rescue, and disaster response in both land and maritime environments.

The Expert Tracker training program aims to help U.S. Border Patrol agents improve their ability to track movement in rough terrain along the nation’s borders.

Port of Entry Based Technology Program seeks to improve illicit cargo detection and legitimate cargo throughput by upgrading legacy scanning systems and linking them to new analysis and information sharing tools that may make the most of personnel resources.

The Port of Entry People Screening Program aims to identify, evaluate and implement combinations of process and technology improvements that facilitate the movement of people through the nation’s air, land and sea ports of entry.

Autopsy is an open-source digital forensics platform that seeks to help law enforcement determine how electronic devices were used in a crime and recover evidence.

Voice Forensics aims to help identify individuals who make hoax rescue calls to the U.S. Coast Guard, which may make it easier to find and prosecute suspects.

Child Exploitation Image Analytics seeks to reduce the amount of time it takes to identify and rescue children from exploitation, as well as identify perpetrators, through automated face recognition algorithms and forensic tools.

The Tunnel Detection and Surveillance Program is designed to help border officials detect and locate clandestine tunnels, as well as gather forensic data to support investigation and prosecution of drug smuggling activities.

The Port of Entry Forensics and Investigations Program aims to help combat transnational crime and investigate child exploitation and human trafficking through open source data and forensic analysis of material collected from suspicious packages and cargo.

DHS S&T seeks to help improve community resilience to natural disasters through technology and tools that support planning, decision making and mitigation efforts.

The Canada-U.S. Enhanced Resiliency Experiment series aims to use real-world exercises to demonstrate that seamless communication is possible between responders on either side of the northern border during a large-scale emergency.

DHS S&T and the Central United States Earthquake Consortium are developing a suite of decision support tools designed to help emergency managers analyze data used when planning, managing, coordinating and communicating during natural disasters

The Mutual Aid Resource Planner is a prototype application designed to help jurisdictions develop more accurate resource plans by incorporating custom data on geospatial hazards, risk assessments and potential mutual aid partners.

The National Mutual Aid Technology Exercise seeks to test existing mutual aid systems to improve users’ ability to exchange information between systems in real time and develop technical guidance for future use.

The Coastal Resilience Center of Excellence aims to conduct targeted research and education to address key challenges facing coastal communities in the United States, including storm surge modeling, pre-disaster planning, communicating risk and more.

The Flood Apex Program is designed to help identify and develop technology that can reduce flood-related fatalities and property loss, increase community resilience and improve flood preparation, response and recovery.

The Internet of Things Low Cost Flood Inundation Sensors project seeks to develop and test sensor technology that can provide real- time updates on rising water levels.

The Kentucky Dam Safety project aims to create technology and processes to better monitor dams and alert communities of potential danger, reducing loss of life and property.

The Advanced CIRCulation modeling tool seeks to accurately predict coastal flooding threats to help emergency managers better coordinate evacuation and response.

The Hurricane Evacuation -eXtended platform is a decision support tool for emergency managers designed to organize and stage resources for hurricane response.

The Simulation-Based Decision Support System for Water Infrastructural Safety Lite™ tool is designed to quickly model the effects of potential dam breaks, helping officials develop accurate emergency response plans and anticipate evacuation needs.

The Tunnel Plug is an inflatable device that aims to seal off subway tunnels to prevent water from flowing into the system, minimizing damage to critical transportation systems.

The Linking the Oil and Gas Industry to Improve Cybersecurity project seeks to facilitate cooperative research, development, testing and evaluation procedures to improve cybersecurity in petroleum industry digital control systems

The Homeowner Flood Insurance Roundtable aims to help reduce future uninsured flood losses by identifying decision support and research and development needs.

The Automated National Structures Inventory project is seeking to build a comprehensive list of private and commercial property at risk for flood damage, which may help promote proper insurance and more effective flood protection efforts.

The Smart Cities IoT innovation project is designed to help first responders improve their situational awareness through advances in autonomous drone navigation, intelligent building sensors and body- worn interoperability platforms.

The Wireless Emergency Alerts Research, Development, Testing and Evaluation program aims to inform changes to the Federal Communications Commission’s alerting system, including increased character length and adding URLs, pictures, videos and geo-targeting capabilities.

The System Assessment and Validation for Emergency Responders program seeks to evaluate available responder technology on affordability, usability, and other criteria to help agencies understand which equipment will best fit their needs.

The Urban Operational Experimentation program is designed to let responders test new technologies in real-world settings, and may help provide developers with direct feedback on how their products can better meet operational needs.

The Enhanced Dynamic Geo-Social Environment training platform is a free virtual tool that aims to allow responders to practice responding to an active shooter incident, whether within a single agency or with multiple jurisdictions and disciplines.

The Surface Transportation Explosives Threat Detection program is aiming to develop screening technology that can identify potential threats on people and in their bags without physically interacting with them.

The Explosives Detection Canine program is designed to help detection canine teams identify new explosive compounds through non-hazardous training aids and increase their proficiency through realistic self-assessment and training events across the country.

The Datacasting project aims to help responders send encrypted video, data files, and other critical information through existing public broadcast television signals, which helps prevent other communication channels from being overwhelmed.

The Next-Generation Incident Command System, a web-based platform, seeks to allow responders to share data and request assistance in real-time, and also allows officials to observe and make critical decisions during evolving situations to better support preparation, response, and recovery.

The Android Team Awareness Kit, a free app, is designed to help responders visually track team members and assets in real time during an incident, as well as share encrypted data across jurisdictions, disciplines, and components.

The Assistant for Understanding Data through Reasoning, Extraction and Synthesis platform aims to help responders overcome information overload by providing actionable insight based on up-to-the-minute sensor data.

The First Responder Electronic Jamming Exercise seeks to identify mitigation tactics against intentional or accidental communications jamming, which responders were able to practice implementing in realistic scenarios.

The Telephony Denial of Service program is designed to help improve 911 emergency call centers’ ability to defend against attacks through cyber security technologies that can analyze incoming calls and may help determine potential threats in real time.

The Finding Individuals for Disaster and Emergency Response is designed to detect human heartbeats under up to 30 feet of rubble, which may help responders more effectively target rescue efforts.

The Rapid DNA technology can complete a DNA test within 90 minutes or less from the field, which seeks to help officials identify victims and inform family members in a timely manner.

The Forensic Video Exploitation and Analysis tool aims to help responders quickly analyze video to identify potential suspects by allowing users to tag a person to a left-behind item and reconstruct that individual’s path across multiple camera views.

The Cyber Risk Economics program seeks to fund applied R&D, knowledge products by gathering stakeholders across government, industry and academia to discuss cyber risk economics capability gaps and needs. Through these stakeholder discussions, along with scholarly cybersecurity economics research literature reviews and authoritative U.S. federal government documents, DHS S&T developed the newly released Cyber Risk Economics Capability Gaps Research Strategy which aims to consider business, legal, technical and behavior factors impacting cyber risk.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgements

GAO Contact

William Russell, (202) 512-8777 or russellw@gao.gov.

Staff Acknowledgements

In addition to the contact named above, Ben Atwater (Assistant Director), Melissa Hargy (Analyst-in-Charge), Nanette Barton and Gary M. Malavenda made key contributions to this report. In addition, key support was provided by Chris P. Currie, Dominick Dale, Michele Fejfar, Richard Hung, Benjamin Licht, John Mingus, Janet Temko-Blinder, and Sarah Veale.