Critical Infrastructure Protection: DHS Could Strengthen the Management of the Regional Resiliency Assessment Program
Released: 2013-07-30T00:00:00

Letter

Background

DHS Roles and Responsibilities in Critical Infrastructure Protection

Various laws and directives guide DHS’s role in critical infrastructure protection, including the Homeland Security Act of 2002, as amended, the Homeland Security Presidential Directive/HSPD-7, and most recently, Presidential Policy Directive/PPD-21, which was issued on February 12, 2013. Consistent with HSPD-7, which directed DHS to establish uniform policies, approaches, guidelines, and methodologies for integrating federal infrastructure protection and risk management activities within and across CI sectors, 18 CI sectors were established. PPD-21, among other things, purports to refine and clarify critical infrastructure-related functions, roles, and responsibilities across the federal government, and enhance overall coordination and collaboration. Pursuant to PPD-21, which expressly revoked HSPD-7, 2 of the 18 sectors were incorporated into existing sectors, thereby reducing the number of CI sectors from 18 to 16 (app. I lists the CI sectors and their SSAs).

PPD-21 directs DHS to, among other things, coordinate the overall federal effort to promote the security and resilience of the nation’s critical infrastructure. PPD-21 also recognizes that DHS, in carrying out its responsibilities under the Homeland Security Act, evaluates national capabilities, opportunities, and challenges in protecting critical infrastructure; analyzes threats to, vulnerabilities of, and potential consequences from all hazards on critical infrastructure; identifies security and resilience functions that are necessary for effective public-private engagement with all critical infrastructure sectors; and integrates and coordinates federal cross-sector security and resilience activities and identify and analyze key interdependencies among critical infrastructure sectors.

Within DHS, NPPD’s IP is responsible for various activities intended to enhance CI protection and resilience across a number of sectors. While other entities may possess and exercise regulatory authority over CI to address security, such as for the chemical, transportation, and nuclear sectors, IP generally relies on voluntary efforts to secure CI because, in general, DHS has limited authority to directly regulate CI. In carrying out its responsibilities, IP leads and coordinates national programs and policies on critical infrastructure issues and, among other things, conducts and facilitates security surveys and vulnerability assessments to help CI owners and operators and state, local, tribal, and territorial partners understand and address risks. In so doing, IP is responsible for working with public and private sector CI partners in the 16 sectors and leads the coordinated national effort to mitigate risk to the nation’s CI through the development and implementation of CI protection and resilience programs.

IP’s Protective Security Coordination Division (PSCD) provides programs and initiatives to enhance CI protection and resilience and reduce risk associated with all-hazards incidents. In so doing, PSCD works with CI owners and operators and state and local responders to (1) assess vulnerabilities, interdependencies, capabilities, and incident consequences; (2) develop, implement, and provide national coordination for protective programs; and (3) facilitate CI response to and recovery from incidents. Related to these efforts, PSCD has deployed 91 PSAs in 50 states and Puerto Rico, with deployment locations based on population density and major concentrations of CI. In these locations, PSAs are to act as the links between state, local, tribal, and territorial organizations and DHS infrastructure mission partners in the private sector and are to assist with ongoing state and local CI security efforts. PSAs are also to support the development of the national risk picture by conducting vulnerability and security assessments to identify security gaps and potential vulnerabilities in the nation’s most critical infrastructures. In addition, PSAs are to share vulnerability information and protective measure suggestions with local partners and asset owners and operators.

The Regional Resiliency Assessment Program

As discussed earlier, DHS developed the RRAP to assess vulnerability and risk associated with dependent and interdependent infrastructure clusters and systems in specific geographic areas. RRAP projects are intended to evaluate CI on a regional level to identify facilities and sectors that are dependent on one another, or interdependent. RRAP projects also identify situations where failures at facilities or sectors would lead to failures at other facilities or sectors, characteristics that make facilities and regions within the study resilient to disruptions, and resilience vulnerabilities that could promote or foster disruptions. According to DHS officials, the sectors selected to be studied as part of a RRAP project may vary based on priorities of IP and the state(s) where the RRAP occurs, that is, the “sector” focus can be narrow or broad, depending on the concerns of the state. For example, a transportation sector RRAP project in one state focused only on bridges, while another RRAP project in another state examined lifeline sectors.

The region or area covered by the RRAP project can also vary substantially. For example, the size of the “region” under study in a RRAP project in Colorado covered a few square miles within a city. Conversely, another RRAP covered an entire industry spread across a large state and yet another RRAP is looking at infrastructure that crosses 12 states. Accordingly, RRAP projects have been conducted in various locations throughout the country covering a wide variety of CI sectors and regions. These RRAP projects include one covering the financial district in Chicago; three covering commercial facilities in cities like Minneapolis, Atlanta, and Las Vegas; and one covering energy production facilities managed by the Tennessee Valley Authority. Figure 1 provides a map showing the states where RRAP projects have been completed or are planned.

According to DHS officials, the current process for conducting a RRAP project can take from 18 to 24 months from start to finish. The process includes selecting and scoping RRAP projects from proposals; assembling and preparing a RRAP team of federal, state and local training the states via webinar (i.e., stakeholder awareness training); conducting an introductory kickoff (i.e., outreach) meeting; gathering preliminary data and selecting sites to be included in the scheduling meetings with asset owners or operators of the sites; conducting ongoing analyses using data derived from performing the aforementioned vulnerability and security assessments at facilities;conducting stakeholders’ meetings for training purposes and to discuss regional resilience issues; preparing a draft report for state review; incorporating the state’s feedback into a final report; and establishing a process to follow up with stakeholders to, among other things, periodically update their progress making RRAP-related enhancements.

The final RRAP report typically includes a description of the key findings of the vulnerabilities in the sector(s) and region under study, including vulnerabilities for individual facilities, a hazard and risk analysis for the region and sector under review, and an analysis of dependencies and interdependencies. Also included in the RRAP report are resilience enhancement options that provide the report recipient suggestions to address key findings and mitigate the indentified vulnerability or weakness, and a list of organizations or funding sources that could provide the state and other stakeholders with support if they choose to implement an identified resilience enhancement option. RRAP reports can provide insights into the resilience of a region and the sector(s) under review and the gaps that could prompt regional disruptions.

Another aspect of the program centers on DHS’s efforts to use RRAP projects to build stakeholder relationships and enhance information sharing and coordination among stakeholders in a particular region. For example, one RRAP report stated that fostering relationships between key facilities and supporting infrastructure providers was necessary to improve response to a hazard or incident. Another RRAP project sought to coordinate a partnership of key players and stakeholders (including both public and private sector stakeholders in the sector of focus and local law enforcement) to improve information sharing necessary for responding to a contamination in the food supply system. According to DHS officials, the creation and continuation of these stakeholder relationships is a major benefit of RRAP projects and the RRAP process. DHS officials said it is often the case that regional CI stakeholders were not acquainted and did not understand how their own operations were related to those of other stakeholders until the RRAP was conducted.

For fiscal year 2013, as in past fiscal years, the RRAP does not have a budget line item; rather the costs for the program are funded with resources budgeted for DHS’s vulnerability assessment program and for PSAs. DHS officials estimated that the cost to PSCD for the average RRAP project is currently less than $1 million, including IP assessments, contractor support, and travel and administrative costs. The estimate does not include costs incurred for services rendered by other IP branches that participate in RRAP projects, like IP’s National Infrastructure Simulation and Analysis Center (NISAC), which, among other things, develops computerized simulations of the effect of an all- hazards event on particular geographic areas. The estimate also does not include costs incurred by other SSAs, or the states and localities participating in a RRAP project.

DHS Has Developed Criteria to Identify RRAP Project Candidates, but Does Not Fully Document Its Project Recommendation and Selection Process

PSCD has developed criteria that consider various factors when selecting possible locations and sectors for RRAP projects. PSCD uses the criteria to develop lists of RRAP project candidates, and officials use this list to make final project selections. However, PSCD officials do not fully document why certain project candidates are or are not recommended for selection by the IP Assistant Secretary.

DHS’s Approach for Identifying and Selecting RRAP Projects Has Evolved

IP’s approach for identifying and selecting RRAP projects has evolved since the program’s inception in 2009. For fiscal years 2009 and 2010, IP headquarters officials stated that they identified and selected RRAP project locations and sectors based on IP interests and preferences while considering input from primary stakeholders. IP officials told us that they relied heavily on IP’s interests and preferences because they considered RRAP projects conducted during this time frame as pilot projects. For fiscal years 2011 and 2012, IP officials stated that they refined their process for identifying and selecting RRAP projects to incorporate more input from primary stakeholders. For example, IP officials developed a RRAP project template for PSAs and states to use when jointly developing RRAP project proposals. The template included information on regional characteristics and risk, the willingness of state and facility stakeholders to participate, potential outcomes of the RRAP analysis, and planning and logistical considerations. While considering project proposals states and PSAs jointly developed using the template, IP headquarters officials also developed their own RRAP project proposals (using open source documents for major metropolitan areas) to ensure IP leadership could consider a range of projects across a variety of sectors and locations. IP officials stated that when selecting projects during fiscal years 2011 and 2012, they considered, among other factors, information obtained from the template and, if applicable, risk-based factors such as the concentration of critical infrastructure, and IP management judgment as to the feasibility of conducting the project.

More recently, for projects planned to begin in fiscal year 2013, IP took two actions to further revise its RRAP project identification and selection process. First, IP revised its process from that used in previous years by considering only RRAP project proposals submitted jointly by PSAs and states. According to IP officials, they made this change to help ensure that RRAP locations and sectors reflected state priorities, particularly in light of lessons learned from past RRAP projects and feedback from SLTTGCC. In a 2011 report on state and local government CI resilience activities, SLTTGCC expressed, among other things, concern about the scope of RRAP projects—particularly when states did not request the RRAP project—and the cost and resources required to be involved in a RRAP project.

Second, IP officials developed nine point selection criteria to identify lists of potential RRAP project candidates. IP officials stated that they developed the criteria to help evaluate proposals and to develop lists of potential candidate projects given the volume of proposals generated by states and PSAs and the DHS resources available to conduct RRAP projects. IP officials told us that they asked PSAs and PSA regional directors who had previously conducted RRAP projects to review the criteria before the criteria were finalized to provide assurance that the criteria reflected lessons learned.

Our review of IP criteria shows that it focuses on nine questions in four broad categories: whether the proposed project (1) is feasible, (2) promotes partnering with important stakeholders, (3) will produce results with broad applicability to other locations, and (4) accounts for risk-based factors. These criteria were used to evaluate the RRAP project proposals used to make the fiscal year 2013 and 2014 RRAP project recommendations. Table 1 lists the criteria IP uses to develop a list of feasible RRAP project candidates. A more detailed explanation of these criteria can be found in app. II, table 3.

DHS analysts may conduct supplemental research or contact PSAs or state officials to gather additional information. For example, to determine whether the proposed project is likely to produce original key findings and resiliency enhancement options, the analyst may reach out to the PSA and other critical infrastructure stakeholders to see if the state or other organization has initiated similar work to avoid duplicative activities. above are then referred to PSCD officials for further consideration, and PSCD officials select among those candidates to develop a list of recommended projects for approval by the IP Assistant Secretary. Figure 2 depicts IP’s current RRAP proposal and selection process, as of May 2013.

According to PSCD officials, the Assistant Secretary for IP selects projects from among those candidates PSCD officials recommend, but PSCD officials did not fully document why specific project candidates were or were not recommended for selection. For fiscal years 2013 and 2014, IP analysts identified 22 project candidates that scored a seven or greater. PSCD officials stated that after further review, they recommended that the Assistant Secretary select 16 of the 22 projects— 10 to be conducted in fiscal year 2013 and 6 to be conducted in fiscal year 2014.10 of PSCD’s recommended project candidates. According to PSCD officials, the Assistant Secretary plans to make final fiscal year 2014 project selections in October 2013. For the 16 projects, IP officials told us For fiscal year 2013, the IP Assistant Secretary selected all they provided the Assistant Secretary information about each of the recommended project candidates. However, PSCD officials did not document why individual projects were recommended over others, including candidate projects that received the same score—they stated that they believe providing such information on the projects that are recommended is sufficient. For example, 1 of the fiscal year 2014 candidate projects recommended to the Assistant Secretary—a health care sector project in New Jersey—had a score of seven. By contrast, 3 other potential candidates—1 food and agriculture sector project in Pennsylvania, a transportation sector project in South Carolina, and a lifeline sector project in the U.S. Virgin Islands—each scored an eight, and none were recommended to the Assistant Secretary for selection.

Although PSCD officials did not provide documentation, PSCD officials explained that there can be a variety of reasons why they recommend that the Assistant Secretary select 1 RRAP project over another— including geographic and sector diversity, IP’s strategic priorities, and the availability of PSCD resources. Additionally, PSCD officials provided examples of why some projects were recommended over others. For example, PSCD officials told us that one PSA had submitted three separate proposals, all of which received scores of seven or above, but PSCD recommended only one of the three for selection by the Assistant Secretary because a PSA can participate in only one RRAP at a time. In another case, PSCD officials told us that an international partner for a cross-border transportation project could not participate because of resource constraints. However, without documentation, we were unable to determine why PSCD recommended 1 project candidate that scored a seven over the 3 other potential candidates that scored an eight.

Standards for Internal Control in the Federal Government states that all transactions and significant events should be promptly recorded to maintain their relevance and value to management in controlling operations and making decisions. The standards further call for all transactions and significant events to be clearly documented, and readily available for examination to inform decision making. Recording and documenting key decisions are among the suite of control activities that are an essential part of an agency’s planning, implementing, and reviewing, and they are essential for proper stewardship and accountability for government resources and achieving efficient and effective program results. Documenting the rationale for making project selections would provide DHS managers and others responsible for overseeing the program valuable insights into why 1 RRAP project was selected over another, particularly among proposals with the same score that appear equally feasible and worthy. DHS officials agreed that maintaining this documentation could be used to support the recommendations and help answer any potential questions about final project selections.

Maintaining documentation about reasons why projects were or were not selected would also provide DHS a basis for defending its selections or responding to queries about them, particularly given the desirability of the program among the states and budgetary constraints facing states and other potential RRAP stakeholders. Regarding the budgetary constraints, states or other stakeholders, such as local, tribal, or territorial government entities, might be interested in knowing why a RRAP project proposal was not selected so that they could make decisions about (1) whether they need to dedicate additional resources to refining a RRAP proposal for future years, or (2) adjust the scope of their involvement in a future RRAP based on anticipated budgetary resource increases or constraints. With documentation on why projects were or were not recommended and selected, DHS would be better positioned to respond to queries about project selections from potential RRAP stakeholders, particularly if senior managers or staff currently involved in the program move to other positions and new managers or staff do not have records about key decisions.

DHS Has Taken Action to Work with States to Improve the RRAP Process, and Has Begun to Engage CI Partners to Ascertain Their Resilience Information Needs

Since 2011, IP has worked with states to improve the RRAP process, and IP officials said these efforts are viewed favorably by primary stakeholders. IP shares the project results of each RRAP with the primary stakeholder, and each report is generally available to IP staff, including PSAs and SSAs within IP, but IP does not share individual reports with others, including other primary stakeholders and SSAs outside of DHS. According to IP officials, IP has begun to conceptualize how it can develop a resilience product or products using multiple sources— including RRAP reports—to distribute to CI partners, and is using various forums to solicit input from CI partners to gauge their resilience information needs.

DHS Has Responded to Concerns Raised by States and Worked with the States to Improve the RRAP Process

In May 2011, SLTTGCC expressed concerns about states being selected to conduct a RRAP project before first being provided information on the time, cost, and scope of conducting a RRAP project. SLTTGCC established its RRAP Working Group in September 2011 in response to states’ experiences participating in the RRAP in fiscal years 2009 and 2010, with the goal to help ensure that other states had better experiences with DHS in future RRAP projects. In addition, some RRAP project participants we interviewed told us that maintaining the RRAP project schedule had been a challenge. Specifically, officials representing 5 of the 10 primary stakeholders we contacted in locations where RRAP projects had taken place from fiscal years 2009 through 2011 told us that they had encountered challenges completing RRAP projects within a specific time frame. Moreover, 12 of the 20 PSAs we contacted agreed that it was challenging to schedule meetings, such as kickoff meetings that required all key stakeholders to be in the same room during the meetings. Six of these PSAs also said it was challenging to get all required surveys and assessments completed in the short (usually 2 months) data-gathering period.

IP officials told us that they took actions to address these challenges by improving communication with participants about the scope of RRAP projects before they were selected and while projects were ongoing. These officials stated that this included setting expectations early on to inform stakeholders when particular RRAP events are scheduled to occur, including scheduling vulnerability assessments, and group discussions among the various stakeholders participating in the RRAP. Officials representing two of the four primary stakeholders that participated in the fiscal year 2009 or 2010 RRAP projects and were active in SLTTGCC stated that they believed IP has improved the conduct of later projects. One of these state officials said including states in the proposal development process and helping states to understand the time, costs, and benefit of the RRAP project prior to initiating the project made the execution of RRAP projects go more smoothly. IP officials told us that that they have since received positive feedback from the states regarding these changes, and our discussions with a representative of SLTTGCC confirmed that they believe that DHS’s revised proposal development process had been beneficial to them.

DHS Shares RRAP Project Results with Primary Stakeholders and Some Partners, but Relies on the Primary Stakeholder for Broader Distribution

IP shares individual RRAP reports with the primary stakeholders— officials representing the state where the RRAP was conducted—but has generally limited the distribution of the reports to those officials. According to IP, individual RRAP project reports are provided directly to primary stakeholders. PSAs and others that have access to the IP Gateway may also view RRAP reports. When the RRAP report contains Protected Critical Infrastructure Information (PCII), distribution and access to those reports is limited to individuals that are authorized to receive such information. Upon the request of a primary stakeholder, IP will also prepare For Official Use Only (FOUO) versions of RRAP reports—which, although sensitive, may be shared with a broader audience than PCII When this occurs, IP versions—to share with primary stakeholders.develops FOUO and PCII versions of RRAP reports—and primary stakeholders can share FOUO results with whomever they deem appropriate or necessary, including other RRAP participants. Otherwise, to share information within PCII reports, states would need to identify the FOUO information within the PCII report or request that IP clear the recipient for access to PCII information. During our review, 13 of 17 RRAP projects had both PCII and FOUO versions of RRAP reports, the other 4 projects had PCII versions only. IP officials told us that state officials can share FOUO versions of RRAP reports more readily than PCII versions of the reports. Furthermore, PSAs told us they share RRAP-derived information with CI partners—both those who participated in the RRAP and those who did not—during the course of their PSA duties as appropriate. IP officials told us that they do not distribute non- PCII versions of RRAP reports more broadly because the individual state is the primary stakeholder for a particular RRAP report. They said that they consider the state to be the owner of the information and believe that any party who wants the information should go to the state. Officials said they provide point-of-contact information for the primary stakeholder of a particular RRAP project to those who want to request a RRAP report from that primary stakeholder.

IP does not proactively distribute RRAP reports to SSAs whose sectors are the focus of the RRAP project. Officials representing the eight of nine SSAs we contacted told us they do not generally receive RRAP reports and may be unaware the reports exist. Representatives of two SSAs stated that they did not know about the existence of certain RRAP reports for their sector, and officials representing two others told us they made multiple requests before receiving RRAP reports from DHS. IP officials stated that SSAs should be able to receive a copy of any RRAP report in which they participated and stated that it was possible that we did not speak to the appropriate SSA representatives—those that participated in the RRAP projects. IP officials also stated that RRAP reports are on the IP Gateway and IP SSAs—chemical, commercial facilities, critical manufacturing, dams, emergency services, and nuclear sectors—have access to these reports, but other SSAs may have to make specific requests to IP or the primary stakeholder in order to receive the RRAP reports because not all of these SSAs have access to the IP Gateway and PCII information. IP officials told us that they intend to share a FOUO copy of a RRAP report on regional energy pipelines with the non-IP SSAs who participated. IP officials stated that the regional energy pipelines RRAP project is not expected to be completed until the latter part of 2013.

DHS Plans to Develop an Approach to Share Resilience Information and Has Engaged CI Partners to Ascertain Their Information Needs

IP is in the early stages of developing an approach—either a product or a series of products—to share resilience-related lessons learned, but plans are in the early concept stage and few specifics are available regarding the contents of these products. According to IP officials, the planned product or products are not to be limited to RRAP project data or findings. Rather they will leverage RRAP data and common observations or findings; data from security surveys and vulnerability assessments done at individual assets or facilities; and open source information to communicate collective results, lessons learned, and best practices that can contribute to ongoing local, state, regional, and national efforts to strengthen the resilience of critical infrastructure systems. IP officials anticipate that the first product, or products, will be available for distribution before the end of fiscal year 2013.

With regard to the planned resilience product(s), IP officials cautioned that (1) this effort is in the conceptual stage, (2) DHS has not approved funding for the product(s), and (3) the product or products are not expected to be ready for distribution until later this year at the earliest. IP officials further stated that it is too early to determine whether this approach will be an effective means to share resilience information across the spectrum of CI partners, to include states and SSAs. Nonetheless, IP officials told us that they engage CI partners, such as SLTTGCC’s RRAP and information-sharing working groups on resilience and information sharing, and during their participation in sector agency meetings and private sector coordination council meetings where, according to officials, the views of SSAs and CI owners and operators are discussed. For example, IP officials said they have had specific discussions with CI partners concerning state resilience information needs, and they are considering this input as they begin to develop a resilience product or products. They said that they also are considering feedback on information needs that they receive at regional conferences attended by various CI partners, and during daily PSA contacts in the field, primarily with CI owners and operators.

IP’s efforts to solicit feedback from CI partners during development of any resilience information-sharing product or products is consistent with the NIPP, which states that when the government is provided with an understanding of information needs, it can adjust its information collection, analysis, synthesis, and sharing accordingly. Through outreach and engagement with CI partners, DHS should be better positioned to understand their needs for information about resilience practices. It also helps DHS clarify the scope of work needed to develop a meaningful resilience information-sharing product or products that are useful across sectors and assets, and ascertain how the information can best be disseminated to the various CI partners—issues that could be critical given current budgetary constraints and uncertainty over the availability of resources.

DHS Gathers Facility Data, but Faces Challenges Measuring Results of RRAP Projects

PSCD uses follow-up surveys at facilities that have undergone vulnerability assessments and security surveys, including those that participate in RRAP projects, and has initiated a broad data-gathering effort with its RRAP CI stakeholders to explore changes in diverse topics such as partnering and state actions based on RRAP participation. These are important steps to provide insight about RRAP projects, but PSCD faces challenges developing performance measures and is not positioned to gauge the RRAP’s impact on regional resilience.

DHS Gathers Data on Resilience Enhancements at Individual Facilities

According to the NIPP, the use of performance measures is a critical step in the risk management process to enable DHS to objectively and quantitatively assess improvement in CI protection and resilience at the sector and national levels. The NIPP states that the use of performance metrics provides a basis for DHS to establish accountability, document actual performance, promote effective management, and provide a feedback mechanism to decision makers.

IP gathers data from individual facilities, including those that participated in RRAP projects, with the intent of measuring the efforts of those facilities to make enhancements arising out of security surveys and vulnerability assessments performed during RRAP projects. As discussed earlier, PSAs support the development of the national risk picture by conducting vulnerability assessments and security surveys to identify security gaps and potential vulnerabilities in the nation’s most critical infrastructure. PSAs perform these surveys and assessments at individual assets and facilities, including those that participate in RRAP projects, across the 16 sectors. In January 2011, IP directed PSAs to follow up with security survey and vulnerability assessment participants to gather feedback on security and resilience enhancements at their facilities using standardized data collection tools. These follow-up tools were to be used by PSAs to ask asset representatives about enhancements in six general categories—information sharing, security management, security force, protective measures, physical security, and dependencies—and focused on changes made directly as a result of IP security surveys and vulnerability assessments.

According to IP officials, PSCD revised its security survey and vulnerability assessment in January 2013 to include additional resilience- related questions intended to focus on facility preparedness, mitigation measures, response capabilities, and recovery mechanisms among facilities that participated in a security survey or vulnerability assessment. In addition, officials said beginning after July 2013, facilities that received a survey or assessment using the revised resilience questions are also to receive a PSA follow-up visit that reflects those same updated questions. IP officials said that revisions to the follow-up tools will also reflect changes associated with security and resilience enhancements at the facility, distinguishing them as either security or resilience changes. Officials said security surveys and vulnerability assessments that were conducted on facilities in support of a RRAP project are noted as such in the IP Gateway, but there is no other additional or separate tracking for the purposes of performance metrics. Furthermore, officials said they continue to gather data on changes initiated at facilities that participated in the RRAP, but they believe it may not be possible to link any changes made at facilities to participation in the RRAP. They added that resilience improvements made at individual facilities do not necessarily address regional vulnerabilities identified in RRAP reports.

DHS Faces Challenges Measuring the Effect of RRAP Projects

IP has considered how it intends to measure results associated with RRAP projects—not just facilities within projects— but faces challenges doing so. In January 2012, IP developed a project management plan (PMP) intended to clarify planned performance metrics for IP’s vulnerability assessment programs, including the voluntary security surveys and vulnerability assessments performed during RRAP projects. The PMP stated that DHS planned to measure the impact of RRAP projects by conducting follow-up checks at RRAP facilities to see if these facilities or systems implemented changes that increased the resilience of the facility. The PMP set a goal of 20 percent of facilities making resilience improvements following a security survey or vulnerability assessment performed for RRAP projects for fiscal year 2013, rising to 50 percent of facilities by fiscal year 2017. The PMP stated that this facility information is to be used to compile resilience information for the region, but it did not explain how this information would be combined to measure regional resilience. In April 2013, IP officials told us that they no longer intended to use the performance targets contained in the PMP. IP officials explained that they believe that individual facility assessment follow-ups are not an effective means to measure the impact of a RRAP project. They said that RRAP findings are written for the primary stakeholder—the state and not the assessed facilities—and RRAP projects most often provide the analyses of larger regional issues rather than specific facility gaps.

Alternatively, PSCD officials stated that they have since developed the RRAP Findings Tracker to engage primary stakeholders about their efforts to address key findings resulting from individual RRAP projects. According to PSCD officials, in March 2013, the RRAP Findings Tracker was distributed to all PSAs who had conducted a RRAP project over the previous 3 years. PSAs were directed by IP to use the RRAP Findings Tracker to follow up with the state and other stakeholders on specific RRAP issues identified in those states. IP updates the tracker on a monthly basis and headquarters officials are to review the results every 6 months. The RRAP Findings Tracker is intended to cover, among other things: developments that demonstrate project relevance since the RRAP project was initiated, for instance, news reports, speeches, or studies that demonstrate the ongoing relevance of the project’s focus; partnership building and information sharing, to include developments that relate to how project stakeholders—whether state, regional, federal, or private sector—have enhanced interaction, awareness, communication, and information sharing; any action taken concerning the RRAP report’s key findings, particularly with regard to enhancement options specified in the RRAP report; and activities at specific individual assets assessed during the RRAP and their efforts to enhance resilience, including the percentage of assessed assets that have made an improvement or planned to make an improvement after 6 and 12 months.

PSCD officials said that they believe that by utilizing the information in its Findings Tracker, they would likely have greater insights into the extent that stakeholders take action following a RRAP project, such as the extent to which the project has improved communication among RRAP stakeholders. According to officials, in May 2013, they began having preliminary discussions about using the RRAP Findings Tracker as one input for developing possible metrics. They added that it would be would be premature for them to provide us with any of the preliminary draft ideas for metrics associated with this effort.

Nonetheless, IP officials also stated they face challenges measuring performance across facilities within a RRAP project, and from project to project. For example, IP officials told us that each RRAP project is difficult to measure because each focuses on unique assets within a unique geographic area or region. For example, our reviews of RRAP reports showed one RRAP project might focus on commercial facilities, such as stadiums and arenas in one urban area, while another project might focus on a shopping district or an urban mall in another. Similarly, a transportation RRAP project in one region may focus on roadways and bridges, while a project in a different region might focus on waterways. IP officials added that participation in a RRAP project is voluntary, as is participation in the completion of the RRAP Findings Tracker. Therefore, the ability to develop measures that represent assets in a region could hinge on the willingness of CI stakeholders, including facility owners and operators, to participate.

IP officials further explained that, given the diversity of assets and regions covered by individual RRAP projects, it could also be challenging to link key RRAP findings and subsequent actions within projects. For example, one RRAP project may identify a planning shortfall, leading to a resilience-enhancing option calling for the creation of a plan. If the affected stakeholder or stakeholders subsequently create such a plan, IP could note that an action or actions were taken toward addressing a key finding, but it would be unable to assess whether the plan addresses the key finding adequately until it was implemented and tested through an exercise or real-world emergency. Reaching that next step may take years, according to officials. Officials also stated that it might be difficult to develop measures of key findings across RRAP projects. Whereas a key finding of one RRAP project might focus on the development of a regional plan as discussed above, a key finding of another might focus on prioritizing the distribution on resources, such as fuel, to ensure that emergency services can remain viable during a hurricane or earthquake. A separate RRAP project might have a key finding that electrical power is provided by single supplier, leaving a region vulnerable to a single point of failure.

We recognize that developing performance measures among and across RRAP projects could be challenging moving forward. We further recognize that the information generated through the administration of the RRAP Findings Tracker with RRAP project primary stakeholders (e.g., states) may provide a foundation for DHS’s development of RRAP performance measures. However, DHS could better position itself to gain insights into a project’s effects if it were to develop a mechanism to assess whether changes made at individual facilities are linked to or influenced by participation in a RRAP project. One approach for doing so could entail IP revising its security survey and vulnerability assessment follow-up process at individual facilities, including follow-ups at facilities that participated in RRAP projects to gather and analyze data on the extent to which participation in a RRAP project influenced owners and operators to make related resilience enhancements. More specifically, IP officials stated earlier that they did believe it was possible to link security and resilience enhancements made at facilities that participated in RRAP projects to RRAP project participation. However, currently the PSA does not specifically ask facility owners and operators whether participation in the RRAP project influenced their enhancement decisions. Developing a mechanism—such as revising the security survey and vulnerability assessment follow-up tool—to ascertain whether changes made at individual facilities are linked to or influenced by findings in RRAP projects could provide IP valuable information on individual facility efforts to address key RRAP project findings and how any enhancements are linked to the RRAP project. Doing so would also enable IP to compare facilities that participated in a RRAP project with those that did not and provide a basis for assessing why RRAP participation may or may not have prompted changes at a facility, thereby providing a building block for measuring IP’s performance and insights into the effect a RRAP project may have on facility resilience. This would also be consistent with the NIPP, which states that the use of performance metrics provides a basis for DHS to establish accountability, document actual performance, promote effective management, and provide feedback to decision makers.

Gathering data on the extent to which participation in a RRAP project influenced facility enhancements might also provide DHS valuable information about the results of its efforts, consistent with the views of PSAs who coordinate RRAP projects among stakeholders in particular regions. For example, 6 of the 10 PSAs we interviewed who had participated in RRAP projects where RRAP reports were issued expressed the belief that facilities that participated in the RRAP are more likely to have made improvements that increased security or resilience than other facilities that were not part of a RRAP project, but had undergone a security survey or assessment. These PSAs said that they believed this would occur because facilities participating in RRAP projects are able to see how their own operations affect the security and resilience of other facilities within the region. IP officials stated that they agreed that understanding whether RRAP participation had an effect on whether enhancements were made at an individual facility could provide useful information to the program. By assessing the linkage between the actions of individual facilities and the results of a RRAP project, DHS would also have a basis to begin to explore the effect of a RRAP project on facility management and operations, especially since RRAP projects are intended to focus on dependencies and interdependencies among facilities in a particular region.

Conclusions

IP has taken important actions to standardize the selection process for RRAP project locations. It has also worked with state stakeholders to better communicate the scope of projects, consider how it can share resilience information with CI partners, and gather information on CI partner actions to enhance resilience after the RRAP project is completed. However, further actions could strengthen these endeavors. First, with regard to the process for selecting RRAP project locations, IP has developed criteria and a process for selecting project candidates, but it has not fully documented why some projects are recommended over others. Documenting why specific RRAP selections were or were not recommended would be consistent with Standards for Internal Control in the Federal Government, and would provide IP managers and others responsible for overseeing the program valuable insights into why one RRAP project was selected over another, particularly among proposals with the same score that appear equally feasible and worthy. Furthermore, maintaining documentation about reasons why projects were or were not recommended would also provide IP a basis for defending its selections or responding to queries about them, particularly given the desirability of the program among the states and budgetary constraints facing states and other potential RRAP stakeholders. With documentation on why projects were or were not recommended and selected, DHS would be better positioned to respond to queries about project selections from potential RRAP stakeholders, particularly if senior managers or staff currently involved in the program move to other positions and new managers or staff do not have records about key decisions.

Second, consistent with the NIPP, IP has taken action to establish an approach for conducting follow-up surveys at facilities that have undergone security surveys and vulnerability assessments—both those that participated in RRAP projects and those that did not—to document changes the facilities make that affect their resilience. Also, IP has taken preliminary steps, via its RRAP Findings Tracker, to gain insights into primary stakeholder efforts to enhance resilience in the regions where RRAP projects have been performed. We recognize that IP faces challenges developing performance measures to gauge results among and across RRAP projects; nevertheless, IP could benefit from assessing how participation in a RRAP project may or may not influence change. Specifically, although the RRAP Findings Tracker may provide a foundation for IP’s overall development of RRAP performance measures, IP could develop a mechanism to assess whether changes made at individual facilities are linked to or influenced by participation in a RRAP project. One such mechanism could entail IP revising its security survey and vulnerability assessment follow-up tool, which is used to query all facilities that have participated in these surveys and assessments— regardless of whether they participated in a RRAP project. Doing so would enable IP to compare the extent to which facilities that participated in a RRAP project made enhancements related to DHS security surveys and assessments with those that did not participate in a RRAP project. This comparison could serve as a building block for measuring IP’s efforts to conduct RRAP projects, thereby providing an avenue to use performance metrics to establish accountability, document actual performance, promote effective management, and provide feedback to decision makers as stated in the NIPP. It would also provide valuable insights on individual facility efforts to address key RRAP findings, and give IP a basis for determining how those finding may have affected facility resilience, particularly as it relates to facility dependence and interdependence.

Recommendations for Executive Action

To help ensure that DHS is taking steps to strengthen the management of RRAP projects and the program in general, we recommend that the Assistant Secretary for Infrastructure Protection, Department of Homeland Security, take the following two actions: document decisions made with regard to recommendations about individual RRAP projects to provide insights into why one project was recommended over another and assurance that recommendations among equally feasible proposals are defensible, and develop a mechanism to assess the extent to which individual projects influenced participants to make RRAP related enhancements, such as revising the security and vulnerability assessment follow-up tool to query facilities that participated in RRAP projects on the extent to which any resilience improvements made are due to participation in the RRAP.

Agency Comments and Our Evaluation

We provided a draft of this report to the Secretary of Homeland Security for review and comment. DHS provided written comments, which are summarized below and reprinted in appendix III. DHS agreed with both recommendations and discussed plans to address one of them. DHS also provided technical comments, which we incorporated as appropriate.

With regard to the first recommendation, that DHS document decisions made with regard to recommendations about individual projects, DHS concurred, stating that the Office of Infrastructure Protection (IP) will develop a mechanism to more comprehensively document the decision- making process and justifications that lead to the selection of each project. DHS stated that it estimates that it will complete this action as of September 30, 2014, for projects in the next RRAP cycle—that is, projects to be conducted in fiscal year 2015.

With regard to the second recommendation, that DHS develop a mechanism, such as revising the security survey and vulnerability assessment follow-up tool, to assess the extent to which individual projects influenced participants to make RRAP related enhancements, DHS also concurred. In its written comments, DHS agreed that it would be insightful to understand whether the implementation rate of security and resilience enhancements at facilities differs between those receiving an assessment as part of a RRAP, and those receiving an assessment unrelated to this program. After we provided a draft of this report to DHS for review and comment, IP officials raised concerns that the recommendation as originally worded did not provide them the flexibility they needed to consider multiple alternatives to gain insights about RRAP-related enhancements. For example, and as noted in the written comments, facilities participate in the RRAP in many ways and surveys and assessments are but one option offered to facilities in a focus area. While we continue to see benefits to revising the security survey and vulnerability assessment follow-up tool, as discussed in the report, we modified the recommendation to acknowledge IP’s concerns about considering other possible mechanisms. In its written comments, DHS stated that IP would review alternatives, including the one we discussed, and would provide additional details on how it will address this recommendation in DHS’s written statement of the actions taken on our recommendations 60 calendar days after the receipt of the final report.DHS stated that its estimated completion date for action on this recommendation is to be determined.

DHS also raised two concerns with the report. First, while concurring with our second recommendation, DHS stated that it is disappointed that the draft report did not have a more extensive discussion on the overall success and effectiveness of the RRAP to identify and address regional security and resilience gaps. DHS noted that since the RRAP’s inception, projects have been conducted in regions throughout the nation and have focused on sectors such as energy, transportation, commercial facilities, water, and food and agriculture. DHS stated that through the RRAP, DHS has provided unique technical expertise to its stakeholders that helps guide their strategic investments in equipment, planning, training, and resources to enhance the resilience and protection of facilities, surrounding communities, and entire regions. We believe that the report did address these issues sufficiently. As noted in the report, IP has taken important actions to (1) standardize the selection process for RRAP project locations, (2) work with state stakeholders to better communicate the scope of projects and consider how it can share resilience information with CI partners, and (3) gather information on CI partner actions to enhance resilience after the RRAP project is completed. Nonetheless, the NIPP states that the use of performance measures is a critical step in the risk management process to enable DHS to objectively and quantitatively assess improvements in CI protection and provides a basis for DHS to document actual performance, promote effective management, and provide a feedback mechanism to decision makers. As discussed in the report, developing performance measures among and across RRAP projects could be challenging moving forward, but, absent these measures, neither we nor DHS is positioned to report on the overall success and effectiveness of the program. Hence, we recommended the development of such a mechanism to assess RRAP-related enhancements.

Second, DHS stated that the draft report did not substantially discuss the significant evolution of the program from a 2009 pilot to a more mature program that is at the forefront of the evolving critical infrastructure security and resilience mission that is responsive to the needs of the federal government and its partners. We disagree and believe that the report sufficiently discusses the evolution of the program, particularly the evolution of DHS’s process for selecting project locations as well as changes DHS has made to address the concerns of stakeholders based on their early experiences with RRAP.

We are sending copies of this report to the Secretary of Homeland Security, the Under Secretary for the National Protection Programs Directorate, and interested congressional committees. In addition, this report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have questions about this report, please contact me at (202) 512-8777 or caldwells@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Critical Infrastructure Sectors

This appendix provides information on the 16 critical infrastructure (CI) sectors and the federal agencies responsible for sector security. The National Infrastructure Protection Plan (NIPP) outlines the roles and responsibilities of the Department of Homeland Security (DHS) and its partners—including other federal agencies. Within the NIPP framework, DHS is responsible for leading and coordinating the overall national effort to enhance protection via 16 critical infrastructure sectors. The NIPP and Presidential Decision Directive/PPD-21 assign responsibility for critical infrastructure sectors to sector-specific agencies (SSA). As an SSA, DHS has direct responsibility for leading, integrating, and coordinating efforts of sector partners to protect 10 of the 16 critical infrastructure sectors. The remaining six sectors are coordinated by seven other federal agencies. Table 2 lists the SSAs and their sectors.

Appendix II: Criteria for Fiscal Years 2013 and 2014 RRAP Projects

This appendix provides the criteria DHS’s Office of Infrastructure Protection (IP) uses to assess RRAP proposals for consideration for selection as RRAP projects. IP officials stated that the criteria were developed based on feedback received from infrastructure protection partners such as the State, Local, Tribal and Territorial Government Coordinating Council and from lessons learned conducting RRAP projects. IP officials said that they asked protective security advisors (PSA) and PSA regional directors who had previously conducted Regional Resilience Assessment Program (RRAP) projects to review the criteria before they were finalized to provide assurance that the criteria reflected lessons learned. As shown in table 3, our review of IP’s criteria shows that they generally focus on the feasibility of the overall proposed project; partnering, such as whether the project has clear sponsorship and willing participants; broad applicability, such as the potential to generate resilience-related findings that can be applied to other locations; and risk-based factors, including the concentration of critical infrastructure in the region and the likelihood that the project will produce resilience- related findings.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgment

GAO Contact

Staff Acknowledgments

In addition to the contact named above, John F. Mortin, Assistant Director, and Anthony J. DeFrank, Analyst-in-Charge, managed this assignment. Chuck Bausell, Orlando Copeland, Katherine M. Davis, Justin Dunleavy, Aryn Ehlow, Michele C. Fejfar, Eric Hauswirth, and Thomas F. Lombardi made significant contributions to the work.

Related GAO Products

Critical Infrastructure Protection: DHS List of Priority Assets Needs to Be Validated and Reported to Congress. GAO-13-296. Washington, D.C.: March 25, 2013.

Critical Infrastructure Protection: Preliminary Observations on DHS Efforts to Assess Chemical Security Risk and Gather Feedback on Facility Outreach. GAO-13-412T. Washington, D.C.: March 14, 2013.

Critical Infrastructure Protection: An Implementation Strategy Could Advance DHS’s Coordination of Resilience Efforts across Ports and Other Infrastructure. GAO-13-11. Washington, D.C.: October 25, 2012.

Critical Infrastructure Protection: Summary of DHS Actions to Better Manage Its Chemical Security Program. GAO-12-1044T. Washington, D.C.: September 20, 2012.

Critical Infrastructure Protection: DHS Is Taking Action to Better Manage Its Chemical Security Program, but It Is Too Early to Assess Results. GAO-12-567T. Washington, D.C.: September 11, 2012.

Critical Infrastructure: DHS Needs to Refocus Its Efforts to Lead the Government Facilities Sector. GAO-12-852. Washington, D.C.: August 13, 2012.

Critical Infrastructure Protection: DHS Is Taking Action to Better Manage Its Chemical Security Program, but It Is Too Early to Assess Results. GAO-12-515T. Washington, D.C.: July 26, 2012.

Critical Infrastructure Protection: DHS Could Better Manage Security Surveys and Vulnerability Assessments. GAO-12-378. Washington, D.C.: May 31, 2012.

Critical Infrastructure Protection: DHS Has Taken Action Designed to Identify and Address Overlaps and Gaps in Critical Infrastructure Security Activities. GAO-11-537R. Washington, D.C.: May 19, 2011.

Critical Infrastructure Protection: DHS Efforts to Assess and Promote Resiliency Are Evolving but Program Management Could Be Strengthened. GAO-10-772. Washington, D.C.: September 23, 2010.

Critical Infrastructure Protection: Update to National Infrastructure Protection Plan Includes Increased Emphasis on Risk Management and Resilience. GAO-10-296. Washington, D.C.: March 5, 2010.

The Department of Homeland Security’s (DHS) Critical Infrastructure Protection Cost-Benefit Report. GAO-09-654R. Washington, D.C.: June 26, 2009.

Information Technology: Federal Laws, Regulations, and Mandatory Standards to Securing Private Sector Information Technology Systems and Data in Critical Infrastructure Sectors. GAO-08-1075R. Washington, D.C.: September 16, 2008.

Risk Management: Strengthening the Use of Risk Management Principles in Homeland Security. GAO-08-904T. Washington, D.C.: June 25, 2008.

Critical Infrastructure: Sector Plans Complete and Sector Councils Evolving. GAO-07-1075T. Washington, D.C.: July 12, 2007.

Critical Infrastructure Protection: Sector Plans and Sector Councils Continue to Evolve. GAO-07-706R. Washington, D.C.: July 10, 2007.

Critical Infrastructure: Challenges Remain in Protecting Key Sectors. GAO-07-626T. Washington, D.C.: March 20, 2007.

Homeland Security: Progress Has Been Made to Address the Vulnerabilities Exposed by 9/11, but Continued Federal Action Is Needed to Further Mitigate Security Risks. GAO-07-375. Washington, D.C.: January 24, 2007.

Critical Infrastructure Protection: Progress Coordinating Government and Private Sector Efforts Varies by Sectors’ Characteristics. GAO-07-39. Washington, D.C.: October 16, 2006.

Information Sharing: DHS Should Take Steps to Encourage More Widespread Use of Its Program to Protect and Share Critical Infrastructure Information. GAO-06-383. Washington, D.C.: April 17, 2006.

Risk Management: Further Refinements Needed to Assess Risks and Prioritize Protective Measures at Ports and Other Critical Infrastructure. GAO-06-91. Washington, D.C.: December 15, 2005.