Critical Infrastructure Protection: DHS Could Better Manage Security Surveys and Vulnerability Assessments
Released: 2012-06-29T00:00:00

Letter

Background

The Homeland Security Act of 2002 created DHS and gave the department wide-ranging responsibilities for, among other things, leading and coordinating the overall national critical infrastructure protection effort. Homeland Security Presidential Directive (HSPD) 7 further defined critical infrastructure protection responsibilities for DHS and SSAs. HSPD-7 directed DHS to establish uniform policies, approaches, guidelines, and methodologies for integrating federal infrastructure protection and risk management activities within and across CIKR sectors. Various other statutes and directives provide specific legal authorities for both cross sector and sector-specific protection and resiliency programs. For example, the Public Health Security and Bioterrorism Preparedness and Response Act of 2002 was enacted to improve the ability of the United States to prevent, prepare for, and respond to acts of bioterrorism and other public health emergencies, and the Pandemic and All-Hazards Preparedness Act of 2006 addresses, among other things, public health security and all-hazards preparedness and response. Also, the Cyber Security Research and Development Act, enacted in January 2002, authorized funding through fiscal year 2007for the National Institute of Standards and Technology and the National Science Foundation to facilitate increased research and development for computer and network security and to support related research fellowships and training. CIKR protection issues are also covered under various presidential directives, including HSPD-5 and HSPD-8. HSPD-5 calls for coordination among all levels of government as well as between the government and the private sector for managing domestic incidents, and HSPD-8 establishes policies to strengthen national preparedness to prevent, detect, respond to, and recover from threatened or actual domestic terrorist attacks, major disasters, and other emergencies.directives are tied together as part of the national approach for CIKR protection through the unifying framework established in HSPD-7.

According to the NIPP, these separate authorities and NPPD’s IP is responsible for working with public and private sector CIKR partners in the 18 sectors and leads the coordinated national effort to mitigate risk to the nation’s CIKR through the development and implementation of CIKR protection and resilience programs. Using a sector partnership model, IP’s Partnership and Outreach Division works with sector representatives, including asset owners and operators, to develop, facilitate, and sustain strategic relationships and information sharing. IP’s Protective Security Coordination Division (PSCD) provides programs and initiatives to enhance CIKR protection and resilience and reduce risk associated with all-hazards incidents. In so doing, PSCD works with CIKR owners and operators and state and local responders to (1) assess vulnerabilities, interdependencies, capabilities, and incident consequences; (2) develop, implement, and provide national coordination for protective programs; and (3) facilitate CIKR response to and recovery from incidents. Related to these efforts, PSCD has deployed the aforementioned PSAs in 50 states and Puerto Rico, with deployment locations based on population density and major concentrations of CIKR. In these locations, PSAs are to act as the link between state, local, tribal, and territorial organizations and DHS infrastructure mission partners and are to assist with ongoing state and local CIKR security efforts by establishing and maintaining relationships with state, local, tribal, territorial, and private sector organizations; support the development of the national risk picture by conducting vulnerability and security assessments to identify security gaps and potential vulnerabilities in the nation’s most critical infrastructures; and share vulnerability information and protective measure suggestions with local partners and asset owners and operators.

Enhanced Critical Infrastructure Protection Security Surveys

As part of their ongoing activities, PSAs are responsible for promoting the ECIP Initiative. Launched in September 2007, the ECIP Initiative is a voluntary program focused on forming or maintaining partnerships between DHS and CIKR owners and operators of high-priority level 1 and level 2 assets and systems, as well as other assets of significant value. According to DHS guidance, PSAs are to schedule ECIP visits with owners and operators in their districts using lists of high-priority and other significant assets provided by PSCD each year, with visits to level 1 assets being the first priority, and visits to level 2 assets being the second priority. Visits to other significant assets are to receive subsequent priority based on various factors, including whether they are of significant value based on the direction of IP; have been identified by the state homeland security office; or represent a critical dependency associated with higher- priority assets already identified. If an asset owner or operator agrees to participate in an ECIP visit, PSAs are to meet with the owner or operator to assess overall site security, identify gaps, provide education on security, and promote communication and information sharing among asset owners and operators, DHS, and state governments.

One of the components of the ECIP Initiative is the security survey, formally called the Infrastructure Survey Tool, which a PSA can use to gather information on the asset’s current security posture and overall security awareness. If the asset owner or operator agrees to participate in the security survey, the PSA works with the owner or operator to apply the survey, which assesses more than 1,500 variables covering six major components—information sharing, security management, security force, protective measures, physical security, or dependencies—as well as 42 more specific subcomponents within those categories. For example, within the category “physical security” possible subcomponents include fences, gates, parking, lighting, and access control, among others. Once the survey is complete, the PSA submits the data to Argonne National Laboratory, which analyzes the data to produce protective measures index scores ranging from 0 (low protection) to 100 (high protection) for the entire asset and for each component of the survey. Argonne National Laboratory also uses the data to produce a “dashboard”—an interactive graphic tool that is provided to the asset owner or operator by the PSA. The dashboard displays the asset’s overall protective measures score, the score for each of the six major components, the mean protective measures score and major component scores for all like assets in the sector or subsector that have undergone a security survey, and high and low scores recorded for each component for all sector or subsector assets that have undergone a security survey. The asset score and the scores for other like assets show the asset owner or operator how the asset compares to similar assets in the sector. The asset owner can also use the dashboard to see the effect of making security upgrades to its asset. For example, if the dashboard shows a low score for physical security relative to those of other like assets, the owner or operator can add data on perimeter fencing to see how adding or improving a fence would increase the asset’s score, thereby bringing it more in line with those of other like assets. Figure 1 provides an example of the dashboard produced as a result of the security survey.

Related to these security surveys, DHS also produced, from calendar years 2009 through 2011, summaries of the results of the security surveys related to sector or subsector security postures, known as sector summaries. These sector summaries were provided directly to SSAs in 2009 and 2010, and according to program officials were made available to SSAs in 2011 for sectors upon request. Unlike the summaries in past years, the 2011 summaries also included an “options for consideration” section that identified specific protective measures that had been adopted by the top 20 percent of assets in the sector or subsector as measured by the overall protective measures score.

Site Assistance Visit Vulnerability Assessments

DHS also uses vulnerability assessments to identify security gaps and provide options for consideration to mitigate these identified gaps. These assessments are generally on-site, asset-specific assessments conducted at the request of asset owners and operators. As of September 30, 2011, DHS had conducted more than 1,500 vulnerability assessments. Generally, vulnerability assessments are conducted at individual assets by IP assessment teams in coordination with PSAs, SSAs, state and local government organizations (including law enforcement and emergency management officials), asset owners and operators, and the National Guard, which is engaged as part of a joint initiative between DHS and the National Guard Bureau (NGB). These assessment teams are staffed via an interagency agreement between DHS and NGB and include two national guardsmen—a physical security planner and a systems analyst, one of whom serves as the team lead. They may also be supplemented by contractor support or other federal personnel, such as PSAs or subject matter experts, when requested. Argonne National Laboratory staff then finalize the vulnerability assessment report—which includes options for consideration to increase an asset’s ability to detect and prevent terrorist attacks and mitigation options that address the identified vulnerabilities of the asset—and provide it to the PSA for delivery. The asset owners and operators that volunteer for the vulnerability assessments are the primary recipients of the analysis. The vulnerability assessment is developed using a questionnaire that focuses on various aspects of the security of an asset, such as vulnerabilities associated with access to asset air handling systems, physical security, and the ability to deter or withstand a blast or explosion. The vulnerability assessment report also contains a section called “options for consideration” where DHS makes suggestions to improve asset security or reduce identified vulnerabilities. For example, one vulnerability assessment report made suggestions to the asset owners or operators to explore the option of installing additional cameras to improve video surveillance in certain locations, install additional barriers to prevent vehicles from entering the facility at high speeds, and increase the training of its security staff.

DHS revised the vulnerability assessment methodology in 2010 to enhance the analytical capabilities of IP. According to DHS officials, vulnerability assessments developed prior to 2010 did not have a consistent approach for gathering data on assets and did not produce results that were comparable from asset to asset. They also did not incorporate an approach for assessing asset resilience. DHS reported that the revised vulnerability assessment is intended to incorporate about 75 percent of the questions currently asked during an ECIP security survey, including questions on resilience, to bring the tool more in line with the security survey. As a result, vulnerability assessments completed beginning in 2011 have the capability to produce a dashboard similar to that produced from security surveys. By revising the assessment methodology, DHS intends to ensure that the data collected during the vulnerability assessment can be compared within and across sectors and subsectors while still providing each asset an assessment specific to that asset, including options for consideration to reduce vulnerability.

Regional Resiliency Assessment Program

While not the focus of this review, DHS has developed the Regional Resiliency Assessment Program (RRAP) to assess vulnerability and risk associated with resiliency. The RRAP is an analysis of infrastructure “clusters,” regions, and systems in major metropolitan areas that uses security surveys and vulnerability assessments, along with other tools, in its analysis. The RRAP evaluates CIKR on a regional level to examine vulnerabilities, threats, and potential consequences from an all-hazards perspective to identify dependencies, interdependencies, cascading effects, resiliency characteristics, and gaps. The RRAP assessments are conducted by DHS officials, including PSAs in collaboration with SSAs; other federal officials; state, local, territorial, and tribal officials; and the private sector depending upon the sectors and assets selected as well as a resiliency subject matter expert(s). The results of the RRAP are to be used to enhance the overall security posture of the assets, surrounding communities, and the geographic region covered by the project and is shared with the state. According to DHS officials, the results of specific asset-level assessments conducted as part of the RRAP are made available to asset owners and operators and other partners (as appropriate), but the final analysis and report are delivered to the state where the RRAP occurred. Further, according to DHS, while it continues to perform surveys and assessments at individual assets, prioritizing efforts to focus on regional assessments allows DHS to continue to meet evolving threats and challenges.

DHS Faces Challenges Managing Its Efforts to Conduct Security Surveys and Vulnerability Assessments on High- Priority Assets

DHS conducted about 2,800 security surveys and vulnerability assessments during fiscal years 2009 through 2011. In so doing, DHS directed PSAs to contact owners and operators of high-priority assets to offer to conduct voluntary security surveys and vulnerability assessments at their assets and PSAs used these as part of their outreach efforts among these assets. However, DHS faces challenges tracking whether security surveys and vulnerability assessments have been performed at high-priority assets. Furthermore, DHS has not developed institutional performance goals that can be used to measure the extent to which owners and operators of high-priority assets participate in security surveys and vulnerability assessments. In addition, DHS is not positioned to assess why some high-priority asset owners and operators decline to participate in these voluntary surveys and assessments so that it can develop strategies for increasing participation.

Inconsistent Data Make It Challenging to Match Security Surveys and Vulnerability Assessments with High-Priority Assets

DHS is not positioned to track the extent to which it is conducting security surveys and vulnerability assessments on high-priority assets because of inconsistencies between the databases used to identify high-priority assets and to identify surveys and assessments completed. Consistent with the NIPP, DHS prioritizes the participation of high-priority assets in its voluntary security survey and vulnerability assessment programs and uses the NCIPP list of high-priority assets to guide its efforts. In February 2011, DHS issued guidance to PSAs that called for them to form partnerships with owners and operators of high-priority assets in their areas. Under the guidelines, PSAs are to use NCIPP lists of high- priority assets to identify and contact owners and operators of the these assets in their areas that could benefit from participation in the security surveys, for the purpose of reducing potential security vulnerabilities and identifying protective measures in place. PSAs are to conduct outreach directly by meeting with the asset owners and operators to provide information about DHS efforts to improve protection and resiliency, sharing information about how an asset owner or operator can request a vulnerability assessment, and offering to conduct a security survey. If the owner or operator agrees to a visit from the PSA, the PSA is to record the date of the visit, and if the owner or operator agrees to participate in a security survey or vulnerability assessment, the PSA is likewise to record the day the security survey or vulnerability assessment was conducted. DHS analysts are then required to record the data provided by the PSAs in DHS’s Link Encrypted Network System (LENS) database—DHS’s primary database for tracking efforts to promote and complete security surveys and annual assessments. According to DHS guidelines, these data are subject to weekly reviews to ensure that data recorded in LENS are accurate, consistent, and complete. Thus, data on each individual asset should be recorded so that asset sector, name, and physical address reflect a single asset in a specified location throughout the database. For example, according to the guidelines, asset names recorded in LENS should not be recorded with stray asterisks, other special characters, and notes, and to the extent possible, address fields, such as “St” should be captured as “Street.”

To determine how many of these activities have been conducted on high- priority assets, we used an automated statistical software program to compare data on security surveys and vulnerability assessments completed in DHS’s LENS database with data on high-priority assets on the NCIPP lists for fiscal years 2009 through 2011—the lists PSAs are to use to contact officials representing high-priority assets in their areas. Out of 2,195 security surveys and 655 vulnerability assessments conducted during fiscal years 2009 through 2011, we identified a total of 135 surveys and 44 vulnerability assessments that matched assets on the NCIPP lists of high-priority assets. We also identified an additional 106 security surveys and 23 vulnerability assessments that were potential matches with assets on the NCIPP lists of priority assets, but we could not be certain that the assets were the same because of inconsistencies in the way the data were recorded in the two different databases. For example, we found instances where assets that appeared to be the same company or organization were listed in different sectors. We also encountered instances where names of companies at the same address did not match exactly or where companies with the same names had slightly different addresses in the two databases. For example, an asset at 12345 Main Street in Anytown, USA, might appear as ABC Company on one list and ABC on another. Conversely, we also found instances where company names appeared to be the same or similar on both lists, but they were listed at different street addresses or on different streets. In this case, for example, ABC Company might appear as being located on Main Street on one list, and E. Main St. on another.

We contacted DHS officials responsible for maintaining the LENS database and the NCIPP list and told them that we had encountered difficulty matching company names and addresses in the two lists. We explained that our results depended on an asset being described in a similar manner—same name, same address, same sector—in both the NCIPP and LENS databases. These officials acknowledged that the two databases do not match and explained that they have had to match the data manually because of the inconsistencies. Specifically, DHS reported that it reviewed over 10,000 records—including records of security surveys, vulnerability assessments, and the NCIPP lists for fiscal years 2009 through 2011—and manually matched assets that had participated in surveys or assessments with the NCIPP lists of high-priority assets using DHS officials’ knowledge of the assets. Based on its efforts, DHS analysts provided a table that showed that DHS conducted 2,128 security surveys and 652 vulnerability assessments, of which it identified 674 surveys and 173 assessments that were conducted on high-priority assets. Thus, by manually matching assets across the two lists, DHS was able to show that the percentage of high-priority assets surveyed and assessed increased significantly. Table 1 illustrates the results of our efforts to match the data using an automated software program and the results of DHS’s efforts to manually match the data.

DHS officials noted that beginning with the fiscal year 2012 NCIPP lists, they have begun to apply unique numerical identifiers to each asset listed in LENS and the NCIPP lists. According to these officials, once a match is made, the application of unique identifiers to the same assets in both databases is intended to remove uncertainty about which asset is which, regardless of variations in the name or address of the asset. Related to this, DHS officials also said that they have initiated a quality assurance process whereby they use descriptive data—such as geographic coordinates (longitude and latitude)—to verify street addresses and names, thereby giving IP the ability to more readily make matches in those instances where it may have previously experienced difficulty doing so. Nonetheless, they said that the NCIPP list continues to present matching challenges because there have been “significant” changes in the NCIPP list from year to year, but they anticipate fewer changes in the future. Most recently, the format and the organization of the list has changed to focus on clusters—groups of related assets that can be disrupted through a single natural or man-made hazard, excluding the use of weapons of mass destruction—rather than on individual assets. Thus, some assets previously considered high priority as stand-alone assets are now listed as part of a system or as clusters that in and of themselves are no longer considered high priority. According to DHS officials, the introduction of clusters has resulted in other data matching challenges, including the duplicate entry of an NCIPP asset that spans two states; multiple entries for a single asset that is listed both individually and in relation to a cluster or a system, and multiple entries for a single asset within several clusters or systems. DHS officials added that with the assignment of the unique identifier, they expect to be better positioned to cross-reference their program activities with the NCIPP list.

DHS officials have stated that the discrepancies between our analyses and the analysis performed by IP, as well as the confusion created by factors such as changing data sets, made it clear that improvements should be made in the collection and organization of the data. Accordingly, DHS officials said that they are continuing to work with various partners within DHS and its contractors to streamline and better organize the list of high-priority assets and data associated with assessments, surveys, and other IP field activities. However, DHS did not provide milestones and time frames for completing these efforts. DHS appears to be heading in the right direction in taking actions to resolve many of the issues we identified with regard to matching data and data inconsistencies. However, moving forward, DHS would be better positioned if it were to develop milestones and time frames for its plans to accomplish these tasks. Standard practices for project management state that managing a project involves, among other things, developing a timeline with milestone dates to identify points throughout the project to reassess efforts under way to determine whether project changes are necessary. By developing time frames and milestones for streamlining and organizing the lists of high-priority assets and data associated with surveys, assessments, and field activities, DHS would be better positioned to provide a more complete picture of its approach for developing and completing these tasks. It also would provide DHS managers and other decision makers with insights into (1) IP’s overall progress in completing these tasks and (2) a basis for determining what, if any, additional actions need to be taken.

DHS’s Efforts to Measure Progress Could Be Enhanced with Realistic Institutional Goals

As DHS moves forward to improve its efforts to track the hundreds of security surveys and vulnerability assessments it performs each year, DHS could also better position itself to measure its progress in conducting these surveys and assessments at high-priority assets. We have previously reported that to efficiently and effectively operate, manage, and oversee programs and activities, agencies need reliable information during their planning efforts to set realistic goals and later, as programs are being implemented, to gauge their progress toward achieving those goals. In July 2011, the PSCD Deputy Director told us that PSCD had a goal that 50 percent of the security surveys and vulnerability assessments conducted each year be on high-priority assets. However, this goal was not documented; PSCD did not have written goals and the results to date indicate that this goal was not realistic. Specifically, according to DHS’s 2010 NAR, less than 40 percent (299 of 763) of security surveys were conducted on high-priority assets from May 1, 2009, through April 30, 2010. For the same time period, DHS’s NAR reported that about 33 percent (69 of 212) of vulnerability assessments were conducted on high- priority assets. Setting institutional realistic goals for the number of security surveys and vulnerability assessments conducted at high-priority assets—consistent with DHS’s efforts to improve its data on these assets—would enable DHS to better measure its performance and assess the state of security and resiliency at high-priority facilities, across the 18 sectors, over time. For example, if there is a high-priority list consisting of 2,000 facilities, a DHS goal of 500 security surveys and vulnerability assessments conducted on high-priority facilities annually would allow for the potential assessment of all high-priority facilities over a defined period of time. Therefore, DHS could be in a better position to identify security strengths and weaknesses at high-priority facilities and within and across sectors and target areas for improvement.

Consistent with HSPD-7, DHS pursues a voluntary approach to critical infrastructure protection and coordination.

DHS officials told us that many of these assets do not receive voluntary surveys and assessments conducted by PSCD. Rather, as we previously reported, PSCD staff told us that they work with the responsible federal entity, such as the U.S. Coast Guard and the Nuclear Regulatory Commission, to identify and address vulnerabilities. Finally, according to the PSCD Deputy Director, shifting priorities based on terrorist threat information, budget constraints, and other department wide priorities, affect the prioritization and distribution of assets participating in these voluntary programs. For example, DHS officials stated that given DHS is placing increased emphasis on regional activities, such as RRAPs, voluntary surveys and assessments are not necessarily focused on individual high-priority assets. They said that expanded focus on regional activities enables IP to meet evolving threats and challenges, but in a budget constrained environment, forces them to prioritize activities so that they can leverage existing resources.

See GAO/AIMD-00-21.3.1. Standards for Internal Control in the Federal Government also calls for accurate and timely recording of information and periodic record reviews to help reduce the risk of errors. DHS officials told us that they conduct data quality checks and DHS guidelines direct such actions. However, the extent to which data were inconsistent indicates that information was not always accurately captured. process to enable DHS to objectively and quantitatively assess improvement in CIKR protection and resiliency. Specifically, the NIPP states that performance metrics allow NIPP partners to track progress against these priorities and provide a basis for DHS to establish accountability, document actual performance, promote effective management, and provide a feedback mechanism to decision makers. Consistent with the NIPP risk management framework, our past work has shown that leading organizations strive to align their activities to achieve mission-related goals. By using LENS and NCIPP data to establish performance goals, DHS could also be better positioned to identify gaps between expected and actual participation, track progress in achieving higher levels of participation, and ultimately gauge the extent to which protection and resiliency are enhanced for the nation’s most critical assets. Relying on institutional goals rather than informal goals would also provide assurance that DHS has a common framework for measuring performance in the face of organizational or personnel changes over time.

DHS Tracks Whether Owners and Operators Decline Surveys and Assessments but Does Not Track Reasons Why They Decline

DHS guidelines issued in February 2011 call for PSAs to document the names and addresses of CIKR asset owners or operators that decline to participate in security survey outreach activities as well as the reasons they declined. DHS officials told us that currently they track aggregate data on declinations but they do not document the reasons why asset owners and operators decline to participate in the security survey and vulnerability assessment programs. In November 2011, DHS provided a list of 69 asset owners or operators that PSAs recorded as having declined to participate in the security surveys from March 2009 through 2011, but these records did not identify reasons for the declinations. Program officials told us that the tool with which they collect declination information is not designed to capture such information. The Deputy Director for PSCD said that in 2012, DHS is developing a survey tool that PSAs can use to record why asset owners or operators decline to participate. Nonetheless, DHS could not provide specifics as to what would be included in the tool, which office would be responsible for implementing it, or time frames for its implementation. Rather, officials told us that they intend to use the results of our review to inform improvements to the process. Regarding vulnerability assessments, the assessment guidance is silent on whether PSAs are to discuss declinations with asset owners and operators and why they declined. However, PSCD issued guidance in January 2012 that states that the vulnerability assessment guidance is designed to complement the ECIP guidance issued in February 2011.

In our survey of PSAs, PSA respondents provided some anecdotal reasons as to why asset owners and operators may decline to participate. For example, when asked how often they had heard various responses from asset owners and operators that declined to participate in security surveys or vulnerability assessments, PSAs responded that reasons for declinations can include (1) the asset was already subject to federal or state regulation or inspections, (2) the identification of security gaps could render the owner of the asset liable for damages should an incident occur, or (3) assets owner or operator had concerns that the information it provides will not be properly safeguarded by DHS. Figure 2 shows the frequencies of PSA responses of either “often” or “sometimes” to our survey question about the various reasons for declinations that they have heard. Appendix III shows the results of our survey in greater detail.

While these PSA perceptions may reflect some reasons asset owners and operators decline to participate, it is important that DHS systematically identify reasons why high-priority asset owners and operators may decline to participate, especially if reasons differ from PSA region to PSA region or by sector or subsector. By doing so, DHS may be able to assess which declinations are within DHS’s ability to control or influence and strategize how the security survey and vulnerability assessment program and DHS’s approach toward promoting it can be modified to overcome any barriers identified. For example, 39 percent (31 of 80) of the PSAs who responded to our survey suggested that senior- level partners, including senior leaders within DHS, could better support the promotion of the security survey program when those leaders interact with CIKR partners at high-level meetings. According to DHS, NPPD and IP officials meet often with nonfederal security partners, including sector coordinating councils (SCC), industry trade associations, state and local agencies, and private companies, to discuss the security survey and vulnerability assessment and other programs to assist in educating mission partners about the suite of available IP tools and resources. Meeting with security partners to discuss IP’s surveys, assessments, and other programs is consistent with the NIPP partnership model whereby DHS officials in headquarters are to promote vulnerability assessments at high-level meetings where corporate owners are present—such as at SCC or Federal Senior Leadership Council meetings—and through the SSAs responsible for sector security. The NIPP also calls for DHS to rely on senior-level partners, such as the SCCs and state representatives, to create a coordinated national framework for CIKR protection and resilience within and across sectors and with industry representatives that includes the promotion of risk management activities, such as vulnerability assessments. Given the barriers to participation identified in our PSA survey, we contacted officials with 12 industry trade associations representing the water, commercial facilities, dams, and energy sectors to get their views on their awareness of DHS security surveys and vulnerability assessments. Officials representing 10 of the 12 trade associations said that they were aware of DHS’s voluntary survey and vulnerability assessment programs, but only 6 of 12 knew if some of their members’ had participated in these programs.

As noted earlier, at the time of our review DHS was not systematically collecting data on reasons why some owners and operators of high- priority assets decline to participate in security surveys or vulnerability assessments. Officials stated that they realize that some of the data necessary to best manage these programs are not currently being collected and said that one example is that PSAs are not consistently reporting assessment and survey declinations from assets. DHS officials added that in an effort to increase efficiency and accuracy, they are developing additional data protocols to ensure that all the applicable data are being collected and considered to provide a more holistic understanding of the programs. Given that DHS efforts are just beginning, however, it is too early to assess the extent to which they will address these data collection challenges. Nevertheless, by developing a mechanism to systematically collect data on the reasons for declinations, consistent with DHS guidelines, DHS could be better positioned to identify common trends for such declinations, determine what programmatic and strategic actions are needed to manage participation among high-priority assets, and develop action plans with time frames and milestones to serve as a road map for addressing any problems. This could enhance the overall protection and resilience of those high-priority CIKR assets crucial to national security, public health and safety, and the economy. Given that DHS officials recognize the need to collect these data to obtain a more holistic understanding of these programs, DHS could be better positioned if it had a plan, with time frames and milestones, for developing and implementing these protocols. Standard practices for project management state that managing a project involves, among other things, developing a plan with time frames and milestones to identify points throughout the process to reassess efforts under way to determine whether project changes are necessary. By having plan with time frames and milestones for developing additional data protocols, IP could be better positioned to provide a more complete picture of its effort to develop and complete this task. This could also provide DHS managers and other decision makers with (1) insights into IP’s overall progress and (2) a basis for determining what, if any, additional actions need to be taken.

DHS Shares Survey and Assessment Results with Asset Owners and Operators and Sector- Specific Agencies but Faces Challenges

DHS shares security survey and vulnerability assessment information with asset owners and operators that participate in these programs and shares aggregated sector information with SSAs. However, DHS faces challenges ensuring that this information is shared with asset owners and operators in a timely manner and in providing SSAs security survey- derived products that can help SSAs in their sector security roles. According to DHS officials, they are working to overcome these challenges, but it is unclear whether DHS actions will address SSA concerns about the use of aggregate security survey data.

Security Survey and Vulnerability Assessment Results Can Be Useful in Helping Asset Owners and Operators Reduce Risk

DHS security surveys and vulnerability assessments can provide valuable insights into the strengths and weaknesses of assets and can help asset owners and operators make decisions about investments to enhance security and resilience. For example, our survey of PSAs showed that most PSAs believe that the survey dashboard and the vulnerability assessment were moderately to very useful tools for reducing risk at CIKR assets. Specifically, 89 percent of PSAs (71 of 80) and 83 percent of PSAs (66 of 80) responded that the security surveys and vulnerability assessments, respectively, were moderately to very useful products for reducing risk. One PSA commented that “The dashboard is the first tool of its kind that allows the owner/operator a clear and measurable quantitative picture of existing security profile” while another commented that “ provide specific, actionable items for the owner/operator to take action on to decrease vulnerabilities.”

Our discussions with various CIKR stakeholders—specifically asset owners and operators and SSA representatives—also showed that these tools can be useful to the asset owners and operators that participate in these programs. As will be discussed later in greater detail, 6 of the 10 asset owners and operators we contacted used the results of these survey and assessment tools to support proposals for security changes at the assets that had been assessed. As one owner and operator said, these voluntary programs provide a fresh look at facility security from a holistic perspective. Another asset operator told us that it is nice to be able to see how its security practices compare to those of others within its sector. The representatives of the four SSAs we spoke with also believe the security survey and vulnerability assessments were beneficial to the asset owners and operators that received them.

DHS Is Not Delivering Results to Owners and Operators in a Timely Manner

The usefulness of security survey and vulnerability assessment results could be enhanced by the timely delivery of these products to the owners and operators that participated in them. For example, facility owners may not see the importance of an identified security weakness if they do not receive this information soon after a security survey or vulnerability assessment is completed. Furthermore, the inability to deliver results within the expected time frame could undermine the relationship DHS is attempting to develop with asset owners and operators. As mentioned earlier, PSAs rely on Argonne National Laboratory to provide them with the results of the vulnerability assessments, which PSAs, in turn, deliver directly to asset owners and operators. While PSAs find the voluntary programs useful, 14 percent of PSAs we surveyed (11 of 80) described late delivery of the reports as a factor that undermines the usefulness of vulnerability assessments. One PSA commented that “the program is broken in regard to timely completion of reports and deliverables (protective measures and resiliency dashboards) for the asset owners/operators. I have yet to receive anything from (a vulnerability assessment conducted several months ago). I have not even received the draft report for review nor the dashboard. This creates a big credibility problem for me with my stakeholders who are looking for the results.” The NIPP states that in order to have an effective environment for information sharing, CIKR partners need to be provided with timely and relevant information that they can use to make decisions. Consistent with the NIPP, DHS guidelines state that PSAs are to provide the results of security surveys in the form of a survey dashboard within 30 days of when the security survey was completed. In addition, according to PSCD officials, although there is no written guidance, PSCD expects that vulnerability assessment results are to be provided to assets within 60 days of completion of the vulnerability assessment.

We analyzed DHS LENS data to determine the extent to which survey dashboards were delivered to asset owners and operators on a timely basis, using DHS’s 30-day criteria for timeliness. Our analysis showed that for fiscal year 2011, more than half of all dashboards and vulnerability assessment reports were delivered to owners and operators late. Specifically, of the 570 dashboard reports that were supposed to be delivered during fiscal year 2011, about 24 percent (139 of 570) were delivered on time and approximately 60 percent (344 of 570) were late, with almost half of those delivered 30 days beyond the 30-day deadline established by DHS guidelines. Data were missing for about 15 percent (85 of 570) of the remaining dashboards.of dashboard deliveries for all security surveys conducted in fiscal year 2011.

DHS Could Better Measure Security Survey and Vulnerability Assessment Results to Improve Program Management

DHS has taken actions to determine whether asset owners or operators have made security improvements based on the results of security surveys. However, DHS has not developed an overall approach to determine (1) the extent to which changes have enhanced asset protection and resilience over time or (2) why asset owners and operators do not make enhancements that would help mitigate vulnerabilities identified during security surveys and vulnerability assessments. As a result, DHS may be overlooking an opportunity to make improvements in the management of its voluntary risk mitigation programs that could also help DHS work with asset owners and operators to improve security and resilience. According to DHS, moving forward, it may consider changes to the types of information gathered as part of its effort to measure improvements, but it has not considered what additional information, if any, should be gathered from asset owners or operators that participate in security surveys and vulnerability assessments.

DHS Has Implemented a Security Survey Follow-up Tool

According to the NIPP, the use of performance measures is a critical step in the risk management process to enable DHS to objectively and quantitatively assess improvement in CIKR protection and resiliency at the sector and national levels. The NIPP states that the use of performance metrics provides a basis for DHS to establish accountability, document actual performance, promote effective management, and provide a feedback mechanism to decision makers. Consistent with the NIPP, DHS has taken action to follow up with security survey participants to gather feedback from asset owners and operators that participated in the program regarding the effect these programs have had on asset security using a standardized data collection tool, hereafter referred to as the follow-up tool or tool.

DHS first began to do follow-ups with asset owners and operators in May 2010 but suspended its follow-up activities shortly thereafter to make enhancements to the tool it used. In January 2011, IP introduced its revised follow-up tool, which was to be used by PSAs to ask asset representatives whose assets had undergone a security survey and received a dashboard about enhancements made in six general categories—information sharing, security management, security force, protective measures, physical security, and dependencies. Whereas the original follow-up tool focused on changes asset owners and operators made to enhance security and resilience, the revised tool focused on changes that were made directly as a result of DHS security surveys. According to DHS guidance, the tool was to be used 180 days after the completion of a security survey at an asset. The tool, which directs PSAs to ask a series of questions about improvements made as a result of the survey, instructs PSAs to request information on specific enhancements within those categories that were discussed in the dashboard provided to the asset owners and operators. For example, within the physical security category, the tool instructs the PSAs to ask about any enhancements to things like fences, gates, parking, lighting, and access control, among others, and to ask asset owners or operators whether an identified change was made as a result of the security survey the asset had received. In February 2011, shortly after the revised tool was introduced, IP issued guidelines that instructed PSAs to implement the follow-up tool. According to IP officials, PSAs used the tool to follow up with owners and operators of 610 assets from January 2011 through September 2011. Data provided by IP showed that about 21 percent (126 of the 610) of the respondents to the PSA follow-ups reported that they had completed improvements, and 81 percent of these (102 of 126) reported that those improvements were implemented as the result of the security survey the asset received. According to IP’s data, the most common types of improvements identified by assets that had completed improvements since receiving the security survey were changes to information sharing, which could include activities such as participating in working groups, and physical security.

Follow-up Data on Results Do Not Reflect Long-Term Program Performance

DHS guidance states that PSAs are to conduct a follow-up with the asset owners and operators 180 days after an asset receives a security survey. We compared DHS data on 522 security surveys conducted from July 1, 2010, through March 31, 2011, with DHS data on the follow-ups performed from January 1, 2011, through September 30, 2011—180 days after DHS completed the security surveys. We found that DHS did not contact some asset owners or operators that should have received a 180- day follow-up and contacted some owners and operators that had participated in a security survey more than 180 days prior to the introduction of the tool. For example, of the 522 security survey participants that participated in a security survey from July 1, 2010, through March 31, 2011, 208 (40 percent) received the 180-day follow-up and 314 (60 percent) did not. Furthermore, DHS recorded an additional 402 follow-ups on assets that had received their security survey more than 180 days prior to the introduction of the tool. Thus, the data DHS reported included improvements assets made beyond the 180-day scope of the follow-up tool, making it difficult to measure the effectiveness of the security survey in prompting enhancements within 180 days of the survey.

According to PSCD officials, there are two key reasons why DHS used the follow-up tool to capture data on changes made beyond 180 days. First, program officials said that completion of the 180-day follow-up depends upon the asset representative’s willingness to participate and availability to answer these questions. If the asset representative does not agree to participate, or neither the representative nor the PSA is available, the 180-day follow-up cannot be completed on schedule. However, when DHS provided the follow-up data in November 2011, officials said that they were not aware of any asset owners or operators that had refused to participate in the 180-day follow-up at that time. Second, program officials noted that the inclusion of assets that had received a security survey more than 180 days prior to the introduction of the revised follow-up tool occurred because they believed that it was necessary to capture data on as many assets as possible. They said that IP intends that follow-ups be completed as close to the 180-day mark as possible, but they believed it was important to initially document whether the security survey resulted in changes to security, regardless of when the change was made. IP officials further explained that they had developed a similar follow-up tool to capture data on enhancements resulting from vulnerability assessments. However, at the time of our review, results were not available from the vulnerability assessment follow-up tool, which was also implemented in January 2011 and was designed to capture data on enhancements made 365 days following the delivery of the vulnerability assessment report. Consistent with the security survey, DHS officials explained that the 365-day follow-up for vulnerability assessments was determined as a means to begin the process of collecting and assessing data on improvements being made as a result of the assessments. They added that as more data are collected, IP will review the information to determine if the follow-up visits for security surveys and vulnerability assessments should remain at 180 and 365 days, respectively, or be moved as a result of information collected from asset owners and operators. Nonetheless, DHS officials did not provide a road map with time frames and milestones showing when they planned to revisit the 180-day follow-up time frame or the intervals between follow-ups. Consistent with the standards for project management, by having a road map with time frames and milestones for revisiting these time frames, IP could be better positioned to provide a more complete picture of its overall progress making these decisions and a basis for determining what, if any, additional actions need to be taken or data inputs need to be made.

GAO/AIMD-00-21.3.1. especially true if asset owners and operators are implementing more complicated enhancements over a longer term because of the need to develop and fund plans for particular types of improvements. For example, gathering these data could help DHS measure not only what improvements asset operators are implementing, but also how long it takes to complete the planning phase of a security enhancement project and how this time frame might vary by the type of improvement.

Furthermore, while it is important to capture information about improvements made as a result of these activities over time, it is also important that DHS either capture the information within the prescribed times outlined in DHS guidance, adjust the time frames based on an analysis of data captured over time, or perform follow-ups at additional intervals beyond those initially performed. This would also be consistent with Standards for Internal Control in the Federal Government, which calls for the establishment and review of performance measures and indicators to monitor activities and top-level reviews by management to track major agency achievements and compare these with plans, goals, and objectives. By doing so, IP could be better positioned to document actual performance, promote effective management, provide a feedback mechanism to decision makers, and enhance overall accountability.

Additional Enhancements Could Better Position DHS to Measure Performance

According to DHS officials, moving forward, DHS may consider additional changes to its follow-up tool depending on the results they gather over time. The NIPP states that performance measures that focus on outputs, called output measures, such as whether an asset completes a security improvement, should track the progression of a task. The NIPP further states that outcome measures are to track progress toward an intended goal by beneficial results rather than level of activity. Our review of DHS’s approach for following up with assets that had undergone a security survey showed that PSAs were instructed to focus on security enhancements completed as result of the security survey, not enhancements that were planned or in process. Nonetheless, our review of DHS’s follow-up results for the period from January through September 2011 showed that DHS reported the following:

41 percent (250 of 610) of the owners and operators surveyed reported that security enhancements were either in process or planned and the results did not indicate whether these planned or in-process enhancements were attributable to DHS’s security survey at these assets.

After we discussed our observation with DHS officials, they informed us that they believe completed improvements are the best initial measurement of the impact of security surveys and vulnerability assessments. They added that other metrics can be added as the process matures and is refined. However, as of March 2012, DHS did not document whether planned or in-process improvements are the result of security surveys. Given that the NIPP calls for CIKR partners to measure performance in the context of the progression of the task, DHS could be missing an opportunity to measure performance associated with planned and in-process enhancements, especially if they are attributable to DHS efforts via security surveys and vulnerability assessments. DHS could also use this opportunity to consider how it can capture key information that could be used to understand why certain improvements were or were not made by assets owners and operators that have received surveys and assessments. For example, the follow-up tool could ask asset representatives what factors—such as cost, vulnerability, or perception of threat— influenced the decision to implement changes, either immediately or over time if they chose to make improvements; what factors—such as perception of risk, cost, or budget constraints— influenced an asset owner or operator to choose to not make any improvements; why were the improvements made chosen over other possible improvements, if improvements were made; and did the improvements, if any, involve the adoption of new or more cost-effective techniques that might be useful as an option for other owners and operators to consider as they explore the feasibility making improvements?

Understanding why an asset owner or operator chooses to make, or not make, improvements to its security is valuable information for understanding the obstacles asset owners or operators face when making security investments. For example, the cost of security upgrades can be a barrier to making enhancements. As one PSA who responded to our survey commented, “there is no requirement for the owner/operator to take action. They are left with making a “risk-reward” decision. Some see great value in making security upgrades, while others are less inclined to make improvements due to costs.” Likewise, one asset representative told us that security is one of the most important things to management until budget time. In this regard, a better understanding of the complexity of the security improvement decision at the asset could also help DHS better understand the constraints asset owners or operators face in making these decisions—information that could possibly help DHS determine how, if at all, to refine its security survey program to assist asset owners or operators in making these decisions. For example, the NIPP states that effective CIKR programs and strategies seek to use resources efficiently by focusing on actions that offer the greatest mitigation of risk for any given expenditure. Additional information on the cost of improvements made and the reasons why improvements were or were not made could also assist DHS in understanding the trade-offs asset owners and operators face when making decisions to address vulnerabilities identified as a result of DHS security surveys and enhancements. IP officials told us they are wary of attempting to gather too much information from asset representatives with the follow-up tool because of a concern that being too intrusive may damage the relationships that the PSAs have established with asset representatives. They said that gathering additional information is not as important as maintaining strong relationships with the asset representatives.

We recognize that DHS operates its security survey program in a voluntary environment and that DHS can only succeed at improving asset and sector security if asset owners and operators are willing to participate, consistent with DHS’s interest in maintaining good relationships with asset representatives. However, by gathering more information from assets that participate in these programs—particularly high-priority assets—DHS could be better positioned to measure the impact of its programs on critical infrastructure security at the sector and national levels. Moreover, by collecting and analyzing this type of information, DHS could be better informed in making decisions about whether adjustments to its voluntary programs are needed to make them more beneficial to CIKR assets—a factor which could help DHS further promote participation by asset owners and operators that may previously have been reluctant to participate in DHS security surveys and assessments. Having this type of information could also be important in light of DHS’s efforts to better understand interdependencies between assets via the RRAPs. For instance, by knowing what factors influence decisions to make an improvement, or not, at one asset or a group of assets, DHS could be better positioned to understand how that decision influences the security of other assets that are also part of the RRAP. As a result, DHS and PSAs could then be better positioned to work with owners and operators to mitigate any vulnerabilities arising out of these decisions. It could also help DHS develop and target strategies for addressing why certain enhancements were not made and ultimately put DHS in a better position to measure outcomes, rather than outputs, associated with its efforts to promote protection and resilience via its voluntary risk mitigation programs.

Conclusions

DHS has taken important actions to conduct voluntary CIKR security surveys and vulnerability assessments, provide information to CIKR stakeholders, and assess the effectiveness of security surveys and vulnerability assessments. However, further actions could enhance each of these endeavors and provide DHS managers the information they need to ensure that IP is taking appropriate steps toward completing them or making adjustments where needed. DHS has not institutionalized realistic goals that could help DHS measure the effects of its efforts to promote and conduct security surveys and vulnerability assessments among high- priority assets. By developing realistic institutional goals, DHS could, for example, better measure the effects of its efforts to promote and conduct security surveys and vulnerability assessments among high-priority assets. Further, developing a road map with milestones and time frames for (1) taking and completing actions needed to resolve issues associated with data inconsistencies and matching data on the list of high-priority assets with data used to track the conduct of security surveys and vulnerability assessments, (2) completing protocols to systematically collect data on the reasons why some owners and operators declined to participate in the voluntary surveys and assessments, and (3) improving the timely delivery of the results of security surveys and vulnerability assessments could better position DHS to target high-priority assets and provide them with the information they need to make decisions related to security and resiliency. Moreover, by revising its plans to include when and how SSAs will be engaged in designing, testing, and implementing the web-based tool, consistent with its recent efforts to coordinate with CIKR partners, DHS could be positioned to better understand and address their information needs.

Consistent with the NIPP, DHS is also continuing to take actions to follow up with asset owners and operators that have participated in security surveys and vulnerability assessments to gauge the extent to which these surveys and assessments have prompted owners and operators to improve security and resilience at their assets. DHS officials said that they intend to review the information it gathers from asset owners and operators to determine if the follow-up visits should remain at 180 days after DHS completed the security surveys. By establishing a road map with milestones and time frames for conducting this review, DHS would be better positioned to provide a picture of its overall progress in making these decisions and a basis for determining what, if any, additional actions need to be taken or data inputs need to be made and whether additional follow-ups are appropriate at intervals beyond the follow-ups initially performed. In addition, collecting detailed data on actions started and planned and, for example, why actions were not taken, could provide DHS valuable information on the decision-making process associated with making security enhancements and enable DHS to better understand what barriers owners and operators face in making improvements to the security of their assets.

Recommendations for Executive Action

To better ensure that DHS’s efforts to promote security surveys and vulnerability assessments among high-priority CIKR are aligned with institutional goals, that the information gathered through these surveys and assessments meet the needs of stakeholders, and that DHS is positioned to know how these surveys and assessments could be improved, we recommend that the Assistant Secretary for Infrastructure Protection, Department of Homeland Security, take the following seven actions: develop plans with milestones and time frames to resolve issues associated with data inconsistencies and matching data on the list of high-priority assets with data used to track the conduct of security surveys and vulnerability assessments; institutionalize realistic performance goals for appropriate levels of participation in security surveys and vulnerability assessments by high-priority assets to measure how well DHS is achieving its goals; design and implement a mechanism for systematically assessing why owners and operators of high-priority assets decline to participate and a develop a road map, with time frames and milestones, for completing this effort; develop time frames and specific milestones for managing DHS’s efforts to ensure the timely delivery of the results of security surveys and vulnerability assessments to asset owners and operators; revise its plans to include when and how SSAs will be engaged in designing, testing, and implementing DHS’s web-based tool to address and mitigate any SSA concerns that may arise before the tool is finalized; develop a road map with time frames and specific milestones for reviewing the information it gathers from asset owners and operators to determine if follow-up visits should remain at 180 days for security surveys and whether additional follow-ups are appropriate at intervals beyond the follow-ups initially performed; and consider the feasibility of expanding the follow-up program to gather and act upon data, as appropriate, on (1) security enhancements that are ongoing and planned that are attributable to DHS security surveys and vulnerability assessments and (2) factors, such as cost and perceptions of threat, that influence asset owner and operator decisions to make, or not make, enhancements based on the results of DHS security surveys and vulnerability assessments.

Agency Comments and Our Evaluation

We provided a draft of this report to the Secretary of Homeland Security for review and comment. In its written comments reprinted in appendix IV, DHS agreed with all seven of the recommendations; however, its implementation plans do not fully address two of these seven recommendations and it is unclear to what extent its plans will address two other recommendations.

With regard to the first recommendation that DHS develop plans to resolve issues associated with data inconsistencies between its databases, DHS stated its efforts to assign unique identifiers to assets on the high-priority list that have received security surveys and vulnerability assessments will make matching easier and that other quality assurance processes have been implemented to better verify individual asset data. We agree these are positive steps; however, to fully address the recommendation, we believe DHS should develop a plan with time frames and milestones that specify how the steps it says it is taking address the data inconsistencies we cited, and demonstrate the results—how many high-priority assets received security surveys, vulnerability assessments, or both in a given year—of that effort. By doing so, DHS would be better positioned to provide a more complete picture of its approach for developing and completing these tasks. It would also provide DHS managers and other decision makers with insights into (1) IP’s overall progress in completing these tasks and (2) a basis for determining what, if any, additional actions need to be taken.

With regard to the second recommendation that DHS institutionalize realistic performance goals for levels of participation in security surveys and vulnerability assessments by high-priority assets, DHS stated that the participation of high-priority assets continues to be a concern but reiterated its view that the voluntary nature of its programs and competing priorities makes setting goals for high-priority participation difficult. DHS stated that its fiscal year 2012 Project Management Plans for Protective Security Advisor and Vulnerability Assessment Projects established realistic goals concerning the total number of assessments to be conducted. However, they said these plans do not include goals for assessments performed at high-priority assets. Furthermore, DHS stated the shift in emphasis to regional resilience suggested metrics and goals intended to measure the participation of high-priority assets in vulnerability assessments and surveys may not be a strong or accurate indicator of the degree to which DHS is achieving its infrastructure protection and resilience goals. We agree that the voluntary nature of these programs and changing priorities make the process of setting goals difficult. However, the NIPP and DHS guidance emphasize the importance of high-priority participation in these programs, and DHS can take factors like the voluntary nature of the program and DHS’s shift toward regional resilience into account when setting realistic goals for the number of security surveys and vulnerability assessments it conducts at high-priority facilities. By establishing realistic performance goals for levels of participation by high priority assets, DHS would be better positioned to compare actual performance against expected results and develop strategies for overcoming differences or adjust its goals to more realistically reflect the challenges it faces.

With regard to the third recommendation that DHS design and implement a mechanism for systematically assessing why owners and operators of high priority assets decline to participate and develop a road map, with time frames and milestones, for completing this effort, DHS stated it recognizes that additional clarification and guidance are needed to ensure effective implementation of existing guidance. Specifically, DHS stated it will review and revise the guidance to (1) determine if revisions to the existing process are required and (2) develop supplementary guidance to aid PSAs in executing the process. DHS stated it will initiate this review in the fourth quarter of fiscal year 2012, after which time it will develop additional milestones for mechanism improvement. We believe that DHS’s proposed actions appear to be a step in the right direction, but it is too early to tell whether DHS’s actions will result in an improved mechanism for systematically assessing why owners and operators decline to participate.

Regarding the fourth recommendation to develop time frames and specific milestones for managing its efforts to improve the timely delivery of the results of security surveys and vulnerability assessments to asset owners and operators, DHS stated it is working with contractors and program staff to advance the processes and protocols governing the delivery of assessment and survey products to facilities. DHS also stated that it had begun a review of assessments lacking delivery dates in LENS and is working with PSAs to populate the missing information. In addition, DHS noted that its plan to transition to a web-based dashboard system will help mitigate the issue of timely report delivery by eliminating the need for in-person delivery of the dashboard product. However, DHS did not discuss time frames and milestones for completing these efforts. Thus, it is unclear to what extent DHS’s actions will fully address this recommendation. As noted in our report, developing time frames and milestones for completing improvements that govern the delivery of the results of surveys and assessments would provide insights into IP’s overall progress.

With regard to the fifth recommendation to revise its plans to include when and how SSAs will be engaged in designing, testing, and implementing DHS’s web-based tool, DHS stated that it is currently taking actions to develop and test a web-based dashboard tool for individual owners and operators, which is expected to be widely available in January 2013. DHS stated that it anticipates the development of a state and local “view,” or dashboard, following the successful deployment of the web-based owner and operator dashboards. Regarding SSAs, DHS stated that a concept for a sector-level view of assessment data has been proposed and that the requirements and feasibility of such a dashboard will be explored more fully following the completion of the state-level web- based dashboard. DHS noted that that IP will engage the SSAs to determine any associated requirements. DHS’s proposed actions appear to be a step in the right direction. However, given that the sector level view of assessment data is in the proposal stage and further action will be explored more fully after completion of the state level web-based dashboard, it is too early to tell when and how SSA’s will be engaged in designing, testing and implementing the web-based tool.

In response to the sixth recommendation to develop a road map with time frames and specific milestones to determine if follow-up visits should remain at 180 days for security surveys, and whether additional follow- ups are appropriate at intervals beyond the follow-ups initially performed, DHS stated it will analyze and compare security survey follow-up results in early calendar year 2013 to determine whether modifications are required. DHS also stated that given that the 365-day follow-up process went into effect in January 2011, the first follow-up evaluations of vulnerability assessments have only recently begun and IP will collect, at a minimum, 1 year of vulnerability assessment data. DHS said that IP intends to review the results for both the security survey 180-day follow- up and the 365-day follow-up in early calendar year 2013 to determine whether modifications to the follow-up intervals are required. DHS’s proposed actions are consistent with the intent of this recommendation.

In response to the seventh recommendation to consider the feasibility of gathering and acting upon additional data, where appropriate, on (1) ongoing or planned enhancements attributable to security surveys and assessments and (2) factors that influence asset owner and operator decisions to make or not make security enhancements, DHS stated that it collects information on ongoing or planned enhancements. However, as noted in the report, DHS does not collect information that would show whether these enhancements are attributable to security surveys and assessments. DHS also stated that IP will continue to work with Argonne National Laboratory and field personnel to determine the best method for collecting information related to those factors influencing an asset’s decision to implement or not implement a new protective measure or security enhancement. However, it is not clear to what extent DHS’s actions will fully address this recommendation because it did not discuss whether it will consider the feasibility of gathering data on whether ongoing or planned enhancements are attributable to security surveys and assessments or how it will act upon the data it currently gathers or plans to gather to, among other things, measure performance in the context of the progression of the task, consistent with the NIPP. By gathering and analyzing data on why an asset owner or operator chooses to make, or not make, improvement to security, DHS would be better positioned to understand the obstacles asset owners face when making investments.

DHS also provided technical comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Homeland Security, the Under Secretary for the National Protection Programs Directorate, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have questions about this report, please contact me at (202) 512-8777 or caldwells@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: Critical Infrastructure Sectors

This appendix provides information on the 18 critical infrastructure sectors and the federal agencies responsible for sector security. The National Infrastructure Protection Plan (NIPP) outlines the roles and responsibilities of the Department of Homeland Security (DHS) and its partners—including other federal agencies. Within the NIPP framework, DHS is responsible for leading and coordinating the overall national effort to enhance protection via 18 critical infrastructure and key resources (CIKR) sectors. Homeland Security Presidential Directive (HSPD) 7 and the NIPP assign responsibility for CIKR sectors to sector-specific agencies (SSA). As an SSA, DHS has direct responsibility for leading, integrating, and coordinating efforts of sector partners to protect 11 of the 18 CIKR sectors. The remaining sectors are coordinated by eight other federal agencies. Table 2 lists the SSAs and their sectors.

Appendix II: Objectives, Scope, and Methodology

To meet our first objective—determine the extent to which DHS has taken action to conduct security surveys and vulnerability assessments among high-priority CIKR—we reviewed DHS guidelines on the promotion and implementation of the security surveys and vulnerability assessments, records of outreach to CIKR stakeholders regarding these tools, and various DHS documents, including DHS’s National Critical Infrastructure and Key Resources Protection Annual Report, on efforts to complete security surveys and vulnerability assessments. We also interviewed officials in the Protective Security Coordination Division, which is part of the Office of Infrastructure Protection (IP) in DHS’s National Protection and Program Directorate, who are responsible for managing and administering DHS’s security surveys and vulnerability assessments to learn about the actions they took to conduct these programs. We obtained and analyzed DHS data on the conduct of voluntary programs for fiscal years 2009 through 2011—which are maintained in DHS’s Link Encrypted Network System (LENS) database and compared those records with the National Critical Infrastructure Prioritization Program (NCIPP) list of the high-priority CIKR assets—to determine the extent to which DHS performed security surveys and vulnerability assessments at high-priority assets. To assess the reliability of the data, we reviewed existing documentation about the data systems and spoke with knowledgeable agency officials responsible for matching the two databases to discuss the results of our comparison and to learn about their efforts to match LENS data with the NCIPP lists. While the information in each database was sufficiently reliable for the purposes of providing a general overview of the program, issues with the comparability of information in each database exist, which are discussed in this report. To do our comparison, we used a Statistical Analysis System (SAS) program to match the different data sets and summarize the results. Because we found that assets in the LENS database and NCIPP lists did not share common formats or identifiers that allowed us to easily match them, we had to match the data based on asset names and addresses. However, names and addresses were generally not entered in a standardized way, so we had to develop a process to standardize the available information and identify potential matches based on similar names or addresses. In our attempt to match the data sets, we did the following:

Standardized the date formats for fields that tracked when assessments were conducted (dates across lists might have formats such as 01/01/10 or 1/1/2010 and needed to be standardized to ensure appropriate matching within certain time frames).

Standardized the labels for sectors (across data sets, a sector might be listed as Chemical & Hazardous Materials Industry, Chemical and Hazardous Materials Indus, or ‘Chemical’).

Standardized state fields (across data sets, a state might be listed as Alabama or AL). Identified exact matches between the data sets on the asset name and the state name. Identified potential matches between the data sets based on asset name, asset address, and state. Specifically, we used a SAS function (SPEDIS) that measures asymmetric spelling distance between words, to determine the likelihood that names and addresses from two data sets did match and to generate possible pairs of matching assets. The possible matches for an asset were written to a spreadsheet, which we reviewed to determine a potential match.

As noted in the report, the inconsistencies between the data sets prevented us from determining definitively the extent to which assets on one list were also present in the other list. For example, in some cases assets seemed to be potential matches but there were differences in the sector listed or inconsistent company names and addresses. Thus we report separately on assets that were exact matches based on asset name and those that were potential matches. We also examined the inconsistencies we found with respect to DHS’s guidance on gathering data on participation in the security survey and vulnerability assessments and compared the findings to the criteria in Standards for Internal Control in the Federal Government. We also compared the results of our analyses with GAO reports on performance measurement, including ways to use program data to measure results.

In addition, to address the first objective, we also interviewed representatives—asset owners and operators—at 10 selected assets, also known as facilities, in 4 of the 18 sectors—the water, dams, commercial facilities, and energy sectors—to discuss their views on DHS efforts to work with asset owners and operators and conduct DHS’s voluntary security surveys and vulnerability assessments. We also contacted industry association representatives from the 4 sectors to discuss their views on DHS efforts to promote and conduct these activities. We selected these asset and industry representatives to take into account (1) sectors with a mix of regulations related to security; (2) sectors where DHS’s IP and non-DHS agencies are the SSAs—DHS for the commercial facilities sector and dams sector, the Department of Energy for the energy sector, and the Environmental Protection Agency for the water sector; (3) sectors where security surveys and vulnerability assessments had been conducted; and (4) geographic dispersion. We selected three states—California, New Jersey, and Virginia—where, based on our preliminary review of DHS’s LENS database and the NCIPP lists, security surveys and vulnerabilities assessments may have been performed at high-priority assets. At these assets, we, among other things, focused on the role of protective security advisors (PSA) who serve as liaisons between DHS and security stakeholders, including asset owners and operators, in local communities. We also reviewed PSA program guidance and interviewed 4 of 88 PSAs—PSAs from California, New Jersey and from the National Capital Region (encompassing Washington, D.C., suburban Virginia, and suburban Maryland)—to discuss the roles and responsibilities in partnering with asset owners and operators and in promoting security surveys and vulnerability assessments. While the results of our interviews cannot be generalized to reflect the views of all asset owners and operators and PSAs nationwide, the information obtained provided insights into DHS efforts to promote participation in its security survey and vulnerability assessment programs.

We also conducted a survey of 83 of 88 PSAs, those who, based on lists provided by DHS officials, had been in their positions for at least 1 year. We conducted the survey to gather information on PSAs’ efforts to promote and implement security surveys and vulnerability assessments, and identify challenges PSAs face when conducting these. GAO staff familiar with the critical infrastructure protection subject matter designed draft questionnaires in close collaboration with a social science survey specialist. We conducted pretests with three PSAs to help further refine our questions, develop new questions, clarify any ambiguous portions of the survey, and identify any potentially biased questions. We launched our web-based survey on October 3, 2011, and received all responses by November 18, 2011. Log-in information for the web-based survey was e- mailed to all participants. We sent one follow-up e-mail message to all nonrespondents 2 weeks later and received responses from 80 out of 83 PSAs surveyed (96 percent).

Because the survey was conducted with all eligible PSAs, there are no sampling errors. However, the practical difficulties of conducting any survey may introduce nonsampling errors. For example, differences in how a particular question is interpreted, the sources of information available to respondents, or the types of people who do not respond can introduce unwanted variability into the survey results. We included steps in both the data collection and data analysis stages to minimize such nonsampling errors. We collaborated with a GAO social science survey specialist to design draft questionnaires, and versions of the questionnaire were pretested with 3 PSAs. In addition, we provided a draft of the questionnaire to DHS’s IP for review and comment. From these pretests and reviews, we made revisions as necessary. We examined the survey results and performed computer analyses to identify inconsistencies and other indications of error. A second independent analyst checked the accuracy of all computer analyses.

Regarding our second objective—to determine the extent to which DHS shared the results of security surveys and vulnerability assessments with asset owners and operators and SSAs—we reviewed available DHS guidelines and reports on efforts to share security survey and vulnerability assessment results with stakeholders and compared DHS’s sharing of information with standards in the NIPP. We accessed, downloaded, and analyzed LENS data for information regarding the asset owners and operators that participated in DHS security surveys and vulnerability assessments during fiscal years 2009 through 2011. To assess the reliability of the data, we spoke with knowledgeable agency officials about their quality assurance process. During the course of our review DHS began taking action to clean up the data and address some of the data inconsistencies we discuss in this report. We found the data to be sufficiently reliable for providing a general overview of the program, but issues with the missing information in the LENS database exist and are discussed in this report. We compared the results of our analysis with DHS criteria regarding the timeliness of security surveys and vulnerability assessments, criteria in Standards for Internal Control in the Federal Government, and the NIPP.

We also used the LENS database, the NCIPP lists, and DHS documentation showing all assets that had received a security survey or a vulnerability assessment to select a nonrandom sample of high-priority assets from 4 sectors—the commercial facilities, dams, energy, and water sectors—and spoke with representatives from these selected assets to garner their opinions on the value of these voluntary programs and how they used the information DHS shared with them. In addition, we reviewed the 2009 and 2010 sector annual reports and the 2010 sector- specific plans for all CIKR sectors to assess if and how results of the security surveys and vulnerability assessments were included. We also interviewed SSA officials from our 4 selected sectors to learn what information DHS shared with them and how that information was used, and to discuss their overall relationship with DHS with respect to receiving and using data from DHS security surveys and vulnerability assessments. While the results of these interviews cannot be generalized to all SSAs, the results provided us with valuable insight into the dissemination and usefulness of information DHS provided from security surveys and vulnerability assessments. Furthermore, we interviewed DHS officials regarding their efforts to enhance the information they provide to SSAs from security surveys and vulnerability assessments.

With regard to our third objective—determine the extent to which DHS assessed the effectiveness of the security survey and vulnerability assessment programs, including any action needed to improve DHS’s management of the programs—we reviewed DHS documents and our past reports, and DHS Office of Inspector General (OIG) reports on DHS efforts to assess the effectiveness of its programs. We interviewed DHS officials and reviewed DHS guidelines on procedures for following up with asset owners and operators that have participated in these programs and to discuss the results of DHS efforts to conduct these follow-ups. We also (1) examined DHS documents that discussed the results of DHS efforts to conduct follow-ups and (2) analyzed the instrument used to contact owners and operators, as well as the questions asked to assess its effectiveness. In addition, we analyzed available data on DHS efforts to perform follow-ups for the period from January 2011 through September 30, 2011, and compared DHS data with DHS guidelines that discussed the number of days DHS officials were to begin follow-ups after providing the results of security surveys and vulnerabilities to asset owners and operators. We also compared the results or our work with criteria in Standards for Internal Control in the Federal Government and the NIPP, particularly those related to performance measurement. Finally, we spoke to CIKR officials in our sample sectors to learn how DHS personnel in the field had followed up on security surveys and vulnerability assessments and whether asset owners and operators were making changes based on the results, and if not why.

We conducted this performance audit from June 2011 through May 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Results of the Survey of Protective Security Advisors

This appendix provides information on our survey of Protective Security Advisors, which we used to gather information on efforts to promote and implement the voluntary programs offered by DHS and the challenges faced when conducting security surveys and vulnerability assessments. We conducted a Web-based survey of all 83 Protective Security Advisors who had been in their positions for at least one year. We received responses from 80, for a response rate of 96 percent. Our survey was composed of closed- and open-ended questions. In this appendix, we include all the survey questions and aggregate results of responses to the closed-ended questions; we do not provide information on responses provided to the open-ended questions. Percentages may not total to 100 due to rounding. For a more detailed discussion of our survey methodology, see appendix II.

Survey Respondent Information

1. Please provide the following information about the Protective Security Advisor responsible for completing this questionnaire.

Number of years as a PSA (Round up to nearest year) 2. Did you receive the Enhanced Critical Infrastructure Protection (ECIP) Initiative Standard Operating Procedures (SOP) guidance dated February 2011? 80 3. (If yes to Q2) How useful did you find the ECIP SOP guidance for promoting ECIPs?

If you answered "slightly useful" or "not at all useful", please explain why: 4. (If yes to Q2) How useful did you find the ECIP SOP guidance for conducting ECIPs?

If you answered "slightly useful" or "not at all useful", please explain why: 5. Did you receive training on the Enhanced Critical Infrastructure Protection (ECIP) Initiative program? 6. (If yes to Q5) How useful did you find the ECIP training?

If you answered "slightly useful" or "not at all useful", please explain why: 7. In your opinion, how useful is the ECIP Initiative program for reducing risk at CI facilities?

Please explain your opinion about the usefulness of the ECIP Initiative program: 8. In your opinion, how useful is the ECIP Infrastructure Survey Tool (IST) for reducing risk at CI facilities?

Please explain your opinion about the usefulness of the ECIP IST: 9. In your opinion, how useful is the ECIP Facility Dashboard for reducing risk at CI facilities?

10. How often have you heard each of the following reasons from facilities who declined to participate in an ECIP site visit? (Select one answer in each row.) a. The facility does not want to participate in additional facility assessments because it is already subject to Federal or State regulation/inspection. b. The facility does not have time or resources to participate.

80 c. Facility owners and operators are not willing to sign Protected Critical Infrastructure Information Express statements due to legal concerns over the protection and dissemination of the data collected. d. The entity that owns/oversees the facility declines to participate as a matter of policy. e. Facility owners and operators have a diminished perception of threat against the facility. f. The facility already received a risk assessment through a private company and participation in the voluntary assessment would be redundant or duplicative. g. Identification of security gaps may render the owner of the facility liable for damages should an incident occur.

What other reasons, if any, have you heard for facilities declining ECIP site visits? 11. Have you found that higher priority facilities (Level 1 or 2) are more or less likely to participate in ECIP site visits than lower priority facilities? 80 12. If you answered somewhat less likely or much less likely, what do you see as the reasons for the lower participation by the higher priority facilities? 13. What factors do you believe are important to facilities considering participating in an ECIP site visit? 14. How often have you heard each of the following reasons from facilities who declined to participate in an ECIP IST? (Select one answer in each row.) a. The facility does not want to participate in additional facility assessments because it is already subject to Federal or State regulation/inspection. b. The facility does not have time or resources to participate. c. Facility owners and operators are not willing to sign Protected Critical Infrastructure Information Express statements due to legal concerns over the protection and dissemination of the data collected. d. The entity that owns/oversees the facility declines to participate as a matter of policy. e. Facility owners and operators have a diminished perception of threat against the facility.

79 f. The facility already received a risk assessment through a private company and participation in the voluntary assessment would be redundant or duplicative. g. Identification of security gaps may render the owner of the facility liable for damages should an incident occur. h. Facility's security program is not yet mature enough to benefit from participation.

What other reasons, if any, have you heard for facilities declining to participate in an ECIP IST? 15. Have you found that higher priority facilities (Level 1 or 2) are more or less likely to participate in ECIP ISTs than lower priority facilities? 16. If you answered somewhat less likely or much less likely, what do you see as the reasons for the lower participation by the higher priority facilities? 17. How much of an incentive do you believe each of the following are for encouraging participation in an ECIP IST? (Select one answer in each row.)

No incentive Don't know d. Appeal to public service (patriotic duty)

If you responded not applicable to any of the sectors above, please explain. 20. Are you aware of any factors that drive differing levels of participation in the voluntary ECIP Initiative program by sector? Please explain. 21. In your opinion, how useful are SAVs as a tool for reducing risk at CI facilities?

Please explain your opinion about the usefulness of SAVs: 22. How often have you heard each of the following reasons from facilities who declined to participate in a SAV? (Select one answer in each row.) a. The facility does not want to participate in additional facility assessments because it is already subject to Federal or State regulation/inspection. b. The facility does not have time or resources to participate. c. Facility owners and operators are not willing to sign Protected Critical Infrastructure Information Express statements due to legal concerns over the protection and dissemination of the data collected. d. The entity that owns/oversees the facility declines to participate as a matter of policy.

80 e. Facility owners and operators have a diminished perception of threat against the facility. f. The facility already received a risk assessment through a private company and participation in the voluntary assessment would be redundant or duplicative. g. Identification of security gaps may render the owner of the facility liable for damages should an incident occur. h. Facility's security program is not yet mature enough to benefit from participation.

If you responded not applicable to any of the sectors above, please explain. 28. Are you aware of any factors that drive differing levels of participation in the voluntary SAV program by sector? Please explain. 29. What challenges, if any, do you face when implementing voluntary CI protection programs associated with ECIPs and SAVs?

30. Are you ready to submit your final completed survey to GAO? (This is equivalent to mailing a completed paper survey to us. It tells us that your answers are official and final.)

No, my survey is not yet complete - To submit your final responses, please click on "Exit" below" save your responses for later, please click on "Exit" below" You may view and print your completed survey by clicking on the Summary link in the menu to the left.

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, John F. Mortin, Assistant Director, and Anthony DeFrank, Analyst-in-Charge, managed this assignment. Andrew M. Curry, Katherine M. Davis, Michele C. Fejfar, Lisa L. Fisher, Mitchell B. Karpman, Thomas F. Lombardi, and Mona E. Nichols-Blake made significant contributions to the work.

Related GAO Products

Critical Infrastructure Protection: DHS Has Taken Action Designed to Identify and Address Overlaps and Gaps in Critical Infrastructure Security Activities. GAO-11-537R. Washington, D.C.: May 19, 2011.

Critical Infrastructure Protection: DHS Efforts to Assess and Promote Resiliency Are Evolving but Program Management Could Be Strengthened. GAO-10-772. Washington, D.C.: September 23, 2010.

Critical Infrastructure Protection: Update to National Infrastructure Protection Plan Includes Increased Emphasis on Risk Management and Resilience. GAO-10-296. Washington, D.C.: March 5, 2010.

The Department of Homeland Security’s (DHS) Critical Infrastructure Protection Cost-Benefit Report. GAO-09-654R. Washington, D.C.: June 26, 2009.

Information Technology: Federal Laws, Regulations, and Mandatory Standards to Securing Private Sector Information Technology Systems and Data in Critical Infrastructure Sectors. GAO-08-1075R. Washington, D.C.: September 16, 2008.

Risk Management: Strengthening the Use of Risk Management Principles in Homeland Security. GAO-08-904T. Washington, D.C.: June 25, 2008.

Critical Infrastructure: Sector Plans Complete and Sector Councils Evolving. GAO-07-1075T. Washington, D.C.: July 12, 2007.

Critical Infrastructure Protection: Sector Plans and Sector Councils Continue to Evolve. GAO-07-706R. Washington, D.C.: July 10, 2007.

Critical Infrastructure: Challenges Remain in Protecting Key Sectors. GAO-07-626T. Washington, D.C.: March 20, 2007.

Homeland Security: Progress Has Been Made to Address the Vulnerabilities Exposed by 9/11, but Continued Federal Action Is Needed to Further Mitigate Security Risks. GAO-07-375. Washington, D.C.: January 24, 2007.

Critical Infrastructure Protection: Progress Coordinating Government and Private Sector Efforts Varies by Sectors’ Characteristics. GAO-07-39. Washington, D.C.: October 16, 2006.

Information Sharing: DHS Should Take Steps to Encourage More Widespread Use of Its Program to Protect and Share Critical Infrastructure Information. GAO-06-383. Washington, D.C.: April 17, 2006.

Risk Management: Further Refinements Needed to Assess Risks and Prioritize Protective Measures at Ports and Other Critical Infrastructure. GAO-06-91. Washington, D.C.: December 15, 2005.

Protection of Chemical and Water Infrastructure: Federal Requirements, Actions of Selected Facilities, and Remaining Challenges. GAO-05-327. Washington, D.C.: March 28, 2005.

Homeland Security: Agency Plans, Implementation, and Challenges Regarding the National Strategy for Homeland Security. GAO-05-33. Washington, D.C.: January 14, 2005.