Critical Infrastructure Protection: DHS Could Better Manage Security Surveys and Vulnerability Assessments
Published: 2012-05-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Natural disasters, such as Hurricane Katrina, and terrorist attacks, such as the 2005 bombings in London, highlight the importance of protecting CIKRassets and systems vital to the economy or health of the nation. DHS issued the NIPP in June 2006 (updated in 2009) to provide the approach for integrating the nations CIKR. Because the private sector owns most of the nations CIKRfor example, energy production facilitiesDHS encourages asset owners and operators to voluntarily participate in surveys or vulnerability assessments of existing security measures at those assets. This includes nationally significant CIKR that DHS designates as high priority. In response to a request, this report assesses the extent to which DHS has (1) taken action to conduct surveys and assessments among highpriority CIKR, (2) shared the results of these surveys and assessments with asset owners or operators, and (3) assessed the effectiveness of surveys and assessments and identified actions taken, if any, to improve them. GAO, among other things, reviewed laws, analyzed data identifying high-priority assets and activities performed from fiscal years 2009 through 2011, and interviewed DHS officials.
section_title: What GAO Found
paragraphs: The Department of Homeland Security (DHS) has conducted about 2,800 security surveys and vulnerability assessments on critical infrastructure and key resources (CIKR). DHS directs its protective security advisors to contact owners and operators of high-priority CIKR to offer to conduct surveys and assessments. However, DHS is not positioned to track the extent to which these are performed at high-priority CIKR because of inconsistencies between the databases used to identify these assets and those used to identify surveys and assessments conducted. GAO compared the two databases and found that of the 2,195 security surveys and 655 vulnerability assessments conducted for fiscal years 2009 through 2011, 135 surveys and 44 assessments matched and another 106 surveys and 23 assessments were potential matches for high-priority facilities. GAO could not match additional high-priority facilities because of inconsistencies in the way data were recorded in the two databases, for example, assets with the same company name had different addresses or an asset at one address had different names. DHS officials acknowledged that the data did not match and have begun to take actions to improve the collection and organization of the data. However, DHS does not have milestones and timelines for completing these efforts consistent with standards for project management. By developing a plan with time frames and milestones consistent with these standards DHS would be better positioned to provide a more complete picture of its progress.
DHS shares the results of security surveys and vulnerability assessments with asset owners or operators but faces challenges doing so. A GAO analysis of DHS data from fiscal year 2011 showed that DHS was late meeting its (1) 30-day time frameas required by DHS guidancefor delivering the results of its security surveys 60 percent of the time and (2) 60-day time frameexpected by DHS managers for delivering the results of its vulnerability assessmentsin 84 percent of the instances. DHS officials acknowledged the late delivery of survey and assessment results and said they are working to improve processes and protocols. However, DHS has not established a plan with time frames and milestones for managing this effort consistent with the standards for project management. Also, the National Infrastructure Protection Plan (NIPP), which emphasizes partnering and voluntary information sharing, states that CIKR partners need to be provided with timely and relevant information that they can use to make decisions. Developing a plan with time frames and milestones for improving timeliness could help DHS provide asset owners and operators with the timely information they need to consider security enhancements.
DHS uses a follow-up tool to assess the results of security surveys and assessments performed at CIKR assets, and are considering upgrades to the tool. However, DHS could better measure results and improve program management by capturing additional information. For example, key information, such as why certain improvements were or were not made by asset owners and operators that have received security surveys, could help DHS improve its efforts. Further, information on barriers to making improvementssuch as the cost of security enhancementscould help DHS better understand asset owners and operators rationale in making decisions and thereby help improve its programs. Taking steps to gather additional information could help keep DHS better informed for making decisions in managing its programs.
section_title: What GAO Recommends
paragraphs: GAO recommends that, among other things, DHS develop plans for its efforts to improve the collection and organization of data and the timeliness of survey and assessment results, and gather and act upon additional information from asset owners and operators about why improvements were or were not made. DHS concurred with the recommendations.

section_title: Letter
paragraphs: In 2005, Hurricane Katrina devastated the Gulf Coast, damaging critical  infrastructure, such as oil refineries; electric power lines; water mains;  and cellular phone towers. More recently, in 2011, a major earthquake  and related tsunami devastated eastern Japan, damaging critical  infrastructure, such as dams, buildings, and power plants. The damage  and resulting chaos disrupted government and business functions alike,  producing cascading effects far beyond the physical location of these  events. Threats against critical infrastructure are not limited to natural  disasters, as demonstrated by the terrorist attacks of September 11,  2001, and the 2005 suicide bombings in London where terrorists  disrupted the city’s transportation and mobile telecommunication  infrastructure. In March 2007, we reported that our nation’s critical  infrastructures and key resources (CIKR)—assets and systems, whether  physical or virtual, so vital to the United States that their incapacity or  destruction would have a debilitating impact on national security, national  economic security, national public health or safety, or any combination of  those matters—continue to be vulnerable to a wide variety of threats.  Because the private sector owns the vast majority of the nation’s CIKR— banking and financial institutions, commercial facilities,  telecommunications networks, and energy production and transmission  facilities, among others—it is vital that the public and private sectors work  together to protect these assets and systems.
In 2006, the Department of Homeland Security (DHS) issued the National  Infrastructure Protection Plan (NIPP), which provides the overarching  approach for integrating the nation’s CIKR protection and resilience  initiatives into a single national effort. The NIPP sets forth a risk  management framework and details the roles and responsibilities for DHS  and other federal, state, regional, local, tribal, territorial, and private sector  partners implementing the NIPP. The NIPP also outlines the roles and  responsibilities of sector-specific agencies (SSA), the various federal  departments and agencies that are responsible for CIKR protection and  resilience activities in 18 specific CIKR sectors, such as the chemical,  commercial facilities, energy, and transportation sectors. The NIPP  emphasizes the importance of collaboration and partnering with and  among the various partners and voluntary information sharing between  the private sector and DHS. Consistent with the NIPP, DHS issues the  National Critical Infrastructure and Key Resources Protection Annual  Report (NAR) that summarizes risk mitigation and resiliency activities  across DHS and the 18 CIKR sectors. Further, as part of its risk  management strategy, DHS has established the National Critical  Infrastructure Prioritization Program (NCIPP), which uses a tiered  approach to identify nationally significant CIKR each year. This high- priority CIKR are categorized as either level 1 or level 2 based on the  consequence to the nation in terms of loss of life or economic impact. The  levels are used to enhance decision making related to CIKR protection  and can include a range of businesses or assets in a local geographic  area, such as refineries, water treatment plants, or commercial facilities,  as well as the information and data systems that ensure their continued  operation. According to DHS, the overwhelming majority of the assets  and systems identified as high priority are classified as level 2. Only a  small subset of assets meet the level 1 consequence threshold—those  whose loss or damage could result in major national or regional impacts  similar to the impacts of Hurricane Katrina or the September 11, 2001,  attacks.
Within DHS, the Office of Infrastructure Protection (IP) in the National  Protection and Programs Directorate (NPPD) is responsible for CIKR  protection and resilience. DHS coordinates through SSAs and other  sector partners for each of the CIKR sectors to identify security overlaps  and gaps as they implement the NIPP framework. While other entities  may possess and exercise regulatory authority over CIKR to address  security, such as for the chemical, transportation, and nuclear sectors, IP  generally relies on voluntary efforts to secure CIKR because of its limited  authority to directly regulate most CIKR. The voluntary efforts include the  Enhanced Critical Infrastructure Protection (ECIP) security surveys and  Site Assistance Visit (SAV) vulnerability assessments DHS conducts at  assets across the 18 sectors. ECIP security surveys are voluntary half to  full-day surveys DHS conducts to assess overall asset security and  increase security awareness, the results of which are presented to CIKR  owners and operators in a way that allows them to see how their assets’  security measures compare to those of similar assets in the same sector.  ECIP security survey results do not provide assets with recommendations  or options to enhance protective measures. Participation in SAV  vulnerability assessments is also voluntary and these assessments can  take up to 3 days to complete. These assessments identify security gaps  at assets and are used to provide options to enhance protective  measures and resilience to CIKR owners and operators. DHS field  representatives, called protective security advisors (PSA), are responsible  for working with CIKR owners and operators to conduct these surveys  and assessments. As of July 2011, there were 88 PSA positions in  various locations throughout the country. DHS shares information on the  results of these efforts with various stakeholders, generally on a need-to- know basis.
Given the voluntary nature of DHS’s CIKR program, the importance of  collaboration and partnering with and among the various partners, and  the need to identify and mitigate security and resilience gaps, you asked  that we examine DHS efforts to manage and measure the impact of its  voluntary security survey and vulnerability assessment programs.  Specifically, we assessed the extent to which DHS has  taken action to conduct security surveys and vulnerability  assessments among high-priority CIKR to improve their security  postures;  shared the results of security surveys and vulnerability assessments  with asset owners and operators and SSAs; and assessed the effectiveness of the security survey and vulnerability  assessment programs and identified actions, if any, to improve  management of these programs.
To meet our objectives, we reviewed applicable laws, regulations, and  directives as well as IP policies and procedures for conducting security  surveys and vulnerability assessments, providing their results, and  assessing the effectiveness of these programs. We also interviewed IP  officials in Washington, D.C., responsible for administering these  programs and obtained and assessed IP data on the conduct and  management of its security surveys and vulnerability assessments. In so  doing, we (1) compared IP data on security surveys and vulnerability  assessments conducted for fiscal years 2009 through 2011 with IP lists of  high-priority assets over the same period, (2) reviewed IP data on the  conduct of security surveys and vulnerability assessments and the  delivery of those surveys and assessments to asset owners and  operators, and (3) analyzed IP data on its efforts to follow up with asset  owners and operators to measure whether they had made changes to  enhance the security of their assets as a result of DHS security surveys  and vulnerability assessments conducted at their assets for fiscal years  2009 through 2011. We then compared the results of our analyses with  various criteria, including DHS policies and procedures outlined in the  NIPP; IP guidelines on the conduct and delivery of and follow-up to  security surveys and vulnerability assessments; our Standards for Internal  Control in the Federal Government; and our reports on performance  measurement, including those on ways to use program data to measure  results. We discussed the sources of the data and IP’s quality  assurance procedures with agency officials and determined that the data  were sufficiently reliable to provide a general overview of the program;  limitations are discussed later in this report.
Also, we interviewed SSA officials in Washington, D.C., representing four  selected sectors—the water, commercial facilities, energy, and dams  sectors—to determine whether these SSAs received the results of  security surveys and vulnerability assessments and, if so, how the  information was used. We selected these sectors because assets in  these sectors underwent numerous vulnerability assessments and  security surveys over the period and had a mix of SSA partners.  Specifically, DHS was the SSA for two of the sectors—the commercial  facilities and dams sectors. The Department of Energy and the  Environmental Protection Agency were the SSAs for the other two  sectors—the energy sector and the water sector, respectively. In addition,  we met with owners and operators of 10 high-priority assets in New  Jersey, Virginia, and California to obtain their views on DHS efforts to  conduct security surveys and vulnerability assessments and provide  results. We selected these locations because they contained assets on  the high-priority list and had a number of security surveys and  vulnerability assessments performed. During our visits to the 10 assets in  these locations, we also met with responsible PSAs from our sample  states to discuss actions that they take to conduct these surveys and  assessments, promote participation among asset owners and operators,  provide them the results of security surveys and assessments at their  assets, and measure results. The information from our interviews with  SSA officials, as well as asset owners and operators and PSAs in the  three states, are not generalizable to the universe of CIKR sectors and  assets and PSAs throughout the country. However, they provide valuable  insights into IP efforts to conduct and manage its security survey and  vulnerability assessment programs. With regard to PSAs, we also  conducted a survey of 83 of the 88 PSAs nationwide—those with 1 or  more years of experience—to obtain their views on various aspects of the  security survey and vulnerability assessment programs, including the  value of the programs in enhancing CIKR protection and resilience and  the challenges PSAs face in promoting and executing the surveys and  assessments. We administered our survey from October to November  2011 and received a 96 percent (80 of 83) response rate.
We conducted this performance audit from June 2011 through May 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix II discusses our  objectives, scope, and methodology and our survey of PSAs in greater  detail.
subsections: section_title: Background
paragraphs: The Homeland Security Act of 2002 created DHS and gave the  department wide-ranging responsibilities for, among other things, leading  and coordinating the overall national critical infrastructure protection  effort. Homeland Security Presidential Directive (HSPD) 7 further  defined critical infrastructure protection responsibilities for DHS and  SSAs. HSPD-7 directed DHS to establish uniform policies, approaches,  guidelines, and methodologies for integrating federal infrastructure  protection and risk management activities within and across CIKR  sectors. Various other statutes and directives provide specific legal  authorities for both cross sector and sector-specific protection and  resiliency programs. For example, the Public Health Security and  Bioterrorism Preparedness and Response Act of 2002 was enacted to  improve the ability of the United States to prevent, prepare for, and  respond to acts of bioterrorism and other public health emergencies, and  the Pandemic and All-Hazards Preparedness Act of 2006 addresses,  among other things, public health security and all-hazards preparedness  and response. Also, the Cyber Security Research and Development  Act, enacted in January 2002, authorized funding through fiscal year  2007for the National Institute of Standards and Technology and the  National Science Foundation to facilitate increased research and  development for computer and network security and to support related  research fellowships and training. CIKR protection issues are also  covered under various presidential directives, including HSPD-5 and  HSPD-8. HSPD-5 calls for coordination among all levels of government  as well as between the government and the private sector for managing  domestic incidents, and HSPD-8 establishes policies to strengthen  national preparedness to prevent, detect, respond to, and recover from  threatened or actual domestic terrorist attacks, major disasters, and other  emergencies.directives are tied together as part of the national approach for CIKR  protection through the unifying framework established in HSPD-7.
According to the NIPP, these separate authorities and  NPPD’s IP is responsible for working with public and private sector CIKR  partners in the 18 sectors and leads the coordinated national effort to  mitigate risk to the nation’s CIKR through the development and  implementation of CIKR protection and resilience programs. Using a  sector partnership model, IP’s Partnership and Outreach Division works  with sector representatives, including asset owners and operators, to  develop, facilitate, and sustain strategic relationships and information  sharing. IP’s Protective Security Coordination Division (PSCD) provides  programs and initiatives to enhance CIKR protection and resilience and  reduce risk associated with all-hazards incidents. In so doing, PSCD  works with CIKR owners and operators and state and local responders to  (1) assess vulnerabilities, interdependencies, capabilities, and incident  consequences; (2) develop, implement, and provide national coordination  for protective programs; and (3) facilitate CIKR response to and recovery  from incidents. Related to these efforts, PSCD has deployed the  aforementioned PSAs in 50 states and Puerto Rico, with deployment  locations based on population density and major concentrations of CIKR.  In these locations, PSAs are to act as the link between state, local, tribal,  and territorial organizations and DHS infrastructure mission partners and  are to assist with ongoing state and local CIKR security efforts by  establishing and maintaining relationships with state, local, tribal,  territorial, and private sector organizations;  support the development of the national risk picture by conducting  vulnerability and security assessments to identify security gaps and  potential vulnerabilities in the nation’s most critical infrastructures; and  share vulnerability information and protective measure suggestions  with local partners and asset owners and operators.
subsections: section_title: Enhanced Critical Infrastructure Protection Security Surveys
paragraphs: As part of their ongoing activities, PSAs are responsible for promoting the  ECIP Initiative. Launched in September 2007, the ECIP Initiative is a  voluntary program focused on forming or maintaining partnerships  between DHS and CIKR owners and operators of high-priority level 1 and  level 2 assets and systems, as well as other assets of significant value.  According to DHS guidance, PSAs are to schedule ECIP visits with  owners and operators in their districts using lists of high-priority and other  significant assets provided by PSCD each year, with visits to level 1  assets being the first priority, and visits to level 2 assets being the second  priority. Visits to other significant assets are to receive subsequent priority  based on various factors, including whether they are of significant value  based on the direction of IP; have been identified by the state homeland  security office; or represent a critical dependency associated with higher- priority assets already identified. If an asset owner or operator agrees to  participate in an ECIP visit, PSAs are to meet with the owner or operator  to assess overall site security, identify gaps, provide education on  security, and promote communication and information sharing among  asset owners and operators, DHS, and state governments.
One of the components of the ECIP Initiative is the security survey,  formally called the Infrastructure Survey Tool, which a PSA can use to  gather information on the asset’s current security posture and overall  security awareness. If the asset owner or operator agrees to participate in  the security survey, the PSA works with the owner or operator to apply  the survey, which assesses more than 1,500 variables covering six major  components—information sharing, security management, security force,  protective measures, physical security, or dependencies—as well as 42  more specific subcomponents within those categories. For example,  within the category “physical security” possible subcomponents include  fences, gates, parking, lighting, and access control, among others. Once  the survey is complete, the PSA submits the data to Argonne National  Laboratory, which analyzes the data to produce protective measures  index scores ranging from 0 (low protection) to 100 (high protection) for  the entire asset and for each component of the survey. Argonne National  Laboratory also uses the data to produce a “dashboard”—an interactive  graphic tool that is provided to the asset owner or operator by the PSA.  The dashboard displays the asset’s overall protective measures score,  the score for each of the six major components, the mean protective  measures score and major component scores for all like assets in the  sector or subsector that have undergone a security survey, and high and  low scores recorded for each component for all sector or subsector  assets that have undergone a security survey. The asset score and the  scores for other like assets show the asset owner or operator how the  asset compares to similar assets in the sector. The asset owner can also  use the dashboard to see the effect of making security upgrades to its  asset. For example, if the dashboard shows a low score for physical  security relative to those of other like assets, the owner or operator can  add data on perimeter fencing to see how adding or improving a fence  would increase the asset’s score, thereby bringing it more in line with  those of other like assets. Figure 1 provides an example of the dashboard  produced as a result of the security survey.
Related to these security surveys, DHS also produced, from calendar  years 2009 through 2011, summaries of the results of the security  surveys related to sector or subsector security postures, known as sector  summaries. These sector summaries were provided directly to SSAs in  2009 and 2010, and according to program officials were made available  to SSAs in 2011 for sectors upon request. Unlike the summaries in past  years, the 2011 summaries also included an “options for consideration”  section that identified specific protective measures that had been adopted  by the top 20 percent of assets in the sector or subsector as measured by  the overall protective measures score.
section_title: Site Assistance Visit Vulnerability Assessments
paragraphs: DHS also uses vulnerability assessments to identify security gaps and  provide options for consideration to mitigate these identified gaps. These  assessments are generally on-site, asset-specific assessments  conducted at the request of asset owners and operators. As of  September 30, 2011, DHS had conducted more than 1,500 vulnerability  assessments. Generally, vulnerability assessments are conducted at  individual assets by IP assessment teams in coordination with PSAs,  SSAs, state and local government organizations (including law  enforcement and emergency management officials), asset owners and  operators, and the National Guard, which is engaged as part of a joint  initiative between DHS and the National Guard Bureau (NGB). These  assessment teams are staffed via an interagency agreement between  DHS and NGB and include two national guardsmen—a physical security  planner and a systems analyst, one of whom serves as the team lead.  They may also be supplemented by contractor support or other federal  personnel, such as PSAs or subject matter experts, when requested.  Argonne National Laboratory staff then finalize the vulnerability  assessment report—which includes options for consideration to increase  an asset’s ability to detect and prevent terrorist attacks and mitigation  options that address the identified vulnerabilities of the asset—and  provide it to the PSA for delivery. The asset owners and operators that  volunteer for the vulnerability assessments are the primary recipients of  the analysis. The vulnerability assessment is developed using a  questionnaire that focuses on various aspects of the security of an asset,  such as vulnerabilities associated with access to asset air handling  systems, physical security, and the ability to deter or withstand a blast or  explosion. The vulnerability assessment report also contains a section  called “options for consideration” where DHS makes suggestions to  improve asset security or reduce identified vulnerabilities. For example,  one vulnerability assessment report made suggestions to the asset  owners or operators to explore the option of installing additional cameras  to improve video surveillance in certain locations, install additional  barriers to prevent vehicles from entering the facility at high speeds, and  increase the training of its security staff.
DHS revised the vulnerability assessment methodology in 2010 to  enhance the analytical capabilities of IP. According to DHS officials,  vulnerability assessments developed prior to 2010 did not have a  consistent approach for gathering data on assets and did not produce  results that were comparable from asset to asset. They also did not  incorporate an approach for assessing asset resilience. DHS reported  that the revised vulnerability assessment is intended to incorporate about  75 percent of the questions currently asked during an ECIP security  survey, including questions on resilience, to bring the tool more in line  with the security survey. As a result, vulnerability assessments completed  beginning in 2011 have the capability to produce a dashboard similar to  that produced from security surveys. By revising the assessment  methodology, DHS intends to ensure that the data collected during the  vulnerability assessment can be compared within and across sectors and  subsectors while still providing each asset an assessment specific to that  asset, including options for consideration to reduce vulnerability.
section_title: Regional Resiliency Assessment Program
paragraphs: While not the focus of this review, DHS has developed the Regional  Resiliency Assessment Program (RRAP) to assess vulnerability and risk  associated with resiliency. The RRAP is an analysis of infrastructure  “clusters,” regions, and systems in major metropolitan areas that uses  security surveys and vulnerability assessments, along with other tools, in  its analysis. The RRAP evaluates CIKR on a regional level to examine  vulnerabilities, threats, and potential consequences from an all-hazards  perspective to identify dependencies, interdependencies, cascading  effects, resiliency characteristics, and gaps. The RRAP assessments are  conducted by DHS officials, including PSAs in collaboration with SSAs;  other federal officials; state, local, territorial, and tribal officials; and the  private sector depending upon the sectors and assets selected as well as  a resiliency subject matter expert(s). The results of the RRAP are to be  used to enhance the overall security posture of the assets, surrounding  communities, and the geographic region covered by the project and is  shared with the state. According to DHS officials, the results of specific  asset-level assessments conducted as part of the RRAP are made  available to asset owners and operators and other partners (as  appropriate), but the final analysis and report are delivered to the state  where the RRAP occurred. Further, according to DHS, while it continues  to perform surveys and assessments at individual assets, prioritizing  efforts to focus on regional assessments allows DHS to continue to meet  evolving threats and challenges.
section_title: DHS Faces Challenges Managing Its Efforts to Conduct Security Surveys and Vulnerability Assessments on High- Priority Assets
paragraphs: DHS conducted about 2,800 security surveys and vulnerability  assessments during fiscal years 2009 through 2011. In so doing, DHS  directed PSAs to contact owners and operators of high-priority assets to  offer to conduct voluntary security surveys and vulnerability assessments  at their assets and PSAs used these as part of their outreach efforts  among these assets. However, DHS faces challenges tracking whether  security surveys and vulnerability assessments have been performed at  high-priority assets. Furthermore, DHS has not developed institutional  performance goals that can be used to measure the extent to which  owners and operators of high-priority assets participate in security  surveys and vulnerability assessments. In addition, DHS is not positioned  to assess why some high-priority asset owners and operators decline to  participate in these voluntary surveys and assessments so that it can  develop strategies for increasing participation.
subsections: section_title: Inconsistent Data Make It Challenging to Match Security Surveys and Vulnerability Assessments with High-Priority Assets
paragraphs: DHS is not positioned to track the extent to which it is conducting security  surveys and vulnerability assessments on high-priority assets because of  inconsistencies between the databases used to identify high-priority  assets and to identify surveys and assessments completed. Consistent  with the NIPP, DHS prioritizes the participation of high-priority assets in  its voluntary security survey and vulnerability assessment programs and  uses the NCIPP list of high-priority assets to guide its efforts. In  February 2011, DHS issued guidance to PSAs that called for them to  form partnerships with owners and operators of high-priority assets in  their areas. Under the guidelines, PSAs are to use NCIPP lists of high- priority assets to identify and contact owners and operators of the these  assets in their areas that could benefit from participation in the security  surveys, for the purpose of reducing potential security vulnerabilities and  identifying protective measures in place. PSAs are to conduct outreach  directly by meeting with the asset owners and operators to provide  information about DHS efforts to improve protection and resiliency,  sharing information about how an asset owner or operator can request a  vulnerability assessment, and offering to conduct a security survey. If  the owner or operator agrees to a visit from the PSA, the PSA is to record  the date of the visit, and if the owner or operator agrees to participate in a  security survey or vulnerability assessment, the PSA is likewise to record  the day the security survey or vulnerability assessment was conducted.  DHS analysts are then required to record the data provided by the PSAs  in DHS’s Link Encrypted Network System (LENS) database—DHS’s  primary database for tracking efforts to promote and complete security  surveys and annual assessments. According to DHS guidelines, these  data are subject to weekly reviews to ensure that data recorded in LENS  are accurate, consistent, and complete. Thus, data on each individual  asset should be recorded so that asset sector, name, and physical  address reflect a single asset in a specified location throughout the  database. For example, according to the guidelines, asset names  recorded in LENS should not be recorded with stray asterisks, other  special characters, and notes, and to the extent possible, address fields,  such as “St” should be captured as “Street.”
To determine how many of these activities have been conducted on high- priority assets, we used an automated statistical software program to  compare data on security surveys and vulnerability assessments  completed in DHS’s LENS database with data on high-priority assets on  the NCIPP lists for fiscal years 2009 through 2011—the lists PSAs are to  use to contact officials representing high-priority assets in their areas.  Out of 2,195 security surveys and 655 vulnerability assessments  conducted during fiscal years 2009 through 2011, we identified a total of  135 surveys and 44 vulnerability assessments that matched assets on the  NCIPP lists of high-priority assets. We also identified an additional 106  security surveys and 23 vulnerability assessments that were potential  matches with assets on the NCIPP lists of priority assets, but we could  not be certain that the assets were the same because of inconsistencies  in the way the data were recorded in the two different databases. For  example, we found instances where assets that appeared to be the same  company or organization were listed in different sectors. We also  encountered instances where names of companies at the same address  did not match exactly or where companies with the same names had  slightly different addresses in the two databases. For example, an asset  at 12345 Main Street in Anytown, USA, might appear as ABC Company  on one list and ABC on another. Conversely, we also found instances  where company names appeared to be the same or similar on both lists,  but they were listed at different street addresses or on different streets. In  this case, for example, ABC Company might appear as being located on  Main Street on one list, and E. Main St. on another.
We contacted DHS officials responsible for maintaining the LENS  database and the NCIPP list and told them that we had encountered  difficulty matching company names and addresses in the two lists. We  explained that our results depended on an asset being described in a  similar manner—same name, same address, same sector—in both the  NCIPP and LENS databases. These officials acknowledged that the two  databases do not match and explained that they have had to match the  data manually because of the inconsistencies. Specifically, DHS reported  that it reviewed over 10,000 records—including records of security  surveys, vulnerability assessments, and the NCIPP lists for fiscal years  2009 through 2011—and manually matched assets that had participated  in surveys or assessments with the NCIPP lists of high-priority assets  using DHS officials’ knowledge of the assets. Based on its efforts, DHS  analysts provided a table that showed that DHS conducted 2,128 security  surveys and 652 vulnerability assessments, of which it identified 674  surveys and 173 assessments that were conducted on high-priority  assets. Thus, by manually matching assets across the two lists, DHS was  able to show that the percentage of high-priority assets surveyed and  assessed increased significantly. Table 1 illustrates the results of our  efforts to match the data using an automated software program and the  results of DHS’s efforts to manually match the data.
DHS officials noted that beginning with the fiscal year 2012 NCIPP lists,  they have begun to apply unique numerical identifiers to each asset listed  in LENS and the NCIPP lists. According to these officials, once a match is  made, the application of unique identifiers to the same assets in both  databases is intended to remove uncertainty about which asset is which,  regardless of variations in the name or address of the asset. Related to  this, DHS officials also said that they have initiated a quality assurance  process whereby they use descriptive data—such as geographic  coordinates (longitude and latitude)—to verify street addresses and  names, thereby giving IP the ability to more readily make matches in  those instances where it may have previously experienced difficulty doing  so. Nonetheless, they said that the NCIPP list continues to present  matching challenges because there have been “significant” changes in  the NCIPP list from year to year, but they anticipate fewer changes in the  future. Most recently, the format and the organization of the list has  changed to focus on clusters—groups of related assets that can be  disrupted through a single natural or man-made hazard, excluding the  use of weapons of mass destruction—rather than on individual assets.  Thus, some assets previously considered high priority as stand-alone  assets are now listed as part of a system or as clusters that in and of  themselves are no longer considered high priority. According to DHS  officials, the introduction of clusters has resulted in other data matching  challenges, including the duplicate entry of an NCIPP asset that spans  two states; multiple entries for a single asset that is listed both individually  and in relation to a cluster or a system, and multiple entries for a single  asset within several clusters or systems.   DHS officials added that with  the assignment of the unique identifier, they expect to be better  positioned to cross-reference their program activities with the NCIPP list.
DHS officials have stated that the discrepancies between our analyses  and the analysis performed by IP, as well as the confusion created by  factors such as changing data sets, made it clear that improvements  should be made in the collection and organization of the data.  Accordingly, DHS officials said that they are continuing to work with  various partners within DHS and its contractors to streamline and better  organize the list of high-priority assets and data associated with  assessments, surveys, and other IP field activities. However, DHS did not  provide milestones and time frames for completing these efforts. DHS  appears to be heading in the right direction in taking actions to resolve  many of the issues we identified with regard to matching data and data  inconsistencies. However, moving forward, DHS would be better  positioned if it were to develop milestones and time frames for its plans to  accomplish these tasks. Standard practices for project management state  that managing a project involves, among other things, developing a  timeline with milestone dates to identify points throughout the project to  reassess efforts under way to determine whether project changes are  necessary. By developing time frames and milestones for streamlining  and organizing the lists of high-priority assets and data associated with  surveys, assessments, and field activities, DHS would be better  positioned to provide a more complete picture of its approach for  developing and completing these tasks. It also would provide DHS  managers and other decision makers with insights into (1) IP’s overall  progress in completing these tasks and (2) a basis for determining what, if  any, additional actions need to be taken.
section_title: DHS’s Efforts to Measure Progress Could Be Enhanced with Realistic Institutional Goals
paragraphs: As DHS moves forward to improve its efforts to track the hundreds of  security surveys and vulnerability assessments it performs each year,  DHS could also better position itself to measure its progress in conducting  these surveys and assessments at high-priority assets. We have  previously reported that to efficiently and effectively operate, manage,  and oversee programs and activities, agencies need reliable information  during their planning efforts to set realistic goals and later, as programs  are being implemented, to gauge their progress toward achieving those  goals. In July 2011, the PSCD Deputy Director told us that PSCD had a  goal that 50 percent of the security surveys and vulnerability assessments  conducted each year be on high-priority assets. However, this goal was  not documented; PSCD did not have written goals and the results to date  indicate that this goal was not realistic. Specifically, according to DHS’s  2010 NAR, less than 40 percent (299 of 763) of security surveys were  conducted on high-priority assets from May 1, 2009, through April 30,  2010. For the same time period, DHS’s NAR reported that about 33  percent (69 of 212) of vulnerability assessments were conducted on high- priority assets. Setting institutional realistic goals for the number of  security surveys and vulnerability assessments conducted at high-priority  assets—consistent with DHS’s efforts to improve its data on these  assets—would enable DHS to better measure its performance and  assess the state of security and resiliency at high-priority facilities, across  the 18 sectors, over time. For example, if there is a high-priority list  consisting of 2,000 facilities, a DHS goal of 500 security surveys and  vulnerability assessments conducted on high-priority facilities annually  would allow for the potential assessment of all high-priority facilities over  a defined period of time. Therefore, DHS could be in a better position to  identify security strengths and weaknesses at high-priority facilities and  within and across sectors and target areas for improvement.
Consistent with HSPD-7, DHS pursues a voluntary approach to critical infrastructure  protection and coordination.
DHS officials told us that many of these assets do not receive voluntary  surveys and assessments conducted by PSCD. Rather, as we previously  reported, PSCD staff told us that they work with the responsible federal  entity, such as the U.S. Coast Guard and the Nuclear Regulatory  Commission, to identify and address vulnerabilities. Finally, according to  the PSCD Deputy Director, shifting priorities based on terrorist threat  information, budget constraints, and other department wide priorities,  affect the prioritization and distribution of assets participating in these  voluntary programs. For example, DHS officials stated that given DHS is  placing increased emphasis on regional activities, such as RRAPs,  voluntary surveys and assessments are not necessarily focused on  individual high-priority assets. They said that expanded focus on regional  activities enables IP to meet evolving threats and challenges, but in a  budget constrained environment, forces them to prioritize activities so that  they can leverage existing resources.
See GAO/AIMD-00-21.3.1. Standards for Internal Control in the Federal Government  also calls for accurate and timely recording of information and periodic record reviews to  help reduce the risk of errors. DHS officials told us that they conduct data quality checks  and DHS guidelines direct such actions. However, the extent to which data were  inconsistent indicates that information was not always accurately captured. process to enable DHS to objectively and quantitatively assess  improvement in CIKR protection and resiliency. Specifically, the NIPP  states that performance metrics allow NIPP partners to track progress  against these priorities and provide a basis for DHS to establish  accountability, document actual performance, promote effective  management, and provide a feedback mechanism to decision makers.  Consistent with the NIPP risk management framework, our past work has  shown that leading organizations strive to align their activities to achieve  mission-related goals. By using LENS and NCIPP data to establish  performance goals, DHS could also be better positioned to identify gaps  between expected and actual participation, track progress in achieving  higher levels of participation, and ultimately gauge the extent to which  protection and resiliency are enhanced for the nation’s most critical  assets. Relying on institutional goals rather than informal goals would  also provide assurance that DHS has a common framework for  measuring performance in the face of organizational or personnel  changes over time.
section_title: DHS Tracks Whether Owners and Operators Decline Surveys and Assessments but Does Not Track Reasons Why They Decline
paragraphs: DHS guidelines issued in February 2011 call for PSAs to document the  names and addresses of CIKR asset owners or operators that decline to  participate in security survey outreach activities as well as the reasons  they declined. DHS officials told us that currently they track aggregate  data on declinations but they do not document the reasons why asset  owners and operators decline to participate in the security survey and  vulnerability assessment programs. In November 2011, DHS provided a  list of 69 asset owners or operators that PSAs recorded as having  declined to participate in the security surveys from March 2009 through  2011, but these records did not identify reasons for the declinations.  Program officials told us that the tool with which they collect declination  information is not designed to capture such information. The Deputy  Director for PSCD said that in 2012, DHS is developing a survey tool that  PSAs can use to record why asset owners or operators decline to  participate. Nonetheless, DHS could not provide specifics as to what  would be included in the tool, which office would be responsible for  implementing it, or time frames for its implementation. Rather, officials  told us that they intend to use the results of our review to inform  improvements to the process. Regarding vulnerability assessments, the  assessment guidance is silent on whether PSAs are to discuss  declinations with asset owners and operators and why they declined.  However, PSCD issued guidance in January 2012 that states that the  vulnerability assessment guidance is designed to complement the ECIP  guidance issued in February 2011.
In our survey of PSAs, PSA respondents provided some anecdotal  reasons as to why asset owners and operators may decline to participate.  For example, when asked how often they had heard various responses  from asset owners and operators that declined to participate in security  surveys or vulnerability assessments, PSAs responded that reasons for  declinations can include (1) the asset was already subject to federal or  state regulation or inspections, (2) the identification of security gaps could  render the owner of the asset liable for damages should an incident  occur, or (3) assets owner or operator had concerns that the information it  provides will not be properly safeguarded by DHS. Figure 2 shows the  frequencies of PSA responses of either “often” or “sometimes” to our  survey question about the various reasons for declinations that they have  heard. Appendix III shows the results of our survey in greater detail.
While these PSA perceptions may reflect some reasons asset owners  and operators decline to participate, it is important that DHS  systematically identify reasons why high-priority asset owners and  operators may decline to participate, especially if reasons differ from PSA  region to PSA region or by sector or subsector. By doing so, DHS may be  able to assess which declinations are within DHS’s ability to control or  influence and strategize how the security survey and vulnerability  assessment program and DHS’s approach toward promoting it can be  modified to overcome any barriers identified. For example, 39 percent (31  of 80) of the PSAs who responded to our survey suggested that senior- level partners, including senior leaders within DHS, could better support  the promotion of the security survey program when those leaders interact  with CIKR partners at high-level meetings. According to DHS, NPPD  and IP officials meet often with nonfederal security partners, including  sector coordinating councils (SCC), industry trade associations, state and  local agencies, and private companies, to discuss the security survey and  vulnerability assessment and other programs to assist in educating  mission partners about the suite of available IP tools and resources.  Meeting with security partners to discuss IP’s surveys, assessments, and  other programs is consistent with the NIPP partnership model whereby  DHS officials in headquarters are to promote vulnerability assessments at  high-level meetings where corporate owners are present—such as at  SCC or Federal Senior Leadership Council meetings—and through the  SSAs responsible for sector security. The NIPP also calls for DHS to rely  on senior-level partners, such as the SCCs and state representatives, to  create a coordinated national framework for CIKR protection and  resilience within and across sectors and with industry representatives that  includes the promotion of risk management activities, such as  vulnerability assessments. Given the barriers to participation identified in  our PSA survey, we contacted officials with 12 industry trade associations  representing the water, commercial facilities, dams, and energy sectors to  get their views on their awareness of DHS security surveys and  vulnerability assessments. Officials representing 10 of the 12 trade  associations said that they were aware of DHS’s voluntary survey and  vulnerability assessment programs, but only 6 of 12 knew if some of their  members’ had participated in these programs.
As noted earlier, at the time of our review DHS was not systematically  collecting data on reasons why some owners and operators of high- priority assets decline to participate in security surveys or vulnerability  assessments. Officials stated that they realize that some of the data  necessary to best manage these programs are not currently being  collected and said that one example is that PSAs are not consistently  reporting assessment and survey declinations from assets. DHS officials  added that in an effort to increase efficiency and accuracy, they are  developing additional data protocols to ensure that all the applicable data  are being collected and considered to provide a more holistic  understanding of the programs. Given that DHS efforts are just beginning,  however, it is too early to assess the extent to which they will address  these data collection challenges. Nevertheless, by developing a  mechanism to systematically collect data on the reasons for declinations,  consistent with DHS guidelines, DHS could be better positioned to identify  common trends for such declinations, determine what programmatic and  strategic actions are needed to manage participation among high-priority  assets, and develop action plans with time frames and milestones to  serve as a road map for addressing any problems. This could enhance  the overall protection and resilience of those high-priority CIKR assets  crucial to national security, public health and safety, and the economy.  Given that DHS officials recognize the need to collect these data to obtain  a more holistic understanding of these programs, DHS could be better  positioned if it had a plan, with time frames and milestones, for  developing and implementing these protocols. Standard practices for  project management state that managing a project involves, among other  things, developing a plan with time frames and milestones to identify  points throughout the process to reassess efforts under way to determine  whether project changes are necessary. By having plan with time  frames and milestones for developing additional data protocols, IP could  be better positioned to provide a more complete picture of its effort to  develop and complete this task. This could also provide DHS managers  and other decision makers with (1) insights into IP’s overall progress and  (2) a basis for determining what, if any, additional actions need to be  taken.
section_title: DHS Shares Survey and Assessment Results with Asset Owners and Operators and Sector- Specific Agencies but Faces Challenges
paragraphs: DHS shares security survey and vulnerability assessment information  with asset owners and operators that participate in these programs and  shares aggregated sector information with SSAs. However, DHS faces  challenges ensuring that this information is shared with asset owners and  operators in a timely manner and in providing SSAs security survey- derived products that can help SSAs in their sector security roles.  According to DHS officials, they are working to overcome these  challenges, but it is unclear whether DHS actions will address SSA  concerns about the use of aggregate security survey data.
subsections: section_title: Security Survey and Vulnerability Assessment Results Can Be Useful in Helping Asset Owners and Operators Reduce Risk
paragraphs: DHS security surveys and vulnerability assessments can provide valuable  insights into the strengths and weaknesses of assets and can help asset  owners and operators make decisions about investments to enhance  security and resilience. For example, our survey of PSAs showed that  most PSAs believe that the survey dashboard and the vulnerability  assessment were moderately to very useful tools for reducing risk at  CIKR assets. Specifically, 89 percent of PSAs (71 of 80) and 83 percent  of PSAs (66 of 80) responded that the security surveys and vulnerability  assessments, respectively, were moderately to very useful products for  reducing risk. One PSA commented that “The dashboard is the first tool  of its kind that allows the owner/operator a clear and measurable  quantitative picture of existing security profile” while another commented  that “ provide specific, actionable items for the  owner/operator to take action on to decrease vulnerabilities.”
Our discussions with various CIKR stakeholders—specifically asset  owners and operators and SSA representatives—also showed that these  tools can be useful to the asset owners and operators that participate in  these programs. As will be discussed later in greater detail, 6 of the 10  asset owners and operators we contacted used the results of these  survey and assessment tools to support proposals for security changes at  the assets that had been assessed. As one owner and operator said,  these voluntary programs provide a fresh look at facility security from a  holistic perspective. Another asset operator told us that it is nice to be  able to see how its security practices compare to those of others within its  sector. The representatives of the four SSAs we spoke with also believe  the security survey and vulnerability assessments were beneficial to the  asset owners and operators that received them.
section_title: DHS Is Not Delivering Results to Owners and Operators in a Timely Manner
paragraphs: The usefulness of security survey and vulnerability assessment results  could be enhanced by the timely delivery of these products to the owners  and operators that participated in them. For example, facility owners may  not see the importance of an identified security weakness if they do not  receive this information soon after a security survey or vulnerability  assessment is completed. Furthermore, the inability to deliver results  within the expected time frame could undermine the relationship DHS is  attempting to develop with asset owners and operators. As mentioned  earlier, PSAs rely on Argonne National Laboratory to provide them with  the results of the vulnerability assessments, which PSAs, in turn, deliver  directly to asset owners and operators. While PSAs find the voluntary  programs useful, 14 percent of PSAs we surveyed (11 of 80) described  late delivery of the reports as a factor that undermines the usefulness of  vulnerability assessments. One PSA commented that “the program is  broken in regard to timely completion of reports and deliverables  (protective measures and resiliency dashboards) for the asset  owners/operators. I have yet to receive anything from (a vulnerability  assessment conducted several months ago). I have not even received the  draft report for review nor the dashboard. This creates a big credibility  problem for me with my stakeholders who are looking for the results.” The  NIPP states that in order to have an effective environment for information  sharing, CIKR partners need to be provided with timely and relevant  information that they can use to make decisions. Consistent with the  NIPP, DHS guidelines state that PSAs are to provide the results of  security surveys in the form of a survey dashboard within 30 days of  when the security survey was completed. In addition, according to PSCD  officials, although there is no written guidance, PSCD expects that  vulnerability assessment results are to be provided to assets within 60  days of completion of the vulnerability assessment.
We analyzed DHS LENS data to determine the extent to which survey  dashboards were delivered to asset owners and operators on a timely  basis, using DHS’s 30-day criteria for timeliness. Our analysis showed  that for fiscal year 2011, more than half of all dashboards and  vulnerability assessment reports were delivered to owners and operators  late. Specifically, of the 570 dashboard reports that were supposed to  be delivered during fiscal year 2011, about 24 percent (139 of 570) were  delivered on time and approximately 60 percent (344 of 570) were late,  with almost half of those delivered 30 days beyond the 30-day deadline  established by DHS guidelines. Data were missing for about 15 percent  (85 of 570) of the remaining dashboards.of dashboard deliveries for all security surveys conducted in fiscal year  2011.
section_title: DHS Could Better Measure Security Survey and Vulnerability Assessment Results to Improve Program Management
paragraphs: DHS has taken actions to determine whether asset owners or operators  have made security improvements based on the results of security  surveys. However, DHS has not developed an overall approach to  determine (1) the extent to which changes have enhanced asset  protection and resilience over time or (2) why asset owners and operators  do not make enhancements that would help mitigate vulnerabilities  identified during security surveys and vulnerability assessments. As a  result, DHS may be overlooking an opportunity to make improvements in  the management of its voluntary risk mitigation programs that could also  help DHS work with asset owners and operators to improve security and  resilience. According to DHS, moving forward, it may consider changes to  the types of information gathered as part of its effort to measure  improvements, but it has not considered what additional information, if  any, should be gathered from asset owners or operators that participate  in security surveys and vulnerability assessments.
subsections: section_title: DHS Has Implemented a Security Survey Follow-up Tool
paragraphs: According to the NIPP, the use of performance measures is a critical step  in the risk management process to enable DHS to objectively and  quantitatively assess improvement in CIKR protection and resiliency at  the sector and national levels. The NIPP states that the use of  performance metrics provides a basis for DHS to establish accountability,  document actual performance, promote effective management, and  provide a feedback mechanism to decision makers. Consistent with the  NIPP, DHS has taken action to follow up with security survey participants  to gather feedback from asset owners and operators that participated in  the program regarding the effect these programs have had on asset  security using a standardized data collection tool, hereafter referred to as  the follow-up tool or tool.
DHS first began to do follow-ups with asset owners and operators in May  2010 but suspended its follow-up activities shortly thereafter to make  enhancements to the tool it used. In January 2011, IP introduced its  revised follow-up tool, which was to be used by PSAs to ask asset  representatives whose assets had undergone a security survey and  received a dashboard about enhancements made in six general  categories—information sharing, security management, security force,  protective measures, physical security, and dependencies. Whereas the  original follow-up tool focused on changes asset owners and operators  made to enhance security and resilience, the revised tool focused on  changes that were made directly as a result of DHS security surveys.  According to DHS guidance, the tool was to be used 180 days after the  completion of a security survey at an asset. The tool, which directs PSAs  to ask a series of questions about improvements made as a result of the  survey, instructs PSAs to request information on specific enhancements  within those categories that were discussed in the dashboard provided to  the asset owners and operators. For example, within the physical security  category, the tool instructs the PSAs to ask about any enhancements to  things like fences, gates, parking, lighting, and access control, among  others, and to ask asset owners or operators whether an identified  change was made as a result of the security survey the asset had  received. In February 2011, shortly after the revised tool was introduced,  IP issued guidelines that instructed PSAs to implement the follow-up tool.  According to IP officials, PSAs used the tool to follow up with owners and  operators of 610 assets from January 2011 through September 2011.  Data provided by IP showed that about 21 percent (126 of the 610) of the  respondents to the PSA follow-ups reported that they had completed  improvements, and 81 percent of these (102 of 126) reported that those  improvements were implemented as the result of the security survey the  asset received. According to IP’s data, the most common types of  improvements identified by assets that had completed improvements  since receiving the security survey were changes to information sharing,  which could include activities such as participating in working groups, and  physical security.
section_title: Follow-up Data on Results Do Not Reflect Long-Term Program Performance
paragraphs: DHS guidance states that PSAs are to conduct a follow-up with the asset  owners and operators 180 days after an asset receives a security survey.  We compared DHS data on 522 security surveys conducted from July 1,  2010, through March 31, 2011, with DHS data on the follow-ups  performed from January 1, 2011, through September 30, 2011—180 days  after DHS completed the security surveys. We found that DHS did not  contact some asset owners or operators that should have received a 180- day follow-up and contacted some owners and operators that had  participated in a security survey more than 180 days prior to the  introduction of the tool. For example, of the 522 security survey  participants that participated in a security survey from July 1, 2010,  through March 31, 2011, 208 (40 percent) received the 180-day follow-up  and 314 (60 percent) did not. Furthermore, DHS recorded an additional  402 follow-ups on assets that had received their security survey more  than 180 days prior to the introduction of the tool. Thus, the data DHS  reported included improvements assets made beyond the 180-day scope  of the follow-up tool, making it difficult to measure the effectiveness of the  security survey in prompting enhancements within 180 days of the  survey.
According to PSCD officials, there are two key reasons why DHS used  the follow-up tool to capture data on changes made beyond 180 days.  First, program officials said that completion of the 180-day follow-up  depends upon the asset representative’s willingness to participate and  availability to answer these questions. If the asset representative does not  agree to participate, or neither the representative nor the PSA is  available, the 180-day follow-up cannot be completed on schedule.  However, when DHS provided the follow-up data in November 2011,  officials said that they were not aware of any asset owners or operators  that had refused to participate in the 180-day follow-up at that time.  Second, program officials noted that the inclusion of assets that had  received a security survey more than 180 days prior to the introduction of  the revised follow-up tool occurred because they believed that it was  necessary to capture data on as many assets as possible. They said that  IP intends that follow-ups be completed as close to the 180-day mark as  possible, but they believed it was important to initially document whether  the security survey resulted in changes to security, regardless of when  the change was made. IP officials further explained that they had  developed a similar follow-up tool to capture data on enhancements  resulting from vulnerability assessments. However, at the time of our  review, results were not available from the vulnerability assessment  follow-up tool, which was also implemented in January 2011 and was  designed to capture data on enhancements made 365 days following the  delivery of the vulnerability assessment report. Consistent with the  security survey, DHS officials explained that the 365-day follow-up for  vulnerability assessments was determined as a means to begin the  process of collecting and assessing data on improvements being made  as a result of the assessments. They added that as more data are  collected, IP will review the information to determine if the follow-up visits  for security surveys and vulnerability assessments should remain at 180  and 365 days, respectively, or be moved as a result of information  collected from asset owners and operators. Nonetheless, DHS officials  did not provide a road map with time frames and milestones showing  when they planned to revisit the 180-day follow-up time frame or the  intervals between follow-ups. Consistent with the standards for project  management, by having a road map with time frames and milestones for  revisiting these time frames, IP could be better positioned to provide a  more complete picture of its overall progress making these decisions and  a basis for determining what, if any, additional actions need to be taken or  data inputs need to be made.
GAO/AIMD-00-21.3.1. especially true if asset owners and operators are implementing more  complicated enhancements over a longer term because of the need to  develop and fund plans for particular types of improvements. For  example, gathering these data could help DHS measure not only what  improvements asset operators are implementing, but also how long it  takes to complete the planning phase of a security enhancement project  and how this time frame might vary by the type of improvement.
Furthermore, while it is important to capture information about  improvements made as a result of these activities over time, it is also  important that DHS either capture the information within the prescribed  times outlined in DHS guidance, adjust the time frames based on an  analysis of data captured over time, or perform follow-ups at additional  intervals beyond those initially performed. This would also be consistent  with Standards for Internal Control in the Federal Government, which  calls for the establishment and review of performance measures and  indicators to monitor activities and top-level reviews by management to  track major agency achievements and compare these with plans, goals,  and objectives. By doing so, IP could be better positioned to document  actual performance, promote effective management, provide a feedback  mechanism to decision makers, and enhance overall accountability.
section_title: Additional Enhancements Could Better Position DHS to Measure Performance
paragraphs: According to DHS officials, moving forward, DHS may consider additional  changes to its follow-up tool depending on the results they gather over  time. The NIPP states that performance measures that focus on outputs,  called output measures, such as whether an asset completes a security  improvement, should track the progression of a task. The NIPP further  states that outcome measures are to track progress toward an intended  goal by beneficial results rather than level of activity. Our review of DHS’s  approach for following up with assets that had undergone a security  survey showed that PSAs were instructed to focus on security  enhancements completed as result of the security survey, not  enhancements that were planned or in process. Nonetheless, our review  of DHS’s follow-up results for the period from January through September  2011 showed that DHS reported the following:
41 percent (250 of 610) of the owners and operators surveyed  reported that security enhancements were either in process or  planned and  the results did not indicate whether these planned or in-process  enhancements were attributable to DHS’s security survey at these  assets.
After we discussed our observation with DHS officials, they informed us  that they believe completed improvements are the best initial  measurement of the impact of security surveys and vulnerability  assessments. They added that other metrics can be added as the  process matures and is refined. However, as of March 2012, DHS did not  document whether planned or in-process improvements are the result of  security surveys. Given that the NIPP calls for CIKR partners to measure  performance in the context of the progression of the task, DHS could be  missing an opportunity to measure performance associated with planned  and in-process enhancements, especially if they are attributable to DHS  efforts via security surveys and vulnerability assessments. DHS could  also use this opportunity to consider how it can capture key information  that could be used to understand why certain improvements were or were  not made by assets owners and operators that have received surveys  and assessments. For example, the follow-up tool could ask asset  representatives what factors—such as cost, vulnerability, or perception of threat— influenced the decision to implement changes, either immediately or  over time if they chose to make improvements; what factors—such as perception of risk, cost, or budget constraints— influenced an asset owner or operator to choose to not make any  improvements; why were the improvements made chosen over other possible  improvements, if improvements were made; and did the improvements, if any, involve the adoption of new or more  cost-effective techniques that might be useful as an option for other  owners and operators to consider as they explore the feasibility  making improvements?
Understanding why an asset owner or operator chooses to make, or not  make, improvements to its security is valuable information for  understanding the obstacles asset owners or operators face when making  security investments. For example, the cost of security upgrades can be a  barrier to making enhancements. As one PSA who responded to our  survey commented, “there is no requirement for the owner/operator to  take action. They are left with making a “risk-reward” decision. Some see  great value in making security upgrades, while others are less inclined to  make improvements due to costs.” Likewise, one asset representative  told us that security is one of the most important things to management  until budget time. In this regard, a better understanding of the complexity  of the security improvement decision at the asset could also help DHS  better understand the constraints asset owners or operators face in  making these decisions—information that could possibly help DHS  determine how, if at all, to refine its security survey program to assist  asset owners or operators in making these decisions. For example, the  NIPP states that effective CIKR programs and strategies seek to use  resources efficiently by focusing on actions that offer the greatest  mitigation of risk for any given expenditure. Additional information on the  cost of improvements made and the reasons why improvements were or  were not made could also assist DHS in understanding the trade-offs  asset owners and operators face when making decisions to address  vulnerabilities identified as a result of DHS security surveys and  enhancements. IP officials told us they are wary of attempting to gather  too much information from asset representatives with the follow-up tool  because of a concern that being too intrusive may damage the  relationships that the PSAs have established with asset representatives.  They said that gathering additional information is not as important as  maintaining strong relationships with the asset representatives.
We recognize that DHS operates its security survey program in a  voluntary environment and that DHS can only succeed at improving asset  and sector security if asset owners and operators are willing to  participate, consistent with DHS’s interest in maintaining good  relationships with asset representatives. However, by gathering more  information from assets that participate in these programs—particularly  high-priority assets—DHS could be better positioned to measure the  impact of its programs on critical infrastructure security at the sector and  national levels. Moreover, by collecting and analyzing this type of  information, DHS could be better informed in making decisions about  whether adjustments to its voluntary programs are needed to make them  more beneficial to CIKR assets—a factor which could help DHS further  promote participation by asset owners and operators that may previously  have been reluctant to participate in DHS security surveys and  assessments. Having this type of information could also be important in  light of DHS’s efforts to better understand interdependencies between  assets via the RRAPs. For instance, by knowing what factors influence  decisions to make an improvement, or not, at one asset or a group of  assets, DHS could be better positioned to understand how that decision  influences the security of other assets that are also part of the RRAP. As  a result, DHS and PSAs could then be better positioned to work with  owners and operators to mitigate any vulnerabilities arising out of these  decisions. It could also help DHS develop and target strategies for  addressing why certain enhancements were not made and ultimately put  DHS in a better position to measure outcomes, rather than outputs,  associated with its efforts to promote protection and resilience via its  voluntary risk mitigation programs.
section_title: Conclusions
paragraphs: DHS has taken important actions to conduct voluntary CIKR security  surveys and vulnerability assessments, provide information to CIKR  stakeholders, and assess the effectiveness of security surveys and  vulnerability assessments. However, further actions could enhance each  of these endeavors and provide DHS managers the information they need  to ensure that IP is taking appropriate steps toward completing them or  making adjustments where needed. DHS has not institutionalized realistic  goals that could help DHS measure the effects of its efforts to promote  and conduct security surveys and vulnerability assessments among high- priority assets. By developing realistic institutional goals, DHS could, for  example, better measure the effects of its efforts to promote and conduct  security surveys and vulnerability assessments among high-priority  assets. Further, developing a road map with milestones and time frames  for (1) taking and completing actions needed to resolve issues associated  with data inconsistencies and matching data on the list of high-priority  assets with data used to track the conduct of security surveys and  vulnerability assessments, (2) completing protocols to systematically  collect data on the reasons why some owners and operators declined to  participate in the voluntary surveys and assessments, and (3) improving  the timely delivery of the results of security surveys and vulnerability  assessments could better position DHS to target high-priority assets and  provide them with the information they need to make decisions related to  security and resiliency. Moreover, by revising its plans to include when  and how SSAs will be engaged in designing, testing, and implementing  the web-based tool, consistent with its recent efforts to coordinate with  CIKR partners, DHS could be positioned to better understand and  address their information needs.
Consistent with the NIPP, DHS is also continuing to take actions to follow  up with asset owners and operators that have participated in security  surveys and vulnerability assessments to gauge the extent to which these  surveys and assessments have prompted owners and operators to  improve security and resilience at their assets. DHS officials said that  they intend to review the information it gathers from asset owners and  operators to determine if the follow-up visits should remain at 180 days  after DHS completed the security surveys. By establishing a road map  with milestones and time frames for conducting this review, DHS would  be better positioned to provide a picture of its overall progress in making  these decisions and a basis for determining what, if any, additional  actions need to be taken or data inputs need to be made and whether  additional follow-ups are appropriate at intervals beyond the follow-ups  initially performed. In addition, collecting detailed data on actions started  and planned and, for example, why actions were not taken, could provide  DHS valuable information on the decision-making process associated  with making security enhancements and enable DHS to better understand  what barriers owners and operators face in making improvements to the  security of their assets.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that DHS’s efforts to promote security surveys and  vulnerability assessments among high-priority CIKR are aligned with  institutional goals, that the information gathered through these surveys  and assessments meet the needs of stakeholders, and that DHS is  positioned to know how these surveys and assessments could be  improved, we recommend that the Assistant Secretary for Infrastructure  Protection, Department of Homeland Security, take the following seven  actions: develop plans with milestones and time frames to resolve issues  associated with data inconsistencies and matching data on the list of  high-priority assets with data used to track the conduct of security  surveys and vulnerability assessments;  institutionalize realistic performance goals for appropriate levels of  participation in security surveys and vulnerability assessments by  high-priority assets to measure how well DHS is achieving its goals; design and implement a mechanism for systematically assessing why  owners and operators of high-priority assets decline to participate and  a develop a road map, with time frames and milestones, for  completing this effort; develop time frames and specific milestones for managing DHS’s  efforts to ensure the timely delivery of the results of security surveys  and vulnerability assessments to asset owners and operators;  revise its plans to include when and how SSAs will be engaged in  designing, testing, and implementing DHS’s web-based tool to  address and mitigate any SSA concerns that may arise before the tool  is finalized; develop a road map with time frames and specific milestones for  reviewing the information it gathers from asset owners and operators  to determine if follow-up visits should remain at 180 days for security  surveys and whether additional follow-ups are appropriate at intervals  beyond the follow-ups initially performed; and  consider the feasibility of expanding the follow-up program to gather  and act upon data, as appropriate, on (1) security enhancements that  are ongoing and planned that are attributable to DHS security surveys  and vulnerability assessments and (2) factors, such as cost and  perceptions of threat, that influence asset owner and operator  decisions to make, or not make, enhancements based on the results  of DHS security surveys and vulnerability assessments.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Secretary of Homeland Security  for review and comment. In its written comments reprinted in appendix IV,  DHS agreed with all seven of the recommendations; however, its  implementation plans do not fully address two of these seven  recommendations and it is unclear to what extent its plans will address  two other recommendations.
With regard to the first recommendation that DHS develop plans to  resolve issues associated with data inconsistencies between its  databases, DHS stated its efforts to assign unique identifiers to assets on  the high-priority list that have received security surveys and vulnerability  assessments will make matching easier and that other quality assurance  processes have been implemented to better verify individual asset data.  We agree these are positive steps; however, to fully address the  recommendation, we believe DHS should develop a plan with time frames  and milestones that specify how the steps it says it is taking address the  data inconsistencies we cited, and demonstrate the results—how many  high-priority assets received security surveys, vulnerability assessments,  or both in a given year—of that effort. By doing so, DHS would be better  positioned to provide a more complete picture of its approach for  developing and completing these tasks.  It would also provide DHS  managers and other decision makers with insights into (1) IP’s overall  progress in completing these tasks and (2) a basis for determining what, if  any, additional actions need to be taken.
With regard to the second recommendation that DHS institutionalize  realistic performance goals for levels of participation in security surveys  and vulnerability assessments by high-priority assets, DHS stated that the  participation of high-priority assets continues to be a concern but  reiterated its view that the voluntary nature of its programs and competing  priorities makes setting goals for high-priority participation difficult.  DHS  stated that its fiscal year 2012 Project Management Plans for Protective  Security Advisor and Vulnerability Assessment Projects established  realistic goals concerning the total number of assessments to be  conducted. However, they said these plans do not include goals for  assessments performed at high-priority assets. Furthermore, DHS stated  the shift in emphasis to regional resilience suggested metrics and goals  intended to measure the participation of high-priority assets in  vulnerability assessments and surveys may not be a strong or accurate  indicator of the degree to which DHS is achieving its infrastructure  protection and resilience goals. We agree that the voluntary nature of  these programs and changing priorities make the process of setting goals  difficult. However, the NIPP and DHS guidance emphasize the  importance of high-priority participation in these programs, and DHS can  take factors like the voluntary nature of the program and DHS’s shift  toward regional resilience into account when setting realistic goals for the  number of security surveys and vulnerability assessments it conducts at  high-priority facilities. By establishing realistic performance goals for  levels of participation by high priority assets, DHS would be better  positioned to compare actual performance against expected results and  develop strategies for overcoming differences or adjust its goals to more  realistically reflect the challenges it faces.
With regard to the third recommendation that DHS design and implement  a mechanism for systematically assessing why owners and operators of  high priority assets decline to participate and develop a road map, with  time frames and milestones, for completing this effort, DHS stated it  recognizes that additional clarification and guidance are needed to ensure  effective implementation of existing guidance. Specifically, DHS stated it  will review and revise the guidance to (1) determine if revisions to the  existing process are required and (2) develop supplementary guidance to  aid PSAs in executing the process. DHS stated it will initiate this review in  the fourth quarter of fiscal year 2012, after which time it will develop  additional milestones for mechanism improvement. We believe that  DHS’s proposed actions appear to be a step in the right direction, but it is  too early to tell whether DHS’s actions will result in an improved  mechanism for systematically assessing why owners and operators  decline to participate.
Regarding the fourth recommendation to develop time frames and  specific milestones for managing its efforts to improve the timely delivery  of the results of security surveys and vulnerability assessments to asset  owners and operators, DHS stated it is working with contractors and  program staff to advance the processes and protocols governing the  delivery of assessment and survey products to facilities. DHS also stated  that it had begun a review of assessments lacking delivery dates in LENS  and is working with PSAs to populate the missing information. In addition,  DHS noted that its plan to transition to a web-based dashboard system  will help mitigate the issue of timely report delivery by eliminating the  need for in-person delivery of the dashboard product. However, DHS did  not discuss time frames and milestones for completing these efforts.   Thus, it is unclear to what extent DHS’s actions will fully address this  recommendation.  As noted in our report, developing time frames and  milestones for completing improvements that govern the delivery of the  results of surveys and assessments would provide insights into IP’s  overall progress.
With regard to the fifth recommendation to revise its plans to include  when and how SSAs will be engaged in designing, testing, and  implementing DHS’s web-based tool, DHS stated that it is currently taking  actions to develop and test a web-based dashboard tool for individual  owners and operators, which is expected to be widely available in  January 2013. DHS stated that it anticipates the development of a state  and local “view,” or dashboard, following the successful deployment of the  web-based owner and operator dashboards. Regarding SSAs, DHS  stated that a concept for a sector-level view of assessment data has been  proposed and that the requirements and feasibility of such a dashboard  will be explored more fully following the completion of the state-level web- based dashboard. DHS noted that that IP will engage the SSAs to  determine any associated requirements. DHS’s proposed actions appear  to be a step in the right direction. However, given that the sector level  view of assessment data is in the proposal stage and further action will be  explored more fully after completion of the state level web-based  dashboard, it is too early to tell when and how SSA’s will be engaged in  designing, testing and implementing the web-based tool.
In response to the sixth recommendation to develop a road map with time  frames and specific milestones to determine if follow-up visits should  remain at 180 days for security surveys, and whether additional follow- ups are appropriate at intervals beyond the follow-ups initially performed,  DHS stated it will analyze and compare security survey follow-up results  in early calendar year 2013 to determine whether modifications are  required. DHS also stated that given that the 365-day follow-up process  went into effect in January 2011, the first follow-up evaluations of  vulnerability assessments have only recently begun and IP will collect, at  a minimum, 1 year of vulnerability assessment data. DHS said that IP  intends to review the results for both the security survey 180-day follow- up and the 365-day follow-up in early calendar year 2013 to determine  whether modifications to the follow-up intervals are required. DHS’s  proposed actions are consistent with the intent of this recommendation.
In response to the seventh recommendation to consider the feasibility of  gathering and acting upon additional data, where appropriate, on   (1) ongoing or planned enhancements attributable to security surveys and  assessments and (2) factors that influence asset owner and operator  decisions to make or not make security enhancements, DHS stated that it  collects information on ongoing or planned enhancements. However, as  noted in the report, DHS does not collect information that would show  whether these enhancements are attributable to security surveys and  assessments. DHS also stated that IP will continue to work with Argonne  National Laboratory and field personnel to determine the best method for  collecting information related to those factors influencing an asset’s  decision to implement or not implement a new protective measure or  security enhancement. However, it is not clear to what extent DHS’s  actions will fully address this recommendation because it did not discuss  whether it will consider the feasibility of gathering data on whether  ongoing or planned enhancements are attributable to security surveys  and assessments or how it will act upon the data it currently gathers or  plans to gather to, among other things, measure performance in the  context of the progression of the task, consistent with the NIPP. By  gathering and analyzing data on why an asset owner or operator chooses  to make, or not make, improvement to security, DHS would be better  positioned to understand the obstacles asset owners face when making  investments.
DHS also provided technical comments, which we incorporated as  appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of  Homeland Security, the Under Secretary for the National Protection  Programs Directorate, and other interested parties. In addition, the report  will be available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have questions about this report, please contact me at  (202) 512-8777 or caldwells@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix V.
section_title: Appendix I: Critical Infrastructure Sectors
paragraphs: This appendix provides information on the 18 critical infrastructure sectors  and the federal agencies responsible for sector security. The National  Infrastructure Protection Plan (NIPP) outlines the roles and  responsibilities of the Department of Homeland Security (DHS) and its  partners—including other federal agencies. Within the NIPP framework,  DHS is responsible for leading and coordinating the overall national effort  to enhance protection via 18 critical infrastructure and key resources  (CIKR) sectors. Homeland Security Presidential Directive (HSPD) 7 and  the NIPP assign responsibility for CIKR sectors to sector-specific  agencies (SSA). As an SSA, DHS has direct responsibility for leading,  integrating, and coordinating efforts of sector partners to protect 11 of the  18 CIKR sectors. The remaining sectors are coordinated by eight other  federal agencies. Table 2 lists the SSAs and their sectors.
section_title: Appendix II: Objectives, Scope, and Methodology
paragraphs: To meet our first objective—determine the extent to which DHS has taken  action to conduct security surveys and vulnerability assessments among  high-priority CIKR—we reviewed DHS guidelines on the promotion and  implementation of the security surveys and vulnerability assessments,  records of outreach to CIKR stakeholders regarding these tools, and  various DHS documents, including DHS’s National Critical Infrastructure  and Key Resources Protection Annual Report, on efforts to complete  security surveys and vulnerability assessments. We also interviewed  officials in the Protective Security Coordination Division, which is part of  the Office of Infrastructure Protection (IP) in DHS’s National Protection  and Program Directorate, who are responsible for managing and  administering DHS’s security surveys and vulnerability assessments to  learn about the actions they took to conduct these programs. We  obtained and analyzed DHS data on the conduct of voluntary programs  for fiscal years 2009 through 2011—which are maintained in DHS’s Link  Encrypted Network System (LENS) database and compared those  records with the National Critical Infrastructure Prioritization Program  (NCIPP) list of the high-priority CIKR assets—to determine the extent to  which DHS performed security surveys and vulnerability assessments at  high-priority assets. To assess the reliability of the data, we reviewed  existing documentation about the data systems and spoke with  knowledgeable agency officials responsible for matching the two  databases to discuss the results of our comparison and to learn about  their efforts to match LENS data with the NCIPP lists. While the  information in each database was sufficiently reliable for the purposes of  providing a general overview of the program, issues with the  comparability of information in each database exist, which are discussed  in this report. To do our comparison, we used a Statistical Analysis  System (SAS) program to match the different data sets and summarize  the results. Because we found that assets in the LENS database and  NCIPP lists did not share common formats or identifiers that allowed us to  easily match them, we had to match the data based on asset names and  addresses. However, names and addresses were generally not entered in  a standardized way, so we had to develop a process to standardize the  available information and identify potential matches based on similar  names or addresses. In our attempt to match the data sets, we did the  following:
Standardized the date formats for fields that tracked when  assessments were conducted (dates across lists might have formats  such as 01/01/10 or 1/1/2010 and needed to be standardized to  ensure appropriate matching within certain time frames).
Standardized the labels for sectors (across data sets, a sector might  be listed as Chemical & Hazardous Materials Industry, Chemical and  Hazardous Materials Indus, or ‘Chemical’).
Standardized state fields (across data sets, a state might be listed as  Alabama or AL).  Identified exact matches between the data sets on the asset name  and the state name.  Identified potential matches between the data sets based on asset  name, asset address, and state. Specifically, we used a SAS function  (SPEDIS) that measures asymmetric spelling distance between  words, to determine the likelihood that names and addresses from two  data sets did match and to generate possible pairs of matching  assets. The possible matches for an asset were written to a  spreadsheet, which we reviewed to determine a potential match.
As noted in the report, the inconsistencies between the data sets  prevented us from determining definitively the extent to which assets on  one list were also present in the other list. For example, in some cases  assets seemed to be potential matches but there were differences in the  sector listed or inconsistent company names and addresses. Thus we  report separately on assets that were exact matches based on asset  name and those that were potential matches. We also examined the  inconsistencies we found with respect to DHS’s guidance on gathering  data on participation in the security survey and vulnerability assessments  and compared the findings to the criteria in Standards for Internal Control  in the Federal Government. We also compared the results of our  analyses with GAO reports on performance measurement, including ways  to use program data to measure results.
In addition, to address the first objective, we also interviewed  representatives—asset owners and operators—at 10 selected assets,  also known as facilities, in 4 of the 18 sectors—the water, dams,  commercial facilities, and energy sectors—to discuss their views on DHS  efforts to work with asset owners and operators and conduct DHS’s  voluntary security surveys and vulnerability assessments. We also  contacted industry association representatives from the 4 sectors to  discuss their views on DHS efforts to promote and conduct these  activities. We selected these asset and industry representatives to take  into account (1) sectors with a mix of regulations related to security; (2)  sectors where DHS’s IP and non-DHS agencies are the SSAs—DHS for  the commercial facilities sector and dams sector, the Department of  Energy for the energy sector, and the Environmental Protection Agency  for the water sector; (3) sectors where security surveys and vulnerability  assessments had been conducted; and (4) geographic dispersion. We  selected three states—California, New Jersey, and Virginia—where,  based on our preliminary review of DHS’s LENS database and the NCIPP  lists, security surveys and vulnerabilities assessments may have been  performed at high-priority assets. At these assets, we, among other  things, focused on the role of protective security advisors (PSA) who  serve as liaisons between DHS and security stakeholders, including asset  owners and operators, in local communities. We also reviewed PSA  program guidance and interviewed 4 of 88 PSAs—PSAs from California,  New Jersey and from the National Capital Region (encompassing  Washington, D.C., suburban Virginia, and suburban Maryland)—to  discuss the roles and responsibilities in partnering with asset owners and  operators and in promoting security surveys and vulnerability  assessments. While the results of our interviews cannot be generalized to  reflect the views of all asset owners and operators and PSAs nationwide,  the information obtained provided insights into DHS efforts to promote  participation in its security survey and vulnerability assessment programs.
We also conducted a survey of 83 of 88 PSAs, those who, based on lists  provided by DHS officials, had been in their positions for at least 1 year.  We conducted the survey to gather information on PSAs’ efforts to  promote and implement security surveys and vulnerability assessments,  and identify challenges PSAs face when conducting these. GAO staff  familiar with the critical infrastructure protection subject matter designed  draft questionnaires in close collaboration with a social science survey  specialist. We conducted pretests with three PSAs to help further refine  our questions, develop new questions, clarify any ambiguous portions of  the survey, and identify any potentially biased questions. We launched  our web-based survey on October 3, 2011, and received all responses by  November 18, 2011. Log-in information for the web-based survey was e- mailed to all participants. We sent one follow-up e-mail message to all  nonrespondents 2 weeks later and received responses from 80 out of 83  PSAs surveyed (96 percent).
Because the survey was conducted with all eligible PSAs, there are no  sampling errors. However, the practical difficulties of conducting any  survey may introduce nonsampling errors. For example, differences in  how a particular question is interpreted, the sources of information  available to respondents, or the types of people who do not respond can  introduce unwanted variability into the survey results. We included steps  in both the data collection and data analysis stages to minimize such  nonsampling errors. We collaborated with a GAO social science survey  specialist to design draft questionnaires, and versions of the  questionnaire were pretested with 3 PSAs. In addition, we provided a  draft of the questionnaire to DHS’s IP for review and comment. From  these pretests and reviews, we made revisions as necessary. We  examined the survey results and performed computer analyses to identify  inconsistencies and other indications of error. A second independent  analyst checked the accuracy of all computer analyses.
Regarding our second objective—to determine the extent to which DHS  shared the results of security surveys and vulnerability assessments with  asset owners and operators and SSAs—we reviewed available DHS  guidelines and reports on efforts to share security survey and vulnerability  assessment results with stakeholders and compared DHS’s sharing of  information with standards in the NIPP. We accessed, downloaded, and  analyzed LENS data for information regarding the asset owners and  operators that participated in DHS security surveys and vulnerability  assessments during fiscal years 2009 through 2011. To assess the  reliability of the data, we spoke with knowledgeable agency officials about  their quality assurance process. During the course of our review DHS  began taking action to clean up the data and address some of the data  inconsistencies we discuss in this report. We found the data to be  sufficiently reliable for providing a general overview of the program, but  issues with the missing information in the LENS database exist and are  discussed in this report. We compared the results of our analysis with  DHS criteria regarding the timeliness of security surveys and vulnerability  assessments, criteria in Standards for Internal Control in the Federal  Government, and the NIPP.
We also used the LENS database, the NCIPP lists, and DHS  documentation showing all assets that had received a security survey or a  vulnerability assessment to select a nonrandom sample of high-priority  assets from 4 sectors—the commercial facilities, dams, energy, and water  sectors—and spoke with representatives from these selected assets to  garner their opinions on the value of these voluntary programs and how  they used the information DHS shared with them. In addition, we  reviewed the 2009 and 2010 sector annual reports and the 2010 sector- specific plans for all CIKR sectors to assess if and how results of the  security surveys and vulnerability assessments were included. We also  interviewed SSA officials from our 4 selected sectors to learn what  information DHS shared with them and how that information was used,  and to discuss their overall relationship with DHS with respect to receiving  and using data from DHS security surveys and vulnerability assessments.  While the results of these interviews cannot be generalized to all SSAs,  the results provided us with valuable insight into the dissemination and  usefulness of information DHS provided from security surveys and  vulnerability assessments. Furthermore, we interviewed DHS officials  regarding their efforts to enhance the information they provide to SSAs  from security surveys and vulnerability assessments.
With regard to our third objective—determine the extent to which DHS  assessed the effectiveness of the security survey and vulnerability  assessment programs, including any action needed to improve DHS’s  management of the programs—we reviewed DHS documents and our  past reports, and DHS Office of Inspector General (OIG) reports on  DHS efforts to assess the effectiveness of its programs. We interviewed  DHS officials and reviewed DHS guidelines on procedures for following  up with asset owners and operators that have participated in these  programs and to discuss the results of DHS efforts to conduct these  follow-ups. We also (1) examined DHS documents that discussed the  results of DHS efforts to conduct follow-ups and (2) analyzed the  instrument used to contact owners and operators, as well as the  questions asked to assess its effectiveness. In addition, we analyzed  available data on DHS efforts to perform follow-ups for the period from  January 2011 through September 30, 2011, and compared DHS data with  DHS guidelines that discussed the number of days DHS officials were to  begin follow-ups after providing the results of security surveys and  vulnerabilities to asset owners and operators. We also compared the  results or our work with criteria in Standards for Internal Control in the  Federal Government and the NIPP, particularly those related to  performance measurement. Finally, we spoke to CIKR officials in our  sample sectors to learn how DHS personnel in the field had followed up  on security surveys and vulnerability assessments and whether asset  owners and operators were making changes based on the results, and if  not why.
We conducted this performance audit from June 2011 through May 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix III: Results of the Survey of Protective Security Advisors
paragraphs: This appendix provides information on our survey of Protective Security  Advisors, which we used to gather information on efforts to promote and  implement the voluntary programs offered by DHS and the challenges  faced when conducting security surveys and vulnerability assessments.  We conducted a Web-based survey of all 83 Protective Security Advisors  who had been in their positions for at least one year. We received  responses from 80, for a response rate of 96 percent. Our survey was  composed of closed- and open-ended questions. In this appendix, we  include all the survey questions and aggregate results of responses to the  closed-ended questions; we do not provide information on responses  provided to the open-ended questions. Percentages may not total to 100  due to rounding. For a more detailed discussion of our survey  methodology, see appendix II.
subsections: section_title: Survey Respondent Information
paragraphs: 1. Please provide the following information about the Protective Security Advisor responsible for completing this questionnaire.
Number of years as a PSA  (Round up to nearest year) 2. Did you receive the Enhanced Critical Infrastructure Protection (ECIP) Initiative Standard Operating Procedures (SOP) guidance  dated February 2011?  80  3. (If yes to Q2) How useful did you find the ECIP SOP guidance for promoting ECIPs?
If you answered "slightly useful" or "not at all useful", please explain why:  4. (If yes to Q2) How useful did you find the ECIP SOP guidance for conducting ECIPs?
If you answered "slightly useful" or "not at all useful", please explain why:  5. Did you receive training on the Enhanced Critical Infrastructure Protection (ECIP) Initiative program?  6. (If yes to Q5) How useful did you find the ECIP training?
If you answered "slightly useful" or "not at all useful", please explain why:   7. In your opinion, how useful is the ECIP Initiative program for reducing risk at CI facilities?
Please explain your opinion about the usefulness of the ECIP Initiative program:  8. In your opinion, how useful is the ECIP Infrastructure Survey Tool (IST) for reducing risk at CI facilities?
Please explain your opinion about the usefulness of the ECIP IST:   9. In your opinion, how useful is the ECIP Facility Dashboard for reducing risk at CI facilities?
10. How often have you heard each of the following reasons from facilities who declined to participate in an ECIP site visit?   (Select one answer in each row.)  a. The facility does not want to participate in additional facility assessments because it is already subject to Federal or State  regulation/inspection.  b. The facility does not have time or resources to participate.
80  c. Facility owners and operators are not willing to sign Protected Critical Infrastructure Information Express statements due to legal  concerns over the protection and dissemination of the data collected.  d. The entity that owns/oversees the facility declines to participate as a matter of policy.  e. Facility owners and operators have a diminished perception of threat against the facility.  f. The facility already received a risk assessment through a private company and participation in the voluntary assessment would be  redundant or duplicative.  g. Identification of security gaps may render the owner of the facility liable for damages should an incident occur.
What other reasons, if any, have you heard for facilities declining ECIP site visits?   11. Have you found that higher priority facilities (Level 1 or 2) are more or less likely to participate in ECIP site visits than lower priority  facilities?   80  12. If you answered somewhat less likely or much less likely, what do you see as the reasons for the lower participation by the higher  priority facilities?   13. What factors do you believe are important to facilities considering participating in an ECIP site visit?   14. How often have you heard each of the following reasons from facilities who declined to participate in an ECIP IST?   (Select one answer in each row.)  a. The facility does not want to participate in additional facility assessments because it is already subject to Federal or State  regulation/inspection.  b. The facility does not have time or resources to participate.  c. Facility owners and operators are not willing to sign Protected Critical Infrastructure Information Express statements due to legal  concerns over the protection and dissemination of the data collected.  d. The entity that owns/oversees the facility declines to participate as a matter of policy.  e. Facility owners and operators have a diminished perception of threat against the facility.
79  f. The facility already received a risk assessment through a private company and participation in the voluntary assessment would be  redundant or duplicative.  g. Identification of security gaps may render the owner of the facility liable for damages should an incident occur.  h. Facility's security program is not yet mature enough to benefit from participation.
What other reasons, if any, have you heard for facilities declining to participate in an ECIP IST?   15. Have you found that higher priority facilities (Level 1 or 2) are more or less likely to participate in ECIP ISTs than lower priority  facilities?   16. If you answered somewhat less likely or much less likely, what do you see as the reasons for the lower participation by the higher  priority facilities?   17. How much of an incentive do you believe each of the following are for encouraging participation in an ECIP IST?   (Select one answer in each row.)
No incentive   Don't know   d. Appeal to public service (patriotic duty)
If you responded not applicable to any of the sectors above, please explain.   20. Are you aware of any factors that drive differing levels of participation in the voluntary ECIP Initiative program by sector? Please  explain.   21. In your opinion, how useful are SAVs as a tool for reducing risk at CI facilities?
Please explain your opinion about the usefulness of SAVs:   22. How often have you heard each of the following reasons from facilities who declined to participate in a SAV?   (Select one answer in each row.)  a. The facility does not want to participate in additional facility assessments because it is already subject to Federal or State  regulation/inspection.  b. The facility does not have time or resources to participate.  c. Facility owners and operators are not willing to sign Protected Critical Infrastructure Information Express statements due to legal  concerns over the protection and dissemination of the data collected.  d. The entity that owns/oversees the facility declines to participate as a matter of policy.
80  e. Facility owners and operators have a diminished perception of threat against the facility.  f. The facility already received a risk assessment through a private company and participation in the voluntary assessment would be  redundant or duplicative.  g. Identification of security gaps may render the owner of the facility liable for damages should an incident occur.  h. Facility's security program is not yet mature enough to benefit from participation.
If you responded not applicable to any of the sectors above, please explain.   28. Are you aware of any factors that drive differing levels of participation in the voluntary SAV program by sector? Please explain.   29. What challenges, if any, do you face when implementing voluntary CI protection programs associated with ECIPs and SAVs?
30. Are you ready to submit your final completed survey to GAO?  (This is equivalent to mailing a completed paper survey to us. It tells us that your answers are official and final.)
No, my survey is not yet complete - To  submit your final responses,  please click on "Exit" below"  save your responses for later, please  click on "Exit" below"  You may view and print your completed survey by clicking on the Summary link in the menu to the left.
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, John F. Mortin, Assistant  Director, and Anthony DeFrank, Analyst-in-Charge, managed this  assignment. Andrew M. Curry, Katherine M. Davis, Michele C. Fejfar, Lisa  L. Fisher, Mitchell B. Karpman, Thomas F. Lombardi, and Mona E.  Nichols-Blake made significant contributions to the work.
section_title: Related GAO Products
paragraphs: Critical Infrastructure Protection: DHS Has Taken Action Designed to  Identify and Address Overlaps and Gaps in Critical Infrastructure Security  Activities. GAO-11-537R. Washington, D.C.: May 19, 2011.
Critical Infrastructure Protection: DHS Efforts to Assess and Promote  Resiliency Are Evolving but Program Management Could Be  Strengthened. GAO-10-772. Washington, D.C.: September 23, 2010.
Critical Infrastructure Protection: Update to National Infrastructure  Protection Plan Includes Increased Emphasis on Risk Management and  Resilience. GAO-10-296. Washington, D.C.: March 5, 2010.
The Department of Homeland Security’s (DHS) Critical Infrastructure  Protection Cost-Benefit Report. GAO-09-654R. Washington, D.C.:   June 26, 2009.
Information Technology: Federal Laws, Regulations, and Mandatory  Standards to Securing Private Sector Information Technology Systems  and Data in Critical Infrastructure Sectors. GAO-08-1075R. Washington,  D.C.: September 16, 2008.
Risk Management: Strengthening the Use of Risk Management Principles  in Homeland Security. GAO-08-904T. Washington, D.C.: June 25, 2008.
Critical Infrastructure: Sector Plans Complete and Sector Councils  Evolving. GAO-07-1075T. Washington, D.C.: July 12, 2007.
Critical Infrastructure Protection: Sector Plans and Sector Councils  Continue to Evolve. GAO-07-706R. Washington, D.C.: July 10, 2007.
Critical Infrastructure: Challenges Remain in Protecting Key Sectors.  GAO-07-626T. Washington, D.C.: March 20, 2007.
Homeland Security: Progress Has Been Made to Address the  Vulnerabilities Exposed by 9/11, but Continued Federal Action Is Needed  to Further Mitigate Security Risks. GAO-07-375. Washington, D.C.:  January 24, 2007.
Critical Infrastructure Protection: Progress Coordinating Government and  Private Sector Efforts Varies by Sectors’ Characteristics. GAO-07-39.  Washington, D.C.: October 16, 2006.
Information Sharing: DHS Should Take Steps to Encourage More  Widespread Use of Its Program to Protect and Share Critical  Infrastructure Information. GAO-06-383. Washington, D.C.: April 17,  2006.
Risk Management: Further Refinements Needed to Assess Risks and  Prioritize Protective Measures at Ports and Other Critical Infrastructure.  GAO-06-91. Washington, D.C.: December 15, 2005.
Protection of Chemical and Water Infrastructure: Federal Requirements,  Actions of Selected Facilities, and Remaining Challenges. GAO-05-327.  Washington, D.C.: March 28, 2005.
Homeland Security: Agency Plans, Implementation, and Challenges  Regarding the National Strategy for Homeland Security. GAO-05-33.  Washington, D.C.: January 14, 2005.