Cybersecurity: DHS's National Integration Center Generally Performs Required Functions but Needs to Evaluate Its Activities More Completely
Published: 2017-02-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Cyber-based intrusions and attacks on federal systems and systems supporting our nation's critical infrastructure, such as communications and financial services, have become more numerous, damaging, and disruptive. GAO first designated information security as a government-wide high-risk area in 1997. This was expanded to include the protection of critical cyber infrastructure in 2003 and protecting the privacy of personally identifiable information in 2015. The National Cybersecurity Protection Act of 2014 and the Cybersecurity Act of 2015 require NCCIC to perform 11 cybersecurity-related functions, including sharing information and enabling real-time actions to address cybersecurity risks and incidents at federal and non-federal entities.
The two acts also contained provisions for GAO to report on NCCIC's implementation of its cybersecurity mission. For this report, GAO assessed the extent to which the NCCIC was performing the 11 required functions. To do this, GAO analyzed relevant program documentation, interviewed officials, and conducted a non-generalizable survey of 2,792 federal and nonfederal recipients of NCCIC products and services.
section_title: What GAO Found
paragraphs: The National Cybersecurity and Communications Integration Center (NCCIC) of the Department of Homeland Security (DHS) has taken steps to perform each of its 11 statutorily required cybersecurity functions, such as being a federal civilian interface for sharing cybersecurity-related information with federal and nonfederal entities. It manages several programs that provide data used in developing 43 products and services in support of the functions. The programs include monitoring network traffic entering and exiting federal agency networks and analyzing computer network vulnerabilities and threats. The products and services are provided to its customers in the private sector; federal, state, local, tribal, and territorial government entities; and other partner organizations. For example, NCCIC issues indicator bulletins, which can contain information related to cyber threat indicators, defensive measures, and cybersecurity risks and incidents and help to fulfill its function to coordinate the sharing of such information across the government.
The National Cybersecurity Protection Act also required NCCIC to carry out its functions in accordance with nine implementing principles, to the extent practicable. However, the extent to which NCCIC adhered to the 9 principles when performing the functions is unclear because the center has not yet determined the applicability of the principles to all 11 functions, or established metrics and methods by which to evaluate its performance against the principles. GAO identified instances where NCCIC had implemented its functions in accordance with one or more of the principles. For example, consistent with the principle that it seek and receive appropriate consideration from industry sector-specific, academic, and national laboratory expertise, NCCIC coordinated with contacts from industry, academia, and the national laboratories to develop and disseminate vulnerability alerts. On the other hand, GAO also identified instances where the cybersecurity functions were not performed in accordance with the principles. For example, NCCIC is to provide timely technical assistance, risk management support, and incident response capabilities to federal and nonfederal entities; however, it had not established measures or other procedures for ensuring the timeliness of these assessments. Until NCCIC determines the applicability of the principles to its functions and develops metrics and methods to evaluate its performance against the principles, the center cannot ensure that it is effectively meeting its statutory requirements.
In addition, GAO identified factors that impede NCCIC's ability to more efficiently perform several of its cybersecurity functions. For example, NCCIC officials were unable to completely track and consolidate cyber incidents reported to the center, thereby inhibiting its ability to coordinate the sharing of information across the government. Similarly, NCCIC may not have ready access to the current contact information for all owners and operators of the most critical cyber-dependent infrastructure assets. This lack could impede timely communication with them in the event of a cyber incident. Until NCCIC takes steps to overcome these impediments, it may not be able to efficiently perform its cybersecurity functions and assist federal and nonfederal entities in identifying cyber-based threats, mitigating vulnerabilities, and managing cyber risks.
section_title: What GAO Recommends
paragraphs: GAO recommends nine actions to DHS for enhancing the effectiveness and efficiency of NCCIC, including to determine the applicability of the implementing principles and establish metrics and methods for evaluating performance; and address identified impediments. DHS concurred with GAO's recommendations.

section_title: Letter
paragraphs: Cyber-based intrusions and attacks on federal systems and systems  supporting our nation’s critical infrastructure, such as communications  and financial services, have become not only more numerous and diverse  but also more damaging and disruptive. This is illustrated by the data  breach at the Office of Personnel Management, reported in July 2015,  which impacted at least 21.5 million individuals and demonstrated the  effect that such an incident can have on an agency’s mission and national  security. Protecting the information and control systems on which federal  operations and national critical infrastructures depend and effectively  responding to cyber incidents is critical because the unauthorized  disclosure, alteration, and destruction of the information on those systems  can result in great harm to those involved.
Since 1997, we have designated federal information security as a  government-wide high-risk area and, in 2003, expanded this area to  include computerized systems supporting the nation’s critical  infrastructure. More recently, in the February 2015 update to our high- risk list, we further expanded this area to include protecting the privacy of  personally identifiable information that is collected, maintained, and  shared by both federal and nonfederal entities.
In 2009, the Department of Homeland Security (DHS) developed an  integration center, the National Cybersecurity and Communications  Integration Center (NCCIC), to provide a central place for the various  federal and private-sector organizations to coordinate efforts to address  and respond to cyber threats. The National Cybersecurity Protection Act  of 2014 requires NCCIC to perform several cybersecurity functions,  including being a federal civilian interface for sharing information on  cybersecurity-related information and facilitating cross-sector coordination  to address cybersecurity risks and incidents. The act also required the  center to adhere to nine principles, to the extent practicable, in carrying  out these functions. One principle, for example, is ensuring that timely,  actionable, and relevant information related to cybersecurity risks,  incidents, and analysis is shared. The Cybersecurity Act of 2015  subsequently established additional functions for the center, among other  things. These acts together identified 11 cybersecurity functions that the  center is to perform.
In addition, the two acts included provisions for us to review NCCIC’s  efforts in carrying out its cybersecurity mission. For this review, our  specific objective was to determine the extent to which NCCIC was  performing its statutorily defined cybersecurity-related functions.
To do so, we reviewed the National Cybersecurity Protection Act of 2014  and the Cybersecurity Act of 2015 to identify the center’s statutorily  defined functions and the principles for carrying out the functions. For  each of the 11 functions identified in the acts, we analyzed relevant  program documentation, including concepts of operations, program  descriptions, analysis products, resource allocations, and performance  measures. We also interviewed relevant NCCIC officials and analyzed the  evidence provided to identify the products and services that the center  develops and disseminates to federal and nonfederal entities and to learn  how information sharing is coordinated and facilitated. We examined how  these products and services are used in support of NCCIC’s mandated  cybersecurity functions. In addition, we assessed the products and  services against the nine principles outlined in the National Cybersecurity  Protection Act of 2014 and interviewed relevant agency officials to  determine how the center adhered to the principles.
Further, to supplement our work, we developed and administered a  survey to a nongeneralizable sample of 2,792 recipients of the center’s  products and services to determine the effectiveness of current  operations, products, and services in meeting their needs. The sample  was randomly selected from a population of over 19,000 recipients  maintained by NCCIC. The recipients included representatives from  federal agencies and nonfederal entities, such as private-sector  companies and organizations; state, local, tribal, and territorial  governments; international partners; and individual citizens. We began  our survey on August 2, 2016, and ended it on September 8, 2016.
The response rate to the survey, calculated as the number of usable  responses received, divided by the number in the original sample found  to be eligible was about 14 percent. Because of the low rate of response  to the survey and other factors, the results can represent only illustrative  responses from those responding, and are not generalizable to any larger  population of recipients.
We conducted this performance audit from January 2016 to February  2017, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objective. Additional details on our  objective, scope, and methodology are contained in appendix I.
subsections: section_title: Background
paragraphs: Federal agencies and our nation’s critical infrastructures—such as  energy, transportation systems, communications, and financial services— are dependent on computerized (cyber) information systems and  electronic data to carry out operations and to process, maintain, and  report essential information. Federal and nonfederal operations are  largely supported by computer systems and electronic data, and  organizations would find it difficult, if not impossible, to carry out their  missions, deliver services to the public, and account for their resources  without these cyber assets. Information security is, thus, especially  important for federal and nonfederal entities to ensure the confidentiality,  integrity, and availability of their systems and data.
Conversely, ineffective information security controls can result in  significant risk to a broad array of operations and assets, as the following  examples illustrate:
Computer resources could be used for unauthorized purposes or to  launch attacks on other computer systems.
Sensitive information, such as personally identifiable information,  intellectual property, and proprietary business information, could be  inappropriately disclosed, browsed, or copied for purposes of identity  theft, espionage, or other crimes.
Critical operations, such as those supporting critical infrastructure,  national defense, and emergency services, could be disrupted.
Data could be added, modified, or deleted for purposes of fraud,  subterfuge, or disruption.
subsections: section_title: The Nation Faces an Evolving Array of Cyber- Based Threats
paragraphs: Threats to systems are evolving and growing. Cyber threats can be  unintentional or intentional. Unintentional or nonadversarial threat sources  include failures in equipment or software due to aging, resource  depletion, or other circumstances that exceed expected operating  parameters, as well as errors made by end users. They also include  natural disasters and failures of critical infrastructure on which the  organization depends, but that are outside of the control of the  organization.
Intentional or adversarial threats include individuals, groups, entities, or  nations that seek to leverage the organization’s dependence on cyber  resources (i.e., information in electronic form, information and  communications technologies, and the communications and information- handling capabilities provided by those technologies). Threats can come  from a wide array of sources, including corrupt employees, criminal  groups, and terrorists. These threat adversaries vary in terms of their  capabilities, their willingness to act, and their motives, which can include  seeking monetary gain, or seeking an economic, political, or military  advantage.
Cyber threat adversaries make use of various techniques, tactics, and  practices, or exploits, to adversely affect an organization’s computers,  software, or networks, or to intercept or steal valuable or sensitive  information. These exploits are carried out through various conduits,  including websites, e-mails, wireless and cellular communications,  Internet protocols, portable media, and social media. Further, adversaries  can leverage computer software programs as a means by which to deliver  a threat by embedding exploits within software files that can be activated  when a user opens a file within its corresponding program.
Reports of successfully executed cyber exploits illustrate the debilitating  effects they can have on the nation’s security and economy, and on  public health and safety. Federal and nonfederal entities have  experienced security breaches in their networks, potentially allowing  sensitive information to be compromised, and systems, operations, and  services to be disrupted. The examples that follow illustrate that a broad  array of personal information and critical infrastructures are at risk:  In October 2016, the International Atomic Energy Agency reported  that a cyber attack had caused a disruption to the operations of a  power plant. The agency did not disclose details about the information  gathered or what specific operations were disrupted.
In September 2016, Yahoo Incorporated, a multinational company,  confirmed that 500 million user accounts were compromised. Yahoo  company officials reported that the account information may have  included names, e-mail addresses, telephone numbers, and dates of  birth.
In August 2015, the Internal Revenue Service reported that  approximately 390,000 tax accounts were potentially affected by  unauthorized third parties gaining access to taxpayer information from  the agency’s “Get Transcript” application. According to testimony from  the Commissioner of Internal Revenue in June 2015, criminals had  used taxpayer-specific data acquired from nonagency sources to gain  unauthorized access to information, although at that time, the  commissioner reported that approximately 100,000 tax accounts had  been affected. The data included Social Security information, dates of  birth, and street addresses.
In July 2015, the Office of Personnel Management reported that an  intrusion into its systems had compromised the background  investigation files of 21.5 million individuals. This was in addition to a  separate but related incident that had affected the personnel records  of about 4 million current and former federal employees, which the  agency announced in June 2015.
In April 2015, the Department of Veterans Affairs’ Office of Inspector  General reported that two contractors had improperly accessed the  agency’s network from foreign countries using personally owned  equipment.
section_title: Federal Law Established NCCIC as the Federal Civilian Center for Cybersecurity
paragraphs: In 2009, DHS developed NCCIC to provide a central place for federal and  private-sector organizations to coordinate efforts to address cyber threats  and respond to cyber attacks. The center’s stated mission is to reduce  the likelihood and severity of incidents that may significantly compromise  the security and resilience of the nation’s critical information technology  and communications networks.
The National Cybersecurity Protection Act of 2014 statutorily established  the center’s role within DHS to act as a federal civilian interface for  sharing information related to cybersecurity risks, incidents, analysis, and  warnings with federal and nonfederal entities, and to provide shared  situational awareness to enable real-time actions to address  cybersecurity risks and incidents to federal and nonfederal entities. The  Cybersecurity Act of 2015 added roles for NCCIC and required DHS to  create and issue several related policies and procedures.
Table 1 describes the 11 cybersecurity functions that NCCIC is to carry  out, as prescribed by the National Cybersecurity Protection Act of 2014  and the Cybersecurity Act of 2015.
Further, the National Cybersecurity Protection Act of 2014 states that the  center shall ensure that it carries out these functions, to the extent  practicable, in accordance with the following 9 principles:  1.  Ensure that timely, actionable, and relevant information related to  risks, incidents, and analysis is shared.  2.  Ensure that when appropriate, information related to risks, incidents,  and analysis is integrated with other information and tailored to a  sector.  3.  Ensure that the activities are prioritized and conducted based on the  level of risk.  4.  Ensure that industry sector-specific, academic, and national  laboratory expertise is sought and receives appropriate consideration.  5.  Ensure that continuous, collaborative, and inclusive coordination  occurs across sectors, with sector coordination councils, information  sharing and analysis organizations, and other nonfederal partners.  6.  Ensure that, as appropriate, the center works to develop and use  mechanisms for sharing information related to cybersecurity risks and  incidents that are technology-neutral, interoperable, real-time, cost- effective, and resilient.  7.  Ensure that the center works with other agencies to reduce  unnecessarily duplicative sharing of information related to  cybersecurity risks and incidents.  8.  Ensure that information related to cybersecurity risks and incidents is  appropriately safeguarded against unauthorized access.  9.  Ensure that activities conducted comply with all policies, regulations,  and laws that protect the privacy and civil liberties of United States  persons.
section_title: Four Branches Within NCCIC Perform Cybersecurity-related Functions
paragraphs: To perform its functions, NCCIC is organized into four branches:  United States Computer Emergency Readiness Team (US-CERT) is  responsible for leading efforts to improve the nation’s cybersecurity  posture, coordinate cyber information sharing, and proactively manage  cyber risks to the government and private sector.
Industrial Control Systems (ICS) Cyber Emergency Response Team  (ICS-CERT) is responsible for taking steps to reduce risk to the nation’s  critical infrastructure by strengthening control systems security and  resilience through public-private partnerships. In executing its mission,  ICS-CERT is to serve its partners as the preeminent federal government  resource for industrial control systems security.
National Coordinating Center for Communications (NCC) is  responsible for helping government, private industry, and international  partners to share and analyze threat information about, assess the  operating status of, and understand the risk posture of the  communications infrastructure. In addition, it is to coordinate efforts to  prepare for, prevent, protect against, mitigate, respond to, and recover  from significant communications disruptions.
NCCIC Operations & Integration (NO&I) is responsible for engaging in  planning, coordination, and integration capabilities to synchronize  analysis, information sharing, and incident response efforts across the  center’s branches and activities.
Figure 1 shows the organizational structure of NCCIC.
According to DHS policy, the cyber situational awareness, incident  response, and management efforts of NCCIC’s four branches are to  occur on a 24-hour-a-day, 7-day-a-week basis at an integrated operations  center known as the Watch Floor, as shown in figure 2.
section_title: NCCIC Works with Federal and Nonfederal Partners in Support of its Cybersecurity Mission
paragraphs: According to DHS policy, the center is to collaborate with federal  departments and agencies most responsible for securing the  government’s cyber and communications systems. DHS policy and law  also states that it is to engage with critical infrastructure owners and  operators; other private sector entities; state, local, tribal, and territorial governments; and international partners. These federal and nonfederal  entities, as well as individual citizens, represent NCCIC’s customers that  are recipients of its products, such as documents about threats, malware  or digital media analyses, or software vulnerabilities; and services, which  can include support for the testing of emergency communications,  incident response, or vulnerability assessments.
subsections: section_title: Federal Organizations and Cybersecurity Centers
paragraphs: According to DHS policy, NCCIC is to coordinate with its federal partners  that focus on securing the federal information infrastructure, with the  intent of integrating cyber center information to provide cross-domain  situational awareness, analysis, and reporting on the composite state of  U.S. cyber networks and communication infrastructure. In addition, the  Comprehensive National Cybersecurity Initiative requires the center to  foster partnerships with other key federal cybersecurity and  communications centers in order to collaborate and improve cybersecurity  and communications infrastructure across the federal government. The  federal centers include:
Defense Cyber Crime Center sets standards for digital evidence  processing, analysis, and diagnostics for Department of Defense  investigations that require computer forensic support to detect,  enhance, or recover digital media, including audio and video.
Intelligence Community Security Coordination Center provides attack  sensing and warning capabilities to characterize cyber threats and  attributions of attacks, and anticipates future incidents.
National Cyber Investigative Joint Task Force, organized by the  Federal Bureau of Investigation, serves as a focal point for all  government agencies to coordinate, integrate, and share information  related to domestic cyber threat investigations.
National Security Agency/Central Security Service Threat Operations  Center establishes real-time network awareness and threat  characterization capabilities to forecast, alert, and attribute malicious  activity.
United States Cyber Command Joint Operations Center establishes  and maintains situational awareness and directs the operations and  defense of the “.mil” networks.
National Infrastructure Coordinating Center is the coordination and  information sharing operations center that maintains situational  awareness of the nation’s critical infrastructure for the federal  government.
DHS policy also states that each of these federal cybersecurity and  communications centers is to provide complementary capabilities and  resources that collectively form the threat characterization, vulnerability  analysis, information sharing, detection and response, investigation, and  defense of civilian federal cyber networks and communication  infrastructures.
section_title: Private Sector and Critical Infrastructure Owners and Operators
paragraphs: NCCIC works with the private sector that owns and operates most of the  nation’s critical infrastructure, such as banking and financial institutions,  telecommunications networks, and energy production and transmission  facilities. Infrastructure owners and operators are to integrate (both  physically and virtually) into the center’s operations so that, during an  incident, information can be aggregated and communicated between  government and appropriate private sector partners in an efficient  manner. As of August 2016, 174 private sector companies had as-needed  access to NCCIC through their participation in the Cyber Information  Sharing and Collaboration Program (CISCP).
As part of this effort, NCCIC is to coordinate on an ongoing basis with  various private sector partners, including information sharing and analysis  centers (ISAC) and technology vendors. ISACs have been formed for a  number of sectors. According to the National Council of ISACs, these  include (1) automotive; (2) aviation; (3) defense industrial base; (4)  emergency services; (5) electricity; (6) financial services; (7) healthcare,  (8) information technology; (9) maritime security; (10) communications;  (11) multistate; (12) national health; (13) oil and gas; (14) public transit;  (15) real estate; (16) retail; (17) research and education; (18) supply chain; (19) surface transportation; and (20) water. As of October 2016,  five nonfederal entities maintained a permanent presence on the NCCIC  Watch Floor (the ISACs of the financial, national health, aviation, and  energy sectors, as well as the Multi-State ISAC).
section_title: State, Local, Tribal, and Territorial Governments
paragraphs: NCCIC maintains partnerships with state, local, tribal, and territorial  governments to support their protection of each respective community.  These governments are responsible for the security and integrity of their  own cyber networks, along with associated preparedness, mitigation, and  response efforts.
The center is to facilitate overarching situational awareness and the  sharing of technical information and best practices with these nonfederal  government partners to help ensure a strengthened national cyber risk  posture. As part of this effort, NCCIC is to coordinate on an ongoing basis  with the Multi-State Information Sharing and Analysis Center (MS-ISAC),  and an MS-ISAC representative is physically located on the NCCIC  Watch Floor.
section_title: International Partners
paragraphs: As part of its mission, NCCIC also engages and collaborates with  international partners, including governments that are members of the  North Atlantic Treaty Organization (NATO), to disseminate bulletins and  perform services. The center works with international partners and their  respective cyber centers while conducting cyber-related exercises —as  part of a March 2016 exercise, representatives from Australia, Canada,  Denmark, Finland, Germany, Hungary, Japan, Netherlands, New  Zealand, Sweden, Switzerland, and the United Kingdom participated.  NCCIC also can assist international partners with responding to cyber  incidents. For example, during the cyber attack against the Ukrainian  power infrastructure in December 2015, the center collaborated with the  Ukrainian government to determine the methods of the cyber attack.
section_title: NCCIC Reported Spending about $480 Million over the Past Three Fiscal Years
paragraphs: NCCIC reported that it spent about $480 million on cybersecurity-related  activities during fiscal years 2014 through 2016. According to the center’s  officials, this included spending for 262, 268, and 301 full-time employees  for each of the three fiscal years, respectively. Figure 3 depicts the  reported expenditures per year by each of the four branches of NCCIC.
section_title: Although NCCIC Has Taken Steps to Perform Required Cybersecurity Functions, the Extent to Which It Carries Them Out In Accordance with Implementing Principles Is Unclear
paragraphs: NCCIC has taken steps to perform each of its 11 statutorily required  cybersecurity functions. It has developed a variety of products and  services in support of these functions, including those related to analyzing  and sharing cyber information, facilitating coordination among federal and  nonfederal partners, and conducting technical assistance and exercises.  However, the extent to which NCCIC carried out these functions in  accordance with the nine principles specified in the National  Cybersecurity Protection Act of 2014 is unclear because the center has  not consistently evaluated its performance against the principles. In  addition, a number of factors impede NCCIC’s ability to more efficiently  perform several of its cybersecurity functions. Although not generalizable  to any larger population, recipients of its products and services that  responded to our survey expressed generally favorable views of its  activities. Nevertheless, NCCIC has limited assurance that it is fully  meeting statutory requirements and efficiently performing its cybersecurity  functions because it has not completely evaluated its performance  against the principles or addressed the impediments to performing its  cybersecurity functions.
subsections: section_title: NCCIC Has Developed a Variety of Products and Services in Support of Required Cybersecurity Functions
paragraphs: NCCIC has developed 43 types of products and services in support of its  11 statutorily required functions. Descriptions of these products and  services as well as the total numbers of each provided to NCCIC’s  customers during fiscal years 2015 and 2016 are discussed in greater  detail in appendix II. The center manages several programs that provide  data used in developing the products and performing the services related  to its cybersecurity functions. These programs include:
The National Cybersecurity Protection System, operationally known  as EINSTEIN, monitors network traffic entering or exiting networks of  federal agencies and provides intrusion detection and intrusion  prevention services. NCCIC analysts use data logged by EINSTEIN to  notify federal and nonfederal partners of potential breaches of  information security.
The Advanced Malware Analysis Center is a set of capabilities  intended to provide a segregated, closed, computer network system  that is used to analyze computer network vulnerabilities and threats.  According to NCCIC officials, information transmitted to NCCIC  through the Advanced Malware Analysis Center may include  malicious codes, computer viruses, worms, spyware, bots, and Trojan  horses. Once received, analysts use the malware analysis capabilities  to analyze the code or images in order to discover how to secure or  defend computer systems against the threat. The corrective action  information is then published in products such as vulnerability reports  or alerts or malware reports.
The Automated Indicator Sharing program was created to provide  real-time sharing of cyber threat indicators and defensive measures  by enabling NCCIC to (1) receive cyber threat indicators and  defensive measures submitted by its nonfederal participants and  federal entities; (2) remove personally identifiable information and  other sensitive information that is not directly related to a  cybersecurity threat; and (3) disseminate the cyber threat indicators  and defensive measures to its nonfederal participants and federal  entities, as appropriate. The Automated Indicator Sharing program  uses the DHS-developed Structured Threat Information  Expression/Trusted Automated Exchange of Indicator Information  formats, a mechanism for sharing cyber threat information in a  common manner. NCCIC uses this program to send out machine- readable cyber threat indicators at near-real-time and is now on- boarding participants across the public and private sectors. NCCIC  officials stated that the Automated Indicator Sharing program was first  disseminated to the 5 cyber centers. Since then, the program has  become accessible to additional entities. According to the officials, as  of August 2016, 32 private sector entities representing 6 critical  infrastructure sectors and 7 federal agencies were connected to the  program. NCCIC officials stated that DHS is in the process of  expanding the service to all 24 Chief Financial Officers Act agencies  in response to guidance from the Office of Management and Budget  from October 2015.
The following summarizes NCCIC’s products and services that support  its 11 statutorily required functions. Details on how all 43 products and  services support each of the cybersecurity functions are in appendix III.
Function 1: Be a federal civilian interface for the multidirectional and  cross-sector sharing of information related to cyber threat indicators,  defensive measures, cybersecurity risks, incidents, analysis and warnings  for federal and nonfederal entities.
NCCIC has nine products and services that support this function. Among  these, it provides products such as Cyber Information Sharing and  Collaboration Program (CISCP) Indicator Bulletins and US-CERT  Indicator Bulletins, which can include cyber threat indicators, defensive  measures, cybersecurity risks, incidents, analysis, and warnings. For  fiscal year 2016, the center developed and disseminated 151 US-CERT  Bulletins. For example, one US-CERT Bulletin identified Internet protocol  addresses that had conducted unauthorized scans of networks of partner  entities.
NCCIC also provides services to interface with federal and nonfederal  entities. Through the center’s Information Sharing and Liaison Services,  representatives from federal and nonfederal sectors are able to reside  permanently or temporarily on the Watch Floor alongside NCCIC officials,  to better ensure multidirectional, cross-sector sharing of information.  According to the officials, there are seven seats available on the Watch  Floor that its partners can reserve as a temporary residence. As of  August 2016, the center reported agreements with 118 entities that could  elect to reside temporarily on the Watch Floor. We observed additional  cross-sector information sharing through the presence on the NCCIC  Watch Floor of liaison officers from the other five cyber centers; members  from the Multi-State, Communications, and Financial Services ISACs; and  the intelligence community working in conjunction with NCCIC analysts.
Function 2: Provide shared situational awareness to enable real-time,  integrated, and operational actions across the federal government and  nonfederal entities to address cybersecurity risks and incidents to federal  and nonfederal entities.
NCCIC supports this function with the use of 12 products and services.  Among these products, it provides situational awareness to its customers  to enable real-time, integrated, operational actions. Through one such  product, Watch Floor Situation Reports, the center provides awareness  of incidents and recommendations on remediation. For example, one  report disseminated to its partners identified current events related to  Ransomware incidents directed towards hospitals. As part of the report,  the center identified immediate and future actions in support of resolving  the incidents. In addition, in providing notifications, such as National  Coordinating Center for Communications (NCC) Watch Train  Derailment and GPS Testing Notices, the center shared information to  support operational actions on behalf of the partner entities. Further,  NCCIC provided situational awareness of potentially malicious Internet  protocol addresses through Victim/Abuse notifications.
Function 3: Coordinate the sharing of information related to cyber threat  indicators, defensive measures, cybersecurity risks and incidents across  the federal government.
NCCIC has nine products and services that support this function. Among  the products are CISCP Bulletins, US-CERT Bulletins, Joint Analysis  Reports, and Joint Indicator Bulletins that can contain information  related to cyber threat indicators, defensive measures, cybersecurity risks  and incidents. For example, the center issued eight Joint Analysis  Reports during fiscal years 2015 and 2016. One report, issued jointly with  the Federal Bureau of Investigation on April 14, 2015, was also  coordinated with the Departments of Treasury and Energy. The report  contained a summation of open-source analysis related to common  vulnerabilities leveraged by state-sponsored cyber operators in products  such as Adobe Flash, Adobe Reader, Microsoft Office, Microsoft server  software, and OpenSSL. The report contained information on the specific  version of the product affected, as well as the associated information to  patch the vulnerability.
According to NCCIC officials, the center relies on the NCCIC Portal as a  mechanism to coordinate the sharing of these products to customers.
Specifically, the portal is comprised of 35 compartments, which include  customers across the globe, and within government and various critical  infrastructures. Each of the compartments represents a grouping of  entities with a similar role or focus. For example, the Government Forum  of Incident Response and Security Teams are comprised of individuals  from federal civilian and military agencies responsible for securing  government information technology systems.
Function 4: Facilitate cross-sector coordination to address cybersecurity  risks and incidents, including cybersecurity risks and incidents that may  be related or could have consequential impacts, across multiple sectors.
NCCIC has six products and services that support this function. For  example, the center facilitates cross-sector coordination to address  cybersecurity risks and incidents through its Industrial Control Systems  Joint Working Group and its Incident Notifications. In particular, the  joint working group holds biannual meetings with the industrial control  system community. For example, the most recent meeting occurred on  May 3–5, 2016, and had over 300 stakeholders represented. According to  the after-action report, representatives from several sectors, including  officials from the energy, water, transportation, and nuclear sectors,  among others, attended the meeting.
Function 5: Conduct and share integration and analysis, including cross- sector, of cyber threat indicators, defensive measures, cybersecurity risks  and incidents with federal and nonfederal entities.
NCCIC has eight products and services that support this function. For  example, the US-CERT Analysis Report is an integrated analysis  document that can contain indicators of compromise and tactics,  techniques, and procedures related to specific threats. Further, US-CERT  officials stated that the center provides common vulnerabilities to the  National Vulnerability Database, which is an established, open source of  indicators used by information security professionals located across the  nation and throughout the world. Further, ICS-CERT officials stated that it  provides industrial control system vulnerabilities to over 15,000 “.gov” e- mail addresses that are signed up to receive ICS-CERT Vulnerability  Alerts.
Function 6: Provide timely technical assistance, risk management  support, and incident response capabilities to federal and nonfederal  entities with respect to cyber threat indicators, defensive measures, and  cybersecurity risks and incidents, which may include attribution,  mitigation, and remediation.
NCCIC supports this function with the use of five products and services.  With these products, it has the capacity to provide technical assistance,  risk management support, and incident response capabilities to  customers upon request. For example, in responding to and conducting  incident response analyses for public or private sector customers, US- CERT developed Incident Response Team Reports that outlined  mitigation recommendations to the customers.
In addition, to support risk management, the center conducted, as  services, Risk and Vulnerability Assessments, which are activities to  assist entities in developing strategies for improving their cybersecurity  posture. According to officials, NCCIC attempts to provide a report of its  findings to the requesting entity within 30 days of the assessment.
Further, NCCIC provided Cyber Assessments of control systems. For  example, the Cyber Security Evaluation Tool (CSET®) can be  downloaded to conduct a self-evaluation of an entity’s cybersecurity  posture against, among other things, best practices and National Institute  of Standards and Technology recommendations. In addition, ICS-CERT  officials stated that upon a customer’s request, NCCIC can provide further  assistance by conducting industrial control system architectural  assessments and network assessments.
Function 7: Provide information and recommendations on security and  resilience measures to federal and nonfederal entities, including  information and recommendations to facilitate information security and  strengthen information systems against cybersecurity risks and incidents;  and share cyber threat indicators and defensive measures.
NCCIC has 16 products and services that support this function. Among  these products, the center provided information and recommendations on  security and resilience measures through its Preliminary Digital Media  Analysis Report and Digital Media Analysis Report products.  Specifically, for these products, it conducted analysis of digital media and  provided a report that includes analysis of the exploits and associated  mitigation strategies. Further, according to NCCIC officials, the US-CERT  and ICS-CERT components conduct incident response activities (known  as US-CERT Incident Response Team Report and ICS Incident  Response Deployment) and develop reports to document their findings  at the request of partner entities. These reports can contain  recommendations to strengthen information systems against  cybersecurity risks and incidents and potentially share cyber threat  indicators and defensive measures.
Function 8: Engage with international partners, in consultation with other  appropriate agencies, to (a) collaborate on cyber threat indicators,  defensive measures, and information related to cybersecurity risks and  incidents; and (b) enhance the security and resilience of global  cybersecurity.
NCCIC supports this function with the use of 10 products and services. It  engages with international partners to collaborate on cyber threat  indicators, defensive measures, and information related to cybersecurity  risks and incidents. For example, the most recent Cyberstorm exercise— the department’s national-level exercise series—was conducted during  the spring of 2016 and involved more than 1,200 participants including  NCCIC’s national and international partners. According to the exercise  after-action report, 12 international partners participated in the  Cyberstorm exercise. To enhance the security and reliance of global  cybersecurity, the after-action report identified areas of improvement  relating to the escalation of incidents and coordination of public and  private efforts. ICS-CERT also collaborated with the Ukrainian  government in the aftermath of a cyber attack on its power infrastructure  to develop and disseminate vulnerability alerts, reports, and briefings on  the attack. US-CERT, on a different occasion, collaborated with Canada  on developing vulnerability alerts associated with ransomware.
Function 9: Share cyber threat indicators, defensive measures, and other  information related to cybersecurity risks and incidents with federal and  nonfederal entities, including across sectors of critical infrastructure and  with state and major urban area fusion centers, as appropriate.
NCCIC relies on four products and services to support this function. For  example, it shared cyber threat indicators, defensive measures, and  information through its Malware Initial Findings Reports and Malware  Analysis Reports. These reports are based on NCCIC malware analysis  conducted at the request of the customer. They can contain indicators,  such as a description of the malware artifact, as well as defensive  measures, such as the Internet Protocol addresses potentially associated  with the malware.
Both types of reports are disseminated via the NCCIC Portal. The center  used a Traffic Light Protocol (TLP), which is a designation to ensure that  sensitive information is shared with the appropriate audience. MS-ISAC  representatives have access to the NCCIC portal and can share  information with its members per TLP protections.
Function 10: Participate, as appropriate, in national exercises run by the  department.
NCCIC has four products and services that support this function. For  example, in addition to Cyberstorm, the center conducted and  participated in external exercises for customers to support the  improvement of national and international cybersecurity. NCCIC officials  stated that these external exercises include federal, state, local, tribal,  territorial, private, and international partners and range from individual  table-top exercises to multi-organization exercises. The center conducted  such an exercise in October 2015 with a state government to improve its  communication capabilities and provided a seminar on the current threats  to control systems worldwide.
Function 11: Coordinate with the Office of Emergency Communication of  the Department, assessing and evaluating consequence, vulnerability,  and threat information regarding cyber incidents to public safety  communications to help facilitate continuous improvements to the security  and resiliency of such communications.
NCCIC has four products and services that support this function. Among  its activities, the center engages with the Office of Emergency  Communications in planning and preparing for disasters and incidents,  including cyber incidents, to ensure continued readiness of the  communications network. NCCIC officials stated they meet weekly with  Office of Emergency Communications Regional coordinators during a  NCC Weekly Operations call to discuss threats and vulnerabilities.  According to NCCIC, the Office of Emergency Communications is a  supporting partner of its execution of national coordinator responsibilities  for Emergency Support Function 2–Communications under the National  Response Framework. In addition, officials from the center have briefed  the National Council of Statewide Interoperability Coordinators on how to  share cyber data using NCCIC’s incident reporting process at a national  conference in April 2016.
section_title: NCCIC’s Adherence to the Implementing Principles in Carrying Out its Cybersecurity Functions Is Unclear
paragraphs: NCCIC is required to carry out its functions in accordance with nine  principles specified in the National Cybersecurity Protection Act of 2014,  to the extent practicable. As previously described, these principles,  among other things, relate to ensuring that industry sector-specific,  academic, and national laboratory expertise is sought and receives  appropriate consideration; the information related to cybersecurity risks  and incidents is appropriately safeguarded against unauthorized access;  and shared information is timely, actionable, and relevant to risks,  incidents, and analysis.
The extent to which NCCIC carried out its 11 cybersecurity functions in  accordance with the nine principles specified in the act is unclear. We  identified instances where, with certain products and services, NCCIC  had implemented its functions in adherence with one or more of the  principles. For example, consistent with the principle that it seek and  receive appropriate consideration from industry sector-specific, academic,  and national laboratory expertise, NCCIC coordinated with contacts from  industry, academia, and the national laboratories to develop and  disseminate vulnerability alerts through the National Vulnerability  Database. In addition, to comply with the principle that the information  related to cybersecurity risks and incidents be appropriately safeguarded  against unauthorized access, the center used the TLP designation to  ensure that sensitive information was shared with the appropriate  audience. Specifically, NCCIC disseminated its products via the NCCIC  portal, using the protocol for products such as Indicator Bulletins, Analysis  Reports, Malware Initial Findings Reports, and Malware Analysis  Reports. (Additional examples of how NCCIC products and services  helped the center implement its functions according to the principles are  provided in appendix IV.)
On the other hand, we also identified instances where the cybersecurity  functions were not performed in adherence with the principles. For  example, with regard to function 6, NCCIC is to provide timely technical  assistance, risk management support, and incident response capabilities  to federal and nonfederal entities. The function is supported, in part, by  Risk and Vulnerability Assessments. However, NCCIC had not  established measures or other procedures for ensuring the timeliness of  these assessments. According to officials responsible for this service, the  assessments have an estimated completion time frame of 8-10 weeks for  each customer. However, the officials stated that this time frame is not an  established metric by which they evaluate the timeliness of the service.  Further, NCCIC had not established measures or procedures to assess  the actionability of its products and services. For example, US-CERT  Indicator Bulletins, a product that supports several functions, typically  contain actionable information, such as specific malicious Internet  addresses to be blocked. NCCIC had not established a means of  determining the extent to which a particular bulletin helped to mitigate a  risk or prevent an incident.
In discussing this matter, NCCIC officials acknowledged that they had not  made a complete determination of the applicability of the principles with  all of the center’s functions and thus had not established measures and  procedures for assessing its products and services against the principles.  The officials stated that they have begun to map activities supporting the  cybersecurity functions to the implementing principles. For example,  according to the officials, the center established a unit for reviewing and  making recommendations to improve overall NCCIC operations. During  fiscal year 2016, this unit completed performance management reviews  across the center’s programs to identify areas in which NCCIC could  better align its operations with its overall requirements, including the  principles. Further, officials from the ICS-CERT branch stated that they  were in the preliminary stages of measuring their activities against one of  the nine principles. Specifically, the officials stated that 20 metrics were  being developed that would measure timeliness, relevance, and  actionability (principle 1) across the components of their organization.
Nevertheless, while these preliminary actions are important steps, they do  not represent a complete determination of the applicability of all nine  principles across all of NCCIC’s statutorily-required cybersecurity  functions. As such, NCCIC officials could not say whether the principles  did or did not apply to all of the 11 functions. Moreover, because a  complete determination of the applicability of the nine principles had not  been done, the center also had not developed metrics and methods for  assessing and ensuring adherence with the principles. Until the center  determines the applicability of the implementing principles for all of its  functions and develops the metrics and methods necessary to ensure that  the principles are met, it will not be positioned to ensure that NCCIC is  effectively meeting its statutory requirements.
section_title: NCCIC Faces Impediments to Performing Its Cybersecurity Functions More Efficiently
paragraphs: In addition to NCCIC not having made a complete determination of how it  is adhering to the principles, a number of factors impede the center’s  ability to more efficiently perform several of its cybersecurity functions. In  particular, the center faces impediments in tracking security incidents;  maintaining current and reliable customer information, to include  obtaining such information on all owners and operators of the most critical  infrastructure assets; working across multiple network platforms; and  collaborating with international partners.
Tracking of security incidents is not centralized or reconciled. The  National Cybersecurity Protection Act of 2014 requires NCCIC to  coordinate the sharing of information across the government. This  includes information related to cyber threat indicators, defensive  measures, and cybersecurity risks and incidents.
However, NCCIC officials were unable to completely track and  consolidate cyber incidents reported to the center, thereby inhibiting its  ability to coordinate the sharing of information across the government. For  US-CERT-related incidents, personnel assigned to the NCICC service  desk generated a daily report of the current status of the open incident  tickets. For example, the July 18, 2016 report had a total of 520 incident  tickets. However, this report did not represent the totality of incidents  across the center because it did not include incidents reported to ICS- CERT. Since the NCCIC service desk did not have access to the data  within the ICS-CERT ticketing system, it could not produce a  management report on the status of all incidents reported to the center.
NCCIC officials attributed the lack of a single, centralized incident tracking  system to the fact that ICS-CERT and US-CERT had operated as  separate entities prior to the establishment of the center. As such, both  ICS-CERT and US-CERT has its own incident ticketing system. Senior  ICS-CERT officials stated they are aware of this challenge and are  exploring options on how best to integrate the two systems. Until such  integration takes place, NCCIC will continue to encounter difficulty in  completely tracking the total efforts of its branches to address reported  cybersecurity incidents. As a result, the center will be challenged in  determining how effective it is in sharing information related to cyber  threat indicators, defensive measures, and cybersecurity risks and  incidents across the government.
The difficulty of logging incident data is further compounded by the  multiple ways in which an incident can be reported to the center. US- CERT officials stated that there are six preferred ways in which NCCIC  receives information related to potential incidents. For example, to  communicate with US-CERT, customers can choose e-mail, a phone call,  an automatic submission form, or an automated machine-to-machine  submission as the means to notify US-CERT of an incident. In addition, to  communicate with ICS-CERT regarding an industrial control system- related incident, customers can choose to submit an e-mail or phone  directly to ICS-CERT (Figure 4 shows the ways in which NCCIC prefers  to receive reported incidents.)
However, contrary to the 6 preferred methods of communicating with US- CERT and ICS-CERT, officials from NCCIC’s Operations and Integration  office provided documentation that identified at least 22 methods by  which the center receives potential incidents. These methods would  include phone numbers and e-mail addresses other than the  aforementioned 6 methods, established by various groups within the four  NCCIC components as a means to communicate with partners.
In addition, according to NCCIC officials, depending on the method of  reporting, incidents are not always logged into the NCCIC incident  ticketing systems. For example, when customers have prior established  relationships, analysts can be called directly and can handle the incidents  without logging them into the system. The lack of control over the entry  points as well as inconsistencies in logging data, together; inhibit the  center in consistently tracking incidents and their status across the entire  NCCIC.
Until the center can reduce, consolidate, or modify the points of entry that  customer entities use to communicate with NCCIC, it will lack the ability to  better ensure that all incident tickets are logged appropriately. Thus,  further contributing to the center being less able to effectively perform its  statutorily-required function in coordinating the sharing of information  related to cyber threat indicators, defensive measures, and cybersecurity  risks and incidents across the government.
Maintaining current and reliable customer information. The National  Cybersecurity Protection Act of 2014 requires NCCIC to be the federal  civilian interface for the multidirectional and cross-sector sharing of  information related to cyber threat indicators, defensive measures,  cybersecurity risks and incidents, and analysis and warnings for federal  and nonfederal entities. To perform this function, the center needs to  have accurate and up-to-date contact information for the potential  recipients of the cybersecurity information it shares.
However, NCCIC’s contact information was not always up to date, thus  impacting its ability to effectively function as a federal civilian interface for  federal and nonfederal entities. Specifically, after e-mailing our survey to  recipients of NCCIC’s products and services, we received 303  undeliverable return messages out of 2,792 recipients contacted. We also  identified individuals who were included on the list of recipients that  NCCIC provided to us that no longer had the role the center indicated or  were no longer with the entity listed.
NCCIC officials were unable to demonstrate that they had any formal  process for maintaining customer contact information. The officials stated  that maintaining customer contact information was an ad hoc process and  acknowledged that capturing changes to that data was a challenge.
Without regularly validating data pertaining to its product and service  recipients, the center may lack quality information it needs to effectively  develop and maintain partnerships and share cybersecurity-related  information with federal and nonfederal entities to support its operation as  required by the statutes.
Obtaining contact information of all owners and operators of the  most critical cyber-dependent infrastructure assets. The National  Cybersecurity Protection Act of 2014 requires NCCIC to facilitate cross- sector coordination to address cybersecurity risks and incidents. This  includes cybersecurity risks and incidents that may be related or could  have consequential impacts across multiple sectors.
However, representatives of federal and nonfederal entities that own  critical cyber assets that could have a catastrophic impact on the nation if  victimized by a cyber attack were not fully represented in the NCCIC  customer log. Specifically, our review found that 23 percent of the  entities owning such critical assets (as determined by DHS) were not  represented within the master NCCIC customer log as of September  2016. Without representation of these entities, NCCIC may not have the  information it needs readily available to facilitate coordination with critical  asset owners. NCCIC officials were unable to demonstrate that they had  a formal internal process for maintaining customer contact information  and acknowledged that doing so remains a challenge for the center.
Without a concentrated effort on ensuring the full representation of the  owners and operators of these critical assets, the center lacks assurance  that it is adequately facilitating the cross-sector coordination of  cybersecurity risk and incidents to the nation’s most critical cyber- dependent assets that, if impacted, could have a catastrophic effect on  the nation.
Working across multiple network platforms. The National  Cybersecurity Protection Act of 2014 requires NCCIC to coordinate the  sharing of information across the government. This includes information  related to cybersecurity risks and incidents.
However, we found that the sharing of information is complicated by  NCCIC analysts having to operate across multiple networks, often  manually entering data into each network, which decreases the rate of  response of coordinating the sharing of incident information to customers  and increases the risk of false entry. For example, officials stated that it  takes on average 3 minutes for a ticket to be closed when working within  one network. Across the 3 systems, it could take up to 15 minutes,  depending on the size of the ticket, and the amount of information needed  to be manually entered into each system.
According to senior NCCIC officials, this impediment was attributed to a  legacy technical infrastructure implemented prior to the center’s  existence. They added that efforts were under way to address this  impediment. However, NCCIC had not developed an implementation plan  or established time frames for consolidating or integrating the networks.
Until NCCIC develops a process to avoid manual data entry, it will  continue to face challenges in efficiently sharing information related to  cyber threat indicators, defensive measures, and cybersecurity risks and  incidents across the government.
Collaborating with international partners using the NCCIC Portal.  The Cybersecurity Act of 2015 requires NCCIC to engage with  international partners, in consultation with other appropriate agencies, to  (a) collaborate on cyber threat indicators, defensive measures, and  information related to cybersecurity risks and incidents; and (b) enhance  the security and resilience of global cybersecurity.
International and other partners have access to the center’s products  through the NCCIC Portal, which has functioned as a mechanism to  disseminate products to recipients since 2003. However, DHS is  migrating the NCCIC portal to the Homeland Security Information  Network. This network is categorized as a FIPS 199 high-impact system  and, thus, requires authentication of individuals with access to the  system. According to NCCIC officials, international partners had  expressed a concern that the new network will have a negative impact on  their collaboration with NCCIC because continued access would require  the submission of international participants’ passports and other sensitive  personal information to a U.S. government entity.
While DHS has a responsibility to ensure the security over its high-impact  systems, NCCIC may face a barrier to engaging with international  partners. Without taking action to address this potential barrier,  international partners may be reluctant to engage with NCCIC. Thus, the  center may be challenged in its ability to collaborate and enhance global  cybersecurity if it does not find alternative methods to engage and share  information with international partners while ensuring the security  requirements of high-impact systems.
section_title: Survey Respondents Varied in Their Reported Use of NCCIC Products and Services but Had Generally Favorable Views of the Center’s Activities
paragraphs: The respondents to our nongeneralizable survey of the center’s activities  reported that they used its products and services to varying extents. The  respondents also expressed generally favorable views of the center’s  activities. Table 2 depicts the extent to which the survey respondents,  who each self-identified as a customer of an NCCIC component (US- CERT, ICS-CERT, NCC, the Watch Floor, and NO&I), used, did not use,  or were unsure if they used a particular NCCIC product or service.
With regard to evaluating the characteristics of products and services, the  respondents to our nongeneralizable survey generally reported that  NCCIC products and services were timely, relevant, and actionable.  Specifically, 289 of 333 respondents (87 percent) found products and  services they had used to be extremely, very, or moderately timely; 286 of  332 respondents (86 percent) found products and services to be  extremely, very, or moderately relevant; and 234 of 332 respondents (70  percent) stated that products and services have led to an actionable  result to a very great, great, or moderate extent (e.g., used to address a  vulnerability or apply a defensive measure) on their part.
In addition, although between 12 and 18 percent of respondents to our  nongeneralizable survey indicated a low level of effectiveness,  respondents had generally favorable views of the center’s provision of  cybersecurity information. Specifically, 236 of 335 respondents (70  percent) evaluated the provision of cyber threat indicators to be at a high  or moderate level of effectiveness. In addition, 219 of 333 respondents  (66 percent) identified risks and incidents to be at a high or moderate  level of effectiveness. Further, 211 of 339 respondents (62 percent)  indicated cyber defensive information to be at a high or moderate level of  effectiveness. Further, the survey respondents evaluated NCCIC’s ability  to provide timely, relevant, and actionable information at a 235 of 331 (71  percent), 245 of 334 (73 percent), and 222 of 339 (65 percent) high or  moderate level of effectiveness, respectively.
Table 3 shows survey respondents’ evaluations of NCCIC’s effectiveness  in providing them with cyber threat indicators, information on risks and  incidents and defensive measures, and information that was timely,  relevant, and actionable.
Survey respondents also evaluated the center’s effectiveness with regard  to its information sharing capability, the uniqueness in the information it  provides, and its partnerships with them in improving the protection of  critical cyber assets and functions, and how well it is fulfilling its mission.  Table 4 shows respondents’ overall evaluation of the center in terms of  the effectiveness of its information sharing capability, customer  partnerships, and the extent to which it is fulfilling its mission, among  other things.
Further, respondents regarded NCCIC as important to the nation’s ability  to protect critical cyber assets and functions. Specifically, 264 of 337  respondents (78 percent) to our nongeneralizable survey stated that there  would be a “very” or “somewhat” negative impact on the nation if the  NCCIC products and services did not exist.
However, not all survey responses were positive. Specifically, survey  respondents reported that they were not aware of all of the products and  services that the center offered. The respondents added that they would  be interested in receiving additional NCCIC products and services but  were unsure about how to begin receiving them. The respondents  reported that the center had not provided information identifying these  products and services. NCCIC officials acknowledged that customers  may not be aware of certain products and services because not all  products and services are meant for every customer.
section_title: Conclusions
paragraphs: NCCIC, as the federal civilian cyber center, is generally performing 11  required cybersecurity functions through the development and  dissemination of 43 products and services. However, the extent to which  NCCIC carried out these cybersecurity functions in accordance with the 9  implementing principles is unclear. Until it determines the extent to which  the implementing principles apply to these functions, NCCIC will not be  able to fully assess the extent to which it is meeting the mandated  principles. Further, without measuring the extent to which principles are  being met, NCCIC will be challenged in articulating how effectively it is  performing the functions in support of its role as a focal point for  cybersecurity incident coordination, information sharing, and incident  response across the federal civilian government and critical infrastructure.
NCCIC also faces several impediments that inhibits it from efficiently  performing its cybersecurity functions. These impediments relate to  consolidating entry points for receiving and logging potential incident data  and maintaining the center’s relationship with customers. Until NCCIC  takes steps to overcome these impediments it may not be able to  efficiently perform its cybersecurity functions and assist federal and  nonfederal entities in identifying cyber-based threats, mitigating  vulnerabilities, and managing cyber risks.
section_title: Recommendations for Executive Action
paragraphs: To more fully address the requirements identified in the National  Cybersecurity Protection Act of 2014 and the Cybersecurity Act of 2015,  we recommend that the Secretary of the Department of Homeland  Security take the following nine actions:  1.  Determine the extent to which the statutorily required implementing  principles apply to NCCIC’s cybersecurity functions.  2.  Develop metrics for assessing adherence to applicable principles in  carrying out statutorily required functions.  3.  Establish methods for monitoring the implementation of cybersecurity  functions against the principles on an ongoing basis.  4.  Integrate information related to security incidents to provide  management with more complete information about NCCIC  operations.  5.  Determine the necessity of reducing, consolidating, or modifying the  points of entry used to communicate with NCCIC to better ensure that  all incident tickets are logged appropriately.  6.  Develop and implement procedures to perform regular reviews of  customer information to ensure that it is current and reliable.  7.  Take steps to ensure the full representation of the owners and  operators of the nation’s most critical cyber-dependent infrastructure  assets.  8.  Establish plans and time frames for consolidating or integrating the  legacy networks used by NCCIC analysts to reduce the need for  manual data entry.  9.  Identify alternative methods to collaborate with international partners,  while ensuring the security requirements of high-impact systems.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from DHS.  In its  comments, the department concurred with all nine recommendations. The  department also provided details about steps that it plans to take to  address each of the recommendations, including estimated time frames  for completion. If effectively implemented, these actions should enhance  the effectiveness and efficiency of NCCIC in performing its statutory  requirements. The department’s written comments are reprinted in  appendix V.
In addition to the aforementioned comments, DHS also provided a  technical comment via e-mail, which we considered and incorporated.
We are sending copies of this report to the appropriate congressional  committees, the Department of Homeland Security, and other interested  parties. In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  Gregory Wilshusen at (202) 512-6244 or wilshuseng@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix VI.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to determine the extent to which the National  Cybersecurity and Communications Integration Center (NCCIC) was  performing its statutorily defined cybersecurity-related functions. To  determine this, we analyzed two acts that establish roles and  responsibilities for the center: the National Cybersecurity Protection Act of  2014 and the Cybersecurity Act of 2015. These laws together require the  center to carry out 11 cybersecurity functions. More specifically, the  National Cybersecurity Protection Act of 2014 prescribed 7 functions and  the Cybersecurity Act of 2015 prescribed 4 additional functions. The  National Cybersecurity Protection Act of 2014 also identified 9  implementing principles. The two acts also contained provisions for GAO  to report on NCCIC’s implementation of its cybersecurity mission.
To determine the extent to which it was addressing the 11 cybersecurity  functions, we analyzed the center’s program descriptions, concepts of  operations, and policies and procedures documenting how each of the  center’s components are to operate. For example, we analyzed the U.S.  Computer Emergency Readiness Team (US-CERT) Strategic Action  Plan, Industrial Control Systems-Cyber Emergency Response Team  (ICS-CERT) Five-Year Plan (2015-2019), and the National Coordinating  Center (NCC) Information Guide and Overview for 2015. In addition, we  analyzed the NCCIC Watch Floor Concept of Operations that describes  its operations. We corroborated information by interviewing center  officials, including the Assistant Secretary for Cybersecurity and  Communications, the Director of the NCCIC, and the directors of each of  the center’s components, as well as other responsible officials.
Based on our analysis of this information, we developed an initial list of  products and services and held interviews with officials to confirm the  total list of products and services. Based on these actions, we determined  that the center develops and disseminates 43 products and services. We  then collected and analyzed examples of each product and service to  determine how they addressed each of the 11 cybersecurity functions  established in the two laws. To gain a greater understanding of the  purposes and methods of developing the products and services, we also  interviewed NCCIC officials.
To identify instances where products and services addressed the 9  implementing principles, we analyzed relevant program documentation  and reviewed the procedures by which the center develops its products  and services. We corroborated our information by interviewing NCCIC  officials responsible for product and service development. To gain a  greater understanding of its operations, we visited the site of the ICS- CERT operations in Idaho Falls, Idaho, to observe its activities, including  the development of its products and services. We also observed  operations of the Watch Floors in Arlington, Virginia and Idaho Falls,  Idaho, and interviewed officials responsible for operating the Watch  Floors, developing  services, and liaising with federal and nonfederal  partners.
We also analyzed the dissemination methods of products and services by  examining the contents of the NCCIC web portal, including how the  customer base was segmented to disseminate products and services in  accordance with information sharing protections. At the ICS-CERT facility   in Idaho Falls, Idaho, we observed the basic ICS training exercise  services provided to customers. We also collected and analyzed  performance measures and interviewed officials about the actions being  taken to improve its measurement efforts and efforts to consolidate  operations across NCCIC components. In addition, NCCIC officials  provided budget execution information that we analyzed to determine the  reported amount spent across three fiscal years for each component.  During the interviews, we discussed with officials the impediments that  the center faced in more efficiently performing the 11 cybersecurity  functions.
To obtain the views of the recipients of the center’s products and  services, we administered a survey to a sample of individuals identified by  NCCIC as having access to a product or service, or participating in a  center group or activity. We asked customers about their awareness and  use of the 43 products and services, and other activities and roles  performed by the center. We then asked them to assess their  experiences, including rating the effectiveness and the implementing  principles of timeliness, actionability, and relevance. We also asked  respondents to rate various elements of NCCIC in terms of importance,  expectations, challenges, and reasons for not using the center’s products  and services.
To develop our questionnaire, we met with NCCIC officials and identified  the activities performed for customers, including the development and  dissemination of 43 products and services, disseminated to customers.  We pretested draft versions of the questionnaire with nonfederal  representatives of two Information Sharing and Analysis Centers and an  information security officer at a federal agency, to reflect some of the  variation in the population.
We defined the target population for the survey to be all organizational  points of contact or other individuals identified by NCCIC, as of June 8,  2016, as having access to a product or service, or participating in a center  group or activity. NCCIC provided us with 19,573 records across14 lists of  customer contact information. Some of the provided lists consisted only of  e-mail addresses of individuals subscribed to a particular NCCIC product  or service, while others consisted of members of a group. Some  organizations were represented by many individual e-mail addresses  across the lists, and some individuals appeared on more than one list.  While the basic unit of the population to be sampled was an individual e- mail address, due to the variability in coverage of the population  mentioned above, an individual survey respondent may be representing  their own personal experiences and opinions, or those of an organization,  and multiple respondents may be representing the same organization.
After removing records with missing, incomplete, or erroneously  duplicative e-mail addresses within each list, our sample was reduced to  19,293 records. We did not remove multiple instances of the same e-mail  address appearing on more than one list; these duplicates were retained  in the sample frame so that each instance of that e-mail address might  have a chance of initial selection proportional to the size of the customer  list it appeared on.
We initially drew a random but nongeneralizable sample of 2,907 e-mail  address records, allocated across the 16 customer types roughly  proportional to the sizes of each type. We then removed 115 of this initial  sample because their e-mail addresses duplicated selections made from  other customer lists, for a total sample of 2,792 customer records with  unique e-mail addresses, which we attempted to contact with our survey.
We began our survey on August 2, 2016. We sent e-mails with login  information to the web-based questionnaire to the sample. We sent up to  three follow-up e-mails during the fieldwork period to those who had not  yet responded. The survey ended on September 8, 2016. The outcomes  of the survey fieldwork are displayed in table 5 below.
The response rate to the survey, calculated as the number of usable  responses divided by the number found to be eligible was about 14  percent. Because of the variability in coverage of the population by the  sample frame, irregularities in the contact information and eligibility of the  records sampled, and the low rate of response to the survey, the results  of this survey only represent those that responded, and are not  generalizable to any larger population of NCCIC customers. We do not  make any inferences about those not sampled or not responding to the  survey.
In addition to this limitation, questionnaire surveys of this kind are subject  to other potential errors. To minimize the possibility of measurement error  (differences between reported and true answers) arising from question  design, interpretation or administration, or the misreporting of answers,  we designed and administered the survey in consultation with survey  methodologists, made improvements to the questionnaire based on  pretest results, and had a separate survey methodologist review the draft  questionnaire to identify potential problems in questionnaire design.
Of the 340 respondents, 14 percent identified themselves as individual  participants in NCCIC activities, 64 percent as representatives of a single  public or private organization, 13 percent as representing an association  or other entity representing a sector or group of organizations, and 9  percent identified in other ways. Thirty-four percent said they represented  federal government entities; 18 percent said they represented state, local,  or tribal entities; 44 percent said they represented private sector entities;  and 4 percent gave other answers.
During the processing and analysis of reported data, we also identified  and corrected for patterns of response across questions that we could  identify as inconsistent or contradictory. Nonresponse error (failure to  obtain a response to a question or the questionnaire) may lead to bias in  the results if those who do not respond would have given materially  different responses from those who did respond. To minimize  nonresponse, we made follow-up contacts throughout the survey. To  minimize processing error (mistakes in converting reported data into  published survey results), data processing and analysis programming  was independently verified by a separate data analyst.
We conducted this performance audit from January 2016 to February  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: NCCIC Product and Service Information
paragraphs: Table 6 below highlights the total number of each product and service  that the National Cybersecurity and Communications Integration Center  (NCCIC) reported providing to its customers in fiscal years 2015 and  2016.
section_title: Appendix III: NCCIC Products and Services Supporting Cybersecurity Functions
paragraphs: The National Cybersecurity and Communications Integration Center  (NCCIC) is required to perform 11 cybersecurity functions. Table 7 below  summarizes how the 43 products and services were being used as of  October 2016 in support of the 11 functions.
section_title: Appendix IV: Examples of How NCCIC Products and Services Address the Implementing Principles
paragraphs: Although the National Cybersecurity and Communications Integration  Center (NCCIC) did not completely determine the applicability of statutory  implementing principles to its products and services, table 8 below  provides examples of our determination of how NCCIC products and  services adhered to the principles.
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Michael W. Gilmore (assistant  director), Kush K. Malhotra (analyst in charge), Chris Businsky, Lee A.  McCracken, Constantine Papanastasiou, David Plocher, Carl Ramirez,  and Priscilla A. Smith made key contributions to this report.