Cybersecurity: DHS's National Integration Center Generally Performs Required Functions but Needs to Evaluate Its Activities More Completely
Released: 2017-02-01T00:00:00

Letter

Background

Federal agencies and our nation’s critical infrastructures—such as energy, transportation systems, communications, and financial services— are dependent on computerized (cyber) information systems and electronic data to carry out operations and to process, maintain, and report essential information. Federal and nonfederal operations are largely supported by computer systems and electronic data, and organizations would find it difficult, if not impossible, to carry out their missions, deliver services to the public, and account for their resources without these cyber assets. Information security is, thus, especially important for federal and nonfederal entities to ensure the confidentiality, integrity, and availability of their systems and data.

Conversely, ineffective information security controls can result in significant risk to a broad array of operations and assets, as the following examples illustrate:

Computer resources could be used for unauthorized purposes or to launch attacks on other computer systems.

Sensitive information, such as personally identifiable information, intellectual property, and proprietary business information, could be inappropriately disclosed, browsed, or copied for purposes of identity theft, espionage, or other crimes.

Critical operations, such as those supporting critical infrastructure, national defense, and emergency services, could be disrupted.

Data could be added, modified, or deleted for purposes of fraud, subterfuge, or disruption.

The Nation Faces an Evolving Array of Cyber- Based Threats

Threats to systems are evolving and growing. Cyber threats can be unintentional or intentional. Unintentional or nonadversarial threat sources include failures in equipment or software due to aging, resource depletion, or other circumstances that exceed expected operating parameters, as well as errors made by end users. They also include natural disasters and failures of critical infrastructure on which the organization depends, but that are outside of the control of the organization.

Intentional or adversarial threats include individuals, groups, entities, or nations that seek to leverage the organization’s dependence on cyber resources (i.e., information in electronic form, information and communications technologies, and the communications and information- handling capabilities provided by those technologies). Threats can come from a wide array of sources, including corrupt employees, criminal groups, and terrorists. These threat adversaries vary in terms of their capabilities, their willingness to act, and their motives, which can include seeking monetary gain, or seeking an economic, political, or military advantage.

Cyber threat adversaries make use of various techniques, tactics, and practices, or exploits, to adversely affect an organization’s computers, software, or networks, or to intercept or steal valuable or sensitive information. These exploits are carried out through various conduits, including websites, e-mails, wireless and cellular communications, Internet protocols, portable media, and social media. Further, adversaries can leverage computer software programs as a means by which to deliver a threat by embedding exploits within software files that can be activated when a user opens a file within its corresponding program.

Reports of successfully executed cyber exploits illustrate the debilitating effects they can have on the nation’s security and economy, and on public health and safety. Federal and nonfederal entities have experienced security breaches in their networks, potentially allowing sensitive information to be compromised, and systems, operations, and services to be disrupted. The examples that follow illustrate that a broad array of personal information and critical infrastructures are at risk: In October 2016, the International Atomic Energy Agency reported that a cyber attack had caused a disruption to the operations of a power plant. The agency did not disclose details about the information gathered or what specific operations were disrupted.

In September 2016, Yahoo Incorporated, a multinational company, confirmed that 500 million user accounts were compromised. Yahoo company officials reported that the account information may have included names, e-mail addresses, telephone numbers, and dates of birth.

In August 2015, the Internal Revenue Service reported that approximately 390,000 tax accounts were potentially affected by unauthorized third parties gaining access to taxpayer information from the agency’s “Get Transcript” application. According to testimony from the Commissioner of Internal Revenue in June 2015, criminals had used taxpayer-specific data acquired from nonagency sources to gain unauthorized access to information, although at that time, the commissioner reported that approximately 100,000 tax accounts had been affected. The data included Social Security information, dates of birth, and street addresses.

In July 2015, the Office of Personnel Management reported that an intrusion into its systems had compromised the background investigation files of 21.5 million individuals. This was in addition to a separate but related incident that had affected the personnel records of about 4 million current and former federal employees, which the agency announced in June 2015.

In April 2015, the Department of Veterans Affairs’ Office of Inspector General reported that two contractors had improperly accessed the agency’s network from foreign countries using personally owned equipment.

Federal Law Established NCCIC as the Federal Civilian Center for Cybersecurity

In 2009, DHS developed NCCIC to provide a central place for federal and private-sector organizations to coordinate efforts to address cyber threats and respond to cyber attacks. The center’s stated mission is to reduce the likelihood and severity of incidents that may significantly compromise the security and resilience of the nation’s critical information technology and communications networks.

The National Cybersecurity Protection Act of 2014 statutorily established the center’s role within DHS to act as a federal civilian interface for sharing information related to cybersecurity risks, incidents, analysis, and warnings with federal and nonfederal entities, and to provide shared situational awareness to enable real-time actions to address cybersecurity risks and incidents to federal and nonfederal entities. The Cybersecurity Act of 2015 added roles for NCCIC and required DHS to create and issue several related policies and procedures.

Table 1 describes the 11 cybersecurity functions that NCCIC is to carry out, as prescribed by the National Cybersecurity Protection Act of 2014 and the Cybersecurity Act of 2015.

Further, the National Cybersecurity Protection Act of 2014 states that the center shall ensure that it carries out these functions, to the extent practicable, in accordance with the following 9 principles: 1. Ensure that timely, actionable, and relevant information related to risks, incidents, and analysis is shared. 2. Ensure that when appropriate, information related to risks, incidents, and analysis is integrated with other information and tailored to a sector. 3. Ensure that the activities are prioritized and conducted based on the level of risk. 4. Ensure that industry sector-specific, academic, and national laboratory expertise is sought and receives appropriate consideration. 5. Ensure that continuous, collaborative, and inclusive coordination occurs across sectors, with sector coordination councils, information sharing and analysis organizations, and other nonfederal partners. 6. Ensure that, as appropriate, the center works to develop and use mechanisms for sharing information related to cybersecurity risks and incidents that are technology-neutral, interoperable, real-time, cost- effective, and resilient. 7. Ensure that the center works with other agencies to reduce unnecessarily duplicative sharing of information related to cybersecurity risks and incidents. 8. Ensure that information related to cybersecurity risks and incidents is appropriately safeguarded against unauthorized access. 9. Ensure that activities conducted comply with all policies, regulations, and laws that protect the privacy and civil liberties of United States persons.

Four Branches Within NCCIC Perform Cybersecurity-related Functions

To perform its functions, NCCIC is organized into four branches: United States Computer Emergency Readiness Team (US-CERT) is responsible for leading efforts to improve the nation’s cybersecurity posture, coordinate cyber information sharing, and proactively manage cyber risks to the government and private sector.

Industrial Control Systems (ICS) Cyber Emergency Response Team (ICS-CERT) is responsible for taking steps to reduce risk to the nation’s critical infrastructure by strengthening control systems security and resilience through public-private partnerships. In executing its mission, ICS-CERT is to serve its partners as the preeminent federal government resource for industrial control systems security.

National Coordinating Center for Communications (NCC) is responsible for helping government, private industry, and international partners to share and analyze threat information about, assess the operating status of, and understand the risk posture of the communications infrastructure. In addition, it is to coordinate efforts to prepare for, prevent, protect against, mitigate, respond to, and recover from significant communications disruptions.

NCCIC Operations & Integration (NO&I) is responsible for engaging in planning, coordination, and integration capabilities to synchronize analysis, information sharing, and incident response efforts across the center’s branches and activities.

Figure 1 shows the organizational structure of NCCIC.

According to DHS policy, the cyber situational awareness, incident response, and management efforts of NCCIC’s four branches are to occur on a 24-hour-a-day, 7-day-a-week basis at an integrated operations center known as the Watch Floor, as shown in figure 2.

NCCIC Works with Federal and Nonfederal Partners in Support of its Cybersecurity Mission

According to DHS policy, the center is to collaborate with federal departments and agencies most responsible for securing the government’s cyber and communications systems. DHS policy and law also states that it is to engage with critical infrastructure owners and operators; other private sector entities; state, local, tribal, and territorial governments; and international partners. These federal and nonfederal entities, as well as individual citizens, represent NCCIC’s customers that are recipients of its products, such as documents about threats, malware or digital media analyses, or software vulnerabilities; and services, which can include support for the testing of emergency communications, incident response, or vulnerability assessments.

Federal Organizations and Cybersecurity Centers

According to DHS policy, NCCIC is to coordinate with its federal partners that focus on securing the federal information infrastructure, with the intent of integrating cyber center information to provide cross-domain situational awareness, analysis, and reporting on the composite state of U.S. cyber networks and communication infrastructure. In addition, the Comprehensive National Cybersecurity Initiative requires the center to foster partnerships with other key federal cybersecurity and communications centers in order to collaborate and improve cybersecurity and communications infrastructure across the federal government. The federal centers include:

Defense Cyber Crime Center sets standards for digital evidence processing, analysis, and diagnostics for Department of Defense investigations that require computer forensic support to detect, enhance, or recover digital media, including audio and video.

Intelligence Community Security Coordination Center provides attack sensing and warning capabilities to characterize cyber threats and attributions of attacks, and anticipates future incidents.

National Cyber Investigative Joint Task Force, organized by the Federal Bureau of Investigation, serves as a focal point for all government agencies to coordinate, integrate, and share information related to domestic cyber threat investigations.

National Security Agency/Central Security Service Threat Operations Center establishes real-time network awareness and threat characterization capabilities to forecast, alert, and attribute malicious activity.

United States Cyber Command Joint Operations Center establishes and maintains situational awareness and directs the operations and defense of the “.mil” networks.

National Infrastructure Coordinating Center is the coordination and information sharing operations center that maintains situational awareness of the nation’s critical infrastructure for the federal government.

DHS policy also states that each of these federal cybersecurity and communications centers is to provide complementary capabilities and resources that collectively form the threat characterization, vulnerability analysis, information sharing, detection and response, investigation, and defense of civilian federal cyber networks and communication infrastructures.

Private Sector and Critical Infrastructure Owners and Operators

NCCIC works with the private sector that owns and operates most of the nation’s critical infrastructure, such as banking and financial institutions, telecommunications networks, and energy production and transmission facilities. Infrastructure owners and operators are to integrate (both physically and virtually) into the center’s operations so that, during an incident, information can be aggregated and communicated between government and appropriate private sector partners in an efficient manner. As of August 2016, 174 private sector companies had as-needed access to NCCIC through their participation in the Cyber Information Sharing and Collaboration Program (CISCP).

As part of this effort, NCCIC is to coordinate on an ongoing basis with various private sector partners, including information sharing and analysis centers (ISAC) and technology vendors. ISACs have been formed for a number of sectors. According to the National Council of ISACs, these include (1) automotive; (2) aviation; (3) defense industrial base; (4) emergency services; (5) electricity; (6) financial services; (7) healthcare, (8) information technology; (9) maritime security; (10) communications; (11) multistate; (12) national health; (13) oil and gas; (14) public transit; (15) real estate; (16) retail; (17) research and education; (18) supply chain; (19) surface transportation; and (20) water. As of October 2016, five nonfederal entities maintained a permanent presence on the NCCIC Watch Floor (the ISACs of the financial, national health, aviation, and energy sectors, as well as the Multi-State ISAC).

State, Local, Tribal, and Territorial Governments

NCCIC maintains partnerships with state, local, tribal, and territorial governments to support their protection of each respective community. These governments are responsible for the security and integrity of their own cyber networks, along with associated preparedness, mitigation, and response efforts.

The center is to facilitate overarching situational awareness and the sharing of technical information and best practices with these nonfederal government partners to help ensure a strengthened national cyber risk posture. As part of this effort, NCCIC is to coordinate on an ongoing basis with the Multi-State Information Sharing and Analysis Center (MS-ISAC), and an MS-ISAC representative is physically located on the NCCIC Watch Floor.

International Partners

As part of its mission, NCCIC also engages and collaborates with international partners, including governments that are members of the North Atlantic Treaty Organization (NATO), to disseminate bulletins and perform services. The center works with international partners and their respective cyber centers while conducting cyber-related exercises —as part of a March 2016 exercise, representatives from Australia, Canada, Denmark, Finland, Germany, Hungary, Japan, Netherlands, New Zealand, Sweden, Switzerland, and the United Kingdom participated. NCCIC also can assist international partners with responding to cyber incidents. For example, during the cyber attack against the Ukrainian power infrastructure in December 2015, the center collaborated with the Ukrainian government to determine the methods of the cyber attack.

NCCIC Reported Spending about $480 Million over the Past Three Fiscal Years

NCCIC reported that it spent about $480 million on cybersecurity-related activities during fiscal years 2014 through 2016. According to the center’s officials, this included spending for 262, 268, and 301 full-time employees for each of the three fiscal years, respectively. Figure 3 depicts the reported expenditures per year by each of the four branches of NCCIC.

Although NCCIC Has Taken Steps to Perform Required Cybersecurity Functions, the Extent to Which It Carries Them Out In Accordance with Implementing Principles Is Unclear

NCCIC has taken steps to perform each of its 11 statutorily required cybersecurity functions. It has developed a variety of products and services in support of these functions, including those related to analyzing and sharing cyber information, facilitating coordination among federal and nonfederal partners, and conducting technical assistance and exercises. However, the extent to which NCCIC carried out these functions in accordance with the nine principles specified in the National Cybersecurity Protection Act of 2014 is unclear because the center has not consistently evaluated its performance against the principles. In addition, a number of factors impede NCCIC’s ability to more efficiently perform several of its cybersecurity functions. Although not generalizable to any larger population, recipients of its products and services that responded to our survey expressed generally favorable views of its activities. Nevertheless, NCCIC has limited assurance that it is fully meeting statutory requirements and efficiently performing its cybersecurity functions because it has not completely evaluated its performance against the principles or addressed the impediments to performing its cybersecurity functions.

NCCIC Has Developed a Variety of Products and Services in Support of Required Cybersecurity Functions

NCCIC has developed 43 types of products and services in support of its 11 statutorily required functions. Descriptions of these products and services as well as the total numbers of each provided to NCCIC’s customers during fiscal years 2015 and 2016 are discussed in greater detail in appendix II. The center manages several programs that provide data used in developing the products and performing the services related to its cybersecurity functions. These programs include:

The National Cybersecurity Protection System, operationally known as EINSTEIN, monitors network traffic entering or exiting networks of federal agencies and provides intrusion detection and intrusion prevention services. NCCIC analysts use data logged by EINSTEIN to notify federal and nonfederal partners of potential breaches of information security.

The Advanced Malware Analysis Center is a set of capabilities intended to provide a segregated, closed, computer network system that is used to analyze computer network vulnerabilities and threats. According to NCCIC officials, information transmitted to NCCIC through the Advanced Malware Analysis Center may include malicious codes, computer viruses, worms, spyware, bots, and Trojan horses. Once received, analysts use the malware analysis capabilities to analyze the code or images in order to discover how to secure or defend computer systems against the threat. The corrective action information is then published in products such as vulnerability reports or alerts or malware reports.

The Automated Indicator Sharing program was created to provide real-time sharing of cyber threat indicators and defensive measures by enabling NCCIC to (1) receive cyber threat indicators and defensive measures submitted by its nonfederal participants and federal entities; (2) remove personally identifiable information and other sensitive information that is not directly related to a cybersecurity threat; and (3) disseminate the cyber threat indicators and defensive measures to its nonfederal participants and federal entities, as appropriate. The Automated Indicator Sharing program uses the DHS-developed Structured Threat Information Expression/Trusted Automated Exchange of Indicator Information formats, a mechanism for sharing cyber threat information in a common manner. NCCIC uses this program to send out machine- readable cyber threat indicators at near-real-time and is now on- boarding participants across the public and private sectors. NCCIC officials stated that the Automated Indicator Sharing program was first disseminated to the 5 cyber centers. Since then, the program has become accessible to additional entities. According to the officials, as of August 2016, 32 private sector entities representing 6 critical infrastructure sectors and 7 federal agencies were connected to the program. NCCIC officials stated that DHS is in the process of expanding the service to all 24 Chief Financial Officers Act agencies in response to guidance from the Office of Management and Budget from October 2015.

The following summarizes NCCIC’s products and services that support its 11 statutorily required functions. Details on how all 43 products and services support each of the cybersecurity functions are in appendix III.

Function 1: Be a federal civilian interface for the multidirectional and cross-sector sharing of information related to cyber threat indicators, defensive measures, cybersecurity risks, incidents, analysis and warnings for federal and nonfederal entities.

NCCIC has nine products and services that support this function. Among these, it provides products such as Cyber Information Sharing and Collaboration Program (CISCP) Indicator Bulletins and US-CERT Indicator Bulletins, which can include cyber threat indicators, defensive measures, cybersecurity risks, incidents, analysis, and warnings. For fiscal year 2016, the center developed and disseminated 151 US-CERT Bulletins. For example, one US-CERT Bulletin identified Internet protocol addresses that had conducted unauthorized scans of networks of partner entities.

NCCIC also provides services to interface with federal and nonfederal entities. Through the center’s Information Sharing and Liaison Services, representatives from federal and nonfederal sectors are able to reside permanently or temporarily on the Watch Floor alongside NCCIC officials, to better ensure multidirectional, cross-sector sharing of information. According to the officials, there are seven seats available on the Watch Floor that its partners can reserve as a temporary residence. As of August 2016, the center reported agreements with 118 entities that could elect to reside temporarily on the Watch Floor. We observed additional cross-sector information sharing through the presence on the NCCIC Watch Floor of liaison officers from the other five cyber centers; members from the Multi-State, Communications, and Financial Services ISACs; and the intelligence community working in conjunction with NCCIC analysts.

Function 2: Provide shared situational awareness to enable real-time, integrated, and operational actions across the federal government and nonfederal entities to address cybersecurity risks and incidents to federal and nonfederal entities.

NCCIC supports this function with the use of 12 products and services. Among these products, it provides situational awareness to its customers to enable real-time, integrated, operational actions. Through one such product, Watch Floor Situation Reports, the center provides awareness of incidents and recommendations on remediation. For example, one report disseminated to its partners identified current events related to Ransomware incidents directed towards hospitals. As part of the report, the center identified immediate and future actions in support of resolving the incidents. In addition, in providing notifications, such as National Coordinating Center for Communications (NCC) Watch Train Derailment and GPS Testing Notices, the center shared information to support operational actions on behalf of the partner entities. Further, NCCIC provided situational awareness of potentially malicious Internet protocol addresses through Victim/Abuse notifications.

Function 3: Coordinate the sharing of information related to cyber threat indicators, defensive measures, cybersecurity risks and incidents across the federal government.

NCCIC has nine products and services that support this function. Among the products are CISCP Bulletins, US-CERT Bulletins, Joint Analysis Reports, and Joint Indicator Bulletins that can contain information related to cyber threat indicators, defensive measures, cybersecurity risks and incidents. For example, the center issued eight Joint Analysis Reports during fiscal years 2015 and 2016. One report, issued jointly with the Federal Bureau of Investigation on April 14, 2015, was also coordinated with the Departments of Treasury and Energy. The report contained a summation of open-source analysis related to common vulnerabilities leveraged by state-sponsored cyber operators in products such as Adobe Flash, Adobe Reader, Microsoft Office, Microsoft server software, and OpenSSL. The report contained information on the specific version of the product affected, as well as the associated information to patch the vulnerability.

According to NCCIC officials, the center relies on the NCCIC Portal as a mechanism to coordinate the sharing of these products to customers.

Specifically, the portal is comprised of 35 compartments, which include customers across the globe, and within government and various critical infrastructures. Each of the compartments represents a grouping of entities with a similar role or focus. For example, the Government Forum of Incident Response and Security Teams are comprised of individuals from federal civilian and military agencies responsible for securing government information technology systems.

Function 4: Facilitate cross-sector coordination to address cybersecurity risks and incidents, including cybersecurity risks and incidents that may be related or could have consequential impacts, across multiple sectors.

NCCIC has six products and services that support this function. For example, the center facilitates cross-sector coordination to address cybersecurity risks and incidents through its Industrial Control Systems Joint Working Group and its Incident Notifications. In particular, the joint working group holds biannual meetings with the industrial control system community. For example, the most recent meeting occurred on May 3–5, 2016, and had over 300 stakeholders represented. According to the after-action report, representatives from several sectors, including officials from the energy, water, transportation, and nuclear sectors, among others, attended the meeting.

Function 5: Conduct and share integration and analysis, including cross- sector, of cyber threat indicators, defensive measures, cybersecurity risks and incidents with federal and nonfederal entities.

NCCIC has eight products and services that support this function. For example, the US-CERT Analysis Report is an integrated analysis document that can contain indicators of compromise and tactics, techniques, and procedures related to specific threats. Further, US-CERT officials stated that the center provides common vulnerabilities to the National Vulnerability Database, which is an established, open source of indicators used by information security professionals located across the nation and throughout the world. Further, ICS-CERT officials stated that it provides industrial control system vulnerabilities to over 15,000 “.gov” e- mail addresses that are signed up to receive ICS-CERT Vulnerability Alerts.

Function 6: Provide timely technical assistance, risk management support, and incident response capabilities to federal and nonfederal entities with respect to cyber threat indicators, defensive measures, and cybersecurity risks and incidents, which may include attribution, mitigation, and remediation.

NCCIC supports this function with the use of five products and services. With these products, it has the capacity to provide technical assistance, risk management support, and incident response capabilities to customers upon request. For example, in responding to and conducting incident response analyses for public or private sector customers, US- CERT developed Incident Response Team Reports that outlined mitigation recommendations to the customers.

In addition, to support risk management, the center conducted, as services, Risk and Vulnerability Assessments, which are activities to assist entities in developing strategies for improving their cybersecurity posture. According to officials, NCCIC attempts to provide a report of its findings to the requesting entity within 30 days of the assessment.

Further, NCCIC provided Cyber Assessments of control systems. For example, the Cyber Security Evaluation Tool (CSET®) can be downloaded to conduct a self-evaluation of an entity’s cybersecurity posture against, among other things, best practices and National Institute of Standards and Technology recommendations. In addition, ICS-CERT officials stated that upon a customer’s request, NCCIC can provide further assistance by conducting industrial control system architectural assessments and network assessments.

Function 7: Provide information and recommendations on security and resilience measures to federal and nonfederal entities, including information and recommendations to facilitate information security and strengthen information systems against cybersecurity risks and incidents; and share cyber threat indicators and defensive measures.

NCCIC has 16 products and services that support this function. Among these products, the center provided information and recommendations on security and resilience measures through its Preliminary Digital Media Analysis Report and Digital Media Analysis Report products. Specifically, for these products, it conducted analysis of digital media and provided a report that includes analysis of the exploits and associated mitigation strategies. Further, according to NCCIC officials, the US-CERT and ICS-CERT components conduct incident response activities (known as US-CERT Incident Response Team Report and ICS Incident Response Deployment) and develop reports to document their findings at the request of partner entities. These reports can contain recommendations to strengthen information systems against cybersecurity risks and incidents and potentially share cyber threat indicators and defensive measures.

Function 8: Engage with international partners, in consultation with other appropriate agencies, to (a) collaborate on cyber threat indicators, defensive measures, and information related to cybersecurity risks and incidents; and (b) enhance the security and resilience of global cybersecurity.

NCCIC supports this function with the use of 10 products and services. It engages with international partners to collaborate on cyber threat indicators, defensive measures, and information related to cybersecurity risks and incidents. For example, the most recent Cyberstorm exercise— the department’s national-level exercise series—was conducted during the spring of 2016 and involved more than 1,200 participants including NCCIC’s national and international partners. According to the exercise after-action report, 12 international partners participated in the Cyberstorm exercise. To enhance the security and reliance of global cybersecurity, the after-action report identified areas of improvement relating to the escalation of incidents and coordination of public and private efforts. ICS-CERT also collaborated with the Ukrainian government in the aftermath of a cyber attack on its power infrastructure to develop and disseminate vulnerability alerts, reports, and briefings on the attack. US-CERT, on a different occasion, collaborated with Canada on developing vulnerability alerts associated with ransomware.

Function 9: Share cyber threat indicators, defensive measures, and other information related to cybersecurity risks and incidents with federal and nonfederal entities, including across sectors of critical infrastructure and with state and major urban area fusion centers, as appropriate.

NCCIC relies on four products and services to support this function. For example, it shared cyber threat indicators, defensive measures, and information through its Malware Initial Findings Reports and Malware Analysis Reports. These reports are based on NCCIC malware analysis conducted at the request of the customer. They can contain indicators, such as a description of the malware artifact, as well as defensive measures, such as the Internet Protocol addresses potentially associated with the malware.

Both types of reports are disseminated via the NCCIC Portal. The center used a Traffic Light Protocol (TLP), which is a designation to ensure that sensitive information is shared with the appropriate audience. MS-ISAC representatives have access to the NCCIC portal and can share information with its members per TLP protections.

Function 10: Participate, as appropriate, in national exercises run by the department.

NCCIC has four products and services that support this function. For example, in addition to Cyberstorm, the center conducted and participated in external exercises for customers to support the improvement of national and international cybersecurity. NCCIC officials stated that these external exercises include federal, state, local, tribal, territorial, private, and international partners and range from individual table-top exercises to multi-organization exercises. The center conducted such an exercise in October 2015 with a state government to improve its communication capabilities and provided a seminar on the current threats to control systems worldwide.

Function 11: Coordinate with the Office of Emergency Communication of the Department, assessing and evaluating consequence, vulnerability, and threat information regarding cyber incidents to public safety communications to help facilitate continuous improvements to the security and resiliency of such communications.

NCCIC has four products and services that support this function. Among its activities, the center engages with the Office of Emergency Communications in planning and preparing for disasters and incidents, including cyber incidents, to ensure continued readiness of the communications network. NCCIC officials stated they meet weekly with Office of Emergency Communications Regional coordinators during a NCC Weekly Operations call to discuss threats and vulnerabilities. According to NCCIC, the Office of Emergency Communications is a supporting partner of its execution of national coordinator responsibilities for Emergency Support Function 2–Communications under the National Response Framework. In addition, officials from the center have briefed the National Council of Statewide Interoperability Coordinators on how to share cyber data using NCCIC’s incident reporting process at a national conference in April 2016.

NCCIC’s Adherence to the Implementing Principles in Carrying Out its Cybersecurity Functions Is Unclear

NCCIC is required to carry out its functions in accordance with nine principles specified in the National Cybersecurity Protection Act of 2014, to the extent practicable. As previously described, these principles, among other things, relate to ensuring that industry sector-specific, academic, and national laboratory expertise is sought and receives appropriate consideration; the information related to cybersecurity risks and incidents is appropriately safeguarded against unauthorized access; and shared information is timely, actionable, and relevant to risks, incidents, and analysis.

The extent to which NCCIC carried out its 11 cybersecurity functions in accordance with the nine principles specified in the act is unclear. We identified instances where, with certain products and services, NCCIC had implemented its functions in adherence with one or more of the principles. For example, consistent with the principle that it seek and receive appropriate consideration from industry sector-specific, academic, and national laboratory expertise, NCCIC coordinated with contacts from industry, academia, and the national laboratories to develop and disseminate vulnerability alerts through the National Vulnerability Database. In addition, to comply with the principle that the information related to cybersecurity risks and incidents be appropriately safeguarded against unauthorized access, the center used the TLP designation to ensure that sensitive information was shared with the appropriate audience. Specifically, NCCIC disseminated its products via the NCCIC portal, using the protocol for products such as Indicator Bulletins, Analysis Reports, Malware Initial Findings Reports, and Malware Analysis Reports. (Additional examples of how NCCIC products and services helped the center implement its functions according to the principles are provided in appendix IV.)

On the other hand, we also identified instances where the cybersecurity functions were not performed in adherence with the principles. For example, with regard to function 6, NCCIC is to provide timely technical assistance, risk management support, and incident response capabilities to federal and nonfederal entities. The function is supported, in part, by Risk and Vulnerability Assessments. However, NCCIC had not established measures or other procedures for ensuring the timeliness of these assessments. According to officials responsible for this service, the assessments have an estimated completion time frame of 8-10 weeks for each customer. However, the officials stated that this time frame is not an established metric by which they evaluate the timeliness of the service. Further, NCCIC had not established measures or procedures to assess the actionability of its products and services. For example, US-CERT Indicator Bulletins, a product that supports several functions, typically contain actionable information, such as specific malicious Internet addresses to be blocked. NCCIC had not established a means of determining the extent to which a particular bulletin helped to mitigate a risk or prevent an incident.

In discussing this matter, NCCIC officials acknowledged that they had not made a complete determination of the applicability of the principles with all of the center’s functions and thus had not established measures and procedures for assessing its products and services against the principles. The officials stated that they have begun to map activities supporting the cybersecurity functions to the implementing principles. For example, according to the officials, the center established a unit for reviewing and making recommendations to improve overall NCCIC operations. During fiscal year 2016, this unit completed performance management reviews across the center’s programs to identify areas in which NCCIC could better align its operations with its overall requirements, including the principles. Further, officials from the ICS-CERT branch stated that they were in the preliminary stages of measuring their activities against one of the nine principles. Specifically, the officials stated that 20 metrics were being developed that would measure timeliness, relevance, and actionability (principle 1) across the components of their organization.

Nevertheless, while these preliminary actions are important steps, they do not represent a complete determination of the applicability of all nine principles across all of NCCIC’s statutorily-required cybersecurity functions. As such, NCCIC officials could not say whether the principles did or did not apply to all of the 11 functions. Moreover, because a complete determination of the applicability of the nine principles had not been done, the center also had not developed metrics and methods for assessing and ensuring adherence with the principles. Until the center determines the applicability of the implementing principles for all of its functions and develops the metrics and methods necessary to ensure that the principles are met, it will not be positioned to ensure that NCCIC is effectively meeting its statutory requirements.

NCCIC Faces Impediments to Performing Its Cybersecurity Functions More Efficiently

In addition to NCCIC not having made a complete determination of how it is adhering to the principles, a number of factors impede the center’s ability to more efficiently perform several of its cybersecurity functions. In particular, the center faces impediments in tracking security incidents; maintaining current and reliable customer information, to include obtaining such information on all owners and operators of the most critical infrastructure assets; working across multiple network platforms; and collaborating with international partners.

Tracking of security incidents is not centralized or reconciled. The National Cybersecurity Protection Act of 2014 requires NCCIC to coordinate the sharing of information across the government. This includes information related to cyber threat indicators, defensive measures, and cybersecurity risks and incidents.

However, NCCIC officials were unable to completely track and consolidate cyber incidents reported to the center, thereby inhibiting its ability to coordinate the sharing of information across the government. For US-CERT-related incidents, personnel assigned to the NCICC service desk generated a daily report of the current status of the open incident tickets. For example, the July 18, 2016 report had a total of 520 incident tickets. However, this report did not represent the totality of incidents across the center because it did not include incidents reported to ICS- CERT. Since the NCCIC service desk did not have access to the data within the ICS-CERT ticketing system, it could not produce a management report on the status of all incidents reported to the center.

NCCIC officials attributed the lack of a single, centralized incident tracking system to the fact that ICS-CERT and US-CERT had operated as separate entities prior to the establishment of the center. As such, both ICS-CERT and US-CERT has its own incident ticketing system. Senior ICS-CERT officials stated they are aware of this challenge and are exploring options on how best to integrate the two systems. Until such integration takes place, NCCIC will continue to encounter difficulty in completely tracking the total efforts of its branches to address reported cybersecurity incidents. As a result, the center will be challenged in determining how effective it is in sharing information related to cyber threat indicators, defensive measures, and cybersecurity risks and incidents across the government.

The difficulty of logging incident data is further compounded by the multiple ways in which an incident can be reported to the center. US- CERT officials stated that there are six preferred ways in which NCCIC receives information related to potential incidents. For example, to communicate with US-CERT, customers can choose e-mail, a phone call, an automatic submission form, or an automated machine-to-machine submission as the means to notify US-CERT of an incident. In addition, to communicate with ICS-CERT regarding an industrial control system- related incident, customers can choose to submit an e-mail or phone directly to ICS-CERT (Figure 4 shows the ways in which NCCIC prefers to receive reported incidents.)

However, contrary to the 6 preferred methods of communicating with US- CERT and ICS-CERT, officials from NCCIC’s Operations and Integration office provided documentation that identified at least 22 methods by which the center receives potential incidents. These methods would include phone numbers and e-mail addresses other than the aforementioned 6 methods, established by various groups within the four NCCIC components as a means to communicate with partners.

In addition, according to NCCIC officials, depending on the method of reporting, incidents are not always logged into the NCCIC incident ticketing systems. For example, when customers have prior established relationships, analysts can be called directly and can handle the incidents without logging them into the system. The lack of control over the entry points as well as inconsistencies in logging data, together; inhibit the center in consistently tracking incidents and their status across the entire NCCIC.

Until the center can reduce, consolidate, or modify the points of entry that customer entities use to communicate with NCCIC, it will lack the ability to better ensure that all incident tickets are logged appropriately. Thus, further contributing to the center being less able to effectively perform its statutorily-required function in coordinating the sharing of information related to cyber threat indicators, defensive measures, and cybersecurity risks and incidents across the government.

Maintaining current and reliable customer information. The National Cybersecurity Protection Act of 2014 requires NCCIC to be the federal civilian interface for the multidirectional and cross-sector sharing of information related to cyber threat indicators, defensive measures, cybersecurity risks and incidents, and analysis and warnings for federal and nonfederal entities. To perform this function, the center needs to have accurate and up-to-date contact information for the potential recipients of the cybersecurity information it shares.

However, NCCIC’s contact information was not always up to date, thus impacting its ability to effectively function as a federal civilian interface for federal and nonfederal entities. Specifically, after e-mailing our survey to recipients of NCCIC’s products and services, we received 303 undeliverable return messages out of 2,792 recipients contacted. We also identified individuals who were included on the list of recipients that NCCIC provided to us that no longer had the role the center indicated or were no longer with the entity listed.

NCCIC officials were unable to demonstrate that they had any formal process for maintaining customer contact information. The officials stated that maintaining customer contact information was an ad hoc process and acknowledged that capturing changes to that data was a challenge.

Without regularly validating data pertaining to its product and service recipients, the center may lack quality information it needs to effectively develop and maintain partnerships and share cybersecurity-related information with federal and nonfederal entities to support its operation as required by the statutes.

Obtaining contact information of all owners and operators of the most critical cyber-dependent infrastructure assets. The National Cybersecurity Protection Act of 2014 requires NCCIC to facilitate cross- sector coordination to address cybersecurity risks and incidents. This includes cybersecurity risks and incidents that may be related or could have consequential impacts across multiple sectors.

However, representatives of federal and nonfederal entities that own critical cyber assets that could have a catastrophic impact on the nation if victimized by a cyber attack were not fully represented in the NCCIC customer log. Specifically, our review found that 23 percent of the entities owning such critical assets (as determined by DHS) were not represented within the master NCCIC customer log as of September 2016. Without representation of these entities, NCCIC may not have the information it needs readily available to facilitate coordination with critical asset owners. NCCIC officials were unable to demonstrate that they had a formal internal process for maintaining customer contact information and acknowledged that doing so remains a challenge for the center.

Without a concentrated effort on ensuring the full representation of the owners and operators of these critical assets, the center lacks assurance that it is adequately facilitating the cross-sector coordination of cybersecurity risk and incidents to the nation’s most critical cyber- dependent assets that, if impacted, could have a catastrophic effect on the nation.

Working across multiple network platforms. The National Cybersecurity Protection Act of 2014 requires NCCIC to coordinate the sharing of information across the government. This includes information related to cybersecurity risks and incidents.

However, we found that the sharing of information is complicated by NCCIC analysts having to operate across multiple networks, often manually entering data into each network, which decreases the rate of response of coordinating the sharing of incident information to customers and increases the risk of false entry. For example, officials stated that it takes on average 3 minutes for a ticket to be closed when working within one network. Across the 3 systems, it could take up to 15 minutes, depending on the size of the ticket, and the amount of information needed to be manually entered into each system.

According to senior NCCIC officials, this impediment was attributed to a legacy technical infrastructure implemented prior to the center’s existence. They added that efforts were under way to address this impediment. However, NCCIC had not developed an implementation plan or established time frames for consolidating or integrating the networks.

Until NCCIC develops a process to avoid manual data entry, it will continue to face challenges in efficiently sharing information related to cyber threat indicators, defensive measures, and cybersecurity risks and incidents across the government.

Collaborating with international partners using the NCCIC Portal. The Cybersecurity Act of 2015 requires NCCIC to engage with international partners, in consultation with other appropriate agencies, to (a) collaborate on cyber threat indicators, defensive measures, and information related to cybersecurity risks and incidents; and (b) enhance the security and resilience of global cybersecurity.

International and other partners have access to the center’s products through the NCCIC Portal, which has functioned as a mechanism to disseminate products to recipients since 2003. However, DHS is migrating the NCCIC portal to the Homeland Security Information Network. This network is categorized as a FIPS 199 high-impact system and, thus, requires authentication of individuals with access to the system. According to NCCIC officials, international partners had expressed a concern that the new network will have a negative impact on their collaboration with NCCIC because continued access would require the submission of international participants’ passports and other sensitive personal information to a U.S. government entity.

While DHS has a responsibility to ensure the security over its high-impact systems, NCCIC may face a barrier to engaging with international partners. Without taking action to address this potential barrier, international partners may be reluctant to engage with NCCIC. Thus, the center may be challenged in its ability to collaborate and enhance global cybersecurity if it does not find alternative methods to engage and share information with international partners while ensuring the security requirements of high-impact systems.

Survey Respondents Varied in Their Reported Use of NCCIC Products and Services but Had Generally Favorable Views of the Center’s Activities

The respondents to our nongeneralizable survey of the center’s activities reported that they used its products and services to varying extents. The respondents also expressed generally favorable views of the center’s activities. Table 2 depicts the extent to which the survey respondents, who each self-identified as a customer of an NCCIC component (US- CERT, ICS-CERT, NCC, the Watch Floor, and NO&I), used, did not use, or were unsure if they used a particular NCCIC product or service.

With regard to evaluating the characteristics of products and services, the respondents to our nongeneralizable survey generally reported that NCCIC products and services were timely, relevant, and actionable. Specifically, 289 of 333 respondents (87 percent) found products and services they had used to be extremely, very, or moderately timely; 286 of 332 respondents (86 percent) found products and services to be extremely, very, or moderately relevant; and 234 of 332 respondents (70 percent) stated that products and services have led to an actionable result to a very great, great, or moderate extent (e.g., used to address a vulnerability or apply a defensive measure) on their part.

In addition, although between 12 and 18 percent of respondents to our nongeneralizable survey indicated a low level of effectiveness, respondents had generally favorable views of the center’s provision of cybersecurity information. Specifically, 236 of 335 respondents (70 percent) evaluated the provision of cyber threat indicators to be at a high or moderate level of effectiveness. In addition, 219 of 333 respondents (66 percent) identified risks and incidents to be at a high or moderate level of effectiveness. Further, 211 of 339 respondents (62 percent) indicated cyber defensive information to be at a high or moderate level of effectiveness. Further, the survey respondents evaluated NCCIC’s ability to provide timely, relevant, and actionable information at a 235 of 331 (71 percent), 245 of 334 (73 percent), and 222 of 339 (65 percent) high or moderate level of effectiveness, respectively.

Table 3 shows survey respondents’ evaluations of NCCIC’s effectiveness in providing them with cyber threat indicators, information on risks and incidents and defensive measures, and information that was timely, relevant, and actionable.

Survey respondents also evaluated the center’s effectiveness with regard to its information sharing capability, the uniqueness in the information it provides, and its partnerships with them in improving the protection of critical cyber assets and functions, and how well it is fulfilling its mission. Table 4 shows respondents’ overall evaluation of the center in terms of the effectiveness of its information sharing capability, customer partnerships, and the extent to which it is fulfilling its mission, among other things.

Further, respondents regarded NCCIC as important to the nation’s ability to protect critical cyber assets and functions. Specifically, 264 of 337 respondents (78 percent) to our nongeneralizable survey stated that there would be a “very” or “somewhat” negative impact on the nation if the NCCIC products and services did not exist.

However, not all survey responses were positive. Specifically, survey respondents reported that they were not aware of all of the products and services that the center offered. The respondents added that they would be interested in receiving additional NCCIC products and services but were unsure about how to begin receiving them. The respondents reported that the center had not provided information identifying these products and services. NCCIC officials acknowledged that customers may not be aware of certain products and services because not all products and services are meant for every customer.

Conclusions

NCCIC, as the federal civilian cyber center, is generally performing 11 required cybersecurity functions through the development and dissemination of 43 products and services. However, the extent to which NCCIC carried out these cybersecurity functions in accordance with the 9 implementing principles is unclear. Until it determines the extent to which the implementing principles apply to these functions, NCCIC will not be able to fully assess the extent to which it is meeting the mandated principles. Further, without measuring the extent to which principles are being met, NCCIC will be challenged in articulating how effectively it is performing the functions in support of its role as a focal point for cybersecurity incident coordination, information sharing, and incident response across the federal civilian government and critical infrastructure.

NCCIC also faces several impediments that inhibits it from efficiently performing its cybersecurity functions. These impediments relate to consolidating entry points for receiving and logging potential incident data and maintaining the center’s relationship with customers. Until NCCIC takes steps to overcome these impediments it may not be able to efficiently perform its cybersecurity functions and assist federal and nonfederal entities in identifying cyber-based threats, mitigating vulnerabilities, and managing cyber risks.

Recommendations for Executive Action

To more fully address the requirements identified in the National Cybersecurity Protection Act of 2014 and the Cybersecurity Act of 2015, we recommend that the Secretary of the Department of Homeland Security take the following nine actions: 1. Determine the extent to which the statutorily required implementing principles apply to NCCIC’s cybersecurity functions. 2. Develop metrics for assessing adherence to applicable principles in carrying out statutorily required functions. 3. Establish methods for monitoring the implementation of cybersecurity functions against the principles on an ongoing basis. 4. Integrate information related to security incidents to provide management with more complete information about NCCIC operations. 5. Determine the necessity of reducing, consolidating, or modifying the points of entry used to communicate with NCCIC to better ensure that all incident tickets are logged appropriately. 6. Develop and implement procedures to perform regular reviews of customer information to ensure that it is current and reliable. 7. Take steps to ensure the full representation of the owners and operators of the nation’s most critical cyber-dependent infrastructure assets. 8. Establish plans and time frames for consolidating or integrating the legacy networks used by NCCIC analysts to reduce the need for manual data entry. 9. Identify alternative methods to collaborate with international partners, while ensuring the security requirements of high-impact systems.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from DHS. In its comments, the department concurred with all nine recommendations. The department also provided details about steps that it plans to take to address each of the recommendations, including estimated time frames for completion. If effectively implemented, these actions should enhance the effectiveness and efficiency of NCCIC in performing its statutory requirements. The department’s written comments are reprinted in appendix V.

In addition to the aforementioned comments, DHS also provided a technical comment via e-mail, which we considered and incorporated.

We are sending copies of this report to the appropriate congressional committees, the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact Gregory Wilshusen at (202) 512-6244 or wilshuseng@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI.

Appendix I: Objective, Scope, and Methodology

Our objective was to determine the extent to which the National Cybersecurity and Communications Integration Center (NCCIC) was performing its statutorily defined cybersecurity-related functions. To determine this, we analyzed two acts that establish roles and responsibilities for the center: the National Cybersecurity Protection Act of 2014 and the Cybersecurity Act of 2015. These laws together require the center to carry out 11 cybersecurity functions. More specifically, the National Cybersecurity Protection Act of 2014 prescribed 7 functions and the Cybersecurity Act of 2015 prescribed 4 additional functions. The National Cybersecurity Protection Act of 2014 also identified 9 implementing principles. The two acts also contained provisions for GAO to report on NCCIC’s implementation of its cybersecurity mission.

To determine the extent to which it was addressing the 11 cybersecurity functions, we analyzed the center’s program descriptions, concepts of operations, and policies and procedures documenting how each of the center’s components are to operate. For example, we analyzed the U.S. Computer Emergency Readiness Team (US-CERT) Strategic Action Plan, Industrial Control Systems-Cyber Emergency Response Team (ICS-CERT) Five-Year Plan (2015-2019), and the National Coordinating Center (NCC) Information Guide and Overview for 2015. In addition, we analyzed the NCCIC Watch Floor Concept of Operations that describes its operations. We corroborated information by interviewing center officials, including the Assistant Secretary for Cybersecurity and Communications, the Director of the NCCIC, and the directors of each of the center’s components, as well as other responsible officials.

Based on our analysis of this information, we developed an initial list of products and services and held interviews with officials to confirm the total list of products and services. Based on these actions, we determined that the center develops and disseminates 43 products and services. We then collected and analyzed examples of each product and service to determine how they addressed each of the 11 cybersecurity functions established in the two laws. To gain a greater understanding of the purposes and methods of developing the products and services, we also interviewed NCCIC officials.

To identify instances where products and services addressed the 9 implementing principles, we analyzed relevant program documentation and reviewed the procedures by which the center develops its products and services. We corroborated our information by interviewing NCCIC officials responsible for product and service development. To gain a greater understanding of its operations, we visited the site of the ICS- CERT operations in Idaho Falls, Idaho, to observe its activities, including the development of its products and services. We also observed operations of the Watch Floors in Arlington, Virginia and Idaho Falls, Idaho, and interviewed officials responsible for operating the Watch Floors, developing services, and liaising with federal and nonfederal partners.

We also analyzed the dissemination methods of products and services by examining the contents of the NCCIC web portal, including how the customer base was segmented to disseminate products and services in accordance with information sharing protections. At the ICS-CERT facility in Idaho Falls, Idaho, we observed the basic ICS training exercise services provided to customers. We also collected and analyzed performance measures and interviewed officials about the actions being taken to improve its measurement efforts and efforts to consolidate operations across NCCIC components. In addition, NCCIC officials provided budget execution information that we analyzed to determine the reported amount spent across three fiscal years for each component. During the interviews, we discussed with officials the impediments that the center faced in more efficiently performing the 11 cybersecurity functions.

To obtain the views of the recipients of the center’s products and services, we administered a survey to a sample of individuals identified by NCCIC as having access to a product or service, or participating in a center group or activity. We asked customers about their awareness and use of the 43 products and services, and other activities and roles performed by the center. We then asked them to assess their experiences, including rating the effectiveness and the implementing principles of timeliness, actionability, and relevance. We also asked respondents to rate various elements of NCCIC in terms of importance, expectations, challenges, and reasons for not using the center’s products and services.

To develop our questionnaire, we met with NCCIC officials and identified the activities performed for customers, including the development and dissemination of 43 products and services, disseminated to customers. We pretested draft versions of the questionnaire with nonfederal representatives of two Information Sharing and Analysis Centers and an information security officer at a federal agency, to reflect some of the variation in the population.

We defined the target population for the survey to be all organizational points of contact or other individuals identified by NCCIC, as of June 8, 2016, as having access to a product or service, or participating in a center group or activity. NCCIC provided us with 19,573 records across14 lists of customer contact information. Some of the provided lists consisted only of e-mail addresses of individuals subscribed to a particular NCCIC product or service, while others consisted of members of a group. Some organizations were represented by many individual e-mail addresses across the lists, and some individuals appeared on more than one list. While the basic unit of the population to be sampled was an individual e- mail address, due to the variability in coverage of the population mentioned above, an individual survey respondent may be representing their own personal experiences and opinions, or those of an organization, and multiple respondents may be representing the same organization.

After removing records with missing, incomplete, or erroneously duplicative e-mail addresses within each list, our sample was reduced to 19,293 records. We did not remove multiple instances of the same e-mail address appearing on more than one list; these duplicates were retained in the sample frame so that each instance of that e-mail address might have a chance of initial selection proportional to the size of the customer list it appeared on.

We initially drew a random but nongeneralizable sample of 2,907 e-mail address records, allocated across the 16 customer types roughly proportional to the sizes of each type. We then removed 115 of this initial sample because their e-mail addresses duplicated selections made from other customer lists, for a total sample of 2,792 customer records with unique e-mail addresses, which we attempted to contact with our survey.

We began our survey on August 2, 2016. We sent e-mails with login information to the web-based questionnaire to the sample. We sent up to three follow-up e-mails during the fieldwork period to those who had not yet responded. The survey ended on September 8, 2016. The outcomes of the survey fieldwork are displayed in table 5 below.

The response rate to the survey, calculated as the number of usable responses divided by the number found to be eligible was about 14 percent. Because of the variability in coverage of the population by the sample frame, irregularities in the contact information and eligibility of the records sampled, and the low rate of response to the survey, the results of this survey only represent those that responded, and are not generalizable to any larger population of NCCIC customers. We do not make any inferences about those not sampled or not responding to the survey.

In addition to this limitation, questionnaire surveys of this kind are subject to other potential errors. To minimize the possibility of measurement error (differences between reported and true answers) arising from question design, interpretation or administration, or the misreporting of answers, we designed and administered the survey in consultation with survey methodologists, made improvements to the questionnaire based on pretest results, and had a separate survey methodologist review the draft questionnaire to identify potential problems in questionnaire design.

Of the 340 respondents, 14 percent identified themselves as individual participants in NCCIC activities, 64 percent as representatives of a single public or private organization, 13 percent as representing an association or other entity representing a sector or group of organizations, and 9 percent identified in other ways. Thirty-four percent said they represented federal government entities; 18 percent said they represented state, local, or tribal entities; 44 percent said they represented private sector entities; and 4 percent gave other answers.

During the processing and analysis of reported data, we also identified and corrected for patterns of response across questions that we could identify as inconsistent or contradictory. Nonresponse error (failure to obtain a response to a question or the questionnaire) may lead to bias in the results if those who do not respond would have given materially different responses from those who did respond. To minimize nonresponse, we made follow-up contacts throughout the survey. To minimize processing error (mistakes in converting reported data into published survey results), data processing and analysis programming was independently verified by a separate data analyst.

We conducted this performance audit from January 2016 to February 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: NCCIC Product and Service Information

Table 6 below highlights the total number of each product and service that the National Cybersecurity and Communications Integration Center (NCCIC) reported providing to its customers in fiscal years 2015 and 2016.

Appendix III: NCCIC Products and Services Supporting Cybersecurity Functions

The National Cybersecurity and Communications Integration Center (NCCIC) is required to perform 11 cybersecurity functions. Table 7 below summarizes how the 43 products and services were being used as of October 2016 in support of the 11 functions.

Appendix IV: Examples of How NCCIC Products and Services Address the Implementing Principles

Although the National Cybersecurity and Communications Integration Center (NCCIC) did not completely determine the applicability of statutory implementing principles to its products and services, table 8 below provides examples of our determination of how NCCIC products and services adhered to the principles.

Appendix V: Comments from the Department of Homeland Security

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Michael W. Gilmore (assistant director), Kush K. Malhotra (analyst in charge), Chris Businsky, Lee A. McCracken, Constantine Papanastasiou, David Plocher, Carl Ramirez, and Priscilla A. Smith made key contributions to this report.