Information Security: DHS Needs to Enhance Capabilities, Improve Planning, and Support Greater Adoption of Its National Cybersecurity Protection System
Published: 2016-01-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Cyber-based attacks on federal systems continue to increase. GAO has designated information security as a government-wide high-risk area since 1997. This was expanded to include the protection of critical cyber infrastructure in 2003 and protecting the privacy of personally identifiable information in 2015. NCPS is intended to provide DHS with capabilities to detect malicious traffic traversing federal agencies' computer networks, prevent intrusions, and support data analytics and information sharing.
Senate and House reports accompanying the 2014 Consolidated Appropriations Act included provisions for GAO to review the implementation of NCPS. GAO determined the extent to which (1) the system meets stated objectives, (2) DHS has designed requirements for future stages of the system, and (3) federal agencies have adopted the system. To do this, GAO compared NCPS capabilities to leading practices, examined documentation, and interviewed officials at DHS and five selected agencies. This is a public version of a report that GAO issued in November 2015 with limited distribution. Certain information on technical issues has been omitted from this version.
section_title: What GAO Found
paragraphs: The Department of Homeland Security's (DHS) National Cybersecurity Protection System (NCPS) is partially, but not fully, meeting its stated system objectives:
Intrusion detection: NCPS provides DHS with a limited ability to detect potentially malicious activity entering and exiting computer networks at federal agencies. Specifically, NCPS compares network traffic to known patterns of malicious data, or “signatures,” but does not detect deviations from predefined baselines of normal network behavior. In addition, NCPS does not monitor several types of network traffic and its “signatures” do not address threats that exploit many common security vulnerabilities and thus may be less effective.
Intrusion prevention: The capability of NCPS to prevent intrusions (e.g., blocking an e-mail determined to be malicious) is limited to the types of network traffic that it monitors. For example, the intrusion prevention function monitors and blocks e-mail. However, it does not address malicious content within web traffic, although DHS plans to deliver this capability in 2016.
Analytics: NCPS supports a variety of data analytical tools, including a centralized platform for aggregating data and a capability for analyzing the characteristics of malicious code. In addition, DHS has further enhancements to this capability planned through 2018.
Information sharing: DHS has yet to develop most of the planned functionality for NCPS's information-sharing capability, and requirements were only recently approved. Moreover, agencies and DHS did not always agree about whether notifications of potentially malicious activity had been sent or received, and agencies had mixed views about the usefulness of these notifications. Further, DHS did not always solicit—and agencies did not always provide—feedback on them.
In addition, while DHS has developed metrics for measuring the performance of NCPS, they do not gauge the quality, accuracy, or effectiveness of the system's intrusion detection and prevention capabilities. As a result, DHS is unable to describe the value provided by NCPS.
Regarding future stages of the system, DHS has identified needs for selected capabilities. However, it had not defined requirements for two capabilities: to detect (1) malware on customer agency internal networks or (2) threats entering and exiting cloud service providers. DHS also has not considered specific vulnerability information for agency information systems in making risk-based decisions about future intrusion prevention capabilities.
Federal agencies have adopted NCPS to varying degrees. The 23 agencies required to implement the intrusion detection capabilities had routed some traffic to NCPS intrusion detection sensors. However, only 5 of the 23 agencies were receiving intrusion prevention services, but DHS was working to overcome policy and implementation challenges. Further, agencies have not taken all the technical steps needed to implement the system, such as ensuring that all network traffic is being routed through NCPS sensors. This occurred in part because DHS has not provided network routing guidance to agencies. As a result, DHS has limited assurance regarding the effectiveness of the system.
section_title: What GAO Recommends
paragraphs: GAO recommends that DHS take nine actions to enhance NCPS's capabilities for meeting its objectives, better define requirements for future capabilities, and develop network routing guidance. DHS concurred with GAO's recommendations.

section_title: Letter
paragraphs: Cyber-based intrusions and attacks on federal systems have become not  only more numerous and diverse but also more damaging and disruptive.  This is illustrated by recently reported data breaches at the Office of  Personnel Management, which affected millions of current and former  federal employees. Protecting the information systems and the  information that resides on them and effectively responding to cyber- incidents is critical to federal agencies because the unauthorized  disclosure, alteration, and destruction of the information on those systems  can result in great harm to those involved.
Due to the cyber-based threats to federal systems and critical  infrastructure, the persistent nature of information security vulnerabilities,  and the associated risks, we continue to designate information security as  a government-wide high-risk area in our most recent biennial report to  Congress, a designation we have made in each report since 1997. In  2003, we expanded this high-risk area to include the protection of critical  cyber infrastructure and we continued to do so in the most recent update  to our high-risk list. In the 2015 update, we further expanded this area to  include protecting the privacy of personally identifiable information—that  is, personal information that is collected, maintained, and shared by both  federal and nonfederal entities.
The National Cybersecurity Protection System (NCPS), designed and  operated by the Department of Homeland Security (DHS), was developed  to be one of the tools to aid federal agencies in mitigating information  security threats. The system is to provide DHS with the capability to  provide four cyber-related services to federal agencies: intrusion  detection, intrusion prevention, analytics, and information sharing.
Senate and House reports accompanying the Consolidated  Appropriations Act, 2014, included provisions for us to review NCPS. Our  objectives were to determine the extent to which (1) the system meets  stated objectives, (2) DHS has designed requirements for future stages of  the system, and (3) federal agencies have adopted the system.
This is a public version of a report we issued in November 2015 that was  designated “for official use only” and released with limited distribution due  to the sensitive nature of the material it contained. Certain information has  been omitted. Although the information provided in this report is more  limited in scope, it addresses the same objectives as the November 2015  report. Also, the overall methodology used for both reports is the same.
To determine the extent to which NCPS meets stated objectives, we  compared NCPS’s four overarching capabilities to leading federal best  practices related to intrusion detection, intrusion prevention, analytics,  and information sharing. Further, we reviewed elements of each NCPS  objective as well as intrusion detection incident notifications sent to five  agencies: the Departments of Energy and Veterans Affairs, the General  Services Administration, the National Science Foundation, and the  Nuclear Regulatory Commission.
To determine the extent to which DHS had designed requirements, we  reviewed DHS’s planning documentation and compared it to federal  guidance issued by the Office of Management and Budget. In addition, for  all new capabilities identified for funding in DHS’s fiscal year 2016 funding  request, we determined if formalized requirements had been documented  and approved. Further, we determined if future capabilities plans for  NCPS’s intrusion prevention objective were developed using a risk-based  approach, including threat, vulnerability, impact, and likelihood.
To determine the extent of federal adoption of NCPS, we reviewed DHS  documentation and agreements to determine the adoption by the 23 non- defense agencies identified in the Chief Financial Officers Act.  Additionally, we identified challenges to adoption by reviewing DHS  program documentation and interviewing officials from DHS, the selected  agencies identified above, and three Internet service providers  participating in NCPS. See appendix I for additional details on our  objectives, scope, and methodology.
We conducted this performance audit from June 2014 to January 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: As computer technology has advanced, federal agencies have become  dependent on computerized information systems to carry out their  operations and to process, maintain, and report essential information.  Virtually all federal operations are supported by computer systems and  electronic data, and agencies would find it difficult, if not impossible, to  carry out their missions, deliver services to the public, and account for  their resources without these cyber assets. Information security is thus  especially important for federal agencies to ensure the confidentiality,  integrity, and availability of their systems and data. Conversely, ineffective  information security controls can result in significant risk to a broad array  of government operations and assets, as the following examples  illustrate:
Computer resources could be used for unauthorized purposes or to  launch attacks on other computer systems.
Sensitive information, such as personally identifiable information,  intellectual property, and proprietary business information, could be  inappropriately disclosed, browsed, or copied for purposes of identity  theft, espionage, or other types of crime.
Critical operations, such as those supporting critical infrastructure,  national defense, and emergency services, could be disrupted.
Data could be added, modified, or deleted for purposes of fraud,  subterfuge, or disruption.
subsections: section_title: The Nation Faces an Evolving Array of Cyber- Based Threats
paragraphs: Threats to systems are evolving and growing. Cyber threats can be  unintentional or intentional. Unintentional or non-adversarial threats  sources include failures in equipment, environmental controls, or software  due to aging, resource depletion, or other circumstances which exceed  expected operating parameters. They also include natural disasters and  failures of critical infrastructure on which the organization depends but are  outside of the control of the organization. Intentional or adversarial threats  include individuals, groups, entities, or nations that seek to leverage the  organization’s dependence on cyber resources (i.e., information in  electronic form, information and communications technologies, and the  communications and information-handling capabilities provided by those  technologies). Threats can come from a wide array of sources, including  corrupt employees, criminal groups, and terrorists. These threat  adversaries vary in terms of their capabilities, their willingness to act, and  their motives, which can include seeking monetary gain, or seeking an  economic, political, or military advantage. Table 1 describes the sources  of cyber based threats in more detail.
Cyber threat adversaries make use of various techniques, tactics, and  practices, or exploits, to adversely affect an organization’s computers,  software, or networks, or to intercept or steal valuable or sensitive  information. These exploits are carried out through various conduits,  including websites, e-mails, wireless and cellular communications,  Internet protocols, portable media, and social media. Further, adversaries  can leverage common computer software programs, such as Adobe  Acrobat and Microsoft Office, as a means by which to deliver a threat by  embedding exploits within software files that can be activated when a  user opens a file within its corresponding program. Table 2 provides  descriptions of common exploits or techniques, tactics, and practices  used by cyber adversaries.
Reports of successfully executed cyber exploits illustrate the debilitating  effects they can have on the nation’s security and economy, and on  public health and safety. Further, federal agencies have experienced  security breaches in their networks, potentially allowing sensitive  information to be compromised and systems, operations, and services to  be disrupted. These examples illustrate that a broad array of federal  information and critical infrastructures are at risk:  In August 2015, the Internal Revenue Service (IRS) reported that  approximately 390,000 tax accounts were potentially affected by  unauthorized third parties gaining access to taxpayer information from  the agency’s “Get Transcript” application. According to testimony from  the Commissioner of the IRS in June 2015, criminals used taxpayer- specific data acquired from non-IRS sources to gain unauthorized  access to information; however, at that time, he reported that  approximately 100,000 tax accounts had been affected. These data  included Social Security information, dates of birth, and street  addresses.  In July 2015, the Office of Personnel Management reported that an  intrusion into its systems compromised the background investigation  files of 21.5 million individuals. This was in addition to a separate but  related incident that affected personnel records of about 4 million  current and former federal employees, which the agency announced  in June 2015.  In September 2014, a cyber-intrusion into the United States Postal  Service’s information systems may have compromised personally  identifiable information for more than 800,000 of its employees.
section_title: Federal Law and Policy Provide a Framework for Securing Federal Information and Systems
paragraphs: The Federal Information Security Management Act of 2002 (FISMA  2002) was enacted into law to provide a comprehensive framework for  ensuring the effectiveness of information security controls over federal  information resources. The law required each agency to develop,  document, and implement an agency-wide information security program  to provide risk-based protections for the information and information  systems that support the operations and assets of the agency. Such a  program includes assessing risk; developing and implementing cost- effective security plans, policies, and procedures; plans for providing  adequate information security for networks, facilities, and systems;  providing security awareness and specialized training; testing and  evaluating the effectiveness of controls; planning, implementing,  evaluating, and documenting remedial actions to address information  security deficiencies; procedures for detecting, reporting, and responding  to security incidents; and ensuring continuity of operations. The act also  assigned oversight responsibilities to the Office of Management and  Budget (OMB) and gave the National Institute of Standards and  Technology (NIST) responsibility for developing standards and guidelines  that include minimum information security requirements.
The Federal Information Security Modernization Act of 2014 largely  supersedes FISMA 2002. This law retains the requirements for agencies  to develop, document, and implement an agency-wide information  security program, as well as OMB oversight and NIST development of  standards and guidelines. Its changes include requiring DHS to assist  OMB with providing oversight by administering the implementation of  information security policies and practices for information systems. DHS  responsibilities include developing and overseeing the implementation of binding operational  directives requiring agencies to implement OMB’s information security  standards and guidelines; operating a federal information security incident center (previously  OMB’s responsibility), which has been established as the DHS United  States Computer Emergency Readiness Team (US-CERT); deploying technology, upon request by an agency, to continuously  diagnose and mitigate against cyber threats and vulnerabilities; and  conducting targeted operational evaluations, including threat and  vulnerability assessments, on agency information systems.
In January 2008, the President issued National Security Presidential  Directive 54/Homeland Security Presidential Directive 23. The directive  established the Comprehensive National Cybersecurity Initiative, a set of  projects with the objective of safeguarding federal executive branch  government information systems by reducing potential vulnerabilities,  protecting against intrusion attempts, and anticipating future threats  against the federal government’s networks. Under the initiative, DHS  was to lead several projects to better secure civilian federal government  networks, while other agencies, including OMB, the Department of  Defense, and the Office of the Director of National Intelligence had key  roles in other projects, including monitoring military systems and  classified networks, overseeing intelligence community systems and  networks, and spearheading advanced technology research and  development. The initiative’s projects can be grouped into three focus  areas:
Establishing front lines of defense. This includes projects intended to  protect the perimeter of federal networks, such as consolidating  connections and deploying intrusion detection and prevention  systems.
Defending against full spectrum of threats. This includes physical and  cyber projects intended to protect national security and intelligence- related information and systems across the federal government.
Shaping the future environment. The initiatives in this area are  focused on expanding cybersecurity education and research and  development efforts for future technologies and cybersecurity  strategies.
As required by FISMA (both the 2002 and 2014 laws), NIST has  developed standards and guidelines for agencies to develop, document  and implement their required information security programs, select  controls for systems, and conduct risk-based cyber threat mitigation  activities. For example, NIST’s Special Publication 800-37 recommends  cost-effectively reducing information security risks to an acceptable level  and ensuring that information security is addressed throughout an  information system’s life cycle. In addition, NIST Special Publication  800-94 establishes guidance for federal agencies to use when designing,  implementing, and maintaining the systems they deploy to perform  intrusion detection and prevention.
section_title: DHS’s Actions Supporting Cybersecurity of the Federal Network Enterprise
paragraphs: DHS designated the National Protection and Programs Directorate to lead  the national effort to strengthen the security and resilience of the nation’s  physical and cyber-critical infrastructure, including supporting federal  agencies in securing their information systems and information.  Specifically, the directorate is responsible for enhancing the security,  resilience, and reliability of federal agencies in the protection of the “.gov”  domain of the federal civilian government.
Within the National Protection and Programs Directorate, the Office of  Cybersecurity and Communications, among other things, operates the  National Cybersecurity and Communications Integration Center (NCCIC)  that is to serve as a 24/7 cyber monitoring, incident response, and  management center and as a national focal point of cyber and  communications incident integration. The US-CERT, one of several  subcomponents of the NCCIC, is responsible for operating the NCPS,  which provides intrusion detection and prevention capabilities to covered  federal agencies. The Network Security Deployment (NSD) division of the  Office of Cybersecurity and Communications is responsible for  developing, deploying, and sustaining NCPS. For example, the division is  to deliver NCPS intrusion detection capability directly to federal agencies  through Trusted Internet Connection Access Providers or through Internet  service providers at Managed Trusted Internet Protocol Service  locations.
section_title: National Cybersecurity Protection System
paragraphs: NCPS is an integrated system-of-systems that is intended to deliver a  range of capabilities, including intrusion detection, intrusion prevention,  analytics, and information sharing. The NCPS capabilities, operationally  known as the Einstein program, are one of a number of tools and  capabilities that assist in federal network defense. Originally created in  2003, NCPS is intended to aid DHS in its ability to help reduce and  prevent computer network vulnerabilities across the federal government.  Its analysts examine raw and summarized data from a wide variety of  information sources to make determinations about potential attacks  across the network traffic of participating federal agencies detected by  NCPS. Table 3 provides an overview of the enhancements DHS has  made to the original iteration of Einstein as well as the corresponding  objective of NCPS the functionality supports.
NCPS is intended to build successive layers of defense mechanisms into  the federal government’s information technology infrastructures. When  NCPS intrusion detection sensors are deployed at a Trusted Internet  Connection location, the system monitors inbound and outbound network  traffic, with the goal of allowing US-CERT, using NCPS and its supporting  processes, to monitor all traffic passing between the federal civilian  networks and the Internet for malicious activity. Figure 1 illustrates how  Trusted Internet Connection portals interact with the NCPS intrusion  detection sensors and the Internet. For more detailed information about  NCPS’s development and functionality, see appendix II.
section_title: NCPS Expenditures through Fiscal Year 2014
paragraphs: As we reported in April 2015, DHS spent over $1.2 billion on the NCPS  system through fiscal year 2014. Figure 2 below depicts the funds spent  for NCPS over the past 6 budget years.
NSD plans to use the fiscal year 2015 funding to sustain currently  deployed capabilities and expand intrusion prevention, information- sharing, and analytics capabilities of NCPS. As of April 2015, the  projected total life-cycle cost of the program was approximately $5.7  billion, through fiscal year 2018.
section_title: NCPS Is Not Fully Satisfying All Intended System Objectives
paragraphs: The overarching objectives of NCPS are to provide functionality that  supports intrusion detection, intrusion prevention, analytics, and  information sharing. While NCPS’s ability to detect and prevent intrusions,  analyze network data, and share information is useful, its capabilities are  limited. For example, NCPS detects signature-based anomalies, but does  not employ other, more complex methodologies and cannot detect  anomalies in certain types of traffic. Further, the intrusion prevention  capabilities can currently mitigate threats to a limited subset of network  traffic. Regarding NCPS’s analytics function, DHS has deployed aspects  of this capability and plans to develop more complex tools. However,  information sharing, which is the fourth stated objective, has only recently  been approved and funded for development; thus, current information  sharing efforts are manual and largely ad hoc. In addition, although DHS  established a variety of NCPS-related metrics, none provide insight into  the value derived from the functions of the system. Developing such  metrics poses a challenge for the agency, according to DHS officials.  Until NCPS’s intended capabilities are more fully developed, DHS will be  hampered in its abilities to provide effective cybersecurity-related support  to federal agencies.
subsections: section_title: NCPS Has Limited Ability to Detect Intrusions within Observed Network Traffic
paragraphs: NCPS’s intrusion detection capability is intended to provide DHS with the  ability to scan network traffic for signs of potentially malicious activity.  Effective intrusion detection provides an organization with the ability to  detect abnormalities within network traffic and can be accomplished  through the use of multiple types of intrusion detection methodologies. In  order to more comprehensively and accurately detect malicious activity,  NIST recommends using a combination of three detection  methodologies: signature-based, anomaly-based, and stateful purpose  analysis.
Signature-based intrusion detection is able to detect malicious traffic  by comparing current traffic to known patterns of malicious behavior,  also referred to as signatures. This method is considered effective at  detecting known threats and is the simplest form of intrusion  detection, because it can only match against known patterns of  malicious traffic.
The anomaly-based and stateful purpose detection methodologies are  more complex approaches, which involve comparing current network  activity to predefined baselines of “normal behavior” to identify  deviations which could be indicative of malicious activity. These  approaches to intrusion detection are more effective than signature- based detection at identifying previously unknown threats, such as  “zero-days,” as well as variants to known threats and threats  disguised by the use of evasion techniques.
NCPS uses only a signature-based methodology for detecting malicious  activity. According to US-CERT officials, NCPS’s intrusion detection  capability is supported by 228 intrusion detection sensors placed  throughout the .gov network infrastructure. The sensors provide a flow of  network traffic to be analyzed. Officials added that there are over 9,000  intrusion detection signatures deployed within NCPS, with approximately  2,300 that are enabled and being used to evaluate traffic at any given  time. A majority of the signatures are available through commercially  available products, though a portion is custom developed.
According to DHS documentation and NSD officials, NCPS was always  intended to be a signature-based intrusion detection system, and thus it  does not have the ability to employ multiple intrusion detection  methodologies. Further, NSD and US-CERT officials stated that NCPS is  just one of the many tools available to federal agencies to help enhance  their cybersecurity posture. They stated that it is the responsibility of each  agency to ensure their networks and information systems are secure  while it is the responsibility of DHS to provide a baseline set of protections  and government-wide situational awareness, as part of a defense-in- depth information security strategy.
By employing only signature-based intrusion detection, NCPS is unable to  detect intrusions for which it does not have a valid or active signature  deployed. This limits the overall effectiveness of the program. Moreover,  given that many federal agencies use commercially available signature- based intrusion detection systems to support their information security  efforts, the addition of another signature-based intrusion detection system  may do little to provide customer agencies with a baseline set of  protections. DHS officials acknowledged that the intrusion detection  systems used by many federal agencies likely have more signatures  deployed than NCPS. Thus, the agencies’ intrusion detection systems  would be able to compare their network traffic against a larger set of  potential exploits, such as exploits that US-CERT determined no longer  needed to be scanned by NCPS. In other cases, US-CERT officials  stated, some agencies do not possess their own robust intrusion  detection capability and thus rely more on the intrusion detection  functionality provided by NCPS.
Regarding zero-day exploits, US-CERT officials stated there is no way to  identify them until they are announced. Once they are announced, US- CERT can develop a signature, as was the case with Adobe Flash  exploits that were recently publicly announced. While there are sources  that can be used to buy zero day exploits, officials stated that DHS does  not pay for zero days. Occasionally, US-CERT will receive notifications of  exploits from partners before they go public, but these are mostly  malware notifications. While we acknowledge the challenge of developing  signatures for zero-day exploits, enhancing NCPS’s current intrusion  detection approach to include functionality that would support the  development of a baseline of network behavioral analysis, as described  in NIST 800-94, would enhance DHS’s ability to combat such threats.
subsections: section_title: NCPS Is Unable to Detect Exploits across All Types of Network Traffic
paragraphs: According to NIST, many intrusion detection products have the ability to  detect attacks carried out through various types of network traffic, such as  traffic related to network browsers, e-mail, and file transfer, as well as  traffic related to supervisory control and data acquisition (SCADA) control  systems. In addition, intrusion detection systems should also have the  ability to detect malicious activity across multiple layers of network  protocols, including Internet Protocol Version 6 (IPv6). Further, NIST  states that some intrusion detection products have the ability to detect  characteristics of encrypted traffic (i.e., whether encryption had been  applied) but not evaluate the traffic itself. Adversaries will often use  encryption to mask malicious traffic to help better facilitate the successful  execution of cyber-exploits, such as zero-day attacks.
However, NCPS is not currently evaluating all types of network traffic.  NSD and US-CERT officials stated there are currently no signatures  deployed with NCPS that will detect threats embedded in certain types of  network traffic.
US-CERT officials stated that they have not deployed signatures related  to these additional types of network traffic for various reasons. They  stated that NCPS customer departments and agencies have not been  clear on the details of the specific types of network traffic present within  their organizations or the amount of traffic allowed to pass through their  network gateways. According to an NSD official, they initially collect such  data and hold meetings with officials from customer departments and  agencies to exchange technical information, but the departments and  agencies are responsible for routing network traffic to the NCPS sensors  and not required to keep DHS abreast of changes. In addition, US-CERT  officials stated that they have not conducted a detailed analysis of  customer departments’ and agencies’ traffic to gain this understanding.  Further, US-CERT officials stated that they were not equally concerned  with the risk posed by all types of network traffic.
Without an ability to analyze all types of traffic, DHS is unable to detect  threats embedded in such traffic and increases the risk that agencies  could be negatively impacted by such threats.
section_title: Current NCPS Signatures Do Not Address Selected Common Vulnerabilities
paragraphs: According to NIST, signature-based intrusion detection systems depend  on the quality of the signatures contained within them, and thus need to  be updated to reflect new vulnerabilities and exploits that emerge.  Organizations can purchase signatures from commercial vendors, custom  develop them, or obtain them from open sources. NIST maintains the  National Vulnerability Database (NVD), which is an open source of  information that can influence many information security activities,  including the development of intrusion detection signatures. Federal  agencies are encouraged to use the information contained within the  database as part of their information security efforts.
In addition, US-CERT has acknowledged the importance of incorporating  the use of common vulnerabilities and exposures (CVE) information in  information security activities. In April 2015 US-CERT issued an alert  which stated that cyber threat adversaries continue to exploit unpatched  software products from vendors such as Adobe, Microsoft, and Oracle.  Vulnerabilities in these products are often a common vector for spear  phishing attacks. The alert stated that as many as 85 percent of these  attacks are preventable through the implementation of patches.  Accordingly, the bulletin contained 30 of the top targeted vulnerabilities  and associated CVE information that security officials could use to  implement within their organizations.
However, the signatures supporting NCPS’s intrusion detection capability  only identify a portion of vulnerabilities associated with common software  applications from vendors such as Adobe, Microsoft, and Oracle.  Specifically, we found that NCPS had limited coverage of vulnerabilities  associated with 10 common client and server applications we evaluated.  At the time of our review, NCPS intrusion detection capability signatures  provided: partial coverage for 7 vulnerabilities, and no coverage for 2 vulnerabilities.  reasonable coverage for 1 vulnerability,  Further, for the 12 advanced persistent threats we evaluated, NCPS’s  intrusion detection capability had signatures that at the time of our review  provided: partial coverage for 4 advanced persistent threats.  reasonable coverage for 8 advanced persistent threats, and  More specifically, for the five client applications we reviewed (Adobe  Acrobat, Flash, Internet Explorer, Java, and Microsoft office), the NCPS  intrusion detection signatures provided some degree of coverage for  approximately 6 percent of the total vulnerabilities selected for review  (i.e., 29 of 489), with coverage for specific applications ranging from 1.2  to 80 percent of vulnerabilities identified in CVE reports published during  2014.
Further, it is unknown how, if at all, US-CERT plans to leverage  vulnerability data from other DHS sources to influence the development of  intrusion detection signatures. For example, the Federal Network  Resilience division is responsible for managing the Continuous  Diagnostics and Mitigation program. The vulnerability information  garnered from this program could be used to develop signatures that  would target exploits that are affecting many federal agencies. US-CERT  officials stated that they plan to use this information to influence NCPS,  but could not provide specific details as how they plan to accomplish this  due to the relative immaturity of the Continuous Diagnostics and  Mitigation program.
One reason that the signatures did not cover all identified vulnerabilities is  that the current tool DHS uses to manage and track the status of intrusion  detection signatures deployed within NCPS does not have the ability to  capture CVE information. US-CERT officials stated that when they  developed the Signature Management System tool, they were not  required to create a link between a signature and a published CVE data.  However, US-CERT has acknowledged this deficiency and stated this is  something it plans to address in the future.
In addition, US-CERT officials agreed with the results of our analysis of  client vulnerabilities, but reiterated that the goal of NCPS was not to  protect against all vulnerabilities. US-CERT officials stated that agencies  with their own internal intrusion detection systems would likely be able to  comprehensively address the common client vulnerabilities we selected.  US-CERT officials stated that the overall intent of the system was to  protect against nation-state level threat actors who often leverage “zero- day” exploits which may not have had a known mitigation or specific CVE  assigned. Accordingly, officials stated, they must consider input from a  variety of classified and unclassified sources, in addition to open source  data such as CVEs, when developing their intrusion detection signatures.  However, NCPS did not possess intrusion detection signatures that fully  addressed all the advanced persistent threats we reviewed, which are  often a type of exploit leveraged by nation-state actors.
US-CERT officials added that they must consider a variety of factors  when deciding which specific signatures to deploy and the length of time  they keep the signatures active. For example, the current version of the  software managing the intrusion detection function does not allow for  custom rules at each sensor. As a result, the signatures deployed must  be uniform across all sensors and cannot be tailored to a specific agency.  This adds an additional layer of complexity when deciding how long to  deploy signatures. For example, a smaller agency may be unaware of a  particular threat or associated signature, and thus could benefit from  having that signature deployed longer than a larger agency, which may  view it as potentially duplicative of signatures employed by its own  internal intrusion detection tool. Officials stated that they expect this issue  to be addressed when they upgrade to the next version of the software  that manages the intrusion detection function.
We acknowledge that NCPS’s intrusion detection capabilities draw on  many sources of vulnerability information and it should not necessarily  duplicate capabilities that agencies already possess. However, updating  the tool used to manage NCPS signatures to draw on and more clearly  link to publicly available, open-source repositories of vulnerability  information, such as the NVD, and using data from the Continuous  Diagnostic and Mitigation program as they become available as an input  into the development and management of signatures could add value by  minimizing the risk that known vulnerabilities will be exploited.
section_title: NCPS Intrusion Prevention Capability Is Limited, but Further Enhancements Are Planned
paragraphs: NCPS’s ability to provide intrusion prevention is another key objective of  the system. Intrusion prevention is an additional technique recommended  by NIST in support of effective information system monitoring. When fully  developed, NCPS will have the ability to proactively mitigate threats  across multiple types of network traffic. This is important because  malicious actors can propagate threats across multiple vectors and types  of network traffic.
NCPS’s intrusion prevention capability provides DHS with the ability to  proactively address network-based threats before they can potentially  cause harm to federal networks. This is accomplished by monitoring  network traffic to and from a customer agency’s network and taking some  action to stop traffic (e.g., blocking an e-mail) that has characteristics  matching pre-defined indicators of malicious traffic.
NCPS has the ability to prevent intrusions in near real time, but is  currently only able to proactively mitigate threats across a limited subset  of network traffic (i.e., Domain Name System, or DNS, blocking and e- mail filtering) at a selected group of customer agencies. Consequently,  there are other types of network traffic (e.g., web content), which are  common vectors of attack not currently being analyzed for potentially  malicious content.
NSD officials noted that initial capabilities for intrusion prevention were  intended to be more robust, but were scaled back due to a change in the  program’s approach. Specifically, these officials stated that the original  intent of the intrusion prevention deployment was to protect all types of  network traffic with classified indicators. Further, the solution was  supposed to provide government-furnished equipment to Internet service  provider networks as the backbone of the intrusion prevention function of  NCPS. However, NSD officials stated that, due to the excessive costs of  operating and maintaining the original solution, the agency decided in  May 2012 to change approaches. The new approach follows a managed  service model, where the Internet service providers would receive  classified indicators within their appropriate facilities and manage the  prevention for their particular customer agencies. Because of this  managed service model approach, NSD officials stated that the first set of  prevention capabilities was based in part on the existing solutions  provided by commercial service providers under the Defense Industrial  Base Opt-In pilot (later renamed the Enhanced Cybersecurity Services).  Officials also added that another motivation for the new approach was to  create a cybersecurity marketplace where the various Internet service  providers would compete with each other to provide better cybersecurity  solutions for federal customers.
DHS officials stated that they are developing prevention capabilities for  other types of network traffic. Specifically, NSD officials stated that they  plan to introduce the ability to filter web content by January 1, 2016. A  review of a recent monthly report from one of the Internet service  providers supporting NCPS intrusion prevention indicated that the  contractor has begun work on web content filtering and provided DHS  with a draft report on the indicators and overall process.
section_title: DHS Has Developed Aspects of the NCPS Analytics Capability
paragraphs: Another key objective of NCPS is to provide DHS with an analytics  capability. NIST recommends that organizations take a variety of actions  with respect to analytics, including analyzing and correlating audit records across different repositories to  gain organization-wide situational awareness,  correlating information from nontechnical sources with audit  information to enhance organization-wide situational awareness, analyzing the characteristics of malicious code, and employing automated tools to support near real-time analysis.
The functionality deployed in support of NCPS analytics capability  developed to date is in accordance with recommended standards. For  example, the security information and event management solution, which  has been operational since February 2012, simplifies cyber analysis by  providing a centralized platform in which the log data from similar events  can be aggregated, thereby reducing duplication. The tool also facilitates  analysts’ ability to correlate related events that might otherwise go  unnoticed and provides visualization capabilities, making it easier to see  relationships. Additionally, NSD has established functionality that enables  the analysis of the characteristics of malicious code. For example, the  Packet Capture tool enables US-CERT analysts to see “inside” the  packet, and inspect the payload to analyze a specific cyber threat.  Further, the Digital Media Analysis Environment (Forensics) and  Advanced Malware Analysis Center provide mechanisms to collect and  contain information on cyber threats in a highly secure environment for  evaluation by US-CERT analysts.
NSD and US-CERT officials stated that DHS initially focused funding and  development efforts on analytical functions associated with supporting the  intrusion detection and prevention functions of NCPS. However, the more  complex analytics development is planned for later stages of system  development. Specifically, DHS has enhancements planned through  fiscal year 2018. These planned enhancements are intended to better  facilitate the near real-time analysis of various data streams and  advanced malware behavioral analysis, and to conduct forensic analysis  in more collaborative way.
section_title: NCPS Information-Sharing Capability Is Not Fully Developed, and Selected Customer Agencies Did Not Always View Incident Notifications as Timely and Useful
paragraphs: Information sharing is a key control recommended by NIST in support of  effective information system security. Additionally, the presence of good  information sharing, particularly the ability to effectively notify an affected  entity of potentially malicious activity, is a key component of effective  intrusion detection and prevention, and thus a key objective of NCPS.  Further, NIST states that organizations should develop standard  operating procedures to ensure that consistent and accurate information  is available for reporting and management oversight. Also, US-CERT’s  Concept of Operations for NCPS establishes monitoring the status of  mitigation actions and strategies as a requirement of the program.
NSD officials stated that the information-sharing capability has only  recently been approved and funded for development, and thus current  information-sharing efforts are manual and largely ad hoc. DHS first  requested funding for the development of information-sharing capabilities  in 2010, but NSD officials stated the effort was given a lower priority than  the intrusion prevention capability and was not funded to begin planning  activities until 2014. As a result, DHS has yet to develop a majority of  planned functionality for the information-sharing capability of NCPS.  Though the operational requirements for the NCPS information-sharing  functionality were approved in November 2014, DHS did not formally  authorize NSD to initiate development of the capability until August 2015.  As a result no substantive actions have yet been taken to develop this  capability.
Regarding the current information-sharing efforts, officials from the five  customer agencies we reviewed stated that DHS is not always effectively  communicating its intrusion detection notifications to customer agencies.  Specifically, DHS officials provided evidence that they sent 74 incident  notifications that they believed were related to NCPS to the five agencies  in our review during fiscal year 2014. However, evidence provided by  the agencies showed that only 56 of these notifications had been  received by the customer agencies. The five impacted agencies and DHS  disagreed as to whether the other 20 incident notifications had been sent  and received. Specifically, for 18 of 20 these notifications, DHS provided  evidence that an e-mail may have been sent, but the agencies had no  record of receiving the notifications. For the 2 additional notifications, one  customer agency had a record of receiving them; however, DHS had no  evidence of transmitting the e-mails.
For the 56 NCPS-related notifications that the five agencies  acknowledged receiving, the agencies stated that
31 incidents notifications were timely and useful,
10 incidents notifications were not timely or useful,
7 incident notifications were identified by agency officials as false
7 incident notifications were not related to an NCPS intrusion- detection.
Additionally, DHS did not always solicit, and agencies did not always  provide, feedback on the notifications. Specifically:
Of the 56 incident notifications mentioned above, DHS requested that  the impacted agency provide feedback on 36 of them. Of these 36,  the agencies stated that they provided feedback on 15 notifications,  but did not provide feedback on 21.
For an additional 10 notifications, officials from 3 of 5 agencies stated  they provided feedback even though DHS had not explicitly requested  follow-up action.
For an additional 10 notifications, DHS did not request feedback and  the customer agencies did not provide any.
As another channel for sharing information, US-CERT holds weekly calls  with representatives of the security operation centers of various federal  agencies. These calls provide a forum for the voluntary exchange of a  variety of information security information, including NCPS-related  information. Officials from the five customer agencies involved in our  review expressed value in the information received from these  discussions.
One reason DHS and agencies do not agree about whether notifications  were received may be that DHS does not always explicitly ask for  feedback or confirmation of receipt of the notification. Additionally,  officials from one customer agency stated that DHS has no way of  determining which of its analysts were responsible for transmitting a  particular notification, so it is difficult to obtain context after a notification  is sent.
US-CERT officials stated that standard operating procedures and a  quality control procedure are being developed as part of the  implementation of a new version of the incident management database.  However, these procedures were not developed during the scope of our  review, fiscal year 2014. In August 2015, US-CERT provided us with a  draft standard operating procedure related to the incident notification  process. The policy provides an overview of the types of questions a US- CERT analyst should ask a customer agency when transmitting a  notification. However, the draft policy does not instruct them specifically to  include a solicitation of feedback within the notification. Further, US- CERT could not provide any information regarding the timetable for when  these procedures would take effect.
Regarding the usefulness of the notifications, two of the agencies in our  review stated that because of the placement of the intrusion detection  sensors on their networks, a significant amount of effort was required to  evaluate the context of the DHS notifications. Thus, both agencies stated,  and DHS agreed, the value of the notifications could be enhanced by  giving US-CERT analysts access to the agencies’ network diagrams,  which could allow them to identify the specific location of the intrusion.
Officials from customer agencies stated that they did not provide  feedback for a variety of reasons. For example, one agency stated that  due to its federated nature, getting a response from the impacted entity  within their agency was a challenge and could only be rectified by  reaching out to site owners for every incident notification they receive.  Consequently, officials stated, they typically only reached out when the  notification had met the threshold of a security event. Officials from this  agency stated that they had instituted a new standard operating  procedure that requires the analyst processing the incident notification to  reach out to DHS prior to closing it out. They added that this policy went  into effect after fiscal year 2014 and did not impact the data set we  reviewed. An additional agency stated that a request for feedback is not  always clearly stated within the notification they receive from US-CERT.
Without verifying the receipt of intrusion detection notifications and  soliciting feedback on their usefulness, DHS may be hindered in  assessing the effectiveness of NCPS’s current information-sharing  capabilities.
section_title: Current Metrics Do Not Comprehensively Measure Effectiveness of NCPS
paragraphs: According to NIST, a number of laws—including the Federal Information  Security Management Act—cite performance measurement in general,  and information security performance measurement in particular, as a  requirement. Further, NIST 800-55 states that an information security  measurement program provides a number of organizational and financial  benefits, including increased accountability for information security  performance, improved effectiveness of information security activities,  demonstrated compliance with laws, and quantified inputs and allocation  decisions. Further, effectiveness or efficiency measures are used to  monitor if program-level processes and system-level security controls are  implemented correctly, operating as intended, and meeting the desired  outcome.
Metrics for NCPS, as provided by DHS, do not provide information about  how well the system is enhancing government information security or the  quality, efficiency and accuracy of supporting actions. DHS has  established three department-wide NCPS-related performance metrics,  as part of its Performance and Accountability report:
Percentage of traffic monitored for cyber intrusions at civilian federal  executive branch agencies: According to Executive Program  Management Office and NSD officials, this measure assesses  NCPS’s intrusion detection capability by providing information on the  scope of coverage for potentially malicious cyber-activity across  participating civilian federal government agencies. During fiscal year  2014, DHS reported that approximately 88.5 percent of the total  Internet traffic of 23 civilian, executive branch agencies was monitored  by NCPS intrusion detection sensors. Though this metric provides  insight into the amount of federal executive-branch traffic that NCPS  is able to provide intrusion detection for, it does not provide insight  into the quality or efficiency of the intrusion detection function for that  traffic.
Percentage of incidents detected by US-CERT that targeted agencies  are notified of within 30 minutes: According to Executive Program  Management Office and NSD officials, this is an additional measure of  NCPS’s intrusion detection capability. Specifically, DHS  documentation stated that there were 297 cyber incidents identified on  federal networks using the NCPS’s intrusion detection capability in  fiscal year 2014. The average time to notify impacted agencies was  18 minutes, with 87.2 percent (259 of 297) of notifications occurring  within 30 minutes. While this metric provides insight into the speed at  which DHS could share information related to detected incidents, it  does not provide a measure for the accuracy or value of those  notifications. Further, of 24 incident notifications for the five selected  agencies that support this metric, DHS could not provide evidence  that 12 of these notifications were sent. Without appropriately sharing  the notifications with the affected agency, we are unsure how DHS  classifies these 12 notifications as incidents.
Percentage of known malicious cyber traffic prevented from causing  harm at federal agencies: According to Executive Program  Management Office and NSD officials, this measure assesses  NCPS’s intrusion prevention capability. Specifically, DHS documents  stated that each currently deployed indicator of a malicious threat is  paired with a countermeasure to prevent the malicious threat from  harming those networks. In fiscal year 2014, 389 indicators were  deployed amongst intrusion prevention sensors. Though this metric  would track whether a particular countermeasure was engaging (i.e., if  prevention occurred) it does not necessarily evaluate the  effectiveness or efficiency of the intrusion prevention capability. DHS  officials agreed with this observation and stated that the agency was  in the process of retiring this metric and developing a new one that  would better measure and evaluate the effectiveness of intrusion  prevention.
Further, NSD has established key performance parameters that provide  an indication of the system’s ability to perform functions supporting  NCPS’s objectives. For example, the following measures were developed  to track the performance of the intrusion detection function:
Detect known cyber events through automated intrusion detection  within 1 minute of event occurrence.
Provide automated notification within the operations center that a  cyber event took place within 1 minute of event detection.
Aggregate and correlate detected cyber events for known indicators  within 30 minutes of event notification.
While these are valuable for determining how NCPS is operating as a  system, officials from the Executive Program Management Office and  NSD agreed that they did not provide a qualitative or quantitative  assessment of the system’s ability to fulfill the aforementioned objectives.  Further, as we reported in April 2015, a DHS acquisition official  questioned whether the NCPS key performance parameters were defined  properly.
Regarding the system’s benefits, NSD and US-CERT officials stated that  the total amount of incident notifications sent to customer agencies does  indicate that NCPS is providing value. However, as our analysis of a  selected group of customer notifications from fiscal year 2014 indicates,  customer agencies do not perceive every notification transmitted as  valuable. Without the deployment of comprehensive measures, DHS  cannot appropriately articulate the value provided by NCPS.
section_title: DHS Identified Future NCPS Needs, but Has Developed Limited Requirements
paragraphs: While DHS developed an executive road map for the intrusion detection,  prevention, analytics, and information sharing objectives that describes  future NCPS capabilities to be developed through fiscal year 2018, it has  not defined requirements, as called for by OMB guidance and best  practice, for two intrusion detection capabilities to be provided in fiscal  year 2016. In addition, although DHS officials stated that they do consider  threat information as part of the required risk-based approach for  determining future capabilities to protect federal information systems, they  do not consider specific vulnerabilities affecting agencies’ networks and  systems, as information on these is not currently available. The lack of  vulnerability information prevents DHS from taking a full risk-based  approach to designing future NCPS intrusion prevention capabilities.
subsections: section_title: DHS Identified Future Needs in Program Planning Documents, but Not All Future Capabilities Are Based on Appropriately Defined Requirements
paragraphs: OMB’s Capital Programming Guide states that requirements should be  developed to support program budgeting activities. The guidance also  states that agencies should avoid “specification creep,” where  requirements become uncontrolled by defining requirements to meet  future potential needs or incorporating emerging technology that would be  “nice” to have. Further, a recognized best practice in requirements  development from the Software Engineering Institute notes that  requirements should be expressed in a way that can be used for design  decisions.
NSD maintains a road map that is used to track potential additional  capabilities for NCPS’s intrusion detection, intrusion prevention, analytics,  and information-sharing objectives to be developed in future fiscal years,  up to fiscal year 2018. For each NCPS objective, the Executive Road  Map identifies the current state of operations (“as-is”) and the desired  state of operations (“target”). According to NSD officials, this road map  facilitates discussions with senior DHS management, and is revised at  several points in the fiscal year.
The road map identifies technology and techniques that may increase the  department’s ability to perform activities to support the four objectives.  For example, DHS plans to begin work on a “web gateway proxy scan  encryption” capability in fiscal year 2016. DHS also plans to seek funding  for a wireless network protection capability in fiscal year 2018, which may  add an additional type of intrusion detection and prevention technology  described in guidance issued by NIST.
Requirements have not been fully defined for all items in the road map.  Specifically, two capabilities DHS stated will be provided in fiscal year  2016—expanding the intrusion detection capability to identify malware  present on customer agency internal networks and identifying malicious  traffic entering and exiting cloud-based service provider services—are  based on requirements that have not been fully defined.
NSD officials stated that these capabilities were based upon the  requirement to detect intrusion attempts in near real time across the  federal government. They added that identifying malware on customer  agency internal networks and malicious traffic entering and exiting cloud- based service providers is a logical expansion of responding to the cyber  threat, and the program office needs flexibility to adapt to the threat.  However, these capabilities could represent a significant departure from  the version of NCPS currently deployed and envisioned in the  governance documents. Specifically, the technical nature of cloud  computing—where customer agency data may be stored and accessed  by multiple physical sites—and the number of cloud service providers that  could be used by customer agencies may require a different infrastructure  deployment methodology than the existing NCPS sensor deployments at  Internet service providers and at customer agency locations. Further,  while the Executive Road Map indicates that NCPS will detect malware  on customer agency internal networks using log data from DHS’s  Continuous Diagnostics and Mitigation program, it is unclear how DHS  plans to accomplish this.
Until it fully defines requirements for these two capabilities, DHS  increases the risk that it will invest in functionality that does not effectively  support information security efforts at the customer agencies and across  the federal government.
section_title: Decisions Regarding Future NCPS Intrusion Prevention Capabilities Incorporate Some Elements of a Risk-Based Approach
paragraphs: The Federal Information Security Modernization Act of 2014 and  guidance issued by NIST call for a risk-based approach to protecting  federal systems. According to NIST’s Guide for Conducting Risk  Assessments, information security risk is assessed by considering the  threats posed to the federal government, the vulnerabilities (or  weaknesses) in information systems, the impact (or harm) that could  occur given the potential for threats exploiting vulnerabilities, and the  likelihood that threats would use the exploits to allow harm to occur.
DHS has incorporated selected elements of a risk-based approach when  considering the next capabilities of the NCPS intrusion prevention  objective. Specifically, NSD coordinated and leveraged threat information  from the National Security Agency and the National Cybersecurity and  Communications Integration Center, along with information provided by  the Internet service providers, to develop a list of countermeasures that  DHS believed would be reasonable to implement.
NSD officials stated that decisions regarding existing and upcoming  countermeasures were made based on the capabilities of the Internet  service providers. Specifically, the e-mail and DNS countermeasures  were used as the first countermeasures for the NCPS intrusion prevention  capability because they were already deployed at Internet service  providers as part of the Enhanced Cybersecurity Services program.
However, NSD did not consider and does not currently have access to  vulnerability information for the agency information systems it is helping to  protect. NSD officials stated that vulnerability data about customer  agency information systems and networks are difficult to obtain. For  example, agency information security reports required under the Federal  Information Security Modernization Act of 2014 do not contain  vulnerability information that NSD could use to inform future capabilities.  Also, NSD officials stated—and the Executive Road Map confirmed—that  DHS’s Continuous Diagnostics and Mitigation program may provide  additional vulnerability information that could be valuable in determining  future capabilities. However, at this time the program is relatively  immature and NSD had not developed processes and procedures on how  to use this vulnerability information to inform decisions on future  capabilities at the time of our review.
Further, DHS also has a separate program to collect vulnerability  information on federal executive branch agency systems and networks  that could be useful for determining future NCPS intrusion prevention  capabilities. In an October 2014 memo, OMB directed DHS to scan  Internet-accessible addresses and public-facing segments of federal  civilian agency systems for vulnerabilities on an ongoing basis, and report  to OMB on the identification and mitigation of vulnerabilities across  federal agency information systems. However, NSD did not provide  details—including processes and procedures—of how this information  could be used to inform future NCPS intrusion prevention capabilities.
Until the department develops processes and procedures for using such  vulnerability information, DHS will not be able to adopt an effective risk- based approach for planning future NCPS intrusion prevention  capabilities.
section_title: NCPS Adoption Varied across the Agencies
paragraphs: OMB Memorandum M-08-05 established the requirement for almost all  federal executive branch agencies to implement the intrusion detection  capabilities (within Einstein 2) of NCPS. In July 2015, the White House  noted that deployment of the NCPS intrusion prevention capabilities was  to be accelerated, with DHS awarding a contract to provide intrusion  prevention services for all federal civilian agencies by the end of 2015.
Agencies have had mixed results in adopting NCPS capabilities.  According to DHS program documentation, all 23 of the non-defense  CFO Act agencies had routed traffic to NCPS intrusion detection sensors.  However, NSD documents indicated that only 5 of the 23 agencies were  receiving intrusion prevention services. Further, NSD documents showed  that for 3 of these 5 agencies, adoption of intrusion prevention services  for e-mail was limited—only 1 agency appeared to have fully adopted  intrusion prevention for e-mail service, another agency had adopted  intrusion prevention for only one part of its network e-mail, and a third  agency was just beginning to adopt the e-mail service.
Further, four of the five selected agencies in our review reported that not  all of their traffic was being sent to NCPS intrusion detection sensors. In  addition, two of the selected agencies reported that they had adopted the  DNS intrusion prevention service, and only one had completed the  adoption process for its e-mail service. See table 4 below for a summary  of NCPS intrusion detection and prevention adoption at the selected five  agencies.
Officials from NSD, the selected agencies in our review, and the Internet  service providers identified several policy and implementation challenges  to adopt the NCPS intrusion prevention capabilities, along with efforts to  address these challenges:
Approval of memoranda of agreement (MOA): An MOA is required in  order to establish NCPS service for an agency. Among other things,  the MOA identifies responsibilities for both DHS and the customer  agency (including interactions with Internet service providers), as well  as identifies points of contact for the respective organizations. Sixteen  of the 23 non-defense CFO Act agencies had an MOA in place with  DHS to provide intrusion prevention services.
Three of the five agencies in our review were in the process of  approving an MOA for intrusion prevention services and cited barriers  to approving the agreement. Specifically, two of these agencies did  not sign an agreement because their Internet service providers had  not been capable of providing NCPS intrusion prevention services.  NSD officials stated that they are in the process of accelerating the  availability of Internet service providers to those agencies which are  not currently provided NCPS intrusion prevention capabilities. Officials  from the third agency stated that there were questions whether  existing law protecting sensitive information in its possession  prohibited participation in NCPS intrusion prevention or not. In July  2015, officials from this agency stated that, working with DHS, it had  agreed to adopt some NCPS intrusion prevention capabilities.
Agency capabilities and concerns: NSD officials noted that the ability  to meet DHS security requirements (e.g., encrypted tunnels) to use  the intrusion prevention capabilities varies from agency to agency.  NSD officials also stated that because each agency has unique  network infrastructures, implementation must be specific to that  agency. Further, NSD officials added that agencies are generally  concerned about interfering with any mission-critical applications,  such as e-mail. Also, while chief information officers usually sign the  MOA, NSD officials noted that network operators within the agency  can be unaware of the agreement, which can pose a potential barrier  to full deployment. To address these issues, NSD staff stated that  they work with agencies to tailor implementation and explain details of  the prevention capabilities to reassure them that business operations  will not be impeded. Additionally, officials from one agency in our  review that had adopted the DNS intrusion prevention capability  initially hesitated to adopt the e-mail capability due to records  management concerns. Agency officials stated in July 2015 that they  are in the process of working with DHS to adopt the e-mail intrusion  prevention capability.
Viability of solution for cloud e-mail: Officials from one agency in our  review stated that they obtain e-mail services from cloud providers,  and added that they hesitate to participate in NCPS intrusion  prevention e-mail capability because there is currently no solution that  is easily implemented. Officials from another agency in the process of  signing an MOA stated that they also use cloud service providers for  e-mail. This agency will also not be able to implement the e-mail  intrusion prevention capability. NSD has noted the challenges  associated with implementing a cloud solution, but plans to refine this  capability over time. However, as we previously stated, the plans to  initiate development efforts on a cloud solution during fiscal year 2016  are not based on fully developed requirements.
Development and operational challenges at Internet service providers:  NSD and two of the Internet service providers noted a challenge with  designing, developing, and operating a classified infrastructure on  unclassified network traffic. For example, the complex and changing  security requirements of one of DHS’s partners who provides threat  information created delays in the service providers’ ability to deliver  intrusion prevention capabilities. In addition, obtaining and retaining  personnel with appropriate security clearances posed a challenge for  the Internet service providers. NSD has acknowledged the inherent  complexity of using classified information to address cyber risks in  non-classified network traffic and has ongoing efforts to work with the  Internet service providers to address this.
Further, NCPS faces additional implementation challenges in ensuring  that agency traffic is sent to the intrusion detection sensors. Specifically,  four of the five agencies in our review cited several challenges in routing  all of their traffic through NCPS intrusion detection sensors, including  capacity limitations of the sensors, agreements with external business  partners that use direct network connections, interagency network  connections that do not route through Internet gateways, use of encrypted  communications mechanisms, and backup network circuits that are not  used regularly. NSD officials stated that agencies are responsible for  routing their traffic to the intrusion detection sensors, and DHS does not  have a role in that aspect of NCPS implementation.
NCPS also faces a challenge in implementing a portion of the intrusion  detection capability and all of the intrusion prevention capability when  routing traffic through sensors at the Internet service providers. Of the  five agencies in our review, four depend on their Internet service provider  to receive NCPS intrusion detection services (through the Managed  Trusted Internet Protocol Service program) and/or intrusion prevention  services. Two of these four agencies had taken steps to securely route  traffic to the sensors, while one agency did not implement an  authentication mechanism to ensure that network routes received by their  router were legitimate. The other agency stated that its Internet service  provider managed its routing configurations and did not provide evidence  for us to verify if secure routing configurations were in place.
This occurred in part because NSD did not provide guidance to customer  agencies on how to securely route their information to the Internet service  providers. NSD officials stated that providing network routing guidance to  customer agencies is not the role of DHS. Rather, they believe that is  best handled by the customer agency and their Internet service provider.  However, without providing network routing guidance, NSD has no  assurance the traffic they see constitutes all or only a subset of the traffic  the customer agencies intend to send. Further, by not providing routing  guidance, NSD has less assurance that customer agency traffic will  actually be picked up at the sensors, since the routing may bypass those  sensors, reducing the effectiveness of NCPS.
section_title: Conclusions
paragraphs: DHS has devoted significant resources to developing and deploying  NCPS, with the goal of strengthening agencies’ ability to detect and  prevent intrusions on their networks, as well as the capability for  analyzing network activity and sharing information between DHS and  agencies. The system’s intrusion detection capabilities are the most fully  developed of the four system objectives, and they provide the ability to  detect known malicious patterns of activity on agency networks. However,  without the ability to effectively detect intrusions across multiple types of  traffic or provide other types of detection capabilities, such as anomaly- based and stateful purpose detection, NCPS is limited in its ability to  identify potential threats. In addition, without making use of publicly  available, open-source repositories to enhance the system’s signatures  and data available from its Continuous Diagnostics and Mitigation  program, DHS may not be providing the ability to detect attacks that  exploit known vulnerabilities.
The system’s intrusion prevention capability is less fully developed, with  limited deployment across different types of network traffic, such as  content from websites, limiting its ability to prevent malicious code from  penetrating agencies’ networks.
Further, NCPS’s support of a number of analytics capabilities, and  ongoing efforts to enhance these, should provide DHS and agencies with  improved ability to analyze potentially malicious traffic in a timely and  efficient manner. However, DHS’s sharing of information with agencies  has not always been effective, with disagreement among agencies about  the number of notifications sent and received and their usefulness.  Finalizing the incident notification process, to include the solicitation of  feedback from customer agencies, could help ensure that DHS is  effectively communicating information that helps agencies strengthen  their security posture. Another step that could assist in ensuring the  effectiveness of NCPS is developing metrics that measure the quality,  efficiency, and accuracy of the services it provides.
DHS has continued to plan for future capabilities of the system, but  without clearly defined requirements, it risks investing in functionality that  does not effectively support agency information security. Moreover, to  ensure a risk-based approach is being pursued to select future NCPS  capabilities, information about vulnerabilities on agency networks could  be a valuable input.
The effectiveness of NCPS further depends on its adoption by agencies.  While the adoption of the intrusion detection capabilities is widespread  among the 23 agencies required to use NCPS, the implementation of  intrusion prevention capabilities is more limited due to policy and  implementation challenges that DHS is working to overcome. However,  addressing a lack of guidance for routing network traffic through NCPS  sensors could help better ensure a wider and more effective use of NCPS  capabilities.
section_title: Recommendations for Executive Action
paragraphs: We recommend the Secretary of Homeland Security direct:
NSD to determine the feasibility of enhancing NCPS’s current  intrusion detection approach to include functionality that would detect  deviations from normal network behavior baselines;
NSD to determine the feasibility of developing enhancements to  current intrusion detection capabilities to facilitate the scanning of  traffic not currently scanned by NCPS;
US-CERT to update the tool it uses to manage and deploy intrusion  detection signatures to include the ability to more clearly link  signatures to publicly available, open-source data repositories;
US-CERT to consider the viability of using vulnerability information,  such as data from the Continuous Diagnostics and Mitigation program  as it becomes available, as an input into the development and  management of intrusion detection signatures;
US-CERT to develop a timetable for finalizing the incident notification  process, to ensure that customer agencies are being sent notifications  of potential incidents, which clearly solicit feedback on the usefulness  and timeliness of the notification;
The Office of Cybersecurity and Communications to develop metrics  that clearly measure the effectiveness of NCPS’s efforts, including the  quality, efficiency, and accuracy of supporting actions related to  detecting and preventing intrusions, providing analytic services, and  sharing cyber-related information;
The Office of Cybersecurity and Communications to develop clearly  defined requirements for detecting threats on agency internal  networks and at cloud service providers to help better ensure effective  support of information security activities;
NSD to develop processes and procedures for using vulnerability  information, such as data from the Continuous Diagnostics and  Mitigation program as it becomes available, to help ensure DHS is  using a risk-based approach for the selection/development of future  NCPS intrusion prevention capabilities; and
NSD to work with their customer agencies and the Internet service  providers to document secure routing requirements in order to better  ensure the complete, safe, and effective routing of information to  NCPS sensors.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Departments of Homeland  Security, Energy, and Veterans Affairs; the General Services  Administration; the Nuclear Regulatory Commission; and the National  Science Foundation for their review and comment. In written comments  signed by the Director, Departmental GAO-OIG Liaison Office, DHS  concurred with each of our nine recommendations. DHS also provided  details about steps that it plans to take to address eight of the nine  recommendations, including estimated time frames for completion. If  effectively implemented, these actions should help address the  weaknesses we identified in the NCPS program. Regarding our  recommendation to develop clearly defined requirements for detecting  threats on agency internal networks and at cloud service providers, the  Director asked that we consider it resolved and closed because a formal  requirements working group and requirements management process had  been developed. We will review the evidence and determine if these  actions address the recommendation. DHS’s written comments are  reprinted in appendix III.
Officials from DHS also provided technical comments via e-mail, which  we incorporated as appropriate. Officials from the Departments of Energy  and Veterans Affairs, General Services Administration, Nuclear  Regulatory Commission, and the National Science Foundation stated that  they had no comments.
We are sending copies of this report to the appropriate congressional  committees, the departments and agencies in our review, and other  interested parties. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  Gregory Wilshusen at (202) 512-6244 or wilshuseng@gao.gov or Dr.  Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine the extent to which (1) the National  Cybersecurity Protection System (NCPS) meets stated objectives, (2) the  Department of Homeland Security (DHS) has designed requirements for  future stages of the system, and (3) federal agencies have adopted the  system.
To determine the extent to which NCPS meets stated objectives, we  compared four of the overarching capabilities of the system (intrusion  detection, intrusion prevention, analytics, and information sharing) to  leading federal practices, including the National Institute of Standards and  Technology’s Special Publication 800-53: Security and Privacy Controls  for Federal Information Systems and Organizations; Special Publication  800-55: Performance Measurement Guide for Information Security, and  Special Publication 800-94: Guide to Intrusion Detection and Prevention  Systems (IDPS). We also examined program information and documents,  as well as interviewed DHS officials within the Office of Cybersecurity and  Communications responsible for designing, developing, maintaining, and  operating NCPS.
For the information-sharing objective we examined NCPS-related incident  notifications DHS stated were sent by the United States Computer  Emergency Readiness Team in fiscal year 2014 to five selected Chief  Financial Officers Act agencies: the Departments of Energy and Veterans  Affairs, the General Services Administration, the National Science  Foundation, and the Nuclear Regulatory Commission. These agencies  were selected based on information provided by DHS regarding the  relative number of NCPS-related incident notifications sent to the  agencies (one with a higher amount of notifications, two with around the  median amount of notifications, and two with the fewest amount of  notifications) and NCPS capabilities received. We also interviewed  information security staff from each of these agencies and collected  information regarding each agency’s perceived usefulness and timeliness  of the incident notifications, along with any feedback provided in response  to the notification.
To evaluate the intrusion detection signatures deployed, we selected 10  common vulnerabilities from 2014 commonly affecting client and server  applications and determined the extent to which the NCPS signatures  provided reasonable coverage for the vulnerability the signature was  intended to mitigate. Additionally, we conducted a similar evaluation for  the signatures associated with a selection of 12 common advanced  persistent threats from 2014. Further we evaluated the number of  intrusion detection signatures DHS had issued for each client vulnerability  during fiscal year 2014 and compared them to the number of signatures  from publicly available repositories, such as common vulnerabilities and  exposures (CVE) published for each corresponding category of  vulnerability during the same period. We then determined the percentage  of DHS coverage by comparing the number of signatures DHS had that  addressed each of the vulnerabilities to the total number of CVEs  released in 2014 for that category.
To determine the extent to which DHS has designed requirements for  future stages of the system, we reviewed NCPS program planning  documentation and interviewed program officials in order to identify how  future capabilities are planned. We compared this information to federal  guidance for planning and requirements development found in the Office  of Management and Budget’s Capital Programming Guide. In addition, for  all new capabilities identified for funding in DHS’s fiscal year 2016 funding  request (such as expanding information sharing, streaming and near-real- time analytics, and deploying intrusion detection sensors at Internet  service providers’ traffic aggregation sites), we determined if formalized  requirements supporting these capabilities had been documented and  approved in program documentation. Further, we determined if plans for  future capabilities to address NCPS’s intrusion prevention objective were  determined using a risk-based approach, including a consideration of  threat, vulnerability, impact, and likelihood.
To determine the extent to which federal agencies have adopted NCPS,  we reviewed policy issued by the Office of Management and Budget and  DHS documentation (such as memoranda of agreement) for the 23 non- defense agencies identified in the Chief Financial Officers Act. We also  discussed any challenges to adoption with DHS officials. To gain a better  understanding of how federal agencies adopt the system, including the  amount of traffic and any challenges or limitations associated with  adoption, we interviewed officials from the five Chief Financial Officers  Act agencies identified previously and reviewed agency network  documentation. We also interviewed officials from the three Internet  service providers currently participating in NCPS to obtain their  perspective on agency adoption of the system.
We conducted this performance audit from June 2014 to January 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Overview of the National Cybersecurity Protection System’s Development and Functionality
paragraphs: The National Cybersecurity Protection System (NCPS), operationally  known as the Einstein program, is an integrated system-of-systems that is  intended to deliver a range of capabilities, including intrusion detection,  intrusion prevention, analytics, and information sharing.
subsections: section_title: Intrusion Detection Capability
paragraphs: The sensors deployed to support the 2003 version of NCPS, or Einstein  1, collect network flow records of data entering and exiting participating  agencies’ networks, which are to be analyzed by U.S. Computer  Emergency Readiness Team (US-CERT) analysts and tools to detect  certain types of malicious activity. If the system detects malicious  activity, US-CERT analysts are to coordinate with the appropriate  agencies to support the mitigation of those threats and vulnerabilities. US- CERT also is to use the information from the sensors to create analyses  of cross-governmental trends that offer agencies an aggregate picture of  external threats against the federal government’s networks.
In 2009, the Department of Homeland Security (DHS) incorporated  network intrusion detection technology into the capabilities of the initial  version of the system, enabling NCPS to monitor Einstein 1 network data  from participating federal agencies for specific predefined patterns of  known malicious activity, referred to as signatures. The NCPS intrusion  detection capability, or Einstein 2, is to use signatures derived from  numerous sources, such as commercial and public computer security  information, incidents reported to US-CERT, information from federal  partners, and independent US-CERT in-depth analysis. When NCPS’s  intrusion detection function detects traffic consistent with malicious  patterns denoted by a particular signature, it provides US-CERT analysts  with a notification. The analyst is then to investigate the detection to  determine if it was in fact an incident and provide mitigation support to the  affected agency, as appropriate.
section_title: Intrusion Prevention Capability
paragraphs: In 2013, DHS’s Network Security Deployment division (NSD) began  deployment of an initial operational capability of the intrusion prevention  function, operationally known as Einstein 3A, which is intended to support  DHS’s ability to actively defend .gov network traffic. One of the major  components supporting the capability is the “Nest,” which is a classified  facility located at each of the participating Internet service providers that  is responsible for off-ramping (i.e., routing traffic to the Nest from the  agency) and on-ramping (i.e., routing traffic from the Nest back to the  Internet) .gov traffic. DHS shares specific indicators of malicious activity  with Internet service providers, who then configure the indicators into  signatures for testing and implementation and match patterns against  established indicators based on known or suspected malicious traffic  traveling to or from the participating agencies. Table 5 below highlights  additional intrusion prevention functions currently available in NCPS.
Once fully deployed across the government, NCPS is intended to  leverage available information from commercial and government sources  to apply in-line protection measures to a wide set of federal network traffic  protocols. When a signature detects a known or suspected cyber threat,  NCPS is supposed to act on that threat to stop malicious traffic and  prevent harm to the intended targets. Figure 3 provides an overview of  how NCPS intrusion prevention capability is designed to work.
section_title: Analytics Capability
paragraphs: NCPS’s analytic capability is intended to capture, organize, and analyze  data collected from NCPS sensors and other data feeds. Table 6 below  highlights key analytics functions currently available in NCPS.
These capabilities are expected to enable US-CERT to fuse information  and correlate malicious network activities across participating federal  executive branch agencies to achieve situational awareness of high- profile cyber threats. US-CERT is responsible for sharing situational  awareness about current and potential cybersecurity threats and  vulnerabilities with federal agencies, state and local governments, private  sector partners, infrastructure owners and operators, and the public.
section_title: Information-Sharing Capability
paragraphs: NCPS’s information-sharing capability is intended to enable enhanced  sharing of information between DHS and its partners through real-time or  near-real-time response; collaboration and coordination; and analysis of  network intrusion attempts, suspicious intrusion activity, and analytical  best practices. When fully developed, NCPS information sharing is  intended to promote the rapid exchange of appropriate cyber threat and  cyber incident information among NCCIC cybersecurity analysts and their  cybersecurity partners, at multiple classification levels. Further, the  capabilities are intended reduce time required to respond to incidents with  better coordination and collaboration, and improved efficiencies with more  automated information sharing and exposure of analysis capabilities.
section_title: Appendix III: Comments from the Department of Homeland Security
section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the contacts named above, Lon C. Chin, Michael W.  Gilmore, Harold Lewis, Christopher Warweg (assistant directors); Andrew  Banister, Bradley Becker, Christopher Businsky, Kush K. Malhotra, Lee  McCracken, and David Plocher made key contributions to this report.