Federal Information Security: Mixed Progress in Implementing Program Components; Improved Metrics Needed to Measure Effectiveness
Released: 2013-09-26T00:00:00

Letter

Background

To help protect against threats to federal systems, FISMA sets forth a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support federal operations and assets. This framework creates a cycle of risk management activities necessary for an effective security program. It is also intended to provide a mechanism for improved oversight of federal agency information security programs.

To ensure the implementation of this framework, FISMA assigns specific responsibilities to agencies, their inspectors general, OMB, and NIST.

FISMA requires each agency to develop, document, and implement an information security program that includes the following components: periodic assessments of the risk and magnitude of harm that could result from the unauthorized access, use, disclosure, disruption, modification, or destruction of information or information systems; policies and procedures that (1) are based on risk assessments, (2) cost-effectively reduce information security risks to an acceptable level, (3) ensure that information security is addressed throughout the life cycle of each system, and (4) ensure compliance with applicable requirements; subordinate plans for providing adequate information security for networks, facilities, and systems or group of information systems, as appropriate; security awareness training to inform personnel of information security risks and of their responsibilities in complying with agency policies and procedures, as well as training personnel with significant security responsibilities for information security; periodic testing and evaluation of the effectiveness of information security policies, procedures, and practices, to be performed with a frequency depending on risk, but no less than annually, and that includes testing of management, operational, and technical controls for every system identified in the agency’s required inventory of major information systems; a process for planning, implementing, evaluating, and documenting remedial action to address any deficiencies in the information security policies, procedures, and practices of the agency; procedures for detecting, reporting, and responding to security plans and procedures to ensure continuity of operations for information systems that support the operations and assets of the agency.

In addition, agencies are to report annually to OMB, certain congressional committees, and the Comptroller General on the adequacy and effectiveness of information security policies, procedures, and practices, and compliance with FISMA. The act also requires each agency inspector general, or other independent auditor, to annually evaluate and report on the information security program and practices of the agency.

OMB’s responsibilities include developing and overseeing the implementation of policies, principles, standards, and guidelines on information security in federal agencies (except with regard to national security systems). It is also responsible for ensuring the operation of a federal information security incident center. The required functions of this center are performed by the DHS United States Computer Emergency Readiness Team (US-CERT), which was established to aggregate and disseminate cybersecurity information to improve warning and response to incidents, increase coordination of response information, reduce vulnerabilities, and enhance prevention and protection. OMB is also responsible for reviewing, at least annually, and approving or disapproving agency information security programs.

Since it began issuing guidance to agencies in 2003, OMB has instructed agency chief information officers and inspectors general to report on a variety of metrics in order to satisfy reporting requirements established by FISMA. Over time, these metrics have evolved to include administration priorities and baseline metrics meant to allow for measurement of agency progress in implementing information security-related priorities and controls. OMB requires agencies and inspectors general to use an interactive data collection tool called CyberScopemetrics. The metrics are used by OMB to summarize agencies’ progress in meeting FISMA requirements and report this progress to Congress in an annual report as required by FISMA.

NIST’s responsibilities under FISMA include the development of security standards and guidelines for agencies that include standards for categorizing information and information systems according to ranges of risk levels, minimum security requirements for information and information systems in risk categories, guidelines for detection and handling of information security incidents, and guidelines for identifying an information system as a national security system. (See app. II for additional information on agency responsibilities under FISMA.)

In the 11 years since FISMA was enacted into law, executive branch oversight of agency information security has changed. As part of its FISMA oversight responsibilities, OMB has issued annual instructions for agencies and inspectors general to meet FISMA reporting requirements. However, in July 2010, the Director of OMB and the White House Cybersecurity Coordinator issued a joint memorandumwas to exercise primary responsibility within the executive branch for the operational aspects of cybersecurity for federal information systems that fall within the scope of FISMA. The memo stated that DHS activities would include five specific responsibilities of OMB under FISMA: overseeing implementation of and reporting on government cybersecurity policies and guidance; overseeing and assisting government efforts to provide adequate, risk-based, and cost-effective cybersecurity; overseeing agencies’ compliance with FISMA; overseeing agencies’ cybersecurity operations and incident response; annually reviewing agencies’ cybersecurity programs.

The OMB memo also stated that in carrying out these responsibilities, DHS is to be subject to general OMB oversight in accordance with the provisions of FISMA. In addition, the memo stated that the Cybersecurity Coordinator would lead the interagency process for cybersecurity strategy and policy development. Subsequent to the issuance of this memo, both OMB and DHS began issuing annual reporting instructions to agenciesand DHS began issuing reporting metrics to agencies and inspectors general instead of OMB.

Within DHS, the Federal Network Resilience division’s Cybersecurity Performance Management Branch is responsible for (1) developing and disseminating FISMA reporting metrics, (2) managing the CyberScope web-based application, and (3) collecting and reviewing federal agencies’ cybersecurity data submissions and monthly data feeds to CyberScope. In addition, the Cybersecurity Assurance Program Branch is responsible for conducting cybersecurity reviews and assessments at federal agencies to evaluate the effectiveness of agencies’ information security programs.

Mixed Progress Has Been Made in Implementing Many FISMA Requirements, but Weaknesses Continue in Agencies’ Security Programs

In fiscal year 2012, agencies and their inspectors general reported mixed progress from fiscal year 2011 in implementing many of the requirements for establishing an entity-wide information security program. According to inspectors general reports, agencies (1) improved in establishing a program for managing information security risk; (2) generally documented information security program policies and procedures; (3) generally implemented certain elements of security planning; (4) declined in providing security awareness training but improved in providing specialized training; (5) generally established test and evaluation programs and are working toward establishing continuous monitoring programs; (6) declined in implementing elements of a remediation program; (7) generally established programs for detecting, responding to, and reporting security incidents; and (8) declined in implementing elements of continuity of operations programs. Notwithstanding the mixed progress made, GAO and inspectors general continue to identify weaknesses in agencies’ information security programs and make recommendations to mitigate the weaknesses identified. In addition, OMB and DHS continued to develop reporting metrics and assist agencies in improving their information security programs; however, the metrics do not evaluate all FISMA requirements, focused mainly on compliance rather than effectiveness of controls, and in many cases did not identify specific performance targets for determining levels of implementation. Finally, inspectors general conducted the required independent evaluations of agency information security programs, and NIST continued to issue guidance to assist agencies with implementing controls to improve their information security posture.

An Increasing Number of Agencies Have Implemented Programs for Managing Information Security Risk, but Weaknesses Remain in Program Elements

FISMA requires that the head of each agency provide information security protections commensurate with the risk resulting from unauthorized access, use, disclosure, disruption, modification, or destruction of agency information and information systems. FISMA specifically requires agencies to assess this risk in order to determine the appropriate controls needed to remediate or mitigate the risk to the agency. To assist agencies in identifying risks, NIST has issued risk management and assessment guides for organizations and information systems. According to NIST’s Guide for Applying the Risk Management Framework to Federal Information Systems, risk management is addressed at the organization level, the mission and business process level, and the information system level. Risks are addressed from an organizational perspective with the development of, among other things, risk management policies, procedures, and strategy. The risk decisions made at the organizational level guide the entire risk management program.

Agencies made progress in implementing programs for managing information security risk in fiscal year 2012. According to inspectors general reports, an increasing number of agencies implemented a program for managing information security risk that is consistent with FISMA requirements and its implementing guidance. Specifically, 18 of 24 agencies in fiscal year 2012 implemented such a program compared to 8 of 24 in 2011. In addition, an increasing number of agencies documented policies, procedures, and strategies—three key components for assessing and managing risk. Figure 1 shows agency progress in documenting and implementing a risk management program and key elements of that program in fiscal years 2011 and 2012.

Although an increasing number of agencies have implemented a risk management program and documented policies, procedures, and strategies, agency inspectors general identified areas for improvement in their agency’s risk assessment and management activities. For example, in fiscal year 2012, 20 of 24 agencies had weaknesses in periodically assessing and validating risks. To illustrate, 1 agency did not conduct a risk assessment to ensure that the impact of mobile devices and their associated vulnerabilities were adequately addressed. Another agency’s risk assessments were not properly updated, as they included references to inaccurate system environment information. Another agency was missing key elements in its approach to managing risk at an agency-wide level, including conducting an agency-wide risk assessment and communicating risks to system owners. In addition, fewer agencies addressed risk from a mission or business perspective in fiscal year 2012 than in fiscal year 2011, declining from 15 to 14 agencies. Risk management is at the center of an effective information security program, and without an effective risk management program agencies may not be fully aware of the risks to essential computing resources, and may not be able to make informed decisions about needed security protections.

Most Agencies Documented Information Security Program Policies and Procedures, but Implementation Was Inconsistent

FISMA requires agencies to develop, document, and implement policies and procedures that are based on risk assessments; cost-effectively reduce information security risks to an acceptable level; ensure that information security is addressed throughout the life cycle of each agency information system; and ensure compliance with FISMA requirements, OMB policies and procedures, minimally acceptable system configuration requirements, and any other applicable requirements.

In fiscal years 2011 and 2012, OMB asked inspectors general to report on whether agencies had documented policies and procedures for 11 information system control categories. These controls are intended to (1) manage risks to organizational operations, assets, and individuals resulting from the operation of information systems; (2) provide reasonable assurance that changes to information system resources are authorized and systems are configured and operating securely and as intended; (3) rapidly detect incidents, minimize loss and destruction, mitigate exploited weaknesses, and restore IT services; (4) inform agency personnel of the information security risks associated with their activities and inform agency personnel of their responsibilities in complying with agency policies and procedures designed to reduce these risks; (5) ensure individuals with significant security responsibilities understand their responsibilities in securing information systems; (6) assist agencies in identifying, assessing, prioritizing, and monitoring the progress of corrective efforts for security weaknesses found in programs and systems; (7) deter, detect, and defend against unauthorized network access; (8) ensure access rights are only given to the intended individuals or processes; (9) maintain a current security status for one or more information systems or for all information systems on which the organization’s mission depends; (10) ensure agencies are adequately prepared to cope with the loss of operational capabilities due to a service disruption such as an act of nature, fire, accident, or sabotage; and (11) assist agencies in determining whether contractor-operated systems have adequate security.

Inspectors general reported that most agencies documented policies and procedures that were consistent with federal guidelines and requirements; however, several agencies had not fully documented policies and procedures for individual control categories. In addition, the number of agencies documenting policies and procedures increased for some control categories, but declined for others. For example, an increasing number of agencies documented policies and procedures for risk management, configuration management, and continuous monitoring, but the number of agencies documenting policies and procedures for security awareness and remote access declined. According to OMB, the decline in the number of agencies documenting certain policies and procedures could be due to agencies’ not updating their policies and procedures after new federal requirements are established or new technologies are deployed. Table 1 provides a summary of the number of agencies that fully documented information security program policies and procedures for fiscal years 2011 and 2012.

Although most agencies documented security policies and procedures, they often did not fully or consistently implement them. To illustrate, most major federal agencies had weaknesses in the following information system controls:

Access controls: In fiscal year 2012, almost all (23 of 24) of the major federal agencies had weaknesses in the controls that are intended to limit or detect access to computer resources (data, programs, equipment, and facilities), thereby protecting them against unauthorized modification, loss, and disclosure. For example, 21 of 24 agencies had weaknesses in their ability to appropriately identify and authenticate system users. To illustrate, although agencies are required to uniquely identify users on their systems, some users shared accounts at 1 agency, and administrators shared accounts for multiple systems at another agency, making it difficult for the agencies to account for user and administrator activity on their systems. Other agencies had weak password controls, including systems with passwords that had not been changed from the easily guessable default passwords supplied by the vendor. In addition, 20 of 24 agencies had weaknesses in the process used to grant or restrict user access to information technology resources. For example, 1 agency had not disabled 363 user accounts for individuals who were no longer employed by the agency, despite a department policy of disabling these accounts within 48 hours of an employee’s departure. Further, 18 of 24 agencies had weaknesses in the protection of information system boundaries. For example, although 1 agency had established a program for remote access to agency systems, it had not ensured that authentication mechanisms for remote access meet NIST guidelines for remote authentication. Lastly, 11 of 24 agencies had weaknesses in their ability to restrict physical access or harm to computer resources and protect them from unintentional loss or impairment. For example, 1 agency had not always deactivated physical access cards for contractors that no longer worked at the agency and had provided physical access to employees that were not approved for such access.

Configuration management: In fiscal year 2012, all 24 agencies had weaknesses in the controls that are intended to prevent unauthorized changes to information system resources (for example, software programs and hardware configurations) and provide reasonable assurance that systems are configured and operating securely and as intended. For example, 20 of 24 agencies had weaknesses in processes for updating software to protect against known vulnerabilities. One agency had not installed critical updates in a timely manner for 14 of 15 systems residing on one if its networks reviewed by the agency’s inspector general. Another agency had multiple database update-related vulnerabilities dating back to 2009. In addition, 17 of 24 agencies had weaknesses in authorizing, testing, approving, tracking, and controlling system changes. For example, most of the system change request records reviewed by 1 agency’s independent auditor did not include the proper approvals for the system change.

Segregation of duties: In fiscal year 2012, 18 of 24 agencies had weaknesses in the controls intended to prevent one individual from controlling all critical stages of a process, which is often achieved by splitting responsibilities between two or more organizational groups. For example, at 1 agency, excessive system access was granted to users of at least seven systems and may have allowed users to perform incompatible duties. The same agency also did not have an effective process for monitoring its systems for users with the ability to perform these incompatible duties.

Illustrating the extent to which weaknesses affect the 24 major federal agencies, inspectors general at 22 of 24 agencies cited information security as a major management challenge for their agency, and 19 agencies reported that information security control deficiencies were either a material weakness or significant deficiency in internal controls over financial reporting in fiscal year 2012. Until all agencies properly document and implement policies and procedures, and implement recommendations made by us and inspectors general to correct weaknesses identified, they may not be able to effectively reduce risk to their information and information systems, and the information security practices that are driven by these policies and procedures may be applied inconsistently.

Agencies Generally Implemented Elements of Security Planning but Did Not Consistently Develop or Update Security Plans

FISMA requires an agency’s information security program to include plans for providing adequate information security for networks, facilities, and systems or groups of information systems, as appropriate. According to NIST, the purpose of the system security plan is to provide an overview of the security requirements of the system and describe the controls in place or planned for meeting those requirements. The first step in the system security planning process is to categorize the system based on the impact to agency operations, assets, and personnel should the confidentiality, integrity, and availability of the agency information and information systems be compromised. This categorization is then used to determine the appropriate security controls needed for each system. Another key step is selecting a baseline of security controls for each system and documenting those controls in the security plan.

In fiscal years 2011 and 2012, OMB asked inspectors general to report on whether their agency appropriately categorized information systems and selected appropriate baseline security controls. Although a few inspectors general reported weaknesses in their agency’s process for categorizing information systems, 21 of 24 reported that agencies appropriately categorized them in fiscal years 2011 and 2012. In addition, in fiscal years 2011 and 2012, 18 of 24 inspectors general also stated that agencies selected an appropriately tailored set of baseline security controls.

However, inspectors general at 19 of 24 agencies reported that security plans were not always complete or properly updated. For example, 11 system security plans at 1 agency did not meet the minimum security requirements required by NIST 800-53.agency were not consistently updating system security plans to reflect the current operating environment. Further, 2 of the 16 system security plans reviewed at another agency had not been updated within the required 3- year period.

Until agencies appropriately develop and update system security plans and implement recommendations made by us and inspectors general to correct weaknesses identified, they may face an increased risk that officials will be unaware of system security requirements and that controls are not in place.

Agencies Declined in Providing Security Awareness Training but Improved in Providing Specialized Training

FISMA requires agencies to provide security awareness training to personnel, including contractors and other users of information systems that support the operations and assets of the agency. Training is intended to inform agency personnel of the information security risks associated with their activities, and their responsibilities in complying with agency policies and procedures designed to reduce these risks. FISMA also requires agencies to train and oversee personnel with significant security responsibilities for information security with respect to those responsibilities. Providing training to agency personnel is critical to securing information and information systems because people are one of the weakest links in attempts to secure systems and networks. In fiscal years 2011 and 2012, OMB required agencies to report on the number of network users that were provided and successfully completed security awareness training for that year. Agencies were also required to report on the number of network users and other staff with significant security responsibilities that were provided specialized training.

In fiscal year 2012, 12 of the 24 agencies provided annual security awareness training to at least 90 percent of their network users, which is a notable decline from fiscal year 2011, in which 22 of 24 agencies provided training to at least 90 percent of their users. Inspectors general at 17 of 24 agencies also reported weaknesses in security awareness programs, including agencies’ ability to track the number of system users provided training that year. For example, 5 of 24 inspectors general reported that their agency’s process for identifying and tracking the status of security awareness training was not adequate or in accordance with government policies, an improvement over 10 of 24 in fiscal year 2011. To illustrate, in fiscal year 2011, 1 agency could not identify evidence of security awareness training for over 12 percent of system users at three component agencies. Another agency lacked a process to ensure all contractors were identified and provided with security awareness training in fiscal year 2012. Without sufficiently trained security personnel, security lapses are more likely to occur and could contribute to further information security weaknesses.

In fiscal year 2012, 16 of 24 agencies provided specialized training to at least 90 percent of their users with significant security responsibilities, a slight increase from 15 of 24 in fiscal year 2011. In addition, inspectors general reported in fiscal year 2012 that 22 of 24 agencies established a specialized training program that complied with FISMA, an improvement over fiscal year 2011, in which half of the major federal agencies had established such a program. Further, in fiscal year 2012, 19 of 24 inspectors general reported that their agency’s mechanism for tracking individuals who need specialized training was adequate, a slight improvement from fiscal year 2011, in which 17 of 24 reported adequate tracking.

Although the number of agencies implementing specialized training programs increased, 16 of 24 inspectors general identified weaknesses with such programs in fiscal year 2012. For example, 1 agency had not yet defined “significant information security responsibilities” in order to identify those individuals requiring specialized training. Another agency’s specialized training process was ad hoc and everyone with significant security responsibilities had taken the same training course, not one that was tailored for their specific job roles. While agencies have made progress in implementing specialized training programs that comply with FISMA, without tailoring training to specific job roles agencies are at increased risk that individuals with significant security responsibilities may not be adequately prepared to perform their specific responsibilities in protecting the agency’s information and information systems.

Most Agencies Established a Test and Evaluation Program and Are Working toward Establishing Continuous Monitoring Programs

FISMA requires that federal agencies periodically test and evaluate the effectiveness of their information security policies, procedures, and practices as part of implementing an agency-wide security program. This testing is to be performed with a frequency depending on risk, but no less than annually. Testing should include management, operational, and technical controls for every system identified in the agency’s required inventory of major systems. This type of oversight is a fundamental element that demonstrates management’s commitment to the security program, reminds employees of their roles and responsibilities, and identifies and mitigates areas of noncompliance and ineffectiveness. Although control tests and evaluations may encourage compliance with security policies, the full benefits are not achieved unless the results are used to improve security. In recent years, the federal government has been moving toward implementing a more frequent control testing process called continuous monitoring. In March 2012, the White House Cybersecurity Coordinator announced that his office, in coordination with experts from DHS, the Department of Defense (DOD), and OMB, had identified continuous monitoring of federal information systems as a cross-agency priority area for improving federal cybersecurity. According to NIST, the goal of continuous monitoring is to transform the otherwise static test and evaluation process into a dynamic risk mitigation program that provides essential, near real-time security status and remediation. In February 2010, NIST included continuous monitoring as one of six steps in its risk management framework described in NIST special publication 800-37. In addition, in September 2011 NIST published special publication 800-137 to assist organizations in the development of a continuous monitoring strategy and the implementation of a continuous monitoring program that provides awareness of threats and vulnerabilities, visibility into organizational assets, and the effectiveness of implemented security controls.

The majority of federal agencies implemented elements of test and evaluation programs in fiscal years 2011 and 2012. For fiscal year 2012, 17 of 24 inspectors general reported that agencies assessed controls using appropriate assessment procedures to determine the extent to which controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements for the system.

However, 17 of 24 inspectors general also identified weaknesses in agencies’ processes for testing and evaluating identified controls. For example, 10 of 23 agencies did not monitor information security controls on an ongoing basis in fiscal year 2012. According to DHS, monitoring information security controls includes assessing control effectiveness, documenting changes to the system or its environment of operation, conducting security impact analyses of the associated changes, and reporting the security state of the system to designated organizational officials. One agency had not performed ongoing assessments of selected security controls on nearly 10 percent of its systems in fiscal year 2012. Another agency had not met the basic test and evaluation requirement for the past 5 years, and this was the major reason the agency’s inspector general classified its information security governance as a material weakness for financial reporting. The identified weaknesses in test and evaluation programs could limit agencies’ awareness of vulnerabilities in their critical information systems.

According to OMB’s annual report to Congress, agencies reported improvements in fiscal year 2012 in implementing tools that provided automated continuous monitoring capabilities for vulnerability, configuration, and asset management for the agency’s information systems.goal of 80 percent for implementing an automated capability to assess vulnerability, configuration, and asset management information for agencies’ information technology assets in fiscal year 2012. According to OMB, 17 of 24 major federal agencies reported at least 80 percent implementation of this capability for asset and configuration management, and 16 of 24 reported at least 80 percent implementation of this capability for vulnerability management. In addition, as figure 2 illustrates, most agencies reported an overall improvement in the percentage of information technology assets with these automated capabilities from fiscal year 2011 to 2012. Specifically, 12 agencies increased the percentage of information technology assets with automated capabilities for asset management, 18 agencies increased the percentage of information technology assets with automated capabilities for configuration management, and 14 agencies increased the percentage of information technology assets with automated capabilities for vulnerability management.

The annual DHS reporting metrics established a minimum telecommunication connections and ensure a set of baseline security capabilities for situational awareness and enhanced monitoring.

Continuous monitoring of federal information systems: Transform the otherwise static security control assessment and authorization process into a dynamic risk mitigation program that provides essential, near real-time security status and remediation, increasing visibility into system operations and helping security personnel make risk management decisions based on increased situational awareness.

Strong authentication: Increase the use of federal smartcard credentials such as Personal Identity Verification and Common Access Cards that provide multifactor authentication and digital signature and encryption capabilities, authorizing users to access federal information systems with a higher level of assurance.

CyberStat reviews: In fiscal year 2011, DHS, along with OMB and National Security Staff (NSS),of seven federal agencies. According to OMB, these CyberStat reviews were face-to-face, evidence-based meetings to ensure agencies were accountable for their cybersecurity posture and assist them in developing focused strategies for improving their information security posture in areas where they were facing challenges. According to OMB, these reviews resulted in a prioritized action plan for the agency to improve overall agency performance. CyberStat reviews were also conducted for seven agencies in fiscal year 2012. According to OMB, these meetings focused heavily on the three administration priorities and not specifically on FISMA requirements. The top challenges raised by agencies in fiscal year 2012 included the need to upgrade legacy systems to support new capabilities, acquire skilled staff, and ensure that the necessary financial resources were allocated to the administration’s priority initiatives for cybersecurity. According to DHS, OMB and NSS are now requiring a CyberStat review of all 24 major federal agencies for fiscal year 2013—a new process that began in December 2012. However, in May 2013 OMB officials stated that while conducting CyberStat reviews of all 24 agencies is their goal, they would not meet that goal this year, and in July 2013 DHS officials stated that they do not have the capacity to meet with all 24 agencies in 1 fiscal year. conducted the first CyberStat reviews

CIO and CISO interviews: In fiscal year 2011, DHS began interviewing agency CIO’s and chief information security officers (CISO) on their agency’s cybersecurity posture. According to OMB, these interviews had three distinct goals: (1) assessing the agency’s FISMA compliance and challenges, (2) identifying security best practices and raising awareness of FISMA reporting requirements, and (3) establishing meaningful dialogue with the agency’s senior leadership.

Baseline metrics: Many of the fiscal year 2010 metrics were carried over into fiscal year 2011, which established a baseline and provided an opportunity to measure progress in federal agencies and the federal government as a whole. According to OMB, establishing these baseline metrics has improved their understanding of the current cybersecurity posture and helped to drive accountability for improving the collective effectiveness of the federal government’s cybersecurity capabilities.

In our 2009 report on efforts needed to improve federal performance measures, we found that leading organizations and experts have identified different types of measures that are useful in helping to achieve information security goals:

Compliance measures, which are used to determine the extent to which security controls were in place that adhered to internal policies, industry standards, or other legal or regulatory requirements. These measures are effective at pointing out where improvements are needed in implementing required policies and procedures but provide only limited insight into the overall performance of an organization’s information security program.

Control effectiveness measures, which characterize the extent to which specific control activities within an organization’s information security program meet their objectives. Rather than merely capturing what controls are in place, such measures gauge how effectively the controls have been implemented.

These categories are consistent with those laid out by NIST in its information security performance measurement guide, which serves as official guidance on information security measures for federal agencies and which OMB requires agencies to follow. In addition, information security experts, as well as NIST guidance, indicated that organizations with increasingly effective information security programs should migrate from predominantly using compliance measures toward a balanced set of measures to include various types of measures.

Further, we found that measures generally have key characteristics and attributes. For example, measures are most meaningful to an organization when they, among other things, had targets or thresholds for each measure to track progress over time and are linked to organizational priorities.

In our report we recommended that OMB, among other things, revise annual reporting guidance to agencies to require (1) reporting on a balanced set of measures, including measures that focus on the effectiveness of control activities and program impact; and (2) inclusion of all key attributes in the development of measures. OMB concurred with our recommendations and revised its fiscal year 2010 reporting instructions and metrics accordingly.

For fiscal years 2011 and 2012, DHS, as part of its recently assigned responsibilities for FISMA oversight, developed a revised set of reporting metrics to assess agencies’ compliance with the act. Specifically, inspectors general were asked to report on 11 information system control categories, and agency chief information officers were asked to report on 12 categories, as indicated in table 6.

For each category, inspectors general and chief information officers were required to answer a series of questions related to the agency’s implementation of these controls.

The metrics developed for inspectors general and chief information officers by DHS for fiscal year 2012 address compliance with six of the eight components of an information security program as required by FISMA. Specifically, the metrics address the establishment of information security policies and procedures; security training; periodic testing and evaluation of the effectiveness of information security policies, procedures, and practices; remedial actions to address information security deficiencies; procedures for detecting, reporting, and responding to security incidents; and continuity of operations plans and procedures. However, these metrics do not specifically discuss two of the eight components—agencies’ processes for conducting risk assessments or developing security plans. For example, while the metrics ask inspectors general to report on their agency’s policies and procedures for risk management and its overall risk management program, they do not specifically require inspectors general or agency chief information officers to report on whether the agency has periodically assessed the risk and magnitude of harm that could result from the compromise of information and information systems that support the operations and assets of the agency, as required by FISMA. The metrics also do not specifically require agencies or inspectors general to comment on the development, documentation, and implementation of subordinate plans for providing adequate security for networks, facilities, and systems or groups of systems, as appropriate. Without measuring agencies’ compliance with these FISMA requirements, DHS, OMB, and other stakeholders will have less insight into the implementation of agencies’ information security programs.

As highlighted in our 2009 report,measures in addition to compliance measures can provide additional insight into how effectively control activities are meeting their security objectives. According to OMB instructions for FISMA reporting, the DHS metrics for inspectors general were also designed to measure the effectiveness of agencies’ information security programs, and OMB relied on responses by inspectors general to these metrics to gauge the effectiveness of information security programs. the use of control effectiveness While some of the metrics for inspectors general were intended to measure effectiveness, many of them did not. The 2012 metrics ask inspectors general to determine whether or not their agency has established a program for each of the 11 information system control categories, and whether or not these programs include key security practices. Several of these metrics were intended to reflect the effectiveness of agencies’ program practices within the control categories. For example, for the incident response and reporting category, inspectors general were asked whether their agency responded to and resolved incidents in a timely manner and whether it reported incidents to US- CERT and law enforcement within established time frames. However, many of the metrics for inspectors general did not provide a means of assessing the effectiveness of the program for control categories. Specifically, the metrics focus on the establishment of the program but do not require inspectors general to characterize the extent to which these program components meet their objectives. For each control category, the metrics ask whether the agency established an enterprise-wide program that was consistent with FISMA requirements, OMB policy, and applicable NIST guidelines. However, these metrics do not allow the inspectors general to respond on how effectively the program is operating. Instead, they capture whether programs have been established.

The lack of effectiveness metrics has led to inconsistencies in inspector general reporting. The following examples illustrate that while inspectors general reported, via responses to the DHS metrics, that their agency had established programs for implementing control categories, they also reported continuing weaknesses in those controls in the same year.

One inspector general responded to the metric for plans of action and milestones (i.e., remediation program) that its agency had a remediation program in place that is consistent with FISMA requirements, tracks and monitors weaknesses, includes remediation plans that are effective at correcting weaknesses, remediates weaknesses in a timely manner, and adheres to milestone remediation dates. However, the inspector general audit of the agency’s information security program identified 4,377 unremediated weaknesses, and the resulting report stated that component agencies were not entering or tracking all information security weaknesses.

Another inspector general reported in response to the contractor systems metric that its agency updates the inventory of contractor systems at least annually; however, a report we issued on this agency’s information security program identified a weakness in the accuracy of the agency’s inventory of systems, including those systems operated by contractors. Specifically, the agency provided three different information systems inventories and none of them had the same information, reducing the agency’s assurance that information systems were properly accounted for.

In response to the configuration management metric, an inspector general at another agency stated that software scanning capabilities were fully implemented. However, the inspector general’s independent evaluation showed that although the systems reviewed had the capability for software scanning, none of the systems were being fully scanned for vulnerabilities in accordance with agency requirements.

Without fully or consistently measuring the effectiveness of controls, DHS, OMB, and other stakeholders will lack insight into the performance of agencies’ information security programs.

In October 2011, we determined that of the 31 metrics for CIOs for fiscal year 2010, 30 of them did not include performance targets that would allow agencies to track progress over time. We recommended that the Director of OMB incorporate performance targets for metrics in annual FISMA reporting guidance to agencies and inspectors general. OMB generally agreed with our recommendation.

In fiscal year 2012, DHS included explicit performance targets for metrics that were linked to the three cross-agency cybersecurity priority goals discussed earlier. For example, agencies were to ensure that 75 percent of all users were required to use personal identity verification cards to authenticate to their systems. While this partially addresses our previous recommendation, no explicit targets were established for metrics that did not relate to the three cross-agency cybersecurity priority goals, such as metrics related to data protection, incident management, configuration management, incident response and reporting, and remediation programs.

DHS officials acknowledged that these targets were needed, but that agency resources and the lack of DHS authority to establish targets have prevented the department from establishing additional targets. The officials also stated that only certain targets were included at this time in order to focus agency resources and senior leadership attention on those items that they believed would create the most change in federal information security. They added that additional targets will be included over time. Developing targets for additional metrics, as we previously recommended, will enable agencies and oversight entities to better gauge progress in securing federal systems.

In June 2013, the DHS inspector general issued a report on the results of its evaluation of whether DHS has implemented its additional cybersecurity responsibilities effectively to improve the security posture of the federal government. It found that DHS had not developed a strategic implementation plan that describes its cybersecurity responsibilities or establishes specific time frames and milestones to provide a clear plan of action for fulfilling those responsibilities. The report also stated that DHS had not established performance metrics to measure and monitor its progress in accomplishing its mission and goals. According to the inspector general, management turnover has hindered DHS’s ability to develop a strategic implementation plan. Specifically, three key individuals essential to the DHS division overseeing FISMA compliance have left the agency since July 2012. The inspector general recommended that DHS coordinate with OMB to develop a strategic implementation plan that identifies long-term goals and milestones for federal agency FISMA compliance.

In addition, the inspector general found that some agencies indicated that DHS could make further improvements to the clarity and quality of the FISMA reporting metrics. Specifically, five agencies indicated that some of the fiscal year 2012 and 2013 metrics were unclear and should be revised. In addition, two agencies stated that the reporting process was a strain on personnel resources because there are too many metrics. Some agency officials we interviewed echoed the need for clearer metrics and agreed that the process was time consuming. The inspector general recommended that DHS improve communication and coordination with federal agencies by providing additional clarity regarding the FISMA reporting metrics.

DHS agreed with the recommendations and officials stated that they are developing a strategic plan and documenting a methodology for metric development with the specific aim of improving the quality of the metrics, but did not state when the plan would be completed.

Inspectors General Conducted Independent Evaluations of Agency Information Security Programs

FISMA requires that agencies have an independent evaluation performed each year to evaluate the effectiveness of the agency’s information security program and practices. FISMA also requires this evaluation to include (1) testing of the effectiveness of information security policies, procedures, and practices of a representative subset of the agency’s information systems; and (2) an assessment of compliance with FISMA requirements, and related information security policies, procedures, standards, and guidelines. For agencies with inspectors general, FISMA requires that these evaluations be performed by the inspector general or an independent external auditor. Lastly, FISMA requires that each year the agencies submit the results of these evaluations to OMB and that OMB summarize the results of the evaluations in its annual report to Congress. According to OMB, instructions for FISMA reporting, the metrics for inspectors general were designed to measure the effectiveness of agencies’ information security programs and OMB relied on responses by inspectors general to gauge the effectiveness of information security program processes.

Our review of reports issued by inspectors general from the 24 major federal agencies in fiscal years 2011 and 2012 show that all 24 inspectors general conducted evaluations, identified weaknesses in agency information security programs and practices, and included recommendations to address the weaknesses. Inspectors general responded to the DHS-defined metrics for reporting on agency implementation of FISMA requirements, and most inspectors general also issued a more detailed audit report discussing the results of their evaluation of agency policies, procedures, and practices. One inspector general responded to the DHS metrics, but chose not to issue an additional detailed report on the results of the evaluation in fiscal year 2012. Three other inspectors general issued reports that summarized weaknesses contained in multiple reports throughout the reporting period.

NIST Continues to Provide Standards and Guidance to Assist Federal Agencies

To fulfill its responsibility to provide standards and guidance to agencies on information security, NIST has produced numerous information security standards and guidelines as well as updated existing information security publications. In April 2013, NIST released the fourth update of a key federal government computer security control guide, Special Publication 800-53: Security and Privacy Controls for Federal Information Systems and Organizations. According to NIST, the update was motivated by expanding threats and the increasing sophistication of cyber attacks. According to NIST, over 200 controls were added to help address these expanding threats and vulnerabilities. Examples include controls related to mobile and cloud computing; applications security; trustworthiness, assurance, and resiliency of information systems; insider threat; supply chain security; and the advanced persistent threat. As with previous versions of special publication 800-53, the controls contained in the latest update, according to NIST, can and should be tailored for specific needs of the agency and based on risk. In addition to this guide, NIST also issued and revised several other guidance documents. Table 7 lists recent NIST updates and releases.

In August 2012, NIST also published the National Cybersecurity Workforce Framework, which established a common taxonomy and lexicon that is to be used to describe all cybersecurity work and workers regardless of where or for whom the work is performed. This framework was developed as part of a larger effort to educate, recruit, train, develop, and retain a highly qualified workforce in the federal government as well as other sectors.

In addition, in partnership with the Department of Defense, the intelligence community, and the Committee on National Security Systems, NIST developed a unified information security framework to provide a common strategy to protect critical federal information systems and associated infrastructure for national security and non-national security systems. Historically, information systems in civilian agencies have operated under different security controls than military and intelligence systems. According to NIST, the framework provides standardized risk management policies, procedures, technologies, tools, and techniques that can be applied by all federal agencies. See table 8 for a list of publications that make up the framework.

Conclusions

According to NIST officials, an update to SP 800-53A is expected to be released this year. weaknesses continued to be identified for all of the components of an information security program, and we and agency inspectors general have made numerous recommendations to address these weaknesses and strengthen agencies’ programs. These weaknesses show that information security continues to be a major challenge for federal agencies, and addressing these weaknesses is essential to establishing a robust security posture for the federal government. Until steps are taken to address these persistent challenges, overall progress in improving the nation’s cybersecurity posture is likely to remain limited. Moreover, while OMB and DHS have continued to oversee agencies’ FISMA implementation, they have not included all FISMA requirements; developed effectiveness measures; or, as we have recommended, established performance targets for many of the metrics agencies and inspectors general use to report on agencies’ progress, making it more difficult to accurately assess the extent to which agencies are effectively securing their systems. Without more relevant metrics, OMB and DHS may lack adequate visibility into the federal government’s information security posture.

Recommendations for Executive Action

We recommend that the Director of the Office of Management and Budget, in coordination with the Secretary of Homeland Security, take the following actions to enhance the usefulness of the annual FISMA reports and to provide additional insight into agencies’ information security programs: develop compliance metrics related to periodic assessments of risk and development of subordinate security plans, and develop metrics for inspectors general to report on the effectiveness of agency information security programs.

Agency Comments and Our Evaluation

We provided a draft of this report to OMB, DHS, the Departments of Commerce, Education, Energy, and Transportation; the Environmental Protection Agency; and the Small Business Administration. The audit liaison for OMB responded via e-mail on September 10, 2013, that OMB generally agreed with our recommendations, but provided no other comments. In written comments provided by its Director of the Departmental GAO-Office of Inspector General Liaison Office (reproduced in appendix III), DHS concurred with both of our recommendations and identified actions it has taken or plans to take to implement our recommendations. For example, the department stated that it plans to work with OMB to include metrics specific to periodic assessments of risk and development of subordinate security plans, as well as to provide OMB with recommendations for metrics that inspectors general can use that focus on measuring the effectiveness of agency information security programs. According to DHS, these actions should be completed by the end of fiscal year 2014.

The audit liaison for NIST, within the Department of Commerce, provided technical comments via e-mail on September 4, 2013, and we incorporated them where appropriate. The audit liaisons for the Departments of Education, Energy, and Transportation; the Environmental Protection Agency; and the Small Business Administration responded via e-mail that the agencies did not have any comments.

We are sending copies of this report to the Director of the Office of Management and Budget, the Secretary of Homeland Security, and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov.

If you have any questions regarding this report, please contact me at (202) 512-6244 or wilshuseng@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objective, Scope, and Methodology

Our objective was to evaluate the extent to which the requirements of the Federal Information Security Management Act (FISMA) have been implemented, including the adequacy and effectiveness of agency information security policies and practices.

We reviewed and analyzed the provisions of the act to identify agency, Office of Management and Budget (OMB), and National Institute of Standards and Technology (NIST) responsibilities for implementing, overseeing, and providing guidance for agency information security to evaluate federal agencies’ implementation of FISMA requirements. To assist in assessing the adequacy and effectiveness of agencies’ information security policies and practices, we reviewed and analyzed FISMA data submissions and annual FISMA reports, as well as information security-related reports for each of the 24 major federal agencies based on work conducted in fiscal years 2011 and 2012 by us, agencies, and inspectors general. We reviewed and summarized weaknesses identified in those reports using FISMA requirements as well as the security control areas defined in our Federal Information System Controls Audit Manual. Additionally, we analyzed, categorized, and summarized chief information officer and inspector general annual FISMA data submissions for fiscal years 2011 and 2012. Further, we compared weaknesses identified by inspectors general to the inspector general responses to the Department of Homeland Security (DHS)-defined metrics on the effectiveness of agency controls.

To assess the reliability of the agency-submitted data we obtained via CyberScope, we reviewed supporting documentation that agencies provided to corroborate the data. We also conducted an assessment of the CyberScope application to gain an understanding of the data required, related internal controls, missing data, outliers, and obvious errors in submissions. We also reviewed a related DHS inspector general report that discussed its evaluation of the internal controls of CyberScope. In addition, we selected 6 agencies to gain an understanding of the quality of processes in place to produce annual FISMA reports. To select these agencies, we sorted the 24 major agencies from highest to lowest using the total number of systems the agencies reported in fiscal year 2011; separated them into even categories of large, medium, and small agencies; then selected the median 2 agencies from each category. These agencies were the Departments of Education, Energy, Homeland Security, and Transportation; the Environmental Protection Agency; and the Small Business Administration. We conducted interviews and collected data from the inspectors general and agency officials from the selected agencies to determine their process to ensure the reliability of data submissions. Based on this assessment, we determined that the data were sufficiently reliable for our work.

We also examined OMB and DHS FISMA reporting instructions and other guidance related to FISMA to determine the steps taken to evaluate the adequacy and effectiveness of agency information security programs. In addition, we interviewed officials from OMB, DHS’s Federal Network Resilience Division, and NIST. We did not evaluate the implementation of DHS’s FISMA-related responsibilities assigned to it by OMB.

We conducted this performance audit from February 2013 to September 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Agency Responsibilities under FISMA

FISMA assigns a variety of responsibilities for federal information security to OMB, agencies, inspectors general, and NIST, which are described below.

FISMA states that the Director of the Office of Management and Budget (OMB) shall oversee agency information security policies and practices, including: developing and overseeing the implementation of policies, principles, standards, and guidelines on information security; requiring agencies to identify and provide information security protections commensurate with risk and magnitude of the harm resulting from the unauthorized access, use, disclosure, disruption, modification, or destruction of information collected or maintained by or on behalf of an agency, or information systems used or operated by an agency, or by a contractor of an agency, or other organization on behalf of an agency; overseeing agency compliance with FISMA; and reviewing at least annually and approving or disapproving, agency information security programs.

FISMA also requires OMB to report to Congress no later than March 1 of each year on agency compliance with the requirements of the act.

FISMA requires each agency, including agencies with national security systems, to develop, document, and implement an agency-wide information security program to provide security for the information and information systems that support the operations and assets of the agency, including those provided or managed by another agency, contractor, or other source.

Specifically, FISMA requires information security programs to include, among other things: periodic assessments of the risk and magnitude of harm that could result from the unauthorized access, use, disclosure, disruption, modification, or destruction of information or information systems; risk-based policies and procedures that cost-effectively reduce information security risks to an acceptable level and ensure that information security is addressed throughout the life cycle of each information system; subordinate plans for providing adequate information security for networks, facilities, and systems or groups of information systems, as appropriate; security awareness training for agency personnel, including contractors and other users of information systems that support the operations and assets of the agency; periodic testing and evaluation of the effectiveness of information security policies, procedures, and practices, performed with a frequency depending on risk, but no less than annually, and that includes testing of management, operational, and technical controls for every system identified in the agency’s required inventory of major information systems; a process for planning, implementing, evaluating, and documenting remedial actions to address any deficiencies in the information security policies, procedures, and practices of the agency; procedures for detecting, reporting, and responding to security plans and procedures to ensure continuity of operations for information systems that support the operations and assets of the agency.

In addition, agencies must produce an annually updated inventory of major information systems (including major national security systems) operated by the agency or under its control, which includes an identification of the interfaces between each system and all other systems or networks, including those not operated by or under the control of the agency.

FISMA also requires each agency to report annually to OMB, selected congressional committees, and the Comptroller General on the adequacy of its information security policies, procedures, practices, and compliance with requirements. In addition, agency heads are required to report annually the results of their independent evaluations to OMB, except to the extent that an evaluation pertains to a national security system; then only a summary and assessment of that portion of the evaluation needs to be reported to OMB.

Under FISMA, the inspector general for each agency shall perform an independent annual evaluation of the agency’s information security program and practices to determine the effectiveness of such program and practices. The evaluation should include testing of the effectiveness of information security policies, procedures, and practices of a representative subset of agency systems. In addition, the evaluation must include an assessment of the compliance with the act and any related information security policies, procedures, standards, and guidelines. For agencies without an inspector general, evaluations of non-national security systems must be performed by an independent external auditor. Evaluations related to national security systems are to be performed by an entity designated by the agency head.

Under FISMA, the National Institute of Standards and Technology (NIST) is tasked with developing, for systems other than for national security, standards and guidelines that must include, at a minimum: (1) standards to be used by all agencies to categorize all their information and information systems based on the objectives of providing appropriate levels of information security according to a range of risk levels; (2) guidelines recommending the types of information and information systems to be included in each category; and (3) minimum information security requirements for information and information systems in each category. NIST must also develop a definition of and guidelines for detection and handling of information security incidents.

The law also assigns other information security functions to NIST including: providing technical assistance to agencies on elements such as compliance with the standards and guidelines, and the detection and handling of information security incidents; evaluating private-sector information security policies and practices and commercially available information technologies to assess potential application by agencies; evaluating security policies and practices developed for national security systems to assess their potential application by agencies; and conducting research, as needed, to determine the nature and extent of information security vulnerabilities and techniques for providing cost-effective information security.

In addition, FISMA requires NIST to prepare an annual report on activities undertaken during the previous year, and planned for the coming year, to carry out responsibilities under the act.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Anjalique Lawrence (assistant director), Cortland Bradford, Wil Holloway, Nicole Jarvis, Linda Kochersberger, Lee McCracken, Zsaroq Powe, David Plocher, Jena Sinkfield, Daniel Swartz, and Shaunyce Wallace made key contributions to this report.