Information Security: Agencies Need to Develop and Implement Adequate Policies for Periodic Testing
Published: 2006-10-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Agencies rely extensively on computerized information systems and electronic data to carry out their missions. To ensure the security of the information and information systems that support critical operations and infrastructure, federal law and policy require agencies to periodically test and evaluate the effectiveness of their information security controls at least annually. GAO was asked to evaluate the extent to which agencies have adequately designed and effectively implemented policies for testing and evaluating their information security controls. GAO surveyed 24 major federal agencies and analyzed their policies to determine whether the policies address important elements for periodic testing. GAO also examined testing documentation at 6 agencies to assess the quality and effectiveness of testing on 30 systems.
section_title: What GAO Found
paragraphs: Federal agencies have not adequately designed and effectively implemented policies for periodically testing and evaluating information security controls. Agencies' policies often did not include important elements for performing effective testing. For example, none of the agencies' policies addressed how to determine the depth and breadth of testing according to risk. Also, agencies did not always address other important elements, including the identification and testing of security controls common to multiple systems, the definition of roles and responsibilities of personnel performing tests, and the frequency of periodic testing. The six case study agencies did not effectively implement policies for periodically testing and evaluating information security controls for the 30 systems reviewed. The methods and practices for testing and evaluating controls at the six agencies were not adequate to ensure that assessments were consistent, of similar quality, and repeatable. For example, these agencies did not always sufficiently document their test methods and results, did not define the assessment methods to be used when evaluating security controls, did not test security controls as prescribed, and did not include previously reported remedial actions or weaknesses in their test plans to ensure they had been addressed. As a result, agencies may not have reasonable assurance that controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements of the agency. In addition, agencies may not be fully aware of the security control weaknesses in their systems, thereby leaving the agencies' information and systems vulnerable to attack or compromise.

section_title: Letter
paragraphs: Federal agencies rely extensively on computerized information systems  and electronic data to carry out their missions. The security of these  systems and data is essential to prevent data tampering, disruptions in  critical operations, fraud, and inappropriate disclosure of sensitive  information. Concerned with accounts of attacks on systems through the  Internet and reports of significant weaknesses in federal computer  systems that make them vulnerable to attack, Congress passed the Federal  Information Security Management Act (FISMA) in 2002.
Among other things, FISMA requires federal agencies to periodically test  and evaluate the effectiveness of their information security policies,  procedures, and practices as part of developing and implementing an  agencywide information security program. In addition, agencies and their  Inspectors General are required to annually report to Congress and the  Office of Management and Budget (OMB) on the adequacy and  effectiveness of information security policies and practices and  compliance with the act. The act also assigns specific responsibilities to  OMB and the National Institute of Standards and Technology (NIST).  OMB’s responsibilities include (1) developing and overseeing the  implementation of policies, principles, standards, and guidelines on  information security and (2) reporting to Congress on the agencies’  compliance with FISMA requirements. OMB also provides instructions to  agencies and Inspectors General to assist them in meeting FISMA  reporting requirements. These instructions have a strong focus on  performance measures, which are the basis of agencies’ annual reports  and Inspectors General independent annual evaluations. The act requires  NIST to develop, for systems other than national security systems,  standards and guidelines to assist agencies in implementing their  information security programs.
As agreed with your office, our objective was to determine whether  agencies have adequately designed and effectively implemented policies  for periodically testing and evaluating information security controls. To  accomplish this objective, we conducted a survey of 24 major federal  agencies and their Inspectors General, analyzed information security  policies, and selected 6 of the 24 agencies to use as case studies for  conducting in-depth evaluations of their periodic testing and evaluation  methods and practices. Specifically, to determine whether the 24 agencies  adequately designed policies for periodic testing, we obtained and  analyzed their policies to determine whether they included elements  important for conducting effective tests and evaluations. To determine  whether the 6 agencies had effectively implemented policies and  procedures, we assessed methods and practices used to test and evaluate  controls for 30 of their systems. We examined instructions, standards, and  guidelines issued by OMB and NIST as a framework for assessing the  adequacy of the 24 agencies’ policies and for determining the effectiveness  of the 6 agencies’ testing and evaluation methods and practices. Details of  our objective, scope, and methodology are included in appendix I.
We conducted our work from November 2005 through July 2006 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Increasing computer interconnectivity—most notably growth in the use of  the Internet—has revolutionized the way that our government, our nation,  and much of the world communicate and conduct business. While this  interconnectivity offers us huge benefits, without proper safeguards, it  also poses significant risks to the government’s computer systems and,  more importantly, to the critical operations and infrastructures they  support. We reported in 2005 that while federal agencies showed  improvement in addressing information security, they have also continued  to have significant control weaknesses in federal computer systems, which  puts federal assets at risk of inadvertent or deliberate misuse, financial  information at risk of unauthorized modification or destruction, sensitive  information at risk of inappropriate disclosure, and critical operations at  risk of disruption.
subsections: section_title: Federal Law and Policy Establish Federal Information Security Testing Requirements
paragraphs: The Federal Information Security Management Act of 2002 requires each  agency to develop, document, and implement an agencywide information  security program. This program should provide security for the  information and information systems that support the operations and  assets of the agency, including those provided or managed by another  agency, contractor, or other source. Among other things, the program is to  include periodic testing and evaluation of the effectiveness of information  security policies, procedures, and practices, to be performed with a  frequency depending on risk, but no less than annually. The testing is to  include management, operational, and technical controls for every system  identified in the agency’s required inventory of major information systems.
The act also assigns specific responsibilities to OMB and NIST. OMB’s  responsibilities include the following:    Overseeing agency information security policies and practices, including  developing and overseeing the implementation of policies, principles,  standards, and guidelines on information security.
Reviewing agency information security programs, at least annually.
Reporting to Congress annually on agency compliance with FISMA  requirements.
As part of the reporting process, OMB provides instructions to agencies  and their Inspectors General on the annual FISMA reporting requirements.  These instructions include performance measures for such things as the  number of systems for which security controls have been tested and  evaluated in the past year. OMB also uses performance measures to assist  in its oversight responsibilities and to annually report to Congress on  agencies’ compliance with the requirements of the act.
FISMA also directs NIST to develop standards and guidelines for systems  other than national security systems. These standards and guidelines  instruct agencies on providing an acceptable level of information security  for all agency operations and assets and contribute to the testing and  evaluation of information security controls within an agencywide  information security program. Recognizing the importance of  documenting standards and guidelines as part of an agencywide  information security program, NIST emphasizes that agencies must  develop and promulgate formal, documented policies and procedures in  order to ensure the effective implementation of security requirements.
NIST standards and guidelines that contain elements applicable to  periodic testing and evaluation include the following:    Special Publication 800-26, Security Self-Assessment Guide for  Information Technology Systems, November 2001. This publication is a  self-assessment guide for agencies to use in determining the current status  of their information security program. The guide includes a standardized  form for reporting the results of system-level assessments and a method  for evaluating the effectiveness of the agency’s information security  program. The guide also emphasizes the importance of establishing levels  of implementation, referred to as the IT security assessment framework.  NIST Special Publication 800-26 is effective through the 2006 FISMA  reporting period and will be rescinded when Special Publications 800-53A  and 800-100 are finalized.
Special Publication 800-37, Guide for the Security Certification and  Accreditation of Federal Information Systems, May 2004. This guide is to  be used for certifying and accrediting nonnational security systems.  Developed as part of NIST’s project to promote the development of  standards and guidelines to support FISMA, this guide specifies the need  for ongoing activities to continuously monitor the effectiveness of security  controls.
Special Publication 800-53, Recommended Security Controls for Federal  Information Systems, February 2005. This publication provides  instructions on selecting and specifying security controls for information  systems. It also provides the set of security controls that satisfy the depth  and breadth of security requirements levied on information systems and  provides the fundamental concepts associated with security controls  selection and specification, including the identification and use of  common security controls. In conducting security assessments, NIST  states that assessment results can be used and shared to enhance the  efficiency of evaluations and reduce security program costs.
Special Publication 800-53A, Guide for Assessing the Security Controls in  Federal Information Systems, April 2006. The publication is a second  public draft to be used by agencies to assess the effectiveness of security  controls employed in federal information systems. NIST establishes  methods and procedures to assess the security controls in federal  information systems, specifically those controls listed in NIST Special  Publication 800-53, Recommended Security Controls for Federal  Information Systems. These methods and procedures are designed for  agencies to use in determining if the controls are implemented correctly,  operating as intended, and producing the desired outcome with respect to  meeting the security requirements of the agency. NIST closed acceptance  of public comments on this draft on July 31, 2006, and plans to issue a final  publication in December 2006.
section_title: Elements Important for Performing Effective Testing and Evaluation
paragraphs: Having well-designed policies is critical for performing effective testing  and evaluation of security controls. To assist agencies, OMB and NIST  developed instructions, standards, and guidelines for testing and  evaluating the controls over information systems. We used the following  six elements to evaluate agencies’ policies for periodically testing security  controls:  1.  Identifying the frequency of periodic testing.  2.  Defining roles and responsibilities of personnel performing the testing.  3.  Selecting a minimum set of security controls evaluated during periodic  tests.  4.  Identifying and testing common security controls.  5.  Determining the depth and breadth of periodic testing.  6.  Including assessment results in remediation plans.
The related federal and NIST references are shown in table 1.
section_title: Agencies’ Policies Do Not Fully Address Elements Important for Effective Testing and Evaluation
paragraphs: Agencies’ policies for periodically testing and evaluating security controls  have not been adequately designed and effectively implemented.  Specifically, none of the federal agencies’ policies fully addressed six  important elements included in OMB and NIST guidelines and standards  for performing effective security testing and evaluations. In addition, there  were weaknesses in the security control assessments for the 30 systems  reviewed at the six case study agencies. As a result, agencies have limited  assurance that controls are implemented correctly, operating as intended,  and producing the desired outcome. In addition, agencies may not be fully  aware of security control weaknesses in their systems, thereby leaving the  agencies’ operations and systems at risk.
subsections: section_title: Agencies’ Policies Have Design Weaknesses
paragraphs: Agencies did not fully address six elements important for testing and  evaluating security controls in their policies. Specifically, the (1) frequency  of periodic testing was not always identified, (2) roles and responsibilities  of personnel performing tests often were not clearly defined, (3) selection  of a minimum set of security controls evaluated during periodic tests was  not always fully addressed, (4) instructions on identification and testing of  common security controls were not addressed, (5) instructions on  determining the depth and breadth of testing were not included, and (6)  descriptions of a process for documenting remedial actions to address  deficiencies were not always addressed. Table 2 indicates weaknesses in  developing and promulgating formal, documented policies to address the  security elements needed for effective testing.
FISMA requires agencies to perform—for all major information systems in  their inventory—periodic testing and evaluation of the effectiveness of  information security policies, procedures, and practices, to be performed  with a frequency depending on risk, but no less than annually.
Of the 23 agencies’ policies we reviewed, 7 agencies did not require that  their security controls (management, operational, and technical) be tested  and evaluated at least annually. For example, policies for 3 of the 7  agencies did not specify the frequency of periodic testing. The other 4  agencies identified the frequency of some testing activities—reviewing the  overall security program annually, testing standard user account  procedures annually, and certifying and accrediting systems at least every  3 years—but did not specify the frequency of periodic testing for other  management, operational, and technical security controls. Unless agencies  specify the frequency for conducting periodic testing and evaluations at  least annually per FISMA, they may not have assurance that controls are  being sufficiently evaluated and producing the desired outcome with  respect to meeting the security requirements of the agency.
subsections: section_title: Policies Did Not Clearly Define Roles and Responsibilities for Periodic Testing
paragraphs: NIST 800-37 identifies the roles and associated responsibilities with regard  to testing and evaluating information security controls. These roles  include the chief information officer, authorizing official, senior agency  information security officer, information system owner, and information  system security officer. In addition, NIST Special Publication 800-26  specifies that agencies should have procedures in place that identify who  is conducting the security testing.
Roles and responsibilities of personnel performing testing were not clearly  defined in policies for 15 of the 23 agencies. Ten of the 15 agencies did not  define roles and responsibilities for personnel performing tests in their  policies and the other 5 agencies defined them only partially. For example,  one agency defined roles and responsibilities for the system owner but not  for other key security personnel such as the chief information security  officer and information system security officer. As a result, agency  officials may not clearly understand their expected responsibilities and  consequently, may not be able to carry out their duties correctly and  effectively.
section_title: Policies Lacked Adequate Instructions for Selecting Minimum Controls Evaluated during Periodic Testing
paragraphs: Baseline controls are the minimum security controls recommended for an  information system based on the system’s security categorization. NIST  Special Publication 800-53 provides guidance to agencies for selecting  these security controls, which serve as a starting point in determining and  designing methods for testing the security controls. NIST specifies that  agency security personnel must develop, document, and implement  policies for consistent identification, testing, and evaluation of baseline  controls.
Policies for selecting the minimum security controls evaluated during  periodic tests for 11 of the 23 agencies were not always adequate. To  illustrate, 7 of the 11 agencies reported having no specific policies or  procedures for selecting the minimum baseline security controls, and the  other 4 agencies’ policies partially addressed the selection of these  controls. For example, one agency’s policy referenced NIST guidance for  identifying controls, but it did not first specify the use of the NIST  standard when determining the system’s impact level. In another example,  an agency referenced NIST 800-53 guidance for selecting baseline controls,  but it provided a checklist of controls to be tested that did not include the  baseline controls as identified in NIST guidance. Without adequate  instruction, security personnel may not consistently identify, test, and  evaluate the baseline controls used to secure their systems.
section_title: Policies Did Not Specify How to Identify and Test Common Controls
paragraphs: Identifying common security controls can increase efficiency in agencies’  periodic testing. NIST 800-37 guidance defines a common security control  as one that can be applied to one or more of an agency’s information  systems. This guidance suggests that many of the management and  operational controls—contingency planning, incident response, security  training and awareness, personnel security, and physical security—needed  to protect an information system may be excellent candidates for common  security control status. By identifying common controls, agencies can  achieve efficiencies by testing common controls and using the results for  multiple systems. For example, NIST states that an organizationwide  approach to reusing and sharing test results can greatly enhance  efficiencies and significantly reduce security program costs.
Policies for 22 of the 23 agencies we reviewed did not specify how to  identify and test common security controls. For example, the security  policies for 15 of the 22 agencies did not address the identification and  testing of common security controls and policies and the other 7 agencies  only partially addressed them. Specifically, the 7 agencies identified and  tested some elements of common controls, but their policies did not  describe how to identify, test, or share testing results with others. For  example, one agency encouraged the use of common controls, but it did  not specify how common controls were to be identified, how to test them,  or how test results should be shared with others. In addition, another  agency made reference to common controls as part of a pilot program, but  no other discussion or reference was made regarding identifying and  testing common security controls. Without policies and procedures that  address or provide guidance for identifying and testing common controls,  agencies may needlessly test common controls multiple times, thereby  reducing efficiency and increasing costs for their periodic testing.
section_title: Policies Lacked Adequate Instructions for Determining the Depth and Breadth of Testing
paragraphs: An important element of efficient and effective testing is the consideration  of the depth and breadth of agency testing. FISMA requires testing of the  management, operational, and technical controls for every system at least  annually. Moreover, Special Publication 800-37 states that it is not feasible  or cost effective to monitor all of the security controls in an information  system on a continuous basis and that the information system owner  should select an appropriate subset of those controls for periodic  assessment. In addition, OMB Memoranda M-05-15 and M-06-20 have  identified three criteria for agency officials to consider when determining  the depth and breadth of a review:    The potential risk and magnitude of harm to the system or data.
The relative comprehensiveness of the past year’s review.
The adequacy and successful implementation of a remediation plan to  address weaknesses in the information system.
None of the 23 agencies’ policies provided adequate instruction for  determining the depth and breadth of periodic tests. Moreover, agencies  did not incorporate the three OMB criteria into their policies as  consideration for determining the depth and breadth of periodic testing.  Security personnel reported that they do not fully understand how to apply  the current guidance on determining the depth and breadth of controls  testing and need further clarification. Until additional guidance clarifies  how to determine the depth and breadth of testing, increased risk exists  that agencies may not sufficiently test security controls in a cost-effective  manner.
section_title: Policies Did Not Always Describe a Process for Documenting Weaknesses in Remediation Plans
paragraphs: FISMA directs agencies to establish a process for remediating identified  weaknesses in their information security policies and procedures. Key to  an effective remediation plan is the accurate and complete inclusion of  weaknesses identified during periodic testing. Remediation plans, also  referred to as plans of action and milestones, should list all identified  weaknesses and show estimated resource needs or other challenges to  resolving them, key milestones and completion dates, and the status of  corrective actions. NIST 800-37 states that remediation plans need to be  updated to address weaknesses identified as a result of periodic testing.
Policies for 10 of the 23 agencies did not fully describe a process for  documenting identified control weaknesses. For example, 7 of the 10  agencies did not have policies that described a process for incorporating  weaknesses identified during periodic security testing into remediation  plans. The remaining 3 agencies had policies on remediation plans, but  these were in draft form only and provided no further description of the  process for addressing weaknesses. Without adequate guidance for  ensuring that identified weaknesses are incorporated into remediation  plans, there is increased risk that weaknesses identified through security  controls testing are not being properly addressed. Thus, agencies may not  realize the full benefits of such testing and have limited assurance that the  controls for their systems are functioning effectively.
section_title: Six Case Study Agencies Did Not Effectively Implement Policies
paragraphs: None of the six case study agencies fully implemented their policies for  periodic information security testing. During our review of 30 systems, we  found implementation weaknesses at all six agencies. These weaknesses  consisted of insufficient testing documentation, inadequately defined  assessment methods, inadequate security testing, and lack of remedial  actions included in testing plans, as shown in table 3.
Testing documentation and supporting material serves as the basis for  verifying that the security controls in the information system are  implemented correctly, operating as intended, and producing the desired  outcome with respect to meeting the security requirements for the  information system. Test documents may include risk assessments, testing  plans, the controls being tested, the results of the testing (security  weaknesses and vulnerabilities), including results from previous security  assessments, security reviews, or audits. Support materials may include  procedures, reports, logs, and records showing evidence of security  controls implementation.
Agencies did not sufficiently document periodic testing activities and  results for 28 of the 30 systems reviewed. These examples ranged from no  documentation to documentation that omitted key elements, such as risk  assessments, testing plans, and test results. For example, testing plans did  not provide enough detail to determine which tests were to be conducted  or the scope of test coverage. In addition, one security manager reported  that maintaining supporting documentation was not a common practice  and that no supporting documentation or test records had been  maintained until recently. Unless agencies develop and maintain sufficient  testing documentation, they will have limited evidence for making  judgments about the security of their systems.
subsections: section_title: Agencies Did Not Always Define Assessment Methods
paragraphs: NIST 800-37 identifies a variety of assessment methods such as  interviewing, inspecting, studying, testing, demonstrating, and analyzing  that agencies can use when evaluating their security controls. NIST  guidelines describe these methods as interview, examine, and test.
The interview method of assessment is the process of conducting focused  discussions with individuals or groups of individuals within an  organization to facilitate assessor understanding, achieve clarification, or  obtain evidence.
The examine method of assessment is the process of reviewing,  inspecting, observing, studying, or analyzing one or more assessment  objects (specifications, mechanisms, or activities). Similar to the interview  method, the primary purpose of the examine method is to facilitate  assessor understanding, achieve clarification, or obtain evidence.
The test method of assessment is the process of exercising one or more  assessment objects (limited to mechanisms or activities) under specified  conditions to compare actual with expected behavior. NIST states that the  results of assessments using these methods are to support the  determination of overall security controls effectiveness.
Agencies did not fully define the assessment methods used to evaluate  their system controls for 7 of the 30 systems reviewed. We found that the  test plans, procedures, and testing results for 4 of the 7 systems did not  identify how agencies evaluated system controls or whether they used  interviews, examinations, or tests to determine the effectiveness of those  controls. For the 3 remaining systems, agencies did not provide  documentation to show what assessment methods were used. If agencies  do not define assessment methods, they may not have information  describing how that control was assessed. Without that information,  agencies have limited assurance that those controls are being effectively  tested or implemented.
section_title: Agencies Did Not Always Adequately Test Security Controls
paragraphs: Once employed within an information system, security controls should be  tested to determine the extent to which the controls are correctly  implemented, operating as intended, and producing the desired outcome  with respect to meeting the security requirements for the system. NIST  states that assessments should be based on an examination of relevant  documentation and a rigorous examination and testing of the controls. The  results of security testing contribute to the knowledge base of organization  officials with regard to the security status of the information system and  the overall risk to the operations and assets of the organization incurred  by the operation of the system.
Agencies did not adequately test security controls for 24 of the 30 systems  reviewed. The testing documentation showed no evidence of how testers  assessed the security controls, whether they had tested the control as  planned, or if they had conducted the test in accordance with the plan. In  one example, testers reviewed management control policies; however, the  testing guidelines required that the control be tested to determine if it had  been effectively implemented. Unless agencies adequately test controls  and document the results, they may not be able to measure the security  status of their information systems, thereby limiting their ability to know  whether controls are protecting their operations and assets.
section_title: Agencies Did Not Include Remedial Actions in Testing Plans
paragraphs: FISMA requires that agencies document remedial actions that address  deficiencies in the information security policies, procedures, and  practices. NIST 800-37 states that the plan of action and milestones should  describe the measures that have been implemented or planned to correct  any deficiencies or weaknesses noted during the assessment of the  security controls. NIST also states that remedial actions should be  evaluated to determine if they effectively mitigate previously identified  weaknesses or vulnerabilities in the information system.
For 18 of the 30 systems, agencies did not consistently test or evaluate the  effectiveness of remedial actions for weaknesses identified through  security control assessments. For example, testing documentation for  some systems did not address the remedial actions that agencies had  identified from prior assessments in their test plans. Unless agencies  document and include remedial actions for previously identified control  weaknesses in testing plans, agencies will have limited assurance that  weaknesses have been corrected.
section_title: Conclusions
paragraphs: Agencies have not adequately designed and effectively implemented  policies for periodically testing information security controls. While  almost all agencies had documented policies for security testing, the  policies did not always adequately address elements important for  effective testing. Ensuring that agencies’ policies’ are sufficient to address  federal standards and guidelines helps to ensure their effective  implementation in meeting FISMA requirements. While NIST has issued  guidance on how agencies should apply the depth and breadth method for  testing security controls, agencies have not been documenting or  implementing this approach in their testing. Also, agency officials reported  that they did not understand this method.
Our review of 30 systems at six major federal agencies found weaknesses  in testing practices and methods: documentation, testing methods,  controls testing, and remedial actions in testing plans. Conducting  effective periodic testing and evaluations of information security controls  is a serious, pervasive, and crosscutting challenge to federal agencies,  warranting increased attention from OMB. If these challenges are not  addressed, federal agencies’ information and operations may be at  increased risk.
section_title: Recommendations for Executive Action
paragraphs: Because of the governmentwide weaknesses in the design and  implementation of agencies’ policies for periodically testing and evaluating  security controls, we recommend that the Director of the Office of  Management and Budget take the following two actions:  Instruct federal agencies to develop and implement policies on periodic  testing and evaluation.
Revise instructions for future FISMA reporting by requesting Inspectors  General to report on the quality of agencies’ periodic testing processes.
We also recommend that the Secretary of Commerce direct the Director,  National Institute of Standards and Technology, to strengthen guidance on  determining the depth and breadth of testing security controls.
section_title: Agency Comments
paragraphs: We received oral comments on a draft of this report from representatives  of the Office of Management and Budget’s Offices of Information and  Regulatory Affairs and General Counsel. The representatives agreed to  consider our recommendations as part of their oversight responsibilities  for information security at federal agencies. The Deputy Secretary of the  Department of Commerce provided written comments in response to our  draft report (see app. II).  He stated that the department agreed with our  characterization of the National Institute of Standards and Technology’s  FISMA responsibilities and activities and also said that NIST is currently  reviewing its guidance, including that for the depth and breadth of testing  security controls.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from  the date of this letter. At that time, we will send copies of the report to  other interested congressional committees; the Director, Office of  Management and Budget; and the Deputy Secretary of the Department of  Commerce. We will make copies available to others on request. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you have any questions about this report, please contact me at (202) 512- 6244 or wilshuseng@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are acknowledged in  appendix V.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: The objective of our review was to determine the extent to which federal  agencies have adequately designed and effectively implemented policies  for periodically testing and evaluating security controls. The scope of our  review included (1) the 24 federal agencies, focusing on reviewing their  policies and procedures and responses to our survey and (2) a selection of  30 systems at 6 of these agencies, focusing on in-depth evaluations of their  periodic controls testing and evaluation practices and methods.
To determine the adequacy and effectiveness of federal agencies’ policies  and procedures for testing and evaluating security controls for their  information systems, we conducted a survey of the 24 major agencies,  which included 21 questions for the agencies and 4 questions for the  agencies’ Inspectors General. We also reviewed the agencies’ policies that  were submitted in response to the surveys and compared them against six  policy elements from the Office of Management and Budget (OMB) and  the National Institute of Standards and Technology (NIST) standards and  guidelines that we considered to be important for performing effective  testing. The survey instruments were pretested with two federal  information technology organizations—the Department of Defense and  GAO’s Office of the Chief Information Officer.
To assess the implementation of Federal Information Security  Management Act of 2002 (FISMA) requirements, we reviewed 30 systems  at the six case study agencies to determine whether policies for testing  and evaluating security controls were effectively implemented. We  selected for review the six agencies that reported the largest number of  systems in their inventories of major systems, excluding agencies that had  been recently reviewed by GAO.
We relied on FISMA standards and guidelines from OMB and NIST as  criteria for evaluating agency testing and evaluation methods, policies, and  procedures. These criteria were used to evaluate agency system  documentation on the results of security controls testing, such as system  security plans, testing results, testing plans and schedules, remedial action  plans, memoranda, and other artifacts used for information security  testing. We collected fiscal year 2005 self assessment and testing artifacts  and fiscal years 2004 and 2005 remediation plans in order to standardize  the data for analysis. To augment our work, we considered the responses  to our survey by the agencies and the Inspectors General.
We selected and examined 5 systems comprised of low, medium, and high  impact general support systems and major applications for a total of 30  systems across the six agencies. Because we were evaluating the extent to  which agencies periodically test and evaluate the effectiveness of security  controls, we avoided selecting systems that had recently undergone  certification and accreditation where more rigorous (independent) testing  is conducted. In cases where an agency had recently certified and  accredited the majority of its systems, we selected those having the oldest  accreditation date within the selected time period. We evaluated  government-owned and operated systems, and government-owned,  contractor-operated systems; all were operational and none were under  development. We did not select systems that were recently or currently  under review by an Inspector General or those classified as national  security or financial.
We performed our work in the Washington, D.C., metropolitan area and in  three agency field offices in Pennsylvania, Texas, and Georgia, from  November 2005 to July 2006, in accordance with generally accepted  government auditing standards.
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Suzanne Lightman (Assistant  Director), Ayannah Buford, Larry Crosland, Neil Doherty, Nicole Garofalo,  Nancy Glover, Joel Grossman, David Hong, John Ortiz, Jerome Sandau,  Donald Sebers, Jenniffer Wilson, and Charles Youman made key  contributions to this report.