Information Assurance: National Partnership Offers Benefits, but Faces Considerable Challenges
Released: 2006-03-24T00:00:00

Letter

Background

The growing sophistication and effectiveness of cyber attacks, and the increase of information assurance and information assurance-enabled information technology (IT) products available for use on national security systems, have heightened federal attention to the need for information assurance. As a result of these trends, acquiring commercial IT products that perform as vendors claim on national security systems has become a governmentwide challenge. While not a complete solution, an important way to increase confidence in commercial IT products is through independent testing and evaluation of their security features and functions during design and development.

Federal Partnership Formed to Promote the Use of Evaluated IT Products

In 1997, NIST and the National Security Agency collaborated to form the NIAP. The purpose of the partnership is to boost consumers’ and federal agencies’ confidence in information security products and enhance the ability of U.S. companies to gain international recognition and acceptance for their products. The five main goals of NIAP are to: promote the development and use of evaluated IT products and systems; champion the development and use of national and international standards for IT security; foster research and development in IT security requirements definition, test methods, tools, techniques, and assurance metrics; support a framework for international recognition and acceptance of IT security testing and evaluations; and facilitate development and growth of a commercial security testing industry within the U.S.

To facilitate achievement of these goals, NIAP developed a national program called the Common Criteria Evaluation and Validation Scheme. The program is based on an international standard of general concepts and principles of IT security evaluations for the international community. The program evaluates, through various evaluation assurance levels (see app. II), commercial-off-the-shelf information assurance and information assurance-enabled products for the federal government. These products can be items of hardware, software, or firmware. As part of the evaluation, agencies can specify a degree of confidence desired in a product through protection profiles. While a protection profile is not required in order to have a product evaluated, a vendor is required to develop a security target.

NIAP evaluations are performed by accredited Common Criteria testing laboratories. While a product is undergoing evaluation, the NIAP validation body—an activity currently managed by the National Security Agency—approves participation of security testing laboratories in accordance with accreditation policies and procedures. It also reviews the results of the security evaluations performed by the laboratories and issues a validation report, which summarizes and provides independent validation of the results. A product is considered NIAP-certified only after it is both evaluated by an accredited laboratory and validated by the validation body. Upon successful completion of these requirements, the validation body issues a Common Criteria certificate for the evaluated product. All evaluated products that receive a NIAP Common Criteria certificate appear on a validated products list available on NIAP’s Web site. According to the Committee on National Security Systems—a forum for the discussion of policy issues that sets federal policy and promulgates direction, operational procedures, and guidance for the security of national security systems—the fact that a product appears on the validated products list does not by itself mean that it is secure. A product’s listing on any Common Criteria validated products list means that the product was evaluated against its security claims and that it has met those claims. Figure 1 outlines the NIAP evaluation process.

In order to maintain the validity of an evaluation when a product upgrades to its next version, a vendor can request either a re-evaluation of the entire new product version or validation of only the changes in the product. To request the latter, a vendor must participate in the NIAP Assurance Maintenance Program. To participate in this program, a vendor must submit a request that addresses how it plans to maintain the product and a report of what will be maintained.

Vendors can select any one of the 10 accredited commercial testing laboratories to perform product evaluations. The vendor and testing laboratory negotiate evaluation costs, which can vary according to the laboratory and the assurance level the product is tested against (see fig. 2).

NVLAP identifies NVLAP-accredited laboratories on its Web site. Accreditation criteria are established in accordance with the U.S. Code of Federal Regulations (CFR, Title 15, Part 285), NVLAP Procedures and General Requirements, and encompass the requirements of ISO/IEC 17025 and the relevant requirements of ISO 9002. the scope of evaluation—the tendency of vendors to include elements in their security target that agencies may not require introduces additional costs; and the design of the product—if a product is designed so that its security functions are performed by a small number of modules, it may be possible to limit the portion of the product that must be examined.

Federal Policy Requires Evaluated Products for National Security Systems

In January 2000, as revised in June 2003, a federal policy was established that required the use of evaluated products for national security systems. Specifically, the Committee on National Security Systems established National Security Telecommunications and Information Systems Security Policy Number 11. The policy required, effective July 1, 2002, that all commercial-off-the-shelf information assurance and information assurance-enabled IT products acquired for use on national security systems be evaluated and validated in accordance with one of the following criteria: 1. The International Common Criteria for Information Security Technology Evaluation Recognition Arrangement,2. The NIAP Common Criteria Evaluation and Validation Scheme, 3. The NIST Federal Information Processing Standards Cryptographic Module Validation Program.

The objective of the policy is to ensure that these products, which are acquired by the federal government, undergo a standardized evaluation validating that a product either performs as its claims or meets the user’s security requirements. The policy requires that the evaluation and validation of such products be conducted by accredited commercial laboratories or by the National Security Agency for government off-the shelf products. It does not require mandatory compliance for information assurance products acquired prior to July 1, 2002, and includes a provision for deferred compliance, on a case-by-case basis, when information assurance-evaluated products do not cover the full range of potential user application, or do not incorporate the most current technology.

Moreover, while not a requirement, the federal policy includes provisions for departments and agencies who may wish to consider using the NIAP process for the acquisition and appropriate implementation of evaluated and validated products for non-national security systems.

NIAP Evaluation Process Contributes to System Security, but Is Not a Complete Solution

The use of commercial products that have been independently tested and evaluated is only a part of a security solution that contributes to the overall information assurance of a product. Other complementary controls are needed, including sound operating procedures, adequate information security training, overall system certification and accreditation, sound security policies, and well-designed system architectures. According to the Committee on National Security Systems, the protection of systems encompasses more than just acquiring the right product. The committee notes that once acquired, these products must be integrated properly and subjected to a system accreditation process, as discussed above, which will help to ensure the integrity of the information and systems to be protected.

For federal agencies, such an overall security solution is spelled out by the Federal Information Security Management Act. The act requires federal agencies to protect and maintain the confidentiality, integrity, and availability of their information and information systems. Among other things, the act requires each agency (including agencies with national security systems) to develop, document, and implement agencywide information security programs to provide information security for the information and information systems that support the operations and assets of the agency, including those provided or managed by another agency, contractor, or other source.

More specifically, the Federal Information Security Management Act stipulates that the head of each agency operating or exercising control of a national security system is responsible for providing information security protections commensurate with the risk and magnitude of harm that could result should a security breach occur. The act also stipulates that agency heads are responsible for implementing information security policies and practices as required by standards and guidelines for national security systems. The Department of Defense and the Director of Central Intelligence have authority under the act to develop policies, guidelines, and standards for national security systems.

The Federal Information Security Management Act also requires NIST, among other things, to provide technical assistance to agencies; to evaluate private sector security policies and practices; to evaluate commercially available IT, as well as practices developed for national security systems; and to assess the potential application by agencies to strengthen information security for non-national systems.

NIAP Offers Benefits for Use in National Security Systems, but Process Faces Considerable Challenges

While the NIAP evaluation process offers benefits to national security systems, its effectiveness has not been measured or documented, and considerable challenges to acquiring and using NIAP-evaluated products exist.

NIAP Evaluation Process Offers Benefits

NIAP process participants—vendors, laboratories, federal agencies, and NIAP officials—identified benefits to using the process for use in national security systems, including independent testing and evaluation of IT products and accreditation of the performing laboratories, which can give agencies confidence that the products will perform as claimed; international recognition of evaluated products, which provides agencies broader product selection and reduces vendor burden; discovery of software flaws in product security features and functions, which can cause vendors to fix them; and improvements to vendor development processes, which help to improve the overall quality of current and future products.

Independent Testing and Evaluation of Products and Accreditation of Laboratories Can Increase Product Assurance

Independent testing and evaluation of commercial IT products and accreditation of the laboratories that perform the test and evaluations can give agencies increased assurance that the products will perform as vendors claim. Independent testing is a best practice for assuring conformance to functional, performance, reliability, and interoperability specifications—especially for systems requiring elevated levels of security or trust. As discussed previously, NIAP requires vendors to obtain independent testing and evaluation of specific security features and functions that are built into their products. Agencies are able to use the results of validation reports to distinguish between competing products and thus make better-informed IT procurement decisions. Further, the Committee on National Security Systems encourages agencies to review the security target of a product and determine its appropriateness for the environment in which the product will operate.

In our survey, 15 of 18 federal agencies reported that they have derived benefits from acquiring and using products evaluated by the NIAP process. Of these 15 agencies, 11 reported that the availability of evaluated products helped the agency make IT procurement decisions; 9 reported that the process provided their agency with thorough and accurate product documentation; and 1 reported that evaluated products provided a common method of performing a particular security service that is implemented in different types of security or security-enabled devices, potentially resulting in a greater degree of standardization of elements (such as audit entries).

Moreover, the NIST-administered National Voluntary Laboratory Accreditation Program (NVLAP) reviews laboratories annually to ensure competence and compliance with standards. Accreditation is granted to laboratories following their successful completion of a process that includes an application submission and fee payment by the laboratory, an on-site assessment, participation in proficiency testing, resolution of any deficiencies identified during the process, and a technical evaluation. The issuance of a certificate formally signifies that a laboratory has demonstrated that it meets all NVLAP requirements and operates in accordance with management and the technical requirements of the relevant standards. However, the accreditation does not imply any guarantee of laboratory performance or test and calibration data; it is solely a finding of laboratory competence and compliance with standards. Figure 3 shows the laboratory accreditation process.

Another benefit of the NIAP evaluation process is NIAP’s membership in the Arrangement on the Recognition of Common Criteria Certificates in the Field of IT Security. As part of the goals of the arrangement, members can increase the availability of evaluated IT products and protection profiles for national use and eliminate duplicate evaluations of IT products and protection profiles, thus giving agencies a broader selection of evaluated products from which to choose. Agencies have the ability to acquire products that have been evaluated at evaluation assurance levels 1 through 4 from any of the countries that have an evaluation scheme. As of February 2006, there were 22 global signatories to the recognition arrangement, and 247 evaluated products available.

The recognition arrangement also reduces the burden on vendors by limiting the number of criteria to which their products must conform and the number of evaluations that a vendor needs to complete in order to sell a product internationally. Because NIAP evaluations (evaluation assurance levels 1-4) are accepted by the arrangement, vendors that go through the NIAP process can sell their evaluated products in any of the 22 member countries. Vendors are able to save time and money since they do not need to complete multiple evaluations to sell their product in different countries.

Product Evaluations Can Uncover Flaws and Cause Vendors to Fix Them

Another benefit of the NIAP process is that it uncovers flaws during product evaluations and can cause vendors to fix them. NIAP, vendor, and laboratory officials stated that the NIAP evaluation process has uncovered flaws and vulnerabilities in evaluated products. According to NIAP officials, software flaws are found in nearly all evaluated products, with an evaluation resulting in an average of two to three fixes. According to the four vendors included in our review, the NIAP evaluation process discovered flaws or vulnerabilities in their products or their product documentation. Also, officials from one of the laboratories included in our review stated that out of the 90 products they have evaluated, all of them had documentation flaws. Although vendors have the option of removing from the evaluation security features or functions in which flaws have been identified, any flaws in the remaining security features or functions must be fixed in order to successfully complete the product evaluation. Nonetheless, agencies procuring NIAP-evaluated products have a higher level of assurance that the product’s security features and functions will perform as claimed in the validation report.

Product Evaluations Can Result in Improvements to Vendors’ Development Processes

Product evaluations can influence vendors to make improvements to their development processes that raise the overall quality of their current and future products. To complete a successful evaluation, vendors submit to laboratories their development documentation, which describes various processes related to security, such as software configuration controls. Officials at six of the seven vendors we visited stated that product evaluations had a positive influence on their development process. According to one of the six vendors, completed product evaluations that result in improvements to their development process would likely transfer to the development process of other products and help improve the overall quality of their products. Laboratory officials also stated that NIAP evaluations often result in vendors improving their software development process because vendors adopt some of the methodologies used to pass evaluation, such as test methods and documentation, for their own quality assurance processes. Additionally, we previously reported that vendors who are proactive and adopt effective development processes and practices can drastically reduce the number of flaws in their products.

NIAP Evaluation Process Faces Challenges

NIAP process participants—NIAP officials and selected vendors, laboratories, and federal agencies—identified challenges to acquiring and using NIAP-evaluated products.

NIAP-evaluated products do not always meet agencies’ needs, which limit agencies’ acquisition and use of these products.

A lack of vendor awareness of the NIAP evaluation process impacts the timely completion of the evaluation and validation of products.

A reduction in the number of validators available to certify products could contribute to delays in validating products for agency use; and A lack of performance measures and difficulty in documenting the effectiveness of the NIAP process makes it difficult to demonstrate the program’s usefulness or improvements made to products’ security features and functions or improvements to vendors’ development processes.

Collectively, these challenges hinder the effective use of the NIAP evaluation process by vendors and agencies.

NIAP-Evaluated Products Do Not Always Meet Agencies’ Needs

Meeting agency needs for NIAP-evaluated products for use in national security systems can be a challenge. According to agency responses to our survey, 10 of 18 agencies that purchased NIAP-evaluated products reported experiencing challenges in acquiring those products. Specifically, 10 agencies noted that products on the NIAP-evaluated product list were not the most current versions; and 7 agencies noted that products needed by their agency were not included on the NIAP-evaluated product list.

Agencies also reported additional challenges for acquiring NIAP-- evaluated products. Specifically, choices for evaluated products are somewhat limited compared to the general product marketplace; and the length of time required for a product to complete the evaluation process can delay availability of the most up-to-date technology.

However, opportunities exist to better match agency needs with the availability of NIAP-evaluated products: Agencies can write protection profiles to define the exact security parameter specifications that they need. For example, two of the vendors we visited stated that they had their products evaluated against the Controlled Access Protection Profile, which provides agencies with a set of security functional and assurance requirements for their IT products and also provides a level of protection against threats of inadvertent or casual attempts to breach the system security.

Vendors can enter the evaluation process before their products are publicly released, which can allow consumers to acquire the most up-to- date technology. One vendor we visited had taken such a proactive approach.

Agencies can use the NIAP-validated products list to identify products that meet their needs. Because the number of available NIAP-evaluated products is increasing, agencies now have a variety of products from which to choose. In January 2002, there were about 20 evaluated products. As of February 2006, there were 127 evaluated products and 142 products in evaluation. These evaluated products span across 26 categories of information assurance products and information assurance-enabled products from which to choose, including operating systems and firewalls. As products continue to enter evaluation, agencies’ needs may be better met.

Vendors can, by participating in the NIAP Assurance Maintenance Program, maintain the validity of an evaluation when a product upgrades to its next version by either requesting a re-evaluation of the entire new product version or validation of only the changes in the product. Vendors’ participation in this program may allow agencies to have the most recent products available to them.

Agencies can increase their selection of products through the Common Criteria Recognition Arrangement—available on the Common Criteria portal Web site—which currently has 247 evaluated products available. The products listed on the Web site give agencies more choices of products evaluated at evaluation assurance levels 4 and below.

Lack of Vendors Awareness of NIAP Evaluation Process Affects Efficiency of Evaluations

Another challenge faced by the NIAP process is the lack of vendor awareness regarding the requirements of the evaluation process. For example, vendors who are new to the evaluation process are not aware of the extensive documentation requirements. Creating documentation to meet evaluation requirements can be an expensive and time-consuming process. According to laboratory officials, about six months is the average time for vendors to complete the required documentation before test and evaluation can begin. However, if vendors consistently maintain their documentation, subsequent evaluations can be faster and less expensive since the vendor has previously produced the documentation and is already familiar with the process.

Also, some vendors are not as active as others in the evaluation process, which can cause varying lengths of time for completing the evaluation. Vendors who are actively involved in the process are usually able to complete the process more quickly, including fixing flaws, than those who are not actively involved. According to one laboratory, the more active a vendor is in the evaluation process, the faster and less expensive it will be for the vendor. As such, the amount of involvement by the vendor during the process and the timeliness with which it fixes discovered flaws affects the length of time the product is in evaluation.

Furthermore, some vendors and laboratories do not have the same perception of the length of time required to perform the evaluation. According to laboratory officials, the length of time needed for conducting product evaluations varies depending on the type of product being evaluated and the evaluation assurance level (see fig. 4). Vendors are often not aware of these requirements and tend to underestimate the length of time required for evaluations.

Vendors and laboratories also perceive the length of evaluations differently because they punctuate start and end dates differently. Vendors measure the length of an evaluation from the day they decide to go into evaluation to the day they receive their product certificate. Their measurement includes selecting and negotiating with a laboratory, preparing required documentation, and testing the security features and functions. Laboratories, on the other hand, consider the length of an evaluation to be from the day they sign a contract with the vendor to the day they complete testing.

While Common Criteria user forums for program participants have been held, which NIAP participated in, NIAP itself has not developed education and training workshops that focus on educating participants on specific requirements—such as the documentation requirements. These workshops could help ensure that vendors and laboratories are aware of the NIAP process and could contribute to the efficiency of product evaluations. NIAP officials acknowledge that such educational offerings could be beneficial.

Reduction in the Number of Validators May Affect Timeliness of Certifications

Over the last year, NIAP has seen a reduction in the number of qualified validators. NIAP officials stated that one of the most significant challenges the NIAP process faces is hiring and maintaining qualified personnel to validate products. In fiscal year 2005, the NIAP program lost approximately four government validators and six contractor validators. According to the NIAP Director, maintaining qualified personnel to perform validation tasks is difficult largely because many validators are nearing retirement age and the job is not an attractive position for recent college graduates. Validators have a complex job with tasks that span the entire evaluation process; they incrementally review the results of the various tests of functional and assurance requirements as they are completed by the laboratory. As a result, once validators are hired, it typically takes 12 to 24 months to train new validators to become proficient in performing validation tasks.

If the NIAP program continues to see a reduction in validators, there could be an increased risk that a backlog of products needing to obtain NIAP certifications will develop, which could also impact the already lengthy evaluation process. The number of products entering evaluation is steadily increasing (in fiscal year 2002 there were approximately 20 products in evaluation and as of February 2006, there were 142 products in evaluation). Additionally, approximately five to seven products enter into evaluation each month. To address the widening gap between the number of products entering the process and the number of validators available to review products, NIAP intends to pursue legislation allowing it to recoup the costs of validations and hire additional staff.

Lack of Performance Measures and Difficulty in Documenting the Effectiveness of the NIAP Process

A best practice in public and private organizations is the use of performance measurements to gain insight into—and make adjustments to—the effectiveness and efficiency of programs, processes, and people. Performance measurement is a process of assessing progress toward achieving predetermined goals, and includes gathering information on the efficiency with which resources are transformed into goods and services, the quality of those outputs, and the effectiveness of government operations in terms of their specific contributions to program objectives. Establishing, updating, and collecting performance metrics to measure and track progress can assist organizations in determining whether they are fulfilling their vision and meeting their customer-focused strategic goals.

The NIAP program lacks performance metrics to measure process effectiveness and thus faces difficulty in documenting its effectiveness. The program has not collected and analyzed data on the findings, flaws, and fixes resulting from product tests and evaluations. NIAP officials pointed out that nondisclosure agreements between laboratories and vendors make it difficult to collect and document such data. According to NIAP officials, there is existing laboratory information on findings, flaws, and fixes, but it has not been collected because of nondisclosure agreements. Nondisclosure agreements are important for protecting vendors’ proprietary data from being released to the public and competitors. However, releasing summary laboratory information on findings, flaws and fixes, while at the same time considering the requirements of nondisclosure agreements, could be beneficial to determining the effectiveness of the NIAP program. Without this type of information, NIAP will have difficulty demonstrating its effectiveness and will be challenged to know and to demonstrate whether the process is meeting its goals.

Expanding NIAP Requirement to Non- national Security Systems May Yield Many of the Same Benefits and Challenges and Could Exacerbate Resource Constraints

While the National Security Telecommunications and Information Systems Security Policy Number 11 already allows agencies with non-national security systems to acquire NIAP-evaluated products, expanding the policy to mandate that such systems acquire NIAP-evaluated products may yield many of the same benefits and challenges experienced by current process participants, and could further exacerbate resources. For example, one identified benefit for national security systems—independent testing and evaluation of IT products—gives agencies confidence that validated features of a product, whether acquired for national or non-national security systems, will perform as claimed by the vendor. Similarly, one challenge—a reduction in the number of validators for certifying products—could contribute to delays in validating products, whether for national or non-nation security systems. Further, expanding the requirement to mandate the policy for non-national security systems may further exacerbate current resource constraints, related to hiring and maintaining qualified personnel to validate products.

Nevertheless, agencies with non-national security systems have in fact acquired NIAP-evaluated products. Specifically, ten of the federal agencies we surveyed indicated that they have used the NIAP process to acquire evaluated products for non-national security systems, even though they are not required to do so. One agency is considering the use of NIAP-evaluated products during its product reviews, and is also considering including NIAP-evaluated products as part of its procurement strategy.

Moreover, agencies seeking information assurance for their non-national security systems, but who do not acquire NIAP-evaluated products, have guidance and standards available to them. Specifically, as required by the Federal Information Security Management Act, NIST has developed and issued standards and guidelines, including minimum information security requirements, for the acquisition and use of security-related IT products for non-national security systems. These standards and guidelines are to be complementary with those established for the protection of national security systems and information contained in such systems. Further, NIST issued additional guidance to agencies for incorporating security into all phases of the system development life cycle process as a framework for selecting and acquiring cost-effective security controls. In August 2000, NIST also issued guidance on security assurance for non-national security systems in NIST Special Publication 800-23: Guideline to Federal Organizations on Security Assurance and Acquisition/Use of Tested/Evaluated Products.

Conclusions

While a range of controls are needed to protect national security systems against increasingly sophisticated cyber attacks, establishing effective policies and processes for acquiring products that have been validated by an independent party is important to the federal government’s ability to procure and deploy the right technologies. Acquiring NIAP-evaluated products can increase the federal government’s confidence that its IT products and systems will perform security features and functions as claimed. Despite the benefits of acquiring and using IT products that have gone through the rigorous tests and evaluations of NIAP, the program faces considerable challenges that hinder its effective use by vendors and agencies. These challenges include the difficulty in matching agencies’ needs with the availability of NIAP-evaluated products, vendors’ lack of awareness regarding the evaluation process, a reduction in the number of validators to certify products, and difficulty in measuring and documenting the effectiveness of the NIAP process. Until these challenges are addressed, they will continue to undermine the efficacy of NIAP. Regarding expanding the NIAP requirement to non-national security systems, pursing this approach may further exacerbate current resource constraints.

Recommendations for Executive Action

To assist the NIAP in documenting the effectiveness of the NIAP evaluation process, we recommend that the Secretary of Defense direct the Director of the National Security Agency, in coordination with NIST under the provisions of the NIAP partnership, to take the following two actions: 1. Coordinate with vendors, laboratories, and various industry associations that have knowledge of the evaluation process to develop awareness training workshops for program participants. 2. Consider collecting, analyzing, and reporting metrics on the effectiveness of NIAP tests and evaluations. Such metrics could include summary information on the number of findings, flaws, and associated fixes.

Agency Comments and Our Evaluation

In providing written comments on a draft of this report (reprinted in app. III), the Deputy Assistant Secretary of Defense (Deputy Chief Information Officer), partially agreed with one of our recommendations, agreed with the other, and described ongoing and planned efforts to address them. While the Deputy Assistant Secretary agreed with our recommendation to develop awareness training workshops for NIAP program participants, she stated that the NIAP must also live with the realities of the challenges that we identified in our report. The Deputy Assistant Secretary noted that, as our report highlights, the NIAP program is facing considerable challenges with resources and funding to sustain the current day-to-day running of the program and that it is not feasible for the NIAP office to increase its current efforts in developing and hosting the recommended training and education. Nonetheless, she also noted that the Secretary of Defense should direct the Director of the National Security Agency, in coordination with the NIST under the provisions of the NIAP, to coordinate with the vendors, laboratories, and various industry associations that have knowledge of the evaluation process to develop awareness training workshops for program participants within the current constraints and to work with the commercial laboratories, vendors, and others to identify ways that organizations outside of NIAP can further this initiative. We agree that NIAP should continue its efforts in awareness and education training, and endorse increasing such efforts as resources permit.

The Deputy Assistant Secretary agreed with our recommendation to collect, analyze, and report metrics on the effectiveness of NIAP tests and evaluations, and stated that the NIAP has already started researching ways to institute metrics to help determine the effectiveness of the evaluation program. She noted that the goal of collecting metrics is to demonstrate to the NIAP constituency that NIAP evaluations do provide value by improving the security of the evaluated products and by providing the end customer with assurance that these products perform their security functions as intended even when faced with adverse conditions.

The Department of Defense and the Department of Homeland Security also provided technical comments, which we considered and addressed in our report, as appropriate.

We are sending copies of this report to the Departments of Commerce (National Institute of Standards and Technology), Defense, and Homeland Security; the Office of Management and Budget; the General Services Administration, and to other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6244 or wilshuseng@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Objectives, Scope, and Methodology

Our objectives were to identify (1) the governmentwide benefits and challenges of the National Information Assurance Partnership (NIAP) evaluation process; and (2) the potential benefits and challenges of expanding the requirement of NIAP to non-national security systems, including sensitive but unclassified systems.

To determine the benefits and challenges for both objectives, we analyzed and reviewed a number of policy documents and reports from both industry and government. We also reviewed relevant federal policies relating to information security issues. To gain insight into the NIAP evaluation process, we met with software vendors and certification laboratories to discuss their experiences with NIAP, their applicable processes, and reviewed their relevant documentation. We selected vendors based on broad or distinguishing product capabilities demonstrating a range of features, brand recognition based on high ratings received in reviews conducted by information security magazines, and vendors mentioned more frequently in various discussions with industry experts and in information security literature. Vendors selected represented different information technology (IT) market sectors, are considered leaders in their field, and varied in size.

To determine the industrywide perspective on NIAP, we met with two IT industry groups: The Information Technology Association of America and Cyber Security Industry Alliance. We selected these industry groups because they represent a cross-section of the IT industry as a whole. To gain insight into the program’s functions and usefulness to agencies, we spoke with government officials from the Department of Commerce (specifically the National Institute of Standards and Technology), Department of Defense, Department of Homeland Security, General Services Administration, and the Office of Management and Budget. We also surveyed officials from the 24 federal agencies designated under the Chief Financial Officers Act of 1990 to determine their current use of NIAP- evaluated products, the perceived usefulness of the program, and the benefits and challenges associated with acquiring and using NIAP- evaluated products.

For each agency survey, we identified the office of the chief information officer, notified them of our work, and distributed the survey instrument to each via an e-mail attachment. In addition, we discussed the purpose and content of the survey instrument with agency officials when requested. All 24 agencies responded to our survey. We did not verify the accuracy of the agencies’ responses; however, we reviewed supporting documentation that agencies provided to validate their responses. We contacted agency officials when necessary for follow-up information. We then analyzed the agencies’ responses.

Although this was not a sample survey, and, therefore, there were no sampling errors, conducting any survey may introduce other kinds of errors. For example, difficulties in how a particular question is interpreted, in the sources of information that are available to respondents, or in how the data are entered into a database (or were analyzed) can introduce unwanted variability into the survey results.

We took steps in the development of the survey instrument, the data collection, and the data analysis to minimize these survey-related errors. For example, we developed the questionnaire in two stages. First, we had a survey specialist design the survey instrument in collaboration with subject-matter experts. Then, we pretested the instrument at two federal departments and internally at GAO to ensure that questions were relevant, clearly stated, and easy to answer.

We conducted our work in Washington, D.C., from May 2005 through February 2006, in accordance with generally accepted government auditing standards.

Comments from the Department of Defense

GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Jenniffer Wilson (Assistant Director), Neil Doherty, Jennifer Franks, Joel Grossman, Matthew Grote, Min Hyun, Anjalique Lawrence, J. Paul Nicholas, Karen Talley, and Amos Tevelow were key contributors to this report.