**Conformance Testing and Certification Model for Software
Specifications**

**Lisa Carnahan, Lynne Rosenthal, Mark Skall**

**Software Diagnostic and Conformance Testing Division**

# **National Institute of Standards and Technology**

##### **Abstract**

The use of conformity assessment as a means by which buyers and sellers
can communicate requirements and conformance will increase as
information technology systems and applications grow more complex.
Models for conformance testing and certification programs are necessary
to understand principles and issues that are essential for successful
conformity assessment programs. This paper presents one such model by
identifying key roles, activities and products involved in any
conformance testing and certification program. The authors have
successfully used this model in helping private-sector organizations
establish their certification programs.

##### **Communication between Buyers and Sellers**

As the pervasiveness of information technology increases, so does the
importance of ensuring the quality of products (i.e., software and
systems). Conformity assessment is defined in ISO/IEC Guide 2: 1996 as
\"any activity concerned with determining directly or indirectly that
relevant requirements are fulfilled.\" In the marketplace, conformity
assessment provides a vehicle for exchanging information between buyer
and seller. It increases a buyer's (and/or user's) confidence in a
product and its ability to meet their needs. It provides an independent,
objective method for evaluating products and not becoming locked-into a
single vendor. For sellers (and developers), conformity assessment can
help to substantiate claims that a product meets the given
specification.

Often conformity assessment is accomplished by conformance testing.
Conformance testing is a means of measuring whether a product faithfully
implements a specification. The level and formality of the testing are
determined by the market -- the requirements of the buyer directly or an
organization acting on behalf of a community of buyers, or by regulation
(e.g., safety, health, national security concerns). For example, some
programs may require a very formal testing and certification approach
consisting of independent (i.e., third party), nationally accredited
testing laboratories while others may be more appropriate for self
declaration and demonstration testing.

The Information Technology Laboratory (ITL), within the National
Institute of Standards and Technology (NIST), develops conformance tests
for forward-looking, publicly available standards and specifications.
The tests are used by companies in the private sector who build
implementations which purport to conform to these specifications. The
intent of these tests is twofold: 1) to be used by the implementors
early on in development to improve the quality of their implementation;
and 2) to be used by industry associations wishing to administer a
testing and certification service. ITL focuses on the technical task of
developing test suites; leaving the regulatory aspects of testing and
certification to the private sector. However, ITL is very active in
helping industry associations set up testing and certification programs
to use the ITL-developed tests.

The sections below describe a generic model for establishing a
conformance testing and certification program. It describes the
processes and procedures for establishing, administering a testing
program. While much has changed regarding conformity assessment given
the growth and changes in the software industry, the conformance and
certification model has not. Examples are used to describe how the model
is applied to support the changes in the software industry.

##### **Conformance Testing and Certification Model**

It is well recognized that conformance testing and certification is a
way to ensure that "standard-based" products are implemented. The
advantage afforded by testing and certification are fairly obvious:
quality products, competitive markets with more choices, commodity
pricing, and less opportunity to become "locked in" to a particular
vendor. Moreover, a testing and certification program based on well
understood and sound principles will be acceptable and credible to its
community of users.

The conformance testing and certification model described herein
contains the fundamental *roles, activities*, and *products* that are
necessary in administering and operating a testing and certification
program (see Figure 1). By adjusting and modifying the various
activities, roles and products, the model can be applied and used in
establishing any testing and certification program. Figure 2 highlights
the interactions between the roles and activities. The model allows for
roles, activities and/or products to be consolidated or further
partitioned.

+------------------+----------------------+---------------------------+
| ***Roles***      | ***Activities***     | ***Products***            |
+------------------+----------------------+---------------------------+
| Buyer            | Require              | Specification             |
|                  | Certification        |                           |
| Seller           |                      | Implementation Under Test |
|                  | Test IUT             | (IUT)                     |
+------------------+----------------------+---------------------------+
| Test Laboratory  | Recognize Test       | Test Method               |
| (TL)             | Method               |                           |
|                  |                      | Test Report               |
| Certificate      | Recognize Tester     |                           |
| Issuer           |                      | Certification Program     |
|                  | Validate Results     | Policy                    |
| Control Board    |                      |                           |
| (CB)             |                      |                           |
+------------------+----------------------+---------------------------+
|                  | Answer Programmatic  | Testing Laboratory        |
|                  | Queries              | Criteria                  |
+------------------+----------------------+---------------------------+
|                  | Answer Test Method   | Certificate of            |
|                  | Queries              | Conformance               |
|                  |                      |                           |
|                  | Resolve Test Method  |                           |
|                  | Disputes             |                           |
|                  |                      |                           |
|                  | Validate Conformance |                           |
|                  |                      |                           |
|                  | Issue Certificate    |                           |
+------------------+----------------------+---------------------------+

Figure 1: Roles, Activities and Products

While actual testing and certification can be carried out by various
organizations, it is essential that there be a centralized sponsor or
owner of the testing and certification program. The sponsor has a
fundamental interest in ensuring the success of the program.

Typically, the sponsor establishes and maintains the conformance testing
and certification program. It assumes responsibility for insuring that
the components of the program are in place and becomes the centralized
source for information about the program. The sponsor may be composed of
one or more organizations. Examples of sponsors are consortia, trade
associations, standards groups, or a government agency. More often than
not, the sponsor of the program is also the Certificate Issuer.

![](media/image1.wmf){width="5.1305555555555555in"
height="3.504861111111111in"}

Figure 2: Interactions Among Roles and Activities

##### Roles

To execute the activities of the model, five roles are defined. In the
realization of this model, some roles may be combined and performed by a
single organization or further distributed among several organizations.

-   *Buyer* requires conformance to the Specification.

-   *Seller* builds the product with the intent of meeting the
    conformance requirement of the purchaser. Products that undergo
    testing are called Implementation Under Test (IUT)

-   *Test Laboratory (TL)* performs the operational testing of the IUT .

-   *Certificate Issuer (CI)*, issues a Certificate of Conformance for
    IUTs that have successfully completed the testing process.

-   *Control Board (CB),* resolves dispute and answers queries on behalf
    of the CI.

**Buyer**

The Buyer requires that a product be tested for conformance. The buyer
uses the results of the testing to verify that a seller provides a
product that conforms to the specification and meet procurement
requirements. In general, the buyer is the impetus for sellers to
undergo conformance testing. Specifically, if buyers don't demand that a
product be tested and show evidence of that testing, it is most likely
that sellers will not undertake having their products tested.

# Seller

The Seller or developer uses the conformance tests and undergoes testing
to demonstrate that the product adheres to the specification and thus,
meets established conformance requirements. Additionally, developers may
use the tests to debug their products prior to market.

**Test Laboratory**

The Test Laboratory (TL) conducts the conformance testing using the
prescribed test method. The testing is performed on the
seller/developer's product. A TL can be an organization or individual. A
TL can be accredited from a formal accreditation organization such as
NIST's National Voluntary Laboratory Accreditation Program (NVLAP) or
recognized by the buyer, seller, and certificate issuer, as qualified to
perform the testing.

## Certificate Issuer

The Certificate Issuer (CI) is responsible for issuing certificates for
conforming products. The decision to issue a certificate is based on the
testing results and established criteria for issuing certificates.

## Control Board

The Control Board (CB) is an impartial body of experts who function on
behalf of the CI. The CB is responsible for resolving queries and
disputes related to the testing process.

##### Activities

The activities comprising the model can be categorized into one of four
areas:

-   *Recognition* of competent testing laboratories;

-   *Testing* with an approved test method; and,

-   *Testing process;*

-   *Resolution* of Queries and Disputes.

**Recognition of Competent Testing Laboratory**

A Testing Laboratory (TL) is an entity that provides services to
measure, examine, test, or otherwise assess conformance of an
implementation with its specification. Within the buyer/seller model, a
TL can be either a first-party, (the seller performs the testing),
second-party (the buyer performs the testing), or third-party[^1] (an
independent organization performs the testing) testing organization. All
three types of testing are used in the software industry. Often there
will be multiple TLs for a conformance testing and certification
program.

The Certificate Issuer (CI) as well as Sellers and other interested
parties, must have confidence in the competency of the TL. Competence is
based on three concepts,

1.  the ability to apply the test method correctly,

2.  the ability to repeat a given test and generate the same results,
    and

3.  the ability to operate the TL in a manner that maintains objectivity
    and neutrality (obviously, first and second party testing
    organizations are not neutral).

The CI defines competence through requirements and criteria. The CI can
then apply the criteria to a TL, determine its level of competency and,
if appropriate, recognize the TL as competent to perform testing. This
practical approach to identifying and recognizing qualified testing
organizations is appropriate when costs, time and efforts do not warrant
seeking accreditation from a formal accreditation organization.

If a more formal and rigorous approach is appropriate, there exists many
accreditation bodies exist that are capable of performing this function.
The National Voluntary Laboratory Accreditation Program (NVLAP) is a
NIST organization that accredits testing organizations based on the
requirements of ISO Guide 25[^2] and additional subject-matter
requirements. NVLAP is responsible for accrediting testing organizations
to perform POSIX and Cryptographic Module testing.

The purpose of the recognition criteria or accreditation is to assure
that TLs are capable and competent to meet the needs of the testing and
certification program. The basic activities to make this determination
include:

-   proficiency testing -- demonstration of a TL's competency to
    successfully perform the conformance testing using the test method,

-   on-site assessment -- visit by a technical expert to determine
    compliance with the recognition criteria and ensure the TL is a
    legally identifiable organization with staff and resource to
    discharge their duties.

-   quality assurance -- documentation and practices to ensure technical
    integrity of testing and analyses and adherence to quality practices
    appropriate to the testing and certification program.

Additional attributes required of a third-party TL include that it:

-   ensure that its personnel are free from any commercial, financial
    and other pressures which might adversely affect the quality of
    their work,

-   ensure that the protection of sellers' confidential information and
    proprietary rights are protected,

-   ensure that sellers are served with impartiality and integrity,

-   maintain a functional record keeping system for each seller testing
    process, and

-   have the adequate facilities and equipment to fulfill the
    requirements of a TL.

**Testing with an Approved Test Method**

For a Certificate of Conformance to be meaningful, all implementations
must be tested in the same manner. Testing reflects the essence of
technical requirements of specifications and measures whether a product
faithfully implements the specification. A *test method* is a defined
technical procedure for performing a test. A *test* is the technical
operation that consists of the determination of one or more
characteristics of a given product, process or service according to a
specified procedure. \[ISO/IEC Guide 2\] A test suite is the collection
of tests. Critical to the success of any conformance testing and
certification program is an appropriate and adequate test method.

An adequate test method is one that provides test results that give
enough information for the CI to be satisfied that conformance can be
measured. An adequate test method meets the requirement of rigor. An
appropriate test method is one that, while adequate, does not place
undue requirements on the IUT and is cost justifiable. If the test
method is too expensive to employ then it will not be used. The
definition of adequate and appropriate is left to the CI to determine.

### **Testing Process**

The Testing Process is described in a conformance testing and
certification policy and procedures document. The document identifies
the administrative as well as testing processes.

The testing process initiates with a seller (or anyone desiring to be
tested) contracting with the TL to have an implementation tested for
conformance. The seller and TL negotiate the scope of testing, the cost
of testing, and the timeliness of testing. For a given seller, the TL
must not be in a position to benefit nor suffer (beyond the testing
fees) from the resulting pass or failure of the implementation under
test (IUT).

Using the approved *Test* *Method*, The TL tests the IUT for conformance
and reports the results in a *Test Report*. The TL forwards the Test
Report and an indication of pass/fail to the CI. If the IUT successfully
completes all the tests and meets the criteria for issuing certificates,
the CI issues a *Certificate of Conformance* to the seller. Typically,
the CI maintains a list and makes available to the public, a register
containing a listing of products that have received certificates of
conformance.

**Resolution of Queries and Disputes**

Queries and disputes involving the test method, procedures, test
results, and program administration are directed to the Control Board
(CB). The purpose of the CB is to resolve these issues and communicate
the decision to all parties involved. The CB acts on behalf of the CI. A
query or dispute can be initiated by a seller, TL or entity (e.g.,
developer) at any point in the testing process. Queries and disputes
should contain a statement of the problem, rationale for dispute, and
desired resolution. All matters to be resolved by the CB should be
determined by consensus or as determined by documented CB policy and
procedures.

Additional activities that may be under the auspices of the CB include:

-   maintain liaison with appropriate standards bodies and test
    laboratories,

-   participate in the assessment of TL's seeking recognition status,

-   recommend changes to new versions of the test method or test
    laboratory recognition criteria,

-   serve as technical advisor to the CI and TLs

-   maintain the test suite, and

-   control changes to the conformance testing process.

##### **Products**

The following products are used in the model:

-   Certification Program Policy;

-   Testing Laboratory Criteria;

-   Specification;

-   Implementation Under Test (IUT);

-   Test Method;

-   Test Report; and

-   Certificate of Conformance.

**Certification Program Policy**

The Certification Program Policy (CPP) defines the certification system.
ISO/IEC Guide 2 defines a certification system as a "system having its
own rules of procedure and management for carrying out conformity
certifications. The CPP addresses the following:

-   responsibilities of the CI;

-   responsibilities of the TLs;

-   responsibilities of the seller (the IUT owner);

-   policy and procedures for test laboratory recognition;

-   policy and procedures for the testing process;

-   policy and procedures for handling queries and disputes;

-   complete definition of the certificate of conformance.

**Testing Laboratory Criteria**

Testing Laboratory Criteria serves three purposes. The first purpose is
to define the competence and quality-related requirements that a testing
laboratory must possess to be designated as a recognized testing
laboratory. The second purpose is to describe the manner in which the
laboratory will be assessed against the requirements. The third purpose
is to show those who want to use the testing laboratory (e.g., sellers)
, or those who want to accept the conformance certificate as evidence of
conformance (e.g., buyers) the rigor under which the testing laboratory
operates.

**Specification**

First and foremost to conformance testing and certification is the
specification. This paper delineates 'standards-based' software
specification from other types of specification. This is because not all
specifications can be objectively tested for conformance. We recognize
that not all 'standards-based' specifications can be objectively tested.
However objective measurement (not necessarily conformance testing per
se) is usually a goal in these specification development efforts.

If the specification can not be objectively tested, then a alternate
approach to conformance testing should be used to measure whether a
produce faithfully implements the specification. This is because an
accepted test method cannot be developed, thus repeatability and
reproducibility cannot be ensured.

**Implementation Under Test**

The implementation under test (IUT) is the object that is being tested
for conformance. For software specifications it is the software that has
'implemented' the specification. For any certification program, the
scope of the IUT must be defined and delineated from the rest of the
supporting software and hardware of the total system (referred to as the
system under test). In many current certification programs the hardware
that is used by the software must also be defined. The software and
supporting hardware constitute the IUT and are listed in both the test
report and certificate of conformance.

**Test Method**

The test method must be adequate and appropriate within the conformance
testing and certification program in which it is used. Beyond these
properties, test methods (and thus the tests) should be objective, have
adequate coverage, and correctly implement the specification. In trying
to meet these requirements, those using and applying the test method
should not make the common mistake of allowing the test method to become
the 'specification'. This means that sellers (builders of IUTs) will
build the IUT to pass the conformance tests, rather than building to the
specification.

An objective test method allows for test results to be reproducable by
the same testing laboratory and to be repeatable by a different
laboratory. Initially some test methods do not quite achieve a
sufficient level of objectivity. However objectivity should be something
that is always strived for in the development and ongoing refinement of
a test method.

**Test Report**

A test report contains the results of the testing effort, along with any
additional information required by the CI. The test report should
provide enough information that, if necessary, the testing effort could
be duplicated. The testing report should contain:

-   a complete description of the IUT;

-   the name of the testing laboratory;

-   the signature of a testing laboratory official;

-   the date that the testing occurred;

-   the name and version number of the test method (and test suite);

-   the results of the test method;

-   an unambiguous statement indicating pass or fail.

**Certificate of Conformance**

The certificate of conformance is typically a summation of the test
report. Since it is often used in the procurement process, it includes
information most pertinent between the buyer and the seller.

The certificate includes statements made by the CI. These statements
articulate what the CI is asserting as being conformant. Typically these
statements indicate that "this IUT was tested in this environment, on
this day, using this test method: the test results produced were
consistent with expected test results". The certificate also includes
the signature of a CI official.

##### Examples

**Cryptographic Module Validation Program**

The Cryptographic Module Validation[^3] (CMV) Program was established to
provide independent testing to *Security Requirements to Cryptographic
Modules, Federal Information Processing Standard, 140-1,*(FIPS 140-1).
When applied appropriately, FIPS 140-1 can help provide strong
protection for equipment that provides security services such as
encryption, authentication, and digital signature generation and
verification. FIPS 140-1 was developed by a joint industry/government
working group.

The requirement for certification is specified in the FIPS 140-1
standard. Therefor, this program is based on a regulatory requirement.
However there is currently an effort underway in the ANSI X9 area to
adopt FIPS 140-1. Thus the requirement for certified products will be
driven from the private sector.

The Security Technology Group at National Institute of Standards and
Technology (NIST) serves as the sponsor and Certificate Issuer (CI) for
the CMV Program. As such, the certifications are considered to be
second-party (i.e., NIST is acting on behalf of user of FIPS 140-1,
primarily federal agencies.) The CI requires that testing laboratories
be accredited by NVLAP under its Cryptographic Module Testing Program.
The CMV Program requires that testing laboratories be independent from
clients (i.e., third-party testing.)

The test method was developed NIST and was vetted by the industry. The
test method, policies and all other program documents are publicly
available.[^4]

The validation certificate, signed by a CMV Program official, contains
the following:

-   the name of the cryptographic module;

-   an indication of pass/fail for each of the requirement areas
    specificed in the standard;

-   the accredited testing laboratory with its NVLAP identification
    code;

-   a statement defining the scope of the validation;

-   the date of the validation.

**ATA Computer Graphics Metafile (CGM) Conformance Testing Program**

The Air Transport Association (ATA)[^5] CGM Program was originally
established and operated by NIST to support the *ATA 2100 Specification,
Graphics Exchange* (a.k.a. ATA CGM profile). The testing program is a
critical component of the ATA's program to represent maintenance manuals
in digital form and move to completely on-line maintenance manuals.
Testing is done to ensure that the fidelity and quality of the digital
information is sufficient to satisfy the airline companies' safety and
quality concerns. The program is a means whereby a seller of a CGM
implementation can formally demonstrate conformance to the ATA CGM
profile.

NIST is currently working with the ATA in its assumption of the testing
program. The ATA CGM Conformance Testing Program will consist of
recognized Testing Laboratories to conduct the testing and a Control
Board to handle disputes and serve as an advisor to the ATA. The ATA
will act as the sponsor and administrator of the program. The ATA or an
ATA designate will issue certificates of conformance. The roles,
activities, and products as described in the generic model apply here
with little modification. The Control Board takes on the additional
activity of assessing the testing laboratories according to
pre-established criteria. Additionally, the ATA Technical Information
Communication Committee's Graphics Working Group[^6] serves as a
technical advisor to both the ATA and the control board.

The test method consists of a NIST developed test suite and test
procedures. The test method has been accepted and used by the community.
It is publicly available along with other program documents[^7].

# IEEE POSIX Validation Service 

The IEEE established a validation service for the POSIX (Portable
Operating System Interface). The IEEE validation service[^8] uses
Accredited POSIX testing laboratories, issue certification of validated
test results, and maintains a register of accredited laboratories and
successfully tested products. The laboratories are accredited by the
NVLAP under its POSIX program.

The requirement for testing is buyer driven. Initially, federal agencies
in their requests for procurement (RFP) of POSIX systems required
certificates of validation prior to purchase. However, the benefits of
POSIX testing and its acceptance in the industry has resulted in sellers
requesting to be tested as a matter of course, rather than a procurement
requirement.

The test suite was produced in a joint effort between NIST and several
computer vendors. The original testing policy and procedures produced by
NIST have been adopted by the IEEE.

#### Conclusion

This model describing the conformance testing and certification process
has been used many times over in certification programs for
standards-based software specifications. The examples above illustrate
just a few of these programs. It will continue to be used as a
communication mechanism between buyers and sellers.

Test method developers must continue to develop test methods that have
adequate coverage with regard to the specification; are well defined in
terms measurement (i.e., what does each test case prove); and be
adequate and appropriate as defined by the Certificate Issuer.

As the industry moves toward component based software, the challenge
will be to develop test methods and associated certification programs
that can provide meaningful measurement in this environment.

**\
**

###### Bibliography

Breitenberg, Maureen, *The ABC's of the U.S. Conformity Assessment
System*, NISTIR 6014, April 1997.

Breitenberg, Maureen, *The U.S. Certification System from a Government
Perspective*, NISTIR 6077, October, 1997.

Carnahan, Lisa, *Developing Federal Standards and Accreditations for
Data Protection Products*, Proceeding of SPIE Conference, October, 1995.

Dashiell, William H., L. Arnold Johnson and Lynne S. Rosenthal,
*Overview of Model for United States Geological Survey Recognition of
Spatial Data Transfer Standard Certification System*, NIST IR 6124, May
1998.

Horlick Jeffrey, and Lisa Carnahan, *Cryptographic Module Testing*,
Handbook 150-17, April, 1995.

ISO/IEC Guide 2: 1996, *Standardization and Related Activities: General
Vocabulary*

ISO/IEC Guide 25: 1990, *General Requirements for the Competence of
Calibration and Testing Laboratories*.

NIST, *Derived Test Requirements for FIPS 140-1, Security Requirements
for Cryptographic Modules*, March, 1995.

NIST, *Procedures and Requirements*, NIST Handbook 150, March 1994.

NIST, *Security Requirements for Cryptographic Modules*, FIPS 140-1,
January, 1994.

[^1]: The use of the term \'independent\' is ambiguous and thus applied
    differently in the testing community. Some programs may determine
    independence based on corporate structures; while others may apply
    financial-interest related measures. It is incumbent for the
    Certificate Issuer to define the term \'independent\' for its
    community of interest.

[^2]: The requirements of ISO Guide 25, General Requirements for the
    Competence of Calibration and Testing Laboratories, are based on the
    ISO 9000 standards. Accreditation based on ISO Guide 25 provides the
    basis for many of the international recognition agreements for
    recognizing test results.

[^3]: The CMV Program uses the term 'validation' rather than
    'certification'. Beyond legal reasons for not using 'certification';
    the program developers chose not to use this term because of its
    many uses within the cryptographic and computer security communities
    (e.g., 'certification authority for digital signatures' and
    'certification and accreditation of systems').

[^4]: CMV Program information can be found at
    http://csrc.nist.gov/cryptval/.

[^5]: Information about the ATA can be found at
    http://www.air-transport.org/.

[^6]: The Graphics Working Group is the organization responsible for
    creating and maintaining the ATA 2100 Specification, Graphics
    Exchange specification.

[^7]: ATA CGM Program information can be found at
    http://www.itl.nist.gov/div897/ctg/graphics/cgm.htm.

[^8]: IEEE POSIX Validation Program information can be found at
    http://standards.ieee.org/regauth/posix/index.html.