Standards Conformance Testing:

Issues and Activities

Sharon J. Kemmerer

U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Institute for Computer Sciences and Technology

Information Systems Engineering Division

Gaithersburg, MD 20899

April 1988

-QC

100

. U56

88-3768

1988

RTMENT OF COMMERCE

JREAU OF STANDARDS

’ i • V i LJL

1913-1988

NBSIR 88-3768

STANDARDS CONFORMANCE TESTING:

ISSUES AND ACTIVITIES

iieseartk i seiner

National bureau of btanuarcis

Gaithersburg, Maryland 20899

Q.c/oo

no. S^374g

/its

t. 3

-

Sharon J, Kemmerer

U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Institute for Computer Sciences and Technology

Information Systems Engineering Division

Gaithersburg, MD 20899

April 1988

U.S. DEPARTMENT OF COMMERCE, C. William Verity, Secretary

NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Director
.

-

__
TABLE OF CONTENTS

INTRODUCTION . 1

Purpose 1

Scope 1

Background 1

Document Organization 2

Terms 3

STATUS OF ICST CONFORMANCE TESTING PROGRAMS 5

Compiler Validation 5

Document interchange 5

Operating Systems 5

Graphics 6

Data Management . . . 6

Raster Compression 6

Networking ........ 6

Storage Media 7

Security 8

CONCEPTUAL FRAMEWORK FOR DEVELOPING

CONFORMANCE TESTING PROGRAMS 9

1. PROGRAM INITIATION 9

Standards Identification 9

2. CONFORMANCE TESTING PROGRAM DEVELOPMENT 9

Guide for Developing a Testing Program ...... 9

Application for Accreditation . 10

3. TESTING METHODS 10

Commonly accepted conformance test methods .... 10

Guidelines for development of test method

approaches 11

Existing test methods 12

Test Suite Structure 13

Maintenance of Test Methods 15

4. CERTIFICATION AND REPORTING CONSIDERATIONS .... 15

Certification Body 15

ICST Role in Conformance Testing As Government

Agent 17

Role of Standards Bodies . 17

Test Report 18

Certificate of Validation Format, Content and

Presentation 19

Applicability of Certificate of Validation .... 19

5. TESTING ADMINISTRATION 20

General Guidelines for Test Method Administration 20

Criteria and Procedures for Alternative Testing

Sources 21

i
Associated Costs ................. 22

Resource and overhead requirements 22

Criteria and Procedures for Requesting Validation

Services .................. 22

6. TESTING LABORATORIES ............... 25

Conditions for Accreditation ........... 25

Criteria for Accrediting Testing Laboratories . . 26

Testing Laboratory Assessment . .... 31

Responsibilities of Testing Laboratories ..... 35

Role of Certification Body in Sponsoring a

Laboratory Accreditation Program ...... 35

7. MUTUAL RECOGNITION OF VALIDATION/CERTIFICATION . . 36

Common Recognition ................ 36

Considerations and Issues 37

8. REFERENCES 3 7

APPENDICES

A. Terms and Definitions . 41

B. Sample Application for Testing Laboratory 47

C. Model Certificate of Accreditation ......... 53

INDEX ........................... 55

ii
STANDARDS CONFORMANCE TESTING: ISSUES AND ACTIVITIES

INTRODUCTION

l Purpose. To provide guidance when developing a conformance

testing program for a Federal Information Processing Standard

(FIPS)

.

Scope. Anyone interested in developing a conformance testing

program may find this document provides a starting point from

which to depart.

Background

.

Technically sound national and international standards are needed

to preserve open competition in international markets and to

support increased productivity and delivery of services at

reduced cost. Through standards, users are provided with offthe-shelf, compatible hardware, software and communications

products for computer and related telecommunications systems.

Validation of these computer products claiming conformance with

standards further reduces risks and uncertainties to vendors and

users? uniform conformance testing procedures should be employed

to perform this validation.

Conformance testing constitutes the examination of a candidate

product or service for the existence of specific characteristics

required by a standard. The Institute for Computer Sciences and

Technology (ICST) at the National Bureau of Standards (NBS)

participates in development of U.S. Government-wide standards for

computer software, hardware, data management, networks and

security. ICST works through voluntary industry standards

organizations to develop standards that will meet the needs of

Government users and be implemented in off-the-shelf commercial

products. Standards that promise sizable benefits to the

Government are issued as Federal Information Processing Standards

(FIPS) . These FIPS and the specifications they adopt are

implemented into computer products for commercial use. Through

past experience in research and testing, ICST sees a need for

expansion of its efforts in structuring conformance testing to

these FIPS, and is in the process of formulating a program.

ICST is developing an initial structure and approach for uniform

conformance testing programs Government-wide for FIPS. This

paper presents a conceptual framework which is modeled after such

established international programs as the National Voluntary

Laboratory Accreditation Program (NVLAP) and language compiler

validation program; as well as the international efforts in the

ISO graphics committee. This framework outlines a progression of

steps needed to develop a conformance testing program for any

given standard. (Although this document does not particularly

address verification or acceptance testing, some of these steps

may also apply for these programs.)

1
One of the issues needing to be addressed separately is the

technical approach to program and test suite development.

Another area of concern is available resources. Staff and

equipment required for development, continual maintenance and

updating, and all documentation are just a few considerations

when developing a conformance testing program. An additional

critical element impacting effective use of the given resources

is timings when does the user community need these conformance

tests, and how long will it take to provide the tests using the

current level of resources?

Part of each standard conformance testing program development

will be recognition of the unique characteristics of. each

standard, such as: stability for conformance test development;

level to which evaluation can be automated; "degree of

completeness" of tests (when is enough, enough?) ; test expansion;

interpretation of the standard for test development; (should

tests be developed before, after or concurrent with standard

development) ; criteria (should all tests have pass/fail results)

;

human intervention allowed during testing; and the potential

implication of excluding portions of the standard for commercial

use since only selected portions of the standard may be tested.

The issues and procedures identified in this paper are among

those that will be considered in developing the overall ICST

Conformance Testing Program. The ICST conformance program and

individual, conformance tests for any given standard will evolve

as more experience is gained.

Document Organization.

The remainder of this document is divided into two parts; Status

of ICST 8 s Conformance Testing Programs , which provides a brief

summary of conformance testing programs already under way at

ICST; and Conceptual Framework for Developing Conformance Testing

Programs . This second part is an outlined process offering one

approach for developing a conformance testing program for any

given standard. Variations to this approach are assumed since

testing requirements and programs will be based on the maturity

and complexity of the standard, available resources, and public

demand. It is broken down into the following sections;

1. Program Initiation

2 . Conformance Testing Program Development

3 . Testing Methods

4. Certification and Reporting Considerations

5. Testing Administration

6. Testing Laboratories

7. Mutual Recognition of Validation/Certification

8. References

APPENDIX Terms and Definitions

2
APPENDIX B. Sample Application for Testing Laboratory

APPENDIX C. Model Certificate of Accreditation

Terms.

Appendix A provides terms and their definitions used within the

body of this report or commonly used when discussing conformance

testing. They are presented to provide a common understanding

from which to depart.

3
j),' j{ »r- {«»«*«< 'Sk ctS* - •• .
STATUS OF ICST CONFORMANCE TESTING PROGRAMS

The following paragraphs provide a glimpse of initiatives in

conformance test development at ICST. Various approaches were

employed for a common goal: testing conformance to a standard.

Some programs are more complex than others due to the varying

requirements and nature of the standards themselves. Development

has been completed on only a few of the programs.

Compiler Validation. ICST provides compiler validation for the

following FIPS languages: COBOL ( FIPSPUB2 1-2 ) , FORTRAN

(FIPSPUB69-1) . PASCAL (FIPSPUB109) , Ada (FIPSPUB119) , and BASIC

(FIPSPUB68-1) . There is a fee associated with each language

compiler test to recover costs since it has been a multimillion

dollar investment to date. An entire ICST certification program

is in place, which includes complete validating and operating

procedures, and publishing a certified list of those compilers

currently under certification. The quantity of tests in the test

suite will vary; for example, PASCAL currently has approximately

800 tests associated with it, Ada has approximately 2600. ICST

cooperates with other testing and certification authorities in

the United States (e.g., Department of Defense Ada Joint Program

Office) , as well as internationally (France, the UK and Germany)

.

The Federal Information Resources Management Regulations2

requires all compilers sold to the U.S. Government must show

conformance to the appropriate language FIPS.

Document interchange. The International Standard Generalized

Markup Language (SGML) (proposed as a FIPS) , requires "parsing"

software to test the conformance of document type definitions and

documents to the standard. SGML tools that parse SGML documents

also need to be checked for completeness and ICST has developed a

validation suite of test cases to do this. ICST has developed

and made publicly available an SGML reference parser, an SGML

validation suite, and supporting documentation. This software is

available for a nominal fee from NTIS (703/487-4650) . No formal

certification program is in place, and ICST does not endorse or

certify any product which has successfully been parsed using this

software

.

Operating Systems. The standard for Portable Operating System

Interface for Computer Environments (POSIX) has been proposed as

a FIPS. A reference implementation test suite is also being

developed to test conformance to the FIPS. This implementation

is based on the draft document being developed in the P1003.3

FIPSPUBS are available through U.S. Department of

Commerce, National Technical Information Service,

Springfield, VA 22161? 703/487-4650.

FIRMR 201-8.109

5
Committee sponsored by IEEE. A number of companies are

participating in this implementation. When completed, the source

code for the test suite will be placed in the public domain.

ICST is investigating the use of third party testing services,

and plans to maintain and update the test suite as the standard

evolves. The initial release of the subset is scheduled CY1988.

Graphics. An effort for the Graphics Kernel System (GKS)

(FXPSPUB120) validation began in 1982 and has been funded by the

Commission of the European Communities (CEC or EC) . ICST has

been an active participant along with Germany, the United Kingdom

and France. The falsification testing methodology has been

employed, tests have been developed for five separate interfaces

(application, data structure, error data consistency, metafile

device, and operator {user}), and ICST has provided 35 beta test

sites with copies for comment. ICST has been welcomed into CEC as

a potential test site for future testing of both GKS and CGM

(Computer Graphics Metafile) (FXPSPUB128) . A one year trial test

service is scheduled to begin in 1988. Work is also progressing

toward developing test suites for CGM and PHIGS, the Programmer's

Hierarchical Interactive Graphics Standard (anticipated as a FIPS

CY1988 )

.

Data Management. The Information Resource Dictionary System

(IRDS) has been the forerunner in the data management standards

for initiating a conformance testing program. Currently

completing the final stages of becoming a national standard, it

will also be a FIPS. ICST has developed a prototype

implementation of a subset of the IRDS command language; future

direction includes a complete reference implementation. The

prototype has been made available for vendors who are willing to

provide feedback on their results. Industry is also cooperating

in the development of tests. Database Language SQL (FIPSPUB127 )

,

comprised of a schema definition language and a module language

which includes structured query language statements, has a

validation test methodology and test suite also under

development.

Raster Compression. In support of the Department of Defense, ICST

is currently prototyping the compression and decompression of

engineering drawings, for storage and transmittal, using the

CCITT , Group 4 Raster Compression standard (FED-STD 1064 & 1065)

.

The testing is demonstrating 8 1/2 "xll" size drawings; ICST

anticipates that a formal compression conformance testing program

will be established in CY1988.

Networking.

FDDI. The Fiber Distributed Data Interface (FDDI) is a token ring

network standard which allows up to 1000 stations in a network.

ICST has established a Fiber Optic Network Laboratory, initially

focusing on FDDI. A station test set and procedures were

developed and a neutral, instrumented testbed for multi-vendor

6
fiber LAN integration testing is being established. Applying

these facilities and documentation, ICST cooperates with industry

for use and participation in developing commercial test

equipment

.

X. 25 . ICST developed a testing methodology for the International

Telegraph and Telephone Consultative Committee (CCITT)

Recommendation X.25 in 1983. The Department of Defense, Navy and

Army have adopted the method, and the ICST test method is being

used as a base to establish a national standard on the testing of

X.25. ISO/TC97/SC6 has established a project (ISO 8882) to test

X.25 terminals. Additional efforts for a conformance testing

framework are underway in the international standards arena.

FIPS PUB 122 provides the conformance tests for FIPS PUB

100/FEDSTD 1041.

ISDN . ICST has solicited support from other government agencies

and industry, and is developing testing specifications and

coordinating efforts with the Open System Interconnection (OSI)

testing initiatives. Next steps include developing a standard

test report format, implementing the test specifications, issuing

a FIPS on the Integrated Services Digital Network (ISDN)

conformance testing, developing a plan on ISDN laboratory

accreditation, and attracting national and international support

of the testing program.

GOSIP . ICST has developed a FIPS for the Government Open System

Interconnection Profile (GOSIP) specification. GOSIP will enable

Federal agencies to acquire and use OSI protocols rather than

vendor-specific or proprietary protocols. GOSIP is expected to

be promulgated as a FIPS in early CY1988. ICST has already

developed a prototype testing network called OSINET, a global

network supporting 25 industry and government organizations.

This cooperative testing among vendors through demonstration

participation, the availability of such an OSINET operation, and

research and development activities at ICST are all initiatives

contributing to the testing program development. Some of the

tests have been developed by ICST; the Corporation for Open

Systems (COS) is also developing tests. Organizations such as

the Industrial Technology Institute (ITI) already offer testing.

Overall, tests and testing programs will be evaluated by ICST for

their suitability and sufficiency in meeting Federal

requirements

.

Storage Media. NBS offers for sale various standard reference

materials (SRM) for evaluating the performance of magnetic

storage media and systems, and for maintaining control over their

production. Each SRM is individually calibrated and certified.

Purchase orders for all SRM's should be addressed to: Office of

Standard Reference Materials, Room B311 Chemistry Building,

National Bureau of Standards, Gaithersburg, MD 20899; telephone

3 Ol/975-OSRM {6776}; FTS 879-OSRM; Telex TRT197674NBS UT.

7
Security. DES Validation. ICST has had in place a Data

Encryption Standard (DES) Validation Program since 1977 which

validates DES implementations for conformance to the standard

{ FIPSPUB46 } . In 1986 a Message Authentication Code (MAC)

Validation System was established, based on FIPS PUB 113 # Computer Data Authentication. For both of these services, ICST

provides Certificates of Validation for devices which

successfully pass the tests. To date, ICST has validated more

than 10 vendor equipments for MAC (two of which were transAtlantic (UK, Denmark}), and more than 25 for DES. A Key

Management Validation System which will support the testing of

ANSI X9.17, Financial Institution Key Management, is anticipated

to be in place in CY1988.

8
CONCEPTUAL FRAMEWORK FOR DEVELOPING

CONFORMANCE TESTING PROGRAMS

1 . PROGRAM INITIATION

1.1. Standards Identification.

Specific FIPS and other standards being prepared for FIPS

publication are currently under study as candidates for

conformance testing. Considering the following will help

identify those FIPS requiring a conformance testing program:

1.1.1. Is the standard straightforward so that

conformance to it would be obvious and tests

unnecessary, e.g., state and country codes.

LL2. What are the needs and scope of the user

population.

1.1.3. What is the nature and content of other relevant

conformance testing programs ... are there already

sufficient programs in place to test the standard.

1.1.4. What is the current or perceived future

importance of the FIPS to commerce, or to consumer

well-being.

1.1.5. What is the economic and technical feasibility

of accrediting testing laboratories for the test

methods, types of test methods, products, or services

requested.

2. CONFORMANCE TESTING PROGRAM DEVELOPMENT

2.1. Guide for Developing a Testing Program.

The following issues need to be addressed in developing a

new testing program; guidelines and recommendations to solve

these issues follow in the remainder of the text.

2.1.1. If multiple test methods are currently

available, establish assessment techniques and

priorities to decide which test method (s) should be

offered for testing.

2.1.2. Decide what the units of evaluation should be,

i.e., should the test methods be offered individually

or should they be logically grouped; should all test

methods of a standard specification be offered as a group

.

2.1.3. Identify the critical elements (environment,

test equipment and apparatus, procedures) , for each

test method, e.g., quality assurance checks.

9
2.1.4. Establish assessment techniques and priorities

based on the complexity and importance of each test

method, e.g„, should the assessor carry any

instruments; should any test methods be demonstrated

during assessment.

2.1.5. Determine the applicability of field testing

important test methods and the means for implementing

new or available field testing programs.

2.1.6. Identify selection criteria and nominate

technical experts who could serve as assessors and

evaluators

.

2.1.7. Recommend any other special technical

requirements which should be considered as necessary

for accreditation in this testing field.

2.2.

Application for Accreditation.

2.2.1. Any laboratory may apply for accreditation in

any conformance testing program following the

instructions provided in notices announcing the formal

establishment of the program. Appendix B provides a sample application for a Testing Laboratory.

2.2.2. Upon receipt of the laboratory's application,

the Certification Body should:

2. 2.

2.1.

acknowledge receipt of the application;

2. 2. 2. 2. request further information, if

necessary;

2. 2. 2. 3. confirm any payment of any fees before

proceeding with the accreditation process; and

2. 2. 2. 4. specify the next steps (s) in the

accreditation process.

2.2.3. In accepting an application from a foreignbased laboratory, consideration should be given to the

policy of the host government regarding the acceptance

of test data from laboratories accredited by other

domestic or foreign accreditation systems.

3. TESTING METHODS

3.1.

Commonly accepted conformance test methods.

10
3.1.1. Falsification Testing. Types of standards which

currently employ this method include:

3 .1.1.1. programming language standards;

3.1. 1.2. graphics standards; and

3 • 1 • 1 . 3 . text standards

.

3.1.2. Proofs of Correctness Testing. This is a more

difficult process, but the more desired approach

because of its completeness. To date, no known

conformance testing program is able to offer proof of

correctness.

3.2.

Guidelines for development of test method approaches.

The test method should be specified in a way that is easy to

understand and to implement. The resulting test suite

should be useful to implementors. The following steps are

recommended when developing a test method:

3.2.1. Management Plan. Describes the overall scheme

for the testing. It should include a specification of

the technical objectives to be achieved, a description

of the test strategy, and an outline of the procedures

to be used for testing. This management plan should be

publicly available.

3.2.2. Non-Testable Features. Although the objective

is always to test all functions in a standard, there

are sometimes parts of a standard which are nontestable. For example, some error conditions,

depending on their nature, cannot be included in a

standardized test method to check conformance. A list

of all areas explicitly not tested should be available

as part of the documentation of the test method.

3.2.3. Test Strategy. A full description of the test

strategy to be used for the standard should be produced

and made publicly available. This should include a

specification of the structure of the proposed test

suite, an indication of the number of tests, and an

identification of the elements of the standard to be

tested.

3.2.4. Test Procedures. The test method should be

automated and include consideration of the procedures

used to carry out testing. The setting up and

execution of the test suite on a system under test, and

the analysis of the output produced should be carried

out in accordance with clearly defined procedures.

These procedures should be documented in a manual to

11
accompany the test suite, and should give criteria for

the analysis of all acceptable outputs for each test in

the test suite. Specific assumptions for failure of

test should also be stated.

3.2.5» Levels of Testing. The development of the

test method should take into account all possible

levels and/or options specified in the standard. The

test method should allow for modularity so that tests

for a specific level or option can be easily selected.

3 0 3 . Existing test methods.

The following areas should be examined when selecting the

appropriate test methods

3.3.1.

Standard specification:

3. 3. 1.1. Definition of conditions for conformance

with standards (conformance clauses)

.

3 .3. 1.2. Specification requirements. The

requirements of the standard should be evaluated

to identify the test method necessary for

determining conformance. This method might

include tests for determining that the

implementation

3. 3. 1.2.1. rejects illegal structures;

3 . 3 . 1 . 2 . 2 . accepts data and other constructs

permitted by the standards, and processes

them in accordance with the standard;

3. 3. 1.2. 3. does not include extensions to a standard;

3. 3. 1.2. 4. properly flags usage of standard

requirement options or levels;

3. 3. 1.2. 5. processes implementation

dependent specifications in accordance with

the standard;

3. 3. 1.2. 6. processes optional implementation

features of the standard correctly.

3 o 3 c 2 . Written test procedures supporting the test

suite to identify:

3. 3. 2.1. test objective;

3 . 3 e 2 . 2 . personnel responsible for test

;

12
3 . 3 . 2 . 3 . necessary equipment?

3. 3. 2. 4. materials, e.g., support software?

3. 3. 2.5. input data?

3 . 3 . 2 . 6 . expected output?

3. 3.2. 7. constraints ?

3 . 3 . 2 . 8 . required test preparation, setup?

3 . 3 . 2 . 9 . operator actions, console setups? and

3.3,2.10. step-by-step execution instructions

.

3.3.3. The test suite to see if it is:

3 . 3 . 3 . 1

.

easy to understand and manage ? 3. 3. 3. 2. portable between different hardware

configurations and designed to take into account

any implementation-dependent criteria allowed by

the standard?

3. 3. 3. 3. designed in such a way that there are no

restrictions in using the test method or

implementations on the size machine on which it

can be executed.

3.3.4. Documentation Requirements for inclusion of:

3. 3. 4.1. description of test cases?

3 . 3 . 4 . 2 . notation used to define test?

3. 3. 4. 3.

and

test method implementation/user guides ?

3.3. 4. 4. evaluation guides for qualifying pass,

fail, or inconclusive test verdicts.

3.3.5. Requirements, if any, for specially designed

test equipment training.

3.4. Test Suite Structure.

The following guidelines should be followed when writing

test suites

:

3.4.1. Clear documentation should be incorporated into

the tests and this documentation should include:

13
3. 4. 1.1. comments, which are clear and concise.

No redundant comments should be included.

3.4. 1.2. a reference to the clauses in the

standard which are under test.

3 .4.1. 3. clear statements of any assumptions made

in the test suite design.

3.4. 1.4. a list of any tests which are performed

for information only, and will not be used to

determine conformance.

3.4.2.

The test suite should be split into a series of

independent test programs or scenarios, such that each

test program tests only one ‘specific requirement of the

specification document.

3.4.3.

Each self-contained test should use the same

format, and:

3. 4. 3.1. list all test tools necessary?

3. 4. 3. 2. provide the testing environment data,

including display of setup instructions to

operator ?

3 . 4 . 3 . 3

.

include program documentation section ?

3. 4. 3. 4. initialize its own workspace?

3. 4. 3. 5. perform the test?

3 .4. 3. 6. provide a verification section?

3. 4. 3. 7. check whether passed/failed?

3. 4. 3. 8. report result?

3. 4. 3. 9. have the pass/fail path clearly defined?

3.4.3.10. include a simple comment describing the

task performed by the test and a reference to the

section of the standard under test? and

3.4.3.11.

be kept as simple as possible.

3.4.4. Only simple features of the test support code

should be used.

14
3.4.5. Translating the Test Suite to Other Languages.

If possible and when applicable, the test suite should

be written in a common format, which can then be

converted, by the use of translators, to each of the

programming languages. Updates to the test suites

across languages must be kept in line at all times by

strict use of the change control procedures.

3.5. Maintenance of Test Methods

3.5.1. Responsible Party. The Certification Body is

responsible to ensure maintenance of current test

methods, authorize changes, define the procedures for

changes, and indicate when any changes become effective

for validation purposes.

3.5.2. Procedures. A published change control

procedure should be established for both the test

software and the associated documentation. The change

control procedures should provide a formal mechanism

for the following:

3 . 5 . 2 . 1

.

3 . 5 . 2 . 2 .

3 c 5 . 2 . 3 . and

reporting errors in the test software;

logging changes to the test software;

reporting errors in the documentation;

3. 5. 2. 4. logging changes to the documentation.

4. CERTIFICATION AND REPORTING CONSIDERATIONS

4.1.

Certification Body.

4.1.1. General Requirements.

4. 1.1.1. Access to the Certification Body shall

not be conditional upon membership in any

association or group, nor shall there be undue

financial conditions to restrict participation.

4. 1.1. 2. The procedures under which the Body

operates should be administered in a

nondiscriminatory manner.

4. 1.1. 3. ICST will identify the members on the

Certification Body.

4.1.2. Responsibilities.

15
4. 1.2.1. Approve test methods to be used by the

testing laboratories when they are not included in

the standard itself. Ensure the test method is

technically correct to the FIPS before using it as

the basis for issuing certificates.

4 . 1 o 2 o 2 » Collaborate with the testing laboratory

in drafting a contract for testing services to

include regulations which define rights and

obligations in detail.

4 . 1 . 2 . 3 . Determine accreditation criteria for

testing laboratories and keep lists of all

recognized to do conformance testing. -Since

technical requirements for laboratory

accreditation are specific for each standard,

technical requirements for accreditation should be

developed using expert advice in that given field.

Appendix C provides a Model Certificate for

Accreditation

.

4. 1.2. 4. Notify any testing laboratory that it is

no longer accredited.

4. 1.2.5. Issue certificates at the request of the

Client, based on the certification criteria

defined by the Certification Body.

4.1.2.6. When applicable, define the requirements

and procedures of a Manufacturer's Declaration of

Conformance Testing Program.

4. 1.2. 7. Define Client/testing laboratory dispute

procedures. Arbitrate appeals in cases of

dispute. Include procedures for disputed tests as

well as disputed testing results.

4 .1.2. 8. Determine if there is a need for a Certified Computer Products List. If there is a need (possibly on a case by case basis) , identify

the maintainer and what information should be

reported.

4 .1.2.9. Determine what process should be

followed in recognizing other published Certified

Computer Products Lists.

4.1.2.10. Notify the public by advertising new

conformance testing programs when the technical

requirements and test methods are developed for

the standard (s) . Publish a notice in the FEDERAL

REGISTER announcing the establishment of the

16
program, requirements, or methods,

will

:

The notice

4.1.2.10.1. identify the scope of the

program ,* and

4.1.2.10.2.

advise how to request validation

services.

4.1.2.11. Develop criteria and procedures for

evaluating, modifying and accepting test methods

for use in conformance testing to the standards.

Keep in mind such test methods sources as:

4.1.2.11.1.

4.1.2.11.2.

4.1.2.11.3.

4.1.2.11.4.

industry

;

government

;

other countries ? and

universities.

4.1.2.12. Publish the effective date for

modifications to test methods.

4.2.

ICST Role in Conformance Testing As Government Agent.

4.2.1. Assist and educate the agencies with regard to

the scope and purpose of the test method as a measure

of product quality.

4.2.2. Assist agencies in developing solicitation

document requirements with regard to product testing

for determining conformance to standards.

4.2.3. Identify publicity and announcement sources for

informing national and international industry,

Government and product users on the availability of

conformance test methods and validation services.

4.2.4.

Work with other government agencies, as

required, to incorporate requirements within the

procurement regulations (i.e., General Services

Administration's Federal Information Resources

Management Regulations) concerning use of conformance

testing and certification of products offered to the

Government

.

4.3. Role of Standards Bodies (National and International).

4.3.1. Consider conformance testing aspects in the

development of standards.

17
4.3.2. Provide advice and give technical committee or

subcommittee recommendations for the approval of a test

method.

4.3.3. Provide technical advice when necessary, on

interpreting a standard under conflict.

4.4. Test Report.

4.4.1. Content.

4. 4. 1.1. Includes a complete test report and a

statement citing laboratory accreditation.

4 .4. 1.2. Includes complete identification of

system under test (SUT)

.

4. 4. 1.3. Provides a description/identification of

the test environment, and any incidents contrary

to routine process.

4.4.2. Format. The following guidelines should be

followed when writing the software tools to produce the

test program reports:

4.4.2. 1. The volume of output produced by each

test program should be kept as concise as

possible. Tabular summaries are encouraged.

4. 4. 2. 2. The following details should be given in

the test report for each test: test number, brief

description of the test, reference to the standard

and implementation under test (IUT) , and message

indicating pass/ fail/ inconclusive , or not

executed.

4. 4. 2. 3. Test discrepancy summary complete enough

to repeat the process.

4.4.3. Style for Test Reporting.

4. 4. 3.1. Each test report should provide a summary of the results giving the following

information: number of tests passed, number of

tests failed, number of inconclusive tests, number

of "information only" tests, number of tests not

executed, and total number of tests.

4. 4. 3. 2. Wherever possible, the output of test

programs should be structured in a way which is

18
easy to check both manually and automatically,

e.g., tabular.

4 .4. 3. 3. If test programs produce visual output

on a graphics display, then this output should be

kept as simple as possible to ease the checking

process, while still examining all necessary

features.

4. 4. 3. 4. All tests which produce visual output

should be accompanied by a comprehensive check

list in the form of an operator script and a set

of reference samples.

4 .4.3. 5. Particular care and attention should be

paid to the arrangement of the test report,

especially with regard to presentation of the test

data and ease of assimilation by the reader. The

format should be carefully and specifically

designed for each type of test carried out, but

the headings should be standardized as far as

possible.

4.5. Certificate of Validation Format, Content and

Presentation

.

4.5.1. Identifies the standard (s) tested.

4.5.2. Includes a statement of the fact that the

product (s) or service (s) meets the standard (s) or an

applicable portion (s) thereof, clearly identifying that

which is being certified.

4.5.3. Includes a clear reference to the testing

laboratory and test report produced.

4.5.4. States the product name and version tested.

4.5.5. The certificate only applies to the product

version operating in the SUT.

4.5.6. Certificate validity depends on the set of

standards, and the specified period of time.

4.6. Applicability of Certificate of Validation on product

tested. Some of the criteria to consider include:

4.6.1. The system configurations that require testing

for the product for which conformance is claimed.

4.6.2. The number and type of errors discovered during

testing.

19
4.6.3. Use of the validated product under different

operating environments e.g., different hardware

configurations and operating systems.

4.6.4. New software releases.

5. TESTING ADMINISTRATION

5.1.

General Guidelines for Test Method Administration.

5.1.1. Test methods need to be approved by the

Certification Body under technical guidance of the

responsible standards committee.

5.1.2. Test methods for which conformance testing is

offered

adequate

are made available

specified conditions.

to everybody through

5.1.3.

binding

Test methods offered

on conformance testing.

are official for and

5.1.4. The same test methods derived from the

standards specification should

approved testing laboratories.

be used by all ICST

5.1.5. Test method upon which certification is based

is s 5. 1.5.1. acceptable to buyer or governmental

regulator if mandated.

5. 1.5. 2. used in its entirety unless limitations

are fully disclosed.

5. 1.5. 3. is based on a FIPS (which is based on a national or international standard) available to

the public.

5. 1.5.4. suitable for use as a basis for

conformance testing.

5.1.6. Declaration of conformance is indicated by a statement, certificate, label or mark which:

5. 1.6.1. includes information on the product,

standards, producer, and procedures used.

5. 1.6.2. presents clearly the intent of the

declaration.

20
5.2.

Criteria and Procedures for Alternative Testing

Sources

.

5.2.1. Manufacturer's declaration of conformance (Self

testing) . Requirements for Vendor include:

5. 2. 1.1. manufacturer's quality management system

and documentation;

5. 2. 1.2. technically appropriate resources for

testing and inspection? -

5. 2. 1.3. use of a test method approved by the

Certification Body;

5. 2. 1.4. retention of complaint records against

product or service? and

5. 2. 1.5. any dispute resolution is in accordance

with the Certification Body for that standard.

5.2.2. Third-party testing. Requirements include:

5. 2. 2.1. products to be tested are fully

identified.

5. 2. 2. 2. procedures by which the laboratory is

operated include provision for:

5. 2. 2. 2.1. fiscally responsible management

and appropriately trained staff.

5. 2. 2. 2. 2. participation on a nondiscriminatory basis.

5. 2. 2. 2. 3. resource commitment to both

initial and continuing validation activities.

5. 2. 2. 2. 4. selection and retention of

qualified testing and inspection services.

5. 2. 2. 2. 5. notification of participants on

changes in standards and procedures.

5. 2. 2. 2. 6. confidentiality of proprietary

information.

5. 2. 2. 2. 7. maintenance of records.

5. 2. 2. 2. 8. safeguarding use of official

declaration of conformance.

21
5 . 2 e 2 . 3 . Documentation required by the laboratory

includes:

5 .2. 2 .3.1. availability of published program

directory listing products, standards,

licensees, and other parties.

5.2. 2. 3. 2. file of legally binding

agreements for licensees.

5.2.2. 3.3-. availability of published

statement of operating procedures.

5.2.2

.4.

Approved by Certification Body.

5.2.3.

Government-managed testing. The same criteria

and procedures for third-party testing would apply.

5.3. Associated Costs.

Identify costs associated with developing and maintaining

conformance test methods and responsible source of funding.

Typically costs associated with conformance testing will be

on a cost-reimbursable basis.

5.4. Resource and overhead requirements. Examine:

5.4 .1. Where and what type of facilities are required

to perform validations. Considerations of competence,

impartiality, and integrity are fundamental to the

acceptability of the testing laboratory. Clients of

the testing services must be guided by this fact in

making the choice of testing laboratories that they

employ.

5 . 4 o 2 . What staff, equipment and office facilities are

required to support a validation service.

5.4.3. What are the retention and storage requirements

for validation materials, project folders and financial

accounting records.

5.4.4. What is the anticipated volume of Clients and

tests conducted over a year.

5.5. Criteria and Procedures for Requesting Validation

Services. This section presents typical procedures which

could be used for requesting and scheduling validations, and

the table below provides only example timeframes associated

with first time validation. These times will vary according

to complexity of tests requested, and number of tests being

performed.

22
SCHEDULE EXPECTATIONS FOR VALIDATION

T - 120

T - 90

T - 60

T - 30

TEST

T + 4~7

T + 60

Contact approved testing laboratory,

Negotiate agreement; vendor testing

Submit disputed tests, if any

Prevalidation test information/material

received by the testing laboratory

Resolve all outstanding issues

Laboratory testing

Testing complete

Final Conformance Summary Report available

Key

T = Testing Day 1; ' or ' + ' = Before/Beyond Test Start

Table I.

5.5.1.

Requesting validation. The Client provides the

following information/material to the approved testing

laboratory in a Letter of Request:

5. 5. 1.1. desired and alternate month of testing;

5. 5. 1.2. product identification, including

version;

5. 5. 1.3. host and target configurations,

operating systems, program language and other

appropriate information necessary for the testing

laboratory to carry out the test;

5. 5. 1.4. site locations and addresses;

5. 5. 1.5. a point of contact for validation,

including full name, address, and telephone

number ; and

5.5. 1.6. which test(s) the Client desires to

take

.

5.5-. 2. Revalidation requests. The Client requests

23
revalidation by the appropriate testing laboratory.

The request includes:

5. 5. 2.1. date of current certificate of

conformance

.

5c5o2«2. a statement clearly confirming the

computer product which is certified was not

modified from the computer product described in

the Conformance Summary Report.

5.5.3. Revalidating. When a modification which is

beyond continuing maintenance has been made to the

computer product previously certified, it should be

revalidated.

5.5.4c Scheduling Validations. These times apply to

the Client's responsibilities directly with the testing

laboratory; the specific month in which the testing

takes place is established by the laboratory.

5. 5. 4.1. Any prevalidation information/material

should be received 60 days prior to the month of

the scheduled validation.

5. 5. 4.2. For products not previously tested,

request for validation should be submitted at

least 120 days prior to the desired testing month.

Data collection dates should not be established

for first time validations until after a successful review of prevalidation materials has

been made by the testing laboratory.

5.5.4 .3. Requests for changes to an existing test

date should be submitted no later than 60 days

prior to that date.

5. 5. 4. 4. Priorities. Priority is given to the

scheduled annual validations. Requests from

Clients for special, procurement-related

validations are given second priority. Third

priority is given to requests for official

validation of new products. All other requests

are given a lesser priority.

5.5.5. Appeal Process.

5. 5. 5.1. Disputed tests. If a Client believes

any tests are in error, or not appropriate, the

Client contacts the testing laboratory, and

provides associated justification for the dispute.

24
A copy of this letter should be forwarded to the

Certification Body as well.

5. 5. 5.

2.

The Client follows the procedures

defined by that Certification Body which approved

the testing laboratory for any appeals over

disputed tests or disputed validation testing

decisions by the Accredited Testing Laboratory.

6 . TESTING LABORATORIES

6.1. Conditions for Accreditation. The conditions for a

laboratory to become accredited and maintain accreditation

depends on the testing requirements. Laboratory

accreditation may require the Testing Laboratory tot

6.1.1. be assessed and evaluated initially and on a

periodic basis.

6.1.2. demonstrate, on request, that it is able to

perform the tests representative of those for which it

is seeking accreditation.

6.1.3. pay any relevant fees.

6.1.4. participate in proficiency testing as required.

6.1.5. be capable of performing the tests for which it

is accredited according to the latest version of the

test method within one year after its publication or

within the time limit specified by the Certification

Body.

6.1.6. limit the representation of the scope of its

accreditation to only those tests or services for which

accreditation is granted.

6.1.7. limit all its test work or services for Clients

to available capacity.

6.1.8. limit advertising of its accredited status to

letterheads, brochures, test reports, and professional,

technical, trade or other laboratory services

publications

.

6.1.9. inform its Clients that the laboratory's

accreditation or any of its test reports in no way

constitutes or implies product certification, approval,

or endorsement by ICST.

6.1.10. maintain records of all actions taken in

25
response to testing complaints for a minimum of one

year.

6cl.ll. maintain an independent decisional

relationship between itself and its Clients,

affiliates, or other organizations so that the

laboratory's capacity to render test reports

objectively and without bias is not adversely affected.

6.1.12. report within 30 days any major changes

involving the location, ownership, management

structure, authorized representation, approved

signatories, or facilities of the laboratory.

6.2.

Criteria for Accrediting Testing Laboratories.

The criteria address a laboratory's quality system, staff,

facilities and equipment, test methods and procedures,

records, and test reports. At a minimum following ISO/XEC

Guide 25, the criteria below should be considered in order

for a testing laboratory to be accredited:

6.2.1.

Quality System

6.2.1.L The laboratory should operate an

internal quality assurance program appropriate to

the type, range, and volume of work performed.

The quality assurance program should be documented

in a quality assurance manual which is available

for use by the laboratory staff. The manual

should be maintained relevant and current by a

responsible member of the laboratory staff. A

person or persons having responsibility for

quality assurance within the laboratory should be

designated by the laboratory management and have

direct access to top management.

6.2.

1.2.

The quality assurance manual should

contain information regarding:

6. 2. 1.2.1. the structure of the laboratory

(organizational charts)

?

6.2. 1.2.2. the operational and functional

duties and services pertaining to quality, so

that each person concerned will know the

extent and the limits of his responsibility;

6.2. 1.2. 3. general quality assurance

procedures

;

6. 2. 1.2. 4. quality assurance procedures

specific for each test, as appropriate;

26
6. 2. 1.2. 5. where appropriate, proficiency

testing, use of reference material, etc;

6 .2.1. 2. 6. satisfactory arrangements for

feedback and corrective action whenever

testing discrepancies are detected?

6. 2. 1.2. 7. procedure for dealing with

technical complaints.

6. 2.

1.3.

The quality system should be

systematically and periodically reviewed by, or on

behalf of management to ensure the continued

effectiveness of the arrangements, and corrective

action initiated. Such reviews should be recorded

together with details of any corrective action

taken

.

NOTE: In small laboratories, the quality system

may fulfill the requirements of this clause in a

simplified way.

6.2.2. Staff

6. 2. 2.1. Staff should have the necessary

education, training, technical knowledge and

experience for their assigned functions.

6. 2. 2. 2. There should be a job description for

each senior technical position category, which

includes the necessary education, training,

technical knowledge and experience.

6 . 2 . 2 . 3

.

The proportion of supervisory to nonsupervisory staff should be such as to ensure

adequate supervision.

6. 2. 2. 4. Suitable staff should be nominated to

deputize for the senior technical and quality

system management staff in their absence.

6. 2. 2. 5. Information on the relevant

qualifications, training, and experience of the

technical staff should be maintained by the

laboratory.

NOTE: In small laboratories, one person may

fulfill more than one function.

6.2.3. Testing Equipment

27
6. 2. 3.1. The testing laboratory should have

access to all items of equipment required for

correct performance of the tests for which it is

recognized.

6 . 2 . 3 .

2

. All equipment should be properly maintained to ensure protection from

deterioration. Instructions for a proper

maintenance procedure for those items of equipment

which require periodical maintenance should be

available.

6 . 2 o 3 . 3

.

Any item of the equipment which has been

subjected to overloading or mishandling, or which

gives suspect results, or has been shown to be

defective, should be taken out of service and

clearly labelled until it has been repaired and

then shown by test to be performing its function

satisfactorily.

6. 2. 3.

4.

Records should be maintained on each

major item of equipment. Each record should

include:

6.

2.

3. 4.1. the name of the item of

equipment.

6 . 2 . 3 . 4 . 2

.

the manufacturer ' s name and type

identification and serial number.

6. 2. 3. 4. 3. date received and date placed in

service.

6. 2.3.4. 4. current location, where

appropriate.

6. 2. 3. 4. 5. details of maintenance.

6.2.4. Test methods and procedures

6.2.4. 1. The testing laboratory should have

adequate documented instructions on the use and

operation of all relevant equipment, on the

handling and preparation of test items (where

applicable) , and on standard testing techniques,

where the absence of such instructions could

jeopardize the efficacity of the testing process.

All instructions, standards, manuals, and

reference data relevant to the work of the testing

laboratory should be maintained up-to-date and be

readily available to the staff.

28
6. 2. 4.

2.

The testing laboratory should use

methods and procedures required by the

specification against which the test items are to

be tested. The specification should be available

to staff performing the test.

6. 2. 4. 2.1. all manual calculation and data

transfers should be subject to appropriate

checks

.

6 . 2 . 4 . 2 . 2 . where these results are derived

by electronic data processing techniques, the

stability of the system should be such that

the accuracy of the results is not affected.

This implies an ability to detect

malfunctions in the hardware during program

execution and to take appropriate action.

6.2.5. Environment

6. 2.

5.1.

The environment in which the tests are

undertaken should not invalidate the test results

or adversely affect the required accuracy. The

testing premises should be protected as required

from conditions such as excessive temperature,

dust, moisture, steam, vibration, electromagnetic

disturbance and interference, and should be

maintained accordingly. There should be

sufficient space to limit the risk of damage or

danger and to allow operators to make practical

and precise movements. The premises should have

the equipment and energy sources needed for the

testing.

6. 2. 5.

2.

Access to and use of all test areas

should be controlled in a manner appropriate to

their designated purpose, and entry by persons

external to the laboratory should be defined.

6. 2 c 5. 3. Adequate measures should be taken to

ensure good housekeeping in the testing

laboratory.

6.2.6. Records

6. 2.

6.1.

The testing laboratory should maintain a

record system to suit its particular circumstances

and comply with any existing regulations. It

should retain on record all original observations,

calculations and derived data, and the final test

report for an appropriate period as designated.

The records for each test must contain sufficient

29
information to permit satisfactory repetition of

the test.

NOTE: In some countries it may be necessary to

maintain records for a period specified by law.

6 c

2

060 2. All records and test reports should be

held secure and in confidence to the Client,

6.2.7.unless otherwise required by law.

Test reports

60 2. 7.1. The work carried out by the testing

laboratory should be covered by a report which

accurately, clearly, and unambiguously presents

the test results and all other relevant

information.

6 . 2. 7.

2.

Each test record should include at least

the following information:

60

2

. 7. 2 . 1 . name and address of testing

laboratory?

6 . 2 . 7. 2 . 2 . unique identification of report

(such as serial number) , and of each page of

the report (sequential numbering)

?

6.2. 7.2.3. name and address of Client;

6 . 2 . 7. 2 . 4. description and identification of

the test item?

6 . 2. 7. 2 .5. date(s) of performance of test,

as appropriate?

6 . 2 . 7. 2 . 6 . a statement to the effect that

the test results relate only to the items

tested?

6 . 2 . 7. 2 . 7. identification of the test

specification, method, and procedure?

6 . 2 . 7. 2 . 8 . description of sampling

procedure, where relevant?

6 . 2 . 7

.

2 . 9. any deviations, additions to, or

exclusions from the test specification, and

any other information relevant to a specific

test

;

30
6.2.7.2.10. disclosure of any non-standard

test method or procedure utilized?

6.2.7.2.11. derived results, supported by

tables, graphs, sketches, and photographs as

appropriate, and any failures identified?

6.2.7.2.12. a statement on test performance

uncertainty where relevant?

6.2.7.2.13. a signature and title of

person (s) accepting technical responsibility

for the test report and date of issue?

6 c 2 . 7 . 2 . 14 . a statement that the report

should not be reproduced except in full o without the approval of the testing

laboratory.

6 . 2 . 7 . 3

.

Corrections or additions to a test

report after issue should be made only by a

further document suitably marked, e.g.,

"Supplement to test report serial number . . . (or

as otherwise identified)," and should meet the

relevant requirements of the preceding paragraphs.

6.3. Testing Laboratory Assessment. (Figure 1, below, is a

pictorial explanation of the procedures which can be

considered for testing laboratory assessment.)

6.3.1. On-Site Assessment. Before initial

accreditation and about every two years thereafter, an

on-site assessment of each laboratory is conducted to

determine compliance with the criteria. Assessors use

standardized checklists so each laboratory receives a

fair assessment in relation to others? however,

assessors have considerable latitude in judgments about

each laboratory's compliance with the criteria

depending on the unique circumstances of each

laboratory. The assessors are selected and assigned on

the basis of their expertise in the testing techniques

to be reviewed. The time needed to conduct an

assessment varies, but generally one week or less is

the norm. Every effort is made to conduct an

assessment with as little disruption as possible to the

normal operations of the laboratory. The assessors:

6. 3. 1.1. Meet with management and supervisory

personnel responsible for the laboratory's

activities for which accreditation is being sought

to acquaint the individuals involved and to set

the assessment agenda.

31
6 .3. 1.2. Examine the quality system employed by

the laboratory, its major equipment, apparatus,

and facilities

o

6.3»1.3. Assessors thoroughly review the

laboratory's quality manual or equivalent, examine

technician notebooks for records pertaining to the

samples, check sample identification and tracking

procedures, determine whether the appropriate

conditions are maintained, and examine copies of

completed test reports.

6. 3 .1.4. Review records of periodic internal

audits, use of check samples or participation in

round robin testing or other similar programs.

6. 3. 1.5. Review representative records including

competency evaluations for all staff members who

perform the tests, verification records, and

sample control records.

6. 3. 1.6. Observe demonstrations of testing

techniques and discuss them with the technical

personnel to assure their understanding of the

procedures. If possible the history of one or

more samples from receipt to final issuance of

test reports is traced.

At the conclusion of the assessment, an exit briefing

is held to discuss assessment findings with laboratory

management and identify any deficiencies uncovered. A

written summary of all identified deficiencies is left

at the laboratory. Assessment forms and a written

report is submitted to ICST for further evaluation.

The laboratory is asked to respond within 3 0 days of

the date of the exit briefing and provide documentation

or certification that the specific deficiencies have

been corrected or that specific actions are being

taken. Any laboratory applying for initial

accreditation may request a delay in responding.

If any deficiencies are noted at laboratories which are

currently accredited, such deficiencies must be

corrected within 30 days after the exit briefing or the

laboratory may face possible suspension, revocation or

expiration of its accreditation. When test systems are

identified as malfunctioning, they must not be used

until corrective action has been completed. Any

deficiencies noted for corrective action will be

subject to thorough review and verification during

subsequent assessments.

32
V/»

ACCREDITATION PROCESS

Figure 1

6.3.2. Monitoring Visits. In addition to regularly

scheduled assessments, monitoring visits can be made at

any time during the accreditation period. Monitoring

visits serve to verify reported changes in the

laboratory's personnel, facilities, and operations, or

to explore possible reasons for poor performance in

proficiency testing. The scope of a monitoring visit

may range from checking a few designated items to a

complete review. Failure to cooperate with assessors

or their representatives, may be grounds for adverse

accreditation action.

6.3.3. Proficiency Testing. Proficiency testing is an

integral part of the ICST accreditation process. While

the existence of facilities, equipment, and personnel

which satisfy the criteria indicates a laboratory's

overall capability to obtain good results, an analysis

of actual test results for certain test methods is also

necessary to determine if the overall capability does

in fact produce the desired results. A laboratory's

failure to participate fully in the conduct of required

proficiency testing is grounds for adverse

accreditation action.

6.3.4. Evaluation. Evaluation of a laboratory is

conducted by ICST-approved technical experts who review

records on the applicant laboratory and base their

decision on:

6. 3. 4.1. information provided on the application;

33
6.3. 4.2. on-site assessment reports?

6. 3.4.3. actions taken by the laboratory to

correct deficiencies?

6 c 3 . 4 . 4

.

results of proficiency testing? and

6. 3 .4. 5. information from any monitoring visits

performed on the laboratory.

If the technical evaluation reveals additional

deficiencies, written notification describing them will

be made to the laboratory. The laboratory must respond

within 3 0 days of such notification and provide

documentation or certification that the specified

deficiencies have been corrected. Clarification of

some issues may be requested by telephone. All

deficiencies must be corrected before accreditation can

be granted or renewed.

6. 3c 5. Accreditation Actions.

The Laboratory Accreditation Program needs to address

the issues of:

6 0 3. 5.1. Recommendation. When accreditation is

recommended, the recommendation forms the basis

for granting accreditation.

6 .3. 5. 2. Denial. In cases where denial is

recommended, the laboratory is notified of such

action and the reasons for denial provided in

writing.

6. 3. 5. 3. Appeal. When denial has been proposed,

the laboratory may request a hearing, under United

States Code (U.S.C.) 556.

6. 3. 5. 4. Renewal. Accreditation is granted for a given period of time with renewal occurring on the

same anniversary date.

6. 3. 5. 5. Termination. A laboratory may

voluntarily terminate its accreditation by written

request at any time.

6. 3 .5.6. Suspension. If an accredited laboratory

develops problems or deficiencies which are of a

temporary nature, its accreditation may be

suspended until such time as the deficiencies are

resolved.

34
6. 3. 5.

7.

Revocation. In cases where a laboratory

is found to have violated the terms of its

accreditation, the accreditation can be revoked.

The laboratory may, however, be given the option

to voluntarily terminate accreditation.

6.3.6c Public Notification. Accreditation actions and

a directory of accredited laboratories is published on

a routine basis.

6.4. Responsibilities of Testing Laboratories.

6.4.1. Carry out conformance testing and provide test

reports to the Client.

6.4.2. Fulfill contractual commitment to the

Certification Body. *

6.4.3. Record technical test details.

6.4.4. Communicate the approved test method to the

Client for conformance testing preparation.

6.4.5. Treat all test results and documents

confidentially, except those which are explicitly

stated as public.

6.4.6. Comply with all requirements for accreditation.

6.4.7. Comply with existing laws. Accreditation does

not relieve the laboratory of the need to observe and

comply with existing Federal, State, and local

statutes, ordinances, or regulations that may be

applicable to its operation, including consumer

protection and antitrust laws.

6.4.8. Accredited laboratories are encouraged, within

specified limits, to publicize their accredited status.

The major restriction is that advertising must not

imply product certification by ICST or the U.S.

Government. (Laboratories and their Clients may not

reference their accredited status in consumer media, in

product advertising, or on product labels, containers,

or packaging.)

6.5. Role of Certification Body in Sponsoring a Laboratory

Accreditation Program.

6.5.1. Ensure the testing laboratory is accessible to

anyone, including foreign Clients.

35
6.5.2. Ensure a model contract (e.g M order form) is

available to serve as agreement between Client and

testing laboratory.

6.5.3. Ensure that accredited laboratories are

available to meet Client needs. Any Client requesting

testing for a given standard, should be able to have

his product tested within one year after his letter of

request to an Accredited Testing Laboratory.

6.5.4. Establish standard-specific criteria for

assessing competence of laboratories to perform

testing. (Information contained in this document should

assist in the generic aspects of assessment plan

development.) On-site assessment can be carried out by

the Certification Body or by independent assessors

approved by 'the Certification Body.

6.5.5. Construct the test method management plan with

enough flexibility to accommodate technological

developments

.

6.5.6. Ensure qualified technical experts, assessors

and evaluators are used for accrediting a testing

laboratory.

6.5.7. Ensure there are procedures for recommending

the laboratory either be granted or denied

accreditation. This recommendation is based on a

review of the evaluation and other records to ensure

that all technical, financial, and administrative

obligations have been satisfied.

7. MUTUAL RECOGNITION OF VALIDATION/CERTIFICATION

7.1. Mutual recognition of validation.

Mutual recognition of validation is issuing a national

certificate solely on the grounds of a test report, without

testing anew. The layout of both test reports and

certificates should be harmonized to improve mutuality among

testing laboratories and certification bodies, and to ease

translation.

7.1.1. Examine bilateral agreements on mutual

recognition of certificates which exist for other

conformance testing programs, e.g., language compilers.

7.2. Common Recognition.

To help achieve common recognition (at both the national and

international level) for test methods, interface is

encouraged through:

36
7.2.1.

The certification system to provide the right

for every testing laboratory to attend the testing by

any other laboratory once a year in order to allow

reciprocal observation for the sake of identical

results. 7.2.2.

Any Client worldwide, should be able to address

the Certification Body.

7.3. Considerations and Issues.

Considerations and issues need to be resolved in order to

achieve common recognition of validation test reports for

products offered by multi-national companies who trade with

other countries.

8 . REFERENCES

8.1. FIPSPUB21-2 , COBOL . 1986 March 18.

8.2. FIPSPUB68-1 , Minimal BASIC . 1985 December 24.

8.3. FIPSPUB69-1 , FORTRAN . 1985 December 24.

8.4. FIPSPUB109 , PASCAL . 1985 January 16.

8.5. FIPSPUB119 , Ada . 1985 November 8.

8.6. FIPSPUB120, Graphical Kernel System (GKS^ . 1986 April

18.

8.7. FIPSPUB128 , Computer Graphics Metafile ( CGM) . 1987

March 10.

8.8. FIPSPUB127 , Database Language SOL . 1987 March 10.

8.9. FED-STD-1064 , Telecommunications; General Aspects of

Group 4 Facsimile .

8. 10. FED-STD-1065 , Telecommunications; Facsimile Coding

Schemes and Coding Control Functions for Group 4 Facsimile

Apparatus . 8.11. FI PS PUB 10 0/ FED- STD- 104 1 , Interface Between Data

Terminal Equipment (DTE) and Data Circuit-Terminating

Equipment (DCE) for Operation with Packet-Switched Data

Communications Networks . 1983 July 6.

8.12. FIPSPUB122 , Conformance Tests for FIPS PUB 100/FEDSTD 1041 Version of CCITT 1980 Recommendation X.25,

Interface Between Data Terminal Equipment (DTE) and Data

37
Circuit-Terminating Equipment fDCE^ for Operation with

Packet-Switched Data Communications Networks . 1986 May 28.

8.13. NBS Standard Reference Materials Catalog 1988-89 . NBS

Special Publication 260 , U.S. Department of Commerce,

National Bureau of Standards.

8.14. FIPSPUB46 , Data Encryption Standard . 1977 January 15.

8.15. FIPSPUB113 , Computer Data Authentication . 1985 May

30.

8. 16. Wood, Helen, "Emerging Software Standards?

Opportunity and Challenge, Proceedings of Computer Standards

Conference - 1986", IEEE Computer Society Press, 1986.

8.17. Kruckeberg, F., "Conformance Testing and

Certification for Information Processing Systems", Computers

& Standards 4, North-Holland, pp 21-32, 1985.

8.18. ISO/IEC Guide 2, 1983-07-01.

8.19. ISO/IEC Guide 38, 1983-12-01.

8.20. Certification - Principles and Practice . ISO, Geneva,

1980.

8.21. Berger, Harvey (X4016)

,

The National Voluntary

Laboratory Accreditation Program, American Laboratory . November 1986.

8.22. ISO/TC 97/SC 22/WG 12 N45 - Test Methods for

Programming Languages Processors - Guidelines for Their

Development and Procedures for Their Approval . 8.23. Validation Guide. LSAR ADP Systems . USAMC Materiel

Readiness Support Activity, Lexington, Kentucky 40511-5101,

January 1987.

8.24. ISO/IEC Guide 40, 1983-12-01.

8.25. Institute for Computer Sciences and Technology

Compiler Testing Procedures . NBS/ICST-87/010 , February 1,

1987.

8.26. ISO/IEC Guide 25, 1982-12-15.

8.27. Planning for Software Validation. Verification, and

Testing . NBS Special Publication 500-98.

8.28. An Overview of Computer Software Acceptance Testing . NBS Special Publication 500-136.

38
8.29. XSO/IEC Guide 28, 1982-12-15.

8.30. XSO/IEC Guide 7, 1982-12-15.

8.31. XSO/TC 97/SC 21, Information Processing Systems - OSI

Conformance Testing Methodology and Framework - Part 1;

General Concents . 1986-12-09.

8.32. Adrion, Branstad, & Cherniavsky, "Validation,

Verification, and Testing of Computer Software", Computing

Surveys, Vol. 14, No. 2, June 1982.

39
'

.

*

*

~

.

40
TERMS AND DEFINITIONS

APPENDIX A

41
TERMS AND DEFINITIONS

ACCEPTANCE TESTING. Formal testing conducted to determine whether

a software system satisfies its acceptance criteria and to enable

the customer to determine whether to accept the system. Formal

testing includes the planning and execution of several kinds of

tests, (e.g, f functional, volume, performance tests) to

demonstrate the implemented software satisfies the customer

requirements for the software system.

ACCREDITATION. The formalized initial and continuing acceptance

of a testing laboratory as competent to carry out specific tests.

APPROVED TEST METHOD. An organized system under which, on a uniform and equitable basis, similar products or services may be

certified to specified standards.

ASSESSORS. Selected to conduct an on-site assessment of a

particular laboratory on the basis of how well their individual

experience matches the type of testing to be assessed. The

laboratory has the right to appeal the assignment of an assessor

and may request an alternate.

CERTIFICATE OF CONFORMANCE. A document attesting that a product

or a service is in conformance with specific standards or

technical specifications as determined through use of a specified

test method.

CERTIFICATION. The procedure by which a product (s) or service (s)

becomes certified after passing tests for conformance to a

standard.

CERTIFICATION BODY. An impartial body, governmental or nongovernmental, possessing the necessary competence and reliability

to operate or accredit operation of a certification system, and

in which the interests of all parties concerned with the function

of the system are represented.

CERTIFICATION MARK. The Certification Body's validating sign,

symbol, or letter that identifies a product (s) or service (s) as

being certified.

CERTIFICATION SYSTEM. A system having its own rules of procedure

and management, for carrying out conformance certification.

CERTIFIED. Attested by the manufacturer/vendor under the

procedures of an accredited testing laboratory as satisfying the

requirements of the referenced standard (s)

.

CLIENT. As used in this Plan, Client refers to any organization

or person who employs a testing laboratory for any purpose.

Thus, "Client" can refer to a commercial organization or

42
Certification Body who uses the services of the testing

laboratory. A Client is someone who wants products to be tested.

CONFORMANCE CERTIFICATION. The action of certifying by means of a

certificate of conformance or mark of conformance that a product

or service is in conformance with specific standards or technical

specifications

.

CONFORMANCE SUMMARY REPORT. A document written after conformance

testing which records details about the IUT and the environment

in which it was tested. The overall summary for the test cases

executed, and, for each test case, the test purpose, analysis,

and verdict.

CONFORMANCE TESTING. The testing of a candidate product for the

existence of specific characteristics required by a standard?

testing the extent to which an IUT is a conforming

implementation.

CONFORMANCE WITH STANDARDS OR TECHNICAL SPECIFICATIONS. The

conformance of a product or a service with all the requirements

of specific standards or technical specifications.

CONFORMING IMPLEMENTATION. An implementation under test which is

shown to satisfy the conformance requirements, consistent with

the capabilities and options stated in the test suite pass/fail

documentation

.

CONTROL BOARD. An impartial body, governmental or nongovernmental, possessing the necessary competence and reliability

to approve and maintain proficient test suites for the testing

program, and in which the interests of all parties concerned with

the function of the tests are represented.

ERROR. A failure of a IUT to meet its test specifications.

EVALUATORS. Selected to provide a second opinion, if necessary,

and to review the record including the application, assessment

report, deficiencies, corrections to deficiencies, and

proficiency test results and, based on this record, to recommend

whether accreditation should be granted.

"FAIL” VERDICT. A verdict given when the observed outcome

matches an outcome identified or categorized as "fail" in the

test case specification.

FALSIFICATION TESTING. Method using sample cases that test as

many of the requirements of the standard as are feasible. The

test suite tries to find errors in the implementation. If errors

are found, one can correctly deduce the implementation does not

conform to the standard; however, the absence of errors does not

necessarily imply the converse. The absence of errors implies

43
either that the implementation conforms to the standard o£ that

the test suite was not comprehensive enough to find errors.

Falsification testing can only prove nonconformance. (Proofs of

Correctness Tests prove conformance to the standard.)

FIELD TESTING. Examining the test method and test suite for the

quality necessary to perform a successful conformance test to the

standard.

IMPLEMENTATION UNDER TEST (IUT) . That part of a standard which

is to be studied under testing, which should be an implementation

of one or more characteristics of the standard.

"INCONCLUSIVE" VERDICT. A verdict given when the observed

outcome either matches an outcome identified or categorized as

" inconclusive" in the test case specification or does not match

any identified outcome in the test case specification.

IUT. see Implementation Under Test.

LABORATORY ACCREDITATION. A formal recognition that a testing

laboratory is competent to carry out specific tests or specific

types of tests.

MANUFACTURER'S DECLARATION OF CONFORMANCE. The action by which a manufacturer declares under his sole responsibility, by means of

a 'declaration of conformance,' that the product is in

conformance with designated standards or other technical

specifications, without being under the procedures of a thirdparty certification system. Note: this term is preferred over

self-certification to avoid confusion with the certification

process in general.

NONCONFORMANCE. When a product which has undergone conformance

testing at an accredited laboratory does not pass the designated

number of tests to qualify for conformance to the standard.

"PASS" VERDICT. A verdict given when the observed outcome

matches an outcome identified as "pass" in the test case

specification.

PROFICIENCY TESTING (LABORATORY) . Determination of laboratory

testing performance by means of interlaboratory test comparisons.

PROOF OF CORRECTNESS. The use of techniques of mathematical logic

to infer that a relation between program variables assumed true

at program entry implies that another relation between program

variables holds at program exit.

REFERENCE MATERIAL. A material or substance one or more

properties of which are sufficiently well established to be used

44
for the calibration of an apparatus, the assessment of a

measurement method, or for assigning values to materials.

SEMANTICS. The functional description, it defines precisely what

must be done, but not how it is to be done (as does the syntax) . Generally specified in narrative form using the English language.

SYNTAX. Can consist of verbs in a programming language to access

the function or, in the case of graphics standards, the

"bindings" to existing programming languages to access the

graphics functions in the most natural way for programmers,

depending on the language they are using.

SYSTEM UNDER TEST (SUT) . The computer hardware and/or software

in which the IUT resides.

SUT. see System Under Test.

TECHNICAL EXPERTS. Respected peers in their field used as

assessors and evaluators, and selected through evaluation of

their professional/academic achievements, experience in the field

of testing, management awareness, potential for conflict-ofinterest, and tact in dealing with people.

TEST METHOD. A specified technical procedure to determine one or

more specified characteristics of a material or product. Note 1:

test method includes the test software. Note 2: part of such a

technical procedure may be described by statements in some

programming language. The relevant software package and any

hardware needed (including reference material) is understood to

be part of the test method.

TEST REPORT. A document which presents the test results and other

information relevant to the test.

TEST SUITE. A complete set of tests necessary to perform

conformance testing for an IUT, together with the information for

determining the order in which . they are to be executed when in

their executable form.

TESTING LABORATORY. A laboratory which measures, examines,

tests, calibrates, or otherwise determines the characteristics or

performance of materials or products. Note: "Testing Laboratory"

in the context of this document may mean (1) a body corporate,

(2) one of its subdivisions, (3) the laboratory proper (office

and equipment) , or (4) the testing service functions for a

specific standard.

VALIDATION. Determination of the correctness of the program or

software produced to support a specific standard or set of

standards

.

45
VERDICT If . Statement of "pass," "fail," or "inconclusive”

concerning conformance of an IUT and made with respect to a test

case that has been executed.

VERIFICATION. The demonstration of consistency, completeness, and

correctness of the product software.

46
SAMPLE APPLICATION FOR TESTING LABORATORY

Appendix B

47
SAMPLE APPLICATION FOR TESTING LABORATORY

i 1. Applicant's name and address

s

Telephone No.

s

Telex No.

2 o Test Laboratory3 name and address (if different from

paragraph 1)2

Telephone No.

Telex No.

s

3 o Senior management

3=1. Names and titles of the senior executives of the

Test Laboratory1 and of the testing laboratories

for which recognition is being sought:

3.2. Name and title of the person responsible for the

Quality Management System in the testing

laboratory:

3.3. Name and title of the principal contact nominated

by the testing laboratory, and of his deputy:

3.4. Operating departments of the testing laboratory

for which recognition is being sought. (Show on

a separate sheet to be attached either as a list

or as an organization chart of the Test

Laboratory1

. )

4 . Employees

4

o 1 . Total number in Test Laboratory1 : 4.2. Total number in testing laboratory for which

recognition is being sought:

5 . Equipment

List on a separate sheet the major items of test equipment

available for use in the area for which recognition is being

sought.

6. Test facilities and services

6.1. List on a separate sheet the testing services for

which recognition is being sought, indicating for

3 "Test Laboratory" refers here to the corporate entity

having final authority over the "testing laboratory" seeking

recognition. In some cases this may mean one of the

corporate's subdivisions or the laboratory proper (office and

equipment)

.

48
each service any limits between which it will

operate, and the published specifications against

which the testing will be performed.

6.2. If recognition by other bodies or authorities are

held in the area for which recognition is being

sought, please give details, e.g., subcontractor.

6.3. What type of testing is to be subcontracted in

respect of the recognition being sought?

7 . Other information

7.1. Document, where applicable, how the testing

laboratory may be related to external

organizations or to components within its own

parent organization.

7.2. Give any other information which you consider

could be of assistance to the assessment team (on

a separate sheet if necessary)

.

8. Quality Management Policy

8.1. Are policy and procedures for the operation of

the testing laboratory contained in a document

such as a Quality Manual?

8.2. Has the person responsible for guality management

the responsibility and authority to identify

quality problems and initiate effective

solutions?

8.3. Does the Quality Manual contain procedures for

the supervision of any unqualified staff?

8.4. Is there a prescribed audit procedure for

checking quality management functions?

9 . Work instructions

9.1. Are manuals, work instructions, and regulations

to be used by staff readily available?

9.2. Is there a system for updating, implementing, and

recording changes to these documents?

9.3. Are documents available for each testing

operation?

9.4. Are documents and reference data maintained in an

up-to-date condition?

9.5. Is obsolete data promptly removed from documents,

etc. ?

49
10. Personnel

10.1. Have standards of professional ability, skills,

and job descriptions been prescribed where

necessary?

10.2. Are training methods applied to attain and

maintain skills with due attention to quality

requirements?

11. Test equipment

11.1. Does the Quality Management System specify that

the equipment is of an accuracy compatible with

the testing undertaken?

11.2. Is a record maintained of all test equipment?

12 . Test procedures

12.1. Are test methods and procedures recorded which

are not called up in specifications, manuals,

etc. ?

12.2. Are the environments in which tests are conducted

and results recorded suitable to ensure their

accuracy?

12 o 3 o Do environmental test facilities exist?

12.4. Is there control of access to the testing areas?

12.5. Is there a prescribed system for detecting

deficiencies in testing and their causes, and for

correcting unfavorable trends?

13 . Handling and storage

13.1. Are work and inspection instructions prescribed

and implemented for the handling, storage, and

return to the submitter of software and

equipment?

13.2. Are appropriate storage areas arranged to prevent

deterioration or damage to the products

concerned?

13.3. Are storage areas accessible only to authorized

persons?

14 o Records

14.1.

Is there a prescribed system for recording the

method and results of testing activities?

50
14.2. Are observations and calculations recorded and

stored as to provide a permanent test record?

14.3. Are there arrangements for ensuring that records

are current, complete, accurate, and held

confidential where required?

15 . Test reports

15.1. Do test reports contain all the information

required for such by ICST?

15.2. Is the testing laboratory prepared to make

arrangements to send copies of test reports to

ICST, where required, on a strictly confidential

basis?

IS. Preparedness for assessment

16.1. Are you satisfied that you can meet all the

requirements prescribed herein?

16.2. At what date will the testing laboratory be ready

for assessment?

16.3. Is there any special urgency for assessment?

If so what is the reason?

Applicant ' s name

Signature and title of person authorized to sign for the

Applicant

( S ignature/Title

)

Date

51
APPLICATION FOR ASSESSMENT AS A RECOGNIZED

TESTING LABORATORY

Name of applicant: Name and address of testing

laboratory if different:

Address:

Name of Contact:

Title:

Name of Contact:

Title:

Test or series of tests for which recognition is sought:

The testing laboratory hereby agrees to:

a) conform to the requirements for a recognized testing laboratory;

b) pay any costs connected with assessment and administration in the

pre-recognition stage irrespective of the eventual granting of

recognition.

Signed:

(Title)

Date

Figure 1

52
MODEL CERTIFICATE OF ACCREDITATION

APPENDIX C

53
MODEL CERTIFICATE OF ACCREDITATION

This is to certify the laboratory named herein has been accredited

using the latest official version, version number , of appropriate

standard test method ; therefore the laboratory conforms to the

accreditation criteria until the expiration date shown below on this

certificate. This laboratory must be recertified at least once every

two years, using this certificate date as the anniversary date for reaccreditation. Accreditation must occur at least 30 days prior to the

expiration date of this Certificate or a new Certificate must be issued

by the National Bureau of Standards, Institute for Computer Sciences

and Technology by the expiration date in order to keep the

accreditation of the testing laboratory in good standing and eligible

for inclusion in the Accreditation Directory List for the following

year.

Laboratory Name

Laboratory Address

Point of Contact: Name/Phone

Accreditation for Standard: Title/Number

Test Method Version Number

Accreditation Assessment Report

EXPIRATION DATE

ICST Director Signature
INDEX

acceptance testing (1), (38), (42)

accreditation (1) , (3) , (7) , (10) , (16), (18), (25), (31-36), (38),

(42), (43), (44), (53), (54)

accreditation actions (34) , (35)

accredited testing laboratory (25)

,

(36), (42)

Ada (5), (37)

ANSI (8)

appeal (24), (34), (42)

conformance testing (1-3) , (5-7)

(9-11), (16), (17), (20), (22)

(35), (36), (38), (39), (43-45)

conformance testing program (1)

(2), (6), (9-11), (16)

control board (43)

COS (7)

costs (5), (22), (52)

criteria for accrediting (26)

data management (1) , (6)

application for accreditation (10) declaration of conformance (16) , (20), (21), (44)

BASIC (5), (37)

definitions (3) , (5) , (41) , (42)

CCITT (6), (7), (37)

DES (8)

certificate (3) , (16)

,

(19) , (20) , (24), (36), (42), (43), (53) , (54) document interchange (5)

certification (2), (5) , (10) , documentation (2), (5), (7), (11) , (15-17) , (20-22)

,

(25) , (32) , (13-15), (21), (22), (32), (34) , (34-38), (42-44) (43)

certification body (10) , (15), (16) , EC (6)

(20-22), (25), (35-37), (42) , (43)

environment (9) , (14)

,

(18) , (29) , CGM (6), (37) (43)

Client (16), (23-25), (30), (35-37),

(42), (43)

COBOL (5), (37)

compiler validation (1) , (5)

Computer Graphics Metafile (6) , (37)

conceptual framework (1)

,

(2) , (9)

conditions for accreditation (25)

conformance certification (42) , (43)

evaluation (2) , (9) , (13) , (32-34)

(36), (45)

falsification testing (6)

,

(11)

(43), (44)

FDDI (6)

fees (10), (25)

field testing (10) , (44)

FIPS (1) , (5-9) , (16) , (20) , (37)

FIRMR (5)

55
format (7) , (14) , (15) , (18) , (19)

FORTRAN (5), (37)

GKS (6), (37 ) GOSIP (7)

government (1) , (5) , (7) , (10)

(17), (22) , (35)

graphics (1) , (6) , (ID # (19) , (37)

(45)

Graphics Kernel System (6)

ICST (1), (2), (5-8), (15), (17),

(20), (25), (32), (33), (35), (38),

(51), (54)

IEEE (6), (38)

Integrated Services Digital Network

(7)

networking (6)

notification (21) , (34) , (35)

NTIS (5)

operating systems (5), (20), (23)

OSI (7), (39)

PASCAL (5), (37)

PHIGS (6)

POSIX (5)

proficiency testing (25) , (27)

,

(33), (34), (44)

proof of correctness (11), (44)

quality system (26), (27), (32)

raster compression (6)

IRDS (6)

ISDN (7)

ISO (1), (7), (26), (38), (39)

ITI (7)

recognition of validation (2),

(36), (37)

records (21), (22), (25), (26),

(28-30), (32), (33), (36), (43),

(50), (51)

IUT (18), (43-46)

laboratory accreditation (1) , (7) , (16), (18), (25), (34), (35), (38),

(44)

laboratory assessment (31)

language compiler (1) , (5)

references (2) , (37)

reporting (2) , (15)

,

(18)

requesting validation (22)

,

revalidating (24)

sample application (3), (10)

(48)

(23)

/ (47),

MAC (8) schedule (23)

maintenance (2), (15), (21), (24), security (1), (8)

(28)

SGML (5)

SQL (6) , (37)

SRM (7)

methodology (6), (7), (39)

NBS (1), (7), (38)

56
staff (2), (21), (22), (26-29),

(32), (49)

standards bodies (17)

standards identification (9)

storage media (7)

summary report (23) , (24) , (43)

SUT (18), (19), (45)

terms (2) , (3) , (35) , (41) , (42)

test procedures (11) , (12) , (50)

test reports (25), (26), (30), (32),

(35-37), (51)

test suite (2), (5), (6), (11-15),

(43-45)

test suite structure (13)

testing administration (2) , (20)

testing equipment (27)

testing methods (2)

validation (1) , (2) , (5) , (6) , (8) , (15), (17), (19), (21-25), (36),

(37), (38), (39), (45)

verdict (43) , (44) , (46)

verification (1) , (14) , (32) , (38) , (39), (46)

57
.

.

'
NBS-114A (rev. 2«>c

)

U.S. DEPT. OF COMM.

BIBLIOGRAPHIC DATA

SHEET (See instructions)

4. TITLE AND SUBTITLE

PUBLICATION OR

REPORT NO.

NB SIR 88-3768

2. Performing Organ. Report NoJ 3. Publication Date

MAY 1988

Standards Conformance Testing: Issues and Activities

5. AUTHOR(S)

Sharon J. Kemmerer

6. PERFORMING ORGANIZATION (If joint or other than NBS. see instructions) 7. Contract/Grant No. NATIONAL BUREAU OF STANDARDS

U.S. DEPARTMENT OF COMMERCE

GAITHERSBURG, MD 20899

S. Type of Report & Period Covered

NBSIR, 1980 - 12/1987

9.

SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street. City. State , ZIP)

10.

SUPPLEMENTARY NOTES

| | Document describes a computer program; SF-185, FIPS Software Summary, is attached.

11.

ABSTRACT (A 200-word or less factual summary of most si gnificant information. If document includes a significant

bibliography or literature survey, mention it here)

This NBSIR suggests a process to follow when developing a conformance testing

program for appropriate NBS/ICST Federal Information Processing Standards (FIPS).

These FIPS and the standards they are promoting are implemented into software

products for commercial use. A consistent application of testing processes and

specific test suites will help align commercial products to conform to these

standards. Application of such a testing process will become more critical as Federal agencies begin to request procurement of software implementing these

standards.

12.

KEY WORDS (Six to twelve entries; alph abeti cal order; capitalize only proper names; and separate key words by semicolon s) certificate; certification; certification body; conformance testing; laboratory

accreditation; test method; test procedures; test suite; testing administration;

validation

13. AVAILABILITY 14. NO. OF

PRINTED PAGES

[ X 1 Unlimited 62

] 1 For Official Distribution. Do Not Release to NTIS

r

1 Order From Superintendent of Documents, U.S. Government Printing Office, Washington, D.C.

20402. 15. Price

[~Xj Order From National Technical Information Service (NTIS), Springfield, VA. 22161 $ 13.95

U SCOMM-DC 6043-P80