# Presentation: 549167

## Toward Credible IT Testing and Certification 

- Rick Kuhn
- National Institute of Standards and Technology
- kuhn@nist.gov
- 301-975-3337

## Outline

- Need for software certification
- Measurement in physical science and information technology
- Laboratory accreditation and certification processes
- Current trends and implications

**Notes:**
Two trends coming together:  
	ever-greater dependence on software in all areas of life
	increase in product liability lawsuits

These trends mean a pressing need for ways to understand and manage risks of computer and communication systems

NIST has nearly a century of experience with measurement science in a variety of fields.  Now moving to apply principles of measurement to information technology.

NIST has already established programs for accrediting third-party labs to do certification of software products.

Finally, discuss the impact of the IT industry’s movement toward component-based software, reliance on commercial off-the-shelf software for everything from medical systems to weapons systems, and improvements in software assurance technology.

## Software is Rarely Certified

- One of the stronger software warranties:

**THIS SOFTWARE IS NOT DESIGNED, MANUFACTURED OR  INTENDED FOR USE OR RESALE FOR ON-LINE CONTROL EQUIPMENT IN HAZARDOUS ENVIRONMENTS REQUIRING FAIL-SAFE PERFORMANCE, SUCH AS IN THE OPERATION OF NUCLEAR FACILITIES, AIR TRAFFIC CONTROL, AIRCRAFT NAVIGATION OR AIRCRAFT COMMUNICATION SYSTEMS, LIFE SUPPORT MACHINES, OR WEAPONS SYSTEMS, IN WHICH FAILURE OF THE SOFTWARE COULD LEAD DIRECTLY TO DEATH, PERSONAL INJURY, OR SEVERE PHYSICAL OR ENVIRONMENT DAMAGE.**

**Notes:**
Most software includes a statement that no warranties apply, and that the software may not be useful for any purpose at all

Warnings like this have been found in software for operating systems, programming language compilers, geographic information systems, and other software that provides a foundation for other software that may be highly critical, and whose failure could in fact lead to death, injury, or severe damage

Not everyone can get away with disclaimers.  Someone has to develop software for these critical systems.

## Examples

- U.S.S. Yorktown dead in the water because Windows NT failed to detect divide by zero error
- $500 Million Ariane 5 rocket explodes because guidance software failed to detect invalid parameter setting

**Notes:**
Most software includes a statement that no warranties apply, and that the software may not be useful for any purpose at all

Warnings like this have been found in software for operating systems, programming language compilers, geographic information systems, and other software that provides a foundation for other software that may be highly critical, and whose failure could in fact lead to death, injury, or severe damage

Not everyone can get away with disclaimers.  Someone has to develop software for these critical systems.

## Measurement in Physical Science

- Measurement requires:
- Traceability to a *reference* - e.g., platinum-iridium meter bar (in years past)
- Measurement *method - *
*Statement of uncertainty - *e.g., using statistical variances

## Measurement in Information Technology-- Examples --

- Traceability to a *reference* - e.g., IEEE standard for POSIX kernel, P1003.1
- Measurement *method - *standard test suite
*Statement of uncertainty - *specific configuration and platform tested

## Reference Data Example -Speech Database

## NIAP Testing Process

## NIST ITL Test Competency Centers

## NIST ITL Test Information

## NIST ITL Test Information

## Laboratory Accreditation

*Goal: Promote development of competitive market to:*
- Increase availability of testing services
- through third-party laboratories
- market for testing encourages development of software testing industry
- Reduce cost
- by increasing supply of testing services

## National Voluntary Laboratory Accreditation Program

- Accredits **third-party testing services**, public and private labs
- Works with other national metrology institutes to establish criteria for **mutual recognition of test results**
- Accreditation in: instrument calibration, **computers and electronics**, dosimetry, environmental standards,  fasteners and metals, product testing

## National Voluntary Laboratory Accreditation Program

- Accredits third-party testing services, public and private labs
- NVLAP Computer/Electronics Group
-              Cryptographic Modules Testing
-              FCC Test Methods
-              GOSIP
-              MIL-STD-462 Test Methods
-              POSIX

## NVLAP Computer/Electronics Group

- Cryptographic Modules - hardware and software
- FCC Test Methods
- GOSIP - OSI communications
- MIL-STD-462 Test Methods - electromagnetic emanations (TEMPEST)
- POSIX - IEEE operating system standard

## Conformance Testing

- Reference: specification or standard
- normally no access to source code
- Measurement method: test cases + test configuration + platform
- Uncertainty statement: accounts for limitation to tested platform, options, and test suite

## Trends

- More standard component-based software
- integrating components without source code
- Increased concern about liability
- Y2K is just the beginning
- Increased demand for testing
- need for consensus on prudent testing practices
- need for precise definition of standards

## Implications of Trends

- More standard component-based software
- means usually no access to source code
- Increased concern about liability
- means stronger assurance needed
- Increased demand for testing
- means more efficient test methods needed

## Bottom Line Implications

- Needs for rigorous assurance of standardized software
- precise specifications of components
- specification-based testing using
- formal specifications
- realistic fault models
- statistical methods