# Presentation: 331847

## NIST’s Role in Standards and TestingDecember 10, 2003

**Mark Skall**
**Chief, Software Diagnostics and Conformance Testing Division**
**National Institute of Standards and Technology**
*mark.skall@nist.gov*

*Building Trust and Confidence in Voting Systems** *
*Panel on Specification, Testability, Accreditation and Qualification*

## Today’s Objectives

- Focus on the specification, testing, and  certification processes
- Panel to address: various aspects, give perspectives
- Informative and stimulus for discussion
- Panel guidelines
- Open Discussion
- Challenges and opportunities
- Moving forward together
- NIST’s Role in Standards and Testing
- Overview of standards and testing, an IT perspective

**Notes:**
First, let me tell you about our panel

Short intro of panelists since full bio in the handout
Each speaker will have 15 minutes, including 1-2 questions, as time permits
At the end of the panel presentations, there will be a Q&A and open discussion – hold questions to end

Apply IT principles to voting processs

Open System and Challenges – look at this whole area in a constructive way and decide how to move forward

Make this whole process as good as we can

Continuing dialog

## Background	

- NIST works with industry and Federal agencies to develop standards and tests to improve the quality of software and achieve interoperable solutions
- NIST has many years experience with
- Formal standards organizations and Consortia
- Developing conformance test suites, tools, reference implementations
- Developing validation and certification testing programs

**Notes:**
 . . . And worked with industry organizations to help them set testing programs

Even know what the languages are (COBOL)

## Good Specs are the Key

- Goal is correct, reliable software
- Requirements are captured in a specification
- Spec should be clear, unambiguous, complete, and testable
- Ideal spec would be defined in a formal language – not English

**Notes:**
If there is one message....

How do we get high quality software?  We work with the IT industry.  Objective is to have high-quality, correct (software) implementations.  Need to start with requirements.

Need to capture the requirements in a clear, unambiguous, precise document.  This is a specification (many times the spec is an industry standard).

Must meet requirements

Need to trace back to requirements

Standards are a means to an end
-standards meaningless if not implemented
Standard worthless if not implemented in a consistent, correct way
Need to ability to ensure that implementations adhere to the standard

Specs need to be: clear and concisely written, understood, precise, technically credible, contain unambiguous requirements 

Formal language – can be semi-formal language (XML)

## What Makes a Good Spec

- NIST is working with W3C to define how a spec should be written
- Addresses what a spec should contain with respect to conformance and testability
- Addresses:
- How to define conformance
- How to sub-divide a spec
- Discretionary items
- Extensions
- Test assertions

**Notes:**
NIST and W3C recognized the importance of achieving a good spec so together we hosted a workshop, at NIST, to address how to accomplish this.

As a result NIST co-chairs a W3C WG that is writing guidelines on how to develop high-quality specs and test suites.  In particular, SpecGL addresses .  . .

Discretionary items – choices.  Implementer gets to pick.

Sub-division – levels, profiles or modules.

Investigate whether any of these (or other) concepts apply to voting stds.

Test Assertions – pulling out testable requirements from the spec

All these things affect conformance

## Conformance Testing

- Methodology
- Falsification testing
- Find errors by means of experimentation
- Outcomes
- Show presence of errors not their absence
- Demonstrates non-conformance; can never prove conformance
- Issues
- How much testing is enough?
- How can we produce more tests with less resources?
- Early involvement improves quality of software

**Notes:**
Probe an implementation, trying to break it – design the test suite with possible scenarios that may cause errors.

In our conf. testing activities, we focus on black box testing (by definition)

Black Box Testing – Treating the implementation as a black box.  We can’t see inside of it, but we can communicate with it.

Test developer’s subjective knowledge of testing, specification, and implementations is key. 

Often times, people think of conformance tests as the stick – If you fail the tests, you cannot be certified.  However, we also use the tests as the carrot.  We try to provide the tests early on, before the implementation is developed so that the implementers can use the tests to improve the quality of their software before it gets tested and before it gets to market.

## Moving Towards Trust and Confidence

- Conformance

*Requirements*

- 100%
- non conforming

- ????
- conforming

*Testing activities*

- Specification

- Implementation

**Moving Towards Trust and Confidence**

**Notes:**
Challenge is to move forward in a way to have trust and confidence in our voting system by increasing the likelihood of correct software implementations to as close to 100% as possible.  How do we ensure the integrity of the voting software given the fact that we can never prove the voting software to be correct.

Mission-critical industry – uses f.s. to tighten up specs
Availability of consensus-based test materials, e.g., test requirements, tools, tests, data sets

How can we maximize probability that implementations are 100% correct?

Increase trust and confidence to the point where we can rely on the software?

Need all 3 – interrelated.  Each one is necessary for the other 2 (feedback loop)

In order to do conformance need to read spec and build test materials

In order to build imp, need to read spec and build imp

In each process, feedback loop

Run imp against conformance.  Need conf tests to improve imp.
Need imp to improve conf tests.

Each contributes to other 2.

Need early on so implementations are improved as they’re being built.

## Relevant NIST Efforts

- Automatic Test Generation From Formal Specs
- Automatic Test Generation using XML Technologies
- Software Component Integration Testing
- National Software Reference Library (NSRL) to determine whether software has been altered
- For more information, please contact:
- Mark Skall
- _[mark.skall@nist.gov](mailto:mark.skall@nist.gov)_
- 301-975-3262
- http://www.itl.nist.gov/div897/

**Notes:**
Formal v. semi-formal 
Automatically generates tests for object-oriented software.

Used to identify the COTS software used in voting software