# Presentation: 212698

## Conformance TestingAn Industry Perspective

- Patrick Curran
- Manager, Java Conformance Testing
- Sun Microsystems
- patrick.curran@sun.com
- December 10, 2003

## Agenda

-  Background
-  What is conformance testing?
-  The elements of conformance testing
-  The specification
- Testable and untestable requirements
-  The conformance test suite
-  The reference implementation
-  Conformance requirements
-  Areas of concern

## Background

-  Conformance Test Development manager at Sun   Microsystems, responsible for:
- The Java Compatibility Kit (JCK) - the conformance test suite for the Java 2 Platform, Standard Edition (J2SE)
- 50,000 tests
- 200 staff-years of development effort
- Conformance test suites for XML technologies
- Conformance tools
- Conformance-testing processes are defined through the Java Community ProcessSM (see http://www.jcp.org/)

## Conformance Testing

*“**Testing, often performed as part of a formal Certification Program, to verify that an implementation of a hardware or software technology conforms to its specification.”*

## Conformance testing elements

**Specification**

**Conformance**
**Test Suite**

**Reference**
**Implementation**

- Is the specification
-  implementable?

- Is the specification
- testable?

- Is the test suite correct?
- Does the RI conform?

## Specification

-  Must be testable
- Testable assertions will be derived from normative statements in the specification (“must”, “must not”...)
- Unspecified, ambiguous, implementation-specific behaviour cannot be tested
-  Optional behaviour is discouraged
- Can be tested, but weakens uniformity and interoperability of implementations
* **Public review improves quality*

## Untestable requirement

*“**Protection Against Malicious Software** *
*Voting systems shall deploy protection against the many forms of threats to which they may be exposed such as file and macro viruses, worms,Trojan horses, and logic bombs. Vendors shall develop and document the procedures to be followed to ensure that such protection is maintained in a current status.**” **VSS: V1S6.4.2*
*NOTE: Testing requirements in VSS V2S6.2.4 ( System Level Integration Testing) supplement this specification by requiring ITAs to test "the ability of the system to detect, prevent, log, and recover from a broad range of security risks as identified in Section 6..."*

## Testable requirement

*Re and Firmware Installation** *
*“**After initiation of election day testing, no source code or compilers or assemblers shall be resident or accessible.” **VSS: V1S6.4.1*

## Conformance Test Suite

-  Test suite contents:
- Comprehensive set of tests derived from testable assertions within the specification
- Test harness
- Helps to ensure that test runs are repeatable & deterministic
- Documentation
*How to run the test suite*
*Conformance requirements*
*Test appeals process*
* **All implementations must pass the same test suite*

## Reference Implementation

-  Developed concurrently with spec and test suite
-  Verifies that specification is implementable
-  Enables the test suite to be tested
-  Serves as *Gold Standard against which other         implementations can be measured*
*Helps to clarify intent of specification where conformance tests are inadequate** *

## Conformance Requirements

* **Specify all requirements for conformance (it's not enough just to pass the tests)*
- How to run the tests
- What it means to pass
- Untested or untestable criteria that must also be met
-  Examples
- Don't modify the tests or test harness
- All tests must pass in all configurations/modes
- Full specification must be implemented even if features are untested

## Issues & Concerns (1)

* **Broad and ambitious specification*
*There's a lot to test!*
* **Fuzzy spec – m*uch is left to implementers' initiative
- Difficult to determine what is “good enough” (e.g., security)
- Difficult to measure level of test coverage
- Customised functional testing must supplement conformance testing
- Not possible to create a single, standardized test suite
- How to verify that the tests are correct?
- How to ensure a level playing-field for implementers?

## Issues & Concerns (2)

-  Human/social processes are very difficult to test
- Don't plug holes in the hardware/software specification by defining processes that humans must implement
- We are non-deterministic!
- “To prevent alteration of executable code, no software shall be permanently installed or resident in the system unless the system documentation states that the jurisdiction must provide a secure physical and procedural environment for the storage, handling, preparation, and transportation of the system hardware” *VSS V1S6.4.1*
- Inter-State variation in requirements
- No reference implementation