Conformance Test

An Industry Perspecti

Patrick Curran

Manager, Java Conformance Te

Sun Microsystems

patrick.curran@sun.com

December 10, 2003
Agenda

● Background● What is conformance testing?● The elements of conformance testing● The specification

● Testable and untestable requirements

● The conformance test suite● The reference implementation● Conformance requirements● Areas of concern
Background

● Conformance Test Development manage

Microsystems, responsible for:

● The Java Compatibility Kit (JCK) - the confor

for the Java 2 Platform, Standard Edition (J2S

● 50,000 tests● 200 staff-years of development effort

● Conformance test suites for XML technologie● Conformance tools

● Conformance-testing processes are define

Java Community ProcessSM (see http://ww
Conformance Testing

“Testing, often performed as parformal Certification Program, to that an implementation of a hardwsoftware technology conforms to specification.”
Conformance testing elements

Specification

ConformanceTest Suite

RefeImp

Is the sp

implem

Is the specificationtestable?

Is the test suite correct?Does the RI conform?
Specification

● Must be testable

● Testable assertions will be derived from norm

statements in the specification (“must”, “mus

● Unspecified, ambiguous, implementation-spe

cannot be tested 

● Optional behaviour is discouraged

● Can be tested, but weakens uniformity and in

of implementations

● Public review improves quality
Untestable requirement

“Protection Against Malicious SoftwareVoting systems shall deploy protection agains

forms of threats to which they may be exposand macro viruses, worms,Trojan horses, anVendors shall develop and document the profollowed to ensure that such protection is macurrent status.

” VSS: V1S6.4.2

NOTE: Testing requirements in VSS V2S6.2.4 ( SyIntegration Testing) supplement this specification to test "the ability of the system to detect, prevent, from a broad range of security risks as identified i
Testable requirement

Re and Firmware Installation“After initiation of election day testing, n

code or compilers or assemblers shall or accessible.” VSS: V1S6.4.1
Conformance Test Suite

● Test suite contents:

● Comprehensive set of tests derived from testab

the specification

● Test harness

● Helps to ensure that test runs are repeatable & det

● Documentation

● How to run the test suite● Conformance requirements● Test appeals process

● All implementations must pass the same 
Reference Implementation

● Developed concurrently with spec and te● Verifies that specification is implementa● Enables the test suite to be tested● Serves as Gold Standard against which 

implementations can be measured

● Helps to clarify intent of specification where 

tests are inadequate
Conformance Requirements

● Specify all requirements for conformance 

enough just to pass the tests)

● How to run the tests● What it means to pass● Untested or untestable criteria that must also be 

● Examples

● Don't modify the tests or test harness● All tests must pass in all configurations/modes● Full specification must be implemented even if 

untested
Issues & Concerns (1)

● Broad and ambitious specification

● There's a lot to test!

● Fuzzy spec – much is left to implementers' 

● Difficult to determine what is “good enough” (e.g.● Difficult to measure level of test coverage● Customised functional testing must supplement co

testing● Not possible to create a single, standardized test suite● How to verify that the tests are correct?● How to ensure a level playing-field for implementers
Issues & Concerns (2)

● Human/social processes are very difficult 

● Don't plug holes in the hardware/software spec

defining processes that humans must implemen● We are non-deterministic!

“To prevent alteration of executable code, no softpermanently installed or resident in the system undocumentation states that the jurisdiction must prphysical and procedural environment for the storapreparation, and transportation of the system hardV1S6.4.1

●Inter-State variation in requirements●No reference implementation