NIST HANDBOOK 150-22 

2008 Edition 

National 

Voluntary

Laboratory 

Accreditation 

Program 

VOTING

SYSTEM

TESTING

Jeffrey Horlick 

Jon Crickenberger 

National Voluntary Laboratory Accreditation Program 

Standards Services Division 

Technology Services 

May 2008 

U.S. Department of Commerce 

Carlos M. Gutierrez, Secretary 

National Institute of Standards and Technology 

James M. Turner, Deputy Director 
NVLAP AND THE NVLAP LOGO 

The term NVLAP and the NVLAP logo are registered 

marks of the Federal Government, which retains 

exclusive rights to control the use thereof. Permission 

to use the term and symbol (NVLAP logo with 

approved caption) is granted to NVLAP-accredited 

laboratories for the limited purpose of announcing 

their accredited status, and for use on reports that 

describe only testing and calibration within the scope 

of accreditation. NVLAP reserves the right to control 

the quality of the use of the NVLAP term, logo, and 

symbol. 
Contents 

Foreword.......................................................................................................................................................v

Introduction..................................................................................................................................................vi

1 General information ..............................................................................................................................1

1.1 Scope..........................................................................................................................................1

1.2 Organization of handbook..........................................................................................................1

1.3 Program description ...................................................................................................................1

1.4 References..................................................................................................................................2

1.5 Terms and definitions.................................................................................................................2

1.6 Program documentation .............................................................................................................4

2 LAP establishment, development and implementation.........................................................................5

3 Accreditation process............................................................................................................................5

3.1 General .......................................................................................................................................5

3.2 Management system review.......................................................................................................5

3.3 On-site assessment .....................................................................................................................6

3.4 Proficiency testing......................................................................................................................7

3.5 NVLAP recommendation to EAC..............................................................................................7

4 Management requirements for accreditation.........................................................................................8

4.1 Organization...............................................................................................................................8

4.2 Management system...................................................................................................................8

4.3 Document control.......................................................................................................................9

4.4 Review of requests, tenders and contracts .................................................................................9

4.5 Subcontracting of tests and calibrations...................................................................................10

4.6 Purchasing services and supplies .............................................................................................10

4.7 Service to the customer............................................................................................................10

4.8 Complaints ...............................................................................................................................10

4.9 Control of nonconforming testing and/or calibration work......................................................11

4.10 Improvement ............................................................................................................................11

4.11 Corrective action ......................................................................................................................11

4.12 Preventive action......................................................................................................................11

4.13 Control of records ....................................................................................................................11

4.14 Internal audits...........................................................................................................................11

4.15 Management reviews ...............................................................................................................12

5 Technical requirements for accreditation............................................................................................12

5.1 General .....................................................................................................................................12

5.2 Personnel..................................................................................................................................12

5.3 Accommodation and environmental conditions.......................................................................13

5.4 Test and calibration methods and method validation...............................................................13

5.5 Equipment ................................................................................................................................14

5.6 Measurement traceability .........................................................................................................14

5.7 Sampling ..................................................................................................................................15

5.8 Handling of test and calibration items......................................................................................15

NIST Handbook 150-22:2008 iii 
5.9 Assuring the quality of test and calibration results..................................................................15

5.10 Reporting the results ................................................................................................................15

6 Additional requirements......................................................................................................................15

iv NIST Handbook 150-22:2008 
Foreword 

The NIST Handbook 150 publication series sets forth the procedures, requirements, and guidance for the 

accreditation of testing and calibration laboratories by the National Voluntary Laboratory Accreditation 

Program (NVLAP). The series is comprised of the following publications: 

• NIST Handbook 150, NVLAP Procedures and General Requirements, which contains the general 

procedures and requirements under which NVLAP operates as an unbiased third-party accreditation 

body; 

• NIST Handbook 150-xx program-specific handbooks, which supplement NIST Handbook 150 by 

providing additional requirements, guidance, and interpretive information applicable to specific 

NVLAP laboratory accreditation programs (LAPs). 

The program-specific handbooks are not standalone documents, but rather are companion documents to 

NIST Handbook 150. They tailor the general criteria found in NIST Handbook 150 to the specific tests, 

calibrations, or types of tests or calibrations covered by a LAP. 

NIST Handbook 150-22, NVLAP Voting System Testing, presents the technical requirements and 

guidance for the accreditation of laboratories under the NVLAP Voting System Testing LAP. The 2007 

edition is based upon the newest editions of ISO/IEC 17025, General requirements for the competence of 

testing and calibration laboratories, and NIST Handbook 150, and supersedes and replaces the 2005 

edition. 

The handbook was written with the participation of technical experts in applicable fields of testing 

concerning voting systems and was approved by NVLAP. 

The body of the handbook has been structured to conform with internationally accepted rules for the 

structure and drafting of standards, where appropriate, to promote ease of use and understanding. 

This handbook is available on the NVLAP web site (http://www.nist.gov/nvlap). 

Questions or comments concerning this handbook should be submitted to NVLAP, National Institute of 

Standards and Technology, 100 Bureau Drive, Stop 2140, Gaithersburg, MD, 20899-2140; phone: 

301-975-4016; fax: 301-926-2884; e-mail: nvlap@nist.gov. 

NIST Handbook 150-22:2008 v 
vi NIST Handbook 150-22:2008 

Introduction 

The Help America Vote Act (HAVA) of 2002 (Public Law 107-252) was signed into law by President 

Bush on October 29, 2002. Section 231 of HAVA requires the Director of NIST to provide for the 

accreditation of laboratories that conduct testing on the hardware and software of voting systems. In 

response to the HAVA, the National Voluntary Laboratory Accreditation Program (NVLAP) has 

established a program for laboratories that test voting systems. 

The Voting System Standards of 2002 (VSS-2002) were approved for release and publication by the 

Federal Election Commission on April 30, 2002. The VSS-2002 provides for entities called “Independent 

Test Authorities” (ITAs): laboratories that test to the requirements of the VSS-2002 and certify products 

that meet the requirements. ITAs were approved by the National Association of State Election Directors. 

Testing is now conducted by NVLAP-accredited Voting System Testing Laboratories (VSTLs). Product 

certification is performed by the U.S. Election Assistance Commission (EAC). 

On December 13, 2005, the EAC unanimously adopted the Voluntary Voting System Guidelines of 2005 

(VVSG-2005). The VVSG-2005 increased security requirements for voting systems and expanded 

access, including the opportunity for disabled individuals to vote privately and independently. The 

VVSG-2005 took effect in December 2007, at which time voting systems could no longer be tested 

against VSS-2002 and all previous national standards became obsolete. 

VSTLs are required to meet the requirements in NIST Handbook 150, the VSS-2002 with HAVA 

requirements, VVSG-2005, and any other criteria deemed necessary by the EAC. VSTLs should be 

familiar with the election and voting communities including the National Association of State Election 

Directors (NASED) and the National Association of Secretaries of State (NASS). 
1 General information 

1.1 Scope 

1.1.1 The purpose of this handbook is to set out procedures, technical requirements, and guidance for 

accreditation of Voting System Testing Laboratories (VSTLs). 

1.1.2 This handbook supplements the procedures and general requirements found in NIST 

Handbook 150. The scope of the Voting System Testing (VST) program is the set of tests contained in 

the Voting System Standards as specified in the Help America Vote Act (HAVA) of 2002 (Public Law 

107-252), VVSG-2005, and any other tests deemed necessary by the U.S. Election Assistance 

Commission (EAC). 

1.1.3 The additional requirements and interpretive comments contained in this handbook make the 

general NVLAP criteria specifically applicable to the Voting System Testing laboratory accreditation 

program (VST LAP). 

1.1.4 The requirements of NIST Handbook 150, this handbook, and the NIST Handbook 150-22 

Checklist are normative (i.e., mandatory) and must be combined to produce the criteria for accreditation 

in the VST LAP. 

1.2 Organization of handbook 

The numbering and titles for first and most second level headings of this handbook match those of NIST 

Handbook 150. Lower level headings are generally specific to the VST LAP. In some cases, upper level 

headings have been included in the document with no additional text. In these cases, refer to NIST 

Handbook 150. 

1.3 Program description 

1.3.1 HAVA states that it is “an act to establish a program to provide funds to States to replace punch 

card voting systems, to establish the Election Assistance Commission to assist in the administration of 

Federal elections and to otherwise provide assistance with the administration of certain Federal election 

laws and programs, to establish minimum election administration standards for States and units of local 

government with responsibility for the administration of Federal elections, and for other purposes.” 

1.3.2 HAVA Section 231 requires EAC and NIST to develop a national program for accrediting voting 

system testing laboratories. On June 23, 2004, NIST published a notice in the Federal Register

announcing the establishment of this program. 

1.3.3 The purpose of the VST LAP is to accredit VSTLs to conduct testing of voting systems and 

components, providing a measure of confidence that such laboratories are capable of performing testing 

to meet the requirements of HAVA. VSTLs provide testing services for core voting requirements and 

may subcontract requirements that are not voting-specific. VSTLs use tests and test methods from VSS2002, VVSG-2005, and other approved test methods, including test methods incorporated by reference 

(e.g., electromagnetic compatibility). 

NIST Handbook 150-22:2008 1 
1.3.4 Laboratories that achieve NVLAP accreditation are recommended by NIST to the EAC for 

designation as EAC-accredited Voting System Test Laboratories (VSTLs). The EAC maintains a list of 

accredited VSTLs to help vendors and elections officials identify resources to fulfill system testing 

requirements. 

1.3.5 EAC-accredited VSTLs test voting systems for conformance with the voluntary voting system 

standards. Laboratory test reports are reviewed by the EAC for compliance with certification 

requirements. 

1.4 References 

The following documents are referenced in this handbook. For dated references, only the edition cited 

applies. For undated references, the latest edition of the referenced document (including any 

amendments) shall apply within one year of publication or within another time limit specified by 

regulations or other requirement documents. 

— NIST Handbook 150, NVLAP Procedures and General Requirements, available at 

 <http://www.nist.gov/nvlap> 

— Help America Vote Act (HAVA) of 2002 (Public Law 107-252), available at <http://www.eac.gov> 

— Voting System Standards of 2002 (VSS-2002), available at 

 <http://ts.nist.gov/Standards/Accreditation/upload/2002VSS.pdf> 

— Voluntary Voting System Guidelines, available at 

 <http://www.eac.gov/voting%20systems/voting-system-certification/2005-vvsg> 

— U. S. EAC Voting System Test Laboratory Accreditation Program Manual, Version 1.0, effective 

January 1, 2007, available at <http://www.eac.gov> 

1.5 Terms and definitions 

The Voting System Testing LAP brings together requirements from a number of different communities. 

Each of these communities may have its own special vocabulary and its own definitions. Where the same 

word is used by more than one community, the laboratory must ensure that all involved parties understand 

which definition the laboratory is using in its contracts, test plans, reports, etc. 

For the purposes of this handbook, the terms and definitions given in NIST Handbook 150; the VSS2002, Volume I, Appendix A; VVSG-2005, Volume I, Appendix A; and the following apply. 

NOTE Any reference to VSS-2002 includes HAVA requirements. 

1.5.1 

authority to do business in the United States 

As a condition of accreditation, all laboratories shall be lawfully entitled or otherwise not prohibited from 

doing business with the United States or its citizens or operating in the United States. 

2 NIST Handbook 150-22:2008 
1.5.2 

Certification Test Report 

Report of results of independent testing of a voting system by an accredited test lab with a 

recommendation to approve the system under test for certification. 

1.5.3 

core voting system tests 

The core voting system tests in the VST LAP include: Technical Data Package review, physical 

configuration audit, source code review, functional configuration audit, system integration test, volume 

tests, telecommunications and security tests. While VSS-2002 and VVSG-2005 include testing for the 

following: electromagnetic compatibility, telecommunications, environmental, electrical, acoustical, and 

cryptographic modules, these test methods are covered through a separate accreditation and are not part of 

the VST accreditation process. 

1.5.4 

Election Assistance Commission (EAC) 

The U.S. Election Assistance Commission (EAC) was established by the Help America Vote Act of 2002 

(HAVA). Central to its role, the Commission serves as a national clearinghouse and resource for 

information and review of procedures with respect to the administration of Federal elections. 

1.5.5 

EAC-accredited laboratory 

A Voting System Testing Laboratory (VSTL) which has the authority to do business in the United States 

and which has been accredited by the EAC per HAVA Section 231. The determination of the authority to 

do business will be determined by the EAC as part of its accreditation program. 

1.5.6 

key personnel 

Laboratory management and technical staff fundamental to the NVLAP accredited testing process. These 

are staff personnel who impact the process in such a way that their loss would negatively affect the 

quality of the testing. 

1.5.7 

letter of intent 

A written agreement between the laboratory and the EAC committing the laboratory to meet the EAC 

requirements. The letter of intent must be sent to the EAC prior to EAC accreditation. 

1.5.8 

National Institute of Standards and Technology (NIST) 

An agency within the U. S. Government’s Department of Commerce tasked with developing, 

maintaining, and disseminating standards within the United States. NIST also conducts research in 

fundamental measurement processes and provides technology services to public and private sector 

organizations. NVLAP is the laboratory accreditation group within NIST. 

1.5.9 

test campaign 

The sum of the work by a VSTL on a single product or system from contract through test plan, conduct of 

testing for each requirement (including hardware, software, and systems), reporting, archiving, and 

responding to issues afterwards. 

NOTE Testing of modified products and systems for re-qualification is a new test campaign. 

NIST Handbook 150-22:2008 3 
1.5.10 

Voluntary Voting System Guidelines 

Voluntary voting system standards developed, adopted, and published by the EAC. The guidelines are 

identified by version number and date. 

1.5.11 

voting system 

The total combination of mechanical, electromechanical or electronic equipment (including the software, 

firmware, and documentation required to program, control, and support the equipment) that is used to 

define ballots; to cast and count votes; to report or display election results; and to maintain and produce 

any audit trail information; and the practices and associated documentation used to identify system 

components and versions of such components; to test the system during its development and 

maintenance; to maintain records of system errors and defects; to determine specific system changes to be 

made to a system after the initial qualification of the system; and to make available any materials to the 

voter (such as notices, instructions, forms or paper ballots). [VVSG-2005, Volume I, Annex A] 

1.5.12 

Voting System Testing Laboratory (VSTL) 

A testing laboratory accredited by NVLAP under the Voting System Testing LAP for core voting system 

tests in the VSS-2002 and VVSG-2005 standards. A VSTL provides attestation to the EAC that a voting 

system conforms to these standards for consideration as a qualified voting system. 

NOTE In the VSS-2002, test laboratories are referred to as Independent Test Authorities (ITAs), test authorities, 

and test agencies. The term Voting System Testing Laboratory includes all test laboratory functions described in 

the VSS-2002 and replaces the other terms.

1.6 Program documentation 

1.6.1 General 

Assessors use NVLAP checklists to ensure that each laboratory receives an assessment comparable to that 

received by others. Checklists assist assessors in documenting the assessment to the NVLAP 

requirements found in NIST Handbook 150, this handbook, and the checklists themselves. Checklists 

contain definitive statements or questions about all aspects of the NVLAP criteria for accreditation, and 

form part of the On-Site Assessment Report (see NIST Handbook 150). The current version of each 

checklist is available on the NVLAP web site <http://www.nist.gov/nvlap>. 

1.6.2 NIST Handbook 150 Checklist 

All NVLAP programs use the NIST Handbook 150 Checklist (formerly called the General Operations 

Checklist), which contains the requirements published in NIST Handbook 150. The checklist items are 

numbered to correspond to clauses 4 and 5 and annexes A and B of NIST Handbook 150. 

1.6.3 NIST Handbook 150-22 Checklist 

The NIST Handbook 150-22 Checklist (also referred to as the VST Program-Specific Checklist) 

addresses the requirements specific to voting system testing given in NIST Handbook 150-22. The 

checklist may contain requirements expressed at a more detailed level than found in this handbook. 

4 NIST Handbook 150-22:2008 
1.6.4 NVLAP Lab Bulletins

NVLAP Lab Bulletins are issued to laboratories and assessors, when needed, to clarify program-specific 

requirements and to provide information about program additions and changes. 

2 LAP establishment, development and implementation 

This clause contains no information additional to that provided in NIST Handbook 150, clause 2. 

3 Accreditation process 

3.1 General 

3.1.1 This clause discusses the assessment and accreditation process for Voting System Testing 

Laboratories. 

3.1.2 An overview of the laboratory accreditation process is provided in NIST Handbook 150, clause 3, 

and includes information pertaining to application for accreditation; on-site assessment; proficiency 

testing; accreditation decision; granting accreditation; renewal of accreditation; changes to scope of 

accreditation; monitoring visits; and suspension, denial, revocation, and voluntary termination of 

accreditation. 

3.1.3 NVLAP may consider a pre-assessment on-site visit to better define the laboratory’s requested 

scope of accreditation. 

3.1.4 Proficiency testing may be required before initial accreditation and periodically thereafter. 

Laboratories will be informed when proficiency testing is required. 

3.2 Management system review 

3.2.1 Prior to applying to NVLAP, the laboratory shall have a fully implemented management system. 

A copy of the quality manual and relevant associated documents will be sent to NVLAP with the 

application forms. 

3.2.2 Prior to an on-site assessment, one or more NVLAP assessors review the documents to ensure 

they cover all aspects of the management system and, if followed, satisfy the requirements in NIST 

Handbook 150 and this handbook. During the review, the assessors may identify nonconformities and 

require changes to the management system so that it meets the requirements. 

3.2.3 It is recommended that the laboratory create a cross-reference document to allow the laboratory 

and the assessors to verify that all NVLAP requirements are addressed in the management system 

documentation. 

NIST Handbook 150-22:2008 5 
3.3 On-site assessment

3.3.1 When the management system review has been completed and nonconformities resolved, 

NVLAP will schedule the on-site assessment. 

3.3.2 A typical NVLAP on-site assessment is conducted by a team of two or three NVLAP assessors 

over a four-day time period. The assessment will normally take place at the main laboratory site. 

3.3.3 The laboratory shall have its facilities and equipment in good working order and be ready for 

examination according to the NVLAP requirements and the laboratory management system manual. 

Efforts will be made to minimize disruption to the normal working routines during the assessment. The 

assessors will need time and workspace to complete assessment documentation during their time at the 

laboratory site. 

3.3.4 The assessors will use the checklists described in 1.6 to ensure that the assessment is complete 

and that all assessors cover the same items at each laboratory. The assessors may request additional 

information in an effort to clarify checklist responses or delve more deeply into a specific issue. 

3.3.5 The activities covered during a typical on-site assessment are described below. The assessor, 

prior to the visit, will provide a specific agenda. 

a) Opening meeting: The assessors meet with laboratory management and supervisory personnel to 

explain the purpose of the on-site assessment and to discuss the schedule for the assessment 

activities. Information provided by the laboratory on its application form may be discussed during 

this meeting. At the discretion of the laboratory manager, other staff members may attend this 

meeting. 

b) Staff interviews: The assessors ask the laboratory manager to assist in arranging times for 

individual interviews with laboratory staff members. The assessors interview staff members filling 

key positions (e.g., laboratory manager and Authorized Representative) and staff members who 

have an effect on the outcome of the testing. It is not necessary for the assessors to talk to all staff 

members; however, they will select staff members representing all aspects of the laboratory. 

 These interviews are conducted to determine if the staff members are properly trained, assigned, 

supervised and technically competent for the tasks assigned to them. The staff are expected to 

know HAVA, the VSS-2002, the VVSG-2005, and the specific technical aspects of the systems that 

the laboratory tests. 

c) Records review: The assessors review laboratory documentation, including the management 

system, quality manual, equipment and maintenance records, record-keeping procedures, testing 

procedures, laboratory test records and reports, personnel competency records, personnel training 

plans and records, procedures for updating pertinent information (e.g., the VSS-2002, VVSG-2005, 

and state regulations), and safeguards for the protection of vendor-sensitive and proprietary 

information. 

 The assessors do not need access to employee information that may be considered sensitive or 

private such as salary, medical information, or performance reviews for work done outside the 

scope of the laboratory’s accreditation. However, this information is often stored together with 

technical information that the assessors need to check (e.g., job descriptions, resumes, and technical 

performance reviews). In these cases, the assessors work with the laboratory to ensure that they are 

6 NIST Handbook 150-22:2008 
able to perform their review without violating individual privacy. At the discretion of the 

laboratory, a member of its human resources department may be present during the review of 

personnel information. 

d) Internal audit and management review: The assessors review and discuss with the laboratory staff 

the laboratory’s internal audit and management review activities. The discussion will include all 

aspects of those activities including the management system procedures, the audit findings, the 

results of the management review, and the actions taken to resolve problems identified. 

e) Demonstrations: Test personnel are requested to demonstrate their competence to perform the test 

methods for which the laboratory is seeking accreditation. The demonstrations include system 

configuration. For tests that cannot be completed during the on-site visit, portions of tests are 

observed. The laboratory should have a voting system available during the on-site visit for the 

purpose of these demonstrations. 

f) Proficiency testing: When applicable, the assessors discuss all aspects of proficiency testing with 

laboratory staff. Test methodology and the records documenting the laboratory’s execution of the 

testing are reviewed and discussed. 

g) On-site assessment report: The assessors prepare an on-site assessment report, which summarizes 

their findings (nonconformities and comments). This report normally consists of the Signature 

Sheet, On-site Assessment Narrative Summary, the NIST Handbook 150 Checklist, and the VST 

Program-Specific Checklist. 

h) Closing meeting: At the end of the on-site assessment, the assessors hold a closing meeting with 

the laboratory manager and staff to discuss the on-site assessment report and the laboratory’s plans 

for resolution of nonconformities. The process for resolving nonconformities is documented in 

NIST Handbook 150. 

 At the conclusion of the discussion, the report is finalized and the assessors and the laboratory’s 

Authorized Representative sign the report. A copy of the complete report is given to the laboratory 

representative. Any disagreements between the laboratory and the assessors are referred to NVLAP. 

3.4 Proficiency testing 

3.4.1 Proficiency testing requirements for this LAP are under development. Proficiency testing may 

include the testing of artifacts and quizzes and written examinations. 

3.4.2 Applicant and accredited laboratories will be informed when proficiency testing is required. 

3.5 NVLAP recommendation to EAC 

3.5.1 NVLAP will accept an application for accreditation from any laboratory. However, HAVA 

permits the EAC to list only “independent, non-Federal” laboratories. 

3.5.2 Once a VSTL has been granted NVLAP accreditation, NVLAP will recommend it to the Director 

of NIST for submission for consideration by the EAC (see HAVA 231 (b)(1)). 

NIST Handbook 150-22:2008 7 
3.5.3 The EAC will determine if the laboratory meets EAC requirements and that a letter of intent has 

been filed before it grants accreditation (see HAVA 231 (b)(2)).

4 Management requirements for accreditation 

4.1 Organization 

4.1.1 The laboratory shall establish and maintain policies and procedures for maintaining laboratory 

impartiality and integrity in the conduct of voting system testing. When conducting testing under HAVA, 

the laboratory policies and procedures shall ensure that: 

a) the laboratory cannot perform both developmental testing and accredited testing of a particular voting 

system or system component; 

b) the laboratory cannot provide consultation or other services to a voting system developer such that the 

independence, or appearance of independence, in the testing of a voting system or system component 

would be compromised. 

4.1.2 The laboratory shall have physical and electronic controls augmented with an explicit policy and 

set of procedures for maintaining separation, both physical and electronic, between the laboratory test 

personnel and laboratory consultants, product developers, system integrators, and others who may have an 

interest in and/or may unduly influence the outcome of the test. 

4.2 Management system 

4.2.1 The controlled version of the laboratory management system documentation may be paper-based 

or computer-based. Version control shall be maintained in either case. If both methods are used, one or 

the other will be identified as a primary source with the other having the status of a copy (e.g., historical, 

archive, working, distribution). 

4.2.2 The following general management system procedures (required, but not limited to) should be 

available for assessor examination prior to the on-site visit (if requested), but in any event shall be part of 

the on-site assessment process: 

a) internal audits and management review; 

b) writing and implementing system procedures; 

c) writing and implementing system instructions; 

d) staff training and individual development plans; 

e) contract review; 

f) staff members who work at home and at alternate work sites outside the laboratory (e.g., 

telecommuting); 

g) referencing NVLAP accreditation and use of the NVLAP symbol. 

8 NIST Handbook 150-22:2008 
4.2.3 The following program-specific procedures (required, but not limited to) should be available for 

assessor examination prior to the on-site visit (if requested), but in any event shall be part of the on-site 

assessment process: 

a) review of the vendor Technical Data Package (VSS-2002, Volume II, Section 2 and VVSG-2005, 

Volume II, Section 2); 

b) selecting the laboratory staff for the certification test campaign; 

c) writing a Certification Test Plan for first-time testing and testing of modified systems (VSS-2002, 

Volume II, Appendix A and VVSG-2005, Volume II, Appendix A); 

d) writing Test Operation Procedures (VSS-2002, Volume II, Appendix A.6.4 and VVSG-2005, Volume 

II, Appendix A.6.4); 

e) conducting testing at a customer’s site (if the laboratory offers such services); 

f) writing a National Certification Test Report (VSS-2002, Volume II, Appendix B and VVSG-2005, 

Volume II, Appendix B); 

g) reviewing the Configuration Management Plan (VSS-2002, Volume II, Section 2.11 and VVSG2005, Volume II, Section 2.11); 

h) ensuring the protection of proprietary information against threats from persons outside the laboratory, 

from visitors to the laboratory, from laboratory personnel without a need to know, and from other 

unauthorized persons; 

i) performing security testing, (VSS-2002, Volume II, Section 6.4 and VVSG-2005, Volume II, 

Section 6.4); 

i) cooperating with the EAC during test campaigns; 

j) witnessing of system build and installation; 

k) matrix cross-referencing the laboratory’s test methods to the voting system standard. Specific test 

methods will be checked for compliance with the standard. 

4.3 Document control 

There are no requirements additional to those set forth in NIST Handbook 150. 

4.4 Review of requests, tenders and contracts 

4.4.1 The procedures for review of contracts shall include procedures to ensure that the customer 

understands that its products and systems must meet the requirements of HAVA, the VSS-2002, VVSG2005, and the EAC. 

NIST Handbook 150-22:2008 9 
4.4.2 The review shall include (but is not limited to): laboratory competencies and resources to provide 

the service, vendor-supplied documentation, tests to be conducted, testing in additional Certification 

Testing, and subcontracting. 

4.4.3 Procedures for the review of requests, tenders, and contracts should include provisions to ensure 

that any state certification testing does not replace or dilute National Certification requirements. 

4.4.4 When conducting a contract review, the VSTL should determine if there are any special or 

changed requirements from the EAC or from state or local election authorities. 

4.5 Subcontracting of tests and calibrations 

4.5.1 Subcontracting of tests is the use of laboratory services outside of the VSTL to perform tests, e.g., 

electromagnetic compatibility testing, environmental testing, shock and vibration testing, and FIPS 140 

validation. It my also include contracting of services for equipment needed to support testing but not part 

of the core test requirements such as test equipment calibration or the monitoring and operation of an 

environmental test chamber to support the 48-hour environment portion of the accuracy and reliability 

tests. The word subcontracting is not used to describe a mechanism by which the laboratory employs 

staff members (see 5.2.7). 

4.5.2 All core voting system testing shall be conducted by a VSTL. If the VSTL subcontracts testing 

for any test within its scope of accreditation, the subcontracted laboratory shall also be an EAC-accredited 

VSTL authorized to do business in the United States. 

4.5.3 Subcontractors for non-core testing do not need to be accredited under the VST LAP. If 

laboratories accredited in another LAP are available for non-core testing, VSTLs shall use accredited 

laboratories. When an accredited laboratory is not available for non-core testing, the VSTL shall conduct 

an audit of the subcontracted laboratory and shall document that the laboratory is competent and qualified 

for use. 

4.5.4 When a VSTL subcontracts to another laboratory, the VSTL is responsible for ensuring that 

setup, configuration, testing, and reporting is competent, appropriate, and conducted by qualified people. 

The VSTL shall ensure that there are no gaps in the knowledge required to conduct the testing. For 

example, a VSTL subcontracting with another laboratory to conduct temperature cycling tests should 

conduct the functional testing itself rather than allowing the subcontractor to do so. The VSTL is 

responsible for ensuring that the entire voting system is properly tested. 

4.6 Purchasing services and supplies 

There are no requirements additional to those set forth in NIST Handbook 150. 

4.7 Service to the customer 

There are no requirements additional to those set forth in NIST Handbook 150. 

4.8 Complaints 

There are no requirements additional to those set forth in NIST Handbook 150. 

10 NIST Handbook 150-22:2008 
4.9 Control of nonconforming testing and/or calibration work 

There are no requirements additional to those set forth in NIST Handbook 150. 

4.10 Improvement 

There are no requirements additional to those set forth in NIST Handbook 150. 

4.11 Corrective action 

There are no requirements additional to those set forth in NIST Handbook 150. 

4.12 Preventive action 

There are no requirements additional to those set forth in NIST Handbook 150. 

4.13 Control of records 

4.13.1 The laboratory shall set policies and procedures on the retention of records that meet the 

requirements of HAVA and the EAC and meet the needs of its customers as agreed in a contract. 

4.13.2 Laboratory records shall be maintained, released, or destroyed in accordance with the laboratory’s 

policy on proprietary information and contractual agreements with customers. 

4.13.3 The Certification Test Report plus the laboratory’s records of the certification test shall contain 

sufficient information to allow repeating, reproducing and/or auditing the entire certification test. 

4.14 Internal audits 

4.14.1 The internal audit shall cover the laboratory management system and the application of the 

management system to all laboratory activities, including compliance with NVLAP, HAVA, VSS-2002, 

VVSG-2005, contractual, laboratory management system, and any additional EAC requirements. 

4.14.2 In the case where only one member of the laboratory staff is competent to conduct a specific 

aspect of a test method, and performing an audit of work in this area would result in that person auditing 

his or her own work, then the audit may be conducted by another staff member. The audit shall cover the 

methodology for that test method and shall include a review of documented procedures and instructions, 

adherence to procedures and instructions, and review of previous audit reports. External experts may also 

be used in these situations. 

4.14.3 The laboratory shall perform at least one complete internal audit of its management system prior 

to the first on-site assessment. 

NIST Handbook 150-22:2008 11 
4.15 Management reviews 

The laboratory shall perform at least one management review prior to the first on-site assessment. 

5 Technical requirements for accreditation 

5.1 General 

The quality manual shall contain, or refer to, documentation that describes and details the laboratory’s 

implementation of procedures covering all of the technical requirements in NIST Handbook 150 and this 

handbook. 

5.2 Personnel 

5.2.1 The laboratory shall maintain a competent administrative and technical staff appropriate for 

testing voting systems to be recognized by the EAC under HAVA. 

5.2.2 The laboratory shall maintain a list of personnel designated to fulfill NVLAP requirements 

including: technical manager, Authorized Representative, Approved Signatories, and team leaders. 

5.2.3 The laboratory shall notify both NVLAP and the EAC within 30 days of any change in key 

personnel. When key personnel are added to the staff, the notification of changes shall include a current 

resume for each new staff member. 

5.2.4 Laboratories shall document the required qualifications for each technical staff position. 

5.2.5 The laboratory shall have documented a detailed description of its training program for new and 

current staff members. Each new staff member shall be trained for assigned duties. The training program 

shall be updated and current staff members shall be retrained when the VSS-2002 and VVSG-2005 

changes, or when the individuals are assigned new responsibilities. 

5.2.6 The laboratory shall review annually the competence of each staff member for each test method 

the staff member is authorized to conduct. A record of the annual review of each staff member shall be 

dated and signed by the supervisor and the employee 

5.2.7 Individuals hired to perform testing activities are sometimes referred to as subcontractors. 

NVLAP does not make a distinction between full-time laboratory employees and individuals hired on a 

contract. NVLAP requires that the VSTL maintain responsibility for and control of any work performed 

within its scope of accreditation. To that end, the VSTL shall ensure all individuals performing testing 

activities satisfy all NVLAP requirements, irrespective of the means by which individuals are 

compensated (e.g., the VSTL shall ensure all test personnel receive proper training and are subject to 

annual performance reviews, etc.). 

5.2.8 The records for each person having an effect on the outcome of the testing shall include: 

a) position description; 

12 NIST Handbook 150-22:2008 
b) resume/bio to match the person to the position; 

c) duties assigned; 

d) annual competence review; 

e) training records and training plans. 

5.2.9 In order to maintain confidentiality and impartiality, the laboratory shall maintain proper 

separation between personnel conducting testing and other personnel inside the laboratory or outside the 

laboratory, but inside the parent organization. 

5.3 Accommodation and environmental conditions 

5.3.1 The laboratory shall have adequate facilities to conduct the voting system testing that it offers. If 

testing activities are conducted at more than one location, all locations shall meet the NVLAP 

requirements. 

5.3.2 A protection system shall be in place to safeguard customer proprietary hardware, software, test 

data, electronic and paper records, and other materials. This system shall protect the proprietary materials 

and information from personnel outside the laboratory, visitors to the laboratory, laboratory personnel 

without a need to know, and other unauthorized persons. 

5.3.3 Laboratories shall have systems (e.g., firewall, intrusion detection) in place to protect internal 

systems from distrusted external entities. The laboratory shall have regularly updated protection for all 

systems against viruses and other malware. 

5.3.4 If the laboratory is conducting multiple, simultaneous tests, it shall maintain a system of 

separation between the products of different customers. This includes the product itself, the test platform, 

peripherals, documentation, electronic media, manuals, testing area, office space, and records. 

5.3.5 If testing activities will be conducted outside of the laboratory, the management system shall 

include procedures for conducting activities at customer sites or other off-site locations. For example, 

procedures may explain how to secure the site, where to store records and documentation, and how to 

control access to the test facility. 

5.3.6 If the laboratory is conducting its tests at a customer site or other location outside the laboratory 

facility, the environment shall conform, as appropriate, to the requirements for a laboratory environment. 

If a customer’s system on which a test is conducted is potentially open to access by unauthorized entities 

during test, the VSTL shall control the test environment. This is to ensure that the systems are in a 

defined state compliant with the requirements for the test before starting to perform testing work and that 

the systems ensure that unauthorized entities do not gain access during testing. 

5.4 Test and calibration methods and method validation 

5.4.1 The test methods for this program are given in the VSS-2002 and VVSG-2005. In the VSS-2002 

and VVSG-2005, there are specified test methods, test methods that require adaptation, and requirements 

for which the laboratory shall have to develop test methods. When the EAC publishes amendments or 

NIST Handbook 150-22:2008 13 
augmentations to the standards or guidelines, the laboratory shall develop procedures for implementation 

of the new requirements. 

5.4.2 Where the laboratory has developed or modified test methods to meet the requirements of the 

VSS-2002 and VVSG-2005, validation of the test methods shall be referenced in the test report. 

5.4.3 For the purposes of achieving product certification under HAVA, laboratories shall comply with 

interpretations of the test methods as provided by the EAC. When exceptions to the testing methodology 

may be necessary for technical reasons, the laboratory shall ask the EAC for an interpretation, the 

customer shall be informed, and details of an interpretation shall be described in the test report. 

5.4.4 As a part of the testing procedure, the laboratory shall describe by whom and how the voting 

system will be configured. If the customer configures any part of the voting system, then the laboratory 

shall verify the configuration, including all software. 

5.4.5 Testing may be conducted at the customer site, the laboratory or another location that is mutually 

agreed to by the laboratory and the customer. When testing activities are conducted outside the 

laboratory, the laboratory shall have additional procedures to ensure the integrity of all tests and recorded 

results. These procedures shall also ensure that the same requirements that apply in the laboratory are 

maintained at the non-laboratory site. 

5.4.6 The laboratory shall clearly identify any test methods included in the test campaign that are 

outside of the laboratory’s scope of accreditation. 

5.5 Equipment 

5.5.1 For the purposes of this section “equipment” is defined as test equipment used in the testing 

process. Test equipment includes software and hardware products or other assessment mechanisms used 

by the laboratory to support the testing of products and systems. 

5.5.2 The laboratory shall document and maintain records on all test equipment used during testing. 

The laboratory shall have procedures to configure and operate all equipment within its control. 

5.5.3 Equipment used during the conduct of testing shall be under configuration control. The 

laboratory shall have procedures to ensure that any equipment used for testing is in a known state prior to 

use for testing. 

5.5.4 Any software test tools shall be validated to be sure that they are accurately testing to the 

standard. They shall also be examined to ensure they do not interfere with the conduct of the test and do 

not modify or impact the integrity of the product under test in any way. VSS-2002 and VVSG-2005 

require the documentation of the test software and supporting hardware in the certification. 

5.6 Measurement traceability 

All developed test methods and tests performed within the test campaign shall be traceable to the VSS2002 and VVSG-2005. This validation shall be documented (e.g., cross-reference matrix). 

14 NIST Handbook 150-22:2008 
NIST Handbook 150-22:2008 15 

5.7 Sampling 

This section does not apply to the VST LAP since testing to the entire standard is required. 

5.8 Handling of test and calibration items 

5.8.1 The laboratory shall maintain separation between and control over the items from different tests, 

to include the product being tested, its platform, peripherals, and all documentation. 

5.8.2 When the product being tested includes software components, the laboratory shall ensure that 

configuration management mechanisms are in place to prevent inadvertent modifications to the software 

components during the testing process. This includes the customer’s software, test tools, and commercial 

off-the-shelf (COTS) software. 

5.9 Assuring the quality of test and calibration results 

The laboratory procedures for test method validation shall include tests for abnormal conditions as well as 

normal operations where the program functionality includes requirements to detect and respond to invalid 

data, operator actions, or hardware malfunctions. 

5.10 Reporting the results 

5.10.1 Reports shall be submitted in the form and by the method specified in VSS-2002 and VVSG2005. Information required to reproduce the test but not included in the Certification Test Report shall be 

kept by the laboratory as part of the testing records. For example, the report shall contain sufficient 

information for state certification officials to identify what testing was completed for the purpose of 

ascertaining what additional testing may be necessary at the state level. 

5.10.2 Reports intended for use only by the customer shall meet customer-laboratory contract 

obligations and be complete, but need not necessarily meet all other requirements. 

5.10.3 The section of a Certification Test Report that meets the VSS-2002 and VVSG-2005 

requirements for a summary or the recommendation section of a test report for a customer shall also meet 

the requirements of NIST Handbook 150 on opinions and interpretations under Reporting the results. 

6 Additional requirements 

There are no additional requirements beyond NIST Handbook 150 and its associated normative annexes, 

and any other normative references previously cited in this handbook.