EAC Interim Accreditation checklist 7/20/06
CIBER record

4.3.2.2

4.3.2.3

4.3.3

4.3.3.1

4.3.3.2

4.3.3.3

43.3.4
44

444

b) Master List. Using the ‘Roadmap’ but the ‘Roadmap is limited to product testing
documents and does not include QA and others. Need to develop

The procedure(s) adopted shall ensure that:

a) authorized editions Need to develop

b) periodically reviewed Need to develop

¢C) invalid or obsolete removed/assured against intended use Need to develop
d) obsolete documents retained Need to develop

Uniquely identified Not being done, need to develop

a) date of issue/revision

b) page numbering

c) the total number of pages or mark for end of document
d) issuing authority

Document changes (5)

Review and Approval process. Only for the Qualification Test process but not for all
controlled documents. PPQA, Sec 7-8.2 for covered documents, Need to be expanded.

New or altered text marked or identified. Document and do.
4) If permitted to amend by hand, document and authorize

b) clearly marked, initialed and date. /f authorized, document.
Making changes to electronic records.

Review of requests, tenders and contracts (5)

Procedures for ibid.

QQTP, Sec 3 TDP Review.

QTP, Sec 1.4

Need to develop

a).Requirements known and understood

(VS-4.4.1) Consider HAVA, VSS/VVSG, EAC directives, and,

(VS-4.4.3) if required, specific state requirements and does not circumvent the Federal
standards.

(VS-4.4.4) If involved, check that state requirements are current.

Page 8 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

2.5.2.2 Project Quality Assurance Process (PQAP) ITA Practices quality document

2.5.3

ITA Testing Process

2.5.3.1 Quality Test Process for Voting System Software (QTP), 4/15/05 Governs testing process

3
34
3.2

4

44
4.44

4.1.3

Terms and definitions (2)
Election Assistance Commission (EAC).
Federal Election Commission (FEC).

Management requirements (2)

Organization (2)

The laboratory or the organization of which it is part shall be an entity that can be
held legally responsible

Legal Name: __ CIBER, Inc. Format Note: Legal name is all caps for CIBER
Itis the responsibility of the laboratory to carry out its testing and calibration
activities in such a way as to meet the requirements of this International Standard
and to satisfy the needs of the customer, the regulatory authorities or
organizations providing recognition.

The EAC shall be identified as the organization providing recognition and as the
governing regulatory authority.

Need to make change
Currently, QTP Sec 17. As part of the Test Complaint Procedure.
Sec 1.3.

The management system shall cover work carried out in the laboratory’s
permanent facilities, at site away from its permanent facilities, or in associated

CQMM 1. Indicated that the ITA Practice, CIBER, Inc. shall to define and use their
quality program independently but compliant to the parent CIBER’s Custom Solution
Division’s program within the terms of the ISO/EC 17025 requirements. The actual QA
Manager is specified in the QA policies and procedures as ?

If the laboratory is part of an organization performing activities other than testing
and/or calibration, the responsibilities of key personnel in the organization that
have an involvement or influence on the testing and/or calibration activities of the
laboratory shall be defined in order to identify potential conflicts of interest.

NOTE 1 Where a laboratory Is part of a larger organization, the organizational arrangements
should be such that departments having conflicting interests, such as production,
commercial marketing or financing do not adversely influence the laboratory's compliance
with the requirements of this International Standard.

NOTE 2 If the laboratory wishes to be recognized as a third-party laboratory, it should be
able to demonstrate that it is impartial and that it and its personnel are free from any undue
commercial, financial and other pressures which might influence their technical judgment.
The third-party testing or calibration laboratory should not engage in any activities that
may endanger the trust in its independence of judgment and integrity in relation to its
testing or calibration activities.

See organization charts:
Overall CIBER, Inc.

Page 2 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

b) The CIBER corporate program ensures training of audit personnel. The ITA QA
Representatives

PQAP provides for creation of CARs which the ITA Director must provide for reviews to
include monitor the actions from the CARs.

4.14.2 PQAP 8.2.2 2” paragraph. The CARs created are to trigger corrective action including
involvement of the ITA Practice Director. Need to add notification of customers in writing if
investigations show that laboratory results may have been affected.

4.14.3 The PQAP CARs procedure would appear to satisfy requirement for recorded.

4.14.4 Need to identify procedures for follow up of CARs or, if not defined, develop.

4.15 Management reviews (10)

4.15.1 May be in Corporate

x Date of most recent management review: (the review last year may qualify as a
management review)

* Actions from previous review (CARs)

¢ Reports from third party assessment groups

* Customer Audit (feedback) reports.

* Internal audit reports, including any associated corrective action

« Documented problems arising from lack of procedural adherence

« Results of proficiency testing and any inter-laboratory comparisons.
* Corrective action requests and any preventative actions taken

* Details of customer complaints and feedback

° — Staff training

* Current adequacy of staff, equipment, and facility resources

« Future plans and projections for new work, staff, equipment, and other
requirements.

¢ Summary of annual review and revision activity for all controlled generic
5 Technical requirements (10)
5.1 General (10)
5.1.1 (no comment)

5.1.2

Page 12 of 21


Core Voting System Tests-Ciber/Wyle Ver 01.01
Rev: 12 Jul 06
Visit: 17-21 Jul 06

EAC Technical Supplement Checklist:

Review test lab procedures/standards for the following elements of the VSS 2002 (and
equivalent VVSG 2005).

(W) Wyle
(C) Clber
Core voting system tests:
1 Technical Data Package review,

a Verify that TDP contains required document content and identify vendor’s document
meeting requirements.
(C) Initial TDP Review
(W) Test Procedures, Sect 1.
b Identification of deliverables: Documents or manuals to be delivered to client
for operation, maintenance, and training.
(C) Not identified.
(W) Not identified.
c Terms and references. Unique usage
(C) Need to add
(W) Need to add
d Review of documents for completeness and consistency
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 4
(W) Test Procedures No VSS-2002,
e Quality Assurance plan
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 4 Step 9
(W) Quality Assurance Test Procedure Need reference identification
f Configuration Management
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 4 Step 9. May need to add
attention to identifying EUT for configuration purposes
(W) Configuration Management Test Procedure. Need reference identification
g Review of System release change log
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 4 TDP Step 7, 9
Sec & Source Code reviews.
(W) Test Procedures. May need to add.
h __ Review of vendor tests. Includes but not limited to:
i Readiness Check
ii Operational Status Check
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 8.1 Test Data and Procedure
Preparation. May need to add specifics for validating Readiness Check and Operational
Status Check.
(W) Test Procedures 2.0 PreQualification Tests. Verification at the Polling Place Test
Procedure (Needs document identification).i May need to add specifics for validating
Readiness/Operational Status Check.
Note: Wyle providing validation of the Readiness/Operational Status Check for Ciber.
i Review of prior test lab tests
(C) Section 7. Qualification Previously Qualified Software. May need to expand
(W) Need to add
-----Deliverables----
j TDP Document Trace matrix directory. Matching the document requirements to the
vendor’s document names or titles.
(W) Test Procedures, Sec 1

Page 1 of 8


EAC Interim Accreditation checklist 7/20/06
CIBER record

442
443
4.4.4
4.45
45

4.5.1

). Capability and resources

¢). Have test method or need to develop.

d). Client has to approve

Records of reviews. Need to develop procedures.
Review of subcontract work. Need fo develop
Reporting deviations from contract. Need fo develop
Contract amendments. Need to develop.
Subcontracting of tests and calibrations (6)

(4.4.3) Needs to be accredited for the scope of test. Needs to develop. Currently comply

with existing or in process subcontract.

45.2

4.5.3

4.5.4
46
47
474

47.2

Advise the customer Need to develop procedure Currently complying

Responsible for subcontracting work (NASED Guideline #4) Complying NASED
Guideline #4. Needs to develop

Approved vendor list. Needs to develop and create.

Purchasing services and supplies (6) Does not apply at this time.
Service to the customer (6)

Cooperation with customer but protect other customers confidentiality

(New) Feedback. PQAP, 8.3 and sample survey, including CIBER Policy and

Procedure for processing survey (Internal Customer Satisfaction Surveys.

48

4.8.1

4.8.2
49
4.9.1

Complaints (7)

Laboratory policy and procedure: Q7P, Sec 17. Need to specify that a record needs
to be made and kept.

Make a record. Have a Test Complaint Process Document

Control of nonconforming testing and/er-calibration-werk (7)

Master Services Agreement (CIBER corporate document-standard contract with vendors)
provides some specific guidance but ITA Practices may need to provide additional
procedures to cover the following:

a) Responsibility and authority for managing of non-conforming.

b) Evaluation and initial determination

¢) Immediate corrective action

d) Where necessary, customer notified and work recalled.

Page 9 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

{VS-5.2.3). The laboratory shall notify both accreditation agency and the EAC within 30

5.3

5.3.1

(VS-5.3.1) a) The laboratory shall have adequate faci

5.3.2

5.3.3

days of any change in key personnel. When key personnel are added to the staff,
the notification of changes shall include a current resume for each new staff
member. This requirement is based on direction given under the initial NASED
accreditation and is to be transferred to EAC. Need to develop/update.

Note 1: ‘Key Personnel’ is considered here to be the personnel identified in VS-4.5.2
above.

Note 2: ‘both accreditation agency’ is a residue from NASED as accreditation agency. It
has not been confirmed that the future accreditation agency will require this but this was
statement is extracted and updated from a draft for that agency. For this accreditation,
the accreditation body is EAC.

Accommodation and environmental conditions (12)

a). Laboratory facilities for testing and/or calibration, including but not limited to
energy sources, lighting and environmental conditions, shall be such as to
facilitate correct performance of the tests and/or calibrations. The office is a basic
administrative office with adequate lighting and support. No special needs outside of
environmental testing requirement performed by other labs.

b) The laboratory shall ensure that the environmental conditions do not invalidate
the results or adversely affect the required quality of any measurement. N/A for
base office

Note: Particular care shall be taken when sampling and tests and/or-calibrations
are undertaken at sites other than a permanent laboratory facility.

c) The technical requirements for accommodation and environmental conditions
that can affect the results of tests and calibrations shall be documented. N/A.

ies to conduct the voting
system testing that it offers. This includes facilities for staff training, record
keeping, document storage, and software storage.

b) If testing activities are conducted at more than one location, all locations
shall meet the EAC requirements, and mechanisms shail be in place to
ensure secure communication between all locations.

(VS-5.3.2)

(VS-5.3.3)

a) The laboratory shall monitor, control and record environmental conditions as required
by the relevant specifications, methods and procedures or where they influence the
quality of the results.

b) Tests and calibrations shall be stopped when the environmental conditions jeopardize
the results of the tests and/or calibrations.

Page 15 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

~b Retention times. Implied kept forever. Open question of what retention should be
required but this is a conservative choice.

4.13.2. Technical records (9)

4.13.21 a) QTP, Sec 15 Archive and Qualification Test Artifacts. Observed checkout log
and directory.

b) Found vendor manuals, sample ballots, test ballots, test voting results, hand written
notes, CDs, scripts, weekly status reports and communication with customer. Have not
accumulated enough to exceed available, local storage.

c) Not noticed. Handwritten copies marked but printed copies did not have

4.13.2.2 No test log requirements defined or practices that shows records are complete
and identifiable. May need to develop better practice.

4.13.2.3 a) Very little handwritten/ hardcopy notes.

b) TDP reviews, spreadsheet makes new entries but don’t lock entries to prevent later
changes. Need develop

4.14 Internal audits (9)

4.14.1 a) PQAP 8.2.3 internal audits are performed by a separate corporate office Process and
Product Quality Audit(s) (PPQA Group) with specific training, independence, from
observed activity, and direct top management access.

PPQA, Sec 7.2 PPQA Reviews When: ITA Practice Director requests a project review by
the PPQA at least once each calendar year and the event of changes in staff, scope of
accreditation, facilities, or equipment.

PQAP, 8.1.2 Plan Quality Assurance Events lists ITA Project Audit (yearly), Quality
Assurance Training on staff change, and Accreditation Audit by ITA Accreditation
Committee

Date of last internal audit: (2005)

This program is actually managed outside of the ITA Practice responsibilities. Copies of
the report were not available to the ITA Practice Director. PQAP defines that any
recommendations are reported in CARs. However, ITA Director does not have a log and
reports there were no previous CARs (procedure was created last year and may not have
been in place for last action). May need to request contact with PPQA group. Terry
Debell, Manager Internal Audit and Compliance. 303-267-3820.

x Also have project oriented internal audits by the [TA QA Representatives which perform a
Process Conformance Audit at the end of each Qualification Test. A report is to be
provided to the ITA Director and CARs for any recommendations for deficiencies. There
have been no Process Performance Audits. The Representatives were assigned last
year.

Page 11 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

4.2.6 The roles and responsibilities of technical management and the quality manager,
including their responsibility for ensuring compliance with this International Standard,
shall be defined in the quality manual.

PPQAP 3.1 Explains the roles and separation of QA Representatives to the testing
activity.

4.2.7 (New) Top management shall ensure that the integrity of the management system is
maintained when changes to the management system are planned and implemented.
Need reference from corporate

(VS-4.2.2- ensuring QA Manual considers topics)
a) internal audits and management review;

[PPQA 8.23

b) writing and implementing system procedures;

[_PPQA, 8.2.2.

¢) writing and implementing system instructions;

[Do not currently have instructions at this level.

d) staff training and individual development plans;

[CIBER Policies and Procedures Training Ver 2.3

) contract review;

[Need to develop

f) staff members who work at home and at alternate work sites outside the laboratory (e.g.,
telecommuting);

[Need to specify

g) referencing EAC accreditation and use of the EAC branding.

Modify for reflect EAC rather than NASED. EAC will need to provide further guidance.

(VS-4.2.3) The following program-specific procedures shall be included with the quality
manual when it is submitted as part of the application package:

a) review of the vendor Technical Data Package (VSS-2002, Volume II, Section 2). This
procedure shall include:

QTP, Sec 4 TDP Review. Need to review and update to include the following.

Use in preparing Qualification/National Certification Test Plan. (Ref VSS Vol
11,2.1,See also VI,9.)

Format. Table of content, abstracts, and cross-index against the VSS/VVSG
documentation requirements (Ref: VSS Vol I1,2.1.1.3)

Provisions for placing the TDP in escrow for reference in state certification and
acceptance testing. (Ref: VSS Vol III, 2.1.2)

Note: Completion of the TDP Review includes the validation of user procedures and
operation manuals against the actual equipment.

Note: vendor diagnostics and simulations must be validated.

b) selecting the laboratory staff for a Qualification/National Certification test team;

Page 6 of 21


Core Voting System Tests-Ciber/Wyle Ver 01.01
Rev: 12 Jul 06
Visit: 17-21 Jul 06
----Deliverables-----
h_ Report on tests performed and their results.
(C) QTP Sect 12, Step 7 Prepare anomaly list. May need to include specifics for HAVA
provisional balloting, absentee ballot consolidation, and write-in resolution.

____6 Qualification Test Report
(C) QTR Template (not uniquely identified/ versioned under document control)
(W) QD V-1 Instructions, Procedures, and Certification Reports
a Introduction.
(C) QTR Template Sec 1 (copied supplied is not current.need update).
1.1 Test Agency History and Capability
1.2 Document Overview
(W) Have an electronic copy that is “cut and paste” but not controlled master. This has
been a source of error in the past. Need to develop.
b Qualification Test Background (B2)
i General Information about the qualification test process. (For outside readers
not familiar with the ITA testing).
(C) standard boilerplate text.
(W) standard boilerplate
ii A list and definition of all terms and nomenclature peculiar to the hardware,
the software, or the test report.
(W) QD V-1. Sec 4.0 Terms and Definitions.
c System Identification (B3). This is the test hardware and software used in this test.
(C) QTR Sec 5.4
(W) QD V-1 Sec 4.9 Test Hardware/Software description Sec 4.
i System name and major subcomponents. Sec 3
ii System Version. Sec 3
iii Test support hardware.
(W) Materials required for testing QTR Sec 5.0 (ISO/IEC 17025 5.10.2 f)
(C) QTR Sec 3. Hardware Support
iv Specific documents (deliverables) from the TDP used to support testing
(W) QTR Sec 5.3.
(C) QTR Sec 3 Documentation provided to support testing. Need to specity
which are part of the vendor deliverables.
d System Overview (B4). Describes the voting system in terms of
i its overall design structure,
ji technologies used,
iii processing capacity claimed by the vendor and
iv modes of operation.
v_ (May) include other products that interface with the voting system. Note:
Shall include components necessary to consolidate and produce final results
including telecommunications.
(©) OTR Sec 4
(W) OTR Sec 4
e Qualification Test Results (B5). “This section provides a summary of the results of
the testing process, and indicates any special considerations that affect the
conclusions derived from the test results. This summary includes:
i Acceptability of the system design and construction based on the
performance and software source code review.
(C) QTR Sect 5
(W) QTR Sect 6

Page 6 of 8


EAC Interim Accreditation checklist 7/20/06
CIBER record

(W) QD V-1, 5.4. Publish as full revision with changes marked.

(C) QTP Sec 14. Point 2. Needs to review to include requirements of this checklist
Section 5.10

Page 21 of 21


EAC Interim Accreditation checklist 7/20/08
CIBER record

5.5

5.6

5.7

58
5.8.1

5.8.2
5.8.3
5.8.4
5.9

5.9.1

5.9.2
5.10
5.10.1

5.10.2

Equipment (15). N/A However, review th ese requirements because they may apply and
give guidance for some concerns and issues for working with the vendor supplied
equipment refer 5.5.5. May need to apply to actual equipment under test.

Measurement traceability (17). N/A except as applies to calibrated equipment

Sampling (19). N/A. Program currently does not deal with sampling from manufacturing
production,

Handling of test and ealibratien items (19).

The laboratory shall have procedures for the transportation, receipt, handling, protection,
storage, retention and/or disposal of test and/or calibration items, including all provisions
necessary to protect the integrity of the test or calibration item, and to protect the
interests of the laboratory and the customer.

a-d)

a-b).

a-d)

Assuring the quality of test and- calibration results (20)

(New) Quality control data analyzed.
Reporting the results (20)

General (20)

Test reports and calibration certificates (20)
--a) title

--b) name and address of laboratory

(W) QD V-1, Cover page,

if testing elsewhere, Need to add provisions for test location if different than
company address in scope

(C) QTR template, cover page,

if testing elsewhere, Need to add provisions for test location if different than
company address

-c) unique identification of test report and identification of end of report

(W) QD V-1 job number and use ‘page x of y’ to identify end of report

Page 18 of 21


Core Voting System Tests-Ciber/Wyle Ver 01.01
Rev: 12 Jul 06
Visit: 17-21 Jul 06

i Demonstrate

f Report of results.

(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 5. Note comments about catalogs of
file.

g Witnessed build from verified source code and COTS
(C) The procedures in the current document is being superceded by a revision to be
provided.

3 Physical configuration audit,

a Configuration verification against Configuration Management plan

(C) Performed during final Functional Configuration Test. 222?

(W) Test Procedures 4.2. Hardware Configuration. See comment in TDP area issue.
b Accessibility standards

(C) Expect hardware ITA

(W) Accessibility Test Procedure , VSS Volume I, Section 2.2.7, Common Standards
2.2.7.1 (svf: physical size and position). Needs to provide specification of table height
for item b. See Title 29, CFR, 1910.

Accessibility Test Procedure , VSS Volume I, Section 2.2.7, Common Standards 2.2.7.2
(svf: acoustical and tacile)

c Construction
(W) Design, Construction, and Maintenance Characteristics Test Procedure, VSS Volume
I, Section 3.4.1, Materials, Processes, and Parts.

Section 3.4.2, Durability. This requirement lacks adequate guidance for test method

Section 3.4.3. Reliability.

Section 3.4.4 Maintainability

Section 3.4.4.1 Physical Attributes supporting maintenance.

Section 3.4.4.2 Additional Attributes.

Section 3.4.5. Availibility. Also, VS Test Procedure 6.9, Need to calculate and report
Ai. Under ISO/IEC 17025 procedures this will need to include reporting the basis for the
calculation including assumptions made to create proposed values for some of the factors.

Section 3.4.6. Product Marking.

Section 3.4.7. Workmanship.

VS Test Procedure, 6.10 Product Safety under product safety review to ensure
compliance with UL 60950-1. This includes review of requirements for features
specified under entire section 3.4 in terms of safety concerns excepting possibility
Durability. In Design, Construction, and Maintenance Characteristics Test Procedure,
need to develop and/or reference test method standard (possibly referencing UL 60950-1)

d Validity of operations provided in deliverable manuals

(C) During functional test, need specification of procedure for software operation.
(W) During functional test, need specification of procedure for equipment operation.
e Hardware transportation and storage tests.

(C)

(W) Environmental Control — Transit and Storage Test Procedure, VSS Volume I,
Section 3.2.2.14. Need to develop reference to Operational Status Test to include
validation and repeatability between all the tests. Should reference use of the test for
both pre-test and post-test determination of operability.

f Hardware operational environmental test.

Page 3 of 8


Core Voting System Tests-Ciber/Wyle Ver 01.01
Rev: 12 Jul 06
Visit: 17-21 Jul 06
Use the Requirements of the FECVSS 2002 Trace to Vendor Testing and Technical
Data Package.
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 4
TDP Inventory (Template)
Initial TDP Review Checklist.doc
k Production of formal Test Plan (VSS 2002-Vol II, App A)
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 8.1 Test Data and Procedure
Preparation.
(W) Test Procedures, Sec I and Appendix A (Volume 1, FEC VSS 2002 Functional
Requirements) Note that this does not include Volume II requirements
QD XI-1, Test Control Program includes development Test Plan.

2 Source code review,
Wyle no longer does source code review. All source code review for Wyle testing is
done by Ciber. This constitutes a change in the scope of accreditation for Wyle/Ciber.

a Catalog of source code

(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 5. Need to develop and add.
Currenily produce a master list of all files submitted as part of the source code and
provide with some reports. This list includes source code, make files, .dils and other files
which may or may not be reviewed or relevant

(W) Defer to Ciber for source code review.
b Catalog of compilation environment including COTS components of build

(C) Needs to add. Request copy of new procedure for Witnessed Build which is expected
to address this.
c Determination of changes from prior review
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 5, Step 4. Perform a diff
comparison on files submitted for a change to verify what has changed and checking with
vendor’s change log. May need to specify documenting what files (source and installed)
are changed.
d__ Review for coding conventions and integrity requirements
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 5. Step 6 lists the exceptions
currently identified from the VSS standards.
i Demonstrate
e Review for security
(C) Document No. ITA 2002 QTP R1.0 04-15-0, Sec 5. Only those items currently listed
in VSS 2002 and documented in Step 6 are examined. Source code reviewers are
expected to report any strange code or process they notice that would be considered a
security breach. Current list includes the following with known security implications::
Q: 5.4.2d Unbound area not protected
R: 5.4.2f Case statement with no default area

S: 5.4.29 Possible vote counter overflow Needs attention. This requirement should
expect that an overflow condition is prevented or detected and reported for operator
action.

FF: 5.4.2v If else operator used more than once. Incorrect. Needs adjustment

HH: 4.2.2 Self modifying, Dynamic Loaded, Interpreted code. Needs development

HH: 4.2.2 Unbound Area, Pointer Values, Dynamic Memory unprotected. Needs
development.

Other items under VSS 2002 code review have security implications in terms of features
to aid in detection or to prevent hiding unsecure code.

Although not specifically required by VSS 2002, issues such as hardcoded passwords or
passphrases or ‘backdoors’ should be included and provisions specified to client on how
such issues will be reported or treated. See topic on reporting anomalies.

Page 2 of 8


EAC Interim Accreditation checklist 7/20/06
CIBER record

43

4.3.1

4.3.2

4.3.21

Need to develop.

c) writing a Qualification/National Certification Test Plan for first-time testing and testing of
modified systems (Ref VSS-2002, Volume IJ, Appendix A);

QTP, Sec 8 Qualification Test Plan.

d) writing Test Operation Procedure (Ref VSS-2002, Volume II, Appendix A.6.4);

Need to develop. Currently, CIBER performs the customized tests from knowledge and information
from the TDP review but does not have a reference copy that defines the common process used in all
test campaigns.

€) conducting testing at a customer's site (if the laboratory offers such services);

NOTE: Reference NASED Tech Guide 3
Need to develop.

f) writing a Qualification/National Certification Test Report (VSS-2002, Volume II, Appendix B);
QTP Sect. 14. Need to review and develop.

g) reviewing the Configuration Management Plan (VSS-2002, Volume II, Section 2.11);
OTP Sec 1.5 Configuration Management During Qualification.

h) ensuring the protection of proprietary information against threat from persons outside the
laboratory, from visitors to the laboratory, from laboratory personnel without a need to know, and
from other unauthorized persons;

(contained elsewhere in ISO/IEC 17025)

i) cooperating with the EAC during test campaigns;
Need to update.

j) witnessing of system build and installation. (Vol-2002, Volume I, 9.2.6.4, NASED Tech Guide 3)
Have created a draft form and need to complete and validate. Needs acceptance review and possible
further expansion based on review.

Document control (4)

General (4)

Under CIBER corporate polices, approved master copies of the QA policies and manuals
are stored under a corporate server Sharepoint subdirectory for the individual divisions.
Personnel within the appropriate division and corporate QA management responsibility
have access.

For internal to ITA Practices, they have a process but have not documented the process.
The samples of documents shown lack identification, version identification and other
required features. Need to document and develop futher.

Document approval and issue. (4)

a) Have a working process requiring approval by ITA Practices Director. Need to
document and develop further.

Page 7 of 21


Core Voting System Tests-Ciber/Wyle Ver 01.01

Rev: 12 Jul 06
Visit: 17-21 Jul 06

(W) VS Test Procedures, Sec 4.4.4

g

Appendix Functional Qualification Checklist
(topic) Test Procedure which specifically reference functional requirement.
Verify functional operation against requirements of Vol I, §2 thru §8 (See
Requirements Checklist)
(C) QTP Section 9 Step 10.

Final Report Template.doc, Appendix C. FEC Requirements Relevant to Software
Functional Testing. (undated and not currently used)

Update and use
(W) VS Test Procedure Section 1. Uses checklist (Needs to be updated against
official version VSS).

Verify functional operation against requirements of vendors technical specification
and manuals

(C) QTP Section 9 Step 10. modify checklist (App C) to include vendor specific
requirements

(W) VS Test Procedures, Sect 1 and slightly Sect 4.4.4, part of TDP review
Verify HAVA functional requirements.
(C) Primarily covered under Wyle testing. Need to use as part of system integration
test.
(W) Casting a Ballot, Vol I, Sec 2.4.3.3
Post-voting functions,
---- Deliverables ----
Provide a Requirement matrix showing which tests performed and requirement
satisfied.
(C) Section 9, App C
(Ww)

Report deficiencies encountered and resolutions of deficiencies.

Note: not all deficiencies will result in a recommendation to not certify.
(©) Sect 9 & 10, App C, comment section Verify against official VSS 2002 and use
(W) QD XV-2. Notice of Anomaly.

5 System integration tests,

a

Accuracy. For non-COTS systems, includes 48 environmental operating test.

(C) QTP 13 COTS Functional and Volume Hardware Testing. Step 3

b

ReHabilit:—For non-COTS systems, includes 48 environmental operating test. For
COTS

(C) including testing for multi-feed as part of accuracy test. Need to specify/reference

c

Volume & Stress tests

(C) Need to document. Ciber does perform tests to exercise maximum limits of system
but do not have procedure identified or documented.

d_— Security tests.

(C) Need to document

e (VVSG 2005) Cryptographic

f Telecommunication, as applicable to system design.

(C) Need to document

g System end-to-end of EMS, vote recording, vote tabulation, consolidation, and

canvass reporting.

(C) QTP Sect 12. Final System Level Testing

Page 5 of 8


EAC Interim Accreditation checklist 7/20/06
CIBER record

5.2

5.2.1

5.2.2

i) human factors. QTP Sect 3.2.2Job Description for ITA Practice Director
Software Analyst
TD Specialist
ITA QA Representatives
Configuration Mgr
Test Engineer
ii) accommodation and environmental QTP, Sec 15.1 Archiving.
PPQAP Sec 3.2.1 Facilities and Equipment
lil) test ... methods and method validation. QTP Sec 3 through 17
iv) equipment. Not prepared
v) measurement traceability N/A except as applies to calibrated equipment
vi) sampling N/A except as applies to calibrated equipment
vii) the handling of test and-calibration items. QTP, Sec 3 for TDP, 6 for equipment
Personnel (11)

a) Ensuring qualification of personnel. C/BER Policies & Procedures Training, Ver 2.2
4/1/08. (on Corporate server) general policy.

Source Code Review Qualification Test. Consists of spreadsheet with sample code and
list of items to find. A partial copy of Vol | 4.2.3, and edited Vol Il coding convention
standards. Does not include issues about problem such as integrity and security issues.
Just used to see if basic competency exists

All other is based on experience performing the tests. No training for security, testing
procedures. Corporate training requirements for corporate procedures and quality
program. No formal training on voting requirements such as the VSS, state laws
variations. Such information is acquired through discussion and vendor designs. May
result in problems in critical evaluation of vendor design.

b) Providing supervision during activity where personnel are becoming qualified.

Only one person has been added since creation of office so training has been informal.
He observed and participated with experienced technical staff until deemed ready. Need
documented policy or procedure

a) Documented goals in the form of formal Position Description containing requirements
for Educational Requirements, Professional Certificates, work related experience and
other requirements.

b) CIBER Training also establishes corporate required training. No supplemental

training is defined for the differences required for the ITA Practices under ISOAEC 17025
based standards (new) or division specific.

Page 13 of 21


Election Assistance Commission (EAC)
Interim Accreditation
Independent Test Authorities (ITA)
Assessment Report
CIBER & Wyle

Conducted: Jul 17-22, 2006
Huntsville, AL

Assessor: Steven V. Freeman


Core Voting System Tests-Ciber/Wyle Ver 01.01
Rev: 12 Jul 06
Visit: 17-21 Jul 06
(W) In attachments B through ---, based on relevant standards appropriate for the
specific tests.
(C) No known requirements under current scope of operation. Will need to adopt/ensure
as part of including specific hardware tests.

Page 8 of 8


Core Voting System Tests-Ciber/Wyle Ver 01.01
Rev: 12 Jul 06
Visit: 17-21 Jul 06
ii The degree to which the hardware and software meet the vendor's
specifications and the standards, and the acceptability of the vendor's
technical and user documentation
(C) QTR Sect 5 by subsection
(W) QTR 1.3 Summary
iii General findings on maintainability
(1) Includes notation of specific procedures or activities that are difficult
to perform.
(C) Need to add to template in System Overview
(W) Attach A as a note.
iv Identification and description of any deficiencies that remain uncorrected
after completion of the qualification test
(1) that has caused or is judged to be capable of causing the loss or
corruption of voting data, providing sufficient detail to support a
recommendation to reject the system being tested.
(2) deficiency in compliance with the security requirements,
(3) deficiency in compliance with the accuracy requirements,
(4) deficiency in data retention, and
(5) deficiency audit requirements are fully described);
Note: In practice, vendors will not allow reports to be published if it has
this level of deficiency.
(C) Atend ofeach Appendice. Need to add to QTR Template/procedure
(W) Located after body of report using a standard Notice of Anomaly (NOA)
v_ Recommendations to EAC for approval or rejection
(C) QTR 5.4, Includes summary description of the system configuration to be
certified
(W) QTR 1.3 (Executive Summary) including system configuration to be
certified.
vi Note: Deficiencies that do not result in a loss or corruption of voting data
shall not necessarily be a cause for rejection. (Identified as “anomaly”)
f Appendix Test Operations and Findings (B6)
i Additional details of test results needed to enable understanding of the
conclusions. B. b. Organized to reflect the Qualification Test Plan.
ii Summaries of the results of
(1) hardware examinations,
(2) operating and non-operating hardware tests,
(3) software module tests,
(4) software function tests, and
(5) system-level tests (including
(6) security and
(7) telecommunications tests, and
(8) the results of the Physical and
(9) Functional Configuration Audits)
__g__ Appendix Test Data Analysis (B7)
i summary records of the test data and
ii the details of the analysis. The analysis includes
(1) a comparison of the vendor's hardware and software specifications to
the test data, together with
(2) any mathematical or statistical procedure used for data reduction and
processing.

Page 7 of 8


EAC Interim Accreditation checklist 7/20/08
CIBER record

(W) In QTR Sec 1.3 Summary with standard defined language. Also includes
recommendation about restricted reproduction

(C) Need to add.
5.10.3. Test reports (21)

5.10.3.1
a) deviations from test methods
(C) Needs to establish reference test method
(W) QTR Sec 6 and QD V-1

b) compliance/non-compliance with requirements (covered under QTR standard App B5
item e)i.

c) N/A except under referenced test standards outside the scope of accreditation
d) (See 5.10.5)

€) additional information required (such as additional tests or information for a requested

5.10.3.2 Sampling (N/A to voting system testing)
5.10.4 Calibration certificates (22) (N/A)
5.10.5 Opinions and interpretations (22)

{C) Need to develop

(W) If accepting prior results without retesting in QTR Sect 6. May need expand
criteria.

5.10.6 Testing and-calibration results obtained from subcontractors (23)
a) identify test was done by a sub-contractor
{W) QTR Sect 6. and App A. Procedure defined in QD VIl-1 Approved Vendor list.
(C) Need to specify

5.10.7 Electronic transmission of results (23) (refers to Control of Data in electronic
media)

(C) Need to give results
(W) QD VII. Document Control (reference 5.4.7)
5.10.8 Format of reports and certificates (23) (covered by VSS II, App B requirements)

5.10.9 Amendments to test reports-and-calibration certificates- (23)

Page 20 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

(C) Need to develop. Current standard identification is only title and version number
on cover page.

—d) name and address of customer.

(W) QD V-1 Sect 3 and cover page

(C) QTR template, name is in Sec 3 Need to provide address.
--€) identification of the method used (VS in requirement matrix)
(W) QD V-1 Sect 4.9

(C) Need to develop. Currently do not have e set of test methods/procedures which can
be referenced.

~-f) description of equipment under test
(W) QD V-1 Sec 4.9, QTR Sec 4.

(C) QTR Sec 3 Test Support Hardware
~-g) date of receipt (N/A to voting system)
--h) sampling plan (N/A to voting system)

--i) test results (VSS Vol II, App B5 Test Result Summary), and B6 Appendix of Test
Results and Findings)

(C) QTR Sect 5 Qualification Test Results and subdivided TDP, Source Code Review,
Functional Test. Where are PCA. Appendices divided into TDP (A), Source Code
Review (B), Functional Test (C) including system integration results and should includes
security and should include telecommunications. Where is Witnessed Build?

(W) QTR Sect 6.1 Summary and Attachment A (Functional Req, Matrix includes Sect
2-8 of functional requirements) .

Specific Data in Appendices B- (required) for specific tests
--j) names, functions, and signature or person(s) authorizing test report.

(C) “Prepared by’ line on QTR cover page. Authority to assign is designated in Project
Quality Assurance Process Tailored for ITA Practice. (Page 6 of 16) as Approved
Signatory as specified in Position Description

(W) Listed on cover page per QD V-1 and includes:
Prepared

Approved
Quality Assurance Manager
Release (Department Manager)
--k) Statement that the results relate only to the items tested or calibrated.

Page 19 of 21


Core Voting System Tests-Ciber/Wyle Ver 01.01

Rev: 12 Jul 06
Visit: 17-21 Jul 06
Note: The system integration tests for accuracy and reliability (e.1. and 2. below) are
conducted in conjunction with this test and the final criteria include all components used
to consolidate polling place and jurisdiction results from individual voting machines.
(W) Wyle considers a system crash or "abend” as a failure. Resetting the machine is not
an acceptable recovery. Check against the VSS 2002 shows the section which specified
‘acceptable’ errors is not in the final version and Wyle’s approach is correct. Need to
address the issue of including extended operation of the user interface and not use
exclusive automated testing.
—— & EMC and clectrical test suit. Iftest is submitted from a third-party source

i Verify test lab is accredited by MRP body

ii Verify equipment under test is for same configuration as being certified

iii Verify that operational status check was appropriate
(C) Defer to Wyle
(W) VS Test Procedure 6.5 Test Operations Procedures ~ Electrical
Performance Requirements Test Procedures,
Electrical Power Disturbance
Electromagnetic Radiation, (CFR Part 15, Class B/FCC Part 15 Class B)
Electrostatic Disruption
Electromagnetic Susceptibility
Electrical Fast Transients
Lightning Surge
Conducted RF Immunity
Magnetic Fields Immunity

h Safety inspection.
(Covered under construction)
----Deliverables----

_... i Reports for the hardware, EMC and electrical, and Safety tests and inspections. If
necessary (i.e. from third party source), provide a statement reporting the results of
the verification on the applicability of the reports.

(C) Need to develop. (Tech Guide #32?)
(W) QD V-1. Instructions, Procedures, and Certification Reports
QD VII-1. Supplier Evaluation and Suppliers List. For third-party report.

____ j___ Directory of deliverables, including hardware and software setup and both
application and COTS installed files. (Part of witnessed build documentation)
(C) Need to develop process. Have form and procedures.

4 Functional configuration audit,

—— 4 Functional Requirement matrix against technical specification and manuals
(©) QTP Sec. 4. TDP Review, Step 8 & 9 (second part of the cross-reference matrix
between the VSS designated documents and the vendor identification).
(W) Test Procedures, Sec 1
Use the second part of the Requirements of the FECVSS 2002 Trace to Vendor
Testing and Technical Data Package.
___ Db ___ Test Specifications for functional requirements
(C) QTP Section 9
Need to develop specific test methods. Ciber has common practices/test case for

most of the functional requirements but needs to document for consistency and
repeatability.

Page 4 of 8


EAC Interim Accreditation checklist 7/20/06
CIBER record

(VS-4.9.1) Where necessary, EAC—especially if for accepted report and certified
system.

e) Responsibilty for authorizing resuming of work, if halted.

4.9.2 Where non-conforming work could recur or doubt exists of laboratory compliance
with own policies and procedures, corrective action in 4.11 shall be promptly
followed. Need to develop

4.10 Improvement (7)
>The laboratory shall continually improve the effectiveness of its management
system through the use of the quality policy, quality objectives, audit results,
analysis of data, corrective and preventive actions and management review.

4.11 Corrective action. (8)

4.11.1 General (8)

PPQAP, Sec 8.2 Suggestions, issues and Corrective Action Requests (CARs)

PQAP, requires ITA Practices fo create Corrective Action Log. Not created. Need
procedures

4.11.2 Cause analysis (8)

4.11.3 Selection and implementation of corrective actions. (8)
4.11.4 Monitoring of corrective actions (8}

4.11.5 Additional audits (8)

4.12 Preventive action (8)

4.12.1 a) Handled as discussions within office. Need to develop procedure and management
tracking process.

b) (New) Action plan for identified preventative action
4.12.2 Procedures to initiate and application of controls
4.13 Control of records (9}
4.13.1 General (9)

4.13.11 QITP, Sec 15 Archive and Qualification Test Artifacts. Observed checkout log
and directory

4.13.1.2

~a Storage of files. Not seen were QA records such as audit reports which are stored
with corporate QA.

Page 10 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

5.3.4 Access to and use of areas affecting the quality of the tests and/or calibrations shall be
controlled. The laboratory shall determine the extent of control based on its particular conditions.

(VS-5.3.6) If the laboratory is conducting its tests at a customer site or other location
outside the laboratory facility, the environment shall conform, as appropriate, to
the requirements for a laboratory environment. If a customer's system on which a
test is conducted is potentially open to access by unauthorized entities during
test, the ITA shall control the test environment. This is to ensure that the systems
are in a defined state compliant with the requirements for the test before starting to
perform testing work and that the systems ensure that unauthorized entities do not
gain access during testing. Ref NASED Technical Guideline #4. Draft procedures exit
and are awaiting approval.

5.3.5 Good housekeeping. Observed reasonable office house-keeping.
5.4 Test and-calibration methods and method validation (12)
5.4.1 General (12)

QTP, Sec 1.4 through 10.

a) The laboratory shall use appropriate methods and procedures for all tests and/or
calibrations within its scope.

b) The laboratory shall have instructions on the use and operation of all relevant
equipment, and on the handling and preparation of items for testing and/or calibration, or
both, where the absence of such instructions could jeopardize the results of tests and/or
calibrations. Not widely needed at Ciber. May need to look at such an instruction to
provide the control of operating system setup as an example.

¢) All instructions, standards, manuals and reference data relevant to the work of the
laboratory shall be kept up to date and shall be made readily available to personnel (see
4.3).
d) Deviation from test and calibration methods shall occur only if the deviation has been
documented, technically justified, authorized, and accepted by the customer.
Need to develop.

5.4.2 Selection of methods (13)
Currently limited to broad based QTP for test areas such as TDP Review, Source Code
Review, Functional Testing. Need to develop more specific test procedures that
provide a standard base for testing between vendors.

a) Preferred test methods from international, regional, or national standards. (VS
5.4.1) methods required in VSS/VVSG shall be used.

b) Additional details to supplement standard method.
d) Customer informed and agrees.

e) Shall confirm that it can properly be performed. (See under review of tenders,
etc.)

(VS-5.4.2) Validation of the test method will be included in documentation.

Page 16 of 21


Steven V. Freeman
Attachments:
1. CIBER Organization

a. ITA Organization
b. Corporate Organization


EAC Interim Accreditation checklist 7/20/06
CIBER record

415

ITA Practice.

QTPVS Para 1, introduction
PQAP, Sec 3, pg 4

(VS 4.1.1) Employee can not develop and test a product or otherwise consult for a client and then
test as ITA the client. Need to develop or confirm from CIBER corporate policy.

The laboratory shalt

a) have managerial and technical personnel who, irrespective of other
responsibilities, have the authority and resources needed to carry out their duties,
including the implementation, maintenance and improvement of the management
system, and to identify the occurrence of departures from the management system
or from the procedures for performing tests and/or calibrations, and to initiate
actions to prevent or minimize such departures (see also 5.2);

PQAP, Sec 3, pg 4

b) have arrangements to ensure that its management and personnel are free from
any undue internal and external commercial, financial and other pressures and
influences that may adversely affect the quality of their work;

Need to add. Billing and contracting are done outside ITA Practices and ITA Practices
Director.

c} have policies and procedures to ensure the protection of its customers’
confidential information and proprietary rights, including procedures for protecting
the electronic storage and transmission of results;

PQAP, Sec 3 geographically separated office with their own filing network and file
system
(VS-4.1.2) Covered.

d) have policies and procedures to avoid involvement in any activities that would
diminish confidence in its competence, impartiality, judgement or operational
integrity;

Certification of Laboratory Conditions and Practices for EAC. 12 Jan 05.
QTP Sec?

e) define the organization and management structure of the laboratory, its place in
any parent organization, and the relationships between quality management,
technical operations and support services;

See organization chart

f) specify the responsibility, authority and interrelationships of all personnel who
manage, perform or verify work affecting the quality of the tests and/or
calibrations;

(See also 5.2)
PQAP, ITA organization and the table of Roles and Responsibilities

Page 3 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

4.2.2

4.2.3

See Documents/internal for a list of Quality manuals and documents. Basic organization
is the QA program plans for the administration of the QA program with a separate set of
documents for the QA for the testing activity.

(VS-4.2.1) See Document Control for master copies.

The laboratory's management system policies related to quality, including a quality
policy statement, shall be defined in a quality manual (however named). The
overall objectives shall be established, and shall be reviewed during management
review. The quality policy statement shall be issued under the authority of top
management. It shall include at least the following:

PPQA, Apr 2005 Sec 1 Purpose, authorized under ITA Practices Director and Director of
Federal Systems

PQAP, Sec 1. Purpose

a) the laboratory management's commitment to good professional practice and to
the quality of its testing and calibration in servicing its customers;

b) the management's statement of the laboratory's standard of service;

Should consider strengthening this part of statement more explicitly.

c) the purpose of the management system related to quality;

d) a requirement that all personnel concerned with testing and calibration activities

within the laboratory familiarize themselves with the quality documentation and
implement the policies and procedures in their work; and

Need to add to ITA Practices QA document. All in office are involved in working with the
QA procedures. Currently, CIBER corporate requires each employee to receive and sign
off on a statement.

e) (New) the laboratory management's commitment to comply with this
International Standard and to continually improve the effectiveness of the
management system.

Need to add.

(New) Top management shall provide evidence of commitment to the development and

implementation of the management system and to continually improving its effectiveness.

4.2.4

Need to add or document from corporate

(New) Top management shall communicate to the organization the importance of

meeting customer requirements as well as statutory and regulatory requirements.

425

Need to add or document from corporate

~a) The quality manual shall include or make reference to the supporting procedures
including technical procedures.

PPQAP does identify and reference both PPQA and QTP.

~-b) It shall outline the structure of the documentation used in the management system.
PPQAP 1.3

Page 5 of 21


descriptions of what testing was performed although they don't always indicate where a test was
only done in an earlier version. In a number of reports over the last year, Wyle has indicated that
CIBER is expected to complete certain tests involving the EMS or Reporting systems. CIBER’s
reports provide limited or no descriptions of the testing performed so a reader or reviewer can not
tell if all the testing was completed. Cross checking between CIBER and Wyle reports has
revealed at times that neither ITA has performed certain tests, expecting that the test was done
by the other.

Wyle has a demonstrated capability to do well in the limited scope of hardware testing and some
related functional testing but does not have the internal resources to perform what is being
identified for the new VSTLs as the core requirement testing. With the right partner Wyle could
potentially be a full scale test lab but needs to develop the internal resources to be able to take a
lead in system integration testing and end-to-end functional testing including more aggressive
security testing.

CIBER has not shown the resources to provide a reliable product. The current quality
management plan requires more time to spend on managing the process than they appear to
have available and it was clear during the assessment visit that they had not accepted that they
have a responsibility to provide quality reviewed reports that show what was done in testing. The
ITA Practice Director indicated during the assessment that their difficulties were that corporate
CIBER did not allow for the personnel resource time for quality management functions but there
may be other alternatives for allocating the resources.

In addition, during the review, ITA Practice Director indicated that the testing for a product tends
to either use vendor developed tests or new tests developed specifically for the product—they
have no standard test methods defined. This makes their testing dependent on the vendor input
and vulnerable to unique vendor interpretations rather than a core validated set of internal
references for training and testing.

A proposal was made that Wyle take the lead and provide direction on qualify management
reviews, audits, test planning, and report writing. CIBER would add software review and election
definition experience with possibly some security expertise through corporate CIBER. Wyle,
under this proposal, would be fully responsible for the coordination of testing and the final report.
CIBER/Wyle would need to work out additional criteria to standardized test plans, determine the
who and how review of the TDP would be conducted, and the contract oversight relations.

All the ITAs need to complete a review of the VSS 2002 and new VVSG 2005 and update the
requirements cross-reference matrix to be used to identify which requirements have been tested
and where or when. The former matrix developed jointly between the ITAs is missing significant
requirements and variations on requirements. (Note: Shawn Southworth, ITA Practice Director,
reports that CIBER does not have that version of the checklist.)

Recommendation

Wyle to continue as a Hardware ITA, eventually serving as a resource lab for environmental
hardware testing for new VSTLs or move to becoming a VSTL by taking responsibility for full
system testing with possible subcontracting to CIBER or another qualified group.

CIBER ITA Practice continues only with the support of Wyle or a commitment from corporate

CIBER to provide management assistance in getting the quality system functioning and fuller
reporting of results with a review in 120 days.

(signed)


EAC Interim Accreditation checklist 7/20/06
CIBER record

¢) The training program as it exists based on corporate training is not completely
relevant.

d) (new).

{VS-4.5.2) Positions assigned

Laboratory Director: Shawn Southworth, ITA Practices Director

Technical Director: Shawn Southworth, ITA Practices Director

Authorized Representative: Shawn Southworth, ITA Practices Director
Approved Signatory Personnel: Shawn Southworth, ITA Practices Director
Team Leaders: Shawn Southworth, ITA Practices Director

Jack Cobb, Systems Software Analyst, _ aod

training record available and reviewed Does not verify J. Cobb is
qualified to test.

Quality Manager: Paul Rainville, Director of Delivery.

prainville@ciber.com
703-610-6400 x 6475

Not listed on Org chart. External to ITA Practices office.

ITA QA Representatives: (defined as local staffemployees with responsibilities to
monitor QA requirements, assigned in org chart and PD)

Tom Dunn No training records or record of designated as qualification
Jennifer Price No training records or record of designated as qualification
5.2.3 Personnel,

a) employed and/or contracted personnel. All personnel assigned qualify under this
requirement

b) CIBER Policy and Procedure. Subcontractor Monitoring, Feed back from client of
manager is collected and a performance assessment is made. Records held at divisional
office. No policy/procedure for training or qualifying for competency. Corporate training
appears to be irrelevant for them.

5.2.4 Job descriptions. Available and complete. Individuals identified as assigned to
position in QTP Organization Chart in Sect 3. Recommended that the names be
removed from the QTP Org Chart and the information be provided in other forms.

5.2.5 Authorizations for testing.

a). Authorize specific personnel to perform specific processes. Informal.

4). Training record reviewed for Jack Cobb. Does not include record of authorization for
performing tests. No record exists for subcontracted employees.

Page 14 of 21


Introduction

This accreditation assessment was conducted to support an interim program pending
implementation of the full EAC Accreditation program in cooperation with the National Voluntary
Laboratory Accreditation Program (NVLAP) under NIST Handbook 150-2006, NVLAP Procedures
and General Requirements and NIST Handbook 150-22- 2005 NVLAP Voting System Testing
(HB 150-22). The interim program is designed to accredit ITAs formerly authorized under the
National Association of State Election Directors (NASED) accreditation program to continue
voting system testing under an EAC accreditation until such time as the NVLAP/EAC joint
accreditation has qualified one or more testing laboratories as Voting System Test Laboratory
(VSTL).

Summary of Findings

Under NASED, Wyle Laboratories and CIBER, Inc. were separately accredited as Hardware and
Software ITAs. Under those roles, Wyle tested the principle voting devices—those components
that received the votes of the voter and performed the basic tally operation. CIBER tested the
Election Management System (EMS) and Reporting System components that performed
consolidation of voting results from multiple voting devices on a general purpose computer such
as a Commercial Off The Shelf (COTS) workstation or server. Wyle and CIBER have since
formed an exclusive team agreement to work with each other as a joint testing cooperative to
perform the full voting system certification testing. The source code review of software resident to
the voting or vote tallying device which had been Wyle’s responsibility has actually been
performed by CIBER or source code reviewers working with CIBER in the last year or more.

Wyle has a long identity as a voting system testing lab being the first accredited under the
NASED program. Wyle brought to the program a strong background in environmental testing of
DOD systems and holds separate accreditation such as the American Association for Laboratory
Accreditation (A2LA) for the major hardware test methods required for voting system
accreditation. Wyle has a well-defined quality management system in the terms of ISO/IEC
17025 which is generally exercised and used. The corporate culture and higher level
management support are compatible with and help support quality management practices.

The CIBER ITA operation is actually a small branch office, [TA Practice, CIBER, Inc, operating
independently from the corporate CIBER operations. Corporate CIBER's quality management
system (which is ISO 9001 compliant) places the responsibility to define and implement the
quality program under the ISO/IEC 17025 requirements at the ITA Practice Director’s level but
places the QA Manager responsible at a branch office (PPQA Group) located remotely from ITA
Practice locatoin. ITA Practice’s Process and Project Quality Assurance Plan (PPQAP) [Apr 2005] policy
document and supporting processes and procedures were created last year but critical processes
were not implemented nor procedures followed. ITA Practice, CIBER is unable to follow their
own defined processes and procedures to ensure the quality of their work.

Although Wyle and CIBER are working together, they have distinctly different quality
management programs and different levels of proficiency about following those programs. in the
Hardware/Software division, Wyle tests only to the boundaries of the device—they do not, as a
rule, perform any operations on the EMS or Reporting system components and limit the
interaction with transfer media to the input/output ports of the specific device.. CIBER performs
more of the system integration testing by producing variations of election definitions which they
either provide to Wyle or operate the voting devices to produce results to use in the Reporting
system testing but generally do not exercise a wide function of the voting device, leaving that
testing to Wyle. Wyle reports follow ISO/IEC guidelines and tend to be reasonably complete


EAC Interim Accreditation checklist 7120/06
CIBER record

5.4.3 Laboratory-developed methods (13)
a)

5.4.4 Non-standard methods (13).

5.4.5 Validation of methods (14).

5.4.5.1 Validation definition

Validation is the confirmation by examination and the provision of objective evidence that
the particular requirements for a specific intended use are fulfilled.

5.4.5.2 The laboratory shall validate non-standard methods, laboratory-designed/developed
methods, standard methods used outside their intended scope, and amplifications and
modifications of standard methods to confirm that the methods are fit for the intended
use. The validation shall be as extensive as is necessary to meet the needs of the given
application or field of application. The laboratory shall record the results obtained, the
procedure used for the validation, and a statement as to whether the method is fit for the
intended use.
Need to develop

5.4.6 Estimation of uncertainty of measurement (14) N/A May need to develop in the
future.

5.4.7 Control of data (15)

5.4.7.1 Calculation and data transfers.

Calculations and data transfers shall be subject to appropriate checks in a systematic
manner.

5.4.7.2

When computers or automated equipment are used for the acquisition, processing,
recording, reporting, storage or retrieval of test or calibration data, the laboratory shall
ensure that:

a) computer software developed by the user is documented in sufficient detail and is
suitably validated as being adequate for
Will need to develop for test tools and utilities used for testing provided by a vendor.

b) procedures are established and implemented for protecting the data; such procedures
shall include, but not be limited to, integrity and confidentiality of data entry or collection,
data storage, data transmission and data processing;

May need to develop.

¢). computers and automated equipment are maintained to ensure proper functioning

and are provided with the environmental and operating conditions necessary to maintain
the integrity of test and calibration data

Page 17 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

1 Purpose and Application.

141 Purpose. The following checklist was developed for use in the Election Assistance
Commission (EAC) Interim Accreditation for Independent Test Authority Labs (ITAs). This
program is an interim program pending implementation of the full EAC Accreditation program in
cooperation with the National Voluntary Laboratory Accreditation Program (NVLAP) under NIST
Handbook 150-2006, NVLAP Procedures and General Requirements and NIST Handbook 150-
22- 2005 NVLAP Voting System Testing (HB 150-22). The interim program is designed to
accredit ITAs formerly authorized under the National Association of State Election Directors
(NASED) accreditation program to continue voting system testing under an EAC accreditation
until such time as the NVLAP/EAC joint accreditation has qualified at least one testing laboratory
as Voting System Test Laboratory (VSTL).

1.2 Background. The NASED ITAs were accredited under the NASED Program Handbook
9201, Accreditation of Independent Testing Authorities for Voting System Qualification Testing,
(Rev A), 7 Apr 2001. (HB 9201). The HB 9201 was based on Department of Defense standards
such as MILSTD-490A and MIL-STD-2167A which had been deleted or superceded by the time
of the Rev A release. Rev A was to have been a temporary revision pending the completion of
the new voting system standards in 2002 (which was to add a much larger scope of accreditation
to include the election management software integration with vote tallying equipment as a voting
system.) The Help America Vote Act (HAVA) provisions took the responsibility from NASED and
the revision was cancelled pending the development of a new program under EAC and NIST.

1.3 Usage. For the purpose of this accreditation, the management documents provided by

the candidate lab were developed under the NASED HBK but will be assessed using ISO/IEC

17025 criteria. As such, it is expected that the documented policy and procedures may not

explicitly follow the language and procedures recommended under ISO/IEC 17025 but that the

underlying program may support a quality management program that meets the intent of ISO/IEC

17025. On the items below, the assessor will place a

«  “X" on substantive discrepancies to be considered in the accreditation decision.

* “C”omments on items where some work is needed to bring the program into compliance with
ISO/IEC 17025 but procedures used support the integrity of the testing process.

* “OK” where published procedures and policies are supported by evidence of implementation/

A technical supplement checklist will include Voting System Standards/HAVA requirements for

specific review, assessment, or testing.

Note: In general, the Voting System testing is not a calibration activity as intended under ISO/IEC

17025. Calibrated instruments are used in the environmental testing.

(The number in parenthsis is a back reference to page reference to ISO/IEC 17025)

2 Reference Documents

21 Normative

2.2 ISO/EC 17025(2005). General requirements for the competence of testing and
calibration laboratories, dated 2005-5-15.

2.3 FEC VSS-2002, General requirements for the competence of testing and calibration
laboratories, dated May 2002 (Note: official version posted on EAC web site and available

since 2004)
24 FEC VVSG-2005,
2.5 Internal

2.5.1 Parent organization

2.5.1.1 CIBER’s Custom Solution Division Quality Management Manual (CQMM) (ISO 9001
compliant)

2.5.2 QA Program for ITA Practices

2.5.2.1 Process and Project Quality Assurance Plan (PPQAP), Ver 3.0, Apr 30, 2005. Parent
document (note: document labeling which says Version 2.0 and Version Release History shows Apr
2004 both are typo errors)

Page 1 of 21


EAC Interim Accreditation checklist 7/20/06
CIBER record

4.16

4.2

4.21

g) provide adequate supervision of testing and calibration staff, including trainees,
by persons familiar with methods and procedures, purpose of each test and/or
calibration, and with the assessment of the test or calibration results;

(See also 5.2)

Need to add. ITA Practices is using processes to make sure personnel are qualified
before working independently but do not have a statement of policy to cover supervision
while new hires are being qualified or changes of position to a function where not
previously qualified.

h) have technical management which has overail responsibility for the technical
operations and the provision of the resources needed to ensure the required
quality of laboratory operations;

ITA Practice Director. See QTAP, Sec 3. Currently there are only three full time
employees so many positions will overlap.

i) appoint a member of staff as quality manager (however named) who, irrespective
of other duties and responsibilities, shall have defined responsibility and authority
for ensuring that the management system related to quality is implemented and
followed at all times; the quality manager shall have direct access to the highest
tevel of management at which decisions are made on laboratory policy or
resources;

PQAP, pg 6 &7 ITA QA Representatives are assigned to advise the ITA Practices
Director directly on QA issues. Currently these are T. Dunn and J. Price (independent
subcontractors)

j) appoint deputies for key managerial personnel (see Note);

Not documented. Currently Jack Cobb but not reflected in organizational chart.

k) (New) ensure that its personnel are aware of the relevance and importance of
their activities and how they contribute to the achievement of the objectives of the
management system.

With three people sitting down at some table.

NOTE Individuals may have more than one function and it may be impractical to appoint
deputies for every function.

(New) Top management shall ensure that appropriate communication processes
are established within the laboratory and that communication takes place
regarding the effectiveness of the management system.

Emails and conversations. Small organization.
Management system (3)

The laboratory shall establish, implement and maintain a management system
appropriate to the scope of its activities. The laboratory shall document its
policies, systems, programmes, procedures and instructions to the extent
necessary to assure the quality of the test and/or calibration results. The system's
documentation shall be communicated to, understood by, available to, and
implemented by the appropriate personnel.

Page 4 of 21