EAC Interim Lab: CIBER

Not applicable

5.7 SAMPLING

5.8 HANDLING OF TEST AND-GAHBRATION ITEMS

(Handling and care of the actual Equipment Under Test (EUT))

(not reviewed in detail due to time but it has parallel issues to 5.5 and the test equipment)N

EAC Interim On Site Lab Report

Page 16 of 19


LAB: CIBER 6-8 Dec 2006
serious issue or risk to the business is identified.

4,12 Preventive action
4.12.1

ok a) Needed improvements and potential sources of nonconformities, either
_technieal or. Lgoncerning- the management system, shall be identified.
Bors VA Pieces Lmproverset” Prevenkeb behen Pacers {Pern
Cie@ b) When improvement opportunities are identifi ied or if preventive action is
required, action plans shall be developed, implemented and monitored to
reduce the likelihood of the occurrence of such nonconformities and to take
advantage of the opportunities for improvement. .
POET Process Bee provement? theverdat Aelin ®
ole 4.12.2 Procedures for preventive actions shall include the initiation of such actions
and application of controls to ensure that they are effective. >
BRON Preees Reverey ommek  Pyvendebrs Ache Yous &
NOTE 1 Preventive action is a proactive process fo identify opportunities for
improvement
rather than a reaction to the identification of problems or complaints.

NOTE 2 Apart from the review of the operational procedures, the preventive action
might

involve _analysi
results.

i

or .

dea ae (a

4.13 Control of records

4.13.1 General
Flow y CRE Becend erosemnegl

24 4.13.1.1 The laboratory shall establish and maintain proced bs for identification,

collection, indexing, access, filing, storage, maintenance and disposal of

quality and technical records. Quality records shall include reports from

internal audits and management reviews as well as records of corrective

and preventive actions. 4
vg a lone tuilectee vv @ Fi epee ey Birny | \peleadion
Bole Gaades

4. 13. 1. 2
_— we rnnnee bt PA
“ lea) All records shall be legible and shall b etained in such a way t at
/ they are readily retrievable in facilities that provide a suitable environment to
prevent (Gamage or deterioration and to prevent loss. i
(Show perk Y STA Porhel Soa 1S Poteet
Cb) Retentiqn times of records shall be established. mi
as FAC
XN NOTE Recotis Hey be in any media, such as hard copy or electronic media.
eas 13.1.3 All records shall be held secure and in confidence. Ln Corks ant
oe Foes iv, & veep” ENS ufene ccece Ligdls av de TES evel
Reg Beesltspeg) 2° Beto On 8. ‘ge bl hehe eucbdacl9

Recor “di. +
4.13.14 The iaborately y shall have procedures to protect and back up records stored
electronically and to prevent unauthorized access to or amendment of these

peers fre ae 1%. y eoeeQs ineinbaen! * wi e- Loenter-
uport bec’ so dee zy,
NIST HANDBOOK 180 hese CKLIST (REV. Pee Atay Be sews Menara. Gauge

bask

PAGE 11 OF 42



ok.

LAB: CIBER 6-8 Def 2006

NOTE For review of routine and other simple tasks, the date
and the identification (e.g., the initials) of the person in the laboratory responsible for
carrying out the contracted work are considered adequate. For repetitive routine tasks,
the review need be made only at the initial enquiry stage or on granting of the contract
for ongoing routine work performed under a general agreement with the customer,
provided that the customer's requirements remain unchanged. For new, complex or
advanced testing and/or calibration tasks, a more comprehensive record should be
maintained. Prajent Pork  eecamined

Poe F BOW alt GbE sleet tiv ad (4

44.3 The review shall also cover any work that is subcontracted by he laboratory.
©. PY. { ‘
“2% .}

gi 4.4.4 — The customer shall be informed of any deviation fram the contract.

Pom: F ~The

445 if a contract needs to. be amended after work has commenced, the same

4.5

contract review process shall be repeated and any amendments shall be
communicated to all affected personnel.
fom 2 Tm. 2.

Subcontracting of tests and calibrations

4.5.1 When a laboratory subcontracts work whether because of unforeseen
reasons (e.g., workload, need for further expertise or temporary incapacity) or
on a continuing basis (e.g., through permanent subcontracting, agency or
franchising arrangements), this work shall be placed with a competent
subcontractor. A competent subcontractor is one that, for example, complies
with this handbook for the work in question, Pom 9 2 ect

Sar ror an cheblhy. a7] 9 "Ke.

4.5.2 The laboratory shall advise the customer of the arrangement in Writing and,

when appropriate, gain the approval of the customer, preferably in writing.
Tk Zhe gots Pda

4.5.3 The laboratoty is responsible to the customer for the subcontractor's work,\ wee
except in the case where the customer or a regulatory authority specifies , °
which sub tractor is to be used. Geter

Sb. Be relch) bebe. ;

4.5.4 The laboratory shail maintain a register of all subcontractors that it uses for
tests and/or calibrations and a record of the evidence of compliance with this
handbook for the work in question

Aypron

Sens ora Br wy Parkas,

Purchasing services and supplies

4.8.1 The laboratory shall have a policy and procedure(s) for the selection and
purchasing of services and supplies it uses that affect the quality of the tests
and/or calibrations. Procedures shall exist for the purchase, reception and
storage of reagents and laboratory consumable materials relevant for the
tests and calibrations.

4.6.2

a) The laboratory shall ensure that purchased supplies and reagents and
consumable materials that affect the quality of tests and/or calibrations are not
used until they have been inspected or otherwise verified as. complying with
standard specifications or requirements defined in the methods for the tests
and/or calibrations concerned. These services and supplies used shall comply

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 7 OF 42



LAB: CIBER 6-8 Deg 2006

at. b) Where contracted and additional technical and key
support personnel are
used, the laboratory shall ensure that such personnel are supervised and
competent and that they work in accordance with the laboratory’s
management system. genuicet mnde,  ebdandra

gow fved etansi. .

5.2.4 The laboratory shall maintain current job descriptions for managerial, _
technical and key support personnel involved ip ipsts and/or calibrations. ick

Rem FL ele © Deepen a lO Lose. a Loch

NOTE Job descriptions can be defined in many ways, As'a minimum, the following

should 1 pedis Fh ALT Spent on
. (Png : s wT. Oyen te eden:
be defined: gies vale Vl fen te a) | dounckel . whi eanreyanurelo
, Toor ees, Diihh spud! ;
i) the responsibilities with respect to performing tests andlor éalibrations: “ on aely
Ba bref [3 I ¥mene offeridonnteisoe dr leans (eflade FTE Seu cohe
ii) the responsibilitiés with regpect to the-planning of tests’ and/or

calibrations and

evaluation of results; a proce Sar,
OA’ pel L 1 Phone [re Yeu
iii) the responsil S for reporting opinions ‘and interpretations;
et E he L Ph. You stew fe [re: és, i
iv) the responsibilities with respect to method modification and

development and
validation of new methods;

Qk inek Chas thle. pret Le
vy) expertise and experience required; a \ . ae
EAS gprs we tached Rak Coerkgvagg GM ab ma $4 rere 47 & pd
vi) qualifications and training programs; bee pe [ pee ve a ane
GAY tape, ghti wetted viele Fx seetgaee fi Para. [Tes fie
vii) managerial duties. ‘ a

. Sepa feels es | PR Be [re pe.
§.2.5

_€a) The management shall authorize specific personnel to perform particular
types of sampling, test and/or calibration, to issue test reports and
calibration certificates, to give opinions and interpretations and to
Operate particular types of equipment.

| Mls drcocrd. Aewel er Gore 4 budlohuakel? acofffen 2

eb) — The laboratory shall maintain recofds of the relevant authorization(s),
competence, educational and professional qualifications, training,
skills and experience of all technical personnel, including contracted

personnel. Revie e( creaky oad PAH TZ

&”_¢) This information shail be readily available and shall include the date on which
authorization and/or competence is confirmed.”
J 4
ves So gle teakoyo
NVLAP Note: This requirement also applies to Approved Signatories (see 1.5.2).

5.3 Accommodation and environmental conditions

53.4

[aid a) Laboratory facilities for testing and/or calibration, including but not limited to
energy sources, lighting and environmental conditions, shall be

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 16 OF 42



1 EAC Interim Lab: CIBER

4.11 CORRECTIVE ACTION

Prior visit observation that the corporate CIBER required CARs log was not active and
showed no reports.has been corrected.. Test cases were added to validate the operation and
use, No action had been taken on the management review and monitoring because the
annual management review that would review and take follow up action had not occurred yet
but is expected within the next week. Prune cavnaltTol fodey ae

No non-conformance items have been identifed

4.12 PREVENTIVE ACTION

Processing of review of management indicators, including the review of audit forwarded
Corrective Action Requests (CARS), was not observed due to lack of activity. Two Audit
directed CARS (ACARS) have been placed into the log to suggest improvements to the
CARS upload operation but have not yet processed by the annual management review (see
above)

No non-conformance items have been identified.

EAC Interim On Site Lab Report Page 10 of 19



LAB: CIBER 6-8 Dec 2006

—— 5.6.2.1.2 There are certain calibrations that currently cannot be strictly made in SI
units. In these cases ibration shall provide confidence in
measurements b stablishing traceability to appropriate
measurement standards such as:

fe a . . i: -.
a) the use of ce! nce materials provided by a competent supplier to give
a reliable physical chemical characterization of a material;

b) the use of speci ef rethods and/or consensus standards that are clearly
described and agreed by all parties concerned.

c) Participation in eéuitable program of interlaboratory comparisons is required
where possible.
§.6.2.2 Testing

—___ §.6.2.2.1 For testing laboratories, the requirements givén in 5.6.2.1 apply for
measuring and test equipme ith measuring functions
used, unless it has bee! ablished that the associated
contribution from the-calibration contributes little to the total
uncertaint e test result. When this situation arises, the
laboratory shall ensure that the equipment used can provide the
uncertainty of measurement needed.

NOTE The extent to which the requirements in 5.6.2.1 should be followed dependson '
the relative contribution of the calibration uncertainty to the total uncertainty. If calibration is *~
the dominant factor, the requirements should be strictly followed.

§.6.2.2.2 Where traceability of measurements Giits is not possible and/or not
relevant, the same requiréments for traceability to, for example,

5.6.3 Reference standards and reference materials
5.6.3.1 Reference standards

a) The laboratory shall have a program a: rocedure for the calibration of its

reference standards.

b) Reference standards shal|-bé calibrated by a body that can provide
traceability as described’in 5.6.2.1.

c) Such refere standards of measurement held by the laboratory shall be
used forcalibration only and for no other purpose, unless it can
be-stiown that their performance as reference standards would
not be invalidated. Reference standards shall be calibrated before and
after any adjustment.

5.6.3.2 "Reference materials

Reference materials sh: ll_-whére possible, be traceable to SI units of measurement, or to
certified reference-1 aterials. Internal reference materials shall be checked as far

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 26 OF 42



5.6

Lue
Bt

iw"

@
.

LAB: CIBER 6-8 Deg 2006

5.5.8 | Whenever practicable, all equipment under the control
of the laboratory and _-.

requiring calibrati -sfrall be labeled, coded or otherwise identified to indicate
the status-of Calibration, including the date when last calibrated and the date
or ration criteria when recalibration is due.

§.5.9 When, for whatever reason; equipment goes outside the direct control of the
laboratory, the lab rafary shall ensure that the function and calibration status
of the equip are checked and shown to be satisfactory before the
equipments returned to service. _

5.5.10 When intermediate checks are-néeded to maintain confidence in the
calibration status of, théequipment, these checks shall be carried out
according to a defined procedure.

5.5.11 Where calibrations give rise’fo a set of correction factors, the laboratory shall
have procedures ta-€nsure that copies (e.g., in computer software) are
correctly updated:

5.5.12 Test and ealibration-equipment, including both hardware and software, shall
BE Safeguarded from adjustments which would invalidate the test and/or

calibration results. yied

New: 5.5 y
Measurement traceability asl

Comer by fh rey Sirmal
5.6.1 General ,° | rey
pefleg Mast /
a) All equipment used for tests and/or calibrations, including equipment for
subsidiary measurements (e.g., for environmental conditions) having a

significant effect on the accuracy or validity of the result of the test,
calibration or sampling shall be calibrated before being put into service.

b) The laboratory shall have an established program and procedure for the
calibratiok of its equipment.

m should include a system for selecting, using, calibrating, checking,
controlling and maintaining measurement standards, reference materials used as
measurement standards, and measuring and test equipment used to perform tests and
calibrations.

NVLAP Note: See Annex B for requirements for the implementation of traceability policy
in NVLAP-accredited laboratories.

5.6.2 Specific requiremen

5.6.2.1 Calibration

5.6.2.1.1

a). For calibration laboratories, the\program for calibration of equipment shall be
designed and operated so as to énsure that calibrations and measurements
made by the laboratory are traceable to the International System of Units (SI)
(Systéme international d’unités). \

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 24 OF 42



LAB: CIBER 6-8 Dec 2006

instructions in the test or calibration methods.

Be b) The results shall be reported, usually in a test report or-a-calibration
certifieate-(see Note 1), and shall include all the information
requested by the customer and necessary for the interpretation of
the test or calibration results and all information required by the
method used. This information is wale 4 that required by 5.10.2, and
5.10.3 or 5.10.4. Arve pave Ayes 4

es 0) , Inthe case of tests or calibrations performed for internal customers, or in the

. rad case of a written agreement with the customer, the results may
br & be reported in a simplified way. Any information listed in 5.10.2
W to 5.10.4 which is not reported to the customer shall be readily
available in the laboratory which carried out the tests and/or
calibrations.

NOTE 1 Test reports and calibration certificates are sometimes called test certificates and
calibration reports, respectively.

NOTE 2 The test reports or calibration certificates may be issued as hard copy or by
electronic data transfer provided that the requirements of this handbook are met.

5.10.2 Test reports and calibration certificates

Each test report or calibration certificate shall include at least the following
information, unless the laboratory has valid reasons for not doing so:

oi a) a title (e.g., “Test Report” or “Calibration Certificate’); ga? ony

2b) the name and address of the laboratory, and the location where the tests
and/or calibrations were carried out, if different from the
address of the laboratory; . Rote few

<vc) unique identification of the test report or calibration certificate (such as the

serial number), and on each page an identification in order to
ensure that the page is recognized as a part of the test report or
calibration certificate, and a clear identification of the end of the test report

anaer ve a. ok
or calibration certificate; bt pace “
Pica d) the name and address of the customer;
: wel ® pare
de) identification of the method used; :
: my ch! .
sli f) a description of, the condition of, and unambiguous identification of the

wae
a

item(s) tésted or calibrated;

- C) Me
Gee ON Bee gx Ae
the date of receipt of-the test or calibration item(s) where this’ is critical to the

validity{and application of the results, and the date(s) of
performance of the test or calibration;

ae h) reference to the sampling plan and procedures used by the laboratory or
other bodies where these are relevant to the validity or
application of the results;

2 i) the test or calibration results with, where appropriate, the units of
NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 30 OF 42



LAB: CIBER 6-8 Dec 2006
c) When it is necessary to issue a complete new test report or calibration

certificate, this shall be uniquely identified and shall contain a
reference to the original that it replaces.

NiST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 34 OF 42


LAB: CIBER 6-8 Dec 2006
systematic manner. Nek Bovucet

5.4.7.2 When computers or automated equipment are used for the
acquisition, processing, recording, reporting, storage or
retrieval of test or calibration data, the laboratory shall ensure that:

_®@ a) computer software developed by the user is documented in sufficient detail
and is suitably validated as being adequate for use;
; he fo Porm py~ Se euih,
Ok b) procedures are established and implemented for protecting the data; such

procedures shall include, but not be limited to, integrity and
confidentiality of data entry or collection, data storage, data transmission
and data processing; for ft Seems

c) computers and automated equipment are maintained to ensure proper
functioning and are provided with the environmental and operating
conditions necessary to maintain the integrity of test and calibration data.

TR LL Basal ing Poolsty L Feat Patee
NOTE Commercial off-the-shelf software (e.g., word processing, dafab: se and statistical an
programs) in general use within their designed application range may be considered to be '* ¢SVe"~
sufficiently validated. However, laboratory software configuration/modifications should
be validated as in 5.4.7.2 a).

5.5 Equipment

5.5.1

e/a) The laboratory shall be furnished h-alkitents GF sampling, measurement
and test equipment re: he correct performance of the tests
and/or calibrati. including sampling, preparation of test and/or

calibratier"items, processing and analysis of test and/or calibration data).

H bp) In those cases where the laboratory needs to use equipment outside its
permanent control, it shall ensure that the requirements of this
handbook are met.

Rerrthe omyre ue fis. ) Mewes

we

5.5.2 —_ . 3

id a) Equipment and its software used for tésting, calibration and sampling shall
be capable of achievirig the accuracy required and shall
comply with-spécifications relevant to the tests and/or calibrations
concerned.

- ee

b) Calibration programs shaitbé established for key quantities or values of the

instruments.where these properties have a significant effect on the results.

c) Before being placed into service, equipment (including that used for

sampling) shall be calibrated-orchecked to establish that it
meets the laborat: Specification requirements and
complies wi € relevant standard specifications. it shall be

checked afid/or calibrated before use (see 5.6).
To St2 Neb Fd Tacha mend

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 22 OF 42



ANIBER WILLBURN
Proposal iVianager

Ms. Willburn is a Proposal Manager with over five years of writing experience. She has played a key role
in furnishing proposal and business development expertise in support of defining/refining company
proposal development processes and procedures. Provided writing and editing for technical
documentation and business development tasks as needed, including design and direction for graphics.
In addition, she has defined and prepared documentation schedules, outlines, writing assignments,
themes, and discriminators. Provided other as-requested technical writing and document review support
to various company business units. Developed and designed marketing collateral for federal clients and
inter-office fact sheets on current CIBER technologies. Lastly, Ms. Willburn has exceptional
communication and interpersonal skills which allow her to work effectively with all levels of management
and personnel.

EDUCATION, CERTIFICATIONS & TRAINING
BA, English, University of Colorado at Colorado Springs, 2004

Certificate in Professional and Technical Writing, University of Colorado at Colorado Springs, 2004

TECHNICAL EXPERIENCE

Software/Tools: Word, PowerPoint, Outlook, Internet Explorer, Project

PROFESSIONAL EXPERIENCE

CIBER, Inc., 7104 — Present
Proposal Manager

® Furnished proposal and business development expertise in support of defining/refining company
proposal development processes and procedures.

e Provided writing and editing for technical documentation and business development tasks as needed,
including design and direction for graphics.

e Defined and prepared documentation schedules, outlines, writing assignments, themes, and
discriminators.

e Provided other as-requested technical writing and document review support to various company
business units.

2 Developed and designed marketing collateral for federal clients and inter-office fact sheets on current
CIBER technologies.

Project Excel Writing Center 2002 — 2004
Writing Tutor, Newsletter Editor

° Worked with students on technical documents and taught grammar and document design.
e Edited student papers over online forum.

° _Instituted first Excel Center newsletter and acted as editor.

2 Designed layout and graphics for newsletter.

» Assigned writing responsibilities to contributors.

HNE (11/10/06)



LAB: CIBER 6-8 Dec 2006

4.13.2 Technical records
4.13,2.1
Cé.-a) The laboratory shall retain records of original observations, derived data and

sufficient information to establish an audit trail, calibration records, staff records
and a copy of each test report or ealibration-certificate issued, for a defined

perios Bon WS ar. wt Shp 2 obscowbn, ¢ ,
ey Chink. | Gaur tex i-< Cate Comet on aa eFlekfnm c. baits
£. b) The re lords for each test or calibration shall contain’sufficient information to the

he facilitate, if possible, identification of factors affecting the uncertainty and to
enable the test or calibration to be repeated under conditions as close as
possible to the original. Pom, Misi6
a alactp-
oc) The records shall include the identity of personnel tésponsible for the alee

sampling, performance of each test and/or calibration and checking of results.
tecorted | Kedobs. hard copy cache re.
NOTE 1 In certain fields it may ibe impossible or impracticable to retain records of all
original
observations,

NOTE 2 Technical records are accumulations of data (see 5.4.7) and information
which

result from carrying out tests and/or calibrations and which indicate whether specified
quality or process parameters are achieved. They may include forms, contracts, work
sheets, work books, check sheets, work notes, control graphs, external and internal test
reports and calibration certificates, customers’ notes, papers and feedback.

ole 4.13.2.2 Observations, data and calculations shall be recorded at the time they are
made and shall be identifiable to the Specie task.
* aa a Obsary ¥ ade ~ CAL. one > bP rechigh “ fe.
413.23 ob hh 4 ohsceeetion. Fm 2, G? * we FIRS ea, Teco
sip ausdse pbuh tomporeck desbas eee
_f <a) When mistakes occur in records, each mistake shall be crossed out, not
erased, made illegible or deleted, and the correct value entered alongside. All
such alterations to records shall be signed or initialed by the person making the
correction. FMB, pom & IS.b Ste 0S dees gee

maticlh, »
J

swb) tn the case of records stored electronically, equivalent measures shall be taken
to avoid loss or change of original data. .
Lahey go Mtb bre Mag Pehensparts ¢ len boty

4.14 Internal audits 1 (*' 4
4.14.1
ota) The laboratory shall periodically, and in accordance with a predetermined

schedule and procedure, conduct internal audits of its activities to verify that
its operations continue to comply with the requirements of the management
system and this handbook. The internal audit program shall address all
elements of the management system, including the testing and/or calibration
activities. It is the responsibility of the quality manager to plan and organize
audits as required by the schedule and requested by management.

Pom tel te fei , 12 etal theft

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 12 OF 42



I EAC Interim Lab: CIBER

To: Election Assistance Commission
Attn: Brian Hancock .
EAC Interim Accreditation Program On Site Assessment- CIBER
SIGNATURE SHEET
Lab Legal Name: CIBER, Inc
Address: CIBER ITA Practices
7501 South Memorial Parkway, Suite 107
Huntsville, AL 35802

. Assessor: Steven V. Freeman

On-Site Assessment Dates: 6-8 Dec 2006
Prior Assessment Visit: 20 Jul 2006

This report was presented on-site at the conclusion of the visit and presents a summary of
the findings.

Resolution of previous findings

The qualify management and technical documents describing the procedures for
documenting and conduction of a test campaign had been completely revised prior to the
assessment visit. Test methods (cases) have been developed and added in the form of the
Test Methods document. In addition, personnel have been brought in to assist in revising
the program and to assist with responsibilities in the Quality Assurance (QA) and Project
Management roles. Within corporate CIBER, an Executive Management Steering
Committee was created consisting of the President of CIBER Federal Civilian, VP
Contracts Federal, and a Consulting Services Manager to perform high management
oversight. Finally, QA records and logs were activated to store, track, and follow-up
records of Corrective Action Reports (CAR) and audit findings.

Latest Internal Audit: 15 November 2006, Terry L. Debell,
py, cee Manger, Internal Audit & Compliance
Latest Management Review: (Bondisiey- Executive Management Steering Committee

Acknowledgement of Receipt
The assessor has discussed the contents of this report with members of the laboratory
management who agree to respond in writing to EAC regarding resolution or correction
of any nonconformities noted within 30 days of the date of this report. |

‘oy c.
Signature of Authorized Representatives or designee Poet BFL LE!

_ c .
Printed Name: Oo ha wh Douthworth

EAC Interim On Site Lab Report Page 1 of 19



LAB: CIBER 6-8 Dee 2006

4.3.2 Document approval and issue
4.3.2.4
ae a) All documents issued to personnel in the laboratory as part of the
management system shall be reviewed and approved for use by authorized
personnel prior to issue. agua Combos pode CLES }
Cf b) A master list or an equivalent document control procedure identifying the

current revision status and distribution of documents in the management
system shall be established and be readily available to preclude the use of
invalid and/or obsolete documents. Pople

4.3.2.2 The procedure(s) adopted shall ensure that:

IE a) authorized editions of appropriate documents are available at all locations
where operations essential to the effective functioning of the laboratory are
performed; Shope, pearly

hb) documents are periodically reviewed and, where necessary, revised to

} ensure continuing suitability and compliance with applicable requirements;
Pore TL PM encgement Pevian ' G1 Er Ber Btw Feel (
dim, C) invalid or obsolete documefits are promptly removed from all points of issu

or use, or otherwise assured agaist unintended use; f

(
Pom G1) obge rprenbal scurry bee byw dh audemck

4%, d) obsolete documents retained for either legal or knowledge preservation
Purposes are suitably marked. .
Pom 6 pravrely bev erehtagy ov Shewpanct Bbolule Lecme:

4.3.2.3 Management system documents génerated by the laboratory shall be reves
uniquely identified. Such identification shall include:
?, 3 Pacticeetpe Sie nderls

Oke sa) the date of issue and/or revision identification,

Gk b) page numbering,

Gk c) the total number of pages or a mark to signify the end of the document, and
abe d) the issuing authority(ies).

4.3.3 Document changes

dike 4.3.3.1 Changes to documents shall be reviewed and approved by the same function
that performed the original review unless specifically designated otherwise.
The designated personnel shall have access to pertinent backgr und
information upon which to base their review and approval. “+l le.

PE 12 petamey VOY Menasemd Revivegs Ke beter o
Where practicable, the altered or new text shall be identified in the document
or the appropriate attachments.

Tech. Chergent Gms Heist.) renand of afi

4 po Fens. oa bos.

~ (EIR Cover

henge poh f

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 5 OF 42



{ EAC Interim Lab: CIBER

yiee
5.9 ASSURING THE QUALITY OF TEST ANS-GALIBRATION RESULTS

Not reviewed due to time.

5.10 REPORTING THE RESULTS

Non-conform. Work by other accredited labs needs to be identified (test plan and contract
also) and validated that the lab is qualified. The results need to be validated that they are
appropriate for the report. If the work is outside of the scope of accreditation for the
contracting lab, this condition needs to be explicitly stated (ref 5.4.6 in the HDBK 150-22)

HDBK 150/150-22 requires specification of processing for reports for other purposes. Note
that this involves branding issues where claiming the authority as an accredited lab may not
be appropriate. Recognized alternate reports are for state certification and internal to the
vendor.

EAC Interim On Site Lab Report Page 17 of 19



KELLY ROHACEK
Project Manager

ROBERT HALF CONSULTING, (Cleveland, OH) 6/03 ~ 7/04
Senior Project Manager

Managed strategic initiatives and the installation of integrated systems Completed 2040 hours in RFP

processes, project initiation, and full project life cycle management for the clients of Key Bank and

National City by;

® Successfully and quickly creating the schedule and demonstration of the Request for Proposal (RFP)
process and defined process attributes.

e Coordinating demonstrations and sandbox sessions for product understanding.

* Created financial proposals that included vendor package and labor costs.

e Learned and applied the new roll out of Rational Unified Process Methodologies and tools with an in-
flight project team.

e Successfully managed an infrastructure project through the redesigning of an external and internal
network solution.

e Preformed and planned validations and test strategies with the RUP methodologies using Clear
Quest, Test Manager and Rational Rose tools.

HANDLEMAN COMPANY, (Troy, Ml) 4101 — 4/03
Senior Project Manager

Managed teams in project activities with staff ranging up to 30 team members Completed 2,800 hours

in Project Management practices by;

° Successfully installing major projects such as a new multi-million dollar sorter, new business system
implementation into Canada, Oracle 3.06/11i upgrade, Discreet Order system and the ERP Oracle
solution.

¢ Lead international! projects in England, Canada, Mexico, Puerto Rico, and the Virgin Islands working
with diverse cultures and business practices to achieve corporate goals.

e Introduced new processes of Project Management PMI and Software development guidelines
(SDLC) as a member of the SEPG committee, ‘ensuring adherence to methodology standards.

e Managed teams in requirement gathering; development, validation and testing, implementation and
deployment.

e Awarded the “Caught in the Act of Excellence”. Awarded twice for strategic project installations

e Demonstrated practices in estimating, budget, and financial tracking of projects.

e Certified/participation in quality concepts, tracking and application of the project life cycle.

MICHIGAN NATIONAL BANK, (Lansing, Ml) 7199 — 3/041
Project Office Manager

DJO(11/06)


1 EAC Interim Lab: CIBER.

ATTACHMENTS

1. Resumes for New Personnel

EAC Interim On Site Lab Report Page 19 of 19



LAB: CIBER 6-8 Dec 2006

aia) a) If the laboratory's document control system allows for the amendment of
documents by hand pending the reissue of the documents, the procedures
and authorities for such amendments shall be defined.

fit. b) Amendments shall be clearly marked, initialed and dated. A revised .
document shall be formally reissued as soon as practicable. Chenge bes,
'PeM & Decundok -Plendak a
ele 4.3.3.4 Procedures shall be established to describe how changes in documents
maintained in computerized systems are made and controlled.

Pom 6. Slergwe WA Poti Cerbe |
44 Review of requests, tenders and contracts

oe 4.44 / The laboratory shall establish and maintain procedures for the review of
b requests, tenders and contracts. The policies and procedures for these
i reviews leading to a contract for ‘esr wee calibration shall ensure that:
? Pan
et a) <2 the requirements, inclaalinG Te the tet das to be’ used, are adequately defined,
"S documented and understood (see 5.4.2);
Pom 7 of * Dive ash GeetiSra, 9 eect
the laboratory has the capability and resources tome

ch b) ne requirements;

se, oP eee “ea

Pam 7) ® 2 Negative ey
sg the appropriate test and/or calibration method i is seleth fed and Is capable of
x meeting the customers’ re yirements (see 5.4.2).
® Medi hl Tab tebe de
d) Any differences between the request or tender and the contract shall be
resolved before any work commences. Each contract shall be acceptable
both to the laboratory and the customer. _
Poa 7 pay 2 Monn Gmle
NOTE 1 The request, tender and contract review should be conducted in a practical
and
efficient manner, and the effect of financial, legal and time schedule aspects should be
taken into account. For internal customers, reviews of requests, tenders and contracts
can be performed in a simplified way.

NOTE 2 The review of capability should establish that the laboratory possesses the
necessary physical, personnel and information resources, and that the laboratory's
personnel have the skills and expertise necessary for the performance of the tests
and/or calibrations in question. The review may also encompass results of earlier
participation in interlaboratory comparisons or proficiency testing and/or the running of
trial test or calibration programs using samples or items of known value in order to
determine uncertainties of measurement, limits of detection, confidence limits, etc.

NOTE 3 A contract may be any written or oral agreement to provide a customer with
testing
and/or calibration services.

fi 4.4.2 Records of reviews, including any significant changes, shall be maintained.
Records shall also be maintained of pertinent discussions with a customer
relating to the customer's requirements or the results of the work during the
period of execution of the contract.

Pen 7

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 6 OF 42

be, Tenelens , otf Ghd 11 Neged abt

OR,

wig

BAA Shek faa kate dbedy



1 EAC Interim Lab: CIBER

5.3 ACCOMMODATION AND ENVIRONMENTAL CONDITIONS

Current space small and crowded, especially with new staffing. Cyphers locks on front and
back door. Site can “accommodate” four vendors by placing them into the offices which also
contain people and storage. This is a temporary situation with a move schedule to place
them closer to their Team Partner Wyle Labs.

For the Accuracy and Reliability Environmental Operating test, environmental conditions are a
critical condition of the test but this level of facilities is to be gained through the services of a
sub-contractor/Team partner, Other tests can require more space but these may be
conducted a the Hardware Team Partner facility where the environmental controls are more
appropriate to their scope of accreditation

Testing at a vendor site may involve more significant conditions. Procedures for remote
operation require the CIBER test team are defined but consist mainly of taking control of
security conditions to ensure reduced risk of interference with testing.

sopen

5.4 TEST AND-GALBRATION-METHODS AND METHOD VALIDATION

The Test Method document lists different sets of test activities required for all voting system
testing. In reviewing the coverage of these sets, test methods were not listed for a number of
test objectives, primarily those which have formerly been the responsibility of the “hardware
ITA" such as the Transportation and Storage, EMC, accessibllity. Most of these have their
own test accreditation process are not part of the core requirements for the test lab.

Non-conform. The TM does not provide for the inclusion of the non-core test requirements in
the test plan or test report. Although not a core requirement, the lab needs to include it in the
test planning and report.for direction and integration with the voting system test report as a
single document supporting a system certification.

Non-Conform, the test method for the core requirements lack validation and reports for the
validation of the tests. They appear to be too general for validation in some cases.

(See section 6 for more specific listing of missing test methods)

Non-Conform, The TM specifies the application will be installed by the vendor and fails to
provide verification that the software installed matches the Witnessed Build including the
operating system and third party software.

Non-conform. The test lab needs to explicitly identiy tests that it does not hold accredition Be,

EAC Interim On Site Lab Report Page 14 of 19



LAB: CIBER 6-8 Dec 2006

e) details ofany environmental conditions during sampling that may affect the
interpretati ae test results;
f) any standard olother specification for the sampling method or procedure,

and deviations, additions to or exclusions from the specification concerned.

5.10.4 Calibration certificates

§.10.4.1 \ In addition to the requirements listed in 5.10.2, calibration certificates shall
iaclude the following, where necessary for the interpretation of
calibration results:

a) the conditions (e.g., environmental) under which the calibrations were made

b) the uncertaiMty of measurement and/or a statement of compliance with an
identified metralogical specification or clauses thereof;

c) evidence that the measurements are traceable (see Note 2 in 5.6.2.1 1).
5.10.4,2
a) *\ The calibration certificate shall relate only to quantities and the results of

Functional tests.

b) If statement of compliance with a specification is made, this shall identify

whictiglauses of the specification are met or not met.

c) When a Statement of compliance with a specification is made omitting the
measurement results and associated uncertainties, the
laboratory: shall record those results and maintain them for possible ;
future reference.

d) When statement® of compliance are made, the uncertainty of measurement
shall be taken into ‘account.

s,

5.10.4.3 Wher\an instrument for calibration has been adjusted or repaired, the
calibration results before and after adjustment or repair, if
available, shall be reported.

5.10.4.4 A calibration certificate (or calibration label) shall not contain any
recommendation on the calibration interval except where
this has béen agreed with the customer. This requirement may
be supersedad by legal regulations.

5.10.5 Opinions and interpretations

ok When opinions and interpretations are included, the laboratory shall document the
basis upon which the opinions and interpretations have been made. rai ions and
interpretations shall be clearly marked as such in a test report. yor \ + if .

NOTE 1 Opinions and interpretations should not be confused with inspections and product
Certifications as intended in ISOMEC 17020 and ISOMEC Guide 65,

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 32 OF 42



1 EAC Interim Lab: CIBER

4.3 DOCUMENT CONTROL

The CIBER Sharepoint Internet service hosts the ITA Portal that is the control point for
documenis, reports, tracking logs, and resources. Access through sharepoint allows
individual access control to define and limit access to specific resources and maintain version
separation of information/procedures with seamless version control showing active versions
and control access to only the current versions as needed.

The ITA Portal home hosts a calendar that the QA manager keeps posted to show upcoming
QA events and required reporting. As a comment, the definition of the periodic cycle for these
events was not as well defined with a tendency to point to the next event rather than show
how these events were to be scheduled but this expected to resolved before the next review.

No non-conformities were identified

4.4 REVIEW OF REQUESTS, TENDERS AND CONTRACTS

A “test method” (TM 2) for Negoitiating supports the policies for the review of requests,
tenders, and contracts. The basic process showed no issues or problems. Specific items
that needed to be identified in a negotiation such as the areas where the CIBER ITA Practice
was not qualified under the scope of accreditation (HDBK 150-22, 5.4.6) were identified in the
later sections where encountered. ~~

No non-confroming items were noted.

EAC Interim On Site Lab Report Page 6 of 19



LAB: CIBER 6-8 Dec 2006

are made on laboratory policy or resources;

Name of person:

Amin, tbo &

PJoly gtr diln

fj) appoint deputies for key managerial personnel (see Note).

Name(s): --(yy. Wecey
Name(s): qaex ten) Got rym Cin Orgehert [143 & Sofhume t

Name(s): Oha- cu. Gebbieet

EM hong ae reece.

* k) ensure that its personnel are aware of the relevance and importance of their
activities and how they contribute to the: achievement of the objectives of the

' management system.

' ' AS ool af an aok Pees

[re _{NOTE Individuals may have more than one function and it may be impractical to appoint
deputies for every function.

—.¢ 4.1.6 Top management shail ensure that the appropriate communication
processes are established within the laboratory and that communication
takes place regarding the effectiveness of the management system.
As peek die shes ry ermine thet, atte pene CaN, Be Cerri fe
Eb ne Bey wpe Enemnnectoreth

hen 3

4.2 Management system Hien prs

4.2.1
ex a) The laboratory shall establish, implement and maintain a management
system appropriate to the scope of its activities. Pom
cb) The laboratory shall document its policies, systems, programs, procedures
and instructions to the extent necessary to assure the quality of the test
and/or calibration results. Pow b reck sede pact
Ek ¢) The system’s documentation shall be communicated to, understood by,

available to, and implemented by the appropriate personnel.
FIM Tesh 7p tte oe wee ehh Be hae pis ee’

Cis.” 4.2.2 The laboratory's management system policies related to quality, including a
quality policy statement, shall be defined in a quality manual (however
named). The overall objectives shall be established, and shall be reviewed
during management review. TTA Hee. Rated eechog & at, TH. aad off

Rofremfict bf Perpeet hocks  Clmee . _ funnta)

Date of most recent quality manual: 2b. 2oce

The quality policy statement shall be issued under the authority of top

management. It shall include at least the following: CABS Gamer, Comer k
. Te Gihihy Cte.)
fe a) the laboratory management's commitment to good professional practice and
to the quality of its testing and calibration in servicing its customers;
fx b) the management's statement of the laboratory's standard of service;
cleo) the purpose of the management system related to quality;

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 3 OF 42



B.3

LAB: CIBER 6-8 Deg 2006

Demonstration of traceability

B.3. 1 NVLAP-accredited laboratories may submit appropriate physical standards

and test and measurement equipment directly to NIST or, when
appropriate, to another national metrology institute. Accredited
laboratories may obtain certified reference materials from NIST
(caNed Standard Reference Materials under copyright) or from
anotter national metrology institute. Use of a national metrology
institute other than NIST shall be documented and will be assessed
by NVI

B.3.2 Testing labor tories that perform calibrations only for themselves do not need to

be borers as calibration laboratories. Calibration laboratories
that perforin specific calibrations only for themselves to support
their accredited services do not need to be accredited for those
calibrations.\For the purpose of assuring traceability, an
accredited lakoratory may calibrate its own equipment if the
appropriate requirements of NIST Handbook 150 have been met.

B.3.3 NVLAP-accredited laboratories that do not demonstrate traceability as

described in B.3.1, or B.3.2, shall use accredited calibration
laboratory services\wherever available. Accredited calibration
laboratories are those accredited by NVLAP or by any accrediting
body with which NVLAP has a mutual recognition arrangement. A
listing of NVLAP-accredited calibration laboratories and of
accreditation bodies with which NVLAP currently has agreements
is available from NVLAP,

B.3.4 If a NVLAP-accredited laboratory submits physical standards or test and

4)

that is not accredited by NVLAP or by an accrediting body with
which NVLAP has a mutual recdgnition arrangement, the laboratory
shall:

measurement equipment We calibration service provider

document that an appropriate accrediteti calibration service provider is not
available;

audit the claim of traceability of the provider tf the calibration service and
document the following areas related\o the calibration and
claim of traceability of its standards and test and measurement equipment:

information regarding assessment of the a 'y system used by the
calibration service provider,

the calibration procedure(s) used by the calibration service provider,

the physical standards or other test and measurement equipment used
by the calibration service provider (including evidence of
traceability to standards maintained by NISK or an
appropriate national metrology institute and copes of relevant
calibration certificates),

information regarding the calibration intervals of relevant standards or
L ni pment.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 39 OF 42



LAB: CIBER 6-8 De¢ 2006

NIST HANDBOOK 150 CHECKLIST
COMMENTS AND NONCONFORMITIES

Instructions to the Assessor: Use this sheet to document comments and nonconformities.
For each, identify the appropriate item number from the checklist. Identify comments with a
“C” and nonconformities with an “X.” If additional space is needed, make copies of this page

(or use additional blank sheets).

item No. C or X Comments and/or Nonconformities

TAS € wecmece CA Aremmuacle mile
a rpg Gh 200 fer. ;
P20 _€ Pe Tae Reg de Hadad

re a Porrk hopes 4 Ga celact dak panesidtax, Pructteds (re,

Bets Luvelecmma va Wnt av ete a eengeccin Phecs ca

Yen Lt
cumel cove & sblec af pas whee aus Spea bive: ul he en ege-,
430 6 fhe veo ihe Je peavelic. bet Plu UM Acbiedts Git Pin Qd Aches
© gvenl ta drcematt A ahered in tin UTA Plefeel book” and
pled no celraden “Flan is tes Clie dilole Chasing “ plnacel
Schutte de decminee but tir celeelere pisceai fl fb the, pact
QA fine afl gehebild, fo daveian,
HSA C Mea teonples be cated fa Ceccina le [nee Clecle. an next vis?
feast etheg,

. i ; , oO ,
Bie Show exon ds of dog fe crnsther GRE Sf
7

silat hea andy “Me -Comslarnes bog oat te
Cecrdeme Complore Leg" tthe weed otic sokemvisio. foram ess

ta dhe Lay Hbecl ft “tales Has i reained Jsiypor the

TTA hese

&

Clecunes’,
iby iS Ode | —
Sky ee gusa winks

FE gees quitelies (Corb feud jes cdlussy \ ie Sica fe 4,

p04. reales Lawle! ka the srerdes Poulesnt (eh

*#i2Q _e@
(Hetgop. Sire. types by book Che Ohocrdus, taba ol ye”
sae sd OE? peletien pe bees ation lidstly
al. 3 ‘ z A id Ree et. fb ws LAG, L as GEp9 more sil, bi dls has beim
‘ to Fe rece s  enioe (hot bation

performs ered. Hes eattigh
Se, JFeliicine Fein ¢
Ceerbediwe! or byob,

hls ALE,

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 41 OF 42



LAB: CIBER 6-8 De¢ 2006

NIST HANDBOOK 150 CHECKLIST
COMMENTS AND NONCONFORMITIES

Instructions to the Assessor: Use this sheet to document comments and nonconformities,
For each, identify the appropriate item number from the checklist. Identify comments with a
“C’ and nonconformities with an "X.” If additional Space is needed, make copies of this page
(or use additional blank sheets).

Item No. C or X Comments and/or Nonconformities
‘ 7 pit ’ .
Wise} wk Hel campletud Te un submits) ec crapletin of Hevea
=

af Phen be Gree. Grae hae bier pena lbs.
—_ =F
BAe & Treks) plewet ces Bvesead cond Cewset peri f def 5
bat Hees te alta, des ptzlon fal ofa CAG estes

tee elade it per fo hepa b Ceaepats a otivity oe.

tified. ach ucts
HT KX Prosslns fy hencilis dle! wa Stizia

Sip eben nels Sow _}

U

BHA t Glccediity Ge diegeling aos! aerette of cue Teles cao esl
_ thee" Tecg ¢ bon lis, & chow spastic & Gree are
Cetpenk Hevt nedjetin Hak: eof paseslle elem tel fos

Lite Aad cebtdry

bore

7 —

Buse thin So ne Sel vrbaitenbea n £ The th cle vel ‘
py saul rm & enwelfc Le ,

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 42 OF 42



VICTOR P. DAILY
Senior Software Engineer

g a i
Cy

Ey

a

Recipient of two Extra Miler Awards in 1990 and 1992 for exceptional job performance.

DJO (7/06)


] EAC Interim Lab: CIBER

8. Senior Software Engineer: * Victor (Vic) Daily
9. Software Engineer: Jack Cobb

# Primary Contacts.

*New Personnel (Resumes attached):

1. Amber Willburn, Proposal Manager, has been working with Shawn Southworth,
the ITA Practices Director, and other resident staff to revise the documentation
and develop the processes for the quality management of the certification testing.
She serves as the acting QA Manager but is expected to be replaced by a
permanent position for a full-time resident QA Manager. Amber has been
mentored within the QA operations by Terry Debell, Manager, Internal Audit &
Compliance

2. Kelly Rohacek, Project Manager, has been added to the staff to assist in the
project management of the test campaigns/projects. In this position she will be
the Technical Lead Contact. She is mentored by John Manning, Consulting
Services Manager.

3. Victor (Vic) P. Daily, Senior Software Engineer, has been added to the Test
Engineers/Software Engineers.

BAC Interim On Site Lab Report Page 3 of 19



1 EAC Interim Lab: CIBER

ITA Practices Organization:

ciber

ITA Organization

Key Personnel:

1. Executive Management Steering Committee:
a. Wally Birdseye, President CIBER Federal
b. Paul Rainville, Director of Delivery, Federal Civilian Branch
c. Terry Debell, Center for Project Performance
Center for Project Performance: Terry Debell, Manager, Internal Audit &
Compliance
Business Unit Leader: Robert MacFarlane, Federal Civilian Branch
ITA Practice Director: #Shawn Southworth
Project Manager: *#Kelly Rohacek
. QA Configuration Manager: *Amber Willburn
Test Engineers:
7. Document Reviewer: Diane Grey

N

ANA

EAC Interim On Site Lab Report Page 2 of 19



I EAC Interim Lab: CIBER

5.5 EQUIPMENT

TM provides for test equipment (support, not the equipment to be tested and certified); to be
checked in and inventoried but does not include provisions for maintenance, setup and
validation that is operating correctly and for the intended purpose, handling of damaged
equipment, or disposal for either CIBER owned or rented equipment or that provided by the
vendor for testing such as certified pieces needed to complete test objectives.

In the latter case, readiness testing, care, validation, and setup verification are equally
important as for the Equipment under test but needs the care extended beyond the actual test
campaign. This area should be relatively minor unless specialized equipment is involved.

This section was not completely reviewed due to time limits and little applicability

5.6 MEASUREMENT TRACEABILITY

Not applicable at this time

EAC Interim On Site Lab Report Page 15 of 19



] EAC Interim Lab: CIBER

4.1 ORGANIZATION

In response to the previous assessment, CIBER has responded by providing resources to
support:

a. QA Configuration Manager. Ms Willburn has been attached to provide support in QA
and QA Management documentation until a full-time QA manager can be approved and
hired.

b. Kelly Rohacel has been assigned as Project Manager to steer the test campaign
projects and the completion of tasks and products with review.

c. The President of CIBER Federal formed an Executive Management Steering
Committee (EMSC) and to provide high level management attention and overview
through the corporate CIBER audit and other QA reporting paths and to ensure that the
process improvement activities received the attention and resources needed.

d. Key management positions were tasked to provide mentoring support to the new key
personnel and for follow up attention.

All this may be an initial flush of action and response but the activities included the activation,
test, and use communication paths to establish use and practice. In the next assessment
review, the organization support indicated in the Organization chart should be checked for
continuation of support.

No non-conformicies were identified.

4.2 MANAGEMENT SYSTEM

The management documentation was changed from four or more major documents to two
major documents supported by forms and some independent procedures.

ITA Practice Operations Manual provides a more streamlined management procedures
document.

CIBER ITA Test Methods captures the technical test procedures, documentation and
reporting, .

Upward management processes and reporting are defined through corporate CIBER
procedures to maintain communications channels and ensure follow up review of issues and
problems.

The CIBER Sharepoint Internet service hosts the ITA Portal that is the control point for
documents, reports, tracking logs, and resources. Access through sharepoint allows
individual access control to define and limit access to specific resources and maintain
separation of information/procedures with version control showing active versions and control
access to only the active versions as needed.

ommenk fee close le WKS

4.2.7 (HDBi150) requirement to maintain integrity during planned change is a new
accreditation requirement that needs some basic attention to initial setup. .

EAC Interim On Site Lab Report Page 5 of 19



ANIBER WILLBURN

Proposal Manager c1iber

Asylum Youth Project Summer 2002
Director

e Implemented youth center for a small rural town.
e Designed all marketing material and developed Procedures and Policies manual for volunteers.
* Provided monthly progress, cost reports, and a large season-end report to Board of Directives.

Scribe 2001 ~ 2002
Reporter

° For a small newspaper, contributed numerous articles, including front page news. Responsibilities
included locating newsworthy stories, interviewing eye witnesses, and writing and editing articles.

HNE (11/10/06)

|
|


EAC Interim Accreditation CIBER 1/18/2007
Revisit (Jul 2006)

Election Assistance Commission (EAC)
Interim Accreditation
Independent Test Authorities (ITA)
Assessment Report
CIBER

Conducted: Dec 06-08, 2006
Huntsville, AL

Assessor: Steven V. Freeman

Page 1 of 4


LAB: CIBER 6-8 Dec 2006

1¢_¢) All instructions, standards, manuals and reference data relevant to the work of
the laboratory shall be kept up to date and shall be made readily
available to personnel (see 4.3),

Der commend Canter! Probate, acl, ceccth wee f deud- pred tet,

Gig d) Deviation from test and calibration methods shall occur only if the deviation has
been documented, technically justified, authorized, and accepted by

‘. the customer. af wAe Methet.. Shen by Ted tedtut
Gx is0-22 08 brah a Ae dae 2 etheke., hop Desyvioks Te a
NOTE International, regionalor national standards or other recognized specifications that fPq,

contain sufficient and concise information on how to perform the tests and/or
calibrations do not need to be supplemented or rewritten as internal procedures if these
standards are written in a way that they can be used as published by the operating staff ina
laboratory. It may be necessary to provide additional documentation for optional steps in
the method or additional details.

5.4.2 Selection of methods

The laboratory shall use test and/or calibration methods, including methods “7?
1 for sampling, which meet the needs of the customer and
t which are appropriate for the tests and/or calibrations it
} undertakes. Methods published in international, regional or nationals

standards shall preferably be used. The laboratory shall ensure
A that it uses the latest valid edition of a standare unless it is net ,
1 appropriate or possible todo so. -g. fccuwe, & Relee oi hy Tad to:
‘ wdotoid om VEEP VVSE bed heeds Bott olialepm och ove Bane
Xe b) (Cr necessary, the standard shall be supplemented with additional details
C

to ensure consistent application.
>
4 Sere” C) When the customer does not specify the method to be used, the laboratory

shall select appropriate methods that have been published
either in international, regional or national standards, or by
reputable technical organizations, or in relevant scientific texts or
journals, or as specified by the manufacturer of the equipment.

| Laboratory-developed methods or methods adopted by the: ..

| laboratory may ‘also be used if they are appropriate for the intended?

\ use and if they are validated. FF 3) Ded {et fe
“nog, need be eel EAC on a be OL ps ledlecmiigy meted,
ie, d) The custéftér shall be informed as to the method chosen, 0
: Tm Zt, Shep 7
Le) The laboratory shall confirm that it can properly operate standard methods

before introducing the tests or calibrations. If the standard
method changes, the confirmation shall be repeated.
MB ape be etl aleclacen point

ak f) The laboratory shall inform the customer when the method proposed by the
customer is considered to be inappropriate or out of date. .
Fm B60 Sep ces che 1072 chaddad Mee TE Septic
5.4.3 Laboratory-developed methods
ck a) The introduction of test and calibration methods developed by the laboratory

for its own use shall be a planned activity and shall be assigned to
qualified personnel equipped with adequate resources.

oie Me Pao Cheng & CiGa (TR teh Leb oF] yh Pak, Shek
mk b) Plans shall be updated as development proceeds and effective
communication amongst all personnel involved shall be ensured.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) “7m. iy. PAGE 18 OF 42



1 EAC Interim Lab: CIBER.

4.7 SERVICE TO THE CUSTOMER

CIBER has a strong customer service orientation. Although not noted in the checklist, it may
be worth noting the emphasis on working with previously prepared and validated test methods
to provide standard conforming tests rather than acceding too quickly to requests to modify
tests at request to vendors. The increased role of providing monitoring and direction by EAC
as a regulatory body will also support more consistent reporting of the conformance of tested
systems.

No non-conforming items were noted.

4.8 COMPLAINTS

CIBER has a Customer Complaint/feedback Survey system which is independently under the
control of upper management and is routinely used as part of the management review cycle.
This process has been actively in use for other units of CIBER but no complaints have been
submitted against the ITA Practice activity and survey reports have been favorable. The
program is restricted to customer complaints and other sources of complaints are not
routinely submitted.

No non-conforming items were noted.

EAC Interim On Site Lab Report Page 8 of 19



1 EAC Interim Lab: CIBER

4.5 SUBCONTRACTING OF TESTS AND CALIBRATIONS

CIBER has an exclusive Team Partnering agreement with Wyle Labs based on Wyle’s current
EAC Interim accreditation.

The relationship of the lead lab under the NVLAP 150/150-22 procedures and EAC
preliminary guidance needs some clarification in the instructions to recognize the accredited
voting system test lab’s increased responsibility under the core requirements as compared
with past practice of software/hardware lab. The current procedures for subcontracting
recognize the need for the subcontracted tests to be with a lab accredited for the appropriate
scope of testing but CIBER is just recognizing that their scope of responsibility for the testing
has shifted and they need to be more responsible in the direction and performance of tests
formally conducted by the ‘hardware’ labs. With Wyle’s experience and current accreditation,
this extended responsibility is blurred as Wyle is in a position to provide more of the service
and management than would expected otherwise.

CIBER will need to pay attention to develop practices in what had formerly been a Hardware
ITA exclusive area. Some later non conformance will be in specific areas where CIBER
needs to include more details on the full range of test requirements, recognizing their own out
of scope status requiring the use of other labs with the appropriate accreditation and CIBERs
responsibility in these cases to provide contractual specification of test Operation and setup
configuration information

4.6 PURCHASING SERVICES AND SUPPLIES

Not applicable

EAC Interim On Site Lab Report Page 7 of 19



LAB: CIBER 6-8 Dec 2006

measurement;
p-}) the name(s), function(s) and signature(s) or equivalent identification of
person(s) authorizing the test report or ‘Calibration certificate;
pol
ak k) where relevant, a statement to the effect that the results relate only to the

items tested or calibrated.

NVLAP Note: NVLAP defines the person(s) who authorizes the test report or calibration
certificate as the Approved Signatory (see 1.5.2).

NOTE 1 Hard copies of test reports and calibration certificates should also include the page
number and total number of pages.

NOTE 2 It is recommended that laboratories include a statement specifying that the test

report or calibration certificate shall not be reproduced except in full, without written
approval of the laboratory.

5.10.3 Test reports

5.10.3.1 In addition to the requirements listed in 5.10.2, test reports shall, where
necessary for the interpretation of the test results, include the following:
¢

oh 1 a) deviations from, additions to, or exclusions from the test method, and
information on specific test conditions, such as environiiental conditions;
perry IA
it b) where relevant, a statement of compliance/non-compliance with
requirements and/or specifications; , pe . .
Sade sy
WAr oc) where applicable, a statement.on the estimated uncertainty of measurement;

information on uncertainty is needed in test reports when it is
relevant to the validity or application of the test results, when a
customer’s“instruction so requires, or when the uncertainty affects
compliance to a specification limit;

of. d) where appropriate and needed, opinions and interpretations (see 5.10.5);
Pp SM Deters BE Loken p NET,
gli e) additional information which may be required by specific methods, customers
or groups of customers. : eapesntyy DL Genel Lefarmst abet

are m . . i matic. btw '
5.10.3.2 In addition to the requirements listed in 5.10.2 and 5.10.3.1, test reports
\containing the results of sampling shall include the

¥ollowing, where necessary for the interpretation of test results:

a) thé\date of sampling;

b) unambiguous\ identification of the substance, material or product sampled
(includitg the name of the manufacturer, the model or type of
designatign and serial numbers as appropriate);

c) the location ef sampling, including any diagrams, sketches or photographs;

d) a reference to the sampling plan and procedures used;

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 31 OF 42

fot



LAB: CIBER 6-8 Dec 2006

__f4j) recommendations for improvement;
7 k) other relevant factors, such as quality control activities, resources and staff _
training.

NOTE 1 A typical period for conducting a management review is once every 12
months.

NOTE 2 Results should feed into the laboratory planning system and should
include the
goals, objectives and action plans for the coming year.

NOTE 3 A management review includes consideration of related subjects at
regular
management meetings.

4.15.2
ke a) Findings from management reviews and the actions that arise from them
shall be recorded. Te bee Lev gupercled
x b) The management shail ensure that those actions are carried out within an

appropriate and agreed timescale.

5 Technical requirements for accreditation
5.1 General

5.1.1 Many factors determine the correctness and reliability of the tests and/or
calibrations performed by a laboratory. These factors include contributions

from:
i) human factors (5.2);
Re
ii) accommodation and environmental conditions (5.3);
we TH Of Tab A.

ili) test and eatibration metheds and method-validation (5.4);

TAL, | Debewmaf
iv) equipment ( fe feral © peels ale

55)F ra 31 Peeckics f sboiifel

v) measurement traceability-(6-6-and Annex-B);
sed CRA. Paget eens de

PPM eed bay Ly a

sheds Ue)

vi) sampling (5.7); “(ase cutteale feta \
vii) the handling of test and calibration items (5.8). _
THB Revcticel: folaotls | Tnventy log

4 5.1.2 The extent to which the factors contribute to the total uncertainty of Tar need >

T peik ohSerset ® measurement differs considerably between (types of) tests and between 2°" “% FS fa,

(types of) calibrations. The laboratory shall take account of these factors in

ee ad developing test and calibration methods and procedures, in the training and

Hh boe aes t “ze Qualification of personnel, and in the selection and calibration of the

om Uhl” equipment it uses.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 14 OF 42

Pee eee



LAB: CIBER 6-8 Dec 2006

445 The laboratory shall:
ale a) have managerial and technical personnel who, irrespective of
— other

responsibilities, have the authority and resources needed to carry out
their duties, including the implementation, maintenance and
improvement of the management system, and to identify the occurrence of
departures from the management system or from the procedures for
performing tests and/or calibrations, and to initiate actions to prevent
or minimize such departures (see also 5.2);

ai. b) have arrangements to ensure that its management and personnel
are free

from any undue internal and external commercial, financial and
other pressures and influences that may adversely affect the quality of
their work;

oc) have policies and procedures to ensure the protection of its
customers’ confidential information and proprietary rights,
including procedures for protecting the electronic storage and
transmission of results;

PR ne a t Beer ratye, Car Zoek ol Seltee,
have Ditist ahd sees ndets 2 Sia iiSiusimeat n° any ‘
activities that would diminish confidence in its competence,
impartiality, judgement or operational integrity;

fic, e) define thé organization and management structure of the laboratory,
its place .
in any parent organization, and the relationships between
quality management, technical operations and support services; ,
Conprorbe OQ yp Kato o Prone, lhe
f) specify the responsibility, authority and interrelationships of all personnel who
manage, perform or verify, work affecting the quality of the tests and/or

calibrations, Pom "D. Cobse jole alecenplin, 5

OR. g) provide adequate supervision of testing and calibration staff, including
trainees, by persons familiar with methods and procedures, purpose of each
test and/or calibratioh, and with the assessment of the test or calibration
results;

Pow S.2.( training  Reopensbr

£1... h) have technical management which has overall responsibility for the technical
operations and the provision of the resources needed to ensure the required

quality of laboratory operations;
ca WR HDR D2 FD

Name of person:

Area of responsibility:

Repeat as necessary:
Ot iy appoint a member of staff as quality manager (however named) who,
irrespective of other duties and responsibilities, shall have defined
responsibility and authority for ensuring that the management system related
to quality is implemented and followed at all times; the quality manager shall

frerre-direct-en tertire-highestte Pmremagenrentetwiteh-cheeisie

S
NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 2 OF 42


LAB: CIBER 6-8 Deg 2006

5.4.4 Non-standard methods
Xa) When it is necessary to use methods not covered by standard methods,
these shall be subject to agreement with the customer and shall
include a clear specification of the customer’s requirements and the
purpose of the test and/or calibration. a
. Nase. Utrolerpaord Sh ae dleiinn Cid mw be teehee
£.._b) The method developed shall have been validated appropriately before use.
“ey, Sp PAZ mates veloWPS needa deuelopmend of val te f&
NOTE” For new test and/or-calibration-methods, procedures should be developed priorto (“#4
the tests-and/or-catibrations-being performed and should contain at least the following
information:
a) appropriate identification;
b) scope;
c) description of the type of item to be tested or-calibrated;
d) parameters or quantities and ranges to be determined;
e) apparatus and equipment, including technical performance requirements;
f) reference standards and reference materials required;
g) environmental conditions required and any stabilization period needed;
h) description of the procedure, including:
i) affixing of identification marks, handling, transporting, storing
and preparation of
items,
i) checks to be made before the work is started,
iil) checks that the equipment is working properly and, where
required, calibration and adjustment of the equipment before each use,
iv) the method of recording the
observations and results, v) any safety measures
Ce
to be observed;
i) criteria and/or requirements for approval/rejection;
j data to be recorded and method of analysis and presentation;
k) the uncertainty or the procedure for estimating uncertainty.
_ ee eee. oe est
5.4.5 Validation of methods
5.4.5.1 Validation is the confirmation by examination and the provision of
objective evidence that the particular requirements for a specific
NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 19 OF 42



LAB: CIBER 6-8 Dec 2006
NIST HANDBOOK 150 CHECKLIST
(Adapted for EAC Interim Accreditation)

Instructions to the Assessor: This checklist addresses the general accreditation criteria
prescribed in NIST Handbook 150, NVLAP Procedures and General Requirements (2006
edition). The checklist items are numbered to correspond to the requirements found in Clauses
4 and 5, and Annexes A and B of the handbook. Items marked with ¢ indicate a change in
requirements from the 2001 edition of NIST Handbook 150.

Place an "X” beside each checklist item that represents a nonconformity, Place a “C” beside
each item on which you are commenting for other reasons. Record the item number and written
nonconformity explanation and/or comment on the comment sheet(s) at the end of the checklist.
Write “OK” beside ail other items you observed or verified as compliant at the laboratory.

EAC Interim Accreditation This checklist is adapted for use in the EAC Interim Accreditation

4 Management requirements for accreditation

41 Organization

_OK_ 4.1.1 The laboratory or the organization of which it is part shall be an entity that
can be held legally responsible.
Legal name of laboratory ownership: CIBER, Inc
Format Note: Legal name is all caps for CIBER

&" _ 4.1.2 It is the responsibility of the laboratory to carry out its testing and calibration
activities in such a way as to meet the requirements of this handbook and to
satisfy the needs of the customer, the regulatory authorities or organizations
providing recognition.

4.1.3 The management system shall cover work carried out in the laboratory's

permanent facilities, at sites away from its permanent facilities, or in
associated temporary or mobile facilities.
ae 4.1.4 If the laboratory is part of an organization performing activities other than

testing and/or calibration, the responsibilities of key personnel in the
organization that have an involvement or influence on the testing and/or
calibration activities of the laboratory shall be defined in order to identify
potential conflicts of interest.

NOTE 1 Where a laboratory is part of a larger organization, the organizational arrangements
should be such that departments having conflicting interests, such as production, commercial
marketing or financing do not adversely influence the laboratory's compliance with the
requirements of this handbook.

NOTE 2 If the laboratory wishes to be recognized as a third-party laboratory, it should be
able to demonstrate that it is impartial and that it and its personnel are free from any undue
commercial, financial and other pressures which might influence their technical judgment. The
third-party testing or calibration laboratory should not engage in any activities that may
endanger the trust in its independence of judgment and integrity in relation to its testing or
calibration activities.

NIST HANDBOOK 150 CHECKLIST (REV, 2006-03-08) PAGE 1 OF 42



LAB: CIBER 6-8 Deg 2006

Annex B (normative)

implementation of traceability policy in accredited laboratories

n
‘\
B.1 Polity overview
It is a fundamental requirement that the results of all accredited calibrations and

the results ofall calibrations required to support accredited tests shall be
traceable to the SI (the International System of Units) through standards
maintained by the National Institute of Standards and Technology (NIST) or
other internationally recognized national metrology institutes (NMIs). NIST
Handbook 150 (and ISO/IEC 17025) details the specific requirements for
traceability to be mét by testing and calibration laboratories. This annex provides
guidance as to how these requirements may be met and how traceability of
measurement can be assured by an accredited laboratory.

Internationally recognized NMIs are those that are signatory to the Comité
International des Poids et Mesures (CIPM) Mutual Recognition Arrangement
(MRA) titled “Mutual recognitiyn of national measurement standards and of
calibration and measurement cértificates issued by national metrology institutes”
and that have the necessary calibration services listed in Appendix C of the MRA,
Calibration and Measurement Capabilities (CMC). For more details on the CIPM
MRA and the CMC database, please see
<http://www.bipm.org/en/convention/mra/> or visit the NVLAP web site.

B.2 General

a) Laboratories shall be able to demonstrate
test and measurement equipment competent laboratory
personnel in a suitable environment ih performing the tests for which
accreditation is desired or held. This demonstration will include the
determination of the appropriate measurelyent uncertainty.

roper use of traceable standards and

b) Calibration certificates received by NVLAP-accredited testing and calibration
laboratories with new or recalibrated equipment shall meet the
requirements of ISO/IEC 17025. The certificatés must include the
uncertainty of measurement and/or a statement of compliance with an
identified metrological specification or clauses thereof.

Note to assessor: The NVLAP assessor(s) must, for each measurement
parameter, indicate which method the laboratory has employed to achieve
traceability. Select from B.3.1, B.3.2, B.3.3, B.3.4, or B.3.5 below. [K8.3.4 or B.3.5 is
selected, supporting documentation is also required as indicated.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 38 OF 42



Tidele
thet

l EAC Interim Lab: CIBER

ON-SITE ASSESSMENT NARRATIVE SUMMARY
CHANGES TO CURRENT OR REQUESTED SCOPE OF ACCREDITATION
EAC Interim Independent Test Authority (ITA)

ACCREDITATION STANDARD
NIST Handbook 150,
NIST Handbook 150-22,
As a documented accreditation standard to be used by the new Voting System Test
Laboratory Accreditation Program, these handbooks apply the requirements of ISO/IEC
17025. The EAC accreditation Bto¥ram-will require accredattion under the NIST
National Volunteer Laboratory Accreditation Program (NVLAP) on completion of the full
accreditation program for new labs and prior labs who complete the upgrade. This
assessment was to update the accreditation of prior ITA labs as an interim program to
ensure availability of testing labs during the transition frorn the NASED accreditation
which was terminated in July 2006 until the VSTL accreditation program had established

OVERALL SUMMARY

The management documents were completely rewritten and updated, reducing the
number of documents and confusion over what requirements were where. The new
structure has two manuals plus associated forms.

-- ITA Practice Operational Manual (POM) Quality Management program

-- CIBER ITA Test Methods (TM ). Technical test procedures

These manuals and associated forms and record files are stored on the CIBER
sharepoint service supporting controlled access for CIBER offices across the country.
The ITA Practice files and document/records are located under the ITA Portal and
provide version and access control to maintain separation of test campaign
documentation between authorized users while giving all the project team access to the
resource they need.

As part of this effort, QA procedures and a test projects (simulated) were performed
using the actual staff to make sure the procedures, tools, and resources such as the
Corrective Action Report (CAR) input forms and logs were linked, active, and accessible
(permissions granted) for ITA Practice personnel.

CIBER has entered a Team Partnering agreement with Wyle where Wyle will be
performing the non-core hardware based test requirements under separate scope of
accreditation programs. In this review, CIBER was reminded that, under the Interim
scope, CIBER will be responsible for the overall program

CIBER ITA Practice currently b as no test projects in house to test and validate these

new procedures and can not e minor corrections and adjustments until they do
so. The large rewrite has resulted in new nonconformist observations listed in the
following sections which CIBER is expected to respond to within 30 days

EAC Interim On Site Lab Report Page 4 of 19



1 EAC Interim Lab: CIBER

6.0 TECHNICAL SUPPLEMENT
Voting System Test Campaign
Procedures and Methods

The review included an overview of required test requirement categories and where they
are to be reported in the Certification test report. The following items were found
missing or requiring further work:

TDP Review:

1.k. TM 4. Test plan should include the tests that are not in the core responsiibilites
but are still required for the certification of the system. The test plan is to be complete
for all requirements. Where the test requirements are outside of the core tests, the plan
should identify the accredited lab to be used, what materials and directions need to be
given to the lab, what support is to be provided, how the labs report will be validated
(correct configuration for the certification, appropriate operations for a voting system)
and how the report is to be included In the final report

Source Code Review
2  d. Need to validate Using Exam Diff Pro and provide validation report.

Physical Configuration Audti

3b Accessibility CIBER provides test cases to Wyle

3c. Construction to be included in planning and reports but identifies as out of the
scope of accreditation for CIBER and performed by accredited lab (Wyle);

3 e. Hardware transportation and storage tests needed to be included but identifies
as out of the scope of accreditation for CIBER and performed by accredited lab
Wyle);

3 f EMC and electrical test suit. identifies as out of the scope of accreditation for
CIBER and performed by accredited lab (Wyle);

3. i. Include tests above in report.

Functional Configuration Audit

4 e. Verify HAVA functional requirements
In the Functional Requiremeins Checklist v.1.1 need to update for HAVA 301
requirements that are not in current VSS 2002 list.

System Integration Tests

a. Accuracy

b. Reliability

Also need Aj to be calculated and reported

c, Volume tests (could not find in review but may be there)
d Security tests, need to perform and add to report layout
f. Telecommunication tests per VSS 2002/HAVA

an

aaa

EAC Interim On Site Lab Report Page 18 of 19



LAB: CIBER 6-8 Dec|2006

NOTE 1 Where test items are to be returned into service after testing, special care
is required to ensure that they are not damaged or injured during the handling, testing or
storing/waiting processes.

NOTE 2 A sampling procedure and information on storage and transport of samples,
including information on sampling factors influencing the test or calibration result, should
be provided to those responsible for taking and transporting the samples.

NOTE 3 Reasons for keeping a test or calibration item secure can be for reasons of
record, safety or value, or to enable complementary tests and/or calibrations to be performed
later.

5.9 Assuring the quality of test and calibration results

5.9.4

a) The laboratory shall have quality control procedures for monitoring the

validity of tests and calibrations undertaken.

b) The resulting data shall be recorded in such a way that trends are detectable
and, where practicable, statistical techniques shall be
applied to the reviewing of the results.

c) This monitoring shall be planned and reviewed and may include, but not be

limited to, the following:

1) regular use of certified reference materials and/or internal quality control
using secondary reference materials;

2) participation in interlaboratory comparison or proficiency-testing
programs;

3) replicate tests or calibrations using the same or different methods;

4) retesting or recalibration of retained items;

5) correlation of results for different characteristics of an item.

NOTE The selected methods should be appropriate for the type and volume of the work
undertaken.

__¢ 5.9.2 Quality control data shall be analyzed and, where they are found to be
outside pre-defined criteria, planned action shall be taken to
correct the problem and to prevent incorrect results from being reported.

ee
5.10 Reporting the results

5.10.1 General 4M VE Hal be ak ree oh

wt a) The results of each test, calibration, or series of tests or calibrations carried
out by the laboratory shall be reported accurately, clearly,
unambiguously and objectively, and in accordance with any specific

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 29 OF 42



f)

—_1)

2)

—___ 3

—_— 9)

h)
1)

2)

LAB: CIBER 6-8 Ded 2006

When the NVLAP symbol is used, the form of the NVLAP logo must conform
to the following guidelines:

The logo shall stand by itself and shall not be combined with any other
logo, symbol, or graphic.

The aspect ratio (width to height) shall be 2.25 to 1.

) The lago and caption shall be of a size that allows the caption to
be\ easily read. The size of the caption shall not
exceed the size of the logo itself.

The logo shall appear in black, blue, or other color approved by
NVLAP\ and may be filled or unfilled. In the case of a
filled loyo, the same color shall be used for the outline and the fill.

The name of at leyst one Approved Signatory shall appear on a test or
calibration repon that displays the NVLAP symbol or references
NVLAP accreditation. A computer-generated report may have
the Approved Sighatory’s name printed along with the test or
calibration results, as long as there is evidence that there is a
system in place to ehsure that the report cannot be generated
without the review and consent of the Approved Signatory.
There may be legal o\ contractual requirements for original
signatures to appear on the report.

When the term and/or eynbo! are used on test or calibration reports,
such use shall be limited to reports in which some or all
of the data are from te¥ts or calibrations performed by
the laboratory under itS scope of accreditation.

A test or calibration report that contains both data covered by the
accreditation and data not govered by the accreditation
shall clearly identify the data that are not covered by the
accreditation.

The report must prominently display the ¥ollowing statement at the
beginning of the report: "This report contains data
that are not covered by the NVLAP actreditation.”

When the term and/or symbol are used on test ‘er calibration reports
that also include work done by subcontracted

laboratories, such use shall be limite
which some or all of the data are from
calibrations performed by the laboratory \nder its
scope of accreditation.

A test or calibration report that contains both data covered ‘hy the
accreditation and data provided by a subcontractor
shall clearly identify the data that were provided by the
subcontracted laboratory.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 36 OF 42


LAB: CIBER 6-8 Dec 2006
48 Complaints ~

sibs 4.8.1 The laboratory shall have a policy and procedure for the resolution of of

complaints received from customers or other parties. v
: Eaedinn Crem bes Pom &. 2 fe
—_* 4.8.2 Records shall be maintained of all complaints and of the investigations and
corrective actions taken by the laboratory (see also 4. 11).
Ne cers & Oe once ted

4.9 Control of nonconforming testing and/or calibration work

—< 4.9.1 The laboratory shall have a policy and procedures that shall be implemented
when any aspect of its testing and/or calibration work, or the results of this
ss -aé work, do not conform to its own procedures or the agreed requirements of the
2 customer. The policy and procedures shall ensure that:
Per lid Cardioe ( af Won Bardeommoner,

the responsibilities and authorities for the management of nonconforming
. work are designated and actions (including halting of work and withholding of

{. test reports and calibration certificates, as necessary) are defined and taken
Se TM when nonconforming work is identified;

whe HOp/

OK a)

; an evaluation of the significance of the nonconforming work is made;

ck) " correction is taken immediately, together with any decision about the
acceptability of the nonconforming work;

@) where necessary, the customer is notified and work is recalled;
an) the responsibility for authorizing the resumption of work is defined.

NOTE _ Identification of nonconforming work or problems with the management
system or
with testing and/or calibration activities can occur at various places within the
management system and technical operations. Examples are customer complaints,
quality control, instrument calibration, checking of consumable materials, staff
observations or supervision, test report and calibration certificate checking,
management reviews and internal or external audits. Ele: al

ia 4.9.2 Where the evaluation indicates that the nonconforming work could recur or
that there is doubt about the compliance of the laboratory’s operations with
its own policies and procedures, the corrective action procedures given in
4.11 shall be promptly followed.

vis obs Lew

hfs Pom LV.

o Ae.
4.10 Improvement

___¢ The laboratory shall continually improve the effectiveness of its management system

through the use of the quality policy, qui objectives, audit results, analysis of data,
corrective and preve iv

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 9 OF 42

Tet



LAB: CIBER 6-8 Dec 2006

4.11 Corrective action

4.11.1 General

&{\- The laboratory shall establish a policy and a procedure and shall designate appropriate
authorities for implementing corrective action when nonconforming work or departures
from the policies and procedures in the management system or technical operations
; ede fea lg”

have been identified, Py IB + Seygebe a * roan ;
=P, ¢ Bebb Eos thy alg Ska CORA) 5) os rec Ree OPE
‘i ria operations of beret

rs Cathe the foe a pe

ensSe 3
NOTE A problem with the management system orwith the t
the
laboratory may be identified through a variety of activities, such as control of
nonconforming work, internal or external audits, management reviews, feedback from
customers and from staff observations. a

4.11.2 Cause analysis

3}¢-The procedure for corrective action shall start with an investigation to determine the
root cause(s) of the problem. Bul. 3 ; Stop Y 7 AwM Lae,

NOTE Cause analysis is the key and sometimes the most difficult part in the
corrective

action procedure. Often the root cause is not obvious and thus a careful analysis of all
potential causes of the problem is required. Potential causes could include customer
requirements, the samples, sample specifications, methods and procedures, staff skills
and training, consumables, or equipment and its calibration.

4.11.3 Selection and implementation of corrective actions

Pans! a) Where corrective action is needed, the laboratory shall identify potential corrective
~ actions. It shall select and implement the action(s) most likely to eliminate the
problem and to prevent recurrence. Pam te, 8 Shy Chm WY, bee

the problem. a

{y /4. b) Corrective actions shall be to a degree appropriate ‘f the magnitude and the risk of
ee,
che

c) The laboratory shall document and implement any required changes. resulting from
corrective action investigations. ds soph 5 lee ih Workathe

4.11.4 Monitoring of corrective actions

___ The laboratory shall monitor the results to ensure that the corrective actions taken
have been effective. Tram INS Qi Clee be oP J eevied awa
Cyrrcedun te be UY (otal

seal, wube pend b> hechete jas ,
Cert, fowls nese 2 pewt aede boll.

4.11.5 Additional audits

ti Where the identification of nonconformities or departures casts doubts on the
laboratory's compliance with its own policies and procedures, or on its compliance
with this handbook, the laboratory shall ensure that the appropriate areas of activity
are, audited in accordance with 4.14 as soon as possible. BA 14.3 wilh iv \o-des

CHAS Cad td Crna
NOTE Such additional audits often follow the implementation of the corrective

actions to
confirm their effectiveness. An additional audit should be necessary only when a

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) : PAGE 10 OF 42



a
LAB: CIBER (8:8 Bec 2006

A calibration laboratory establishes traceability of its own
measurement standards and measuring instruments to the SI by
means of an unbroken chain of Calibrations or comparisons
linking them to relevant pritary standards of the SI units of
measurement. The link Ae Sl units may be achieved by
reference to national-heasurement standards. National
measurement standards may be primary standards, which
are primary rgafizations of the SI units or agreed representations
of SI units pated on fundamental physical coristants, or they
may be sécondary standards which are Standards calibrated by
anotherfational metrology institute, —

b) When using externa! calj ration services, traceability of measurement shail
be assured by Use of calibration-Services from laboratories

that can d nstrate ore ene capability and traceability.

c) The ‘calibration certificates issued by these laboratories shall contain the

# including the measurement uncertainty
@nt of compliance with an identified
metrolog}: specification (see also 5.10.4.2).

NOTE 1 Calibration laboratories fulfilling the requirements of this handbook are considered
to be competent. A calibration certificate bearing an accreditation body logo from a calibration
laboratory accredited to this handbook, for the calibration concerned, is sufficient evidence of
traceability of the calibration data reported.

NOTE 2 Traceability to S| units of measurement may be achieved by reference to an
appropriate primary standard (see VIM:1993, 6.4) or by reference to a natural constant,
the value of which in terms of the relevant SI unit is known and recommended by the
General Conference of Weights and Measures (CGPM) and the International Committee
for Weights and Measures (CIPM).

NOTE 3 Calibration laboratories that maintain their own primary standard or representation
of SI units based on fundamental physical constants can claim traceability to the SI
system only after these standards have been compared, directly or indirectly, with other
similar standards of a national metrology institute.

NOTE 4 The term “identified metrological specification” means that it must be clear from
the calibration certificate which specification the measurements have been compared with,
by including the specification or by giving an unambiguous reference to the specification.

NOTE 5 When the terms “international standard’ or “national standard” are used in
connection with traceability, it is assumed that these standards fulfill the properties of primary
standards for the realization of SI units.

NOTE 6 Traceability to national measurement standards does not necessarily require the
use of the national metrology institute of the country in which the laboratory is located.

NOTE7 Ifa calibration laboratory wishes or needs to obtain traceability from a national
metrology institute other than in its own country, this laboratory should select a national
metrology institute that actively participates in the activities of BIPM either directly or
through regional groups.

NOTE 8 The unbroken chain of calibrations or comparisons may be achieved in several
steps carried out by different laboratories that can demonstrate traceability.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 25 OF 42



KELLY ROHACEK
Project Manager

Incorporated and lead the induction of a Project Office into the IT division of Michigan National

operations center. Completed 1850 hours in Project Management practices managing a staff of five that

supported the project office time tracking and project portfolio business by;

e Training and managing compliance of the CMM and Methodology rollout in IT

° Supporting the strategic vision of system development through holding an active position on the
board of project initiatives

« Lead the intraduction and training of Risk Management rollout in IT

® Implemented processes for Quality Assurance teams using the Mercury Automated tool for validation
and test scripting. ( customized process for Michigan)

e Presented a project management concept to students of the University of Namibia, Africa.

e Lead the planning effort and implemented the business plan model for management of IT System
Solutions division

e Achieved CMM level three award by implementing business practice improvements in the project
office and through out the IT divisions

¢ Implemented the Quality Management and assurance models to the development staff through
training and process presentations

e Preformed the instruction and implementation of the time tracking tool - Primavera Team Play,
retiring the Niku / ABT workbench product from production

NATIONAL CITY / FIRST OF AMERICA, (Kalamazoo, iil) 6/89 — 6/99

Project Manager Il 1996-1999

Managed 12 programmers of the Special Projects Force and completed 4120 hours in Project

Management practices by;

e Successfully managing multiple projects related to the year 2000 preparations.

e Lead the conversation of Desk Top applications for Lotus/Ami Pro to Microsoft Word/Excel office
suites

e Preformed /planned validations in unit, system, integration, production readiness, and end user
testing

° Lead the consolidation efforts of the Project Office of First of America and National City

Business Analyst 1989-1996

e Successfully implemented small project work and gained project leadership responsibilities by
completing 3250 hours of Project Management practice by;

» Coordinating all project work for the Hogan systems by priority and business needs

e Leading requirements gathering and feasibility studies in the initiation and requirements phase of the
project life cycle

® Performing/executing test plans and validations in unit, system, integration, production readiness,
and end user testing for Hogan system

e Using and applying Project Management practices as a junior project manager and moving into a
senior project manager's role.

DJO(11/06)



LAB: CIBER 6-8 Dec 2006
based upon.

5.8 Handling of test 4nd-calibrattonitems

CG. 5.84 The laboratory shall have procedures for the transportation, receipt, handling,
protection, storage, retention and/or disposal of test and/or

calibration ttems, including all provisions necessary to protect

the integrity of the test or-calibration item, and to protect the
interests of the laboratory and the customer.

" ob Pes
THe 2 hed Bawentey bos Veh Pes
5.8.2 .
a) The laboratory shall have a system for identifying test and/or-calibratien
items. qe chvly
b) The identification shall be retained throughout the life of the item in the
laboratory.  litty
c) The system shall be designed and operated so as to ensure that items
cannot be confused physically or when referred to in records or
other documents. “yin dite
d) The system shall, if appropriate, accommodate a sub-division of groups of
items and the transfer of items within and from the laboratory.
cma Ladies
58.3 TS diate

a) Upon receipt of the test or calibration item, abnormalities or departures from
normal or specified conditions, as described in the test or
calibration method shall be recorded.

- sbotis
b) When there is doubt as to the suitability of an item for test or calibration, or
when an item does not conform to the description provided, or
the test or calibration required is not specified in sufficient detail,
the laboratory shall consult the customer for further instructions
before proceeding and shall record the discussion.

Two Aide

z

5.8.4

a) The laboratory shall have procedures and appropriate facilities for avoiding
deterioration, loss or damage to the test or calibration item during
storage, handling and preparation.

b) Handling instructions provided with the item shall be followed.

c) When items have to be stored or conditioned under specified environmental
conditions, these conditions shall be maintained, monitored and recorded.

__. d) Where a test or calibration item or a portion of an item is to be held secure,
the laboratory shall have arrangements for storage and security
that protect the condition and integrity of the secured items or portions
concerned.

NIST HANDBOOK 150 CHECKLIST (REV. 2008-03-08) PAGE 28 OF 42



—_))

k)

!) Laboratories shall not use the terms certifi

LAB: CIBER 6-8 Deq 2006

The report must prominently display the following statement at the
beginning of the report: “This report contains data that

ere produced under subcontract by Laboratory X." If the
sukcontracted laboratory is accredited by NVLAP, then its Lab
Codé&should also be stated.

If the sulgcontracted laboratory is accredited by a body other than
NVLAP, ‘then the name of the accreditation body and the
laboratory\s number or other unique identifier should
also be stated. If the subcontracted laboratory is not accredited,
then this must Be stated.

Each test or calibration réport bearing the term and/or symbol shall include a
statement that the repert must not be used by the client to claim
product certification, alyproval, or endorsement by NVLAP, N
IST, or any agency of tha Federal Government.

When used in a contract or propdsal, the term and/or symbol shalt be
accompanied by a description df the laboratory’s scope of
accreditation and current accreditation status.

or registered when referencing
their NVLAP accreditation or conformance to ISO/IEC 17025
requirements. The correct term is accredited.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 37 OF 42



EAC Interim Accreditation CIBER 1118/2007
Revisit (Jul 2006)

(Other tests and requirements may need to be developed in the TM but were not reviewed in this
assessment.)

Recommendation

CIBER has made substantive improvements in the documentation and implementation of the
quality assurance policies and procedures. A similar level of improvement is provided in the
CIBER Test Method procedures and the potential exists to develop and respond as additional
requirements are recognized as needing to be included.

If CIBER is accredited, their initial test plans and some of the more critical tests such as accuracy,
reliability, and security should be subject to test plan review and on-site observers until the TM
has had a chance to be validated and samples of the test reports are available to verify the level

of reporting is adequate. This should include clarification of the tasks to be performed by CIBER
personnel and Wyle personnel.

(signed)

Steven V. Freeman

Page 4 of 4


nib)

Bi& 5.3.3

Mla 6.3.4

LAB: CIBER 6-8 Dec 2006
such as to facilitate correct performance of the tests and/or calibrations.

mTM Sf al neste re a
,

Herteot Meme
The laboratory shall ensure that the environmental
conditions do not invalidate the results or adversely affect the
required quality of any measurement. Particular care shall be
taken when sampling and tests and/or calibrations are
undertaken at sites other than a permanent laboratory facility.

The technical requirements for accommodation and environmental conditions
that en affect the results of tests and calibrations shall be documented.
PBA ty fa, A 8s stu aEaghet firb Cow crea
Tr Io. ( andi. frill, nee C
Td kaye
The laboratory shall monitor, contro! and record environmental conditions as
required by the relevant specifications, methods and
procedures or where they influence the quality of the results.
Due attention-Shall be paid, for example, to biological sterility,
dust, electromagnetic disturbances, radiation, humidity, electrical
supply; temperature, and sound and vibration levels, as appropriate to
the téchnical activities concerned
ISG Gasvieady pe
Tests and calibrations shall be stopped when the environmental conditions
jeopardize the results of the tests and/or calibrations. , \ del
ae a

Thecus Howl) be tepile oy enilery.
There shall be effectivé separation between neighboring areas in which there

are incompatible activities. Measures shall be taken to prevent
cergss<contamination.

Access to and use of areas affecting the quality of the tests and/or
calibrations shal be controlled. The laboratory shall determine the

extent Of contro! based on its particular circumstances.

colt 5.3.5 Measures shall be taken to ensure good housekeeping in the laboratory.

Special procedures shall be prepared where necessary.
Free Sewxiily

5.4 Test and calibrationametheds and method validation

5.4.1

_bA by

a ee calibrations within its scope. These include‘Sampling,
2 Bra

peer, tested-ahd/erealibrated, and, where appropriate, an
we time nes

General

The laboratory shall use appropriate methods and procedures for all teste
ndlingStransport,“Storage and preparation of iteris to be”

. ation of the méasurement uncertainty as well as statistical
” techniques for analysis of test and/or calibration data. Peomed Seg

seni MY Kannhiodt carmecc ppm foie Ter bitelel on
Cee]

The laboratory shall have instructions on the use and operation of all relevant Ch aloceeneng-
equipment, and on the handling and preparation of items for °
testing and/or-calibratien; or-beth, where the absence of such
instructions gould jeopardize the results of tests and/or calibrations.

te theft

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 17 OF 42



LAB: CIBER 6-8 Deg 2006

5.2 Personnel
5.2.1
au a) The laboratory management shall ensure the competence of all who operate

specific equipment, perform tests and/or calibrations, evaluate results, and
sign test reports and calibration certificates. Pon Set Pecan nate
S20 Fpeskedy Rewapeensle ber, :

When using staff who are undergoing training, appropriate supervision shall

be provided. Personnel performing specific tasks shall be qualified on the

basis of appropriate education, training, experience and/or demonstrated

skills, as required. 2, Urelasda: pernannad ebtaueler
ade We tr. Sper ui be. .< were’ « .

NOTE 1 In sometechnical areas eg. nondestructive testing) it may be required that

the

personnel performing certain tasks hold personnel certification. The laboratory is

responsible for fulfilling specified personnel certification requirements. The

requirements for personnel certification might be regulatory, included in the standards

for the specific technical field, or required by the customer.

NOTE 2 The personnel responsible for the opinions and interpretation included in test
reports

should, in addition to the appropriate qualifications, training, experience and satisfactory
knowledge of the testing carried out, also have:

i) relevant knowledge of the technology used for the manufacturing of the items,
materials, products, etc. tested, or the way they are used or intended to be
used, and of the defects or degradations which may occur during or in service;

ii) knowledge of the general requirements expressed in the legislation and
standards; and

ii) an understanding of the significance of deviations found with regard to the
normal
use of the items, materials, products, etc. concerned.

5.2.2
Gl a) The management of the laboratory shall formulate the goals with respect to
the education, training and skills of the laboratory personnel. ~ im
ih Bee Bey vend opiate td an ( $7.0) pee yo bs fuk
ef b) The laboratory shall have a policy and procedures for identifying training
needs and providing training of personnel. Pon 57, 1
if 0) The training program shall be relevant to the present and anticipated tasks of
the laboratory. $2 ga. © Trainin 3} Gr TIA personne trek
@ deovden (Ta weoeyny E person pel prog asi
wz ¢ d) The effectiveness of the training actions taken shall be evaluated. Seton
frm 82,1 “Pthsne, pda orranbe fof , _
523 | be ekn. fe ABB BOP twhefhodls &Indarell erygi
of a) The laboratory shall use personnel who are employed by, or under contract .
to, the laboratory. Pin 8 Subcentiin fey amen euFes f gurtlafree

nb fe ee

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 15 OF 42



LAB: CIBER 6-8 Dec 2006

5) the environmental conditions of the laboratory,

6) \ the method(s) by which uncertainties are determined (e.g., Guide to the
\ pression of Uncertainty in Measurement (GUM), and

7) \ the relative uncertainties achieved at all steps of the process;
\

c) pursue the traceability chain until traceability to appropriate stated references is
completel) validated, when a calibration service provider submits
physical staydards and/or test and measurement equipment used in
the calibration to another laboratory(s) not accredited by NVLAP;

d) enter the audit doclymentation, including all findings of nonconformance and
resolutions of thase findings, into the laboratory’s quality
management record-keeping system.

not required as long as the informatidxy listed above is obtained and otherwise verified,
Self-declaration of compliance to | EC 17025 or other relevant standards by a
calibration service provider is not acceptable evidence of verification of traceability.
Citation of a NIST Test Number by the cal ipration service provider likewise is not
acceptable evidence of verification of traceability.

NOTE An on-site visit to the provi se ite calibration service is encouraged, but is
so

B.3.5 If traceable calibration services are not available or appropriate, laboratories
may demonstrate comparison to\a widely used standard that
is clearly specified and mutually agreeable to all parties concerned,
particularly in measurements where NIST does not maintain a U.S.
national standard. For example, NIST does not maintain a standard
for all hardness testing scales. There aré\several widely used
commercial standards available for hardness; However, these
standards may not all give equivalent measurement results;
therefore, it is important to specify which standard is used and to
obtain agreement among all parties involved that the chaice made is
acceptable.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 40 OF 42



1 EAC Interim Lab: CIBER

4.13 CONTROL OF RECORDS

The sharepoint ITA Portal supports the storage, maintenance and control of records as well
as documents. Under this the records are identified, ownership is established and versioning
is automatically performed, All management and test records are uploaded to the appropriate
sharepoint directory with the exception of HR sensitive records which are processed, along
other networks to preserve confidentiality. Hardcopy manual records are stored in QA/CM
secure filing cabinets and logged in a directory log in a directory within the ITA Portal. Test
campaign records are stored in Project Workbooks with different directories for each vendor
to maintain separation and protection of records.

Current record retention is the life of the project + 3 years. CIBER is to review the new EAC
Certification Program Manual and consider adopting the matching retention for election
records. No disposal procedures are specified and are to be developed. Check with next
assessment review.

Backup copies are saved daily and maintained in another location by corporate policy. The
policy includes the periodic test of backup recovery.

The records for each test were not checked for the identity of personnel responsible as there
are no Project Workbooks active (pending accreditation). However, procedures/test methods
do require such records.

No non-conforming issues were identified

4.14 INTERNAL AUDITS

The last internal audit was completed 15 Nov in response to the previous assessment report
by Terry Debell from the CIBER Center for Project Improvements. Improvements have been
initiated out of that audit and it is to be reviewed in the next management review (pending) by
the Executive Management Steering Committee (see organization information).

No non-conforming issues identifed

EAC Interim On Site Lab Report Page 11 of 19



5.7

LAB: CIBER 6-8 Dec 2006
as is technically and economically practicable.

5.6.3.3 Intermediate checks

Checks needed to maintain confidence in the calibration status of reference, primary,
transfer or wor ig standards and reference materials shall be carried out according to
defined pr dures and schedules.

5.6.3.4 Transport and storage

The laboratory shall Gedures for safe handling, transport, storage and use of
reference standafds and reference materials in order to prevent contamination or
it ind in order to protect their integrity.

ea
NOTE Additional procedures may be necessary when reference standards and reference
materials are used outside the permanent laboratory for tests, calibrations or sampling.

Sampling
571
\ , ,
a) \ The laboratory shall have a sampling plan and procedures for sampling when

it carries out sampling of substances, materials or products for
\ subsequent testing or calibration.

b) The sampling plan as well as the sampling procedure shall be available at
thé\location where sampling is undertaken. Sampling plans shall,
whohever reasonable, be based on appropriate statistical

methods. The sampling process shall address the factors to be

controlled to ensure the validity of the test and calibration results.

NOTE 1 Sampling is ddefined procedure whereby a part of a substance, material or
product is taken to provide for testing or calibration of a representative sample of the whole.
Sampling may also be required by the appropriate specification for which the substance,
material or product is to be tested or calibrated. In certain cases (e.g., forensic analysis),
the sample may not be represehtative but is determined by availability.

NOTE 2 Sampling procedures sfguid describe the selection, sampling plan, withdrawal
and preparation of a sample or samples from a substance, material or product to yield the
required information.

5.7.2 Where the customer requires deviations, additions or exclusions from the
documented sampling procedure, these shall be recorded in detail
with the appropriate sampling ata and shall be included in all
documents containing test and/sr calibration results, and shall be
communicated to the appropriate

5.7.3 The laboratory shall have procedures for
operations relating to sampling that ms part of the testing or
calibration that is undertaken. These rederds shall include the
sampling procedure used, the identification of the sampler,
environmental conditions (if relevant) and dkagrams or other
equivalent means to identify the sampling location as necessary
and, if appropriate, the statistics the samplikg procedures are

cording relevant data and

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) ‘, PAGE 27 OF 42



LAB: CIBER 6-8 Ded 2006

NOTE 2 Opinions and interpretations included in a test report may comprise, but not
be limited to, the following:

i) an opinion on the statement of compliance/noncompliance of the
results with
requirements;

ii) fulfillment of contractual requirements;

iii) recommendations on how to use the results;

iv) guidance to be used for improvements.

NOTE 3 In many cases it might be appropriate to communicate the opinions and
interpretations by direct dialogue with the customer. Such dialogue should be written down.-

5.10.6 Testing anétalibration-results obtained from subcontractors «

fa) When the test report contains results of tests performed by subcontractors,

these results shall be clearly Haentifer ce5 CL unl del sxalarl- 8 “ich

Lew

tak 49. G rare
até b) The subcontractor shall report the results in writing of Lieto Binz &
. MOT %, 2 ‘
wht ¢) When a calibration has been subcontracted, the laboratory performing the
work shall issue the calibration certificate to the contracting laboratory.

5.10.7 Electronic transmission of results

he In the case of transmission of test or calibration results by telephone, telex, facsimile or
other electronic or electromagnetic means, the requirements of this handbook
shall be met (see also 5.4.7). Pris | Yo Deut

5.10.8 Format of reports and certificates

of The format shall be designed to accommodate each type of test or calibration carried
out and to minimize the possibility of misunderstanding or misuse.
tere he of BS kaa B signa
NOTE 1 Attention should be given to the layout of the test report or calibration certificate,
especially with regard to the presentation of the test or calibration data and ease of
assimilation by the reader.

NOTE 2 The headings should be standardized as far as possible.
5.10.9 Amendments to test reports and calibration certificates
oh a) Material amendments to a test report or calibration certificate after issue shall

be made only in the form of a further document, or data transfer,
which includes the statement:_.. (2. (

at A
_ Le Lo
"Supplement to Test Report [or Calibration Certificate], serial

y fiumber... for as otherwise identified],” or an equivalent form of wording.
he b) Such amendments shall meet all the requirements of this handbook.
a) ian oo

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 33 OF 42



1 EAC Interim Lab: CIBER

5.1 GENERAL

The technical test procedures have been completely rewritten into the CIBER ITA Test
Methods. The organization of the document is oriented to the description used in the HDBK
150-22 Technical Supplement (Section 6.0) showing the test/review areas from the Voting
System Standards (VSS) 2002 core responsibilities.

5.2 PERSONNEL

Resumes fosition description, and training records were reviewed for the new personnel:
Amber Willburn QA Manager (temp)

Kelly Rohcek Test Project Manager

Vic Daily Software Engineer (Tech Engineer)

Amber was brought in primarily as a technical writer to assist in preparing the QA
management documentation and resources. She has little prior experience of the position but
here work on developing the program under the Mentorship of Terry Debell has prepared to
some degree.

Kelly Rohcek is highly qualified as a technical Project Manager but lacks voting system
experience

Vic Daily has extensive technical testing background but lacks voting experience.

Position descriptions were adequate and gave relevant information.
The training records, while showing an active training program were inconsistent with different
names Scope of training for the same activity. No standardized training plan appears to
exist beyond corporate policy of Security and 30 day training.

sahsek Po
Non-conforming. No clear designator that a test engineer is qualified or for.what methods.

EAC Interim On Site Lab Report Page 13 of 19



EAC Interim Accreditation CIBER 1/18/2007
Revisit (Jul 2006)

Summary of Findings

A thorough internal audit was performed after the July assessment and a plan for correction
developed [NC Item a] Within corporate CIBER, an Executive Management Steering Committee
was created consisting of the President of CIBER Federal Civilian, VP Contracts Federal, and a
Consulting Services Manager to perform high management oversight and ensure adequate
resources would be available. Personnel have been brought in to assist in revising the program
and to assist with responsibilities in the Quality Assurance (QA) and Project Management roles
pending the staffing of additional staffing positions for ongoing support. [NC Item c].

The management documents were completely rewritten and updated, reducing the number of
documents and confusion over what requirements were where. The new structure has two
manuals plus associated forms.

-- ITA Practice Operational Manual (POM) Quality Management program

- CIBER ITA Test Methods (TM ). Technical test procedures

QA procedures and a test projects (simulated) were performed using the actual staff to make sure
the procedures, tools, and resources such as the Corrective Action Report (CAR) input forms and
logs were linked, active, and accessible (permissions granted) for ITA Practice personnel. The
CAR contained entries resulting from the internal audit. [NC Item b.]

CIBER has entered a Team Partnering agreement with Wyle where Wyle will be performing the
non-core hardware based test requirements under separate scope of accreditation programs. In
this review, CIBER was reminded that, under the Interim scope, CIBER will be responsible for the
overall program. CIBER still needs to develop procedures in what had formerly been a Hardware
ITA exclusive area, recognizing their own out of scope status for some requirements requiring the
use of other labs with the appropriate accreditation. The procedures shall include CIBER’s
responsibility in these cases to provide contractual specification of test operation and setup
configuration information

CIBER ITA Practice currently has no test projects in house to test and validate these new
procedures and can not complete minor corrections and adjustments until they do so.

Minor revisions and non-compliant items identified as needed include:

1. Updating record retention periods to support the EAC Voting System Testing and Certification
Program Manual (EAC Certification Program).

2. Records showing when and for what methods test engineers are qualified.

3. Including reporting of non-core requirement testing in the Test Plan and Report

4. Validation of test methods for the core requirement test methods.

5. The TM specifies the application will be installed by the vendor and fails to provide verification
that the software installed matches the Witnessed Build including the operating system and third
party software.

6. The ITA needs to explicitly specify in contracts and reports which test methods that the ITA
does not hold accreditation for and what accredited lab will be used for such tests.

7. Test methods and tools need to be validated and the validation documented.

8. In the Functional Requirements checklist, need to incorporate HAVA 301 requirements in with
the Voting System Standard (VSS-2002) requirements.

9. Need to include Accuracy test details in TM. Some issues exist over whether automated
testing using simulated voter inputs are acceptable for this test, especially in the environmental
phase of the test.

10. Need to include Reliability test details in TM.

11. Need to calculate and report the Availability Index in TM.

12. Need to include volume tests

13. Need to include required security tests in TM and in Test Report.

14. Need to include Telecommunication tests (in System Integration testing)

Page 3 of 4


LAB: CIBER 6-8 Dec 2006

with specified requirements.
N A b) Records of actions taken to check compliance shall be maintained.

wk 4.6.3 Purchasing documents for items affecting the quality of laboratory output
shall contain data describing the services and supplies ordered. These
purchasing documents shall be reviewed and approved for technical content
prior to release.

NOTE The description may include type, class, grade, precise identification,
specifications,

drawings, inspection instructions, other technical data including approval of test ©
results, the quality required and the management system standard under which they

were made.
4.6.4
ye a) The laboratory shall evaluate suppliers of critical consumables, supplies and

services which affect the quality of testing and calibration, and

Lh b) shall maintain records of these evaluations and list those approved.

47 Service to the customer

fk 4.7.1 The laboratory shall be willing to cooperate with customers or their
representatives in clarifying the customer's request and in monitoring the
laboratory's performance in relation to the work performed, provided that the
laboratory ensures confidentiality to other customers.
Pon L Bek & ls Ue 2 TROT Sump
NOTE 1Such cooperation may include:

a) providing the customer or the customer's representative reasonable

access to relevant areas of the laboratory for the witnessing of tests

and/or calibrations performed for the customer; ep ~7

Pom § Seven te Cie! bey oe Selah Co tier fed beh.

b) preparation, packaging, and dispatch of test and/or calibration items needed

by the customer for verification purposes.

Pe

NOTE 2 Customers value the maintenance of good communication, advice and
guidance in
technical matters, and opinions and interpretations based on results. Communication
with the customer, especially in large assignments, should be maintained throughout
the work. The laboratory should inform the customer of any delays or major deviations
in the performance of the tests and/or calibrations.

*¢ 47.2 The laboratory shall seek feedback, both positive and negative, from its
customers. The feedback shall be used and analyzed to improve the
management system, testing and calibration activities and customer service.

ak

: beh Contre £e il bsce
NOTE Examples of the types of feedback include customer satisfaction surveys and

review
of test or calibration reports with customers.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 8 OF 42



KELLY ROHACEK et
Project Manager Chloe!

A Senior Project Manager professional with expertise in Banking, Information Technology (IT),
Manufacturing, Supply Chain, and Defense and Aerospace programs. Seventeen years of proven
management expertise with a strong business sense for converting business needs into development
solutions.

TECHNICAL EXPERIENCE

Languages:

Software/Tools: MS Project 98/2000/2003 ; Niku/Clarity, Primavera/Team Play, Lawson/Account
4, Planveiw/5.3 and 7.2, time tracking, project portfolio, and reporting; WBS
Chart Pro, Microsoft suite; Lotus Notes office suite; Mercury Test Tool, Aristotle,
Sharepoint, Clear Quest, RUP, PVCS.

Systems: Windows, AS400. Pobre 4 Uedke se

Pt tae:

Databases: Oracle. UR

PROFESSIONAL EXPERIENCE

SANMINA SCI, (Huntsville, AL) 42105 -— 10/06

Program Manager /IPT Lead

Managed “build to print” programs in the Defense and Aerospace division of Sanmina — SCI. Completed

1440 hours in project initiation, planning, execution, and control by;

» High level performance in the Program Managers role for a 37M program of In Flight Entertainment
(IFE) products for an Avionics customer — France and US, managing two of the five products

¢ Promoted to Program Manager with in three months by learning the manufacturing industry, its
standards and processes as an ITP Lead.

e Lead New Product Implementation (NPI) team’s in preparation for new electronic product induction.

Planned, executed and controlled “build to print” schedules at the manufacturing, supply chain- material

planning, and change controls levels of the program.

NATIONAL CITY BANK, (Cleveland, OH) 7104 ~ 11/05
Senior Project Manager

Managed the installation of large integrated projects for loan origination in the Lending Portfolio of

National City’s IS organization. Completed 1920 hours in project initiation, planning, execution, control

and closure by applying;

e The application and management of iterative development using Fusion /Agile methodology and
coordinating it with the life cycle of waterfall methodology.

* Managed a 4.5M National Home Equity broker on line origination system development project.

° Coordinated and lead the project through initiation, project approval and the first two releases of five
and then successfully transitioned future releases with the sustaining Project Manager.

° Managed the implementation of a 2.5M Loan Express system for a best in class initiative.

e Managed a, 1.3 M project for the new business venture of on-line loan applications with Lending Tree.

Participated on the board of reviews as a subject matter expert for the new project office rewrites of

processes and new methodologies for project management.

DJO(41/06)


EAC Interim Accreditation CIBER 1118/2007
Revisit (Jul 2006)

Introduction

This accreditation assessment was a follow-up to an earlier assessment of CIBER Inc.'s ITA
Practices office under the Election Assistance Commission (EAC) Interim Accreditation Program
conducted in July 2006. The assessment used the NIST Handbook 150-2006, NVLAP
Procedures (HB 150) and General Requirements and NIST Handbook 150-22- 2005 NVLAP
Voting System Testing (HB 150-22) as the criteria for certification. The interim program is
designed to accredit ITAs formerly authorized under the National Association of State Election
Directors (NASED) accreditation program to continue voting system testing under an EAC
accreditation until such time as the NVLAP/EAC joint accreditation has qualified one or more
testing laboratories as Voting System Test Laboratory (VSTL).

The July assessment found that the CIBER ITA Practice program in non-compliance in four
areas:

a. There was no record of an internal audit or management review performed since the creation
of the policy and procedures for ITA Practice. That was not serious in itself because there may
be delays but queries about records from the last audit to the corporate office with related
responsibilities did not indicate the results were available for corrective action at the ITA Practice
level.

b. The corporate quality manual requires the ITA Practice Director to maintain a log of Corrective
Action Reports (CARs). The CARS are also required at other points to capture and track
complaints, reports of non-compliance, and other problems which should receive management
review and follow-up.

c. QA Representatives were named last year to provide local people responsible for QA
requirements in the absence of the remote QA manager. Their major role was to perform
Process Conformance Audit at the end of each test campaign of the test and test report to ensure
conformance to processes and procedures. No Process Conformance Audit has been done
since the creation of the requirement.

d. During review of requirements for standard testing methods, the ITA Practice Director stated
that they could not use standard methods but either used the vendor's tests or created new tests
for each system's test. The requirement for standard test methods is necessary to ensure that
consistent and conforming testing is applied for each tested system. The standard test method
provides a base for comparison against the vendor's tests and, when system variations require
some adaptation, the changes needed are reported based on the pre-defined established test
method.

In addition to these items, the July assessment went through issues and expected features
needing development based on the application of the new ISO 17025 requirements as presented
in the checklists prepared from the HB 150 and HB 150-22 criteria. CIBER ITA Practices and
Wyle Labs had formed a Team Partnership for performing the testing but the policies and
procedures needed further development to define each labs responsibility with one lab (the
‘VSTL’ in HB 150 and HB 150-22 terms) taking the primary responsibility for managing and
reporting the complete voting system test results. Additional requirements resulting from applying
the ISO 17025 standards for more complete reporting were also reviewed and copies of the
checklists were left.

CIBER was not granted an interim accreditation but was given a 120 day response period to
submit corrections. Based on the documentation presented as a response, this follow-up
assessment was conducted in Huntsville during the period of December 6-8 at the CIBER ITA
Practice office in Huntsville.

Page 2 of 4


LAB: CIBER 6-8 Dec 2006
intended use are fulfilled.

5.4.5.2

co

_a) The laboratory shall validate non-standard methods, laboratory-
designed/developed methods, standard methods used outside their
intended scope, and amplifications and modifications of standard
methods to confirm that the methods are fit for the intended use.
The validation shall be as extensive as is necessary to meet the
needs of the given application or field of application.

Th 22 Vekcitebny , Bae OY. Yb,
The laboratory shall record the results obtained, the procedure used for the
validation, and a statement as to whether the method is fit for
the intended use. woo

dows scl puchik slocerebetin & Ro pettiy ghee

NOTE 1 Validation may include procedures for sampling, handling and transportation:

NOTE 2 The techniques used for the determination of the performance of a method should
be one of, or a combination of, the following:

i) calibration using reference standards or reference materials;

ii) comparison of results achieved with other methods;

iti) interlaboratory comparisons;
iv) systematic assessment of the factors influencing the result;

v) assessment of the uncertainty of the results based on scientific understanding of the
theoretical principles of the method and practical experience.

NOTE 3 When some changes are made in the validated non-standard methods, the
influence of such changes should be documented and, if appropriate, a new validation
should be carried out.

wi §.4,5.3 The range and accuracy of the values obtainable from validated methods
(e.g., the uncertainty of the results, detection limit, selectivity of the
method, linearity, limit of repeatability and/or reproducibility,
robustness against external influences and/or cross-sensitivity
against interference from the matrix of the sample/test object), as
assessed for the intended use, shall be relevant to the customers’ ceagteize 7
needs. AL Comuenk die aha, byg lice de nd popper CFP of,
Ab comb hme, robs; ageio de ad epee i Hs cf,
, NOTE 1 Validation includes specification of the requirements, determination of the
characteristics of the methods, a check that the requirements can be fulfilled by using the
jmethod, and a statement on the validity.

[NOTE 2 As method-development proceeds, regular review should be carried out to verify
;that the needs of the customer are still being fulfilled. Any change in requirements
requiring modifications to the development plan should be approved and authorized.

{NOTE 3 Validation is always a balance between costs, risks and technical possibilities.
There are many cases in which the range and uncertainty of the values (e.g., accuracy,

lactecton limit, selectivity, linearity, repeatability, reproducibility, robustness and cross-

| sensitivity) can only be given in a simplified way due to lack of information.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 20 OF 42



LAB: CIBER 6-8 Dec 2006

a requirement that all persorinel concerned with testing and calibration
activities within the laboratory familiarize themselves with the quality
documentation and implement the policies and procedures in their work; and

¢ e) the laboratory management's commitment to comply wittrthisstandbook and
to continually improve the effectiveness of the management system.
townen\ed
NOTE _ The quality policy statement should be concise and may include the requirement that
tests and/or calibrations shall always be carried out in accordance with stated methods and
customers’ requirements. When the test and/or calibration laboratory is part of a larger
organization, some quality policy elements may be in other documents.

_ ti 4.2.3. Top management shall provide evidence of commitment to the development
and implementation of the management system and to continually improve its
offecveness. : Spates ae Ne doce lath. Be Sey 40 bg rtfenatss
C) Exaati " Morage gayi Dive. Ze avery ESE &
wok 4.2.4 Top management shall communicateto the organi n the importance of
meeting customer requirements as well as statutory and regulatory

requirements. ob hedadicn ow Date Pa beb

giya) The quality manual shall include or make reference to the supporting
procedures including technical procedures. Pow f ! Groen (Te Tab

Malioks | lnc Beye 5
Cue. b) It shall outline the structure of the documentation tisedsathe anagement ay ba,

system. Pov t.2  deumet Cocianeted ——

fhe 4.2.6 The roles and responsibilities of technical management and the quality
tmanager, including their responsibility for ensuring compliance with this
handbook, shall be defined in the quality manual. per. {0 Degen Zerbun
ord 4.2, Rela el Reparsal Lee .
Ce ¢ 4.2.7 Top management shall ensure that the integrity of the management system
is maintained when changes to the management system are planned and
implemented.

fete altace Lip peel

4.3 Document contro}
4.3.1 General

¢\~" The laboratory shall establish and maintain procedures to contro! all documents that
form part of its management system (internally generated or from external sources),
such as regulations, standards, other normative documents, test and/or calibration
methods, as well as drawings, software, specifications, instructions and manuals.
om 4 he arnt Cobrel  frry Bit gx Shintiapa oe
NOTE 1 In this context "docliment” could be policy state! drs, procedures, specifications,
calibration tables, charts, text books, posters, notices, memoranda, software, drawings, plans,
etc. These may be on various media, whether hard copy or electronic, and they may be digital,
analog, photographic or written. Cm Tesh bet je epoyh Four Code exhsel

OE Pega saparte.L- proguizles awk) peanees.
NOTE 2 The control of data related to testing and calibration is covered in 5.4.7. The control
of records is covered in 4.13.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 4 OF 42



LAB: CIBER 6-8 Dec 2006

mien: LEY Ie

pee is, Fis

Note to assessor: Attach a copy of the full int internal | audit schedule. tee

b) Such audits shall be carried out by trained and and qual qualified personnel who aré,
wherever resources permit, independent of the activity to be audited.

Porm VR> Pewter, fe Geran by Terry Debet, ade poor A f

NOTE The cycle for internal auditing should normally be completed i in one year. be

Dates of most recent internal audit:

wae

cV414.2 When audit findings cast doubt on the effectiveness of the operations or on
the correctness or validity of the laboratory's test or calibration results, the
laboratory shall take timely corrective action, and shall notify customers in
writing if investigations show that the laboratory resuilig may have been
affected. Tem 1H. Seypesbion Dee | Conentiny Ack,
gi4,14.3 The area of activity audited, the audit findings and corrective actions that
arise from them shall be recorded. TH Aak ¢ Te Ae & nA Choe. prea of | =
Sig facut tC Biuas fox etn
2414.4 Follow-up audit activities shall verify and rec record thei implementation and
effectiveness of the corrective action taken.

Pam AUB REALS 5

[rake
Carrer fun ae ronda es

doitees u
4.15 Management reviews Gey

wk Shere

eatin) FEA He Byer
4.15.1 In accordance with a predetermined schedule and procedure, the
taboratory’s top management shall periodically conduct a review of the
laboratory's management system and testing and/or calibration activities to
ensure their continuing suitability and effectiveness, and to introduce 2 psi. [B, 4
necessary changes or improvements. Sebepee ap Ee he

Date(s) of most recent management review: fra ra
Aoruvd copy triples orga} A.
What is the review schedule?

Que. dest) Geel

genre! ob hahaa}

“4) wis) gh © MS Gees
__b) teports from managerial and supervisory personnel;
__) the outcome of recent internal audits;
__/4) corrective and preventive actions;
__“e) assessments by external bodies;
__ vf the results of interlaboratory comparisons or proficiency tests;
_ #4 changes in the volume and type of the work;
hy customer feedback;

complaints;

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 13 OF 42



VICTOR P. DAILY ag fies sm
Senior Software Engineer ELIS

Mr. Daily is a Senior Software Engineer with over 12 years experience specializing in certification,
support and deployment of technical software applications. Self-motivated team member possessing
excellent interpersonal and communication skills. Familiar with ISO 9000 and CMM methodology.

EDUCATION, CERTIFICATIONS & TRAINING
B.S. in Industrial Operations, Auburn University, Auburn, Alabama, March 1984

Microsoft Certified Professional - 2000

TECHNICAL EXPERIENCE

Languages:
Software/Tools:
Systems:
Databases:

PROFESSIONAL EXPERIENCE
Intergraph Corporation, Huntsville, Alabama 1994 — Present

Senior Software Engineer, 1997 — Present

Responsible for software certification and validation of Intergraph’s SmartPlant 3D application software.
Develop and execute formal test plans which correspond to detailed software specifications. Document
findings during testing and submit detailed program change requests and trouble reports to software
developers. Retest software before deployment to verify conformance to customer's expectations.

Provide support and direct interface with customers to coordinate on-site deployment of complete
operating system and application software.

Senior Software Engineer, 1994 — 1997

System implementation specialist of the U.S. Army's Digital Storage and Retrieval Data System
(DSREDS) and Joint Engineering Data Management System (JEDMICS). Responsible for configuring
UNIX, Windows NT and IBM PC (DOS / Windows) hardware to various software platforms. Edited vector
and digital data utilizing Intergraph application software conforming to MIL-STD-1840B file standards.
Provided technical support to MICOM user base in addition to eight additional Army installations.

Sverdrup Technology, Huntsville, Alabama 1989 - 1993
Mechanical Designer

Utilized Intergraph's Engineering Modeling Software (I/EMS) to support NASA's Marshall Space Flight
Center Science & Engineering Directorate. Design tasks ranged from intricate man-rated flight hardware
to large structural steel components. Duties included performing conceptual design, three dimensional
modeling, two dimensional detailing, interference checking, extracting mass properties, stress analysis,
trade studies, producibility and procurement for all design tasks. All final drawings conformed to
standards, DOD-D-1000B, DOD-STD-100C, MSFC-STD-555B and ANSI Y14.5M.

JO (7/06)



I EAC Interim Lab: CIBER

4.9 CONTROL OF NONCONFORMING TESTING AND/OR CALIBRATION WORK

Calibration work does not apply at this time

Basic policy and procedures follows recommended practices based on the NIST Handbook
standards (ISO 17025) but there is no evidence that the process work since there have been
no non-conforming reports submitted. Test case on this and other processes were made to
ensure that the form linkage and operation worked and would post a report. In the
assessment it was noted that the only source of non-conforming reports was from internal
audit reports. CIBER is to consider changes to recognize EAC or related stakeholder reports
of non-conformance, possible through official EAC process communication or otherwise
provide a path (see comments on Complaints) for recognizing such inputs from legitimate
stakeholders.

No formal non-comformance issues were identified.

4.10 IMPROVEMENT

An active program and policy statements recognizing and supporting on-going improvement
was noted in the management procedures.

No non-conformance issues were noted.

EAC Interim On Site Lab Report Page 9 of 19



A

LAB: CIBER 6-8 Ded 2006

Annex A (normative)

Referencing NVLAP accreditation

Cohditions for referencing the NVLAP term, logo, and symbol
The term NVLAP and the NVLAP logo are registered marks of the
Federal Government, which retains exclusive rights to control the use thereof.
Permission to \se the term and symbol (NVLAP logo with approved caption) is
granted to NVLAPaccredited laboratories for the limited purpose of announcing
their accredited Status, and for use on reports that describe only testing or
calibration within the scope of accreditation. NVLAP reserves the right fo control
the quality of the use of the NVLAP term, logo, and symbol.
!n order to become ’nd remain accredited, laboratories shall comply with
the following condition’ pertaining to the use of the term NVLAP, the NVLAP logo,
and NVLAP symbol. Failkre to comply with these conditions may result in
suspension or revocation of a laboratory's accreditation.

a) An applicant laboratoty that has not yet achieved accreditation may make
reference to its applicant status. If the NVLAP Lab Code is used, it
shall be accompanied by a statement accurately reflecting the
laboratory’s status. An applicant laboratory shall not use the
NVLAP term, logo or\symbol in a manner that implies accreditation.

b) The laboratory shall have a policy and procedure for controlling the use of
the term NVLAP and the NVLAP symbol.

c) The term and/or symbol shall no\be used in a manner that brings NVLAP
into disrepute or misrepresents a laboratory’s scope of
accreditation or accredited status.

d) When the term NVLAP is used to reference a laboratory’s accredited status,
it shall be accompanied by the NVLAR Lab Code.

e) When the NVLAP symbol used to reférence a laboratory's accredited status,

it shall be comprised of the NVLAP logo and the NVLAP Lab Code in an
approved caption. The caption shall apRear below and in close proximity to
the logo. The following captions have been approved by NVLAP:

“For the scope of accreditation under NVLAP\Lab Code 000000-0”
« “NVLAP Lab Code 000000-0".

See Annex A of NIST Handbook 150 for example§ of the logo with captions.

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 35 OF 42



CO

LAB: CIBER 6-8 Ded 2006

5.4.6 Estimation of uncertainty of measurement

5.4.6.1 Aicalibration laboratory, or a testing laboratory performing its own

alibrations, shall have and shall apply a procedure to
estimate the uncertainty of measurement for all calibrations and types of
callbrations.

uncertainty of measurement. In certain cases the nature of the
test method may preclude rigorous, metrologically and statistically
valid, ca\culation of uncertainty of measurement. In these cases
the laboratory shall at least attempt to identify all the components
of uncertaikty and make a reasonable estimation, and shall ensure
that the form of reporting of the result does not give a wrong
impression of the uncertainty. Reasonable estimation shall be
based on knowledge of the performance of the method and
on the measurement scope and shall make use of, for
example, previous experience and validation data.

5.4.6.2 Testing ratores shall have and shall apply procedures for estimating

NOTE 1 The degree of rigor néeded in an estimation of uncertainty of measurement
depends on factors such as: .
\
i) the requirements of the test method;
\
ii) the requirements of the customer, s,
\

iii) the existence of narrow limits on which decisions on conformity to a specification are
based. \

NOTE 2 in those cases where a well recognized test method specifies limits to the values

of the major sources of uncertainty of measurement and specifies the form of

presentation of calculated results, the laboratory is considered to have satisfied this

clause by following the test method and reporting Inetructions (see 5.10).

5.4.6.3 When estimating the uncertainty of measurement, all uncertainty
components which are of importance In the given situation shall be
taken into account using appropriate methods of analysis.

reference standards and reference materials used, method$,and equipment used,
environmental conditions, properties and condition of the item ‘being tested or calibrated,
and the operator.

NOTE 1 Sources contributing to the uncertainty include, but ee necessarily limited to, the

‘

NOTE 2 The predicted long-term behavior of the tested and/or calibrated item is not
normally taken into account when estimating the measurement uncertainty.

NOTE 3 For further information, see ISO 5725 and the Guide to the Expression of
Uncertainty in Measurement (see 1.4).

NVLAP Note: ANSI/NCSL Z540-2-1997 and NIST Technical Note 1297, 1 aoe Sarton, are
considered to be equivalent to the Guide to the Expression of Uncertainty’ in
Measurement (GUM). \

Poe Tm |L Sly 37

5.4.7 Control of data a Sot Fete
CCH ATES

5.4.7.1 Calculations and data transfers shall be subject to appropriats checked “t

NIST HANDBOOK 150 CHECKLIST (REV. 2006-03-08) PAGE 21 OF 42



5.5.3

F554

5.5.5

Va)
TL»)

NOTE

LAB: CIBER 6-8 Deg 2006

Equipment shall be operated by authorized
personnel. Up-to-date

instructions ery the use and maintenance of equipment
(includi a“any relevant manuals provided by the manufacturer
of the-€quipment) shall be readily available for use by the appropriate

laboratory personnel.

Each item of equipment and its software used for testing and calibration and
significant to the result shall, when practicable, be uniquely identified. .

Tin 3 Peoslucd Sabor Ha C ivente Cay Je, deleweiey
Records shall be maintained of each item of equipment and its software fet “9
significant to the tests and/or calibrations performed. The
records shall include at least the following:

THB NS hoe Tevente, Cae

the identity of thé item of equipment and its softwaré: 5

the manufacturer’s name, type identification, and serial number or other
unique identification;

checks that equipment complies with the specification (see 5.5.2);
the current location, where appropriate;
the manufacturer's instructions, if available, or reference to their location;

dates, results and copies of reports and certificates of all calibrations,
adjustments, acceptance criteria, and the due date of next calibration;

the maintenance plan, where appropriate, and maintenance carried out to
date;

any damage, malfunction, modification or repair to the equipment.
The laboratory shall have procedures for safe handling, transport, storage,
use and planned maintenance of measuring equipment to ensure
proper functioning and in order to prevent contamination or deterioration.

Additional procedures may be necessary when measuring equipment is used

outside the permanent laboratory for tests, calibrations or sampling.

Equipment that has been subjected to overloading or mishandling, gives

suspect results, or has been shown to be defective or outside

specified limits, shall be taken out of service. It shall be isolated

to prevent its use or clearly labeled or marked as being out of

service until it has been repaired and shown by calibration or test to

perform correctly. tthe! heppars GUT & olizcorcen! L hemssug

ae

The laboratory shall examine the effect of the defect or departure from
specified limits on previous tests and/or calibrations and shall
institute the “Control of nonconforming work” procedure (see 4.9).

pa Lord

NIST HANDBOOK 150 CHECKLIST (REV, 2006-03-08) : PAGE 23 OF 42



1 EAC Interim Lab: CIBER

4.15 MANAGEMENT REVIEWS
clean 92 ve mage arent Keene! Penton} Bes. GE

i Fem. No management reviews had been made yet. However, the prior
assessment report and internal audit have started major performance improvement actions
including the management review requested by next week. A copy of the management
review report was made available and | confirmed that the items in 4.15.1 were all included,

(Copied from the HDBK 150 checklist).

This item is pending evidence to show that it is functioning but seems to be more a matter of
timing at this time.

EAC Interim On Site Lab Report Page 12 of 19