REFERENCE

NBS

PUBLICATIONS

NAT'L INST. OF STAND & TECH

AlllOt 0 3 7 4 0 3

IMBSIR 85-3137

NVLAP Assessment and Evaluation

Manual

Peter S. Unger

U S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Office of Product Standards Policy

Gaithersburg, Maryland 20899

April 1985

U S. DEPARTMENT OF COMMERCE

-QC

100

.1156

85-3137

1985

AL BUREAU OF STANDARDS

NBSIR 85-3137

NVLAP ASSESSMENT AND EVALUATION

MANUAL

NATIONAL BUREAU

OF STANDARDS

library

D He l DO

Peter S. Unger

U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Office of Product Standards Policy

Gaithersburg, Maryland 20899

April 1985

U.S. DEPARTMENT OF COMMERCE, Malcolm Baldrige, Secretary

NATIONAL BUREAU OF STANDARDS. Ernest Ambler. Director

NVLAP ASSESSMENT AND EVALUATION MANUAL

FOREWORD

The National Voluntary Laboratory Accreditation Program (NVLAP),

established in 1976, is administered by the NBS Office of Product Standards

Policy. NVLAP is a voluntary system for assessing and evaluating testing

laboratories and accrediting those found competent to perform specific test

methods or types of test methods. Individual laboratory accreditation

programs (LAPs) for specified product or service areas are established in

response to requests and demonstrated need.

This publication is the main body of written guidance provided to NVLAP

technical experts (TEs), who serve as assessors and evaluators. The

general policies and practices of NVLAP assessment and evaluation are described. The specific, technical details for assessing and evaluating

laboratories under each LAP are provided in appendices tailored for each

LAP. The appendices are not part of this publication.

ACKNOWLEDGEMENTS

The author wishes to recognize the significant contributions of several sources. NVLAP staff members, Harvey Berger, Robert Gladhill, Wiley Hall,

Jeffrey Horlick, and John Locke, each made helpful suggestions on the

structure and content of the document. NVLAP assessors, Charles Gilbo and

Sidney Levinson, also provided useful insights. Several concepts developed

in this publication were based on writings of and discussions with staff of

Australia's National Association of Testing Authorities. Finally, Lisa

Drake, NVLAP staff secretary, provided the tedious, but necessary, typing

and production support.

NVLAP ASSESSMENT AND EVALUATION MANUAL

TABLE OF CONTENTS

Page

Title Page i

Foreword ii

Acknowledgements ii

Table of Contents iii

I. INTRODUCTION 1

Purpose 1 NVLAP Approach to Accreditation 1 Role of NVLAP Technical Experts 1 Assessment Assignments 2

Monitoring Visits 3

II. ON-SITE ASSESSMENT 3

Assessment Forms 3

Preparation for an Assessment 4

Entry Briefing 5 Typical Steps of an Assessment 5

Preparing an Assessment Report 6

Exit Briefing 6

Completing an Assessment Report 7

III. ASSESSMENT CONDUCT AND TECHNIQUES 8

Conducting an Assessment 8

Information Collection 10

Questioning Techniques 11

IV. DEFICIENCIES 12

Identifying Deficiencies 12

Typical Deficiencies 14

V. TECHNICAL EVALUATION 15

i i i
'
NVLAP ASSESSMENT AND EVALUATION MANUAL

I. INTRODUCTION

Purpose

This manual is for use by NVLAP technical experts (TEs). It describes the

role of a NVLAP TE in performing the assessment and evaluation functions of

NVLAP.

The information presented here is intended to spell out your

responsibilities as a TE. Since you will be representing NVLAP directly

with applicants, you should understand the entire NVLAP accreditation

process, including NVLAP policies and procedures, sequence of events, as well as the on-site assessment. The NVLAP accreditation process is

described in handbooks specially prepared for each laboratory accreditation

program (LAP). LAP handbooks are part of the application package provided

to potential applicants. In addition to describing the accreditation

process, each LAP handbook identifies the specific criteria and other

technical requirements for accreditation.

NVLAP Approach to Accreditation

NVLAP is a system for accrediting testing laboratories found competent to

perform specific testing services. Before an accreditation decision is

made, a thorough evaluation is performed. The NVLAP evaluation consists of

a review of application information, periodic on-site assessments, results

of required proficiency testing, and correspondence between the applicant

and NVLAP. The goal of an evaluation is to verify that a laboratory has

attained a minimum level of competence as defined by the conditions and

criteria for accreditation. However, we also encourage laboratories to

improve testing performance above the minimum level through the use of

proficiency testing and by using technical experts as assessors and

evaluators.

The NVLAP approach to accreditation is basically non-adversarial . We view

our role as one of assisting applicants to attain and maintain their

accredited status. However, we are firm in requiring compliance with

accreditation requirements and a laboratory's accreditation may be denied,

suspended or revoked if compliance is not achieved.

Role of NVLAP Technical Experts

NVLAP TEs have a dual role in the accreditation process. One role is to

conduct on-site assessments of applicants. The other role is to evaluate
I

information on applicants, including assessment reports, responses to

deficiencies, and proficiency test results, for the purpose of recommending

appropriate accreditation actions.

Your main contribution is your technical expertise. You and the other TEs

assigned to your LAP(s) provide the technical direction and opinions

required to evaluate each applicant's compliance with the accreditation

requirements. NBS/NVLAP staff provide the structure and administrative

procedures; you provide the technical support. NBS technical staff and

others may also be available for consultation.

One important aspect of NVLAP is that each applicant be evaluated, in

relation to others, as uniformly as possible. Assessments and evaluations

must be fair and even-handed. Interpretations of requirements should be

consistent among all TEs within a LAP. If you are in doubt about a requirement, please feel free to discuss it with other TEs as well as the

relevant NVLAP staff.

Assessment Assignments

We assign TEs to perform assessments by matching individual expertise with

a laboratory's requested scope of accreditation, as well as by trying to

avoid any conflict of interest. If you are assigned to assess or evaluate

a laboratory for which possible conf lict-of-interest questions could be

raised, please advise us. Although it is not always possible, assignments

are scheduled so that every participant gets a different assessor on successive visits.

Laboratories have the right to refuse acceptance of an assessor assignment.

Although this rarely happens, if a laboratory does not want you to be the

assessor, notify NVLAP staff immediately. When we have before-the-fact

information about whether a laboratory will accept a certain TE, we contact

the laboratory before making the assignment.

When you are assigned to perform an assessment, we will send you copies of

the laboratory's application form, proficiency test data, and any other

pertinent information. Please keep these documents in confidence. After

completing the assessment, you should return the documents to NVLAP.

Normally, you will have at least 60 days to schedule and complete the

assessment(s) that you have been assigned. Occasionally, a faster response

may be requested.

On your first one or two assessments you will normally be accompanied by a NVLAP staff person. After this initial visit(s) you will generally be on your own. Occasionally, you may be accompanied by a NVLAP staff person or possibly a new TE in training.

2
Monitoring Visits

In addition to regularly scheduled on-site visits, you may be asked to

conduct monitoring visits which may be scheduled for cause or on a random

selection basis during the accreditation period. These visits serve to

verify reported changes in the laboratory facilities and operations or to

explore possible reasons for poor performance in proficiency testing. The

scope of the monitoring visit may range from checking a few designated

items from the checklists to a complete review. An assessment report is

still written and left with the laboratory as is done during a regular

assessment, but not all checklist items need be covered or completed.

When we make assignments for monitoring visits, we will specify what should

be examined, but you are not bound by our instructions if, while you are at

the laboratory, you find other items that need to be reviewed. Monitoring

visits generally last no longer than one day.

II. ON-SITE ASSESSMENT

The on-site assessment, a key part of the accreditation process, is our face-to-face examination of the laboratory. Before initial accreditation,

and periodically thereafter, on-site assessments of each NVLAP participant

are conducted to assess compliance with accreditation requirements.

Assessments may last from one to several days and may involve more than one assessor depending on the scope of accreditation desired by the laboratory.

Specific aspects of a NVLAP on-site assessment are described under separate

headings below.

Assessment Forms

NVLAP uses on-site assessment forms (checklists and an assessment report

form) to:

o ensure uniformity of assessments among assessors;

o guide each assessment;

o provide a place for recording notes during an assessment;

o help assessors prepare exit briefings and assessment reports;

o provide evidence that an assessment was performed and that all

applicable aspects of the laboratory were examined; and

o provide information for future assessments, thus facilitating

continuity.

3
The structure and content of assessment forms can vary depending on the

LAP. Each assessment form contains checklists and a report form. Specific

instructions are provided with the on-site assessment form used in your

LAP(s)

.

Some general tips in completing the checklists:

o Make sure that each item on the checklists either is answered or that

there is a definite indication that no answer was feasible or applicable.

o Keep as many notes as possible on the checklists; where there is

insufficient space, use additional sheets of paper;.

o Although many of the items on the checklists are arranged in a suggested sequence, you may alter the sequence as you see fit.

Preparation for an Assessment

The overriding purpose of preparing for an assessment is to organize

yourself to use the time available during the assessment to the best

possible advantage. Failure to adequately prepare will slow down the

assessment, create an unfavorable impression of NVLAP and yourself, waste

the laboratory's time, and render the results less valuable. The following

steps are recommended when preparing for an assessment:

1. Notify the laboratories at least one month in advance whenever

possible.

2. Discuss with each laboratory, by telephone, the laboratory location,

directions, dates, time of arrival, security arrangements, names of

laboratory staff you will meet, suitable lodging arrangements,

proposed agenda, requested demonstrations of specific tests, etc. Ask

the laboratory to send you appropriate information to review before

the visit (e.g., QA manual and procedures). Assure the laboratory

that any documents they are willing to send you in advance will be

held in confidence, will be returned upon completion of the

assessment, and will not be retained in NVLAP files nor your own personal files.

3. Review all relevant information (e.g., criteria, test methods, QA

manual, application information, previous assessment reports,

deficiency correction letters, proficiency test results). Prepare

your own questions based on previous assessment reports and

correspondence to address weaknesses previously observed. Review the

laboratory's proficiency test results and look for trends that might

indicate an apparatus or procedural weakness. This information may

indicate what tests need to be demonstrated.

4. Develop an agenda. If time allows, send a proposed agenda to the

laboratory beforehand and state who at the laboratory you would like

to have available, particularly to attend the exit briefing.

4
5. Complete what you can on the assessment forms before your

assessment.

Entry Briefing

Upon arrival at a laboratory you are going to assess, conduct an entry

briefing during which you should:

o meet with the laboratory's authorized representative and all others

who will be involved in assisting the assessment;

o explain the purpose of the assessment and set the agenda;

o ask for one escort who is knowledgeable about the laboratory's

management, quality assurance, and procedural systems to ensure

cooperation of laboratory personnel;

o confirm that any laboratory documents that you may have previously

reviewed are still current;

o request a meeting room or place where you can be by yourself to

review laboratory documentation and records, to complete the

checklists, and to compose your findings;

o discuss the schedule for the day, conforming as far as possible to

the laboratory's working hours, lunch hour, and coffee breaks;

o agree on a tentative time for holding the exit briefing;

o write down all names and positions of relevant persons for future

reference;

o note who speaks for the laboratory and who is knowledgeable so that

you may refer back to the appropriate "information resource" later in

the assessment; and

o walk through the portion of the site to be assessed to see the layout

and to meet the relevant laboratory personnel.

Typical Steps of an Assessment

After the entry briefing, an assessment may be performed in any order. To some extent, the checklist questions follow a logical sequence and dictate

the order to pursue for some aspects of the assessment, but you need not

follow that order. Some assessors plan a sequence of activities with which

they are more comfortable. However, keep in mind that the goal of an assessment is to verify that the laboratory complies with all applicable

NVLAP accreditation requirements. A summary of a typical sequence of an assessment is as follows:

5
1. Conduct an entry briefing.

2. Examine QA manuals, personnel competency records, equipment

maintenance and calibration records, sample handling, and

recordkeeping procedures.

3. Examine test plans, reports, data logs, and related records.

4. Trace one or more samples from receipt to final test report.

5. Witness demonstrations of selected procedures by laboratory staff.

6. Interview technicians.

7. Physically examine equipment and facilities.

8. Formulate impressions about the overall management and organization.

9. Conduct an exit bri ef ing--discuss findings, particularly

deficiencies.

10.

Complete written assessment report, have laboratory sign it, and

leave a copy.

Preparing an Assessment Report

After you have completed the checklists, find a place where you can review

your notes on the checklists and collect your thoughts. Summarize your

findings on the assessment report form. Write anything that is significant

about what you have observed, whether complimentary or negative. However,

before an exit briefing, be sure to:

o respond to all applicable checklist items;

o decide which findings must be presented as deficiencies requiring

corrective action by the laboratory; and

o describe each deficiency on the appopriate assessment report sheet.

Exit Briefing

At a minimum, the laboratory's QA manager or equivalent and his superior

should attend the exit briefing. It is useful for future reference to

identify the attendees at the exit briefing on the first page of an assessment report. An exit briefing will usually not last more than one

hour.

For each deficiency, describe the finding, the justification or reason for

stating the finding and, if appropriate, the proposed corrective action.

Make sure that the laboratory understands each deficiency and what is

required to achieve compliance.

6
Some of the deficiencies you identify may be challenged. Remember that the

laboratory's technical staff may also be experts and differences of opinion

may arise. Try to work out any differences and be satisfied that the NVLAP

criteria have been met. However, don't argue. As appropriate, record your

finding as a deficiency and ask the laboratory to refute it in their

response to NBS. The NVLAP accreditation process provides a second level

evaluation which includes a review by other technical experts and NVLAP

staff of your assessment report, the checklists, results of proficiency

testing, and correspondence from the laboratory responding to the cited

deficiencies.

Do not be overly concerned about differences that may arise; they can be

worked out later. Our experience to date indicates that participants look

to NVLAP assessors for guidance and do not consider them as adversaries.

Advise the laboratory to write NBS within 30 days describing action taken

or planned to be taken for each specific deficiency identified in the

assessment report. The letter must be signed by an authorized official of

the laboratory who can certify that the statements made in the letter are correct.

If any deficiencies are noted at laboratories which are currently

accredited, such deficiencies must be corrected within 30 days or they may

face suspension or revocation of accreditation. When out-of-calibration

apparatus is cited, explain that the apparatus should not be used until

corrective action has been completed.

During the exit briefing, watch for warning signs of difficulties in

obtaining compliance. For certain difficult assessments, a private

conference with top management may be necessary or a separate follow-up

note to NBS should be considered if there were major problems at the

laboratory that you could not be openly or comfortably discussed with the

laboratory management.

Completing an Assessment Report

Make any necessary revisions or additions to the assessment report in light

of the exit briefing. At least one laboratory representati ve, preferably

the authorized representative, must sign the assessment report indicating

that it has been discussed during the exit briefing. Leave a copy of the

report with the person who signs it or allow the laboratory to photocopy

the report. Do not leave a copy of the checklists (because of the

possibility that your "rough notes" will contradict your official

assessment report to the laboratory).

After you have left the laboratory, mail the original copy of the

checklists along with the signed original of the report to NBS and retain a copy of all sheets in case the originals are lost in the mail or we need

you to refer back to them during the technical evaluation. After an accreditation decision is made by NBS, you should return all documents

pertaining to the assessment to NBS. Mail all documents to:

7
NVLAP

National Bureau of Standards

ADMIN A531

Gai thersburg, MD 20899

III. ASSESSMENT CONDUCT AND TECHNIQUES

More than your technical expertise will be required to conduct an assessment successfully. There are certain procedural items, ethical

issues, assessment concepts, and questioning techniques to bear in mind.

These are covered in this section.

Conducting an Assessment

To conduct an assessment properly, the following guidelines should be

followed:

o Make every effort to disrupt as little as possible the normal

operating schedule of the laboratory while conducting an assessment.

o Be determined, decisive and direct. (Once there is enough evidence to

form the basis for a sound judgement, there is no point in going over

the same ground.)

o Keep the assessment moving; be aware of the overall progress of the

assessment so as to avoid wasting time on trivia.

o Be honest and fair. (Personal dislikes/prejudices must not interfere

with the assessment.)

o Be independent. (You decide what will be examined)

o Recognize the difference between a "clean-up" job and a "cover-up"

job. There is normally no need to worry about clean-ups that a laboratory carries out before you arrive. Clean-ups rarely solve

major problems, but show that the laboratory is concerned about the

outcome of the assessment. Dispense with cover-ups and get to the

heart of each problem.

o Discuss any problems right away. (This deters arguments later on when

people's memories have grown hazy, and allows you to clarify the

problem and collect new information relating to it. If the

discussion reaches an impasse, do not get involved in an argument; go

on to the next item.)

o Be prepared to return to an area, if necessary, to obtain new information and to reassess the operation in the light of new information.

8
o Keep a sense of proportion. (The magnitude and significance of a deficiency is a vital issue.) Make allowances. (Put errors in

perspective.) Don't pursue unimportant errors, which wastes time and

effort and risks alienating the laboratory.

o Be constructive at all times; when a deficiency is found, suggest

possible corrective action if you are certain that such action would

satisfy the requirement. If wrong in your assumptions or conclusions, it is best to admit it and apologize. Such an attitude

elicits cooperation and respect from the laboratory.

o Be helpful; Suggest any procedures which would be beneficial if the

opportunity presents itself. Your helpful suggestions need not be

NVLAP-oriented. Do what is reasonable to establish an attitude that

leads the laboratory to eagerly welcome the next assessor. Be

careful, however, that you do not give away some other laboratory's

"trade secret" in an attempt to be helpful. Open literature sources

are preferred.

o At the end of each day, give the laboratory representatives a brief

summary of what has been happening. The summary is a courtesy

measure. It limits the need to make embarrassing revelations at the

exit briefing and in the assessment report by forewarning the people

of the nature of any deficiencies encountered. In some situations

the deficiency you observe early in your assessment can be corrected

before the exit briefing. Most laboratories are thankful to be given

the chance to put their house in order discreetly.

o Try to answer all questions posed; if you lack information, offer to

provide an answer at a later date and be sure to follow up, or refer

questions to the NVLAP staff.

o Thank the laboratory for its assistance and hospitality. Even if

there were some contentious issues and you had to differ, you should

still indicate that you appreciated the laboratory's cooperation.

o Treat privileged information as such.

o Avoid all situations which could be construed as undue or improper

influences on your assessment findings.

o Do not request favors from laboratories which could be construed as improper or an imposition.

o Plan to pay for all meals. You will be reimbursed by NBS. Be

prepared to eat lunch by yourself at all times, but if the laboratory

staff wishes to join you, that is acceptable. In some cases the only

reasonable facility is a laboratory-supplied eating facility. Lunch

supplied by the laboratory in these cases is acceptable. However,

always offer to pay.

o Avoid making derogatory remarks about specific manufacturers or suppliers of equipment or about their products.

9
o Avoid making derogatory remarks about individuals either within or outside of the laboratory.

o Avoid becoming involved in intra-laboratory personnel problems.

Information Collection

Assessment is an information collection process that entails verifying

application information, questioning staff, examining facilities,

equipment, and records, and reviewing applicable proficiency testing

results. Collect information as you go by observing and listening

(laboratory staff should be doing the majority of the talking) to what is

going on around you. To assist in completing the checklists, it may be

helpful to note the following:

o key persons interviewed;

o applicable document designations, revision dates, where found, and

descriptions;

o equipment numbers and identification to cross reference with calibration and maintenance records;

o sample/specimen identification system.

o identification of document/information/equipment recipients in

laboratory;

o flow charts showing how the laboratory functions in terms of input

(sample receipt), prerequisites (people, equipment, etc.), processing

(testing procedures), and output (test report). (This systems

approach to viewing a laboratory's operation may be useful in

identifying weaknesses and underlying causes of deficiencies); and

o latest revision dates in the QA manual, procedures, and

instructions

.

It may also be useful as you go along to take mental notes concerning the

following questions:

o Do the laboratory staff members know their jobs?

o Do the lab managers/supervisors want to answer all questions? Some

want to cover all questions either in the fear that their

subordinates will commit an error that will reflect adversely on them, or because they feel it is their right and responsibility to

speak up since they know what is going on. In this situation, you

must make it clear that the "worker" should be the one to answer the

questions, explaining that this is the purpose of the assessment to see what is actually being done and to verify that the proper

procedures are being implemented.

10
o Does the staff try to bluff its way out of a tight corner?

o Does the staff know the QA manual and associated procedures and where

these documents are?

o What is the condition of facilities and equipment? Are the requisite

calibration tags or stickers on the equipment?

o Does it appear that the QA and procedural documents are used?

Every assessment involves collecting information on a laboratory's quality

assurance system. The objective of this "information collection" is to

ascertain whether the laboratory can recall the necessary facts to

substantiate its test reports. Three approaches for assessing a laboratory's quality assurance system are to:

1. trace a test through the testing process forward from receipt of

sample to final test report;

2. trace a test back from final test report to receipt of sample; or

3. randomly select certain files, data sheets, reports, etc.

The first two approaches are generally more efficient and effective. Some

questions on the checklists are sequenced to use these two approaches.

Random selection may be too haphazard and not give you a complete

understanding of the system. We recommend that either of the first two

approaches form the baseline investigation. The third approach is used

when many additional test methods must be covered. Keeping careful notes

is essential to help you prepare a complete assessment report.

Questioning Techniques

Effective questioning is vital to an assessment's success. It is of great

value in assessing to keep the systems concept in mind, to separate the

laboratory's work into input, prerequisites, process, and output, and to

ask questions on each of these individually and in turn. You should voice

your understanding of how the system works so that the laboratory can correct any misconceptions.

Ask questions that require a substantive response ( e . g . , "Who, What, How,

When, Where, and Why" questions force a substantive response) rather than a "Yes/No" answer. Questions that require a "Yes/No" answer are useful when

you need to clarify particular points. Statements or questions such as: Show me what you do./ Where do you put that?/ How do you do that?/ are recommended. If there is a variation from written procedures, then ask

"Why do you do it that way?"

Hypothetical questions are good when little objective evidence is

available. The "What if . . ."or "Let us suppose . . ." or "I don't

understand . . ." type of question is usually effective in such situations.

Avoid asking questions that suggest the answers you expect.

11
Silence can be extremely powerful. Some people find silence uncomfortable

and to break it they may volunteer something they "shouldn't." When you

look at people and say nothing, they may feel a response is expected, or because they are not quite sure what the response should be, they may say more than they otherwise would. This often produces information of use.

Observe what is not said by the laboratory staff. If answers are superficial or evasive, be prepared to continue the line of questioning.

Be alert to differences in information presented to you from different

sources ( i - e . , conflicting answers from staff members or differences

between documented procedures and what is actually observed or said).

Attempt to put the staff at ease at the beginning of a questioning session

since they will probably be nervous and, if so, react by becoming withdrawn

or defensive, or worse--by responding aggressively. You should not be

aggressive in return, but you must be thorough (and occasionally

persistent) enough to establish the factual situation. Keep the following

points in mind:

o Direct questions to the person who performs the task being assessed,

and not to that person's superior.

o Never talk down to anyone.

o Talk the "lab's language".

o Give credit where credit is due. A compliment, sincerely given, goes

a long way towards eliciting cooperation.

o Be interested in the laboratory's work and responses.

o Don't appear to be distrustful of people or to regard their responses

with criticism.

o Be calm and courteous and thank people for their time.

IV. DEFICIENCIES

Identifying Deficiencies

A deficiency is a departure from or an instance of noncompliance with a condition or criterion for accreditation. Keep in perspective the

magnitude of any deficiency found and avoid making an issue out of trivial

deficiencies, human errors, or isolated mistakes. Isolated errors may be

due to someone having an off-day. Examine records for clarity,

completeness, and consistency, and be alert to anomalies which may be a symptom of much deeper problems. If you find no deficiencies after tracing

the testing process by a random selection of tests, then you can probably

12
conclude that there is a system present, working and known to the personnel

concerned. If only one or a few minor deficiencies are found, then the

system, rather than not being implemented, may be in a period of

development or may need to be better instilled into the staff. If several

deficiencies are found, then:

o the system may need to be revised or changed;

o the system may need to be more fully developed and documented;

o the staff members may need to be more adequately trained to perform

their functions as part of the system; or

o a combination of some or all of these corrective actions need to take

place.

Make sure that observed deficiencies are discussed. Sometimes further

information can be provided and change your first impressions. When a deficiency is found, ask questions to discover its cause. After

determining the underlying cause, you should try to suggest corrective

action that would satisfy the requirements. Always remember that cheap,

quick, simple, but effective corrective action is more likely to be acted

upon promptly by the laboratory, particularly if it sounds easy to

impl ement

.

It is helpful to note whether a laboratory is already aware of a deficiency. If so, determine what action has been taken to analyze the cause of the deficiency and to implement corrective action. If the

laboratory was unaware of the deficiency, encourage the laboratory to make

a proposal for corrective action.

Obviously, some deficiencies are more critical than others. It is

virtually impossible, however, to provide decision rules for objectively

ranking deficiencies. This is where your expert technical judgment rules.

The causes of deficiencies can be categorized as follows:

o Lack of system-measures have not been taken to ensure that an activity is performed or performed properly.

o Lack of indoctrination/training of the personnel --system has been

developed and documented, but staff is unaware or doesn't know how to use it.

o Lack of discipline on the part of the personnel involved in

implementation of the system as documented--the personnel have been

trained in how to operate the system but don't comply.

o Lack of time--too much pressure of work, overwork, or inadequate

manpower.

o Lack of resources—incorrect equipment,

o Lack of top management support.

13
Typical Deficiencies

Deficiencies are usually found in a laboratory's documentation, equipment,

or personnel. Typical documentation deficiencies include:

o Required items in the criteria are not addressed.

o Contents of instructions are not adequate.

o Procedures/test plans do not have sufficient detail.

o Outdated copies of documents are still in use.

o The system, as described in the QA manual or instructions, has been

ignored.

o Samples/specimens bear no identification to permit traceability.

o Responsibility for keeping documents updated has not been assigned.

o No planned maintenance records of equipment are available.

Typical equipment deficiencies include:

o Equipment is out-of-tolerance and/or has not been calibrated or verified at requisite intervals.

o Cleanliness is inadequate.

o No provision is made for protected storage in the laboratory.

o Substitute equipment is used without acceptable evidence that it

performs as specified in the standard involved.

Typical personnel deficiencies include:

o Supervision is inadequate.

o On-the-job practice is careless.

o Personnel are not adequately trained to perform their tasks.

o Objective evidence is not available in laboratory files that attests

to the training/certif ication of competence of the personnel.

14
V. TECHNICAL EVALUATION

Technical evaluations (second level reviews) of all applicants are normally

conducted at NBS. The usual procedure for new applicants is to have one or more TEs and NVLAP staff, who were not involved in the assessment, review

the NVLAP evaluation folder for each laboratory. The evaluators provide

technical judgments on whether an applicant complies with the conditions

and criteria for accreditation and make recomnendations on appropriate

accreditation actions to take accordingly. In some cases a special form is

used to record TE evaluation recommendations. A TE recommendation is based

on:

o information provided on the application;

o assessment reports;

o actions taken by the laboratory to correct deficiencies observed

during the on-site visit;

o results of proficiency testing from current and previous assessment

periods; and

o information from any monitoring visits that may have been done.

When a laboratory is not considered to be in compliance with all

requirements, we notify it and request additional information to allow the

laboratory to demonstrate compliance. Occasionally, an additional

assessment may be necessary.

The NVLAP staff prepares the paperwork including the transmittal of

additional correspondence to any laboratory which the evaluators deem needs

to respond more fully to unresolved deficiencies before accreditation can

be recommended.

When notification describing specific deficiencies is made, the laboratory

must respond within 30 days of notification and provide documentation or certification by an authorized member of management that the specified

deficiencies have been corrected. Clarification of some issues may be

requested by telephone. All deficiencies must be corrected before

accreditation is granted.

When a laboratory satisfies all requirements, the Director of the NBS

Office of Product Standards Policy signs and transmits a certificate and

scope of accreditation to the laboratory.

15
NBS-114A (REV. 2-8 U.S. DEPT. OF COMM.

BIBLIOGRAPHIC DATA

SHEET (See instructions

)

. PUBLICATION OR

REPORT NO.

NBSIR 85-3137

2. Performing Organ. Report No, 3. Publication Date April 1985

4.

TITLE AND SUBTITLE

NVLAP Assessment and Evaluation Manual

5.

AUTHOR(S)

Peter S. Unger

6.

PERFORMING ORGANIZATION (If joint or other than NBS. see instructions)

national bureau of standards

DEPARTMENT OF COMMERCE

Gaithersburg, MD 20899

7. Contract/Grant No.

8. Type of Report & Period Covered

9.

SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street. City. State. ZIP)

10.

SUPPLEMENTARY NOTES

J Document describes a computer program; SF-185, FIPS Software Summary, is attached.

11.

ABSTRACT (A 200-word or less factual summary of most significant information. If document includes a significant

bi bl iography or literature survey, mention it here)

This manual explains the role of an assessor and evaluator under the National

Voluntary Laboratory Accreditation Program (NVLAP). Policies, procedures,

and techniques for conducting a NVLAP on-site assessment of a testing laboratory

are described. Deficiencies (or departures from the accreditation criteria)

and the technical evaluation leading to accreditation recommendations are also

discussed. 12.

KEY WORDS (Six to twelve entries; alphabetical order; capitalize only proper names; and separate key words by semicolon s) accreditation; assessment; evaluation; laboratory; testing

13. AVAILABILITY

[j£\ Unlimited

Q] For Official Distribution. Do Not Release to NTIS

[~] Order From Superintendent of Documents, U.S. Government Printing Office, Washington, D.C.

20402.

Order From National Technical Information Service (NTIS), Springfield, VA. 22161 '

14. NO. OF

PRINTED PAGES

20

15. Price

$7.00

USCOMM-DC 6043-P80