O*"

*

Of

Research Information Center

National Bureau of Standards

Gaithersburg, Maryland 20899

NBSIR 87-3651 NOV 2 1987

NVLAP Assessor Selection and

Training

Harvey W. Berger

U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Associate Director of Industry

and Standards

Gaithersburg, MD 20899

September 1987

0*

U.S, DEPARTMENT OF COMMERCE

NATIONAL BUREAU OF STANDARDS

NBSIR 87-3651

NVLAP ASSESSOR SELECTION AND

TRAINING

Harvey W. Berger

U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Associate Director of Industry

and Standards

Gaithersburg, MD 20899

September 1987

U.S. DEPARTMENT OF COMMERCE, Clarence J. Brown, Acting Secretary

NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Director

ATIONAL VOLUNTARY LABORATORY ACCREDITATION PROGRAM (NVLAP)

k SSESSOR selection and training

[arvey W. Berger, Manager

fational Voluntary Laboratory Accreditation Program (NVLAP)

:ntroduction

?he National Voluntary Laboratory Accreditation Program (NVLAP) is

idministered by the National Bureau of Standards (NBS) for the purpose of

recognizing competent testing laboratories. As an agency of the US Department

)f Commerce, NBS is concerned with international acceptance of test data,

Increased trade, and elimination of technical barriers to trade. NVLAP

iddresses these objectives and also contributes to improved quality of testing

md improved test methods.

rtie opportunity for accreditation is offered for specific test methods or

:ypes of tests in specified fields of testing to all domestic and foreign

Laboratories. NVLAP does not offer "blanket accreditation" by field of

:esting. Test methods developed by US and foreign standards organizations or government agencies can be offered for accreditation by NVLAP. Private

company test methods can also be offered for accreditation if those methods

are accessible to all applicant laboratories. A condition for inclusion of

any test method in a NVLAP program is that it be of sufficient technical

quality to permit the differentiation between laboratories that can or cannot

perform it in a competent manner.

DEMONSTRATION OF LABORATORY COMPETENCE

Laboratories demonstrate their competence through compliance with NVLAP

accreditation criteria, which address the laboratory's quality systems, staff,

facilities, equipment, calibration schedules, implementation of test methods,

documentation, operational procedures, records management, and test reporting.

The general criteria for accreditation are published as part of the NVLAP

Procedures in the US Code of Federal Regulations. Criteria which address test

methods in a particular testing technology (or field of testing) are published

in NVLAP Technical Handbooks.

The determination of a laboratory's competence is based on two, fundamental,

NVLAP-directed activities: proficiency testing and on-site assessments by

technical experts.

PROFICIENCY TESTING

Successful proficiency testing is a condition of continuing accreditation in

all programs. The requirements of the programs differ, depending on opportunities, costs, difficulty in performing specific tests, availability of

test artifacts, cooperation of other organizations (such as standards

committees), and other external factors. Proficiency testing is required for

selected test methods if such testing is deemed necessary to determine or reaffirm laboratory competence to perform that method. Laboratories seeking

accreditation for those selected methods must participate in the associated
proficiency testing programs. The results of an individual laboratory’s

proficiency testing provide an indication of the laboratory's overall ability

to carry out a specific test method. If a laboratory's performance is an outlier compared with the group mean, or if it fails to meet a pass/fail

criterion, the precise nature of the testing problem cannot be pinpointed.

However, unsatisfactory performance clearly indicates that the laboratory is

not competent to perform the test method.

In the NVLAP scheme, proficiency testing is viewed as a necessary complement

to on-site assessments. Proficiency testing is a means for comparing each

participating laboratory to the group performance and assuring that all

laboratories do comparable work. It is also a means for maintaining

surveillance of the laboratory's competence at scheduled intervals between

assessments. The on-site assessment is a means for determining if laboratory

practices, personnel, and physical plant comply with the criteria by which

NVLAP defines a quality laboratory. Unlike proficiency testing, the

assessment process does not directly address the question of the laboratory's

ability to "get the right answer."

ON-SITE ASSESSMENTS

On-site assessments of laboratories are performed by technical experts in the

fields of testing for which accreditation is sought. These "peer assessors"

are not NBS employees: they are consultants, retired persons, currently

employed individuals, academicians, and employees of other government

organizations, contracted by NBS for services on an as-needed basis. On-site

assessments are the basis for determining the capability of a laboratory to

perform specific tests and determining the overall quality of work being done

by the laboratory.

NVLAP assessors have both formal and informal objectives in carrying out their

on-site tasks. Using checklists, they review and critique laboratory

performance in each area specified by the NVLAP criteria. They identify,

discuss, and document the laboratory's "deficiencies" relevant to specific

criteria. Deficiencies are failures to comply with specific conditions or criteria for accreditation and must be satisfactorily resolved by the

laboratory before initial accreditation, or within 30 days of the assessment

if the laboratory is already accredited. When opportunities present

themselves and if laboratory personnel are receptive, assessors offer

suggestions for improving testing quality and the efficiency and effectiveness

of laboratory operations, and promote laboratory adoption of the principles of

quality assurance inherent in the NVLAP system.

Assessors are vital to NVLAP because their findings play such an important

part in the final accreditation decision. In their role as assessors, they do

not make formal recommendations to NBS regarding accreditation actions.

However, their evaluations of laboratory performance, in terms of the NVLAP

criteria, are a major element of the decision to accredit or to deny

accreditation. In effect, the assessors are the interpreters of the technical

criteria for accreditation and are, therefore, largely responsible for

maintaining the quality and reputation of the program. The assessors are indeed representatives of NVLAP and the National Bureau of Standards, and must

understand the program and its place within NBS so as not to misrepresent NBS

or the implications of NVLAP accreditation.
;dentification, evaluation, and selection of assessors

tdentif ication of potential assessors is accomplished in many ways. The

jrocess of development of a new accreditation program offers opportunities to

inform the technical community of NVLAP's needs for technical experts. NVLAP

mblishes announcements of new and expanded programs in trade and technical

journals and other media. Public workshops are advertised and held to

establish technical requirements for accreditation. NVLAP staff develop

ixtensive contacts with technical committees of standards organizations, trade

ind professional organizations, other government agencies, and are in contact

#ith NBS technical staff who are aware of other experts in their field.

Potential assessors are identified through many informal interactions and at

Industry and professional organization meetings. Assessors assigned to

established programs are asked to recommend colleagues and co-workers who may

3e qualified.

Potential assessor candidates are contacted by telephone and letter and, if

they indicate an interest, are sent an information package describing the

program and the role of assessors in it. They are asked to submit a resume

describing their technical qualifications and experience relevant to NVLAP,

and are also asked to identify the test methods (in the applicable field of

testing) for which they feel competent to perform assessments and evaluations.

Technical and non-technical criteria are used for evaluating and selecting

assessors. Minimum technical qualifications may include the number of years

of experience, training, professional degrees, and relevant . certifications

.

Prospective NVLAP assessors must have extensive technical experience in those

fields for which they will evaluate laboratories. In addition, potential

assessors should have experience in assessing and evaluating laboratories and

direct experience in laboratory management and operations. They should have a sound understanding of laboratory practices for assuring quality testing. The

nature and breadth of their activities and participation in technical

societies and standards committees should be such that they are widely

recognized as "peers" in their fields.

Two non-technical criteria bear heavily on the selection of assessors. First,

they must have no past or current affiliations that would be perceived as a conflict of interest either by NVLAP or the laboratories seeking

accreditation. Second, the candidate assessors must be able to interact with

laboratory personnel in a firm, constructive, and professional yet nonadversarial manner.

The assessor selection process depends on the particular nature of the NVLAP

program in which they will participate and the number of assessors sought. A

formal selection process involves establishing a selection panel, comprised of

NVLAP staff, other technical experts from NBS, and representatives of public

and private organizations in the relevant technical community. The panel

develops and assigns weighting factors to selection criteria, reviews each

resume, and evaluates and ranks each candidate. This type of formal process

was used effectively to select 14 assessors for the Radiation Dosimetry

program.

For a program initially requiring few assessors, the selection process is

usually informal. NVLAP staff review resumes and select assessors on the
basis of technical training and experience and the recommendations of other

technical experts in the field.

Final selection of an assessor is conditional on the impressions given and the

evaluations made by NVLAP staff throughout the training process. An

individual may have outstanding technical credentials and experience, but not

the personal qualifications or appropriate temperament for performing on-site

assessments

.

ASSESSOR TRAINING

Training of assessors is an on-going, developmental process. Clerical and

administrative tasks are fairly straightforward, but the technical aspects of

the assessment process are evolutionary as a particular program develops from

its early stages to a mature program with an extensive menu of test methods,

well-established standards of performance, and proficiency testing programs.

The assessor training program can, nonetheless, be divided into three distinct

stages: orientation at MBS , the first on-site assessment, and follow-up.

ORIENTATION AT NBS

After a new program has been established, the selected assessors are invited

to attend an orientation and training session conducted at NBS. The sessions

generally last two days and the participants are paid for their travel and

time. Orientation consists of an overall description of NVLAP goals,

objectives, philosophy, and methodology. Training pertains to the description

of assessment techniques and procedures and use of NVLAP documents to prepare

for, implement, and report on assessments.

The sessions are structured and follow a formal agenda conducted as a seminar

by the NVLAP staff. Discussions are interactive and all participants are encouraged to contribute by describing their experiences in visiting and

evaluating laboratories and by sharing information on test methods and other

technical aspects of the program. Although the primary intent of the

orientation is to convey assessment-related information to the technical

experts, NVLAP takes advantage of their technical expertise to benefit the

program and upgrade the technical criteria for accreditation.

It is important that all assessors have a common understanding of what

constitutes a competent laboratory in order that all laboratories receive

equivalent evaluations. The orientation session is the first opportunity to

accomplish a "calibration” of assessors. This session is also the first step

in the process of evaluating the assessors themselves. Note is taken of their

preparation for the session, their apparent interest in the proceedings, their

apparent understanding of what is expected of them, their communication and

listening skills, personality, self-confidence, cooperativeness,

professionalism, and general demeanor.

Orientation emphasizes the following areas:

NVLAP Procedures, Policies, and Philosophy —

The NVLAP Procedures prescribe the criteria upon which NVLAP accreditation

decisions are based. The Procedures are reviewed in detail to ensure that the assessors thoroughly understand the conditions and implications of NVLAP
accreditation. The differences between accreditation and product

certification, acceptable references to NVLAP in laboratory advertising, use

of the NVLAP logo, ethical business practices expected of the laboratory, and

acceptable relationships between a manufacturer's laboratory and its parent

company, are discussed. One of NBS ' objectives, to be implemented through

NVLAP, is the improvement of the quality of testing by laboratories doing

commercial work. While the on-site assessment is not meant to be a consulting

session for the laboratory, the assessors are encouraged to help and inform

the laboratories, in an appropriate manner, in their areas of expertise.

Criteria for Accreditation —

The test methods and NVLAP technical documents, including critical elements,

checklists, and program Handbook, are reviewed and discussed to promote a common knowledge of the essential aspects of the methods that are focal points

during an assessment. The experts are asked to share their experiences in

performing and observing the tests and to offer suggestions on key elements of

technician performance, equipment, test conditions, and facilities. Review of

the test methods allows the participants, rather than NVLAP staff, to lead the

discussion and affords the staff an opportunity to observe them in their

capacity as technical experts. Discussion of proficiency testing requirements

provides insight into the experimental aspect of NVLAP and develops the

concept that NVLAP is free to go beyond the limits of the test methods

themselves to ascertain laboratory competence.

Assessor Checklists and Reports —

The assessment report is thoroughly reviewed to assure that the assessors

understand the level of detail to which they must evaluate the laboratory and

report on their findings. This is their first opportunity to see the report

form and the extent of the evaluation required and the seriousness of the task

are emphasized. They are shown how the report is keyed to the NVLAP

Procedures and general criteria, and NVLAP staff emphasize the need for

preparation before the on-site assessment, as well as thoroughness during the

assessment, in order to carry out a high quality evaluation.

Assessment Techniques —

Techniques for interviewing laboratory personnel and making observations of

laboratory procedures, equipment and procedures, are described from the

perspective of many years of NVLAP experience. NVLAP staff recount prior

experiences and encounters to describe how to deal with various personal

interactions that may occur during an assessment. The experts are encouraged

to share their own prior experiences in evaluating testing laboratories.

Advice and guidance are given by NVLAP staff on the key elements of the

assessment, including:

(1) the first meeting with laboratory management and key personnel,

and explanation of the assessment procedures and objectives;

(2) the walk-through of the laboratory and introduction to other

laboratory personnel;

(3) discussion of findings with laboratory personnel throughout the

assessment

;

(4) the exit debriefing, during which the assessor's findings are discussed with laboratory management to ensure that they understand

what must be done to achieve or maintain accreditation; and
(5) avoidance of confrontations or adversarial situations by displaying

courtesy, professionalism, and firm commitment to NVLAP objectives.

Role Of Technical Experts

The experts are briefed on their other responsibilities as evaluators and

technical experts. In their role as evaluators (not as assessors) they review

on-site assessments performed by their colleagues, develop consensus views

regarding controversial laboratory assessments and findings, review deficiency

corrections, consult with each other on borderline recommendations, and review

proficiency test results or other technical information to develop

recommendations for accreditation of individual laboratories.

As technical experts, they provide assistance to NVLAP in developing and

updating the technical criteria for accreditation in their field of expertise,

and advise NVLAP on proposed additions of test methods or proficiency testing

requirements to existing programs. In general, they are expected to provide

technical advice on any matter relating to requirements for accreditation and

implementation of the program.

Constraints on Assessors

The role of the assessor as NVLAP' s representative is emphasized and the

constraints on the assessor are discussed in terms of that role. It is

important that the assessors, in performing an on-site evaluation, do not put

their personal interests ahead of NVLAP concerns. Similarly, they must not

indulge in discussions or unwelcome advice that, might lead to internal

conflict, are not in consonance with NVLAP goals and intentions, or reflect

badly on NBS . It is emphasized that assessors are bound by the same ethical

practices as NBS employees.

An issue that is always raised is the potential for a NVLAP assessor to work

as an independent consultant for a laboratory seeking accreditation or already

accredited. NVLAP does not object to such contractual arrangements, but they

must be dealt with openly and honestly and not compromise the integrity of the

program or of the assessor. It is stipulated that assessors do not evaluate

the results of their own independent consulting work.

Administrative Details

Administrative details, including billing and payment procedures are discussed. NVLAP staff provide information on average costs and levels of

effort for assessments and other tasks. NVLAP costs for assessments are fully

reimbursed through fees collected from the laboratories. Assessors are encouraged to be sensitive to the need to keep fees as low as practical.

At the orientation session, each assessor is provided with an Assessor Manual

which contains copies of reference materials, assessment forms, and other

documents related to the program(s) for which assessments are to be performed.

The manuals are periodically updated and new or revised documents are sent to

the assessors to keep them current on all phases of the program.

Prior to a scheduled assessment , the assessor is provided with all appropriate

documentation relevant to the assigned laboratory, including copies of the
application forms submitted by the laboratory, proficiency testing data, and

previous assessment reports, if any, as well as any other pertinent

information on the laboratory.

INITIAL ON-SITE ASSESSMENT

The first on-site assessment is a critical phase of the new assessor's

training since the abstract discussions at the orientation session become real

and have immediate impact. Each new assessor is accompanied by a member of

(

the NVLAP staff on at least the first assessment. The staff person assists

the assessor by providing an on-going critique of the proceedings, advising on how to deal with unusual situations, helping to explain NVLAP procedures and

policies to laboratory personnel, and generally supplying guidance and

I

emotional support.

Throughout the assessment the NVLAP staff member evaluates the performance of

the assessor and, if necessary, privately discusses his:

(1)

understanding of NVLAP criteria, procedures, and policy;

(2)

knowledge of the test methods and related technology;

(3)

effectiveness in the use of the assessment form;

(4)

knowledge of the steps in performing the assessment;

(5)

ability to identify and discuss deficiencies;

(6)

effectiveness of oral and written communication;

(7)

handling of any disagreements or arguments;

(8)

appropriateness of advice given;

(9)

poise in performing the assessment; and

(10)

efficient use of time.

If the assessor does not demonstrate confidence in carrying out an assessment,

he will be accompanied by a NVLAP staff member or an experienced assessor on a second assessment for further evaluation. If an assessor is found to be

unacceptable for reasons of personality, temperament, ability to communicate,

or other reason, he will not be asked to perform further assessments.

FOLLOW UP

After a program has been operating for approximately 2 years, the assessors

are reconvened to share their experiences. The sessions are conducted by a NVLAP staff person and are generally informal interactive sessions. This

provides an opportunity for exchanging ideas on assessment techniques, problem

areas encountered and problem-solving, on refinements to the evaluation

criteria and for improving the evaluation process. These sessions are held

either at NBS or are scheduled in conjunction with a professional society

meeting which the assessors are likely to attend.

Occasionally a program is substantially modified by adding test methods or changing test methods already included in the program. When such changes

occur, a session is held to update the assessors as to the changes required in

the assessments.
The continuing acceptability of assessors is conditional on their performance

of assessments and evaluations in an objective, professional manner and in

accordance with NVLAP policies. Assessor performance is reviewed at least

annually, and more frequently if questions arise as to their ability to carry

out assigned duties or before a particularly sensitive or critical assessment

is assigned.

FUTURE ACTIVITIES IN ASSESSOR TRAINING

The requirement that NVLAP be self-supporting and the determination to keep

laboratory fees as low as practicable preclude the maintenance of sufficient

staff, or use of more than one assessor, to implement a team approach to on- site assessments. Our reliance on a single, unaccompanied assessor to

evaluate a laboratory demands that assessor training receive a very high

priority. Frequent informal discussions and interactions form the basis for

on-going training, but there is a need to increase the scope of formal

training. Within the limits of time and funding, plans are in progress to

develop training films, begin limited use of a "buddy" system for particularly

difficult or controversial assessments, and to use formal debriefing and

feedback following assessments. In addition, a NVLAP Operations Manual

provides detailed information on how NVLAP operates and describes standards of

performance to assure a high quality program. The Manual will be one of

several training devices for assessors to give them more detailed information

on NVLAP functions and operations relevant to the planning and implementation

of on-site assessments. NVLAP will continue to review assessor training

programs used by other national systems to learn of new approaches to training

that can be adopted by the NVLAP system.
BS-114A (REV. 2-«C> U.s. DEPT. OF COMM.

BIBLIOGRAPHIC DATA

SHEET (See instructions)

1. PUBLICATION OR

REPORT NO.

NBSIR 87-3651

2. Performing Organ. Report No. 3. Publication Dace

SEPTEMBER 1987

TITLE AND SUBTITLE

. NVLAP Assessor Selection and Training

—

i — 1 » — 1 1

5. AUTHOR(S)

i. PERFORMING ORGANIZATION (If joint or other than N BS. see instructions)

NATIONAL BUREAU OF STANDARDS

U.S. DEPARTMENT OF COMMERCE

GAITHERSBURG, MD 20899

7. Contract/Grant No.

*. Type of Report & Period Covered

}. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street, City . State, ZIP)

Same as 6.

10. SUPPLEMENTARY NOTES

[~~1 Document describes a computer program; SF-185, FiPS Software Summary, is attached.

11. ABSTRACT (A 200-word or less factual summary of most si gnificant information. If document includes a significant

bibliography or literature survey, mention it here)

The role and importance of "peer" assessors in the National Voluntary

Laboratory Accreditation Program (NVLAP) is discussed. Procedures for

identification, evaluation, and selection of assessors are described

and the elements of assessor orientation and training are presented.

12. KEY WORDS (Six to twelve entries; alphabetical order; capitalize only proper names; and separate key words by semicolon s) accreditation; assessment; assessor; laboratory accreditation; proficiency testing;

NVLAP

13. AVAILABILITY

1X2 Unlimited

n For Official Distribution. Do Not Release to NTIS

1 1 Order From Superintendent of Documents, U.S. Government Printing Office. Washington. D.C.

20402.

IXYl Order From National Technical Information Service (NTIS), Springfield, VA. 22161

14. NO. OF

PRINTED PAGES

11

15. Price

$9.95

USCOMM-OC 8043-P80
.