should be few surprises when an external audit is conducted. Indeed a good
system of internal audits with adequate records thereof should minimize the
need for external audits. The NBS system for Certification of the Capability
of State Measurement Laboratories recognizes this in making self-appraisal a
part of the certification process [21].

Performance audits consist of activities used to quantitatively evaluate
measurement proficiency. A laboratory's internal quality assessment program is
essentially an ongoing internal performance audit. External performance audits
(MAPs and RMMPs are elegant examples of such) can identify bias that might be
difficult to evaluate, internally.

Documentation
One aspect of quality assurance that merits emphasis is that of
documentation. All data must be technically sound and legally defensible (that
is to say, supportable by evidence of unquestionable reliability). Accor-
dingly, a metrologist must keep adequate and accurate records on such things
as:
¢ What is measured
+« Who measures
+ When measurements are made
¢ How measurements are made
- Equipment
- Calibration
- Methodology
+ Data obtained
* Calculations
* Quality assurance support
* Reports
Good metrologists have historically kept such information and a
well-managed laboratory will automatically acquire and manage it. Its quality
assurance program should address in detail the way that documentation is to be
maintained.
Quality Assurance Program Document
The various aspects of the quality assurance practices that should be

followed in. a laboratory should be developed and described in a quality
assurance program document [25]. This document will formally declare

3.5


the same item or of similar items can be used to evaluate precision (see
Chapter 8.4). At least 15 such duplicates are needed to estimate a standard
deviation with reasonable confidence such as would be needed to establish
sontrol chart limits (see Chapter 7.4) which is a recommended way to use such
data.

A check standard of reasonable stability can be used to monitor both
precision and bias, if its value is known with sufficient accuracy. Historical
data on a laboratory’s own check standard can be used to devélop a control
chart and thus monitor and assess measurement precision.

The frequency of use of any of the above internal approaches to assess
quality depends on several considerations. For an ongoing process, historical
information on its stability will provide guidance together with the risks that
are involved. Obviously all data taken within the period between the last
known in-control and first known out-of-control are suspect. A prudent
metrologist will design quality assessment procedures that will minimize such a
risk.

Measurement operations carried out intermittently or occasionally present
difficult problems for their quality assurance. In such cases, a sufficient
number of preliminary measurements need to be made to assure that the process
is in statistical control. This could require more effort than the actual
measurements of the test item.

External Approaches: Any laboratory can evaluate its own precision and should
do so before seeking external evaluation of its measurement accuracy. Until it

has acquired the capability to do so, a laboratory is virtually unqualified to
perform reliable measurements. The use of an externally provided check
standard, certified by or on the basis of NBS traceability is an excellent way
to evaluate the bias of a measurement system already demonstrated to be in
statistical control. Participation in a MAP (measurement assurance program)
[11] or in an RMMP (regional measurement management program) is an elegant way
for measurement quality assessment. Participation in less formalized round
robins provides other opportunities for quality assessment. Again, it is
emphasized that the attainment of acceptable precision, based on a laboratory's
internal quality assessment Program is a prerequisite for meaningful
participation in any external quality assessment activity.

Laboratory Audits: Audits are a valuable technique for quality assurance and
may be of both internal and external origin. System audits consist of appro-
priate inspections to assess the adequacy of various aspects of the quality
assurance system including facilities, equipment, records, and control charts.
Some audits even include investigation of the qualifications of staff.

The objective of a system audit is to determine the operational
characteristics of a laboratory's quality assurance practices. Internal audits
usually use the laboratory's stated quality assurance policy or program as the
basis of comparison. Externally conducted audits may use external standards
for this purpose. The details of either type of audit are beyond the scope of
this presentation but guidance will be found in the literature [26].

Laboratories are encouraged to conduct internal system audits at a level
of scrutiny exceeding that of any external audit. When this is done, there

3.4


management's commitment to all aspects of quality assurance and its support and
enforcement of good laboratory practices and the quality assurance plan to be
followed.

The quality assurance document should describe the maintenance procedures
for facilities and equipment, the control charts to be maintained and the
records of “out-of-control” that should be kept. The document should indicate
the procedure to be followed for review of test reports and the mechanism by
which the quality of data is assessed. Matters of safety and safe laboratory
practices should be addressed. .

An example of a quality assurance program document suitable for a State
laboratory is given in reference [21].

Responsibilities

A quality assurance program is only as effective as it is systematically
implemented. Ordinarily, this means that a formal QA program must be esta-
blished that documents the policy and the procedures to be followed. The
establishment of policy is the responsibility of management. The development
of QA procedures is a joint responsibility in which the technical staff has a
major role because of its superior knowledge of technical requirements. When
quality output is the objective of all concerned, the quality assurance program
is not a disciplinary document but one that sets forth the way in which there
is common belief that the work of the laboratory should be done.

3.6


3. QUALITY ASSURANCE IN A METROLOGY LABORATORY
Introduction

The limits of uncertainty of measurement data are of concern to both the
serious methologist and to the user of measurements. They permit the strength
and weakness of each measured value to be evaluated. They support valid data
and prevent the over interpretation of poor data. Precision estimates may be
based on the results of replicate measurements while limits for bias depend on
a critical analysis of sources of error of the measurement process. In
research investigations, all of the above is done. In practical measurement
situations, repetition must be minimized due to time and cost considerations,
and bias is often evaluated on the basis of past experience. In fact, the
objective of a good metrologist should be to conduct operations so that
“individual measurements are good enough for their intended use" [7].

The experience of metrologists has demonstrated that data reliability is
best achieved by a well-designed and operational quality assurance program.
For this reason, most certification plans, and the NBS system for Certification
of Capability of State Measurement Laboratories [21] require the existence of a
quality assurance program as one of the criteria for certification.

What is Quality Assurance

Quality assurance consists of two separate but related activities, quality
control and quality assessment. Both must be operational and coordinated. The
following definitions are offered.

Quality assurance: A system of activities whose purpose is to provide
to the producer or user of a product or a service the assurance that it
meets defined standards of quality with a stated level of confidence.

: The overall system of activities whose purpose is to
control the quality of a product or service so that it meets the needs
of users. The aim is to provide quality that is satisfactory, adequate,
dependable, and economic.

Quality assessment: The overall system of activities whose purpose is

to provide assurance that the overall quality control job is being done
effectively. It involves a continuing evaluation of the products

produced and the performance of the production system.

Quality assurance is based on the premise that measurements can be made
systematically by what may be called a measurement process, analogous to a
manufacturing process. The product of a measurement process is measurement
data that can be envisioned to attain a high degree of reproducibility similar
to the reproducibility of the products of a well-controlled production process.
In each case, reproducibility is obtained by quality control of the process.
While the techniques used may be different, the concept is the same in both
situations.

3.1


The quality of the product in each case can be evaluated by the second
aspect of quality assurance, which may be called quality assessment. In the
manufacturing »rocess, the product is tested for conformance with specifica-
tions. In the measurement process, the assessment may be made by replicate
measurements and by the measurement of a check standard. Ordinarily, and
especially in the case of measurement, the production output is sampled and
evidence is accumulated and maintained using control charts, for example, to
verify the stability of the process and to set limits on the reliability of the
data. .

Quality Control

Anything that may affect the production process must be optimized and
stabilized to the extent necessary and possible, if reproducible products are
to be obtained. In measurement processes, it is widely recognized that quality
is influenced by many factors that can be classified in three categories:
management practices; personnel; technical operations [21].

lanagement Pr: : A well-managed laboratory is essential for reliable
measurements. Both the calibre of the management staff and the policies it
develops can influence data quality. Management sets the goals of the labora-
tory, provides resources and staff, and supervises the laboratory activities.
Good management recognizes its responsibility to train and maintain staff
competence, and develops policy and provides resources to accomplish this.

Needless to say, management must be skilled in its management
responsibilities, with the skill requirement dependent on the size and complex-
ity of the laboratory. In technical organizations, managerial skill is not
enough. Managers must have a high level of technical competence, at least in
the general aspects of the laboratory's operations. They must be able to
evaluate the general quality of the laboratory output and to develop and
administer the quality assurance aspect of its operations.

Personnel: A competent staff is an absolute necessity for quality
measurements. This is often overlooked in today's highly mechanized and
automated laboratories. Each member must have an educational background,
supplemented with specific training and experience, sufficient for the duties
to be performed. Each person must understand the responsibilities of his/her
position (by suitable position descriptions and ‘indoctrination) and must have
the personal desire to perform them at a high level of competence.

Laboratory personnel are critical factors in the operational aspects of
quality assurance [24]. Not only must they perform technical operations
intelligently and skillfully, but in the full spirit of the quality assurance
requirements. Strict adherence to appropriate GLPs, GMPs, and SOPs (see
below), is essential and quality assessment must be carried out in the spirit
of its purpose -- to realistically evaluate the measurement process and its
outputs.

: All technical operations must be carried out in a
reliable and consistent manner. The first requirement is for the use of
suitable and properly maintained equipment and facilities. The equipment to be
used ordinarily is specified in the SOPs (see below) but maintenance usually is

3.2


the responsibility of the operator. At least, the operator must verify its
serviceability at the time of use. Calibration and calibration standards are
closely related to equipment and these must be "operational" as well. Mainte-
nance of facilities includes good housekeeping as well as environmental
control. Both are essential for good measurements and both can affect data
quality, introducing both systematic and random error.

Consistent and reliable data are dependent on the use of GLPs, GMPs, and
SOPs as discussed in the following. GLPs (Good Laboratory Practices) denote
those practices that the metrological community has developed to facilitate and
promote reliable and reproducible measurements. GLPs are general and relate to
most if not all of the activities and operations that a laboratory may conduct.
For example, recording and maintaining data and records is related to all of
the measurements of a laboratory. Because carelessness and inconsistencies can
introduce error or uncertainties and raise questions of reliability, the
procedures used for record keeping should be the subject of a GLP.

GMPs (Good Measurement Practices) describe recommended ways that specific
technical operations are carried out that are closely related to but are not in
themselves methods of measurement. GLPs may address some of the steps in SOPs
(see later) that are assumed to be part of the art of measurement, hence are
included only as general instructions. The method of reading a meniscus is an
example. Because variability in such operations can introduce imprecision or
bias, both within and between laboratories, critical ones are subjects for
GMPs.

SOPs (Standard Operations Procedures) describe procedures to carry out
methods for specific measurements. They consist of step-by-step instructions
and all critical operations are specified. SOPs are central to the concept of
measurement as a process. To qualify as a process, a measurement must be
carried out in a highly specific and consistent manner. The SOP defines the
modus operandi of the process. The term standard has several connotations in
this regard. It may be, but is not necessarily, a method produced by the
consensus action of a standards - writing organization. When such a method is
available, its use should be given serious consideration. Whether or not one
is used, an SOP adopted by a laboratory becomes the procedure to be followed
precisely when that kind of measurement is made; thus it becomes the "standard
method" for that laboratory. .

Quality Assessment

The term quality assessment describes those activities and procedures
utilized to monitor the effectiveness of the quality control program and to
evaluate the quality of the data output. There are both internal and external
approaches for quality assessment and some of the features of each are
described below.

Internal Approaches: Repetitive measurement is the key to evaluation of
precision. Repetitive measurements of a test item (or sample) are always
useful but this is expensive since at least 7 repetitions are required to
estimate a standard deviation within any reasonable limits of uncertainty and
30 are desirable (see Chapter 8.7). Pooling of duplicate measurement data of

3.3