NBSIR 73-162

Executive Summary of A Proficiency

Test Assessment of Clinical Laboratory

Capability in the United States

Peter W. Finkel. Ted R. Miller

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

May 1973

-Final Report

Available NTIS - #COM74-10552 (12pages)

superseded by IR 73-163

Prepared for Division of Health Evaluation

Office of the Assistant Secretary for Planning and Evaluation

Department of Health, Education, and Welfare

Washington, D. C. 20201

NBSIR 73-162

EXECUTIVE SUMMARY OF A PROFICIENCY

TEST ASSESSMENT OF CLINICAL LABORATORY

CAPABILITY IN THE UNITED STATES

Peter W. Finkel, Ted R. Miller

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

May 1973

Final Report

Prepared for Division of Health Evaluation

Office of the Assistant Secretary for Planning and Evaluation

Department of Health, Education, and Welfare

Washington, D. C. 20201

U. S. DEPARTMENT OF COMMERCE, Frederick B. Dent. Secretary

NATIONAL BUREAU OF STANDARDS, Richard W. Roberts, Director

\

Background

The primary objective o£ this study was to obtain performance

capability measures for various classes of clinical laboratories in the

United States and to determine if there are significant differences in

analytical accuracy which would warrant remedial action by public agencies

or tlie private sector.

The procedure utilized was to: (1) establish a Scientific Advisory

Committee of governmental and health industry representatives, (2) develop

a survey design, (3) contact professional and regulatory groups to solicit

laboratory participation, (4) procure and distribute two sets of laboratory

samples in clinical chemistry, hematology and bacteriology to laboratories

participating in the study, and (5) statistically analyze the results

reported by the laboratories.

Participants

Six types or groups of clinical laboratories participated in the

study:

1. Interstate—laboratories engaged in interstate commerce and

licensed by the Center for Disease Control, Health Services and Mental

Health Administration, under the Clinical Laboratory Improvement Act

of 1967 (CLIA '67) in one or more of the specialties under consideration

in this study.

2. American Academy of Family Physicians—^private physician

laboratories, generally small, which are currently affiliated with AAFP.

3. American Society of Internal Medicine—^laboratories operated in

conjunction with a private physician's practice which, like AAFP affiliated

laboratories, are presently exempt from any Federal licensure program.

4. Joint Commission on Accreditation of Hospitals—laboratories

within hospitals which are accredited under JCAH.

5. Medicare Certified Hospitals—^laboratories in hospitals which

are Medicare providers under Title XVIII, Health Insurance for the Aged.

Hospitals and laboratories accredited under JCAH were excluded from this

category.

6. Medicare Certified Independent—private and commercial laboratories

which are reimbursed for certain laboratory procedures under Medicare, but

which are not normally licensed under CLIA '67 or accredited by JCAH.

i
Approximately 1,000 laboratories participated in the study. The

number within a category ranged from 43 (AAFP) to 231 (Interstate) . A

seventh category of 18 reference laboratories served as a control group.

Methodology

Separate proficiency test specimens were prepared for clinical chemistry,

hematology and microbiology. Criteria for selection of the clinical

chemistry and hematology constituents to be analyzed were that laboratory

analysis should be routine, and that fairly well developed analytical

procedures exist. Each clinical chemistry shipment included normal and

abnormal samples. The laboratories were asked to determine the concentration

of eight constituents: glucose, urea nitrogen, calcium, total bilirubin,

cholesterol, uric acid, sodium, and total protein. Each hematology shipment required analysis of red blood count, white blood count, hemoglobin,

hematocrit, and mean corpuscular volume levels in both normal and abnormal

specimens. Five pure cultures of ordinary bacteria were used as sample

cultures for identification in the microbiological portion of the study.

Two shipments of specimens and/or cultures were sent to each

participating laboratory. All analytic results were reported by the

laboratory on forms shipped with the test specimens.

Results - Clinical Chemistry

There were no significant differences (at the 90% confidence level)

among the average laboratory results obtained by the groups participating

in the study. The interlaboratory consistency ("interlaboratory precision")

of the laboratory groups can be exhibited as follows, where groups joined

by the same line did not exhibit significantly different precision

at the 901 confidence level.

Rank Order Laboratory Group

Most Precise Medicare Independent

Interstate

JCAH

AAFP/ASIM

Least Precise Medicare Hospital

The techniques used had a considerable effect on the accuracy and

precision of reported analyses. Table 1 lists the techniques which

were most satisfactorily applied and the percentage of the participating

laboratories which applied each technique. In most instances, automated

methods were applied with equal or better average accuracy and considerably

better precision than the corresponding manual methods. Results reported by

laboratories using diagnostic kits were consistently less precise than

other determinations.

ii
•H cn

CO

CD

+->

t/) O

pq

CDN

r—

OP

•Hu

•H

<D

CD

+-> op

CDO

•HQ

Id

CD

+->

CDO0o oooHI<

CD r—

03 XiP

1—( o

(/) CD

n3

CD P

ojI

+->

U5-o

CD OI oN

•HQ

O

•HPo

•Po

•H

5h •p

•H

O

J-<

CD •p

nJo

•p

CD P

Oj p

CM

O I t—

CD c3 Po

ino

CD •P

CT3 p

CD •P

CD

-po

tin

•H

(A to

O

to

O CNI \0 P

CD

CD

Jh

CD

t+-l CD

•pp

•HP

(AO

CDm

oJ •HoX

CDtoou

pH

CD

x:si

+-> (DU

g

•HO

c/) OH

rHO

Ph 5h o

CJ U5-

5h ^CD rd

•P o

Cfl (A

!h 0 t3

01 CD

0}

Pi

CD

(D

C/i u

•H

5h 13

CDtoo

fH P

•H

ct3 0

5-1 13

•HU

I—

•H

PQ

O

f-i 0P

t/5 0

t—

O

rCU

t3•HO

•H

5-4

5h0P0§

-PO0

Tdo

CO iii
Insufficient information was available to assess the medical usefulness

of the total bilirubin determinations. Of the remaining seven constituents,

only cholesterol was analyzed by the study participants -with sufficient

precision to permit the interlaboratory monitoring over time of the

variation in an individual patient's constituent concentrations. In

contrast, reference laboratory analyses of cholesterol, uric acid, urea

nitrogen, sodium and total protein were all sufficiently precise to

permit interlaboratory monitoring of individual variation. Those participating laboratories using the best applied techniques also achieved

acceptable interlaboratory precision in analyses of these five constituents.

Results - Hematology

As with clinical chemistry, the average laboratory results obtained

by the participating groups did not differ significantly at the 951

confidence level. The interlaboratory precision of the laboratory groups

can be exhibited as follows where groups joined by the same line did not

exhibit significant differences at the 90% confidence level.

Rank Order Laboratory Group

Most Precise Medicare Independent

JCAH

Interstate

Medicare Hospital

Least Precise AAFP/ASIM

Table 2 shows the best applied techniques and the percentages of participating laboratories using these techniques.

Results - Microbiology^

The performance of the Interstate group was significantly better

than the performance of the other groups at the 95% confidence level;

7.6% of the Interstate laboratory determinations were incorrect while

19.9% of all other determinations were incorrect. However, even a 7.6%

mis- identification rate is not satisfactory. Most troublesome are such

mis- identifications as Neisseria NOS, N. gonorrhoeae or N. meningitidis

for the pure culture of Streptococcus faecalis.

The relative perfomance of the laboratory groups can be portrayed

as shown, where the lines join groups whose performance did not differ

significantly at the 95% confidence level.

iv
o

p! •H cn o (/) 1 LD o 1 !>- o 1 rH

rH

rH

t/) iH D rH

Oj

cr" •Hs CD u 'Td CD CD

E-i •H

. iH CD

(D t— a

1

i-H & Td

•H CD CD 03

1—

1

n3 +-> 'r-i •H ^

CD O

u +-> o cr

+-> (D 0 4-> •H (D

(/) +-> 4-> rH

0) t—

1

rH X3

PQ D u •H O o CD t/) U u o -M on

to o

rH +J I-H +-> u iH ^ t5 Mi TV t

CD

iH

nt

CD

03

+-> p rH •H O :3

+-> u (J o C/) u rH M c

o iH •H •H u 1—

1

CD 1— U U O O

CD O rH u CD -M tiO M

•H i

CD CD Pi X

V
Rank Order Laboratory Group

Best Performance Interstate

JCAH

Worst Performance

Medicare Independent

AAFP/ASIM

Medicare Hospital

Conclusions

The data indicate that high volume laboratories may be more proficient

than smaller laboratories, such as those which serve Doctors' Offices and

Medicare Certified Hospitals. In microbiology, 7.6% of the Interstate

laboratory determinations were incorrect, while 16.5% of the determinations

by other large laboratories (JCAH and Medicare Independents) were incorrect.

Thus, it would appear that the CDC proficiency testing program has

considerably improved the microbiology performance of the enrolled laboratories. Conversely, clinical chemistry and hematology analyses by the

Interstate laboratories were no better than comparable analyses by other

large laboratories, many of whom do not engage in routine proficiency

testing programs. This seems to indicate that the CDC proficiency testing

programs in clinical chemistry and hematology have had relatively little

effect upon the performance of laboratories participating in the program.

This conclusion is further substantiated in a companion report.* It is

particularly important to improve the effectiveness of these programs

because the interlaboratory consistency of study participants with

respect to clinical chemistry and hematology was too often insufficient

to support monitoring of an individual's constituent concentrations over

time. It appears that poor selection of techniques is an important factor

in the low rate of acceptability of laboratory determinations.

Limitations

It must be clearly understood that the results of this survey are limited by four important considerations:

1. Because all of the laboratories participated on a purely voluntary

basis, no straightforward extrapolation can be made to the larger universe

of unsampled clinical laboratories.

2. It is probable that the results of this study do not represent

routine laboratory performance for two reasons: (a) a laboratory probably

would not volunteer if its management felt that to do so would be

disadvantageous, and (h) the sample materials probably received special

attention in many of the smaller laboratories which were unfamiliar with

analyzing proficiency test samples.

*Clinical Laboratory Performance Analysis Using Proficiency Test Statistics,

NBS Report, NBSIR 73 197, 1973 (to be published).

VI
3. The clinical chemistry test specimens were prepared by a dialyzation process which removes naturally occurring reducing agents

and other substances. As a result, the accuracy of some methods, as applied to the test specimens, might differ from their accuracy in

analyses of human serum.

4. The true constituent concentrations of cholesterol and the

hematology constituents could not be exactly determined. For these

constituents accuracy was assessed relative to the mean reference

laboratory assays.

Recommendat ions

Satisfactory performance in a microbiology proficiency testing

program conducted under the auspices of either Federal or other approved

authorities should be a legislative requirement for all clinical laboratories analyzing microbiological specimens.

A Technical Advisory Committee consisting of government and professional society representatives should be established to identify the most

accurate and precise analytical methods available and encourage their

use by the largest possible number of clinical laboratories. Zones of

acceptable performance for proficiency testing should be constructed in

a manner which reflects the variability associated with the more accurate

and precise methods and systems. In this way, failure to accept the

recommended procedures would increase the risk of unacceptable performance

ratings

.

An experimental study should be undertaken to determine a better

design for proficiency testing programs in clinical chemistry and hematology.

An empirical description of the causes of inadequate laboratory work should

be used in defining alternative testing strategies for consideration. This

study should deal with such questions as frequency of sair^ling; feedback

to participants; number of levels at \^^ich to test; long-term monitoring

of intralaboratory variability; follow-up procedures on outlier values;

and the criteria for scoring, ranking or rating laboratory performance

and its medical usefulness. Until the results of this recommended study

become available, it does not appear justified (or warranted) to alter

the frequency of CDC proficiency testing in clinical chemistry and

hematology.

vii

PORM NBS-n4A (1-71) U.S. DEPT. OF COMM. 1 . PU B LIC AT ION OR R E PORT NO. 2. Gov't Accession

BIBLIOGRAPHIC DATA mhotd 7-5 i AO

5l^£gj NBSIR 7J-iDZ

3. Recipient's Accession No.

4. TITLE AND SUBTITLE

Executive Summary of A Proficiency Test Assessment of Clinical

Laboratory Capability in the United States

5. Publication Date

6. Performing Organization Code

7. AUTHOR(S)

Peter W. Finkel and Ted R. Miller

8. Performing Organization

NBSIR 73-162

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

UEFAKIMENI Or CUMMhRCb

WASHINGTON, D.C. 20234

10. Project/Task/Work Unit No. 4314457

11. Contract/Grant No.

12. Sponsoring Organization Name and Address

Division of Health Evaluation

Office of the Assistant Secretary for Planning, and Evaluation

Department of Health, Education and Welfare

Room 5526 HEW North Building, Washington, D. C. 20201

13. Type of Report & Period

Covered

Final 3/71 - 4/73

14. Sponsoring Agency Code

15. SUPPLEMENTARY NOTES

16. ABSTRACT (A 200-word or less factual summary of most significant information. If document includes a significant

bibliography or literature survey, mention it here.) Executive summary of a report abstracted as follows:

The proficiency of a selected sample of physician, hospital and independent

laboratories was assessed with respect to their ability to analyze clinical chemistry

and hematology samples and to identify microbiological organisms. For the assessment

of clinical chemistry and hematology proficiency, the laboratories were grouped and

determinations of group accuracy and group precision were made. Further analyses

were performed to determine relative accuracy and precision of the techniques

presently applied by these groups. There was no significant difference at the 951

confidence level in the accuracy achieved by the various laboratory groups involved in

Clinical chemistry and hematology analysis. In clinical chemistry, the MedicareCertified Independent Laboratories, CDC Tested Laboratories and JCAH-Members generally

proved more precise than Physicians' Office and Medicare- Certified Hospital Laboratories.

However, none of the laboratory groups were sufficiently accurate to permit the

monitoring over time of variation in an individual patient's constituent concentrationf;

.

It would appear that poor selection of techniques was an important contributor to this

low performance level. In hematology the Physicians' Office Laboratories proved to be

the least precise of the groups. There was no noticeable difference in precision

between participants in the CDC proficiency testing program and non-participants. With

respect to microbiology, 7.6% of the identifications by laboratories participating in

the CDC testing program were incorrect, while 19.4% of all other identifications

were incorrect.

' 17. KEY WORDS (Alphabetical order, separated by semicolons)

i| Accuracy; clinical chemistry; hematology; medical usefulness; microbiology, proficienc)'

, r.estmg — 1 18. AVaILA^ILLi 1 STATEMEiMx 19. SECURITY CLASS

(THIS REPORT)

21. NO. OF PAGES

UNLIMITED.

UNCL ASSIFIED

1 1 FOR OFFICIAL DISTRIBUTION. DO NOT RELEASE

TO NTIS.

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

22. Price

USCOMM-DC 66244-P71