NBSIR 73-163

A Proficiency Test Assessment of

Clinical Laboratory Capability in

the United States

Peter W. Finkel, Ted R. Miller

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

May 1973

Final Report Available NTIS - #COM74 10542 (163pagei

^ see IR73-197

Supersedes NTIS Nos. COM 73-11190 and COM 73-11193

Prepared for Division of Health Evaluation

Office of the Assistant Secretary for Planning and Evaluation

Department of Health, Education, and Welfare

Washington, D. C. 20201

NBSIR 73-163

A PROFICIENCY TEST ASSESSMENT OF

CLINICAL LABORATORY CAPABILITY IN

THE UNITED STATES

Peter W. Finkel, Ted R. Miller

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

May 1973

Final Report

Prepared for Division of Health Evaluation

Office of the Assistant Secretary for Planning and Evaluation

Department of Health, Education, and Welfare

Washington, D. C. 20201

U. S. DEPARTMENT OF COMMERCE, Frederick B. Dent. Secretary

NATIONAL BUREAU OF STANDARDS, Richard W. Roberts. Director
ACKNOWilDGMLVrS

The authors gratefully acknowledge the participation of the following persons during

the conduct of this study:

Dr. Margaret M. Gurney, Statistical Research Division, Bureau of the Census, for con- sultation in the sanpling of the various laboratory groups;

Dr. Joan R. Rosenblatt, Statistical tngineering Section, Applied Mathematics Division,

NBS for technical review and critique of the many drafts and the final version;

Drs. Roy N. Bamett, Joseph H. Boutwell, Harvey Gralnick, James D. MacLowry, W. Wayne

Meinke, M. Keith Weikel and Donald S. Young—members of the Scientific Advisory Committee

without whose contributions the study could not hav^e been completed;

Ms. Patricia Cullen, Mr. liugene Balzer, Mr. Donald Kronish, Mr. William Young, Mr. Charles Sloane and so many other dedicated people at Warner-Lambert, for unstinting, sin- cere and tireless support and technical consultation;

Mrs. Ruth Wamock and Miss June Watkins for arranging and preparing the many drafts

and final version;

Dr. John Rowen, Mr. John Flannagan, Ntrs. Mary Friend, Nfr. Robert Lemieux, Mr. Eli

Mantel, Mr. Rolfe MacCullough, Ms. Marilyn Westfall, Mr. Barry Hocks and other members of

the staff of the Technical Analysis Division for their work in support of the project.

ii
ExnamvE sijnwary

A. BACKGROUND

There are estimated to be at least 15,000 clinical laboratories in the United States,

ranging in size from small laboratories operated in conjunction with a physician's practice

to extremely large laboratories which process millions of specimens each year on a mail order

basis. Various accreditation and licensing programs periodically monitor the performance of

some clinical laboratories through inspection and/or proficiency testing. Other laboratories

are subject to little or no control.

The effect of regulatory activities on the performance of clinical laboratories has been

investigated in prior evaluations. In general, these studies and surveys have been of limited scope and directed toward specific groups of laboratories, such as those within a state,

a subset of hospital laboratories, or laboratories engaged in interstate coimerce.

Unfortunately, it is not possible to "pool" data from these experiments in order to establish relative performance measures for various classes of lal)oratories . Analysis of

the relevant literature shows a lack of compatibility: variations in tenporal aspects,

sanple sizes, constituents analyzed, and a host of other discrepancies which make valid data

aggregation or extrapolation impractical.

In order to obtain comparable data on laboratory performance, the Division of Health

Evaluation, Department of Health, Education, and Welfare, contracted with the Technical

Analysis Division of the National Bureau of St.-indards in January 1971 to develop and implement a comprehensive national survey of clinical laboratories.

The major objective of the study was to obtain measures of capaliility for several different types or groups of clinical laboratories and to determine if there are basic differences in analytical accuracy among these groups which would warrant remedial action by public

agencies or the private sector. /"Analytical capability was judged by the performance of participating laboratories in analyzing proficiency test specimens in the areas of clinical

chemistry, hematology, and microbiology. A secondary objective was to conpare the accuracy

and precision of the various analytical methods used by the participating laboratories.

The procedure utilized was to: (1) establish a Scientific Advisory Committee of governmental and health industry' representatives to assist in the selection of proficiency test

conponents and monitor the progress of the study from a clinical viewpoint, (2) develop a survey design, (3) contact professional and regulatory groups to solicit laboratory participation, (4) procure and distribute two sets of laboratoiy samples in clinical chemistry,

hematology and bacteriology to laboratories participating in the study, and (5) statistically analyze the results reported by the laboratories.

B. PARTICIPANT I^\BORATORIES

Six types or groups of clinical laboratories participated in the study:

1. Interstate—laboratories engaged in interstate connerce and licensed by the Center

for Disease Control (CDC), Health Services and Mental Health Administration, under

the Clinical laboratory Improvement Act of 1967 (CLIA '67) in one or more of the

specialties under consideration in this study. This Act requires licensees to participate in either the CDC or the College of American Pathologists proficiency

testing program.

2. American Academy- of Family Physicians (.V\fT)—private physician laboratories, generally small, which are currently affiliated with .\AFP. These laboratories are presently exempt from any Federal licensure program.

3. American Society of Internal Medicine-—laboratories operated in conjunction with a private physician's practice which, like VAFP affiliated laboratories, are presently

exempt from any Federal licensure prograjn.

i i i
4. Joint Comnission on Accreditation of Ftospitals (JCAH) —laboratories within hospitals

vAich are accredited under JCAH. The JCAH inspects these laboratories, but does

not require them to participate in proficiency testing.

5. Medicare Certified Hospitals—laboratories in hospitals which are Medicare providers

under Title XVIII, Health Insurance for the Aged. "Hiese laboratories are also

Medicaid providers. Hospitals and laboratories accredited under JCAH were excluded

from this category. As a result, the laboratories in this category tend to be the

smallest laboratories which are not operated in ccnjunction with a private physician's practice.

6. Medicare Certified Independent—private and conmercial laboratories which are reimbursed for certain laboratory procedures under Medicare and Medicaid, but which are

not normally licensed under CLIA '67 or accredited by JCAH. .The law requires that

these laboratories participate in a proficiency testing program administered or approved by the regulatory program in the state in which they operate.

Approximately 1,000 laboratories participated in the study. The number within a cate- gory ranged from 45 (AAFP) to 231 (Interstate).* A seventh category of 18 reference laboratories served as a control group.

C. NETHODOLOGY

Separate proficiency test specimens were prepared for clinical chemistry, hematology

and microbiology. Criteria for selection of the clinical chemistry and hematology constit- • uents to be analyzed were that laboratory analysis should be routine, and that fairly well

developed analytical procedures exist.

Each clinical chemistry shipment included one normal and one abnormal sample. TTie laboratories were asked to determine the concentration of eight constituents: glucose, urea

nitrogen, calcium, total bilirubin, cholesterol, uric acid, sodium, and total protein. Each

hematology' shipment required analysis of red blood count, white blood count, hemoglobin,

hematocrit, and mean corpuscular volume levels in both normal and abnormal specimens. Five

pure cultures of ordinary and easily identifiable bacteria were used as sauple cultures for

identification in the microbiological portion of the study.

Two shipments of specimens and/or cultures were sent to each participating laboratory.

All analytic results were reported by the laboratory on forms shipped with the test specimens.

D. RESULTS

For analysis, the AAFP and ASIM data were pooled together into a Doctor's Office group.

All groups were then sufficiently large to allow extrapolation of the study results to the

unsajrpled laboratories with 95% confidence. However, since participation was on a voluntary

basis, the sample is a selected rather than a random sample, and it woul.d be invalid to extrapolate to the unsanpled population.

Clinical Chemistry

There were no significant differences (at the 95% confidence level) among the average

clinical chemistry laboratory results obtained by the groups participating in the study.

The interlaboratory consistency (interlaboratory precision) of the laboratory groi4)S can be

exhibited as follows, where groups joined by the same line did not exhibit significantly

different precision at the 90% confidence level.

*Accurate information is not available on the nunfcer of laboratories which are members of

each group. Even if such information were available, it would be improper to use the

relative numbers of laboratories as measures of group activity because the average number

of analyses performed by a laboratory in a year is much higher for some groins than for

others.

iv
Rank Order Laboratory Group

Most Precise

Least Precise

Medicare Independent

Interstate

JCAJI AAFP/ASIM

Medicare Hospital

The line diagram indicates that the Medicare Independent Laboratory analyses were significantly more precise than those of all other groups except the Interstate group. No

other differences in group precision were statistically significant.

The techniques used had a considerable effect an the accuracy and precision of reported

analyses. Table 1 lists the techniques which were most satisfactorily applied, and the percentage of the participating laboratories which applied each technique. As the table

implies, a large percentage of the participating laboratories are using outmoded clinical

chemistry techniques. In most instances, automated methods were applied with equal or better average accuracy and considerably better precision than the corresponding manual

methods. Results reported by laboratories using diagnostic kits were consistently less

precise than other determinations.

Medical usefulness of the clinical chemistry analyses was assessed using the criterion

that analyses should be sufficiently precise to permit the interlaboratory monitoring over

time of the variation in an individual patient's constituent concentrations. This criterion

was not applied to total bilirubin because data on individual bilirubin variation was not

available. Of the remaining seven constituents, only cholesterol was analyzed by the study

participants with sufficient precision to satisfy the criterion. In contrast to the performance of the study participants, reference laboratory analyses of cholesterol, uric acid, urea nitrogen, sodium and total protein were all sufficiently precise to permit interlaboratory monitoring of individual variation. Those participating laboratories using the best

applied techniques also achieved acceptable interlaboratory precision in analyses of these

five constituents.

Hematology

As with clinical chemistry, the average hematolog)' results obtained by the participating groups did not differ significantly at the 951 confidence level. The interlaboratory

precision of the laboratory groups can be exhibited as follows where groups joined by the

same line did not exhibit significant differences at the 90% confidence level.

Rank Order Laboratory Group

Most Precise Medicare Independent

JCAH

Interstate

1 Medicare lk)spital

Least Precise 1 A/\FP/ASIM

Table 2 shows the best applied techniques and the percentages of participating laboratories

using these techniques.

V
Table 1. Clinical Chemistr)' - Best Applied Techniques

Constituent Best Applied Techniques % Applying

Glucose Ferricyanide AutoAnalyzer 12.9

Urea Nitrogen Uiacetyl Monoxijne Automated 34.

Calcium Atomic Absorption

Cresolphthalein Complexone Automated

4.0

33.4

Bilirubin Diazo-Other Coupl ing (J 5 G) Automated 24.9

Cholesterol FeCl3-H2S04 with Prior Extraction

Manual or Automated

5.9

Uric Acid Uricase

Phosphotungstate Automated

2.2

24.2

Sodium Flame Photometer Automated 12.6

Total Protein Biuret Automated

Refractometer

34.5

15.7

Table 2. Hematology - Best Applied Techniques

Constituent Best Applied Techniques % Applying

Red Cell Count All Coulter Models

Kits

43.3

White Cell Count All Coulter Models

Kits

45.7

Hematocrit Microhematocrit 75.9

Hemoglobin All techniques applied equally well 100.0

Mean Corpuscular Volume Impossible to judge
Microbiology

Tlie performance in microbiology of the Interstate group was significantly better than

the performance of the other groups at the 951 confidence level ; 7.6% of the Interstate laboratory determinations were incorrect while 19.')°6 of all other determinations were incorrect.

However, even a 7.6% misident i f ication rate is not satisfactory' because the organisms

selected arc common and should be easily identifiable. Most worrisome are such mis identifications as Neisseria N()S, N. gonorrhoeae or N. meningitidis for the pure culture of

Streptococcus faecal is.

Tlie relative performance of the laboratory groups can be portrayed as shown, where the

lines join groups whose performance did not differ significantly at t)ie 95% confidence level.

Rank Order Laboratory' Groty

j Best Performance

Worst Performance

Interstate

JCAli Medicare Independent

MIT/,\SIM I Medicare Hospital !

K. CONCLUSIONS

The data indicate that high volume laboratories may be more proficient than smaller

laboratories, such as those which serve Doctors' Offices and Medicare Certified Hospitals.

In microbiology, 7.61 of the Interstate laboratory determinations were incorrect, while

16.5% of the determinations by other large laboratories (JCAH and Medicare Independents) were incorrect. Thus, it would appear that the CIX^ proficiency testing program has considerably improved the microbiology performance of the enrolled laboratories (although further

improvement is still needed). Conversely, clinical chemistry and hematology analyses by the

Interstate laboratories were no better than comparable analyses by other large laboratories,

many of whom do not engage in routine proficiency testing programs. This seems to indicate

that the CAT. proficiency testing programs in clinical chemistry and hematology have had

relatively little effect upon the performance of laboratories participating in the program.

This conclusion is further substantiated in a companion report.* It is particularly impor- tant to improve the effectiveness of these programs because the interlaboratory consistency

of study participants with respect to clinical chemistry and hematology was too often

insufficient to support monitoring of an indi vidu^il ' s constituent concentrations over time.

It appears that poor selection of techniques is an important factor in the low rate of

accept£ibil ity of laboratory determinations.

r. LINniATIONS

It must be clearly understood that the results of this survey are limited by four impor- tant considerations:

1. Because all of the laboratories participated on a purely voluntary basis, no straightforward extrapolation can be made to the larger universe of unsairpled

clinical laboratories.

2. It is probal-)le that the results of this study do not represent routine laboratory

performance for two reasons: fa) a laborator>' probably would not volunteer if its

*Clinical Laboratory' Performance Analvsis Using Proficiency Test Statistics, NBS Report,

NBSIft 7M97, June, 197^.

Vll
management felt that to do so would be disadvantageous, and O"") the sajiple materials

probably received sjxjcial attention in many of the smaller laboratories which were unfamiliar with analyzing proficiency test sanples.

3. The clinical chemistry test specimens were prepared by a dialyzation process which

removes naturally occurring substances. As a result, the accuracy of some methods,

as applied to the test specimens, might differ from their accuracy in analyses of

human serum.

4. The true constituent concentrations of cholesterol and the hematology constituents

could not be exactly detemined. For these constituents, accuracy was assessed

relative to the mean reference laboratory assays.

C. RECONMI:NI)ATIONS

The following recommendations are offered as logical outgrowths of the conclusions

noted earlier in E:

1. Satisfactory performance in a microbiology proficiency testing program conducted

under the auspices of either Federal or other approved authorities should be a requirement for all clinical laboratories analyzing microbiological specimens.

It is unclear whether this reconmendat ion should be implemented through new

legislation or a re interpretation of existing legislative authority.

2. A Technical Advisory (Committee consisting of government and professional society

representatives should be established to identify the most accurate and precise

analytical methods available, and encourage their use by the largest possible

number of clinical laboratories. Zones of acceptable performance for proficiency

testing should be constructed in a manner which reflects the variability associated

with the more accurate and precise methods and systems. In this way, failure to accept the recommended procedures would increase the risk of unacceptable performance ratings.

3. An experimental study should be undertaken to determine a better design for

proficiency testing programs in clinical chemistry and hematology. An empirical description of the causes of inadequate laboratory work should be used

in defining alternative testing strategies for consideration. This study should

deal with such questions as frequency of sanpling; feedback to participants;

number of levels at which to test; long-term monitoring of intralaboratory variability; follow-up procedures on outlier values; and the criteria for scoring,

ranking or rating laboratory performance and its medical usefulness. Until the

results of this recommended study become available, it does not appear justified

(or warranted) to alter the frequency of CDC proficiency testing in clinical

chemistry and hematology.

viii
CONTEVrS

Page

Executive Summary ; iii

1. Introduction 1

2. Methodology 3

2.1. Scientific Advisory Conmittee 3

2.1.1. Proficiency Test Components 3

2.2. Types of Clinical Laboratories Represented 3

2.2.1. Interstate Laboratories 4

2.2.2. American Academy of Family I'hysicians 4

2.2.3. American Society of Internal Medicine 5

2.2.4. Joint Comnission on Accreditation of Hospitals 5

2.2.5. Medicare Certified Itospitals and Independent Laboratories 6

2.2.6. Reference Laboratories 6

2.3. Selection of Contractor for Sample Materials 6

2.4. Specifications for Sample Materials 6

2.4.1. LstablLshing Target Values for Clinical Chemistry 6

2.4.2. Desired Levels of Hematology Samples 8

2.4.3. Selection of Organisms for Microbiology 10

2.5. Hxecution of the Proficiency Test Program 11

2.5.1. Shipment of Samples 11

2.5.2. Response , 11

2.5.3. Follow-Up to Non-Respondents 13

2.5.4. Other F>roblems 13

2.5.5. Treatment of Outliers 13

2.6. Limitations 15

3. Clinical Chemistry 15

3.1. Definition 15

3.1.1. Accuracy 15

3.1.2. Precision 20

3.1.3. Acceptability of Laboratory Detenni nations 20

3.2. Topics Addressed 22

ix
CONTENTTS (continued)

Page

3.3. Analysis Plan. .. * 23

3.3.1. Relative Accuracy 23

3.3.2. Precision 24

3.3.3. Medical Acceptability of Results 24

3.3.4. Target Value Surrogates 24

3.3.5. Dialyzed Versus Pooled Serum Specimens 24

3.3.6. Comparison with Other Proficiency Tests 24

3.4. Reporting of Results 24

3.5. Precision and Accuracy 25

3.5.1. Glucose 25

3.5.2. Urea Nitrogen 28

3.5.3. Calcium 28

3.5.4. Total Bilirubin 28

3.5.5. Cholesterol 34

3.5.6. Uric Acid 34

3.5.7. Sodium 40

3.5.8. Total Protein 40

3.5.9. Summary of Group and Technique Performance 40

3.6. Medical Acceptability and Usefulness of Results 47

3.6.1. Acceptability of Laboratory Group Precision 47

3.6.2. Likelihood of a Medically Useful Laboratory Analysis 49

3.7. Target Value Surrogates 51

3.7.1. Adequacy of the Manufacturer's Average Assay as a Target Value 51

3.7.2. Other Possible Targets for Proficiency Testing 54

3.8. Comparison with Other Proficiency Surveys 54

4. Hematology 56

4.1. Precision and Relative Accuracy 56

4.1.1. Red Cell Count 56

4.1.2. White Cell Count 56

4.1.3. Hematocrit 61

X
COhTTENTS (continued)

Page

4.1.4. Hanoglobin 61

4.1.5. Mean Corpuscular Volume 61

4.1.6. Summary 61

4.2. Comparison with Other Proficiency Surveys 68

4.3. Acceptability of Laboratory Group Precision 68

5. Microbiology 71

5.1. Background 71

5.2. Overall Performance 73

5.3. Comparison of Group Performance 74

5.3.1. Non-parametric Approach ; 74

5.3.2. Parametric Approach 76

5.3.3. Performance Ranking 76

5.4. Summary of Group Identification 76

5.4.1. Overview of All Participants 77

5.4.2. Doctors' Offices 79

5.4.3. Interstate Licensure (CDC) Laboratories 80

5.4.4. JCM{ Hospital Laboratories 81

5.4.5. Medicare Hospital Laboratories 82

5.4.6. Medicare Independent Laboratories 84

6. Summary of Results 85

6.1. Clinical Chemistr>' 85

6.2. Target Value Surrogates. 86

6.3. Hematology i 86

6.4. Microbiology 86

6.5. Conclusions 86

6.6. Limitations 87

7. Recommendations and Their Implementation 87

7.1. Recormendations 87

7.2. Implementation 88

xi
romm'S (continued)

Page

Appendix A. Scientific Advisory Committee for Study of Clinical Laboratory

Capability 89

Appendix B. Reference Laboratories for Study of Clinical Laboratory Capability 89

Appendix C. Differences in Laboratory Performance When Analyzing Dialyzed and

Serum Pool Specimens 90

Appendix D. Medical Usefulness Plots 93

Appendix L. References for Tables 3.8.1 and 4.2.1 101

xi i
A PROFICIENCY TEST ASSESSMENT OF CLINICAL LABORATORY CAPABILITY IN TfE UNITED STATES

Peter W. Finkel

Ted R. Miller

The proficiency of a representative sample of physician, hospital and

independent laboratories was assessed with respect to their ability to analyze

clinical chemistry and hematology sanples and to identify microbiological

organisms. For the assessment of clinical chemistry and hematology proficiency,

the laboratories were grouped, and determinations of group accuracy and group

precision were made. Further analyses were performed to determine relative

accuracy and precision of the techniques presently applied by these groups.

There was no significant difference at the 951 confidence level in the accuracy

achieved by the various laboratory groups involved in clinical chemistry and

hematology analysis. In clinical chemistry, the Medicare Certified Independent

laboratories, CDC Tested laboratories and JCJMI Members generally proved more precise than Physicians' Office and Medicare Certified Hospital laboratories.

Ftowever, none of the laboratory groups were sufficiently accurate to permit

the monitoring over time of variation in an individual patient's constituent

concentrations. It would appear that poor selection of techniques was an important contributor to this low performance level. In hematology the

Physicians' Office laboratories proved to be the least precise of the groups.

There was no noticeable difference in precision between participants in the

CDC proficiency testing program and nonparticipants. With respect to microbiology, 7.6% of the identifications by laboratories participating in the CDC

testing program were incorrect, while 19.41 of all other identifications were

incorrect.

Key words: Accuracy; clinical chemistry; hematology; medical usefulness;

microbiology; proficiency testing.

1. INTRODUCTION

There are estimated to be at least 15,000 clinical laboratories in the United States,

ranging from small laboratories operated in conjunction with a physician's practice to extremely large laboratories which process millions of specimens each year on a mail order

basis. A major group of clinical laboratories are involved in interstate commerce. These

laboratories are normally licensed in one or more "procedure" or "category of procedure"

(microbiology, serology, etc.) under Public I^w 90-174 (Partnership for Health Amendments

1967). Section 5 of this law includes provisions relating to the application for and issu- ance of licenses, quality control, personnel standard setting, and proficiency testing. The

Center for Disease Control (CDC), Department of Health, Education, and Welfare (DHEW)

serves as the Federal regulatory agency for this legislation, which applies to between

5 to 10 percent of all clinical laboratories in the U. S.

Recently, many states have passed new legislation or strengthened existing laws in an effort to insure high quality laboratory results. Such legislation is primarily directed

to hospitals and independent laboratories not involved in interstate coimerce. Regulatory

activities, including inspection, staff accreditation and proficiency testing, are dependent

on the strength of the legislation involved. Twenty-six states are currently conducting

some form of regulatory activity. However, in all instances, laboratories operated in con- junction with a physician's practice are excluded from legislative actions as long as the

laboratory is operated solely for the benefit of the physician's patients.

In addition to Federal and state regulatory' activities, three other accreditation

programs are worthy of mention. In 1962 the College of American Pathologists (CAP) initiated its own program of laboratory inspection and accreditation for member laboratories.

This program involves on-site inspections, proficiency testing and the use of certification

criteria pertaining to laboratory facilities, services, procedures, and staff qualifications.

A second laboratory accreditation program, which does not presently include proficiency

testing, is administered by the Joint Commission on Accreditation of Hospitals (JCAH)

Currently, more than 2,000 hospitals throughout the U. S. arc surveyed annually by the

1
Joint Commission. Ihe third accreditation program, also lacking proficiency tests, is

associated with the Medicare program; state agencies accredit hospitals and independent

comnercial firms as being eligible for payment of Medicare funds.

Thus, various accreditation and licensing programs periodically monitor clinical laboratory performance through inspection and/or proficiency testing. Most independent and

hospital laboratories are accredited or licensed under one or more of these programs. However, almost all laboratories operated exclusively by and for the private physician in con- junction with his practice are not similarly covered although such laboratories p)erform

approximately 2S% of all laboratory tests each year.

The effect of regulatory activities on the performance of clinical laboratories has

been the subject of prior evaluations. A review of the literature indicates that a sizable

amount of laboratory performance data has been collected since the late 1940's. In general,

these studies and surveys have been of limited scope and were directed to specific groups

of laboratories, such as those within a state, a subset of hospital laboratories, or laboratories engaged in interstate commerce. Unfortunately, it is not possible to "pool" data

from these experiments in order to establish relative performance measures for various

classes of laboratories. Analysis of the relevant literature shows lack of compatibility

variations in temporal aspects, sample sizes, constituents analyzed (where proficiency

testing was involved) , and a host of other differences which make valid data aggregation or extrapolation impractical.

In order to obtain ccsnparable data on laboratory performance, the Division of Health

Evaluation, UHEW, contracted with the Technical Analysis Division (TAD) of the National

Bureau of Standards (NBS) to develop and implement a comprehensive national survey of clinical laboratories. Laboratories participating in this study were to include laboratories

involved in various licensure/accreditation programs (CLIA '67, JCAH, Medicare), as well as laboratories which do not participate in such programs. Since disguising samples for study

purposes is impractical, proficiency tests were to be administered to each laboratory openly.

Work commenced in January, 1971.

Concurrent with this study program, a complementary analysis was performed of quantitative proficiency test results compiled by the CDC. These data relate to the performance of

roughly 270 clinical laboratories licensed under the Clinical Laboratory Improvement Act of

1967 (CLIA '67) and approximately 400 additional laboratories that participate in this program on a voluntary basis. The results of that study are described in a companion (TAD/NBS)

report entitled "Clinical Laboratory Performance Analysis Using Proficiency Test Statistics."

Some of the results derived from this work affected the data collection and analysis phase

of the national survey reported herein. An additional report which deals with CLIA '67 was prepared by CDC and is entitled "Evaluation Report on the Effectiveness of the Clinical

Laboratories Improvement Program" (July 1, 1972).

Two general caveats on the interpretation of the results of the current study should be

emphasized. The absence of any legal requirement for participation in the study implied a self-selection aspect that negates extrapolating results to the unsampled universe of clinical laboratories. In addition, the relationship between laboratory performance in open

proficiency testing and routine analysis is unknown. Itowever, a laboratory's routine level

of performance is unlikely to be superior to the effort it expends in the analysis of known

proficiency test samples.

The major objective of the study was to obtain measures of performance from several

different types or groups of clinical laboratories, and to determine if there are basic

differences in the capabilities of these laboratory groups from the standpoint of analytical

accuracy. Secondary objectives of the study were to compare the precision of clinical

chemistry and hematology determinations by these groups and to compare the accuracy and

precision of the various analytical methods used. Analytical capability of participating

laboratories was measured through the use of proficiency test samples in the areas of

bacteriology, clinical chemistry, and hematology.

2
2. MLT1I0IK3L(X.T

The specific tasks performed to carry out this study included:

(a) Establishment of a Scientific Advisory Committee to act as an advisory body to the

study

(b) Contact with professional groups and their member laboratories to insure widespread

participation in the study

(c) Development of a sample survey design

(d) Production of laboratory specimens through contractual agreement

(e) Design of reporting forms and a computer card layout to facilitate analysis

(f) Mailing of specimens to the laboratories and coordination of the return of laboratory results

(g) Statistical analysis of the data and preparation of a report summarizing the study

results.

2.1. Scientific Advisory Committee

A Scientific Advisory Committee of recognized authorities was formed to assist in the

selection of those proficiency test coirponents which could be regarded as representative of

laboratory analytical capability, and to monitor the progress of the study fi*om a clinical

viewpoint. The members of the Coimittee are listed in App>endix A.

2.1.1. Proficiency Test Components

There was general agreement among the Committee members that the study should concentrate on routine laboratory procedures which arc coimonly performed and for which fairly

well developed analytical procedures exist. It was assumed that a laboratory which is

unable to adequately accomplish routine analyses is unlikely to adequately perform less common or more complicated procedures. Based on this rationale, the following quantitative

procedures were selected for inclusion in the study:

Clinical Chemistry - glucose, urea nitrogen, calcium, total bilirubin, cholesterol,

uric acid, sodium, total protein

Hematology - red cell count, white cell count, hemoglobin, hematocrit, mean

corjjuscular volume.

In the area of microbiology the samples consisted of pure cultures of five rather ordinary

bacteria: Diplococcus pneumoniae. Salmonella oranienberg, Klebsiella pneumoniae, Pseudo- monas aeruginosa, and Streptococcus faecalis. Mach participating laboratory was asked to analyze two shipments of proficiency test samples.

2.2. Tv-pes of Clinical Laboratories Represented

The study encompassed voluntary participation by a sample of the most common types of

clinical laboratories, as evidenced by the following list:

Doctors' Offices - physician members of the American Academy of Family Practice

(AAFP) and the American Society of Internal Medicine (ASIM)

Interstate - laboratories licensed by the Center for Disease Control under

the Clinical Laborator>' Improvement Act of 1967

Large Hospitals - laboratories in hospitals which are accredited by the Joint

Commission on Accreditation of Hospitals

Smaller Hospitals - laboratories in hospitals which are accredited for the Medicare

program by state agencies

Smaller Independents - commercial laboratories which are accredited for the Medicare

program by state agencies.

The next few paragraphs briefly discuss each of the participating laboratory groups. The

anonymity of all participants was assured through the use of code nunters known only to the

3
individual laboratory and the National Bureau of Standards. The manner in which laboratories were solicited for the study apparently played an important part in the decision for or against participation. Since each laboratory was contacted by the NBS study team, WarnerLambert, and at least one other group (JCAIl, ASIM, etc.). some laboratories were uncertain

as to whether they were being asked to cooperate in one, two or three studies.

2.2.1. Interstate Laboratories

One of the major objectives of the study was to compare the performance capability of

laboratories covered by CLIA '67 with laboratories exempt from the Act.

The assistance of the Laboratory Division, CDC, was secured in order to solicit the

voluntary participation of those clinical laboratories which are licensed under CLIA '67 in one or more of the specialties included in this study. Of the 257 laboratories originally

contacted by CDC for the study team, there were 9 declinations and 248 acceptances.

2.2.2. American Academy of Family Physicians

The American Academy of Family Physicians (AAFP) was one of two professional societies

contacted in an effort to bring private physicians' laboratories into the study. Through

the cooperation of Dr. William L. Loterhos, then President of AAFP, the approval of the

Board of Directors, and the efforts of Mr. Arthur T. Smith (Secretary of the Committee on Clinical Investigation, AAFP), a list of 300 research-minded Academy members was made

available.^ Dr. Roger A. Peatee, Chairman of the Committee on Clinical Investigation, sent

letters to the 300 physicians informing them that their names had been given to the study

group and advising them that they might be asked to participate in the testing program.

Subsequently, written contact was made with each physician, asking for an expression

of interest in the study and an indication of the areas of testing (bacteriology, chenistry,

and hematology) in which the physician was currently active. Negative replies were received

from 61 doctors—39 indicated tliat they did little or no laboratory testing in conjunction

with their practice, and 22 said that they preferred not to participate. There were 150

physicians who expressed an interest in participating, and who also indicated which tests they were capable of performing. No replies were received from the remaining 89 doctors.

Table 2.2.1 below summarizes this information.

Table 2.2.1. Physician Participation, AAFP

Number Percent

Total Number of Physicians Sent the AAFP Letter 300 100

Dated 8/18/71 and the TAD/NBS Letter Dated 8/23/71

Total Number of Physicians Declining to Participate 22 7.3

(but were capable)

Total Number of Physicians Unable to Participate 39 13.0

(no lab facilities, no longer in private practice,

retired, ill, etc.)

Total Number of Physicians Not Responding 89 29.7

Total Number of Physicians Volunteering to 150 50.0

Participate

^ Ihe Academy had previously polled its members in order to ascertain the extent of their

interest in participating in research studies. The list given to NBS contained the names

of all physicians who had indicated a pxDsitive response to the 1969 query.

4
2.2.3. American Society of Internal Medicine

The second professional group contacted, the American Society of Internal Medicine,

has a membership of approximately 12,000 physician specialists (internists) many of whom

operate a clinical laboratory in conjunction with their practice. Because of their significant role in the health care delivery system, and the fact that their laboratories are generally exempt from the Federal licensure program, their participation in this study was considered to be essential.

On July 20, 1971, Dr. Otto C. Page, then President of the ASIM, wrote to each of 496

physicians who had previously expressed an interest in having their laboratory evaluated.

Dr. Page's letter requested an expression of continued interest. Replies were received from

306—19 stated that they did not want to be in the study, an additional 23 reported that they

did not have a laboratory in their practice, and 264 agreed to participate. All conriunication was between the Society and the physician members. The TAD study group did not contact

any of the ASIM participants except in connection with the data reporting or to respond to questions. Table 2.2.2 below surmarizes the numbers of contacts, responses, etc.

Table 2.2.2. Physician Participation, ASIM

Number Percent

Total Number of Internists Written to by Dr. Page 496 100 on 7/20/71

Total Number of Internists Declining (but were 19 3.8

capable)

Total Number of Internists Unable to Participate 23 4.6

(no lab, etc.)

Total Number of Internists Not Responding 190 38.2

Total Number of Internists Volunteering to 264 53.4

Participate

2.2.4. Joint Commission on Accreditation of Hospitals

The JCAU was created in 1952 through a joint effort of the American College of Surgeons,

the American College of Physicians, the American Hospital Association, the American Medical

Association, and the Canadian Medical Association. (The latter group discontinued participation in 1959, however, upon the creation of the Canadian Council on Hospital Accreditation.) Basically the JCAH exists to: (1) establish standards for the operation of

hospitals and other health care facilities and services; and (2) conduct survey and

accreditation programs that will encourage members of the health professions, hospitals

and other health care facilities and services to voluntarily:

(a) Promote high quality of care in all aspects in order to give patients the optimal

benefits that medical science has to offer;

(b) Apply certain basic principles of physical plant safety and maintenance, and of

organization and administration of function for efficient care of the patient; and

(c) Maintain the essential services in the facilities through coordinated effort of

the organized staffs and the governing bodies of the facilities.

Through the supp>ort of the JCAH, the study was able to include more than 200 hospital

clinical laboratories selected strictly randomly from all JCAH hospitals. This was accomplished through the use of a table of random numbers, after assigning a unique number to each

accredited hospital. After 222 hospitals were selected, each was sent a letter frcmi the

Commission, advising them that participation in the study was considered mandatory as part

of the accreditation program. Seven hospitals received JCAH approval to drop out of the

study for various reasons, leaving 215 hospital laboratories in this group.

5
2.2.5. Medicare Certified Hospitals and Independent Laboratories

Most of the Nation's hospitals and independent clinical laboratories are providers of

Medicare services under Title XVIII, Health Insurance for the Aged. These same laboratories

are approved providers of Medicaid services. It was considered extremely important that the

study include a sample of these laboratories in order to permit conparisons between laboratory groups involved in different Federal certification programs.

Personnel in the Community Health Service ("Dr. Sheriden Weinstein and Mrs. Barbara

Nagel) and the Division of State Operations (DSO) , Ikireau of Health Insurance (BHI)/Social

Security Administration (SSA) (Messrs. Brown, Burk and Byers) were instrumental in securing

the participation of several hundred Medicare certified hospitals and independent laboratories. Through the ccmibined efforts of these individuals and the TAD study team, a letter

was issued on July 23, 1971, over the signature of the Director, BHI , to every BHI/SSA

regional office. The regional offices contacted the state health departments who, in turn,

contacted the individual hospitals and independent laboratories. The manner in which the

study was described to the laboratory by the state agency apparently played an important role

in the laboratory's decision on whether to participate.

The names and addresses of 300 hospitals and 200 independent laboratories (randomly

selected from those who were not JCAH members), broken down by state, were supplied to DSO/BHI/SSA for transmittal to the regional offices. Of the 300 hospitals, positive replies were received from 236; 153 of the 200 independent laboratories contacted said that they

would participate in the study.

2.2.6. Reference Laboratories

To obtain a check on the accuracy of information supplied by the manufacturer of the

test sanples, the participation of 19 reference laboratories was solicited. These laboratories were selected so as to be representative of the "best" possible state-of-the-art.

Eighteen of the laboratories (see Appendix B) elected to participate in the study. Of

these, 16 laboratories normally performed clinical chemistry determinations (one of these

labs did not do cholesterol), 15 performed hematology, and 16 performed microbiology.

2.3. Selection of Contractor for Sample Materials

At the very early discussion stages of the study, consideration was given to the

possibility of having the Center for Disease Control prepare the sample materials. The

major reason for not pursuing this course of action was that CDC was heavily conmitted to preparing samples for another program and could not redirect their resources toward this

effort during the established time frame.

Accordingly a contract was awarded to the General Diagnostic Division of WarnerLambert (through competitive bid) to prepare and mail all samples, specimens and cultures

used in the testing program.

2.4. Specifications for Sample Materials

2.4.1. Establishing Target Values for Clinical Chemistry

It was possible to stipulate desired levels for seven of the eight constituents in

clinical chemistry^ because of Warner -Lambert's unique manufacturing process. In this

^A desired level of cholesterol could not be specified in advance because it is functionally

related to other blood constituents, some of \^^ich are removed in the dialysis and ion

exchange procedure.

6
process, many of the normally occurring constituents are removed from the blood through an ion exchange and dialysis procedure until measurable amounts can no longer be recovered.

Known amounts of the desired constituents are then weighed and remixed ("weighed back") in

the blood serum to arrive at specified levels.^ Unfortunately, the process also removes naturally occurring reducing substances which are not replaced. Consequently, comparisons

of laboratory methods based on performance in analyzing these study samples may not accurately

reflect differences in performance in analyzing normal patient samples.

The first column in Table 2.4.1 shows the constituent values requested by the Advisory

Committee, and the second column shows the actual amounts weighed back by Warner-Lambert.

These latter values are alleged to be exact measures of the contents of the test sample

material (nonhomogeneity in the material results in some vial-to-vial variability, but this

variability never exceeds .SI). The close agreement between requested and actual values is

evident. I'he third column, headed MRA, is the Mean Reference Assay—the arithmetic mean

assay reported by the reference laboratories. Thirty reference laboratory assays are included in these means. One of the reference laboratory determinations for normal glucose

and one for abnormal uric acid were reported as more than seven standard deviations from

their respective means and were therefore discarded as outliers. Only 4 of the 14 Warner-Lambert weigh-back determinations (both levels of calcium and bilirubin) differed from

the MRAs by as much as 1%, and no determinations differed by more than 10%. Warner-Lambert'

average cholesterol assay at the normal level seriously differs from the mean reference

assay. A reference technique for determining exact constituent concentration was available

only for calcium. Because calcium sometimes binds to glass and the calcium MRAs and weigh

back values differed fairly widely, the mean reference assay by the reference technique

(MRART) was examined. The MRART values were 9.0 and 11.7 mg/100 ml.

Table 2.4.1. Specification of Constituent Levels, Chemistry

Constituent

Normal Abnormal

Requested Actual mk Requested Actual MRA

Glucose (mg/100 ml) 84 82.2 83.4 307 304 301

Sodium (mEq/1) 143 141 141 118 117 116

Urea Nitrogen (mg/100 ml) 19 18.6 18.6 66 65.4 65.7

Bilirubin (mg/100 ml) 0.9 0.9 1.0 4.7 4.7 4.5

Uric Acid (mg/100 ml) 6.3 6.2 6.2 11.1 11.0 11.

Calcium (mg/100 ml) 9.2 9.2 8.7 12.2 12.2 11.5

Total Protein (g/100 ml) 6.8 6.8 6.9 3.3 3.3 3.4

Cholesterol (mg/100 ml)"* 160 133 70 74

In this study, laboratory performance was judged by the ability to reproduce the actual

specimen contents as displayed in Table 2.4.1. These actual values were defined as the

target values for clinical chemistry. Analysis presented in Section 3.7 was used to justify

the use of mean reference assays as target values for cholesterol. In analyses where minor

differences in the mean might prove important, calcium was assessed in terms of both the

weigh-back and MRART values.

^A more complete description of the process is available upon request to the authors.

'*A desired level of cholesterol could not be specified in advance because it is functionally

related to other blood constituents, some of which are removed in the dialysis and ion

exchange procedure. The "actual" cholesterol values shown are the average Warner-Lambert

assays since weigh-back values do not exist.

7
In the first shipment, dialyzed specimens were sent to all 900 laboratories participating in the clinical chemistry portion of the study. In the second shipment, 200 laboratories, randomly selected frcsn the 900, were sent specimens prepared from serum pool batches

(validate specimens), while the other 700 were again sent dialyzed sample material. This

substitution was designed to provide insight into the method conparison problem. The assay

values for these specimens, as experimentally determined by Warner- Lambert , and the mean of

the 200 laboratory analyses are shown in Table 2.4.2.

Table 2.4.2. Specification of Constituent Levels, Validate Chemistry

Normal Abnormal

W-L Assay Mean Analysis W-L Assay Mean Analysis

Glucose 88.0 80.6 250 241

Sodium 134 135 130 129

Urea Nitrogen 10.6 10.6 51.2 49.7

Bilirubin .5 .5 5.3 4.8

Uric Acid 5.5 5.4 9.9 9.2

Calcium 8.4 8.3 12.1 12.3

Total Protein 6.3 6.1 5.3 5.2

Cholesterol 174 172 183 167

1 It is apparent that two of the Warner-Lambert average assays (normal glucose, abnormal

cholesterol) differ substantially from the mean assays. Consequently, a choice of proper

serum pool "target values" is prerequisite to an evaluation of differences in performance on dialyzed and serum pool specimens. This topic is discussed at length in Appendix C.

2.4.2. Desired Levels of Hematology Samples

The hematology samples, prepared by Charles Pfizer S Co., Inc. under contract to WarnerLambert, were the most critical materials with regard to shelf life. According to WarnerLambert, the lyophilized chemistry and bacteriology samples were stable for at least two years; in contrast, Pfizer recommends a shelf life of only 21 days for their comnercially

prepared blood standards. As a result, it was necessary to prepare separate hematology

batches for the two shipments.

One of the reference laboratories agreed to monitor the stability of the hanatology

samples used in the study. Manual and automated determinations were made on each working

day for a 30-day period starting on the day that each batch of samples was mailed. During

this period, each batch remained stable. Previous studies^ have reported that mailing does

not affect the stability of hematology samples, and it seemed unnecessary to reverify this

result.

Pfizer did not make special hematology batches for this study; the samples \^^ich they

supplied to Warner-Lambert were part of their routine commercially prepared batches. As a result, the Advisory Committee could not control the choice of normal and abnormal levels

used. This was particularly unfortunate in the case of white cell count since the Committee

would have preferred an abnormally low value to the abnormally high value in the prepared

batches.

^See, e.g., I. Schoen, G. Thomas, and S. Lange, "["he Quality of Performance in Physician's

Office Laboratories," American Journal of Clinical Pathology , Vol. 55, ^Jo. 2, 2/71, pp. 163-170.

8
Table 2.4.3 contains values for each constituent in each hematology batch, as reported

by Pfizer and the reference laboratories (MRA) . Within each cell the top line refers to the

Coulter Model S assay and the bottcmi line is for all other methods. At least five assays were used in computing each NRA.

Pfizer had reported separate Coulter S targets because of the occasional presence of

substances in their artificially prepared samples which do not occur naturally and have

different effects on Coulter S measurements than on others. The Advisory Committee suggested

that it would be preferable to use a single set of targets if such problems were uncomnon in

the serum batches. The frequency of bias therefore had to be checked before proceeding with

analysis of the study results. The Pfizer reconmendations and the reference laboratory

results provided sufficient data for resolution of this question. Because of the small

reference laboratory sample sizes, a non-parametric test was constructed. First, the difference between each pair of Coulter S and non-Coulter S values was determined for the Pfizer

data and the Nf^. Under the hypothesis that Coulter S determinations are unbiased, for

any constituent in any batch at any level, the probability is one-half that the sign of the

difference between Pfizer' s assays using Coulter S and other methods will match the sign of

the difference in MRA assays. In actuality 10 of the 19 differences agree in sign.^ It is

thus possible to state that bias occurred rarely, if at all, for specimen analyses performed

in this study, and it is not necessary to use separate targets for the Coulter S determinations. However, it is important to note any analytic results which arise strictly as a result of using a single set of hematology target values.

2.4.3. Selection of Organisms for Microbiology

The bacteria were selected jointly by the Advisory Committee and the Microbiology

Department of Warner- Lajnbert. This was done so that the samples would be representative of

organisms routinely encountered in clinical laboratories and to insure that Warner- Lambert

had prior experience in isolating and growing pure cultures of the chosen bacterium.

Another very real concern on the part of Warner-Lambert was that the organisms should not be

too pathogenic in case a vial accidently burst or was dropped.'

A very brief description of the "source" of each vial of freeze-dried organisms was

included in the package insert. Table 2.4.4 below shows the organism and its vial number

and simulated source in the two shipnents. The p)ermutation of the organisms was intended

to identify any laboratories which, upon noting the same "source" for the same vial on the

second shipment, concluded that the organism contained therein was the same as they had

previously identified. Care was taken to assure that each organism was conmonly found in

both of the sources ascribed to it. There were only two laboratories which misidentified

all five vials in the second shipment by reporting them as they were in the first shipment.

Table 2.4.4. Similated Source of Organisms

Vial Organism Organism

Number Source Shipment 1 Shipment 2

1 Sputum, 46 yr. old male Diplococcus pneumoniae Klebsiella pneumoniae

2 Stool, 9 yr. old girl Salmonella oranienberg Streptococcus faecalis

3 Sputum, 60 yr. old female IQebsiella pneumoniae Pseudomonas aeruginosa

4 Blood, 22 yr. old male Pseudomonas aeruginosa Diplococcus pneumoniae

5 Urine, 25 yr. old female Streptococcus faecalis Salmonella oranienberg

^One data point was discarded because the difference between MRA values was unsigned (0)

'a description of the manufacturing procedure for the sample cultures is available upon

request to the authors.

9
u

•ft a.

CT) 00

00 un .—I o

O) o

.-H 00

CT) 00

to o

a] C7-!

00 CTl

o 00 c

i

00 r-.

>—I 00

5

0,1

00 o ,—1 <^J

r J O i-H

00

fO oo oc

00 cr> 00 CTl

a,

to 00 to

I—( 1—I to «3-

to rj oo 00 o to

CTl O

u

PQ (Li

.—I I—t to "-T

rj to o

t-~. to

to T

1^- \0

LO

r^j to

00 00

00 Oi

LO LO

f—

t—

o

X)

t- 3 ^ o 0; 1- x: !- s- o +-> Hi (-J (LI 4-J

4-' o 1

4-J o •(-J o

i 3 t—

O O O < u <

LO

o

c

5:3

g

s

0 0 '-_) c ^ 4-1 x: '71 0 i u

0

oa i- 2Q

eg o>°

:8 Si ^

0

o

o

o

ii x:

g

(Js

f—(3 U

U -H

1/1 ^3 3

o

10
2.5. Execution of the Proficiency Test Program

2.5.1. Shipment of Samples

Approximately 7% of the laboratories that originally agreed to participate in the

study did not carry through with their earlier declaration. Samples were therefore mailed

to 1,195 laboratories, 87 less than the number originally agreeing to be in the study.

Table 2.5.1 summarizes these dropouts by laboratory type. The majority of the dropouts were AAFP members.

Table 2.5.1. Number and Percent of Dropouts

Lab Type

AAFP

ASIM

Interstate

JCAIl

Medicare

Hospitals

Medicare

Independent

Total

Number Originally

Agreeing

150

264

257

222

236

153

1282

Number Dropjjed

Out

n

49

13

9

7

4

5

87

32.7

4.9

3.5

3.2

1.7

3.3

6.8

Shipments

f«lailed to

n

101

251

248

215

232

148

1195

67.3

95.1

96.5

96.8

98.3

96.7

93.2

2.5.2. Response

Of the 1,195 laboratories to which samples were sent, some did not respond. Table

2.5.2 contains the number of laboratories in each group which received shipments and re- turned analytical results for neither, one, or both of the shipments (as of 5/25/72).

Table 2.5.2. Partial and Total Participation, by Laboratory Group

Lab Type

Shipments

Mailed to No Response

n %

AAFP 101 58 57.4

ASIM 251 44 17.5

Interstate 248 17 6.8

One (First)

Shipment Analyzed

n

17

43

23

16.8

17.1

9.3

Both

Shipanents Analyzed

n

26 25.7 ( 8.7)1

164 65.3 (33.2)

208 83.9 (80.9)

( ) figures are percent of original list.

11
Table 2.5.2. Partial and Total Participation, by Laboratory Group (continued)

Lab Type

Shipments

Mailed to

JCAII

Medicare

Hospitals

Medicare

Independent

Total

215

232

148

1195

hfo Response

n J_

14 6.5

31 13.4

18

182

12.

15.2

One (First) ! Both

Shipment Analyzed j Shipanents Analyzed

n

24

30

24

161

11.2

12.9

16.2

13.5

177 82.3 (79.7)

171 73.7 (57.0)

106

85:

71.6 (53.0)

71.3 (48.0)

Fifteen percent of the laboratories did not return results for either shipment. Approximately another 14% of the laboratories analyzed and returned the report forms for only one of the two shipments. Consequently, only 71% (852 out of 1195) of the laboratories

analyzed and completed the rejxjrt forms for both shipments. Sample sizes were sufficient

to permit statistically significant method comparisons in clinical chemistry and hematology

for all groups except the AAFP. Fortunately, statistical tests showed that the AAFP and

ASIM pooled method data did not differ significantly, and it was possible to retain all

data by creating a pooled Doctor's Office group. Not every laboratory had the capability

to perform all of the assays routinely; consequently, the number of laboratory results

reported will change frcmi one constituent to another. Table 2.5.3 shows the number of

usable analysis reports returned.

Table 2.5.3. Number of Analyses by Constituent

Dialyzed Chemistry Serum Pool Chemistry

Glucose 1292 174

Sodium 887 144

Urea Nitrogen 1256 172

Bilirubin 1163 176

Uric Acid 1433 199

Calcium 969 164

Total Protein 1136 175

Cholesterol 1284 180

Hematology Hematology

Shipment 1 Shipment 2

Hemoglobin 864 784

Hematocrit 847 770

RBC 790 717

WBC 866 789

MCV 731 653

12
2.5.3. Follow-Up to Non- Respondents

On April 27, 1972, a follow-up letter was sent from tlie National Bureau of Standards

to all 475 laboratories which had not returned one or both of the two report forms.

Because of the large number of laboratories from whom little or nothing liad been heard it was interesting to tally their responses to our follow-up letter. As of May 31, 1972, more

than four weeks after the letter was mailed, no reply had been received from 275 of the

laboratories to whom it had been sent. The majority of the responses which were received

fell largely into the following five categories:

(aj Failed to receive either one or both shipments - 44;

(b) Samples were broken upon arrival - 11;

(c) Time and personnel not available - 64;

(d) Insufficient knowledge about study - 3^;

(e) All laboratory work sent out - 24.

Some other problems which arose were:

(a) (tospital administrators agreed to participate but did not tell the laboratory

personnel—consequently when the unexpected samples arrived they were discarded;

(b) Physicians in private practice as well as hospital personnel received a letter

from Warner-Lambert two weeks prior to receiving the first shipment, but never

opened the envelope because it looked like "junk mail";

(c) A consulting pathologist told a hospital not to participate because the laboratory

was not very good;

(d) Laboratories discarded the second shipment because they did not remember that there

would be two shipments.

It is reasonable to expect that direct contact with the laboratories would have minimized

these problems.

2.5.4. Other Problems

A great deal of time was spent reviewing the inccmiing laborator>' report forms. Almost

all of the problems were associated with laboratories which have little or no routine contact

with data reporting forms Ce.g., private physicians offices and small, rural hospitals which

are not normally engaged in proficiency testing programs). Quite a few laboratories did not

honor the decimal point when entering their results on the form; hence, order of magnitude

errors were not uncommon.

Other problems which arose were:

(aj Laboratories whose submitted report forms were incomplete, and whose telephone

numbers were not listed in their local telephone director)', making it impossible

to complete the form and include their data;

(b) Laboratories returned the report form intended for their files fbecause it did

not contain their name and address on it) and kept the copy they were supposed to return (which had the only means of identifying the source of the data)

(c) Physicians and laboratories not sending the report forms back to NBS in the return

envelope provided—and addressing their return envelopes to GX, JCAH, state health

departments , etc .

(d) Physicians and laboratories not observing the deadlines for sample analyses and submission of reports.

2.5.5. Treatment of (Xitliers

Every possible effort was expended toward identifying the rational basis for roughly 500

reported analyses which, upon first glance, appeared to be totally unrealistic. Through a series of letters and telephone calls almost 200 of these aberrant values were resolved

dilution errors, inversion of normal and abnormal results, arithmetical mistakes, etc. Table 2.5.5.1 shows the distribution by constituent of the 315 determinations which were

categorized as obviously deviant and were eliminated from the data base.

13
14
After removing the obvious deviants, it was necessary to determine whether the data

base still contained any outliers; i.e., values so extreme tJiat they should not be included

in further data analysis. Data was excluded which did not meet the criteria of plus and

minus four (4) standard deviations from the mean of tlx? determinations by the given analytical technique. Applying the rule of + 4 standard deviations eliminated data from the various

laboratory groups as summarized in the Tables 2.5.5.2 through 2.5.5.4. The tables are interpreted as follows: each table entry gives the number of results which were more than

+ 4 standard deviations away from the target value for the indicated constituent and the

Total number of results reported for the constituent. In all there were 273 analyses that were classified as outliers and not included in <'my subsequent treatment of the data.

Table 2.5.5.5 summarizes the overall percentage reductions by constituent and laboratory

group. In all, 1.58% of the data were discarded. It was iimediately obvious that, on a laboratory group basis, the largest frequency of outliers belonged to the private physicians'

laboratories. On a percentage basis, the federally- licensed laboratories had the smallest

number of extreme values rejected; the Medicare Independents were next, followed by the

Joint Commission accredited laboratories and smaller hospitals, in that order.

On a constituent by constituent basis, the largest percentage of deletions were associated with cholesterol, mean corpuscular volume and hematocrit. The constituents with the

smallest percentage of deletions were calcium, hemoglobin, uric acid and total bilirubin.

2.6. Limitations

It must be clearly understood that any results derived from this survey will be limited

by four important considerations:

(a) Because all of the laboratories participated on a purely voluntary basis, no straightforward extrapolation can be made to the larger universe of unsampled

clinical laboratories.

(b) It is probable that the results of this study do not represent routine laboratory

performance for two reasons: (1) a laboratory probably would not volunteer if its

management felt that to do so would be disadvantageous, and (2) the sample materials probably received special attention in many of the smaller laboratories

which were unfamiliar with analyzing proficiency test samples.

(c) The clinical chemistry test specimens were prepared by a dialyzation process

which removes naturally occuring substances. As a result, the accuracy of some methods, as applied to the test specimens, might differ from their accuracy in

analyses of human serum.

(d) The true constituent concentrations of cholesterol and the hematology constituents

could not be exactly determined. For these constituents accuracy was assessed

relative to the MRAs.

3. CLINICAl, CIIBIISTRY

3.1. Definition

3.1.1. Accuracy

The accuracy of a laboratory determination is a measure of its distance from the true

value. Unfortunately, there is no known method for determining true values of most clinical

chemistry and hematology specimens. Therefore, accuracy could only be assessed relative to measured values which were considered to be close to the true values. Initially, the

decision was made to assess relative accuracy as either: (a) the relative percent difference

between the observed analytical result and the target value established during the weigh-back

and dialysis procedure^ for all chemistry constituents except cholesterol; or (b) the

^ [(Obs. Value - Target) Target] x 100 = Relative Accuracy for Qaemistry (except cholesterol).
16
17
o

'J

•i-i -a

(U o 00 00 o o o o o

00 1—

(^1 rj r 1 un r- o

o" o \o

1—

i-H r—

1—1 cr> IT) 1 00 o o o I-H o o

>X3 00 "3' ri tn to I-H 1 r I r 1 o o r 1 f—

00o c

LO

roo r 

LO vO o LO

T LO LO LO

oo LO LO LO o T-H r-t r—

O)

r-i (^J 1—

ri l-O 'J o

LO

tnooLOo

00o

LO

cr,

'130)

+-> a

03

If)

nJ Mho

+-> oJ

^1oM

0}C

•HEa

T3M

!D

r-l

-t3

If) >^

Acou

OO

LO to

ro hO to to to r 1 LO 1^ 1 t

uO LO 1—to

to to to to ri o f—

rj (^1 r-i

LO

I- o

to to to to to

to -t- 1—

o00 o

I-- to o r 1 to to to r J to to to to to o o to o to o LO LO 1—

to to to 1—

to f—

to CO f-H

LO

D LO LO

LO LO LO to r 1 1

^ O o

•_) -ooo ou

"3Oo

r—

aa

11

O

cxoeo o

+-> 3

to

LOo otoo

oo

-rr 1 o +^on o

.. -3

a. ca

LO o o 'I- to to to »t to to to to to \ M to r J

lO ^0 OOLO r 1 r J o r—

to I—

-O LO LO o to to to r J

r-i 1—

to LO rj

00 to 1-H o

to o O oo

•:3a; o

Vol

•fHi- eO u o o u

Mea

•H

LOo

to

rj

to

toOLO

1^o

00

00

LO

I/) >- (/I

1/1

1/5 < 73 o

L|-( OJ O (-J M XI

•H

a>M

u

i-c 1/1 !U • 1-1 Cm

18
Table 2.5.5.5. Total Reduction in the IJata Base

Constituent Nlunber %

Glucose 37/2947 1.26

Urea Nitrogen 43/2883 1.49

Calcium 22/2278 0.97

Bilirubin 32/2692 1.19

Cholesterol 86/2967 2.90

Uric Acid 38/3282 1.16

Sodium 32/2070 1.55

Total Protein 33/2630 1.25

Red Blood Count 44/3020 1.46

IVhite Blood Count 51/3321 1.54

Hemoglobin 38/3300 1.15

Hematocrit 59/3237 1.82

Mean Corpuscular Volume 78/2811 2.77

Lab Group Number %

AAJ-P 32/846 3.78

ASIM 116/5686 2.04

Interstate 110/8775 1.25

JCAM 126/8858 1.42

Medicare fbspital 140/8087 1.73

Medicare Lab 69/5195 1.33

All 593/37446 1.58

^Numerator is number of assays deleted and denominator is total number of

assays reported.

19
relative percent difference between the observed analytical result and the manufacturer's mean assay^^ for cholesterol and the hematology sauries (manufacturer-relative accuracy). However, the Advisory Comnittee felt that this definition was not wholly satisfactory,

particularly with regard to hematology and cholesterol, because it relied on determinations

by a single laboratory. Therefore, the decision was made to also assess accuracy relative

to: (c) the mean reference assay (MRA) ( reference-relative accuracy); and (d) the mean of

all non-outlier laboratory analyses. These further assessments, rejKirted in Section 3.7,

showed that reference- relative accuracy was a more reliable measure tJian manufacturerrelative accuracy.

Regardless of which definition of relative accuracy is considered, the clinical chonistry results obtained will be subject to question because the weigh- back and dialysis

procedure, as used by the manufacturer in preparing the sair^les for this study, "removes naturally occurring substances. Consequently, some analytic methods will not

perform on these samples in the same manner as on normal patient samples. This becomes

particularly problematic when con^arisons of performance by different methods are attempted.

A laboratory technique is statistically acoirate if its mean does not differ significantly, at the 951 confidence level, from the target value.

3.1.2. Precision

The precision of laboratory determinations is a measure of reproducibility or consistency of results. It would be desirable to assess precision within individual laboratories,

but it was not possible to obtain the necessary replicate analyses. Therefore, only interlaboratory precision was assessed in this study.

3.1.3. Acceptability of Laboratory Determinations

The definition of criteria for measuring the acceptability of laboratory results is a topic which has received considerable attention. In 1968, the Subcommittee on Criteria of

Medical Usefulness of the College of American Pathologists developed a set of basic subjective guidelines for evaluation of the medical usefulness of laboratory determinations.^^

These included:

(a) Desirable limits for accuracy and precision must be defined at each level of medical usefulness. Maximal accuracy and precision are necessaiy at levels where

decisions are made regarding diagnosis or treatment.

(b) Accuracy and precision of a degree greater than is clinically useful should not be

required if extra time or expense is thereby necessitated.

(c) Desirable precision should be such that errors induced by the measurement process

do not significantly widen the range of values for the normal population.

(d) Desirable accuracy (of a technique) should be such that the method will create no substantial divergence from generally accepted values for normal and disease states.

(e) Ability to distinguish normal from abnormal values is often more important than the

determination of absolute values.

(fj A less precise analytic technique free of large error may be preferable to a more precise method subject to erratic performance.

Unfortunately, the Subcommittee did not develop quantitative techniques for setting allowable

limits for medical significance; however, other investigators have, and a review of such

developments is appropriate.

^^[(Obs. Value - Mfg. Mean) Mfg. Mean] x 100 = Relative Accuracy for hematology and

cholesterol

^^R. Barnett, Clinical Laboratory Statistics , Little Brown, 1971, pp. 132-137.

20
Early enqsirical atten^Jts^^ to define allowable limits of error in accuracy as a percentage deviation from the mean relied on the fonnula: Allowable limits of error (in %) =

+ (1/4 of the normal range) ^ ^^q^ — (mean of the normal range)

"If the normal range for sodium is taken to be 135-145 mEq/L, then the allowable

limits of error calculated by this formula are + 1.8%. The maximum limits for

any determination, however, were set at +_ 10%, even though in some cases those

calculated by the above formula exceeded this figure."

The normal range, although intended as a measure of human deviation, is compounded with

analytical variability. As a result, allowable limits calculated by this formula may not

have medical significance when analytical variability is large relative to the human

variations.

CLIA '67 defines satisfactory laboratory performance as the ability to achieve values

which fall between the lowest lower limit and the highest upper limit of three superimposed

subsets of empirically-derived limits for the sample under consideration.

These subsets of limits were obtained from data accumjlated from (a) Volunteer

Laboratories, (b) Reference Laboratories, and (c) Clinical Requirements.

Subset No. 1 (Volunteer or Licensed Laboratory Data)

This subset of limits enconpasses the central 951 of all volunteer laboratory results.

Obviously deviant results are not used in establishing the limits.

Subset No. 2 (Reference Laboratory Data)

This subset of limits encompasses all reference laboratory results and therefore

consists of the lowest and highest results. Obviously deviant results are not used

in establishing the limits. The median reference laboratory value (MRLV) for each

sample is used as an estimate of the true concentration value for the desired

constituent.

Subset No. 5 (Clinical Requirements)

This subset of limits is centered on the median reference laboratory value. For

san^jle values in the normal range, the limits enconpass an interval equal to one- half of the normal range of values given in several literature references. The

limits for sample values outside the normal range encong^ass an interval equal to one-half of the value calculated by dividing the normal range of values by the midpoint of this normal range and multiplying this value by the median reference laboratory results.

Two problems exist with limits based on these criteria: (a) they are difficult to interpret; and (b) the use of the volunteer laboratory data subset forces these to be limits

for interlaboratory comparisons of ability rather than for measurement of acceptable quality.

By criteria Subset L, at most 5% of the volunteer laboratories can be performing in an unsatisfactoiy manner.

Recent research directed by the Clinical Pathology Department at the National Institutes

of Health has yielded a theoretical basis for judging the medical acceptability of laboratory

^^See, e.g., D. Tonks, "A Study of the Accuracy and Precision of Clinical Chemistry Determinations in 170 Canadian Laboratories," Clinical Chemistry , Vol. 9, No. 2, 10/63, pp. 217-233.

^^CDC, Proficiency Testing , Clinical Chemistry Summary Analysis Survey I, 2/2/72.

21
precision and for appraising the limits of good laboratory performance as measured against

what can be achieved within the state-of-the-art in clinical chemistry.^'' The basic

hypothesis is that analytic variability in measurement within the normal range should not

exceed one-half of the composite biological variation (personal and group) of the normal

population. In cases where the abnormal target is larger than the normal target, Dr. Young

suggests that the acceptable abnormal variability be computed by multiplying the acceptable

normal variability by the ratio of the abnormal target value to the normal target value.

Suggested maximum allowable variations for normal and abnormal clinical chem^str)' are labeled S^^ and respectively in Table 3.1.3.1. Replicate analyses were run over 30-day

periods in the reference laborator>' at NTH in an attempt to compare the state-of-the-art

analytic capability with these requirements for medical acceptability of results. Ansdytically achieved standard deviations are displayed in Table 3.1.3.1 as S^. As can be seen by

comparing S,^ and Sg, the current state-of-the-art does not provide the necessary capability

to perform medically acceptable determinations for sodium and calcium. This problem is

intensified in a proficiency testing situation where the test specimens are not completely

homogeneous. In the current study, the vial-to-vial coefficient of variation was .5% of the

target values for the normal specimens and .45% for the abnormal. The columns labeled

and display the best normal and abnormal test specimen standard deviations achievable

within the state-of-the-art. These deviations were calculated from the vial-to-vial deviations and the achievable analytic standard deviations.

Table 3.1.3.1.

s

^AN' AA

Glucose 1.5 3.3

• 12.2 1.6 2.0

Sodium 1.4 1.3 1.3 1.6 1.5

BIJN .52 1.7 6.0 .53 .60

Bilirubin Not Available

Uric Acid .10 .39 .69 .10 .11

Calcium .085 .046 .061 .096 .101

Total Protein .13 .13 .13 .13 .13

Cholesterol 9.9 15. 15. 9.9 9.9

For the purposes of this study, acceptable precision of clinical chemistry results is

defined as the larger of medically required precision (Sgj^ or Sg^) or 1.5 times the best

achievable precision for the specimen (S^ or S^) as displayed in Table 3.1.3.1. Acceptable relative accuracy of clinical chemistry results is defined by the range of acceptable

precision around the target value for the test specimen.

3.2. Topics Addressed

Examination of the clinical chemistry data indicated that the following seven topics

could be addressed:

^^D. Young, E. Cotlove, E. Harris, et al . , "Biological and Analytic Components of Variation

in Long Term Studies of Serum Constituents in .Niormal Subjects," Clinical Chemistry , Parts I through III, Vol. 16, No. 12, 1970, pp. 1016-1032, Part IV, Vol. 17, No. 5, 1971, pp. 403-

410.

22
(a) Differences in both precision and relative accuracy among laboratory groups;

(b) Differences in both precision and relative accuracy among laboratory techniques ;2

(c) The acceptability of clinical laboratory determinations;

(d) The accuracy of manufacturer's average assays which are frequently used as target

values in routine laboratory quality control programs;

(e) The feasibility of using the mean reference assays, the mean laboratory assays, or

the mean assays performed with the techniques used by the manufacturer as target

values

(f) The likelihood that results of a proficiency test using dialyzed clinical chemistry

specimens would differ significantly from results of a test using serum pool

specimens

(g) The performance of laboratories in this study in comparison to performance in other

published proficiency tests.

Since analysis of topic (f) proved inconclusive, the relevant discussion has been consigned

to Appendix C.

3.3. Analysis Plan

Both precision and accuracy can vary depending on the type of laboratory performing the

analysis and on the laboratory technique used. Consequently, comparison of techniques must

control for differences in laboratory group, and conparison of groups must control for

differences in laboratoiy technique used. In the clinical chemistry analysis, the two physician groups will be pooled.

To siinplify the ensuing discussion, the technique mean has been defined as the mean

value of determinations by a single technique pooled over all groups. The group mean has

been defined as the mean value of determinations by a single laboratory group pooled over

all techniques. The technique variance and group variance have been defined as the variances

analogous to the technique mean and group mean. The analysis discussed below has been

carried out for each constituent at both the normal and abnormal levels. A plan for evaluation of the differences in techniques is discussed in subsections 3.3.1 and 3.3.2; the same plan will be applied in evaluating the differences in groups.

3.3.1. Relative Accuracy

To analyze whether a group applied a technique in a statistically accurate manner, a t-test was used to compare the group's technique mean with the appropriate target value.

Some techniques were applied much more precisely or by many more laboratories than others.

The t-test results may, therefore, be counter- intuitive. For example, a veiy imprecise

technique might be judged statistically accurate, while a more precise technique which has a mean considerably closer to the target value might be judged statistically inaccurate. A

second measure of relative accuracy, the average percentage difference of the group's normal

and abnormal technique means from the target values, was therefore introduced. Comparative

judgments on the relative accuracy of techniques applied by a group were based mainly on average percentage differences.

A second question addressed was whether the technique means differed significantly from

one another. This question was analyzed using a one-way analysis of variance (ANOVA) . ANOVA

allows evaluation of the significance of differences in technique means based on the distribution of determinations by each group.

^°Laboratory technique, in this report, is considered to be a unique combination of a particular analytical method and the system employed; i.e., the glucose oxidase/^1A-12 is

considered to be different from the glucose oxidase/AutoAnalyzer.

23
3.3.2. Precision

Differences in precision of analyses can be assessed by a comparison of standard deviations using Bartlett's test for the homogeneity of variances. In cases whore Rartlett's test indicates that technique variances cannot be considered homogeneous at tlic 951 confidonco

level, the precision of the techniques has been ranked, with the highest ranking (1) assigned

to the most precise determination and subsequent rankings assigned in order of decreasing

precision. A two-stage procedure was used to formulate the rankings. First, two-way

analysis of variance results were examined to determine whether individual differences in

technique precision were significant. A count was then made of the number of times (over

applications by different groups and applications to normal and abnormal specimens) that

each technique was significantly more precise than other techniques. This relative performance count was used as the basis for the rankings.

3.3.3. Medical Acceptability of Results

The acceptability of the results was determined by using x^-tests to evaluate whether

the achieved group standard deviations significantly exceeded the medically acceptable

precision as defined in subsection 3.1.3. In addition, for each constituent, 251 samples of

the determinations by each group were selected randomly and plotted. The medically acceptable accuracy range was then plotted. The fraction of the determinations which lie outside

the acceptable range provides a reasonable estijnate of the likelihood that a laboratory re- port will not be medically useful.

3.3.4. Target Value Surrogates

The accuracy of manufacturer's average assays as target values can be judged by

comparisons (using t-tests) with the weigh-back target values. Similar comparisons can be

used to assess other possible targets.

3.3.5. Dialyzed Versus Pooled Serum Specimens

Because the laboratories receiving the validate specimens were randomly selected,

pooling of the data over groups is permissible for the analysis of dialyzed versus pooled

serum specimens. If the use of dialyzed specimens did affect the performance of a laboratory technique, then a t-test would confirm whether the percentage bias of the mean with

respect to the target value for the dialyzed specimens differed significantly from the

percentage bias with respect to the target value for the pooled serum specimens.

3.3.6. (Comparison with Other Proficiency Tests

Since different studies use proficiency test samples with different assay values, it

would be impossible to use means and variances as a basis for conparison. Itowever, the

coefficient of variation (the ratio of the standard deviation to the mean) provides a basis

for comparison and has been used in this study.

3.4. Reporting of Results

In this stuly very large quantities of data were collected. Analysis of the data by

laboratory group, sample batch, and analytical technique resulted in over 1,000 calculations

of different means and standard deviations. These values are likely to be useful to many

researchers and proficiency test program administrators, and consequently it seemed desirable

to report this information. If all of the test statistics and intermediate statistical

calculations performed in this study were also reported, there would be an overwhelming

quantity of data. Therefore, the results of the statistical analysis are reported, but

details such as the exact values of test statistics are omitted. All results are significant

at the 9S% confidence level unless otherwise stated.

24
3.5. Precision and Accuracy

One part of each table in this section displays information on both levels (normal and

abnormal) of a single constituent; the information is broken down by laboratory group and

technique. Results were entered only for those laboratory group/ technique combinations

which include 15 or more determinations. For each line item, the sanple size (n) , mean (x)

and standard deviation (s) are shown. For ease of reference, the table includes techniquedependent results pooled over all groups and group -dependent results pooled over all

techniques. The weigh-back values for the normal and abnormal determinations (T) and the

manufacturer's average assays (A) are indicated in parentheses in the table headings. At

the bottom of each column, the relative ranking of the precision of each of the groups is

indicated.

A discussion of differences in the group means has been omitted because the ANOVA

analyses revealed that, at the 951 confidence level, there is no evidence that the technique

means obtained by any laboratory group differed significantly from the technique means of

any other group ; thereby indicating tnat relative accuracy among groups does not differ

significantly at this level . Because means do not differ among groups, it was possible to pool the data on all groups to analyze the difference in technique means. Differences in

technique means could thus be discussed for any technique used for 15 or more laboratory

determinations.

For each constituent, the discussion following the table contains a relative ranking

of the precision of the techniques and an analysis of the relative accuracy of the techniques.

For each constituent an associated table contains the pooled data used to make this analysis.

This table displays, by technique, the means, standard deviations, sample sizes and percentage biases from the target values in cases where targets exist or from mean reference assays

in cases where no targets exist. The percentage difference {% d) is defined as 100 times

the average bias from the target value divided by the target value. A star appears after

those technique means which a t-test indicates are significantly different from the target

values.

Bartlett's test revealed that precision differed by both group and technique for all

constituents. Because it was demonstrated that the groups did not differ significantly in

their overall accuracy, it was possible to judge them based on their relative precision.

However, since laboratory techniques differed in both accuracy and precision, comparisons

among them must consider both descriptors.

3.5.1. Glucose

The data are displayed in Tables 3.5.1.1 and 3.5.1.2. The precision of techniques'^

is ranked:

1 Ferricyanidc fAutoAnalyzer)

1 Copper Neocuproine (SMA 12-60)

2 Glucose Oxidase

2 Folin-Wu

3 o-Toluidine

3 Kits.

Table 3.5.1.2 reveals that only three techniques (Somogyi Nelson, Ferricyanide (AutoAnalyzer) and Folin-Wu) were statistically accurate at both the normal and abnormal levels.

Two techniques (Glucose Oxidase and Copper Neocuproine (SMA 12-60)) were not accurately

applied at either level.

Of the glucose techniques used in this study, Ferricyanide (AutoAnalyzer) appears to be

the most successfully applied. It is interesting to note that this is not one of the techniques which, when correctly applied, are considered to be the "best" glucose techniques.

'^Throughout Section 3.5, techniques are manual or partially autcmated unless otherwise

stated.

25
O .-(

.-. O r-<

—1 o

5 5 ^X o

it?) Ou

u o -« u —

o

(sen II

?6
O

II

00 00 LO OO

r

J LO LD CT) ri 00 o CT^ 00 to r 1 ri to i

00 r-1 00 -T

r 1 LO OO CT. LO

rH 1—11 1 r 1 1 »—

r-i to1 to1 K « *

LO rj vC

r 1 LO LO 1—

to o LO 00 vO O C o CTl r ) r-j o CTi vO cn 00 O cn t—

CT. 1—

a\ o 00 CT1 rsi r 1 r 1 to ri fO r 

to

II

II

X o

.e.oo3

LO

to

E00

II H>

_1

LO CTi LO

r

JO 00 00

to

O r—

to 00 to LO

f—t LO .-H

O CT1 to LO r4

r I "T r-^ 'J- LO

r-^ 00 LO 00 1—

LO \0 00 1—

rH

CT) 1—

LO •o LO 1 o o 00 00 o 00 1—

1—

1 * « « « «

LO -t to to 00 CT1 to

r

J

1—to CTl CT» o to f-H 00 OO OO 00 00 CTl OO 00 00 00

to CT^ LO •—! LO

to

to to

rH VO LO

r

J 00

to

00

00

"3§U

LO

CTl

(30

4)3

cru

4) H

CO

(/> rH

OO

J?

u

Kl rH

to;3

<*;

<u rH t3H 111 T3

r(3 o rl rH u O 3 frH 1 a o coX

<u o o c c c • rl ,— H O 1- o o u

1- 0; -3

&. t-i 3 >. 3 O 3 O U rH U ro 'J O

O

O 1 O 1 II O <^> rH i?: rH o

S5 3 D-<

rHU o o u o o o a

IT3 ouc

<u

<u

<4H

-3C

fcO

I/)

27
3.5.2. Urea Nitrogen

Table 3.5.2.1 contains the data stratified by group and technique. Precision of

techniques is ranked:

1 Diacetyl monoxiJiie (SMA 12-60)

2 Diacetyl monoxime (AutoAnalyzer)

3 Berthelot Reaction

3 Chromatographic

3 Urease Nessler's

4 Diacetyl monoxime (Manual)

4 Kits.

The data displayed in Table 3.5.2.2 indicates that none of the techniques, as applied,

accurately reproduced both the normal and abnormal target values. Two techniques were

statistically accurate, however, at the normal level: Urease Nessler's and Kits. The

Diacetyl monoxime determinations performed with the SMA 12-30 and SMA 12-60 were statistically accurate in the abnormal range. Only the automated Diacetyl monoxime techniques had

average percentage differences (normal and abnormal combined) of less than 5%.

The automated Diacetyl monoxime techniques appear to outperform other techniques as they are currently being used.

3.5.3. Calcium

The relevant data are displayed in Tables 3.5.3.1 and 3.5.3.2. The Doctors' Offices were excluded from the ranking of groups because of insufficient sample size. Precision

of tecliniques is ranked:

1 Cresolphthalein Complexone (SMA 12-60)

2 Cresolphthalein Complexone (AutoAnalyzer)

3 Calcein Fluorimetric

3 Chloranilate precipitate

4 Ijnission flame photometer

5 Kits.

Table 3.5.3.2 shows that, on average, all of the techniques underestimate the calcium

weigh-back value at the abnormal level. Nine of the 11 tecliniques also underestimate the mean reference assay by the reference technique (MR-XRT) . Four techniques yielded statistically accurate normal detemiinations: Oxalate precipitate, Chloranilate precipitate.

Emission flame photometry and Cresolphthalein Complexone (AutoAnalyzer). Averaging the errors at the normal and abnormal levels yields a mean error of less than S% for Oxalate

precipitate, Chloranilate precipitate, Atomic Absorption Spectrometry and all three auto- mated Cresolphthalein techniques.

Sample sizes were insufficient to support statistically valid conclusions about the

precision of the Oxalate precipitate and Atomic Absoi-ption techniques, itowever, it would

appear that Atomic Absoi-ption was applied with a precision roughly equal to that of the

automated Cresolphthalein techniques. Similarly, the precision achieved with the Oxalate

precipitate technique appears to be roughly equivalent to the precision achieved with tlie Chloranilate precipitate technique. Thus, when both accuracy and precision are considered,

it would appear that calcium determinations performed with either Atomic Absorption

Spectrometry or the autonated Cresolphthalein Complexone techniques were the most satisfac-

.tory.

3.5.4. Total Bilirubin

Table 3.5.4.1 presents the data for total bilirubin stratified by technique and group.

Precision of techniques is ranked:

28
§

8 ,x o

ao « —I O

o

o «

00 00

O

00

00 oc

O 00

o r~ O ^0

1/1 i

td

C

2

.-I

o

N

2 £

Kl (NJ

C -1 CO

X CI O w

c >.

-I 5

O 3

u <IS ~-'

1= o

Q O

K K U

o

a 5

29
cn 00 to rr LO

LO o to

r 1 to 1—

1—

rH rH

rH o o LO oo Ol to

rH 1—

1 >—

f-H1 ( 1

UO o r 

o LO LO

ocO Ol LO LO LO LO K * * « « oo

10

1/1 ISO

.5 . 6 o o

to inonc -6o;

<-'l r-H r-4 si

•H

r-l '—I r-H r—4 O

0)u

oa a! o vofi Q O £ Si u

to

.—

vO

11

II

X

f-He C

to

to

00

00

1—

r

J LO

1—

c

f—

SOT'

1246 oo1—

to

O a\ 00 I—

(/) to rj to to

a;bOo

tr to to

r 1 rsi 00

1—4 to 1- ri o to

1—1 1 03 II 1 1 O

t-^

»—

o 1—1

T o LO

O o> cc '*00 cn LO oo

1—1 1—1 (—1 1—

* * •K * K *

LO II to

1—

II

X

(U

1—1 J3

re H LO 1- 1 LO t-- LO O oo "-r o LO

to r-l 1—

rsi 1—

35

:5 :5

muc

0)

•T3 • Hou

cN» LOa

0)3

1—1 rt>

<-> (Uoo

t-1

1)§c

•H

13O

•HH

•rH «

30
a o II

-H o

CO o oc o

3/
r—

vO vO

11 1/1 1—1 1—1 o »—

1—

<—

O o 1-H i—

vO to 1^

ro LO rj to in i 1—1 1—1 r—

1 1 1—

1 1—1

II 00

r - to LO »-r r—

O t—

o o O (—

t—

O o 1 X « > II -J II X

r-HOc in r- o 00 o to C r 1 o o to to oo sD < I—

1—

o

II

t-!

II t1-^ o to

LO to o ^ o ^

.—( I—I r-i

to

00 o

to o o

CO 00 « « oo

too

to

to

<u c u

03 o o

•H § eu <u

0) •H

+-> o

ct) c *-> •M a. o o

•H

o.

J 1 O e-

•t-J o 3 c

(—

o 1—

c c1 o o C c

•rH (J •r-l

(/I

m o o (/I s u u

iH <—t 1—

X 3 »-> ri O < u oco

1) >,

CI. 1—1 S CD O --^

^.

C rj

"-1

C

r-t

O T3 ^ O

«-'—(-' ^ a M

a, 3 S —ICO o eg ^

too

toto

u a> c c c o o o X X

pie <u 1—

o

Com Com Com C I- c c

~< i) >-< dj tj <u

1—1 >. rH

O r—o n i-( rt to

j=; nj x; 1 4-' £^ x; 1—

Q.Oa•-^ ^ 1—

SO Au si so

(i> >—

0)

^1 ;-i u u o

too

00 00 00 «

r-- 00

II

II

X

1^ 00 00 LO o LO OO 1—1 CI o o to to 1—

LO 00

1—

r-H 1—

o

x:

IDuc

<u T3

•Hcou

LO

CTl (/) - 1^ x; a01 4-1 3

r-j 0) n3 o > C

It" ^

<U Of 60 U-i

4-> (U = >^

i/i

M (U

LMC ..

to

U

32

1 Diazo-Other (AutoAnalyzer)

1 Diazo-Other (St-\A 12-60)

2 Diazo-Alcohol

3 Diazo-Other (Manual)

Table 3.5.4.2 indicates that none of the techniques, as applied, were statisticallyaccurate at the normal level. Furthermore, all techniques overestimated the normal bilirubin

content. Only three techniques (Diazo-Other Coupling (Manual), Spectrophotometric , and Kits) were statistically accurate at the abnormal level.

The data indicates that, as presently applied, there are no techniques which are both

relatively precise and statistically accurate. Therefore, a choice between techniques will

require a decision on whether an accurate technique with low precision or a biased, but

precise, technique is preferable. A feasible basis for this choice is that if a reasonably

precise but biased technique is used to redefine the normal range, then determination of the

normality of specimens tested with that technique will be reasonably consistent. Based on

this reasoning Diazo-Other (AutoAnalyzer) and Diazo-Other (SMA 12-60) are the techniques that

are preferred as they are presently utilized.

3.5.5. Cholesterol

Tables 3.5.5.1 and 3.5.5.2 present the data for cholesterol. The precision of the

techniques is ranked:

1 Lieberman-Burchard w/o Extraction (SMA 12)

2 Lieberman-Burchard w/o Extraction (Manual)

2 FeCl3-H2SOL, with Extraction (AutoAnalyzer)

3 Kit with Extraction

4 Kit w/o Extraction

4 Paratoluene-Sulfonic Acid w/o Extraction.

Cholesterol was unique among the chemistries in that there was no weigh-back value

associated with the samples. Consequently, to determine the relative accuracy of the various

techniques it was necessary to use a proxy for this value. In Section 3.7, the mean

reference assay is explained to be the most reasonable target value. Table 3.5.5.2 shows

that the FeCl3-H2SOi+ technique, with prior extraction (Manual) was statistically accurate

at both normal and abrwrmal levels. The FeCl3-H2SOi^ (AutoAnalyzer) technique, with prior

extraction, also proved statistically accurate at the abnormal level and had an average

error of less than 5%. Among the many techniques used to analyze cholesterol samples in this

stixiy, these two appear to be the most successfully applied.

3.5.6. Uric Acid

The data for uric acid are displayed in Tables 3.5.6.1 and 3.5.6.2. The precision of

the techniques is ranked as:

1 Phosphotungstate (AutoAnalyzer)

1 Phosphotungstate (SMA 12)

2 Phosphotungstate (Manual)

2 Phosphotungstate (Kit)

3 Other Kit.

The data in Table 3.5.6.2 indicates that participants in this study who analyzed uric

acid with the Uricase technique achieved statistical accuracy. However, the small sample

size made it impractical to include this method in the rankings of precision. Of the other

techniques used, Kits were statistically accurate at the normal level but were more than 5% inaccurate at the abnormal level; while Phosphotungstate (SMA 12) determinations were

statistically accurate at the abnormal level and less than S% inaccurate at the normal level.

In addition, the Phosphotungstate (AutoAnalyzer) determinations were less than S% inaccurate

at both levels.

34
t/1

1—

oo. oo

C7> 00 to

t~- in rH

to

to

to 00 o o o o o o rH rH o

CO

1—

00 rH to to 00 t to

II 1—

1 rH 1 >

di

00 to 00 a. LT, ro cr, ' X *

un«

LO rr

II r-(

II

X oc

ml) c lo r-(

a-. rH

rj

to 00or 1 rH

LO

SO LO

rH

rH o

1—

i t/l

00r4

CTi rH

00toC\r 1 rH

vD

rH

r-- LO o rH

to

lb

in o o o O o O o o o

4.2.

Total

Bilin o

II H,

f-(

1—1

toto

to

toto

to ^

-atoto

to t LO

LO

LO

—

> O rj

rH«

rjrH«

to

rH«

to

rH

r 1 rH«

to

rH« « r

J rH

II lA 1—

II

X

tn

Table c 1—

rH -r to rHCI

rH

LOooo

LO

rH

rH

>. rH rH

•H • H DfirH rH C

CX-H

a

H .H

i-> 3 rH *-l

O »-> a. i- u 03 3 rs a o cx, rH -H -—

o (-1 . o o ^H J= o -o i- o o O 1 a u rH -M 'J Csl JZ rH

:3

rH rH 2 e O 3 torn o 6 ^ N 3

•H •H •H

CJQ

a

COc c . •H • H 'J rH rH Hu 3" ll § 3 O OJ o o 1 u 

U

>. o o o

r—t to

u o o OJ 01 " <u > x; rsl

-Ot

toA

4-" rH O rop o o ^

+-I

rsl

u

CU

n ^ o H •H •H ex

Ci

T3O

rC

4-1

rH

a; t3• H

<JHcou

«**> to

rH «

03

x:1

Mhc

t|H

t3XrHc

DO

LO

35
O H-1 LO O >/i rv< t-l

Medicare

Labs

n

X

s

97

149.

12.1

33

150.

8.7

20

150.

18.6

194

147.

13.3

96

95.

9.2

34

96.

7.6

20

97.

19.4

194

93.

12.1

en

(T'

»

134, A

=

160)

Medicare

Hospitals

n

X

s

172

152.

15.6

30

149.

26.3

19

149.

8.1

273

150.

18.8

-

74, A

-

70)

172

99.

11.7

30

93.

18.6

19

97.

13.2

273

97.

14.6

^ § .X

a) Q

rHi

c

'

144

150.

10.9

53

149.

10.7

18

124.

11.

Q

17

149.

20.6

300

146.

IS.S

Dialyzed

Specimen

(T'

141

99.

10.1

51

95.

12.8

18

71.

9.7

17

90.

15.3

294

93.

15.2

3.S.5.I.

Cholesterol

,

Interstate

n

X

s

55

151.

11.3

137

149.

8.9

16

127.

18.9

286

147.

12.4

Cholesterol

Abnormal

54

101.

13.0

140

95.

8.9

IS

72.

8.1

287

93.

14.0

Table

Doctors'

Offices

n

X

s

95

149.

14.1

18

159.

17.3

32

148.

18.0

214

150.

16.1

93

97.

12.8

18

100.

18.8

31

91.

15.8

212

95.

16.8

Tecluiique

Liebermann-Burchard

w/o

Extraction

(Manual)

Liebermann-Burchard

w/o

Extraction

{3^

12)

Paratoluene

Sulfonic

w/o

Extraction

FeClj-HzSOu

w/Extr.

(AutaXnalyzer)

Kit

w/o

Extraction

Kit

with

Extraction

All

Methods

Liebeiinann-Burchard

w/o

Fj(t

Taction

(Manual)

Liebermann-Burchard

w/o

Extr.iction

(»1A

12)

Paratoluene

Sulfonic

w/o

Extraction

FeCl3-H2SOw

w/Extr.

(AutoAnalyzer)

Kit

w/o

Extraction

Kit

with

Extraction

All

Methods

•St

36
o «

rj r-H

00 «

i-H 00 vD Ln

1—

1—

1—

LO r~

00 to .—

»—

i~o oo un

LD 00 to

oc en « « « « o 00

to 00 00 o

rO

i—i 1—

1—1—

o

to

LO

t—

o

t—

I-H 1—

oo 00

r—

oo to I-H LO T T to to

LO f—i oo r 1 «

1—

I—i to« O LO

r-J —I

II IlXO

to o fVl rg to LO o(V|

(^g

Orj

LOrHLO

I—I oc p

10

(Uo

i^g LO

LO

too

r3

MO

I-H r J o

to

LO o

(VI LO 00 UO

LO LO (7> 00 LO r-) o 00 00 (VI to 00 (^g LO O

LO LO lO rg (^g LO

f-H 1—

I-H 1—* I-H I-H I-H * « +: « r «

00LO

00

II

II

X

(VI

4) 5c

(h

II

X DO

Du

(U

u

)-.

in a, 3C

S}

-3

X)

Ih

'-I >- ^ o

S'-J. c %

XI < Xo

i^g -D 3J t J <;

I-H 'J

I-H -3 r-H

C 03 c u o 3 o 3 o I I-H 1

ch cn

rac Bur < o -3 H J- en oJ 5 3 (SI 3 I—I Ih O 1 o 1 g-> ro (U

OJ I-H !h

U •H o

1*; 5 —1 -J

1/1 T3O

re

reooc

(U

Ih

(U

l4H(Uu

i-soX

ii'x'

IIXI so

LWc

(Uo

Lm

•rH -3

>.C

37
38
II>oc

to to

vD LO oto

f—

r-i

r—OO l-H l-H I—

rr LO to ri 1

to to

T

c-> a.

r 1 vD

O l-H

O o o O « «

II

II

X

vO to to l-H

rsi

r i to l-H I—

r 1 <-H o

DO 3

vO

LO

to

(L> l-H XI

75

11

00 to

0-, r 1 o or 1 00

00 o O O o l-H o O o O

l-H1 l-H to o o

l-H to -r «

vD *

II

II

K

to

rsi to

l-H l-H M-

"l-H oc

4) T3

•r-l <+-! co

L>

t»f> LO

x:O co

4-1

DCoX

D,

(/) oX o

Hi

O

4-1

4-J 4-> S S oXcoX oX

(/I oX o

x:M co

Uh

rH

30

'J~> «

39
Therefore, it would appear that study participants achieved adequate results using the

Uricase, Phosphotungstate (StAA 12) and Phosphotung state (AutoAnalyzer) techniques.

3.5.7. Sodium

The stratified data is displayed in Table 3.5.7.1. Precision of techniques is ranked:

1 Flame photometer (SMA 6)

2 Flame photometer (Manual ).

Table 3.5.7.2 indicates that all of the techniques except Flame photometer (Manual)

yielded statistically accurate determinations at both the normal and abnormal levels.

Although the sample sizes are too small to make operationally significant statements about

the relative precision of three of the four acceptably accurate techniques, it would appear

that the Flame photometer (AutoAnalyzer) and Flame photometer (SMA 12) techniques were

applied with a precision roughly equivalent to that of the Flame photometer (StiA 6) , while

the kits were applied considerably less precisely. Thus, it appears that the automated Flame

photometer techniques were applied most successfully by participants in this study.

: 3.5.8. Total Protein

Based upon the data displayed in Table 3.5.8.1, the precision of the techniques is

ranked:

1 Biuret (AutoAnalyzer)

1 Biuret (SMA 12)

1 Refractometer

2 Biuret (Manual).

Examination of Table 3.5.8.2 reveals that Biuret (Manual) and two types of kits were

applied with statistically accurate results at the normal level. None of the techniques were

statistically accurate at the abnormal level. Consideration of the average percentage

inaccuracy reveals that all techniques, except the kits, achieved less than a 5% average

inaccuracy

It appears that the Refractometer, Biuret (AutoAnalyzer), and Biuret (SMA 12) techniques were all adequately applied in analyzing total protein.

3.5.9. Summary of Groiqj and Technique Performance

There were no significant differences in the relative accuracy of constituent analyses

by the different laboratory groups. Table 3.5.9.1 summarizes the ranked precision of the

groups. In order to facilitate analysis, tied ranks have been assigned an average ranking;

for example, if two analyses were tied as the most precise, they would both be ranked 1.5.

the average of 1 and 2. Application of Friedman's Two-Way Analysis of Variance by Ranks^^

revealed that the groups differ significantly in precision (at the 991 confidence level).

^^For further details on this test, see R. G. Miller, Jr., Simultaneous Statistical Inference , McGraw-Hill, 1966, pp. 171-173.

40

l-H ^o r 00 00

LO to ri (^) I—

to

too

to

to

o

(-) LO

to a. r—

00 to 1—

cc LO 00

II

1 X

00

II bl

r 1 to o

O

o

o

LO

LO 'a- LO i—t to > o o o O a! o

rr -T •-r to

_1

1—1 «

I—II

II X

en to r-j o

c to to LO 00

u

c

r-l 'ao

'J

CT3

(LI

3

0;So

o

"5.

o

0)o

6o

i-> o

3

0)

5

o

a.

1

Ooa

O

i-H 5

eo

c

u

a» lm

4-4 •H-o

u• l-l

•iH&• -I CO

42
a.3

Oo

,-4 rsj I ^ r4 ^

43
II oc

J2

00 00 o OC hO

1—

1—1r I rr LO o o O O o o o o r 1 o r 1 LO

LO LO

r 1 t ~ 00 LO

r'-. hO « •K «

II

hO ,C OC rr •—( ''I '-~

II H

LOr 1 to ZZ 00

r~- OS

to

to o O o o o o O o LO LO LO o o f-H f-H o r 1 ouc

t3• Hou o

00 o CO O « o

II

II

X

LO

CTl r-4 to

0)OO

}h

c3

(U

4-1 eo

(-. c

•r-( CQ

d

nu

a: o O

4i

a

0)^.

•H

44
Table 3.5.9.1. Ranking of Precision of Groups by Constituent

Doctors

Offices Interstate .ICAU flospitals

Medi c Lab

Glucose 5 4 3 2 1

Urea Nitrogen S 2.5 2.5 4 1

Calcium 2.5 2.5 4 1

Bilirubin 4.5 2.5 2.5 4.5 1

Uric Acid 2.5 2.5 4.5 4.5 1

Cholesterol 4 2 2 5 2

Sodium 4 2.5 2.5 5 1

Total Protein 4 1.5 4 4 1.5

The simultaneous confidence interval for multiple comparisons based on the Friedman

test (as given by Miller) was adjusted using the correction factor given by Gibbons. This

adjustment was necessary because ties are present in the data. Applications of the adjusted

multiple conparison test to the mean ranks when calcium was and was not included in the

analysis yielded identical conclusions, all of which are significant at the 90% confidence

level:

(a) The precision of the Independent Medicare laboratories was significantly better

than the precision of any other group except the Interstate Hospital group.

(b) The JCAH, AAFP-ASIM, Interstate and Nledicare Itospital groups did not differ

significantly in precision.

In summary, the precision of the different groups of laboratories can be exhibited as follows, where groups joined by the same line are not significantly different from each

other, at the 901 confidence level.

Rank Order

Most Precise

Least Precise

Laboratory Group

I Medicare Laboratories

I Interstate

JCAII AAFP/ASIM

Medicare Hospitals

Table 3.5.9.2 displays the referee analytical methods suggested by a member of the

Advisory Committee, the percentage of study participants who employed these methods, the

techniques which the study participants appeared to have applied most successfully, and the

percentage of the study participants who used these techniques. In examining this table,

it is inportant to recall that these conclusions are based on performance in analyzing

dialyzed serum samples and may not be directly transferable to human serum analyses.

Examination of Table 3.5.9.2 indicates that, for six of the eight constituents, at least one of the best applied techniques was based on the suggested referee method; very few

^^J. Gibbons, Non- Parametric Statistical Inference, McGraw-Hill, 1971, pp. 256-257.

45
C

r 1 O ^ 8U

E-

<1> I<

T3

reQ

+-> 3

0) <ucoX o Q>

1—1 1 c u o—< c X »->

no e-^ o re

I/)

x

1—

4-)

O <-! o •H O u e i« o <u rH J t- O < u

reCo

v<o

x:

tj oo

t)

re Q

4-" j= re 5 4^

<N

O

X 3< O O

re PL

rH

O 3 c-

•4-> §oc

r3 o o

T3O

x;

0)

S-i

1/) dou o

rec

•i-t o o c ^^ o 3 UO^^ t/1 co O re u; o -o I—

1) c

re o 0) :§

0)to

reu

•H

Z3

re o

*->

1/) o

r-l o

x:u

tj <

:3 o

46
laboratories used the suggested referee methods in analyzing the other two constituents.

For three of the constituents (urea nitrogen, calcium and total protein) more than 30% of

the analyses were performed with the best applied techniques. Conversely, less than 15% of

the cholesterol, glucose, and sodium analyses were p)erformed with the techniques which were

most effectively applied.

Assessment of the results reported in Subsections 3.5.1 through 3.5.8 reveals that

automated techniques, as applied by the laboratories in this study, yielded equal or better

accuracy and considerably superior precision than manual techniques employing the same analytical methods. In contrast, the various techniques comnonly referred to as diagnostic

kits were consistently the most imprecisely applied.

3.6. Medical Acceptability and Usefulness of Results

In Section 3.5, the precision and accuracy of the reported analyses were assessed. In this section, the relationship between the reported analyses and medical utility will be

treated. For each constituent, discussion will focus on the acceptability of the group

precision and the likelihood that determinations made by members of the group will be

medically useful. The criterion chosen for measuring medical utility is that interlaboratory

precision and accuracy should be within the limits necessary to support the monitoring over

time of the variation in an individual's constituent concentrations. Such monitoring is

important in therapeutic medicine as a means of checking on the progress of treatment and in

preventive medicine as a means of early detection of medically alarming changes in a person's

physical and functional condition. An intralaboratory monitoring capability is insufficient

Decause people move from one area to another area, physicians sometimes switch laboratories

or split their work between laboratories, and patients sometimes switch physicians.

3.6.1. Acceptability of Laboratory Group Precision •

For each of the constituents except total bilirubin. Table 3.6.1.1 displays the

acceptable precision (Sg) as defined in Subsection 3.1.3^"* and the sample size and standard

deviation for:

the reference laboratory determinations,

each laboratory group, and

the best applied technique with the smallest standard deviation.

Total bilirubin has been emitted from this analysis because acceptable limits for this

constituent were not available. Standard deviations which were not significantly greater

than the acceptable precision values at the 99% confidence level have been starred in the

table.

Examination of the table indicates the following:

(a) Precision of the reference laboratory analyses was within the medically acceptable

range for five of the seven constituents—urea nitrogen, cholesterol, uric acid,

sodium, and total protein.

(b) Precision of the cholesterol determinations was, in general, medically acceptable

for all of the laboratory groups.

(c) The Interstate laboratory group achieved medically acceptable precision for urea nitrogen and sodium at the normal level, and total protein at the abnormal level.

(d) All other group precisions exceeded the medically acceptable precision values.

(e) Those laboratories using the best applied technique achieved acceptable precision

for urea nitrogen, cholesterol, uric acid, sodium, and total protein. Thus, it

is possible to state that a better selection of techniques would probably increase

the acceptability of interlaboratory precision.

2'*Some of the acceptable precision values contained more significant digits than were desired

for the analysis; these values were rounded upwards.

47
o o o ao CM rg o o

to K) O to 00

TT i~n t/>^ OO • r^fsj Or^ roro

I

IOO tn^J "T^T cDLTJ <—40 fsj—

•—»>—< OO u-iLTi Ovc?% )— r^fsj r^r3ain

rsioo rsjr- r^r- oo sO O ^

f-t 1-1 »0 00 O o

0\ <^ ^0 l/l 00 0% 00 OO

r-4 rs| rsi rsi rsj rg

OO

r~ in

tn fo m

rg OO ^ ^ m rj

LO in o o

00 CTl O

f-l <Jl 00 rJ

rg o

00 00 O O

0«T I~-vO r^fM O>00 OO* ^ ^ 00 00 OO Klrg rOtO rsirsi KlfO o o o

o .-< O O

O

r^j r-j

LTl

rsi rsi r- OO GO O O to rj (7^ CTi r^jr4 fNirsi »orO rarg rgrsi

ro f-H 00

r-s ^ sO sO

rsi in vO 00 —1-4 Kl »0 O o

in OO rg rg rsi (^jro oin OOt

tntn f—If—I OO* i-orsi rg rsi rsi

sO Ot O O

ro *r OO r-~- O O r-J ^

rsi rj O O

rO rs| ro rsj

1^ O

d d

to fM rg

rg fsi d d

is

55

- 2J 5l 5:§ 55

c

o

o 3 00

4«
The conclusions just cited are open to question on two grounds. First, they are based on precision values calculated about the group means. It would be f)ossible to argue that

the precision values were inflated by the procedure of pooling over techniques. It is important to investigate whether the conclusions would have changed if techniques had been

differentiated in the precision calculations. Secondly, the conclusions are based on performance in analyzing dialyzed specimens. It would be desirable to compare the precision

values in this study with values from a survey which employed serum pool specimens. These

further assessments should yield insight into the validity and transferability of the

conclusions

Table 3.6.1.2 displays data which can be used to make these further assessments. These

data come from two sources, the present study (labeled MBS in the table) and a document

entitled The 1971 Chemistr>^ Survey Program—An Analysis of the Data by Dr. Roger Gilbert

which was published by the College of American Pathologists and is based on their 1971

survey data (labeled CAP in the table). It should be noted that the CAP program is the

only proficiency program which the CDC certifies as equivalent to its own. The CAP report

gives coefficients of variation which, when multiplied by the average of the normal and

abnormal target values used in the NBS study, yield projected approximations of the precision CAP participants would have achieved in analyzing the NBS test samples. This procedure

is of questionable validity in the case of cholesterol where CAP supplied abnormally high

specimens while the NBS abnormal specimen was abnormally low. The reported NBS data are an average of the normal and abnormal precision values.

The first two columns in Table 3.6.1.2 show the constituents and the medically acceptable precision (Sg) which should be used when the normal and abnormal precisions are

averaged. The third column, headed NBS All-Lab Pooled Teclmique Precision, displays, for

each constituent, the standard deviation of all reported non-outlier analyses. The fourth

column, headed NBS Median Technique Precision, displays the precision of that technique

which had the median technique precision (as defined in Section 3.3) among technique precisions calculated for the constituent. Comparison of these data should permit assessment

of whether the conclusions on medical acceptability were biased by using data pooled over

techniques. Since these two sets of data are roughly consistent in magnitude, it is unlikely

that pooling affected the conclusions.

The last six columns in Table 3,6.1.2 provide the data needed for intersurvey comparisons. The three sets of precisions selected for comparison were:

(a) The NBS Interstate and CAP data on the standard deviations of all reported non- outlier analyses (Pooled Technique Precision).

(b) The NBS Interstate and CAP data on the median standard deviations among the

deviations of the different techniques used in analyzing a constituent (Median

Technique Precision).

(c) The NBS All-Lab and CAP data on the standard deviations of those techniques which

were applied with the least variability by study participants (Minimum Variance

Technique Precision)

For the most part, the data prove to be canparable. Thus, the precision achieved by an analysis of dialyzed specimens, as reported in this study, is not atypical of precision

achieved in analyzing serum pool specimens, and the conclusions on medical acceptability

are probably robust ones.

3.6.2. Likelihood of a Medically Useful Laboratory Analysis

To estimate the probability that a correct laboratory assay of one of the seven

clinical chemistry constituents (total bilirubin is again excluded) could be acceptably

reproduced by laboratories in a given group, a multi-step procedure was utilized. First,

a 251 random sample of the shipment 1 assay repxjrts (including outliers) returned by each

group was selected, and separate plots of the seven constituents were prepared. In each

case, normal determinations reported by the group were plotted on the horizontal axis and

abnormal determinations were plotted on the vertical axis. The medically acceptable region was then plotted about the target value. (For calciim, the region about the Nfil^T was also

plotted.) The percentage of points lying in the acceptable region gives the desired estimate

49
so
of the percent of medically useful results. The plots are displayed in Appendix D. Table

3.6.2.1 summarizes the estimates. Reference laboratory capability assessment was based on

all reported analyses.

Examination of Table 3.6.2.1 reveals that more than half of the reference laboratory

determinations for all constituents except glucose and calcium were within medically acceptable precision limits from the target values. In only four instances were more than half

of the analyses of a constituent by a laboratory group medically useful —the Interstate and

Medicare Independent uric acid analyses and the Interstate and JCM\ urea nitrogen analyses.

Overall, the average likelihood of a correct assay being reproduced within medically acceptable limits was 28%. Such a low likelihood of a medically acceptable determination is

particularly noteworthy in view of the frequency with which these laboratory tests are performed and the cost of their administration.

Further examination of the plots in Appendix D reveals that the medically unacceptable

values tend to cluster about a 45" line. Youden has demonstrated that such clustering

results from systematic errors. Known causes of systematic error include miscalibration

of equipment, inadequate calibration specimens, incorrect application of procedures, and

inherent bias in procedures.

3.7. Target Value Surrogates

Exact determination of the composition of pooled serum test specimens is impossible

within the current state-of-the-art of clinical chemistry. Consequently, manufacturer's

average assays may be used as the target values for pooled serum calibration and proficiency

test specimens. In this proficiency test, the availability of actual weigh-back values

makes it possible to assess whether the manufacturer's average assay is a good choice for

the target value or whether another choice might be preferable in proficiency testing

situations. To address these questions three possible target surrogates were considered in

addition to the manufacturer's average assay. These were the mean reference assays, the mean laboratory assays, and the mean assay of laboratories using the technique that the

manufacturer used. Table 3.7,1 contains all of the data used in this analysis. An asterisk

next to the mean value indicates that this value differs significantly from the weigh-back

value at the 95% confidence level.

3.7.1. Adequacy of the Manufacturer's Average Assay as a Target Value

In all cases, the manufacturer's determinations were performed with either the recom- mended referee method or the best applied technique identified in Section 3.5.9. A series

of t-tests run on the manufacturer's average assays indicates that 15 of 14 differ significantly from the weigh-back values. However, an examination of the percentage difference of

the manufacturer's average assays from the weigh-back values reveals that 10 of the 14 assays

differ from the weigh-back values by less than 3%. The statistical significance of the

differences between these assays and the weigh-back values may have resulted from the use of

the precision values of the manufacturer's assays in performing the t-tests. These values were used because the precision values of the techniques were not available. Precision was extremely high for the manufacturer's assays. This may be partially attributed to the fact

that all 25 replicate determinations were run on the same day on the same piece of equipment

by the same technician. Thus, the precision of the manufacturer's assays is largely a measure of the vial-to-vial variation, and the significance of the t-tests for the manufacturer's assays may be artificial.

Examination of the reference laboratory data facilitates an evaluation of whether the

apparent inaccuracy of the manufacturer's assays results from unavailability of data on the

precision of the manufacturer's technique. Reference laboratory sample sizes and average

2^W. J. Youden, Statistical Techniques for Collaborative Tests , Association of Official

Analytical Chemists, 1967.

51
52
d ^

00

—1 Lr> rsi I—f in ^ r-t O O

o o

d d

o ^

o —• «T r-< «

C

O O

tn CTi

-T 00

t^l r-H

r-- VO r-H O rH

O O

^ 00

in 00

<s —I o o

bO

.-I CO < 3

30 S

OO *-p< —4 ^

C7) to

%o to

C7» LOlO

1—1 rvl

00 o

GO r o to O O O 4/1

I —I o o

La

I/) -rr

oo >—

.-1 vO « «

00 o

00 to 1-4 OO '

oo oo

r 1 r 1

r i \0 to 00 ^ LO ^ vO LA r r oo oo

1-^ —

to

U-i r-4 O O

LO Oi to O r-i oooo oo

lA - X

U K

3

C O

CO \0 o r-~

to CT> o — o c o r 00 to

to

oo lo

—1 a> t3> LO d >TLO to d

d d

—I « «

Ef

c«

re 3

^ 3 5^

o

O 4) < I/"

o^ .—

3 —

x:

tJ ;/)

enUa]

j

tungst Syst. C

o d

oX • C 0 .21 a ^

Abel

ly *-* O 3

3

5 O C C

-,3 ^

01 O

3 3

53
percent differences from the weigh-back values (Table 3.7.1) are comparable to those in the

manufacturer's determinations, but the deviations are much larger because of interlaboratory

variability. None of the nine reference assays which differ from the weigh-back values by

less than 5% are statistically different from the weigh-back values. In contrast, 9 of the

10 manufacturer's assays which differed from the weigh-back values by less than 3% are statistically different from the weigh-back values. Thus, it would apf)ear that the accuracy

of manufacturer's average assays may, in general, be acceptable, but these assays are performed in a manner which does not protect against occasional human and calibration errors.

3.7.2. Other Possible Targets for Proficiency Testing

The first surrogate considered as a replacanent for the manufacturer's mean assay, the mean value of all reported determinations, differed significantly from the weigh-back value

in 13 of 14 instances and differed from the weigh- back value by more than 3% in 8 of 14 instances. This level of performance is far below that achieved by the manufacturer, and

this possible target was rejected. The mean value of all determinations performed by the

technique employed by the manufacturer was the second target considered. This measure

differed significantly from the weigh-back value in 12 of 14 cases and differed from the

weigh- back value by more than 3% in 5 of 14 cases. This level of performance is similar to the level of performance achieved by the manufacturer. However, the average percentage

difference of manufacturer's means from the weigh-back values is only 2.6% while this group's

average is 5.2%. Therefore, this possible target was also rejected.

The final target considered was the mean reference assay. Only 4 of the 14 reference

assays differed significantly from the weigh-back values. Five of the 14 assays were more

than 31 from the weigh- back values and the average percent difference was 2.7%. Thus, the

accuracy of the mean reference assay is comparable to the accuracy of the manufacturer's

average assay. Furthermore, the mean reference assay is determined from heterogeneous data

sources and forms a more reliable base than the replicate analyses of a single laboratory.

When available, mean reference assays (MRAs) appear to be a better choice of target values

than manufacturer's average assays. This finding lends credence to the choice of MRAs as the target values for cholesterol and hematology in this study.

3.8. Comparison with Other Fh-oficiency Surveys

The coefficients of variation (CV) provide a basis for comparing the proficiency with

which individual laboratory methods were applied. Unfortunately, although information identified by method is regularly reported by seme groups administering open proficiency tests,

many studies that are one-time assessments of laboratory capability report only CV's grouped

over methods. In order to make comparisons with these surveys, the CV's from this study were combined over all groups, methods, and levels (normal and abnormal) . Although this

procedure increases the number of surveys available for comparison, the concomitant masking

of information decreases the reliability of the comparisons.

Table 3.8.1 displays the CV's observed in this study and 11 other studies. Bibliographic

references for these studies are given in Appendix E.

The relative variability observed in 4 of the 12 other studies, the initial (1949) CAP

survey, a Canadian study in 1963, an assessment of Italian laboratory proficiency (1968),

and, surprisingly, the initial CDC survey in 1968, was greater than in this study. Variability displayed in the other seven surveys seems to be consistent with the relative

variability in this study.

^^Al though they are not shown, the ungrouped CV's can be calculated from the data in Section

3.5.

54
H

4-> o

CO

IT)

00LO 00

O

e2

••-> Iu

I-t rto

•i-t a1—

uogH

•HI

(Acsuogw

•H o

•l-J

i-H

00 cr.

T LO to 00

<N| (^1 to to 1—

(Nl

LO

O CT> (SI 00 (Nl o

l-t LO 00 a. 1—

LO

1—1 vO1—

!*.<» LO cr> c LO rsi 'Jr 1 o (Ni CD

i-H ^ ) I—

tor 1 00

1—1 00

r4

to

1—1 to LO

r—

'-' (NJ o o LO rj

(Nl a>

ro

to

1—1

Or 1

CO CT) 1—

t^. fNl o o

1—1 oo LO

t—

o 1—

1—1

r-- o vO r-t LO to Tt r-- 1—

LO

I-H I—

l-H to

r-> to to

1-) bO

1—

to

1—

<N| to

rH 1—1 00

1—1

O'3-

d*> O 00 LO (Nl

O (^4 LO to 1—1 to o> 00 00 LO

r-j LO ol-H OO t^ oI—

1—

1—1—1 1—1

u

l-H 00

to

(1> l-t 43

rt Hi

+-> -o c 4)

tg 3 >N

4-1 1—

•H nj

*-> •iH

!/)

Q o 1 u

4-> to

"2

-3

4-1

00I

a,c

Si

(U o 4-> 1^ ^2

uso

(U3

a> fNl

«3- o 1^

<T> 1—

l-H

O

l-H o

l-H

4-> to 1 1 1—1

I-. a\ T3 1 1—1 > an nj

.—1 •iH to c l-H 3

re 1

ii

reu • iHu

iH

00so

1—

re 3 ii

55
4. HEMATOLOGY

The topics addressed in analyzing hematology include:

(a) Differences in precision and relative accuracy among laboratory groups;

(b) Differences in precision and relative accuracy among laboratory techniques;

(c) The performance of laboratories in this study in comparison to the performance of

laboratories in other published proficiency tests.

The analysis follows the same plan which was outlined for clinical chemistry in Section 3.3,

and the discussion snploys the definitions presented in Section 3.1. The mean reference

assays were used as the target values in the hematology analysis.

Because the two shipments of hematology specimens differed in composition, it was necessary to analyze each separately. The rankings of precision in these two separate

analyses were virtually identical. Therefore, it was deemed sufficient to present the data

on precision for the first shipment only. There was no significant difference in relative

accuracy among the laboratory groups; consequently, it was possible to pool data over

groups for the analysis of relative accuracy of the techniques.

4.1. Precision and Relative Accuracy

The table formats used in the previous chapter will be adopted here. The ranking of

group performance again appears at the botton of the first table in each set.

4.1.1. Red Cell Count

The red cell count data for shipment 1, stratified by group and laboratory technique,

appear in Table 4.1.1.1. Precision of techniques is ranked:

1 Coulter Model S

2 Coulter Model A, B, D2

.. ^ 2 Kits, Coulter F

3 Hemocytometer

Examination of Table 4.1.1.2 reveals that only the Fischer Autocytometer technique was applied in a statistically accurate manner for all four determinations. However, the average

percentage difference from the target value did not exceed 3% for any of the techniques, and

only the Hemocytometer technique had a percentage difference which exceeded 5%. Although

the sample size was insufficient to make operationally significant statements about the

precision of the Fischer Autocytometer, it appears that this technique was applied with a precision comparable to that achieved with the Hemocytometer.

Based on criteria of precision and accuracy, it would appear that Kits and the Coulter

Models A, B, D2, F, and S were successfully applied techniques for red cell counting.

4.1.2. White Cell Count

Analysis of the white cell count data displayed in Table 4.1.2.1 reveals that the

precision of techniques is ranked:

1 Coulter Model S

2 Coulter Model A, B, D2, F

2 Kits

3 Hemocytometer

Table 4.1.2.2 indicates that statistically accurate performance was achieved most

frequently with Kits and the Coulter Model S. Six techniques, Kits and the Coulter Models

A, B, D2, F and S, were applied with average inaccuracies of less than 3%; and appear to be

the most successfully applied techniques for white cell counts.

56
57
c

•H

r I O

O

f—

oo CT> 1 ro Csl hO

rsi Ti f—

1 t

CT> o 00 00

t-H rsi r—t

X 'a- « « N

00

I—

rt

II c

•r-(

IT)

UO

to

vO l/l 1—

r-l

O r-l o r-4 to t—

~o rsi CT) ^0 o

to -J rj fO

un §

to o I—O o I 1 1 u

f-H •—

u 1—1 LO ri >*

to —1 1—1 r-l o

a: * « «

II

II

X

LO o LO 1 c 00 00 rr r 1 o 00

to 1-H 1—1 1 Q

er CO

ets < c/) o LU 3 *-> a r—

>, •H (U u

-a T3 o *-> (/) me J2

«i Si 4-> 3

r—

3

uu o < O x;

•M

U

>, 0; 'J M •(-> o

.—1 x: ;/) 3 t/j J o O •r-( u •< cMco

'7^

T—

\0

O 1—

00 CTl LO Ol to

r—t

O to 1—

LO CT> to tor o r-H1

rsi to to

LO o a\ r—

LO

>—

t—

rsi o r-l rj rsi rsi « « o

to

00 00 00

T—l oo oo n 1—1 o 1—1

I*oo

to oo

to

oo o

rj

:=>

uo

CQ .p

<-r J- o u.

•—1 r-l )-• o u o

-a T3 o •tJ !-> 1—

3 3 o < O

^- >s (D u t-> QJ

1—1 1—1 J= 3 3 Ul +-> O O •iH U a- a

II MX

O

O 1—1

oo

II

II

X

1—( 1—1 LO LO

LO rj

O 00

1—( 1-1 t--

toO

4-1

r-H

(1) uc

<u -o

•Hcou

LO

x:

1/) uC

^1%uc

(-1 LM

I—

cu

l-M

•r-(

•H

58
ormal

Medicare

Itospitals

n

X

s

120

6.9

1.02

24

7.3

.74

19

7.4

.64

20

7.1

.84

184

7.0

.94

118

20.8

2.67

23

19.7

1.01

18

19.5

.96

20

19.5

2.12

180

20.4

2.39

K)

1,

White

Cell

Count

N

JCAH

n

X

s

55

7.2

.84

55

7.2

.52

42

7.3

.67

29

7.2

.50

Cell

Coimt

Abnormal

55

21.1

2.56

56

19.9

1.18

45

18.9

1.43

30

19.8

.95

189

20.0

1.89

*N|

Table

4.1.2.

Interstate

n

X

s

24

7.1

.98

32

7.3

.63

70

7.5

.56

23

7.5

.59

168

7.4

.68 White

24

21.2

1.91

31

19.5

1.30

69

19.4

1.13

23

20.4

1.75

165

19.7

1.82

Kl

Doctors'

Offices

n

X

s

175

6.8

.95

21

7.3

.71

208

6.9

.94

168

20.4

2.58

17

18.9

3.54

197

20.2

2.63

K)

a* 1

Henocytometer

Coulter

Model

A,

B,

D2,

F

Coulter

Model

S

Kits All

Methods

Ikanocytometer

Coulter

Model

A,

B,

U2,

F

Coulter

Mixlel

S

Kits All

Methods

Ranking

of

Group

Precision

59
60
4.1.3. Hematocrit

Based on the data displayed in Table 4.1.3.1, the precision of the hematocrit techniques was ranked:

1 Microhematocrit

1 Coulter Model S.

Examination of Table 4.1.3.2 shows that the abnormal determinations using Kits and one of the Coulter Model S abnormal determinations were the only statistically accurate determinations. It appears that use of a single normal target may have affected the results in

this instance. The average inaccuracies of all techniques exceeded 3%, but only the Coulter

Model S had an average inaccuracy of more than 4%.

Operationally significant statements about the relative precision of Kits cannot be

made based on the small number of determinations reported in this study. However, these

techniques appear to be more imprecise than the others used in hematocrit determinations.

Among techniques applied by the study participants, the Microhematocrit technique appears

to have been the most satisfactory.

4.1.4. Hemoglobin

Table 4.1.4.1 suimarizes the hemoglobin analyses stratified by group and technique.

Precision of the techniques is ranked:

1 Coulter Model S

2 Cyanmethaixjglobin (Source Group A)

2 Cyanmethemoglobin (Source Group B).

Table 4.1.4.2 contains the hemoglobin data pooled over groins. The Hemoglobinometer

(IL) was statistically accurate in all four determinations. None of the technique means

was inaccurate by more than 5%, and the mean technique inaccuracy never exceeded 3%. The

techniques where sample sizes were insufficient for reliable evaluation of relative precision did not appear to be less precise than the Cyanmethemoglobin techniques or the Kits.

It would appear that all of the hemoglobin techniques were equally well applied by the

study participants.

4.1.5. Mean Corpuscular Volume

The mean corpuscular volume data is displayed in Table 4.1.5.1. Precision of techniques

is ranked:

1 Coulter Model S

2 Best Chart

2 Hand Calculations.

Two of the Hand Calculation means and one Best Chart mean were statistically accurate.

None of the Best Chart or Hand Calculation means were more than 51 inaccurate, and the aver- age inaccuracy with these techniques was less than 3%. However, it should be noted that

obviously deviant values were removed from these determinations at eight times the rate they were removed from the Coulter S determinations. As a result, it is virtually inqwssible to judge which techniques were applied best.

4.1.6. Sunmary

Relative accuracy did not differ significantly among the laboratory groups. However,

application of Friedman's Two-Way Analysis of Variance by Ranks to the ranked precision data

displayed in Table 4.1.6.1 showed that the precision of the groups differs significantly at the 991 confidence level.

61
62
reuo

rH

IIC

•rH to

I'd c

•H

00

1^

l-H l-H

1—

I-H LO un tn T

l-H • l-H • Orf « •K «

l-H i-H

•<* o l-H o in o 1—

I—l-H rj

r--1 r)- o o • o O * * K

II

II

X

>-H 00

to •-r rt 00 coE

CI. ce

f-H 00o

l-H l-H c LO o 1—

to o

l-H

cr.

l-H

•»: o O Oo

cr.

rH OO \0 00

LO fO ^ LO rH I o c

T3• r^

LlHCOO

o«

LO

Ol

(U

ro o «* ro o

CO 00 00 OO c

l-H rH rH 0)u

fl OJ

U-i

II

X 0) ^ rH VO

hO

T

O rH

toOO ceEo

LM

0) 3O

O

Si ouu

-.3 o

Si

)ho3o c

Cl

!hV

U-i

14H

>.

rHMC

reJc

63

65
J32

0\ »0 fO

a> a> CO cft

r-t LD rs|

rj CO

vO i-n o

c

3 2"

o o

rj r

66
00 o oo

00

00o CTl

to

II 1T3

II

C

IH

cn« «

oo«

II

II X

c

5^

LD 00

Ol 00

II

II X

O 1-1 to r-l

to

OO

o

OO

oc

XI<

LO 00

to o

ou

co

• H

co

to

c

•r-l

1^3

1 X

(^1 to to

00 CTl

II

II X

u

1—1 3U

t/l 3

e-o

II O

o

o

to

r4 00 o a>

<-i «

u

II X

I—I LO I (^J I—( rH I

CO

0)U

C

01

l-l c

rts

0) x:

14-1

o u

C/3

3

o

o

03

4->

-3

3

O

t/l

CO

67
Table 4.1.6.1. Ranking of Precision of Groups by Constituent and Shipnent

Constituent

Oil Shin- J.L) ment Offices , Interstate

1

' JCAH

PTeuicdre

Hospitals

pTeuica

Labs.

Red fell fount 1 5 j 2.5 2 5

2 5 2.5 2.5 2.5 2.5

Whi tp Pel 1 fount 1 4 i 4 4 1 2 5 2 3.5 1

1 3.5 1

1 iv7i 1 la. c X X i> X

i 7 2 5 3.5 1.5

' 1.5 3.5

Hemoglobin 1 4.5 2.5 2.5 4.5 1

2 4.5 3 1 : 4.5 2

Mean Corp. Vol. 1 3.5 1 3.5 3.5 3.5

2 3.5 1 3.5 1.5 , 5 1.5

Further evaluation of the differences in precision was based on application of the

Friedman multiple comparison procedure adjusted for ties. The precision of the groups of

laboratories can be exhibited as follows, where groups joined by the same line are not

significantly different from each other at the 90% confidence level.

Rank Order Laboratory Groiq)

Ntost Precise Medicare Laboratories

JCAH

Interstate

! Medicare Hospitals

Least Precise ; AAFP/ASIM

Table 4.1.6.2 indicates the techniques which the study participants ap)peared to have

applied most successfully, and the percentage of study participants used these techniques

in shipment 1. For all constituents, more than 40i of the laboratories used one of the best

applied techniques. In contrast, there was only one clinical chemistry constituent which

more than 40^ of the laboratories analyzed with one of the best applied techniques.

4.2. Comparison with Other Proficiency Surveys

The data displayed in Table 4.2.1 have been formatted in the same manner as the

clinical chemistry information in Section 3.8. The coefficients of variation in all the

listed surveys, with the exception of the initial CDC survey, appear to be comparable to those observed in this study.

.4.3. Acceptability of laboratory Group Precision

A literature search did not yield any data which could be used to assess the medical

inplications of the honatology analyses. However, one article indicated that Dr. Edward

Bums of the Medical College of Ohio at Toledo had monitored the white cell count, hemoglobin,

and hematocrit levels of a group of individuals from the Owen-Illinois Canpany.^'' Dr. Bums

2''E. Bums, "Individual Jfealth Pattern Profile Studies in Fifty- Six Healthy Adults," Joumal

of Occupational Medicine , Vol. 9, No. 10, October 1967, pp. 511-517.

68
5

S

to

ou

<1> u

5i2

o

o

0)

69
00

O

CO

05

>

ir. c o

o

4h8

u

Oco

tn

•i-t uIU

o o

00

o

CD

O

i)

B

OO

cn

5

u

a.c

o

*-> vD < r—

r-l rr-t \0

o r-H

t 1—

M-iO at- at

i/l CL,

arte s

lin. lin.

• rH u u

'J

'/) u

1 hy

70
was contacted and agreed to supply data on the individual variability of thirty-nine male

and two female subjects whose blood was tested 20 times under standardized conditions over a period of 60-80 days. These data on individual variability are, of course, confounded with

the analytical variability of Dr. Bums' laboratory. Consequently, a delineation of the

medically useful range based on Ur. Bums' data may be overly broad; and the estimates of

acceptability of the precision of the hematology analyses may be overestimates.

The medically acceptable interlaboratory precision, as defined in Chapter 3, can be

computed as one-half of the mean individual variability determined in Dr. Bums' study.

These values are shown in the column headed Sg in Table 4.3.1. The table also displays the

sanple size and standard deviation for the reference laboratory determinations, and each

laboratory group. Standard deviations which were not significantly greater than the acceptable precision values (defined and computed as above) at the 99% confidence level have been

starred in the table.

Examination of this table indicates:

(aj The reference laboratories achieved medically acceptable precision for hematocrit

at the abnormal level, and hemoglobin and white blood count at both levels of

concentration.

(b) The precision of the JCAU- Accredited laboratory analyses of hemoglobin was medically

acceptable at the abnormal constituent level.

(c) All other group precisions exceeded the medically acceptable precision values

defined and computed in this report.

5. MICROBIOLOGY

5.1. Background

This phase of the evaluation study consisted of an analysis of two shipments of five

vials, where each vial contained a lyophilized pure culture (see Subsection 2.4.3). The

participating laboratory was asked to identify the organism in each vial, giving both genus

and species wherever possible. According to information supplied by the Scientific Advisory

Conmittee, more than one answer was acceptable for three out of the five organisms. The

preferred and otherwise acceptable answers are displayed in Table 5.1.1.

Table 5.1.1. Preferred and Other Acceptable Identifications of Microbiological Specimens

Preferred

Streptococcus pneumoniae

Diplococcus pneumoniae

Pneumococcus pneumoniae

Pneumococcus

Other Acceptable

None

Vial ^ Shipment

Vial 1, Shipment 1 and Vial 4, Shipment 2

Salmonella oranienberg

Salmonella Cj

Salmonella montevideo

Salmonella enteritidis

Salmonella NOS

Salmonella arizona Vial 2, Shipment 1 and Vial 5, Shipment 2

Klebsiella pneumoniae

Klebsiella NOS

Pseudomonas aeruginosa

KlebsiellaEnterobacter group

PseudonK)nas fluorescens

Pseudomonas stutzeri

Pseudomonas NOS

Vial 3, ShipHnent 1 and Vial 1, Shipmient 2

Vial 4, Shipment 1 and Vial 3, Shipment 2

71
0) Q

CO

O". oc- 00 o LT) ^

rt O 1—

rH t-H o o

O

>-< reu

C30 O r

. * O 00

"3 c 1—1 l^J Cj 1—1 r—

00 oo

re 10 I—1 O o o

cx

(/) r\

V-U o T O vD sO

00 00 00 00 OO 00

t—t f—

O f-H O c o o

*

U1

O

•Jc

ou

o

(LI

-a- o \0 00 LO oo C oc 00 00 oo 00 00

r-- 00 o ^

oo LT,

(^1 .—I *T ^

r-i r

00 o

O r 1

OO r o CT) o cri oo

o o

00 O CTi

O r-H ^ o o o

•a to o o o o

co

3 ^

E re C c

ou

C C

O X3 ^ <

8

o

u

o

4-1 I

Sio

1—

o

eo

72
Table 5.1.1. Preferred and Other Acceptable Identifications of Microbiological Specimens

(continued)

Preferred Other Acceptable Vial S Shipment

Streptococcus faecalis hkme Vial 5, Shipment 1 Enterococcus , group D and Vial 2, Shipment 2 Alpha Streptococcus, group D Beta hemolytic Strep. , group D

Gainna Strep. , group D Enterococcus

Streptococcus enterococcas

Streptococcus zymogenes

Upon the advice of the Scientific Advisory Committee, each instance of multiple identifications (a laboratory reporting that it had identified two or more organisms within the same vial) was treated separately. When one of the organisms identified was a pathogen

which would not be expected to be found in the usu^l laboratory environment, the analysis of

the vial was considered incorrect regardless of whether the correct organism was also one of

those identified. If the correct organism was identified along with one of the innocuous

bacteria which normally occur on the skin or in the air, the harmless contaminant was

ignored, and the analysis of the vial was considered correct. There were several instances

where the study team sought the advice of Dr. George Dillard, resident public health service

physician at the National Bureau of Standards, as to whether a specific analytical report

would have resulted in a medical diagnosis other than that vtiich was intended.

5.2. Overall Performance

It is informative to look at the percent of incorrect answers by each laboratory group,

recalling that participation in the study was completely voluntary, that these findings

relate more closely to laboratory capability than they do to routine performance, and that

the organisms were not unusual ones.

Although the overall frequency of incorrect identifications is 16.8%, it should be

noted that deleting the Interstate group raises this figure to 19.4%—almost one out of

every five.

Incorrect Identifications

—a 1

78 7.6

259 15.8

139 17.9

64 20.8

257 26.0

797 16.8

^^Sanple Size = total number of identifications, not number of laboratories.

Laboratory Group Sajiple Size^^

Interstate Licensure 1021

JCAii Hospitals 1639

Medicare Independent 778

Doctors' Offices 308

Medicare Hospitals 987

OVERALL 4733

73
5.3. Comparison of Group Performance

Two statistical approaches—one parametric and one non-parametric—were considered for

use in evaluating the relative performance of the groups. Each approach has some drawbacks.

The non-parametric approach, based on a ranking procedure, ignores the quantitative nature

of the data. Unnecessary reduction of the significance level of the performance rankings

may result. The parametric approach relies upon assumptions concerning the normality of the

distribution of the data and does not provide an adequate means for distinguishing "other

acceptable" from "preferred" identifications. Since neither approach seemed totally satisfactory, both were tried. Some credence can be attached to the resulting performance • rankings because they were consistent, although the significance levels differed. The

higher parametric significance levels were used in reporting the results. Both analytic

approaches are described ih this section.

5.3.1. Non-parametric Approach

In order to examine relative performance non-paraiT«trically, the submitted report forms were scored as follows: any of the Prefeired identifications arbitrarily received a score

of 20, identification of "Other Acceptable" received a score of 10, and all other identifications were scored as zero. It must be recognized that this scoring system assumes that

the five organisms were equally difficult (or easy) to identify.

No attempt was made to analyze the laboratory scores on different vials parametrically

because the assignment of equal scores to correct answers for all organisms was arbitrary,

and parametric analysis would be invalid if the organisms were not equally easy to identify.

Instead, the scores on each vial were ranked across laboratory groups. Although both shipments contained the same five cultures (with different vial numbers), their identifications were treated independently (e.g., 10 vials rather than 5) because of the variation in the

number of laboratories reporting results of shipments 1 and 2. This variation is interesting

to note because of the rather consistent decrease in nunber of participating laboratories

between the first and second shipments. Doctors' Offices, the Interstate group, and Medicare

Hospitals lost approximately 10 each, while the JCAH group was reduced by almost 20 laboratories in the second shipment.

The results were placed in a 10 x 6 matrix, each of the ten vials representing a row

and the six laboratory groups representing the columns. Table 5.3.1 contains rankings by

vial (or row) ; a rank of one indicates the lowest average score and the rank of six is

given to the highest average score for that vial. For exainple consider S. pneumoniae in

shipment 1. The lowest average score was received by the physician members of the AAFP,

followed in order by the Medicare Certified Hospitals, the Medicare Certified Independent

laboratories, the JCAH Hospitals, the ASIM physicians, and finally the CDC Interstate

laboratories (the highest ranking group). In this and subsequent tables, n is the total

number of analyses of the specimen reported by members of the group.

Table 5.3.1. Ranking of Group Perfonnance, Microbiology

Specimen

Shipment AAFP ASIM

Inter- state

fax:) JCAH

Medic.

Hosp.

Medic.

Indep.

n Rank n Rank n Rank n Rank n Rank n Rank

S. pneumoniae 1 9 1 27 5 107 6 174 4 108 2 83 3

S. pneumoniae 2 4 1 22 3 97 6 151 4 91 2 74 5

Salmonella NOS 1 9 1 23 2 107 6 170 4 105 3 80 5

Salmonella NOS 2 3 2 24 1 98 6 154 5 88 3 74 4

Klebsiella NOS 1 10 1 28 2 107 6 176 5 109 3 83 4

74
Table 5.3.1. Ranking of Group Perfonnance, Microbiology fcontinued)

Inter- Ship- state Medic. Medic.

Specimen ment AAFP ASIM (CDC) JCAU Itos^ Indep. n Rank

1 " Rank n Rank n Rank ank n Rank

Klebsiella NOS 2 4 6 24 3 98 5 155 2 90 1

' 75 4

r. aeruginobd 1X 10 2 27 4 105 6 175 5

! 1 nfi 1 fin \

P. aeruginosa 2 4 6 23 4 98 5 156 2 91 1 75 3

S. faecalis 1 8 4 28 3 107 6 173 5 108 1 81 2

S. faecalis 2 4 1 - 22 4 97 _^ 155 5 89 __2 74 _J_

R = 2.6 ,R = 3.1 1 R = 5.8 1 R = 4.1 R = 1 .9 R = 3.6

The statistical test used to determine whether the group rankings were significantly

different was Friedman's Two-Way Analysis of Variance by Ranks. If all of the groups had

equal capability to identify the organisms, then the distribution of ranks in each column

would be a matter of chance. Thus, one would expect ranks of 1, 2, —6 to appear in each

column with about equal frequency. The Friedman test determines whether the rank totals

differ significantly.

The first application of the Friedman test involved all six laboratory groups and

resulted in rejection of the hypothesis of equal capability at the .05 level of probability.

In other words there is a very strong reason to believe that the six laboratory groups

differ in ability. The multiple comparison portion of tl e test indicated that data from the

two physician-office groups (the AAFP and the ASIM) could be pooled together. Analysis to justify pooling was not repeated in the parametric analysis.

After combining the AAFP and ASIM it was necessary to re-rank all of the groups, as shown in Table 5.3.2.

Table 5.3.2. Revised Ranking of Group Performance, Microbiology

Specimen

Shipment

Doctors

Offices

Inter- state

(CDC) JCAll Medic.

Hosp.

f fedic,

Tndep.

n Rank n Rank n Rank n Rank n Rank

S. pneumoniae 1 36 2 10" 5 174 4 108 1 83 3

S. pneumOTiiae 2 26 2 97 5 151 3 91 1 74 4

Salmonella NOS 1 32 1 107 5 170 3 105 2 80 4

Salmonella NOS 2 27 1 98 5 154 4 88 2 73 3

Klebsiella NOS 1 38 1 107 5 176 4 109 1 83 3

Klebsiella NOS 2 28 3 98 5 155 2 90 75 4

P. aeruginosa 1 37 2 105 5 175 4 108 80 3

P. aeruginosa 2 27 4 98 5 156 2 91 75 3

S. faecalis 1 36 3 107 5 173 4 108 81 2

S. faecalis 2 26 3 97 5_ 155 4 89 74 2

R = 2.2 R = 5.0 R = 3 4 R = 1.3 R = 3.1

29r. Miller, op. cit . 75
Friedman's test was run again, and the differences among the laboratory groups were

still significant at the 951 confidence level. The multiple comparison portion of the test was then applied at various confidence levels. The results of this analysis are discussed

in Subsection 5.3.3.

5.3.2. Parametric Approach

A number of different additive analysis of variance (ANOVA) models were run. Scheffe's

procedure was applied to the ANOVA results in order to rank the laboratory groups.^" The

models used were:

(a) Proportion of incorrect identifications by vial and shipment versus laboratory

group;

(b) Same data as in (a) with a variance- stabilizing arcsine transformation;

(c) Proportion of incorrect identifications averaged over shipments versus laboratory

group;

(d) Same data as in (c) with arcsine transformation.

An analysis weighted by sanqjle sizes was also done for each of the four ceises. >Jone of

the models fit the data as well as one would like; but the conclusions of all, significant

at the 95% confidence level, were consistent.

5.3.3. Performance Ranking

The parametric ranking at the 95% confidence level is identical to the non-parametric

ranking at the 75% confidence level. This ranking will be ascribed a confidence level of 95%.

The performance (capability of the different groups of laboratories can then be pwrtrayed

as follows, where groups joined by the same vertical line are not significantly different

from each other at the 95% confidence level.

Rank Order Laboratory Group

High Interstate (CDC)

JCAH

Medicare Independent

AAFP/ASIM

Low I Medicare Hospitals

Several statanents can be made based on this ranking:

(a) The demonstrated microbiological capability of the Interstate Licensure group (CDC) was significantly better than the capability of every other gtoup.

(b) The JCAH Hospital and Medicare Independent laboratories were significantlylaetter

than the Medicare Hospital laboratories but were not significantly different from

the Doctors' Office laboratories.

(c) The Medicare Hospital laboratories were significantly pxjorer than all other groups

except the Doctors' Office laboratories.

5.4. Suranary of Group Identification

It is interesting to review the various laboratory groups insofar as their identification

capabilities are concerned. To facilitate such a review, two tables have been constructed

for each particular group and for an aggregate of all groiqjs. For each group the first of

these tables contains information on the frequency of correct, marginal and incorrect identifications, while the second table lists these incorrect identifications. Of particular

30r. Miller, pp. 48-53.

76
interest is the heterogeneity of identifications within a vial. The medical implications

of these incorrect identifications are beyond the scope of this report.

5.4.1. Overview of All Participants

The pooled data in Table 5.4.1.1 indicate that Streptococciis faecalis was much more

difficult to identify properly than any of the other organisms. The easiest organism to identify properly was Pseudomonas aeruginosa, and the other three bacteria fell in between

with very little difference in misidentification rate. The fact that S. faecalis was the

most difficult organism to properly identify is a bit disconcerting because this organism

is not that unusual, especially to a large hospital laboratory. From the list of misidentifications in Table 5.4.1.2, it will be evident that many of the organisms mistaken for

Streptococcus faecalis were in the Streptococcus group—viridans, pyogenes, non hemolytic

(gamma), etc. However, there were other instances wher6 groups such as Staphylococcus,

Pseudomonas, Neisseria, Salmonella and Enterobacter were named in place of S. faecalis.

Table 5.4.1.1. Overview of All Study Participants

Specimen

Shipment

Number

of labs Lorrec

1 u s jvjcirgiiicij 11' Tn' r> ID

n 1 n \ n

S. pneianoniae 1 508 431 85 N/A 77 15

S. pneunoniae 2 438 364 83 N/A 74 17

Salmonella N06 1 494 402 81 20 4 72 15

Salmonella N06 2 439 345 78 26 6 68 16

Klebsiella NOS 1 513 387 76 32 6 94 18

Klebsiella NOS 2 445 353 79 25 6 67 15

P. aeruginosa 1 505 335 66 119 24 51 10

P. aeruginosa 2 446 301 67 112 25 33 8

S. faecalis 1 505 354 70 N/A 151 30

S. faecalis 2 440 330 75 N/A 110 25

Table 5.4.1.2. Misidentifications by All Study Participants

Shipment 1, S. pneumoniae: Yeast (1), Pseudomonas NOS (2), Salmonella NOS (2), Klebsiella

NOS (IJ, K. pneumoniae (1), Klebsiella-Enterobacter group (1), H. influenzae

(1), Staphylococcus NOS (2), S. aureus (4), S. epidermis (4), Neisseria NOS

(3), M. polymorpha (2), Uiplococcus (10), Streptococcus NOS (7), S. pyrogenes (4), S. viridans (22), S. faecalis (1), S. anaerobis (1), Non

hemolytic strep (3), Mycobacterium NOS (1)

Salmonella NOS: Pseudomonas NOS (1), P. aeruginosa (2), E. coli (2), S. alkalescens (1), Edwardsiella (1), Shigella NOS (1), S. dysenteriae (1),

S. typhi (13), S. paratyphi A (1), S. paratyphi B (8), S. paratyphi C (6),

S. choleraesuis (7), S. typhimurium (2), Citrobacter group (7), KlebsiellaEnterobacter group (1), Enterobacter NOS (1), E. aerogenes (2), P. aerugenoides (2), Proteus NOS (4), P. vulgaris (1), P. mirabilis (2), S. aureus (1), S. epidermis (1), Neisseria NOS (1), S. viridans (1), S. faecalis (1), Bacillus NOS (1)

77
Table 5.4.1.2. Misidentifications by All Study Participants (continued)

Shipment 1, Klebsiella NOS: Pseudomonas NOS (1), E. coli (19), Salmonella NOS (1),

Citrobacter group (2), K. ozaenae (1), Klebsiella-Enterobacter group (1),

Enterobacter NOS (13), E. cloacae (16), E. aerogenes (16), E. liquefaciens

(3) , E. cloacae, atypical (1), P. aerugenoides (2), Proteus NOS (2), P. vulgaris (1), P. morganii (1), P. rettgeri (1), H. influenzae (2),

Staphylococcus NOS (2), S. aureus (2), S. epidermis (1), S. pneumoniae (2),

S. viridans (1), Corynebacterium (2), Bacillus NK)S (1)

P. aeruginosa: P. pseudomallei (6), P. maltophilia (1), P. aeruginosa, aberrant

strain (2), Alcaligenes (4), Achromobacter (1), E. coli (1), Edwardsiella

(1), Salmonella NOS (3), Klebsiella NOS (4), Klebsiella-Enterobacter group

(1), Enterobacter NOS (3), E. aerogenes (1), E. hafniae (2), Serratia (3),

S. marcescens (2), Proteus NOS (4), P. vulgaris (1), P. mirabilis (1), H. influenzae (1), Mima (1), M. polymorpha (2), Herellea (1), Streptococcus NOS

(1), S. faecalis (2), Bacillus NOS (1), B. subtilis (1)

S. faecalis: Pseudomonas NOS (3), P. aeruginosa (6), A. hydrophila (1), E. coli

(4) , S. alkalescens (2), Shigella NOS (1), S. sonnel fl), Klebsiella M3S

(1), K. pneumoniae (1), Klebsiella-Enterobacter group (1), Enterobacter

NOS (1), E. cloacae (1), E. liquefaciens (1), P. aerugenoides (1), Proteus

NOS (1), Staphylococcus NOS (1), S. aureus (16), S. epidermis (17), Neisseria

NOS (3), N. gonorrhoeae (1), N. meningitidis (1), Herellea (1), S. pneumoniae

. (1), Streptococcus NOS (17), S. pyogenes (9), S. viridans (19), Streptococcus MG, Beta hemolytic strep. Group B (4), Non hemolytic strep (Ganma) (19),

Beta hemolytic strep (15), Salmonella NOS (1), Alcaligenes (1)

Shipment 2, Klebsiella NOS: Pseixiomonas NOS (1), E. coli (10), S. typhi (1), K.ozaenae (1),

Enterobacter NOS (9), E. cloacae (14), E. aerogenes (10), E. liquefaciens

(1), E. hafniae (1), P. aerugenoides (1), Proteus NOS (2), P. vulgaris (1),

P. mirabilis (2), P. rettgeri (1), Haemophilus NOS (1), H. influenzae (1),

Neisseria NOS (1), M. polymorpha (1), S. pneumoniae (5), S. faecalis (1),

Bacillus NOS (2)

S. faecalis: P. aeruginosa (1), A. hydrophila (1), E. coli (3), Shigella NOS (1)

S. dysenteriae (1), Salmonella NOS (7), S. t>phi (1), S. paratyphi C (1),

Klebsiella NOS (2), E. cloacae (1), E. aerogenes (3), Proteus NOS (1), P. vulgaris (1), Staphylococcus NOS (4), S. aureus (19), S. epidermis (11), M. polymorpha (2), S. pneumoniae (1), Streptococcus NOS (10), S. pyogenes (4),

S. viridans (12), Non hemolytic strep (Gamma) (11), Beta hemolytic strep (12)

P. aeruginosa: P. pseudomallei (1), P. maltophilia (1), P. aeruginosa, aberrant

strain (1), Alcaligenes (3), A. species (1), E. coli (1), Klebsiella NOS

(1)',' K. pneumoniae (2), Klebsiella-Enterobacter group (2), Enterobacter NOS

(1)> E. cloacae (2), E. aerogenes (3), Serratia (1), Proteus NOS (2), H. influenzae (2), Moraxella (1), Staphylococcus NOS (2), S. aureus (1), S. epidermis (2), S. pneumoniae (1), Streptococcus NOS (2)

S. pneumoniae: Pseudomonas NOS (5), P. aeruginosa (5), Achromobacter (1), E. coli (1), Salmonella NOS (1), Klebsiella NOS (2), K. pneumoniae (2), E. aerogenes (1), S. aureus (3), S. epidermis (2), Neisseria NOS (1), N. meningitidis (1), M. polymorpha (1), Streptococcus NOS (5), S. pyogenes (2),

S. viridans (29), S. faecalis (6), Non hemolytic strep (Gamma) (1), Beta

hemolytic strep (2), Diphtheroids NOS (1), L. monocytogenes (2)

Salmonella NOS: Pseudcmonas NOS (2), P. aeruginosa (2), Alcaligenes (1), E. coli

(1), S. alkalescens (2), Shigella NOS (1), S. typhi (6), S. paratyphi A (1),

S. paratyphi B (5), S. choleraesuis (5), Citrobacter group (16), Enterobacter

NOS (1), E. cloacae (2), Serratia (1), S. marcescens (1), Proteus NOS (2),

P. vulgaris (2), P. mirabilis (3), Staphylococcus NOS (1), Mima (1), S. viridans (1), S. faecalis (8), Non hemolytic strep (Gaiima) (1), Bacillus NOS

(2) 78
5.4.2. Doctors' Offices

Table 5.4.2.1 indicates that the number of incorrect identifications of Klebsiella NOS,

P. aeruginosa and S. faecalis significantly decreased from shipment 1 to shipment 2. Further

investigation of the data disclosed this to be related to two causes. First, approximately

half of the decrease is attributed to doctors' laboratories which m.is identified these

specimens in the first shipment and did not analyze the second. The remaining decrease could

be attributed to laboratories which had improved their laboratory analyses. Dropouts accounted

for proportionate decreases in correct and incorrect identifications. Thus, the change in

rate is related to an improvement in performance. S. faecalis was the most frequently misidentified organism, while P. aeruginosa was least frequently misidentified. Table 5.4.2.2

lists misidentifications made by the Doctors' Offices.

Table 5.4.2.1. Doctors' Office Laboratories

Specimen

ShipmentNuirber

of labs Correct ID'S Marginal ID'S Incorrect ID' n % n % n %

S. pneumoniae i 36 29 81 N/A 7 19

S. pneumoniae 2 25 IP 76 N/A 6 24

Salmonella NOS 1 32 21 66 4 12 7 22

Salmonella NOS 2 26 16 61 2 8 8 31

Klebsiella NOS 1 38 24 63 4 10 10 26

Klebsiella NOS 2 27 -> -> 81 3 11 2 7

P. aeruginosa 1 37 22 59 10 27 5 14

P. aeruginosa 2 26 19 73 7 27 0 0

S. faecalis 1 36 24 6" N/A 12 33

S. faecalis 1 25 18 72 N/A 7 28

Table 5.4.2.2. Misidentifications by Doctors' Office Laboratories

Shipment 1, S. pneumoniae: S. epidermis (1), Staphylococcus NOS (1), S. viridans (2), S. faecalis (1), Klebsiella (1), S. aureus (1)

Salmonella NOS: E. coli (1), S. paratyphi B (1), S. paratyphi C (1), Proteus

NOS (2), Edwardsiella (1), P. aeruginosa (1)

Klebsiella NOS: E. coli (4), Pseudomonas NOS {1) , Enterobacter NOS (1), E. cloacae (1), E. aerogenes (2), S. pneumoniae (1)

P. aeruginosa: Salmonella NOS (1), Enterobacter NOS {I), Proteus NOS (1), P. aeruginosa (1), Alcaligenes (1)

S. faecalis: S. aureus (2), S. epidermis (2), Streptococcus NOS (1), S. pyogenes (1), S. viridans (2), Non hemolytic strep (Gamma) (2), Beta

hemolytic strep (1), E. coli [1)

Shipment 2, Klebsiella NOS: Enterobacter NOS (1), Proteus NOS (1)

S. faecalis: Proteus NOS (1), Staphylococcus NOS (1), S. aureus (3), Non

hemolytic strep (Ganma) (1), Beta hemolytic strep (1)

P. aeruginosa:

79
Table 5.4.2.2. Misidentifications by Doctors' Office Laboratories (continued)

Shipment 2, S. pneumoniae: Pseudomonas NOS (1), S. \dridans (2), S. faecalis (2), S. aureus

(1)

Salmonella NOS: Pseudomonas NOS (1), P. aeruginosa (2), S. paratyphi B (1), S. choleraesuis (1), Citrobacter group (1), Proteus NOS (1), Alcaligenes (1)

5.4,3. Interstate Licensure (OXT) Laboratories

The Interstate laboratories were found to have the highest percentage of correct

identifications. As Table 5.4.3.1 indicates, the organism most frequently misidentified

was S. faecalis, while that most frequently identified correctly was P. aeruginosa. Table

5.4.3.2 lists misidentifications made by the CDC members.

Table 5.4.3.1. Interstate Licensure (CDC) Laboratories

Specimen

S. pneumoniae

S. pneumoniae

Salmonella NOS

Salmonella NaS

Klebsiella NOS

Klebsiella NOS

P. aeruginosa

P. aeruginosa

S. faecalis

S. faecalis

Ship- Number

ment of labs

1 107

2 97

1 107

2 98

1 107

2 ^ ^ 98

1 105

2 98

1 107

2 97

Correct ID's

102

89

97

86

97

91

83

74

95

83

95

92

91

88

91

93

79

76

89

86

Marginal ID's --^

T, It 0

N/A

N/A

0

2

1

1

20

22

N/A

N/A

2

1

1

19

22

Incorrect ID's "IT

5 5

10

10

9

6

2

2

12

14

9

10

8

6

2

2

11

14

Table 5.4.3,2. Misidentifications by Interstate Licensure Laboratories

Shipment 1, S, pneumoniae: M, polymorpha (1), Diplococcus (2), Streptococcus NOS (1),

S, viridans (1)

Salmonella NOS: S. paratyphi A (1), S, paratyphi B (2), S, paratyphi C (3),

S. choleraesuis (1), S. typhimurium (1), Citrobacter group (1), Proteus

NOS (1)

Klebsiella NOS: E. coli (3), K. ozaenae (1), E. cloacae (1), E. aerogenes (2),

E. liquefaciens (1), E. cloacae, atypical (1)

P. aeruginosa: P. pseudcmallei (1), Achromobacter (1)

S. faecalis: A. hydrophila (1), E, liquefaciens (1), S. aureus (2), Neisseria

NOS (1), Streptococcus NOS (3), S. viridans (2), Non hemolytic strep (Ganma)

(1), Beta hemolytic strep (1)

Shipment 2, Klebsiella NOS: E. coli (1), Enterobacter NOS (1), E. liquefaciens (2), P. vulgaris (1), M. polymorpha (1)

80
Table 5.4.3.2. Misidentifications by Interstate Licensure Laboratories (continued)

Shipment 2, S. faecalis: A. hydrophila (1), E. coli (1), Salmonella NOS (2), E. cloacae (1),

S. aureus (1), M. polymorpha (2), Streptococcus NOS (1), S. viridans (1),

Hon hemolytic strep (Ganma) (2) , Beta hemolytic strep (2)

P. aeruginosa: E. cloacae (2)

S. pneunoniae: P. aeruginosa (1), K. pneumoniae (1), Neisseria NOS CI).

Streptococcus NOS (3), S. viridans (1), Beta hemolytic strep CI)

Salmonella NOS: S, paratyphi A (1), S. paratyphi B CI), Citrobacter group (A),

Serratia (1), Proteus NOS (1), S. faecalis C2)

5.4.4. JCM\ Hospital Laboratories

Similar to the first two groups, the JC^ Hospital laboratories more frequently misidentified S. faecalis than any of the other four organisms, and correctly identified the

P. aeruginosa organism most frequently. Table 5.4.4.1 shows the identification frequencies,

and Table 5.4.4.2 lists the misidentifications.

Table 5.4.4.1. JCAH Hospital Laboratories

Ship- Nuntter Specimen ment of labs

S. pneumoniae 1

S. pneumoniae 2

Salmonella NOS 1

Salmonella NOS 2

Klebsiella NOS 1

Klebsiella NOS 2

P. aeruginosa 1

P. aei-uginosa 2

S. faecalis 1

S. faecalis 2

174

151

170

154

176

155

175

156

173

155

Correct ID's

n

146

126

142

127

133

116

124

107

124

123

84

83

84

82

76

75

71

69

72

80

Marginal ID's

n

N/A

N/A

6

8

13

9

42

34

N/A

N/A

4

5

7

6

24

22

Incorrect ID's

_n

28

25

22

19

30

30

9

15

49

32

16

17

13

12

17

19

5

10

28

21

Table 5.4.4.2. Misidentifications by JCAH Hospital Laboratories

Shipment 1, S. pneumoniae: Yeast (1), K. pneumoniae Cl)» S. epidermis (2), Neisseria NOS

(1), Diplococcus C5), Streptococcus NOS C4), S. pyogenes C2) , S. viridans

(7), Non hemolytic strep CGamma) C2), Beta hemolytic strep C2) , Pseudanonas

NOS CI)

Salmonella NOS: S. alkalescens CI), S. typhi (t) , S. paratyphi B C2) , S. paratyphi C (2), S. choleraesuis C4), Citrobacter group CI), P- aerugenoides Cl)»

P. vulgaris CI), P. mirabilis CI), S. epidermis CI), Neisseria NOS CI),

Bacillus NOS CI)

Klebsiella NOS: E. coli C6) , Salmonella NOS Cl), Citrobacter group CI),

Enterobacter NOS (5), E. cloacae C4), E. aerogenes C5), P. aerugenoides C2)

81
Table 5.4.4.2. Misidentifications by JCAU Hospital Laboratories (continued)

Shipment 1, Klebsiella NOS (continiwd) : P. rettgeri (1), H. influenzae (1), S. aureus (1),

S. epidermis (1), S. viridans (1), Bacillus NOS (1)

P. aeruginosa: P. pseudomallei (2), Alcaligenes (1), E. aerogenes (1), E. hafniae

(1), S. marcescens (1), P. mirabilis (1), Mima (1), Bacillus NOS (1)

S. faecalis: Pseudomonas NOS (1), P. aeruginosa (4), E. coli (1), E. cloacae (1),

P. aerugenoides (1), Staphylococcus NOS (1), S. aureus (4), S. epidermis (6),

Herellea (1), Streptococcus NOS (5), S. pyogenes (3), S. viridans (8), Non

hemolytic strep (Gajima) (5), Beta hemolytic strep (8)

Shipment 2, Klebsiella NOS: E. coli (5), S. typhi (1), K. ozaenae (1), Enterobacter NOS (1),

E. cloacae (7), E. aerogenes (4), E. hafniae (1), P. aerugenoides (1),

Proteus NOS (1), P. mirabilis (1), P. rettgeri (1), Haemophilus NOS fl),

S. pneumoniae (3), S. faecalis (1), Bacillus NOS (1)

S. faecalis: Salmonella NOS (2), S. paraty-phi C (1), Klebsiella NOS (1), E. aerogenes (3), P. vulgaris Cl)» Staphylococcus NOS (1), S. aureus (6), S. epidermis (4), Streptococcus NOS (5), S. pyogenes (2), S. viridans (1),

Non hemolytic strep (Ganma) (2), Beta hemolytic strep (3)

P. aeruginosa: P. maltophilia (1), P. aeruginosa (1), Klebsiella NOS (1), K. pneumoniae (2), Klebsiella-Enterobacter group (1), E. aerogenes (3),

Serratia (1), H. influenzae (1), S. aureus (1), S. epidermis (1), S. pneumoniae (1), Streptococcus NOS (1)

S. pneumoniae: Pseudomonas NOS (1), P. aeruginosa f4), Klebsiella NOS (2),

E. aerogenes (1), S. aureus fl), S. epidermis (2), M. polymorpha (1), S. viridans (9), S. faecalis (3), Non hemolytic strep (1)

Salmonella NOS: Shigella NOS (1), S. typhi (3), S. paratyphi (1), S. choleraesuis (2), Citrobacter group (1), E. cloacae (1), P. vulgaris (2),

P. mirabilis (2), Mima (1), S. viridans (1), S. faecalis (3), Bacillus NOS

(1)

'\

5.4.5. Medicare Hospital Laboratories

The Medicare Ffospital laboratories most frequently misidentified the S. faecalis

organism, and most frequently identified the P. aeruginosa organism, as did the other labora-

' tory groups. Similar to the Doctors' Office data, the "ledicare Hospital laboratoiy data

indicate that a significant decrease in the number of misidentifications occurred from

shipment 1 to shipment 2, especially with regard to S. faecalis where the percentage of misidentifications decreased from 451 to 27%. Further review of the data revealed that eight

laboratories which misidentified this organism in the first shipment identified it correctly

in the second shipment. The change in rate was largely a consequence of this improvement in

performance. Tables 5.4.5.1 and 5.4.5.2 further detail the Medicare Kfospital analyses.

Table 5.4.5.1. Medicare Hospital Laboratories

Specimen

ShipmentNunfcer

of labs Correct m's Marginal ID'S Incorrect ID'S

n

0,0 n J_ n J_

S. pneumoniae 1 108 84 78 N/A 24 22

S. pneumoniae 2 91 64 70 N7A 27 30

Salmonella NOS 1 105 75 71 6 6 24 23

Salmonella NOS 2 88 59 67 8 9 21 24

82
Table 5.4.5.1. Medicare Hospital Laboratories (continued)

Specimen ment

Ml imK^iT* of labs Correct ID'S Marginal ID'S Incorrect ID

0 1 n

Klebsiella NC6 L luy 7i uo 9 8 26 24

Klebsiella NOS 2 yu 0 J / U

0O 9 19 21

P. aeruginosa 1 lOo r r r 1 78 23 21

P. aeruginosa 2 91 so 55 29 32 12 13

S. faecalis 1 108 60 55 N/A 48 44

S. faecalis 2 89 56 63 N/A 33 37

Table 5.4.5.2. Misidentifications by Medicare Hospital Laboratories

Shipment 1, S. pneumoniae: Pseudomonas NOS (1), Klebsiella-Enterobacter group (1), H. influenzae (1), Staphylococcus NOS (1), S. aureus (1). S. epidermis (1),

Neisseria :I0S (2), M. polymoi-pha (1), Streptococcus .\'0S (1), S. pyogenes (1)

S. viridans (9), Non hemol>'tic strep (Gairana) (2), Beta hemolytic strep (1).

Mycobacterium (1)

Salmonella NOS: P. aeruginosa (1), E. coli Cl)» Shigella NOS (1), S. dysenteriae

(1) , S. typhi (5), S. paratyphi B (3), S. typhimurium (1), Citrobacter group

(4), Klebsiella-Enterobacter group (1), Enterobacter NOS (1), E. aerogenes

(2) , P. aerugenoides (1), S. aureus (1), S. faecalis (1)

Klebsiella NOS: E. coli (3), Enterobacter NOS (4), E. cloacae (7), E. aerogenes

(4), E. liquefaciens (1), Proteus NOS (2), P. vulgaris (1), H. influenzae,

Staphylococcus NOS (2)

P. aeruginosa: P. aeruginosa; aberrant strain (1), Alcaligenes (1), E. coli (1),

Edwardsiella (1), Salmonella NOS (1), Klebsiella NOS (4), KlebsiellaEnterobacter group (1), Enterobacter NOS (1), E. hafniae (1), Serratia (3),

Proteus NOS (3), P. vulgaris (1), M. influenzae 0), Herellea (1), S. faecalis (1), B. subtilis (1)

S. faecalis: Pseudomonas NOS (1), P. aeruginosa (2), E. coli (1), S. alkalescens

(2), Shigella NOS (1), Klebsiella NOS (1), Klebsiella-Enterobacter group (1),

Enterobacter NOS (1), S. aureus (6), S. epidermis (7), Neisseria NOS (2),

N. gonorrhoeae (1), N. meningitidis (1), S. pneumoniae (1), Streptococcus

NOS (6), S. pyogenes (2), S. viridans (2), Streptococcus MG (1) , Beta

hesnolytic strep, Group B (2), Non hemolytic strep (Gairana) (5), Beta

hemolytic strep (2)

Shipment 2, Klebsiella NOS: Pseudomonas NOS (1), E. coli (2), Enterobacter NOS (4), E. cloacae (6), E. aerogenes (3), H. influenzae (1), Neisseria NOS (1), S. pneumoniae (1)

S. faecalis: E. coli (1), Shigella NOS (1), S. dysenteriae (1), Salmonella NOS

(2), S. typhi (1), Klebsiella NOS (1), Staphylococcus NOS (2), S. aureus (6),

S. epidermis (5), S. pneumoniae (1), Streptococcus NOS (4), S. pyogenes (2),

S. viridans (3), Non hemolytic strep (Gama) (2), Beta hemolytic strep (1)

P. aeruginosa: Alcaligenes (2), E. coli (1), Klebsiella-Enterobacter group (1),

Proteus NOS (2), H. influenzae (1), Moraxella (1), Staphylococcus NOS (2),

S. epidermis (1), Streptococcus NOS (1)

83
Table 5.4.5.2. Misidentifications by Medicare Hospital Laboratories (continued)

Shipment 2, S. pneumoniae: Pseudomonas NOS (3), Achrcmvobacter (1), E. coli (1), Salmonella

NOS (1), S. aureus (1), N. meningitidis (1), Streptococcus NOS (2), S. pyogenes (2), S. viridans (12), Diphteroids NOS (1), L. monocytogenes (2)

Salmonella NOS: Pseudomonas NOS (1), E. coli (1), S. alkalescens (2), S. typhi

(2), S. paratyphi B (2), S, choleraesuis (1), Citrobacter group (8),

S. marcescens (1), Staphylococcus NOS (1), S. faecalis (2)

5.4.6. Medicare Independent Laboratories

The data for the Medicare Independent laboratories, as shovm in Tables 5.4.6.1 and

5.4.6.2, bear a very strong resemblance to the JCAH Hospital laboratory data. Once again,

the most frequently misidentified organism was S. faecalis, while P. aeruginosa was the

most frequently correctly identified organism.

Table 5.4.6.1. Medicare Independent Laboratories

Specimen ment of labs Correct ID'S Marginal ID'S Incorrect ID'S n \ n n

S. pneunoniae 1 83 70 84 N/A 13 16

S. pneumoniae 2 74 66 89 N/A 8 11

Salmonella NOS 1 80 67 84 4 5 9 11

Salmonella NOS 2 73 57 78 6 8 10 14

Klebsiella NOS 1 83 59 71 5 6 19 23

Klebsiella NOS 2 75 61 82 4 5 10 13

P. aeruginosa 1 80 51 64 17 21 12 15

P. aeruginosa 2 75 51 68 20 27 4 5

S. faecalis 1 81 51 63 N/A 30 37

S. faecalis 2 74 50 67 N/A 24 , 32

Table 5.4.6.2. Misidentifications by Medicare Independent Laboratories

Shipment 1, S. pneumoniae: Salmonella NOS (2), S. aureus (2), Diplococcus (3), Streptococcus

NOS (1), S. pyogenes (1), S. viridans (3), S. anaerobis (1)

Salmonella NOS: Pseudomonas NOS (1), S. typhi (2), S. choleraesuis (2),

Citrobacter group (1), Proteus NOS (1), P. mirabilis (1), S. viridans (1)

Klebsiella NOS: E. coli (3), Citrobacter group (1), Enterobacter NOS (3),

E. cloacae (3), E. aerogenes (3), E. liquefaciens (1), P. morganii (1),

S. aureus (1), S. pneumoniae (1), Corynebacterium (2)

P. aeruginosa: P. pseudcmiallei (3), P. maltophilia (1), Alcaligenes (1),

Salmonella NOS (1), Enterobacter NOS (1), S. marcescens (1), M. polymorpha

(2), Streptococcus NOS (1), S. faecalis (1)

S. faecalis: Pseudomonas NOS (1), E. coli (1), S. sonnei (1), K. pneumoniae (1)

Proteus NOS (1), S. aureus (2), S. epidermis (2), Streptococcus NOS (2), S. pyogenes (3), S. viridans (5), Beta hemolytic strep. Group B (2), Non

honolytic strep (Garana) (6) , Beta hemolytic strep (3) 84
Table 5.4.6.2. Misidentifications by Medicare Independent Laboratories (continued)

Shipment 2, Klebsiella NOS: E. coli (2), Enterobacter NOS (2), E. cloacae (1), E. aerogenes

(1), E. liquefaciens (1), P. mirabilis (1), S. pneumoniae (1) , Bacillus

NOS (1)

S. faecalis: P. aeruginosa (1), E. coli (1), Salmonella NOS (1), S. aureus (3),

S. epidermis (2), S. viridans (7), Non hemolytic strep (Gamma) (4), Beta

hemolytic strep (5)

P. aeruginosa: P. pseudomallei (1), A. faecalis (1), A. species (1), Enterobacter

NOS (1)

S. pneumoniae: K. pneumoniae (1), S. viridans (5), S. faecalis (1), Beta hemolytic

strep (1)

Salmonella NOS: S. typhi (1), S. choleraesuis (1), Citrobacter group (2),

Enterobacter NOS (1), E. cloacae (1), P. mirabilis (1), S. faecalis (1),

Non hemolytic strep (Gaimia) (1), Bacillus NOS (1)

6. SUMMARY OF RESULTS

This chapter summarizes the study results, presents some conclusions, and states the

limitations of these conclusions.

6.1. Clinical Chemistry

There were no significant differences (at the 95% confidence level) among the average

clinical chemistry results obtained by the groups participating in the study. The interlaboratory consistency ("interlaboratory precision") of the laboratory groups can be exhibited

as follows, where groups joined by the same line did not exhibit significantly different

precision at the 90% confidence level.

Rank Order

Most Precise

Least Precise

Laboratory Group

Medicare Independent

Interstate

JCAH

AAFP/ASIM

Medicare Hospital

The techniques used had a consideraible effect on the accuracy and precision of reported

clinical chemistry analyses. Table 3.5.9.2 on page 46 lists the techniques which were most

satisfactorily applied and the percentage of the participating laboratories which applied

each technique. As the table inplies , a large percentage of the participating laboratories

are using outmoded techniques. In most instances, automated methods were applied with equal

or better average accuracy and considerably better precision than the corresponding manual

methods. Results reported by laboratories using diagnostic kits were consistently less

precise than other determinations.

Medical usefulness of the clinical chemistry cinalyses was assessed using the criterion

that analyses should be sufficiently precise to permit the interlaboratory monitoring over

time of the variation in an individual patient's constituent concentrations. This criterion

was not applied to total bilirubin because data on individual bilirubin variation was not

available. Of the remaining seven constituents, only cholesterol was analyzed by the study

participants with sufficient precision to satisfy the criterion. In contrast to the performance of the study participants, reference laboratory analyses of cholesterol, uric acid, urea

nitrogen, sodium and total protein were all sufficiently precise to permit interlaboratory

monitoring of individual variation. Those participating laboratories using the best applied

techniques also achieved acceptable interlaboratory' precision in analyses of these five con- stituents.

85
6.2. Target Value Surrogates

In general, manufacturer's average assays appear to be acceptably accurate pooled serum

target values, but these assays are overly reliant on human factors. As a result, when mean

reference assays are available, they appear to be a better choice for target values.

6.3. Hematology

As with clinical chemistry, the average hematology results obtained by the participating

groups did not differ significantly at the 951 confidence level. The interlaboratory

precision of the laboratory groups can be exhibited as follows where groups joined by the

same line did not exhibit significant differences at the 90% confidence level.

Rank Order Laboratory Group

Most Precise Medicare Independent

JCAH

Interstate

Medicare Itospital

Least Precise A\FP/ASIM

Table 4.1.6.2 on page 69 shows the best applied techniques and the percentages of participating laboratories using these techniques.

6.4. Microbiology

In microbiology, the performance of the Interstate group was significantly better than

the performance of the other groups at the 95^ confidence level; 7.6% of the Interstate

laboratory determinations were incorrect while 19.9% of all other determinations were

incorrect. However, even a 7.6% misidentification rate is not satisfactory because the

organisms selected are common and should be easily identifiable. Most worrisome are such

misidentifications as Neisseria NOS, N. gonorrhoeae or N. meningitidis for the pure culture

of Streptococcus faecalis.

The relative performance of the laboratory groups can be portrayed as shown, where the

lines join groups whose performance did not differ significantly at the 95% confidence level.

Rank Order Laboratory Groups

Best Performance Interstate

1 JCAH

Ntedicare Independent

1

1 A/\FP/ASIM

Worst Performance 1 Medicare Hospital

6.5. Conclusions

The data indicate that high volume laboratories may be more proficient than smaller

laboratories, such as those which serve Doctors' Offices and Medicare Certified Hospitals.

In microbiology, 7.6% of the Interstate laboratory' determinations were incorrect, while 16.5%

of the determinations by other large laboratories (JCAH and Medicare Independents) were

incorrect. Thus, it would appear that the CDC proficiency testing program has considerably

improved the microbiology performance of the enrolled laboratories (although further improvement is still needed). Conversely, clinical chemistry and hematology analyses by the Inter- state laboratories were no better than comparable analyses by other large laboratories, many

of whom do not engage in routine proficiency testing programs. This seems to indicate that

the CDC proficiency testing programs in clinical chemistry and hematology have had relatively

little effect upon the performance of laboratories participating in the program. This

86
conclusion is further substantiated in a companion report.^' It is particularly important

to improve the effectiveness of these programs because the interlaboratory consistency of

study participants with respect to clinical chemistr)' and hematology was too often insufficient to support monitoring of an individual's constituent concentrations over time. It appears that poor selection of techniques is an important factor in the low rate of

acceptability of laboratory determinations.

6.6. Limitations

It must be clearly uj^derstood that the results of this survey are limited by four

important considerations

(a) Because all of the laboratories participated on a purely voluntary basis, no straightforward extrapolation can be made to tlie larger universe of unsampled

clinical laboratories.

(h) It is probable that the results of this study do not represent routine laboratory

performance for two reasons: (1) a laborator>' probably would not volunteer if its

management felt that to do so would be disadvantageous, and (2) the sample

materials probably received special attention in many of the smaller laboratories

which were unfamiliar with analyzing proficiency test samples.

(c) The clinical chemistry test specimens were prepared by a dialyzation process which

removes naturally occurring substances. .'\s a result, the accuracy of^ some methods, as applied to the test specimens, might differ from their accuracy in

analyses of human serum.

(d) The true constituent concentrations of cholesterol and the hematology constituents

could not be exactly determined. For these constituents accuracy was assessed

relative to the mean reference laboratory assays.

7. RECOWENDATIONS AND THEIR IMPLBENTATION

It is beyond the scope of this report to provide a complete plan to remedy existing

problems with laboratory performance. However, it is possible to recommend several of the

elements of such a plan, particularly with regard to proficiency testing.

7.1. Recommendations

(a) Satisfactory performance in a microbiolog>' proficiency testing program conducted

under the auspices of either Federal or other approved authorities should be a legislative requirement for all clinical laboratories analyzing microbiological

specimens, regardless of their inter- or intrastate status. It is unclear whether

this recomnendation should be implanented through new legislation or a re- interpretation of existing legislative authority.

(b) An experimental study should be undertaken to determine a better design for

proficiency testing programs in clinical chemistry and hematolog)'. .^n empirical

description of the causes of inadequate laboratory work should be used in defining

alternative testing strategies for consideration. This study should deal with

such questions as frequency of sampling; feedback to participants; number of

levels at which to test; long-term monitoring of intralaborator>' variability;

follow-up procedures on outlier values; and the criteria for scoring, ranking or rating laboratory performance and its medical usefulness. Until the results of

this recommended study become available, it does not appear justified (or war- ranted) to alter the frequency of CDC proficiency testing in clinical chemistry

and hanatology.

^^Clinical Laborator>' Performance .'\nalysis Using Proficiency Test Statistics, NBS Report,

NBSIR 73 197, 1973 (to be published).

87
(a) Through the cooperative efforts of the Federal Government, State and local

agencies, professional societies and accreditation agencies, a concerted effort

should be made to induce laboratories to use the best available analytical method/

systan combinations,

(d) In those instances where routine proficiency testing in one of the quantitative

specialties is maintained, target values and zones of acceptable perfomance should

be so constructed as to be consistent with generally accepted statistical concepts

in quality control and sampling theory. This would include clearly defined rules

for deletion of outliers and the use of appropriate standard deviations for the

construction of confidence intervals or tolerance limits.

7.2. Implementation

Implementation of two of the recommendations enumerated in Section 7.1 may be facilitated

if DHEW requests additional authority.

(a) The licensing authority of the Clinical Laboratory Improvement Act of 1967 could

be extended to include those laboratories which are presently exempt but whose

normal workload includes the identification of microbiological specimens. An

alternate means of insuring the desired level of laboratory' performance is to identify and accredit proficiency testing programs currently being operated by

State or local governments or professional societies. Under this alternative, the

Federal Government should maintain the authority to withhold or withdraw the license

of any laboratory whose demonstrated f)erformance in the identification of microbiological specimens does not at least meet existing minimum standards under

CLIA '67.

(b) A Technical Advisory" Committee on Clinical Laboratory Procedures consisting of

government and professional society representatives should be established to identify the most accurate and precise analytical methods available and encourage

their use by the largest possible number of clinical laboratories. Zones of

acceptable performance for proficiency testing should be constructed in a manner which reflects the variability associated with the more accurate and precise

- methods and systems. In this way, failure to accept the reccsnmended procedures

would increase the risk of unacceptable performance ratings. It may be useful for

this Committee to have legislated interfaces with Federal regulatory programs.

88
.^pp^vnICIs

Appendices A, B, and E contain report backj^round information and references. Appendix

C reports an inconclusive investigation of possible differences in clinical chemistry

technique performance in analyses of serum pool and dialyzed specimens. Appendix D displays a series of graphs which were discussed in the body of the report.

Appendix A. Scientific Advisory Coimiittee for Study of Clinical Laboratory Capability

Roy Bamett, M.D. Chief, Pathology Department

Norwalk Hospital

Howard L. Bodily, Ph.D.

•Assistant Director and Chief

of Laboratory Services

California State Department

of Health

Joseph H. Boutwell, Ph.D., M.D. Chief, Licensure and Development Branch

Center for Disease Control

Harvey R. Gralnick, M.D. Chief, Hematology Service

Clinical Center

National Institutes of Health

James D. MacLowry, M.D. Qiief, Clinical Pathology Division

Clinical Center

National Institutes of Health

W. Wayne Meinke, Ph.D.

Chief, ;\nalytical Chemistry Division

Institute for Materials Research

National Bureau of Standards

Keith IVeikel, Ph.D. (Chairman)

Office of the Msistant Secretary

for Planning and Evaluation

Department of Health, Education and

Welfare

Donald S. Yomg, Ph.D., M.D. Chief, Clinical Chemistry Service

Clinical Center

National Institutes of Health

y^pendix B. Reference Laboratories for Study of Clinical Laboratory Capability

Norwalk Hospital

ATTN: Dr. Roy N. Bamett

Norwalk, Connecticut 06556

California State Department of Health

ATTN: Dr. Howard L. Bodily

2151 Berkeley Way

Berkeley, California 94704

Center for Disease Control

ATTN: Dr. Joseph H. Boutwell

Atlanta, Georgia 30333

Hartford Hospital

ATTN: Dr. George W. Bowers

Hartford, Connecticut 06115

University of Oklahoma Nfcdical Center

ATTN: Dr. Kurt Dubowski

P. 0. Box 26901

Oklahoma City, Oklahoma 73104

University of Kansas Medical Center

ATTN: Dr. I^ssell Eilers

Rainbow Boulevard and 39tfi Street

Kansas City, Kansas 66103

University of Alabama Medical Center

ATTN: Dr. John W. Foft

619 South 19th Street

Birmingham, Alaliama 35233

Colun^iia-Presb>i:erian Hospital

ATTN: Dr. S. Raymond Gajifcino

630 West 168th Street

New York City, New York 10032

National Institutes of Health

ATTN: Dr. lUirvey R. Gralnick

Clinical Center

Bethesda, Maryland 20014

University of Washington Hospital

ATTN: I>r. Alex Kaplan

Chemistry Department

Seattle,' Washington 98105

State Laboraton' of ffygiene ATTN: Dr. Ronald Laessig

437 Henr>' Mall

Madison, Wisconsin 53706

89
University of California

San Francisco Nfedical Center

ATTN: Dr. fl. Loken

Clinical Laboratory

San Francisco, California 94122

National Institutes of Health

ATTN: Dr. Jajnes MacLowry

Microbiology Department

Clinical Center

Bethcsda, Mar>'land 20014

Illinois Department of Health

Division of Laboratories

ATTN: Dr. Robert Nlartinek

Laboratory Evaluation Section

Chicago, Illinois 60612

Jackson Memorial Hospital

ATTN: Dr. Thomas Noto

1700 NW. 10th Avenue

Miami, Florida 33136

Appendix C. Differences in Laboratory Performance Klicn /Xnalyzing Dialyzed

and Serum Pool Specimens

In Section 3.5 of the main report, the conclusions regarding method performance were considerably weakened by the inclusion of a caveat about possible nonrepresentativeness of

results obtained with dialyzed test specimens. In this Appendix, the representativeness

of such results will be investigated. The question of interest is whether some methods

perform differently in analyses of dialyzed and serum pool specimens. To siirplify the

ensuing discussion, it is convenient to define the relative bias of the technique mean with respect to the target value as fx - T)/T where x is the technique mean and T is the

target value. Using this definition, the hypothesis to be tested can be stated as follows:

the relative bias of the technique mean of the dialyzed detenninations with respect to the

target value for the dialyzed specimen will not differ in a statistically significant manner from the relative bias of the serum pool teclinique mean with respect to the target

value for the pooled serum specimen.

For each constituent, a t-test can be used to cx;unine the statistical significance of

the difference between the serum pool and dialyzed relative biases. However, there are two difficulties associated with defining the appropriate test statistic. The first difficulty

arises because the levels of some constituents in the pooled serum specimens are very

different from the levels in the dialyzed specimens. For example, the dialyzed normal

bilirubin sajnple has more than twice the concentration of the serum pool normal sample.

Since most laboratory methods for bilirubin are sensitive to changes in concentration, it

would be unrealistic to expect all methods to have the same relative bias in analyzing both

samples

The second problem which arises is in the choice of a target value. Several possibilities exist: the mean laboratory analyses fMI-A) , the mean reference analyses (MRA) , the mean analyses by the techniques which were best applied in the dialyzed analyses (M'\BAT)

the manufacturer's average assays f MAA) , and the ueigli-back values (Iv'BV') . Two of these

possibilities, the MRA's and WBV's, were immediately rejected because they were not available

for the pooled serum specimens. There is no readily apparent rationale for a choice among

the remaining possibilities. However, a criteiion can be established which may reduce the

number of possibilities: any target is unsat is factors' if its use leads to the conclusion

that a substantial number of techniques performed differently in the first and second

hematology shipments. This appears to be a logical criterion because the same process was used to prepare the two hematology batches. Thus, there should be no differences in the

bias of technique means for the two batches if the target value is properly chosen.

William Pepper Laboratory' ATTN: Dr. Royden Rand

3400 Spruce Street

Philadelphia, Pennsylvania 19104

C^'orgetovvn University Hospital

ATTN': IH-. Martin Rubin

3800 Reservoir Road, N^V. Washington, D.C. 20007

Yale-New Haven Hospital

ATTN: Di-. David Seligson

"^89 Howard Avenue

New Haven, Connecticut 06504

National Institutes of Health

ATTN: Dr. Donald Young

Clinical Chemistry Service

Rethesda, Man-land 20014

90
Application of this criterion to the throe available target choices was rather disconcerting. None of the targets satisfied the criterion. Table ('1 shows the nuiriier and percentage of technique mciins (for a normal or al)nor7nal analysis) with relative biases which

differ significantly at the 901 level of probability for the two shipments of hcmatolog>'

specimens

Table CI Number and Percent of Techniques

with Significantly Different Relative

Biases for Various Targets 1

Target » of

Techniques

" of Biased

Techniques

1 of Biased

Techniques

MLA 46 18 39

MARAT 18 30

MAA 1 46 28 61

Ml of these percentages exceeded 351. Furthermore, a high rate of technique bias occurred

in analysis of each constituent. Tlius it would appear that the data from the two shipments

were not sufficiently consistent to support the analysis initiated in this Appendix, and

there is no choice but to label the results as inconclasive. For reference purposes, basic

data on the serum pool specimen analysis ref)orts is displayed in Table C2. The table con- tains spearate entries on sainple sjze (n) , mean fx), and standard deviation fs) for each

teclmique ased by 15 or more study participants.

91
CO

(J

(DP.

c/r

§1

•HI

H

I X

UO fsi r-- 0-. O O «»• T O rvj LO I—

O 00 -(

rr

00 rsi (—I O O r-t ^ OO d rsl 00 to .—

to VO 00 to r-1 r-( to O O O t-H o o o o

,—I -It r-- TJ- to rj- rj rj r-4

I—I rj t—I I—

00 r-l r—( 00

"Tj- LO

t-H to to Tt

f^j rj (Vi rsi

r-. CTi

•-r rfO \D CTi to IT) to

cr> »^ to to t^; 'S- n

LO vO O <T>

t-H r-l lO rj

I-H T OO

r to ^

to .—

1^ r~-

\0 i-l \0

1—I to TJ- CTi

CTl CTl C OO

to Csl (Vj ^

LO LO LO LO

r-- cxi r-- (^j ^ f—

o o <*

r~ to f^i

o to

LO r4 to O

CTl LO LO

to 00 to to

to CM cn

I—( r^j (^1

J- I r-(

1—1 O O I—

o o o o

I—I LO

.—I \0

00 Tt to to

to n Lo o o o o to

r~ to rsi ,—( to o o o o

00 O r-t 00 00 O LO to O

o o O t-l

(Nl ,—I LD

00 00 OO OO

rr r~- 'S- o o o

to to O T

LO LO LO LO to

rJ O rH OO

VO O VO LO

fvl t-^ 00

to LO to cn to r-j to to -rf r J LO cr> o 1—( t—t to or-^ to

lO o uo vO 00 rj 'S- r-(

l-H O LO

1^ to r-4

o

^10}

< i-l O rt C ^ J

•H C O

c3 T) 2

u d (-. •H rH (U

V-i o sx

4) I O O u

'a3 O

D (LI X X

o o

c c

.o

^. ^.

<D O

X u

o n c (u o

u

i)

t-l xo

nj I 1^

§ X X

'^-i r-t .—

o o

*-> u u

o

e c c

r-H O

>.'-l

+J 4-> -M 01 o o <K x; u u u *-> rC CT3 •H -r-l -rH OJ Q a Q m

U 0) O -H 3 eg c

x: x: eg

•<-( r-< r-t 01 o o s-

(J </l 1/1 <l> r-H 0) a> x;

03 t-i U +-1 'J u o

o

4-1 oX

Q Q

o M »-' .M U

0) 0) (U n] ^ !-> ^ (h 3 d

•H •r^ T-t (U pa OQ PQ Oi

c

c (UOA e o rH

rH H •H

4-1 1> OQ W w z o •H rH o o CJ rt 3 <u rH *-> r-l (- CTJ o 3 U I—

cH

rH <U o 4-) i- -a o

<u

*-> rH

iri a> rH

rH u •r-l •-H 4JO

u IT) H

92
(\ppendix [). Medical Usefulness Plots

This Appendix contains the plots used in c-stimating the probability that a laboratory

in any given group would "correctly" assay one of seven clinical chemistry constituents.

Details on the estimation procedure are given in Section .3.6.2 (page 49).

Each page displays plots for a single constituent. Table Dl gives a key to the group

numbers on the plots and the corresponding group names used in the report. On each plot,

sample 1 is the normal sajnple and sample 2 is the abnormal sainple. Fach point indicates

the report of a single laboratory. Points beyond the scaled axes of the graphs are indicated in two manners. In cases where both the nonnal and abnormal values are off- scale,

an arrou- and two numbers arc shown. The first number gives the value of the reported nonnal

analysis and the second gives the value of the abnomial analysis. If only one value is offscale, an arrow and a nunt)cr appear. Tlie tail of the arrow lies on the on-scale value and

the head points in the direction of the off- scale value.

Table 01

GroL^ Number Group Naine

1 MIP

2 .IS I VI

3 Interstate

4 .JCMl

5 Nfcdicare Hospitals

6 Medicare Independent Labs

93
CLUCosr

(n^./lOO ml.

Requests for larger copies of these plots should be addressed to Public Policy and Program Analysis Area

Division 431.00, National Bureau of Standards

Building 220, Room 7^60

Washington, D. C. 20234
UREA NITROGEN

(mg./lOO ml.)

Hefinac* Uboritorm

» 12 14 16 II » 23

Group 1

•

•

1 ii

;

1—

: .| •

•

•

•

•

•

•

13

10 12 14 16 la 20 22

• 26

10 12 16 18 20 22

4—j- • t

95
CALCIUM (S./lOO Bl.)

19 U

Cr*^ 1 > 1 1} K.I

1 • « ^ ^^^^ r

-

« >

-

t

1

i

•

•

«.f.l.J ^

111

si —

•

t_ r "J

• • •/».

• •

• •

•

•

•

1 i-3

aojp 4

T 111

f

• ^

* •* •

• • • • •

•

•

>.».i.i *

^.

• ll.l.ll.a :

• •

•

•

•J,T.l ^

!.•,».« '

It II 10 11

l« II

96
CHOLESTEROL

ui ISO i*\ la m iM m

•«

•

wmr 1 • 1 IM in M* IB

4

•

. 1

•

»» « <

1 « • » • IT « »

t

• h

I

US 110 MS 1&9 in

lU in lU IM ITS 1« ut ut us iH in m m

ii> ill i<>

• • •

' .1

ii> » i» IM m i«t «

97
PRIC ACID

ris.i

S.S «.t

«

'i I.*

• f

•

•

• •

• •

•

s s I. M l.« I.I

•

•.I. 11 s

•

•

• •

• •

• •

•

• •

•

•

•

•

1 •

1 •

•

• •• •

••

• t

• •

•

• •

•

*

s

• .1 . *t

I.I < • f 1 11

1 X

I

• I

^ •i.s • 1 a ^

^« '. U S

,^•10. 12 so

4.M. a.e ^

I 4.*e. t.M^

It t.i

11 M • ixi li.T.

•

•

• •

•

•

« •

• •

•• •

•

•

•

• •

•

• • • •

• •

1.St

• 4.

• It •-1 * • M • • l.t

4.4. l.t

• • •

If

t.> It «l T« T.I l.c •»

9a
SOCiUM

99
TOTAL PKOTl IN

JOO
Appendix E. References for Tables 3.8.1 and 4.2.1

CDC Analyses - P. Finkel and J. Rcwen, Clinical Laboratory Performance Analysis

Using Proficiency Test Statistics , NBS Report 10 685, 1972.

CAP 1949 and 1969 - L. Skendzel , D. Hanson, and W. Civin, "The 1949 College of

American Pathologists Survey Revisited," American Journal of Clinical Pathology , Vol. 54, No. 3, September 1970, pp. 493-495.

3. AAB - various including American Association of Bio-Analysts, 1971 Programs

Proficienc

October 19

Proficiency Testing Service , 1970 and the AAB publication Bulletin , Vol. 13, No. 10,

4. Canada - D. Tonks , "A Study of the Accuracy and Precision of Clinical Chemistry

Determinations in 170 Canadian Laboratories," Clinical Chemistry , Vol. 9, No. 2,

April 1963, pp. 217-235.

5. Scandinavia, Australia - J. Stromme and L. Eldjom, "Surveys of the Routine Work of

Clinical Chemistry Laboratories in 116 Scandinavian Hospitals," Scandinavian Journal

of Clinical Laboratory- Investigation , Vol. 25, 1970, pp. 213-222.

6. New Zealand - F. Desmond, "A Clinical Chemistr>' Proficiency Survey," New Zealand Nfedical

Journal , Vol. 63, 1964, p. 716.

'

7. LA Physicians - I. Schoen, G. Thomas, and S. Lange, "The.Quality of Performance in

Physicians' Office Laboratories," American Journal of Clinical Pathology , Vol. 55,

No. 2, February 1971, pp. 163-170.

8. Italy - R. Saracci, "Factors .Effecting Accuracy and Precision in Clinical Chemistry,"

American Journal of Clinical Pathology' , Vol. 52, No. 2, .'Uigust 1969, pp. 161-166.

9. CAP 1965 - J. Koepke and J. Eilers, "Survey of Red Cell Parameters in 1600 United

States Hospitals," Bibliotheca Haematologica , Vol. 24, 1966, pp. 137-144.

10. American Society of Clinical Pathologists - W. Sunderman, "Status of Clinical

Hemoglobinometry in the United States," /\merican Journal of Clinical Pathology , Vol. 43, No. 1, January 1965, pp. 9-15.

11. CAP 1972 - R Gilbert, The 1971 Chemistry Survey Program , College of American

Pathologists, 1972.

I.S. fjcn tMN.MKNI CRINT INC: OtFICK I 9' 3 — I'l? - 65 ? /9fi

101
FOH'/i Ne$-1I4A 11.711

1 us DEPT. OF COMM. I. PUBLICATION OR BE POR I .\0. 2. Gov t Acctmoo

BIBLIOGRAPHIC DATA ^^^^

3. Recipient s Accessioo No.

4. TITLE AND SUBTirLE

A Prn-f1 ri fnrv Tp<>t Assessment of Clinical Laboratorv ranahilitv

in the United States

5. Publication Date

6. Performing Orgaoizstion Coce MSIR 73-163

7. AUTHORtS' Peter W. Finlcel and Ted R. Miller

8. Peiformiog Otgaaizatioo

/

9. PERFOR.MiNG ORGANIZAT'ON NAME AND .ADDRESS NATIONAL BUREAU OF STANDARDS

DEP.^RTMENT OF CO.<LMERCE WASHINGTON, D.C. 20234

10. Project/Task/'Work Unit No. 4314457

y ^» Oontfftct/Orsnt No*

12. Sponsoring Organization Nime and A ^dress Division of Health revaluation

Office of tlie Assistaiit Secretary for Planning, and Evaluation

Department of Health, Education, and Welfare

Room 5526 HBV North Building, V.'ashington, D. C. 20201

13. Tvn^ of R^nnrr St Pr»r<rt.'l Coveted Final 3/71 - 4/73

14. Sponsoring Agency CoJe

15. .SUPPLEMENTARY NOTES

• 6. ABSTRACT i A 200-wort! or less (actua. su.tunary of most sijrnif ic ant inforira. ion. If Jc'Cument includes a significant bibliography or literature -.uivcy, menrionic hero

The proticiency of a selected sainple of physician, hospital a.nd independent laboratories

was assessed uith respect to their ability to analyze clinical chemistr/ and hematology

samples and to identify microbiological organisms. For the assessment of clinical

chemistry- aiiJ hematology proficiency, the laboratories were grouped and determinations

of group accuracy and group precision were made. Further analyses were performed to

determine relative accuracy and precision of the techniques presently applied by these

groups. There was no significant difference at the 9SI confidence level in the accuracy

achieved by the various laboratory groups involved u< clinical chemistry and hematology

analysis. In clinical chemistry, the Medicare-Certified Independent Laboratories, CDC

Tested Laboratories and JC/'Jl-Members generally proved more precise tlian Physicians'

Office and Medicare-Certified Hospital Laboratories. Itowever, none of the laboratory

groups were sufficiently accurate to permit the monitoring over time of variation in an individual patient's constituent concentrations. It would appear that poor selection

of techniques was an important co;itributor to tliis low performance level. In hematology

the Physicians' Office Laboratories proved to be the least precise of the groups. There

was no noticeable difference in precision between participants in the CDC proficiency

testing program and non-participants. With respect to microbiology, 7.61 of the

identifications by laboratories participating in the CDC testing program were incorrect,

while 19.41, of all other identifications were incorrect.

17. KF.Y WORDS 'A!phabnical order, separated b/ scmcolonb)

Accuracy; clinical chemistr>'; hematology; medical usefulness; microbiology; proficiency

testing

18. AVAILABILITY STATEMENT :9. SECURITY CLASS

(THIS REPORT)

21. NO. OF PAGES

XH UNLIMITED. UNCL AS.SIFIED FOR OFFICIAL DISTRIBUTION. DO NOT RELEASE

TO NTIS.

20. SECURITY CLASJ

(THIS PAGE)

UNCLASSiFIED

22. Price

01C0MM-0C eS244>P7l