NBSIR 73-197

Clinical Laboratory Performance

Analysis Using Proficiency

Test Statistics

Peter W. Finkel, John W. Rowen

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

June 1973

Final Report

(Supersedes NBS 10 685 - PB 213-287/6)

Available NTIS - #C0M73-11253 (53pages)

revised December 1973

Prepared for Division of Health Evaluation

Office of the Assistant Secretary for Planning and Evaluation

Department of Health, Education, and Welfare

Washington, D. C. 20201

NBSIR 73-197

CLINICAL LABORATORY PERFORMANCE

ANALYSIS USING PROFICIENCY

TEST STATISTICS

Peter W. Finkel, John W. Rowen

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

June 1973

Final Report

(Supersedes NBS 10 685 - PB 213-287/6)

Prepared for Division of Health Evaluation

Office of the Assistant Secretary for Planning and Evaluation

Department of Health, Education, and Welfare

Washington, D. C. 20201

U. S. DEPARTMENT OF COMMERCE, Frederick B. Dent, Secretary

NATIONAL BUREAU OF STANDARDS, Richard W. Roberts. Director

Executive Summary

This report summarizes the first part of a two-part study performed

for the Division of Health Evaluation, Office of the Assistant Secretary

for Planning and Evaluation, Department of Health, Education, and Welfare

by the Technical Analysis Division (TAD) of the National Bureau of

Standards. The study reviewed the first two years (July 1969 to October

1971) of a proficiency testing program for laboratories licensed under

the Clinical Laboratory Improvement Act (CLIA) of 1967 on the basis of

data collected under the regulatory authority of the Center for Disease . Control (CDC).

The major study effort was associated with statistical analyses of

the data to:

(1) Evaluate the temporal effect of continued participation in the

CDC proficiency testing program.

(2) Describe the population of licensed (interstate) laboratories

in terms of various profile descriptors.

(3) Determine, through one-way analyses of variance, whether the

mean differences associated with the various levels of each

independent variable (excluding number of employees) were

significantly different.

(4) Contrast the relative performance of licensed and volunteer

laboratories

(5) Detemine if bias is introduced by pooling various analytical

methods in calculating the reference laboratory sample target

value.

Statistical analyses were performed on two sets of data supplied by

CDC, the agency within the Public Health Service (PHS) responsible for

administering the CLIA '67 program. The first set, covering 268 licensed

clinical laboratories, contained the analytical results (raw data, not

scores) of proficiency tests conducted in clinical chemistry and hematology. Also contained in this set were the results of laboratory testing

for several hundred other clinical laboratories who participate in the same proficiency tests but who are not licensed under the CLIA program

(volunteer and reference laboratories).

The second set of data contained profile information on 237 of the

268 licensed laboratories, including a record of the number of full-time

employees, the highest academic achievement level of the working supervisors, and the total number of accreditation or licensure programs in

which the laboratory was involved. Corresponding profile data were not

available for the volunteer or reference laboratories.

The temporal effect was evaluated through pooling the data over realtijne. In this manner, laboratories were grouped according to their

absolute point on the learning or experience curve. Inference to the

performance of individual laboratories was not considered, nor was it ever intended to be a part of this study. Also inherent in the analyses

ii

was the assumption of methods consistency independent of the level of

each constituent. The major findings of the analysis are as follows:

1. It was not possible to discern meaningful changes over the twoyear period in the overall level of licensed or volunteer

laboratory performance as measured by the mean- accuracy approach

(average relative difference from target value established by

CDC reference laboratories) for any of the 13 constituents

analyzed.

2. The variability among the licensed laboratories as measured by

the standard deviation has decreased over the two-year period

under study. This is an indication that the group performance

has become more consistent. The same remark is true for the

volunteer laboratory group as well, where the improvement was

often much more marked.

3. Of the four independent variables studied, including Accreditation, State Program Intensity, Supervisory Education, and

Analytical Method/Laboratories, only the last one appeared to be

significant. That is, the general level of capability was about

the same whether laboratories were or were not involved in proficiency testing programs other than CDC; whether they were

located in states which did or did not have proficiency testing

programs of their own; or whether the working supervisor had a

B.S. , M.S., Ph.D., or M.D. degree. Only the Analytical Method

used (or the difference among laboratories) appeared to be significant (p < .001).

4. The data do not appear to support an argument in favor of

establishing reference group target values on a method-by-method

basis. More data, collected through the use of a statistically

designed experiment which would avoid confounding analytical

methods with laboratories by having each laboratory use several

methods, would be helpful in reaching more rigorous and definitive

conclusions

The Clinical Laboratory Improvement Act of 1967, administered by

the Center for Disease Control, provides for a many- faceted interlocking

regulatory program. This study has looked at only one aspect of the

CLIA '67 program, namely proficiency testing. Not included in the study,

but also of major importance, are such functions as on-site laboratory

reviews, training, written critiques in the qualitative specialties, and

method development. There are few indications of meaningful changes in

the average level of performance as measured by proficiency testing

results since the licensed laboratories entered the CLIA proficiency

testing program. Consideration should be given to alternative sampling

methods, such as reduced or skip-lot sampling, for those constituents

which appear to present no analytical challenge to the licensed and

volunteer laboratories. Greater emphasis might then be placed on those

constituents which give the laboratories the most difficulty (cholesterol

and creatinine, for example).

iii
This report includes a review of the legislative intent behind the

law (CLIA '67) and an annotated bibliography of pertinent literature

in the field of laboratory proficiency testing for the years 1963 to 1971.

iv
Contents

Page

Executive Summary ^

1. Introduction 1

2. Background 2

2.1. Proficiency Testing Programs 3

3. Study Methodology 5

3.1. Inputs to the Analysis 5

3.2. Method 5

4. Analytical Results 7

4.1. Temporal Effect 7

4.1.1. Interstate Laboratories 8

4.1.2. Volunteer Laboratories 9

4.2. Profile Information: Licensed Laboratories Only 9

4.2.1. Additional Program Involvement 9

4.2.2. Number of Full-Time Employees 9

4.2.3 Number of Supervisors per Laboratory 10

4.2.4. Supervisor Academic Achievement 10

4.2.5. Academic Achievement of Supervisors Within

Laboratories 11

4.3. Analyses of Variance 12

4.3.1. Accreditation 12

4.3.2. State Program Intensity 12

4.3.3. Supervisory Education 13

4.3.4. Analytical Method/Laboratories 13

4.4. Method Bias in Determining Target Values 13

v
Contents (continued)

Page

5. Summary and Conclusions 15

5.2. Conclusions 15

Appendix A. A Review of the Legislative History of CLIA '67 ....... 17

Appendix B. Abstracts 21

Appendix C. Temporal Effect of Continued Participation in the CDC

Proficiency Testing Program 35

Appendix D. Measuring Laboratory Performance in the CDC Proficiency

Testing Program 48

Appendix E. Mean Measures of Relative Performance 50

vi
Clinical Laboratory Performance Analysis

Using Proficiency Test Statistics

Peter W. Finkel

John W. Rowen

The proficiency testing aspects of the Clinical Laboratory

Improvement Act of 1967 program was assessed. The overall

ability of licensed or volunteer laboratories to accurately

determine mean values for any of the 13 constituents was not

significantly altered during the first two years of program

operation. However, the variability of the laboratories has

decreased over the two-year period. It appears that the

program has increased the consistency of laboratory performance.

The general level of laboratory capability seemed to be

independent of involvement in state -supported or voluntary

proficiency testing programs other than the CDC program and

of whether the working supervisor had a B.S. , M.S., Ph.D., or

M.D. degree. Choice of analytical method did significantly

affect performance. Although insufficient evidence was available to make a definitive statement, the data do not

appear to support arguments favoring establishment of methoddependent reference group target values.

Finally, it appears that consideration should be given

to alternative sampling methods, such as reduced or skip-lot

sampling, for those constituents which appear to present no analytical challenge to the licensed and volunteer laboratories.

Greater emphasis might then be placed on those constituents

which give the laboratories the most difficulty (cholesterol

and creatinine, for example).

Key words: Accuracy; CLIA '67; clinical chemistry; hematology;

microbiology; precision; proficiency testing.

1 . Introduction

This report summarizes the first part of a two-part study of clinical laboratory performance in the United States. The study was conducted

by the Technical Analysis Division (TAD) of the National Bureau of

Standards for the Division of Health Evaluation, Office of the Assistant

Secretary for Planning and Evaluation, Department of Health, Education,

and Welfare.

The objectives of this study were to determine if the participation

of interstate (and volunteer) laboratories in the CDC proficiency testing

program:

(1) Has resulted in significant changes in average laboratory

performance as measured by the accuracy (distance from target

1
value) of proficiency testing results for these laboratories

over a two-year period.

(2) Reveals classes of laboratories which are more/less accurate

from the standpoint of proficiency test results.

, (3) Offers insights, conclusions or recommendations pertaining to

the frequency of testing, constituents tested and laboratory

analysis procedures used.

The purpose of this report is to publish the results of statistical

analyses performed on an extensive data base \\^ich was compiled and

maintained by the Center for Disease Control (CDC) , Public Health

Service, Health Services and Mental Health Administration, Department of

Health, Education, and Welfare. The data for these analyses were the

results of proficiency tests administered by the Center for Disease

Control, extending for a two-year period beginning in July 1969; and

various profile descriptors of the laboratories licensed under CLIA '67.

2. Background

Public Law 90-174 (Partnership for Health Amendments, 1967),

Section 5, was enacted to assure consistent performance of accurate

laboratory procedures and services. The law also includes provisions

relating to the application for and issuance of licenses, quality control,

personnel standards setting, and proficiency testing. The legislation

impacts on clinical laboratories engaged in interstate commerce. The

law was enacted in 1967, but was not fully implemented until July 1, 1969.

The Center for Disease Control serves as the regulatory agency for this

legislation.

Clinical laboratories that meet eligibility requirements under the

Clinical Laboratory Improvement Act (CLIA) legislation are licensed by

"procedure or category of procedure" such as: microbiology, serology,

bacteriology, glucose, hemoglobin, etc. After receiving their license

they are thereafter subject to periodic proficiency testing in many of

the licensed areas. These tests are used as one means for determining

the competency of the laboratory staff and the adequacy and quality of

facilities, equipment, reagents and laboratory procedures. Normally,

samples for the proficiency testing program are prepared and mailed to

the licensed laboratory every third month. The analytical results

obtained by the laboratory are subsequently returned to CDC for scoring,

evaluation, and critique where applicable. (Appendix D provides a brief

description of the program.) When a laboratory's performance reflects

lack of proficiency (significant deviation from norms established by the

CDC reference laboratories and/or peer group analysis) corrective measures

are taken by CDC. The extent of the corrective action varies according

to the individual situation. In instances where poor performance occurs

repeatedly, where cooperation is lacking or suggested remedial action is

not taken by the laboratory, suspension or revocation of the license can

result.

2
Appendix A of this report provides a brief summary of events leading

to the enactment of CLIA '67, changes in the administration of the act

which have occurred since its enactment, and details concerning the purpose and sub- objectives of the program as administered by CDC.

2.1. Proficiency Testing Programs

Proficiency testing programs are monitoring systems designed to

provide a basis for ascertaining the reliability of laboratory performance.

These programs serve to prompt the appropriate actions needed to assure

results that are medically useful and interconvertible (the results from

one method can be translated to comparable results from a different

method) . Proficiency testing often involves the analysis of submitted

samples; however, it is important to remember that this is but one com- ponent of any total proficiency assurance program.

Some other objectives, or components, of a multifaceted program such

as CLIA '67 are:

a. To define the "state-of-the-art" in terms of:

(1) Methodology

(2) Reagents

(3) Standards

(4) Personnel performing test

b. To define sources of error in terms of:

(1) Procedure

(2) Education and performance of analyst

(3) Reagent

(4) Equipment (includes improper use of equipment as well as use of improper equipment for an assay)

(5) Monitoring systems

c. To define criteria to substantiate "medical usefulness"

d. To define educational needs and stimulate remedial measures.

The Federal Regulations published on December 31, 1968, clearly

describe CDC's regulatory authority with regard to unacceptable performance by a licensed laboratory in the proficiency testing program. The

reader is referred to the published regulations for a complete description

of this topic, but in brief, it is incumbent upon a licensed laboratory

to maintain satisfactory performance in all proficiency testing programs

in which it is involved. Failure to do so can result in revocation of

licensure or exemption therefrom.

The extent to which the existing proficiency programs do, in reality,

impact on general laboratory performance improvement is still not widely

agreed upon. The difficulty in determining whether proficiency testing

programs have improved performance is due to one or more of the following:

a. Changes in methods of interpretation of the results.

3
b. The effect of new laboratory entries in the program whose

initial results obscure the improved changes of the established

participating members.

c. Technological changes in the laboratory testing.

d. The fact that virtually all proficiency testing programs,

including CLIA '67, require laboratories to analyze samples and

specimens which are identifiable as testing material. The

: . laboratory that knows it is being tested can use more experienced

personnel, better quality control procedures and more accurate

testing procedures for these sample analyses. In some cases,

the licensed laboratory has split the sample received from CDC,

analyzed one portion itself, and sent the other portion to

another laboratory for confirmative analysis. Thus, proficiency

testing can result in measures which are representative of the

best work that a laboratory is capable of instead of its routine

performance.

A review of the literature indicates that it has been extremely

difficult to determine whether these programs have indeed accomplished

the goal of improA/ing laboratory performance. There is a paucity of

data and much of what is available is conflicting. For example, proficiency test data^ from the College of American Pathologists (CAP)

indicates a significant improvement over the last 20 years. Proficiency

testing data obtained by the New York City Health Department^, however,

indicates no significant improvement, albeit for a shorter period of time,

over the years 1964-1968. Other published studies ^ support the New York

City findings by demonstrating the inability of a laboratory to perform

equally well in two different proficiency testing programs.

The aforementioned findings and speculations should be considered

when proficiency testing is suggested as anything other than one of

several indices to be used for laboratory evaluation. Perhaps a more

germane question to ask is whether proficiency testing in combination

with other measurement and control procedures has any utility.

iL. P. Skendzel, D. J. Hanson, and W. H. Civen, "The 1949 College of

American Pathologists Survey Revisited," American Journal of Clinical

Pathology , lIV (1970), 493.

2m. Schaeffer, D. Wedelock, P. S. May, S. Blatt, and M. Wilson, "The

Clinical Laboratory Improvement Program in New York City II, Progress

After Five Years of Experience," Health Laboratory Science , VII (1970),

242.

3r. G. Martinek, et al., "Comparison of Methods for Conducting Laboratory

Proficiency Surveys," Health Laboratory Science , V (1968), 239.

4
3. Study Methodology

3.1. Inputs to the Analysis

The analysis was performed on data compiled and maintained by the

Center for Disease Control. The data were of two types:

a. Profile information for most of the licensed (interstate)

laboratories, including types of accreditation or licensure

programs in which active, state in v^ich located, number of

full-time employees, number and academic training of working

supervisors

.

b. Proficiency test results for licensed, voluntary, and reference

laboratories covering the period of July 1969 through September

1971. These data pertain to the following constituents of human

blood: glucose, creatinine, urea nitrogen, uric acid, total

protein, cholesterol, calcium, sodium, potassium, chloride,

hematocrit, red blood cell count (RBC) and hemoglobin. The

analytical method used by the laboratory is also reported. If a laboratory is licensed in a category (e.g., clinical chemistry)

or subcategory (e.g., cholesterol) for \\diich CDC has developed a proficiency testing program, it receives a shipment of test samples

every three months. The general procedure calls for three samples

per shipment, usually one at each of the high, medium and low

level of each constituent.

In addition to the technical data, information was obtained on the

"strength" of legislation and controls within states in which interstate

and volunteer laboratories are located. An extensive literature search

(Appendix B) was also conducted to gather information for the development

of a suitable analysis design and for the interpretation of results.

3.2. Method

The methods used for analysis of the data were relatively simple,

with the major activities devoted to re- formatting to detect discernible

temporal effects, editing, and augmentation of basic data already compiled

on magnetic tape. Computer programs were developed to group and statistically analyze this information as follows:

a. The raw data (analytical test results) were separated on a constituent-by- constituent (test-type) basis.

b. A target value was calculated for each sample, using the reference

laboratory analyses. In each instance where there were at least

30 reference laboratories reporting results on a method-by-method

basis, the target value consisted of the average analysis for

those 30 or more reference laboratories. When there were fewer

than 30 reference laboratories using one particular method, the

data were pooled and one target value was calculated for the

methods involved. That one target value was then used for all of

the methods which had been pooled.

5
c. The difference between the analytical result reported by a laboratory in the test and the target value calculated from the

reference laboratory data was used as the measure of performance.

The difference was normalized by dividing it by the target value.

For example, a licensed laboratory analyzed a sample of cholesterol by method A and reported 185 mg/100 ml as its result. The

average reference laboratory analysis (calculated from 63 different laboratories all using method A) was 166.5 mg/100 ml. The

difference (185 - 166.5 = 18.5) was divided by the target value;

18.5/166.5 = 0.11, and the result was the "normalized" measure of

performance for that laboratory for that sample.

d. Each laboratory's data file was searched in order to determine

when it first entered the CDC proficiency testing program. This

entry date was designated "time zero" for that laboratory, establising its position in a pseudo-time scale.

e. The normalized difference (performance) data for interstate

laboratories were grouped according to the pseudo-time scale, and

group averages were calculated. That is, all of the differences

representing "first"- analyses of cholesterol were pooled together

(by method of analysis) to give an average difference for "time

zero." The same procedure was followed for each subsequent time

period. This data is summarized in Table C-1, Appendix C.

f . The volunteer laboratory data were treated exactly as was described

in the preceding paragraph, with one exception. Whereas the

first entry into the data bank for almost all of the licensed

laboratories actually represented their first sample analyses

,

there are a small number of licensed laboratories which were

r participating in the CDC proficiency testing program prior to the

enactment of CLIA '67. Hence, their "first" analyses were at a much earlier time, and these records are not part of the CLIA '67

proficiency testing data base. Consequently, even though the

volunteer laboratory data covers the same time frame as the interstate data, it does not represent the same segment of the overall

. learning or experience curve. This data is summarized in Table

C-2, Appendix C.

Target values established by CDC reference laboratories were used as the standard of comparison in this study. No analysis has been done,

however, to test the validity of these target values. It might be informative to point out some characteristics of the reference laboratory group.

For example, a laboratory which is licensed in one or more categories may

serve as a reference (or referee for the qualitative specialties) laboratory in areas in which it is not licensed. Many of the CDC reference

laboratories rank among the leading state health departments and teaching

hospitals. There do not appear to be any prescribed ground rules for

selecting reference laboratories except that they must be willing to

analyze the samples and report the results back to the originating source.

The computer programs used in this study provided the following

outputs

6
1. A temporal analysis of laboratory proficiency for all participating laboratories for each constituent test method. If laboratories were "improving" over time, mean differences from the

estimated target values should have become smaller. If no "improvement" occurred, convergence would not have been noted.

2. An analysis of variance program determining whether performance

(accuracy with which laboratories approximated target values) is

dependent upon laboratory characteristics such as size, accreditation, and/or academic training of the supervisor.

These outputs were analyzed and are presented in this report, together

with conclusions upon which to base future action on the part of the

Department of Health, Education, and Welfare.

4. Analytical Results

4.1. Temporal Effect

Participation in a proficiency testing program may result in changes

in individual laboratory performance. Changes in overall group performance are not as easily discerned. However, there are at least two ways

in which changes among the laboratories might be detected. An analysis

of the general level of accuracy which clinical laboratories achieve as a consequence of continued participation in a proficiency testing program

would provide one indication of change. The "scatter" (standard deviation)

of results obtained among the participating laboratories would be a second indication. As the group of laboratories becomes more homogeneous

the between- laboratory standard deviation will be smaller.

Several assumptions are inherent in the statistical interpretation of

the data. The following assumptions are representative:

1. The distribution of the relative differences between a laboratory's

result and the target value is not functionally related to the

analytical method used, the level of the constituent, or the

constituent itself, except for a possible learning effect l^^ich

may vary from one constituent to another.

2. The normalizing technique employed, whereby the observed differ- ence between a laboratory's result and the target value is

divided by the target value [(0-E) •^ E] , satisfactorily negates

any effect which the level of the constituent might have had on

the difference.

3. The groups of laboratories (the size of i\diich increased by a factor of at least three during the two years under study) are equally representative at each point in time. That is, those

laboratories which entered the interstate licensure program early

in its development do not vary significantly in their general

level of performance from laboratories which entered the program

at a later date. This assumption also implies that early or late

entrance into the program is not associated with the analytical

methods used.

7
Paragraphs 4.1.1 and 4.1.2 discuss measures o£ performance for both

the interstate and volunteer laboratories.

4.1.1. Interstate Laboratories

This phase of the statistical analysis examined the question of

whether participation in CDC's proficiency testing program results in a changed level of performance over time for laboratories in general. The

question was examined on the basis of overall accuracy at various time

points after entry; i.e., by the mean performance among participating

laboratories and by the consistency among the participating laboratories,

at various time points after entry into the program. Improved performance

would be reflected by smaller mean differences and by smaller standard

deviations, respectively, with continued program participation. A linear

regression analysis was performed on these values, for each constituent,

over the two-year period 1969-1971.

The constituents for which quantitative data was available and for

which the regression analyses were performed were: glucose, creatinine,

urea nitrogen, uric acid, total protein, cholesterol, triglycerides,

calcium, sodium, potassium, chloride, hematocrit, red blood cell count,

and hemoglobin.

On the basis of nine or less quarters of testing, it was not possible

to identify significant shifts in mean differences for any constituents

at the five percent level of probability.

The analysis also identified a phenomenon which has not been satis

factorally explained. Sometimes there was a consistent (but not statistically significant) bias between the mean differences for licensed

laboratories and reference laboratory target values. For example, Table

C-1 (Appendix C) shows that when creatinine was analyzed by the alkaline

picrate (w/o Lloyds) method, the target value (the average analysis for

all the reference laboratories which used that same method) was always

larger than the mean computed for the interstate group using the same method. This was also true for uric acid when analyzed by the phospho-

.

tungstate carbonate method and cholesterol analyzed by the AutoAnalyzer

and SMA methods. Positive biases (i.e., the target value was consistently

smaller) were not as prevalent; they were observed only in the following

instances: (1) urea nitrogen, all methods; (2) red blood cell count, all

methods; (3) hematocrit, specifically the micro Hematocrit method, as well

as for all methods; and (4) chloride. No definitive information has been

found to explain these biases.

The variability of the interstate laboratories, as demonstrated by

the pooled standard deviations, indicates that the group is becoming more

homogeneous. From Table C-1, Appendix C, it is evident that most of the

standard deviations become smaller as time in the proficiency testing

program increases. Exceptions to this trend are found for creatinine and

red blood cell count.

8
4.1.2. Volunteer Laboratories

Volunteer laboratories were, on the average, performing at slightlylower proficiency levels for some constituents in 1971 than they were in

July 1969, (i.e., cholesterol and red blood cell count). Obvious exceptions to this are urea nitrogen (all methods) , uric acid (all methods)

,

and hemoglobin (all methods), where the average differences are small.

As with the interstate group, the volunteer laboratories are generally becoming more homogeneous. In fact, for seven constituents,

the standard deviation for volunteer laboratories was smaller after eight

or nine quarters than the comparable interstate statistic. Considering

that the volunteer laboratory standard deviation was initially larger for

10 out of the 13 constituents, the group as a whole has made greater

strides towards increased homogeneity than have the interstate laboratories.

4.2. Profile Information: Licensed Laboratories Only

The licensed laboratories were characteri2ed according to several

attributes reported to CDC by the laboratories. For instance, it was

possible to determine those licensed laboratories which are involved in one or more additional accreditation or licensure programs, the number of

employees, supervisor's educational achievements, and other characteristics.

Paragraphs 4.2.1 through 4.2.5 discuss the various characteristics of the

237 licensed laboratories for which data was available.

4.2.1. Additional Program Involvement

Of the 237 licensed laboratories for which data was available, 12

percent (29) do not participate in any other inspection, accreditation or licensure programs. The 208 laboratories that do participate in other

programs are distributed as follows:

Number Percent

Participate in one additional program 68 29 • Participate in two additional programs 93 39

Participate in three additional programs 47 20

Subtotal 208 88

Do not participate in other programs 29 12

Total 237 1001

4.2.2. Number of Full-Time Employees

More than half of the 237 laboratories have less than eight full-time

employees; 18 percent, or 42 laboratories, have more than 20 en^loyees.

The frequency distribution below describes this characteristic:

9
Number Percent

Laboratories with 3 or less emnlovees

Laboratories with 4-7 employees 75 35

Laboratories with 8-11 employees 33 14

Laboratories with 12-20 employees 34 14

Laboratories with more than 20 employees 42 18

Total 237 1001

4.2.3. Number of Supervisors per Laboratory

Six percent of the licensed laboratories do not have any supervisors

,

while 71 percent of the laboratories have 1-4 supervisors. The breakdown

is as follows

Number Percent

Laboratories with no supervisors 15 6

Laboratories with 1 supervisor 100 43

Laboratories with 2 supervisors 46 19

Laboratories with 3 or 4 supervisors 46 19

Laboratories with 5-12 supervisors 24 10

Laboratories with more than 12 supervisors 6 3

Total 237 100%

4.2.4. Supervisor Academic Achievement

There were 620 individuals identified as working supervisors in the

237 laboratories. Of these 620 supervisors, roughly 17 percent have no college degree. The following table contains the overall frequency of

supervisors having attained various levels of academic achievement.

Number Percent

Supervisors with no college degree 106 17

Supervisors with a B.S. or A.B. Degree 259 42

Supervisors with an M.S. or M.A. Degree 53 9

Supervisors with a Ph.D. Degree 72 11

Supervisors with an M.D. Degree 80 13

Supervisors with other degrees 50 8

Total 620 100%

10
4.2.5. Academic Achievement of Supervisors Within Laboratories

A cross -tabulation by academic achievement and number of supervisors

per laboratory was also prepared; that is, how the academic degrees are distributed among laboratories with different numbers of supervisors.

The breakdowns were obtained for laboratories with one, two, and three

supervisors. These tabulations follow:

Laboratories with One Supervisor

Number Percent

Supervisors with no college degree 19 19

Supervisors with a B.S. or A.B. Degree 46 46

Supervisors with an M.S. or M.A. Degree 9 9

Supervisors with a Ph.D. Degree 5 5

Supervisors with an M.D. Degree 14 14

Supervisors with other degrees 7 7

Total 100 100%

Laboratories with Two Supervisors

First Supervisor Second Supervisor Frequency

No college degree No college degree 6

No college degree B.S. or A.B. 9

No college degree M.D. 1

No college degree Other 1

B.S. or A.B. B.S. or A.B. 9

B.S. or A.B. M.S. or M.A. 2

B.S. or A.B. Ph.D. 4

B.S. or A.B. Other 1

M.S. or M.A. Ph.D. 1

M.S. or M.A. M.D. 1

M. S. or M.A. Other ' 1

Ph.D. Ph.D^ 1

Ph.D. M.D. 2

Ph.D. Other 1

M.D. M.D. 5

Other Other 2

47

Laboratories with Three Supervisors

First Supervisor Second Supervisor Third Supervisor Frequency

None B.S. or A.B. M.S. or M.A. 2

None B.S. or A.B. M.D. 1

None B.S. or A.B. Other 2

11
Laboratories with Three Supervisors (continued)

First Supervisor Second Supervisor Third Supervisor Frequency

INUliC uuier 1

RDm O. n-r Ui A "R rV. £>. "RJj. O.S n-r UI JAm A RDm None 11

R 9 o-r A R Dm Jm or A. X3. 7J

RDm 0« nrUl S\m A RDm R 9 r,-r A R M . XJn . 1

B S nr A R RiJ. O

<^

. n-r UX I\m A RDm ULXiei X MS nr M A Ph Ti MM . XJn . 11

M.S. or M.A. M.S. or M.A. B.S. or A.B. 1

M.S. or M.A. M.S. or M.A. Ph.D. 1

r^ii. D m X XI mU m i:^n. XJ

.

1

Ph.D. Ph.D. M.S. or M.A. 1

M.D. M.D. M.D. 3

M.D. . M.D. B.S. or A.B. 1

M.D. M.D. Ph.D. 3

26

All other combinations had a zero frequency; i.e., in none of the

licensed laboratories that had three supervisors were all three without

any college degree.

4.3. Analyses of Variance

A one-way analysis of variance of differences among test results

associated with each of the independent variables (excluding number of

employees) was performed for every constituent, using the NBS OMNITAB II

computer program. Specifically, the values of relative differences for

all laboratories and all time periods were classified according to one of the independent variables, and the differences among the means for

this classification were investigated. The results of these analyses are summarized in Paragraphs 4.3.1 through 4.3.4.

4.3.1. Accreditation

The extent to which interstate laboratories participated in other

accreditation or licensure programs did not significantly (tested at the

. 05 level of probability) affect their average performance in the CDC

proficiency testing program. That is, the average difference for laboratories not involved with any other program was not significantly different

from a comparable value associated with laboratories involved with several

other programs. This finding was consistent for all 14 constituents.

4.3.2. State Program Intensity

Analysis of variance indicates that performance in the CDC proficiency testing program is not functionally related to intensity of programs

at the state level, for all 14 constituents.

12
4.3.3. Supervisory Education

Different levels of education achieved by the working supervisors

did not effect the performance of licensed laboratories in the proficiency

testing program. For all constituehts , there was no indication that

laboratories whose work is supervised by persons with Ph.D. degrees

performed at different levels than their counterparts whose supervisors

had baccalaureate degrees or no degree at all.

4.3.4. Analytical Method/Laboratories

The data indicate that interstate laboratory performance in the CDC

proficiency testing program is significantly (p < .001) affected by the

analytical method used. This finding was consistent for all 14 constituents. However, the laboratories included in the study were not assigned

at random to the various methods, and each laboratory used one and only

one method to analyze each constituent. Therefore, unless one is willing

to make the necessary assumption that interstate laboratories are a homogeneous group, and that the differences observed are truly due to the

analytical methods alone and not to factors already analyzed, there are no firm conclusions which can be drawn from this simple one-way analysis

of variance. Additional analyses of variance, which consider the possible

interactions of the four independent variables, will be conducted in the

near future and reported separately. Appendix E consists of summary

tables developed from this one-way analysis of variance.

4.4. Method Bias in Determining Target Values

An original premise underlying this entire evaluative effort was

that one measure of a laboratory's analytical performance is its ability

to come close to the "right answer." In this study the "right answer,"

for each sample, was considered to be the average analytical value

reported by a group of at least 30 reference laboratories, all using the same analytical method as the licensed laboratory. The limited number of

cases where at least 30 laboratories used the same method restricted the

number of method comparisons which could be made. Thus, the available

data base was insufficient to conclusively analyze the importance of

establishing target values on a method-by-method basis. Nevertheless,

the limited results were suggestive. Table 4.1 summarizes the number of

times the method comparisons were possible for each constituent.

The tests of significance, which were performed on the means (t

test with unequal sample size, known variability) of the groups, did not

indicate any statistical significance, at the 10 percent level, between

the methods compared.

13
Table 4.1. Summary of Number of Times Method Comparisons

Were Possible for Each Constituent

Constituents Analytical Methods Number of Method

Comparisons

Glucose AutoAnalyzer vs SMA. 3

Creatinine Alkaline Picrate (w/o Lloyds) vs AutoAnalyzer 5

Urea Nitrogen AutoAnalyzer ys 3AA 6

Uric Acid AutoAnalyzer ys SMA 6

Uric Acid Phosphotungstate (carbonate) vs AutoAnalyzer 3

Uric Acid Phosphotungstate (carbonate) vs

SMA 3

Total Protein Biuret (serum) ys_ Refractive index 3

Total Protein Biuret (serum) ys AutoAnalyzer 3

Total Protein Biuret (serum) ys_ S^A 3

Total Protein Refractive index ys_ AutoAnalyzer 3

Total Protein Refractive index ys_ SMA 3

Cholesterol Direct serum (acetic anhydride) vs AutoAnalyzer 3

Cholesterol Direct serum (acetic anhydride) ys^

SMA 3

Cholesterol AutoAnalyzer ys_ avIA 3

Calcium Atomic absorption ys AutoAnalyzer 3

Calcium Atomic absorption ys_ SVIA 3

Calcium AutoAnalyzer ys 3^ 3

Sodium / Flame photometry vs AutoAnalyzer 8

Sodium Flame photometry ys S^IA 3

Potassium . .. Flame photometry ys AutoAnalyzer 11

Potassium Flame photometry ys_ SMA 3

Potassium AutoAnalyzer ys SAk 3

Chloride Cotlove titrator vs AutoAnalyzer 3

Chloride Cotlove titrator ys SMA 3

Chloride AutoAnalyzer ys_ SVIA 3

Hemoglobin Cyanmethemoglobin (manual) ys

Coulter S 3

14
5. Summary and Conclusions

5.1. Summary

This report summarizes the first part of a two-part study conducted

for the Division of Health Evaluation, Office of the Assistant Secretary

for Planning and Evaluation, Department of Health, Education, and Welfare.

The study consisted of an examination of data from the chemistry and

hematology portions of the first two years of a proficiency testing

program for laboratories licensed under the Clinical Laboratory Improvement Act of 1967. Also included in the report is Appendix A, a review

of the legislative history behind the law, and Appendix B, an annotated

bibliography of pertinent literature in the field of laboratory proficiency testing from 1963 to 1971.

Statistical analyses were performed on two sets of data supplied by

the Center for Disease Control. One set, covering 268 licensed clinical

laboratories and several hundred volunteer laboratories, included the

results of proficiency tests conducted in these blood constituents

glucose, creatinine, urea nitrogen, uric acid, total protein, cholesterol,

triglyceride, calcium, sodium, potassium, chloride, hematocrit, red blood

cell count and hemoglobin. The second set of data contained certain

profile information on 237 of the 268 licensed laboratories. This information included the number of full-time employees, the highest academic

achievement level of the working supervisors, a measure of the strength

of the proficiency testing program operated by the state in which the

laboratory was located, and the total number of accreditation or licensure

programs in which the laboratory was involved.

5.2. Conclusions

It is very important to keep the following three aspects of proficiency testing programs in general, and the CDC program in particular,

in mind when reviewing the data and formulating conclusions.

1. Almost every proficiency testing program measures the capability

of participating laboratories instead of their routine level of

performance. The extent to which capability and performance are related is unknown.

2. Factors such as scoring methods, criteria for determining outliers, and inter- convertibility among methods are known to

affect measured performance in proficiency testing programs.

Greater standardization among the major testing programs would

be desirable.

3. Proficiency testing is only one aspect of a broad based quality

assurance program. It is also necessary to have well trained

laboratory personnel, fresh reagents, standard solutions of

high quality, and timely feedback to the laboratory.

The following conclusions fall into two major categories. The first

three deal specifically with the data analyses. The fourth conclusion

15
is identified with the proficiency testing program currently being

conducted by the Center for Disease Control.

1. The only independent variable for vdiich the mean differences

were significantly different was Analytical Method/Laboratories.

The other three variables, Accreditation, State Program Intensity, and Supervisory Education, were apparently unimportant.

2. The standard deviation, a measure of the variability among

laboratories , has generally decreased over the two-year period

under study for both the interstate and volunteer laboratory

groups. This is an indication that both groups have become

more homogeneous in their testing of proficiency program

samples

.

3. The data do not support establishing reference group target

i values on a method- by-method basis. More data, collected in a statistically designed experiment, would be helpful in reaching

more rigorous and definitive conclusions.

4. There are few indications of meaningful changes in the average

level of performance of laboratories participating in the CLIA

proficiency testing program. Consideration should be given to

a sampling scheme somewhat akin to skip- lot sampling for those

constituents which appear to present no analytical challenge to

laboratories. Greater emphasis might then be placed on those

constituents which give licensed laboratories the most difficulty (cholesterol and creatinine for example)

.

16
Appendix A. A Review of tJie Legislative History of CLIA '67

Introduction

A review of the pertinent legislative material, including House and

Senate report hearings and Federal Register (FR) announcements was made

in search of answers to questions such as:

1. At the time of the hearings, what was the projected magnitude

of the licensing effort? Approximately how many clinical

laboratories were engaged in interstate commerce?

2. What were the ostensible problems in the clinical laboratory

industry which led to the hearings and ultimate legislation?

Public Law and Rule Making

The Honorable Harley 0. Staggers, Chairman of the House Committee on Interstate and Foreign Commerce, introduced HR 6418 early in 1967, at the

request of the Administration. A similar version, S 724, was introduced

in the Senate, and the bill which was eventually enacted was entitled

"Clinical Laboratory Improvement Act of 1967, Section 353 of the Public

Health Service Act." The intent of the Administration was to extend and

strengthen the Partnership for Health Provision, Public Health Service

Act, PL 89-749, by placing primary reliance upon state health authorities

and their services programs. Broadly stated, the administration viewed

the partnership as one where the Federal government would assist the

states with financial and technical resources without assuming any

authority for the conduct of the state programs.

Three major witnesses for the administration were Undersecretary of

HEW, Wilbur J. Cohen; Surgeon General, Dr. William H. Stewart; and

Communicable Disease Center Director, Dr. David J. Sencer. Dr. Sencer

testified that 25 percent of all clinical laboratory analyses were so gross as to be medically useless. This testimony, from so eminent a witness, made a significant impact upon the congressional committee

before which Dr. Sencer was appearing.

The major mileposts with respect to publishing the new legislation

and its projected impact on the industry were:

1. October 15, 1968—A notice of rule making was published in the

Federal Register (33 FR 15297-15303) which proposed the addition

of a new Part 74 to the Public Health Service Regulations.

2. December 31, 1968—A notice that the new Part 74 was added to

the PHS Regulations, effective 30 days after publication (January 31, 1969). This notice described the nine major subparts

of the new Part 74 as follows:

Subpart A - Definitions and Applicability

Subpart B - License: Application and Issuance

Subpart C - Quality Control

17
Subpart D - Personnel Standards

Subpart E - Proficiency Testing

Subpart F - Accreditation—State Loans—Stringency of

Standards—Termination

Subpart G - General Provisions

Subpart H - Revocation, Suspension, and Limitation of

Licenses and Letters of Exemption Notice

Subpart I - Hearings (Reserved)

3. April 5, 1969—Effective date of the Act postponed again to

July 1, 1969 (it had already been postponed from January 31 to

April 1, 1969 in paragraph 2, Subpart I of FR No. 253).

4. July 1, 1969—The College of American Pathologist proficiency

testing program was determined to be equal to or more stringent

than the Federal program. Laboratories enrolled in the CAP

program were granted exemption from CLIA upon submitting an

application for same.

5. August 15, 1969—Effective date for cytotechnologists to become

qualified extended from January 1, 1960 to January 1, 1969.

6. October 14, 1970—Notice in the FR that DHEW proposed to amend

the CLIA '67 regulations with regard to granting hearings for

affected laboratories.

7. October 25, 1969—HEW published a notice of intent in the FR to

require a letter of exemption from laboratories accepting fewer

than 100 specimens per year in a specific category or subcategory.

8. December 3, 1969—A notice published in the FR to amend CDC's

Statement of Organization, Functions, and Delegation of Authority.

9. February 7, 1970—A list of laboratories which were granted

exemption from CLIA '67 was published by HEW in the FR.

10. April 16, 1970—4IEW published a notice in the FR that item 7

above was now in effect; exempt laboratories would have to

have a letter of exemption, issued by the director, CDC.

Licensure—Eligibility and Exemption

There has never been a complete census of all operating clinical

laboratories in the United States. Such an undertalcing would be facilitated by a consensus as to a working definition of "a clinical laboratory."

Current and past estimates of the total number of laboratories range from

15,000 to perhaps 200,000; depending upon whether physicians' offices are

included.

When the director of the Center for Disease Control (CDC) testified

at the House hearings, he estimated that "Ten percent of the clinical

laboratories in this country are in interstate commerce." There are other references to the number of laboratories in interstate commerce,

such as that which appears on page 2083 of the Congressional and Administrative News, 90th Congress (First Session, 1967): "Estijnates given to

the Committee by the Department of Health, Education and Welfare indicate

that there are approximately 1,000 laboratories substantially operating

in interstate commerce. These are primarily independent laboratories

and a few hospital laboratories." The Administration's intent, in

18
supporting this legislation, was to impact on the largest segment of the

industry as evidenced by the following quotation from the same 90th

Session Congressional and Administrative News: "The committee believes

that the Secretary of Health, Education, and Welfare should be required

to provide an exemption from licensing in those cases where laboratory

operations in commerce are so small or infrequent as not to constitute

a significant threat to the public health."

There were many forces acting on the Congress during its consideration of the legislation, such as the life insurance industry which wanted

exemption for laboratories whose primary business was associated with

performing tests in conjunction with physical examinations for policy

holders, the medical community which wanted any physician directed laboratory to be exempt from federal intervention, and various professional

(society) accreditation organizations which operate financially successful quality control programs. Consequently, when Public Law 90-174,

"The Partnership for Health Amendments of 1967" was enacted on December

5, 1967, it contained the following exemptions on page 580 of the U.S.

Code

(2) A license issued under this section shall be valid for a period

of three years, or such shorter period as the Secretary may establish for any clinical laboratory or any class or classes thereof;

and may be renewed in such manner as the Secretary may prescribe.

The provisions of this section requiring licensing shall not apply

to a clinical laboratory in a hospital accredited by the Joint

Commission on the Accreditation of Hospitals or by the American

Osteopathic Association, or a laboratory which has been inspected

and accredited by such commission or association, by the Commission

on Inspection and Accreditation of the College of American Pathologists, or by any other national accreditation body approved for the

purpose by the Secretary, but only if the standards applied by such

Commission, association, or other body in determining whether or not to accredit such hospital or laboratory are equal to or more

stringent than the provisions of this section and the rules and

regulations issued under this section, and only if there is adequate

provision for assuring that such standards continue to be met by

such hospital or laboratory; provided that any such laboratory shall

be treated as a licensed laboratory for all other purposes of this

section.

and on page 582

(i) The provisions of this section shall not apply to any clinical

laboratory operated by a licensed physician, osteopath, dentist,

or podiatrist, or group thereof, who performs or perform laboratory

tests or procedures, personally or through his or their employees,

solely as an adjunct to the treatment of his or their own patients;

nor shall such provisions apply to any laboratory with respect to

tests or other procedures made by it for any person engaged in the

business of insurance if made solely for purposes of determining

19
whether to write an insurance contract or of determining eligibility or continued eligibility for payments thereunder.

(1) Where a State has enacted or hereafter enacts laws relating to

matters covered by this section, which provide for standards equal

I to or more stringent than the provisions of this section or than

the rules and regulations issued under this section, the Secretary

may exempt clinical laboratories in that State from compliance with

this section.

As of January 18, 1972, there were 552 clinical laboratories

licensed by the Center for Disease Control out of approximately 1,500

which are "probably" eligible from the standpoint of being engaged in

interstate commerce. The 63 percent not licensed are assumed to be

exempt because they accept less than 100 samples per year in interstate

commerce or are otherwise accredited or licensed.

Summary

The committee hearings and related evidence indicated that the

performance of clinical laboratories was not what it could or should be.

The committee recognized that improvement of the clinical laboratories

and their performance was a long run effort for the state governments,

their health agencies and their public health service authorities. However, the committee also recognized that it would take time for the state

licensing programs to be developed throughout the nation. In the meantime, the state programs would vary considerably with respect to procedures and quality. Because of this variance, laboratories doing business

in interstate commerce could locate in states with weaker licensing

programs or in states without programs. In addition to the need for

licensing interstate business of the clinical laboratories, there was

the above-mentioned administration and congressional plan to extend and

strengthen the partnership for health provision of the PHS Act. These

elements, the need for regulations of interstate samples and the extention of the partnership for health, were the motivating factors for

CLIA '67. However, an unforeseen development in certain states has,

to some extent, bypassed the anticipated effect of the law. At least

two states have passed laws which prohibit solicitation of laboratory

work from laboratories outside that state.

20
Appendix B. Abstracts

The entries in this annotated bibliography on proficiency testing

programs in the open literature have been obtained from a search of the

Department of Commerce National Technical Information Center; the Defense

Documentation Center; the Index Medicus of the National Library of Medicine, DHEW library; from references in the papers themselves, and from

a report entitled "Clinical Laboratory Evaluation Programs" dated July

31, 1970, by the Auerbach Corporation, Philadelphia. This last report

was procured by the Department of Health, Education and Welfare on

Contract 110-69-434.

The 34 entries on proficiency programs are arranged by years starting with 1963, and are alphabetical within years.

1963

1. Tonks, D. B. , "A Study of the Accuracy and Precision of Clinical

Chemistry Determinations in 170 Canadian Laboratories," Clinical

Chemistry , 9:217 (1963).

The data obtained have shown that there was a lack of both accuracy

and precision in many laboratories. There is obviously a need for an improvement in the methods of detecting errors and of evaluating p cision in the laboratory. An adequate quality control program and the

proper usage of control serums are indispensable for these purposes. It

is also useful for every laboratory to take part in periodic studies such

as this one in order to obtain an unbiased evaluation of its performance.

1967

1. Copeland, B. F. , W. J. Blake, R. J. Muelling, and L. P. Skendzel,

"A Report of the Standards Committee of the College of American

Pathologists," American Journal of Clinical Pathology , 48:104 (1967).

The purpose of this survey was to stimulate interest in the accuracy

and precision of chemical measurements commonly made in laboratories for

the diagnosis and therapy of disease. The following measurements were

included in the survey: glucose, urea nitrogen, bilirubin, cholesterol,

Calcium, phosphorous, sodium, potassium, bicarbonate, uric acid, and

chloride

.

2. Kaufmann, William and R. Vanderlinde, "Medical Laboratory Evaluation,"

New England Journal of Medicine , 277:1024 (1967).

The overwhelming evidence points to rather serious deficiencies in

the performance of tests, in their reproducibility and thus in their

accuracy and dependability as aids in clinical diagnosis. A built-in

fallacy in assessing laboratory performance is the assumption that the

test specimen is treated in the same fashion as the daily routine specimen

21
1967 (continued)

whereas this, in fact, rarely happens. The data show that low quality

in testing is related to a low annual number of tests performed. The

fact that there is a serious laboratory problem in this country should

be recognized.

3. Schaeffer, M. , D. Widdock, S. Blatt, and M. E. Wilson, "The Clinical

Laboratory Improvement Program in New York City: I. Methods of

Evaluation and Results of Performance Tests," Health Laboratory

Science , 4:72 (1967).

This report is limited to the data obtained in evaluating the

capabilities of all the laboratories within the jurisdiction of New York

City and to factors that relate to the quality of performance.

In October 1966 there were 359 clinical laboratories with permits to

operate in New York City. A different group of laboratories was tested

each week, and each laboratory was tested at least four times in a two-year period. The tests performed included microbiology, syphilis

serology, cross-matching and hemoglobin determinations. The standard

deviation of all the results of the reference laboratories was calculated, the acceptable range being two standard deviations from the mean. About 50% of the independent laboratories and about 40% of all of the

hospital laboratories were rated as unsatisfactory in bacteriology, while

about 25-30% were rated as excellent. In clinical chemistry, about 50%

were fair, and the remainder equally divided between unsatisfactory and

excellent. The poorest results appear among the independent laboratories,

although the other laboratories were not much better off. About 18% of

the cross -matching tests were unsatisfactory; in syphilis serology 84%

of the results of independent laboratories and more than 90% of the results

of hospital laboratories were excellent.

Laboratory performance did not appear to be related to whether the

director was full-time or not. Qualifications of the director did not

appear to matter. The organization of the laboratory did appear to influence performance. Higher test accuracy was noted in the laboratories

that processed high volumes of specimens.

There appeared to be a direct relationship between the use of

professionally trained supervisors and the quality of performance. Forty

excellent laboratories, representing only 29.6% of voluntary hospitals,

employed 63.3% of all the supervisors in voluntary hospitals. This level

of supervision is difficult to obtain and too expensive for a small

laboratory. This paper expresses the opinion that the existing situation

(1964-1966) in clinical laboratories cannot be permitted to continue

indefinitely.

22
1968

1. Barnett, Roy N. , "Medical Significance of Laboratory Results,"

American Journal of Clinical Pathology , 50:671 (1968).

Tests performed in clinical laboratories are among the main

diagnostic aids available to physicians. The great demand for these

tests and the enormous number performed have led to Federal specification through the Medicare Act and the Clinical Laboratory Improvement

Act of 1967 (CLIA '67). Regulations under both Acts set basic standards

of operation and require participation in a proficiency testing program.

The basic question is how accurate must clinical laboratory work be?

The Standards Committee of the College of American Pathologists has been

the major proficiency surveying body in the United States for a number of

years, and these surveys lead to two questions: (1) How well are laboratories perfoming certain common analyses? and (2) Is this kind of

performance adequate for good medical practice? This provisional report

examines these questions. Six (sic) terms are defined, including accuracy,

precision, standard deviation, coefficient of variation, percentiles,

normal range, medically significant, state-of-the-art, and decision level.

Other topics include guidelines in establishing limits for reporting

values of medical significance, a table of medically significant values,

interlaboratory and intralaboratory evaluation, and the utilization of

specific limits in medical practice.

2. College of American Pathologists, "Guidelines for Evaluating

Laboratory Performance in Survey and Proficiency Testing Programs,"

American Journal of Clinical Pathology , 49:457 (1968).

Yearly surveys have produced data that describe the accuracy and

reproducibility of procedures. Variability increases as variables are

introduced. Single laboratory precision is better than multiple laboratory precision.

In consultation with Dr. W. J. Youden, NBS, the CAP established

criteria for acceptability in interlaboratory surveys, but not for precision objectives by individual laboratories. Reference laboratory

results will be provided to survey participants. Performance terminology

is given, as are good performance criteria for hemoglobin, glucose, urea

nitrogen, calcium, phosphorous, total protein, chloride, uric acid,

albumin, bilirubin, sodium, potassium, creatinine, and cholesterol.

3. Martinek, R. G., et al., "Comparison of Methods for Conducting

Laboratory Proficiency Surveys," Health Laboratory Science , 5:239-

256 (1968).

The program reported here was planned and carried out by the Illinois

State Department of Public Health. The survey method used, whereby

unknowns are analyzed by participating laboratories, is reported and

summarized in the report. Another method relied on the use of patients'

data (PD) , wherein each laboratory was instructed to list 90 consecutively

23
1968 (continued)

determined values. These distributions were plotted, and normal curves

fitted, each being described by the mean or average and the corresponding

standard deviation. Normal ranges were established for each test used.

From the data from the conventional surveys by the American Association

of Blood Bariks and CAP correlation analyses were made as well as studies

of within and between surveys using Kendall's coefficient of rank

correlation. Results "...indicated as close a relationship between a proficiency sample and the patients' means as between proficiency samples

from the two agencies." Considerable clarification of interpretation

was obtained. "When the conventional proficiency survey results are interpreted in this manner the conclusion is inescapable that the patients'

data is informative and directly related to our concern." The results of

testing patients' specimens represents everything that the laboratory does.

The analyses of patients' data yield standard deviations that are similar

in magnitude to those from conventional and extensive surveys of laboratories for prothrombin time. Conclusions from these analyses are (1)

do not use either technique as the basis for clinical laboratory regulatory action at this time, and (2) carefully evaluate the various quality

control methods.

4. Prier, J. E. , L. Sideman and I. J. Yankevitch, "Clinical Laboratory

Proficiency Testing," Health Laboratory Science , 5:12 (1968).

Clear evidence documents deficiencies in hospital and independent

clinical laboratories (CL) . Available methodology provides ways to

obtain high accuracy and consistency if properly applied. CL evaluation

involves three factors: (1) personnel qualifications, (2) on-site

inspection, and (3) proficiency testing. Proficiency testing (PT) in

Pennsylvania was made mandatory in 1961. Two responses to law are noted:

(a) well trained pathologists provide the least objections; (b) "notable"

objections are from directors who are marginal in both training and

facilities. PT must be carried out to assure competency of CL work. It

is doubtful that an adequate CL control program can be obtained by voluntary participation. (Gives history of PT in Pennsylvania.) Any effective

program must be enforced. Ten characteristics of a new PT program are discussed. Competency in CL work must include PT, and PT without enforcement is inadequate, collecting only disquieting data. A PT program must

reach all laboratories equally. There should be no provision for waiver

because of education of director or other reasons. Accurate laboratory

work is related to close supervision. Inspections and "on-the-spot"

testing are essential parts of a total evaluation program. Programs

provided by legislation with provisions for compulsory PT are mandatory.

5. Report from the Ad Hoc Committee on "Documentation of Procedures,

Diagnostic Criteria, and Results in Laboratory Medicine of the

Committee on Pathology," Division of Medical Science NRC-NAS-NAE,

American Journal of Clinical Pathology , 49:757-760 (1968).

Conclusions: (1) That there is a need for documentation of medical

laboratory data in the United States in order to promote the comparability

24
196S [continued)

of data over long periods and bet^veen different sections of the country

as patient mobility increases.

(2) That the questionnaire method has been successful in determining

the state-of-the-art as practiced in leading medical institutions.

(3) That information on the state-of-the-art should continue to be

collected and should be published.

(4) That a Conference on Documentation of Procedures, Diagnostic

Criteria, and Results in Laboratory Medicine should be held.

(5) That there appears to be a definite need for a Center for

Documentation of Procedures, Diagnostic Criteria, and Results in Laboratory Medicine. The obvious benefit to patient care makes this an important goal.

1969

1. Eilers, R. J., "Total Quality Control in the Medical Laboratory,"

Hospital Progress , 50:92 (1969).

The concept of total quality control for the medical laboratory offers

a means of achieving the professional objective of iinproving the quality

of the service that it renders. Quality Control does not mean best, but

it could. The word "control" in the phrase Quality Control includes a four-part management tool

Establishment of standards

Determination of conformance to these standards

Taking corrective action

Planning for improved standards.

Nine benefits from improved Quality Control in laboratories include:

Improvement in quality of laboratory results

Improvement of employee morale

Improvement of inspection methods

•Establishment of firm time standards

Establishment of preventive maintenance schedules

Development of a factual basis for cost accounting.

The cost of obtaining and maintaining a level of quality in a laboratory is consolidated with the cost from failure to obtain the given

level of quality. Four elements of cost are cited: (1) Prevention cost,

(2) Appraisal costs, (3) Costs of internal failure, and (A) Costs of

external failures. A total Quality Control program can be divided into

three phases:

Assign quality responsibility to all employees

Employ a quality control specialist

Develop special techniques for all laboratory functions.

Six phases of Quality Control are:

Xew design control

Incoming material control

25
1969 (continued)

Process control

Production output control

Product reliability

Special process studies.

The College of American Pathologists has an inspection and accreditation activity for total laboratory services that is accepted by the JCAH

and CDC. This service has been in operation for many years. Participants

in 1968 numbered 3900.

To obtain high Quality Control, all in the chain must recognize the

existing deficiencies and initiate programs for raising the existing

level, otherwise responsibility for this function—Quality Control—will

become the responsibility of state or Federal bureaucrats.

2. Hendry, I. A., Editor, "Reports of Clinical Chemistry Surveys,"

Board of Education of the College of Pathologists of Australia,

Sydney meeting, August 1969.

The program reported here was based on the premise that most of the

testing done in clinical laboratories is on specimens from people in

relatively good health. The test plan was designed to provide information

(a) about the level of laboratory daily precision, and (h) on the ability

of the laboratories to maintain precision over a three-month interval.

Two classes of data were collected, the one from a conventional survey

where two sets of different sera were mailed for testing, and the other

consisting of two groups of data from tests on patients' specimens, each

group containing the results of 90 tests. The sets and groups consisted

of data taken in the same two time intervals. The patients' test data

were the same as for the conventional survey, glucose, urea, total protein,

and cholesterol. Unknown to the laboratories, only two different con- centrations were used, with at least one specimen of each in any given

mailing of the six. Conclusions cited are that:

(a) The specimen concentrations in the human sera remained constant

throughout the three months of the study.

(b) Information regarding the accuracy and precision of individual

laboratories can be obtained from patients' test data.

(c) Surveys of laboratories consisting of a few survey unknowns are of limited value.

(d) Surveys are useful for determining the performance of laboratories as groups, but are of practically no value for estimating

the performance of individual laboratories.

(o) The information from patients' test data is at least as informative as that obtained from tests performed on survey samples.

(f) Patients' test data is sufficiently detailed to be reasonably

valid for estimating individual laboratory performance.

26
1969 (continued)

This report concluded by recommending that the College of Pathologists of Australia establish a computing service for laboratories, with

feedback to individual laboratories.

3. Hoffman, Robert G., "Patients' Test for Quality Control," Clinical

Chemistry , 15:533 (1969).

A brief outline of the method of "average-of-normals" is given. It

consists in the plotting of the mean +_ the 951 confidence limits as a function of the tests or time. Each point was the average of 16 tests

in one demonstration, and of 10 in the other. In this way it is stated

that out-of-normal shifts can be seen.

4. Kanon, Dor, "Computer Analyses Lab Tests, Does Own Quality Control,"

Modern Hospital , 113:105 (1969).

In a form- phase, all-day operation, this chemistry laboratory computer

updates patient files, standardizes laboratory values, does quality control, and produces cumulative reports. The clinical chemistry laboratory

at St. Vincents Hospital, New York, handles 285,000 tests a year. Online operation has form phases that describe in detail the day's work.

The use of standards is described, as is the reliance on quality control

programs. Loss of sensitivity from reagent deterioration, and increase

in standard deviation from dialyzer diaphragm deterioration have been

detected. Monthly cost is about $7,000. Biochemistry production

increased 51% in the first six months of operation with no increase in

personnel

.

5. Kaufmann, William and six co-authors, "Clinical Laboratory Performance

Experience with the New York State Clinical Laboratory Program," New

York State Journal of Medicine , 15:1989 (1969).

In 1964 the New York State Legislature passed the Laboratory Improvement Act, which requires all clinical laboratories to operate under permit

and to have in charge an individual who holds a valid certificate of

qualification. There were 430 clinical laboratories included in the 1967

to 1968 survey, made up of 229 hospital laboratories and 201 independent

laboratories, privately or municipally owned. The six areas of bacteriology, mycobacteriology, mycology, serology, hematology, and clinical

chemistry were covered.

6. McSwiney, R. R. and D. A. Woodrow, "Types of Errors Within a Clinical Laboratory," Journal of Medical Laboratory Technology , 26:340 (1969).

In this study all errors were counted over four weeks covering a total of approximately 7,000 tests. , There were 163 errors found, 63 of

them by senior technicians and 57 by pathologists. In addition, two are known to have "escaped" into the wards. "Organizational" errors consisted

;27
1969 (continued)

mostly o£ omitted or delayed analyses, and accounted for 341 of the total

Over half the errors (54%) were clerical in origin. A normal day's work

produced 15 to 20 errors. Errors are made more than realized; they are forgotten as a familiar part of life but are not permissible within the

clinical laboratory.

7. Sideman, L. , D. T. Wilson, and J. J. Murphy, Jr., "A Survey of

Immunohematological Proficiency Among 131 Clinical Laboratories in

Pennsylvania Health Laboratory Science , 6:156-161 (1969).

Survey objectives were: (1) to uncover laboratory errors in the

identification of major blood groups and Rh type, and (2) having found

the errors, to offer consultation and training in order to eliminate

the causes. The overall improvement shown in a subsequent re-evaluation

of these same laboratories emphasizes the value of a comprehensive

evaluation program which includes interlaboratory studies, consultative

services, training, and re- evaluation.

8. Seracci, R. , "Factors Affecting Accuracy and Precision in Clinical

Chemistry," American Journal of Clinical Pathology , 52:161-166 (1969)

Accuracy and precision in four clinical chemistry determinations

(glucose, urea, total cholesterol, total proteins) have been investigated

in a survey of 404 Italian laboratories. Inaccuracy and imprecision were

found which appear, under survey conditions, to be little related to

factors such as class and size of laboratories, qualifications and

seniority of analyst, and method of analysis. This would imply that

acting upon these factors is less important in improving laboratory

performance than is the systematic use of quality control methods.

9. Skendzel, L. P. and W. J. Youden, "A Graphic Display of Interlaboratory Test Results," American Journal of Clinical Pathology , 51:161

(1969).

A simple graphic system worked out by Youden is described and

advantages noted. One requirement for this system is that laboratories

analyze two samples for the same constituent. Reference laboratory

results are plotted against each other, with intersecting lines passing

through each mean, the SD is multiplied by 3.035 to obtain the radius

of a circle that is drawn with the intersection of the means as the

center. This circle will then include 991 of the values of reports for

the two samples. Two tangents with positive slope at 45° are drawn to

the circle one at each end of the diameter. . Pairs of values from other

participants are then used to plot additional points.

28
1970

1. Barnett, Roy N. and W. J. Youden, "A Revised ScJieme for the Comparison of Quantitative Methods ," American Journal of Clinical Pathology , 54:454-462 (1970).

The authors, on the basis of experience gained in evaluation of 50

different "kits" for chemical analysis, have developed a revised and

simplified method for conducting such evaluations. The scheme, which is

applicable for any comparison of quantitative methods , describes reproducibility studies (20 once-a-day analyses), recovery studies in

triplicate at two or three levels, and comparison studies with patient

serum, all performed by the test method and a reference method. Statistical calculations and criteria of acceptability are described in detail

and a complete example given.

2. Barnett, Roy N. , W. Harold Civin, and Irwin Schoen, "Multiphasic

Screening by Laboratory Tests - An Overview of the Problem," American

Journal of Clinical Pathology , 54:483-492 (1970).

The authors considered the problems of jnultiphasic screening by

laboratory tests in the framework of the concept "Total Quality Control

in the Clinical Laboratory," After reviewing the data, the problems, and

the opinions of others, they drew the following conclusions. It is

entirely premature to recommend the routine use of extensive multiple

screening tests for either hospital admission or general populations,

considering the present ignorance of physicians about every one of the

six categories we used. At present these multiple screening tests should

be considered research rather than service activities. Although the

laboratory problems in testing are still formidable, they are being

solved far more rapidly than are the problems relating to the medical

usefulness of the test results. There is an urgent need for approximately

controlled, large scale, multidisciplinary studies to answer the basic

questions concerning the utility of the data. We cannot accept the

assumption that the production of huge volumes of "screening" information

will by itself contribute to human knowledge or health. A selected

bibliography is appended.

3. Eilers, Russell J., "Total Quality Control for the Medical Laboratory: The Role of the College of American Pathologists Survey

Program," American Journal of Clinical Pathology , 54:435 (1970).

Six phases of Quality Control are set forth as follows:

I. Selection of proper test method

II. Selection of proper standards control and SRM's

III. Internal Quality Control of process and proficiency testing

IV. Proper format of results

V. Reliability and interpretation of results

VI. Inspection and accreditation.

29
1970 (continued)

4. Evenson, M. A., "The Need for Analytical Chemistry in Clinical

Chemistry Training Programs," Analytical Chemistry , 42:53A (1970).

The shortage o£ adequately trained clinical chemists will soon become

a manpower crisis, but recent Federal fund tightening activities have

dampened enthusiasm for new programs. There are 1,600 persons in the

U.S. who have more than a bachelor's degree in chemistry and are members

of the American Association of Clinical Chemists. The 20 most common

clinical chemistry tests are listed. Each clinical chemistry laboratory

doubles its own size every five years. A short history of clinical is

given, starting with Paracelsus in 1526. Comments on the administrative

and academic arrangements usually formed by clinical chemists are made.

Need for analytical chemical excellence in clinical chemistry laboratories

is stressed.

5. Gavan, Thomas L. and John W. King, "An Evaluation of the Microbiology

Portions of the 1969 Basic, Comprehensive, and Special College of

American Pathologists Proficiency Testing Surveys," American Journal

of Clinical Pathologists , 54:514 (1970).

The proficiency testing program of the CAP was markedly expanded in

1969 by adding three new surveys: Special Bacteriology, Special Mycobacteriology, and Special Mycology. The use of mycobacterial and

mycologic specimens in the Comprehensive and Basic Surveys was expanded.

The proficiency testing surveys of the CAP and those of other agencies

are limited to evaluating a laboratory's ability to identify a given

organism or organisms. No satisfactory methods exist for large scale

surveys in microbiology, based on simulated clinical specimens, to enable

laboratory proficiency evaluation in isolation techniques as well as the

identification of bacteria and fungi.

In all, 83 specimens of various bacteria and fungi were distributed,

with 19 being common to two or more surveys. A total of 34,908 reports

from participants have been evaluated. Methods of rating participant

performance in terms of referee performance are given, and the results of

both are presented in seven tables. In general, 80% or more of the participant laboratories were able to identify the pyogenic bacteria submitted

in a good or acceptable manner. The exceptions are cited and discussed.

Mycobacteria were acceptably identified by significantly fewer laboratories than the pyogenic bacteria. Fungi were more poorly identified

than pyogenic bacteria, but were correctly identified more frequently

than the mycobacteria.

The quality and scope of survey material have been improved, as well

as the results reported by participants. Improved techniques for the

conduct of such surveys are given, including specimen checking and in

freeze drying and handling specimens.

30
1970 (continued)

6. Gilbert, Roy K. , "Analysis of Results of the 1969 Comprehensive

Chemistry Survey of the College of American Pathologists," American

Journal of Clinical Pathology , 54:463-482 (1970).

The samples of this survey consisted of eight vials mailed out in

pairs on four occasions during the year. The results are described for

analyses for glucose, calcium, total bilirubin, potassium, sodium, urea

nitrogen, uric acid, creatinine, chloride, phosphorous, protein-bound

iodine, and total protein. The total number of analyses was 140,730.

The results were categorized by method used, and a clear pattern of

relationship was apparent between the method and both the mean values

and the standard deviations. Some methods showed a constant relationship in mean values with one method higher or lower throughout the series.

In others there was variability and the relationship changed. This

fluctuation could occasionally be traced to interaction with other con- stituents. The methods also showed a relationship to the standard

deviations throughout the series, with one method usually higher or lower

than another. The relationship tended to be more constant than that for

the mean values.

Variance is ascribed to three sources : the basic analytical performance, calibration standards, and systematic differences between

laboratories; these are discussed at length. The results show that many

methods give mean values that differ from those of other methods. The

data also show that this is not a constant relationship and that interaction between constituents occurs. The spread of values is wider by

some methods than others, and the automated method tends to produce a narrower range although a survey offers some bias in favor of automated

means.

7. Hanson, Daniel J. , "Suggested Definitions for Clinical Laboratory

Standards and Reference Materials," American Journal of Clinical

Pathology", 54:451 (1970).

The CAP is concerned about the existing confusion regarding definition

and nomenclature of standards that are applicable to clinical pathology.

The situation is an impediment to laboratory performance evaluation and

to the application of standards ^d controls in the clinical laboratory.

There is an impelling and immediate need for agreement on terms and

definitions by pathologists, technologists, and manufacturers to resolve

confusion and permit evaluation of reports. Definitions are presented

and discussed for Primary Standard, Primary Volumetric Standard, and

Secondary Volumetric Standard. The clinical versions of these standards

are also presented, as is the term Reference Material or Reference

Control, originally referred to by Radin as "reference samples."

31
1970 (continued)

8. Koepke, John A. , " Iiranunohematology Proficiency Testing, 1966-1969,"

American Journal of Clinical Pathology , 54:508-511 (1970).

Immunohematology proficiency testing as measured by the Laboratory

Surveys of the College of American Pathologists, from 1966 to 1969, is

reviewed. During this period there has been a gradual expansion of this

portion of the survey. Over the years studied there has been an evident

improvement in ABO grouping and the detection of atypical antibodies.

Crossmatching has continued to be well done; however, performance of Rh

typing has been spotty, presumably due to variation in typing sera or red cell testing specimens, or both.

9. Reed, A. H. , "Use of Patient Data for Quality Control of Clinical

Laboratory Tests," Clinical Chemistry , 16:129 (1970).

The average-of-normals quality- control method is examined mathematically. Two modifications are proposed. After modification, the essential

unchanged feature of the method is the idea of censoring the laboratory

test results so that only means of test values inside the normal range are plotted on the quality- control chart. Several limitations of the

modified method are discussed, and it is concluded that, although

censoring does improve the probability of detecting drift, more statistical research and further modifications are needed before the averageof-normals method will be useful for quality control in the clinical

laboratory.

10. Rodwell, V. W. , Editor, "Test Theory, Proficiency Tests, and

Unevaluated Tests," Test of the Month ISb. 1 , Proficiency Testing

Service, American Association of Bioanalysts (1970).

Research is needed to gain a greater understanding of procedure and

objectives in carrying out proficiency testing and on-site inspections.

PT under Medicare is cited as is CLIA '67. Cost problems of private

groups are noted. CAP participant cost is $881/year, while CDC will do

the same thing for less than $125.

There is little understanding of the theory of testing. It was

first given formal expression by Dr. Reuben L. Kahn: first, a procedure

is employed to derive a probability statement with regard to any determination; second, group validity in regard to any determination is always

more reliable than individual validity; third, the parameters involved in

any test are sensitivity, specificity, and reproducibility of results.

An exception is blood typing where there is no room for error in determining ABO and Rh types.

32
1970 (continued)

11. Schaeffer, M. , D. Widelock, P. S. May, S. Blatt, and M. E. Wilson,

"Tlie Clinical Laboratory Improvement Program in New York City: II.

Progress After Five Years of Experience," Health Laboratory Science , 7:242 (1970).

In previous studies, clinical laboratories in New York City were

analyzed with respect to organization and technical competence. From

that evaluation it became apparent that laboratories performing satisfactorily are those that employ full-time competent supervisors. This

experience led to efforts to in^rove the performance of consistently poor

laboratories.

Two methods designed to improve the proficiency of clinical laboratories in either bacteriology or chemistry were evaluated. On-site

consultation by experts in these two disciplines, and workshop instruction

were studied for their effect on results of proficiency testing of laboratories selected for assistance. The latter were compared with similar

laboratories treated routinely, without advice or instruction.

Analysis of results appears to indicate that some improvement in

performance was achieved simply through repeated proficiency tests,

laboratories participating in bacteriology workshops showed improved

performance as compared with control laboratories. However, laboratories

enrolled in clinical chemistry workshops showed no greater improvement

than control laboratories.

12. Schoen, I., "Guidelines to Laboratory Management ," California State

Department of Public Health, Some Proficiency Testing Experiences

in California for 1968 , Volume D-1 (1970)

.

Proficiency testing is mandatory for many licensed laboratories in

California. There were 226 laboratories in the State Comprehensive

Survey, and 201 in the Basic Survey; the Basic Survey includes sufficient

specimens and examinations to meet the mandatory PT requirements, while

the comprehensive survey includes more specimens and examinations than

required by the mandatory regulations. Overall, the performance of

California laboratories and all participating laboratories was very

similar with no notable difference in the data. The means of reporting

and selection of specimens by current survey techniques appears to lead

to unsatisfactory erroneous mixed-up reports involving up to 1.5% of

reports, and up to 51 of laboratories.

Automation seems more precise for some procedures. This precision

might be due not to automation itself, but may be more appropriately due

to standardizing or controlling instrumentation methods, reagents, and

standards. A "correct" report on a survey specimen does not mean a "correct" interpretation of a report on a patient's specimen from a laboratory with an incorrect set of normal values. Medical Usefulness Criteria

as well as data from Peers or Reference Laboratories should be utilized

in evaluating performance.

33
1970 (continued)

Local or regional data accumulation and evaluation should be considered to achieve more prompt evaluation with follow-up testing and con- sultation for unsatisfactory performance.

To best serve the interest of reliable, better, and improved laboratory services by a specific laboratory for the patient, a program of

standards for quality control analogous to standards for a Blood Transfusion Service should be considered.

13. Skendzel, Laurence P., "Guidelines for the Design of Laboratory

Surveys," American Journal of Clinical Pathology , 54:437-447 (1970).

This report outlines the system used in the College of American

Pathologists National Survey Program for medical laboratories with special

emphasis on data handling and evaluation of a laboratory's performance.

The statistical techniques for data analysis of quantitative procedures

include: (1) exclusion of widely deviant values, (2) calculation of mean,

SD, and CV after exclusion of deviant values to describe the distribution

of values, (3) calculation of the average CV to summarize the distribution

of values from several surveys, (4) reliance on the two-sample plot and

the "t" test to combine results derived by different methods. Evaluation

of laboratory performance is based on the "State-of-the-Art" concept which

uses participant's values to set acceptance and not acceptable limits.

Other systems for evaluation are discussed.

34
Appendix C. Temporal Effect of Continued Participation

in the CDC Proficiency Testing Program

35
•Ha

•H. +J

5-1

>^

tor ram

031^

O

O

(DO

CD •H

-M -M

oj t/5 CDU

I—<

a» •H

<+-fu

O •H

+J

O

u

<D. & m u m Q

rH o

t/)

(-> p: V) o

•H^ iPh O -H

t4-( CO o

•H

(D

O ^ U

I-H

CO

(i> COo X

oj I5

in

CD

+->

Cti

5-1 O

•H

NE Oh o

1—I

CU 1—1 Pi —I

t—1

•H

al o

IV 1 1 1 1 1 LO 1 1 1 oo OJ

rH 1 1 1 1 1 1 o •vj- (Nl 1 1 1 to LO

O 1 1 1 1 1 1 o O Oi 1 t 1 o t~-- iH 1 1 1 1 1 <^ ' 1 1 /*—

o o o OJ 1 I , CT> 00 to VO (N3 rg rH OO

00 rH 1 1 1 o VO <T> VO 00

I-H o o j 1 i o O CM o rH "sd- o rH rH • • 1 1 1 • • • • • • 1 o 1 o 1 O O

O)

cn 1 1 1 00 CSI oo LO 00

LO o 1 1 I rH

O CTi vO oo vO vO rH

I-H o o 1 1 1 O rH (sa o rH r-~- o rH to • • o 1 1 1 •1 • o •1 • o •1 • O

\o rsi iH rH oo o vO iH to rH O LO o tH vO O iH (N! to O o I-H o O LO o o to o o to iH eg O) rH OJ t:^- • • • • • • • • • • o o o o o o 1 o 1 O

I—1

to o LO to \o rH oo tNl OO oo LO LO

o> o to VO (NJ rH 00 CTi VO CTJ rH CTi CO o O VO o o LO O o -^t O iH rH o rH • • • • • • • • CD o o 1 CD 1 CD o VO o to

I-H to LO VO Ol o oo !>. vO

vO I-H LO CM to to to •<:^- -^t to oo LO LO o O rH \o o rH vO o CNl rH o oa LO • • • • • • • e • • 1 o 1 O 1 O 1 o 1 o

rH LO to Ol

oo to cn o to rH VD iH oo OJ

to r—1 O N-) o l>- CSI o rH (Nl LO

O LO VO

O Ol o rH OO O O 00 o rH OO o OJ rH O OJ !>- o • O • o • o 1 • o •1 • o •1 • o

LO o iH iH OJ

to o VO o rH

O to LO o to r-- o t—1

iH CD CD CD rH LO iH tH r*J 0~) 00 o iH iH O o rH o <NI rH o LO rH O • « O o o o O o 1 o /—\ /—\

ovo /—\ 1—\ /—\ > > i V ' > \ / >o Q Q Q Q Q Oh* Ti * Id

•(-> +-> CO z, 2 to :z; s :^ S to ;2: S c/5 2:

10o oo

4-1o

Oh>• H

4-> 03 rHO

OS

PL,

36
•HO

•HP

•HU

•H

+-> X

nti:

tor (CO

03

f-i O

bOO

<u ^ +-> OS p

•HP

(/i P (U

f—t m o o

CD

p •H u u

(D •H

Mho m

rH

03

?-l

Q o0

T—1

t—t

03 o

bo

•H

p

+-> o

Te>^p u

CD H

a

u •H

'Ho

top

(/)

p;

H

•H Ph Q (Uu

•H

c/:i

0) w

CD

+-> (D

to

CJ3 Q

CD

tsi Xt—I

03 I+->

QI—I <

37

<D P

oJ p

VI

bfl CD 3 03 P C O O

o3 I"

CDMXi-H03O

+-> 5 1 1 1 t-H Ln O vD cn 1 1 1 1 1 CTi (3~l 1 1 1 I-H vO cxi rH rH 1 1 1 1 I CXI rH 1 1 1 O O to O O rH 1 1 1 1 o rH to 1 1 1 o o O O 1 1 ' 1 '

LO

O rH

oo o oo OO (XI o OO o 1 1 1 o r-i \0 CTi rH ^ o o (XI CO rH LO rH 1 1 1 o o O O K) O rH o rH CXI O rH to 1 1 1 o to r 1 1 O O O o O

to cn (XI (X) o 00 !-H to OO O OO rH LO to OO

O (XIO MD OO (XI to to to LO to VO

O I—1 OO o rH to o rH to

O rH to o I—1

i' o O

O o O 1

(XI vO C7>

"^J- rH

O \o LO to OO to LO

r-H r-- to (XJ to o rH (J^ to \o o o OO CXI rH vO VO O O LO O t—1

vO o (^ o rH LO o (N l-O o (XI LO 1 o o o o O 1 1

I-H LO OOO to

(XJ 'd- vO LO

O (X] 1>> CTi to

O MD LO to o \o vO (XJ CTi LO VO VO (XI vO OO

O OO vO

O I-H vOO rH VO

O OO LO o rH LOO rH LO

O rH vO • • • • 1* o 1 o O o 1

(X3 to OO OO vO OO I-H (X! (X! to o to OO

I-H OO (XJO (XI (XI vO (XI LO o OO OO

OO l>-O rH OO o rH I>-

O CXI vO o rH VO o rH OO

1* o O

O o o 1 1

rH OO rH

O \o o LO

CJ\ I—1

LO LO to rH rH to

O CT> (N| rH CXI o LO LO

O rH I>, VO (XI LO to CO o O O Oi O rH I-H o <X) rH O (XI OO o rH o rH rH I O 1 O o O 1 ( /—\ /—\ / V o\o /—\ > > > >

CD CD (D CD CD • Q Q Q Q Oh

oi. ^ pi Id ^. 13 • +-> • -P p -p p> -P S 2: SCO s U5 2; CO 2: CO s 2 CO
r<l LT) t-H O t—I LO

oo o I—\

•H -M M C

to

O

i2£

CD e

•Hu -H •H ^o

fl +->

•H tin

CD <U

CD

t/5 %

CNI

oo

CNJ 1 1 1 1 i 1 LO CNJ o 1 1 1 to 1 1 1 1 1 1 ^ CNJ to 1 1 1 o o• • 1—1 11 1 1 1 1 1 1 1 1 11 O rH 1

rH 11 1 1 1 1

to

I—1 K) CNI 1 1 1 1 1 1 vO to rH 1 1 1 tn i>- iH 1 1 1 1 1 1 CN] to 1 1 1 O i-H CNI 1 1 1 1 1 1 1 1 1 11 O rH CNJ 11 1 1 1 1

to

m

O CNI I—I

r—1

vD oo o rH 1 1 1 rH rH lO LO to to \0 LO iH to 1 1 1 O CNI t-O o rH to O iH to O iH to J 1 1 • • 1 1 • 1 • • 1 ' 1 1

CNI CNI LO OO

t—1

<yt CN!

O Cn iH LO CN] 1—1

CN] to CT) tH

O CNJ CNI to iH LO LO

O r-l 1 o1

rH to • O iH 1 O rH 1 o1 o to o OO CNJ Csl vO O OO iH LO

1—1 ^ I—1

LO CN] vO to to LO rH C-~- CN]

O CNI LO

O rH •«:JO iH LO

O rH LO

O

O 1 1 • • 1

to

O \o

-tt C30 t-- o iH CNI ^ o rH CNI rct- I—1

CNJ to iH cn CN] to r-- o O rH \o o iH "vj- O iH VO O rH \0 o O LO • •

,* • ,* * 1 •

rH

\C> so to C30 rH 00 LO LO to o O CTi LO to to rH CN] ri- OO o O T-\ I>. O iH LO O rH 00 O rH OO o O vO 1 1 1 1 1

to rH LO OO oo to

CO

OO LO CT> CNI C30 cr> CNJ CNI I>- iH 1 CNa o 1—1 rH r-- O O o ^ CNJ rH vO O

CP CNI I—1 o iH

O iH 1—1 O iH rH o O rH • • 1 1 • • 1 • • 1 \

OX'S /—\

o\=> • w > >

^—

(1) i Q Q Oh a, • • • +-> • +-> • (-> +-> 2 00 CO 2 S to S S 00 12: CO 2:

13M

CD H rCP +-> CD

CD

QHH<

h-1

COWs

CD

tNl XrH

OJ I(-> rH

(D

CD

5-1 •H 38
o

•H

CX •HU /->

-t-J CD ^ 3

4-> o u

+-> ^O TO ^ J03 00O

03 bO •H

+-)

(/) ^ +-> (D (/) -M O C H I—

M-i

U

O

CD

+-> -H U U

CD -H ^ t+H M-i O

03

U^ Q O U U

r-

03 H

O

exdO </) C X • H +-I M C

CD

O

E^^

X-M

•H Oc U -H •H X U-l CO o

CD 0) •H C

CO 'd

CD

(/)

03 I—

m

CM

<D

+-> -P CD

CD

13

CD

•H

+-> oooo

to

O I-- I—I to oo o o to I

to o to O O rj- 1

C7^ to o 1^ r<i O O LO O LO \0 o o \o

to CO o o i:--

LO

O LO r:J- O O OO O LO CO o o cn I O to OO

I—I >0 Cs] O O I—

I S to Z O CO o to o O O ,-1 I

LO

to to

rH to O O r-H I oO I—

O vD O CNJ I o £^ to O '^t (XI o o to I

t-^ to \o

CD in -rt o o

LOto en

rH to O O LO

LO CXI <N1 O VO LO O VO

to CNJ O \0 I—

o o <»

to

O I—

O O I—

I w >

CD • Q• 4-> 2 CO ;z; o to \0 O CNl I—

O o o

LO

LO CJ) O (St \D o o o o

CT) to to O (Nl a> o cTt o o

OO

to LO \0 O (Nl I—

O O 1—

I o

to CT> rH O CNl to O O ,—

r-- to CO o to ^ O O o o

to vO to O \0 O O t-i o o

\0 OO CD O to OO O O 1—

o o

to

CNI to \0 o to o O O CXI o o

<D S CO 2:

39 I

fXl

CXI LO OO O CXI CXI O o o

CM

LO LO O CXI to O CXI o o

to

to

O CXI o o o to o o

CXI LO rH O (XI OO o o to

CXI CO O (XI CXI O OI o

OO OO to cn O to OO O O "St o o

\0

OO rH

O

O OO O O LO o o

CXI

!>. OO \0 o to O O M3 o o

CXI \0 CT> o to l>- O O i>- o o

(D • +-> S CO

LO o to O to rH o o -^i- o o

OO

t—t LO

LO rH CXJ O rH r-- o o

CD \0 LO

CXI OO r-- O O (3% o o

vo \o

rH r-- rH O O rH o o

CXI

O rH CXI rH OO to O O rH o o

LO

(XI [>~ O LO O O rH o o

CXI rH to O \0 O O rH 1 o

cn "vt OO O OO O O rH

to OO o o o O O CXI w >

(D • +-> S CO 2;

<DI

+-> o

a,
OH

CDa•H

•HO

to

•H

•a c

CO

CD

in

1—I W

00

(Vl (Nl 0 0 to to 0 1—1 • • 1 0

to

LO vO

LO 1—1

I— CNl • •

vO

<N1 00 0 0 to • • 0

(N) rH0 0 to • • 0 0

LO 000 to 0• • 0

to00 LO LO 00 0• • 0

to0 LO • • 0

00 \o i>- 00 0• • 1 0

i-H vO t—1 0 00 0 0 0 00 •1 0

(D S to Z

I—t LO O O LO o o

to

CTi 00 LO O to <:J- O O rH O O

CO

•ri- Ol 0 OJ to iH CNl 0 0 0 0 CNl • 0 0 • 0 • CD

CNl LO

00 0 CNl LO to 0 vO CNl LO 0 0 0 LO 0 to 0 0

to 00

CN3 T-i to 0 LO CNl LO \o 0 \o to 0

00 CNl to LO vD (NI I—1

\D \0 LO LO 0 0 0 0 0 0

00

1—(

to rH to vO 0 0 0 0 vO 0 0 0 0

CNl to 0 00 0 iH to CD r-H LO 00 0 0 rH 0 r-- 0 0 0

LO \o vO 0 <30 \o rg 0 kO to 0 \o iH 0 0 rH 0 0 rH 0 0 CD d /—\ > >

CD Q

4-> S 00 12; 1 1 1 to 1 1 1 0 to 00 1 1 1 0 0 00 1 1 1 •1 • 0

cn VD

rH t>> rH CTi to VO 0 to vD 0 0 0 0 iH • • • * 1 0 1 0

iH r-- LO to LO

rH LO 00 0 LO 0 0 CNl • • • • 1 1

rH cr> !> CNl iH to CNJ 0 LO 0 LO 0 to » • • • 1 0 1

LO

LO CNl LO

LO 0 LO CNl VO 0 • • • 1 0 1

VO CNJ LO to VO to vO VO 0 vO to 0 0 LO 1 0 1 0

CNJ 0 CNJ 0 \o 0 0 0 LO LO 0 00 0 0 0 CD 0 0

LO LO

t-~. CNl ^ rH

rH C--- CT) iH Cr> CNl 0 0 0 0 00 0 0 0 00 0

rH 00 VO 0 r-- 0 0 LO CNJ rH 0 <NJ 0 0 rH 0 rH iH • • 0 0 0 0/—V

v_y /—\

c,\o . V—' >

pi • +-> • 4-> S CO iz:

CD

fr-i USOu

CO

CO2 wQhH

Pio

rH

40
•Sco

•HM

•Hu

Oh •H

+-) o o u ao Co

>HO

CD

+-> 03 M

tr> •H

+-> cu to

+-> (D 1—

U o

0)

+-> •H u u

•H

m o w

I—

oj

u5h Q O U

mp

0)

CI.

I—

X)

cti Hi

O

•H 4->

tn o

•H Ph U -H

•H

X

O

to

•H P-H

<U CD •H C

00

<D

03 rHW

+-> <D

CD

eg

oo 1 1 1 to CNl ( 1 1 1 1 o to o 1 1 1 11 11 11 o1 o vO O CNl LT) 11 11 11

CN

!>- r—

K) 00 vO 1 1 , O CO o CTi CNl I—

r—

1 1 1 O o o o I—

o I—) I-H 1 1 1 • • 1 1 1

CTi oo

T—

CT) "d- 1 1 1 O o \o O r—

t—

LT) 1 1 1 o O o o CNl O t—

I-H 1 i 1 1 1 1

OO oo CTi vf CTi CT^ CT^ oo LO CNl LO o CSl o CTi o o vO t—

to o o CTi o O CNl o I—I CNl o o to . 1

oo CO o i>- I—

o CTi o tn I—

1—

[>. CNl CTi CNl o LO to o o 1—

o o to o o to o o LO

CNl CO t—

I—

CT> LO o MD vO \o oo CTi o LD oa o LO to CT^ CNl 1—1 o O I—

o o o O o o \0 1

IT) CTl LO LO (Nl LO to LO

O o (Nl I-H oo I—

LO CTi

O o 1—o o LO o rH LO o o oo 1

(Nl LO vo' r—

LO LO o oo LO to

O LO vO o o O to CN! CTi CNl o o i-H o o vO o I-H I>~ o o r—

1

1—

I-H vO

OO LO (Nl to to oo CO o !>~ CO o CT^ CNl NO (3^ CNl LO o O r-l o O o t—

oo o o r—

1 /—

/—

w >

CD • +-> S c/^ Z uo

03eou

•H

(D a,S c/^

w >

CD a,pi ^

41 o

rH (Nl cn O O CNl

to

to 1—

O (Nl \0 o o to OrH to vO O OO (Nl o to oo O O LO I

(NJ vO oo

to ^ CTi o (Nl > >

(D (D Q Q

ci 13 13

+-> •M 00 2

CO

4-> t—

O
n,

•HU

•H T)

-M

?H

03

B

•H

)nt u o u

03 tao

+-> 03 P • H

•M

(/)

•4-> (U

l-H

HO o c: M • H O u

CD •H

M-( <4-f O W Pr rH

03

u 5-1 Q O

5hOP

c/i

O

^«c: CD •HU -H

•H ^ M-l CO o

in P

P-i t/^ C -H

•H iX,

0) 0)

CO

CD

1—1

CD

p

CD

C30

cm

vO

oo

rH C^ O O £>

oo ^ 1—I OO O l-H

CSl LO LO oo O 1—I

T—I CNl

K5

LO I>- "^1- O (Nl O

\0 \o

OO

O -St-

(NJ I—I l>- rH O vO O O O vO • • I

I—I LD \0 O t—I O rH t» OLn

T-H r-- 1—I O r-H r-4 w >

CD • p 2 w
g

•H

-P

nJPh •Hu

•H

+J

?-i nJ Oh

tor

03

O

5h fi O

•H

4-> <D (/) d) <U M H

t-H

un

:^ CD •Hu o •H

M-l -M O U f-i (U

<+H CJ M Q

1—1O oC ^ •H +-)

c/1

O

•H

gPh

U -H

•H ^o

5-1 +->

f-l

C -H

•H PL,

CD <D

t3

(D

t/)

03

r—l W

<u I t t I I I

(Nl o 1 1 1 1 iH OO 1 I r-l 1 1 1 CTi 1 1 1 O O iH 1 1 1 CNJ 11 1 1 I 1 O O• • 1

LO 1 11 11 O iH to • • 1 1i 1 1 1 1 o to 1 1 1 I—

LO vO

CO 1 1 1 O CO <^ 1 1 1 iH to (DO 1 1 1 t-H 1 1 1 1 1 o o• • 1

I-H 1( 11 1 O rH• I1 1 1 1 1 o T—

r—l 1 LO LO LO 1 LD to 1 1 1 vO iH 1 1 1 i-H o o O O to 1 1 O OO (Nl 1 1 • • • • 1 1 1 • • 1 1 1

1—

to

LO CO t^- OO OO (Nl

(NJ I-H LO vO o O (Nl to o LO to

I-HO

O t-H bO

O CM toO (N (Nl o tH to • • • • • • • • • •

LO \^ CM 00 vO to

cn o LO

O OJ LO [>- 1-H

O \D o 00 iH o O LO O t—

LO

O CNI toO iH to . o \0 •1 • " • • • • • 1 • •

LO

CN3 I—

[\ to LO to ^H- LO o LO NO K5 t-H rH O vO to (Nl o O LO O I-H LO O rH O iH to O • • • • • • • • 1 •1 •

LO vO OO rH o o O \0 to LO to (Nl OO LO o OO o LO to CTi o o to LO !>- o OO cr> o o \o O rH 1-H o LO LO O LO o O "If • • • • • • • • • •

(N! LO 00 OO (N] rH LO LO oa LO LO vD iH CT) OO o o o LO OO t-H o o CZ:> r-\ T-\ o ^ OO o O I>- O (XI t-H o to vO o to o o rH LO 1 /—

o\o o\a > > >

CD 0) (D

QQ

Q Ph* Ph' Ph Ph* Pi pi pi -13 pi

+-> • +-> +-> • 4-" 2 W 2 s 2; to 2; S w 2 CO 2 winou

(L) Nl rH

03op

p

03JhU t/l •H 13) Ph Xo

(D rH •H

OS o

^-1 oN

g

>s

iH

HH 03 Anop<

43
in

on •H

On

•HU

•H

+-> --ci 5h CD

cti

iti

t02

cor o o

er M

I—1

(->

t/5 CU H o Xu o

•Hu

•H

w o

i-Haj

fHO

OiQ S U0 o• H

(-) V)

CD Hu f:^

•H^ g

;^

M-i CO O^ to

fH •H Ph

<D 0) •H (=: CO

CD

t/i

rH

CD

4->

COCNI

OO 1 1 1 1 i 1 1 1 1 ) 1 1 1 1 1 1 1 vH vo r~> 1 1 1 1 1 1 1 1 1 1 i 1 11 O o to 1 1 1 1 11 11 1 1

LO o to 1 1 1 rH r-- o lO CNI LO i I 1 rH o o o• • 1 o o1

to

O rH r-- 1 1 1 1 11 o1 O VO o r-t o OO

O OO (Nl LO LO rH LO LO rH vO o fvi rj 1—1 CT) rH LO CT> O 00 o rt LO O I—1 C7> o o rH o O rH O O o O iH • • 1 • • 1 • •

to (Nl rH to vD \0 CT) OO t-~- OO CTi LO O CT> rH to (yx O K) LO O tH iH CNI iH LO O VO vO O CNJ (SJ O 1—1

CN

O rH iHOO LO o O iH • • •

iH cr> LO I—(

OO I-O CNJ OO iH LO to LOO CNI 1—1 cr> OO to rH to to ^ o ^^ rH o O i-H 1 o to CNI o iH CNi o o• • 1 o o c^a

lO • to CNI tn "vj- to rH OO VO to iH LO OO OO LO

LO to LO rH

O LOOO o LO iH <p O iH o iH CNI o o 00 o o to • . f • • 1 • • • 1 «

to to LO to

-=t OO CTi

O to LO CTi to to VO iH t^ o CT> £~-- LO (O to to VO CNI rH LO o rH vO O O to "Tl- O iH o rH '^l- O rH iH o O LO • • 1 *

C30 to rH OO "=1- r-H to OO rH «^ OO LO tH o a\

LO r-- o o CSI to to CT»

O VO CNI rH vO vO

O bO o CO o rH -vt- o o rH

O o uo • • • • 1 " rH 00 to 00 CNI to \0 I—1

r-j o o rH o cr> LO rHO OO cn cr>

r~- t-H LO to toO LO o !>•O o O iH 00 O O rH o O rH . • • • • • • • 1 9 • /—\ <—\ /—\ \ ' ^—

—

/—\ /—\ /—\ /—\ /—\

o\o o\=> o\o • w > > v^ > > o 0) <D CD 0) Q Q Pi Ph' • +-> • +-> • +-> +-> S 2: S to ^ S 2: :s LO 2:

2: o

1—1 w % i

•H +J 1

(D CO rH g

OLE

tlO CTj rH ^

•Ho

rH

00

44
o M

to o

Teu

CD

<u •H & o •H

•HO

4-> to •H

•H

<D CDU H •H

CO

CU

to

CI, 03 I—1 m

+-> CD

CO 2 0)

C\l

CNl

\0

COLO vo (y> o (XI cr> O O (Nl o o

(XI "^t O CX] O OI o o t-~- O (XI \0 O -"^J- o o

CXI o o o to o O O LO o o

(XI

O CXI OO O LO

LO oO CXI LO O O I>. o o

CXI o vo O O OO o o O K-) LO O LO o O O r-H o o w >

CD • +-> S S

5-1 pio

a,

CD

Pi

CXI o (XI O O LO o o

LO OO O CXI O CT) o o O (XI CO O (XJ 1—I O O CN o o O vo K) O (X] to O O CXI o o O OO vo O (XJ OO O O CNJ o o

LO

O

O (X! CXI OI o O vO \0 O CXI O O 'd- o o

vo

(XI CTi

O

O bO OO o o o o

OJ

O r-t

O LO I—I O O OO

p:°

Pi

CO

CO

vo CD o en O O (XI I o

i-H to

1—I vO

O o o «^ o o

(XI (J^ [>- O vo o o ^ o O o O LO o O vOO LO o o

to

N-)

O

t—i vO O OI o

LO

to t—I O vO vO O O 00 1 o ^ ^ OO vO LO O O (XI

45

w >

CD Pipi 13 • +-> S W2

4-> CDI

4-> o

Ph

CD

CXI 1—I to O "d- O O LO

vO vO O t-- vO O

to

I--- CT) CTi O LO o O (XI o o

vO

OO o to O l>- O O (XI

to I—I CXI O O CXJ I o

\o to

O LO o o to o o

to

O £7) vO

I—I vO (XI 0 0-=^ I o

VO

LO CTl (X] O LO CD O OI o

OO CO o o to to O CXI OO I o' >

CD Q

Pi Id

OO

!—I to O o o o to o oo CXI c-~- O CXI CXI o o r-^ o o

vO

O o o to CX) O O rH o oo CvJ O LO O to CD O r—I o o O 00 CTl o to O O CNI o o

<X1 to to o O LO OO O (XI

(XJ

O ^ O OI o O vO O LO [>- O O '^t o o

LO

l>-

O LO (XI o cr> o o w >

0)

p., p;• -p s z QI—I

Pio
o

tlO IT) • H

+->

if) CD

CD S

•H PL, U -H

•H ^o

i-( -M C -H

•H pH

CD CD

E- -H

CO

CD

CD 1 1 rH 1 1 OO CT> r—1 o I 1 vO to 1 1 LO CO o O 1 i o o \o ( 1 o O LO o o 1 1 1 1 1 o LD LO r

o cvi rH rH rg VO (XJ rsi rH 1—1 o LD O O «^ o LO LO vO 1—1 o ^ LO

fNl o O \0 o o o o ,—( o rH UO o O CT> 1 o 1

to to (XI rH to C3^ CT) l>- o VO CT» O VO

CO o t—1 o to o to o CTi to oo o CT^ T^j- r—1 o O 1—( o o vO o o cx) o to o O rH • 1* oo (XI (NJ (XI LD I—1 LO 00 oo CNI to o r-. (SI rH <y\ to

U~)

O o LO o oo LO vO "^1" o CSJ T—1 o 1—1 rH o o 00 o o o rH rH o O fsj « 1 O 1

«^ to Csl

O o LO to vO VO I>- vO

(XI o o to oo

O LO LO (SI LOO ,—1 o O CN o o CT> O o to o f—1 CN) o O fNl o o 1

lO to VO CT> rH t-s rH oo rH LO en vO rH rH to r-- o CT> 00 o o O !>- LO o CTi vO o O CsJ o o (—1 o o o rH CXJ o o to o o ,* 1 1 o oo CO rH

t—1

oo oo oo oo C3^ CNJ LO LO o to to rH to oo o LO to o OO o , 1 ^ o (XI o CNJ o rH to o r—1 1 o

(XI ro rH oo \0 t>- oo oo 00 o o O VO vO VO o r—1 t—. o »^ O vD o CTi to ,—1 o 1—1 "sS" o o ,—1 O o LO o 1—1 "^t o rH LO

,* o o CNI to CsJ (XI oo o LO oo t~-- rH to rH CT» o un CD l-O o oo to to rH rH Q o o (XI vO o • 1 o 1 ^

^,.\'' ' ^ ^ /—\

/•^ /•rS ' ' . • • V ' / \ o\o > > > > >

0) CD CD CD 0) Q Q Q Q Q

Ph* •

+->

M . . S 00 z S to ;^ S w s ^ 2 S ;z:

CD

CD

t/1 CD

•Hoo

•ptfouo

•H

46
Table C3 lists the most frequently used analytical methods for each

of the constituents in the CDC proficiency testing data base.

Constituent

Glucose

Creatinine

Urea Nitrogen

Uric Acid

Total Protein

Cholesterol

Triglyceride

Calcium

Sodium

Potassium

Chloride

Table C3. Most Popular Analytical Methods

Methods

(1) AutoAnalyzer; (2) SMA

(1) Alkaline picrate (w/o Lloyds);

(2) AutoAnalyzer

(1) AutoAnalyzer; (2) SMA

(1) AutoAnalyzer; (2) SMA; (3) Phosphotungstatecarbonate

(1) AutoAnalyzer; (2) SMA; (3) Biuret, serum

standard; (4) Refractive index

(1) AutoAnalyzer; (2) SMA; (3) Direct serum

(acetic anhydride)

(1) Acetylacetone reagent; (2) Enzymatic

(1) AutoAnalyzer; (2) SMA; (3) Atomic . Absorption

(1) Flame Photometry; (2) SMA

(1) Flame Photometry; (2) AutoAnalyzer;

(3) SMA

(1) Cotlove Titrator; (2) AutoAnalyzer;

(3) SMA

Hematocrit

Red Blood Cell Count

Hemoglobin

(1) Micro- Hematocrit

(1) Coulter counter

(1) Coulter counter, model S; (2) Cyanmethemoglobin (manual)

47
Appendix D. Measuring Laboratory Performance in

the CDC Proficiency Testing Program

In very general terms, clinical laboratory analyses fall into two

procedural categories; quantitative tests (chemistry, hematology, differential blood films, some seriology and the radioisotopes) and qualitative

tests (bacteriology, cytology, urinalysis, some serology, and most of

immunohematology, and toxicology). The next few paragraphs discuss the

CDC proficiency testing program in these two basic categories.

Quantitative Tests

Reference laboratories, usually 100 or more, are used as prescribed

by regulation (Federal Register No. 253, Vol. 33, December 31, 1968,

p. 20047). Criteria for satisfactory performance for each sample are enumerated in paragraph 74.42; they basically require the use of three

subsets of limits. The first subset encompasses the central 95 percentile

calculated from the results of all applicant and licensed laboratories.

Laboratory results which are "obviously deviant from the other results"

are not used. No specific criteria or procedures are provided for determining which values are "obviously deviant," however, there is a stipulation that, for any sample, no more than five percent of the results

are to be discarded on the basis of being "obviously deviant." The

second subset of limits is determined from the reference laboratory results;

they encompass all the results and simply consist of the highest and

lowest values (the same procedure is followed for outliers as was just

described) . The final subset of limits (for clinical requirements) is

centered on the median reference laboratory result. "They encompass one- half of the normal population range, when the sample is in the normal

range, or one-half of the appropriate range in the case of a sample which

is outside the normal range." The normal population range or other

appropriate range is determined by the Secretary, DHEW. The results for

applicant and licensee laboratories must fall between the lowest lower

limit and highest upper limit of the three subsets of limits just

described.

For chemistry and hematology, a score of three is assigned to all

laboratory results within the tightest combination of limits around the

target value. A score of two is assigned to all laboratory results

within the next tightest combination of limits (but not within the

tightest combination) , and a score of one is assigned to all laboratory

i^esults falling within the outermost limits determined from the three

subsets of limits (but not within the two previously described limits)

.

All results falling outside of the outermost limits are assigned a score

of -1. All laboratory results having a score of one or greater are con- sidered acceptable.

The scoring methods for differential blood films and non- syphilis

serology are slightly more complicated. They can be found in a report

entitled "Development of Grading Methods," a report of the Study Group

48
for the Development of Grading Methods, presented at the First National

Conference on Proficiency Testing, Atlanta, Georgia, October 4-6, 1971.

Qualitative Tests

Qualitative tests differ from the quantitative tests in that in

addition to using a select group of three or more reference laboratories,

the samples are also sent to ten or more referee laboratories. The

reference laboratories analyze the samples in great detail while the

referee laboratories handle the samples in the same manner as do the

licensed laboratories being tested.

The grading procedures for the qualitative tests are much more

complex than those which were described for the quantitative tests. The

reader is directed to pages 20047-8, in the December 31, 1968 Federal

Register for a complete discussion of this subject.

49
Appendix li. Mean Measures of Relative Performance

The tables included in this appendix list mean measures of relative

performance (MRP) which have been found to be significantly different from

each other or from other subgroups. A separate table appears for each of

the 14 constituents. In each instance j the difference between individual

means (calculated from all samples covered in the two-year period) or subgroups was significant at the 0.001 level of probability. This significance was based on the approximate F-test made necessary by significantly

heterogeneous variances (p < .001).

Table El. Glucose

At least 15 laboratories used each of the following six analytical

methods. Means which are underscored by the same line are not significantly different from each other. Hence, methods numbered 7, 9, 4, 8

and 6 are in the same subgroup, as are methods 6 and 3; while method 3

is significantly different from methods 7, 9, 4 and 8.

Analytical Method Mean Relative

Performance

Number of

Laboratories

#7 - AutoAnalyzer -1.51 86

#9 - Other Automated -1.31 20

#4 - Direct Serum - .2% 47

#8 - 3VIA or Neocuproine 0.2% 93

#6 - Other Manual 1.7% 21

#3 - Ortho Toluidine 5.5% 17

7, 9, 4, 8, 6, 3

Table E2. Creatinine

There were five methods which were used by more than 15 laboratories

each. These five methods fall into four subgroups with only one instance

of commonality.

Analytical Method Mean Relative Number of

Performance Laboratories

#6 - Other Manual -22.3% 17

#2 - Alkaline Picrate,

w/o Lloyds - 5.1% 149

#8 - SMA - 3.5% 22

#7 - AutoAnalyzer - 3.3% 53

#1 - Alkaline Picrate,

Lloyds 16.1% 29

6 2 8, 7 1

50
Table E3. Urea Nitrogen

Analytical Method Mean Relative Number o£

Performance Laboratories

#8 - SMA -1.91 97

#1 - Urease-Nesolerization - .7% 15

#7 - AutoAnalyzer - .6% 80

#2 - Urease- Berthelot 2.8% 23

#3 - Diacetyl Reaction 3.6% 35

#6 - Other Manual 7.2% 15

#9 - Other Automated 39.0% 16

#5 - Kits 39.7% 17

8, 1, 7 2, 3 6 9, S

Table E4. Uric Acid

Analytical Method Mean Relative Number of

Performance Observations

#6 - Other Manual -8.6% 19

#1 - Phosphotungstatecyonide -3.8% 18

#2 - Phosphotungstatecarbonate -3.0% 73

#8 - SMA -1.6% 94

#7 - AutoAnalyzer 0.7% 64

6 1, 2 8 7

Table E5. Calcium

' Analytical Method Mean Relative Number of

Performance Observations

#7 - AutoAnalyzer -1.0% 32

#5 - Flame Photometer - .9% 24

u - mk - .1% 112

#2 - EDTA Titrimetric 0.4% 48

#3 - EDTA Colorijnetric 1.2% 27

#4 - Atomic Absorption .2.0% 27

#6 - Other Manual 2.4% 48

#9 - Other Automated 2.6% 20

7, 5 8 2 3 4, 6, 9

51
Table E6. Chloride

Analytical Method Mean Relative Number of

Performance Observations

//8 - SMA -.51 20

- Cotlove Titrator - 4% 54

#3 - Other Titrimetric -A% 44

#1 - Schales 0.6% 111

#7 - AutoAnalyzer 0.7% 33

IT J - Kits 1.0% 76

8, 2 3 1, 7 5

' Table E7. Total Protein

Analytical Method Mean Relative Number of

Performance Observations

#7 — Ai Tt'oA'n^i 1 vTi^T* - 7?; 32

#2 - Biuret, Serum -.6% 79

#8 - SMA -.5% 114

TT O - Refractive Index 0.6%

7 ? 8 =1

Table E8. Potassium

Analytical Method Mean Relative Number of

Performance Observations

#9 - Other Automated -4.8% 16

#7 - AutoAnalyzer - .3% 16

#1 - Flame Photometry 0.4% 169

#8 - 3^ 0.9% 17

9 7, 1, 8

Table E9. Sodium

Analytical Method Mean Relative Number of

Performance Observations

#9 - Other Automated -.7% 15

#1 - Flame Photometry 0.3% 168

#8 - SMA. 0.5% 18

#7 - AutoAnalyzer 0.6% 18

9 1 8, 7

52
Table ElO. Cholesterol

Analytical Method

#7 - AutoAnalyzer Extract

#3 - Ferric- Sulfuric

#8 - SMA

#5 - Direct Serum,

Ferric- Sulfuric

#4 - Direct Serum,

Acetic Anhydride

#6 - Other Manual

#9 - Other Automated

Mean Relative

Performance

-3.3

-2.5

-2.5

7 3

2A\

2.V

0.6!

5 4 6, 9

Number of

Observations

67

18

97

21

87

48

18

Analytical Method

Table Ell. Triglycerides

Mean Relative

Performance

#1 - Acetylocetone Reagent

#2 - Chromotropic Acid

#5 - Other

1.21

10.0%

13.7%

Number of

Observations

132

18

28

12 5

Table E12. Hemoglobin

Analytical Method Mean Relative Number of

Performance Observations

#9 - Other Automated -2.11 47

#1 - Cyanmethemoglobin -1.9% 124

#6 - Coulter S 2.1% 75

#8 - SMA 2.3% 20

#5 - Other Manual 21.1% 16

9, 1 6, 8 5

53
Table El 3. Red Blood Cell Count

Analytical Method Mean Relative Number of

Performance Observations

#3 - SMA

#2 - Coulter

#5 - Other Automated

#1 - Manual

3 2 5 1

-.6%

0.7%

2.6%

4.9%

16

126

15

30

Table E14. Hematocrit

Analytical Method Mean Relative Number of

Performance Observations

#3 - Other -4.8% 15

#2 - Micro Hematocrit ^ 0.4% 159

^'3 2

54
FORM NBS-114A

U.S. DEPT. OF COW, 1. Prui K. A l ION rv,;. Ki:!'(>Kr N'O. | 2. wov t Sca-sKion

BIBLIOGRAPHIC DATA ,\BSI R 75- 19 7 |

3. Recipient's .Accession .No.

4. TITLE .\Nn SbB lT riTClinical Laboratory^ PcrfoiTiiance .Ajialysis Using Proficiency

Test Statistics

5. Publication L)ate

6. Performing Organization Code

7. AUTHORlS)

Peter U'. Finkel and John V,'. Rown

8. Pcrf orr:iin £• Oi r.Jn i tion NB5IR 75-197

9. PERFORMING ORGANIZATION NAM)-, AND ADDRESS

N.ATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COM.MERCE

WASHINGTON, D.C. -0254

10. Project/Task,'V,ork Unit No. 4514457

11. Contract/Grant No.

12. Sponsoring Organization .Name .ir.J .'liJrcss Departnr:^nt of Health, Education and IVelfare

Office of the Assistant Secretaiy for Planning

and Evaluation

Division of Health Evaluation, Washington, D.C. 20201

13, Type of Report & Period Covered

Pinal 1/71 - 5/75

14. Sponsoring .Agency Code

15. SUPPLEMENTARY .N'OTES

This report supersedes NBS 10 685 - PB 215-287/6

16. ABSTRACT (A 200-word or less factual sumr.iary of most significant information. If document includes a sig bibliography or literature survey, mention it here.) The proficiency- testing aspects of the Cliiiical Laborator^.^ Iinprovemcnt Act

of 1967 prograjii were assessed. Tiie overall ability of licensed or volmteer

laboratories to accurately detenrine meai' values for any of the 15 constituents

was not sigiTificantl)^ altered cluring the first two years of program operation.

However, tlie variability of the laboratories has decreased over the t\\'o-year period.

It appears tliat the prograin has increased the consistency of laboratory performance.

The general level of laboratory ca'.Tability seem.ed to be independent of

involvement in state- supported or volurntar)' p^'oficiency testing programs other

than the CDC program i\tA of whether the working supennsor had a B.S. , M.S. , Ph.D. , or M.D. degree. Choice of analytical metliod did significantly affect performance.

Mthougli insufficient evidence v;as available to make a definitive statem.enr, the

data do not appear to support arguments favoring establishment of method- dependent

reference groirp target values.

Finally, -it appears that consideration should be given to alternative sampling

methods, such as reduced or skip-lot samj:)ling, for those constituents which appear

to present no analytical challenge to the licensed and volunteer laboratories.'

Greater emphasis r.dght tlien be placed on those constituents which give the

laboratories the most difficulty (cholesterol and creatinine, for example).

17. KEY Ut)R!U-> 'Alphabetical ord-.-r, separated '..y semicolon^.)

Accuraq.-; CLIA '67; clinical chemistiy; hem.atolog}- ; microbiology; precision;

proficicno.' tcstine

18. A\'A1LA iULlTV ^rAiT;MENl

[X^ L^NLIMI l ED.

r^FOR OFIKJAL Dl ST iU !<l TION . DO NOT RELEASE

TO N ITS.

19. SF.cURirV CLAsS

(THIS REPORT)

UNCL ASSIFIED

21. NO. OF PAGES

20. SECURITY' CLASS

(THIS PAGE)

UNCLASSIFIED

22. Price

USCOMM-DC e£244-P>71

II
I

I