# Presentation: 809562

## Developing an External Quality Assessment (EQA) Program

**Robert Martin, MPH, Dr.PH**
**Director, Division of Laboratory Systems, PHPPO, CDC**

## External Quality Assessment (EQA)What is it, and is there value?

**Thomas L. Hearn, PhD**
**Deputy Director**
**Division of Laboratory Systems**
** ****Public Health Practice Program Office**

## Quality Assurance

**Quality Control**

**External Quality Assessment or**
**Proficiency Testing**

**?????**

## Quality Assurance

**“****Planned and systematic activities to provide adequate confidence that requirements for quality will be met”  (ISO 8042, 3.4; NCCLS)**** **

## Mailing a Payment

- Reference bill

- Write check

- Address envelope

- Add return address

- Put on postage

- Add payment to envelope

- Seal envelope

- Put in mailbox

- Next Bill Mailed

- Payment rec’d and credited

## Quality Assurance

**Staffing**
- Qualifications
- Training
- Competency evaluation
- Periodic performance evaluation
**Infrastructure**
- Space
- Supplies

## Quality Control

**“****Operational techniques and activities that are used to fulfill requirements for quality”**
** ****(ISO 8042, 3.4; NCCLS)**

## Mailing a Payment

- Reference bill

- Write check

- Address envelope

- Add return address

- Put on postage

- Add payment to envelope

- Seal envelope

- Put in mailbox

- Next Bill Mailed

- Payment rec’d and credited

## T h e

**T h e**
** ****W a l l**
**  ****S t r e e t**
**   ****J o u r n a l **
**Monday: February 2, 1987**
**“****Medical Labs, Trusted as**
**  ****Largely Being Error -Free,**
**  ****Are Far From Infallible”**

## Testing Process

- _Analytic_
- Specimen processing
- Specimen storage
- Reagent preparation
- Equipment checks
- Test performance

- _Post-Analytic_
- Results review
- Quality control review
- Report test results
- Results interpretation

- _Pre-Analytic_
- Test selection
- Patient instructions
- Test requisition
- Collect, label,
-  and transport specimen

## Laboratory Quality Assurance

**Staffing / personnel**
**Quality control (QC)**
**Proficiency testing (PT) aka External quality assurance (EQA)**
***Infrastructure***

## Quality Assurance Model

**U.S. Laboratory Regulations**

***P***

***T***

***Q***

***C***

**S****T****A****F****F**

## Benefits of Laboratory Quality Assurance

**Laboratory mistakes are prevented**
**Significant improvements in testing performance can be achieved* **
***Research findings from the College of American Pathologists, the University of Wisconsin, and the Centers for Disease Control and Prevention**

## External Quality Assessment or Proficiency Testing

**“****A program in which multiple samples are periodically sent to members of a group of laboratories for analysis and/or identification; whereby each laboratory’s results are compared with those of other laboratories in the group and/or with an assigned value, and reported to the participating laboratories and others.”  (NCCLS)**** **

## Proficiency Testing or External Quality Assessment

**Central organization sends out challenge specimens for testing, laboratories’ results evaluated, laboratories sent scores**
**Split specimens exchanged with referral laboratories**
**Alternative methods**

## External Quality Assessment 

**Early warning-system for problems**
**Measure of laboratory quality**
**Valuable benchmarking tool (standardization and traceability)**
**Indicator of where to direct improvement efforts**
**Monitor of changes in technology and testing practices (evaluation component)**

## If even the most conservative estimates of gains in improved testing quality are used, thousands of testing mistakes may be prevented yearly by laboratory quality assurance programs.

## March 2000

## Value of EQA in HIV Testing  

**Example A**
- Does running external quality control samples with each EIA “batch” improve quality?

## Study  Design

**Study population: HIV labs in MPEP**
- voluntary participation
- approximately 600 U.S. labs per survey
- panel of 6 “patient” samples per lab per survey
- 5 surveys (Aug 94 - Aug 96)
- n=18,600
**Dependent variable: accuracy testing MPEP samples for HIV Ab by EIA**

## Using External Quality Control Reduced the HIV Error Rate on Performance Evaluation Samples

**  ****Sample HIV****	**** Number****	**** ------------Errors-------------****	****    ****	****	**
**  **_**Ab Reactivity**_**	**** **_**Correct**__**	**__**With QC**__**	**__**Without QC**__**	**__**p-value**_
**  ****Negative****	****    6168****	****	****      3****	****   ****	****     19****	****	****  *****0.014***
**	**
**  ****Strong****	****    4896****	****	****      4****	****     ****	****       3****	****	****  0.830**
**  ****Positive**
** **
**  ****Weak****	****	****    7021****	****	****  152****	****  ****	****   307****	****	****  *****0.0067***
**  ****Positive**
**  ****------------------------------------------------------------------------------------------------**
**  ****Total****	****	****  18085****	****	****   159****	****  ****	****   329****	****	****  *****0.0023***
**	****             ****(97.37%)****	****(2.20%)****	**** ****	****( 2.90%)**

**Using External Quality Control Reduced the HIV Error Rate on Performance Evaluation Samples**

**Source: Model Performance Evaluation Program, CDC**

## Value of EQA in HIV Testing

**Example B**
- Do technologists interpret Western blot bands accurately, and can they identify individual bands reliably?

## Western blot testing practices
scope of testing
how testing performed
interpretative criteria being used 
Testing performance on HIV...
highly reactive samples
weakly reactive samples and, 
non-reactive samples

- March 2000

**Western blot testing practices**
- scope of testing
- how testing performed
- interpretative criteria being used
**Testing performance on HIV...**
- highly reactive samples
- weakly reactive samples and,
- non-reactive samples

## Study Design

- March 2000

**Study Design**

**Convenience sample: participants in MPEP HIV testing program**** **
**Testing practices: from shipment results and a survey sent March 1999**** **
**Performance data: results for performance evaluation samples mailed in August 1998 and January 1999**** **

## Overall HIV Testing Performance August 1998, All Participants

- March 2000

**Overall HIV Testing Performance August 1998, All Participants**

**Source: August 1998 Performance Evaluation Shipment**

- 1171 (89.9%) - reactive
-       0 (  0.0%) - non-reactive
-   131(10.1%) – indeterminate

- WB

- 1302

- Number of EIA laboratories = 701
- Number of WB laboratories =  261

- Number of reactive samples = 16
- (5 weakly reactive)

- Positive

- 3545

## Overall HIV Testing Performance August 1998, All Participants

- March 2000

**Overall HIV Testing Performance August 1998, All Participants**

**Source: August 1998 Performance Evaluation Shipment**

- 129 (98.4%) - non-reactive
-     0 (  0.0%) - reactive
-     2 (  1.6%) – indeterminate

- WB

- 131

- Number of EIA laboratories = 701
- Number of WB laboratories =  261

- Number of non-reactive samples = 2

- Negative

## Overall HIV Testing Performance January 1999, All Participants

- March 2000

**Overall HIV Testing Performance January 1999, All Participants**

**Source: January 1999 Performance Evaluation Shipment**

-   986 (92.1%) - reactive
-       2 (  0.2%) - non-reactive
-     82 (  7.7%) – indeterminate

- WB

- 1070

- Number of EIA laboratories = 724
- Number of WB laboratories = 267

- Number of reactive samples = 16
- (5 weakly reactive)

- Positive

- 2919

## Overall HIV Testing Performance January 1999, All Participants

- March 2000

**Overall HIV Testing Performance January 1999, All Participants**

**Source: January 1999 Performance Evaluation Shipment**

-  253 (98.1%) - non-reactive
-      1 (  0.3%) - reactive
-      4 (  1.6%) – indeterminate

- WB

- 258

- Number of EIA laboratories = 724
- Number of WB laboratories =  267

- Number of non-reactive samples = 2

- Negative

## Sources of Discordant

- March 2000

**Sources of Discordant**
**Western Blot Interpretations**

**Incorrect bands with correct interpretation   ****~62%**
**Correct bands with incorrect interpretation  ****~36%**
**Incorrect bands with incorrect interpretation  ****~2%**

**Source: Serologic Testing for HIV-1: Status of Western Blot (WB) Testing and Inconsistent Interpretation**
**of WB Testing Results in a Model Performance Evaluation Program (MPEP) - 1989-1992. S.O.Blumer, W.O.Schalla, J.H. Handsfield, J.S.Hancock, G.D. Cross, and T.L. Hearn. [Presented June 7-11, 1993 at the IX International                     Conference on AIDS in Berlin, Germany]**

## Number of Different Blot Patterns, August 1998 and January 1999

| 
Samples’ HIV Infection Status | 
Number of Samples | Number of Different Blot Patterns for Each Sample |
| --- | --- | --- |
| Negative | 2 | 3, 5  |
| Positive | 11 | 7, 8, 9, 13 (2), 14,
16 (3), 28, 29  |
| Positive (Weakly reactive) | 5 | 29 (2), 38, 42, 48  |

**-Bands reportable: p17/18, p24, p31/32, gp41, p 51, p55, p66, gp120, and gp160 **
**-U.S. laboratories only**
**n= 187 in 1998 and n= 197 in 1999**

## Value of EQA

**Example C**
- Do laboratories that have participated in mandated laboratory EQA programs perform better than laboratories that have not participated?

## Proficiency Testing Failure Rates

- 1994 HHS approved proficiency testing providers

## External Quality Assessment (EQA)What is it, and is there value?

## Slide 34

## THANK YOU!