# Presentation: 891917

## Review of External Quality Assessment (EQA) Systems

**Review of External Quality Assessment (EQA) Systems**

## International Guidelines for 

**World Health Organization (WHO)**
**International Union Against TB and Lung Disease (IUATLD)**
**Royal Netherlands Tuberculosis Association (KNCV)**
**Association of Public Health Laboratories (APHL)**
**Centers for Disease Control and Prevention (CDC)**
**Japan Anti-Tuberculosis Association (JATA)**

**International Guidelines for **
**External Quality**** ****Assessment **
**(EQA) of AFB Smear microscopy**

## External Quality Assessment 

- Early warning-system for problems
- Measure of laboratory quality
- Valuable benchmarking tool (standardization and traceability)
- Indicator of where to direct improvement efforts
- Monitor of changes in technology and testing practices (evaluation component)

## PL

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

**DL**

- NRL

**DL**

## Take Home Message

- The primary responsibility for National TB Reference Laboratories is supporting AFB microscopy networks through implementation of EQA and training

## International GuidelinesEQA – AFB Microscopy

- External Quality Assessment (EQA) Components
- onsite evaluation
- rechecking
- proficiency testing

## Key Features of EQA guidelines

- Resource analyses and phased approach for EQA
- Identify laboratories with performance problems---not intended to identify individual errors or validate patient diagnosis
- Emphasizes the need for onsite evaluation of laboratories using standard indicators
- Recognizes panel/proficiency testing as a method of EQA

## Key Features of EQA Guidelines Rechecking

- Emphasis on blinded random sample
- Emphasis on the smallest possible sample size that provides some information on test performance (sensitivity)
- Provides information to measure performance---based on the number and types of errors

## EQA  --  AFB MicroscopyOn-site Evaluation -- Background

- Many countries lack the resources for annual visits of peripheral laboratory by central laboratory staff
- Laboratories in most countries are visited by a non-laboratory District supervisor
- Optimum evaluation is performed by trained laboratory staff in supervisory role

## EQA – AFB Microscopy Guidelines Key FeaturesOn-site Evaluation

- Develop a standard checklist of questions and indicators
- Include minimal evaluation that can be performed by non-laboratory trained personnel (e.g., inventory supplies, reagents, equipment)
- Include detailed evaluation that can be performed by supervisory laboratory staff
- Train laboratory and non laboratory staff to assure consistent application

## On-site Evaluation Results - Uganda

| No  sand bucket | 96% | 27% |
| --- | --- | --- |
| No decontamination of sputum cups | 60 | 23 |
| No disinfectant | 31 | 4 |
| No biohazard waste bin (covered) | 48 | 8 |
| No lab coats | 48 | 46 |
| Technicians do not wash hands | 17 | 4 |
| Shortage of lab reagents | 75 | 27 |
| Improper sputum collection | 60 | 31 |
| Improper filling of lab register | 29 | 4 |
| Improper labeling slides/containers | 32 | 13 |

- Aziz, M. and G. Bretzel, Int J Tuberc Lung Dis 2002:6(4):340-349

**BEFORE**

**AFTER**

## On-site Evaluation Performed by Non-laboratory Staff Examples:

- Are all staining reagents available and within expiration dates?
- How are wire loops cleaned?
- Is the laboratory register present and all columns completed properly?
- How is maintenance on the microscope performed?

## Slide 13

**Notes:**
Quality Control in the TB Laboratory

Quality Control (QC) should be routinely
performed in the laboratory to ensure
reliability as well as reproducibility of results.

Perform QC on each new batch of stain. Observe these results before reading patient slides. This important step will verify correct staining procedure and microscope function.

When controls are not demonstrating proper staining,
determine the cause and correct it.

If the negative control appears red after the staining procedure, this is the result of incomplete decolorization. 

## On-site Evaluation Performed by Laboratory Staff Examples:

- Does the technician verify that the container is properly labeled?
- How are slides labeled?
- How often is the carbol fuchsin filtered?
- How many fields are examined to report a negative smear?

## EQA - AFB MicroscopyProficiency Testing - Background

- Uncommon in resource-limited countries
- Prepared smears (South Africa) or patient slides (Senegal) sent  from central laboratory
- Consistent challenge of laboratory test performance
- PT test performance may be different from testing  routine patient specimens

## EQA – AFB Microscopy Guidelines Key FeaturesProficiency Testing

- Laboratory may re-use patient slides but a procedure is provided to produce test slides for consistent slide sets
- Recommended slide set is 10 slides: 5 stained and 5 unstained
- Simple forms for slide production and collection of test results

## PT Implementation in Mexico

- Inspected 587 of 637 laboratories
- 604 microscopists given a 2 hour, 10 slide test
- 54% had score _>_80
- 33% had score 60-79
- 13% had score <60
- 536/604 (88.7%) finished all 10 slides
- 216 persons with score <80 received training followed by second PT:  average scores improved from 65 to 90 (P-value < 0.0001).

**Ref: Martinez-Guarneros et al, Int J Tuber Lung Dis 2003: 7(6) 516-521**

## EQA - AFB MicroscopyRechecking - Background

- Recommended by IUATLD and WHO
- Usually 100% of positive and 10% of negative smears
- Usually un-blinded — adds bias*
- Reviews patient testing; including smear preparation, staining, and interpretation
-   *Lan N.T.N. et al, 1999 Int J Tuberc Lung Dis 3(1): 55-61

## EQA – AFB Microscopy Guidelines Key FeaturesRechecking

- Emphasizes “blinding” and random sample using the laboratory register
- Sample size is based on Lot Quality Assurance Sampling (LQAS) with parameters selected for test volume and desired sensitivity
- Positives and negatives sampled
- Minor errors (FP or FN with 1-9 AFB/ 100 f) are included as a measure of performance

## Comparison of random blinded re-checking to non-blinded rechecking of AFB Smears in Mexico

|  | State A |  | State B |  |
| --- | --- | --- | --- | --- |
|  | 1998 | 2001 | 1998 | 2001 |
| True positives | 3400 |   314 |   517 |     72 |
| False positives |     11 |     55 |     2 |     11 |
| True negatives | 6599 | 6366 | 1728 | 3155 |
| False negatives |     11 |     48 |     6 |     13 |
| Total number of slides | 10022 | 6783 | 2253 | 3251 |
| Number of laboratories | 33 | 35 | 15 |     17 |
| Number of laboratories
with no errors detected | 25 (76%) | 4 (11%) | 11 (73%) | 6 (35%) |

**Provided by A. Martinez (InDRE) and B. Madison (CDC)**

## Comparison of random blinded re-checking to non-blinded rechecking of AFB Smears in Mexico

|   Performance characteristics measured in rechecking program | Non-blinded sample
(1998) | With random blinded sample (2002) |
| --- | --- | --- |
| State A - Sensitivity | 99.7% | 86.7%	 |
| State B  - Sensitivity | 98.9% | 84.7%	 |
| State A  - Specificity | 99.8% | 99.1% |
| State B  - Specificity | 99.9% | 99.7% |

**Provided by A. Martinez (InDRE) and B. Madison (CDC)**

## Sample Size Comparison of the old method* and new EQA guidelines

| Annual Volume | Positive | Negative | Old method
Rechecking | New EQA Rechecking |
| --- | --- | --- | --- | --- |
| 501 | 76 | 425 | 119 | 62 |
| 2219 | 44 | 2175 | 262 | 197 |
| 6650 | 138 | 6512 | 789 | 208 |

*** ****Sample 100% positive and 10% negative smears**

## EQA - AFB Microscopy Guidelines Key FeaturesResource Analysis

**Inventory available resources (actual/projected)**
**Manpower, supplies, communication, administrative, financial**
**Examine effectiveness of current EQA activities**
**Gather laboratory service information**
**Planning—options for the evolution of EQA**
**Pilot test and document changes**
**Expansion based on availability of resources **

## Resources to implement on-site evaluation The NTP will need to consider:

- Capacity of the reference laboratory staff to provide on-site evaluation of all intermediate laboratories at least annually.
- Capacity of intermediate laboratories to provide on-site inspection of the peripheral laboratories at least annually, and more frequently to correct problems identified.
- Availability of properly trained non-laboratory personnel to make supervisory visits at least quarterly (as required for DOTS).
- Capacity to implement necessary QI measures.

## Resources to implement panel testingThe NTP will need to consider:

- Available financial support.
- Proficiency of reference laboratory staff to perform ZN AFB smear microscopy.
- Capacity of the reference laboratory staff to prepare panel testing slide sets for the laboratories to be evaluated
- Available mechanisms to deliver slides to the peripheral sites, including mail and couriers.
- Capacity of the reference laboratory staff to review and evaluate results from peripheral laboratories, and provide recommendations and follow-up for corrective action.

## Resources to implement blinded rechecking The NTP will need to consider:

- Available financial support.
- Capacity of peripheral laboratories to store smears for rechecking.
- Availability of properly trained personnel to collect appropriate samples of slides from peripheral sites.
- Capacity of the reference laboratory staff at central and intermediate level laboratories to reread smears from peripheral sites, including second rereading to resolve discrepancies as needed.
- Capacity of reference laboratories to provide results of rechecking as well as feedback to implement effective corrective action.

## EQA – AFB Microscopy GuidelinesResource AnalysisPhased Approach

**Assure the five elements of DOTS **
**Develop a central reference and intermediate laboratories to carry out EQA **
**Determine the existing capacity for EQA**
**Train district health officials to evaluate the minimal functions of microscopy laboratories**

## EQA – AFB Microscopy GuidelinesResource AnalysisPhased Approach (cont)

- Proficiency testing to evaluate performance
- Pilot rechecking program
- Determine resources: additional PT or phased implementation of rechecking

## EQA References 1998 -2003

**2. Nguyen TN, et al.  Quality control of smear microscopy for acid-fast bacilli: the case for blinded re-reading. Int J Tuberc Lung Dis. 1999 Jan;3(1):55-61. **
**3. Nguyen TN, et al.  The importance of quality control of sputum smear microscopy: the effect of reading errors on treatment decisions and outcomes. ****Int J Tuberc Lung Dis. 1999 Jun;3(6):483-7. **
**4. de Kantor IN, et al.  ****Periphery to centre quality control of sputum smear microscopy and 'rapid fading' of Ziehl-Neelsen staining. Int J Tuberc Lung Dis. 2000 Sep;4(9):887-9. **
**5. de Kantor IN, et al.  More on periphery to centre quality control of sputum smear microscopy and**
**'rapid fading' of Ziehi-Neelsen staining. Int J Tuberc Lung Dis. 2001 Apr;5(4):387-9. **
**6. Fujiki A, et al. Quality control of sputum smear examination in Cebu Province.**
**Int J Tuberc Lung Dis. 2002 Jan;6(1):39-46. **
**7. Aziz M, Bretzel G.  Use of a standardised checklist to assess peripheral sputum smear microscopy**
**laboratories for tuberculosis diagnosis in Uganda. Int J Tuberc Lung Dis. 2002 Apr;6(4):340-9. **
**8. Buzingo T, et al   Systematic restaining of sputum smears for quality control is useful in Burundi. Int J Tuberc Lung Dis. 2003 May;7(5):439-44. **
**9. Paramasivan CN, et al.  Quality assurance studies in eight State tuberculosis laboratories in India. Int J Tuberc Lung Dis. 2003 Jun;7(6):522-7. **
**10. Martinez-Guarneros A, et al.  ****Implementation of proficiency testing in conjunction with a rechecking system for external quality assurance in tuberculosis laboratories in Mexico.****Int J Tuberc Lung Dis. 2003 Jun;7(6):516-21. **

**EQA References 1998 -2003**