# Presentation: 806977

## External Quality Assessment (EQA)

## The Quality System

**Occurrence Management**

**Information Management**

## The Problem

- A newly appointed supervisor of the laboratory has begun reviewing past EQA performance reports The reports indicated a trend in poor performance on interpreting gram stains.
- How should the laboratory investigate this problem?
- What are the implications for patient test results performed and reported during the same time period?
- What corrective actions might be taken?

## EQA

- Definitions
- Responsibilities
- EQA benefits
- EQA methods
- Documentation

## External Quality Assessment

- 	A system of objectively checking the laboratory’s results by means of an external agency

## EQA Methods

- Proficiency Testing
- Rechecking/Retesting
- On-site Evaluation

## Proficiency Testing

- “A program in which multiple samples are periodically sent to members of a group of laboratories for analysis and/or identification; whereby each laboratory’s results are compared with those of other laboratories in the group and/or with an assigned value, and reported to the participating laboratories and others.”  (NCCLS)

## Rechecking/Retesting

- 	Sampling of specimens or slides for rechecking by a reference laboratory
- Rechecking of slides is blinded
- Retesting (e.g. HIV) may not require blinding

## Onsite Evaluation and Review

- Periodic visits to the laboratory to assess laboratory practices
- Why?
- Obtain a realistic picture of laboratory practices
- Provide assistance with problem areas

## Laboratory Responsibilities

- Should participate in organized interlaboratory comparisons, e.g., EQA schemes
- Participation in EQA should encompass the extent and complexity of examination procedures used by the laboratory
*ISO Standard – prEN ISO/DIS 15189.2*

**Notes:**
This evolving ISO standard is entitled “Medical Laboratories – Requirements for Quality and Competence”.  It is now in the final phase of adoption.  Among its numerous clauses is this one relating to interlaboratory comparisons.  Once adopted at a national level then all medical laboratories in-country will have to meet this standard.  Where a recognized EQA scheme is not available then other mechanisms of interlaboratory comparison will have to be invoked.

## Management Responsibilities

- Develop policy
- Assign responsibility
- Monitor the results of external quality assessments
- Participate in the implementation of corrective actions
*ISO Standard – prEN ISO/DIS 15189.2*

**Notes:**
The ISO standard covers the same ground as GP26A.  It specifies the actions to be taken by the management of the laboratory concerning the results of interlaboratory comparisons.  
The requirements are twofold:  monitor the results and implement corrective actions when performance does not meet expectations.
The laboratory must identify the problem in the documented performance and carry out a root cause analysis before designing and implementing corrective actions.

## EQA Benefits

- Allows comparison of performance and results
- Serves as an early warning-system for problems
- Identifies systematic kit problems
- Provides objective evidence of laboratory quality
- Serves as an indicator of where to direct improvement efforts
- Identifies training needs

**Notes:**
Results may be compared between laboratories offering not only an opportunity for performances to be compared but also for systematic kit problems to be identified.
Training needs may be identified and evaluated.
The problems encountered in a panel may then be addressed in the next panel, creating a circular process.

## Proficiency Testing  Process

- Central organization sends out challenge specimens for testing
- Laboratories analyze specimens, return results to central organization
- Results evaluated, laboratories sent scores, performance report
- Laboratories take appropriate corrective actions and document

## PT Analysis / Reporting Protocols

- Handle in the same manner as patient/client samples
- Follow your testing strategy
- Use same method
- Analysis by the tech who routinely performs testing
- Conduct testing within required timeframe
- Communicate problems to supervisors, PT organizers
- Inter-laboratory discussion of PT specimens or  results are not permitted

## Rechecking/Retesting

- Most commonly used for smears, e.g., TB
- Must consider:
- Sample size
- Proper specimen handling
- Procedures to ensure blinding (for smears)
- Procedures to resolve discrepancies
*Details outlined in:  EQA for AFB Microscopy*

## Comparison of random blinded re-checking to non-blinded rechecking of AFB Smears in Mexico

|   Performance characteristics measured in rechecking program | Non-blinded sample
(1998) | Random blinded sample (2002) |
| --- | --- | --- |
| State A - Sensitivity | 99.7% | 86.7%	 |
| State B  - Sensitivity | 98.9% | 84.7%	 |
| State A  - Specificity | 99.8% | 99.1% |
| State B  - Specificity | 99.9% | 99.7% |

- Martinez et al. Int J Tuberc Lung Dis (In press)

## Results of laboratory quality assurance, 

| Laboratory | Number of results used in surveillance | Number of specimens re-tested | % of total laboratory specimens | Number discordant | Percent
discordant |
| --- | --- | --- | --- | --- | --- |
| Atholone Laboratory
 | 387 | 67 | 17.3 | 8 | 11.9 |
| Gweta Hosp Laboratory | 64 | 9 | 14.1 | 0 | 0 |
| Tutume Hosp Laboratory | 306 | 52 | 17 | 1 | 1.9 |
| Gansi Hosp Laboratory | 112 | 16 | 14.3 | 2 | 12.5 |
| Mahalapye Hosp Laboratory | 236 | 25 | 10.6 | 4 | 16 |
| Sefare Laboratory | 67 | 13 | 19.4 | 1 | 7.7 |
| Nyangabwe Hosp Laboratory | 459 | 75 | 16.3 | 5 | 6.7 |
| Hukuntsi Hosp Laboratory | 82 | 15 | 18.3 | 0 | 0 |
| DeborahHosp Laboratory | 310 | 52 | 16.8 | 1 | 1.9 |
| 11 Additional laboratories | --- | --- | --- | --- | --- |
| Total | 6407 | 1051 | 16.4 | 95 | 9.0 |

**Results of laboratory quality assurance, **
**2002 HIV Sentinel Surveillance, Botswana**

## On-Site Evaluation

- Conducted by:
- PT organization
- Laboratory leaders, e.g., provincial, district supervisors
- Consider:
- Frequency, e.g., monthly, quarterly
- Use of checklists
- Follow-up to visits
- Monitoring corrective actions
- Training

**Notes:**
Very poor performance may warrant an expanded visit to perform a comprehensive evaluation of laboratory procedures, assess quality practices, and assist with implementing corrective actions including providing training.

## On-site Evaluation Results – AFB Microscopy

| No  sand bucket | 96% | 27% |
| --- | --- | --- |
| No decontamination of sputum cups | 60 | 23 |
| No disinfectant | 31 | 4 |
| No biohazard waste bin (covered) | 48 | 8 |
| No lab coats | 48 | 46 |
| Technicians do not wash hands | 17 | 4 |
| Shortage of lab reagents | 75 | 27 |
| Improper sputum collection | 60 | 31 |
| Improper filling of lab register | 29 | 4 |
| Improper labeling slides/containers | 32 | 13 |

- Aziz, M. and G. Bretzel, Int J Tuberc Lung Dis 2002:6(4):340-349

**BEFORE**

**AFTER**

## Evaluation of EQA ReportsWhere are the problems?

- Pre-analytic
- Specimen compromised during preparation, transport, or after receipt by improper storage or handling
- Analytic
- Reagents, instruments, methods, calibration, QC - Analytical error
- Calculation
- Competency of Staff*
- Post-analytic
- Report format
- Interpretation
- Data
- Clerical/transcription error

**Notes:**
The problems may lie anywhere along the path of workflow.  
The integrity of the challenge may have been compromised during preparation, shipping or after receipt by improper storage or handling.
On the examination front great care should be taken to ensure that the EQA challenge materials do not exhibit a matrix effect in the examination system used by the participating laboratory.  Analytical problems should be investigated to determine whether any error is random or systemic
My next slide speaks to competency of staff but before going there let me emphasise that post-examination problems occur frequently and that incorrect data capture by the EQA scheme organizer is certainly not unknown.

## Investigating Unacceptable EQA Performance

- Clerical error
- Methodological problem
- Technical problem
- Problem with PT materials
- Problem with evaluation of results
- No explanation

**Notes:**
In 1999, NCCLS issued three approved guidelines dealing specifically with quality issues.  I have spoken about GP26A.  GP22A deals with continuous quality improvement and the third, GP27A addresses the effective use of external quality assessment to improve quality in the clinical laboratory.  
There is a very helpful section (section 7) on investigating unacceptable EQA performance.
It speaks to classifying the problem.  Note – problem not error.  The scheme shown here can be very helpful in carrying out a root cause analysis.  

(Random error)

## EQA Documentation 

- Establish written procedures for:
- Handling of samples
- Analysis of sample
- Recording of results on report forms
- Verification of clerical accuracy
- Use of statistical tools
- Handling corrective actions

## International Guidelines for 

**World Health Organization (WHO)**
**International Union Against TB and Lung Disease (IUATLD)**
**Royal Netherlands Tuberculosis Association (KNCV)**
**Association of Public Health Laboratories (APHL)**
**Centers for Disease Control and Prevention (CDC)**
**Japan Anti-Tuberculosis Association (JATA)**

**International Guidelines for **
**External Quality**** ****Assessment **
**(EQA) of AFB Smear microscopy**

## Proposed Quality System for Rapid HIV Testing: EQA

- Proficiency testing
- onsite administration)
- Retesting (serum or dried blood spot)
- statistical sample
- priority of new sites/personnel
- Onsite evaluation
- emphasize visitation for support and evaluation

## “...effective EQA never 

**“...****effective EQA never **
**comes cheap”**
**Van Deun, A.  External quality assessment of **
**sputum smear microscopy: a matter of careful **
**technique and organizations. IJTLD 2003: 7(6), **
**507-8.**