# Presentation: 019662

## Module 13: External Quality Assessment (EQA)

**On-site Evaluation and Re-Testing**

## The Quality System

**Notes:**
BE SURE THESE STATEMENTS ARE IN MODULE 5 WHEN THIS GRAPHIC IS FIRST INTRODUCED.
This framework has been widely introduced in African region as a common framework for ensuring quality.   It was based on:
Importance of the International Organization for Standardization (ISO), and recognition that some countries use ISO as a terms of reference
Need for simplicity - 
Emphasis on performance, not process
Convey importance of systems approach
Ensure quality of overall process
Detect and reduce errors

## Learning Objectives

- At the end of this module, you will be able to:
- Assess operations at test site to determine if quality requirements are met
- Take corrective actions following External Quality Assessment (EQA)
- Keep appropriate records related to EQA
- Avoid common problems associated with EQA specimen management

## Content Overview

- What is EQA and why is it important?
- EQA Responsibilities
- EQA Methods
- Proficiency Testing
- On-Site Evaluation
- Re-testing
- How to implement EQA

## External Quality Assessment (EQA):Definition

- 	Objective assessment of a test site’s operations and performance by an external agency or personnel

## Why EQA?

- Allows comparison of performance and results among different test sites
- Provides early warning for systematic problems associated with kits or operations
- Provides objective evidence of testing quality
- Indicates areas that need improvement
- Identifies training needs

## EQA: Conducted at All Levels of Testing

**NRL = National Reference Lab**
**PL = Provincial or Intermediate Lab**
**TS = Test Site (Point of Service)**

## Management Responsibilities:  Overview

- Determines policies for EQA (WHO, WHAT, WHEN, HOW)
- Assigns responsibility
- Establishes and maintains a system for assessment visits
- Schedules visits
- Conducts evaluations
- Receives EQA results and supports corrective action measures
- Monitors and maintains records
- Investigates deficiencies
- Manages corrective action efforts
- Communicates outcomes

**Notes:**
Some of the responsibilies here may be carried out by different individuals within the Ministry of Health;  Supervisor/Managers, Quality Assurance Committee, or Site Quality Officers

## Testing Personnel’s Responsibilities:  Overview

- Participate in the EQA program
- Take corrective actions
- Maintain EQA records
- Communicate outcomes to supervisors

## EQA Methods

**Proficiency **
**Testing**

**On-site**
**Evaluation**

**Re-checking/**
**Re-testing**

## What is Proficiency Testing?

- Panels of specimens are sent to multiple test sites by reference laboratory
- Test sites perform tests and report results
- Results indicate quality of personnel performance and test site operations
- Results are often compared across several testing sites

**Proficiency **
**Testing**

**On-site**
**Evaluation**

**Re-checking/**
**Re-testing**

## What is On-site Evaluation?

- Periodic site visits to systematic assessment of lab practices
- Focuses on how the lab monitors its operations and ensures testing quality
- Provides information for internal process improvement

**Proficiency **
**Testing**

**On-site**
**Evaluation**

**Re-checking/**
**Re-testing**

## What is On-site Evaluation? – Cont’d

- Also referred to as audits, assessments, or supervisory visits
- Learn “where we are”
- Part of every lab quality system
- Measures gaps or deficiency
- Collect information for:
- Planning & implementation
- Monitoring
- Continuous improvement

**Proficiency **
**Testing**

**On-site**
**Evaluation**

**Re-checking/**
**Re-testing**

## What is Re-testing?

- The process by which a random selection of specimens are collected from the routine workload at the test site and sent to the reference laboratory for validation
- Used to detect errors

**Proficiency **
**Testing**

**On-site**
**Evaluation**

**Re-checking/**
**Re-testing**

## EQA Should Lead to Corrective Actions

**Take **
**Corrective **
**Action**

**EQA Should Lead to ****Corrective Actions**

***“******Corrective Action”***
- An action taken to correct a problem or deficiency

- Examples:
- Production of an incorrect result
- Not following procedures

**Identify**
**Problems**

## Problems May Occur Throughout the Testing Process

- Specimen compromised during preparation, transport, or after receipt by improper storage or handling

- Reagents, test methods, QC
- Competency of staff

- Report format
- Interpretation

**Pre-Testing**

**Testing**

**Post-Testing**

## Take Corrective Actions

- Use problem-solving team:
- Investigate root causes
- Develop appropriate corrective actions
- Implement corrective actions
- Examine effectiveness
- Record all actions and findings

## Sample of Corrective Action Form

## How To Implement EQA

**Proficiency **
**Testing**

**On-site**
**Evaluation**

**Re-checking/**
**Re-testing**

## On-Site Evaluation Process

**On-Site Evaluation Process**

- 
**Pre-Evaluation**
**Preparation**

- 
**Exit **
**Conference**

- 
**Information**
**Gathering**

- 
**Reporting**

- 
**Entrance**
**Interview**

- 
**Outcome**
**Assessment**

- Observation of facilities,
- process, procedures

## On-Site Evaluation Checklist

## Tester Responsibilities: Ensuring a  Productive Site Visit

- Before Visit:
- Record keeping is essential. Get organized
- Confirm date of visit
- Review written policies and procedures
- Conduct internal assessment in preparation of site visit
- During Visit
- Participate in visits – cooperate
- Ask questions of site assessors
- After Visit
- Take corrective actions, where necessary
- Remember – visits are instructive, not punitive

## On-site Evaluation:   Pre-Evaluation Preparation

- Assign Responsibility
- Laboratory management
- Quality Manager
- Use Integrated team approach
- Determine who will conduct on-site evaluations
- Select auditors with necessary skills:
- Attention to detail
- Ability to communicate effectively
- Diplomacy
- Provide appropriate training
- Schedule site visits

## On-site Evaluation:   Pre-Evaluation Preparation

- Determine site visit frequency:
- Established sites - At least twice per year
- New Sites - Quarterly
- For efficiency, cluster evaluations geographically
- Schedule in advance
- Announced or unannounced visits

## On-site Evaluation:   Entrance Interview

**The entrance interview sets the tone for the entire visit**
- Be prepared, positive and courteous
- Introduce evaluation team - show identification
- Provide overview of process in terms of what will be done
- Review of facility
- Record review
- Observation
- Interview with testing staff
- Use of proficiency panel
- Exit interview

## On-site Evaluation:   Information Gathering

- Observe physical layout of the site
- Evaluate testing operations
- Specimen collection
- Observation of test performance
- Quality control
- Inventory of kits
- Record-keeping
- Use quality systems checklist
- Conduct in an instructional, not punitive, manner.

## On-site Evaluation:   Outcome Assessment

**Evidence of**
**implementation **
**& maintenance of **
**quality system?**

**Ensure visit is sufficient to objectively document evidence**
**Expand visit or observation**
**Reference deficiencies**

**Conclude visit**
**No deficiencies**

**YES**

**NO**

## On-Site Evaluation: Exit Conference

- Review findings with supervisory and testing staff
- Make positive statements first – acknowledge staff cooperation and support
- Address negative findings – allow test site to discuss findings and provide additional information
- Provide instructions and timeframe for submitting plan for correcting problems

## On-site Evaluation:   Reporting

- Include information:
- Site Name & Location
- Date of Visit
- Assessment Team Members
- Major Findings
- Recommendations for corrective actions
- Submit completed checklist and report to relevant authorities

## Example: Assessment Report

## Role Play: On-site Evaluation Visit

- Objective:  To experience situations which may compromise test results that an evaluator may observe
- Volunteers to play the following roles:
- Patient/Client
- Quality Manager
- Laboratory Manager
- Person performing tests
- Rest of the group will observe
- Role play time: 20 minutes

**Notes:**
Improper workstation setup
Possible things to observe leading to problem areas:
Cell phone ringing while conducting testing
Used lancet on floor – missed placing in discard bin
Time for test cut short...in hurry to go home
Missing or incomplete test records
Missing temperature logs
Improper performance of test
Breaching confidentiality

## How To Implement EQA

**Proficiency **
**Testing**

**On-site**
**Evaluation**

**Re-checking/**
**Re-testing**

## Issues to Consider Prior to Implementing a Re-testing Program

- What is the purpose of re-testing?
- Is re-testing feasible?
- Does technical capacity exist at reference lab?
- Can turnaround of re-testing be accomplished in a timely manner allowing for immediate corrective actions?
- What type of specimen should be collected for re-testing?
- How should EQA specimens be labeled and recorded?
- When should specimens be shipped/transported to reference laboratory?
- Which laboratory should re-test specimens submitted by test sites?

## Statistical Basis for Re-testing:Error Detection

| Volume
(Per Site) | 1%* error |  5%* error |   Retesting  Feasibility |
| --- | --- | --- | --- |
| Very low
50 spec |  Re-test 48
    (96%) | Re-test 31
(62%) | No |
| Low
500 spec | Re-test 225
    (45%) | Re-test  56
(11%) | Possible |
| High
5000 spec | Re-test 290
    (5.8%)      | Re-test 59
(1.2%) | Yes |

***95% Confidence**

## Re-testing: Example Sampling Plan

| Volume
(Per Site) | 1%* error | 5%* error |   Retesting
 Feasibility |
| --- | --- | --- | --- |
| Low
500 spec | Re-test 225
(45%) | Re-test  56
(11%) | Possible |

- For monthly re-testing:
- Determine the number of specimens required to detect desired error detection rate
- For 1% error detection =  225 specimens/month
- Divide 225 by 4 = ~56 specimens/week = ~11 specimens/day
- Select number of specimens per day randomly – e.g., every 3rd specimen
***95% Confidence**

## Re-testing Process

- Determine specimen type
- Determine sampling plan and time interval
- Collect specimens
- Store specimens until transport
- Package and transport specimens along with paperwork to designated laboratory
- Compare re-test results with site results
- Take Corrective actions, if needed

## Tester Responsibilities: Re-testing

- Follow written policies and procedures
- Collect appropriate specimen
- Record keeping is essential
- Take necessary precautions to avoid transcription errors
- Package and transport EQA specimens to designated reference laboratory
- Take necessary corrective actions

## Specimen Requirements

- Dried Blood Spots (DBS)
- 100 μl collected on labeled filter paper
- Store refrigerated in appropriately packaged re-sealable plastic bag
- Serum or Plasma
- 0.5 ml aliquot in labeled cryovial
- Store at 2-8oC for up to 1 week
- Store at -20oC or below if longer than 1 week

## EQA Specimen Transfer Log

## Specimen Management :Common Problems

- Transcription errors
- Mislabeling cryovial or DBS card
- From Lab register to specimen transfer log
- From reference lab to testing site
- Inadequate specimens

## Summary

- Describe your responsibilities in EQA.
- What is proficiency testing? On-site evaluation? Re-testing?
- Explain the process for on-site evaluation.
- What are some issues to consider prior to implementing a re-testing program?
- Explain the process for re-testing.
- What are some common problems associated with specimen management?