# Presentation: 112225

## External Quality Assessment for AFB Smear Microscopy

- Blinded Rechecking

## WHY THIS SYSTEM?

_**Not to :**_
**confirm diagnoses**
**rate individual technicians / labs**
**prove that controllers are better**

## RECHECKING MERITS

**Final aim: improve quality**
**screening for unacceptable performance**
**problem solving**
**Control of routine smears = **_**reality**_
**Continuous **_**motivation**_** and education**

## RECHECKING PROBLEMS

**Disadvantage:****	**_**workload!**_
**Impossible / useless without good program**
**regular supervision visits**
**highly functional network : intermediate labs !!**
**corrective measures**

## OVERVIEW

**First step : correct identification**
**of possible problem centres**
**LQAS sampling for least effort**
**not sure that all failing are weak :**
**	****      ****feed-back visit ; evolution ; type / nos. error**
**of **_**type of problem: FP or FN ?**_
**correct technique for QA !**
**Laos example**

## Slide 6

## OVERVIEW

**Second step: investigation for causes**
**False positives**
**lack of training / experience**
**very poor microscope**
**administrative errors**
**(crystals, artifacts)**
**False negatives**
**poor stain or staining technique**
**very thick smear**
**poor microscope / poor light**
**poor examination or overload**

## Slide 8

## ORGANIZATIONAL ASPECTS

_**Integrated**_** with supervision system**
_**Feed-back**_** for maximum reliability**
_**Complementary**_** levels of control**
_**Workload**_** depends on:**
**total turn-over & positivity-rate of the labs**
**level of decentralization**
**desired statistical accuracy**

## SAMPLING PRINCIPLES

_**As few as possible**_
**overload is counterproductive**
**balance statistical and technical accuracy**
**False positives: none, or systematic error**
**statistics not required for significance**
**---> check only few : **
**	****	****at frequency of occurrence**
**	****	****controllers / controlled directly comparable**

## Slide 11

## SAMPLING PRINCIPLES (2)

**False negatives: a few are unavoidable**
**target minimal sensitivity**
**---> **** Lot Quality Assurance Sampling**
**	****	****critical value of false negative surpassed?**
**Include low positives (scanty IUATLD/WHO)****	**
**difficult to confirm, but sensitive**

## TECHNICAL ISSUES

**Sampling:****	**_**random and representative**_
**all slides kept, properly identified****	**
**sampling by supervisor, not technician!!**
**select from register**
**check on completeness of collection**
**calculate sampling interval**

## TECHNICAL ISSUES (2)

**Screening: first controllers**
**blind checking absolutely necessary**
**avoid to overload the controllers**
**better sacrifice statistical power**
**max. 10 slides per day (+ other work)**
**same technique (i.e. no. of fields) ; good microscopes**

## TECHNICAL ISSUES (3)

**Re-staining**_** **_
**fading of fuchsin colour**
**true positives --> “false positives”**
**very fast under extreme conditions**
**always restain discordants**

## <footer>

- <date/time>

## Slide 17

## TECHNICAL ISSUES (4)

**Re-staining before first screening : **_**no consensus **_
**effect of fading more extreme on low positives**
**so false negatives ---> undetectable by controllers**
**poorly stained AFB fade faster**
**effect not only in extreme climates**
**restain not to miss gross deficiencies of stain/staining**
**so better restain all slides before first controls ??**

## Slide 19

## TECHNICAL ISSUES (5)

**Gold standard ???**
**controllers also make errors**
**countercheck : second control on discordants only**
**higher level, or head technician**
**no new bias : “false positive” --> false negative controller**
**false negative usually confirmed**
**may require long search**
**re-stain if not done earlier**
**blinding only to origin of both results**

## TECHNICAL ISSUES (6)

**Controllers must also be evaluated !**
**sample from their routine work ??**
**First controllers: compare FN rates with periphery**
**second control results ---> error by whom? what type?**
**rates denominators = numbers of reported results**
**only for sum of controlled vs. controller**
**validity problem if: **
**	****no errors for a bigger number of centres together**
**	****controller made more FN than periphery**
**Evaluation second controllers**
**send back slides with serious errors **

## SAMPLE SIZE

_**Sample size:**_** LQAS method**
**from industry, smallest sample**
**one-sided test: no exact results, wide confidence limits**
**	****except for bigger area, i.e. country**
**	****example: 3% false negative when sample size = 90 **
**	****	****1 centre : 0.8 to 9.1%**
**	****	****average 12 centres : 2.1 to 4.2 %**
**	****	****average 850 centres : 2.9 to 3.1%**
**“****probably not more than x% error in the lot”**
**accept if not more than “d” error in the sample**
**reject if more than “d” error in the sample**
**	****	****but not sure to be the reverse !!**

## SAMPLE SIZE (2)

**LQAS sample size tables : parameters**
_**acceptance number "d"**_** (~ specificity, power)**
**confidence level (95%)**
**definition & size of a lot: **
**lab annual turn-over recommended**
_**critical value false negative **_**: calculate using **
**positivity rate of labs **
**acceptable lower limit of sensitivity**

## Slide 24

## Critical value

**An upper threshold of the proportion of  false negatives**
**To be chosen from an estimate**
**Historical false negative rates**
**Calculation based on;**
**Prevalence of positives**
**Expected parameters of sensitivity and specificity comparing to controller.**

## Slide 26

## Slide 27

## Slide 28

## Blinded Rechecking

- -Determine sample size
- -Properly store slides
- -Collect a random representative sample
- -Ensuring blinding
- -Recheck smears
- -Resolve discrepancies between original result and result of rechecking laboratory
- -Feedback to province, district and local labs

## Determining Sample Size

**Slide Positivity Rate (SPR)** –
-    The number of AFB+ smears among all slides (diag. and monit.) in the laboratory from which the sample is being taken.
- This number is estimated using the lab registers from the previous year or the preceding four quarters.
- Sample sizes can be set using the average positivity rate for laboratory, region, or country.

## Slide Positivity Rate (SPR)

- 	**	****	****	**SPR=
- (Number of positive smears per year/Annual slide volume) X 100

## 
SENSITIVITY

The sensitivity, as defined here, is the detection of all positives, including low positives (1-9 AFB/100)

 The expected performance in detecting positives, as compared to the microscopists rechecking the slides. 

Acceptable sensitivity is determined by the National Reference Laboratory ( NRL).

A sensitivity of 75-80% is recommended 

- SENSITIVITY
- The sensitivity, as defined here, is the detection of all positives, including low positives (1-9 AFB/100)
-  The expected performance in detecting positives, as compared to the microscopists rechecking the slides.
- Acceptable sensitivity is determined by the National Reference Laboratory ( NRL).
- A sensitivity of 75-80% is recommended

## Determination of annual sample size

## Sample Size Determination Example

|           Procedure | Example |  |  |  |
| --- | --- | --- | --- | --- |
| Step 1 | Laboratory | Slides/yr | Pos/Yr
 | Neg/Yr |
|       Make a list of the microscopy laboratories in your country (or region in large countries( with the following information:

number of slides done per year
number of positive slides per year
number of negative slides per year | A
B
C
D
E
F
G
H
I
J
Total  | 1 500
2 550
1 990
2 085
900
1 158
1 250
885
2 569
500
15 387  | 200
351
156
151
85
100
125
101
335
55
1 659   | 1 300
2 199
1 834
1 934
815
1 058
1 125
784
2 234
445
13 728  |

##    Sample Size Determination Example

|                  Procedure | Example |  |  |  |
| --- | --- | --- | --- | --- |
| Step 2 | Laboratory | Slides/yr | Pos/Yr | Neg/Yr |
| Calculate the slide positivity rate (SPR) in each laboratory and round off to the nearest %.

SPR = (Number of positive slides per year / number of slides done per year) x 100

This is best done using Laboratory Register data from the previous year.  Both diagnostic and follow-up slides  should be included.
 | A
B
C
D
E
F
G
H
I
J
Total 
 | 1 500
2 550
1 990
2 085
900
1 158
1 250
885
2 569
500
15 387 | 200
351
156
151
85
100
125
101
335
55
1 659  
 | 1 300
2 199
1 834
1 934
815
1 058
1 125
784
2 234
445
13 728 
 |

## Sample Size Determination Example

| Procedure | Example |
| --- | --- |
| Step 3 |            

 Average SPR= (1659/15387) x 100 = 10.8% or 10% (rounded off)
 |
| Calculate the average SPR for your country (or region) and round off to the nearest %

Average SPR= )total positive slides/total number of slides) x 100 |  |

**Note:  If variation in slide volume or positivity rate among the centers in a supervisors' area is considered to be excessive, a few choices depending on the ranges of volume and positivity rate may be given.  In areas with extreme variability, collectors might even be given a list with individual sample sizes per laboratory based on each laboratory’s performance the previous year. **

## Sample Size Determination Example

| Procedure | Example |
| --- | --- |
| Step 4 | 
Average Workload = 13728/10 = 1373
           or 1000 (rounded off)
  |
| Calculate the average annual number of negative and round off to the nearest 1000

Average workload = number of the slides done/number of laboratories |  |

- Note: the sample size does not vary considerably when the annual workload exceeds 1000; therefore the rounding off will not affect the calculations.

## Sample Size Determination Example

| Procedure | Example |
| --- | --- |
| Step 5 | 

       80% selected

 0 selected |
| Decide on acceptable limits for performance in your country (or region)
Relative sensitivity (ability of technicians to detect AFB relative to the rechecking technicians)
        Recommended:
        75% if new program
        85% if established program
Acceptance number (maximum number of errors allowed before action is taken)
        Recommended:
        0 if limited resources available
        1 if adequate resources available |  |

## Sample Size Determination Example

| Procedure | Example |
| --- | --- |
| Step 6 | 
  Table V.1
  When choosing a different sensitivity or
  acceptance number, refer to the table in 
  Appendix D.2

  Average Number of Negative Slides = 1000

  Average SPR = 10%

  Sample Size = 96
  |
| Select appropriate sample size table.  Table V.1 can be used by most laboratories or regions if a sensitivity of 80% and acceptance number of 0 is chosen.

On the left side of Table V.1 look down the first column to find the average workload of negative slides in your country/region, as calculated above

At the top of the Table, identify the average SPR in your country/region, as calculated above

Locate the corresponding sample size at this point |  |

## Sample Size Determination Example

| Procedures | Example |
| --- | --- |
| Step 7 | 

96/4 = 24 slides to be collected every quarter |
| Decide on a convenient interval to select the slides

Recommended 4 x per year, i.e. Quarterly

Divide the required sample size by the interval to calculate the number of slides to be collected at every interval   |  |

## Sample Size Determination Example

| Procedures | Example |
| --- | --- |
| Step 8 | 

Suppose 250 slides have been processed during the last quarter.  24 slides need to be collected, therefore:   250/24= 10.4

Collect every 10th slide
 |
| Systematically collect the slides using the Laboratory Register

Divide the number of slides processed during the interval (e.g. quarter) by the sample size

If the slide is missing, select the next slide in the Laboratory Register, irrespective of the result and continue systematically using the sampling interval |  |

## Collecting Slides (Sampling)

- Slide Storage
- Slide Collection
- Slide Selection

##  

It is recommended that one quarter of the total sample size be collected during the quarterly supervisor visit.

- It is recommended that one quarter of the total sample size be collected during the quarterly supervisor visit.

## Laboratory Register:  
Begin with a random number  - consider using digit from currency (KZT) 
Example - sample every 10th slide

- Begin with a random number  - consider using digit from currency (KZT)
- Example - sample every 10th slide

** ****Sampling Example**

## Slide 45

## ANALYSIS OF RESULTS

**Numbers of errors**
**Grading of errors**
**cut-off for positivity: serious / minor error**
**HFP, HFN vs. LFP, LFN**
**priority for intervention: serious errors**
**quantification errors : secondary**
**indicative of stain problems if restaining was done **

## Slide 47

## ANALYSIS OF RESULTS

**Identify possibly poor centres **
**FN acceptance number surpassed? HFP?**
**Feed-back: supervision visit **
**listing + slides with serious errors**
**identify and solve problems**
**further validation (especially FP)**
**check on second level controllers**

## Slide 49

## Interpretation

- If there are no errors, the performance goal has been met. (95% confidence level)
- Even though the sample size is based on  sensitivity, it is still likely that one or more errors will be found even in laboratories that are performing at or above the expected level.
- If errors are detected, the interpretation and appropriate action may be different depending on the number and type of error, as well as the resources and capacity of the program.

## LIMITATIONS OF THE SYSTEM

**False alarm : chance errors**
**LFP always doubtful**
**(ignore when few & equally distributed)**
**Deliberate cheating ....**

## CAUSES OF ERRORS

**Many HFP, also many HFN (nonsense results)**
**does not know at all**
**gross technical problem (microscope?)**

## CAUSES OF ERRORS (2)

**HFP**
**single: administrative mistake**
**few, with some LFP :**
**poor identification, registration**
**confusing AFB with artefacts**
**(contaminated stain)**

## CAUSES OF ERRORS (3)

**LFP:**
**ignore if just a few & equally divided**
**majority of scanty is false, plus some HFP**
**confusing AFB with artefacts**
**excessive decentralization ?**

## CAUSES OF ERRORS (4)

**HFN (& LFN):**
**poor stains**
**poor technique (smearing, staining)**
**poor equipment (light, fungus)**
**often poor work/overload**
**rarely poor training **
**chance finding (by exclusion; **_**follow-up**_**)**

## CAUSES OF ERRORS (5)

**QE:**
**little experience**
**poor work/overload **
**poor equipment (light, fungus)**
**may point to **
**stain(ing) problems (if re-stained for controls) ---> “too low”**
**fading problem (if not re-stained) ---> “too high”**