# Presentation: 045463

## Assuring the Quality of Laboratory Testing in Countries Fighting the HIV/AIDS Epidemic

- CDC
- November 29-30, 2000

## Test Verification&Test Validation

- Niel T. Constantine, Ph. D.
- Professor of Pathology
- Director Clinical Immunology

- University of Maryland
- School of Medicine
- And
- Institute of Human Virology

## Test Verification &Test Validation

- Considerations when determining the utility of   tests
- 	A.  Protocols for Evaluation of Tests
- 	B.  Reference Tests
- 	C.  Algorithms
- 	D.  Choice and Number of Samples
- 	E.  Testing Conditions
- 	F.  Resolution of Discordant Results
- 	G.  Indicators of test performance

## Considerations When Determining the Utility of Tests

- Protocols for Evaluations of Tests

## Protocols for Evaluation of Tests

- Essential to set guidelines.
- Must be followed exactly.
- Must outline all characteristics of samples and procedures.
- Must describe _detailed_ algorithm to follow for discordant results.
- Must include QA/QC section.

## Considerations When Determining the Utility of Tests

- Reference Tests

## Reference Tests

- Needed to fully characterize samples.
- Choice depends on purpose of testing.
- Concordance – against reference screening test.
- Accuracy – against confirmatory test.
- Must be careful about “pre-selected samples” to evaluate false positives.
- Should be tests that are recognized by the scientific community.

## Considerations When Determining the Utility of Tests

- Algorithms

## UNAIDS and WHORecommended Alternative Algorithms

- To maximize accuracy while minimizing cost
- Depends on objectives of the test and the prevalence of infection

## Objective of testing			Prevalences of infections		Testing strategy

-              according to test objective and prevalence of infection in the                                     de depistage du VIH, en fonction de l’obectif du test et de la
-              sample population				                prevalence de l’infection dans la population

- Objective of testing			Prevalences of infections		Testing strategy
- Objectif du dépistage			Prévalences de l’infection	Stratégie de dépistage

- Transfustion/transplant safety		All Prevalences
- Sécurité des transfusions/transplantations	Toutes prévalences		

- Surveillance				>10%			
- 				10%			

- 	Clinical signs/symptoms of	>30%			
- 	HIV Infection- Signnes
- 	Cliniques/symptôms de 		 30%			
- 	l’infection à VIHa

- 	Asymptomatic		>10%			
- 	Asymptomatique
- 			 	10%			

- aWorld Health Organizaion, Intenm proposal for a WHO staging system for HIV infection and desease (WER no.29, 1990, pp 221-228)- Organisation mondiale de la sante. Echelle provisoire OMS proposee pour la determinationdes strades de l’infecrtiono VIH et de la malodie (REN no 29, 1990. P.221-228)

## Slide 11

## Considerations When Determining the Utility of Tests

- Choice and Number of Samples

## Choice and Number of Samples

- Samples:
- Should represent population where test will be performed.
- Same matrix of sample (e.g. plasma).
- Must meet guidelines stated by manufacturer (e.g. not lipemic).
- Avoid multiple freeze/thaw, etc.
- Use “clean”samples.
- Multiple aliquots if possible.
- Must be well categorized.

## Choice and Number of Samples

- Samples:
- Should represent population where test will be performed.
- Same matrix of sample (e.g. plasma).
- Must meet guidelines stated by manufacturer (e.g. not lipemic).
- Avoid multiple freeze/thaw, etc.
- Use “clean”samples.
- Multiple aliquots if possible.
- Numbers of Samples:
- The more the better (min. 30 positives, 200 negatives).
- Depends on purpose of testing (e.g. blood donors).
- Include appropriate percent of variants.
- Perform precision and reproducibility studies (lg. Volumes).

## HIV Classification

- <date/time>

**HIV Classification**

- HIV

- HIV-1

- HIV-2

- O

-  M

-    A, B, C, D,
- E, F, G, H, I, J

*Types*

*Clades*

- ANT 70,
- MVP5180,
- VAU

- ROD           NIH2

*Groups*

-            _**Guidelines for Classification**_
- Types: 		HIV-1 and HIV-2
- 		     50% homology
-  Subtypes/Groups:  	HIV-1 group M, N and O
- 		     60-70%  homology
-  Clades:  		HIV-1 Clades A-J
- 		     >70% homology

- N

## Considerations When Determining the Utility of Tests

- Testing Conditions

## Testing Conditions

- Must test under identical conditions.
- (e.g. same lab, equipment, day, tech).
- Use non-expired kits that have been properly stored.
- Follow manufacturer’s recommendations.
- Sample integrity.
- Test in a blinded fashion.

## Considerations When Determining the Utility of Tests

- Resolution of Discordant Results

## Resolution of Discordant Results

- Check sample integrity, labeling, paperwork, and procedures.
- Repeat by same technologist.
- Repeat blindly by another technologist.
- Repeat reference test blindly.
- Repeat at different laboratory.
- Determine true status by other means.
- What parameters would these investigate?

## Resolution of Discordant ResultsPossible Variants

- Synthetic peptide tests
- Specific Western blots
- Specific IFAs
- Combination tests
- Dot blots
- Immunoconcentration tests
- Augmented blots and LIA
- PCR - specific

## Rapid Assay Evaluation Algorithm

- Rapid Assay +
- ELISA -

- Rapid Assay -
- ELISA +

- Discordant Results

- Repeat Rapid & ELISA

- Western Blot Assay (FDA Licensed)

- Negative

- Indeterminate

- Positive

- Resolved

- IFA (FDA Licensed)

- Negative

- Indeterminate

- Positive

- Resolved

- Sample Volume
- _>_ 1 mL

- Sample Volume
- (<1 mL & >0.2 mL)

- RT-PCR Assay

- Negative

- Positive

- Inconclusive

- Resolved

- P24 Ag Assay  (FDA Licensed)

- Negative

- Positive

- Inconclusive

- Ag Neutralization

- Positive

- Negative

- Resolved

- Resolved

## Considerations When Determining the Utility of Tests

- Indicators of Test Performance

## Indicators of the Value of a Diagnostic Assay

- Sensitivity
- Specificity
- Test efficiency
- Delta values
- Predictive values

## Sensitivity of Tests

- Sensitivity (epidemiologic)
- Sensitivity (analytical)
- Low titer
- Seroconversion
- Dilutions

## Indicators of the Value of a Diagnostic Assay

- Sensitivity = 	True Positives
- 			   	True Positives + False Negatives

- Specificity = 	True Negatives
- 			   	True Negatives + False Positives

- X 100%

- X 100%

## Indicators of the Value of a Diagnostic Assay

- Positive Predictive = True Positives
- 		Value		True Positives + False Positives

- X 100%

- X 100%

- Negative Predictive = True Negatives
- 		Value		True Negatives +False Negatives

## Predictive Values

*Assume: Test Sensitivity = 100% / Specificity = 99.5%*
- _Population #1_, where the prevalence of infection is high (5%)
- Population:		1000 sera tested
- 				50 	sera from infected individuals
- 				950 	sera from non-infected individuals
- Test Results:	50 positives:	45 from the infected group
- 						5 false pos from the non-infected group
- Therefore, the positive predictive value is:
- PPV = 45      = 90%
- 					     45+5
- 9 out of 10 positive results will be from infected persons

## Predictive Values

*Assume: Test Sensitivity = 100% / Specificity = 99.5%*
- _Population #2_, where the prevalence of infection is low (0.7%)
- Population:	1000 sera tested
- 				7 	sera from infected individuals
- 				993 	sera from non-infected individuals
- Test Results:	7 positives:	2 from the infected group
- 						5 false pos from the non-infected 						group
- Therefore, the positive predictive value is:
- PPV = 2      = 28.6%
- 					    2+5

## Predictive Values

- Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations

## Predictive Values

- Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations.
- The chance of a positive result being from a truly infected individual in the low prevalence population is only 28.6% (2 true positive detected by the test and 5 false-positives).

## Predictive Values

- Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations.
- The chance of a positive result being from a truly infected individual in the low prevalence population is only 28.6% (2 true positive detected by the test and 5 false-positives).
- This indicates that a positive result by the test will be from an infectd individual in only one of four cases (a guess could yield better chance!).

## Test Verification &Test Validation

- Quality Assurance and Errors
- 	A.  Common Errors
- 	B.  Quality Assurance Needs
- 			1.  Fundamentals of QA
- 			2.  Quality Control
- 			3.  Quality Assessment
- 			4.  Equipment Issues
- 			5.  10 Key Issues for QA

## Most Common Errors

- Transcription
- Carelessness
- Procedures
- Specimens
- Environmental conditions
- Pipettes and pipetting

## Clerical Errors

- Logging specimens
- Aliquoting
- Worksheets
- Result printouts
- Translating results
- Computer entering
- Reports
- Supervisory Review

## Specimen Problems

- Insufficient volume for repeating
- Hemolysis, lipemia, and bacterial contamination
- Insufficient and inadequate labeling
- Misidentified specimens
- Frozen / Thawed (multiple)

## Other Types of Errors

- Kit Dependent Problems.
- Technologist – dependent errors.
- Inter-lot variations and Intra-lot variations.
- Environmental problems.
- Non repeatable results.
- Inter-laboratory and Intra-laboratory variations.
- Equipment problems.

## Quality AssuranceFundamental for Quality Test Results

- Record keeping
- Monitoring laboratory staff
- Vigilance in the laboratory
- Verification of true positive and true negatives
- Parallel testing of resubmitted samples
- Reporting of results
- Confidentiality
- Interaction with physicians
- Storage of specimens for follow-up testing
- Laboratory efficiency
- Total quality management

## Components of Quality ControlRecord Keeping

- Kit lot numbers (expiration and open dates).
- Clearly label reagents with date opened or prepared (include open and expiration date) on each label.
- Daily temperature monitoring and recording i.e. Incubators water baths, ambient.
- Performance of controls and action taken when out-of-range.
- Photograph or clear photocopies of Western blots.
- Ratios of in-house controls to cut-off values.

## Components of Quality ControlControls

- _Kit controls_:  Use as directed by the manufacturer.
- _In-house controls_: preferably three levels to monitor variability between runs and lot numbers of kits.
- Low  positive – absorbance enough above cut-off that it should not be misclassified because of expected run-to-run variability.
- High positive – well above the cut-off.
- Negative – well below cut-off.
- _Storage of in-house control sera_:
- Dispense in aliquots sufficient for one week of use.
- Freeze at -20°C in a non-self-defrosting freezer.
- Thaw each aliquot once, store at 4 °C  when not in use, do not refreeze and discard after 1 week.

## Monitoring by External Controls

**Trend**

**Shift**

## Quality Assessment

- Internal Quality Assessment
- Known Reactors
- Unknown Reactors
- Blind Testing
- External Quality Assessment
- Proficiency Panels
- Blind Proficiency Panels

## Equipment Issues

- Pipette Calibrations
- _ESSENTIAL FOR ACCURACY_
- Frequency
- At least every 6 months
- Reasoning
- 1l inaccuracy = 10% error (total volume of 10 l)
- Controls – o.k., borderline specimens – loss of sensitivity

## Quality Assurance: What Must Be Done?10 Key Issues

- Detailed SOP with total compliance.
- Supervising review of all paperwork.
- Develop checklists for monitoring all activities.
- Dev. Organizational schemes for processing, documentation, and assessment.
- Monitor staff – blind proficiencies.
- Neat and complete documentation of all results.
- No deviation from procedures.
- Maintain confidentiality.
- Endorse safety measures.
- Vigilance.

## Test Verification &Test Validation

- III.  Introduction of a New Test
- 	A.  Selection
- 	B.  Characteristics
- 	C.  Approved versus Non-Approved tests
- 	D.  Continual Monitoring

## Selection

- Availability
- Appropriateness
- Cost and bulk purchases
- Shelf life and robustness
- Storage
- Publications and WHO evaluations
- Regulations

## Characteristics

- Laboratory capabilities
- Testing Purpose
- Simplicity
- Cost Concerns
- Sample type
- Test limitations
- Test principles and antigens
- Test indices

## Approved VersusNon-approved Tests

- Which can be used?
- When approved tests are unavailable.
- Validation of non-approved tests.
- Documentation necessities and qualifications.

## Continual Monitoring

- Necessity to monitor new tests.
- How long to monitor.
- Methods of monitoring.
- Looking for trends.
- Changing tests – Parallel testing.
- Documentation.

## Test Verification &Test Validation

- IV.  Special Considerations for Developing Countries
- 	A.  Selection of Tests and algorithms
- 	B.  Testing under non-optimal conditions.
- 	C.  Best fit Strategies
- 	D.  When Systems Fail

## Special Considerations for Developing Countries

- Selection of Tests and Algorithms

## Selection of Tests

- Infrastructure
- Supportability
- Expertise
- Accessibility
- Cost Concerns
- Algorithms

## Algorithms

- What’s effective?
- What can be used?
- Established and recommended algorithms.
- Use of _additional_ strategies.
- Differences due to geographical origins of samples.
- Cost effectiveness.
- Sample pooling.
- Blood donations vs. diagnostic testing.
- Different  algorithms within the same country.
- Epidemiological testing.

## Simple, Rapid Test Alternative Algorithm

- <date/time>

**Simple, Rapid Test Alternative Algorithm**

- Rapid Test #1

- Positive

- Negative

-  P/N OR P/P

-    Rapid Test #2*

- Repeat in Duplicate

*REPORT*

-  N/N

*REPORT*

- Positive

- Negative

*REPORT indeterminate*

- Resolve with other tests

- Report as Positive

- *Different configuration or antigens

## Special Considerations for Developing Countries

- Testing Under Non-optimal Conditions

## Testing Under Non-optimal Conditions

- Use of expired kits.
- Unsatisfactory environmental conditions.
- Limited number of test kits.
- Limited equipment (e.g. thermometers).
- Non-calibrated pipettes.
- Old equipment.
- Poor integrity of samples.
- Questionably labeled specimens.

## Special Considerations for Developing Countries

- Best Fit Strategies

## Best-fit Strategies(to Test or Not to Test?)

- Consequences and necessities.
- Cost effective strategies.
- Pooling of samples.
- Saving reagents.
- Parallel testing.
- Sequential testing.
- Mixing reagents.
- Alternate testing areas.
- Testing when temperatures and conditions fail.

## Pooling of Samples

- In what situations can pooling be used?
- How many samples can be pooled?
- Accuracy.
- Final sample dilution of pools.
- Proper sample size for evaluation.
- Effects of the presence of HIV _Antigens._

## Non-approved Testing Strategies

- Re-use of rapid tests.
- Modification of test kits:
- Cutting WB strips.
- Halving reagents.
- Pooling of samples.

## Special Considerations for Developing Countries

- When Test Systems Fail

## When Test Systems Fail

- Trouble shooting.
- Repeat testing.
- Alternative testing.
- Other personnel, other laboratories.
- Getting help.
- Documentation.
- Reporting.
- Consequences.

## Reasons for the Need for Improved Assays

- Early diagnosis.
- Resolution of indeterminate results.
- Differentiation of retroviral infections.
- Less expensive tests.
- Simple and foolproof tests.
- Detection of viral types and variants.
- Multiple combination tests.
- Detection of infection in the newborn.