# Presentation: 036326

## Verification of Procedures in the Molecular Diagnostic Laboratory

- Jeffrey P. Massey, Dr.P.H., HCLD(ABB)
- Manager, Molecular Biology Section
- Michigan Department of Community Health
- Masseyj@michigan.gov

- 1

## Difficulties with Verification of Molecular Assays

- Lack of samples for testing (calibration, QC, or PT material)
- Molecular assays often measure a different target
- Molecular assays are more sensitive than the current “Gold Standard”

- 2

## Another Problem....

- CLIA regulations state _what_ must be done, but don’t give any specifics on _how_ it must be done.

- 3

## Accuracy

- The lab must document how it determined the new method produces accurate results based upon:
- Clinical sensitivity
- Clinical specificity
- Positive predictive value
- Negative predictive value

- 4

## Accuracy

- Verify the method produces the correct result (test reliabilitity)
-   _Number of correct results_ x 100%
-     Total number of results

- 5

## How to Demonstrate Accuracy

- Accomplished by:
- Test reference materials
- Compare test results vs. reference method (shown to provide clinically valid results)
- Compare split sample results

- 6

## Samples for Accuracy Study

- Patient samples with known result
- QC material
- PT material
- Calibration material

- 7

## Precision

- Measure of the extent that repeat testing is in agreement (reproducibility)
-   _# of repeated results in agreement_ x 100%
-           total number of results
- _     _

- 8

## Precision

- The lab will document how it evaluated:
- Day-to-day variance
- Run-to-run variance
- Within-a-run variance
- Operator variance

- 9

## How to Demonstrate Precision

- Repeat testing of known patient samples over time
- Test QC samples in duplicate and over time
- Repeat testing of calibration materials over time

- 10

## Analytical Sensitivity

- Measure of the smallest quantity of an analyte that is reproducibly distinguished from background

- 11

## Analytical Sensitivity

- The lab will document the lowest concentration or amount of the analyte that can be consistently measured.
- Minimum detection limits (how much analyte must be present to be consistently measured)

- 12

## How to Establish Analytical Sensitivity

- Dilutions of virus or bacteria of known quantity
- Pfu, ID50, direct quantitation by EM
- Quantitate amount of RNA or DNA extracted
- Control material of known concentration or copy number

- 13

## Analytical Specificity

- Ability of a method to detect only the analyte it was designed to measure

- 14

## Analytical Specificity

- The lab will document that the procedure will measure only the analyte intended to be measured
- The lab will also document interfering substances

- 15

## What should be tested?

- Closely related organisms
- Organisms that may be present from the same site
- Organisms that may cause similar symptoms

- 16

## Analytical Specificity:  Interfering Substances

- Document information regarding interfering substances from product information, literature, or own testing
- Specimen hemolysis
- Anticoagulant
- Lipemia
- Specimen type (stool, urine, etc.)

- 17

## Quality Systems:  Control Procedures(493.1256(3)(iv-v))

- Each day of testing:
- For tests with extraction phase, 2 control materials.  One must detect errors in extraction process
- Molecular amplification procedure:  2 control materials and another to detect inhibition (if inhibition is a significant source of false negative results)

- 18

## Reportable Range

- Establish the upper and lower limits of the test system
- The lab will document how the reportable range of patient test results was established

- 19

## Reference Range

- If the procedure is a quantitative assay, the lab will document how the normal range was established
- Reference range must be appropriate for the lab’s patient population
- Reflects type of specimen and demographics (e.g., age and sex)

- 20

## Conducting a Verification Study

- Planning
- Testing
- Analyze data & resolve discrepancies

- 21

## Conducting a Verification Study #1

- Planning
- Determine the number and type of specimens
- Establish acceptance criteria
- State the methods to resolve discrepancies

- 22

## How Many to Test?Suggestion #1

- At least 1 week parallel testing with existing method (minimum 50 samples)
- Example:  when converting from standard PCR to Real-time PCR

- 23

## Suggestion #2

- Test known representative samples (12 –15 common isolates, stock cultures or clinical samples):  total of 50 or more tests
- Example:  Implementing a new molecular assay.  Use samples confirmed as present or absent by another method

- 24

## Suggestion #3

- Confirm that 20 to 50 organisms agree in concurrent testing with current method or with results of split testing with another lab
- Example:  Use when implementing a method already in use by another lab

- 25

## Establish Acceptance Criteria

- A general rule of thumb is 95% agreement
- This can vary depending on the analyte being tested
- Performance criteria is not stated in the CLIA regulations

- 26

## Conducting a Verification Study – Step #2

- Testing
- Become proficient with the assay
- If FDA test – complete training, personnel competency, etc.
- If in-house – establish performance characteristics using a small verification panel
- Perform testing on blinded samples

- 27

## Evaluation of Non-FDA Approved HSV Assay

- Detection of HSV 1 & 2 by rapid-cycle real-time PCR using FRET
- Amplifications performed on Roche LightCycler
- HSV 1 and HSV 2 differentiated by melting temperature
- HSV 1 = 54oC
- HSV 2 = 68oC

- 28

- Amplification

- 29

- Determination of melting temperature

- 30

- HSV 1 Tm = 54oC

- 31

- HSV 2 Tm = 68oC

- 32

-          HSV 1                    HSV 2
-         Tm = 54                 Tm = 68

- 33

## Conducting a Verification Study – Step #3

- Data Analysis & Resolving Discrepancies
- Calculate accuracy, precision, reportable range, and reference range (and analytical sensitivity & specificity if in-house assay)
- Resolve discrepancies

- 34

## How to Resolve Discrepancies

- Sequence amplicon
- Determine clinical status of patient
- Test samples by another lab

- 35

## Summary of Results

|  | Neg | PCR HSV1 | PCR
HSV2  | Total |
| --- | --- | --- | --- | --- |
| Culture Neg | 9 | 0 | 1 | 10 |
| Culture HSV 1 | 0 | 8 | 1 | 9 |
| Culture HSV2 | 0 | 0 | 11 | 11 |

- 36

## Ct and Tm Data

|  | Culture | # | Ct | Tm |
| --- | --- | --- | --- | --- |
| HSV-1 | 9 | 8 | 26.5 | 54oC |
| HSV-2 | 11 | 11 | 25.8 | 68oC |

- 37

## Summary of Performance Characteristics

|  | Actual
Results | Lab 
Goal |
| --- | --- | --- |
| Accuracy
 | 95.2%
 | 95% |
| Precision | 100%
 | 95% |
| Analytical Sensitivity
 | 6 viral particles | 10 viral particles |
| Analytical Specificity | VZV(-), EBV(-), 
CMV(-)
 | No cross reactions |

- 38

## Evaluation of Non-FDA Approved HSV Assay

- Detection of HSV 1 & 2 by rapid-cycle real-time PCR using FRET
- Amplifications performed on Roche LightCycler
- HSV 1 and HSV 2 differentiated by melting temperature
- HSV 1 = 54oC
- HSV 2 = 68oC

- 39

## Summary of Evaluation Study

- Accuracy and precision within predetermined range of acceptability
- Excellent correlation results with cell culture
- Acceptable lower limit of detection
- No cross reactivity with other viruses
- Distinctive differences in melting temperatures:  54oC  vs. 68oC

- 40

## Final Step - Implementation

- Review and approval by Lab Director
- Maintain all documentation for 2 years
- Lab report to incorporate ASR disclaimer as needed
- Establish twice a year verification of assay ( in place of 3X PT)
- Assure ongoing QA

- 41

## References

- The CLIA regulations, _[www.phppo.cdc.gov/clia/default.asp](http://www.phppo.cdc.gov/clia/default.asp)_
- CLIA Interpretive Guidelines Subpart K, _[http://www.cms.hhs.gov/clia/appendc.asp](http://www.cms.hhs.gov/clia/appendc.asp)_
- Cumitec 31, Verification and Validation of Procedures in the Clinical Microbiology Laboratory, _[http://www.asmpress.org/browse/bound/index.asp?SelectedItem=10122](http://www.asmpress.org/browse/bound/index.asp?SelectedItem=10122)_
- CAP molecular pathology checklist, _[http://www.cap.org/apps/docs/laboratory_accreditation/checklists/checklistftp.html](http://www.cap.org/apps/docs/laboratory_accreditation/checklists/checklistftp.html)_
- NCCLS document MM3-A, Molecular Diagnostic Methods for Infectious Diseases
- http://www.nccls.org/es/source/orders/free/mm3-a.pdf
- NCCLS document MM6-A, Quantitative Molecular Methods for Infectious Diseases.
- http://www.nccls.org/es/source/orders/free/mm6-a.pdf

- 42

## Acknowledgements

- NLTN
- APHL
- Nancy Anderson
- Karen Mulawski, MA, MT(ASCP)SC
- MDCH Molecular Biology Staff
- Eric Belk
- Steve Dietrich
- Laura Guild
- Virginia Leykam
- Sonia Lugo
- Teresa Miller
- Laura Mosher
- Kevin Rodeman

- 43