The Challenges of Comparing a Clinically Validated Test to Other Methods (2013)

In the January 2012 issue of The Journal of Molecular Diagnostics, Lade-Keller et al1Lade-Keller J. Romer K.M. Guldberg P. Riber-Hansen R. Hansen L.L. Steiniche T. Hager H. Kristensen L.S. Evaluation of BRAF mutation testing methodologies in formalin-fixed, paraffin-embedded cutaneous melanomas.J Mol Diagn. 2013; 15: 70-80Abstract Full Text Full Text PDF PubMed Scopus (59) Google Scholar compared four research-use-only BRAF mutation detection assays-Sanger sequencing, pyrosequencing (PyroMark Q24 platform; Qiagen, Valencia, CA), TaqMan and Competitive Amplification of Differentially Melting Amplicons (CADMA), and the CE-IVD-marked and FDA-approved Cobas 4800 BRAF V600 Mutation Test (Cobas test; Roche Molecular Diagnostics A/S, Hvidovre, Denmark)-for their detection of BRAF mutations within a serial dilution of DNA blends and 28 FFPE cutaneous melanoma samples. Although we agree that improved assay performance is always critical to making appropriate clinical treatment decisions, there were several steps during this evaluation that may have impacted the overall results. The authors state that Sanger sequencing and the Cobas test failed to detect mutations in a significant proportion of the samples, which contained small fractions of tumor cells. We understand the challenges of obtaining adequate material for method comparison studies from limited samples, and it is important to adhere to each assay protocol to provide adequate comparison data. Nevertheless, we agree with the authors that the variability of tumor content estimation as reflected by the κ values obtained in the study may render different results and that it is important to standardize pre-analytic procedures. Though the authors state that macrodissection introduces an additional step (required for samples with <50% tumor content), this process has been successfully implemented in numerous molecular laboratories that interrogate cancer markers without disruption to the testing workflow. Despite alternate DNA isolation and macrodissection, the Cobas test had 100% concordance with the other assays for the identification of BRAF mutations that were 10% to 49% tumor content by all tissue (n = 9; four mutant samples) or by cell-dense tissue (n = 20; nine mutant samples). It has also been shown that there is no direct correlation of percent tumor content to percent mutant allele.2Halait H. Demartin K. Shah S. Soviero S. Langland R. Cheng S. Hillman G. Wu L. Lawrence H.J. Analytical performance of a real-time PCR-based assay for V600 mutations in the BRAF gene. Used as the companion diagnostic test for the novel BRAF inhibitor vemurafenib in metastatic melanoma.Diagn Mol Pathol. 2012; 21: 1-8Crossref PubMed Scopus (128) Google Scholar Further assessment of the low tumor content specimens should be performed to verify percent mutant allele compared to threshold sensitivity for all assays. Published validation studies have shown that the Cobas test has a >95% detection rate across formalin-fixed, paraffin-embedded tissue (FFPET) specimens with 5% mutant alleles using 125 ng per PCR reaction, a DNA amount typically obtained from a single 5 μm section.2Halait H. Demartin K. Shah S. Soviero S. Langland R. Cheng S. Hillman G. Wu L. Lawrence H.J. Analytical performance of a real-time PCR-based assay for V600 mutations in the BRAF gene. Used as the companion diagnostic test for the novel BRAF inhibitor vemurafenib in metastatic melanoma.Diagn Mol Pathol. 2012; 21: 1-8Crossref PubMed Scopus (128) Google Scholar Tested across multiple clinical sites, the Cobas test showed 100% reproducibility between sites for all specimens tested with >5% mutant alleles.3Anderson S. Bloom K.J. Vallera D.U. Rueschoff J. Meldrum C. Schilling R. Kovach B. Ju R.-J.L. Ochoa P. Langland R. Halait H. Lawrence H.J. Dugan M.C. Multisite analytic performance studies of a real-time polymerase chain reaction assay for the detection of BRAF V600E mutations in formalin-fixed paraffin-embedded tissue specimens of malignant melanoma.Arch Pathol Lab Med. 2012; 136: 1385-1391Crossref PubMed Scopus (117) Google Scholar We agree with the authors that molecular tests could falsely classify other BRAF mutations, such as the c.1798_1799GT>AA (V600K) mutation as c.1799T>A mutations. The Cobas test can reliably detect V600E mutations in cell lines and clinical FFPET specimens at >5% mutation level but does display cross-reactivity to V600K at >31% mutation level. Clinical and validation studies have shown that the Cobas test can detect >66% of V600K mutations.2Halait H. Demartin K. Shah S. Soviero S. Langland R. Cheng S. Hillman G. Wu L. Lawrence H.J. Analytical performance of a real-time PCR-based assay for V600 mutations in the BRAF gene. Used as the companion diagnostic test for the novel BRAF inhibitor vemurafenib in metastatic melanoma.Diagn Mol Pathol. 2012; 21: 1-8Crossref PubMed Scopus (128) Google Scholar, 3Anderson S. Bloom K.J. Vallera D.U. Rueschoff J. Meldrum C. Schilling R. Kovach B. Ju R.-J.L. Ochoa P. Langland R. Halait H. Lawrence H.J. Dugan M.C. Multisite analytic performance studies of a real-time polymerase chain reaction assay for the detection of BRAF V600E mutations in formalin-fixed paraffin-embedded tissue specimens of malignant melanoma.Arch Pathol Lab Med. 2012; 136: 1385-1391Crossref PubMed Scopus (117) Google Scholar It is difficult to know the impact of the nonrecommended DNA isolation method used for the Cobas test on the overall assay performance. Surely, the burden on the lab would be minimized if all DNA-based assays relied on a single, unified, DNA isolation method. The Cobas test requires the use of the Cobas DNA Sample Preparation Kit; however, DNA was extracted from all FFPET samples in this study with the Qiamp DNA FFPE Tissue Kit (Qiagen). In a previous study, the Qiamp kit showed a similar nucleic acid yield to the Cobas kit but with a higher level of RNA contamination in nucleic acids isolated with the Qiamp kit. Isolation of DNA from the Cobas kit resulted in significantly less PCR interference from contaminating RNA relative to the Qiamp kit.4Malhotra K.T. Gulati U. Blazer B. Wu H.Y. Comparison of DNA extraction methods from formalin-fixed, paraffin-embedded tissue and their impact on real-rime PCR-based mutation assays.J Med Diagn Meth. 2012; 1 (http://dx.doi.org/10.4172/2168-9784.1000107): 107Crossref Google Scholar Provision of accurate molecular diagnostic results that impact treatment decisions is paramount to the success of personalized health care strategies. The selection of patients for targeted therapeutics based on companion diagnostics could i) target the intended responder population whereas spare nonresponder unnecessary treatment, ii) predict effective dosing, iii) monitor response to treatment, iv) identify subgroups of a larger disease group that are eligible for treatment due to a poor prognosis, and v) minimize the potential for adverse drug events. Although technology drives molecular assays toward greater sensitivity/performance, lab-validated assays can appear to be equivalent to their approved counterparts. However, it is important to keep in mind the clinical utility and reproducibility of testing methodologies. Analytically and clinically validated companion diagnostics, under regulatory oversight, support issues of safety, utility, and reliability for downstream clinical decisions. Although this study was limited in terms of clinical sample size, the collected assays showed good performance relative to each other. The analytical performance of each test across a comprehensive set of samples, multiple test sites, operators, kits, and days will ultimately reveal the true robustness of each of these assays. Such analytical studies are required by pharmaceutical and diagnostic companies and regulatory agencies to ensure the safety of patients. The Cobas test has demonstrated excellent performance relative to the decision for vemurafenib treatment in clinical studies.5Flaherty K.T. Puzanov I. Kim K.B. Ribas A. McArthur G.A. Sosman J.A. O'Dwyer P.J. Lee R.J. Grippo J.F. Nolop K. Chapman P.B. Inhibition of mutated, activated BRAF in metastatic melanoma.N Engl J Med. 2010; 363: 809-819Crossref PubMed Scopus (2907) Google Scholar, 6Flaherty K.T. McArthur G. BRAF, a target in melanoma: implications for solid tumor drug development.Cancer. 2010; 116: 4902-4913Crossref PubMed Scopus (100) Google Scholar, 7Chapman P.B. Hauschild A. Robert C. Haanen J.B. Ascierto P. Larkin J. Dummer R. Garbe C. Testori A. Maio M. Hogg D. Lorigan P. Lebbe C. Jouary T. Schadendorf D. Ribas A. O'Day S.J. Sosman J.A. Kirkwood J.M. Eggermont A.M. Dreno B. Nolop K. Li J. Nelson B. Hou J. Lee R.J. Flaherty K.T. McArthur G.A. BRIM-3 study groupImproved survival with vemurafenib in melanoma with BRAF V600E mutation.N Engl J Med. 2011; 364: 2507-2516Crossref PubMed Scopus (5965) Google Scholar The improved clinical outcomes based on BRAF mutation-positive samples from melanoma patients for objective response rate and progression-free and overall survival support the benefit of the results from this clinically reliable test. We are grateful for the opportunity to respond to the comments raised in the letter by Palma and Shieh. The comments only concern our results using the Cobas 4800 BRAF V600 Mutation Test. The main issues raised were that macrodissections were not performed to increase the percentage of tumor cells for samples with <50% tumor cells and DNA isolation was performed with the Qiamp DNA FFPE Tissue kit. Since pre-analytic procedures for the estimation of tumor content have not been standardized, may give rise to variability in estimations made by different observers, are relative labor intensive, and require a skilled pathologist, we found it important to compare how different mutation testing methodologies are affected by low tumor cell content. Therefore, we decided not to perform macrodissections before DNA isolation. From our study it was clear that the more analytically sensitive assays8Kristensen L.S. Andersen G.B. Hager H. Hansen L.L. Competitive amplification of differentially melting amplicons (CADMA) enables sensitive and direct detection of all mutation types by high-resolution melting analysis.Human Mutation. 2012; 33: 264-271Crossref PubMed Scopus (19) Google Scholar, 9Lang A.H. Drexel H. Geller-Rhomberg S. Stark N. Winder T. Geiger K. Muendlein A. Optimized allele-specific real-time PCR assays for the detection of common mutations in KRAS and BRAF.J Mol Diagn. 2011; 13: 23-28Abstract Full Text Full Text PDF PubMed Scopus (90) Google Scholar were less affected by low tumor content compared to the less analytically sensitive assays.1Lade-Keller J. Romer K.M. Guldberg P. Riber-Hansen R. Hansen L.L. Steiniche T. Hager H. Kristensen L.S. Evaluation of BRAF mutation testing methodologies in formalin-fixed, paraffin-embedded cutaneous melanomas.J Mol Diagn. 2013; 15: 70-80Abstract Full Text Full Text PDF PubMed Scopus (59) Google Scholar Thus, our results underscore the notion that macrodissections should be performed for samples with low tumor cell content when using the Cobas test for diagnostic purposes. To perform meaningful comparisons between the different methods studied, it is important that the DNA samples tested are the same for all methods. Furthermore, if DNA is extracted from different slides for each method, intratumor heterogeneity with respect to BRAF-mutated cells would be a confounding factor. In our study, we used the standard method for DNA isolation in diagnostic molecular pathology at the Department of Pathology at Aarhus University Hospital (Aarhus, Denmark). After DNA isolation, we standardized the amount of DNA to what is recommended by Roche using a NanoDrop ND-1000 spectrophotometer (NanoDrop Technologies, Wilmington, DE). Also, the Cobas test, despite alternate DNA isolation, had a 100% concordance with the more analytically sensitive assays (pyrosequencing, TaqMan, and CADMA) for the identification of BRAF mutations when tumor content was >10%. Therefore, we do not believe that the overall results have been impacted by the choice of DNA isolation procedure. Evaluation of BRAF Mutation Testing Methodologies in Formalin-Fixed, Paraffin-Embedded Cutaneous MelanomasThe Journal of Molecular DiagnosticsVol. 15Issue 1PreviewPatients diagnosed with BRAF V600E mutated cutaneous melanoma show response to treatment with the BRAF inhibitor Vemurafenib. Different methods for BRAF mutation detection exist; however, only the Cobas 4800 BRAF V600 Mutation Test has been approved by the US Food and Drug Administration for patient selection. The results from this test depend on the percentage of tumor cells in the samples, which clinically may be estimated with substantial variation. We have evaluated five different methods: the Cobas test, Sanger sequencing, pyrosequencing, TaqMan-based allele-specific PCR, and Competitive Amplification of Differentially Melting Amplicons (CADMA), for detection of BRAF c.1799T>A (V600E) mutations in 28 formalin-fixed paraffin-embedded (FFPE) cutaneous melanoma samples. Full-Text PDF Open Archive