Title: Ensuring test accuracy evidence is fit for clinical and policy decision making: a new paradigm of contrast-based test evaluation
Funder: Department of Health and Social Care

Abstract: Background Medical tests used in the NHS should be safe and improve health outcomes for patients. Healthcare organisations, like the National Institute for Health and Care Excellence (NICE), use evidence from relevant, high quality primary studies and systematic reviews to inform their decision making. Evidence for the appraisal of tests includes diagnostic accuracy evidence obtained from studies that assessed how well a test identifies patients with disease (sensitivity) and those without disease (specificity). Such studies are known as diagnostic test accuracy (DTA) studies.Historically, researchers have focussed on estimating absolute values of the sensitivity and specificity of individual tests, even in studies and systematic reviews that compare the accuracy of tests. However, most clinical management questions are comparative: we really want to know about test selection for making clinical and policy decisions, i.e., how does the sensitivity and/or specificity of a new test compare to that of an existing test or current practice (comparative accuracy)?Sensitivity and specificity values for a test in a systematic review often show high heterogeneity, limiting interpretation and applicability of the findings of a DTA meta-analysis. Observing high heterogeneity downgrades the certainty of the evidence. My hypothesis is that if we examine differences in sensitivity and in specificity between two tests (contrasts), heterogeneity in the differences may be less, enabling better healthcare decisions to be made using high quality comparative evidence. Conventional methods for comparing test accuracy in a meta-analysis (comparative meta-analysis), focus on estimating summary estimates of absolute values of sensitivity and specificity. Therefore, a paradigm shift is needed from conventional approaches to a contrast-based approach for comparative meta-analysis. The new contrast-based approach will meta-analyse comparisons of test accuracy made within each study.Aims To develop new methods for contrast-based comparative meta-analysis of DTA studies and formats for presenting results.To develop and evaluate measures for quantifying heterogeneity in contrast-based meta-analyses.To investigate study size as a potential source of heterogeneity in contrast-based meta-analyses.To illustrate impact of contrast-based methods on decision making, and to develop guidance for rating certainty of comparative accuracy evidence. Plan of investigation I will develop new meta-analytic methods for pooling measures of absolute and relative differences in test accuracy (contrast-based methods). Using empirical data from 58 comparative reviews and simulation, I will assess the performance of the new methods against conventional methods. Various ways of translating the meta-analytic results into meaningful results for decision making will be explored through a focus group.To identify existing methods for describing heterogeneity in a meta-analysis, I will perform a review of the literature. The principles and assumptions underpinning each method will be summarised, and their applicability for DTA meta-analysis will be examined. Existing methods will be adapted or new methods devised for use in contrast-based DTA meta-analysis as required. The methods will be evaluated as part of the simulation study.Using a cohort of DTA reviews identified from the literature, I will use contrast-based methods to assess the impact of study size on comparative accuracy. Finally, using a case study of nine systematic reviews on diagnosis of melanoma, impact of the new methods on decision making will be demonstrated through assessments of the certainty of comparative evidence. Based on insights from the case study and with input from the advisory group, I will then develop guidance for rating comparative accuracy evidence.Potential benefits to patients and the NHS Correct diagnosis is essential for appropriate treatment. Using the new methods to obtain comparative accuracy evidence, we will produce higher quality evidence and robust answers to clinically important questions relevant to the NHS and patient care. Better decisions about test selection should be made.

Lay abstract: Background Medical tests used in the NHS should be safe and lead to improvements in health outcomes for patients.The best evidence comes from randomized controlled trials (RCTs) comparing tests, including the effects of subsequent interventions on patients. However, RCTs are rarely available. Therefore, healthcare organisations, like the National Institute for Health and Care Excellence (NICE), rely on information from studies that assess how well a test gets the diagnosis right in people who have the disease (sensitivity) and people who do not have the disease (specificity). Such studies are known as diagnostic test accuracy (DTA) studies. Studies that compare the accuracy of two tests (comparative accuracy) for diagnosing the same disease can be used to identify which test is more accurate and should be considered for use in the NHS.A systematic review is a scientific way of finding and summarising data from multiple studies on the same topic. Meta-analysis is a statistical procedure for combining data from the studies in a review to get numerical summaries. Inevitably, studies brought together in a meta-analysis will differ. Variability in the results of studies beyond what we would expect by chance is termed heterogeneity. Evidence-based decision making must consider the quality of evidence from systematic reviews. Observing high heterogeneity downgrades the quality of the evidence and reduces our confidence in the evidence. If we look at sensitivity and specificity values for a test in a systematic review we usually see high heterogeneity.My research will develop new contrast-based methods that allow meta-analysis of differences in sensitivity and specificity values between two tests. The methods will help us find out if there is less heterogeneity when we look at differences and so obtain high quality comparative accuracy evidence for making decisions about which tests should be used for patient care.Aims My research aims to:develop new contrast-based meta-analysis methods for obtaining numerical summaries of differences in the diagnostic accuracy of two medical tests and ways of presenting results;develop methods so that we can describe heterogeneity in contrast-based DTA meta-analyses;assess whether the size of studies partly explains heterogeneity in contrast-based DTA meta-analyses; andshow how contrast-based methods can be used for better decision making. The aims will be addressed in four work packages.Work package 1 I will develop new contrast-based methods that pool differences (absolute or relative) in the accuracy of two tests from comparative studies. The performance of the methods will be assessed against routinely used DTA meta-analysis methods. For the assessment, I will use data from 58 comparative reviews identified from a previous project. I will also use simulated data which mimic the sort of data that may be encountered in real life. I propose a new graphical tool which will enable decision makers and other users of comparative analyses to easily understand the findings. A focus group comprising patient representatives and other stakeholders will help refine this tool to ensure it is fit for purpose.Work package 2 Through a comprehensive review of the literature and contact with experts, I will find existing statistical methods for describing heterogeneity. I will examine if they can be used in contrast-based DTA meta-analysis. Promising methods will be adapted or new methods will be devised for contrast-based DTA meta-analysis. I will assess the methods using simulated data from work package 1.Work package 3 Using statistical techniques on data from meta-analyses identified from the literature, I will investigate whether study size is related to the size of differences in the accuracy of two tests. This will help review authors to know what to do about including small studies in a meta-analysis.Work package 4 I am working with researchers on nine systematic reviews looking at diagnosis of melanoma. I will use these reviews as a case study to show how the new methods can affect decision making. I will assess the quality of the evidence obtained from the reviews. Based on my experience during the case study and with input from expert researchers and patient representatives, I will then develop guidance for how people should rate quality of comparative test accuracy evidence. Outcome of the research Methods, guidance and software programs produced in the work packages will be shared with those producing evidence for NICE and the UK National Screening Committee, and other researchers around the world. This will enable use of valid methods to get correct answers for making policy decisions and improving the care of patients in the NHS and beyond.

Research activity codes: 4.1 DISCOVERY AND PRECLINICAL TESTING OF MARKERS AND TECHNOLOGIES
Health categories: GENERIC HEALTH RELEVANCE