Title: Improving the robustness of health technology funding decisions when evidence is from single-arm trials
Funder: Department of Health and Social Care

Abstract: Background Health technology assessment (HTA) is a multidisciplinary process to evaluate the clinical, economic and broader impact of the use of a health technology. Evidence on clinical effectiveness can arise from multiple sources. Randomised controlled trials (RCTs) are considered to be the gold standard for evaluating an intervention because potential biases are minimised; other study designs may distort results. However, RCTs may be infeasible in some disease areas, e.g., if it is unethical to allocate patients to a comparator arm, and single-arm trials have to be used instead. Estimates of treatment effects are at risk of bias if the data are from single-arm trials. Existing statistical methods can only correct for some of the biases. Decision-makers such as the National Institute for Health and Care Excellence (NICE) have had to make decisions in cases when not all biases have been addressed. Consequences of an incorrect funding decision could be extreme: patients may be harmed because they do not receive an effective treatment, and the wider society loses out because of an inefficient use of scarce healthcare budgets. Aims and methods This fellowship aims to develop new methods to improve the robustness of healthcare funding decisions when evidence is from single-arm trials. We will address two important issues: (i) bias due to difference between patients in different trials ('internal' bias); (ii) bias due to generalisability of the study results to clinical practice ('external' bias). There are four related work packages: WP1 (months 1-17): develop a new analytical approach to adjust for imbalance in observed confounders; evaluate and compare the proposed approach to existing methods through simulation studies and real RCT data (objective 1 and 2) WP2 (months 18-31): develop bias adjustment models to adjust for residual biases due to the remaining internal and external bias; and develop methods for the formal elicitation of experts' opinion regarding potential residual biases (objective 3) WP3 (months 32-45): conduct three case studies using real-world examples to pilot the proposed methods; and determine the use value of information analysis for quantifying the impact of potential bias on decision uncertainty (objective 3 and 4) WP4 (months 46-48): formulate best practice guidance regarding the use of single-arm studies in estimating treatment effect in HTA (objective 5)Anticipated impact and dissemination The findings of this fellowship will benefit NICE by addressing gaps in the existing methods and providing a basis for more robust decision-making. Making better decisions will improve outcomes for patients and the public. Health gains would be maximised in terms of the extent and quality of life of patients, given the available resources. Healthcare resources would be distributed to patients and the public more effectively and fairly in the National Health Service (NHS). It would also allow manufacturers to prepare more thorough submissions, which will improve the efficiency of the appraisal process. The research outputs will be disseminated via publications in relevant open-access peer reviewed scientific journals, presentations at relevant UK and international conferences, an Impact Event with stakeholders, project webpage and social media channels.

Lay abstract: Background The National Health Service (NHS) has a limited budget to meet the demand for healthcare. To decide what the NHS should pay for, the National Institute for Health and Care Excellence (NICE) considers whether a treatment works and whether it offers value for money. The best way to test if a new treatment works is to use a randomised controlled trial (RCT). In an RCT, patients are randomly chosen to receive either the new or the existing treatment. This makes the comparison fair between treatments. Sometimes, 'single-arm trials' have to be used instead, in which there is no comparison against an existing treatment and all patients receive the new treatment. This is used when it may be unethical to randomly assign patients or difficult to recruit them. When NICE makes funding decisions, the new treatment still needs to be compared with an existing treatment. However, the data for the existing treatment would come from other sources. There is now a risk of bias in the comparison as without the random allocation of patients to treatments, the patients in some trials may be healthier than in others. If such biases are not accounted for in the decision-making process, funding may not be distributed fairly, which will negatively impact patients and the public. Unfortunately, existing statistical methods cannot correct for all the potential biases. Currently, new treatments that only use single-arm trials as evidence are 17% less likely to be recommended by NICE. Aim of the research This fellowship aims to improve the robustness of healthcare funding decisions when data are from single-arm trials. This study will explore the use of statistical models and expert opinion to improve the strength of the evidence used. If the study aim can be achieved, healthcare resources will be distributed to patients more effectively and fairly within the NHS. Proposed methods Biases associated with single-arm trials are classed as 'internal' (due to differences between patients in different trials) or as 'external' (due to differences between patients in the trial setting and real clinical practice). I will: develop a statistical model which allows for the correction of these biases as existing methods cannot adjust for both, develop a method for obtaining experts' opinion to estimate how much the biases are likely to affect trial results as it is unlikely data would be available in the literature, explore the use of modelling techniques to estimate how much the biases might affect the decision about whether the treatment works and provides values for money, test the proposed methods in real-world examples, compare the proposed and existing methods, and provide guidance to help researchers and decision-makers to use more appropriate methods in the future.Patient and public involvement (PPI) A PPI panel has been set up at the planning stage of this fellowship. Five panel members have attended two workshops to help develop the research question, aims and the strategy of PPI throughout the fellowship. The PPI panel will advise on the design of the project webpage and dissemination. Panel members will also be involved in the real-world examples and helping to check the quality of the use of experts' opinion. Dissemination I plan to publish four reports about the findings of this study with one written in plain English for the public. I will also present the work at relevant UK and international conferences. An Impact Event will be held at end of this fellowship to communicate the project and its findings to researchers, decision-makers, pharmaceutical companies, patients and the public. A project website will also be set up to aid dissemination and communication of the research findings.

Research activity codes: 8.4 RESEARCH DESIGN AND METHODOLOGIES (HEALTH SERVICES)
Health categories: GENERIC HEALTH RELEVANCE