Title: Multiplicity issues in randomised controlled trials: identifying optimal approaches according to the taxonomy and context of the research questions being investigated
Funder: Department of Health and Social Care

Abstract: Background Analysis of the majority of randomised controlled trials (RCTs) consists of multiple statistical comparisons. Such multiple comparisons arise from situations including: measuring multiple outcomes, performing multiple treatment comparisons, subgroup analyses and interim analyses. Current methods to address multiplicity mainly consist of adjusting the significance level for inference. However viewpoints on the need for and use of such approaches vary.Although the issue of multiplicity is generally well-documented, specific issues are particularly relevant to pragmatic RCTs commissioned by the National Institute for Health Research (NIHR). These issues include: the source of multiplicity (e.g. multiple outcomes, multiple treatment comparisons, etc); which comparisons will be made; and whether outcomes measure efficacy, safety and/or cost-effectiveness. This fellowship application arises from the fact that, while the issue is widely recognised and the notion of adjustment' recommended in general terms, there is very little guidance which methods should be applied in particular circumstances.Aims The primary aim/research question of the fellowship is to determine how optimal approaches to address multiplicity issues in RCTs should vary according to the taxonomy of the situation and context of the research questions being investigated.Plan of investigation A systematic review of the literature will be performed to identify the range of existing methods used to address multiplicity and current guidelines and professional viewpoints on the topic. In addition, a survey of UK registered clinical trials units (CTUs) will be undertaken to describe existing practices, understand typical research questions people are aiming to answer, and obtain opinions on addressing multiplicity in various scenarios. Alongside this current practice will be assessed via review of recent non-commercial RCT reports.The findings of the work outlined above will be triangulated to create: a) a collection of research questions and situations in which multiplicity arises, and b) suggested approaches. This will be used to guide empirical work. Simulated datasets will be drawn from distributions representing different known treatment effects, each dataset will be analysed to determine the treatment effect of interest using a certain approach to the multiplicity problem, and the proportion of false-positive and false-negative findings calculated. The effect on Type I and II error rates has implications for the sample size of future studies, which will be calculated. Various parameters within the simulations will then be varied to determine their impact.The optimal methodology will consider: efficacy, safety and cost-effectiveness outcomes; the nature of the outcome, e.g. mortality vs. less-severe harms; and efficiency, i.e. optimal trade-off between Type I and II errors. The findings of the empirical work will be validated on real data from NIHR-funded RCTs.The findings from all aspects of the fellowship will be used to produce guidance for researchers. It is envisaged this will be a multi-dimensional matrix of points to consider and resultant recommendations, which should assist with the writing of statistical analysis plans (SAPs) and sample size calculations. Despite the large number of permutations to explore, the fellowship is designed to ensure it delivers on the areas most relevant to the NHS.Summary of potential benefits to patients and the NHS The vast majority (if not all) of RCTs commissioned by the NIHR will have a need to at least consider multiplicity in their analysis. If consensus can be reached on the approaches that should be used, this could impact on the design of all such RCTs and their analyses, make interpretation of findings more rational, and thus improve subsequent decision making by patients, practitioners and policy-makers.

Lay abstract: Aims of the research To investigate the best approaches to address issues that arise from performing multiple statistical tests when analysing clinical trial data. This will be within the context of trials commissioned by the National Institute of Health Research (NIHR) to answer questions important to the NHS.Background to the research When analysing clinical trials, multiple statistical tests are usually performed. This is due to a variety of reasons, such as: Multiple outcomes being measured More than two treatments, leading to multiple comparisons between pairs of treatments (e.g. treatment A and treatment B, treatment A and treatment C, etc.) Researchers wanting to look at the effect of treatments in different subgroups of patients (e.g. in males and females, diabetics and non-diabetics) When any statistical test is performed there is always a chance of finding a statistically significant difference between treatment groups in the trial sample, when in fact there is no such difference in the whole population (often called a false-positive' finding). When one test is performed this chance is usually 5%. However when multiple statistical tests are performed this chance increases substantially; if ten comparisons are performed the chance of at least one false-positive is 40%. A number of statistical methods have been proposed to address this problem, but opinions on their value are varied. Therefore the approaches used to analyse clinical trials vary.Although the problem of multiple statistical tests arises in most clinical trials, I believe there are specific issues for the types of trials frequently commissioned by the NIHR. The issues include: the cause of the multiplicity (e.g. multiple outcomes, multiple comparisons between pairs of treatments, etc); the types of statistical comparisons that will be made; and the outcomes needed to address a research question (e.g. should any rules' be different for mortality compared to a less severe complication). Any standard approaches established to address the issues of multiple statistical tests have potential to be applied to the majority of clinical trials commissioned by the NIHR, and so could have direct impact on clinical research in most disease areas.Design and methods used The first three tasks I will perform are as follows: Review and summarise what has already been published about this topic. Survey researchers at other Clinical Trial Units in the UK to establish their current practice on multiple statistical tests, and obtain opinions on the areas and issues they consider to be of particular importance. Review recently completed trials in the published literature and see what approaches were taken (if any) to overcome issues relating to multiple tests. I will pool the findings from tasks 1-3 to form lists of important areas that cause multiple testing issues, and suggested statistical methods to address these issues.Next, I will create computer-generated data, where the size of the difference between the treatments is known. I will then apply the statistical methods identified in the first part of the project and observe the consequences. I will take lots of samples from the computer-generated data, analyse each sample and calculate the number of times the correct' trial results are obtained. The effect on the required sample sizes of future trials will be calculated. I will investigate if different causes of multiple statistical tests lead to different conclusions. Similarly I will explore if the results differ according to other factors, e.g. an outcome measuring effectiveness versus one measuring harm. Once this work has been completed I will test the findings on real data from clinical trials that are either underway or have been completed.Dissemination I will bring together the findings from all aspects of the fellowship and produce recommendations and guidance for researchers planning clinical trials. I anticipate that these recommendations will depend on many factors and summarising them will be challenging. However, I consider it crucial to ensure this guidance is as succinct and understandable as possible, and therefore accessible to a large audience of researchers planning and delivering clinical trials.I plan to produce a number of articles throughout the fellowship. Some of these articles will cover details of the rationale and background for decisions made, or technical and statistical aspects of the project, and will be targeted at clinical trial statisticians and methodologists. I will also produce a publication aimed at public health professionals with little experience of clinical trials, to help them interpret clinical trial reports and weigh up evidence.

Research activity codes: 8.4 RESEARCH DESIGN AND METHODOLOGIES (HEALTH SERVICES)
Health categories: METABOLIC AND ENDOCRINE