Multiplicity and replicability: two sides of the same coin

Replicability is a fundamental principle of good scientific practice: A claimed research finding is more convincing if it is replicated in subsequent studies. This requirement becomes essential in pharmaceutical drug development, where late-stage confirmatory clinical studies are conducted to provide the necessary evidence to support regulatory decision-making for drug approval. However, an increasing evidence of lack of replicability of scientific findings has been reported, be it in genomic research [1,2], drug development [3,4], or medical research in general [5]. While there are many reasons for the observed lack of replicability, failure to adjust for multiplicity is likely to be a culprit [6]. That is, seemingly significant effects occur more often than expected by chance alone when performing selective inference and not compensating for multiplicity can lead to misleading conclusions. In the following, we illustrate the close link between multiplicity and replicability by means of a simple simulation experiment and describe the potential consequences in terms of drug development decisions and impact on patients. In what follows, we illustrate the problem in the context of selecting the best out of several subgroups, although the same arguments apply equally to other relevant situations, such as selecting the best dose or endpoint. Consider the situation of conducting two independent studies in sequence, say Studies I and II. Assume that in Study I, a new drug is compared with placebo in four non-overlapping classes of patients (i.e., populations) defined by predictive biomarkers, demographic factors, or any other classifier. Suppose further that at the end of Study I, the ‘best’ population is selected for a subsequent study. More specifically, assume that the population with the largest observed effect size is selected to confirm the treatment benefit of the new drug over placebo in Study II. This set-up roughly mirrors current clinical drug development practices, if Study I is interpreted as an exploratory study at the end of Phase II to learn about the therapeutic effect in different patient populations and Study II as a Phase III study intended to replicate and confirm the findings from the preceding exploratory clinical development phase(s). A natural question is then to compare the observed effect sizes of the selected population across both studies. To this end, we simulated a large number of times pairs of independent Studies I and II and compared the observed effect sizes of the selected population in both studies. For the simulation study, we assumed identical conditions in both Studies I and II (i.e., same clinical endpoint, dose, centers, etc.) and the same treatment effect sizes across all four populations. That is, at the end of Study I, any of the four populations has equal chance to be selected for Study II. More specifically, we simulated – across both studies and all populations – independent observations from a normal distribution with mean 0 or 1 (depending whether the patient received placebo or drug, respectively) and standard deviation 1. The only difference between Studies I and II is the sample size: We simulated observations for 100 patients per population in Study I and 1000 patients from the selected population in Study II, resulting in 50 and 500 patients for each combination of treatment and population in Study I and II, respectively (assuming a balanced allocation between drug and placebo). Figure 1 displays the results from 10,000 simulation runs. In case of good replicability, the simulated effect sizes would be scattered symmetrically around the diagonal line displayed in Figure 1. The triangle displays the true effect size from which we simulated from (i.e., mean improvement of drug over placebo by 1 unit in each study), and the cross displays the mean observed effect size averaged across all simulation runs. Clearly, the simulation results demonstrate that the observed effect sizes in Study I are not replicable and on average much larger than the ones in Study II. In clinical drug development practice, these results could explain partially the high attrition failure observed in Phase III as a consequence of overoptimistic expectations raised by clinical teams if the Phase II results are interpreted without taking multiplicity into account. Furthermore, such lack of replicability also has an impact on patients if effective drugs receive market approval, but the reported effect sizes do not acknowledge the selection effect. Note that the pattern observed in Figure 1 remains the same regardless of the actual effect size. This is particularly true if there is no difference between drug and placebo in any of the four populations, which means that a non-effective drug could be taken forward to the confirmatory phase due to a chance result