Comments on Berry et al.’s response-adaptive randomization platform trial for Ebola

We commend Berry et al. for their ambitious design to evaluate multiple Ebola virus agents and combinations. Their focus is finding the best treatment, and their randomization allocation adapts over time to favor regimens with better performance. Such trial designs can have efficiency gains, but it is important to understand their disadvantages and limitations. First, a big concern is the double-edged sword of response-adaptive randomization (RAR). Although RAR hones in on effective treatments, it can lead to lopsided sample sizes that are especially problematic if there are temporal trends. Korn and Freidlin showed in a two-arm setting that, with RAR, if a stratified test is used to combat a temporal trend, the loss in efficiency necessitates a sample size increase that also leads to more patients being exposed to inferior treatment than in a fixed randomization scheme. Berry et al. account for temporal trends by including a month effect in their model, but that may be inadequate if the model is wrong, temporal changes occur more frequently than monthly, estimates are poor, or there is a treatment by time interaction. The latter could happen if treatment helps only those recently infected, and the proportion of such patients changes over time. The authors do include a ‘‘burn-in’’ period before RAR begins to ameliorate treatment assignment volatility, but the effect of burn-in length on trial properties must be carefully evaluated. Even without temporal trends, very small sample sizes cause problems (e.g. was treatment really ineffective or did those few patients happen to be sicker? ) One problem in the famous extracorporeal membrane oxygenation (ECMO) trial using RAR was that the lone baby assigned to standard therapy was sicker than other babies. In short, RAR’s laudable goal of minimizing the number of patients assigned to inferior treatments can be outweighed by sample size imbalances resulting in misleading results, especially under temporal trends. Second, the chosen priors are intended to be uninformative, but are they? The prior distribution for interaction terms has mean 0 and small variance, stacking the deck toward additivity of treatment effects. Given that agents with 0.95 probability of being in the optimal regimen are declared superior, additivity is a big assumption that has an important effect on results. In a similar setting, we found considerable differences in results when we dramatically increased variances of single treatment effects and interactions. Third, is the stated goal of finding the best treatment better than that of quickly identifying any treatment better than standard of care? It is difficult to conclude that one arm is best if there are many treatments (begging the questions: how many treatments are too many, and when is it better to focus on just a few of the more promising treatments? ). Large sample size imbalances are similarly problematic. For instance, a tiny sample size in arm 1 means that its response rate is poorly estimated, which in turn makes it difficult to be confident that arm 2 is best. The alternative of comparing each arm with control preserves autonomy and allows us to simultaneously evaluate safety and efficacy of treatments. Finally, given the importance of comparisons with control stated above, it is important to avoid chance declarations that control is inferior to another arm. Berry et al. drop the control if, at any week, the control arm has 0.95 probability of being inferior to any regimen. Even in a two-arm trial, weekly testing dramatically inflates type I error rate. For example, 20 weekly analyses, each at 1-tailed alpha of 0.05, result in a type 1 error rate of 0.25. Having multiple arms compounds the problem. Again we applaud Berry et al. for a very detailed plan dealing with an extremely complex and difficult problem. This design may be lifesaving under the right circumstances, but could potentially produce misleading results under the wrong circumstances. We hope indepth evaluation through simulation, as was done in Saville and Berry in a simpler setting, will clarify its appropriate role and bring focus to modifications that would enhance it.