Commentary on Price and Scott: Complex innovative trial design

The issue of complex innovative trial design raises many methodological challenges. Price and Scott refer to several methods and designs that they illustrate via applications. They define complex innovative designs as designs that feature, for example, novel use of external or historical control data, formal incorporation of prior knowledge into the design, and adaptations to multiple design aspects based on accumulating data. For its part, the Food and Drug Administration (FDA) has defined a transparent process outlining the steps and timeline for submission, review, and meetings, in order to facilitate the implementation of the Complex Innovative Trial Design Pilot Meeting Program. Designs accepted into the pilot program serve as educational examples to facilitate the science and use of complex innovative trial designs. Price and Scott present elements of the design and analysis as case studies. The five trial examples presented have a common theme: the use of historical data or augmentation of the trial’s data with prior data in the Bayesian setting. The authors concluded that ‘‘The pilot meeting program has been successful in further educating stakeholders on the potential uses of complex innovative designs in trials intended to provide substantial evidence of effectiveness. The selected submissions, thus far, have all utilized a Bayesian framework. ’’ We limit our commentary to those questions raised by Bayesian methodology mindful of the fact that there is overlap with some of the other key questions, for example, questions relating to seamless studies and adaptive designs. While the Bayesian school of thought may tend to readily embrace broader application of Bayesian methods, statisticians with a more classical view may be more circumspect. That said, both schools of thought recognize that, in a bid to make full use of all the available data, Bayesian methods clearly have a role to play in clinical trials. The Complex Innovative Design initiative is therefore timely and needed. The challenges of novel Bayesian designs though cannot be underestimated. The most immediate difficulty is understanding what is meant by the term ‘‘Bayesian. ’’ Not all approaches coming under this heading are going to be of help in the learning process. We see three main ones that require deeper discussion: (1) an approach that views statistical modeling as a way to mimic as closely as possible the unknown physical probabilistic mechanisms that can be judged to underlie the observed outcomes, and, in consequence, ‘‘the richer the model, the better,’’ (2) an approach that views any model as no more than a working model, a means to some specific end and, in consequence, ‘‘the simpler the model, the better,’’ and (3) an approach that focuses on the outcomes and will put probabilistic limits on how far these might plausibly stray from what is anticipated. This third approach can make use of pseudo-observations, that is, a certain amount of invented (plausible) data that can both stabilize and refine inference. An appeal can also be made to historical data. The first of these three approaches can also lean on historical data but, typically, will appeal to so-called expert opinion expressed via chosen prior distributions. Of the above three approaches, the second one is possibly the furthest removed from what trialists have in mind when they think ‘‘Bayesian. ’’ Early phase trials based on the continual reassessment method make use of this approach and, interestingly, have shown that predictive performance improves, the further the assumed model is from the observed outcomes. That is not something that one would anticipate initially, at least intuitively, but was shown very clearly in the first paper on the continual reassessment method where, in trials generated under a two-parameter logistic model, basing an analysis on this same (correct) model resulted in poorer performance than is obtained from a more brutally under-parametrized (incorrect) model. Iasonos et al. showed why this situation prevails and helps understand why Bayesian approaches coming under heading (1) of the above paragraph are unlikely to meet