Commentary on Willan & Thabane: Bayesian methods are valuable in pilot trials but the choice of prior needs careful consideration

The article by Willan and Thabane highlights the utility of Bayesian methods in pilot trials, particularly the role of the beta-binomial conjugacy method in this context. These methods have the benefit of allowing direct inference on key quantities in pilot trials such as the proportion of eligible patients randomised or the proportion of patients withdrawing post-randomisation. For example, if investigators believe that a future main trial is feasible if the probability randomised is greater than 50%, then the probability of this occurring can be directly calculated from a Bayesian posterior distribution, as illustrated by Willan and Thabane. Other benefits include being able to incorporate prior information from any similar previous studies (if available) to augment the information available from small pilot studies. Although the main focus of the article by Willan and Thabane is on beta-binomial predictive distributions, the importance of the posterior beta distribution itself in making direct inference should not be overlooked. In particular, intuitive progression criteria that are based on the posterior beta distribution can greatly facilitate the decision to proceed from pilot to main trial. For example, progression criteria such as ‘The main trial will have at least 80% probability of randomising at least 50% of eligible patients’, focus attention helpfully on what will happen in a future larger main trial, rather than simply on what the pilot trial shows, and it is these progression criteria that can be addressed using Bayesian methods. Furthermore, posterior distribution graphs can be presented to provide helpful visual information which can assist with interpretation. Willan and Thabane present an interesting Bayesian method to calculate the ‘expected power’ in a future main randomised controlled trial (mRCT) based on simulation from beta-binomial predictive distributions, using data from a pilot trial (pRCT). I have a few concerns however about the use of this method in general. First, Willan and Thabane’s Bayesian method is similar to the standard method in so far as it is based on expected values, albeit it is influenced by two prior distributions. I would therefore expect that any difference in power for the mRCT between the Bayesian and standard methods would be mainly due to the influence of the prior information. If this is all true, then is it really fair to label the standard method as ‘naive’? After all, the standard method is also based on expected values, and it is not inherently biased unless the estimated proportions themselves are biased. In contrast, the proposed Bayesian method will give misleading results if inappropriate priors are used. Second, Willan and Thabane’s Bayesian method can lead to an expected power that is greater than the power determined by the standard method (e.g. . 80%) under certain conditions. For example, this happens if vague priors such as Beta (1,1) are used, and the observed proportions randomised (Pr) and evaluated (Pe) are both less than the prior means. In this case, the use of vague priors will not be conservative and will attenuate the posterior means towards the midpoint of the distribution. Therefore, investigators should consider carefully the selection of prior distributions and perhaps favour those where most of the prior probability density is below the observed proportions in the pilot trial (pRCT) if prior beliefs are uncertain. As an alternative, I propose a different approach to the problem of ‘expected power’ and that is to calculate the lower bound proportion randomised (Lr) and proportion evaluated (Le) above which there is X3 100% power that the main trial proportions lie, where X should be chosen to be reasonably high (e.g. 0.8). Specifically, the aim is to numerically solve the