Abstract: A total of 18 studies (systematic reviews or meta-analyses) met our inclusion criteria. Ten compared random allocation versus non-random allocation and nine compared adequate versus inadequate or unclear concealment of allocation within controlled trials. All studies were at high risk of bias. For the comparison of randomised versus non-randomised studies, four comparisons yielded inconclusive results (differed between outcomes or different modes of analysis); three comparisons showed similar results for random and non-random allocation; two comparisons had larger estimates of effect in non-randomised studies than in randomised trials; and two comparisons had larger estimates of effect in randomised than in non-randomised studies. Five studies found larger estimates of effect in trials with inadequate concealment of allocation than in trials with adequate concealment. The four other studies did not find statistically significant differences. The results of randomised and non-randomised studies sometimes differed. In some instances non-randomised studies yielded larger estimates of effect and in other instances randomised trials yielded larger estimates of effect. The results of controlled trials with adequate and inadequate/unclear concealment of allocation sometimes differed. When differences occurred, most often trials with inadequate or unclear allocation concealment yielded larger estimates of effects relative to controlled trials with adequate allocation concealment. However, it is not generally possible to predict the magnitude, or even the direction, of possible selection biases and consequent distortions of treatment effects from studies with non-random allocation or controlled trials with inadequate or unclear allocation concealment.

Plain-language summary: Randomised controlled trials (RCTs) use the play of chance to allocate participants to comparison groups to prevent selection bias. Other means of treatment allocation are more prone to bias because decisions about which treatment to use can be influenced by the preferences of the physician or patient. This review compares random allocation (allocated to treatment using a random method) versus non-random allocation (allocated to treatment using a non-random method, such as alternation or external, uncontrollable factors, with no clinical judgement involved) and controlled trials with adequate versus inadequate/unclear concealment of allocation. Concealed treatment allocation is best described in general terms as the process used to prevent foreknowledge of group assignment in a controlled trial (such as the use of sequentially numbered opaque, sealed envelopes). The results of randomised and non-randomised studies sometimes differed. Sometimes non-randomised studies yielded larger estimates of effect, and sometimes randomised trials yielded larger estimates of effect. On the other hand, not using concealed random allocation resulted in larger estimates of effect, but sometimes it resulted in similar estimates of effect (from harmful to beneficial or vice versa). It is a paradox that the unpredictability of random allocation is the best protection against the unpredictability of the extent to which non-randomised studies may be biased.