Letter to the Editor: Efficacy and Degree of Bias in Knee Injury Prevention Studies: A Systematic Review of RCTs

I thank Grimm et al. for their considered response [6] to my letter [3] regarding their recent article in CORR® [5], although I fear that I may not have adequately articulated my concerns (or we are just using different terminology). Clearly the trial by Brushoj et al. [4] was not masked; there is no way it could be masked, since, for example, nobody would confuse a squat with a curl. The participants knew which set of exercises they were doing; ergo they were not masked. Now if they did not know which set of exercises comprised the active treatment, then we may have had de facto masking, which is still a good thing; but it is not true masking, and the two should not be confused. Nor was each subject “randomly divided” as stated, and we would hope, for the safety of the subjects, that they were not divided (or dissected) at all. Even making allowances for this imprecise language, and recognizing what was meant, it also is not true that the subjects were randomized. Cluster randomization was used instead of individual randomization, and a variation of permuted blocks was used to allocate two platoons per company to one treatment, and one platoon to the other, so the companies were the blocks. Apparently the 2:1 allocation in the blocks had a built-in swapping of which treatment was to be allocated twice as often, to ensure overall balance, which in fact was achieved (12 platoons per treatment) [4]. I can appreciate the reasons for using cluster randomization, and I will not argue against it, if its benefits were judged to outweigh its drawbacks. However, this does not mitigate its drawbacks [2], which include a tremendous reduction in information or degrees of freedom or effective sample size. With 24 platoons randomized, we have 24 units of analysis; the sample size is 24, not 1030 or 977. This is a rather small sample size on which to base a conclusion. Moreover, there are other issues with combining permuted blocks with cluster randomization, although in fairness these issues may be less of an issue in the military setting in the article by Brushoj et al. [4] than they would be in a clinical setting, the distinctions being (1) greater homogeneity among members of a platoon than among patients in a clinic and (2) the likelihood that all participants in a “block” (company) in their study were recruited at the same time, as opposed to the sequential recruitment even within blocks that is typical in trials conducted in clinics. There are still plausible scenarios that call into question the success of the allocation concealment. For example, the dependence of which way the 2:1 allocation would go for one company on how it went for prior companies certainly would give some clue regarding which treatment upcoming participants were to receive. The nature of this dependency is unclear. Were the eight companies randomized so that four had to go one way and the other four the other way, or was it just a fortuitous coincidence that it turned out this way? Were all platoons in a company treated at the same time? Was there discretion in selecting the order in which the platoons were treated? I have no smoking gun with which to establish any actual bias, but the risk of bias is certainly present. In addition, the use of parametric analyses is rather questionable [1], especially with such small sample sizes. Grimm et al. asked how to better ensure proper randomization, masking, and allocation concealment in the future [6]. In the context of the trial by Brushoj et al. [4], only a partial answer is possible, since the use of cluster randomization may have been necessary. Ideally, however, when individual randomization is an option, it should be used. When it is not an option, cluster randomization may be used, but with the understanding that the number of clusters sets the sample size, so a large number of clusters would be needed. Masking may be more elusive in this context, and certainly the de facto masking that was used is a good idea. Another possibility, if this was deemed feasible, might have been to describe the two treatments in vague terms, such as (1) strengthening and stretching, emphasis on stretching versus (2) strengthening and stretching, emphasis on strengthening. Participants in both groups would work out both ways, so given this description, they may not be able to determine to which treatment group they had been allocated. Of course, if the descriptors were instead upper body versus lower body exercises, then it would be obvious, but it also might have been possible to add in some arm exercises to the active group and some lower body exercises to the control group, and then the descriptors could have been (1) lower body emphasis versus (2) upper body emphasis, and there would be enough of each type in each group to keep everyone guessing. The primary point I was trying to make, however, was a general one, and not specifically about the trial by Brushoj et al. [4]. This point is that there are many ways to randomize, some better than others, and so no assessment of the quality of randomization or allocation concealment can be complete without specific details of how the randomization was conducted. We cannot just assume that every study with a claim randomization was in fact properly randomized, and this remains true even if all concerns regarding the trial by Brushoj et al. are addressed and this is found to be a truly valid trial.