A Source of False Findings in Published Research Studies: Adjusting for Covariates

Concern about erroneous conclusions ofmany published research findings has led to the conclusion thatmost published research findings are wrong.1,2 What can be done about that? In what follows, I will focus on one common source of false findings: adjusting for covariates. Here, adjusting means allowing variables to vary as they will but then using a mathematical model to assess their influenceon theoutcome. In contrast, to controlmeansmanipulation of variables by the researcher for a particular purpose (eg, in experimental design). Unfortunately, the terms adjust and control are often used as if they were synonymous. Adjustingoften leadstofalseconclusionsbecausethemodelsused may not correspond to reality. To illustrate thispoint, consider a randomizedclinical trial (RCT) inwhich those sampled from thepopulation of interest are randomlyassignedto2 treatmentgroups,T1andT2.Avalid test simply comparing the outcomes in the 2 groups tests the overall effect size (overall ES) that a randomly sampled patient from T1 has an outcome clinically preferable to that of a randomly sampled patient from T2.3 Often, the first table of an RCT report compares the baseline characteristics of the T1 vs T2 samples to assess the success of randomization, ignoring the fact that randomization (1) is a process, not an outcome, and (2) ismeant to generate 2 random samples from the same population, not 2 matched samples. When a few baseline variables significantly differentiate the 2 groups at the 5% level, researchers often propose to adjust for those covariates in testing the treatment effect. This isposthoc testing (likeoffering tobet atpreraceodds on a horse as it approaches the finish line), which frequently leads to false-positive results. Any covariates to be used in adjusting should be specified apriori, listed in theRCT registration, and taken into consideration in thepower analysis. Suchadjustment changes the hypothesis to be tested from comparing all T1 patients vs all T2 patients (overall ES) to comparing T1 patients onlywith T2 patients matched in one way or another on the particular covariates proposed. Let’s say that covariate ES is the ES for patients with one particular configuration of the covariates and typicalES is theESspecifically forpatientswhoareat themean of each such covariate (ie, for the typical patient). Overall ES, typical ES, and all possible covariate ES are the same only if the covariates are irrelevant to the treatment outcome. If the covariates are irrelevant, adjusting for those simply leads to a loss of power. If the covariates arenot irrelevant, then estimation and testing of overall ES, typical ES, and covariate ESprovide answers to different research questions. The linear model used for covariate adjusting (eg, analysis of covariance [ANCOVA]) assumes, for all possible values of the covariates, that covariate ES is equal to typical ES; that is, that there is no interaction between the covariates and the treatment effect. If this assumption is violated, then the interactions that exist in the population (but are not included in the model) can bias the statistical tests and estimation of the treatment ES. Furthermore, not finding statistically significant interactions in the sample does not prove thenull hypothesis that they do not exist in the population. Given these risks for bias, ANCOVA should not generally be used for such adjustment. Whentreatment interactionsare includedina linearmodel, howthevariables are codedcan impact the results.4The treatment effect refers to the treatment effect for patients having thezerovalueofall includedcovariates.Thus, ifT1andT2were 2 treatments for Alzheimer disease, and the single covariate were chronological age at disease onset, the treatment effect would be the effect of the treatment for individuals with Alzheimer disease diagnosed as having the illness at age 0 years, which is a ludicrous result. Instead, age is better coded as deviations fromthemeanageat onset (centeringat themean). Then the treatment effect is typical ES and the interaction effect reflects the change in covariate ES as the covariate value changes. Examinationof the covariateESmaywell indicate to clinicianswhichpatientswill respondbetter toT1 orT2.5With multiple covariates, if each is centered at itsmean,4 the treatment effect tested is typical ES, the treatment effect for those at themeanofeverycovariate,which is sometimesaverysmall subpopulation. There are still additional problems. For example, when multiple covariates are included, omitting interactions between them can introduce bias to the estimation of treatment ES. However, to include all interactions involving m covariates in a linear model requires estimation of 2m +1 parameters. To make matters even worse, the advantage of an RCT with random assignment is that, over replications, treatment choice and each covariate are uncorrelated. However, covariates may be correlated with each other (collinearity). Correlated variables share information. In fitting a model to the data, the computer is instructed to allocate the information shared between 2 variables to one variable or the other. The computer does this using information from within the sample. Because such information will change from one sample to another, the estimates of the adjusted treatment effects (typical ES and covariate ES) are unstable and difficult to replicate. Opinion