Statistics Commentary Series: Commentary No. 30: Propensity Scores.

T he gold standard research design for evaluating the effectiveness or efficacy of a treatment is the randomized controlled trial (RCT), and there are a number of reasons for this. First, most of the commonly used statistical tests are based on the assumption of random assignment, although especially in the case of the analysis of covariance, this assumption is often violated. Second, and more importantly, randomization lessens the chances of baseline differences between groups. It doesn’t guarantee that the groups will be similar, but it does reduce the possibility of confounding by baseline characteristics of the participants. The real strength of randomization comes not from its ability to balance out differences on variables we suspect could affect the outcome – after all, we can use stratified sampling or post-hoc adjustments to deal with those – but also on variables we are either unaware of or cannot easily measure, such as genetic and environmental factors. But we are not always able to carry out RCTs. This is the case when we are interested in adverse effects of some agent. For example, ethics committees would look askance if we proposed to randomly assign 500 pregnant women to either use or not use an antidepressant in order to determine its teratogenicity; or to randomly assign kids to play or not play violent games on their hand-held devices to see if this results in aggressive behavior later in life. In other situations, it may be possible to do an RCT, but it would be much faster and cheaper to use existing databases to get at least preliminary evidence about the effectiveness of some treatment. We could, for instance, take 6 months to write a grant application to do an RCT comparing electro-convulsive therapy (ECT) with drugs in elderly patients with a major depressive disorder; wait 9 months to a year to have it approved by them (assuming, of course, that it is approved); and then spend 2 years recruiting and treating people. Or, we can use already existing data to identify 2 cohorts of patients who meet the eligibility criteria, 1 of which received medication and the other ECT, and look at the outcomes; this may take 2 or 3 months in total (mainly waiting for the ethics board to give its OK). In many jurisdictions, there are data repositories that can link hospital admission and treatment data with many other databases, making this feasible. The problem that we face, however, is the one plaguing all cohort studies, that the groups may not be comparable. In particular, there may be factors, such as disease severity or comorbidity, that may affect (1) which treatment the person receives and (2) the outcome – in other words, confounders. The question is whether there is any way of making the groups comparable to one another on the basis of a number of possible confounders. As I mentioned earlier, if there are only a few of them, we could use stratification or post-hoc adjustments to balance the groups. This becomes increasingly more difficult, though, when there are more than 2 or 3 variables; it may not be possible to find 2 men, both 75 years old, with a previous history of an infarct, 3 previous hospitalizations, and the presence of psychotic symptoms, one of whom received ECT and the other medication. Rosenbaum and Rubin developed a method to dealwith a multitude of covariates in cohort studies, called propensity score matching (PSM). The first step consists of deriving a single score for each subject – the Propensity Score (PS) – which can range between 0 and 1, based on all of the covariates to be considered. Reducing a number of scores to a single number is a well-established technique; it’s what we do every time we derive a regression equation. The difference is that, with ordinary multiple regression, the dependent variable (DV) is normally distributed. In the case of PSs, however, the DV is group membership, which is dichotomous. Consequently, we must use a variant of multiple regression, called logistic regression. The PS itself that is produced by the logistic regression is a meaningless number – a high value is not better than a low one; it is simply a summary score of the variables we wish to control for, and reflects the probability of being in one group or the other, based on the covariates. By using the PSs, the aim is to mimic the effect of randomization by creating groups that are comparable insofar as the covariates are concerned. When running regressions, we’re oftenwarned to keep the number of predictor variables as small as possible, to guard against over-fitting and problems with multicollinearity, and to exclude variables that