Discussion of ‘A critical appraisal of propensity‐score matching in the medical literature between 1996 and 2003’

I would like to begin by thanking the editors of Statistics in Medicine for providing this forum to discuss my recent review of propensity-score matching in the medical literature. I would also like to thank Drs Hansen, Hill, and Stuart for their discussion of issues raised by my article and of other issues related to propensity-score matching in general. Hopefully, this focus on propensity-score matching will lead to improved practice in applied research and to the identification of areas of future research. My response to the discussants’ comments is structured as follows: First, I will discuss issues concerning the comparison of balance between treated and untreated subjects in the propensity-score matched sample. Second, I will address issues relating to estimating the treatment effect and its statistical significance. In my initial review [1], I described the need to assess the degree to which matching on the estimated propensity score resulted in a matched sample in which measured baseline covariates had a similar distribution (were balanced) between treated and untreated subjects. This allows one to assess whether matching on the propensity score has reduced or eliminated observed systematic differences between treated and untreated subjects. Matching on the true propensity score eliminates systematic differences between treated and untreated subjects. However, in nonrandomized settings, the true propensity score is not known and must be estimated using the data. Balance tests can function as a test that the propensity-score model has been specified correctly. In my review, I asserted that the use of significance testing is inappropriate for assessing balance, as the resultant p-values are confounded with sample size. The matched sample is invariably smaller than the initial sample. This can result in the appearance of better balance in the matched sample solely due to the lower statistical power to detect imbalance. Both Drs Hill and Stuart concur with this proscription against the use of significance testing. Furthermore, both Drs Hill and Stuart suggest that rather than only examining means of continuous variables between treated and untreated subjects, applied researchers also examine balance in squared terms and interactions. Examining the variances of baseline variables between treated and untreated subjects can complement the comparison of means. In practice, examining all interactions between baseline variables could prove cumbersome. Further research is required into what constitutes acceptable balance in measured covariates between treated and untreated subjects in the matched sample. Currently, there are two options when researchers are faced with imbalance in the propensity-score matched sample. Researchers can modify the propensity-score model by including additional variables, higher order terms, or interactions. Alternatively, researchers can change the caliper width used in matching. By using a narrower caliper for matching, researchers may be able to achieve better balance in the matched sample. Arguably, each of these approaches could result in improved balance in the matched sample. The relative advantage of each approach requires further examination. Furthermore, there is a paucity of research into the relative strengths of different methods used in the medical literature for forming matched pairs. In a recent study, I have compared, both