Discussion of Adjustment Uncertainty and Propensity Scores

I am delighted to comment on this provocative and innovative paper. My comments will address the intersection between adjustment uncertainty and propensity score (PS) techniques. Adjustment uncertainty, as defined by Wang, Parmigiani and Dominici (WPD, 2011), refers to the uncertainty about which covariates should be accounted for to adjust properly for confounding. PS techniques are a class of methods for controlling confounding in observational studies. In a PS analysis, we combine information from a vector of covariates for each study participant into a single summary score, which is the probability of exposure given the covariates. The score is then used to adjust for confounding. We can stratify on the PS, use inverse probability weighting and matching, or we can include it as a covariate in a regression model for the outcome. To estimate the PS, we need a model for the exposure. Ideally, the investigator has identified the confounders beforehand and can elicit a realistic model for probability of exposure. But, in practice, researchers are often faced with a large array of covariates, and this demands a data-driven approach to model selection. Variable selection for PS models is an exciting area of innovation in statistics. Much recent work has been championed by Schneeweiss et al. (2009) in pharmacoepidemiology. Schneeweiss argues that when choosing variables to include in the model for exposure, we must take into consideration the relationship with the outcome. Including variables unrelated to the outcome in the PS increases the variance of the exposure-effect estimate, with no commensurate reduction in bias. Essentially, the message is that we should only include genuine confounders in the PS. Schneeweiss et al. (2009) describes novel procedures for variable selection. One thing that struck me when reading the Schneeweiss paper was that the methods are similar to those described by Crainiceanu, Dominici, and Parmigiani (2008). For example, compare table 3 from Schneeweiss et al. (2009) with figure 1 of Crainiceanu et al. (2008). Both papers attempt to fit joint models for the exposure and outcome to identify confounders. Thus, it appears that an immediate application of adjustment uncertainty is that it provides a principled framework for variable selection in PS models. How to combine adjustment uncertainty with PS techniques? A simple strategy would be to modify the procedure of Crainiceanu et al. (2008). At Steps 3 and 8, we would first calculate the estimated PS based on each subset of U = (U1, . . . , UM) in the dominant model class. Next, we would plot point and interval estimates for the exposure effect calculated by adjusting for the estimated PS. In other words, we swap out the U ’s in place of a linear predictor calculated from the PS. A Bayesian approach has additional advantages. Rather than adjusting for the estimated PS based on a single exposure model, we could average over different models. This incorporates uncertainty in the PS that arises from uncertainty in the PS model. I suspect that such a method would be superior to conventional techniques in the face of model misspecification. In addition, a Bayesian approach would permit the incorporation of standard Bayesian machinery, such as prior information, complex modeling, and MCMC computation. I was intrigued by the discussion of feedback from the outcome in BAC and TBAC. Feedback describes the situation in which the outcome variable “interferes” with estimation of the exposure model. One consequence of combining the PS with Bayesian model averaging is that feedback could potentially inform the choice of exposure model, and therefore, the estimated PSs. To see how this would work, consider a simple illustration involving a Gaussian response variable Y , a dichotomous 0/1 exposure X , and two Gaussian covariates C1 and C2. Suppose that the true data generating mechanism is,