Sensitivity Analysis in Observational Research: Introducing the E-Value

Key Summary Points Motivation: Observational studies that attempt to assess causality between a treatment and an outcome may be subject to unmeasured confounding. Rationale: Sensitivity analysis can assess how strong an unmeasured confounder would have to be to explain away an observed treatmentoutcome relationship. A sensitivity analysis technique that is easy to use, present, and interpret, and does not itself make strong assumptions, is desirable. Definition of E-value: The E-value is the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need to have with both the treatment and outcome, conditional on the measured covariates, to fully explain away a specific treatmentoutcome association. Calculation: The E-value for an estimate, and for the limit of a 95% CI closest to the null, can be calculated in a straightforward way for risk ratios (Table 1) and for other measures (Table 2). Conclusions: The E-value allows an investigator to make statements of the following form: The observed risk ratio of 3.9 could be explained away by an unmeasured confounder that was associated with both the treatment and the outcome by a risk ratio of 7.2-fold each, above and beyond the measured confounders, but weaker confounding could not do so; the confidence interval could be moved to include the null by an unmeasured confounder that was associated with both the treatment and the outcome by a risk ratio of 3.0-fold each, above and beyond the measured confounders, but weaker confounding could not do so. Much empirical research is concerned with establishing causation. It is well-known, however, that with observational data, association (111) need not imply causation (1222). A central concern with observational data is bias by unmeasured or uncontrolled confounding, that is, that some third factor related to both the treatment and outcome might explain their association, with no true causal effect (1222). With observational data, we can never be certain that efforts to adjust for confounders or common causes are adequate. An important approach to evaluating evidence for causation in the face of unmeasured confounding is sensitivity analysis (or bias analysis) (1422). Sensitivity analysis considers how strong unmeasured confounding would have to be to explain away the association, that is, how strongly the unmeasured confounder would have to be associated with the treatment and outcome for the treatmentoutcome association not to be causal. In this tutorial, we discuss a sensitivity analysis technique that makes minimal assumptions, and we propose that observational studies start reporting the E-value, a new measure related to evidence for causality. The E-value represents the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need to have with both the treatment and outcome to fully explain away a specific treatmentoutcome association, conditional on the measured covariates. Implementing these sensitivity analysis techniques and obtaining E-values are relatively simple. If reporting E-values for sensitivity analysis were standard practice, our ability to assess evidence from observational studies would improve and science would be strengthened. Example As a motivating example, several observational studies have reported associations between breastfeeding and various infant and maternal health outcomes. A common concern is that the effects of breastfeeding may be confounded by smoking behavior or by socioeconomic status. In a population-based casecontrol study, Victora and colleagues (23) examined associations between breastfeeding and infant death by respiratory infection. After adjusting for age, birthweight, social status, maternal education, and family income, the authors found that infants fed with formula only were 3.9 (95% CI, 1.8 to 8.7) times more likely to die of respiratory infections than those who were exclusively breastfed. The investigators controlled for markers of socioeconomic status but not for smoking, and smoking may be associated with less breastfeeding as well as greater risk for respiratory death. Sensitivity Analysis for Unmeasured Confounding Sensitivity analysis considers how strongly an unmeasured confounder would have to be related to the treatment and outcome to explain away the observed association. Several sensitivity analysis techniques have been developed for different statistical models (1422, 2441). Often, these techniques involve specifying several parameters corresponding to the strength of the effect of the unmeasured confounders on the treatment and on the outcome and then using analytic formulas to determine what the true effect of the treatment on the outcome would be if an unmeasured confounder of the specified strength were present. Such techniques are helpful in determining the strength of the evidence for causality. These techniques sometimes are criticized for being too subjectivethat is, regardless of the estimate obtained, investigators could choose the sensitivity parameters that make the result seem robust to confounding. Another criticism is that sensitivity analysis techniques themselves make simplifying assumptions about the unmeasured confounder. Such assumptions often stipulate that the unmeasured confounder is binary (22, 24, 26) or that only 1 unmeasured confounder exists (2428), or that there is no interaction between the effects of the unmeasured confounder and of the treatment on the outcome (2528). The criticisms then assert that these assumptions are needed to assess the effect of assumptions so that, in fact, the approach is not very useful after all. These criticisms are not unreasonable; however, on the basis of recent developments, addressing them is now possible (37). Specifically, some techniques make no assumptions about the underlying structure of unmeasured confounders and still allow conclusions about the strength the unmeasured confounders must have to explain away an observed association (37). We begin by describing such a technique and then introduce the new E-value measure. Suppose that an observational study controls for several covariates thought to be confounders of the treatmentoutcome association. After adjustment, suppose that the estimated relative risk equals RR. We may, however, still be concerned that this estimate is subject to unmeasured confounding. Suppose that all confounding would be removed if the study had controlled for 1 or more unmeasured confounders U, along with the observed covariates. The sensitivity analysis technique requires 2 parameters to be specified. One corresponds to the strength of the association between the unmeasured confounders U and the outcome D; the other corresponds to the strength of the association between the treatment or exposure E and the unmeasured confounders. Once these parameters are specified, we can calculate the extent to which such a set of unmeasured confounders could alter the observed relative risk. We let B denote the largest factor by which the observed relative risk could be altered by unmeasured confounders of a particular strength. In practice, we do not know the strengths of the unmeasured confounder associations, but we could, in principle, specify many different values and determine how the estimate is affected by each setting. Let RRUD denote the maximum risk ratio for the outcome comparing any 2 categories of the unmeasured confounders, within either treatment group, conditional on the observed covariates. Let RREU denote the maximum risk ratio for any specific level of the unmeasured confounders comparing those with and without treatment, with adjustment already made for the measured covariates. Thus, RRUD captures how important the unmeasured confounder is for the outcome, and RREU captures how imbalanced the treatment groups are in the unmeasured confounder U. For example, if 40% of nonbreastfeeding mothers smoked, as compared with 20% of breastfeeding mothers, we would have RREU = 2. The relationships are shown in Figure 1. Figure 1. Unmeasured confounder of the treatmentoutcome relationship. The maximum risk ratio for the outcome comparing any 2 categories of the unmeasured confounders, with adjustment already made for the measured covariates, is denoted in the diagram by RRUD . The maximum risk ratio for any specific level of the unmeasured confounders comparing those with and without treatment, with adjustment already made for the measured covariates, is denoted in the diagram by RREU . The measured covariates are allowed to affect the unmeasured confounders, and vice versa. Once these variables are specified, the maximum relative amount by which such unmeasured confounding could reduce an observed risk ratio is given by the following formula (37): To obtain the maximum amount this set of unmeasured confounders could alter an observed risk ratio RR, one simply divides the observed risk ratio by the bias factor B (37). In fact, one also may divide the limits of the CI by the bias factor B to obtain the maximum the unmeasured confounder could move the CI toward the null (37). The formula applies when the observed risk ratio RR is greater than 1. If the observed risk ratio is less than 1, then one multiplies by this bias factor rather than dividing by it. We illustrate this approach with the association between maternal breastfeeding and respiratory death from the study by Victora and colleagues (23), in which RR= 3.9 (CI, 1.8 to 8.7) for infants formula-fed rather than breastfed. Again, we might be worried that this estimate is confounded by smoking status. Suppose that the maximum ratio by which smoking could increase respiratory death is RRUD = 4 and the maximum by which smoking differed by breastfeeding status was RREU = 2. Our bias factor is then B= 42/(4+21)= 1.6. The most that unmeasured confounding could alter the effect estimate is obtained by dividing the observed risk ratio and its CI by