Causal Inference in Data Science: A/B Testing and the need for Marginal Structural Modeling
By: Andrew Rothman
Tags: Data Science, Machine Learning, Artificial Intelligence, Mathematics, Editors Pick

Causal Inference in Data Science: A/B Testing and the need for Marginal Structural Modeling

Required Adjustment of A/B Tests via G-Methods in the Presence of Informative Censoring

Photo by Arnold Francisca on Unsplash

1: Background and Motivation

Causal Inference is of interest to a wide range of practitioners including Statisticians, Data Scientists, Machine Learning Scientists, and other Computational Researchers. Recovery of unbiased estimates of Causal Effects is at times a tough task. In my previous pieces on Doubly Robust Estimation and G-Estimation of Structural Nested Models, we discussed leveraging G-Methods in non-randomized settings. We also discussed issues with M-Bias and confounding identification in non-randomized settings.

Contrary to popular belief, recovery of valid estimates of Causal Effects can be equally as difficult (or sometimes even unidentifiable and impossible) with A/B Testing (aka randomized trials). Regarding the use of A/B Testing in industry settings, I’ve noticed an often over-simplified view of randomized trial designs; a view of A/B Testing that corresponds to “idealized randomized trials”. For A/B Tests conducted at scale, particularly those that are longitudinal and carried-out over lengthy periods of time, very rarely do they meet the assumptions of an idealized randomized trial.

For example, let’s assume you work for a technology company that hosts an online app. Your department is interested in measuring the impact of a new feature design on the user experience. You are tasked with recovering an unbiased estimate of the Mean Causal Effect of Intervention Z (the new feature design) on outcome Y (the click through rate, or some other success metric). You design your A/B Test by randomly assigning users via IP-address to a version of Intervention Z (experience the new feature design vs. not). You carry-out your experiment collecting data on Outcome Y (success metric).

As a function of your observed data, you recover an empirical estimate of the mean marginal effect difference of Z on Y. You may naively assume the “true causal structure” of your A/B Test corresponds to the Causal Directed Acyclic Graph (DAG) below:

Image by Author

… with the assumption that the recovered marginal empirical estimate is an unbiased estimate of the Mean Causal Effect Difference of Z on Y:

Image by Author

Well, the above Causal DAG is under the following conditions of an “idealized randomized trial”:

No Intervention cross-over, with perfect subject adherence

No informative censoring of outcome Y due to loss-to-follow-up

The above conditions rarely hold in real-world A/B Tests. As inconvenient as this may be to hear, the true causal structure of most large-scale real-world A/B Tests usually look something more like this:

Image by Author

Due to informative censoring (conditioning on C, and by proxy condition on L), randomized intervention assignment Z and outcome Y are clearly not d-separated in the DAG above due to the open path from Z -> A -> [L] <- U2 -> Y. Therefore, the marginal empirical estimate of the effect of Z on Y over the available uncensored data will not recover an unbiased estimate of the Mean Causal Effect of Z on Y, and our inferences will be incorrect.

Image by Author

In this article, we will clearly walk through an example where a large-scale long-term A/B Test corresponds to something like the complex DAG above, and how G-Methods (specifically Inverse Probability Weighting with Marginal Structural Modeling) is the only appropriate method for adjustment due to informative censoring. We will also conduct a computational simulation to investigate these methods.

The contents of this piece are as follows:

Image by Author

2. Subject non-adherence, Intervention Cross-over, and the Intent-to-Treat (ITT) Estimator

Again, starting with an idealized randomized trial, we have the following Causal DAG:

Image by Author

Given Z is randomized, there are by definition no “causes” (i.e. arrows) into Z. However, it’s important to recognize that Z is in truth just the “randomized intervention assignment”, the key-word being assignment. There may be a small percentage of subjects that non-adhere to their randomized assignment and instead experience (i.e. observe) the other intervention arm. The intervention observed is A, of which Z is a direct cause of. And given A is not randomized, there may be unmeasured common-causes U1 of A and Y.

Let us postulate a descriptive use-case. We randomize subjects to intervention assignment Z based on their IP address. However, some heavy-tech using subjects may have multiple IP addresses (represented by U1). This allows those subjects to possibly experience the intervention arm they explicitly want (i.e. observed intervention A). Being heavy-tech savvy with multiple IP addresses is also a direct determinant of the outcome Y (click-through rate).

The Causal DAG we are left with is below:

Image by Author

It’s additionally important to recognize from a scientific perspective the causal contrast we are most interested in estimating? We are not really interested in the Causal Effect of intervention assignment Z on outcome Y. Rather, we are really interested in the Causal Effect of observed intervention A on outcome Y.

Given A and Y are not marginally d-separated in the Causal DAG above due to unmeasured common-cause U1 (and given it’s unmeasured we cannot condition on it), what are we to do? We reasonably have two options:

Note that randomized intervention assignment Z is in fact an Instrumental Variable (IV) of the Causal Effect of A on Y . Assuming our use-case meets certain homogeneity conditions, we could specify an IV estimator to recover an unbiased estimate of the Mean Causal Effect of A on Y among the compliers. I will not cover this method here, but rather write a future piece specifically on Instrumental Variable estimation.

is in fact an Instrumental Variable (IV) of the Causal Effect of on . Assuming our use-case meets certain homogeneity conditions, we could specify an IV estimator to recover an unbiased estimate of the Mean Causal Effect of on among the compliers. I will not cover this method here, but rather write a future piece specifically on Instrumental Variable estimation. Recognizing that our problem is a superiority trial, the Mean Causal Effect of randomized intervention assignment Z on Y is the Intent to Treat (ITT) Estimator of the Mean Causal Effect of observed intervention A on outcome Y. This is the method we will leverage.

In the context of superiority trials with a binary intervention, and under the assumption of no informative censoring, the expected value of the ITT estimator is “null-preserving”. Mathematically this means the expected value of the estimated Mean Causal Effect of Z on Y will always be closer to the null than the true Mean Causal Effect of A on Y. Therefore, by recovering an estimate of the Mean Causal Effect of Z on Y, we are indirectly also recovering a hard lower-bound for the Mean Causal Effect we are interested in (A on Y). And importantly, because the ITT estimator is “null preserving”, if the true Mean Causal Effect of A on Y is null, then the expected value of the estimated Mean Causal Effect of Z on Y will also be null.

Image by Author

However, we’re not quite done yet. Recall how I stated above that the ITT estimator is only valid under the assumption of no informative censoring, or that informative censoring has been adjusted for. This is the issue we will next tackle.

3. Informative Censoring due to loss-to-follow-up, and adjustment with Marginal Structural Modeling

Starting with the DAG from the previous section where we described the ITT estimator, we now acknowledge that in our experiment we have informative censoring leading to the Causal DAG shown below:

Image by Author

Specifically:

There is a variable L that is a direct determinant of observed outcome A

that is a direct determinant of observed outcome There are unmeasured common causes U2 of L and outcome Y

of and outcome Variable L is a direct and sole cause of variable C, being “censored” in the analysis. By “censored” we mean we “lose the subject to follow-up”. For censored subjects (C=1), we do not observe and measure their outcome Y. Therefore, these subjects are implicitly dropped from the analysis, given we do not observe the outcome information to include them. This is why in the Causal DAG above we condition on variable C. We do so because we have no choice. We only observe outcome information for subjects with C=0, and are restricted to this population for the analysis.

Let’s add to our descriptive use-case. Let’s say before deploying our AB-test, we did not thoroughly check that the new feature was correctly configured or default-compatible with all popular (and some less popular) internet browsers. Let’s say the new feature is inaccessible with the default settings on Internet Explorer. Explorer users first need to remove a default firewall block (represented by variable L). Predominant users of Explorer are older in age (unmeasured variable U2) which is also a direct determinant of the outcome Y (click-through rate). Many Explorer users therefore don’t experience the intervention A at all, we don’t measure their outcome Y, and they are “censored” from the analysis (C=1). We call this structure informative censoring.

Let’s now discuss how we can adjust for informative censoring under the given conditions. As noted earlier in this article, randomized intervention assignment Z and outcome Y are clearly not d-separated in the DAG above due to the open path from Z -> A -> [L] <- U2 -> Y. Therefore, what are our options?

The ITT estimator addresses confounding by U1 , but does not address the informative censoring issue. So this alone cannot save us.

, but does not address the informative censoring issue. So this alone cannot save us. An IV estimator alone cannot save us either for the same reasoning as the ITT estimator.

If we had recorded data on U1 and U2 , we could adjust for these variables using standard conditional methods. However, given U1 and U2 are unmeasured, this is not an option. We additionally may not be willing to assume we even know what U1 and/or U2 represent; simply that they exist.

and , we could adjust for these variables using standard conditional methods. However, given and are unmeasured, this is not an option. We additionally may not be willing to assume we even know what and/or represent; simply that they exist. We may be able to leverage the class of G-Methods? Perhaps Standardization with the g-formula or Inverse Probability Weighting (IPW) with Marginal Structural Modeling? We cannot use Standardization via the g-formula, given we don’t observe the outcome Y for those that are censored (C=1). We can however leverage IPW with Marginal Structural Modeling, given we observe data on the intervention for all subjects. We can recover IPWs for each observation via specifying the following Causal SWIG (Single World Intervention Graph):

Image by Author

Image by Author

Image by Author

As we are only interested in recovering estimates of the Mean Causal Effect in a pseudo-population where all observations where uncensored (c=0), we specify the IPWs for uncensored observations accordingly, and IPWs of zero for all observed observations that were censored (C=1):

Image by Author

If we empirically compute the appropriate IPW for each observation in our observed dataset, we can generate a pseudo-population by weighting each observation by it’s corresponding IPW. This pseudo-population will correspond to the Causal SWIG in the figure above. We then fit the following Marginal Structural Model on the pseudo-population by Weighted Least Squares (WLS) Regression:

Image by Author

And that’s it! We’ve recovered a null-preserving lower-bound for our mean causal contrast of interest. A bit more involved than your typical marginal estimate.

Let’s proceed to our computational simulation.

4. Computational Simulation of Toy Example with Marginal Structural Modeling

We’re going to conduct a computational simulation in Python to investigate A/B testing with adjustment for informative censoring by Marginal Structural Modeling. We will:

Create a simulated dataset with the true Causal DAG as shown below in Figure with the true Causal Effect Difference of A on Y being null (i.e. equal to zero) Recover an unbiased estimate of the ITT estimator of Z on A in a pseudo-population where all subjects are uncensored (c=0) We will repeat steps 1 and 2, but with a simulated dataset with the true Causal Effect Difference of A on Y is 0.11

Image by Author

Let’s import our needed libraries: