Apply Propensity Score Methods in Causal Inference — Part 1: Stratification | by Shuangyuan (Sharon) Wei | Towards Data Science

This article introduces and implements the framework of propensity score method from Dehejia and Wahba (1999) “Causal Effects in Non-Experimental Studies: Reevaluating the Evaluation of Training Programs,” Journal of the American Statistical Association, Vol. 94, No448 (December 1999), pp. 1053–1062. I will briefly go over the theories and then walk through how I implemented the stratification matching step by step. The full Python code is provided at the end of the article.
The intuition of propensity score method is: instead of conditioning on the full vector of covariates Xi, which can get difficult when there are many pre-treatment variables and when the treatment and comparison groups are very different, we try to condition on the propensity score estimated with Xi.
If you need a recap of covariates matching, my previous article introduces the exact matching on the covariates with the Conditional Independence Assumption (CIA):
towardsdatascience.com
Define propensity score p(Xi) as
We have the following: Suppose the CIA holds such that
Then this holds:
Propensity score matching works in the same way as covariate matching except that we match on the score instead of the covariates directly. By the propensity score theorem:
In practice, the estimation is usually done in two steps. First, we estimate the propensity score. Second, we estimate the effects of treatment by using one of the matching methods.
The data used by Dehejia and Wahba (1999) are from the National Supported Work (NSW) project designed as a transitional, subsidized work experience program for people with long- standing “employment problems”. Eligible applicants were randomly assigned either to receive training (the treatment group) or to receive no training (the control group). The goal of the evaluation is to estimate the effect of trainings on earnings.
The analysis illustrated in the steps below is organized as follows: First, since the project was conducted as an RCT, the original data (NSW) are experimental data, which are used to estimate the treatment effect as a benchmark. Next, we use additional government survey data (CPS and PSID) to mimic the observational data. Then, we estimate and apply propensity score and stratification method to estimate the effect of the offer of training on earnings using the observation data .
If the randomization is done properly, we expect there should be no systematic differences in pretreatment characteristics between the control and treatment groups. Otherwise we may suspect that they differ also in their unobserved characteristics which may lead to biased estimator of the causal effect of the treatment.
It is therefore always a good idea to check that pretreatment variables are indeed balanced across the two groups. To test the covariate balance of the pre-treatment variables (re78 is the realized outcome variable and all the other variables are predetermined before the assignment to the treatment or control groups), I ran 8 separate simple linear regressions and report the results in the Table 1. It shows that the sample averages are not different across control and treatment groups.
Also, I estimate the average treatment effect (ATE) using the experimental data as the causal effect of being offered on-the-job training on the earnings (re87) post the training . Controlling the covariates, I obtained an effect of $1,676. I will keep this in mind as the benchmark.
Dehejia and Wahba (1999) used two survey data samples (CPS and PSID surveys) as comparison groups with the NSW treatment group to construct the observation database (CPS stands for Current Population Survey and PSID stands for Panel Study of Income Dynamics).
Similarly, I compared the sample means across the NSW (experimental data) and the added comparison samples (CPS and PSID). In the Table 3, it is obvious that The NSW-Treated differ quite a lot from CPS/PSID in the demographic characteristics as well as the pre-treatment earnings (re74 and re75). The NSW treatment groups are younger, less educated, more likely to be nonwhite, and have much lower earnings before the program. It makes no sense to directly compare the NSW-Treated with the comparison groups.
From this step onwards, I use PSID as the comparison group to construct the observation database with the NSW-Treated.
I used logistic regression model to estimate the propensity score. In particular, I explored two specifications regressing on two different sets of covariates (in the second set, I added the squared age and education years):
X1 = [‘age’, ‘ed’, ‘black’, ‘hisp’, ‘married’, ‘nodeg’, ‘re74’,’re75']X2 = [‘age’, ‘ed’, ‘black’, ‘hisp’, ‘married’, ‘nodeg’, ‘re74’,’re75',’age2',’ed2']
Next, I used cross validation to tune the parameter C which is the strength of the regularization. Higher values of C means less regularization.
best = 0.0for strength in [0.00001,0.0001,0.001,0.01,0.1, 1, 10, 100, 1000, 10000, 100000, 1000000] : model = LogisticRegression(C=strength, max_iter=10000) curScore = cross_val_score(model,df_combine[X1], df_combine[T],scoring=’f1').mean() if curScore > best : best = curScore print(strength) print (curScore)
Next, I ran the logistic regression model with the best C value and added the estimated propensity score back to the observation set. Below I plotted the estimated p-score across comparison and treatment groups below. Not shockingly, there is not much overlap between the two groups. The mean p-score of treatment group (n=185) is 0.62 while the mean p-score of comparison group is 0.03 (n=2490). In fact, as Dehejia and Wahba (1999) pointed out, one of the strengths of the propensity score method is that it dramatically highlights the fact that most of the comparison units are very different from the treated units.
The drastic differences between the treatment and comparison groups naturally lead to the steps of trimming and matching, which in my Python code are done in the same function stratification_function. The purpose of trimming is to find the common support so that no p-score from either treatment or control group falls outside it. After the trimming, we discarded roughly 1,300 observations from the PSID comparison group that are outside the range of common support, most of which have estimated propensity scores less than the minimum estimated propensity scores for the treated units. The reason for trimming is to prevent situation in which we end up with treated units without matching control units, and vice versa, when we enter the stratification/matching stage.
The basic idea of stratification is to define strata based on estimated propensity score and then group treated and comparison units into each strata. Within each stratum, we take a difference in means of the outcomes between the treatment and comparison groups, then we calculated the weighted mean across strata by the number of treated units within each stratum. The final weighted mean difference across treatment and control groups (ATT) is calculated as:
In my code, it is done in the function stratification_function. I also tested two ways of defining strata:
blocks1 = np.array_split(df_ps.query(“treat==1”)[“pscore”].sort_values(), nostrata)blocks2 = np.linspace(0,1,11)
Suppose we want 10 strata. The first way divides the estimated propensity score of treated units into 10 number of equal sized blocks and then match comparison units with estimated propensity score within the range of each block. The second way simply divides 1 into 10 equally spaced intervals.
Remember in the Step 3 Propensity Score Estimation, I tested two logistic regression models. Combining the two logistic regression models with the two definitions of strata, I have four models at the output which I name as: M11 (reg1, strata1), M12(reg1, strata2), M21(reg2, strata1), and M22(reg2, strata2). The results are presented in the Table 4 :
Out of these four options, the final model should be chosen based on the how well the covariates within each stratum are balanced across treated and comparison units. Therefore, I think Model21 is the best. Model21 estimates the effect = $1,632 (with std. error=1,514), which is pretty close to our experimental benchmark effect = $1,676.
Lastly, as promised, below is my full Python code for Step 1 to 5 which also output the summary tables 1–4 :
I downloaded all the data from NYU’s website