DOI: 10.3886/icpsr37291
title: Panacea or Poison: Can Propensity Score Modeling (PSM) Methods Replicate the Results from Randomized Control Trials (RCTs)?, United States, 1983-2013
title: Archival Version

description: With the growing popularity, technological ease of using propensity score modeling (PSM), and the concern over its reliability and validity among scholars and practitioners, the researchers aimed to answer whether PSM methods can replicate the results from randomized controlled trials (RCTs). In this secondary data analysis, the researchers gathered the datasets of 10 publicly available and restricted RCT studies from the National Archive of Criminal Justice Data (NACJD), introduced an artificial selection bias into the treatment groups of these investigations, and then used each PSM technique to remove this selection bias. The team then compared the results generated from the PSM methods to those derived from the original RCT experiments, and meta-analyzed the findings across all studies to reveal the true reliability and validity of PSM in relation to RCTs using criminal justice data. For each study used in this analysis, the researchers created SPSS syntax for variable recodes and artificial bias creation and a codebook with original study items, recoded variables, and analytic variables. (In one study, two RCTs were conducted and thus two sets of syntax and codebooks were created.) Seven text files contain the Stata and R code used to run each PSM technique. These materials have been zipped into a package and are available for restricted download. Please refer to the ICPSR README for more information.
description: The purpose of the study was to determine whether results obtained via randomized controlled trial can be replicated by using propensity score matching in criminal justice research studies.
description: A total of seven propensity score matching (PSM) techniques were used in this analysis due to their common usage in criminal justice research: one-to-one matching (with and without caliper), one-to-many matching (with and without caliper), inverse probability of treatment weighting (IPTW), stratified weighting scheme, and optimal pairs matching. In order to test the different PSM techniques, selection bias was systematically introduced into the treatment groups by identifying the most critical measures in each study and those that best predicted the likelihood of being in the treatment group, constructing an additive scale from variables that successfully predicted treatment group membership, and selecting only the treatment cases above the scale's mean. This process was done in a blind manner, meaning that the measures used to create the bias were unknown to the researcher who conditioned the propensity score. The biased and PSM treatment and control groups were compared to each other as well as the RCT using six measures of balance (e.g., percentage of covariates with statistically significant differences and the standardized percent bias). For each dataset, a common metric effect size for all dependent variables in the original RCT and for the seven PSM techniques after removing the artificial selection bias. The PSM and RCT results were compared in relation to their effect sizes and direction (i.e., positive or negative). The researchers calculated the difference in the estimated outcomes between the RCT and PSM methodologies and averaged across the total number of outcomes included. The overall effect size for the difference between each PSM and RCT approach was calculated using a random-effects model meta-analysis.
description: For each dataset, measures were created for predictors for treatment membership group and artificial selection bias (if a case had a significant bias relative to comparison cases).
description: Not applicable.
description: Response Rates: Not applicable.
description: Presence of Common Scales: Not applicable.
description: In order to be included in this evaluation, studies needed to use a random assignment procedure in assigning cases to groups and have at least 130 cases in each condition group (based on the threshold needed to properly detect effect size with enough power). A systematic search of NACJD identified 46 potential studies, from which only nine studies of different types met the sample size requirements. A quasi-experimental study that identified comparison group participants prior to program rollout was added to raise the number of studies to 10.
description: Datasets: DS1: Panacea or Poison: Can Propensity Score Modeling (PSM) Methods Replicate the Results from Randomized Control Trials (RCTs)?, United States, 1983-2013
description: Criminology and criminal justice research studies that involved a randomized controlled trial. Smallest Geographic Unit: None
description: For the 10 datasets upon which this analysis is based, please refer to each study's ICPSR page (see Data Source for a complete listing) for data, documentation, and more information on background, study design, methodology, sampling, and variables.