DOI: 10.3886/icpsr39485
title: Evaluating Observational Data Analyses: Confounding Control and Treatment Effect Heterogeneity [Methods Study], United States, 2013-2019
title: Archival Version

description: A randomized trial is one of the best ways to learn if one treatment works better than another. Randomized trials assign patients to different treatments by chance. But they are not always affordable, and they take a long time to complete.
When randomized trials aren't possible, researchers can use observational studies to learn how treatments work. In observational studies, researchers look at what happens when patients and their doctors choose the treatments. Traits such as age or health may affect treatment choices. These traits may also affect patients' responses to treatment, making it hard to know if the treatment or the traits affected the patients' responses.
Some study designs and statistical methods may help address this problem and make results from observational studies more useful. These methods can give researchers more data about whether treatments work and how the same treatment can affect groups of patients differently.
The research team conducted three studies to test different methods of designing and analyzing observational studies. They wanted to know if observational studies that used these methods produced results similar to randomized trials.
description: To evaluate whether methods for estimating treatment effects and heterogeneity of treatment effects (HTE) from observational data produce results consistent with evidence from randomized trials. The study aimed to address the following research questions: (1) If state-of-the-science analysis methods are used in information-rich observational data, do observational data analyses emulating target trials produce marginal (population) effects consistent with prior randomized evidence and background knowledge? (2) Do observational analyses emulating target trials agree with randomized trials and background knowledge when assessing heterogeneity of treatment effects? (3) How do different methods for assessing effect heterogeneity perform in observational studies?
description: In this project, researchers conducted three studies to test different methods of designing and analyzing observational studies to estimate treatment effects and HTE. Treatment effects refer to differences in the outcome distributions when comparing two or more treatments applied to the same individuals. HTE means the treatment effect is not the same for all individuals.
In study 1, researchers compared treatment effects from observational studies with treatment effects from randomized trials of the same interventions. Using data from claims and electronic health records, researchers developed four observational studies to emulate one hypothetical randomized trial comparing the effectiveness of antihypertensive medicines. Researchers used multiple methods, including propensity score-based methods, to estimate treatment effects in each emulation. They then estimated treatment effects for antihypertensive medicines in a pooled analysis of 11 trials and compared results from the emulations with the pooled trial analysis.
In study 2, researchers evaluated a novel approach of baseline risk stratification to assess HTE. Researchers used data from 32 randomized trials and modeled the predicted outcome risk in each trial using covariates associated with the outcome. Based on the predicted outcome risk, researchers stratified the trial population into four strata of increasing predicted risk. Researchers then assessed HTE by comparing absolute and relative risk reduction across the four risk strata.
In study 3, researchers compared the performance of different HTE estimation methods using simulated and real data. Researchers simulated different treatment scenarios and patient subgroups. They assessed HTE using outcome regression, inverse probability weighting, and doubly robust methods.
Clinicians, patients, clinical practice guideline developers, regulatory authorities, and funders provided input on this study.
description: Datasets: 
DS1: Evaluating Observational Data Analyses: Confounding Control and Treatment Effect Heterogeneity [Methods Study], United States, 2013-2019