Automated Detection of Causal Inference Opportunities: Regression Discontinuity Subgroup Discovery
Authors: Tony Liu
Patrick Lawlor
Lyle Ungar
Konrad Kording
Rahul Ladhania
Venue: Accepted by TMLR ()

Abstract: The gold standard for the identification of causal effects are randomized controlled trials (RCT), but RCTs may not always be feasible to conduct. When treatments depend on a threshold however, such as the blood sugar threshold for diabetes diagnosis, we can still sometimes estimate causal effects with regression discontinuities (RDs). RDs are valid when units just above and below the threshold have the same distribution of covariates and thus no confounding in the presence of noise, establishing an as-if randomization. In practice however, implementing RD studies can be difficult as identifying treatment thresholds require considerable domain expertise -- furthermore, the thresholds may differ across  subgroups (e.g., the blood sugar threshold for diabetes may differ across demographics), and ignoring these differences can lower statistical power. Finding the thresholds and to whom they apply is an important problem currently solved manually by domain experts, and data-driven approaches are needed when domain expertise is not sufficient. Here, we introduce Regression Discontinuity SubGroup Discovery (RDSGD), a machine-learning method that identifies statistically powerful and interpretable subgroups for RD thresholds. Using a medical claims dataset with over 60 million patients, we apply RDSGD to multiple clinical contexts and identify subgroups with increased compliance to treatment assignment thresholds.
As treatment thresholds matter for many diseases and policy decisions, RDSGD can be a powerful tool for discovering new avenues for causal estimation.

Lyle Ungar Published in Transactions Machine Learning Research (10/2023)
Automated Detection of Causal Inference Opportuni- ties: Regression Discontinuity Subgroup Discovery
Tony Liu
litoony@seas. upenn, edu
University of Pennsylvonia, Robloz
Patrick Lawlor
lawlory @chop. edu
Children's Hospital of Philadelphia
ungar upenn.edu edu
University of Pennsylvonia
Konrad Kording University of Pennsyluonia
koerding Bectongg@@mal..mmm com
Rahul Ladhania University of Michigon
ladhania @umich. edu
Reviewed on OpenReview: https: // openreview. net/ forum? id= cdRYo
Abstract
The gold standard for the identification of causal effects are randomized controlled trials (RCT), but RCTs may not always be fcasible to conduct. When treatments depend on a threshold however, such as the blood sugar threshold for diabetes diag- nosis, we can still sometimes estimate causal effects with regression discontinuities (RDs). RDs valid when units just above and below the threshold have the same distribution of covariates and thus no confounding in the presence of noise, establishing an as-if randomization. In practice however, implementing RD studies can be difficult as identifying treatment thresholds require considerable domain ex- pertise - furthermore, the thresholds may differ across subgroups (c.g., the blood sugar threshold for diabetes may differ across demographics), and ignoring these differences can lower statistical power. Finding the thresholds and whom they apply is an important problem currently solved manually by domain experts, and data-driven approaches are needed when domain expertise is not sufficient. Here, we introduce Regression Discontinuity SubGroup Discovery (RDSGD), machine- learning method that identifies statistically powerful and interpretable subgroups for RD thresholds. Using a medical claims dataset with over 60 million patients, we apply RDSGD to multiple clinical contexts and identify subgroups with in- creasod compliance treatment assignment thresholds. As treatment thresholds matter for many diseases and policy decisions, RDSGD can be powerful tool for discovering new avenues for causal estimation.

---

Published in Transactions Machine Learning Research (10/2023)
1 Introduction
Many questions in data science are ultimately causal in nature, yet evaluating causal questions through experimental randomization can be costly or otherwise infeasible (Musci and Stuart 2019) There are numerous methods that estimate cansality from observational data, but many rely on the key assumption of no unobserved confounding, which is generally difficult to justify in realistic data settings (Hernán and Robins, 2020). However, econometricians have been developing study designs that can make credible causal claims from observational data (Leaner 1983 Angrist and Pischke, 2010). These study designs address confounding by exploiting naturally ly occurring randomness in the data, so-called quasi- cousi-czperiments (Angrist and Pischke 2008 Liu et al. 2021
We focus on the regression discontinuity (RD), specific quasi-experimental method for evaluating causal effects from observational data where a cutoff in an observed continuous runming variable determines treatment assignment (Hahn et al. 2001]. ) Such a situation may arise when treatment depends on a threshold. For example, when a patient's 's blood sugar level (measured by A1C %) above 6.5%, they are diagnosed as diabetic (American Diabetes Association 2010) and hence eligible for treatment assignment. Critically, RDs more robust to confounding than other observational causal inference methods (Lee and Lemieux 2009), the cutoff in treatment assignment provides "as-if" randomization for individuals just above and just below the cutoff: patient with an A1C of 6.5%, on average, is not materially different from a patient with an A1C of 6.4%, yet the former is diagnosed with diabetes and treated for the discase while the latter is not. If other covariates smoothly vary with the running variable, then the "just below" subjects arguably have the same (observed and unobserved) covariate distribution as the "just above' subjects. Because of this "as-if" randomization, RDs allow to estimate treatment effects the threshold without explicit randomization. RD opportumities present in education (Valentine et al. 2017), political science (Skovron and Titiunik, 2017), criminal justice (Berk and Rauma 1983), and are particularly natural in medicine, where thresholds govern decisions in many discases, c.g. diabctes, coronary artery disease, and cancer (Petersen el al. 2020: Scott et. al. 2022 Oellinger et al. 2015).
Despite the ubiquity of such RD opportunities, RDs are underutilized Moscoe et al. 2015 Mari- neseu et al. 2018). Because a priori knowledge of the treatment threshold needed use an RD, the typical study design approach is "top-down" process, in which domain expert hypothesizes that a particular data-generating process might yield RD, followed by verification of study valid- ity by examining the data. Often enough, the RD opportunity is underpowered due to sample size limitations [Naddech et al., 2020 McKenzie, 2016). Identifying potential RDs is an ad-hoc process that relies heavily on human intuition and domain expertise, and thus does not scale well to the vast amounts of high-dimensional data we have available today.
This holds especially true as treatment thresholds in practice are often faceted, with hetero- geneity in treatment assignment as function of other covariates. For example, in medicine, though diagnostic criteria for diabetes ostensibly are made only according to blood sugar levels, the risk for the disease varies by gender, race, and age categories, leading to different clinical decisions where official guidelines may not always be followed. Ignoring these differences can lower statistical power. As treatment assignment thresholds bocome more complex, it bocomes difficult for domain experts to generate study designs and verily them. Given the many domains where RDDs are a useful tool for causal inference, a "bottom-up" data-driven approach would streamline and scale RDD study discovery, unlocking opportunities for more interpretable and well-powered causal studies.
2

---

Published in Transactions Machine Learning Research (10/2023)
Full pallppplattan
Beoounter dele 20147
EnGRuESo dats 2314
10000
1. Low compliance in full population RD yllld low effactiva sample sie
Unknown subgroups in population
3. Data drived tree discovery more compliant subgrouns
Improved effective sample feasiblitty RDSGD
$51,37 Gaarple six) 0.26 (oomp
a- 193 209
Figure 1: This work develops RDSGD, a machine learning method for discovering RD op- portunities that improve study feasibility by searching for subgroups with higher TAU com- pliance. We show our colon cancer case study (Section ass an example.
Here we propose data-driven method, Regression Discontinuity Subgroup Discovery (RDSGD), to learn RD subgroups with different treatment assignment thresholds (see Figure I for an illustration of our approach). We frame regression discontinuity discovery similarly to the task of conditional average treatment effect (CATE) estimation (Section 3 Note that our method differs from CATE estimation by focusing on heterogeneity in treatment assignment rather than in treatment effects. We int roduce novel framework largeting higher effective sample sizes of the discovered subgroups maximize statistical power and maintain interpretability (Section 4 We show the utility of our approach through both synthetic experiments (Section 5) and case study using a medical claims dataset consisting of over 60 million patients (Section G We apply our method to three clinical contexts and discover subgroups for each setting, with some that are validated by domain knowledge and others that show promise for potential studies. RDSGD can not only discover new opportunities for experiexperi studies but also provide actionable interpretability in the form of treatment assignment subgroups, which can be easily understood by practitioners (Section
2 Related Work
Automatic regression discontinuity discovery has been explored in related contexts. Porter and Yu (2015) propose statistical testing framework for regression discontinuity treatment effect inference where the discontimuity point is unknown. In particularly relevant work, Herlands et al. (018)) define an automated RD search procedure called local regression discontinuity discovery (LoRD3) that first requires fitting a "smooth" background function to the probability treatment, and then computing test statistics for candidate RDs based on the residuals of the background function. However, neither LoRD3 nor Porter and Yu 2015)'s 's approach consider heterogeneity in treatment. assignment. Our method is the first RD discovery algorithm that frames heterogencity in additional machine learning task.
covariates which affect treatment
Other bodies of work consider treatment effect discovery (McFowland 111 et al. 2018 Ogallo et al. 2021 and CATE estimation for posthoc subgroup analysis (Spicss and Syrgkanis 2021 Lipkovich el al. 2017), which require randomized controlled experiments with interventions. Though there are some parallels with our work, our method focuses on the treatment assignment estimation problem (in contrast to the treatment effect estimation problem), making method amenable to
3

---

Published in Transactions Machine Learning Research (10/2023)
the discovery of causal inference opportunities agnostic of the downstream outcome. Furthermore, method only requires observational data that are amenable to regression discontiniity analysis.
Within the regression discontinuity literature, recent work examines the use of covariates in RDs, highlighting the potential for efficiency gains in inference (Cattaneo et al. 2022) and tree-based CATE estimation for a given RD cutoff (Reguly 2021). Our method advances this space by framing RD study discovery as a data-driven task that utilizes covariates to uncover the most promising RD candidates.
Specifically, we take novel approach by: 1) formulating the discovery procedure trealment assignment uptake (TAU) estimation and 2) identifying heterogeneous subgroups with higher effective sample size for both improved interpretability and statistical power. Our method improves on prior work by using ML to identify statistically powerful interpretable subgroups for RD studies.
3 RD Discovery Framework
In the following section we build upon well-established (and Nobel-prize awarded) econometric estimation frameworks (Imbens, 2014 Angrist and Pischke, 2008) as well as the conditional average treatment effect (CATE) estimation literature (Athey and Imbens 2016 Chernozhukov et al. 2018) to frame our RD discovery procodure. We then target the effective sample size of the discovered RD and show how oplimizing this quantity increases statistical power.
3.1 Regression Discontinuity Preliminaries
Here we review the potential outcomes framework for analyzing regression discontinuities (RDs): see, e.g. Imbens and Lemieux (0077) and Cattaneo et al. (2019a) for comprehensive overviews of
RDs. We define the following notation for an individual i:
X; rumning variable for individual
assignment threshold for X;
W = vector of pre-treatment covariates for individual i
Z, [[, c), threshold indicator for individual
Y: observed outcome for individual
T:(-) potential treatment assignment for individual
Yı(-) potential outcome for individual i
We note here that the potential treatment assignments are defined in terms of the threshold indicator (Z,). T1( corresponds to the potential treatment assignment for X; 2 c, and T,(0) to X, < c. We focus on the "fuzzy" regression discontinuity (FRD) case, which assumes that the probability of treatment assignment uptake jumps c, but not necessarily from 0 to 1 (Hahn et al. 2001 (without loss of generality, also assume that the jump in probability at the threshold is positive) (Hahn
et al. 2001):
Assumption (FRD). The limits x lim E x] and x lim ETTX exist, and x x=" is non-zero, 0 < T < 1.
4

---

Published in Transactions Machine Learning Research (10/2023)
We also define compliers as individuals for which Ti(1) > T,(0), namely that they receive the treatment when above the threshold, and do not receive treatment when below the threshold. Under the assumptions of continuity, monotonicity, and threshold excludability (Lee and Lemieux (209)) Appendix B.1 the treatment effect estimate y can be written as a ratio:
limm E x| limato E x] limg E = x limato E = x]
Where X is the jump in outcome Y at the cutoff and is the jump in treatment assignment uptake (TAU).
In practice, Equation 1 is often estimated by fitting two local linear regressions using the disconti- nuity indicator Z within a bandwidth h the left and right of the cutoff: a "first-stage" treatment regression for t and "second-stage" oulcome regression for À (Imbens and Lemieux, 2007), with
data-driven methods developed to select the bandwidth h (Imbens and Kalyanaraman 2009
As a rumning example throughout the text, we consider evaluating the effect of breast cancer screening age guidelines, which recommend beginning screening at age 40 for women (Oelfinger et al. 2015). Setting the recommended screening age is an important decision that can impact millions of patients the U.S. However, evaluating the causal impact of particular screening it would disrupt standards
age through a randomized experiment would be difficult logistically of clinical practice. Because the screening decision T is made based on the continuous variable age X being above the threshold of 40 years, such a causal question can be evaluated using a regression discontinuity design, and has been done in prior work (Kadiyala and Strumpf, 2016). Additionally, since compliance with the threshold assignment is imperfect (not everyone at the age of 40 will deterministically be sereened for brcast cancer), the sccnario lends itsolf naturally to the Tuzzy RD framework of T estimation we consider here.
C h
c h
To formulate a data-driven RD discovery pro- codure, now turn attention to the "first- slage" lask of estimating in the context of predicting unit's compliance status.
Probability Treatment P(T)
X
local regressions
3.2 Treatment Assignment Uptake (TAU)
Treatment Assignment
Uptake (TAU)
As the goal of our method is to identify RD opportunities rather than explicitly estimating treatment effects, we focus on estimating treat- ment assigument uptake (TAU) т, and in par- ticular will look to maximize TAU using het- crogeneity in observed covariates. In our breast cancer example, the TAU increases with greater adherence to the clinical guidelines: the more women who begin screening at the age of 40, the larger the discontinuity is treatment assign- ment uptake. т often modeled using a local linear regression within a bondwidth around the cutoff (Hahn et al. 2001):
Running variable x
Figure 2: The anatomy of the RD first stage regression, where a local linear regression used estimate TAU т. local linear regression with threshold indicator 7 1[x; 2 c) is fit using data within the shaded bandwidth of X E [h-c, h+ (Eq. 2. An analogous regression with oulcome Y on the vertical axis is used to estimate X.
5

---

Published in Transactions Machine Learning Research (10/2023)
T TeZ 3o + Z) - 32Z(X -c) + +c
(2)
Where treatment assignment uptake is indexed by the cutoff c, is homoskedastic noise, and samples are within X, E [c lc--hce h,c+ h] (Figure 2). We use this linear probability model estimation strategy in order to ensure causal validity, which is commonly used in the econometric literature as an efficient approach to estimate treatment assignment (though other nonpmaraettric methods can be used as well) (Imbens and Lemieux 2007). Our RD study discovery task is formalized as a hypothesis test of the existence of a treatment discontinuity at threshold e: Ho Te 0, H Te # O, which be tested via significance of the estimated & in Equation 2 Our estimation and
subsequent maximization of TAA can be equivalently framed as estimation of compliance probability for subgroups at the threshold (Aronow and Carnegie 2013 (see Appendix B.2
Proposition 1. For given bandwidth h and culoff c, eslimating f.. is equivalend to estimating the probubility of compliance ((T(1) > T(0)).
We leverage this connection to maximize TAU heterogeneity and discover the most promising subgroups for RD analysis.
3.3 Heterogeneity in TAU
Beyond identifying candidate thresholds that produce significant TAU estimates, we want to find heterogeneous subgroups among our sample population at a given cutpoint e to propose more statistically powerful RD studies. This problem can be seen as conditional compliance estima- tion (Kennedy al., 2020). where we identify the individuals (the compliers) to which the threshold cutoff applics, using the other pre-treatment covariates W. In our breast cancer screening example, we would clearly want to exclude all men, lest their inclusion reduce the treatment discontinuity due to their non-compliance with the screening guideline. Other factors such as family history or genetic risk may also influence whether individuals adhere to the guideline.
In order to identify such subgroups, we define the heterogeneous TAU estimation task. Given the standard FRD assumptions presented in Section 3.1 Kennedy et al. (020)) and Coussens and Spiess (2021) have shown that estimating the probability unit is complier, T<(W,) (their TAU proba- bility), can be framed as conditional average treatment effect (CATE) estimation (Appendix B.3
Proposition TWW)) given e can be identified as the conditional probability of compliance.
P(T(1) > T(O) w = TWWW)
3)
Heterogencous TAU TW) can thus be estimated using data-driven machine learning mothods developed in recent years for CATE estimation (e.g., Chernozhukov et al. (2018); Oprescu et al. (2019), and Padilla et al. 2).1)). The machine-learned estimates of T<(W) will be unbiased due to the sample-spliting honesty property of such estimators (Athey and Imbens 2016 Chernozhukox
et al. 2018). We can thus estimate T.(W)) for a given RD threshold c.
Because our goal is to identify subgroups of individuals where treatment assignment uptake varios, we choose use tree-based approaches (Oprescu et al. 2019 Athey el al. 2019) for the estimation problem, which provide valid, honest, and interpretable subgroup populations defined by the learned causal trec's nodes. In particular, can distill treo-based model that estimates 4 (W) into
6

---

Published in Transactions Machine Learning Research (10/2023)
single decision tree, and extract heterogeneous subgroups that correspond the learned tree's nodes (Athey and Imbens] 2016 Battocchi et al. 2019 Tree-based CATE models thus provide data-driven approach for identifying interpretable subgroups that have heterogeneous TAU.
3.4 From TAU effective sample size
Though we have established the TAU objective for heterogeneous treatment uptake at a threshold and an approach to identify subgroups using CATE estimators, actually increase power in finite samples cannot only account for T. TW): we also need to consider the sample size of the subgroup. Solely maximizing for TAU when discovering subgroups may not yield higher power, as it is possible for such an objective to select unfcasibly small subgroups with higher TAU: in our breast cancer example, a subgroup of ten women may have a higher TAU than & subgroup of size 1,000 with 50% women, but we would much prefer the latter subgroup in terms of study feasibility. Thus, we propose to target the effective sample size (Liu et al. 2022 Heng et al. (020) of a given subgroup G TiG, which explicitly accounts for both the TAU as well as the size of the subgroup. Let P= (W; , Zi, Tú, X represent the "baseline population" " samples i.e. all of the samples within the bandwidth for a cutoff e where Np is the sample size, and G = (W,, Zj, T3. X1) represent the samples that are part of subgroup G, where Ng is the sample size. The TAU for a subgroup G is defined as TO == Te),), where WG are the pre-treatment covariates that define a sample's membership in group G; similarly, Tp is the TAU for all of the samples in the bascline population. Letting [a[:] be expectations over subgroup G, the effective sample size B|a is then:
NG NoT (o 1] E 0))
(4)
The intuition behind 76G that only units compliant with the threshold indicator Z contribute to the treatment effect estimation. Non-compliers contribute noise to the estimate, so the "elfective" sample is the nominal sample size scaled by quantity of ta. which is the probability of compliance with the threshold indicator (Proposition D. We want to maximize this as the variance of a fuzzy RD estimator will decrease as the effective sample size increases (Coussens and Spiess 2021 Liu et al. 2022
Heng et al. 2020 Power thus increases as effective sample size increases ( (Appendix B.4
Proposition 3. Power is non-decreasing function of MJa regardless of subgroup size G.
1.0
Maximizing effective sample size v is superior objective to maximizing heterogencous TAU 08
alone as it is possible to select a small subgroup G that has a high TAU but will still have lower power than the baseline population sample. We demonstrate this empirically in Figure B] which together with Proposition motivates the of 1JG in our algorithm.
0.5
0.15 0.5
Figure 3: Optimizing for effective sample size (left) increases statistical power regardless of subgroup sample size. We simulate 1,000 random subgroups and show power against the effective sam- ple size no (left) and treatment assignment uptake (Tc) (right), where the shading indicates different sub- group sizes as proportion of the total population.
3.5 A test statistic for effective sample size
When discovering subgroups with higher effec- tive sample size than the baseline population, want to ensure that the differences are not
7

---

Published in Transactions Machine Learning Research (10/2023)
-0075
Point piot
0.4
0.2
0.0
0.05
0.1
0:15
0.
0.35
0.45
0.5
0.7
0.05
0.95
ducorimati
(b) RDSGD
1|X 0.00)
0.25
x|x
2 0.73)
0.88)
0,0001
0.0001
Figure RDs with heterogeneous cutoff's have smaller TAU (a), but RDSGD can correctly identify cutoffs with hetereogeneity (b). a) treatment probabilities across running variables (95% Cls). b) representation RDSGD (Algorithm where causal trees are fit to each candidate threshold which generate subgroups with higher effective sample sizes (step and statistically significant subgroups selected (orange nodes, step 2).
due to noise in the selected samples. We thus test if the effective sample size for a subgroup G greater than that of the whole population P: Ho ya - 1jp = 0, HA la - 1Jp > The corre- sponding test statistic is:
bn GG -
(5)
Varlna - npl
Though Var[ng 7p) can be difficult to derive groups G and P are overlapping, we leverage properties of influence functions (Newey and McFadden 1994 Kahn 2022) to construct a consistent estimator for this variance term (Appendix B.5). As empirical 1 (Equation 4 can be casily calculated using sample means, we can estimate ty. We verify this test statistic behaves correctly asymptotically under the null hypothesis (Figure B.I). We can thus leverage heterogeneity in treatment assignment uptake for improved study power.
4
Methodology
We calculate the elfective sample size presented in Section to implement our R.SGG (Regression Discontinuity SubGroup Discovery) method, which is outlined in Algorithm [ and visualized in Figure RDSGD comprises of two main steps: 1) identification of candidate thresholds and subgroups with higher effective sample size, and 2) subsequent selection of subgroups"]
As we use a data-driven process to select the subgroups to test (Kuchibhotla et al. 2022 Athey and Imbens 2016), to enable valid inference in this two-step procedure we assume the existence of two separate datasets S₁, S2 through sample splitting or holdout set.
Source code and the data needed reproduce all figures available httpp:://ii htpps://githbb.... com//t1l15526/rdaa

---

Published in Transactions Machine Learning Research (10/2023)
4.1 Identifying Discontinuities and Subgroups
To discover regression discontinuities with potential heterogeneity, we first identify candidate thresh- olds. Given set of cutpoints Cx {ci, c2,-.. for a running variable X, RDSGD analyzes thresh- olds E Cx. It first generates threshold indicator Z == 1[X c| and selects bandwidth he (Algorithm 1, step la), which can be chosen by the user or by a data-driven selection process (Cat-
taneo el al. 2019a Imbens and Kalyanaraman 2009).
Algorithm 1: RD SubGroup Discovery (RDSGD)
1. Identify discontinuities and subgroups.
For c E Cx:
(a) Select bandwidth le of analysis and generate threshold indicator Z == 1[x 2 c]
(b) Select baseline population P ((W, Z, Tú, T.) i (c he, c he]} from S and compute effective sample size fyp
(c) Fit subgroup tree model f estimating +(W) (Eq. 3p
(d) Obtain subgroups C.,e [(W;, Z,, Ti, i s} from S and fGa for each node in
(e) Output subgroups with stat. sig. greater effective sample size G. {Ga,e fa, hpp
2. Select subgroups.
(a) For each subgroup definition G.,e E U Ge:
Select data XG = (X, I (j Gs.e)} from S2 ii. Fit local linear estimator T(Xo,e) (Eq. 2 obtain TAU estimate to and p-value pia
(b) Compute corrected significance level &
(c) Output discovered cutoffs and subgroups: Dx = ((c. Ga,e) (Pta < â)
RDSGD then computes the baseline effective sample size ûr (step 1b) from data sample St. Because X is real-valued, it can theoretically yield infinite potential cutpoints. However, in many situations (such as the clinical contexts we consider) the grid of candidate cutpoints for X can be sensibly defined terms of the running variable e-g., in whole years for age-based clinical guidelines or at the precision of lab result readings. Where the candidate cutpoints do not have a sensible definition, other methods such LoRD3 (Herlands et al. 2018) can be used to provide Cx (Section 2
Next, RDSGD generates subgroups for each e basod on the pro-treatment covariates W by estimat- ing (W) from S₁ for the given cutoff and bandwidth (Steps Ic-d). As discussed in Section 3.3 RDSGD uses a causal tree approach to estimate (WW and produces candidate subgroups G.,c for a given cutoff e for RD study evaluation. RDSGD then determines the subgroup effective sample size fia, (Section 3.5 is statistically greater than the baseline fiP (Step le).
4.2 Selecting subgroups
Once we have candidate heterogeneous subgroups, we need to select the most promising subgroups terms of study power while preserving statistical validity by using the separate sample S2 to test. Given a subgroup Gs,e for a cutpoint e, RDSGD evaluates the local linear regression for TAU test for the discontinuous jump indicative of potential RD study (Step 2a). In order for the TAU test to be valid, RDSGD must account for the multiple comparisons across the set Gx of all
9

---

Published in Transactions Machine Learning Research (10/2023)
Power for
Power for C2
(b) RDSGD discovers more powerful sub- groups over baseline higher dimensions. We run 500 trials with 7 = 0.5, recording mean power and comparing with baseline powers.
Power
RDSGD
no het: mAz
00
dim(W)
c power
C2 power
0.5
0.6 streamment arsignmenn uptake
7. trreotment assignment uptake
baseline
0.52
0.48
2
0.79+0.15
0.78++0.1
(a) RDSGD improves the statistical power of discovering RD opportunities by considering heterogeneity. We simulate RDs over 500 trials for each 7 and record the number of correct discoveries at ej and for an empirical power estimate.
4
0.78:10.15
0.77:00.16
8
0.79::0.1
0.78#0.16
16
0.78:10.15
0.78
Figure 5: Synthetic experiments demonstrate the benefit of RDSGD.
the subgroups considered for running variable X, including the candidate subgroups generated in Step 1d. RDSGD thus applies a Bonferroni correction to produce the adjusted significance level ā (Step 2b). Finally, RDSGD outputs discovered subgroups and cutoffs based on á (Step 2c). By leveraging connections between 1) TAU estimation and machine-learned CATE estimation as well as 2) our statistical testing framework for effective sample size (Sections 3.4 3.5 RDSGD is a data-driven RD discovery procedure that uses potential heterogeneity among subgroups to identify more powerful RD studies (Algorithm !
Figure 4
5 Synthetic Experiments
We first validate RDSGD using synthetic data where multiple discontinuities in a given running variable can be distinguished via heterogeneity in other observable covariates. We compare RDSGD to a baseline method that only tests the TAU regression of Equation 2 for each cutpoint e (Algo- rithm A and thus does not consider heterogeneity. We also make comparisons to the LoRD3 method proposed by Herlands et al. (2018). Full simulation details can be found in Appendix
5.1 Heterogeneity in One Covariate
Data Generation. Here we generate data where half the units in our sample follow a fuzzy RD threshold for rumning variable X E [0, 1] at ei 0.25, Z = =1 2 ci). while the other half follow a fuzzy RD threshold c2 0.75, Z 1[x 2 02). The threshold a particular unit follows can be identified by observed covariate W E [0, with units W < 0.50 following threshold c and units W > 0.50 following threshold 02 (Appendix C. Such scenario might arise in real-world settings where the clinical threshold varies depending on other patient attributes; from our running breast cancer example, women with high risk of breast cancer due to hereditary factors (W = 1) should begin screening carlier than the recommended age of 40 for women without risk factors (W = 0) (Center for Disease Control (021)). The TAUs al ei and 02 will appear much smaller if covariate W is not accounted for, thus this synthetic data scenario one where we would expect RDSGD to be useful.
10

---

Published in Transactions Machine Learning Research (10/2023)
Power Calculations. To quantify RDSGD's performance, we calculate the theoretical power that can be achieved for given RD study. Given the regression framework for TAU ostimation (Equation 2), we analytically derive the theoretically achievable power levels (Appendix B.6 We can then use these power calculations and our synthetic data to evaluate the bascline method, LoRD3, and RDSGD. We simulate RD datasets as described above, evaluating empirical power for each ground-truth 7 E 0.3... 0.7], to correctly identify discontinuities at C1 and C2.
Simulated results. Our empirical results show the benefit of RDSGD (Figure 5a We also calculate the theoretical power without considering heterogeneity in W (red dashed lines) and find that the baseline method (Algorithm A.1 blue lines) matches that power level across RDSGD (Algorithm [ orange lines) improves upon the baseline method well as LoRD3 (Appendix C.4 RDSGD maintains empirical false positive rates below the nominal a = 0.05 for all levels duc to the multiple testing corrections (Figure C.3 Empirical power for RDSGD approaches the theoretical power when heterogeneity in W accounted for (dashed green lines). The gap between the power levels of RDSGD and the theoretical power is expected, as we lose power due testing corrections and the data-driven tree fitting, which does not perform an exhaustive search.
5.2 Heterogeneity in Multiple Covariates
The improvement in power over baseline also extends to multidimensional heterogeneity (Table 5b where we increase the dimensionality of the covariates dim(W) E 12,4, 16] that determine whether an individual complies with cutoff C1 or c2 and record the power of the discovered subgroups (Appendix C.5 Though power (as expected) decreases slightly dim(W) incrcases, RDSGD
scales well to higher dimensions, as the average subgroup powers for both cutoffs across all dim W are greater than the baseline theoretical powers. These simulation results (Figure 5a Table 5b) provide empirical evidence that RDSGD can improve RD discovery in the presence of heterogeneity.
6 Case Study: Medical Claims Data
To evaluate RDSGD in real-world settings, we target a variety of clinical contexts where we believe RDs exist: breast cancer screening, colon cancer screening, and diabetes diagnosis. We use Optum's de-identified Clinformatics Data Mart Database (2007-2018), which contains claims data on diag- noses, procedures, prescriptions, and lab results. We use all adults with demographic data, roughly 60 million unique patients over the twelve-ycar period. We note that cach clinical sctting uses a different subset as there are inclusion criteria that are specific to that setting, e.g. the presence of a lab result. Full details on demographics and sample selection be found in Appendix D
6.1 Data Extraction and Featurization
For each clinical setting, we target a specific running variable X that corresponds to treatment T (see the first two columns of T. We use LOINC, CPT, and ICD codes (MeDonald et al. 2003 WHO 2016) to identify lab results, procedures, and diagnoses. To convert longitudinal data in the claims database into tabular form, we index a pationt by the first recorded presence of the running variable (Appendix D.2). We then assume a fixed time window after the running variable is recorded for the treatment occur, in order to account for lags claims data reporting. If the treatment of interest appears for the patient within the window, they are coded "treated," " otherwisc they
11

---

Published in Transactions Machine Learning Research (10/2023)
Table 1: Discovered RD thresholds and subgroups in medical claims data. We report f and / (higher better both cases) for baseline RDs and discovered subgroups.
Clinical guideline
Running Threshold variable
Subgroup
Baseline (SE)
RDSGD (SE)
Baseline RDSGD
discovered
"
"
Breast cancer
Gender
Age
240
0.039 (0.0013)
0.088 (0.003)
471.7
1074.6
screening
Female
Colon cancer screening
Encounter date
Age
250
0.062 (0.002) 0.12 (0.002)
1335.1
1759.0
> 2014-05-07
Type 2 diabetes
Encounter date 0.093 (0.005) 0.12 (0.006) 1212.1 1533.6
A1C %
26.5
diagnosis
> 2010006-07
are "untreated." " For example, in the setting where we wish estimate the "treatment" uptake of diabetes diagnosis, we find patient's first recorded A1C measurement and then search the database for a Type II diabetes diagnosis within the following week. In the screening settings where age is the rumning variable, use patient's age at their first preventative care visit.
We additionally query the claims database for covariates that may impact TAU heterogeneity. For all clinical settings, we consider heterogeneity across patient demographics (age, gender, race), socio- economic status (education level, household income) and claims-specific features (initial encounter date, insurance type). Our data extraction method provides a pipeline for converting raw claims data into feature matrices for RDSGD.
We use the EconML package (Battocchi et al. 2019) to estimate the hetcrogeneous TAU tree model needed for RDSGD (Algorithn)
step 1d). We split our data inlo equally sized samples Sr. S2 for each clinical context. Due to our larger sample size, most candidate RDs that RDSGD returns will have power approaching 1, so here we compare the estimated TAU and effective sample size ny of the discovered subgroups to the baseline RD without considering heterogeneity.
6.2 Results
Breest cencer screening
Colon cancer screening
Diabetes diegnozis
The most promising discovered RD thresh- olds and subgroups for each clinical setting shown in Table D with treatment probability point plots shown in Figure
zutoff ROSGO
01
0.1
5.4
Breast Cancer. RDSGD correctly identifies that breast cancer screening only applies to women age 40 (Oeffinger et al.) 2015). doubling the effective sample size from 471.7 to 1074.6.
38.32
50
40
49 51 52.52255
62636463856768
400
0g0
AIC
Figure 6: RDSGD discovers subgroups that improve TAU in different clinical settings. We show the discovered cutoff and the probability of treatment for the entire sample (blue) and for sub- group discovered by RDSGD (orange).
Colon Cancer. RDSGD correctly identifies the recommended screening age of 50 for colon cancer, and additionally discovered a subgroup of patients who were more likely to be screened at the threshold; these individuals had an en- counter date later than 2014 4-00-77, producing subgroup with higher effective sample size than
12

---

Published in Transactions Machine Learning Research (10/2023)
the baseline population (1335.1 vs. 1759.0). This could be due to increased adherence screen- ing from guideline update that occurred approximately in the same time period (US Preventive Services Task Force 2016).
Type 2 Diabetes. RDSGD identifies the AIC cutoff of 6.5% for diabetes diagnosis (American Diabetes Association 2010 and also identifies a subgroup more likely to be compliant with the cutoff, increasing the effective sample size from 1212.1 to 1533.6. This subgroup excludes patients who have encounters before 2010-06-07, which aligns with clinical practice intuition as A1C was not introduced a diagnostic criteria until 2010.
7 Discussion
Here we have proposed RDSGD, a method for regression discontinuity (RD) discovery that produces interprctable subgroups by optimizing for the effective sample size through a machine learning framework. We demonstrate through synthetic studies how RDSGD provides power improvements in the presence of heterogeneity, We apply RDSGD to a variety of clinical settings, both validating method well discovering new RDs investigate. We now discuss how our method fits into real-world workflows as well as highlight limitations and future work.
7.1 Real-World Workflows
RDSGD is most useful in scenarios for treatment effect estimation when explicit randomization of the treatment not possible. While our method only discovers RD opportunities within the data (the so-called treatment assignment regression, Section 3.1 and does not make treatment effecl estimates, the primary goal is to identify quasi-experimental randomness that can be used estimate the downstream effects of the given treatment T on an outcome of interest Y. For example, in our A1C diabetes case study, practitioner may wish to study the effect of the A1C cutoff on different outcomes, such as metformin prescription rate or heart attack incidence. By identifying both the cutoff as well as subgroups where the effective sample size is stronger, any downstream treatment effect estimation a practitioner wishes to conduct has both: 1) an identified variable that provides quasi-cxperimentall randomness and 2) an interpretable cohort which applics. By performing causal inference opportunity discovery, our method increases research efficiency and scalability by identifying the most promising RD studies to pursue.
We encourage researchers to think about RDSGD as a scientific discovery tool: instead of driving a manual process where practitioners have to generate RD candidates based on domain expertise alone, RDSGD produces candidates from the data which the practitioners can then verify. This not only accelerates research using larger observational data as is much easier to verify potential candidates rather than generate them, but also provides means to increase research efficiency. For researcher, there is opportunity cost in terms of time and effort when analyzing RD, and method can inform which studies they should pursue, well as which studies they potentially should not pursue. For example, Naidech et al. (2200) investigates a seemingly promising RD opportunity in stroke guidelines, but due to sample size and compliance issues had inconclusive results. RDSGD could have complementary utility by providing an carly signal to practitioners on which opportunities may not ultimately be fruitful if their hypothesized culoff does not appear as a subgroup. We note that the "bottom-up" data-driven approach we advocate for may reduce the power of subsequent studies that use the discovered design if the design was known apriori due to
13

---

Published in Transactions Machine Learning Research (10/2023)
the use of sample splitting. However, we believe this tradeoff' is worthwhile in many situations when RD designs are unknown to identify the most promising opportunities for new causal studies.
We believe that our method is well-positioned to discover RDs that depend on a moderate number of covariates this is particularly relevant within the medical domain as clinical practice evolves to become more nuanced. To give an example in existing work, Scott et al. 2021 [6] use cardiovascular risk score cutoffs as an RD design to study the effect of statins on adverse outcomes; they perform manual analogues of our subgroup method by 1) identifying hospitals that have better adherence to the score cutoff guidelines and 2) by excluding patients with very high cardiovascular risk and comorbidities that might interfere with the treatment discontimuity e.g., patients with diabetes are not recommended to be prescribed statins. Furthermore, we show in our case studies (Section G when working with longitudinal datasets it is possible that the candidate RD cutoffs may shift over time due to changes in clinical best. practice, and it is important lo surface such shifts to the practitioner. We note that these subgroup examples we give here are relatively low dimensional. We want to caution that highly complex RDs that are found via our data-driven discovery approach might have limited empirical value, and warrant careful attention before being categorized as valid studies. Nevertheless, RD subgroups that are defined by moderate number of covariates may still be missed if only the rumning variable is considered in isolation from the other pre-treatment variables, and it is these scenarios that RDSGD fits well.
Furthermore, RDSGD could be used to investigate implicit differences in treatment assignment. Because the subgroups produced by RDSGD define clear inclusion criteria based on the path of the fitted causal tree, it can be used to identify sources of bias in treatment decisions such as those documented in FitzGerald and Hurst (2017); Hausmann et al. 2013 Hoffman et al. (2016).
7.2 Limitations and Future Work
We highlight some limitations and opportunities for future work. First, we do not make treatment effect estimates as part of our method and defer that step to practitioners, who are free to choose which outcome Y they wish to study. Care must be taken when moving forward to treatment effect estimation (7. Scction (.1).. When making treatment effect estimates with identified cutoffs we neod to be mindful of the exclusion restriction assumption Imbees and Lemieux (2007), which can be violated when the cutoff decision Z affects the outcome Y outside of its affects T. Critically, validation tests specific to RDs, such whether the running variable has been manipulated and continuity in the covariates, also need to be run to ensure an appropriate causal design (McCrary) 2008 Imbens and Lemieux, 2007). Tt is straightforward to apply tests of RD validity post our method, and they are important verification steps a practitioner needs to conduct after selecting a discovered RD subgroup to evaluate.
Moreover, there are limitations in using medical claims data, as there may be selection biases in healtheare utilization well potential under-reporting of diagnoses and treatments of inter- est (Jensen et al.) 2015). Thus, when moving forward with RD studies identified by RDSGD, practitioners need to work closely with domain experts to ensure that causal validity is maintained.
We also note that bocause RDSGD uses causal basod methods to identify hotcrogencous TAU subgroups, is inherently greedy (see Figure 5a where RDSGD approaches, but does not achieve max power). Though our use of trees was a deliberate design decision made maintain inter- pretability and scalabililty to large datascts (Wu et al.) 2022), future work could investigate other
14

---

Published in Transactions Machine Learning Research (10/2023)
methods that are optimal in terms of 'AA maximization: for example, applying policy learning methods that maximize power in randomized trials to RDs (Spiess and Syrgkanis 2021 Fur- thermore, use sample splitting to ensure valid inference, additional efficiency gains could be achieved in extensions through cross-fitting, conditional inference, or other bias correction meth-
ods (Kuchibhotla et al. 2022 Zhao et al.) 2023).
Though we have focused on clinical use cases in our motivation and case study, we emphasize that RDSGD can be utilized in any settings where treatment assignment heterogeneity in RD thresholds may exist, such as in education |Chay et al., 2005; Goodman 2008), political science (Klašnja and Titiunik] 2017), and labor market programs [Lalive] 2007), among others |CCattaneo et al. 2016). Future work could look to apply RDSGD to these contexts. Our method could also be extended to identify regression kink designs, another related framework popular in cconomics, where there is a kinkpoint rather than a discontinuity at Irealment threshold (Card et al. 2015).
7.3 Conclusion
Here we introduce RDSGD, machine-learning method for RD discovery which identifies inter- pretable subgroups with higher effective sample size, increasing study feasibility. RDSGD is effec- tive on both simulated and real data, and could provide new avenues for more credible obscrvational causal studies across medicine and social science through quasi-experimental designs.
Statement of Broader Impact
Data-driven RD discovery promises to enable more studies that estimate treatment effects from observational data, providing valuable causal estimates when conducting a randomized experiment would be expensive, unethical, or infeasible. However, there some risk in using our method, shared with much of ML, as can leverage and propagate biases in the data. For example, if a demographic indicator predicts non-compliance, whether true or due to e.g. dataset bias, the treatment effect estimates made in RD studies utilizing our method would exclude the non-compliant group. This could disadvantage that group as the study would not apply to them. The discovered subgroups need to be scrutinized for potentially perpetuating bias, and one advantage of our method is the transparency of to whom the discovered thresholds apply to. Healtheare data are of particular interest, which we utilize as examples throughout this work, and disparities against minority groups such data are common MMurthy et al. 2004). Data could also include selection biases (such as demographic variation in healthcare utilization resulting differences in collected claims data), and we must be mindful of these sources of bias when interpreting results. To mitigate these potential harms, practitioners should carefully consider the source of their data and whenever possible verify their results with a different dataset e.g., claims data from a different healthcare system. When using data-driven methods like RDSGD for causal analysis from observational data, stakeholders must weigh the strength of evidence supporting the causal claim as well as the result's external validity before making decisions based off of the study.
Acknowledgments
We thank Morgan McGuire for valuable discussion and feedback. We thank the anonymous review- ers whose detailed feedback have improved this work.
15

---

Published in Transactions Machine Learning Research (10/2023)
References
American Diabetes Association. Standards of medical care in diabetes 2010. Diabetes care, 33 (Supplement 1):S11-S66 2010.
Joshua D Augrist and Jörn-Stellen Pischke. The credibility revolution in empirical economics: How better research design is taking the con out of econometrics. Journal of economic perspectives, 24(2):3 30, 2010.
Joshua D Angrist and Jörn-Steffen Pischke. Companion. 2008.
Mostly Harmless Econometrics: An Empiricist's
Peter M. Aronow and Allison Carnegie. Beyond LATE:
Estimation of the Average Treatment Effect with an Instrumental Variable. Political Analysis, 21(4):492 506, 2013. ISSN 1047-1987, 1476-4989. doi:
Susan Athey and Guido Imbens. Recursive partitioning for hetcrogencous causal effects. Procoodings of the National Academy of Sciences, 113(27):7353 -7653-7300, July 2016. ISSN 0027-8424, 1091-6490. doi: 10.1073 (.107//pna..ii 15144891.
Susan Athey, Julie Tibshirani, and Stefan Wager. Generalized random forests. The Annals of Statistics, (7(2):1148-1177 April 2019. ISSN 0090-5364. doi: 10.1214/18-A0S1700.
Michael Baiocchi, Jing Cheng, and Dylan S. Small. Instrumental variable met for causal inference. Statistics in Medicine, 33(1))22297 2340, 2014. ISSN 1097-0258. doi: https://
URL htttp:://omlnnellbrryy
6128
eprint:
Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, and Vasilis Syrgkanis. EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estima- tion. https///github.com, /otoo//mcoooo///EEoeoo /FconMI, 2019. Version 0.x.
Richard A Berk and David Rauma. Capitalizing on nonrandom assignment to treatments: A regression-disconiiniig evaluation of crime-control program. Journal of the American Statistical Association, 78(381):21 27, 1983.
David Card, David Lee, Zhuan Pei, and Andrea Weber. Regression Kink Design: Theory and Practice. 2015.
Matias D Cattaneo, Rocio Titiuník, Gonzalo Vazquez-Bare, and Luke Keele. Interpreting regression discontinuity designs with multiple cutoffs. The Journal of Politics, 78(4):1229-1228, 2016.
Matias D. Cattanco, Nicolas Idrobo, and Rocio Titiunik. Practical Introduction to Regression
Discontinuity Designs: Foundations. arXiv:1911.09511 lecon, stat], November 2019a. doi: 10. 1017/9781108688466,
Matias D Cattanco, Rocio Titiunik, and Gonzalo Vazquez-Ban Bare. Power calculations for regression- discontinuity designs. The Stata Journal, 19(1):210 20-, 2019b.
Matias D. Cattaneo, Luke Keele, and Rocio Titiunik. Covariate Adjustment in Regression Dis- continuity Designs, August 2022. URL http:://axxiv.org/bb/////2 08410 arXiv:2110.084100 [econ, stat].
16

---

Published in Transactions Machine Learning Research (10/2023)
Center for Disease Control. Strategies for Managing Risk: Preventive Screen- ing Recommendations for Women Increased Risk for Heroditary Breast and Ovarian Cancer CDC.
http://wwww...deeeo///////e///s/ women/
bringyourbrave/heee hheaaeeeeeeehhhheeeeeee care ppwoider_eduuaalleeeee cateec/po.lllkkeeeseottttnnnnonn management htm,, October 2021.
Kenneth Chay Patrick J McEwan, and Miguel Urquiola. The central role of noise in evaluating interventions that use Lest scores to rank schools. American Economic Review, 95(4):1237 1258, 2005.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double (obbased machine learning for treatment and structural parameters. The Econometrics Journal, (1(1):C1-OC February 2018. ISSN 1368-4221. doi:
10.1111/eetj.120977
Stephen Coussens and Jann Spicss. Improving inference from simple instruments through compli- ance estimation. arXiv preprint arXiv:2108.03726, 2021.
Timothy Erickson and Toni M. Whited. Two-Step
GMM Estimation of the Errors- in-Variables Model Using High-Order Moments. Rconometric Theory, 18(3):776- doi: 10.0017/52066666021133100
799,
June 2002. ISSN 1469-4360, 0266-4666.
URL
FFCEECCCSSSPAPAA44TBB7AAAAAAAAA Publisher: Cambridge University Press.
Chloë FitzGerald and Samia Hurst. Implicit bias in healthcare professionals: a systematic review.
BMC medical ethics, 18(1):19, 2017.
Guy Freeman, Benjamin Cowling, and C Mary Schooling. Power and sample size calculations for Mendelian randomization studies using genetic instrument. International Journal of Epidemiology, 42(4):1157 1163, August 2013. ISSN 0300-5771. doi: 1e1109//je/dyytli URL https://doi. org/ wtrg/10.11 1tt/110...11///1 100.10/3//je/yy1100
Joshua Goodman. Who merits financial aid?: Massachusetts' adams scholarship. Journal of public Economics, 92(10-11):2121 2121-2111, 2008.
Jinyong Hahn, Petra Todd, and Wilbert Van der Klaauw. Identification and Estimation of Treat- ment Effects with
Design. Econometrica, (9(:)2201 209, 2001. ISSN
0012-9682.
Leslie RM Hausmann, Shasha Gao, Edward Lee, and C Kent Kwoh. Racial disparities in the monitoring of patients on chronic opioid therapy. PAIN@, 154(1):46-52, 2013.
Siyu Heng, Bo Zhang, Xu Han, Scott A. Lorch, and Dylan S. Small. Instrumental Variables: To Strengthen or Not to Strengthen? arXiv:1911.09177 [stat], August 2020. URL http:///rxiv. org/abs/19k 09171. arXiv: 1911.09171.
William Herlands, Edward McFowland I1I, Andrew Gordon Wilson, and Daniel B. Neill. Automated Local Regression Discontinuity Design Discovery. In Proceedings of the 24th
17

---

Published in Transactions Machine Learning Research (10/2023)
ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1512 1520, London United Kingdom, July 2018. ACM. ISBN 978-1-4503-6552-0 doi: 10.1145/3219819. 3219982.
Miguel A Hernán and James M Robins. Causal Inference: What If. Boca Raton: Chapman & Hal//CRC, 2020.
Kelly M Hoffman, Sophie Trawalter, Jordan R Axt, and M Norman Oliver. Racial bias in pain
assessment and treatment recommendations, and false beliefs about biological differences between blacks and whites. Proceedings of the National Academy of Sciences, 113(16):4296-44)) 4301, 2016.
Guido Imbens. Instrumental variables: an econometrician's perspective. Technical report, National Bureau of Economic Research, 2014.
Guido Imbens and Karthik Kalyanaraman. Optimal Bandwidth Choice for the Regression Dis- contimuity Estimator. Working Paper 14726, National Bureau of Economic Research, February 2009.
Guido Imbens and Thomas Lemicux. Regression Discontinuity Designs: A Guide to Practice. Working Paper 13039, National Bureau of Economic Research, April 2007.
Guido Imbens and Donald Rubin. Causal inference: for statistics, social and biomedical sciences : an introduction. 2015. ISBN 978-1-139-02575-1 OCLC: 985493948..
Elizabeth T. Jensen, Suzanne F. Cook, Jeffery K. Allen, John Logie, Maurice Alan Brookhart, Michael D. Kappelman, and Evan S. Dellon. Enrollment factors and bias of discase prevalence estimates in administrative claims data. Annals of Epidemiology, 25(7):519 525.e2, July 2015. http://aaameidde.222.55....
ISSN 1047-2797. doi:
Srikanth Kadiyala and Erin Strumpf. How Effective is Population-Based Cancer Screening? Re- gression Discontinuity Estimates from the US Guideline Screening Initiation Ages. Forum for Health Economics & Policy, 19(1))87-139,, June 2016. ISSN 1558-9544, 2194-6191. 20015///hee2214-00144. doi: 10151///ee---a 2014-0014. URL https: http::/wwwwwwwo www.wwwwwwuutter... hwt.//rie//fourraalla///////////
article-p87 xml Publisher: De Gruyler Section: Forum for Health Economics Policy.
Jay Kahn. Influence Functions for Fun and Profit. page 7, 2022.
Edward H. Kennedy, Sivaraman Balakrishnan, and Max G'Sell. Sharp instruments for classifying compliers and generalizing causal effects. The Annals of Statistics, 48(4), August 2020. ISSN 0090-5364. doi:
Marko Klasnja and Rocío Titiunik. The incumbency curse: Weak parties, term limits, and unful filled accountability. American Political Science Review, 111(1))129-1444) 2017.
Arun K. Kuchibhotla, John E. Kolassa, and Todd A. Kuffner. Post-Selection In- forence. Annual Review of Statistics and Its Application, 9(1):505-527, 2022. doi:
10. 10014//// 1111///amuueeettaaaaiee----11110 annurev annurev-statistee wtw.:.stttiiii---01000000333 100421-044639.
100421-044639.
URL https://doiiorr/100 eprint: attps://doi.on 10.1111..111111 sat.i/////mmmrresstttttttisss
18

---

Published in Transactions Machine Learning Research (10/2023)
Rafael Lalive. Unemployment benefits, unemployment duration, and post-unemployment jobs: A
regression discontinuity approach. American Economic Review, 97(2):108-112, 2007.
Edward E. Leamer. Let's Take the Con Out of Econometrics. The American Economic Review, 73
(1):31-43, 1983. ISSN 0002-8282.
David S Lee and Thomas Lemieux. Regression Discontimuity Designs in Economics. Working Paper 14723, National Bureau of Economic Research, February 2009.
Nla Lipkovich, Alex Dmitrienko, and Ralph B. Agostino Sr. Tutorial biostatistics: data-driven subgroup identification and analysis in clinical trials. Statistics in Medicine, 36(1):136 196, 2017. ISSN 1097-0258. doi: 10.1002/sim.
URL htttps://omllneelbrraayyy www.....ww....c................ http://wwww.lle..oo//////a/////
10. 20.1002/sim.700 7064
eprint:
Tony Liu, Lyle Ungar, and Konrad Kording. Quantifying causality in data science with quasi- experiments. Nature Computational Science, (1):24432, January 2021. ISSN 2662-8457. doi: 10.1038/s43555 0005588802000005---
Tony Liu, Patrick Lawlor, Lyle Ungar, and Konrad Kording, Data-driven exclusion criteria for instrumental variable studics. In Procoedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pages 485 508. PMLR, 11- 11-13/ Apr 2022. URL https://prroceedings. swwiiiinnnsmmmrrr.....mm mlr pttpa/eree///////..a.rrtttt
Ioana E. Marinescu, Patrick N. Lawlor, and Konrad P. Kording. exterieaperiitttall causality in neuroscience and behavioural research. Nature Human Behaviour, 2(12):891 898, December 2018. ISSN 2397-3374. doi: 10. N0.108/s41162-014
Justin McCrary. Manipulation of the running variable in the regression discontinuity design: A densily lest. Journal of Econometrics, 142(2):698 714, February 2008. ISSN 0304-4076. doi: 10.1016/1jecommm.2200 jeconom.2007.05.000
Clement J McDonald, Stanley M Huff, Jeffrey G Suico, Gilbert Hill, Dennis Leavelle, Raymond Aller, Arden Forrey, Kathy Mercer, Georges DeMoor, John Hook, Warren Williams, James Case, Pat Maloney, and for the Laboratory LOINC Developers. LOINC, Universal Standard for Identifying Laboratory Observations: A 5- Y-Year Update. Clinical Chemistry, 49(4):624466) 633, April 2003. ISSN 0009-9147. doi: 10.1373/49.4.624
Edward McFowland III, Sriram Somanchi, and Daniel B. Neill. Efficient Discovery of Het- erogencous Treatment Effects in Randomized Experiments via Anomalous Pattern Detection. arXiv:1803.09159 Jecon, stal), June 2018.
David McKenzie. Power Calculations for Regression Discontinuity Evaluations: Part htwp::/wwww....ooooooooooooooooa evaluations 2016.
Ellen Moscoe, Jacob Bor, and Till Bärnighausen. Regression discontinuity designs are under- utilized in medicine, epidemiology, and public hcalth: a review of current and best prac- lice. Journal of Clinical Epidemiology, 68(2):132 143, February 2015. ISSN 0895-4356. doi:
11.1016/.jclmmep22111....... URL http: /www://wwwww.../ http://wo..oooo/////////o///ooo. http://wwwaacccccccccccc////aaaa 40895435614003990
19

---

Published in Transactions Machine Learning Research (10/2023)
Vivek H. Murthy, Harlan M. Krumholz, and Cary P. Gross. Participation in Cancer Clinical Trials: Race-, Sex-, and Age-Based Disparities. JAMA, 291(22):2720, June 2004. ISSN 0098-7484. doi: 10. 10.1001 10.000//am.29111. 291.a....1........ 2720. URL http:////pooana.......ooooooo/om
j ama.291.2 2720
Rashelle J. Musci and Elizabeth Stuart. Ensuring causal, not casual, inference. Prevention science the official journal of the Society for Prevention Rescarch, 20(3):452-44) 444-246,, April 2019. ISSN 1389-4986. doi: 10.
Andrew M. Naidech, Patrick N. Lawlor, Haolin Xu, Gregg C. Fonarow, Ying Xian, Eric E. Smith, Lee Schwamm, Roland Matsouaka, Shyam Prabhakaran, Ioana Marinescu, and Konrad P. Ko- rding. Probing the Effective Treatment Thresholds for Alteplase Acute Ischemic Stroke With Regression Discontinuity Designs. Frontiers in Neurology, 11, 2020. ISSN 1664-2295.
Whitney K. Newey and Daniel McFadden. Large sample estimation and hypothesis testing. In Handbook of Econometrics, volume 4, pages 2111-2245 Elsevier, Jamuary 1994. doi: 10.0011//157334412205880003 URL p11/8157341205500055
Kevin C. Oeffinger, Elizabeth T. H. Fontham, Ruth Etzioni, Abbe Herzig, James S. Michaelson, Ya-Chen Tina Shih, Louise C. Walter, Timothy R. Church, Christopher R. Flowers, Samuel J. LaMonte, Andrew M. D. Wolf, Carol DeSantis, Joannie Lortet-Tieulent, Kimberly Andrews, Deana Latasssrra--appiiste Debbie Saslow, Robert A. Smith, Otis W. Brawley, and Richard Wender. Breast Cancer Screening for Women at Average Risk: 2015 Guideline Update From the American Cancer Society. JAMA, 314(15):1599 1614, October 2015. ISSN 0098-7484. doi: 10.1001/jama.2011 12783.
William Ogallo, Girmaw Abebe Tadesse, Skyler Speakman, and Aisha Walcott-Bryant. Delection Anomalous Patterns Associated with the Impact of Medications on 30-Day Hospital Readmission Rates in Diabetes Care. AMIA Summits Translational Science Procoodings, 2021:495-504, May 2021. ISSN 2153-4063.
Miruna Oprescu, Vasilis Syrgkanis, and Zhiwei Steven Wu. Orthogonal Random Forest. for Causal Inference. In Proceedings of the 36th International Conference Machine Learning, pages 4932 4941. PMLR, May 2019.
Oscar Hernan Madrid Padilla, Peng Ding, Yanzhen Chen, and Gabriel Ruiz. A causal fused lasso for interpretable heterogeneous treatment effects estimation. arXiv:2110.00901 [stat]. October 2021.
Irene Petersen, Sia Kromann Nicolaisen, Federico Ricciardi, Manuj Sharma, Reimar W Thomsen, Gianluca Baio, and Lars Pedersen. Impact being eligible for type 2 diabetes treatment on all-cause mortality and cardiovascular events: regression discontinuity design study. Clinical Epidemiology, 12:569, 2020.
Jack Porter and Ping Yu. Regression discontinuity designs with unknown discontinuity points: Test- ing and estimation. Journal of Econometrics, 189(1):132 147, November 2015. ISSN 03044076. doi: 10.1016/3 n..101666..ccooon 2015.06.002.
20

---

Published in Transactions Machine Learning Research (10/2023)
Agoston Reguly, Heterogeneous Treatment Effects in Regression Discontinuity Designs, October 2021. URL http://aarxv.orr//aa//2000 11640 1rXiv:2106.11640 [econ, stat].
Lauren Scott, Maria Thoresa Redanicl, Matthew Booker, Rupert Payne, and Kate Tilling. Regres- sion discontinuily analysis for pharmacovigllance: statin example reflected trial findings showing little evidence of harm. Journal of clinical epidemiology. 141:121-13) 2022.
Christopher Skovron and Rocio Titinii. A Practical Guide to Regression Discontinuity Designs in Political Science. page 47, 2017.
Jann Spiess and Vasilis Syrgkanis. Evidence-Based Based Policy Learning. arXiv:2103.07066 [cs, econ, stat], March 2021.
US Preventive Services Task Force. Screening for Colorectal Cancer: US Preventive Services Task Force Recommendation Statement. JAMA, 315(23): :2564 2575, June 2016. ISSN 0098-7484. doi: 10.1001/amm..016.500
Jeffrey C. Valentine, Spyros Konstantopoulos, and Sara Goldrick-Rab. What Happens to Students Placed Into Developmental Education? A Meta-Analysis Regression Discontinuity Studies. Review of Educational Research, 87(4):806 833, August 2017. ISSN 0034-6543. doi: 10.3102/ Publisher: American
0034654331709237.. URL https::/doi....oo org/ 10. 010.001/0//0333447779993337 Educational Research Association.
WHO. ICD-10.
2016.
Han Wu, Sarah Tan, Weiwei Li, Mia Garrard, Adam Obeng, Drew Dimmery, Shaun Singh, Hanson Wang, Daniel Jiang, and Eytan Bakshy. Interpretable Personalized Experimenta- tion. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, pages 4173 4183, New York, NY, USA, August 2022. Association for Computing Machinery. ISBN 1--1-----935----1 https://doi.ont org/ 10. 1144111343337/ 35391755
doi: 10.1145/3534678.55
URL
Beibo Zhao, Anastasia Ivanova, and Jason Fine. Inference on subgroups identified based on a
heterogeneous treatment effect in a post hoc analysis of a clinical trial. Clinical Trials (London,
England), 20( 3700-379, August 2023. ISSN 1740-7745. doi: 10.1177/1740774555 173055. URL
https:// htt::://wwww.. htwp/wwww...i....ocmmm nlm. www..nih..oo.o. htt.://///caarrrrlcele//////0000
21

---

Published in Transactions Machine Learning Research (10/2023)
A Baseline algorithm details
We give the full baseline RD discovery procedure used for comparison with RDSGD in Algo- rithm A.1
Algorithm A.1: Baseline method: RD threshold discovery
1. For E Cx:
(a) Select bandwidth he for treatment regression (Section 3.1
(b) Sclect data = {X; X; E [c - he, c + h2]}
(c) Fit estimator f(X, c) of Equation 2 to obtain TAU estimate f. and output p-value Pt.
2. Compute corrected corrected significance level à = a/|Cxl
3. Output discovered cutoffs and bandwidths: Dx [[(,(( pt. < <ā}
B Mathematical details
B.1 FRD and IV assumptions
We assume the following standard FRD assumptions for valid estimation (Lee and Lemieux to identify Equation
2009
Continuity. Both the potential outcomes ((1) and (() are continuous as a function of the running variable X:
EX1(I)X)) [[V continuous over domain of X
(6)
Monotonicity. X crossing the cutoff cannot simultaneously cause some units to take up and others to reject the treatment (also known the "no defiers" assumption):
T(1) T(0)
Excludability of crossing the threshold. X crossing the cutoff cannot impact except through impacting the treatment:
=t==
(8)
Where YT'= is the potential outcome to be observed T = t and 2
Given the equivalence between FRDs and IVss (Imbens 2014), two-stage least squares (TSLS)
estimation uses analogous assumptions:
Consistency. If and T t, then the observed outcomes of T and the potential outcomes under Z and =tt.
22

---

Published in Transactions Machine Learning Research (10/2023)
T T= + 21-2))
(9)
Y +
(10)
Unconfounded instrument. The instrument is unconfounded with the potential treat- ment given the observed covariates.
Z 1 ((1), W
(11)
Monotonicity. The instrument cannot simultaneously cause some units to take up and others to reject the treatment.
T(1) > T(O)
(12)
Exclusion restriction. The instrument cannot impact Y except through the treatment:
Y(T = t. ==
(13)
B.2 Proposition 1 details: compliance and TAU equivalence
Imbens and Lemieux (0077) : Hahn et al. 2001 show how a fuzzy regression discontinuity estimated via local linear regression for a given cutoff c and fixed bandwidth is mumerically equivalent to a two-stage least squares (TSLS) estimation problem with the following additional regressors:
1[x; 2 c(x; =
1[X; < c(x,
(14)
Note that the instrument Z defined Z, 1[2 c), the same our RD cutoff indicator. These regressors thus give the form of the treatment regression in Equation
Imbens and Rubin (2055) further show that under IV assumptions of consistency, unconfounded instrument and monotonicity (Appendix B.1, the treatment effect estimate can be expressed as:
(1) Y,00) Junit is complier)
ccomply
TIV
(15)
Teomply
Where P(T(1)= > T(O)) = Teomply the probability of compliance. Within the data bandwidth of analysis h, we can equivalently write of Equation I in terms of Z Hahn et al. 2001):
T == F 7 = E = 0]
= P(T = 1 = 1 P(T = =)
P(T 0 = 1) P(T = 1|2 =)
nnever taker Talways taker; (unconfounded instrument)
Teomply (monotonicity)
(16)
23

---

Published in Transactions Machine Learning Research (10/2023)
Where always taker and never taker are the proportions of always-takers and never-takers. We then have that is equivalent to the probability of compliance, and that the first stage (denominator of DI from Equation 15 regression of the TSLS framework estimates the probability of compliance. Thus we can use the TSLS framework for our analysis and estimation of RD TAU.
B.3 Proposition
2 details: conditional compliance identification
Identification of the conditional probability of compliance follows similar argument as Ap- pendix B.2 Given the cutoff choice c generating Z 1[X 2 c) and fixed bandwidth h, we can use the equivalent analysis of Z as an IV like we do in Appendix B.2 The conditional proba- bility of compliance is given by: P ((T(1>>((0))
From monotonicity, there are no defiers (units where T(1 < T(0)). We can thus write:
P(T(1) T)
=1- (T = -
- = 0 0|
(17)
The latter two terms are the probability of always-takers and never-takers given covariates W respectively. From the unconfounded instrument assumption, these quantitics can be identified (converted from causal quantities to estimable statistical quantities) (Imbens and Rubin 2015):
P(T(1) = 1, (()) = 1|W) P(T== 1/2 = 0,W)
P(T(I) T(0) 0 P(T 0|2 1,
This then gives:
P(T(1) T(O) = - - - - 0z 1,W)
1 - 1(T 1 = -, - -1- 1| = 1, W))
P(T = 1 1)
1/2 0, 0,
(18)
Allowing to identify the conditional probability of compliance, and thus the conditional TAU To(W) as desired.
Equivalence of CATE and heterogeneity TAU. Next, we make a connection between the conditional TAU and conditional average treatment estimation (CATE).
The CATE of treatment T outcome Y is given by:
CATE = 1
= -ET
(19)
Given the two-stage process of RD treatment effect estimation, we not only have potential outcomes Y-)) but also potential treatments T(.). We can thus equivalently analyze the "treatment effect"
the cutoff indicator Z has "outcome" T, yielding Equation 3 The same standard fuzzy RD
24

---

Published in Transactions Machine Learning Research (10/2023)
assumptions that enable estimation of the treatment effect y cutoff c (Section 3.1 estimation of the heterogencous TAU TWW through CATE estimation frameworks
also enable
(Kenedy et al.
2020
Coussens and Spiess 2021
B.4 Proposition 3 details: power and effective sample size
We describe the relationship between 7 and power. From Cattaneo et al. 2019b), have the following power function for an a-level two-sided test for given a local linear treatment regression:
B(r) 1 + ( 1 VV 2a/2) 1 (
(VV + 2a/2
(20)
where & the Normal distribution CDF, 2, is its tth percentile (e.g. 2, (-)a))), and V the variance of T. Next, leveraging the equivalence between instrumental variable (IV) analysis and fuzzy regression discontintuities (FRD) established in Proposition 1 the variance of an FRD
estimator under constant treatment effects has been shown to be (Coussens and Spiess 2021
V Var[Y| compliers) E (1-E
(21)
Thus, variance decreases as increases. We note that even under the relaxation of the constant treatment effects assumption, Heng et al. (2020) and Baiocchi et al. (2014) have shown that the IV variance with n samples is al least as large as the variance with samples of known compliers. Thus, the two-sided power function is non-decreasing as V decreases, and hence when 1) increases. Freeman et al. (2013) also cquivalently show this relationship between IV power and n in their power calculation analysis of Mendelian randomization studies.
Note that our statements bounded (3(r) [0, 1J). We show empirically in Figure 3 that power in practice increases as f increases, regardless of the size of the subgroup. Data simulated in Figure B.1 follow the TAU regression data generation described in Appendix C.1 with n = 1000, t = 0.2, and a bandwidth of 0.5. Random subgroups of sizes uniformly distributed between 450 and 950 are drawn to show the relationship between To and Tr and power across varying G sizes.
of the power non-decreasing as function of 7 because power is
B.5 Effective sample size test statistic derivation
From Section 3.5 in order to construct test statistic for Equation of VarlnG
5 we need consistent estimator
np]
Varlno m/p Var[nc] Var[yr] @Cov[na,
(22)
We use influence functions empirically estimate this variance term (Newey and McFadden 1994..
To derive the influence function for 'J., first give the influence function of sample which can be as difference-in-means estimator (Imbens and Rubin 2015
TG
Kahn 2022
NG (T, T1a) - 1[2 0] NiG
No NoG
Via. 1/2, 1]
(T. -Tu))
(23)
25

---

Published in Transactions Machine Learning Research (10/2023)
Where 1[: is the indicator function and N& the number of samples in subgroup G where We can then apply the influence function chain rule (Kahn 2022) to obtain the influence function for ng. For an estimator & such that 0 T(0j..... ..., 8n), the influence function of 0 is:
Va,- = Σ
07
Ways
(24)
00,
The influence function thas yi for a given sample subgroup is thus:
8 Noto
nna
Oic
2Nofourra
NG (T; Tic) 1[2; = 0] NG
= =2Nofa
NiG ||| -
(T; Tuc)
(25)
Nog
Note that in following text we refer to tha as &G,i to reduce notational clutter.
Va.; consists of products and differences of empirical means over different subgroups which are straightforward and fast to compute. Following the properties of influence functions (Newey and McFadden]] 1994] Erickson and Whited 2002), we can next derive the variance-covariance matrix of TJ,, NJ follows. Let:
V
Tr. 15
(26)
Npx2
where tG is vector of length Np with values the ith index of va., if G and otherwise. As the distribution of an estimator is equivalent to IN Ei Va, (Erickson and Whited 2002), we have:
Np
hp 1P
1
Σ
No
4 N
(0) E
Vap.eVnoss ))
(27)
fiG GG
VNp
nas
Kahn 2022 then shows that the following variance-covariance matrix gives us the variances of Gas UP on the diagonal and the covariance of "JG and Tpp on the off-diagonal:
V
( NZ
(28)
The elements of V give us empirical, consistent estimates of Var[na], Var[yn), and Cov(nG mP], allowing us to calculate t, (Equation 5)
As a sanity chock, we verify that the distribution of the computed test statistic under the null hypothesis behaves correctly in empirical simulations (Figure B.1 Data simulated in Figure B. follow the TAU regression data generation described in Appendix C.1 with 200, 0.5, and bandwidth of 0.5. Random overlapping subgroups of size 100 are drawn to test the null distribution.
26

---

Published in Transactions Machine Learning Research (10/2023)
n=300 trials
n=1000 trials
n-200D trials
4C0000 trials
Nomallo.11
0.3
Density
0.0
Istat
Figure B.1: Our overlapping 7) hypothesis test produces well-behaved mull distributions, yielding valid p-values. We conduct six different simulations with varying numbers of trials under the null hypothesis that the overlapping groups have the same effective sample size.
B.6 Closed-form power calculations
We use Equation 20 shown in Appendix B.6 to compute the theoretical power of treatment gression. To calculate V, Imbens and Lemicux (2007) give closed form solution for the asymptotic variance of the treatment regression, assuming symmetric bandwidth h:
8 Pbw (o oT
(29)
n
where n is the total sample size, 07.1 is the TAU variance below the cutoff, o7,,, is the TAU variance above the cutoff, and Pow is the fraction of units in the sample in the bandwidth h.
Since T is binary, have that:
7 lim Var(T)X x) PT. pT,))
ztc
TT
lim Var(T]X x) UT (1 UT,,))
zic
where μT, limate Pr(T 1|x æ) and MT,, lim Pr(T = 1|X x). MT, and μT, be computed given our simulated data generating process, giving us closed form solution of the theoretical power that can be achieved in our synthetic experiments.
Synthetic experiment details
C.1 TAU regression setup
For our blended RD simulation scenario, data are generated in the following manner. Let X; 1ni be the running variable for unit i, We generate threshold indicator Z, 1|X; > c], where e is the chosen treatment threshold. Each unit's probability of treatment assignment is defined as:
p= TZ, vX; + 11 + th
(30)
where is the true TAU, the coefficient determining the running variable's effect on the outcome, 77 constant, and th Gaussian noise term. For cach generated data set, vary and draw
27

---

Published in Transactions Machine Learning Research (10/2023)
Scatter (35%
Density treated
1.0
1.-
06-
0.6
64
0.4
xed.25 cutoff
0.0
02
03
06
65
06
077
08
0.
0.0 0.
0.0
0.6 0.0 1.0
Figure C.2: Heterogeneity in TAU can be observed when considering the additional covariate W. Left is a pointplot of TAU probabilities as a function of the rumming variable X (the same as pictured in Figure 4 for a single trial our simulation, with error bars as 95% CIs and a nominal 0.7. Right is a density plot of treated units as a function of both and W.
N (0,0.1) and set 7 0.2, 0.05. pi values are clamped to [0, 1], with an individual's treatment assignment then defined as T; Bern(p;).
C.2 Blended RD in one covariate
For the simulation setting shown in Section for each unit we additionally draw W; Unif(0, 1), with the cutoff e unit complies with being determined by:
0.25 if W, 0.5 0.75 W, 2 0.5
(31)
For each trial of our synthetic experiment, we draw 1000 units for each sample S1, S2 with half of the units complying with the lower cutoff and half complying with the upper cutoff. Differences in TAU can be observed when visualizing heterogeneity in W; (Figure C.2
C.3 Treatment assignment uptake reduction in the presence of heterogeneity
As our simulated data contains two equally-sized populations that comply with different cutoffs, the observed 7 will effectively be reduced by half, since for each cutoff C1, c2, half of the units do not comply with the jump in TAU at that point. For example, at given and n, estimate the observed TAU ci for our sample by computing pr,- Pr.
PT.
lim Pr(T 1|X = =x)
sicr
v. n n n (T 2) =C + n 2
PT. = lim Pr(T = 1|X = x)
stei
V. + n.
VC1
Thus we have Pru- PT, T.
28

---

Published in Transactions Machine Learning Research (10/2023)
C.4
Heterogeneity in one covariate: simulation details
Results presented in Section 5. data gen- crated according to Appendix C. C.2
We use fixed bandwidth h = 0.25 to ensure that the closed form theoretical powers calculated in Appendix B.6 are valid. Causal forests were fit according to default paramcters specified in the EconMT package (Battocchi et. al.) 2019) (with honesty enabled for valid and unbiased inference), and fixed depth of 3 and mini- mum leaf size of 100 were used for subsequent CATE causal trees distilled from the forests to ensure subgroups remained interpretable. We note that the causal forest. implementation in EconML by default runs two-fold cross val- idation internally when sclecting hyperparam- eters for the LogisticRegrressinnNY scikit-learn models for treatment, which searches over L2 regularization parameters in a grid of 10 values between 1e 4 4 and le4 using the default accuracy criterion. Seeds were passed to machine learning models to ensure reproducibility. All 500 trials were seeded with their trial number, and once imple- mentation was complete experiments were run twice to validate reproducibility. We run Herlands et al. (2018)ss 's RDD discovery method, LoRD3, according to recommended parameters in their code repository, setting (in their notation) &o 100 and {X, W) so that information from both X and W are used. All simulations were run on Ubuntu 20.04 LTS server, with 24-core Intel 19-7920X CPU and 94 GB RAM.
False Positive Rates (FPR) running variable
0.10
nominal false positive rate, a 0.05 empincal FPR, RDSGD
0.08
0.06
0.04
0.02
0.00
0.2
0.3
0.4
0.5
0.6
0.7
T treatment assignment uptake jump
Figure C.3: RDSGD maintains false positive rates below the nominal level. We simulate RDs with multiple running variables over 500 trials for each r and record the number of false discoveries.
The false positive rates shown in Figure C.3 are computed based on the number of significant discontinuities discovered that do not equal 61 or 02 divided by the total number of tests over the 500 trials for each level. As our empirical power metric amounts to count of "successful" detections of statistically significant discontinuities at с and C2 out of the 500 trials, use x² test (or corresponding Fisher's exact test if count numbers are not sufficient) to compare RDSGD's performance with the bascline discovery method. All the differences between RDSGD and the baseline algorithm are statistically significant with < 0.001 With the exception of the = 0.2 case, and the differences between LoRD3 and RDSGD are statistically significant at 7 levels.
C.5 Heterogeneity in multiple covariates: simulation details
Results presented in Section 5.2 used data generated according to Appendix C.1 Instead of having a single covariate govern the choice of c as in Appendix C.2 we generate (im(W) covariates W1, W for each unit. using scikit-learn's make regression() method, producing output w;. We then scale wi fall in the range [0, 1] with sample mean 0.5, and determine which cutoff a unit complies with by:
29

---

Published in Transactions Machine Learning Research (10/2023)
0.25 if w/ < 0.5 0.75 if wi 2 0.5
(32)
We fix ground truth 0.5 for the simulation trials and calculate the baseline oracle power according to Appendix B.6 All hyperparameters, hardware, and seeding strategy for the 500 trials are the same as described in Appendix C.4
All statistically siguificant subgroups discovered by RDSGD are recorded, and their power is com- puted according to Equation 20] Means and standard deviations reported in Table 5 are taken across all trials. For significance testing, we use a one-sample t-test comparing the subgroups in each cell of Table 5b with their corresponding baseline oracle powers. All tests were statistically significant al p < 0.001.
D Clinical setting and cohort details
D.1 Justifying use of private claims dataset
In order to evaluate RDSGD in a real-world setting where our clinical collaborators can help verify discovered candidate RD studics, we needed a largo-scale clinical data source that spanned general healthcare settings with enough data granularity on individuals our method can leverage potential TAU heterogeneity across common demographic covariates (described in Appendix D.2). The claims dataset that we use has the advantage of having an array of discase classes and visibility into patient information in general healtheare settings, as opposed to publicly available datasets such as MIMIC which focuses on a very specific context, (critical care). Working with such detailed patient information necessitates adherence to federal HIPAA privacy rules concerning privacy, which restrict access "protectable health information': ; we do however provide descriptive statistics of the presented in Table D.1 as well anonymized datasets that are sufficient recreate all figures in this paper.
D.2 Data extraction per clinical setting
Breast cancer screening. We extract a patient's first recorded routine preventative visit designated by ICD and CPT codes. The treatment indicator T for a patient is whether they received a breast cancer screen as designated by ICD code within 7 days of the recorded encounter date. The running variable X is the patient's 's age at the initial encounter date (note that in order to protect patient privacy, the claims database only has resolution to a patient's year birth). We consider candidate thresholds of Cx = [40, 45, ..., 60] with the data-driven bandwidth selected to be 4, age increments of years to align with typical screening guideline values.
Colon cancer scrcening. Similar to the brcast cancer setting, we paticnt's first recorded routine preventative care visit. The treatment indicator T for a patient is whether they received colon cancer screen within days of the recorded encounter date, and age is the running variable X with candidate thresholds of Cx = [40, 45...,, 60) and bandwidth 4, at age increments years to align with typical screening guideline values.
Type 2 diabetes diagnosis. We extract patient's first recorded AIC measure as designated by LOINC codes and use it the running variable X. The treatment indicator T for patient
30

---

Published in Transactions Machine Learning Research (10/2023)
is whether a type 11 diabetes diagnosis 1CD code appears in their record within 30 days of the first recorded A1C mcasure. We consider candidate thresholds of Cx [5.0, 5.1,,..1,..3 5.,, 7.5] and data-driven bandwidth selected to be 0.4 this is the standard range of A1C values, with the lab readings having precision to one decimal place.
In all three clinical settings, we exclude patients that have recorded treatment indicator code prior to their initial encounter date, as well as patients that do not have recorded demographic information. The following covariates are included as W for each patient (unordered categorical variables are one-hot encoded, while ordinal variables are coded integers): gender, encounter date, insurance type (Medicare commercial), race, education level, and household income range.
D.3 Full result and cohort details
Claims data analyses were run on secure CentOS Limux server with 40-core Intel Xeon E5- 4650 CPU and 504 GB RAM. We follow the same hyperparameter and model training strategy described in Appendix C.4 For the RD candidate cutoff identified in each clinical setting, we show the best subgroup in terms of effective sample size in Table 1 We describe the demographics of patients within the RD bandwidth of analysis in Table D.1
Table D.I: Demographic details for each clinical setting within RD bandwidth.
Breast cancer
Colon cancer
Type diabetes
screen, age 2 40
screen, age 2 50
diagnosis, A1C 2 6.5
Sample size
315,225
346,191
133,826
Mean age (SD)
39.9 (1.98)
49.9 (1.98)
59.6 (13.4)
Gender (%)
Male
169,050
(53.6)
171,601 (49.6)
61,583
(46.0)
Female
146,175 (46.4)
174,590 (50.4)
72,242 (54.0)
Race (%)
White
212,441 (67.4)
247,165 (74.3)
77,945 (58.2)
Black
32,700 (10.4)
33,387 (9.6)
23,159 (17.3)
Asian
26,158 (8.3)
16,578 (1.8)
10,481 (7.8)
Hispanic
43,926
(13.9)
39,057 (11.3)
22,241 (16.6)
Education level (%)
Less than 12th grade
1,285 (0.4)
1,349 (0.4)
1,098 (0.8)
High school diploma
59,943 (19.0)
73,995 (21.4)
40,783 (30.5)
Less than Bachelor's
166,612 (52.9)
187,171 (54.1)
71,527 (53.4)
Bachelor's degree plus
87,385 (27.7)
83,676 (24.2)
20,418 (15.3)
Houschold income (%)
<$40k
17,529 (15.1)
50,651 (14.6)
35,456 (26.5)
$10k 19k
17,756 (5.6)
19,290 (5.6)
11,312 (8.5)
$50k 59k
18,388 (5.8)
21,196 (6.1)
12,234 (9.1)
$60k 74k
28,648 (9.1)
33,265 (9.6)
15,938 (12.0)
$75k 99k
47,319 (15.0)
56,234 (16.2)
21,417 (16.0)
S100k
155,585 (49.4)
165,555 (47.8)
37,469 (27.9)
31


Reviews:
[{"review_id": "2r0Qy0tHlA", "reviewer": "Reviewer_Kxby", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1) Very clear exposition with great appendix materials\n2) Novel approach even conceptually to conduct a subgroup search and optimize /tau. Very clever and intuitively appealing\n3) Important work on the effective subgroup size, a nice approach that can inspire future work and allow for methodological comparison\n4) Good combination of synthetic and real world results with strong comparisons to relevant baseline methods. Great to see the approach discovering real world RDs in a complex dataset.\n\nWeakness:\n1) The authors utilize a hypothesis testing framework (and correction for multiple hypothesis testing). However, there is no discussion about what it means to have an experimental design with 95% confidence. How does this impact subsequent results utilizing that experimental design? In traditional econometric research, the RD is not discovered programmatically but by reason/intuition. While this might not be scalable, there is no computational concern that the RD reduces the power of the subsequent research. How would a researcher think about utilizing the results from this model for their research?\n\n2) In their real world experiments the authors note relatively \"simple\" subgroups with  only 1 dimension. That is reasonable since much more complicated subgroups would quickly become uninterpretable and thus not very useful. From a different perspective, you can say that even if the algorithm discovered an RD in high dimensions, we would be skeptical it is a true RD, since RDs are traditionally thought of as artifacts of human/bureaucratic rules. These are, by definition simple or dumb rules, not likely to be discovered in high dimensional searches. \nSaid simply: what is the econometric value of an RD discovered across multiple covariates? Are there are practical examples/papers you can point to that utilize such an RD?\n\n3) In section 7.2 the authors briefly mention that they have not considered testing \"whether the running variable has been manipulated\" as per McCrary, 2008. However, I believe LoRD3 runs that test. Did the authors ensure to modify LoRD3 to not run that test as to ensure equitable comparison of the results between the methods? Or did they take another approach?\n\n4) In section 7.2 the authors briefly mention that they have not considered testing \"whether the running variable has been manipulated\" as per McCrary, 2008. I have a concern that perhaps the subgroup optimization, especially around effective sample size, can somehow cause the algorithm to falsely identify an RD due to bunching, or else produce a subgroup that falsely passes a bunching test. Honestly I can't think of a good example here but it was concerning that this wasn't addressed in the paper as bunching is a quite common issue with RDs in practice.\n\n5) It seems that LoRD3 performs less testing than RDSGD. Thus in a \"simple\" fuzzy RD we would expect LoRD3 to have more power than RDSGD. Is that correct? The synthetic experiments presented in section 5 assume a subgroup-based RD. Thus it's reasonable that RDSGD outperforms all baselines. However, what would be the results for synthetic experiments with a simple fuzzy RD? It would be useful to understand not only where RDSDG outperforms, but also where it underperforms (if ever).\n\n6) A bit disappointing to only have 1 real world case study, especially since the results were not very surprising/insightful from a practitioner point of view.\n\n(I really like the article and the author's approach. If any of my critiques are too harsh, it is from a place of excitement about the paper and I hope the authors will forgive any slight)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors provide a novel approach for discovering regression discontinuity designs in observational data. Specifically, their approach is distinguished by the ability to detect high-powered, coherent subsets of data for an RD. The paper provides both synthetic and real world experiments validating the approach and showing the potential for improving the speed and nuance of scientific and social scientific research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My requested changes follow the weaknesses I noted above:\n1) Not critical but highly recommended.\n2) Critical.\n3) Critical.\n4) Not critical but would love the author's thoughts.\n5) Not critical.\n6) Potentially critical.", "broader_impact_concerns": "No concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A0pQTY2cYx", "reviewer": "Reviewer_ifu5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* I like the different perspective of this work within treatment effects. While I am quite unfamiliar with RDs, this seems like a novel persective, at least in machine learning, which mostly focusses on inference, rather than clustering (?).\n* Overall the paper looks good. There are some illustrative figures, examples and intuitive experiments. There are a few points which I think could still be improved (listed below).\n* I look forward to follow-up work.\n\n### Weaknesses\n\n* Being unfamiliar with RDs, I think a more gentle introduction would help at least me in reading this paper. Introducing RD in 3.1 is done rather compact. Since TMLR is a journal (rather than a conference), I would encourage the authors to spend a bit more space on this. If the main text is too limited, I would definitely advise in favour of a detailed appendix.\n* I would really enjoy a running example. The authors mention diabetes a few times, but I did not find that very helpful. Would it be possible to provide an example where standard treatment effect inference methods are insufficient and one better uses an RD method? Furthermore, would it be possible to use the example to differentiate RDSGDs objective (of finding subgroups) is superior to standard RD methods?\n* cfr. my question below, I think it would also be beneficial to discuss (albeit briefly) the (C)ATE field in general. Currently, RDs are presented as an island of literature but I would prefer to see it in relation to more well-known methods in treatment effects. \n\n\n### Questions\n* I am quite unfamiliar with IDs for treatment effects. Perhaps you can clarify, for me, how having a _deterministic_ assignment policy ($Z_i = \\mathbf{1}[X_i > c_i]$, aligns with typical TE assumptions such as overlap (which assumes _anyone_ can receive treatment, including $X_i < c_i$).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary.** The authors propose a method (RDSGD) which identifies subgroups in a set of covariates $X$. For each subgroup, the authors aim to maximise a regression discontinuity (RD), which manifests whenever a strict threshold determines treatment assignment. RDSGD is interesting for situations where the threshold is not known prior to inferring a treatment effect in RDs + for when it is reasonable to assume there can be different thresholds in different areas in $X\\in\\mathcal{X}$.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My requested changes are to clarify my weaknesses/questions above, in the main text or the appendix. I am also open to not include these clarifications, but I would need strong motivation from the authors for not doing so.", "broader_impact_concerns": "n.a.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "prIJUN6pIP", "reviewer": "Reviewer_14oM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength #1: The RGSD procedure is simple to implement and leverages off-the-shelf tools for summarizing treatment effect heterogeneity to discover combinations of thresholds and subgroups. As a result, there are no new technical results in the paper, but that is ok. The paper's contribution is to combine existing results on heterogeneous treatment effect estimation to the problem of causal discovery for RD designs. \n\nStrength #2: The authors illustrate their procedure in a real medical claims dataset and extensive simulations. The methods appear to be practically useful given these experiments. \n\nWeakness #1: In selecting subgroups (step #2), RDSGD should construct the test statistics and p-values on a hold out sample. Otherwise, the selected thresholds + groups will suffer from an post-selection inference problem (e.g., Berk et al. 2013 and many others). In other words, because the data are used twice -- first to select which hypothesis to test and second test the selected hypotheses -- the resulting tests will not control size correctly. \n\nAlternatively, the authors could try to characterize the selection event and try to apply recent results on conditional inference (e.g., Lee et al. 2016) but that seems difficult since the first step of identifying thresholds and subgroups is quite complex. Sample splitting seems like a simple solution. The conditional inference problem here does share some similarities to the problem of doing inference after estimating a break point in econometrics (e.g., Andrews et al. 2021). \n\nQuestion #1: Could these ideas be simply extended to regression kink designs? E.g., similar to Porter & Yu (2015) that consider estimating a regression discontinuity with an unknown threshold, Hansen (2017) does the same for regression kink designs (i.e., breaks in the first-derivative of the conditional expectation function). \n\nReferences:\nBerk et al. 2013: Valid post-selection inference\nLee et al. 2016: Exact post-selection inference with application to Lasso.\nAndrews et al. 2021: Inference After Estimation of Breaks\nHansen (2017): Regression Kink With an Unknown Threshold", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "  In this paper, the authors develop Regression Discontinuity Subgroup Discovery (RDSGD), which is a machine learning method that identifies RD thresholds for different subgroups. \n\nThe main insight of the paper is that for a particular conjectured threshold in a regression discontinuity design, searching for subgroups with a large change in treatment uptake probabilities can be recast as a compliance prediction problem (i.e., a conditional average treatment effect on the potential treatment function Di()). The authors therefore can apply ideas from heterogeneous treatment effect estimation to search for heterogeneity across subgroups in treatment response probabilities at the conjectured threshold using ideas from Kennedy et al. (2020) and Coussens & Spiess (2021). \n\nPractically, for a considered running variable and discrete grid, the authors in Algorithm 1 consider a search procedure for combinations of thresholds and subgroups that are have large treatment takeup effects. Then, given the groups and thresholds with the largest such effects, the authors estimate the report a corrected significance level (Bonferroni correction) of the change in treatment uptake and return the thresholds and subgroups with the most significant effects. The authors apply their method to a large real-world dataset on medical claims, and show that RDSGD returns known thresholds that meaningfully affect treatment decisions and disease risk.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the discussed weakness above.", "broader_impact_concerns": "I see no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The three reviewers recommended Accept or Learning Accept. There are however two suggestions to improve the manuscript before it is published, particularly:

1. The manuscript promotes a “bottom-up” data-driven approach to streamline and scale RD study discovery, unlocking opportunities for more interpretable and well-powered causal studies, this statement should be followed by a caution with respect to the power implications of data-driven design discovery since the bottom-up approach might reduce the subsequent power of any studies that use the design.

2. The manuscript should be specific about where and how it views the limits of the utility of multi-dimensional RD discovery. The current manuscript is certainly useful for discovering 1-3 dimensional RDs (case studies with the medical claim dataset discovers  "simple" subgroups with only 1 dimension), however more complex RDs might neither exist nor have econometric value.
Decision: Accept (poster)