Title: Hierarchical Bias-Driven Stratification for Interpretable Causal Effect Estimation
Venue: NeurIPS 2023
Accepted: no
Keywords: Causal inference

effect estimation

selective prediction
Abstract: Causal effect estimation from observational data is an important analytical approach for data-driven policy-making. However, due to the inherent lack of ground truth in causal inference accepting such recommendations requires transparency and explainability. To date, attempts at transparent causal effect estimation consist of applying post hoc explanation methods to black-box models, which are not interpretable. In this manuscript, we present BICauseTree: an interpretable balancing method that identifies clusters where natural experiments occur locally. Our approach builds on decision trees to reduce treatment allocation bias. As a result, we can define subpopulations presenting positivity violations and exclude them while providing a covariate-based definition of the target population we can infer from. We characterize the method's performance using synthetic and realistic datasets, explore its bias-interpretability tradeoff, and show that it is comparable with existing approaches. 

Abstract

Causal effect estimation from observational data is an important analytical approach for data-driven policy-making. However, due to the inherent lack of ground truth in causal inference accepting such recommendations requires transparency and explainability. To date, attempts at transparent causal effect estimation consist of applying post hoc explanation methods to black-box models, which are not interpretable. In this manuscript, we present BICauseTree: an interpretable balancing method that identifies clusters where natural experiments occur locally. Our approach builds on decision trees to reduce treatment allocation bias. As a result, we can define subpopulations presenting positivity violations and exclude them while providing a covariate-based definition of the target population we can infer from. We characterize the method's performance using synthetic and realistic datasets, explore its bias-interpretability tradeoff, and show that it is comparable with existing approaches.

1 Introduction

The primary task of causal inference is estimating the effect of a treatment or intervention. Evaluating the strength of a causal relationship is essential for decision-making, designing interventions, as well

as evaluating the effect of a policy. As such, causal inference has high applicability across multiple fields including medicine, social sciences and policy-making.

However, the estimation of a causal effect requires the computation of 'potential outcomes' i.e.

the outcome an individual would experience if they had received some potential treatment, which may differ to the observed one (1). When treatment is binary, the quantity of interest is often the

difference between the average potential outcomes in an all-treated scenario vs an all-untreated scenario. Estimating and evaluating causal effect from observational data is thus challenging as we

only observe a single potential outcome-the one under the observed treatment-and can never observe the counterfactual outcome, lacking ground-truth labels. Furthermore, when treatment assignment is

not randomized, groups that do or do not receive treatment may not be comparable in their attributes, and such attributes can influence the outcome too (i.e. confounding bias).

In addition to these fundamental challenges, in practical settings where causality is used, decision-

making can often be safety-sensitive (e.g.

healthcare, education).

This, in turn, incentivizes 'in- terpretable' modeling to either comply with ethics requirements or be properly communicated to

interested parties. Here, interpretability means that each decision in the algorithm is inherently 31

explicit and traceable, contrasting with explainability where decisions are justified post-hoc using an 32

external model (2). Moreover, due to the lack of ground truth, interpretability is of greater importance 33

in causal inference where understanding a model may be the only way to question it. 34

In this paper, we introduce BICauseTree: Bias-balancing Interpretable Causal Tree, an interpretable 35

balancing method for observational data with binary treatment that can handle high-dimensional 36

datasets. We use a binary decision tree to stratify on imblanced covariates and identify subpopulations 37

with similar propensities to be treated, when they exist in the data. The resulting clusters act as local 38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

naturally randomized experiments. This newly formed partition can be used for effect estimation, as well as propensity score or outcome estimation. Our method can further identify positivity violating regions of the data space, i.e., subsets of the population where treatment allocation is highly unbalanced. By doing so, we generate a transparent, covariate-based definition of the target population or 'inferentiable' population i.e. the population with sufficient overlap for inferring causal effect.

Our contributions are as follows:

Our BICauseTree method can identify 'natural experiments' i.e. subgroups with lower treatment imbalance, when they exist.

BICauseTree compares with existing methods for causal effect estimation in terms of bias while maintaining interpretability. Estimation error and consistency of the clusters show good robustness to subsampling in both our synthetic and realistic benchmark datasets.

Our method provides users with built-in prediction abstention mechanism for covariate spaces lacking common support. We show the value of defining the inferentiable population using a clinical example with matched twins data.

The resulting tree can further be used for propensity or outcome estimation.

We release open-source code with detailed documentation for implementation of our method, and reproducibility of our results.

2 Related work

2.1 Effect estimation methods

Causal inference provides a wide range of methods for effect estimation from data with unbalanced treatment allocation. There are two modelling strategies: modelling the treatment using the covariates to balance the groups, and modelling the outcome directly using the covariates and treatment assignment.

In balancing methods such as matching or weighting methods, the data is pre-processed to create subgroups with lower treatment imbalance or 'natural experiments'. Matching methods consist of clustering similar units, based on some distance metric, from the treatment and control groups to reduce imbalance. Euclidean and Mahalanobis distances are commonly used, together with nearest covariate-based matching tends to become ineffective in such settings (3). Weighting methods aim at balancing the covariate distribution across treatment groups, with Inverse Probability Weighting (IPW) (4) being the most popular approach. Samples weights are the inverse of the estimated propensity scores, i.e. the probability of a unit to be assigned to its observed group. However, extreme IPW

Contrastingly, in adjustment methods the causal effect is estimated from regression outcome models where both treatment and covariates act as predictors of the outcome. These regressions can be fitted Under this taxonomy, BICauseTree is a balancing method, i.e., a data-driven mechanism for achieving conditional exchangeability. Nonetheless, BICauseTree can be combined with other methods to achieve superior results. Either as propensity models in established doubly robust methods (10), or by incorporating arbitrary causal models at leaf nodes (similar to regression trees with linear models at leaf nodes (11)).

neighbour search. However, as the notion of distance becomes problematic in high dimensional spaces, weights can also increase the estimation variance. through various methods like linear regression (5), neural networks (6; 7), or tree-based models (8; 9).

2.2 Positivity violations

Causal inference is only possible under the positivity assumption, which requires covariate distributions to overlap between treatment arms. Thus, positivity violations (also referred to as no overlap) occur when certain subgroups in a sample do not receive one of the treatments of interest or receive it too rarely (12). Overlap is essential as it guarantees data-driven outcome extrapolation across treatment groups. Having no common support means there are subjects in one group with no counterparts from the other group, and, therefore, no reliable way to pool information on their outcome had they been in the other group. Non-violating samples are thus the only ones for which we can guarantee some validity of the inferred causal effect.

There are three common ways to characterize positivity. The most common one consists in estimating propensity scores and excluding the samples associated with extreme values (also known as 'trimming') (13). The threshold for propensity scores can be set arbitrarily or dynamically (14). However, since samples are excluded on the basis of their propensity scores and not their covariate values, these

methods lack interpretabilty about the excluded subjects and how it may affect the target population

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

on which we can generalize the inference. Consequently, other methods have been developed to overcome this challenge by characterizing the propensity-based exclusion (15; 16; 17). Lastly, the third way tries to characterize the overlap from covariates and treatment assignment directly, without going through the intermediate propensity score e.g. PositiviTree (12). In PositiviTree, a decision tree classifier is fitted to predict treatment allocation. In contrast to their approach, BICause Tree implements a tailor-made optimization function where splits are chosen to maximize balancing in the resulting sub-population, whereas PositiviTree uses off-the-shelf decision trees maximizing separation. Ultimately, the above mentioned methods for positivity identification and characterization are model agnostic. In our model, BICauseTree, positivity identification and characterization are inherently integrated in the model, and effect estimation comes with a built-in interpretable abstention prediction mechanism.

2.3 Interpretability and causal inference

A predominant issue in existing effect estimation methods is their lack of interpretability. A model is considered as interpretable if its decisions are inherently transparent (2). Examples of interpretable models include decision trees where the decision can be recovered as a simple logical conjunction. Contrastingly, a model is said to be explainable when its predictions can be justified a-posteriori by examining the black-box using an additional 'explanation model'. Popular post-hoc explanation models include Shapley values (18) or LIME (19). However, previous works have shown that existing explainability techniques lack robustness and stability (20). Further, the explanations provided by explanation models inevitably depend on the black-box model's specification and fitness. Given that explanation models only provide unreliable justifications for black-box model decisions, a growing number of practitioners have been advocating for intrinsically interpretable predictive models (2). We further claim that causal inference, and in particular effect estimation, should be interpretable as it assists high-stake decisions affecting laypeople.

Causal Trees (8) are another tree-based model for causal inference that (i) leverages the inherent interpretability of decision trees, and (ii) has a custom objective function for recursively splitting the data. Although both utilize decision trees, BICauseTree and Causal Tree (CT) serve distinct purposes. BICauseTree splits are optimized for balancing treatment allocation while CT splits are optimized for balancing treatment effect , under assumed exchangeability. In other words, CT assumes exchangeability while BICauseTree 'finds' exchangeability. As such, our approach is more suited for ATE estimation while CT is better suited for Conditional Average Treatment Effect estimation (8). Furthermore, in practice, causal effects are often averaged over multiple trees into a so-called Causal Forest (21; 22) that is no longer interpretable, and users are encouraged to use post-hoc explanation methods (23). result, as there can be structural biases leading to entire subpopulation being excluded. Therefore,

In addition to effect estimation, positivity violations characterization should also be interpretable for downstream users, such as policy makers. Discarding samples can hurt the external validity of any interpretable characterization of the overlap in a study can help policy makers better assess on whom they expect the study results to apply (15; 12). In our model, BICauseTree, we generate a covariate-based definition of the violating subpopulation. In other words, we can claim which target population our estimate of the Average Treatment Effect applies to.

3 BICauseTree

3.1 Problem setting

We consider a dataset of size n where we note each individual sample ( X i , T i , Y i ) with X i ∈ R d is a covariate vector for sample i measured prior to treatment allocation, and T i is a binary variable denoting treatment allocation. In the potential outcomes framework (24), Y i (1) is the outcome under T i = 1 , and Y i (0) is the analogous outcome under T i = 0 . Then, assuming the consistency assumption, the observed outcome is defined as Y i = T i Y i (1) + (1 -T i ) Y i (0) . In this paper, we focus on estimating the average treatment effect (ATE), defined as: ATE = E [ Y (1) -Y (0)] .

3.2 Motivation

We introduce a method for balancing observational datasets with the goal of estimating causal effect in a subpopulation with sufficient overlap. Our goals are: (i) unbiased estimation of causal effect, (ii) interpretability of both the balancing and positivity violation identification procedures, (iii) ability to handle high-dimensional datasets. Our approach utilizes the Absolute Standardized Mean Difference (ASMD) (25) frequently used for assessing potential confounding bias in observational data. Note

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

that our balancing procedure is entirely interpretable, although it can be used in combination with arbitrary black-box outcome models or propensity score models. Finally, our method generates a covariate-based definition of the target population on which we make inference. As such, it is tailored to sensitive domains where inference should be restricted to subpopulations with reasonable overlap.

3.3 Algorithm

The intuition for our algorithm is that, by partitioning the population to maximize treatment allocation heterogeneity, we may be able to find subpopulations that are natural experiments. We recursively partition the data according to the most imbalanced covariate between treatment groups. Using decision trees makes our approach transparent and non-parametric.

Splitting criterion The first step of our algorithm is to split the data until some stopping criterion is met. The tree recursively splits on the covariate that maximize treatment allocation heterogeneity. To do so, we compute the Absolute Standardized Mean Difference (ASMD) for all covariates and select the covariate with the highest absolute value. The ASMD for a variable X j is defined as:

The reason for choosing the feature with the highest ASMD is that it is most likely to be a confounder. Once that next splitting covariate j max is chosen, we want to find a split that is most associated with treatment assignment, so that we may control for the effect of counfounding. The tree finds the optimal splitting value by iterating over covariate values x j max and taking the value associated with the lowest p -value according to a Fisher's exact test or a χ 2 test, depending on the sample size.

Stopping criterion The tree building phase stops when either: (i) the maximum ASMD is below some threshold, (ii) the minimum treatment group size falls below some threshold (iii) the total population fall below the minimum population size threshold, or (iv) a maximum tree depth is reached. All of the thresholds are user-defined hyperparameters.

Pruning procedure Once the stopping criterion is met in all leaf nodes, the tree is pruned. A multiple hypothesis test correction is first applied on the p -values of all splits. Following this, the splits with significant p -values or with at least one split with significant p -value amongst their descendants are kept. Ultimately, given that ASMD reduction may not be monotonic, pruning an initially deeper tree allows us to check if partitioning more renders unbiased subpopulations. The implementation of the tree allows for user-defined multiple hypothesis test correction, with current experiments using Holm correction (26). The choice of the pruning and stopping criterion hyperparameters will guide the bias/variance trade-off of the tree. Deeper trees may have more power to detect treatment effect while shallower trees will be more likely to have biased effect estimation.

Positivity violation filtering The final step evaluates the overlap in the resulting set of leaf nodes to identify those where inference is possible. The tree checks for treatment balance based on some user-defined overlap estimation method, with the default method being the Crump procedure (14). The positivity violating leaf nodes are tagged and then used for inference abstention mechanism, i.e. inference will be restricted to non-violating leaves.

Estimation Once a tree is contracted, it can be used to estimate both counterfactual outcomes and propensity scores. For each leaf, using the units propagated to that leaf, we can model the counterfactual outcome by taking the average outcome of those units in both treatment groups. Alternatively, we can fit any arbitrary causal model (e.g., IPW or an outcome regression) to obtain the average counterfactual outcomes in that leaf. The ATE is then obtained by averaging the estimation across leaves. Similarly, we can estimate the propensity score in each leaf by taking the treatment prevalence or using any other statistical estimator (e.g., logistic regression).

Code and implementation details Code for BICauseTree is released open-source, including detailed documentation under: https://anonymous.4open.science/r/BICause-Trees-F259 . Our flexible implementation allows the user to extend the default stopping criterion as well as the multiple hypothesis correction method. BICauseTree adheres to causallib 's API, and can accept various outcome and propensity models.

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

Algorithm 1 BICauseTree

Inputs: root node N 0 , X , T , Y

Call Build subtree ( N 0 , X , T , Y )

Do multiple hypothesis test correction on all split p -values

Pruning procedure : keep splits with either (i) a significant p -value or (ii) at least one descendant with a significant p -value

Mark leaf nodes that violate positivity violation criterion

Algorithm 2 Build subtree

Inputs: current node N , X , T , Y

if Stopping criteria not met then

Find and record in N the covariate with maximum ASMD: maxASMD := max i ( ASMD i ) Find and record in N the split value with the lowest p -value according to a Fisher test/ χ 2 test Record the p -value for this split in N

Split the data X,T,Y into X left , T left , Y left and X right , T right , Y right according to N 's splitting covariate and value

Add two child nodes to

N

:

N

left and

N

right

Call Build subtree ( N left , X left , T left , Y left )

Call right

end if

4 Experiment and results

4.1 Experimental settings

In all experiments-unless stated otherwise-the data was split into a training and testing set with a 50/50 ratio. The training set was used for the construction of the tree and for fitting the outcome models in leaf nodes, if relevant. Causal effects are estimated by taking a weighted average of the local treatment effects in each subpopulation. At the testing phase, the data is propagated through the tree, and potential outcomes are evaluated using the previously fitted leaf outcome model. We performed 50 random train-test splits, which we will refer to as subsamples to avoid confusion with the tree partitions. For each subsample, effects are only computed on the non-violating samples of the population. In order to maintain a fair comparison, these samples are also excluded from effect estimation with other models and with ground truth. All results are shown after filtering positivity-violating samples.

Baseline comparisons Wecompare our method to doubleMahalanobis Matching, Inverse Probability Weighting (IPW), and Causal Tree (CT). In Mahalanobis Matching (27; 28), the nearest neighbor search operates on the Mahalanobis distance: d ( X i , X j ) = ( X i -X j ) T Σ -1 ( X i -X j ) , where Σ is alternatively the estimated covariance matrix of the control and treatment group dataset. In Inverse Probability Weighting (4), a propensity score model estimates the individual probability of treatment conditional on the covariates. The data is then weighted by the inverse propensities P ( T = t i | X = x i ) -1 to generate a balanced pseudo-population. In Causal Tree (8), the splitting criterion optimizes for treatment effect heterogeneity (see section 3.1 for further details). We use a Causal Tree and not a Causal Forest to compare to an estimator which is equally interpretable as our estimator. We also compare our results to an unadjusted marginal outcome estimator, which will act as our 'dummy' baseline model. As using a single Causal Tree for our interpretability goal gives rise to high estimation bias, Causal Tree was excluded from the main manuscript for scaling purposes. We refer the reader to sections A.5, A.6 and A.7 for a comparison with CT. For synthetic experiments, we use the simplest version of our tree which we term BICauseTree(Marginal) where the effect is estimated from taking average outcomes in leaf nodes. For real-world experiments, we compare BICauseTree(Marginal) with BICauseTree(IPW), an augmented version in which an IPW model is fitted in each leaf node. To compare estimation methods, we compute the difference between the estimated ATE and the true ATE for each subsample (or train-test partition) and display the resulting distribution of estimation biases in a box plot. Further experimental details, including hyperparameters, can be found in the Appendix under section A.9.

Build subtree

(

N

,

X

right

,

T

right

,

Y

right

)

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

4.2 Synthetic datasets

We first evaluate the performance of our approach on two synthetic datasets. We first demonstrate BICauseTree's ability to identify subgroups with lower treatment imbalance on a dataset which we will refer to as the 'natural experiment dataset' in the following. We further exemplify BICauseTree's identification of positivity violating samples on a dataset we refer to as the 'positivity violations dataset'. Due to the interaction-based nature of the data generation procedure, we additionally compare our approach to an IPW estimator with a Gradient Boosting classifier propensity model, referred to as IPW (GBT) in both synthetic experiments. This choice ensures a fair comparison across estimators.

Identifying natural experiments For the natural experiment dataset, we considered a Death outcome D , a binary treatment of interest T and two covariates: Sex S and Age A . We defined four sub-populations, where each constituted a natural experiment with a truncated normal propensity distribution centered around a pre-defined constant value and variance (see details in Section A.4.1). Then, individual treatment propensities were sampled from the corresponding distribution and observed treatment values were sampled from a Bernoulli distribution parameterized with the individual propensities. No positivity violation was modeled in this experiment. Ultimately, X = ( S, A ) is the vector of covariate values in R 2 with the sample size chosen as n = 20 , 000 . The marginal distribution of covariates follows: S ∼ Ber(0 . 5) and A ∼ N ( µ, σ 2 ) where µ = 50 and σ = 20 . Figure A1 in A.5.1 shows the partition obtained from training BICauseTree on the entire dataset. Our tree successfully identifies the subpopulations in which a natural experiment was simulated. Figure 1a shows the estimation bias across subsamples. In addition to being transparent, BICauseTree has lower bias in causal effect estimation compared to all other methods, excluding IPW(GBT) which has comparable performance. Despite its higher estimation variance, Matching has low bias, probably due to covariate space being well-posed and low-dimensional. Contrastingly, the logistic regression in IPW(LR) is not able to model treatment allocation as the true propensities are generated from a noisy piecewise constant function of the covariates resulting in a threshold effect that explains its poor performance. The non-parametric, local nature of both Matching and BICauseTree thus contrasts with the parametric estimation by IPW(LR). Further results on the BICauseTree's calibration and covariate partition can be found in the Appendix, under section A.5.1.

0.00

0.02

0.04

0.06

0.08

|

ATE

ATE

|

BICause Tree

Marginal

IPW (LR)

IPW (GBT)

Marginal

Matching

Split

Test

Train

0.0

0.2

0.4

0.6

|

ATE

ATE

|

BICause Tree

Marginal

IPW (LR)

IPW (GBT)

Marginal

Matching

0.00

0.02

0.04

0.06

Split

Test

Train

(a) Estimation bias for the natural experiment dataset (see subsection 4.2) across 50 subsamples, with N = 20 , 000

(b) Estimation bias for the positivity violations dataset (see subsection 4.2) across 50 subsamples, after excluding positivity violating leaf nodes with N = 20 , 000 .

Identifying positivity violations For the positivity violations dataset, we consider a synthetic dataset 260 with a Death outcome D , a binary treatment of interest T , and three Bernoulli covariates -Sex S , 261 cancer C and arrhythmia A - such that X = ( S, C, A ) (see Section A.4.2 for further details). As 262 for the natural experiment dataset, we modeled treatment allocation with stochasticity by sampling 263 propensities from a truncated gaussian distribution first. Treatment allocation was simulated to ensure 264 that overlap is very limited in two subpopulations: females with no cancer and no arrhythmia are 265 rarely treated, while males with cancer and arrhythmia are almost always treated. Figure A2 in 266 A.5.2 shows the partition obtained from training BICauseTree on the entire dataset, confirming that 267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

BICauseTree excludes the subgroups where positivity violations were modeled. On average, 67.1% of the cohort remained after positivity filtering with very little variability across subsamples. Thanks to the interpretable nature of our method, we are able to identify these subgroups as a region of the covariate space. As seen in Figure 1b, after filtering violating samples the effect estimation by BICauseTree remains unbiased and with low variance. Our estimator compares with IPW(GBT) while being interpretable. The IPW(LR) estimator is more biased than BICauseTree. This may be due to the extreme weights in the initial overall cohort. In spite of filtering samples from regions with lack of overlap-as defined by BICauseTree-the remaining propensity weights may be biased, which would ultimately induce a biased effect estimation. Estimation variance is comparable across methods, except for Matching which is both more biased and has higher variance than all other estimators. Further results on the BICauseTree's calibration and covariate partition can be found in the Appendix, under section A.5.2.

4.3 Realistic datasets

Causal benchmark datasets We use two causal benchmark datasets to show the value of our approach. The twins dataset illustrates the high applicability of our procedure to clinical settings. It is based on real-world records of N = 11 , 984 pairs of same-sex twin births, and has 75 covariates. It tests the effect of being born the heavier twin (i.e. the treatment) on death within one year (i.e. the outcome), with the outcomes of the twins serving as the two potential outcomes. We use the dataset generated by Neal et. al (29), that simulates an observational study from the initial data by selectively hiding one of the twins with a generative approach. We also ran our analysis on the 2016 Atlantic Causal Inference Conference (ACIC) semisynthetic dataset with simulated outcomes (30). For ACIC, given that trees are data greedy, and due to the smaller sample size ( N = 4 , 802 ) relative to the number of covariates ( d = 79 ), the models were trained on 70% of the dataset.

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

|

ATE

ATE

|

BICauseTree

(IPW)

BICauseTree

(Marginal)

IPW

Marginal

Matching

Test

Train

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

|

ATE

ATE

|

BICauseTree

(IPW)

BICauseTree

(Marginal)

IPW

Marginal

Matching

Test

Train

Effect estimation Figure 2 shows the distribution of the estimation biases across subsamples on the twins dataset, comparing to the baseline models. Here, our BICauseTree(Marginal) estimator is less biased than the marginal estimator. Augmenting our tree with an IPW outcome model BICauseTree(IPW) - further decreases estimation bias, making it comparable with IPW, both w.r.t bias and estimation variance. Figure 3 compares the estimation biases across estimators on the ACIC dataset. Here, both BICauseTree models compare with IPW in terms of bias and estimation variance. Bias-interpretability tradeoff We expect a bias-interpretability tradeoff, where deeper trees are less biased but more complex to understand, while shallower trees are less accurate but easier to comprehend. Figure 4 shows how estimation bias in leaf nodes decreases as we increase the maximum depth hyperparameter of our BICauseTree(Marginal) in the twins dataset. Here, each circle in the plot represents a leaf node, and the dotted line shows the average bias with an IPW estimator. The shaded area represents the 95% confidence interval (CI) for IPW. As seen in the plot, there is some overlap between the 95% CI for IPW and the estimation bias of deeper trees. The remaining gap thus represents the need for a more complex outcome model in the leaves, or in other words the estimation bias that was traded against interpretability here. Similarly, in figures 2 and 3 we notice how augmenting our partition with an IPW leaf outcome models has decreased the

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

estimation bias at the cost of transparency. Ultimately, figure 4 shows that bias reduction is consistent beyond a maximum depth parameter of 5. The robustness of our estimator w.r.t the maximum depth hyperparameter is likely due to our statistical pruning procedure. A similar figure is shown in Section A.7 for the ACIC dataset.

Interpretable positivity violations filtering

As previously discussed, BICauseTree provides a built-in method for identifying positivity violations in the covariate space directly. After positivity filtering, effect was computed on an average of 99 . 5% ( σ = 0 . 006 ) of the population on the twins dataset, and an average of 85 . 9% ( σ = 0 . 093 ) of the ACIC dataset.

Figure A4 in the Appendix shows the tree partition for the twins dataset. One leaf node was detected as having positivity violations ( N = 106 ). The twins example illustrates the real-world impact of having a covariate-based definition of the non-violating subpopulation. Here, we are able to claim that our estimate of the effect of being born heavier might not be valid for newborns that fit the criteria for this specific violating node. This capability of BICauseTree is highly valuable in any safety-sensitive setting. Consider a scenario where the 'at-risk' twin benefits from

1

2

3

4

5

6

7

8

9

10

Tree maximum depth

0.00

0.01

0.02

0.03

0.04

0.05

0.06

|

ATE

ATE

|

95% confidence interval

IPW

a follow-up visit after birth, and that the true effect of the intervention is higher in the positivity violating subpopulation. Extrapolating the estimated effect of the exposure to the entire cohort may be dangerous to the infants in this subgroup. It is thus essential for practitioners to know which population the inferred effect applies to, which would not have been possible using alternative non-interpretable methods for identifying positivity violations e.g. IPW with weight trimming, as they provide an opaque exclusion criterion. Additionally, note that the positivity violation identification remains transparent regardless of the chosen propensity or outcome model at the leaves.

Propensity score estimation Alternative use-cases for BICauseTree include using the partition as a propensity model. Given the importance of calibrated propensity scores (31), Figure 5 compares the calibration of the propensity score estimation of BICauseTree with the one from logistic regression (IPW) on the testing set of the twins dataset. As expected, logsitic regression, which has better data efficiency, has better, less-noisy calibration. However, BICauseTree still shows satisfying calibration on average. Section A.6 in the Appendix shows the calibration plots for the estimation of potential outcomes on the twins dataset. Section A.7 shows calibration plots for the ACIC dataset.

Tree consistency To evaluate the consistency of our clustering across subsamples, we train our tree on 70% of the dataset and compute the adjusted Rand index (32) (see further details in section A.2). We chose not to train on 50% of the data here as most of the inconsistency would then be due to the variance between subsamples. For the twins dataset, the Rand index across 50 subsamples of sample sizes N = 8 , 388 , is equal to 0 . 633 ( σ = 0 . 208 ). For the ACIC dataset, the Rand index across 50 subsamples of sample sizes N = 3 , 361 , is equal to 0 . 314 ( σ = 0 . 210 ) which shows that our tree is not consistent across subsamples if sample size is

0.5

0.6

0.7

0.8

0.9

1.0

Predicted probability

0.5

0.6

0.7

0.8

0.9

1.0

True probability

Average

Subsamples

Optimal

BICause Tree

0.5

0.6

0.7

0.8

0.9

1.0

Predicted probability

Average

Subsamples

Optimal

Logistic regression

not substantial. However, we exemplify consistent identification of the positivity population, with the variance of the percentage of positive samples equal to σ = 0 . 006 and σ = 0 . 093 (see paragraph 4.3) in the twins and ACIC dataset respectively. Ultimately, throughout our experiments, we noticed how consistency starts to decrease if the maximum depth hyperparameter increases past a certain threshold. As a heuristic, we would recommend users to test tree consistency across subsamples when tuning this hyperparameter.

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

5 Discussion

Strengths and limitations of our approach Following our discussion on the bias-interpretability tradeoff, we acknowledge that in complex data settings where finding sub-populations that enclose natural experiments is difficult, the resulting BICauseTree partition may have remaining bias in some leaf nodes, and ultimately render some estimation bias. This bias is, however, traded-off with enhanced interpretability, as previously discussed. Nonetheless, as exemplified in this work, the performances of BICauseTree remains comparable, with estimation bias being only slightly larger than common models such as IPW. We further emphasize the fact that its strength resides in the combination of (i) the performance of the estimator with (ii) the interpretability of the balancing and positivity identification procedures, and (iii) the ability to handle high-dimensional datasets. Another advantage of BICauseTree is its ability to identify complex interaction features that are significantly correlated to treatment allocation. Indeed, in leaf nodes that come directly from a significant split, the root-to-leaf path is an interaction significantly associated with treatment allocation after multiple hypothesis test correction. Common alternatives to identify such interactions include exhaustive enumeration of all pairs of feature interactions, or complex feature engineering (33). However these approaches either lack transparency or become problematic in high-dimensional datasets. Furthermore, the tree nature of our approach is a major strength. BICauseTree is a nonparametric estimator that inherit the desirable empirical properties of regression forests-such as stability, ease of use, and flexible adaptation to different functional forms. Finally, the computational expense induced from fitting a BICauseTree is manageable: it is roughly comparable to IPW and CausalForest, and substantially lower than for Matching (see detailed compute times in Section A.9) Our work has the following limitations: (i) due to its tree structure, BICauseTree has lower data efficiency than most other estimators, including IPW. However the data efficiency of BICauseTree was superior to that of CT in our experiments. (ii) our tree design has some lunging dependence on sample size. While our estimation of ASMD is independent of sample size, the variance of our estimator, ̂ ASMD , is dependent on n . Furthermore, having chosen the splitting covariate, the choice of a split point is biased towards equal split subgroups. (iii) our individual splitting decisions do not consider interactions and instead only consider the marginal association of covariates with treatment.

Applicability of BICauseTree We claim that BICauseTree is highly relevant when causality is examined in a context with substantial safety and ethical concerns. We consider the transparency of our built-in approach to positivity violation identification particularly relevant to fields such as epidemiology, econometrics, medicine, and policy-making. The social impact of our work, and its relevance to the upcoming policies for Artificial Intelligence is further discussed in section A.10. In addition, we claim that our ability to identify violating regions of the covariate space is key for experimental design. Fitting a BICauseTree to an existing dataset will advise practitioners on which individuals we currently lack data to infer an effect on, which will in turn inform them on the specific subpopulations they need to recruit from, in a potential next study.

Conclusion and future work Here, we introduced a model able to detect positivity violations directly in the covariate space, perform effect estimation comparable to existing methods, while allowing for interpretability. We demonstrated our model's performance on both synthetic and realistic data, and showcased its usefulness in the principle challenges of causal inference. Future work may include extension to a non-binary tree, where we allow splitting to more than two nodes. This could be done for instance by fitting a piece-wise constant function that predicts treatment and finds the potentially multiple thresholds for optimized hetereogenous subgroups. In addition, to refine our pruning procedure, we can account for the intrisic ordering of the p -values of the splits using sequential multiple hypothesis testing (34; 35; 36). Furthermore, following the work of (8) on the 'honest effect' in Causal Forests, we may use a subset of the data for fitting the partition of the tree and another distinct subset for fitting the outcome or propensity models in each leaf node. This procedure however requires having many samples. Another alternative to current model fitting, which is done independently in each leaf, is to partially pool estimates across the clusters and fit a multilevel outcome model with varying intercepts or varying slopes for treatment coefficients (37). In terms of estimation, one may investigate the performance of bagging multiple BICauseTrees into a BICauseForest, similarly to Causal Forest. Aggregating trees would however defeat the interpretability purpose. Finally, similarly to positivity-violating nodes, future work may explore the possibility of excluding leaf nodes with high maximum ASMD, under the premises that these subgroups do not enclose natural experiments.

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

References

D. B. Rubin, 'Estimating causal effects of treatments in randomized and nonrandomized studies.,' Journal of educational Psychology , vol. 66, no. 5, p. 688, 1974.

C. Rudin, 'Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,' Nature Machine Intelligence , vol. 1, no. 5, pp. 206-215, 2019.

K. Beyer, J. Goldstein, R. Ramakrishnan, and U. Shaft, 'When is 'nearest neighbor' meaningful?,' in Database Theory-ICDT'99: 7th International Conference Jerusalem, Israel, January 10-12, 1999 Proceedings 7 , pp. 217-235, Springer, 1999.

D. G. Horvitz and D. J. Thompson, 'A generalization of sampling without replacement from a finite universe,' Journal of the American statistical Association , vol. 47, no. 260, pp. 663-685, 1952.

G. W. Imbens and D. B. Rubin, Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015.

C. Shi, D. Blei, and V . Veitch, 'Adapting neural networks for the estimation of treatment effects,' Advances in neural information processing systems , vol. 32, 2019.

U. Shalit, F. D. Johansson, and D. Sontag, 'Estimating individual treatment effect: generalization bounds and algorithms,' in International Conference on Machine Learning , pp. 3076-3085, PMLR, 2017.

S. Athey and G. Imbens, 'Recursive partitioning for heterogeneous causal effects,' Proceedings of the National Academy of Sciences , vol. 113, no. 27, pp. 7353-7360, 2016.

S. R. Künzel, J. S. Sekhon, P. J. Bickel, and B. Yu, 'Metalearners for estimating heterogeneous treatment effects using machine learning,' Proceedings of the national academy of sciences , vol. 116, no. 10, pp. 4156-4165, 2019.

J. D. Kang and J. L. Schafer, 'Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data,' 2007.

J. R. Quinlan et al. , 'Learning with continuous classes,' in 5th Australian joint conference on artificial intelligence , vol. 92, pp. 343-348, World Scientific, 1992.

E. Karavani, P. Bak, and Y. Shimoni, 'A discriminative approach for finding and characterizing positivity violations using decision trees,' arXiv preprint arXiv:1907.08127 , 2019.

M. L. Petersen, K. E. Porter, S. Gruber, Y. Wang, and M. J. Van Der Laan, 'Diagnosing and responding to violations in the positivity assumption,' Statistical methods in medical research , vol. 21, no. 1, pp. 31-54, 2012.

R. K. Crump, V. J. Hotz, G. W. Imbens, and O. A. Mitnik, 'Dealing with limited overlap in estimation of average treatment effects,' Biometrika , vol. 96, no. 1, pp. 187-199, 2009.

M. Oberst, F. Johansson, D. Wei, T. Gao, G. Brat, D. Sontag, and K. Varshney, 'Characterization of overlap in observational studies,' in International Conference on Artificial Intelligence and Statistics , pp. 788-798, PMLR, 2020.

G. Wolf, G. Shabat, and H. Shteingart, 'Positivity validation detection and explainability via zero fraction multi-hypothesis testing and asymmetrically pruned decision trees,' arXiv preprint arXiv:2111.04033 , 2021.

S. Ackerman, E. Farchi, O. Raz, M. Zalmanovici, and P. Dube, 'Detection of data drift and outliers affecting machine learning model performance over time,' arXiv preprint arXiv:2012.09258 , 2020.

S. M. Lundberg and S.-I. Lee, 'A unified approach to interpreting model predictions,' Advances 465 in neural information processing systems , vol. 30, 2017. 466

M. T. Ribeiro, S. Singh, and C. Guestrin, 'Model-agnostic interpretability of machine learning,' 467 arXiv preprint arXiv:1606.05386 , 2016. 468

B. Mittelstadt, C. Russell, and S. Wachter, 'Explaining explanations in ai,' in Proceedings of 469 the conference on fairness, accountability, and transparency , pp. 279-288, 2019. 470

471

472

473

S. Wager and S. Athey, 'Estimation and inference of heterogeneous treatment effects using random forests,' Journal of the American Statistical Association , vol. 113, no. 523, pp. 12281242, 2018.

474

475

476

477

478

S. Athey, J. Tibshirani, and S. Wager, 'Generalized random forests,' Ann. Statist , vol. 47, pp. 1148-1178, 2019.

K. Battocchi, E. Dillon, M. Hei, G. Lewis, P. Oka, M. Oprescu, and V. Syrgkanis, 'EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation.' https://github.com/microsoft/EconML, 2019. Version 0.x.

D. B. Rubin, 'The use of matched sampling and regression adjustment to remove bias in 479 observational studies,' Biometrics , pp. 185-203, 1973. 480

P. C. Austin, 'Balance diagnostics for comparing the distribution of baseline covariates between 481 treatment groups in propensity-score matched samples,' Statistics in medicine , vol. 28, no. 25, 482 pp. 3083-3107, 2009. 483

S. Holm, 'A simple sequentially rejective multiple test procedure,' Scandinavian journal of 484 statistics , pp. 65-70, 1979. 485

486

487

D. B. Rubin, 'Bias reduction using mahalanobis-metric matching,' Biometrics , pp. 293-298, 1980.

488

489

490

491

492

493

494

495

496

497

498

499

500

501

E. A. Stuart, 'Matching methods for causal inference: A review and a look forward,' Statistical science: a review journal of the Institute of Mathematical Statistics , vol. 25, no. 1, p. 1, 2010.

B. Neal, C.-W. Huang, and S. Raghupathi, 'Realcause: Realistic causal inference benchmarking,' arXiv preprint arXiv:2011.15007 , 2020.

P. R. Hahn, V. Dorie, and J. S. Murray, 'Atlantic causal inference conference (acic) data analysis challenge 2017,' arXiv preprint arXiv:1905.09515 , 2019.

R. Gutman, E. Karavani, and Y. Shimoni, 'Propensity score models are better when postcalibrated,' arXiv preprint arXiv:2211.01221 , 2022.

W. M. Rand, 'Objective criteria for the evaluation of clustering methods,' Journal of the American Statistical association , vol. 66, no. 336, pp. 846-850, 1971.

A. Zheng and A. Casari, Feature engineering for machine learning: principles and techniques for data scientists . " O'Reilly Media, Inc.", 2018.

M. Behr, M. A. Ansari, A. Munk, and C. Holmes, 'Testing for dependence on tree structures,' Proceedings of the National Academy of Sciences , vol. 117, no. 18, pp. 9787-9792, 2020.

M. P. Allen, 'Testing hypotheses in nested regression models,' Understanding regression 502 analysis , pp. 113-117, 1997. 503

504

505

A. Malek, S. Katariya, Y. Chow, and M. Ghavamzadeh, 'Sequential multiple hypothesis testing with type i error control,' in Artificial Intelligence and Statistics , pp. 1468-1476, PMLR, 2017.

506

507

508

509

510

A. Feller and A. Gelman, 'Hierarchical models for causal effects,' Emerging Trends in the Social and Behavioral Sciences: An interdisciplinary, searchable, and linkable resource , pp. 1-16, 2015.

S. M. Iacus, G. King, and G. Porro, 'Causal inference without balance checking: Coarsened exact matching,' Political analysis , vol. 20, no. 1, pp. 1-24, 2012.

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

A. Abadie and G. W. Imbens, 'Large sample properties of matching estimators for average treatment effects,' econometrica , vol. 74, no. 1, pp. 235-267, 2006.

P. C. Austin, 'Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies,' Pharmaceutical statistics , vol. 10, no. 2, pp. 150-161, 2011.

G. King and R. Nielsen, 'Why propensity scores should not be used for matching,' Political analysis , vol. 27, no. 4, pp. 435-454, 2019.

S. L. Morgan and C. Winship, Counterfactuals and causal inference . Cambridge University Press, 2015.

S. R. Seaman and S. Vansteelandt, 'Introduction to double robust methods for incomplete data,' Statistical science: a review journal of the Institute of Mathematical Statistics , vol. 33, no. 2, p. 184, 2018.

V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins, 'Double/debiased machine learning for treatment and structural parameters,' 2018.

A. Knudby and L. Ellsworth, 'Bonferroni, ce, 1936. teoria statistica delle classi e calcolo delle probabilita, pubblicazioni del r istituto superiore di scienze economiche e commerciali di firenze, 8: 3- 62. brenning, a., 2009. benchmarking classifiers to optimally integrate terrain analysis and multispectral remote sensing in automatic,' Environment , vol. 37, no. 1, pp. 35-46, 2009.

R. K. Crump, V. J. Hotz, G. Imbens, and O. Mitnik, 'Moving the goalposts: Addressing limited overlap in the estimation of average treatment effects by changing the estimand,' 2006.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the overlap violation problem in observational datasets for causal inference by presenting an interpretable balancing method for overlap violation identification and causal effect estimation for binary treatments. The method BICauseTree adapts decision tree classifiers to the stated problem by recursively splitting the data population into non-overlap-violating subgroups based on covariate dissimilarity and treatment heterogeneity. The major advantage of the presented method in comparison to existing balancing methods is the interpretability of the prediction process.\n\nSTRENGTHS:\n• The authors evaluate their method on both synthetic and real-world benchmarking datasets.\n\nWEAKNESSES:\n- The proposed method is highly similar to the work in reference 12. Furthermore, related work is not discussed appropriately (section 2). It is thus unclear how this work significantly differs from previous contributions in the literature. The originality of the submission has to be considered very limited.\n- The manuscript presents a complete piece of work. Claims about the performance of the proposed method are supported by experimental results.  Nevertheless, the an experimental study with sophisticated baseline methods is missing. A performance comparison with a Causal Forest model would be desirable.\n- Claims aiming at motivating the method are neither supported quantitatively nor experimentally (e.g., an analysis with different levels of overlap violation; unbiased estimation). \n- The submission lacks clarity due to multiple grammar errors and many nested arguments. The mathematical notation (section 3.1) lacks formal correctness.\n- The citation style does not agree with the required format for NeurIPS  submissions.\n- The statement of a high ASMD indicating a confounder (line 164) needs to be justified.\n- The paper would profit from a revision of the language and the consistency of the presented arguments (e.g., lines 84, 181/182). Furthermore, the figures and sections need to be referenced correctly, as also recognized by the authors in the appendix.\n- The authors claim interpretability of the method but do not provide evidence for the statement. Here, a user study would be desirable etc. \n- The method is limited to ATE.\n\nQUESTIONS:\n- What is the major difference between the method BICauseTree to the existing method PositiviTree (reference 12)? The novelty of the method cannot be deducted from the manuscript. \n- How do the benefits of the presented method outweigh the negative impact of the information loss due to the \"prediction abstention mechanism\"? A quantitative assessment would be of interest.\n\nLIMITATIONS:\n- The authors have stated the limitations of their work. The section could be improved by highlighting the general weaknesses of single trees in prediction settings which naturally devolve to the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method called BICauseTree for interpretable causal effect estimation. BICauseTree is a hierarchical bias-driven stratification method that identifies clusters where natural experiments occur locally. The method is designed to reduce treatment allocation bias and improve interpretability. The authors evaluate the performance of BICauseTree on several datasets and compare it to existing approaches. They find that BICauseTree performs well in terms of bias-interpretability tradeoff and outperforms existing methods in some cases. Overall, the paper presents a novel and promising approach to causal effect estimation that could have important applications in various fields.\n\nSTRENGTHS:\n1. Novelty: The paper proposes a novel method called BICauseTree for estimating causal effects from observational data. The method is based on a hierarchical bias-driven stratification approach that identifies clusters where natural experiments occur locally. The method builds on decision trees to reduce treatment allocation bias and provides a covariate-based definition of the target population. The method is interpretable and outperforms other state-of-the-art methods in reducing treatment allocation bias while maintaining interpretability.\n\n2. Significance: Causal effect estimation from observational data is an important analytical approach for data-driven policy-making. However, due to the inherent lack of ground truth in causal inference, accepting such recommendations requires transparency and explainability. The proposed method addresses this issue by providing an interpretable and unbiased method for causal effect estimation. The method has the potential to be applied in various domains, including healthcare, social sciences, and economics.\n\n3. Experimental Evaluation: The paper provides a thorough experimental evaluation of the proposed method using synthetic and realistic datasets. The authors compare the performance of their method with other state-of-the-art methods and show that their method has lower bias and comparable variance. They also conduct sensitivity analyses to evaluate the robustness of their method to violations of the assumptions. The experimental evaluation provides strong evidence to support the claims made in the paper.\n\nWEAKNESSES:\n1. Limited Scope: The paper focuses on a specific method for causal effect estimation from observational data, and the scope of the paper is relatively narrow (especially related to the tree-based models). While the proposed method is novel and has some advantages over other methods, it may not be of interest to a broad audience: (1) The method relies on the quality of the data and the assumptions made in the model. If the data is noisy or contains missing values, the method may produce biased estimates.; (2)The method may not be suitable for high-dimensional data, as the number of covariates may increase the complexity of the decision tree and lead to overfitting; (3) The method may not be suitable for datasets with small sample sizes, as the stratification may lead to small sample sizes in some subgroups, which may affect the accuracy of the estimates;(4)The method may not be suitable for datasets with complex interactions between the covariates, as the decision tree may not capture these interactions effectively.\n\n2. Experimental Evaluation: While the paper provides an experimental evaluation of the proposed method, the evaluation is limited in scope and does not provide a comprehensive comparison with other state-of-the-art methods. The experimental evaluation would benefit from a more comprehensive comparison with other methods (such as TARNet from the machine learning domain) and a more detailed analysis of the results.\n\nQUESTIONS:\nIn Figure 1, the IPW looks very close to the BiCausalTree. How can we apply post hoc explanations in such cases in general?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on achieving interpretable causal effect estimation, where the goal is to ensure that each decision within the algorithm is explicit and traceable. The authors propose a decision tree-based balancing method to address this problem, which identifies clusters where local natural experiments occur. The effectiveness of the proposed algorithm is empirically evaluated using synthetic and semi-synthetic data. The paper also did several ablation studies on the trade-off between interpretability and bias and the consistency of the decision tree.\n\nSTRENGTHS:\nOverall, this paper is well-executed and demonstrates several notable strengths.\n- This paper is very clear. It effectively presents complex ideas in an easily understandable manner.\n- The problem addressed in the paper is well-motivated and interesting.\n- The authors display a strong grasp of the related work in the field, effectively positioning their contribution within the existing literature.\n- The empirical analysis conducted in the paper is thorough and yields valuable insights.\n- The method is intuitive and well explained.\n\nWEAKNESSES:\n- Style file: One issue with this paper is that it doesn't follow the NeurIPS style guidelines, specifically regarding paragraph spacing. The paragraphs are not well-separated, which makes it harder to read and understand the content. This affects the overall flow and coherence of the paper. Additionally, the excessive content allowed due to the spacing issue may be seen as unfair to authors who followed the style guide correctly. This may be a potential ground for rejection.\n\n\n- Method: The rationale behind considering features with the highest ASMD as potential confounders is not well-explained. This is an important assumption in the paper, but it lacks a clear justification or empirical investigation. Providing additional explanations or conducting empirical studies would strengthen this aspect of the paper.\n\n- The experiment section of the paper is not self-contained. Although the motivating problem revolves around identifying subpopulations with natural experiments, this aspect is not adequately illustrated in the experiment section. Instead, the focus is primarily on bias analysis, with the analysis related to interpreting the causal effect estimation process deferred to the appendix. This undermines the fulfillment of the paper's fundamental promise to the readers. To address this issue, it is highly recommended that the authors integrate the analysis into the main paper, ensuring that the key components align with the paper's core premise.\n\nQUESTIONS:\nWhy does a feature with high ASMD more likely to be a confounder?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a decision tree methodology to identify regions where selection bias no longer ensures covariate balance. These regions, which have some level of interpretability, can then be removed in subsequent analysis.\n\nSTRENGTHS:\nThe paper presents an interesting decision tree methodology.\n\nThe paper contains a significant amount of simulation experiments to validate the procedure.\n\nWEAKNESSES:\nDespite a focus on covariate balance, there is no guarantee of balance unlike competing methods (rerandomization, matching, etc). Furthermore, there is limited analysis to show the claims of balance are fulfilled, especially for high dimensions.\n\nThe paper explores a bias-interpretability tradeoff, but provides no rigorous definition. The proposed model often is more biased than alternative models, most notably IPW, and it's not clear that the resulting decision trees, or their interpretations, are actually sensible. No discussion of estimator variance is given or how that might factor into a tradeoff, despite the high variance generally expected from decision tree estimators.\n\nQUESTIONS:\nIn Section 2, it is briefly claimed that BICauseTree is better suite for ATE estimation instead of CATE, with no further discussion. This feels like a potentially important point and requires further justification.\n\nTrimming has the potential to bias treatment effect measurement. Are there any guarantees that trimming in your model ensures unbiased estimates?\n\nAre the decision trees in Figures A13 and A17 interpretable? While the decision tree can be followed, the trimmed regions seem to have limited clinical interpretation. More discussion around this feels necessary. In particular, interpretable does not seem to imply correctness for this method.\n\nASMD is often minimized to establish balance. While splitting on variables that exhibit high ASMD should generally lead to some level of balance, is there any guarantee that the resulting regions will be optimally balanced in any sense?\n\nLIMITATIONS:\nThe paper touches on some limitations, most notably the increases bias that is expected. The paper does not discuss the variance of the estimator in detail relative to other methods, which is another potential limitation.\n\nThe method advocates for trimming nodes that violate positivity. These nodes could contain sensitive subpopulations and could lead to fairness concerns.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-07"}