Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In the observational data, this paper consider the case where covariates and the outcome are collected at multiple timestamps, $t \in \{ t_{1}, \ldots, t_{m} \}$, and only implement treatment $D=d, d \in$ {$0,1$} at the last timestamp $t_m$. In this case, the historical controls means that all treatments are $D=0$ at the timestamp $t \in ${$ t_{1}, \ldots, t_{m-1} $}. The treatment sequence is an known sequence, i.e.,{$0,0,\ldots, 0, d$}.

Using observational data and historical controls, this paper proposes a new notion confounding entropy measure the discrepancy between the conditional outcome distribution of the treated and control groups. Then, they use confounding entropy as splitting rule to divide the data into multiple subgroups and estimate  conditional average treatment effect on the treated group. 

----
I thank the authors for addressing my comments. I increase my score to 5.
strengths_and_weaknesses: **Strengths**

This paper considers a intersting setting about historical controls, and proposes a new notion confounding entropy measure the discrepancy between the conditional outcome distribution of the treated and control groups. The computational feasibility and statistical power are evidenced by simulations and credit card balance dataset. 

**Weaknesses**

- Using observational data and historical controls, we can directly and accurately estimate potential control outcomes $\hat{Y}^0$ with a large amount of historical control data. The conditional average treatment effect on the treated group can be easily obtained: $CATT = \mathbb{E}[Y|D=1,X] - \mathbb{E}[\hat{Y}^0|D=1,X]$. 
- In Lines 98-102, proper stratification [26] can help mitigate confounding effects from observed confounders but can't reduce the bias from unmeasured confounders that are independent with observed confounders.
- In practical, time-invariant covariates and unmeasured confounders assumption is hardly satisfied in a causal scenario. 
- In Lines 104-107, without any prior knowledge, neither constant function nor polynomial function  guarantees robust results. 
- Checklist in Lines 492-493: The Assumption 1 is not the full set of assumptions. SUTVA, positive/overlap assumption are missing in this paper. The authors should reframe the assumptions required. 
- Figure 1 and Figure 2 in Appendix, what data was used to draw this picture? the authors did not give a clear explanation. 
- Not Reproducible. The data generation mechanism, and estimators $\{\alpha(\cdot), \beta(\cdot), f(\cdot), g(\cdot)\}$ are not detailed in this paper.
questions: - Using observational data and historical controls, can we directly and accurately estimate potential control outcomes with a large amount of historical control data? Is the estimation of conditional average treatment effect on the treated group still a challenge? 
- Lines 126-127, what is the relationship between Theorem 1 and the identification of instrumental variables? 
- Why not use observed outcomes on the treated group when estimating CATT?
- The data generation mechanism, and estimators $\{\alpha(\cdot), \beta(\cdot), f(\cdot), g(\cdot)\}$ are not detailed in this paper.
limitations: I don't believe that this method proposed in this paper can reduce the bias from unmeasured confounders that are independent with observed confounders. Besides, in practical, time-invariant covariates and unmeasured confounders assumption is a strict condition that is difficult to satisfy.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new notion, confounding entropy, to measure the discrepancy between the treated and controlled groups. Based on this, the authors propose a new method, the debiased causal tree (more specifically, GBCT), to estimate treatment effects.
strengths_and_weaknesses: Originality: 5. This paper gives a new insight into causal inference with tree-based methods. The confounding entropy impresses me a lot.

Quality: 5. This paper is well written, especially for the mathematical parts.

Clarity: 4. It would be great if the context can be connected with former works closely. Readers may be very familiar with Athey's works on regression trees and causal forests, but the content in Section 2 is too dense.

Significance: 3.5. I don't think the improvement is as fair and significant as the paper states. I think the improvement should lie in comparing with the tree-based or forest-based causal ML methods.

___________Discussion period_________________

Since the authors have added adequate experiments to support their theoretical contributions. I strongly recommend an acceptance. I encourage other reviewers to have more discussions in case we miss a good paper. I thus raise my score by 1.
questions: 1. What are the differences between your work and "Recursive partitioning for heterogeneous causal effects" "Estimation and inference of heterogeneous treatment effects using random forests" "Generalized random forests"?

2. Why is confounding entropy valid? Is it like the individual version of the integral probability metric? Is there any connection?

3. Why do you need to compare GBCT with meta learners? They are just ways to estimate treatment effects, but the base models can be chosen as linear/tree/neural nets. Similarly, why DML-RF? DML is a doubly robust estimator, and the base models for nuisance parameters can be chosen as linear/tree/neural nets, and you choose RF. In my view, you only need to compare all the regression trees and random forests models that are related to causal inference and analyze the source of gain produced by confounding entropy.

4. In Section 4.2, I don't find any source of data. Where is the real-world dataset and code to reproduce your experiments? I hope code/datasets can be released in the rebuttal stage.

5. I am confused about the Biased dataset in Table 3. Is the control group increasing the credit line by 0? Why increasing the credit line for the low-risk group can bring bias into both low-risk and medium-risk groups? How does the RCT experiments conduct? Is it randomly assigning 0/2k/3k/6k to some people, where 0 is control and 2k/3k/6k is treat?

6. In my view, the most important part is to analyze the gain of confounding entropy, which is missing in other tree/forest-based causalML models. You are not developing a new triply robust estimator or a novel meta learner, so I think there is no need to compare with doubly robust estimators or other meta-learners. So, I think 4-5 well-known tree models for causalML are enough as baselines.
limitations: The authors do not discuss their limitations. But I think more extensions can be done based on this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers heterogeneous treatment effect estimation when you have longitudinal observations of outcomes pre-treatment. The main contributions are (i) a partial identification result of the CATE in this setting, (ii) a new objective function for finding optimal tree splits, and (iii) a boosting procedure that appears to work well on simulated and real-world data. 
The tree splitting criteria seeks a recursive partition of the space of measured confounders that minimizes the sum of two terms. The first term measures the error in fitting the observed post-treatment outcomes using piece-wise constant step functions (i.e. a regression tree). The second term aims to regularize towards partitions for which the distributions of outcomes for treated and control subjects in the pre-treatment period are thought to be small. Under a technical assumptions about how similar confounding bias in the pre-treatment period is to confounding bias in the post-treatment regime, the authors show that they can partially identify the CATE in the potential presence of unmeasured confounding.

=== 

After the response period, I have raised my score to 6.
strengths_and_weaknesses: Strengths: I commend the authors for an original approach to the difficult problem of estimating the CATE in the potential presence of unmeasured confounding. On the whole, I thought the paper was well-written and reasonably clear. I think that the proposed method has the potential to be quite useful to practitioners.

That said, my enthusiasm for the paper is tempered by several concerns about specific modeling and implementation choices and the corresponding assumptions. I will summarize these concerns here and expand on them in the Questions section. 

* I am somewhat concerned about the transition from Equation 8 to Equation 9. As written, the assertion in line 159 about the log density of $P_{t,1,j}$ being $-(y_t - \theta^{*(d)}_{j})^{2}$ is true only when $y_t$ is normally distributed with unit variance. You seem to implicitly assume that no matter the partition $Q_j$ or value of $D,$ the outcomes are normally distributed with known and fixed variance. I find the fixed & known variance assumption somewhat unrealistic. 

* Additionally, since $\theta_{t,j}^{*(d)}$ is typically unknown, you plug in the sample mean in Equation 8. This may be reasonable when $Q_{j}$ contains a large number of treated and control observations. However, it seems much less reasonable when there are not many of either.  As such the expression in Equation 9 may not be a particularly accurate estimate of the true symmetric cross-entropy between $P_{t,1,j}$ and $P_{t,0,j}.$ I'd go further and say that the expression in Equation 9 represents the symmetric cross-entropy between $P_{t,1,j}$ and $P_{t,0,j}$ under extremely restrictive conditions, namely that the $y_{t,j}$'s are centered around the corresponding sample mean with variance 1 for all choices of $Q_{j}.$

* I found the motivation for regularizing based on historical cross-entropy loss somewhat lacking. The identification result suggests that one should strive to make $\sum_{k = 1}^{m-1}{\sum_{j = 1}^{q}{b_{t_{k}}^{2}(Q_{j}^{**})}}$ as small as possible. You write in lines 150-151 that minimizing this expression can "give rise to partitions that yield estimators with large variance." It is not at all clear that your suggested regularizer avoids this pathology.  


questions: * Can you prove hat regularizing with \hat{H} instead of the sum of squared biases actually results in estimators with smaller variance?If not, you should provide, at a minimum, more justification for not using the sum of squared biases (i.e. the righthand side of Equation 4) as the regularizer in Equation 12. Better would be an empirical demonstration that \hat{H} yielded better point estimates or better uncertainty quantification than using the sum of squared biases

* Relatedly, why did you try to minimize the symmetric cross-entropy between $P_{t,1,j}$ and $P_{t,0,j}$? There are several other choices of distributional differences/distances/divergences. Does closeness in this distance imply closeness in moments? 

* To what extent does your framework allow treatment assignment to depend on the potential outcomes in the pre-treatment period? One can readily imagine scenarios where the decision to treat is based on past history. It would be useful to clarify explicitly whether you need to assume treatment is assigned independently of past outcomes.

* Relatedly, what specific causal assumptions do you need to make in order to establish Theorem 1? Presumably you are assuming some sort of positivity assumption and SUTVA. You likely do not assume strong ignorability (i.e.$ (Y(1), Y(0))$ and $D$ are conditionally independent given $X$) and it may be helpful to point this out.

* How does one set $\lambda$ in practice (generally) and how did you set it in your experiments? Is it just through cross-validation?

* Does your method provide any uncertainty quantification? As presented, I don't think it can easily. It may be useful to compare the proposed method to Bayesian Causal Forests, which does provide a type of uncertainty quantification while still flexibly targeting the CATE. 


limitations: The authors do not address any negative societal impacts nor can I easily imagine any for this particular work.
The authors do not explicitly mention any methodological or theoretical limitations. I would encourage them to elaborate on some ways to extend their results in the discussion, e.g., by discussing potential relaxations of specific assumptions or implementation choices.
review_title: null

Meta-review: metareview: The authors propose a novel way of dealing with unobserved confounding when having access to historical untreated data on the treated subjects and using transportability ideas and finding partitions of the covariate space that minimize the confounding bias using the historical untreated data. They introduce an interesting notion of confounding entropy and use it successfully in practice within a gradient boosted forest framework. 

Despite initial reviewer concerns, the authors rebuttal has addressed main concerns.
confidence: Certain
Decision: Accept