Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of estimating causal effect with overlap violations. The authors focus on a specific application---causal estimation for text data. The authors start with writing down the model in terms of a DAG and discuss identification results based on the DAG. The authors then proceed with discussing estimation strategies; they propose an outcome only estimator and a doubly robust TI-estimator. They further establish a theorem that allows inference on the proposed TI-estimator. Finally, the authors demonstrate the performance of the proposed estimator through simulations and real data analysis.
strength_and_weaknesses: Strengths

-	The paper is very well-written. The proposed method is motivated and explained in a very nice way. 

-	The paper addresses an interesting and important question: causal estimation with overlap violations.

-	The proposed method is very elegant conceptually. 

-	The proposed method appears to have good performance both theoretically and empirically. 

Weaknesses

-	The identification result is not new: it is the same as using prognostic score as deconfounding score in D’Amour & Franks (2021). See the next section for more details. 

clarity,_quality,_novelty_and_reproducibility: See the next section for more details. 
summary_of_the_review: Overall, I find this paper a good contribution to the literature. Here’re some comments/questions I have. 

-	Correct me if I’m wrong, but the identification result is not new: think about the line of research in prognostic score. It will be helpful if the authors can discuss the connection of the paper to the prognostic score literature. The general theme of the proposed method is the same as using prognostic score as deconfounding score in D’Amour & Franks (2021). The authors discuss briefly on page 3 the differences between this paper and D’Amour & Franks (2021). To me, the paper is still a good one with an interesting application---text data and a nice neural network model, but emphasizing more on the differences and new contributions can help readers understand the novelty of the paper better. 

-	Assumptions in Theorem 2. Do the two rates of convergence have to be o(n^(1/4))? Is it possible for the theorem to hold true if one is smaller while the other is larger (in the sense that as long as the product is o(n^(1/2)))? Thanks!

-	It will be helpful to state the full name of “CDE”. 

-	There are some typos. Here are a few examples:

o	Page 3 line 3. “We assume there” -> “We do not assume”

o	Page 7 line 1. “It remains to given” -> “It remains to give”
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of estimating the causal effect of an attribute of a text on some outcome variable under a setting where overlap is violated, i.e., the treatment variable is fully determined by the text features. Under the assumption that the problem satisfies the constraints of a given causal model (Figure 1), the authors propose an identification formula for the target effect that adjusts for part of the text to block confounding while resolving the overlap violation. Then, they propose an estimation procedure for this formula using standard double machine learning. Empirical evaluation shows the advantage of the proposed technique over baseline work.
strength_and_weaknesses: Strengths: The evaluation of the proposed method shows a clear advantage over standard previous methods for estimation.

Weaknesses and Comments:
- CDE formulation: The authors state the following while explaining the formulation of the CDE expression on page 4: "our formal causal effect aims at capturing the effect of A through only the first, direct, path". Obviously, identifying the total effect of A on Y is different than identifying the direct effect, or at least the variant suggested in Equation 3.1. I'm missing the reasoning for why this is the purpose of the computation. Further clarification from the authors would be appreciated.

- Generality of the causal DAG (Figure 1): The model assumes the absence of confounding between $X$ and $Y$ and between $A$ and $Y$. In general, it is not clear how valid this assumption is for estimating the causal effects of attributes of a text document even if it is justified for the suggested problem of sentiment effect on sales.

- Citations: Most of the introduction states technical claims without any citation to back it. For example, the second paragraph discusses the identification of causal effects and the required conditions but there are no citations to support that.
Another example is in the following paragraph which states that "it is often reasonable to assume that text data has information about all common causes..."

- typo: p.5, "this estimator is yields".
clarity,_quality,_novelty_and_reproducibility: In general, the paper is clear and the authors attempt to explain most of the methodology. Some concerns about clarity are raised in the section above.
summary_of_the_review: The score reflects the concerns raised regarding the assumptions in the causal model and the derivation of the CDE expression which are discussed in the weaknesses section. I would be happy to revisit the score based on further clarification from the authors.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a way to overcome the violation of positivity in causal inference with text data. Following the problem setup of Pryzant et al. (2020), a piece of text X contains both the treatment (assumed binary) and confounders. The authors find a sufficient statistic $\eta$ that prevents the violation of positivity. Another contribution of this paper is to combine the approach with the double machine learning idea and obtain easy valid confidence intervals.
strength_and_weaknesses: ### Strengths
* The problem is this paper is quite practical and interesting.
* Theorem 1 seems to be clever. It finds a sufficient statistic that blocks the paths that lead to violation of positivity.
* The paper is clear and easy to follow.

### Weaknesses
* Theorem 2 is a rather simple application of the DML approach. The theoretical results are also straightforward corollaries based on Chernokhukov et al (2016).

### Request for clarifications:
I am confused about several parts of the paper and I would like the authors to respond before I make the final decision.

1. **The positivity issue**: Given positivity violation, the estimator of $Q(\neg A, X)$ should be quite unreliable, where I assumed that the natural pair is $(X, A)$ and $\neg A$ is the treatment that is not observed. There should be very few examples of $X$ and $\neg A$. This might lead to violations of the assumptions of Theorem 2. Can you comment on this?
2. **Beginning of Section 4.2**: Why is $\eta$ a good sufficient statistic for $\hat{g}$?
3. **End of Section 3**: The phrase "propensity score of $\eta(X)$" does not make sense.
4. **Start of Section 4.1**: The phrase "naive outcome _regression_" is the right choice of words.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow.

Most of the novelty is in Theorem 1 and the architecture of Q-Net. The paper meets the bars of novelty.
summary_of_the_review: This paper proposes a way to overcome the violation of positivity in causal inference with text data. Following the problem setup of Pryzant et al. (2020), a piece of text X contains both the treatment (assumed binary) and confounders. The authors find a sufficient statistic $\eta$ that prevents the violation of positivity. Another contribution of this paper is to combine the approach with the double machine learning idea and obtain easy valid confidence intervals.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper concerns causal effect estimation when the confounders and treatment are both derived from texts. One issue herein is that when adjusting for the entire text, there is a violation of the overlap assumption which is required for drawing valid causal conclusions. The proposed solution is to use supervised representation learning to produce a data representation that preserves confounding information while eliminating information that is only predictive of the treatment. 
strength_and_weaknesses: +1. the paper is well-written and technically correct.
+2. there are theoretical justifications for the proposed methods.
+3. the proposed method is backed up by simulations and two real data applications (Amazon reviews and consumer complaints.)
+4. the topic of the paper is increasingly important and the paper is very timely.

-1. there seems to be some small gap between the advocated main idea in Figure 2 (namely, one should eliminate the components in X that are predictive of A only) and the proposed method, which does not specifically make an effort to eliminate these components. Specifically, the authors proposed to use a three-head dragonnet where two heads predict Q0 and Q1 and one head predicts A. In the end, only the estimated Q0 and Q1 are used for making the causal effect estimation. Should I understand that the estimated A in the dragonnet represents the components in X predictive of A? If that's the case, how can we ensure that the two heads for Q0 and Q1 do not use information in X_A?  If not, how should we make the connection between the conceptual idea in Figure 2 and the implementation in Figure 3?
-2. related to (-1) above, it seems that a critical point is that Equation (3.3) Q(\tilde A, X) := E(Y | \tilde A,X) is in fact E (Y | \tilde A, X_A∧Z, X_Z). This seems to suggest that although conceptually CDE as in (3.2) should be based on X_A∧Z, X_Z which are unknown, in reality, one only needs to build models for Q(A,X) using the entire X. In this case, how does the proposed method differ from the naive method that "naively adjust for all the text as the confounding part"? Some clarification is needed.
-3. It would be helpful to add the "naively adjust for all the text" method to the simulation, to see how this method is biased and problematic.
-4. Clarification needed: following Theorem 1, the author said "We emphasize that, regardless of whether the overlap condition holds or not, the propensity score of η(X) in condition 2 is accessible and meaningful." There is no η(X) in condition 2, and since η(X) is the outcome model, it does not have a propensity score.
-5. abbreviation: On page 7, it says "AIPTW is double robust". AIPTW was not previously defined. AIPTW = augmented inverse probability T??? (treatment?) weighting? If you mean the estimator in (4.8), it should be said so.
-6. bias: about Table 1, you said "bias of the new method is significantly lower than the bias of the outcome-only estimator". Bias can be both positive and negative, and hence lower bias is not necessarily better. Do you in fact mean average estimation error or root mean square estimation error? In fact, it would be good for you to define the bias that you are reporting in the paper. 
-7. simulation. It is counter-intuitive that low confounding actually leads to a low coverage rate of the confidence intervals. In randomized trials (zero confounding), shouldn't the confidence interval be valid? Some explanation/discussion is needed.
-8. Please explain the specific differences between the current work and Claudia Shi, David M. Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. In Advances in Neural Information Processing Systems, 2019.
-9. I am not sure how novel is the proof of Theorem 2. The stated results seem to inherit from double machine learning in Chernozhukov's work. The author should comment on how their proof differs from that of Chernozhukov. This is important as this part is claimed to be the second main contribution of the paper. 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written, technically sound, and provides theoretical justifications. Its clarity can be improved in a couple of places. The authors should make a more compelling case for the paper's novelty (both in methods and theory). 

summary_of_the_review: This paper analyzes the identification conditions for causal effect estimation when there is an apparent violation of overlap. The paper then proposed to use a neural network to learn a representation of the text to predict the outcome, which is then fed into the common double machine learning method to estimate the average treatment effect for the treated. There is a seeming gap between the conceptual idea and the implementation (which should be addressed after some revision/clarification). The author should also address how their paper is novel compared to previous methods (Shi et al.) and theory (that of double machine learning).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Causal effect estimation is particularly hard when treatments and confounders are text, which are high-dimensional and may be necessary to be used in their raw form, as abstract concepts that may underly a treatment-outcome relation and its common causes will not be observed directly. This paper addresses this issue by framing it in the context of the challenges of adjusting for confounders where the overlap under different treatment levels.  

Strengths: the paper appropriately uses existing results in the literature to directly tackle a problem of practical relevance and test it with some less usual real-world dataset (not always a given in a treatment effect estimation contribution). Presentation also tries to tackle hard questions about the operational meaning of text in a causal effect estimation problem.

Weaknesses: in an abstract sense, the paper in much debt to other contributions on the lack of overlap problem, a large literature which contributes to other solutions to these problem. The manuscript acknowledges that other papers work on different assumptions that may relax some of the conditions in their presentation.
justification_for_why_not_higher_score: In an abstract sense, the paper in much debt to other contributions on the lack of overlap problem, a large literature which contributes to other solutions to these problem. The manuscript acknowledges that other papers work on different assumptions that may relax some of the conditions in their presentation.
justification_for_why_not_lower_score: The paper appropriately uses existing results in the literature to directly tackle a problem of practical relevance and test it with some less usual real-world dataset (not always a given in a treatment effect estimation contribution). Presentation also tries to tackle hard questions about the operational meaning of text in a causal effect estimation problem.
Decision: Accept: poster