Learning Representations of Instruments for Partial Identification of Treatment Effects

Authors: Jonas Schweisthal, Dennis Frauen, Maresa Schröder, Konstantin Hess, Niki Kilbertus, Stefan Feuerriegel
Primary area: causal reasoning
Keywords: causal inference, partial identification, instrumental variables, treatment effect, representation learning

Abstract:
Reliable estimation of treatment effects from observational data is important in many disciplines, such as medicine. However, estimation is challenging when unconfoundedness as a standard assumption in the causal inference literature is violated. In this work, we leverage arbitrary (potentially high-dimensional) instruments to estimate bounds on the conditional average treatment effect (CATE). Our contributions are three-fold: (1) We propose a novel approach for partial identification through a mapping of instruments to a discrete representation space so that we yield valid bounds on the CATE. This is crucial for reliable decision-making in real-world applications. (2) We derive a two-step method that learns tight bounds using a tailored neural partitioning of the latent instrument space. As a result, we avoid instability issues due to numerical approximations or adversarial training. Furthermore, our procedure aims to reduce the estimation variance in finite-sample settings to yield more reliable estimates. (3) We show theoretically that our procedure obtains valid bounds while reducing estimation variance and we perform experiments to demonstrate the effectiveness across various settings. Overall, our procedure offers a novel path for practitioners to make use of potentially high-dimensional instruments (e.g., as in Mendelian randomization).

Decision: reject
Reviewer scores: 8, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730294904154

thread_type:
official_review

<thread_object>
creator: reviewer_epq5

<content>
summary:
This paper introduces a method for estimating treatment effects from observational data despite violations of unconfoundedness. It presents a two-step procedure using high-dimensional latent instruments to derive valid bounds on the Conditional Average Treatment Effect.

strengths:
1.	The study addresses an intriguing and valuable problem in causal inference.

2.	The writing is clear and accessible, making complex concepts easier to understand.

weaknesses:
1. The method appears to be a straightforward combination of existing techniques, lacking clear theoretical innovation.  

2. Although the paper highlights the advantage of utilizing high-dimensional instruments, there are limited experiments involving such instruments in the evaluation phase.  

3. The experimental results show no substantial improvements, casting doubt on the method's effectiveness.  

4. While the authors discuss the lack of comparison implementations, it would strengthen the argument to adapt baseline methods to the current setting and compare their performance with that of the proposed method, making the results more persuasive.

5. The final loss function includes two hyperparameters, making it crucial to conduct a sensitivity analysis to evaluate the method's robustness. However, the absence of such experiments raises concerns about the reliability and generalizability of the proposed approach.

6.  The assumption that all observed variables are confounders may be stringent and difficult to satisfy in practical scenarios.

7. The two-stage method presents several limitations, such as potential inefficiencies in estimation and sensitivity to the parameters selected at each stage, which may compromise the overall robustness of the results.

questions:
See Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730354567089

thread_type:
official_review

<thread_object>
creator: reviewer_bx94

<content>
summary:
Reliable estimation of treatment effects from observational data is crucial in many disciplines and is a challenge when unconfoundedness assumption in the causal inference literature is violated. In this paper, the authors leverage arbitrary instruments to estimate bounds on the conditional average treatment effect (CATE). First, the authors propose a novel approach for partial identification by mapping instruments to a discrete representation space such that valid bounds on the CATE are yielded. Second, the authors derive a two-step procedure that learns tight bounds using a tailored neural partitioning of the latent instrument space. Thus, instability issues due to numerical approximations or adversarial training are avoided. Third, the authors provide theoretical studies on the proposed approach. Finally, the authors perform extensive experiments to demonstrate the effectiveness across various settings.

strengths:
This paper investigates the conditional average treatment effect (CATE), assuming the presence of unobserved confounders. Further, researchers encounter complex instrument information (e.g., continuous/high-dimensional), which makes the computation hard in many circumstances. The authors aim to develop estimators that can provide reliable analysis of the above assumptions. Unlike the existing literature, the authors establish bounding estimators of the target quantity and provide theoretical studies of the constructed estimators. A merit of the paper is that it is well-written, so readers can easily follow its notions.

weaknesses:
There are some notational problems in the paper. For example:
1. Notational problems in Eqn. (10).
2. According to Eqn. (8) (or Eqn. (14)), $\phi$ should be a representation map, and $\Phi$ should be a set of all representations. It is quite confusing to write $\Phi(Z)=l$ (e.g., see line 323). 
3. Please also state the explicit form of the constants $c$, $d$ in Theorem 2.
Please recheck the notations to avoid any confusions.

questions:
Marks will be adjusted according to the replies. Below are some feedback/questions:
1. What is the point of building bounding estimators? As mentioned in the paper, existing literature focuses on point estimators. Indeed, we can build the confidence interval based on the point estimator. It is well-known that these are two concepts, but it is odds to build bounding estimators present in the paper.
2. Theorem 2 gives the asymptotic properties of the nuisance estimators $\hat{\mu}$ and $\hat{pi}$ when $\phi$ is arbitrary. It would be better to study the asymptotic properties of the nuisance estimators due to optimal $\phi^*$ after Eqn. (8).
3. It is strange to have $N_b$ in Algorithm 1 (see lines 387, 388).
4. The auxiliary guidance loss is not clear enough. It would be better if the authors could provide an explicit formula for the loss similar to bandwidth minimization loss and regularization loss.
5. The size of $k$ should affect the bounded estimators. Please demonstrate a scientific way to choose $k$. In the existing experiments, authors study results for different $k$. In any case, it seems that increasing $k$ would give better results. It should not be correct in general.
6. Could you provide the results of the following experiment:
assuming linear generative model, nonlinear ML methods are used to estimate the nuisance functions and then construct the corresponding bandwidth;
assuming a nonlinear generative model, linear ML methods are used to estimate the nuisance functions and then construct the corresponding bandwidth.
7. Would the constructed bounding estimators be doubly robust?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730503816380

thread_type:
official_review

<thread_object>
creator: reviewer_ewm3

<content>
summary:
This paper proposes a partial idetification approach of the conditional average treatment effect (CATE) that allows for arbitrary instruments. In particular, the proposed appraoch first maps the intstruments to a discrete representation space, and then leverages partial identification results for discrete intruments to bound the CATE. The representation mapping is learned by minimizing the length of the bounding interval while stabilizing finite sample performance. The authors also provided a neural implementation of the proposed method and performed simulation experiments.

strengths:
- I appreciate the clear comparison with existing works. 
- The paper provided clear description of a neural implementation of the proposed approach, and there is some innovation in the custom loss function.

weaknesses:
Overall, the theoretical contribution of the paper appears limited (some of them are very straight forward applications of the existing results for discrete instruments) and may benefit from further exploration.
- The paper only provided finite samle guarantees for the second stage nuisance functions, but not for the final bounds, which might be more useful (for e.g. guaging the quality of the estimation) and easier to interpret. For instance it would be nice to have some theoretical results on the coverage (or scaled coverage - coverage/length of interval) of the estimated bounds.
- The number of partitions ($k$) is an important hyperparameter, but the paper did not provide anytheoretical results on how the estimation error or the tightness of the bounds would scale with $k$.

questions:
- Even the experiements in the high dimensional settings had only 20 dimensional instruments. I am not familar with Mendelian randomization with SNPs, so I wonder if this falls into the normal range of such applications. 
- Is the bound from Theorem 1 sharp?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730832434119

thread_type:
official_review

<thread_object>
creator: reviewer_hfwx

<content>
summary:
This paper provides a method for the partial identification of treatment effects when potentially high-dimensional instruments belonging to a broad class of domains are available. To overcome the potential of overly conservative bounds, it proposes a discrete embedding scheme for the instruments. It provides theoretical results for the validity of the proposed bounds and for estimator efficiency. It provides an empirical evaluation of the method using a well motivated simulated data generating process.

strengths:
This paper, to my knowledge, provides a novel solution to a well motivated problem. It provides thorough theoretical results and adequate empirical evaluation. The writing quality and organization is excellent. The availability of high dimensional instruments is common in many settings beyond the medical domain, so I believe the potential for impact is high.

weaknesses:
A discussion on potential applications outside of medicine may help to increase the visibility of this work. Perhaps by extending Appendix B.

questions:
I have no further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732100229620

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you very much for the constructive evaluation of our paper and your helpful comments! We addressed all of them in the comments below. Furthermore, we uploaded an updated version of our paper and marked all significant changes to the original version in **blue color**. 

Our **main improvements and clarifications** are the following:
- **Novel contributions**: We clarified our contribution as compared to previous work and now explicitly spell out how our work is novel.\
(i) We show that we can transfer the bounds for discrete instruments to our more complex setting such that they are valid for arbitrary mappings $\phi$ and clarify that also our optimization objective of learning representations to learn tight bounds has not been considered before.\
(ii) We elaborate on how _our Theorem 1 is novel and non-trivial_. Our Theom 1 allows for more effective, stable, and robust training by avoiding a straightforward implementation of our objective that would require alternating learning.\
(iii) We further explain how our finite sample perspective from Theorem 2 does not only focus on guarantees of the final bounds. Instead, we demonstrate that it is used to motivate our regularization loss to further robustify the bound evaluation already during training of $\phi$, which then also results in robust estimation of the final bounds.

- **Better writing & more details**: We **improved the presentation and motivation** of different parts in our paper:\
(i) We improve inconsistencies in our notation and explicitly add expressions such as constants in the proofs and the form of the auxiliary loss\
(ii) We discuss the role of the parameter $k$ and show that our method is evidently more robust against suboptimal selection against the baseline. Further, we added guidelines on how to select $k$.\
(iii) We improve the presentation of our experimental results. For that, we display new metrics focusing on the tradeoff between tightness and reliable estimation to further point out the robust estimation and clear performance gains of our method.

- **New experiments**: We **added new experiments to show the robustness of our method in various challenging settings**. Specifically, we provide new experiments in our **new Appendix E** for:\
(i) a new data set with 100-dimensional instruments (see **Appendix E.2**) including new baselines (see **Appendix E.1**),\
(ii)  ablations with different DGPs and models for nuisance estimation (see **Appendix E.3**), and\
(iii) sensitivity analysis over $\lambda$ and $\gamma$ (see our **Appendix G**). In all experiments, _our method performs robustly over varying scenarios_, showing the added value of our approach for settings when instruments are continuous or potentially high-dimensional.

We highlight all changes in the **updated PDF** in **blue color**. We specifically incorporated all changes (marked with **Action**) into the updated version of our paper. Given these improvements, we are confident that our paper provides valuable contributions to the causal machine learning literature and is a good fit for ICLR 2025.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733154861740

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you again for your thoughtful feedback and engagement. We are confident that we have addressed all the points raised in your reviews and incorporated the suggested improvements into the revised manuscript in blue and red color.

We noticed that some reviewers have not yet responded to our latest updates and responses. Please let us know if there are any remaining questions or concerns — we would be more than happy to clarify further. Otherwise, if you feel our responses have addressed your concerns satisfactorily, we would kindly appreciate it if you would consider updating the rating.

We appreciate your time and effort and look forward to any additional feedback you might have.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the problem of bounding treatment effects (i.e. partial identification) when there is unobserved confounding in observational data. Here, the paper leverages instruments to estimate bounds on the conditional average treatment effect (CATE). This is done by first transforming instruments into a discrete representation space via the Gumbel Softmax trick followed by a two-step procedure to learn bounds on the CATE by partitioning the representation space. The work proposes using a loss function to learn the mapping by simultaneously minimizing the length of the bounding interval while stabilizing finite sample performance. The methodology was tested on simulated data. 

Both the problem setting and the idea are interesting; but at the end of the discussion period this came down to a borderline decision. I closely followed the discussions and rebuttal. I think the authors did an excellent job of responding to the reviewers' questions but my reading of the discussion and responses was that the reviewers felt the manuscript in its current form read more like a proof of concept rather than a strong standalone piece of work either theoretically or empirically. Such a paper is reasonable but the reviewers did not find common ground on whether the idea alone, with simple experiments, sufficed for publication.

The authors clarify that their their major goal is "*not* to derive new identification bounds or new theoretical results for finite sample guarantees. Instead, the main contribution is to "provide a novel stable learning procedure in an important widely overlooked setting and to show its strong performance". Section 6 in the paper starts off by saying that no fair comparison is possible because there is not other work tailored to this setting. To that end, one suggestion to strengthen the manuscript is with further incorporation of baselines and empirical results in similar, even if not identical settings. For example, Appendix D1 in https://arxiv.org/abs/2202.10806 contains a semi-synthetic experiment based on Mendelian randomization that could form a point of comparison to the work. Similarly, the line of work on partial identification with neural causal models (https://arxiv.org/abs/2405.15673v2) also contains experiments on binary data (e.g. Table 1 in Appendix E https://arxiv.org/abs/2210.08139); this too could form yet another point of comparison. These would strengthen the claims around the efficiency of the estimator. Theoretically, an understanding of how the hyperparameter k influences the bounds would also improve the work. I encourage the authors to resubmit with improvements along the aforementioned axes.