Predicting perturbation targets with causal differential networks

Authors: Menghua Wu, Umesh Padia, Sean H. Murphy, Regina Barzilay, Tommi Jaakkola
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: perturbation experiments, Perturb-seq, transcriptomics, causality

Abstract:
Rationally identifying variables responsible for changes to a biological system can enable myriad applications in disease understanding and cell engineering. From a causality perspective, we are given two datasets generated by the same causal model, one observational (control) and one interventional (perturbed). The goal is to isolate the subset of measured variables (e.g. genes) that were the targets of the intervention, i.e. those whose conditional independencies have changed. Knowing the causal graph would limit the search space, allowing us to efficiently pinpoint these variables. However, current algorithms that infer causal graphs in the presence of unknown intervention targets scale poorly to the hundreds or thousands of variables in biological data, as they must jointly search the combinatorial spaces of graphs and consistent intervention targets. In this work, we propose a causality-inspired approach for predicting perturbation targets that decouples the two search steps. First, we use an amortized causal discovery model to separately infer causal graphs from the observational and interventional datasets. Then, we learn to map these paired graphs to the sets of variables that were intervened upon, in a supervised learning framework. This approach consistently outperforms baselines for perturbation modeling on seven single-cell transcriptomics datasets, each with thousands of measured variables. We also demonstrate significant improvements over six causal discovery algorithms in predicting intervention targets across a variety of tractable, synthetic datasets.

Decision: reject
Reviewer scores: 6, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1729624901431

thread_type:
official_review

<thread_object>
creator: reviewer_oza9

<content>
summary:
The paper proposes a method to predict perturbation targets in biological data using the proposed causal differential networks (CDN). The idea is to infer the causal graph from observational and interventional datasets through a pre-trained model, and then learn the mapping from features to the variables targeted by interventions. The proposed method is validated on both synthetic and real-world datasets with STOA performance. However, the reliance on accurate causal graphs and labeled data presents challenges that may need to be addressed for broader applicability.

strengths:
In my view, the major contribution of this paper is to propose a potential way to scalably identify the perturbed variables, which is especially meaningful in biology tasks. 

The proposed method is abundantly validated on both synthetic and real-world datasets, demonstrating the performance of CDN is non-trivial. 

The presentation is generally good, despite a few expressions that might be confusing for readers without a biology background, e.g. DE.

weaknesses:
Supervised learning for causal discovery is an intriguing concept. While I personally believe these two ideas are somewhat contradictory, the performance of supervised causal discovery in this paper is notable. The biggest limitation so far is the lack of discussion around the pretrained causal featurizer, which will clearly influence the identified targets. The paper mentions using a fixed pretrained model. I am not sure if there is any modification of the model across different datasets. Using the same model for every dataset is not entirely reasonable. 

Another limitation is the lack of discussion of out-of-distribution. In the real-world scenario where there are no labels at all, the supervised learning-based method doesn't have any guarantee. Investigation of the performance of this scenario is appreciated and valued. (For the experiment on unseen cell lines, I am not sure I understand it correctly. Currently, I understand it as in distribution generalization rather than out-of-distribution).

questions:
- What is DE in Figure~3? 
- Can the authors elaborate more on "unseen cell lines"? Does that mean we have the same variables as "seen cell lines" but only different data samples, or even different distributions? 
- For all datasets, exactly the same pretrained model is used or there is a modification for different datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730453454712

thread_type:
official_review

<thread_object>
creator: reviewer_br1e

<content>
summary:
This work proposes a method for predicting the (set of) variables targeted by a perturbation from observations of the variables under control and perturbation conditions. The method, called CDN, first maps both datasets to embeddings that represent the interactions of the NxN variables using a pretrained (frozen) causal discovery transformer model of prior work. Given this input representation, it then trains a shallow transformer-style prediction model in classification of the perturbation targets. The approach is evaluated on several single-cell sequencing screens involving gene and drug perturbations, as well as synthetic datasets.

strengths:
The problem studied in this work is a relevant problem in the application of machine learning to biology. Contrary to several related works in the causal discovery literature, a strength of this work is its evaluation on several real-world datasets. The empirical results appear good. However, the paper does not compare to any prior specialist methods that are specifically used to predict intervention targets in biology applications, so it is hard to assess how useful the method is to the biology community.

weaknesses:
-	From the perspective of the machine learning community, the method contribution is very marginal. The idea consists of extracting edge embeddings from an existing, pre-trained transformer model, and then training a 1-layer prediction layer in (multi-hot) binary classification from these features. This is a straightforward supervised learning task, and nothing about the conception or design of the method is specifically motivated by the (biological) problem at hand. The main text of the paper does not adequately motivate that the specific architecture by Ho et al. (2020) used here, apart from arguing that the function class is sufficiently descriptive (Section 3.4).
-	In particular, framing the algorithm as based on “causality” is misleading. There is no algorithmic component underlying the approach that is rooted in a mechanistic model, intervention modeling, or a causal graph. The CDN is a simple classifier that maps Xs to Ys (interaction embeddings to whether or not variables were targeted). 
-	The formalization of what a “target” may not make much sense in the context of biology. In biological (dynamical) systems, changing conditional independencies may not be restricted to the variables targeted by a perturbation. This is because, if the true data-generating process is not a structural causal model, then interventions may not leave other conditional independencies invariant (see e.g. example by [A]). I also don’t agree with the statement that “it is common to assume” that the data was generated by a structural causal model (l. 50). This assumption is rarely made in computational biology and, for example, due to acyclicity assumptions of the causal graph, not commonly regarded a good assumption.
-	Motivation: The discussion around forward vs backward prediction in the introduction is confusing as a motivation of the task of perturbation target prediction. These are not the same task, and neither of them are more nor less suited for experimental design. In the “forward” task, we assume we know the targets. Why does that make these classes of “forward” methods worse at larger combinatorial search spaces than when predicting targets in a “backward” approach? In both cases, we must design new experiments in a combinatorial search space.
-	The “claims” in Section 3.4. on “Theoretical Context” are misleading/vacuous and do not provide any theoretical insight. It is not informative to say that the function class used to train the classifier (here, axial attention) is “well-specified” (i.e. complex enough) to map the covariate X to the target Y. A lot of function classes can do this (e.g. an MLP). The question is whether they will do so, for example, in the large sample limit, but no such questions are addressed or studied here.

[A] Alejandro Tejada-Lapuerta, Paul Bertin, Stefan Bauer, Hananeh Aliee, Yoshua Bengio, Fabian J Theis: Causal machine learning for single-cell genomics

questions:
-	How confident can we be in the ground-truth “targets” used for the drug perturbation datasets for the evaluation? 
-	“To ensure high quality labels, we filtered perturbations to those that induced over 10 differentially-expressed genes” (l. 294). This does not make the **labels** more high quality. If anything, it ensures that the considered perturbations are strong. But whether or not the target labels are accurate does not depend on the variable selection. Please provide some more information on the quality of the labels.
-	In the description of the correlation metric, what does this sentence mean?: “[…] similarity between the (ground truth) mean log- fold change of the top prediction that was observed as a perturbation, and that of the actual perturbation (given as input)” (l. 335). Specifically, what is the “top prediction that was observed as a perturbation”?
-	“LINEAR and MLP take as input the mean expression of all perturbation targets, plus the top 2000 highly-variable genes” Since the perturbation targets are what need to be predicted, what does it mean here to take as input the mean expression of all targets, which are supposedly unknown? Moreover, what does it mean to “take as input a gene”? Do you mean the mean expression of that gene again? I don’t think this baseline makes sense, because it is impossible for a method to predict the **interactions/targets** based on the mean of the variables, without any covariance/correlation information. It would make much more sense to perform classification based on either correlation/covariance information of all NxN variables or the pre-trained edge embeddings used by CDN.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730548423826

thread_type:
official_review

<thread_object>
creator: reviewer_kqt8

<content>
summary:
The authors tackle the problem of identifying the intervention target from two (namely, observational and interventional) datasets on e.g., single cells. Supervised causal learning approaches are used to estimate two structures separately for each dataset, and then target is identified by comparing the two estimated structures.

strengths:
- The use of supervised causal discovery in biology field is interesting -- though without solid asymptotic guarantee, supervised method usually impose priors about the domain to the model (which is exactly the case for biology data; while the priors can usually be hard to formalize) to improve the empirical performance.

- The experiments are relatively comprehensive.

weaknesses:
1. The motivation is quite unclear, making the contribution also questionable:
   - To me the problem of identifying intervention target (at least in the presence of latent variables) is too easy and standard: theoretically one variable is intervened iff given any other set of variables, its conditional distribution changes between the two distributions, i.e., there is an direct edge between the domain index variable and it.
   - The authors claim that existing methods "scale poorly", "must search combinatorial spaces": I don't quite get that. According to above, one only need to locally learn the Markov boundary of the domain index variable (which is a binary variable 0/1 indicating for each sample which dataset it is from in the pooled dataset). Since we usually assume that the domain index is a root variable (interventions are applied exogenously), such MB consists of the children and spouses, and then some trivial CI tests within the MB can reveal the true children. The MB can be learned very efficiently, using e.g., graph LASSO or its nonlinear variants.
   - Speaking of efficiency, instead, what the authors proposed (to learn two graphs separately, and then compare the two only for learning the intervention target) seems too overkill.
   - So could the authors give more explanations for where and how the proposed method is better than existing ones?

2. For a identifiability guarantee, a formal list of assumptions (e.g., which kind of faithfulness is needed) are expected.

3. More details and insights about the usage of supervised model is needed:
   - The choice of supervised learning isn't merely due to its availability. We choose it because it can leverage abundant data from either real world or synthetics, and hopefully gain some prior knowledge (which can be hard to formalized) about the domain to the model. So, more elaboration on these are expected: how authors simulate the data or collect real-world data for this specific case, for the model's reliability and generalizability?
   - Also, how are the two graphs learn separately? Are there any constraints to ensure they don't vary too much? How to solve the conflicts between the two learned graph, e.g., some difference that cannot be explained by any intervention?
   - Please explain why it is necessary to learn two graphs separately. It appears way too overkill.

4. More literature review are expected (especially on the early developments):
   - For (generally) identifying changes:
     - Kevin D Hoover. The logic of causal inference: Econometrics and the conditional analysis of causation.
Economics & Philosophy, 6(2):207–234, 1990.
     - Jin Tian and Judea Pearl. Causal discovery from changes. arXiv preprint arXiv:1301.2312, 2001.
     - Whitney K Newey and James L Powell. Instrumental variable estimation of nonparametric models. Econometrica, 71(5):1565–1578, 2003.
     - Kevin B Korb, Lucas R Hope, Ann E Nicholson, and Karl Axnick. Varieties of causal intervention. In PRICAI 2004: Trends in Artificial Intelligence: 8th Pacific Rim International Conference on Artificial Intelligence, Auckland, New Zealand, August 9-13, 2004. Proceedings 8, pp. 322–331. Springer, 2004.
      - Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research, 21(89):1–53, 2020.
      - Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. Causal discovery from soft inter- ventions with unknown targets: Characterization and learning. Advances in Neural Information Processing Systems, 33:9551–9561, 2020.
   - For supervised causal discovery:
      - Isabelle Guyon. Cause-effect pairs kaggle competition, 2013. URL https://www. kaggle. com/c/cause- effect-pairs, pp. 165, 2013.
      - Li, Hebi, Qi Xiao, and Jin Tian. "Supervised whole dag causal discovery." arXiv preprint arXiv:2006.04697, 2020.
      - Dai, Haoyue, et al. "Ml4c: Seeing causality through latent vicinity." Proceedings of the 2023 SIAM International Conference on Data Mining (SDM). Society for Industrial and Applied Mathematics, 2023.

questions:
See in "weaknesses".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668319665

thread_type:
official_review

<thread_object>
creator: reviewer_tn1g

<content>
summary:
The authors focus on the problem of Identifying the direct targets of interventions in single cell perturbation data (perturb-seq). The challenge consists in filtering solely the direct targets out of a large number of downstream affected genes. The proposed approach, namely causal differential networks (CDN), first extract the causal networks underlying unperturbed and perturbed data. The networks are then compared with an axial-attention based classifier, so that to identify nodes (genes) that are the direct targets of the intervention. Finally, the authors assess CDN's performances on five perturb-seq datasets, as well as on data from two chemical perturbation studies.

strengths:
This work addresses an interesting problem, i.e., the identification of direct targets in perturbation experiments.  The manuscript is well structured and written, and an extensive evaluation is presented both on real-world and simulated data.

weaknesses:
The novelty of the CDN method is debatable, being the direct combination of two other already published algorithms, i.e., an amortized causal discovery model (Wu et al.,2024) and the axial attention-based classifier (Ho et al., 2020).

Interventions may directly affect genes other than their intended targets (off target effects). However, it is not clear if this issue was taken into consideration during the analysis of real-world data.

Experiments would benefit from a comparison against a trivial classifier, so that to establish a truly minimal baseline.

questions:
I would like to ask the authors to address the weaknesses highlighted above:
- please elaborate further on the novelty of your proposed approach. Did you devise any algorithmic improvement? Or is CDN solely the combination of previously published algorithms?
- Can you explain whether the current analysis setting would be robust against biases introduced by off-target effects?
- As reported in Table 6, the direct target of many interventions can be trivially identified as the gene with the largest fold-change in expression. It would be useful to include a trivial classifier in all computational comparisons that simply considers the first n genes with the largest log fold change as direct targets (n is a user-set parameter). Such a classifier would help gauging what additional contribution complex causal approaches provide with respect to a very simplistic solution.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732038829221

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Note to all reviewers

comment:
Dear reviewers,

Thank you for reading our paper and providing thorough feedback. We would like to highlight three aspects that we believe are central to the motivation behind this work.

1. Biological understanding has long been a motivation for causal modeling [1]. In this work, we aimed to bridge the gap between classical ideas and pragmatic application. Specifically, our primary focus was to facilitate experimental design and biological interpretation, by predicting variables responsible for changes to cellular systems. This concept is naturally realized within a causal discovery framework, which while imperfect, provides useful inductive biases for approaching the problem.

[1] Sachs et al. Causal Protein-Signaling Networks Derived from Multiparameter Single-Cell Data. Science. 2005.

2. Identifying unknown intervention targets has been solved in many theoretical settings, but it remains a challenge and is underexplored in real applications. As we note in Section 2.2, this task has been studied extensively in the causality literature. However, guarantees (e.g. of identifiability) often require infinite data, with parametric assumptions regarding the noise and/or causal mechanisms. In single-cell perturbation experiments, data are limited, while graphs are large, and assumptions unknown. To address these issues, we draw inspiration from amortized / supervised causal discovery. These pretrained models, coupled with our attention-based architecture, achieves state-of-the-art performance across a number of real datasets.

3. We envision several practical applications of this work, including not limited to the following.
- It is widely reported that the first genome-wide Perturb-seq experiment cost $1M+ [2]. While costs are decreasing, new experiments must still be conducted for *each* new cell line (e.g. different cancers). Our model (CDN) offers a way to prioritize perturbations to screen for unseen cell lines, while learning from past screens.
- Despite clinical approval, a large number of drugs still lack [clear mechanisms of action](https://en.wikipedia.org/wiki/Category:Drugs_with_unknown_mechanisms_of_action). CDN is complementary to existing approaches for elucidating mechanisms and potential targets.

[2] Replogle et al. Mapping information-rich genotype-phenotype landscapes with genome-scale Perturb-seq. Cell. 2022.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a causal differential network for identifying intervention targets in single-cell perturbation data, combining causal discovery and an axial attention-based architecture. While addressing a relevant biological problem, reviewers have concerns about the limited methodological novelty, reliance on existing methods, and robustness to off-target effects and out-of-distribution data. Although the authors clarified design choices and provided detailed rebuttals, key issues regarding scalability, reliance on accurate causal graphs, and comparisons to simpler baseline methods remain unresolved. Overall, the work demonstrates potential but lacks sufficient innovation and clarity to justify its impact.