SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we provide additional evaluations of the experiments on the Perturb-seq dataset. The computation of RMSE are computed for individual interventional distributions. The computation of  $R^2$  (we capped the minimum by 0 to avoid overflow) records the coefficient of determination by regressing the mean of the generated samples on the ground-truth distribution mean.
G.1 Single-node interventions
Figure 9 shows the same visualization as Figure 5 in the main text for the remaining  $11 = 14 - 3$  single target-gene interventions with more than 800 cells. Figure 10 presents this side-by-side for the training samples. For the entire 105 single interventions, we visualize for each individual intervention the empirical MMD between the generated populations and ground-truth populations in Figure 11, where the bars record the MMD in different batches.
Figure 9: For single-node interventions, the distribution of generated test samples visually mirrors the distribution of the actual samples. A UMAP visualization of 11 single target interventions shows that the generated and the actual distributions closely match.
Figure 10: For single-node interventions, the distribution of generated training samples visually mirrors the distribution of the actual samples. As with the test samples, the distributions of the generated training samples closely match the actual distributions.
Figure 11: For single-node interventions, the distribution of generated training samples quantitatively mirrors the distribution of the actual samples. The figure shows the empirical MMD, defined in Appendix C.1, between the generated populations and ground-truth populations for 105 single target-node interventions.
G.2 Double-node interventions
We plot the generated samples for 11 random double target-gene interventions in Figure 12. In Figure 13, we highlight two interventions for which the generated samples differ from the actual samples. The plots for all 112 interventions are provided at https://github.com/uhlerlab/discrepancy_vae.
- Actual Cells
- Generated Cells
- NA
Figure 12: UMAP visualization for a random sampling of double-node interventions. Compared to single-node interventions, the generated samples of the double-node interventions match only for certain pairs.
Figure 13: For some double-node interventions, the generated samples match the actual samples, and for some combinations they do not. The model accurately predicts the effect of the combinations ETS2+CEBPE and SET+IRF1, but does not accurately predict the effect of ETS2+DUSP9.
The MMD losses for all 112 interventions are summarized in Figure 14. Similar to Figure 6 in the main text, Figure 15 shows the distribution of RMSE and  $R^2$  of the 112 interventions.
We remark here that this task has also been studied in previous works (e.g., [39, 8, 68, 49]) with different setups. Formally benchmarking the empirical results under a unified setting would be of interest in future works.
Figure 14: For some double-node interventions, the distribution of generated samples quantitatively mirrors the distribution of the actual samples. The figure shows the empirical MMD, defined in Appendix C.1, between the generated populations and ground-truth populations for 105 single target-node interventions.
Figure 15: Our model accurately predicts the effect of many double-node interventions. 'All genes' indicates measurements using the entire 5000-dimensional vectors; 'DE genes' indicates measurements using the 20-dimensional vectors for the top 20 most differentially expressed genes.
G.3 Structure Learning
In Figure 16, we show the learned latent structure between gene programs, along with descriptions of each gene program.
Figure 16: Regulatory relationships between programs learned in  $\mathcal{G}$  and full list of genes in each program.
H Extended Experiments
In this section, we provide additional experimental results. First, we perform ablation studies of different components of the proposed architecture on biological data. Then, we provide a simple simulation study to examine the performance of the framework on different tasks.
H.1 Ablation Studies
For the ablation studies of different components, we compared the performance of our final model (depicted in Figure 4) against three alternative versions. All models are trained with the same setting (data split, schedule, learning rate, etc). In particular, we compared against
- Models without the discrepancy loss. These models learn the distributions similar to conditional VAE [51], where both an interventional sample and its interventional label are fed in to learn the exogenous  $Z$ . Then inside the latent space, we use the same causal layer as our model to generate a virtual sample. During inference, we can generate interventional samples via two approaches. One is sampling the exogenous  $Z$  from  $p(Z)$  and decoding. The other is sampling an observational sample, obtaining its exogenous  $Z$  using the encoder
then decoding. These two approaches correspond to the second and third rows of Table 3 respectively.
- A model without the causal layer. This model uses a similar workflow as our final model in Figure 4, where we do not use a causal-based decoder but a simple MLP decoder. This corresponds to the fourth row of Table 3.
We note that the encoder, decoder, DSCM, and intervention encoder are needed to learn distributions and the latent causal graph from this setting where observational and interventional data are present.
For the metrics, we report both MMD and  $R^2$  in Table 3. However, MMD is more meaningful as we are assessing the quality of generating a distribution. We observe that models without discrepancy perform much worse due to mode collapses, whereas the sampling approach using observational data performs slightly better. Our final model works the best in general; however on the MMD for double-node interventions, the version without a causal layer seems to work slightly better. This is potentially because some double-node interventions that act non-additively can be captured better without imposing the structure.
Method	MMD (single)	R2 (single)	MMD (double)	R2 (double)
ours	0.324±0.007	0.986±0.001	0.432±0.006	0.978±0.001
ours w/o discrepancy	2.966±0.054	0.984±0.003	3.358±0.031	0.972±0.002
ours w/o discrepancy (obs)	2.965±0.054	0.984±0.002	3.355±0.030	0.972±0.002
ours w/o causal layer	0.348±0.009	0.982±0.002	0.427±0.006	0.978±0.002
H.2 Simulation
For the simulation study, as a proof-of-concept, we tested on a simple 5-node graph, where we generate 2048 samples in each of the 5 interventional datasets. We map this to a 10-dimensional observation space, where we pad zeros to the additional dimensions. This ensures clear visualization of the generated samples in Figure 17, where we compare the zero-shot learned double-node interventional samples against ground truth. In Table 4, we report the quantitative metrics. In addition to the MMD on left-out single and double-node interventions, we also report the training MMD and Structural Hamming Distance (SHD) of the learned graph.
Due to the combinatorial nature of learning a DAG and the small sample sizes in this setting, we observe that the learned intervention targets can be quite sensitive to initializations. Therefore during evaluation, we report the metrics while fixing the intervention targets to be of different transposition distances to the true targets. For single-node generations, different transposition distances return similar results, meaning that the model is expressive enough to learn these distributions, although we observe that the result with zero transposition distance is marginally better. This also holds during training, which can potentially be used as model selection to overcome the initialization issue. For double-node extrapolation, the result with zero transposition distance shows a larger benefit, as expected from our theory.
Table 3: Ablation studies. We report testing metrics and their standard error on the biological datasets. The results on single-node interventions are computed over 14 interventions. The results on double-node interventions are computed over all 112 interventions.
Transposition Distance	MMD (training)	MMD (single)	MMD (double)	SHD
0	0.030±0.007	0.047±0.008	0.041±0.004	2
1	0.057±0.028	0.058±0.030	0.181±0.048	6
10	0.042±0.007	0.041±0.009	0.119±0.023	11
Table 4: A simple simulation study. On a 5-node DAG, we test the model performance with varying transposition distances of the identified intervention targets. For sample generations, we report MMD and its standard error. The training metric is evaluated on all single-node interventions, where the third and forth rows are evaluated based on held-out samples of single and double-node interventions.
Figure 17: An illustration of double-node intervention extrapolation in simulation. We visualize 16 samples of the double-node intervention on nodes 2, 3. The generated samples are shown on the left, where the ground-truth samples are shown on the right.
I Extended Discussion
I.1 Limitations and Future Work
This paper opens up several directions for future theoretical and empirical work, which we now discuss.
Theoretical Perspective. We have focused on the setting where a single-node intervention on each latent node is available, similar to prior works on causal disentanglement [3, 53]. However, we highlight three issues in this setup and discuss potential remedies. First, by assuming access to data from intervening on every single latent node, we inherently possess partial knowledge of all the latent variables, even though we are unaware of their specific values or whether multiple interventions act on the same variable. The setups that do not assume interventions but the existence of anchored observed variables (i.e., variables with only one latent parent) [19, 9, 64, 65] face the same issue. This assumption can be unsatisfying in the context of causal representation learning, where the causal variables are assumed to be entirely unknown. Second, it may be impossible to intervene on all latent causal variables, especially in scenarios involving latent confounding. For instance, in climate research, it might be impossible to intervene on a variable like the precipitation level in a particular region. Finally, the assumption of single-node interventions can be overly optimistic in many applications. For example, in the case of chemical perturbations on cells, it is known that drugs often target multiple variables.
Nevertheless, the results obtained in the current setup can serve as a foundation and stepping stone towards the ultimate goal of general causal representation learning. On one hand, our analysis showed what can be learned from each intervention. This is helpful when considering cases where only a subset of the latent causal variables can be intervened on. On the other hand, the key techniques employed in our proofs can be extended to the multi-node setting. Specifically, in the latent space, one should expect only the marginals of variables downstream of a multi-node intervention to change.
Moreover, we have primarily focused on the infinite data regime for analyzing identifiability. Considering the expensive nature of obtaining interventional samples in practice, there is ample room for further investigation concerning sample complexity. Aside from the feasibility of identifiability, many applications are concerned with specific downstream tasks. Full identification of the underlying causal representations provides a comprehensive understanding of the system and would be beneficial for multiple downstream tasks. However, in certain cases, full identification may be unnecessary or inefficient for a particular task. Therefore, it is of interest to develop task-specific identifiability criteria for causal representation learning.
Empirical Perspective. We make two remarks on the VAE framework proposed in this work. First, as shown in our experiments in Section 6, our proposed framework can still be applied in settings with multi-node interventions and fewer single-node interventions. For instance, one can model multi-node interventions by reducing the temperature in the softmax layer. Second, due to the permutation symmetry of CD-equivalence, we impose an upper-triangular structure on the adjacency matrix in the deep SCM and learn the intervention targets. Alternatively, when there is exactly one intervention available for each latent node, one can instead prefix the intervention targets and learn the adjacency matrix. Specifically, we can set the intervention targets of  $I_{1}, \ldots, I_{p}$  to be
a random permutation of  $[p]$ . Subsequently, the adjacency matrix can be learned for example via the nontears penalty [70] to enforce acyclicity. However, both methods inherit the combinatorial nature of learning a DAG, and therefore their performance may require large sample sizes and can be sensitive to initialization [27]. Consequently, endeavors to improve the optimization process and robustness of such models would be valuable.
I.2 Discussion of Contemporaneous Works.
This work is concurrent with a number of other works in interventional causal representation learning. Unless otherwise noted, all of these works consider single-node interventions, as we do in this paper. Most similar to our setting is [59], which studies identifiability of nonparametric latent SCMs under linear mixing. They consider the case where exactly one intervention per latent node is available, which is an easier setting as we discussed in Section 2. In that setting, they provide a characterization of the learned causal variables. On the other hand, [7] studies identifiability of a linear latent SCM under nonparametric mixing. They also consider both hard and soft interventions, but in the form of linear SCM with additive Gaussian noises. Three concurrent works [25, 60, 35] consider both nonparametric SCMs and nonparametric mixing functions: [60] prove identifiability for the case of  $p = 2$  latent variables when there is one intervention per latent variable. They provide an extension to arbitrary  $p$  for settings where there are paired interventions on each latent variable. Meanwhile, [25] consider arbitrary  $p$ , without paired interventions. However, they use only conditional independence statements over the observed variables  $X$  to recover the latent causal graph. As a result, their identifiability guarantees place restrictions on the latent causal graph, unlike the other works discussed here. The third work [35] studies the Causal Component Analysis problem, where the latent causal graph is assumed to be known. Finally, we note that other concurrent works study causal representation learning without interventional data [40, 32] or with vector-valued contexts instead of interventions [31].