SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof intuition The recent work [84] studies the special case when  $f$  is linear, and the proof is linear algebraic. In particular, they consider row spans of the precision matrices of  $X^{(i)}$ , project them to certain linear subspaces and use those subspaces to construct a generalized RQ decomposition of the pseudoinverse of the linear mixing matrix  $f$ . However, once we are in the setting of non-linear  $f$ , such an approach is not feasible because we can no longer reason about row spans of the precision matrices of  $X$ , which have been transformed non-linearly thereby losing all linear algebraic structure. Instead, we take a statistical approach and look at the log densities of the  $X^{(i)}$ . By choosing a Gaussian prior, the log-odds  $\ln p_X^{(i)}(x) - \ln p_X^{(0)}(x)$  of  $X^{(i)}$  with respect to  $X^{(0)}$  can be written as a quadratic form of difference of precision matrices, evaluated at non-linear functions of  $x$ . For simplicity of this exposition, ignore terms arising from shift interventions and determinants of covariance matrices. Then, the log-odds looks like  $\theta(x) = f^{-1}(x)^{\top} (\Theta^{(i)} - \Theta^{(0)}) f^{-1}(x)$ , where  $\Theta^{(i)}$  is the precision matrix of  $Z^{(i)}$ .
At this stage, we again shift our viewpoint to geometric and observe that  $\Theta^{(i)} - \Theta^{(0)}$  has a certain structure. In particular, for single-node interventions, it has rank at most 2 and for source node targets, it has rank 1. This implies that the level set manifolds of the quadratic forms  $\theta(x)$  also have a certain geometric structure in them, i.e., the DAG leaves a geometric signature on the data likelihood. We exploit this carefully and proceed by induction on the topological ordering until we end up showing that  $f$  can be identified up to a linear transformation, which is our main Theorem 3. Here, the presence of shift interventions adds additional complexities, and we have to generalize all of our intermediate lemmas to handle these. Once we identify  $f$  up to a linear transformation, we can apply the results of [84] to conclude Theorems 1 and 2.
On the optimality and limitations of the assumptions We make a few brief remarks on our assumptions, deferring to Appendix D a full discussion and the technical construction of several illustrative counterexamples.
1. Number of interventions: For our main results, Theorems 1 and 2, we assume that there are at least  $d$  interventions (Assumption 4). This cannot be weakened even for linear mixing functions [84]. In addition, we also show in Fact 1 that for the linear identifiability proved in Theorem 3,  $d - 2$  interventions are not sufficient. Thus, the required number of interventions is tight in Theorems 1, 2 and is tight up to at most one intervention in Theorem 3.
2. Intervention type: In the setting of imperfect interventions, the weaker identifiability guarantees in Theorem 2 cannot be improved even when the mixing is linear [84, Appendix C]. We also show in Fact 2 that if we drop the condition that interventions are not pure noise interventions, we have non-identifiability. Concretely, we show that when all interventions
are of pure noise type, any causal graph is compatible with the observations. Finally, in contrast to the special case of linear mixing, we show in Lemma 8 that we need to exclude non-stochastic hard interventions (i.e.,  $\mathrm{do}(Z_i = z_i)$ ) for identifiability up to linear maps.
3. Distributional assumptions: We assume Gaussian latent variables, while allowing for very flexible mixing  $f$ . This model has universal approximation guarantees and is moreover used ubiquitously in practice. While the result can potentially be extended to more general latent distributions (e.g., exponential families), we additionally show in Lemma 9 that the result is not true when making no assumption on the distribution of  $\epsilon$ .
5 Experimental methodology
In this section, we explain our experimental methodology and the theoretical underpinning of our approach. Our main experiments for interventional causal representation learning focus on a method based on contrastive learning. We train a deep neural network to learn to distinguish observational samples  $x \sim X^{(0)}$  from interventional samples  $x \sim X^{(i)}$ . Additionally, we design the last layer of the model to model the log-likelihood of a linear Gaussian SCM. Due to representational flexibility of deep neural networks, we will in principle learn the Bayes optimal classifier after optimal training, which we show is related to the underlying causal model parameters. Accordingly, with careful design of the last layer parametric form, we indirectly learn the parameters of the underlying causal model. Similar methods have been used for time-series data or multimodal data [27, 32] but to the best of our knowledge, the contrastive learning approach to interventional learning is novel.
Denote the probability density of  $x \sim X^{(i)}$  (resp.  $z \sim Z^{(i)}$ ) by  $p_X^{(i)}(x)$  (resp.  $p_Z^{(i)}(z)$ ). The next lemma describes the log-odds of a sample  $x$  coming from the interventional distribution  $X^{(i)}$  as opposed to the observational distribution  $X^{(0)}$ . We focus on the identifiable case (see Theorem 1) and therefore only consider perfect interventions. As per the notation in Section 3 and Appendix A.1, let  $s^{(i)}$  denote the row  $t_i$  of  $B^{(0)}$  and let  $\eta^{(i)}, \lambda_i$  denote the magnitude of the shift and scale intervention respectively.
Lemma 1. When we have perfect interventions, the log-odds of a sample  $x$  coming from  $X^{(i)}$  over coming from  $X^{(0)}$  is given by
$$
\ln p _ {X} ^ {(i)} (x) - \ln p _ {X} ^ {(0)} (x) = c _ {i} - \frac {1}{2} \lambda_ {i} ^ {2} \left(\left(f ^ {- 1} (x)\right) _ {t _ {i}}\right) ^ {2} + \eta^ {(i)} \lambda_ {i} \cdot \left(f ^ {- 1} (x)\right) _ {t _ {i}}) + \frac {1}{2} \langle f ^ {- 1} (x), s ^ {(i)} \rangle^ {2} (6)
$$
for a constant  $c_{i}$  independent of  $x$ .
The proof is deferred to Appendix F. The form of the log-odds suggests considering the following functions
$$
g _ {i} \left(x, \alpha_ {i}, \beta_ {i}, \gamma_ {i}, w ^ {(i)}, \theta\right) = \alpha_ {i} - \beta_ {i} h _ {t _ {i}} ^ {2} (x, \theta) + \gamma_ {i} h _ {t _ {i}} (x, \theta) + \langle h (x, \theta), w ^ {(i)} \rangle^ {2} \tag {7}
$$
where  $h(\cdot, \theta)$  denotes a neural net parametrized by  $\theta$ , parameters  $w^{(i)}$  are the rows of a matrix  $W$  and  $\alpha_{i}, \beta_{i}, \gamma_{i}$  are learnable parameters. Note that the ground truth parameters minimize the following cross entropy loss
$$
\mathcal {L} _ {\mathrm {C E}} ^ {(i)} = \mathbb {E} _ {j \sim \mathcal {U} (\{0, i \})} \mathbb {E} _ {x \sim X ^ {(j)}} \operatorname {C E} \left(\mathbf {1} _ {j = i}, g _ {i} (x)\right) = - \mathbb {E} _ {j \sim \mathcal {U} (\{0, i \})} \mathbb {E} _ {x \sim X ^ {(j)}} \ln \left(\frac {e ^ {\mathbf {1} _ {j = i} g _ {i} (x)}}{e ^ {g _ {i} (x)} + 1}\right). \tag {8}
$$
Note that (compare (6) and (7))  $W$  should learn  $B = D^{-1/2}(\mathrm{Id} - A)$  thus its off-diagonal entries should form a DAG. To enforce this we add the NOTEARS regularizer [102] given by  $\mathcal{R}_{\text{NOTESARS}}(W) = \operatorname{tr}\exp(W_0 \circ W_0) - d$  (see Appendix F.3) where  $W_0$  equals  $W$  with the main diagonal zeroed out. We also promote sparsity by adding the  $l_1$  regularization term  $\mathcal{R}_{\text{REG}}(W) = \|W_0\|_1$ . Thus, the total loss is given by
$$
\mathcal {L} (\alpha , \beta , \gamma , W, \theta) = \sum_ {i \in I} \mathcal {L} _ {\mathrm {C E}} ^ {(i)} + \tau_ {1} \mathcal {R} _ {\text {N O T E A R S}} (W) + \tau_ {2} \mathcal {R} _ {\text {R E G}} (W) \tag {9}
$$
for hyperparameters  $\tau_{1}$  and  $\tau_{2}$ . Our identifiability result Theorem 1 implies that when we assume that the neural network has infinite capacity, the loss in (9) is minimized,  $\tau_{1}$  is large and  $\tau_{2}$  small, and we learn Gaussian latent variables  $h(X^{(i)},\theta)$ , then we recover the ground truth latent variables up to the
Figure 2: Dependence of performance metrics for  $ER(d,2)$  graphs with  $d' = 100$  and nonlinear mixing  $f$  on the dimension  $d$ .
tolerable ambiguities of labeling and scale, i.e.,  $h$  recovers  $f^{-1}$  and  $W$  recovers  $B$  up to permutation and scale. Thus, we estimate the latent variables using  $\hat{Z} = h(X, \theta)$  and estimate the DAG using  $W_0$ . Full details of the practical implementation of our approach are given in Appendix H.
Our experimental setup is similar to [84, 2], we consider  $d$  interventions with different targets (our theory holds in full generality) and therefore, we can arbitrarily assign  $t_i = i$  based on the intervention index  $i$  which removes the permutation ambiguity. We focus on non-zero shifts because the cross entropy loss together with the quadratic expression for the log-odds results in a non-convex output layer which makes it hard to find the global loss minimizer, as we will describe in more detail in Appendix G.1. We emphasize that this is no contradiction to the theoretical results stated above. Even when there is no shift intervention the latent variables are identifiable, but our algorithm often fails to find the global minimizer of the loss (9) due to the non-convex loss landscape. For the sake of exposition and to set the stage for future work, we also briefly describe an approach via Variational Autoencoders (VAE). VAEs have been widely used in causal representation learning and while feasible in interventional settings, they are accompanied by certain difficulties, which we detail and suggest how to overcome in Appendix F.4.
In this section, we implement our approach on synthetic data and image data. Complete details (architectures, hyperparameters) are deferred to Appendix H. Additional experiments investigating the effect of varisortability [66] and the noise distribution can be found in Appendix G.
Data generation For all our experiments we use Erdős-Rényi graphs, i.e., we add each undirected edge with equal probability  $p$  to the graph and then finally orient them according to some random order of the nodes. We write  $\mathrm{ER}(d,k)$  for the Erdős-Rényi graph distribution on  $d$  nodes with  $kd$  expected edges. For a given graph  $G$  we then sample edge weights from  $\mathcal{U}(\pm [0.25,1.0])$  and a scale matrix  $D$ . For simplicity we assume that we have  $n$  samples from each environment  $i\in \bar{I}$ . We only consider the setting where each node is intervened upon once and thus the latent dimension is also known. We consider three types of mixing functions. First, we consider linear mixing functions where we sample all matrix entries i.i.d. from a Gaussian distribution. Then, we consider non-linear mixing functions that are parametrized by MLPs with three hidden layers which are randomly initialized, and have Leaky ReLU activations. Finally, we consider image data as described in [2]. Pairs of latent variables  $(z_{2i+1},z_{2i+2})$  describe the coordinates of balls in an image and the non-linearity  $f$  is the rendering of the image. The image generation is based on pygame [77]. A sample image can be found in Figure 3.
Evaluation Metrics We evaluate how well we can recover the ground truth latent variables and the underlying DAG. For the recovery of the ground truth latents we use the Mean Correlation Coefficient (MCC) [37, Appendix A.2]. For the evaluation of the learned graph structure we use the Structural Hamming Distance (SHD) where we follow the convention that we count the number of edge differences of the directed graphs (i.e., an edge with wrong orientation counts as two errors). Since the scale of the variables is not fixed the selection of the edge selection threshold is slightly delicate. Thus, we use the heuristic where we match the number of selected edges to the expected number
Figure 3: Sample image with 3 balls.
of edges. As a metric that is independent of this thresholding procedure, we also report the Area Under the Receiver Operating Curve (AUROC) for the edge selection.
Methods We implement our contrastive algorithm as explained in Section F where we use an MLP (with Leaky ReLU nonlinearities) for the function  $h$  for the linear and nonlinear mixing functions and a very small convolutional network for the image dataset, which are termed "Contrastive". For linear mixing function we also consider a version of the contrastive algorithm where  $h$  is a linear function, termed "Contrastive Linear". As baselines, we consider a variational autoencoder with latent dimension  $d$  and also the algorithm for linear disentanglement introduced in [84]. Since the variational autoencoder does not output a causal graph structure we report the result for the empty graph here which serves as a baseline.
Results for linear  $f$  First, for the sake of comparison, we replicate exactly the setting from [84], i.e., we consider initial noise variances sampled uniformly from [2, 4] and we consider perfect interventions where the new variance is sampled uniformly from [6, 8]. We set  $d = 5$ ,  $d' = 10$ ,  $k = 3/2$ , and  $n = 50000$ . Results can be found in Table 1. The linear contrastive method identifies the ground truth latent variables up to scaling. The failure of the nonlinear method to recover the ground truth latents will be explained and further analyzed in Appendix G.1. Note that the nonlinear contrastive and the linear contrastive method recover the underlying graph better than the baseline.
Table 1: Results for linear  $f$  with  $d = 5,{d}^{\prime } = {10},k = 3/2,n = {50000}$  .
Method	SHD ↓	AUROC ↑	MCC ↑	R² ↑
Contrastive	4.6 ± 0.5	0.84 ± 0.02	0.05 ± 0.02	0.02 ± 0.00
Contrastive Linear	5.4 ± 1.6	0.80 ± 0.07	0.90 ± 0.03	1.00 ± 0.00
Linear baseline	7.0 ± 0.5	0.64 ± 0.05	0.83 ± 0.04	1.00 ± 0.00
Results for nonlinear  $f$  We sample all variances from  $\mathcal{U}([1,2])$  (initial variance and resampling for the perfect interventions) and the shift parameters  $\eta^{(i)}$  of the interventions from  $\mathcal{U}([1,2])$ . Results can be found in Table 2. We find that our contrastive method can recover the ground truth latents and the causal structure almost perfectly, while the baseline for linear disentanglement cannot recover the graph or the latent variables (which is not surprising as the mixing is highly nonlinear). Also, training a vanilla VAE does not recover the latent variables up to a linear map as indicated by the  $R^2$  scores. In Figure 2 we illustrate the dependence on the dimension  $d$  of our algorithm in this setting.
Table 2: Results for nonlinear synthetic data with  $n = {10000}$  .
Setting	Method	SHD ↓	AUROC ↑	MCC ↑	R² ↑
ER(5, 2), d' = 20	Contrastive	1.8 ± 0.5	0.97 ± 0.01	0.97 ± 0.00	0.96 ± 0.00
VAE	10.0 ± 0.0	0.50 ± 0.00	0.48 ± 0.03	0.57 ± 0.07
Linear baseline	10.6 ± 1.9	0.48 ± 0.11	0.32 ± 0.03	0.34 ± 0.06
ER(5, 2), d' = 100	Contrastive	1.0 ± 0.0	1.00 ± 0.00	0.99 ± 0.00	0.98 ± 0.00
VAE	10.0 ± 0.0	0.50 ± 0.00	0.59 ± 0.02	0.68 ± 0.04
Linear baseline	3.4 ± 1.2	0.85 ± 0.07	0.18 ± 0.04	0.11 ± 0.04
ER(10, 2), d' = 20	Contrastive	3.6 ± 1.3	0.98 ± 0.01	0.93 ± 0.00	0.87 ± 0.01
VAE	18.6 ± 0.9	0.50 ± 0.00	0.59 ± 0.02	0.72 ± 0.02
Linear baseline	29.6 ± 2.5	0.49 ± 0.02	0.44 ± 0.02	0.51 ± 0.02
ER(10, 2), d' = 100	Contrastive	1.6 ± 0.5	1.00 ± 0.00	0.98 ± 0.00	0.97 ± 0.00
VAE	18.6 ± 0.9	0.50 ± 0.00	0.62 ± 0.02	0.78 ± 0.01
Linear baseline	28.4 ± 2.1	0.51 ± 0.04	0.17 ± 0.03	0.13 ± 0.03
Results for image data Finally, we report the results for image data. Here, we generate the graph as before and consider variances sampled from  $\sigma^2 \sim \mathcal{U}([0.01, 0.02])$  and shifts  $\eta^{(i)}$  from  $\mathcal{U}([0.1, 0.2])$  (i.e., the shifts are still of order  $\sigma$ ). We exclude samples where one of the balls is not contained in the image which generates a slight model misspecification compared to our theory. Again we find that we recover the latent graph and the latent variables as detailed in Table 3.
Table 3: Results for image data with  $\operatorname{ER}(d,2)$  graphs with  $d = 2 \cdot \#$  balls and  $n_{\mathrm{int}} = 25000$  (per environment),  $n_{\mathrm{obs}} = n_{\mathrm{int}} \cdot d$ .
# Balls	Method	SHD ↓	AUROC ↑	MCC ↑	R² ↑
2	Contrastive Learning	1.4 ± 0.4	0.95 ± 0.03	0.87 ± 0.03	0.84 ± 0.03
VAE	6.0 ± 0.0	0.50 ± 0.00	0.19 ± 0.06	0.16 ± 0.08
5	Contrastive Learning	2.0 ± 0.3	1.00 ± 0.00	0.94 ± 0.01	0.91 ± 0.01
VAE	18.6 ± 0.9	0.50 ± 0.00	0.31 ± 0.02	0.36 ± 0.03
10	Contrastive Learning	11.0 ± 3.3	0.98 ± 0.02	0.89 ± 0.01	0.83 ± 0.01
VAE	37.2 ± 3.1	0.50 ± 0.00	0.22 ± 0.01	0.33 ± 0.02
In this work, we extend several prior works and show identifiability for a widely used class of linear latent variable models with non-linear mixing, from interventional data with unknown targets. Counterexamples show that our assumptions are tight in this setting and could only potentially be relaxed under other additional assumptions. We leave to future work to extend our results for other classes of priors, such as non-parametric distribution families, and also to study sample complexity and robustness of our results. We also proposed a contrastive approach to learn such models in practice and showed that it can recover the latent structure in various settings. Finally, we highlight that the results of our experiments are very promising and it would be interesting to scale up our algorithms to large-scale datasets such as [49].
We thank anonymous reviewers for useful comments and suggestions. We acknowledge the support of AFRL and DARPA via FA8750-23-2-1015, ONR via N00014-23-1-2368, and NSF via IIS-1909816, IIS-1955532. We also acknowledge the support of JPMorgan Chase & Co. AI Research. This work was also supported by the Tübingen AI Center.
[1] K. Ahuja, J. S. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 35:15516-15528, 2022.
[2] K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
[3] S. Arora, R. Ge, Y. Halpern, D. M. Mimno, A. Moitra, D. A. Sontag, Y. Wu, and M. Zhu. A practical algorithm for topic modeling with provable guarantees. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pages 280-288. JMLR.org, 2013. URL http://proceedings.mlr.press/v28/arora13.html.
[4] K. Bello, B. Aragam, and P. K. Ravikumar. DAGMA: Learning DAGs via m-matrices and a log-determinant acyclicity characterization. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=8rZYMpFUgK.
[5] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[6] J. Brehmer, P. de Haan, P. Lippe, and T. S. Cohen. Weakly supervised causal representation learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,
[101] S. Zhao, J. Song, and S. Ermon. Infovae: Balancing learning and inference in variational autoencoders. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 5885-5892. AAAI Press, 2019. doi: 10.1609/AAAI.V33I01.33015885. URL https://doi.org/10.1609/aaai.v33i01.33015885.
[102] X. Zheng, B. Aragam, P. K. Ravikumar, and E. P. Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018.
[103] Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: sparsity and beyond. In NeurIPS, 2022. URL http://papers.nips.cc/paper_files/paper/2022/bit6801fa3fd290229efc490ee0cf1c5687-Abstract-Conference.html.
[104] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
A Proof of Identifiability up to Linear Maps
In this section we give a full proof of Theorem 3. We try to make this essential self-contained. To make this easy to read, we first introduce and recall the required notation in Section A.1, then we prove two important key relations required for the proof in Section A.2. Finally, we prove first a simpler version of Theorem 3 with more assumptions in Section A.3 and then give the full proof of Theorem 3 in Section A.4.
A.1 General Notation
Let us introduce some notation for our identifiability proofs. For a full list of assumptions we refer to the main text, here we just recall the relevant notation concisely. We will assume that there are two representations that generate the same observations, i.e., we assume that there are two sets of Gaussian SCMs
$$
Z ^ {(i)} = A ^ {(i)} Z ^ {(i)} + \left(D ^ {(i)}\right) ^ {\frac {1}{2}} \left(\epsilon + \eta^ {(i)} e _ {t _ {i}}\right) \tag {10}
$$
$$
\widetilde {Z} ^ {(i)} = \widetilde {A} ^ {(i)} \widetilde {Z} ^ {(i)} + (\widetilde {D} ^ {(i)}) ^ {\frac {1}{2}} (\widetilde {\epsilon} + \widetilde {\eta} ^ {(i)} e _ {\widetilde {t} _ {i}}) \tag {11}
$$
where  $\epsilon, \widetilde{\epsilon} \sim N(0, \mathrm{Id})$  and  $i = 0$  corresponds to the observational distribution while  $i > 0$  correspond to interventional settings where we intervene on a single node  $t_i$  (or  $\widetilde{t}_i$ ) by adding a shift and changing the relation to its parents (without adding new parents). Moreover, there are two functions  $f, \widetilde{f}$  such that  $X^{(i)} \stackrel{\mathcal{D}}{=} f(Z^{(i)})$  and  $X^{(i)} \stackrel{\mathcal{D}}{=} \widetilde{f}(\widetilde{Z}^{(i)})$ . It is convenient to define
$$
B ^ {(i)} = \left(D ^ {(i)}\right) ^ {- \frac {1}{2}} \left(\operatorname {I d} - A ^ {(i)}\right) \tag {12}
$$
$$
\mu^ {(i)} = \eta^ {(i)} \left(B ^ {(i)}\right) ^ {- 1} e _ {t _ {i}} \tag {13}
$$
so that  $Z^{(i)} = (B^{(i)})^{-1}\epsilon + \mu^{(i)}$  with  $\epsilon \sim N(0, \mathrm{Id})$ . Note that all model information can be collected in  $((B^{(i)}, \eta^{(i)}, t_i), f)$ .
As in the main text, it is convenient to use the shorthand  $B = B^{(0)}$ ,  $A = A^{(0)}$ .
We denote the row  $t_i$  of  $B^{(i)}$  by  $r^{(i)}$ , i.e.,
$$
r ^ {(i)} = \left(B ^ {(i)}\right) ^ {\top} e _ {t _ {i}}. \tag {14}
$$
Note that for perfect interventions where we cut the relation to all parents of node  $t_i$  we have  $r^{(i)} = \lambda_i e_{t_i}$  for some  $\lambda_i \neq 0$ . Similarly, for the observational distribution we refer to the row  $t_i$  of  $B^{(0)}$  by  $s^{(i)}$ , i.e.,
$$
s ^ {(i)} = \left(B ^ {(0)}\right) ^ {\top} e _ {t _ {i}}. \tag {15}
$$
We thus find
$$
B ^ {(0)} - B ^ {(i)} = e _ {t _ {i}} \left(e _ {t _ {i}} ^ {\top} B ^ {(0)} - e _ {t _ {i}} ^ {\top} B ^ {(i)}\right) = e _ {t _ {i}} \left(s ^ {(i)} - r ^ {(i)}\right) ^ {\top}. \tag {16}
$$
We also consider the covariance matrix  $\Sigma^{(i)}$  and the precision matrix  $\Theta^{(i)}$  of  $Z^{(i)}$  which are given by
$$
\Theta^ {(i)} = \left(B ^ {(i)}\right) ^ {\top} B ^ {(i)}, \quad \Sigma^ {(i)} = \left(\Theta^ {(i)}\right) ^ {- 1} \tag {17}
$$
$$
\Sigma^ {(i)} = \mathbb {E} [ (B ^ {(i)}) ^ {- 1} \epsilon \epsilon^ {\top} (B ^ {(i)}) ^ {- \top} ] = (B ^ {(i)}) ^ {- 1} \mathbb {E} [ \epsilon \epsilon^ {\top} ] (B ^ {(i)}) ^ {- \top} = (B ^ {(i)}) ^ {- 1} (B ^ {(i)}) ^ {- \top}
$$
Finally, the mean  $\mu^{(i)}$  of  $Z^{(i)}$  given by
$$
\mu^ {(i)} = \mathbb {E} Z ^ {(i)} = \eta^ {(i)} \left(B ^ {(i)}\right) ^ {- 1} e _ {t _ {i}}. \tag {18}
$$
We also define similar quantities for  $\widetilde{Z}$ . To simplify the notation, we will frequently use the shorthand  $v\otimes w = v\cdot w^{\top}\in \mathbb{R}^{d\times d}$  for  $v,w\in \mathbb{R}^d$
A.2 Two auxiliary lemmas
The key to our identifiability results is the relation shown in the following lemma.