SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1: Hyperparameters:  $\gamma > 0$ ,  $\alpha \in (0,1)$ .
2: Input: Matrix of observations  $X^{e}\in \mathbb{R}^{d_{e}\times n_{e}}$  for all  $e\in [m]$
3: Output: Number of shared latent variables  $\widehat{\ell}$ , matrix  $\widehat{B}$  and probability measure  $\widehat{P}$ .
4: for  $e \in [m]$  do
5: Linear ICA: Use any linear ICA algorithm to obtain a mixing matrix  $\widetilde{B}^e \in \mathbb{R}^{d_e \times \widehat{s}_e}$ , where  $\widehat{s}_e = \mathrm{rank}_{\gamma}(\pmb{X}^e (\pmb{X}^e)^\top)$ . Compute the matrix  $\tilde{\eta}^e = (\widetilde{B}^e)^\dagger \pmb{X}^e \in \mathbb{R}^{\widehat{s}_e \times n_e}$ , where  $(\widetilde{B}^e)^\dagger$  is the Moore-Penrose pseudoinverse of  $\widetilde{B}^e$ .
6: Scaling: Let  $\Delta^{\tilde{e}}$  be a  $\widehat{s}_e\times \widehat{s}_e$  diagonal matrix with entries  $\Delta_{ii}^{e} = \frac{1}{n_{e}} [\tilde{\eta}^{e}(\tilde{\eta}^{e})^{\top}]_{ii}$ . Define  $\overline{\widehat{B}^{e}} = \widetilde{B}^{e}(\Delta^{e})^{-1 / 2}$  and  $\pmb {\eta}^{e} = (\Delta^{e})^{-1 / 2}\tilde{\pmb{\eta}}^{e}$ .
7: Let  $\widehat{P}^e$  be the estimated probability measure with independent marginals such that  $\widehat{P}_i^e$  is the empirical measure of the row  $\pmb{\eta}_{i,*}^{e}$
8: end for
9: Matching: Let  $\widehat{\ell}$  be the maximal number such that there is a signed permutation matrix  $Q^{e}$  in each domain  $e\in [m]$  such that
$$
\Omega_ {\alpha_ {t}} ([ (Q ^ {e}) ^ {\top} \# \widehat {P} ^ {e} ] _ {i}, [ (Q ^ {f}) ^ {\top} \# \widehat {P} ^ {f} ] _ {i}) = 1
$$
for all  $i = 1,\dots ,\widehat{\ell}$  and all  $f\neq e$  , where  $\alpha_{t} = \alpha /t$  with  $t = 2\sum_{e <   f}\widehat{s}_e\widehat{s}_f$  . Let  $\widehat{\mathcal{L}} = \{1,\ldots ,\widehat{\ell}\}$
10: Construct the matrix  $\hat{B}$  and the tuple of probability measures  $\hat{P}$  given by
$$
\widehat {B} = \left( \begin{array}{c c c c} {[ \widehat {B} ^ {1} Q ^ {1} ] _ {\widehat {\mathcal {L}}} & {[ \widehat {B} ^ {1} Q ^ {1} ] _ {[ \widehat {s} _ {1} ] \setminus \widehat {\mathcal {L}}} & & \\ \vdots & & \ddots & \\ {[ \widehat {B} ^ {m} Q ^ {m} ] _ {\widehat {\mathcal {L}}} & & & {[ \widehat {B} ^ {m} Q ^ {m} ] _ {[ \widehat {s} _ {m} ] \setminus \widehat {\mathcal {L}}}} \end{array} \right) \text {a n d} \widehat {P} = \left( \begin{array}{c} {[ (Q ^ {1}) ^ {\top} \# \widehat {P} ^ {1} ] _ {\widehat {\mathcal {L}}} \\ {[ (Q ^ {1}) ^ {\top} \# \widehat {P} ^ {1} ] _ {[ \widehat {s} _ {1} ] \setminus \widehat {\mathcal {L}}} \\ \vdots \\ {[ (Q ^ {m}) ^ {\top} \# \widehat {P} ^ {m} ] _ {[ \widehat {s} _ {m} ] \setminus \widehat {\mathcal {L}}}} \end{array} \right).
$$
11: return  $(\widehat{\ell},\widehat{B},\widehat{P})$
Algorithm 4 IdentifySharedGraphEmpirical
1: Hyperparameters:  $\gamma > 0$ .
2: Input: Matrix  $B^{*} \in \mathbb{R}^{|V| \times \ell}$
3: Output: Parameter matrix  $\widehat{A} \in \mathbb{R}^{\ell \times \ell}$ .
4: Remove rows  $B_{i,\mathcal{L}}^{*}$  with  $\| B_{i,\mathcal{L}}^{*}\|_{2} \leq \gamma$  from the matrix  $B^{*}$ .
5: Find tuples  $(i_k,j_k)_{k\in \mathcal{L}}$  with the smallest possible scores  $\sigma_{\mathrm{min}}(B_{\{i_k,j_k\},\mathcal{L}}^*)$  such that
(i)  $i_k \neq j_k$  for all  $k \in \mathcal{L}$  and  $\{i_k, j_k\} \cap \{i_q, j_q\} = \emptyset$  for all  $k, q \in \mathcal{L}$  such that  $k \neq q$  and
(ii)  $\sigma_{\min}(B_{\{i_k,i_q\},\mathcal{L}}^*) > \gamma$  for all  $k,q\in \mathcal{L}$  such that  $k\neq q$
6: Let  $I = \{i_1,\dots ,i_\ell \}$  and consider the matrix  $B_{I,\mathcal{L}}^{*}\in \mathbb{R}^{\ell \times \ell}$ .
7: Find two permutation matrices  $R_{1}$  and  $R_{2}$  such that  $W = R_{1}B_{I,\mathcal{L}}^{*}R_{2}$  is as close as possible to lower triangular. This can be measured, for example, by using  $\sum_{i < j}W_{ij}^2$ .
8: Multiply each column of  $W$  by the sign of its corresponding diagonal element. This yields a new matrix  $\widetilde{W}$  with all diagonal elements positive.
9: Divide each row of  $\widetilde{W}$  by its corresponding diagonal element. This yields a new matrix  $\widetilde{W}'$  with all diagonal elements equal to one.
0: Compute  $\widehat{A} = I - (\widetilde{W}^{\prime})^{-1}$
1: return  $\widehat{A}$
$2\exp (-2nx^{2})$  . Given a tuple  $C = (i_{1},\dots ,i_{m})$  that defines a shared node, Condition (ii) is an assumption on the number of wrongly matched components. The most extreme case is when the shared node does not actually exist and all components are wrongly matched. That is, the measures  $\widehat{P}_{i_e}^e$  and  $\widehat{P}_{i_f}^f$  are matched even though  $d_{\mathrm{KS}}(P_{i_e}^e,P_{i_f}^f)\neq 0$  and  $d_{\mathrm{KS}}(P_{i_e}^e, - P_{i_f}^f)\neq 0$  for all  $e,f\in [m]$  . On the other hand, if  $|E|\ll m$  , then  $C$  determines a shared node where the majority of the components are correctly matched.
If  $g(n, x) \to 0$  for  $n \to \infty$  and  $x > 0$ , the statement of the theorem becomes meaningful under the constraint  $\sqrt{n_{\max}} / n_{\min} \to 0$ . In this case, the probability that a given tuple  $C$  with wrong components  $E$  determines a shared node goes to zero for large sample sizes  $n_{\min}$ . As noted, the probability of falsely discovering a shared node decreases exponentially with the number of wrongly matched components  $|E|$ . In the extreme case, this means that the probability of falsely discovering shared nodes with all components wrongly matched, i.e.,  $E = [m]$ , decreases exponentially with the number of domains  $m$ .
Theorem D.3 also tells us that the probability of falsely matching two measures  $\widehat{P}_i^e$  and  $\widehat{P}_j^f$  becomes zero if the sample size grows to infinity and the linear ICA algorithm is consistent. However, with finite samples we might fail to match two measures where the underlying true measures are actually the same, i.e., we falsely reject the true null hypothesis  $H_0$ . Thus, we might be overly conservative in detecting shared nodes due to a high family-wise error rate caused by multiple testing. We suggest to correct the level  $\alpha$  to account for the amount of tests carried out. One possibility is to apply a Bonferroni-type correction. The total number of tests is given by  $t = 2\sum_{e < f}\widehat{s}_e\widehat{s}_f$ . This means that an adjusted level is given by  $\alpha_t = \alpha / t$  and instead of the critical value  $c_{\alpha}$  we consider the adjusted critical value  $c_{\alpha_t}$ .
Algorithm 3 is the finite sample version of Algorithm 1 with the matching  $\Omega_{\alpha}$  defined in Definition D.1. To determine the number of independent components for the linear ICA step in each domain, we need to check the  $\mathrm{rank}(\mathbf{X}^e (\mathbf{X}^e)^\top)$ . We specify the rank of a matrix  $M$  as number of singular values which are larger than a certain threshold  $\gamma$  and denote it by  $\mathrm{rank}_{\gamma}(M)$ .
In Algorithm 4 we also provide a finite sample version of Algorithm 2 where we only have the approximation  $B^{\star} = \widehat{B}_{\mathcal{L}} \approx B_{\mathcal{L}} \Psi_{\mathcal{L}}$  for a signed permutation matrix  $\Psi_{\mathcal{L}}$ . For a matrix  $M$ , we denote by  $\sigma_{\min}(M)$  the smallest singular value.
E Error Distributions in Simulations
We specify  $\mathcal{L} = \{1,2,3\}$ ,  $I_1 = \{4,5\}$ ,  $I_2 = \{6,7\}$  and  $I_3 = \{8,9\}$ . Note that the set  $I_3$  does not exist if the number of domains is  $m = 2$ . The error distributions in all simulations are specified as follows if not stated otherwise.
$$
\mathcal {L}: \varepsilon_ {1} \sim \overline {{\mathrm {B e t a}}} (2, 3), \varepsilon_ {2} \sim \overline {{\mathrm {B e t a}}} (2, 5), \varepsilon_ {3} \sim \overline {{\chi}} _ {4} ^ {2},
$$
$$
I _ {1}: \varepsilon_ {4} \sim \overline {{\operatorname {G u m b e l}}} (0, 1), \varepsilon_ {5} \sim \overline {{\operatorname {L o g N o r m a l}}} (0, 1),
$$
$$
I _ {2} \colon \varepsilon_ {6} \sim \overline {{\text {W e i b u l l}}} (1, 2), \varepsilon_ {7} \sim \overline {{\text {E x p}}} (0. 1),
$$
$$
I _ {3}: \varepsilon_ {8} \sim \overline {{\operatorname {S k e w N o r m a l}}} (6), \varepsilon_ {9} \sim \overline {{\operatorname {S k e w N o r m a l}}} (1 2),
$$
where the overline means that each distribution is standardized to have mean 0 and variance 1. Figure 6 shows histograms of the empirical distributions.
F Additional Simulation Results
In this section, we make additional experiments. First, we consider another setup where all our assumptions are satisfied but we have more domains and more shared latent variables. Then, we also consider two setups where some of our assumptions are not satisfied.
Different Setup. We make additional experiments on a similar scale as in Section 5, but with more shared nodes and less domain specific nodes. This time, we consider  $\ell = 5$  shared latent nodes and  $|I_e| = 1$  domain-specific latent node in each domain. Moreover, we also consider  $m = 4$  domains. The dimensions are given by  $d_e = d / m$  for all  $e\in [m]$  and  $d = 48$ . The graphs and edge weights
Figure 6: Histograms showing the frequency of 25000 values sampled from the random variables  $\varepsilon_{i}$  with distribution as specified in Appendix E. Each distribution has mean zero and variance one. The first row shows the empirical distributions from  $\varepsilon_{1}$  to  $\varepsilon_{5}$  and the second row from  $\varepsilon_{6}$  to  $\varepsilon_{9}$ .
(c)
Figure 7: Simulation results for  $\ell = 5$  shared latent nodes. Logarithmic scale on the  $x$ -axis. Error bars in (a) are one standard deviation of the mean and in (b) and (c) they are the interquartile range.
are sampled equivalently as in Section 5 in the main paper. We also consider the same distribution of the error variables as specified in Appendix E, where we specify  $\mathcal{L} = \{1,2,3,4,5\}$ ,  $I_{1} = \{6\}$ ,  $I_{2} = \{7\}$ ,  $I_{3} = \{8\}$  and  $I_{4} = \{9\}$ .
Figure 7 shows the results where the scores are equivalent as in the main paper. Once again, we see that the estimation error for the matrices  $B_{\mathcal{L}}$  and  $A_{\mathcal{L},\mathcal{L}}$  decreases with increasing sample size. This supports our proof of concept and shows that the adapted algorithms are consistent for recovering  $B_{\mathcal{L}}$  and  $A_{\mathcal{L},\mathcal{L}}$  from finite samples.
Violated Assumptions. We consider the same setup as in the main paper in Section 5 with  $l = 3$  shared latent nodes, but we fix the number of domains to  $m = 3$ . In this experiment, we compare the results where data was generated such that all our assumptions are satisfied with two setups where we violate some of the assumptions. In the first setup, we violate Condition (C1) that requires pairwise different error distributions. We specify the error distributions as follows.
$$
I _ {1}: \varepsilon_ {4} \sim \overline {{\mathrm {B e t a}}} (2, 3), \varepsilon_ {5} \sim \overline {{\mathrm {L o g N o r m a l}}} (0, 1),
$$
$$
I _ {2} \colon \varepsilon_ {6} \sim \overline {{\operatorname {B e t a}}} (2, 5), \varepsilon_ {7} \sim \overline {{\operatorname {L o g N o r m a l}}} (0, 1),
$$
$$
I _ {3}: \varepsilon_ {8} \sim \overline {{\chi}} _ {4} ^ {2}, \varepsilon_ {9} \sim \overline {{\operatorname {L o g N o r m a l}}} (0, 1),
$$
where, as before, the overline means that each distribution is standardized to have mean 0 and variance 1. In the second setup, we do not change the error distributions but we violate Condition (C3) that requires two partial pure children per shared latent node. In this experiment, we do not make any sparsity assumptions on the mixing matrix  $G_{V,\mathcal{L}}$ .
Figure 8 shows the results of our experiments. As expected, we see in (a) and (b) that identifyability of the joint distribution fails if we do not require pairwise different error distributions. Recovering the joint distribution still works well in the second setup where we violate the partial pure children
(b)
Figure 8: Simulation results where assumptions are not satisfied. Logarithmic scale on the  $x$ -axis. Error bars in (a) are one standard deviation of the mean and in (b) and (c) they are the interquartile range.
conditions. However, identifying the shared latent graph is impossible in this setup, as we explained in Appendix C. This is supported by the experimental results displayed in Figure 8 (c), where we can see that recovery of the shared latent graph does not work when the partial pure children assumption is not satisfied.
We see many directions for future work, which include the following.
- Our work and algorithms rely on linear ICA. It would be interesting to study a more direct approach to recover the joint distribution and the causal graph. This might potentially be done by testing certain constraints implied by the model similar as the developments in the LiNGAM literature; see e.g. Shimizu et al. (2011) and Wang and Drton (2020).
- Our results require non-Gaussianity and that both the latent structural equation model and the mixing functions are linear. We consider the linear setup as a basis for any subsequent study of nonlinear cases. For example, recent advances in non-linear ICA allow identifiability of up to linear transformations, see e.g. Khemakhem et al. (2020), Buchholz et al. (2022) and Roeder et al. (2021). Identifiability of a causal representation might then be obtained from identifiability results for the linear model.
- Our sufficient condition for identifiability of the shared latent graph requires two partial pure children per shared latent node. In this regard, it would also be interesting to study necessary conditions; c.f. our discussion in Appendix C.
- This work focused purely on the observational case. However, considering interventional data can be expected to permit relaxing some conditions in both of the key steps, i.e., recovering the joint distribution and the shared latent graph. For example, recent work shows that interventional data allow for identification of the latent graph without sparsity constraints in a single-domain setup (Squires et al., 2023; Ahuja et al., 2023). Extending this to the multi-domain setup is an interesting problem for future work.
- It would be interesting to study the statistical properties of our setup such as theoretical bounds on the accuracy of recovering the matrices  $B$  and  $A_{\mathcal{L},\mathcal{L}}$  as well as developing algorithms that meet these bounds. Currently, our adapted algorithms for finite samples determine the rank of a matrix by using a threshold for singular values. The algorithms depend on the choice of this parameter and it would be worth studying optimal choices. Moreover, one might consider different methods for determining the rank of a matrix.
- There might be different matching strategies of the estimated error distributions in the finite sample setting. For example, instead of matching pairwise consistently across all domains as we propose in Appendix D, one might find an optimal matching by solving a linear program.