SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Their main results assume that the mixing  $f$  is an injective polynomial of finite degree  $p$  (see Assumption 2 in their paper). The injectivity requires their coefficient matrix to be full rank, which in turn implies  $d' \geq \sum_{r=0}^{p} \binom{r+d-1}{d-1} = \binom{p+d}{d}$  (see the discussion below Assumption 2 in their work). This means that we cannot set the degree of the decoder polynomial to be arbitrarily large for fixed output dimension  $d'$  (which is required for universal approximation). In contrast, we only require  $d' \geq d$ . For a general discussion of the relation between approximability and identifiability we refer to Appendix E.
2. All their results for general nonlinear mixing functions rely on deterministic do-interventions which is a type of hard intervention that assigns a constant value to the target. However, we focus on randomized soft interventions (and also allow shift interventions which have found applications [73, 58]), which as also pointed out by [84, 87] is less restricted. Moreover, they require multiple interventions for each latent variable (as they use an  $\epsilon$ -net argument) to show approximate identifiability, while the structural assumptions on the causal model allow us to show full identifiability with just one intervention for each latent variable. Thus, their setting is not directly comparable to ours and neither is strictly more general than the other. In particular, their proof techniques and experimental methodology don't translate to our setting, and we use completely different ideas for our proofs and experiments.
D Counterexamples and Discussions
In this section, we first present several counterexamples to relaxed versions of our main results in Section D.1 and then provide the missing proofs in Section D.2.
D.1 Main counterexamples
In this section, we will discuss the optimality and limitations of various assumptions of our main results. In particular, we consider the number of interventions, the distribution of the latent variables, and the types of interventions separately.
Number of interventions Our main results all require  $d$  interventions, i.e., one intervention per node. This is necessary even in the simpler setting of linear  $f$  as was shown in [84, Proposition 4], directly implying it is also necessary for the more general class of non-linear  $f$  that we handle here. Therefore, the number of interventions in our main theorems 1, 2 is both necessary and sufficient. Going a step further, it is natural to ask whether Theorem 3 remains true for less than  $d$  interventions. However, as we show next,  $d - 2$  interventions are not sufficient.
Fact 1. Suppose we are given distributions  $X^{(i)}$  generated by  $((B^{(i)},\eta^{(i)},t_i)_{i\in I},f)$  satisfying Assumption 1-3. If the number of interventions satisfies  $|I|\leq d - 2$  it is not possible to identify  $f$  up to linear maps. Consider, e.g.,  $d = 2$  and  $Z = N(0,\mathrm{Id})$  and no interventions. Then  $h(Z)\stackrel {\mathcal{D}}{=}Z$  for (nonlinear) radius dependent rotations as defined in [29, 9].
Let us next consider the case with a total of  $d$  environments which corresponds to  $d - 1$  interventional distributions plus one observational distribution. We can provide a non-identifiability result for a general class of heterogeneous latent variable models that satisfy an algebraic property.
Lemma 7. Assume that  $d = 2$  and  $Z^{(0)} \sim N(0, \Sigma^{(0)})$ ,  $Z^{(1)} \sim N(0, \Sigma^{(1)})$  and  $\Sigma^{(1)} \succ \Sigma^{(0)}$  or  $\Sigma^{(1)} \prec \Sigma^{(0)}$  in Löewner order (i.e., the difference is positive definite). Then there is a non-linear map  $h$  such that  $h(Z^{(i)}) \stackrel{\mathcal{D}}{=} Z^{(i)}$  for  $i = 0, 1$ .
We provide a proof of this lemma in Appendix D.2. Note that this non-identifiability lemma requires the assumption on  $\Sigma^{(i)}$ . In Lemma 3 in Appendix A we have seen that for the interventions considered in this work the relation  $\Sigma^{(i)} > \Sigma^{(0)}$  or  $\Sigma^{(i)} < \Sigma^{(0)}$  generally does not hold. This is some weak indication that for Theorem 3,  $d - 1$  interventions might be sufficient. We leave this for future work.
As an additional note, in the special case when  $f$  is a permutation matrix, we are in the setting of traditional causal discovery and here,  $d - 1$  interventions are necessary and sufficient [10, 18, 84].
Type of intervention Even when we restrict ourselves to only linear mixing functions as considered as in [84], we need perfect interventions in Theorem 1. Otherwise, only the weaker identifiability guarantees in Theorem 2 can be obtained. This is shown in [84] for linear  $f$ . Therefore, in the more general setting of nonlinear  $f$ , we cannot hope to obtain Theorem 1 with imperfect interventions, showing that this assumption is needed.
Next, we clarify that for the identifiability result in Theorem 2, the condition that the interventions are not pure noise interventions is also necessary.
Fact 2. Recall that we defined  $B^{(i)} = (D^{(i)})^{-\frac{1}{2}}(\mathrm{Id} - A^{(i)})$ . Suppose  $X^{(i)}$  is generated by  $((B^{(i)}, \eta^{(i)}, t_i)_{i \in \bar{I}}, f)$  satisfying Assumptions 1-4 where all interventions are pure noise interventions. By definition, this implies  $A^{(i)} = A$  for all  $i$ . Consider any matrix  $\widetilde{A}$  such that the corresponding DAG is acyclic. Then  $((\widetilde{B}^{(i)}, \eta^{(i)}, t_i)_{i \in \bar{I}}, \widetilde{f})$  with  $\widetilde{B}^{(i)} = (D^{(i)})^{-\frac{1}{2}}(\mathrm{Id} - \widetilde{A})$  for all  $i$  and
$$
\widetilde {f} = f \circ (B ^ {- 1} \widetilde {B}) = f \circ (\operatorname {I d} - A) ^ {- 1} (D ^ {(0)}) ^ {- \frac {1}{2}} (D ^ {(0)}) ^ {\frac {1}{2}} (\operatorname {I d} - \widetilde {A}) = f \circ (\operatorname {I d} - A) ^ {- 1} (\operatorname {I d} - \widetilde {A}) \tag {91}
$$
generates the same distributions  $X^{(i)}$ . Indeed,
$$
\begin{array}{l} \widetilde {f} (\widetilde {Z} ^ {(i)}) = f \circ (B ^ {- 1} \widetilde {B}) (\widetilde {B} ^ {(i)}) ^ {- 1} (\epsilon + \eta^ {(i)} e _ {t _ {i}}) \\ = f \left(\left(\operatorname {I d} - A\right) ^ {- 1} (\operatorname {I d} - \widetilde {A}) (\operatorname {I d} - \widetilde {A}) ^ {- 1} \left(D ^ {(i)}\right) ^ {\frac {1}{2}} \left(\epsilon + \eta^ {(i)} e _ {t _ {i}}\right)\right) \\ = f \left(\left(\operatorname {I d} - A\right) ^ {- 1} \left(D ^ {(i)}\right) ^ {\frac {1}{2}} \left(\epsilon + \eta^ {(i)} e _ {t _ {i}}\right)\right) \tag {92} \\ = f \left(\left(B ^ {(i)}\right) ^ {- 1} \left(\epsilon + \eta^ {(i)} e _ {t _ {i}}\right)\right) \\ = f (Z ^ {(i)}). \\ \end{array}
$$
This implies that any underlying causal graph could generate the observations. In other words, for pure noise interventions we can identify the scale of the noise variables, but we obtain no intervention
about the causal variables and structure (exactly because the interventions do not target the actual causal variables but the noise variables).
Finally, we remark that in contrast to the case with linear mixing functions, we need to exclude non-stochastic hard interventions for linear identifiability.
Lemma 8. Consider  $d = 2$  and  $Z^{(0)} = N(0, \mathrm{Id})$  and the non-stochastic hard interventions  $\mathrm{do}(Z_1 = 0)$  and  $\mathrm{do}(Z_2 = 0)$  resulting in the degenerate Gaussian distributions  $Z^{(1)} = N(0, e_2 \otimes e_2)$  and  $Z^{(2)} = N(0, e_1 \otimes e_1)$ . Then there is a nonlinear  $h$  such that  $h(Z^{(i)}) \stackrel{\mathcal{D}}{=} Z^{(i)}$  for all  $i$ .
Intuitively, to show this, we choose  $h$  to be identity close to the supports  $\{Z_1 = 0\}$  and  $\{Z_2 = 0\}$  but some nonlinear measure preserving deformation in the four quadrants. The full argument can be found in Appendix D.2. A similar counterexample was discussed in [2]. However, here we in addition constrain the distribution of the latent variables, which adds a bit of complexity.
Distribution of the Noise Variables Our key additional assumption compared to the setting considered in [84, 2, 88] is that we assume that the latent variables are Gaussian which allows us to put only very mild assumptions on the mixing function  $f$ . However, we do this in a manner so that we still preserve universal representation guarantees.
[88] consider arbitrary noise distributions but restrict to linear mixing functions. We believe that our result can be generalized to more general distributions of the noise variables, e.g., exponential families, but those generalizations will require different proof strategies, which we leave for future work. However, the result will not be true for arbitrary noise distributions, as the following lemma shows.
Lemma 9. Consider  $Z^{(0)} = (\epsilon_1, \epsilon_2)^\top$ ,  $Z^{(1)} = (\epsilon_1', \epsilon_2)$ , and  $Z^{(2)} = (\epsilon_1, \epsilon_2')$  where  $\epsilon_1, \epsilon_2 \sim \mathcal{U}([-1, 1])$  and  $\epsilon_1', \epsilon_2' \sim \mathcal{U}([-3, 3])$ . Also define  $\widetilde{Z}^{(0)} = (\epsilon_1, \epsilon_1 + \epsilon_2)$ ,  $\widetilde{Z}^{(1)} = (\epsilon_1', \epsilon_1' + \epsilon_2)$ , and  $\widetilde{Z}^{(2)} = (\epsilon_1, \epsilon_2')$ . Then there is  $h$  such that
$$
h \left(Z ^ {(i)}\right) \stackrel {{\mathcal {D}}} {{=}} \widetilde {Z} ^ {(i)}. \tag {93}
$$
A proof of this lemma can be found in Appendix D.2. The result shows that even with  $d$  perfect interventions and linear SCMs it is not possible to identify the underlying DAG (empty graph for  $Z$  and  $\widetilde{Z}_1\to \widetilde{Z}_2$ ) and we also cannot identify  $f$  up to linear maps. While our example relies on distributions with bounded support, we conjecture that full support is not sufficient for identifiability.
Structural Assumptions A final assumption used in our results is the restriction to linear SCMs. This is used for most works on causal representation learning (see Section 2), even when they restrict to linear mixing functions. Although this may potentially be a nontrivial restriction, the advantage is that the simplicity of the latent space will enable us to meaningfully probe the latent variables, intervene and reason about them. Nevertheless, it's an interesting direction to study to what generality this can be relaxed. Indeed, even for the arguably simpler problem of causal structure learning (i.e., when  $Z^{(i)}$  is observed) there are many open questions when moving beyond additive noise models. We leave this for future work.
D.2 Technical constructions
In this section, we provide the proofs for the counterexamples in Section D.1. For the first two counterexamples we construct functions  $h$  such that  $h(Z^{(i)}) \stackrel{\mathcal{D}}{=} Z^{(i)}$  by setting  $h = \Phi_t$  where  $\Phi_t$  is defined as the flow of a suitable vectorfield  $X$ , i.e.,
$$
\partial_ {t} \Phi_ {t} (z) = X \left(\Phi_ {t} (z)\right). \tag {94}
$$
This is a common technique in differential geometry and was introduced to construct counterexamples to identifiability ICA with volume preserving functions in [9, 8]. To find a suitable vectorfield  $X$  we rely on the fact that the density  $p_t$  of  $\mathbb{P}_t = (\Phi_t)_*\mathbb{P}$  satisfies the continuity equation
$$
\partial_ {t} p _ {t} (z) + \operatorname {D i v} \left(p _ {t} (z) X (z)\right) = 0. \tag {95}
$$
In particular  $\mathbb{P}_t = \mathbb{P}$  holds iff
$$
\operatorname {D i v} \left(p _ {t} (z) X (z)\right) = 0. \tag {96}
$$
Therefore it is sufficient to find a vectorfield  $X$  such that  $\mathrm{Div}(p^{(i)}X) = 0$  for all environments  $i$  to construct a suitable  $h = \Phi_1$ .
Proof of Lemma 7. Let us first discuss the high-level idea of the proof. The general strategy is to construct a vectorfield  $X$  whose flow preserves  $Z^{(i)}$  for  $i = 0,1$ . This holds if and only if the densities  $p_0,p_1$  of  $Z^{(0)},Z^{(1)}$  satisfy
$$
\operatorname {D i v} \left(p _ {0} X\right) = \operatorname {D i v} \left(p _ {1} X\right) = 0 \tag {97}
$$
Using  $\operatorname{Div}(fX) = \nabla f \cdot X + f\operatorname{Div}X$  we find
$$
X \cdot \nabla (\ln p _ {0} (z) - \ln p _ {1} (z)) = \frac {X \cdot \nabla p _ {0} (z)}{p _ {0} (z)} - \frac {X \cdot \nabla p _ {1} (z)}{p _ {1} (z)} - \frac {p _ {0} (z) \operatorname {D i v} X}{p _ {0} (z)} + \frac {p _ {1} (z) \operatorname {D i v} X}{p _ {1} (z)} = 0. \tag {98}
$$
We conclude that any  $X$  satisfying (97) must be orthogonal to  $\nabla (\ln p_0(z) - \ln p_1(z))$  and thus parallel to the level lines of  $\ln p_0(z) - \ln p_1(z)$ . This already fixes the direction of  $X$  and the magnitude can be inferred from (97). To simplify the calculations we can first linearly transform the data so that the directions of  $X$ , i.e., equivalently the level lines of  $\ln p_0(z) - \ln p_1(z)$  have a simple form. We assume that  $\Sigma_0 \prec \Sigma_1$ . Clearly, it is sufficient to show the result for any linear transformation of the Gaussian distributions  $Z^{(i)}$ . Note that generally for  $G \sim N(\mu, \Sigma)$  we have  $AG \sim N(A\mu, A\Sigma A^\top)$ . Applying  $\Sigma_0^{-\frac{1}{2}}$  we can reduce the problem to  $\mathrm{Id} \prec \Sigma_1' = \Sigma_0^{-\frac{1}{2}}\Sigma_1\Sigma_0^{-\frac{1}{2}}$ . Diagonalizing  $\Sigma_1'$  by  $U \in \mathrm{SO}(d)$  it is sufficient to consider  $\mathrm{Id} \prec \Lambda = U\Sigma_1'U^\top$  where  $\Lambda = \mathrm{Diag}(\lambda_1, \lambda_2)$ . Finally, we can rescale  $z_1$  and  $z_2$  such that the resulting covariance matrices  $\Lambda^0 = \mathrm{Diag}(\lambda_1^0, \lambda_2^0)$ ,  $\Lambda^1 = \mathrm{Diag}(\lambda_1^1, \lambda_2^1)$  satisfy
$$
\frac {1}{\lambda_ {1} ^ {1}} - \frac {1}{\lambda_ {1} ^ {0}} = \frac {1}{\lambda_ {2} ^ {1}} - \frac {1}{\lambda_ {2} ^ {0}} = 1. \tag {99}
$$
Thus, it is sufficient to show the claim for such covariances  $\Lambda^0$ ,  $\Lambda^1$  and the result for arbitrary covariances follows by applying suitable linear transformation.
For such covariances we find for some constant  $c$
$$
\ln \left(p _ {0} (z)\right) - \ln \left(p _ {1} (z)\right) = - \frac {z _ {1} ^ {2}}{2 \lambda_ {1} ^ {0}} - \frac {z _ {2} ^ {2}}{2 \lambda_ {2} ^ {0}} + \frac {z _ {1} ^ {2}}{2 \lambda_ {1} ^ {1}} + \frac {z _ {2} ^ {2}}{2 \lambda_ {2} ^ {1}} + c = z _ {1} ^ {2} + z _ {2} ^ {2} + c. \tag {100}
$$
The level lines are circles. Thus, we consider the vector field
$$
X _ {0} (z) = \left( \begin{array}{c} - z _ {2} \\ z _ {1} \end{array} \right). \tag {101}
$$
We see directly that  $\operatorname{Div} X_0 = 0$ . We consider the Ansatz  $X(z) = f(|z|)X_0(z) / p_0(z)$ . We now fix the radial function  $f: \mathbb{R}_+ \to \mathbb{R}$  such that it has compact support away from 0. We find using  $\operatorname{Div} X_0 = 0$
$$
\begin{array}{l} \operatorname {D i v} (X (z) p _ {0} (z)) = \operatorname {D i v} (X (z) p _ {0} (z)) (102) \\ = X _ {0} (z) \cdot (\nabla | z |) f ^ {\prime} (z) (103) \\ = \binom {- z _ {2}} {z _ {1}} \cdot \frac {z}{| z | ^ {2}} f ^ {\prime} (z) (104) \\ = 0. (105) \\ \end{array}
$$
Note, that close to 0 where  $|z|$  is not differentiable the vector field vanishes by our construction of  $f$ . We find similarly using (100)
$$
\begin{array}{l} \operatorname {D i v} (X (z) p _ {1} (z)) = X _ {0} (z) \cdot \nabla \left(f (| z |) \frac {p _ {1} (z)}{p _ {0} (z)}\right) (106) \\ = X _ {0} (z) \cdot \nabla \left(f (| z |) e ^ {- | z | ^ {2} - c}\right) (107) \\ \end{array}
$$
Figure 4: Sketch of the setting in Lemma 9. The map  $h$  agrees with  $(z_1, z_2) \to (z_1, z_1 + z_2)$  on the red rectangle and maps  $R_i$  to  $Q_i$  such that the uniform measure is preserved.
$$
= X _ {0} (z) \cdot \nabla \tilde {f} (| z |) \tag {108}
$$
$$
= 0. \tag {109}
$$
Finally we remark that the flow  $\Phi_t$  of the vectorfield  $X$  generates a family of functions  $h$  as desired, i.e., for all  $t$  we have  $(\Phi_t)_*Z^{(i)} = Z^{(i)}$  and  $\Phi_t$  is clearly nonlinear as it is the identity close to 0 but not globally.
The proof of the next lemma is similar but simpler.
Proof of Lemma 8. Pick any smooth divergence free vectorfield  $X_0$  with compact support in  $[\epsilon, \infty)^2$  for some  $\epsilon > 0$ . Then the vector field  $X = X_0 / p_0$  satisfies
$$
\operatorname {D i v} X p _ {0} = \operatorname {D i v} X _ {0} = 0. \tag {110}
$$
Thus the flow  $\Phi_t$  preserves the distribution of  $Z^{(0)}$ . But it also preserves the interventional distributions  $Z^{(i)}$  for  $i = 1,2$  because  $X(z) = 0$  and thus  $\Phi_t(z) = z$  for  $z$  close to the supports of  $Z^{(i)}$ .
Finally we prove Lemma 9 that shows that we cannot completely drop the assumption on the distribution of the noise variables  $\epsilon$ .
$$
h \left(Z ^ {(i)}\right) \stackrel {\mathcal {D}} {=} \tilde {Z} ^ {(i)}. \tag {93}
$$
Proof of Lemma 9. A sketch of the setting can be found in Figure 4. We define  $h_0(z) = (z_1, z_1 + z_2)$ . If  $h(z) = h_0(z)$  for  $-1 \leq z_2 \leq 1$  we find  $h(Z^{(i)}) \stackrel{\mathcal{D}}{=} \widetilde{Z}^{(i)}$  for  $i = 0$  and  $i = 1$ . It remains to modify  $h_0$  such that  $h$  preserves the uniform measure on  $[-1, 1] \times [-3, 3]$ . We do this in two steps, first we
construct a modification  $\widetilde{h}$  of the map  $h_0$  such that  $[-1,1] \times [-3,3]$  is mapped bijectively to itself but  $\widetilde{h} = h_0$  for  $|z_2| \leq 1$  (so that  $\widetilde{h}(Z^{(i)}) \stackrel{\mathcal{D}}{=} \widetilde{Z}^{(i)}$  for  $i = 0$  and  $i = 1$  holds) and then we apply a general result to make the map measure preserving. We start with the first step. Let  $\psi: \mathbb{R} \to [0,1]$  be differentiable such that  $\psi(t) = 1$  for  $-1 \leq z \leq 1$  and  $\psi(t) = 0$  for  $-5/2 \leq z \leq 5/2$  with  $|\psi'(t)| < 1$ . Then
$$
\tilde {h} (z) = \psi \left(z _ {2}\right) z + \left(1 - \psi \left(z _ {2}\right)\right) h _ {0} (z) = \binom {z _ {1}} {z _ {2} + \psi \left(z _ {2}\right) z _ {1}} \tag {111}
$$
is injective on  $[-1,1] \times [-3,3]$  (note that the second coordinate is increasing in  $z_2$ ) and maps  $[-1,1] \times [-3,3]$  bijectively to itself and agrees with  $h_0$  for  $|z_2| \leq 1$ . However, it is not necessarily volume preserving. But applying Moser's theorem (see [56]) to the image of the rectangles  $R_1 = [-1,1] \times [-3, -1]$  and  $R_2 = [-1,1] \times [1,3]$  which are the quadrilaterals  $Q_1 = \tilde{h}(R_1)$  with endpoints  $(-1, -3)$ ,  $(1, -3)$ ,  $(1,0)$ ,  $(-1, -2)$  and  $Q_2 = \tilde{h}(R_2)$  with endpoints  $(1,3)$ ,  $(-1,3)$ ,  $(-1,0)$ ,  $(1,2)$  with the same size as  $R_1$  and  $R_2$  we infer that there is  $\varphi$  supported in  $Q_1 \cup Q_2$  such that  $h = \varphi \circ \tilde{h}$  satisfies  $h(Z^{(2)}) \stackrel{\mathcal{D}}{=} \widetilde{Z}^{(2)} \stackrel{\mathcal{D}}{=} \mathcal{U}([-1,1] \times [-3,3])$ .
E Identifiability and Approximation
In this section, we explain that approximation properties of function classes cannot be leveraged to obtain stronger identifiability results. The general setup is that we have two function classes  $\mathcal{G} \subset \mathcal{F}$  and we assume that  $\mathcal{G}$  is dense in  $\mathcal{F}$  (in the topology of uniform convergence on  $\Omega$ ), i.e., for  $f \in \mathcal{F}$  and any  $\epsilon > 0$  there is  $g \in \mathcal{G}$  such that
$$
\sup  _ {z \in \Omega} | f (z) - g (z) | \leq \epsilon . \tag {112}
$$
We now investigate the relationship of identifiability results for mixings in either  $\mathcal{G}$  or in  $\mathcal{F}$ .
While this point is completely general, we will discuss this for concreteness in the context of nonlinear ICA and considering polynomial mixing functions for  $\mathcal{G}$ . We also assume that  $\mathcal{F}$  are all diffeomorphisms (onto their image). Suppose we have latents  $Z\sim \mathcal{U}([-1,1]^d)$  and observe a mixture  $X = h(Z)$ . We use the shorthand  $\Omega = [-1,1]^d$  from now on. Let us suppose that  $h\in \mathcal{G}$ . Then the following result holds.
Lemma 10. Suppose  $\widetilde{h}(Z) \stackrel{\mathcal{D}}{=} X \stackrel{\mathcal{D}}{=} h(Z)$  where  $Z \sim \mathcal{U}(\Omega)$ ,  $h, \widetilde{h} \in \mathcal{G}$  and  $h$  satisfies an injectivity condition as defined in Assumption 2 in [2]. Then  $h$  and  $\widetilde{h}$  agree up to permutation of the variables.
Proof. This is a consequence of Theorem 4 in [2]. Essentially, they show in Theorem 1 that  $h$  and  $\tilde{h}$  agree up to a linear map and then use independence of supports to conclude (in our simpler setting here, one could also resort to the identifiability of linear ICA).
The injectivity condition is a technical condition that ensures that the embedding  $h$  is sufficiently diverse, but this is not essential for the discussion here. Let us nevertheless mention here that it is not clear whether this condition is sufficiently loose to ensure identifiability in a dense subspace of  $\mathcal{F}$  because it requires an output dimension depending on the degree of the polynomials.
We now investigate the implications for  $\mathcal{F}$ . It is well known that ICA in general nonlinear function is not identifiable, i.e., for any  $f\in \mathcal{F}$  there is a  $\widetilde{f}\in \mathcal{F}$  such that  $f(Z)\stackrel {\mathcal{D}}{=}\widetilde{f} (Z)$ . To find such an  $\widetilde{f}$  one use, e.g., the Darmois construction, radius dependent rotations or, more generally, measure preserving transformations  $m$  such that  $m(Z)\stackrel {\mathcal{D}}{=}Z$ . Then we have the following trivial lemma.
Lemma 11. For every  $\epsilon >0$  there are functions  $g,\widetilde{g}\in \mathcal{G}$  such that
$$
\sup  _ {\Omega} | g - f | <   \epsilon , \quad \sup  _ {\Omega} | \widetilde {g} - \widetilde {f} | <   \epsilon \tag {113}
$$
and then the Wasserstein distance satisfies  $W_{2}(f(Z),g(Z)) < \epsilon$ ,  $W_{2}(\widetilde{f} (Z),\widetilde{g} (Z)) < \epsilon$ .
Remark 6. For the definition and a discussion of the Wasserstein metric we refer to [90]. Alternatively we could add a small amount of noise, i.e.  $X = f(Z) + \nu$  and then consider the total variation distance.
Proof. The first part follows directly from the Stone-Weierstrass Theorem which shows that polynomials are dense in the continuous function on compact domains. The second part follows from the coupling  $Z \to (f(Z), g(Z))$  and (113).
The main message of this Lemma is that whenever there are spurious solutions in the larger function class  $\mathcal{F}$  we can equally well approximate the ground truth mixing  $f$  and the spurious solution  $\widetilde{f}$  by functions in  $\mathcal{G}$ . In this sense, the identifiability of ICA in  $\mathcal{G}$  does not provide any guidance to resolve the ambiguity of ICA in  $\mathcal{F}$ . So identifiability results in  $\mathcal{G}$  are not sufficient when there is no reason to believe that the ground truth mixing is exactly in  $\mathcal{G}$ .
Actually, one could view the approximation capacity of  $\mathcal{G}$  also as a slight sign of warning as we will explain now. Suppose the ground truth mixing satisfies  $g\in \mathcal{G}$ . Since ICA in  $\mathcal{F}$  is not identifiable we can find  $\widetilde{f}\in \mathcal{F}$  which can be arbitrarily different from  $g$  but such that  $g(Z) = \widetilde{f} (Z)$ . Then we can approximate  $\widetilde{f}$  by  $\widetilde{g}$  with arbitrarily small error. Thus, we find almost spurious solutions, i.e.,  $\widetilde{g}\in \mathcal{G}$  such that  $W_{2}(\widetilde{g} (Z),g(Z)) < \epsilon$  for an arbitrary  $\epsilon >0$  but  $\widetilde{g}$  and  $g$  correspond to very different data representations, in this sense the identifiability result is not robust.
When only considering the smaller space  $\mathcal{G}$  this problem can be resolved by considering norms or seminorms on  $\mathcal{G}$ , for polynomials a natural choice is the degree of the polynomial. Indeed, suppose that we observe data generated using a mixing  $g\in \mathcal{G}$ . Then there might be spurious solutions  $\widetilde{f}\in \mathcal{F}$  which can be approximated by  $\widetilde{g}\in \mathcal{G}$  but this approximation will generically have a larger norm (or degree for polynomials) than  $g$ . Therefore, the minimum description length principle [69] favors  $g$  over  $\widetilde{g}$ .