Multi-View Causal Representation Learning with Partial Observability
Authors: Dingling Yao
Danru Xu
Sebastien Lachapelle
Sara Magliacane
Perouz Taslakian
Georg Martius
Julius von Kügelgen
Francesco Locatello
Venue: ICLR 2024 spotlight (2024)
Keywords: causal representation learning; identifiability
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: We present a unified framework for studying the identifiability of representations learned from simultaneously observed views, such as different data modalities. We allow a partially observed setting in which each view constitutes a nonlinear mixture of a subset of underlying latent variables, which can be causally related.
We prove that the information shared across all subsets of any number of views can be learned up to a smooth bijection using contrastive learning and a single encoder per view. 
We also provide graphical criteria indicating which latent variables can be identified through a simple set of rules, which we refer to as identifiability algebra. Our general framework and theoretical results unify and extend several previous work on multi-view nonlinear ICA, disentanglement, and causal representation learning. We experimentally validate our claims on numerical, image, and multi-modal data sets. Further, we demonstrate that the performance of prior methods is recovered in different special cases of our setup. 
Overall, we find that access to multiple partial views offers unique opportunities for identifiable representation learning, enabling the discovery of latent structures from purely observational data.

Published as a conference paper at ICLR 2024
BULTI-VIER CAUSAL REPRESENTATION LEARNING WITH PARTIAL OBSERVABILITY
Dingling Yao Danru Sébastien Lachapelle Sara Magliacane Perouz Taslakian', Georg Martius', Julius von Kügelgenn". and Francesco Locatello
IInstitute of Science and Technology Austria Max Planck Institute for Intelligent Systems, Tübingen *University of Amsterdam *University of Tübingen *University of Cambridge MIT-IBM Watson AI Lab "Samsung- SAIT AI Lab *Mila, Université de Montréal
ServiceNow Research
ABSTRACT
We present a unified framework for studying the identifiability of representations learned from simultaneously observed views, such as different data modalities. We allow a partially observed setting in which each view constitutes a nonlinear mixture subset of underlying latent variables, which can be causally related. We prove that the information shared across all subsets of any number of views can be learned up to a smooth bijection using contrastive learning and a single encoder per view. We also provide graphical criteria indicating which latent variables can be identified through a simple set of rules, which we refer to as identifiability algebra. Our general framework and theoretical results unify and extend several previous works on multi-view nonlinear ICA, disentanglement, and causal representation learning. We experimentally validate claims on numerical, image, and multi- modal data sets. Further, we demonstrate that the performance of prior methods is recovered in different special cases of our setup. Overall, find that access multiple partial views enables us to identify a more fine-grained representation, under the generally milder assumption of partial observability.
INTRODUCTION
Discovering latent structure underlying data has been important across many scientific disciplines, spanning neuroscience (Vigário al., 1997; Brown et al., 2001), communication theory (Ristaniemi. 1999; Donoho, 2006), natural sciences (Wunsch, 1996; Chadan & Sabatier, 2012; Trapnell et al., 2014), and countless more. The underlying assumption that many natural phenomena measured by instruments have simple structure that is lost in raw measurements. In the famous cocktail party problem (Cherry, 1953), multiple speakers talk concurrently, and while we can easily record their overlapping voices, we are interested in understanding what individual people are saying. From the methodological perspective, such inverse problems became common in machine learning with breakthroughs in linear (Comon, 1994; Darmois, 1951: Hyvärinen & Erkki, 2000) and non non-linear (Hy- varinen et al., 2019) Independent Component Analysis (ICA), and developed into deep learning methods for disentanglement (Bengio et al., 2013; Higgins et al., 2017). More recently, approaches to causal representation learning (Schölkopf et al., 2021) began relaxing the key assumption of inde- pendent latents central prior work (the independent in ICA), allowing for and discovering (some) hidden causal relations (Brehmer et al., 2022; Lippe et al., 2022; Lachapelle et al., 2022; Zhang al., 2023; Ahuja et al., 2023; Varici et al., 2023: Squires et al., 2023; von Kügelgen et al., 2023).
This problem often modeled as two-stage sampling procedure, where latent variables are sampled i.i.d. from a distribution Pz, and the observations x are functions thereof. Intuitively, the latent variables describe the causal structure underlying a specific environment, and they are only observed through sensor measurements, entangling them via so-called "mixing functions".. Unfor- tunately, if these mixing functions are non-lineat the recovery of the latent variables is generally impossible, even if the latent variables independent (Locatello et al., 2019; Hyvärinen & Pajunen, 1999). Following these negative results, the community has turned to settings that relax the i.i.d. condition in different ways. One particularly successful paradigm has been the assumption that data

---

Published as a conference paper at ICLR 2024
is not independently sampled, and in fact, multiple observations may refer to the same realization of the latent variables. This multi-view setup has generated a flurry of results in ICA (Gresele et al., 2020; Zimmermann et al., 2021; Pandeva & Forré, 2023), disentanglement (Locatello et al., 2020; Klindt et al., 2021; Fumero et al., 2023; Lachapelle et al., 2023; Ahuja et al., 2022), and causal representation learning (von Kügelgen et al., 2021; Daunhawer al., 2023; Brehmer al., 2022).
This paper provides a unified framework for several identifiability results in observational multi-view causal representation learning under partial observability. We assume that different views need not be functions of all the latent variables, but only of some of them. For example, person may undertake different medical exams, each shedding light on some of their overall health status (assumed constant throughout the measurements) but none offering comprehensive view. An X-ray may show a broken bone, an MRI how the fracture affected nearby tissues, and a blood sample may inform about ongoing infections. Our framework also allows for an arbitrary number of views, each measuring partially overlapping latent variables. It includes multi-view ICA and disentanglement as special cases.
More technically, prove that any shared information across arbitrary subsets of views and modal- ities can be learned up to a smooth bijection using contrastive learning. Non-shared information can also be identified if it is independent of other latent variables. With single identifiability proof, our result implies the identifiability of several prior works in causal representation learning (von Kügelgen al., 2021; Daunhawer al., 2023), non-linear ICA (Gresele et al., 2020), and disentangled represen- tations (Locatello et al., 2020; Ahuja et al., 2022). In addition to weaker assumptions, our framework retains the algorithmic simplicity of prior contrastive multi-view (von Kügelgen et al., 2021) and multi- modal (Daunhawer et al., 2023) causal representation learning approaches. Allowing partial observ- ability and arbitrarily many views, our framework significantly more flexible than prior work, allow- ing us to identify shared information between all subsets of views and not just their joint intersection.
We highlight the following contributions:
1. We provide unified framework for identifiability in observa- tional multi-view causal representation learning with partial observability. This generalizes the multi-view setting in two ways: allowing (i) any arbitrary number of views, and (ii) par- tial observability with non-linear mixing functions. We prove that any shared information across arbitrary subsets of views and modalities be learned up to smooth bijection us- ing contrastive learning and provide straightforward graphical criteria to categorize which latents can be recovered.
Zz
Z
Z
Z
S
S
fg
2. With single proof, our result implies the identifiability of several prior works in causal representation learning, non- linear ICA, and disentangled representations as special cases.
Figure 1: Multi- Setting with Partial Observability, for Exam- ple 2.1 with K=4 views and N=6 latents. Each view X is generated by a subset zs, of the latent variables through view-specific mixing func- tion fx. Directed arrows between la- tents indicate causal relations.
3. We conduct experiments for various unsupervised and super- vised tasks and empirically show that (i) the performance of prior works can be recovered using special setup of our framework and (ii) our method indicates promising disentan- glement capabilities with encoder-only networks.
2 PROBLEM FORMULATION
We formalize the data generating process as a latent variable model. Let z = (Zi,..., 2N) ~ Pz be possibly dependent (causally related) latent variables taking values in Z -, x ... x 2N. where Z C - is an open, simply connected latent space with associated probability density Pr. Instead of directly observing z, we observe set of entangled measurements views = (x xk.. Importantly, we assume that each observed view X may only depend on some of the latent variables,
which we call "view-specific latents" ZSE. indexed by subsets Sj.... Sx S [N] {1. (L.... N). For any A C [N], the subset of latent variables n, and corresponding latent sub-space Z are given by:
A == {zj E A}:
Z == jen 2,.
Similarly, for any V C [K]. the subset of views xv and corresponding observation space Xy are:
xv (x+ k k V),
Xy X-e Xx.
2

---

Published as a conference paper at ICLR 2024
The view-specific mixing functions [f+ : 2s. Xy [E[K are smooth, invertible mappings from the view-specific latent subspaces 2s, to observation spaces X S (ddi(x)) with X₃ = ft (zs,). Formally, the generative process for the views (xI..... xxx is given by:
2 ~ Pxa
X = (u(2s,) vk E [K],
i.., each view X depends on latents zs, through a mixing function fk. as illustrated in Fig.
Assumption 2.1 (General Assumptions). For the latent generative model defined above:
(i) Each view-specific mixing function f& is a diffcomorphism:
iii Ps is smooth and continuous density Z with pz > 0 almost everywhere.
Example 2.1. Throughout, we illustrate key concepts and results using the following example with K=4 views, N-6 latents, and dependencies among the shown graphical model in Fig. 1.
X1 fi 22. 23. Za, Zs):
X2 fz(Z1. Z2. Z3. Z5 Zo):
(2.1)
X3 fa (z) Z2 Z3, Zz Z6):
X4 = fa Zz. Zz Zg).
Consider a set xy of jointly observed views, and let V == {v S V |V] 2 2} be the set of subsets V, V indexing two more views. For any subset of views Vi, we refer to the set of shared latent variables (i.e., those influencing each view in the set) as the "content" or "content block" of V.. Formally, content variables zc, are obtained intersections of view-specific indexing sets:
C = N Sk
(2.2)
Similarly, for each view & E V. we can define the non-shared ("style") variables as Zs, (C We use C' and zc without subscript to refer the joint content across all observed views XV.
For Example 2.1, the content of xv; = (x1, X2) is 2c, = (Z1, 22, Z3, 2s); the content for all four views x = [x1, X2, X3, X4) jointly is ac (Z), 22), and the style for x is 2 = (Z3. Z4, 2s).
Remark 2.2 ("Content-Style Terminology). We adopt these terms from von Kügelgen et al. (2021), but note that, in our setting, they are relative to specific subset of views. Unlike in some prior works (Gresele et al., 2020; von Kügelgen et al., 2021; Daunhawer et al., 2023), style variables are generally not considered irrelevant, but may also be of interest and can sometimes be identified by other means (c.g., from other subsets of views or because they are independent of content).
Our goal is show that observed views under weak assumptions. This extends previous work (Gresele et al., 2020; von Kügelgen al., 2021; Daunhawer et al., 2023) where only one block of content variables is con- sidered. Isolating the shared content blocks from the rest of the view-specific style information, the learned representation (estimated content) can be used in downstream pipelines, such as classification tasks (Lachapelle et al., 2023; Fumero et al., 2023). In the best case, if each latent component is represented as one individual content block, we can learn a fully disentangled representation (Higgins et al., 2018 8; Locatello et al., 2020; Ahuja et al., 2022). To this end, we restate the definition of dentiiddetii (von Kügelgen al., 2021, Defn 4.1) for the molti-modal, multi-view setting:
simultaneously identify multiple content blocks given a set of jointly
Definition 2.3 (ioooiildetiffabiiittt).. The true content variables are block- bock-iidetifiedd by function g: X + R dim(c) if the inferred content partition è = g(x) contains all and only information about i.c., if there exists some smooth invertible mapping h : [gdim(e) + Ediu(e) s.t. è h(()..
Note that the inferred content variables è can be of entangled latent variables rather than single one. This differentiates our paper from the line of work on disentanglement (Locatello et al., 2020; Fumero al., 2023: Lachapelle et al., 2023). which seek to disentangle individual latent factors and can thus be considered as special cases of our framework with content block sizes equal to one.
IDENTIFIABILITY THEORY
High-Level Overview. This section presents unified framework for studying identifiability from multiple partial views: we start by establishing identifiability of the shared content block zc from any number of partially observed views (Thm. 3.2). The downside of this approach is that if we seek to
3

---

Published as a conference paper at ICLR 2024
learn content from different subsets, need to train an exponential number of encoders for the same modality, one for each subset of views. We, therefore, extend this result and show that by considering any subset of the jointly observed views, various blocks of content variables can be identified by one single view-specific encoder (Thm. 3.8). After recovering multiple content blocks simultaneously, we show in Cors. 3.9 to 3.11 that qualitative description of the data generative process such as in Fig. 1 can be sufficient to determine exactly the extent to which individual latents or groups thereof can be identified and disentangled. Full proof' are included in App.
Definition 3.1 (Content Encoders). Assume that the content size |C]| is given for any jointly observed
views Xv. The content encoders G := {g* Xs + (0, 1)|"" bKEv consist of smooth functions mapping from the respective observation spaces to the (CI-dimmnsional unit cube.
Theorem 3.2 (Identifiability from a Set of Views). Consider a set of views xy satisfying Asm. 2.1, and let G be a set of content encoders (Defi. 3.1) that minimizes the following objective
((G) = -keKey E (1lgx(Xx))
gke (xx Ml -kev H (gu (gx(xx));
(3.1)
Content aligument
Entropy regalarization
where the expectation is taken w.r.t. p(xv) and H-) denotes differential entropy. Then the shared content variable zc == {z, : E C} dlockideentified (Defn. 2.3) by ga E G for any h E V.
The alignment enforces the content encoders 9& only to encode content and discard styles, while the maximized entropy implies uniformity and thus invertibility. For Exam- ple 2.1, recall that the joint content is C = [ktt4 = 2}. Thm. 3.2 then states that, for each k 1,2, 3, 4, the content encoders G {gr X - (0. 1)1C which minimize the loss in eq. (3.1) are actually invertible mappings of the ground truth content {2), 22), i.e., gh (xx)x h4 (2) 22) for smooth invertible functions hgg 2, Z2 - 1)
Intuition.
Discussion. Thm. 3.2 provides a learning algorithm to infer one jointly shared content block for all observed views in a set, extending prior results that only consider two views (von Kügelgen et al., 2021; Daunhawer et al., 2023; Locatello et al., 2020). However, to discover another content block C; w.r.t. a subset of views V, V as defined in 2, we need train another set of encoders, since the di- mensionality of the content might change. Ideally, we would like to learn one view-specific encoder T& that maps from the observation space X's to some | -ddimensional manifold and can block-identify all shared contents zc, using one training run, combined with separate content selectors.
Definition 3.3 (View-Specific Encoders). The view-specific encoders == {rk Xx + Zs, hRv
consist of smooth functions mapping from the respective observation spaces to the view-specific latent space, where the dimension of the 1*th latent space |S&| is assumed known for all E V.
Intuition. The view-specific encoders learn all view-related content blocks simultaneously, instead of training combinatorial number of networks (as would be implied by Thm. 3.2). The view-specific encoders should learn not only a single block of content variables, but instead learn to recover all shared latents in way that makes easy to extract various different content blocks using simple readout functions. This is possible by construction, e.g., if each learns to invert the ground truth mixing ft. Inspired by this idea, we introduce content selectors.
Definition 3.4 (Selection). A selection 0 operates between two vectors a E [0,1]4 ,b Rd s.t.
a 2 b (b3 0 1,j [d]
Definition 3.5 (Content Selectors). The content selectors & (olitt Jev.Nen with (1.)) (0, 1>1اal perform selection (Defn. 3.4) the encoded information: for any subset V, and view is V, we have the selected representation: ((i.)) @ àss = (b.))) @ rx(x), with | llo |
for all V. E V, k. Ke E V.
Intuition.
Using learnable binary weights T. the content selectors (b(+))) should pick out those
latents among the representation åsç extracted by r'k that belong to the content block C₁ shared among V. For Example 2. consider a learned representation (x1) (2), 2g, 23, 2,, 2z)) Applying a content selector with weight ((i=1A=1) = 11, 1, then yields: (2), 22, 23, 255
4

---

Published as a conference paper at ICLR 2024
What is missing? While aligning various content blocks based on the same representation r' (xx)) should promote disentanglement, maximizing the entropy H(T (x+))xx)) of the learned representation (as in Thm. 3.2) promotes uniformity. The latter implies invertibility of the encoders (Zimmermann et al., 2021), which is necessary for block-identifibiiity (Defn. 2.3). However, since a uniform representation has independent components by definition, disentanglement and uniformity cannot be achieved simultancously unless all ground truth latents are mutually independent (a strong assumption we are not willing to make). Thus, to theoretically achieve invertibility while preserving disentanglement, we introduce set of auxiliary projection functions.
Definition 3.6 (Projections). The of projections T hev consist of functions tx Zst (0, 1)15kl mapping each view-specific latent space hyper unit-cube of the same dimension J.S&l.
The projection functions can be understood as mathematical tools: by maximizing the entropy and thus enforcing uniformity of projected representations l Th (xx)), ). can show that rkk needs to be invertible without interfering with the disentanglement of different content blocks.
Intuition.
What if the content dimension is unknown? In Thm. 3.2 we assumed that the size |C] of the shared content block is known, and the encoders map to a space of dimension |C]. In the following, we do not assume that the content size is given. Instead, we will show that the correct content block can still be discovered by ensuring that as much information as possible is shared across any given subset of views. To this end, we define the following information-sharing regularizer.
Definition 3.7 (Information-Shaaing Regularizer). The following regularizer penalizes the Lo- norm ll of the content selectors c: Reg((): == - Evvvv Σkev, ||o(0")]|
Intuition. Reg(4) (P) sums the number of shared latents over V. V and k E V,. It decreases when (f+.k) contains more ones, i.e., more latents are shared across views k V₁. Thus, Reg( ( encourages the encoders to reuse the learned latents and maximize the shared information content.
Theorem 3.8 View-Specifie Encoder for Identifiability). Let R, T. respectively be any view- specific encoders (Defn. 3.3), content selectors (Defn. 3.1) and projections (Defn. 3.6) that solve the following constrained optimization problem:
min Reg( (b)
subject to:
R, +. T E argmin L. (R, T,
(3.2)
where
C (
Σ V.EV K.E'VV kCk'
Σ
E
|1 ((ikk) @ @xx(xx) - ((LN) @ rw (xx) ||]
Σ
H (th (x(x))
REV
Entropy
Content aligament
(3.3)
Then for any subset of views V; E V and any view k E V,, (((.k) @ rk block-identifies (Defn. 2.3) the shared content variables 2c, as defined in eq. (2.2).
Intuition. For Example I. the representation for x obtained by minimizing eq. (3.2) is given by 2 == F1 (x₂) ) = Tı (fi (zs,) )) = 71 o fi Z2, Za. Za, 25). Consider the following two subsets of views V1, V/2 V containing X1, but sharing different content blocks C1, C2:
Xv. = [x1, x2}
zc, = { 1 Z2, Z3, ks},
XV,2 {x1, x3} ZC, = {zı, Z2, Z3, z1}.
Then one of the optimal solutions of the selectors learned by Thm. 3.8 could be
(1= = [1, 0, 1],
((==2 = [1, 1,
Hence, the composed results the selectors and the view-specific encoder "1 give:
)(i=1,k=1) @ à h;=1,k= (Z), Z2, Z3, 25),
((i=2,k=1) O 2 h; 22, 23, 21)
where hs,k=1 is some smooth bijection, for both = 1,2.
Discussion. Note that Equation (3.2) can be rewritten as regularized loss LReg (R, 4 T. L (R, & T) + a (eg( with a sufficiently small regularization coefficient a 2 Overall, Thm. 3.8 further weakens the assumptions of Thm. 3.2 in that content size is required. However,
5

---

Published as a conference paper at ICLR 2024
minimizing the information-shate sharing regularizer is highly non-convex, and having only finite number of samples makes finding the global optimum challenging. In practice, we could use Gumbel Softmax (Jang et al., 2016) for unsupervised learning, and consider content sizes as or follow the approach by Fumero et al. (2023) for supervised classification tasks. Empirically, we will that some of the requirements that needed in theory can be realistically dropped, and different approximations are possible, c.g., incorporating problem-specific knowledge.
After discovering various content blocks, we are further interested in how to infer more information from the learned content blocks. For example, can we identify 2c, == {Z1, Z2, Zz} 2cyn0s ? This perspective motivates our next results, which focus on how to infer new information based on the previously identified blocks: Identifiability Algebra.
Let 2C,; zC2 with C₁ C2 S [N] be two identified blocks of latents. Then holds for C1, C2 that:
Corollary 3.9 (Identifiability Algebra: Intersection). The intersection zc,nc, can be block- ineetideet
Corollary 3.10 (Identifiability Algebra: Complement). CfCn is independent of C1 C2. then the complement z XO2 can be block-identified.
Union). IfC, C and C2\C₁ are mutually inde- pendent, then the union 2CruC, can be block-identifid.
Corollary 3.11 (Identifiability Algebra:
Discussion. While Cor. 3.9 refines the identified block of information into smaller intersections, Cors. 3.10 and 3.11 allows extract "style" variables as defined w.r.t. some specific views, under the assumption that they are independent of the content block, as discussed by Lyu et al. (2021). However, our setup more general, as we can not only explain the independent style variables between pairs of observations, but also between learned content representations. Thus, by iteratively applying Cor. 3.10 we can generalize the statement to any number of identified content blocks. Combining Cors. 3.9 to 3.11 I we can immediately tell which part can be block-identified from a set of views V. given a graphical model representation such as Fig. I and subject to technical assumptions underlying our main results. Applying Cors. 3.9 to 3.11 iteratively on identified blocks can possibly disentangle each individual factors of variation, providing a novel approach for disentanglement. If all variables can be isolated up element-wise nonlinear transformations, we learn the causal relations by assuming the original link are nonlinear with additive noise. This exactly corresponds to a post-nonlinear model, whose graphic structure can be further identified using causal discovery algorithms (Zhang& & Chan, 2006; Zhang & Hyvärinen, 2009).
4 RELATED WORK AND SPECIAL CASES OF OUR THEORY
Our framework unifies several prior work, including multi- view nonlinear ICA (Gresele et al., 2020), weakly-supervised disentanglement (Locatello et al., 2020; Ahuja et al., 2022) and content-style identification (von Kügelgen et al., 2021; Daunhawer et al., 2023). Tab. shows summarized (non- exhaustive) list of related works and their respective graphical models that can be considered as special cases. The graphical setups of the individual works can be recovered from framework (Fig. 1) by varying the number of observed views and causal relations.
In addition, we present short overview of other related work which can be connected with theoretical results, including causal representation learning (Sturma et al., 2023; Silva et al., 2006; Adams et al., 2021; Kivva al., 2021; Cai et al., 2019; Xie et al., 2020; 2022; Morioka & Hyvärinen, 2023; Morioka & Hyvarinen, 2023), mutual information-bat based contrastive learning (Tian et al., 2020; Tsai al., 2020; Tosh et al., 2021), latent correlation maximization (Andrew et al., 2013; Benton et al., 2017; Lyu & Fu, 2020; Lyu et al., 2021), nonlinear ICA without auxiliary variables (Willetts & Paige, 2021) and multitakk disentanglement with sparse classifiers (Lachapelle et al., 2023; Fumero et al., 2023). Further discussion is given in App. B. We remark that several approaches here consider the setting where two observations are generated through an intervention on some latent variable(s). This is sometimes written in the graphical model two nodes connected by an arrow (shown in the graphs in Tab. as dashed lines- --+) indicating the pre- and post-intervention versions of the same variable(s). We stress that this does not constitute example of partial observability. In our setting, latent variables can be simply unobserved, regardless of whether or not they were subject to an intervention.
Causal representation learning. In the context of causal representation learning (CRL), Sturma et al. (2023) also explicitly consider partial observability in a linear, domain setting. Several other works on linear CRL from i.i.d. data could also be viewed as assuming partial observability, since

---

Published as a conference paper at ICLR 2024
Table 1: A non-exhaustive summary of special cases our theory and their graphical models. An asterisk (*) indicates works that have view-specific latents that are not of interest for identifiability.
Method
Graph
Dependent Latents
Multi-Modul
Partial Observability
Views
Schölkopp et (2016)
Gresele al. (2020)
K
Locutello (2020)
K
X
K
X
Abuja et al. (2022)
X
X
Kügelgen (2021)
X
x
x
Daunhawer et al. (2023)
x
Ours
Fig.
they often rely on graphical conditions which enforce each measured variable to depend on a single (a "pure" child) or only few latents (Silva al., 2006; Adams et al., 2021; Kivva et al., 2021; Cai et al., 2019; Xie et al., 2020; 2022). our framework, each view X instead constitutes a nonlinear mixture of several latents. Merging partially observed causal structure has been studied without representation learning component by Gresele et al. (2022); Mejia et al. (2022); Guo et al. (2023).
Mutual Information- Contrastive Learning. Tian et al. (2020); Tsai et al. (2020); Tosh et al. (2021) empirically showcase the success of contrastive learning in extracting task-related in- formation across multiple views, if the augmented views are redundant to the original data regarding task-related information (Tian et al., 2020). From this point of view, the redundant task- informnnonttnnn can be interpreted as shared content between the views, for which our theory (Thms. 3.2 and 3.8) may provide theoretical explanations for the improved performance in downstream tasks.
Latent Correlation Maximization, Prior work (Andrew et al., 2013; Benton et al., 2017: Lyu Fu, 2020; Lyu et al., 2021) showed that maximizing the correlation between the learned representation equivalent to our content alignment principle (eq. (3.1)). The additional invertibility constraint on the learned encoder in their setting is enforced by entropy regularization (eq. (3. explained by Zimmermann et al. (2021). However, their theory is limited to pairs of views and full observability, while we generalize it to any number of partially observed views.
Nonlinear ICA without Auxiliary Variables. Willetts & Paige (2021) shows nonlinear ICA problem be solved using non-observable, learnable, clustering task variables u, replace the observed auxiliary variable in other nonlinear ICA approaches (Hyvarinen et al., 2019). While we explicitly re- quire the learned representation be aligned continuous space within the content block, Willetts & Paige (2021) impose a soft alignment constraint to encourage the encoded information to be similar within cluster. practice, the soft alignment requirement can be easily coded in our framework by relaxing the content alignment with an equivalence class in terms of cluster membership.
Multi-task Disentanglement with Sparse Classifiers. Our setup is slightly different from that of Lachapelle et al. (2023); Fumero et al. (2023) as they focus multiple classification tasks using shared encoding and sparse linear readouts. Their sparse classifier head jointly enforces the sufficient representation (regarding the specific classification task, while we aim for the invertibility of the en- coders) and a soft alignment up to a linear equivalence class (relaxing our hard alignment). However, the identifiability principles use are similar: sufficient representation (entropy regularization). alignment and information sharing. While our results can be easily extended to allow for alignment up linear equivalence class, their identifiability theory crucially only covers independent latents.
7

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
First, we validate Thms. 3.2 and 3.8 using numerical simulations in a fully controlled synthetic setting. Next, we conduct experiments on visual (and text) data demonstrating different special cases that are unified by our theoretical framework ($ 4) and how we extend them. We use InfoNCE (Oord al., 2018) and BarlowTwins (Zbontar et al.. 2021) estimate eqs. (3.1) and (3.2). The content alignment is computed by the numerator (positive pairs) in InfoNCE and the entropy regularization is estimated by the denominator (negative pairs). Further remarks on contrastive learning and entropy regularization are in App. E. For the evaluation, we follow standard evaluation protocol (von Kügelgen et al., 2021) and predict the ground truth latents from the learned representation gk (Xx), using kernel ridge regression for continuous latent variables, and logistic regression for discrete ones, respectively. Then, we report the coefficient of determination R² to show the correlation between the learned and ground truth latent variables. An R² close to one between the learned and ground truth variables means that the learned variables are modelling correctly the ground truth, indicating ilock-identifiabiiityy (Defn. 2.3). However, R² is limited as a metric, since any style variable that strongly depends on content variable would also become predictable, thus showing a high H?² score.
5.1 NUMERICAL EXPERIMENT: THEORY VALIDATION
Experimental Setup. We generate synthetic data following eq. (2.1). The latent variables sampled from Gaussian distribution
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.8
(0, M,), where possible causal dependencies can be encoded through Σz. The view-specific mixing functions fx are implemented by ran- domly initialized invertible MLPs for each view k E (1)...4). We report here the R²² scores for the case of independent variables, because it is easier to interpret than the R² scores in the causally dependent case, for which we show that the learned representation still contains all and only the content information in App. D.1.
1.0
1.0
0.0
1.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.6
0.0
1.0
1.0
0.0
0.0
1.0
0.0
1.0
0.0
0.0
0.4
1.0
0.0
1.0
0.0
1.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
1.0
1.0
0.0
0.0
0.0
(K1.30)
(K).20)
(81:%)
(Ko (4o m
(Ru Ko]
(x) (s) 3
(Ro
(K:
Figure 2: Theory Validation: Average It across mul-
tiple views generated from independent latents.
Discussion. Fig. 2 shows how the averaged R² changes when including more views, with the y y-axis denoting the ground truth latents and the x-axis showing learned representation from different subsets of views. As shown in Example 2.1 and Fig. 1, the content variables are consistently identified. having R²2 x 1, while the independent style variables are non-predictable ( (R2 ~ 0). This numerical result shows that the learned representation explains almost all variation in the content block but nothing from the independent styles, which validates Thms. 3.2 and 3.8.
5.2 SELF-SUPERVISED DISENTANGLEMENT
Table 2: Self-Supervised Disentanglement Performance Comparison on MP1-3D com- plex (Gondal ct al., 2019) and 3Dldent (Zim- mermann et al., 2021), between our method and Ada-GVAE (Locatello et al., 2020).
Experimental Setup. We compare our method (Thm, 3.8) with Ada-GVAE (Locatello et al., 2019), on MPI-3D complex (Gondal et al., 2019) and 3DIdent (Zimmermann et al., 2021) image datasets. We did not compare with Ahuja et al. (2022), since their method needs know which latent is perturbed, even when guessing the offset. We experiment on a pair of views (x) where the second view X2 is obtained by randomly perturbing subset of latent factors of x1 following (Locatello et al., 2019). We provide more details about the datasets and the experiment setup in App. D.2. As shown in Tab. 2, our method outperformed the autoencoder- baasedddr--- Ada-GVAE (Locatello et al., 2020), using only encoder and computing contrastive loss in the latent space.
DCI disentanglement 1 MP13D complex
3DIdent
Ada-GVAA Ours
0.11 0.008
0.091 0.019
0.42.L 0,020
0.30.1 0.34
Discussion. As both methods are theoretically identifiable, we hypothesize that the improvement comes from avoiding reconstructing the image, which is more difficult on visually complex data. This hypothesis is supported by the fact that self-supervised contrastive learning has far exceeded the performance of autoencoder-base based representation learning in both classification tasks (Chen et al., 2020; Caron et al., 2021; Oquab et al., 2023) and object discovery (Seitzer et al., 2022).
8

---

Published as a conference paper at ICLR 2024
Dentoer
1.1
Content
1.0
style
0.3
0.5
cojoctshapp
cojectapcs
coject spos
0.5
2po0iggt pos
oojectsppce
0.0
cbpec, spos
otjer joiier Inder
cbject Jeetanor
tox phvasing
0.4
Spoeii Loolor
0.4
Object
0.2
bookground color
0.2
object alphart
object.zoloo
6.1
0.0
Ca
Cocar
Cuiar
So sotected Blocks
Selected Blodks
Figure 3: Simultaneous on Mildimodal331deett Left: Image latents (averaged hetween two image views) Right: Text latents.
Identification using View-Specific Encoders, Experimental results
5.3 MULTI-MODAL. CONTENT-STYLEE IDENTIFIABILITY UNDER PARTIAL OBSERVABILITY
Experimental setup. We experiment both (Daunhawer et al., 2023; von Kügelgen et al., 2021), which are limited to two views, either two images one image and its caption. The second image view img is generated by perturbing a subset of latents of imgo as in (von Kügelgen et al., 10.1). Notice that this setup provides perfect partial observability, because the text is generated using text-specific modality variables that are not involved in any image views e.g., text phrasing. We train view-specific encoders to learn all content blocks si- multaneously and predict individual latent variables from the each learned content blocks. We assume access to the ground truth content indices to better match the baselines, but we relax this in App. D.4.
set of three views (imgo, img1; txto) extending
Discussion. Fig. reports the R² on the ground truth latent values, predicted from the simultaneously learned multiple content blocks (Cx,,x, Cx,,xa X2 Cx xa, Cx X2, respectively). We remark that this single experiment recovers both experimental setups from (von Kügelgenn et al., 2021, Sec 5.2), (Daun- hawer et al., 2023, Sec 5.2): Cx represents the content block from the image pairs (imgo- img, ) which aligns with the setting in (von Kügelgen et al., 2021) and Cx..x. shows the content block from the multi-modal pair (imgo: txto), which is studied by Daunhawer et al. (2023). We observe that the same performance for both prior works (von Kügelgen et al., 2021; Daunhawer et al., 2023) has been successfully reproduced from single training process, which verifies the effectiveness and efficiency of Thm. 3.8. Extended evaluation and more experimental details are provided in App. D.4.
5.4 MULTI-TASK DISENTANGLEMENT
Experimental setup We follow Example 2.1 with latent causal relations to verify that: (i) the improved classification performance from (Lachapelle et al., 2023; Fumero et al., 2023) originates from the fact that the task-related information is shared across multiple views (different observations from the same class) and (ii) this information can be identified (Thm. 3.2), even though the latent variables are not independent. This explains the good performance of (Fumero et al., 2023) on real-world data sets, where the latent variables are likely not independent, violating their theory.
Discussion, We synthetically generate the labels by linear/nonlinear labeling functions on the shared content values to resemble (Lachapelle et al., 2023; Fumero al., 2023). As expected, the learned representation significantly eases the classification task and achieves an accuracy of 0.99 with linear and nonlinear labeling functions within IK update steps, even with latent causal relations. This experimental result justifies that the success in the empirical evaluation of (Fumero et al., 2023) can be explained by our theoretical framework, as discussed 4.
DISCUSSION AND CONCLUSION
This paper revisits the problem of identifying possibly dependent latent variables under multiple partial non-lincar measurements. Our theoretical results extend to an arbitrary number of views, each poten- tially measuring strict subset of the latent variables. In our experiments, we validate our claims and demonstrate how prior work can be obtained as special case of our setting. While our assumptions are relatively mild, we still have notable gaps between theory and practice, thoroughly discussed in App. E. In particular, we highlight discrete variables and finite-sample errors as common gaps, which we ad- dress only empirically. Interestingly, our work offers potential connections with work in the causality literature (Triantafillou et al., 2010; Gresele al., 2022; Mejia et al., 2022; Guo et al., 2023). Discov- ering hidden causal structures from overlapping but not simultancously observed marginals (c.g.. via views collected in different experimental studies different times) remains open for future works.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
The datasets used in ($ 5) are published by Gondal et al. (2019); Zimmermann et al. (2021); von Kügelgen et al. (2021); Daunhawer et al. (2023). Results provided in the experiments section ($ 5) can reproduced using the implementation details provided App. D. The code available at https://github..
The part of implementation to replicate the experiments Fumero et al. (2023) $ 5.4 was kindly provided by the authors upon request, and we do not include it in the git repository.
ACKNOWLEDGEMENT
This work was initiated at the Second Bellairs Workshop on Causality held at the Bellairs Research Institute, January 6-13, 2022; we thank all workshop participants for providing stimulating research environment. Further, we thank Cian Eastwood, Luigi Gresele, Stefano Soatto, Marco Bagatella and A. René Geist for helpful discussion. GM is member the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. JyK and GM acknowledge support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 011S (IIS18039B). The research of DX and SM was supported by the Air Force Office of Scientific Research under award number FA8655-222 21-5-222-1-7155 1-7155. Any opinions, findings, and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force. We also thank SURF for the support in using the Dutch National Supercomputer Snellius. SL was supported by IVADO excellence Phl) scholarship and by Samsung Electronics Co., Ldt. DY was supported by an Amazon fellowship, the International Max Planck Research School for Intelligent Systems (IMPRS-IS) and the ISTA graduate school. Work done outside of Amazon.
REFERENCES
Jeffrey Adams, Niels Hansen, and Kun Zhang. Identification of partially observed linear causal models: Graphical conditions for the non- gaussian and heterogencous cases. Advances in Neural Information Processing Systems, 14-22822-22833. 2021. 6,
Kartik Ahuja, Jason Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 35:15516-15522 2022. 2, 3, 6, 7, 8, 17
Kartik Ahuja, Divyat Mahajan, Yixin Wang. and Yoshua Bengio. Interventional causal representation learning. In International Conference on Machine Learning, pp. 372-407. PMLR, 2023. 1
Rana Ali Amjad and Bernhard C. Geiger. Learning representations for neural Hltwork-based classification using the information bottleneck principle. IEEE Transactions Pattern Analysis and Machine Intelligence, 42(9))2225-2233) sep 2020. 18
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1247-1255, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. 6, 6,7,
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Adrian Benton, Huda Khayrallah, Biman Gujral, Dee Ann Reisinger, Sheng Zhang, and Raman Arora. Deep generalized canonical correlation analysis. arXiv preprint 10Xiv= 2017. 6, 7, 18
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319-38331, 2022. 1,2
Glen D Brown, Satoshi Yamada, and Terrence J Sejnowski. Independent component analysis at the neural cocktail party. Trends in neurosciences, 24(1):54-63, 2001. 1
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf. and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. arXiv preprint arXiv:2306.022333 2023. 27, 32
10

---

Published as a conference paper at ICLR 2024
Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad constraints for learning causal structure of latent variables. In Advances in Neural Information Processing Systems, volume 32, pp. 12883-12892, 2019. 7
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the International Conference Computer Vision (ICCV), 2021. 8
Khosrow Chadan and Pierre C Sabatier. Inverse problems in quantum scattering theory. Springer
Science & Business Media, 2012.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020. 31
Colin Cherry. Some experiments on the recognition speech, with one and with two ears. The Journal of the acoustical society of America, 25(5):975-995 9999, 1953. 1
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
George Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, pp. 231, 1951. I, 19, 23
Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifiability results for multimodal contrastive learning. In The Eleventh International Conference Learning Representations, 2023. 2, 3, 4, 6, 7, 9, 10, 18, 22, 26, 29, 30, 31
David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4):1289-1306 2006. 1
Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolá, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning, 2023. 2, 3, 6, 9, 10, 18, 30, 31
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. the transfer of inductive bias from simulation to the real world: new disentanglement dataset.
Advances in Neural Information Processing Systems, 32, 2019. 8, 10. 27
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf.
The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence, pp. 217-227. PMLR, 2020. 2, 3, 3, 6,
Luigi Gresele, Julius von Kügelgen, Jonas Kübler, Elke Kirschbaum, Bernhard Schölkopf, and Dominik Janzing. Causal inference through the structural causal marginal problem. In International Conference on Machine Learning, pp. 7793-7824. PMLR, 2022. 7, 31
Siyuan Guo, Jonas Wildberger, and Bernhard Schölkopf. -atiil-variable generalization. arXiv
preprint arXiv:2304.07896, 2023. 7,9
Simon Haykin. Neural networks: a comprehensive foundation. foundaattiiie Prentice Hall PTR, 1994. 26
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition,
pp. 770-778, 2016. 27
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition disentangled representations. arXiv preprint arXiv: (8/2 2018.
11

---

Published as a conference paper at ICLR 2024
Aapo Hyvärinen and Oja Erkki. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411-300 2000.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-444 4399, 1999. 17
Aapo Hyvarinen, Hiroaki Sasaki, and Richard E. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning, 2019. 7, 17
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.0114 2016.
lless Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. volume 33, pp. 12768-12778, 2020.
27
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint orXiv::412.660 2014. 27
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114, 2013. 17
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal graphs via mixture oracles. In Advances in Neural Information Processing Systems, volume 34, pp.
11887-11101, 2021. 7
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 15687-15701. Curran Associates, Inc., 2022. 17
David A Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations, 2021.
Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International Conference on Machine Learning, pp. 1455-11472. PMLR, 2022. 18
Sébastien Lachapelle, Rodriguez Lopez, Pau, Yash Sharma, Katie E. Everett, Rémi Le Priol, Alexan- dre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In First Conference on Causal Learning and Reasoning, 2022.
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: General- ization and identifiability in multi-task learning. In International Conference on Machine Learning, pp. 18171-18206. PMLR, 2023. 2, 3, 6, 7, 18, 30
Wendong Liang, Armin Kekić, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal component analysis. arXiv preprint arXiv:2305.17225. 2023. 32
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pp. 13557-13603. PMLR, 2022.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying weight- variant latent causal models. arXiv preprint 2208 14153, 2022. 17
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. international conference on machine learning, pp. 41 4114-4124, PMLR,
2019. 1,
12

---

Published as a conference paper at ICLR 2024
Francesco Locatello, Ben Poole, Gunnar Raetsch, Bernhard Schölkopf, Olivier Bachem, and Michael
Tschannen. Weakly-supervine disentanglement without compromises. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 6348-6688 6359. PMLR, 13-18 Jul 2020. 2, 3, 4, 6, 7, 8, 27
Qi Lyu and Xiao Fu. Nonlinear multiview analysis: Identifiability and neural network-assisted implementation. IEEE Transactions Signal Processing, 68:2697-2712, 2020. 6, 7, 18
Qi Lyu, Xiao Fu, Weiran Wang. and Songtao Lu. Understanding latent correlation- prased multiview learning and self-supervision: An identifiability perspective. arXiv preprini arXiv:2106.0711 15, 2021. 6, 18, 26
Sergio H Garrido Mejia, Elke Kirschbaum, and Dominik Janzing. Obtaining causal information by merging datasets with maxent. In International Conference on Artificial Intelligence and Statistics, pp. 581-603. PMLR, 2022. 7,9
Nathan Juraj Michlo. Disent a modular disentangled representation learning framework for pytorch. Github, 2021. 27
Hiroshi Morioka and Aapo Hyvärinen. Causal representation learning made identifiable by grouping of observational variables. arXiv preprint (xXXi::2110.17009 2023. 6
Hiroshi Morioka and Aapo Hyvarinen. Connectivity-connrastiv learning: Combining causal dis- covery and representation learning for multimodal data. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (eds.), Proceedings of The 26th International Conference on Artifi- cial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 3399-3426. PMLR, 25-27 Apr 2023. 6
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018. 8, 31
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 2023. 8
Teodora Pandeva and Patrick Forré Multi-view independent component analysis with shared and individual sources. In Uncertainty in Artificial Intelligence, pp. 1639-1650. PMLR, 2023. 2
Tapani Ristaniemi. On the performance of blind source separation in edma downlink. In Proceedings of the International Workshop on Independent Component Analysis and Signal Separation (ICA '99), pp. 437-441, 1999.
Bernhard Schölkopf, David W Hogg, Dun Wang, Daniel Foreman- Mackey, Dominik Janzing, Carl- Johann Gabriel,, and Jonas Peters. Modeling confounding by half-sibling regression. Proceedings of the National Academy of Sciences, 113(27):7391-7398, 2016.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, (09(5):612-634 2021. 1
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap real-world object-centric learning. In The Eleventh International Conference on Learning Representations, 2022. 8
Ricardo Silva, Richard Scheines, Clark Glymour. Peter Spirtes, and David Maxwell Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research, (2), 2006. 6, 7
Chandler Squires, Anna Seigal, Salil S. Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In International Conference on Machine Learning, volume 202, pp. 22540-32560 PMLR, 2023.
13

---

Published as a conference paper at ICLR 2024
Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler. Unpaired multi-domain causal representation learning, 2023. 6
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020. 6, 7, 18
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning. multi-view redun- dancy, and linear models. Algorithmic Learning Theory, pp. 1179-12066 PMLR, 2021. 6, 7, 18
Cole Trapnell, Davide Cacchiarelli, Jonna Grimsby, Prapti Pokharel, Shuqiang Li, Michael Morse, Niall J Lennon, Kenneth J Livak, Tarjei S Mikkelsen, and John L Rinn. The dynamies and regulators of cell fate decisions are revealed by pseudotemporal ordering of single cells, Nature biotechnology, 32(4):381-386 2014.
Sofia Triantafillou, Ioannis Tsamardinos, and Ioannis Tollis. Learning causal structure from over- lapping variable sets. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 860-867, 2010. 9, 31
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self-supervised learning from multi-view perspective. In International Conference on Learning Representations, 2020. 6, 7, 18
Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.082.30 2023. 1
Ricardo Vigário, Veikko Jousmäki, Matti Hämäläinen, Riita Hari, and Erkki Oja. Independent component analysis for identification of artifacts in magnetoenceephalograpicc recordings. Advances in neural information processing systems, 10, 1997.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supprvised learning with data augmentations provably isolates content from style. Advances neural information processing systems, 2021. 2, 3, 4, 6, 7, 8, 9, 10, 18, 20, 22, 26, 29, 30, 31
Julius von Kügelgen, Michel Besserve, Wendong Liang. Luigi Gresele, Armin Kekić, Elias Barein- boim, David M Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representa- tions from unknown interventions. arXiv preprint
2023. 27, 32
Matthew Willetts and Brooks Paige. don't need Identifiable non-linear ica without side informa- tion. arXiv preprint vrXiv:2106.05288 2021. 6, 7, 17
Carl Wunsch. The ocean circulation inverse problem. Cambridge University Press, 1996.
Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in Neural Information Processing Systems, volume 33, pp. 14891-14902. 2020. 6,
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identification of linear non-gausse non-gaussian latent hierarchical structure. In International Conference on Machine Learning. pp. 20320-244425 24387. PMLR, 2022. 6,7
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1rXvv:55050085:: 2015. 26
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint 10X.v:2103.032.0, 2021. 8, 28, 29, 30,
Jiaqi Zhang, Chandler Squires, Kristjan Greenewald, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. arXiv preprint arXiv:2307.06250 2023.
K Zhang and A Hyvärinen. On the identifiability of the post-nonlinear causal model. In 25th Conference Uncertainty in Artificial Intelligence (UAI 2009), pp. 647-655. AUAI Press, 2009. 6
14

---

Published as a conference paper at ICLR 2024
Kun Zhang and Lai Chan. Extensions of ica for causality discovery in the hong kong stock market. In International Conference on Neural Information Processing, pp. 400-409. Springer, 2006. 6
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. 18
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. Proceedings of Machine Learning Research, 139: 19:11997-122990, 2021. 2, 5, 7, 8, 10, 17, 19, 22, 24, 25, 26, 27,
15

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Notation and Terminology
16
B Related work and special cases of our theory
16
Proofs
18
C.I Proof for Thm. 3.2
18
C.2 Proof for Thm. 3.8
22
Proofs for Identifiability Algebra
26
D Experimental Results
26
Numerical Experiment Theory Validation
26
D.2 Self Supervised Disentanglement.
27
D.3 Content-Styl Identifiability on Images
29
D.4 Multi-modal Content-Style Identifiability under Partial Ohservability
29
Multi-Task Disentanglement with Sparse Classifiers
30
E Discussion
31
A NOTATION AND TERMINOLOGY
C Index set for shared content variables
zc Shared content variables
Collection of subset of views from V
V, Index set for subset of views in set of views V
C Index set for shared content variables from subset of views V, V
zc. Shared content variables from subset of views V;
N Number of latents
K Number of views
j
Index for latent variables
Index for views
2
Latent space
X
Observational space for k-th view
X
Observed k-th view
Sp Index set for k-th view-specific latents
V
(I.....(}
B
RELATED WORK AND SPECIAL CASES OUR THEORY
We present our identifiability results from $ 3 as a unified framework implying several prior works in multi-view nonlinear ICA, disentanglement, and causal representation learning.
16

---

Published as a conference paper at ICLR 2024
Multi- View Nonlinear ICA Gresele et al. (2020) extend the idea of nonlinear ICA introduced by Hyvarinen et al. (2019, Sec. 3) by allowing a more flexible relationship between the latents and the auxiliary variables: instead of imposing conditional independence the shared source information c on some auxiliary observed variables, Gresele et al. (2020) associate the shared information source with some view-specifie noise variable n; through some smooth mapping 9k:
X fk (c, n))),
k E [K]
Define the composition of the view-specific function fs and the noise corruption function 9t: as a new mixing function fx := f& o 9k., then each view Xs, k E [K] is generated by a view-specific mixing function für which takes the shared content с and some additional unobserved noise variable ny as imput, that is: Xx = fu (c. ng). In this case, the shared source information s together with the view-specific latent noise nu defines the view-specific latents Sy in notation. The shared source corresponds to our content variables. Our results Thm. 3.2 can be considered as a generalized version of Gresele et al. (2020, Theorem 8) for multiple views, by removing the additivity constraint of the corruption function g (Gresele et al., 2020, Sec 3.3).
Willetts & Paige (2021); Kivva et al. (2022); Liu et al. (2022) show the nonlinear ICA problem can be solved using non-observable, learnable, clustering task variables u to replace the observed the auxiliary variable in the traditional nonlinear ICA literature (Hyvärinen & Pajunen, 1999). Conditioning on the same latents on various clustering task enforces recovering the true latent factors (up bijective mapping). The idea of utilizing the clustering task goes hand in hand with contrastive self-supervised learning. Clustering itself can be considered as a soft relaxation of our hard global invariance condition in eq. (3.2), in the sense they enforce the shared, task-relevan features to be similar within a cluster but not necessarily having the exact same value.
Weaaly-SSperrvseed Representation Learning Locatello et al. (2020) consider a pair of views (e.g., images) X2) where x2 is obtained by perturbing a subset of the generating factors of x Formally,
X1 = fz)
X2 = f(z)
z, 2 E Rd
(B.1)
where 2y = 2 while zg F 2g for some subset of latents S S [d]. In this case, 25 is the shared content between the pair of views X2, According to the adaptive algorithm (Sec 4.1 Locatello et al., 2020), the shared content is computed by averaging the encoded representation from X1, X2 across the shared dimensions, that is:
g(xx); +
j EES
(B.2)
By substituting the extracted representation using the averaged value, Locatello et al. (2020) achieve the same invariance condition enforced in the first term of eq. (3.1). The amortized encoder is trained to maximize the ELBO (Kingma & Welling, 2013) over the pair of views, which equivalent to minimizing the reconstruction loss
E xx dec(g(xx))] E [x2- - dec(g((x))]
(B.3)
The reconstruction loss minimized when the compression is lossless, or equivalently, the learned representation is uniformly distributed (Zimmermann et al., 2021). The uniformity of the represen- tation, i.e., the lossless compression, can be enforced by maximizing the entropy term, as defined in eq. (3.1). Theoretically, Locatello et al. (2020, Theorem 1.) have shown that the shared content z, can be recovered up to permutation, which aligns with our results Thm. 3.8 that the shared content can be inferred up to smooth invertible mapping.
Ahuja et al. (2022) extend Locatello et al. (2020) by exploring more perturbations options to achieve full disentanglement. The main results (Ahuja et al., 2022, Theorem 8) state that each individual factor of a d' -dimensionnl latents can be recovered (up to a bijection) when we augment the original observation with d views, each obtained perturbing one unique latent component. This can be explained by Thm. 3.8 because any (d 11) views from this set would share exactly one latent component, which makes it identifiable. Although the theoretical claim by Ahuja et al. (2022) is to some extent aligned with our theory, in practice, they explicitly require knowledge of the ground truth content indices while we do not necessarily.
17

---

Published as a conference paper at ICLR 2024
Mutual Information-basee Framework Tian al. (2020); Tsai et al. (2020) argue that the self-supervised signal should be approximately redundant to the task-related information. The self- supervised learning methods are based on extracting the task-relevant information (by maximizing the mutual information between the extracted representation 2x of the input x and the self-supervised signal s: T(xx,)) and discarding the task-irrelevant information conditioned task T: T(x, S (7). The mutual information 1(2x.s) s)) is maximized if s is deterministic function (for example, MLP) of & (Amjad & Geiger, 2020, Theorem 1.). Since the mutual information remains invariant under deterministic transformation of the random variables, we have:
max (2, s) max / (2, g()) max / (2. = max (2)
(B.4)
which is equivalent to maximizing the entropy of the learned representations, as given in eq. (3.1). Coupled with the empirically shown strong connection between the task-related information and shared content across multiple views (Tian et al., 2020; Tsai et al., 2020; Tosh et al., 2021; Lachapelle al., 2023; Fumero et al., 2023), our results (Thms. 3.2 and 3.8) provides theoretical explanation for these approaches. As the shared content between the original view and self-supervised signal is proven to be related the ground truth task-related information through a smooth invertible function, it is reasonable to see the usefulness of this high quality representation in downstream tasks.
Latent Correlation Maximization Similar alignment conditions, as given in eq. (C.1), have been widely studied in the latent correlation maximization latent component matching literature (Andrew et al., 2013; Benton et al., 2017: Lyu & Fu, 2020; Lyu et al., 2021). Lyu et al. (2021, Theorem 1.) show that, by imposing additional invertibility constraint on the encoders latent correlation maximization across two views leads to identification of the shared component, up to an invertible function. This theoretical result can be considered as explicit special case of Thm. 3.2, where we extend the identifiability proof to more than two multi-modal views.
Content-Style Identification Our work is most closely related to (von Kügelgen et al., 2021; Daunhawer et al., 2023), while our results extended prior work that purely focused on identifiability from pair of views (von Kügelgen et al., 2021; Daunhawer et al., 2023). von Kügelgen et al. (2021, Theorem 4.2) presented special case of Thm. 3.2 where the set size = 2 and the mixing function fi f2 for both views; Daunhawer et al. (2023, Theorem 1) formulate another special case of Thm. 3.2 by allowing multi modality in the pair of views, but coming with the restriction that the view-specific modality variables have to be independent from others. From the data generating perspective, our work differs from prior work in the sense that all of the entangled views are simultaneously generated, each based on view-specific set of latent, while prior work generate the augmented (second) view by perturbing some style variables. In our case, "style" is relative to specific views. Style variables could become the content block for some set of views (Thm. 3.2) and thus be identifiable or can be inferred as independent complement block of the content (Cor. 3.10) Kong et al. (2022) have proven identifiability for independent partitions in the latent space but mostly focus on the domain adaptation tasks where additional targets are required as supervision signals.
Multi-Task Disentanglement Lachapelle et al. (2023); Fumero al. (2023); Zheng et al. (2022) differs from our theory in the sense that their sparse classifier head jointly enforces the lossless compression (which we do with the entropy regularization) and a soft alignment up to a linear transformation (relaxing our hard alignment). In their setting, the different views are images of the same class and their augmentations sampled from a given task and the selector variable is implemented with the linear classifier. The identifiability principles we use of lossless compression, alignment, and information sharing are similar. With this, we can explain the result that task-related and task- irrelevant information can be disentangled as blocks, given in Lachapelle et al. (2023, Theorem 3.1), Fumero et al. (2023, Proposition 1.). With our theory, their identifiability results extend to non-independent blocks, which is an important case that is not covered in the original works.
C PROOFS
C. PROOF FOR THM. 3.2
Our proof follows the steps from von Kügelgen al. (2021) with slight adaptation:
1. We show in Lemma C.1 that the lower bound of the loss eq. (3.1) is zero and construct encoders (o) X' - (0, 1)1o))eev that reach this lower bound;
18

---

Published as a conference paper at ICLR 2024
2. Next, show in Lemma C.3 that for any set of encoders (9x hev that minimizes the loss, each learned 9 (x+) depends only on the shared content variables zc, i.c. g& (xx)) = h (zc)
for some smooth function lyg Zc - (0, 1)(c1.
3. We conclude the proof by showing that every h4 is invertible using Proposition I (Zimmer- mann et al., 2021, Proposition 5.).
We rephrase each step separate lemma and use them to complete the final proof for Thm. 3.2.
Lemma C.1 (Existence of Optimal Encoders). Consider a jointly observed set of views xv, sat- isfying Asm. 2.1. Let Sk - [N], k V be view-specific indexing sets of latent variables and define the shared coordinates == NREV Sy- For any content encoders G == {9k Xx + (0, (8.1)1l) kEEv (Defn. 3.1), we define the following objective:
(G)) = ΣE [19x
Σ H (gx(xx)) KEV
(C.1)
K.K'EV
KEW
where the expectation is laken with respect to p(xv) and where H() denotes differential entropy. Then the global minimun of the loss (eq. (C.1)) is lower bounded by zero, and there exists set of content encoders Defrn. 3.1 which obtains this global minimum.
Proof. Consider the objective function C(G) defined in eq. (C.1), the global minimum of L(G) is obtained when the first term (alignment) minimized and the second term (entropy) is maximized. The alignment term is minimized to when ga are perfectly aligned for all & E V. i... 9k (Xx)) ge (xx) for all Xy ~ Pxy. The second term (entropy) is maximized to zero only when go (xx)) ) is uniformly distributed on (0, 1)|CI for all views k E V.
To show that there exists set of smooth functions: G (9k that minimizes L(G), consider the inverse function of the ground truth mixing function fx 1:|C1' w.l.o.g. assume that the content variables are at indices 1 |C',. This inverse function exists and a smooth function given by Asm. 2. 1() that each mixing function fy is a smooth invertible function. By definition, we have fx 1:|C] (xx) ze for l V.
Next, we define a function d using Darmois construction (Darmois, 1951) as follows:
d (zc) := Pj (z) 111-1) jE |C]] where Fj denotes the conditional cumulative distribution function (CDF) of 2j given Z1:--1. i.c. F, (1)( P (Z, S 23 (1)- By construction, d (Rc) is uniformly distributed on (0. 1)
(C.2)
Moreover, is smooth because Ps is a smooth density by Asm. 2. 1ii) and because conditional CDF of smooth densities is smooth
Finally, we define
9k = do 1:| X + (0, E
(C.3)
which is smooth function as a composition of smooth functions.
Next, we show that the function set G as constructed above attains the global minimum of L(G).
Given that SE 1:11 (xx) = - 1:1c](XN) (xx) = zc, Vki k'' E V, we have:
(G) Σ
E [llg - gk (xx))122
Σ H (xx))
K.V'VV
REV
k<ck
= Σ
(zlld - (2c))la)) - Σ H (d
(C.4)
K.K'eV
RgV
beck'
where ae is the shared content variables thus the first term (alignment) equals zero; and since d (zc) uniformly distributed on (0, 1)| the second term (entropy) is also zero.
To this end, we have shown that there exists a set of smooth encoders G == (9* with 9a as
defined in eq. (C.3) which minimizes the objective L(G) in eq. (C.1).
19

---

Published as a conference paper at ICLR 2024
Lemma C.2 (Conditions of Optimal Encoders). Assume the same set of views xy as introduced in
Lemma C.I, then for any set of smooth encoders G := {gk: X₂ - (0, 1) to obtain the global
minimum (zero) of the objective (()) in eq. (C.1), the following two conditions have to be fulfilled:
Invariance: All extracted representations 25 gx(xx) must align across the views from the set almost surely:
gk (xx) = (xx) Ak, EV 0.8.
(C.5)
Uniformity: All extracted representations 2n == gh (Xx) must be uniformly distributed over the hyper-cube (0, (0,1)(c)
Proof. Given that G = argmin L(G), we have by Lemma C.I:
C (G)
Σ
E (xl))(xx))
ghe (xx) - Σ
H (xx()x)) =
(C.6)
N.VVV
REV
The minimum L(G) =0 leads to following conditions:
E [llax (xx) gh (xx l12] 0 Vk, E V H (gx( = V
(C.7)
(C.8)
where eq. (C.7) indicates the invariance condition holds for all views 2y and smooth encoders Is G almost surely; and eq. (C.8) implies that the encoded information go (xx) must be uniformly distributed on (0, I)(C).
Lemma C.3 (Content- -ooteentSStyle Isolation from Set of Views). Assume the same set of views xy as
introduced in Lemma C.I, then for any set smooth encoders G = {gg= X (0, 1) that satisfies the Invariance condition (eq. (C.5)), the learned representation can only be dependent on the content variables zc = {zj J E C), not any style variables z Z for all V.
Proof. Note that the learned representation can rewritten as:
gu (xx) = (f(((z)))) EVV,
(C.9)
we define
:k fk V.
(C.10)
Following the second step of the proof from von Kügelgen al. (2021, Thm. 4.2), we show by contradiction that both h& (Zs,) for all & E V can only depend on the shared content variables zc.
Let k E V be any view from the jointly observed set, suppose for a contradiction that hg =
ha (M(2s))1:c depends some component 2g from the view-specific latent variables zi:
Thg Ozg
3g E f1...., dim (zz)). zs = (z zx E Zk: s.t.
[zc
2
+ 0,
(C.11)
which means that partial derivative of hy w.r.t. some latent variable Z, E 2 is non-zero at some point 2s, (z 2x ) E Zz. Since hy smooth, its first-order (partial) derivatives are continuous. By continuity of the partial derivatives, dz,
ane must be non-zero in a neighborhood of (zo, 2x i.e.
3n > 0 s.l. Zq + (2c. 2h,, 2n) is strictly monotonic on (zg - 11 2g + 1n,
(C.12)
where z=_ denotes the remaining view-specific style variables except 2g-
Next, we define an auxiliary function for each pair of views (k:, k²) with k R E V < R: Uh.
Zc x Zs.i Esgii
R20
(Zc, zi, 24.) |/f (zc, z1.) hw (zc, 2i-)
(C.13)
= -ng (zs,.) - hi (zs.)) ) 2 0.
20

---

Published as a conference paper at ICLR 2024
Summarizing the pairwise auxiliary functions, have v Zc IIKEv 25,/0 Rz0 follows:
Σ
lng (zc, 2 hw (Zc, 23,)
A.V'EV
k<ck'
(C.14)
Σ AWVVV
hi (Zs,,) Kgg (zsw) 2
R<k'
To obtain a contradiction the invariance condition in Lemma C.2, it remains show that s from eq. (C.14) is strictly positive with a probability greater than zero w.r.. the true generating process pt in other words, there has exist least one pair of views (k,k)) s.t. th,, with probability greater than zero regarding p.
Since g E Sy \ C, there exists least one view k + s.t. q Sy (otherwise the content block C would contain q). We choose exactly such a pair of views k. k..
Depending whether there zero point 2g of &b within the region (Zg TJ 2q 17), there are two cases to consider:
If there is no zero-point zig E (Zy - 17, + 17) s.t. vlk. (zo. (z =g)) 2 = 0, then it implies
tkk, (20 (2k ,) 2
0 V2g (2g 77 2g 17)
(C.15)
So there is an open = (2g - 17, 24 77) By such that the equation & in eq. (C.14) is strictly positive.
Otherwise, there is zero point zy from the interval (zn 7) Fq + 17) with
4/a,k' c (2% -2) = O zq E (2g - 17, 2g + 17),
(C.16)
then strict monotonicity from eq. (C.12) implies that the. > 0 for all 2g in the neighborhood 20 therefore:
(zzc (KE))))) > 0 Vzg E A = (2g 17, 2 U (=g Fgg + +))
(C.17)
Since v is sum of compositions of two smooth functions (absolute different of two smooth functions), is also smooth. Consider the open set IR and note that, under continuous function, pre-images of open sets are always open. For the continuous function vh, its pre-image I corresponds to an open set:
u C Zc X II ZSAc
(C.18)
KEV
in the domain of eb on which v is strictly positive. Moreover, since eq. (C.17) indicated that for all Zg A, the function vp is strictly positive, which means:
II
(Iz x
A
II S
(2 Cu,
(C.19)
AegESAAC
hence, U is non-emply.
Given by Asm. 2.1 (ii) that Ps smooth and fully supported (px almost everywhere), the non-empty set U is also fully supported by Pz, which indicates:
P (wize, (ev) > 0) 2 P (M) > 0,
(C.20)
where IP denotes the probability w.r.t the true generative process
According to Lemma C.2, the invariance condition and uniformity conditions has to be fulfilled. To this end, we have shown that the assumption eq. (C.11) leads to an contradiction to the invariance condition eq. (C.5). Hence, assumption eq. (C.11) cannot hold, i.c., hg does not depend on any view-specific style variable 2y from zi. It is only a function the shared content variables Zg, that is, 2 hg(2)).
21

---

Published as a conference paper at ICLR 2024
We list Zimmermann al. (2021, Proposition 5.) for future use in proof:
Proposition 1 (Proposition 5 of Zimmermann et al. (2021))). Let M.N be simply connected and oriented C' manifolds without boundaries and h M + N be differentiable map. Further, let the random variable z E .M be distributed according to z ~ p(z) for a regular density function p. i.e, O p < cc. If the push-forward p=t(2) (2) through h is also regular density, i.e., 0 < pæh co, then h is a bijection.
Theorem 3.2 (Identifiabilitty from Set of Views). Consider set of views xy satisfying Asm. 2.1, and lel G be a sel of content encoders (Defn. 3.1) that minimizes the following objective
L(G) EncKev E (xx) gke (xx 1121
Σkev H (gk (xx))
(3.1)
Content aligument
Entropy regularization
where the expectation is taken W.E.f. p(xv) and H()) denotes differential entropy. Then the shared content variable 2e == (2, j E C} is block-identifie (Defn. 2.3) by 9k E for any l E V.
Proof. Lemma C.1 verifies the existence of such a set of smooth encoders that obtains the global minimum of eq. (3.1) zero; Lemma C.2 derives the invariance conditions and the uniformity that the learned representations 9& (X&) have to satisfy for all views & E V. Based on the invariance condition eq. (C.5), Lemma C.3 shows that the learned representation ge (xx), E V can only depend on the content block, not on any style variables, namely 9& (xx) = ha (zc) for some smooth function h2 Zc + (0, (0,1)ICI.
We now apply Zimmermann et al. (2021, Proposition 5.) to show that all of the functions h2 k E V are bijections. Note that both 2c and (0. 1)íci are simply connected and oriented C¹ manifolds, and ha are smooth, thus differentiable, functions that map the intersection set of random variables ze from C to (0, I)Ici, Given by Asm. 2ii) that Pze and the push-forward function through h2 (uniform distributions) are regular densities, we conclude that all h2 are diffeomorphisms for all k V.
Thus we have shown that any content set of encoders G that minimizes L(G) (eq. (3.1)) can extract the ground-trum content variables z from view X2 E Xg up to a bijection h2 : Zc + (0, 1)
(x(xx)) hg(2));
(C.21)
That is, shared content ze is block-identified by the content encoders G = kev.
Remark on the proof technique for Thm. 3.2. For Thm. 3.2, one could imagine alternate proof by induction over the number of views, where the proofs by von Kügelgen et al. (2021); Daunhawer et al. (2023) would be the base case. We opted for direct proof technique as the induction proof may have been perhaps more intuitive at a high level but was significantly longer. Additionally, we present the current version because it would be generally more accessible more familiar proof technique.
C.2 PROOF FOR THM. 3.8
Our proof consists of the following steps:
1. We show Lemma C.4 the loss eq. (C.22) is lower bounded by zero and construct optimal R' (Defn. 3.3), x0 (Defn. 3.5), 7 (Defn. 3.6) that reach this lower bound;
2. Next, we show in Lemma C.6 that, if the content sizes |Ca| are known for all Vi E V. then any view-specific encoders, content selectors, and projections (R, 4 7") that minimize the loss eq. (C.22), block-identify the content variables zc, for any V E V. using similar steps as in the proof for Thm. 3.2.
3. As the third step, we show that any minimizer R (Defn. 3.3), & (Defn. 3.5), T (Defn. 3.6) of eq. (C.22) also minimizes the iittommation-ssarin regularizer (Defn. 3.7); and show that the optimal solution (R", $. T*) we constructed in the first step reaches this lower bound of Defn. 3.7.
4. Then, we show by contradiction that any optimal content selector t² that solves the con- strained optimization problem in eq. (3.2) recovers the correct content size [C,] for each subset Vú, using the invariance condition in Lemma C.5.
5. Lastly, apply the results from Lemma C.6 and conclude our proof for Thm. 3.8.
22

---

Published as a conference paper at ICLR 2024
We rephrase each step as a separate lemma and use them to complete the final proof for Thm. 3.8.
Lemma C.4 (Existence of Encoders, Selectors and Projections), Consider a jointly observed set of views xy satisfying Asm. 2.1. For any set of view-specific encoders R (Defn. 3.3), content selectors Rs (Defn. 3.5) and projections! T (Defn. 3.6), we define the following objective:
C (R)
Σ VEV.REEV.
E
((..)) (rk(xe) ((ak) 2 Tk (xx((x)))))
[b]
-
Σ KEV
H (tx o ox(()))
KCc
(C.22)
which is lower bounded by zero; and there exists such combination of R, 4 T that obtains this global minimum zero.
Proof. Consider the objective function C(R, 4, T) (eq. (C.22)), the global minimum of C(R, D, T) is obtained when the first term (alignment) is minimized and the second term (entropy) is maximized. The alignment term is minimized to zero when selected representations Φ(iAA) Ø T are perfectly aligned for all & E V almost surely. The second term (entropy) is maximized to zero only when to T (xx) is uniformly distributed on (0, 1)(5kl for all view k E V. Thus we have shown that the loss (eq. (C.22)) is lower-bounded by zero.
The optimal view-specific encoders can be defined via the inverse of the view-specific mixing functions {ʃs kev, which by Asm. (i) are smooth and invertible. By definition, we have fx (xx) = zs, for all ls E V. Formally, we define the set of optimal view-specific encoders
: Uf Jkev.
(C.23)
Next, we define the optimal auxiliary transformation l for each view i using Darmois construction, writing ba o (xx) t o f (xx) 6g (zsa), we have:
fg (as,) = P3t ((,, (1 11) P ([Z,,,) I [.s,bil(ss,,)g-1))) j (1....... [kk]) (C.24) where F.J denotes the conditional cumulative distribution function (CDF) of [ıs.]j given [ss, Thus, ty (zs²) is uniformly distributed on (0, 1). and tx is smooth by Asm. 2ii which states that Pa is smooth density.
As for the optimal content selectors +- = (Q(iih) VEV..EE. choose g(i.)) such that
glick) 2 25. == 2c.
(C.25)
Writing t n the loss C(R, T) from eq. (C.22) takes the value:
L CRR G. TT = Σ
E
(li
rxa(xe) -(iAk) 2 r (xxx
Σ H (tx rx((x)) WEV
ViEY KV
kick"
Σ
Σ
E
((iN) 2 (xxx) - g(())) 2 fxw) [b]]
- Σ
H (tx fr (xx))
ViEV KW Rick'
KEV
Σ
Σ
((())) A ZS. (link) 2 n. [1]
Σ
H (tx(2)))
ViEV K.K'E.. ASck"
KEV
Σ VIEV KV
Σ
E Ellzz. 2ci12] Σ
H (tx (zs,))
AgV
kCk""
(C.26)
Note that the first term is minimized to zero because the shared content values zc, align among the views in one subset V; V: the second term is maximized to zero because & (Zs,) is uniformly distributed on (0, 15al given by the property of Darmois construction (Darmois, 1951). To this end, we have shown that there exists such optimum R, Φ T as defined in eqs. (C.23) to (C.25) that minimizes the objective in eq. (C.22).
23

---

Published as a conference paper at ICLR 2024
Lemma C.5 (Conditions of Optimal Encoders, Selectors and projections). Given the same set of views Xy as introduced in Lemma C.4, to minimize C(R, T in eq. (C.22), any optimum R. T (Defns. 3.3, 3.5 and 3.6) has to satisfy similar invariance and uniformity conditions from Lemma C.2:
Invariance: All selected representations (l Ork(Xe), k E V must align across the views from the set V; E almost surely:
((i.)) @ rr((x)) (6ik) (x)()))
V; 0.8.
(C.27)
Uniformity: All extracted representations to (x(())), E V must be uniformly distributed over the hyper unit-cube (0, 1)1s
Proof. The minimum of CR, $ T) = 0 can only be obtained when both terms are zero. For the first term (alignment) to be zero, it is necessary that ((ikk) @rx(xx) = g(i.k) Ø Tx (xx) almost surely for all V; V, ,, Vi w.r.t. the true generating process. The second term (entropy) is upper-bounded by zero; this maximum can only be obtained when the auxiliary encoding 4 o (xt))) k E V follows uniformity, also indicated by Lemma C.2.
Lemma C.6 View-Speciffc Encoder for Identifiability Given Content Sizes). Consider a jointly observed set of views Xv satisfying Asm. 2 and assume that the dimensionality of the subset-specific content [C] is given all subset V; E V. We consider a special type of content selectors & with ( ll = |C,] for all l E Vi. Let R, T respectively denote some view-specific encoders (Defn. 3.3), and projections (Defn. 3.6), which jointly minimize the following objective together with the special content selectors +;
C (R,
Σ
EE
E
( (orx(x)) - -6(ik) (xw) ||]
Σ KEV
H ((x())))))))
K.VEE k<k
(C.28)
Then for any view V, any subset of views V, E V with k. V,, the composed function ((6.)) s rk block-identifee the shared content variables zc, in the sense that the learned representation 2 g(0,2) Ø rx(xx) is related to the ground truth content variables through some smooth invertible mapping by : Zc. - Ec, with 2 (5) (zc:)-
Proof. Lemma C.4 verifies that there exists such optimum which minimizes the loss eq. (C.28) to zero; the invariance and uniformity conditions have to be satisfied by any optimum, as shown in Lemma C.5. Following Lemma C.3, the composition Tj := ((i,k) @rr can only encode information related to the subset-specific content C₂ for any subset VÚ V otherwise it will lead contradiction to the invariance condition from Lemma C.5. The last step is to prove the invertibility of the encoders G. Notice that
f o Fx (xx) = ta Fh fa (Zs,)
By applying Zimmermann al. (2021, Proposition 5.) with similar arguments in the proof for Thm. 3.2, we can show that composition & rs fú is smooth bijection of the subset-specific content zc, Since fü is a smooth invertible mapping by Asm. 2.1 (i), we have:
(t& o o fk) f = (t& o r4) o (fx o fE = ty o lk-
Hence, 4 o Tkk is bijective as the composition of bijections is a bijection. Next, we show that Th is bijective. Showing that Tą is bijective By contradiction, suppose Fk is not injective. Thus there exists distinct values x x2 E X's s.t. (x()) T (x²). This implies that t, Tr (x,) t4 (x²), which violate injectivity of tg rr- Thus, F% must be injective.
its image is equivalent to showing that it is injective.
To this end, conclude that any R, Q. T that minimizes eq. (C.28) block-identifies the shared content variables zc, for any subset of views V; E V.
Claim 1. For any (R, T. (Defns. 3.3nd 3.6) that minimizes the loss eq. (C.22), the Reg(@) (Defn. 3.7) is lower bounded by - Eveev [C2l |V[] and this minimum is obtained at the optimal content selectors defined in eq. (C.25).
24

---

Published as a conference paper at ICLR 2024
Proof. Suppose for a contradiction that there exists some binary weight parameters & + & with
Reg ④ = Σ Σ
< (2).
(C.29)
lo
VEE REV:
which means, there exists least one vector ((1,k) for some view l E V, subset V4 E V. such that
g(i.k) Ø rx(xx)) = A where C S& an index subset of the view-specific latents S₁.- Given that R, Φ minimizes C(R, Φ, T) from eq. (C.22), these minimizers have to satisfy the invariance and uniformity constraint as shown in Lemma C.5. Since uniformity implies invertibility (Zimmermann et al., 2021), the learned representation ri rx(x))) contains sufficient information about the original view X s.t. the view X can be reconstructed by some decoder given enough capacity. Given that the number of selected dimensions |A| > [Cal, at least one latent component j A will contain information that is not jointly shared by V. That means the composition rk = (ff.A) @ rı encodes some information other than just content Cj. As shown in Lemma C.3, any dependency from the learned representation non-content variables leads to contradiction to the invariance condition as derived Lemma C.5. Therefore, the optimal content selectors & following the definition in eq. (C.25) must obtain the global minimum of the information-sharing regularizer (Defn. 3.7), which equals - Eviev Σkev, |C:]. O
[ > |Cil
(C.30)
Theorem 3.8 (View-Specific Encoder for Identifiability). Let R. T,T respectively be any view-
specific encoders (Defn. 3.3), content selectors (Defn. 3.1) and projections (Defn. 3.6) that solve the following constrained optimization problem:
min Reg(O) subject to: T. E argmin L CR,
(3.2)
where
C 2,,
E VEVR K,V'VV
E ||| (iihk) O (x((xx) - (iN) O rH (Xx) (12
Σ KEV
H1 (tx rr((x)))
Entropy
kCk
Content aligrament
(3.3) Then for any subset of views V; E V and any view & E V,, ((i.k) @ block-identifies (Defn. 2.3) the shared content variables 2c,, as defined in eq. (2.2).
Proof. Lemma C.4 confirms that there exist view-specific encoders R, content selectors O and projections T that obtain the minimum of the unregularized loss eq. (C.22) (equals zero); Additionally, any optimal R, Φ T fulfills the invariance condition and uniformity (Lemma C.5) s.t. they obtain the global minimum zero. Using the invariance condition, Claim substantiates that the optimal content selectors as defined in eq. (C.25) minimizes the regularization term (Defn. 3.7). We have thus shown that with R, 4, T' (as defined in eqs. (C.23) to (C.25)), eq. (3.2) obtains the global minimum.
Next, we show that the number of selected dimensions from each selector (i.,), i.c., the Lo norm of (((,k) align with the size of the shared content [C]].
Among the content selectors that minimize the unregularized loss (eq. (C.22)), we consider some content selectors & E argmin Reg(1) ( that also minimize the information-sharins regulariser defined in Defn. 3.7, that is:
Reg( (+')= = E
- J.,).
VEV.
Suppose contradiction that there exists a pair of binary selectors (olid)) (lC' with ((iAA) {0, 1) 1)15 and gl'' ((', E (0011]1szl such that
|ostko 1. [Ca::
[atRoo 11.
|Crr.
(C.31)
which indicates that there exists at least one latent component j E S& \ C, being selected by (((,)) similarly, this contradicts the invariance condition shown in Lemma C.3. Hence, the number of dimensions selected by each ((i,,)) has to equal the content size [Ca).
At this stage, the problem setup is reduced to the case in Lemma C.6 where the size of the content variables [C,l are given for all subset of views V, V. Hence, applying Lemma C.6, conclude that any R, d, T (Defns. 3.3, 3.5 and 3.6) that minimize eq. (3.2) block-identify the shared content variables zc, for any subset of views V, V and for all views l V..
25

---

Published as a conference paper at ICLR 2024
C.3 PROOFS FOR IDENTIFIABILITY ALGEBRA
Let ZC1, &C2 be two sets of content variables indexed by C1 C2 S [N] that are block-identified by some smooth encoders 91 X1 2c1 92 X2 + Zc2, then it holds for C1. C2 that:
Corollary 3.9 (Identifiability Algebra: Intersection). The intersection 4cinC2 can be dlock-identffiel.
Proof. By the definition of block-identifiabiiiyy. construct synthetic views using the learned representation from X and X2:
x (1) = 91 (x1) = In (2c,)
(C.32)
x x(2) = (=(x)((x) = hz(Zc2)
for some smooth invertible mapping h4 Ec, Ec, k E {1,2). Applying the Thm. 3.2 with two views, we can block-identify the intersection C1 n using this pair of views (x("), (2)).. L
Corollary 3.10 (Identifiability Algebra: Complement). If C2C22 is independent of C1 C2z then the complement \C,, can be block-idene ionetideet
(2) in the proof for Cor. 3.9. We then can consider the intersection C1 n C22 as the content variable and C₁ C2 as the style variable from these two synthetic views (x(1), x(2)). Private Component Extraction from Lyu et al. (Theorem 2. 2021) has shown that if the style variable is independent of the content, then the style variables also be extracted up to a smooth invertible mapping. Therefore, we conclude that the complement 201/ also be block-identified.
Proof. Construct the same synthetic views x
Corollary 3.11 (Identifiability Algebra: Union). Cf Ci\C2 and C2\Cs are mutually inde- pendent, then the union AC;u0, be block-identifier
Proof. We rephrase CLUC₂ as a union the following disjoint parts:
C₁UC₁ (GGnCc)JUCC/CC)UUCCC..
(C.33)
Following the definition from Cors. 3.9 and 3.10 have shown that:
Ën = iin (n(,
2 hnzl (C2)
(C.34)
2ز11 = h2g (Zcs)c,),
By concatenate the learned representations, we define hu Zciuc, + Zciucs as (uc, (2 Z2\1) = [2c, 21,2, 22113 = hu(ze,ucs),
(C.35)
hence, the union CnnC2 can be block-identified.
D EXPERIMENTAL RESULTS
This section provides further details about the datasets and implementation details in Š 5. The implementation is built upon the code open-sourced by Zimmermann et al. (2021); von Küggelgen al. (2021); Daunhawer al. (2023).
D.1 NUMERICAL EXPERIMENT THEORY VALIDATION
Data Generation. For completeness, we summarize the setting of our numerical experiments. We generate synthetic data following Example 2. which we also report below. The latent variables sampled from Gaussian distribution (0, Ez), where possible causal dependencies encoded through Σz. Note that this setting the ground truth causal variables will be related linearly each other.
(1 Z2 23, Zz, 23;
X2 = f2(21, 22, Zz Zz, 26):
(D.1)
X3 fz 22, Z3, 24, 26),
X4 f4(2), 22, 24, 25, Z6).
Implementation Details. We implement each view-specific mixing function f.. for each view k 1, 3, 4, using 3-layer invertible, untrainable MI.P (Haykin, 1994) with LeakyReL.U (Xu
26

---

Published as conference paper at ICLR 2024
Table 3: Linear Evaluation: Mean Correlation Coefficients across multiple views.
Dig No)
Dey wat
[N: Ka]
19y Ds.
[8,
Na.
(N) (No NS (i
0,850
0,790
0.0.02
0.744 0.000
al., 2015)(a 0.2). The weight parameters in the mixing functions randomly initialized. For the learnable view-specific encoders, we use a 7-layer MLP with LeakyReLU (a = -.00)) for each view. The encoders trained using the Adam optimizer (Kingma & Ba, 2014) with /r= le-4. All implementation details are summarized in Tab.
Additional Experiments. We experiment on causally dependent synthetic data, generated by z ~ (0, Σx) with Σ, ~ Wishart I). The results are shown Fig. The rows denote ground truth latent factors, and the columns represent the learned representation from different subsets views. Each cell reports the R² score between the respective ground truth factors and the learned representation. For example, the cell with col=(x1, xx) and row=z1 shows the R score when trying to predict ž using the learned representation from subset {x1,x) Since dependent style variables become predictable, discussed in App. D.I, aim to verify that the learned rep- resentation contains all and only content variables. In other words, block-identifies the ground truth content factors. For that, we consider all the views (x....., X4} and train a linear regression from the ground truth content variables Zi, Z2 to the individual style variables Z3, Z4, 25, 25. We report the coefficient of determination R in Tab. 6. We observe that the R values obtained from the ground truth content are highly similar to the ones in the last column of the heatmap (Fig. 4). Based on this, have showcased that the learned representation indeed block ieeeiiieesss the content variables.
Additional Evaluation Metrie. We report the Mean Correlation Coc)ficien((MCC) (Khemakhem al., 2020) the numerical experiments. MCC has been used in several recent works iden- tifiability of causal representation learning (Buchholz et al., 2023; von Kügelgen et al., 2023), measures the component-wic linear correlation up permutations. A high MCC (close one) indicates a clear 1-to-1 linear correspondence between the learned representation and the ground truth latents. We remark our theoretical framework considers block-identing boock-idetiffiabiitty. which could imply any type of bijective relation to the ground truth content variables, including nonlinear transformations. Nevertheless, we observe high MCC score on both independent and dependent cases, showing that the learned representation having high linear correlation to the latent components which indicates stronger identifiability results.
D.2 SULE-SUPERVISED DISENTANGLEMENT
Datasets. In this experiment, test MPI-3D complex (Gondal al., 2019) and 3DIdent (Zim- mermann al., 2021). Both are high-dimensional image datasets generated from mutually independent latent factors: MPI-3D complex contains real-world complex shaped images with ten discretized latent factors while 3Dldent renders a teapot conditioned on ten continuous latent factors.
Ohiend
Augneed HOY
Orginalloo
Example Input: MPI-3D complex
(b) Example Input:
3Dddent
Implementation Details. We used the implementation from (Michlo, 2021) for Ada-GVAE (Lo- catello et al., 2020), following the same architecture (Locatello al., 2020. Tab. 1 Appendix ). For our method, we use ResNet-18 (He et al., 2016) as the image encoder, details given in Tab. 8. For both approaches,
ENCODING SIZE=10 following the setup in Locatello et al. (2020).
27

---

Published as a conference paper at ICLR 2024
Table 4: Parameters for numerical simu- lation ($ 5.1 and App. D.I).
Table 5: Parameters for experiments $$ 5.2 and 5.3 and Apps. D.3 and D.4. *: for both image and text coders. **; hyper-arapmeter for Barlow Twins (Zbontar al., 2021).
Parameter
Value
Mixing function
3-layer MLP
Parameter
Values
Encoder
7-layer MLP
Optimizer
Adam
Content encoding size* View-specific encoding size*
Adam: learning rate
le-4
11
Adam: betal
0.9
Image hidden size
100
Adam: beta2
0.999
Text embedding dim
128
Adam: epsilon
le-8
Text vocab size
111
Batch size
4096
Text fbase
25
Temperature 7 # Iterations
1.0
Batch size
128
100,000
Temperature
1.0
# Seeds
3
Off-diagonal constant *
1.0
Similarity metric
Euclidian
Optimizer
Adam
Adam: betal
0.9
Adam: beta2
0.999
Adam: epsilon
1e-8
Adam: learning rate
le-4
# Iterations
300,000
# Seeds
Similarity metric
Cosine similarity 2-norm; max value
Gradient clipping
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.5
1.0
0.8
1.0
0.9
0.9
0.7
0.9
0.4
Table 6: Linear R² from ground fruth content variables to styles when consider
0.7
0.8
0.9
0
0.8
0.9
0.9
0.8
0.8
0.9
0.9
0.7
0.6
{x1 X2, Xs, x2 ), these values align with the last column of Fig. 4, showing that have block-identipe the content variables {z) Z2)
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.7
0.7
(c))
(stu M4)
(a))
For aou K
(as foroto R4)
(in Mas Rc))
Kc)
content
style
(21, 22)
2
2,4
Z5
26
Figure 4: Theory Verfication: Average R2 across multiple views generated from causally dependent latents.
1.0
0.32
0.65
0.58
0.71
28

---

Published as a conference paper at ICLR 2024
Table Thm. 3.2 Validation on Causal3Dident: R ttdd Green: content, bold: Ra 0.50.
Views penerated
positions
by clanging
spoti
obj
spoti
bkg
hugs
1.00=0.00
0.76.0.01
0.56:0.02
0.0010000
0.8210.01
0.2720.03
0.0010.01
0.0010000
0.2510.02
0:27110.22
0:27:0000
positions
1,00s0.00
0.00
0.46:0002
0.00s:0:01
0.00+6.01
0.32e0.02
0.00x0.01
0.92s0.00
0.26x10.22
0.29x0.02
0.27:00.0
rotations
1.00:0000
0.1120.01
0.50g8802
0.00:0.00
0.0660.01
0,3129002
0.00+0001
0.83:0.01
0.25+0.01
0.2740.02
0.06:0.01
bues+pos
1.00:0.00
0.0010.00
0.2040.02
0.000001
0.0010.01
0.1440.02
0.0010.00
0.0010.01
0.0740.01
0.1810.02
0.12a0.02
0.0910.02
0.36x0.02
0.51s0.01
0.00:0.01
0.00s0.01
0.2510.02
0.25a001
pos-oot
1.00:0.00
0.00:0.00
0.0010.00
0.0700.01
0:00:0.01
0,0540.01
0.2040.02
0.13:0002
huct-poe-roo
1.00:0.00
0.0010.00
0.4260.02
0.00.0.01
0.0010.00
0.25
0.0010:00
0,0010.00
0.0110.01
0.2610.02
0.26.0.02
D.3 CONTENT-STYLE IDENTIHABILITY ON IMAGES
Datasets. Causal3Dident (von Kügelgen et al., 2021) extends 3Dident (Zimmermann et al., 2021) by introducing different classes of objects, thus object shape (or class) is added discrete factor of variation. We extend the image pairs experiments from (von Kügelgen et al., 2021) by inputting three views, shown in Fig. 6b, where the second and third images are obtained by perturbing different subsets of latent factors of the first image. To perturb one specific latent component, we uniformly sample one latent in the predefined latent space (Unifl-1, 1]. details see (von Kügelgen et al., 2021, App. B)), then we use indexing search to retrieve the image in the dataset that has the closest latent values as the sampled ones. Note that only a finite number of images are available; thus, there is not always perfect match. More frequently. we observe slight changes in the non-perturbing latent dimensions. For instance, the hues of the third view is slightly different than the original view, although we intended to share the same hue values.
additional
pos
class
huebog
huespi
pts
rolobi
huceoij
Orignnl vIR
Changehuss
Change DoN
(a) Underlying causal relation in Cadsal3Dldenr and Multimodal3Ddden images. Figure adopted from (von Kügelgen al., 2021, Fig. 2)
(b) Example Input: Causal3Dident
Figure 6: Causal3Dldent: Underlying causal relations and input examples.
Implementation Details. The encoder structure and parameters are summarized in (Tabs. and 8). We train using BarlowTwiie (Zbontar et al., 2021) with cosine similarity and off-diagonal importance constant 1. BarlowTwins another contrastive loss that jointly encourages alignment and uniformity, by enforcing the cross-correlation matrix of the learned representations to be identity. The on-diagonal elements represent the content alignment while the off-diagonal elements approximate the entropy regularization.
Thm. 3.2 Validation. We train content encoders (Defn. 3.1) on Causal3Diden to verify Thm. 3.2. Note that we experiment on three views (von Kügelgen et al., 2021) cannot naively handle. Tab. 7 summarizes results for all possible perturbations among the three views. We can observe that the discrete factor class learned perfectly: dependent style variables become predictable from the content (class) latent causal dependence. Note that this table shows similar results as in (von Kügelgen et al., 2021, Table 6. Latent Transformation (LT)). We remark that there is a reality between theory in practice: In theory, we assume that the content variables share the exact same value across all views; however, in practice, finding a perfect match of all of the continuous content values become impossible, since there is only a finite number of training data available. We believe this reality gap negatively influenced the learning performance on the content variables. thus preventing efficient prediction on certain content variables, such as object hues.
D.4 MUL.TIMODAAL CONTENT-STYA IDENTIFIABILITY UNDER PARTIAL OBSERVABILITY
Dataset. Multimodal (Daunhawer et al., 2023) augments Causal3DIdent (von Kügelgen al., 2021) with text annotations for each image view, and discretizes the objection positions (I, y,
29

---

Published as a conference paper at ICLR 2024
Table 8: Encoder Architectures for Causal3Dldent and Multimodal3lddeut
Image Encoder
Text Encoder
Input size H x x
Input size vocab size Linear([base, text embedding dim)
DesNet-18(hiddee size)
LeakyReLu(a = 0.01)
Conv2D(1, fbase, 4, 2. 1, bias=True)
Linear(hidden size, image encoding size)
BatchNorm((Base); ReLU
Conv2D(fbasee fbase-2. 2. 4, 2, 1, bias=True)
BatchNorm((base-22 2); ReLU
Cony2D(fbase-2 fbase.4, 4, 2, 1, bias=Trrue)
BatchNorm((aase44) ReLU
4-ncaa((base-4331B text encoding size)
Table 9: Thm. 3.2 Validation on Multimodal3Dident R mtan Green: content, bold: Ra 0.50.
Views generated
clucs
Ing pos
ing lizes
class
1M0 pcs
Dd bue
pleasing
changing
sper]
spod
cbisstor_ja
luss
0.82 20.01
1.00 1.00
A0L00
1.00
20.00
0.00 20.00
0.57 #0,01
0.00 20
0.85 =0013
1.00 20.00
1.00 20.00
0.15 20.02
0.21 #0,12
ps
stJl
0.0000000
0.00
0.94:8001
0.16
s002
to categorical variables. particular, object-2pos is constant and thus not shown in our evalua- tion (Fig. 3). Our experiment extends (Daunhawer et al., 2023) by adding one additional image to the original image-text pair, perturbing the hues, object rotations and spotlight positions the original image (Uniformly sample from Uni f0, 1). Thus, (imgo: img,) share object shape and backgrownd color; Thus, (imgo, (xto) share object shape and object x-y positions; both (imgg, txlg) and the joint set (imgo- img1, txte) share only the object shape. One example input is shown in Fig. 7.
Implementation Details. Tabs. 5 and 8 shows the network architec- ture and implementation details, mostly following (Daunhawer et al., 2023). Note that we use the same encoding size for both image and text encoders for the convenience of implementation. We train using BarlowTwins with = 1. In practice, we treat the unknown content sizes list of hy typer-per typer-parametes parameters and optimize over different validations.
Text description: A hare of bright yel- low green color is visible, positioned at the mid-left of the image.
Figure Example input: Multimodall Left: pair of images that are original and perturbed images. Right: Text annotation for the original view.
Further Discussion about $ 5.3. The fundamental difference between the Multimodal3D and Dldst datasets, mentioned above, makes direct comparison between our results in Fig. and (von Kügelgen et al., 2021) harder. However, Tab. 7 shows similar R²² scores as the results given von Küggeggen al. (2021, Sec 5.2), which verifies the correctness of our method.
Thm. 3.2 Validation. We additionally learn content encoders on three partially observed views (imga- img txte) using the loss from Zbontar chitecture and parameters as summarized in Tabs. 5 and 8. Tab. 9 shows the content encoders consis- tently predict the content variables well and that our evaluation results highly align with (Daunhawer al., 2023, Fig. 3) on image-text pairs (imgo txto) inputs, which empirically validates Thm. 3.2.
(2021), to justify Thm. 3.2. We use the same ar-
D.5 MULTI-TASK DISENTANGLEMENT WITH SPARSE CLASSIFIERS
Following Example 2. we synthetically generate the class labels by linear/nonlinear labeling func- tions on the shared content values, which resembles the underlying inductive bias from (Lachapelle al., 2023; Fumero et al., 2023), that the shared features across different images with the same label should be most task-relevant. Here, we use the same sparse-classifier implementation from (Fumero al., 2023). We remark that the goal of this experiment is to verify expectation from Thm. 3.2
30

---

Published as a conference paper at ICLR 2024
that the method of (Fumero al., 2023) can be explained by theory, although they assume mutu- ally independent latents, which is a special case of our setup. In our experimental setup, an input gets a label when the following labeling function value is greater than zero:
Linear: Σj= 2.3 where el denotes the encoding size.
Nonlinear: tanh
(ش) 2))
Thus, we have resembled the inductive hypothesis in Fumero al. (2023) that the classification task is only dependent on the shared features. The fact that the binary classification is solved in several iterations verifies that Fumero et al. (2023) used the same soft alignment principle as described in Thm. 3.2.
E DISCUSSION
The Theory-Practin Gap It is noticeable that some of the technical assumptions we made in the theory may not exactly hold in practice. A common assumption in identifiability literature is that the latent variables z are continuous, while this is not true for e.g. the object shape in Causal3Dldent (von Kügelgen et al., 2021) and object shape, positions in MulimooualSDdden (Daunhawer et al., 2023). Another related gap regarding the dataset is that the additional views are generated by uniformly sampling a subset of latents from the original view and then trying to retrieve an image among the existing finite dataset, whose latent value is closest to the proposed one. However, having only finite number of images implics that always finding a perfect match for each perturbed latents is almost impossible in practice. As consequence, the designed to be strictly aligned content values between different views could differ from each other by a certain margin. Also, both Thms. 3.2 and 3.8 holds asymptotically and the global minimum is obtained only when given infinitely amount of data. Given that there is no closed-form solution for entropy regularization term eqs. (3.1) and (3.2). it is approximated either using negative samples (Oord et al., 2018; Chen et al., 2020) or by optimizing the cross- correlatrnl between the encoded information to be close to Identity matrix (Zbontar et al., 2021); in both cases there is only a finite number of samples available, which makes converging to global minimum almost impossible in practice.
Discovering Hidden Causal Structure from Overlapping Marginals. Identifying latent blocks }zg.} provides us with access to the marginal distributions over the corresponding subsets of latents {p(zg.)}: With observed variables and known graph, this has been termed the "causal marginal problem" (Gresele et al., 2022), and our setup could therefore also been seen as generalization along those dimensions. It may be possible to extract some causal relations from the inferred marginal distributions over blocks, either by imposing additional assumptions or through constraint- bassd approaches (Triantafillou et al., 2010).
How to Learn the Content Sizes? Thm. 3.8 shows that content blocks from any arbitrary subset of views can be discovered simultaneously using view-specific encoders (Defn. 3.3), content selec- tors (Defn. 3.5) and some projections (Defn. 3.6). We remarked in the main text that optimizing the information-sharing regularizer (Defn. 3.7) is highly non-convex and thus impractical. We pro- posed alternatives for both unsupervised and supervised cases: for self-supervised representation learning, one could employ Gumble-Softmax to learn the hard mask. We hypothesize that if there is an additional inclusion relation about the content blocks available, for example, know that C C C2 C Cg. then the learning process could be eased by coding this inclusion relation in the mask implementation. This additional information naturally inherited from the fact that the more views we include, the smaller the shared content will be. Another idea would be manually allocating individual content blocks in the learned encoding in a sequential manner, e.g. we set index=1, 3 for the first content block and index=4, 5 for the second content block, and enforcing alignment correspondingly. Thus, for each view, we learn a concatenated representation of the shared content. Although this method does not perfectly follow the theoretical setting in the Thm. 3.8, it still learn all of the contents simultaneously and shows faster convergence. In classification tasks, the hard align- ment constraint is relaxed to some soft constraint within one equivalence class e.g. samples which have the same label. In this case, we can replace the binary content selector with linear readouts, as studied and implemented by Fumero et al. (2023). Another, yet the most common approach to deal with this problem is to treat the content sizes as hyperparameters, as shown in (von Kügelgen et al., 2021; Daunhawer al., 2023).
31

---

Published as a conference paper at ICLR 2024
Trade-off between Invertibility and Feature Sharing. An invertible encoder implies that the extracted representation is a lossless compression, which means that the original observation can be reconstructed using this learned representation,, given enough capacity. On the one hand, the invertibility of the encoders is enforced by the entropy regularization, such that the encoder preserves all information about the content variables; the other hand, the info-sharing regularizer (Defn. 3.7) encourages reuse of the learned feature, which potentially prevents perfect reconstruction for each individual view. Intuitively, Thm. 3.8 seeks the sweet spot between invertibility and feature sharing: When the encoder shares more than the ground truth, then it loses information about certain views, and thus the compression maximized, then the learned encoder is not optimal solution either, given by the regularization penalty from the infor-sharing regularizer.
lossless; When the invertibility fulfilled but the info-sharing not
Causal Representation Learning from Interventional Data. Our framework considers purely observational data, where multiple partial views are generated from concurrently sampled latent variables using view-specific mixing functions. Recent works (Liang et al., 2023; Buchholz et al., 2023; von Kügelgen al., 2023) have shown identifiability in non-parametric causal representation learning using interventional data, allowing discovering (some) hidden causal relations. Since simultaneously identifying the latent representation and the underlying causal structure in a partially observable setup has been long standing goal, believe incorporating interventional data into the proposed framework could be one interesting direction to explore.
32


Reviews:
[{"review_id": "v7pyVn2vTr", "reviewer": "Reviewer_RoXZ", "summary": "The authors study latent variable identifiability in the setting where there are multiple observations (views), each being a nonlinear mixture of a subset of latent components. Existing work identifies the shared latents for a given set of views $V$. This work generalizes the existing results, where the shared latents are identified for any subset of views $V_i \\subseteq V$. Moreover, this is done simultaneously for all possible $V_i$ in a single training run. Many related identifiability results can be framed as a special case of the authors’ general framework.", "questions": "I understand that it is more general to be able to simultaneously identify the content blocks for all possible $V_i$, but what practical benefit does this generality bring? Was this necessary in order to unify a wider range of existing identifiability results? If so, can you give a  specific example how this generality helped?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper is written extremely well, and reads like a chapter from a textbook. The authors frequently refer to a running example and include “intuition” paragraphs to make it easier to understand their definitions and results. The loss that leads to identifiability (Eq. 3.1) is simple and intuitive, which makes me optimistic about the (eventual) practical applicability of this approach. The authors’ framework unifies many existing theoretical results in nonlinear ICA and causal representation learning, and also explains the empirical success of certain existing methods. This is a very strong paper that made me want to learn more about the related literature.", "weaknesses": "The main weakness of this work is its impracticality and lack of experimental results on realistic datasets. However, this is also acknowledged by the authors, and can also be said regarding most papers in this research area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QtQSrJz8El", "reviewer": "Reviewer_rWVP", "summary": "This work studies causal representation learning under partial observability. Causal representation learning is the emerging field of learning generative representations of data that could potentially have causal relationships among them. Identifiability refers to the existence of a unique generative model and is an important aspect of this field. Many works on CRL usually make certain simplifying functional assumptions or access to interventions, in order to exhibit identifiability. In this work, the authors provide a unified framework for identifiability in multi-view CRL, under various assumptions.\n\nThe data is assumed to be split into multiple views which are observed, where each view is a non-linear mixing function of a subset of the latent variables. Standard assumptions are made, including that the prior density is smooth and mixing functions are diffeomorphic. Under these assumptions, block-identifiability results are derived, when we have access to multiple partial views of the data. Because of the generality of the result, various prior results in this field can be viewed as special cases. The conceptual idea is that if we have a set of views, then we can identify a set of content encoders using a mix of an alignment and a regularization term. However, this leads to a potentially exponential number of encoders, which the authors handle via a single view-specific encoder. Experiments on synthetic data and visual/text data (Causal3DIdent and Multimodal 3DIdent) validate their theoretical ideas. The R^2 metric is reported, which measures how well the latents have been recovered. Overall, the paper is technically well-executed and fits the conference.\n\n### References:\n\n- [1] Learning Linear Causal Representations from Interventions under General Nonlinear Mixing\n\n- [2] Nonparametric Identifiability of Causal Representations from Unknown Interventions\n\n- [3] Identifiability of deep generative models without auxiliary information\n\n- [4] Identifying Weight-Variant Latent Causal Models", "questions": "Some questions were raised above. Additional suggestions:\n\n- The recent work [1] also uses contrastive learning in CRL. Are there any resemblances to their approach, i.e. how do their loss function compare to content alignment?\n\n- The works [3], [4] also study nonlinear ICA without auxiliary variables. I believe they use variants of VaDEs in their experiments, can this also be viewed as a soft alignment inductive bias?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Causal representation learning has gotten a lot of attention recently, due to the promises it holds. This work is another important step in this direction.\n\n- The general identifiability result captures a variety of prior works on CRL, therefore it serves as a neat unifying contribution.\n\n- The approach to avoid using exponential set of encoders for each subset of views is very neat and is crucial for this work, for efficiency purposes.", "weaknesses": "- R^2 metric is reported for validating identifiability, however as the authors also note, there are instances when R^2 scores can be inflated. Why didn't the authors also cite other standard identifiability metrics like MCC (such as the ones reported in [1], [2])?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OT0PxP37y4", "reviewer": "Reviewer_QCXK", "summary": "This work studies identifying latent causal representations given multi-view data. This work shows that the latent subspaces if shared by more than one view, can be identified up to bijjective mappings. Further, a view-specific encoder is devised to make learning a large number of shared blocks tractable. Experiments on multi-modality/task data are provided to verify the theoretical insights.", "questions": "I would like to learn about the authors' response to the weaknesses listed above, which may give me a clearer perspective on the paper's contribution.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "• The paper is well-written and communicates clearly motivations, formulation, technical details, and theoretical implications.\n\n• This paper extends previous content-identification results (mainly von Kugelgen et al. 2021, Daunhawer et al. 2023)  to multi-view distributions and further (identifiability algebra).\n\n• The empirical evaluation is thorough (over multi-view/modality/task datasets) and well-designed to substantiate the theoretical findings in the paper.", "weaknesses": "• My primary concern is the novelty of the work. In comparison with prior work (von Kugelgen et al. 2021; Daunhawer et al. 2023), the technical contribution appears somewhat incremental.\n\n• The assumption that each view-specific generating function is invertible is strong, especially over multiple modalities. Intuitively, this requires the shared blocks to be duplicated multiple times and thus simplifies the identification problem. A recent work [1] has attempted to weaken this assumption.\n\n[1] https://arxiv.org/abs/2306.07916", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "83DEm5KW6x", "reviewer": "Reviewer_BPer", "summary": "This paper generalizes existing theories on causal representation learning, multi-view learning by allowing multi-views, partial observability and dependency between latents. It provides identifiability theory for content variables and so-called identifiability algebra to combine content variables.", "questions": "1. For definition 5.3, why assume l0 to be the same and not the two vectors are the same? For a given view, the content variables are fixed, right?\n\n2. Could you provide a little more motivation as to why view-specific encoders are needed? I understand the theoretical reason. But in practice, one would expect to train with all views available, right?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The literature review is extensive, although there are some recent works on CRL including identification under the nonparametric settings that might be relevant as well.\n\n[1] Liang, Wendong, et al. \"Causal Component Analysis.\" arXiv preprint arXiv:2305.17225 (2023).\n[2] Jiang, Yibo, and Bryon Aragam. \"Learning nonparametric latent causal graphs with unknown interventions.\" arXiv preprint arXiv:2306.02899 (2023).\n[3] Buchholz, Simon, et al. \"Learning Linear Causal Representations from Interventions under General Nonlinear Mixing.\" arXiv preprint arXiv:2306.02235 (2023).\n\n2. I like how the paper is structured with an intuition behind definitions and theorems for readability.", "weaknesses": "1. Although the setting is different, the theoretical contributions and proof techiniques seem to be mostly derived from [1] and [2].\n\n[1] Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. Proceedings of Machine Learning Research, 139:12979–12990, 2021\n\n[2] Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467, 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)