Identifiability Results for Multimodal Contrastive Learning
Authors: Imant Daunhawer
Alice Bizeul
Emanuele Palumbo
Alexander Marx
Julia E Vogt
Venue: ICLR 2023 poster (2023)
Keywords: multimodal learning
multi-view learning
contrastive learning
causal representation learning
nonlinear ica
identifiability

Abstract: Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, a line of recent work reveals that contrastive learning can invert the data generating process and recover ground truth latent factors shared between views. In this work, we present new identifiability results for multimodal contrastive learning, showing that it is possible to recover shared factors in a more general setup than the multi-view setting studied previously. Specifically, we distinguish between the multi-view setting with one generative mechanism (e.g., multiple cameras of the same type) and the multimodal setting that is characterized by distinct mechanisms (e.g., cameras and microphones). Our work generalizes previous identifiability results by redefining the generative process in terms of distinct mechanisms with modality-specific latent variables. We prove that contrastive learning can block-identify latent factors shared between modalities, even when there are nontrivial dependencies between factors. We empirically verify our identifiability results with numerical simulations and corroborate our findings on a complex multimodal dataset of image/text pairs. Zooming out, our work provides a theoretical basis for multimodal representation learning and explains in which settings multimodal contrastive learning can be effective in practice.

Published as a conference paper at ICLR 2023
IDENTIFLABILITY RESULTS FOR MULTIMODAL CONTRASTIVE LEARNING
Imant Daunhawer' Alice Bizeul* Emanuele Palumbo 12 Alexander Marx & Julia E. Vogt
Department of Computer Science, ETH Zurich ETH AI Center, ETH Zurich
ABSTRACT
Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, line of recent work reveals that contrastive learning can invert the data generating process and recover ground truth latent factors shared between views. In this work, we present new identifiability results for multimodal contrastive learning, showing that it possible to recover shared factors in a more general setup than the multi-view setting studied previously. Specifically, we distinguish between the multi-view setting with one generative mechanism (e.g., multiple cameras of the same type) and the multimodal setting that is charactcrized by distinct mechanisms (c.g., cameras and microphones). Our work generalizes previous identifiability results by redefining the generative process in terms of distinct mechanisms with modality-seccific latent variables. We prove that contrastive learning can block-identify latent factors shared between modalities, even when there are nontrivial dependencies between factors. We empirically verify our identifiability results with numerical simulations and corroborate on a complex multimodal dataset of image/text pairs. Zooming out, our work provides theoretical basis for multimodal representation learning and explains which settings multimodal contrastive learning can be effective in practice.
findings
INTRODUCTION
Multimodal representation learning is an emerging field whose growth is fueled by recent develop- ments in weakly-supervised learning algorithms and by the collection of suitable multimodal datasets. Multimodal data is characterized by the co-occurence of observations from two more dependent data sources, such as paired images and captions (c.g., Salakhutdinov and Hinton, 2009; Shi et al., 2019; Radford et al., 2021), and more generally, multimodal observations measurements from different types of sensors (Baltrušaitis et al., 2019). Co-occurrence is a form of weak supervision (Shu et al., 2020; Locatello et al., 2020; Chen and Batmanghelich, 2020), in that paired observations can be viewed as proxies (i.e., weak labels) for a shared but unobserved ground truth factor. Among suitable representation learning methods for weakly supervised data, contrastive learning (Gutmann and Hyvärinen, 2010; Oord et al., 2018) stands out because it is designed to leverage co-occurring observations from different views. In practice, contrastive learning achieves promising results for multi-view and multimodal learning-2 prominent example is the contribution of CLIP (Radford et al., 2021) the groundbreaking advancements in text-to-image generation (Ramesh et al., 2021; 2022; Rombach al., 2022; Saharia et al., 2022).
comprised of aligned
Despite its empirical success, not sufficiently well understood what explains the effectiveness of contrastive learning in practice. Recent works attribute its effectiveness to the recovery of shared latent factors from the underlying causal graph (Gresele et al., 2019; Zimmermann et al., 2021; von Kügelgen al., 2021). From the perspective of multi-view independent component analysis, it was shown that contrastive learning can invert a nonlinear mixing function (i.e., a nonlinear generative process) that is applied to latent variable with mutually independent components (Gresele et al., 2019; Zimmermann et al., 2021). More recently, von Kügelgen et al. (2021) show that contrastive learning can recover shared factors up block-wise indeterminacies, even if there are nontrivial causal and statistical
"Joint authorship. 'Correspondence dimant 1mant@tth..ch

---

Published as a conference paper at ICLR 2023
dependencies between latent components. Collectively, these results suggest that contrastive learning can identify parts of an unknown data generating process from pairs of observations alone- even from nigh-dimensional multi-view observations with nontrivial dependencies. In our work, we investigate the identifiability of shared latent factors in the multimodal setting.
We consider generative process with modality-specifie mixing functions and modality latent variables. Our design is motivated by the inherent heterogeneity of multimodal data, which follows naturally when observations are generated by different types of sensors (Baltrušaitis et al., 2019). For example, an agent can perceive its environment through distinct sensory modalities, such as cameras sensing light or microphones detecting sound waves. To model information that is shared between modalities, we take inspiration from the multi-view setting (von Kügelgen et al., 2021) and allow for nontrivial dependencies between latent variables. However, previous work only considers observations of the same data type and assumes that the same input leads to the same output across views. In this work, we introduce model with distinct generative mechanisms, each of which can exhibit a significant degree of modality-speciftt variation. This distinction renders the multimodal setting more general compared to the multi-view setting considered by previous work.
In nutshell, our work is concerned with identifiability for multimodal representation learning and focuses on contrastive learning as a particular algorithm for which we derive identifiability results. In Section 2, we cover relevant background on both topics, identifiability and contrastive learning. We then formalize the multimodal generative process as a latent variable model (Section 3) and prove that contrastive learning can block-identify latent factors shared between modalities (Section 4). We em- pirically verify the identifiability results with fully controlled numerical simulations (Section 5.1) and corroborate our findings on a complex multimodal dataset of image/tex pairs (Section 5.2). Finally, we contextualize related literature (Section 6) and discuss potential limitations and opportunities for future work (Section 7).
PRELIMINARIES
2.1 IDENTIHIABILITY
Identifiability lies at the heart of many problems in the fields of independent component analysis (ICA), causal discovery, and inverse problems, among others (Lehmann and Casella, 2006). From the perspective of ICA, we consider the relation x = f(z). where an observation x is generated from a mixing function f that is applied to latent variable z. The goal of ICA is to invert the mixing function in order to recover the latent variable from observations alone. In many settings, full identifiability impossible and certain ambiguities might be acceptable. For example, identifiability might hold for subset of components (i.e., partial identifiability). Typical ambiguities include permutation and dement-wise transformations (i.e., component-wise indeterminacy), or identifiability up to groups of latent variables (i.e., block-wit wise indeterminacy). In the general case, when f is nonlinear function, a landmark negative result states that the recovery of the latent variable given i.i.d. observations is fundamentally impossible (Hyvärinen and Pajunen, 1999). However, a recent line of pioneering works provides identifiability results for the difficult nonlinear case under additional assumptions, such as auxiliary variables (Hyvärinen and Morioka, 2017: Hyvärinen al., 2019; Khemakhem et al., 2020) or multiple views (Gresele et al., 2019; Locatello et al., 2020; Zimmermann et al., 2021).
Most relevant our investigation are previous works related multi-view nonlinear ICA (Gresele et al., 2019; Lyu and Fu, 2020; Locatello et al., 2020; von Kügelgen et al., 2021; Lyu et al., 2022). Generally, this line of work considers the following generative process:
Z ~ Pz: X1 = (z), x2 = fz(z),
where a latent variable, or a subset of its components, is shared between pairs of observations (x1, x2) ~ Px,,xy where the two views X1 and X2 are generated by two nonlinear mixing functions, fi and f2 respectively. Intuitively, a second view can resolve ambiguity introduced by the nonlinear mixing, if both views contain a shared signal but are otherwise sufficiently distinct (Gresele et al., 2019). Previous works differ in their assumptions on the mixing functions and dependence relations between latent components. The majority of previous work considers mutually independent latent components (Song et al., 2014: Gresele et al., 2019; Locatello et al., 2020) independent groups of shared and view-specific components (Lyu and Fu, 2020; Lyu et al., 2022). Moreover, some of these works (Song et al., 2014; Gresele al., 2019; I Lyu and Fu, 2020; Lyu et al., 2022) consider
2

---

Published as a conference paper at ICLR 2023
view-specific' mixing functions. Venturing beyond the strict assumption of independent (groups of) components, von Kügelgen et al. (2021) consider additional causal and statistical dependencies between latents and show that the subset of shared components can be identified up to block-wise indeterminacy. Our work considers heterogeneous modalities with causal and statistical dependencies between latents. We prove that shared factors be block-identified in novel setting with modality- specific mixing functions and -specicy latent variables.
2.2
CONTRASTIVE LEARNING
Contrastive learning (Gutmann and Hyvärinen, 2010; Oord et al., 2018) is a self-supervised repre- sentation learning method that leverages weak supervision in the form of paired observations. On a high level, the method learns to distinguish "positive" pairs of encodings sampled from the joint distribution, against "negative" pairs sampled from the product of marginals. The popular InfoNCE objective (Oord et al., 2018) is defined as follows:
K log Σラ= exp(sim(( (xi) B2 (x2))/=} : (2)
exp(sim(g, go gx
LIMGONC (2)) = =EE (x( PPK
Σ
where 81 and g, are encoders for the first and second view, X₁ and X2 respectively. It is common to use single encoder g1 82 when X² is augmented version of when two augmentations are sampled from the same distribution to transform X1 and X₂ respectively (e.g.. Chen et al., 2020). The set of hyperparameters consists of the temperature т, similarity metric sim(-.-), and an integer K that controls the number of negative pairs -K-1) 1) used for contrasting. The objective has information-theorriicc interpretation as variational lower bound the mutual information (gg, (x1);gg(xx) (Oord et al., 2018; Poole et al., 2019) and it can also be interpreted as the alignment of positive pairs (numerator) with additional entropy regularization (denominator), where the regularizer disincentivizes degenerate solution in which both encoders map to constant (Wang and Isola, 2020). Formally, when instantiating the LIMINNE objective with T and sim(a, b) (a (a - b)², it asymptotically behaves like the objective
CAlignMactt gg) = E (xxx)xa)=Pp |llg (x1) - g(x²) [12) - (g(x))
(3)
for a single encoder g, when + oo (Wang and Isola, 2020; von Kügelgen et al., 2021).
In the setting with two heterogeneous modalities, it is natural to employ separate encoders gi g2- which can represent different architectures. Further, it is common to use symmetrized version of the objective (e.g., see Zhang al., 2022; Radford al., 2021), which can be obtained by computing the mean of the loss in both directions:
CSymlnfoNCE 82) 1/2 C 82) + 1/2 (12
(4)
Akin to Equation (3), can approximate the symmetrized objective for 1 and sim(a, b) - -aa - b)2, with a large number of negative samples (K + 2o), as follows:
SsymAlignMaattal (g) 82) ((xxxa)))
[llg (x1) g2 (x2))la 1/2 (H() + (xgg(x)))
(5)
Since the similarity measure is symmetric, the approximation of the alignment term is identical for both LIsONCE( (g) and LIMONCE (g2: g1). Each entropy term is approximated via the denominator of the respective loss term, which can be viewed as a nonparametric entropy estimator (Wang and Isola, 2020). For our experiments, we employ the finite-sample estimators LINGANCE and LSyminfaNCE. while for our theoretical analysis we use the estimand LsymAlignMaxEnt to derive identifiability results.
3 GENERATIVE PROCESS
In the following, we formulate the multimodal generative process as a latent variable model (Sec- tion 3.1) and then specify our technical assumptions on the relation between modalities (Section 3.2).
Note that we define mrodllity-ppccity functions similar to the way Gresele et al. (2019), Lyu and Fu (2020), and Lyu et al. (2022) define view-specific functions. To clarify the distinction, we generally assume that observations from different modalities are generated by distinct mechanisms fi a fz with modality-specift latent variables, and treat the multi-view setting special case, where f, fz without view-specific latents.
3

---

Published as a conference paper at ICLR 2023
22
Z1
m1
8
m2
X1
X2
Figure I: Illustration of the multimodal generative process, Latent variables are denoted by clear
nodes and observations by shaded nodes. We partition the latent space into zı (c, mi) and Z2 = (2. m2), where č = almost everywhere (Assumption 1) and hence we consider only c. Further. is a perturbed version of (Assumption 2) and mi. ma are modality-seecifie variables. The observations and X2 are generated by two distinct mixing functions fi * f2, which are applied to the subsets of latent variables 21 and Zg respectively.
3.1 LATENT VARIABLE MODEL
On a high level, we assume that there exists continuous random variable that takes values in the latent space Z C IR", which contains all information to generate observations of both modalities." Moreover, we assume that z (c, s, m1, m2) can uniquely partitioned into four disjoint parts:
(i) an invariant part e which is always shared across modalities, and which we refer to as content; (ii) a variable part s which may change across modalities, and which we refer to as style; (iii) two modality-speciin parts, mj and m2, each of which is unique the respective modality.
Let 2 = (c, s, mı) and Z = (2, m2), where ē = almost everywhere and š is generated by perturbations that are specified in Section 3.2. Akin multi-view ICA (Equation 1), we define the generative process for modalities x1 and x2 as follows:
Pei X1 (zz),
x2 f2(2))
(6)
where f1 : Zi + Xi and f2 - X2 are two smooth and invertible mixing functions with smooth inverse (i.e., diffcomorphisms) that generate observations and x₂ taking values in XG C Rdi and X2 C dR respectively. Generally, we assume that observations from different modalities are generated by distinct mechanisms fl á f2 that take prodality-speciff latent variables as input. As for the multi-view setting (von Kügelgen et al., 2021), the considered generative process goes beyond the classical ICA setting by allowing for statistical dependencies within blocks of variables (e.g., between dimensions c) and also for causal dependencies from content to style, as illustrated in Figure 1. We assume that Pa is smooth density that factorizes Pz Pe Psic Pm, Pm, in the causal setting, and as the product of all involved marginals when there is no causal dependence from c to s.
The outlined generative process is fairly general and it applies to wide variety of practical settings. The content invariance describes a shared phenomenon that is not directly observed but manifests in the observations from both modalities. Style changes describe shared influences that are not robust across modalities, e.g., non-invertible transformations such as data augmentations, or non- deterministic effects of an unobserved confounder. Modality-specific factors be viewed as variables that describe the inherent heterogeneity of each modality (c.g., background noise).
3.2 RELATION BETWEEN MODALITIES
Next, we specify our assumptions on the relation between modalities by defining the conditional distribution Pe.|z, which describes the relation between latent variables and 22, from which observations X; and X2 are generated via Equation (6). Similar to previous work in the multi-view setting (von Kügelgen et al., 2021), assume that content is invariant, i.e., č c almost everywhere (Assumption 1), and that 5 a perturbed version of s (Assumption 2). To state our assumptions for the multimodal setting, we also need to consider the modality-specinn latent variables.
Put differently, we assume that all ohservations lie continuous manifold, which can have much smaller dimensionality than the observation space of the respective modality.
4

---

Published as a conference paper at ICLR 2023
Assumption 1 Content-invaaiaacce) The conditional density Pez|a: over Z2 X 21 takes the form Pea (22/2)) ö(ē c))P (5) (m2)
for some continuous density psls on S x where 5(-) is the Dirac delta function, i.e., č C a.e.
To fully specify Pza/z: it remains to define the style changes, which are described by the conditional distribution ps There are several justifications for modeling such a stochastic relation between and (Zimmermann et al., 2021; von Kügelgen et al., )0 one could either consider š to be noisy version of s, or consider & to the result of an augmentation that induces a soft intervention on s.
Assumption 2 (Style changes). Let A be the powerset of style variables {1...., Raf and let PA be distribution on A. Then, the style conditional Paix is obtained by conditioning on a set A:
(5]s
=E (A(A) (8(5A- - -SA-) (5AlsA))
(8)
AEA
where Paals is a continuous density on SA x SA. SA S S denotes the subspace of changing style variables specified by A, and A = [I,,..., 3 denotes the complement of A.
Further, for any style variable 1 E (1,.... M,,,, there exists set A S (1,....,n,) with I E A, S.L.
(i) PA(A) >
(ii) Psalsa is smooth w.r.t. both S, and S. and
(iti) for any SA PsatMa (-1s)) > 0, in some open non-empty subset containing SAA
Intuitively, to generate a pair of observations (x1: X2). we independently flip a biased coin for each style dimension to select subset of style features A C {l, na), which are jointly perturbed to obtain 5. Condition (i) ensures that every style dimension has a positive probability to be perturbed, while (ii) and (iii) are technical smoothness conditions that will be used for the proof of Theorem 1.
Summarizing, in this section we have formalized the multimodal generative process as a latent variable model (Section 3.1) and specified our assumptions the relation between modalities via the conditional distribution Pz//22 (Section 3.2). Next, we segue into the topic of representation learning and show that, for the specified generative process, multimodal contrastive learning can identify the content factors up to a block-wise indeterminacy.
IDENTIFIABILITY RESULTS
First, we need define block-identifiabiiit (von Küggelgen al., 2021) for the multimodal setting in which we consider modality -specffcc mixing functions and encoders. In the following, denotes the number of content variables and the subscript 1:n, denotes the subset of content dimensions (indexed from to n. w.l.o.g.).
(x2) is block iookk-identified by function g, : X, + Z, with i E {1,2], if there exists an invertible function h R. + R=" s.t. for the inferred content partition é₁ g, (x,).,, it holds that è = hç(c).
Definition 1 (Block-ideniifaabiitty) The true content partition с = (xi)
is important to note that block-identifiabiiity does not require the identification of individual factors, which is the goal in multi-view nonlinear ICA (Gresele et al., 2019; Locatello et al., 2020; Zimmermann et al., 2021; Klindt et al., 2021) and the basis for strict definitions of disentanglement (Bengio et al., 2013; Higgins et al., 2018; Shu et al., 2020). Instead, our goal is to isolate the group of invariant factors (i.e., the content partition) from the remaining factors of variation the data.
Specifically, our goal is to show that contrastive learning can block-identify the content variables for the multimodal setting described in Section 3. We formalize this in Theorem I and thereby relax the assumptions from previous work by allowing for distinct generating mechanisms fi f2 and additional modality latent variables.
Note that the asymmetry between z and za (or between s and 5) is not strictly required. We chose write 72 as a perturbation of z; to simplify the notation and for consistency with previous work. Instead, we could model both z and z2 via perturbations of z, as deserihed in Appendix A.2. "If style variable would he perturbed with zero prohahility, it would content variable.
5

---

Published as a conference paper at ICLR 2023
Generative process
R' (nonlinear)
Generative process
R (nonlinear)
p(chg.)
Stut.
Cun.
Content
Style
p(chg.)
Cau.
Content c
Style s
Modality m.
1,0
K
x
1,00
:: 0.00
0.00 1: 0.05
K
0.99 = 0.30
0.00 0.00
0.00 0.00
x
0,99
0.00
0.00 0.00
0.75
/
K
0.99
1: 0.00
0.52
1 0.09
0.75
0.95
0.01
0.56 0.23
0.00 1: 0.00
0.25
K
1.00
0.00
0.79 0.04
0.75
0.98
0.00
0.87 +-7---0000
0.00 +:000
0.75
0.99 0.01
0.81 0.04
0.75
0.95 0.03
0.89 0.07
0.00 0.00
(a) Original setting
(b) Multimodal setting
Table 1: Results of the numerical simulations. We compare the original setting (f, = fz. left table) with the multimodal setting (f1 f2, right table). Only the multimodal setting includes modality- specific latent variables. Fach row presents the results of different setup with varying style-change probability p(chg-) and possible statistical (Stat.) and/or causal (Caus.) dependencies. Each value denotes the Rt coefficient of determination (averaged across 3 seeds) for a nonlinear regression model that predicts the respective ground truth factor (c, or m;) from the learned representation.
Theorem 1. Assume the data generating process described in Sec. 3.1, i.e. data pairs (X1; X2) generated from Equation (6) with Pz: [a),mma) and Pna i as defined in Assumptions I and 2. Further, assume that Pz is a smooth and continuous density on Z with Pa p^(z) >> O almost everywhere. Let B1 : Xr (0, 1 th and 82 : X2 - (0,1)a be smooth functions that minimize LsymAlignMaddn as defined in Eq. (5). Then, 81 and g2 block-identify the true content variables in the sense of Def. 1.
A proof of Theorem I is provided in Appendix A.1. Intuitively, the result states that contrastive learning can identify the content variables up to block-wise indeterminacy. Similar to previous work, the result is based the optimization of the asymptotic form of the contrastive loss (Equation 5). Moreover, Theorem 1 assumes that the number of content variables is known or that it can be estimated (e.g.. with a heuristic like the elbow method). We address the question of selecting the encoding size with dimensionality ablations throughout our experiments. In Section 7, we will return to the discussion of the assumptions in the context of the experimental results. experiinte
5 EXPERIMENTS
The goal of our experiments is to test whether contrastive learning can block-identify content in the multimodal setting, as described by Theorem 1. First, we verify identifiability in fully controlled setting with numerical simulations (Section 5) Second, we corroborate our findings on a complex multimodal dataset of image/text pairs (Section 5.2). The code is provided in our github repository."
5.1 NUMERICAL SIMULATION
We extend the numerical simulation from von Kügelgen et al. (2021) and implement the multimodal setting using modality-speciff mixing functions (f, * f2) with modality-speciffi latent variables. The numerical simulation allows to measure identifiability with full control over the generative process. The data generation is consistent with the generative process described in Section 3. We sample (0, m, (O, Σm,) ). and s A(aa + Bc, E,). Statistical dependencies within blocks (e.g., among components of c) are induced by non-zero off-diagonal entries in the corresponding covariance matrix (e.g. in Ex). To induce causal dependence from content to style, we set d, B3 ~ AO. 00 1): otherwise, we set 0,, B. = O. For style changes, Gaussian noise is added with probability æ independently for each style dimension: 5; St + e, where
E.,,)..
(O, Σ.) with probability π. We generate the observations x; = f, (c S, m, and X2 = f2(c, 5, m2) using two distinet nonlinear mixing functions, i.e, for each E {1, 2). f. dd - .Rd is a separate, invertible 3-layer MLP with LeakyReLU activations. We train the encoders for 300,000 iterations using the symmetrized InfoNCE objective (Equation 4) and the hyperparameters listed in Appendix B. We evaluate block-identifiabiin by predicting the ground truth factors from the learned representation using kernel ridge regression and report the n² coefficient determination on holdout data.
'https: //gi htt///gtttttuuu chtb.ccom/mmatttuuuuaveeeeeeeeee modal -contrast learning parningg

---

Published as a conference paper at ICLR 2023
Results We compare the original setting (f, f2, Table la) with the multimodal setting (f1 + fz. Table 1b) and find that content can be block-identified in both settings, as the R² score is close to one for the prediction of content, and quasi-random for the prediction of style and modality-specinff information. Consistent with previous work, we observe that some style information can be predicted when there are statistical and/or causal dependencies; this is expected because statistical dependencies decrease the effective dimensionality of content, while the causal dependence с - s makes style partially predictable from the encoded content information. Overall, the results of the numerical simulation are consistent with our theoretical result from Theorem 1, showing that contrastive learning can block-identify content in the multimodal setting.
5.2
IMAGE/TEXT PAIRS
Next, we test whether block iieck-ideniifiabiiiyy holds in a more realistic setting with image/text pairs two complex modalities with distinct generating mechanisms. We extend the Causal3DDdent dataset (von Kügelgen et al., 2021; Zimmermann et al., 2021), which allows us to measure and control the ground truth latent factors used to generate complex observations. We use Blender (Blender Online Community, 2018) to render high-dimensional images that depict a scene with a colored object illuminated by differently colored spotlight and positioned in front of a colored background. The scene is defined by 11 latent factors: the shape of the object (7 classes), position of the object (z, y, 2 coordinates), orientation of the object (a, B, angles), position of the spotlight (0 angle), as well as the color of the object, background, and spotlight respectively (one numerical value for each).
Didde We extend the Cadsal3DIdent dataset the multimodal setting as follows. We gener- ate textual descriptions from the latent factors by adapt- ing the text rendering from the CLEVR dataset (Johnson et al., 2017). Each image/text pair shares information about the shape of the object (cow, teapot, etc.) and its position in the scene (e.g., bottom-righh). For each position factor, we use three clearly discernable values (opp/cntee/bbottom;; left/center/right)), which can be de- scribed in text more naturally than coordinates. While shape and position are always shared (i.e., content) be- tween the paired image and text, the color of the object is causally influenced by position and is stochastically shared (i.e., style). For the object color, we use a continuous hue value, whereas for the text we mathh the RGB value with the nearest value from a given palette (i.e., a list of named colors, such brown, beige, olive, etc.). The color palette is randomly sampled from a set of three palettes to ensure the object color depicted in the image does not uniquely determine the color described in the text. As modality-specific factors for the images, we have object rotation, spotlight position, and background color, while for the textual descriptions, we follow Johnson et al. (2017) and use 5 different types of phrases to introduce modality-speciff variation. Examples of image/text pairs are shown in Figure 2. Further details about the dataset are provided in Appendix B.I.
The battom-right of the image represents
On the top-center A teapot of bright the image, there
teal color visible, cyan object, lawngreen colored the form of head. cow.
positioned the mid-center of the image.
Figure 2: Examples of image/text pairs.
We train the encoders for 100,000 iterations using the symmetrized InfoNCE objective (Equation 4) and the hyperparameters listed in Appendix B.1. For the image encoder we use ResNet- 18 architecture (He al., 2016) and for the text we use convolutional network. As for the numerical simulation, we evaluate block block-identifiabiiiy by predicting the ground truth factors from the learned representation. For continuous factors, we use kernel ridge regression and report the R² score, whereas for discrete factors we report the classification accuracy of an MLP with a single hidden layer.
Results Figure presents the results on Multimodal3Dldent with dimensionality ablation, where we vary the size of the encoding of the model. Content factors (object position and shape) are always encoded well, unless the encoding size is too small (i.e., smaller than 3-4 dimensions). When there is sufficient capacity, style information (object color) is also encoded, partly because there is a causal dependence from content to style and partly because of the excess capacity, as already observed in previous work. Image-specifie information (object rotation, spotlight position, background color) is mostly discarded, independent of the encoding size. Text-specific information (phrasing) is encoded to a moderate degree (48-80% accuracy). which we attribute to the fact that phrasing is a discrete factor that violates the assumption of continuous latents. This hints at possible limitations in the
7

---

Published as a conference paper at ICLR 2023
1.
prediction per perfrmance
Object sposition
prediction per performacce
Object position
Object shape
0.6
Otject color
0.6
Object rotation
Obyect rotation
0.4
Object rotation
0.4
Spotlight posttion
Object x position
Spetlight color
Otject position
0.2
Background
0.2
Object shape
Object color
Text phrasing
0.0
0.0
2
6
8
10
12
1
3
8
10
12
encoding size
encoding size
(a) Prediction of image factors
(b) Prediction text factors
Figure 3: Results on Multimolalllideei as a function of the encoding size of the model. We assess the nonlinear prediction of ground truth image factors (left subplot) and text factors (right subplot) to quantify how well the learned representation encodes the respective factors. Content factors denoted bold and style factors in italic. Along the x-axis, we vary the encoding size, i.e., the output dimensionality of the model. We measure the prediction performance in terms of the R² coefficient of determination for continuous factors and classification accuracy for discrete factors respectively. Each point denotes the average across three seeds and bands show standard deviation.
presence of discrete latent factors, which we further investigate in Appendix B.2 and discuss in Section 7. Overall, our results suggest that contrastive learning can block-identify content factors in complex multimodal setting with image/text pairs.
6 RELATED WORK
Multi-view nonlinear ICA The goal of multi-view nonlinear ICA is identify latent factors shared between different views, as described in Section 2.1. There is thread of works (Gresele et al., 2019; Locatello et al., 2020) that recover the latent variable up in setting with mutually independent latent components, or up to block-wise inderterminacies in the case of independent groups of shared and view-specific components (Lyu and Fu, 2020; Lyu et al., 2022). Beyond the assumption of independent (groups of) components, there a line of works (von Kügelgen al., 2021; Kong al., 2022) that partition the latent space into blocks of invariant and blocks of changing components and show that the invariant components can be identified up to block-wiss indeterminacy, even when there are nontrivial dependencies between latent components. Our work advances in this direction and considers heterogeneous modalities with nontrivial statistical and causal dependencies between latents. We prove that shared factors can be block- bock-iidet in novel setting with modality-specifff mixing functions and modality-specioe latent variables.
component-wist indeterminacy
Multimodal representation learning Multimodal representation learning seeks to integrate infor- mation from heterogeneous sources into joint representation (Baltrušaitis et al., 2019; Guo et al., 2019). There is a myriad of methods designed to learn representations of multimodal data either directly indirectly. Among methods that learn representations indirectly, there multimodal autoencoders (Ngiam et al., 2011; I; Geng et al., 2022; Bachmann et al., 2022: Aghajanyan et al., 2022) and large variety of multimodal generative models (e.g., Suzuki al.. 2016: Wu and Goodman. 2018; Shi et al.. 2019; Huang et al., 2018; Tsai et al., 2019; Ramesh et al., 2021) that learn repre- different forms of reconstruction and/or masked prediction error. A more direct approach is taken by decoder-free methods that maximize the similarity between the encodings of different modalities. This class methods includes nonlinear canonical correlation analysis (Akaho, 2001; Bach and Jordan, 2002; Andrew et al., 2013; Wang al., 2016: Tang et al., 2017; Karami and Schuurmans, 2021) 1))e))))) as well as multi-view and multimodal contrastive learning (Tian et al., 2019; Bachman et al., 2019; Federici et al., 2020; Tsai al., 2021; Radford et al., 2021; Poklukar et al., 2022). While all of the named methods aim to integrate information across modalities, they do not answer the underlying question of identifiability, which our work seeks to address.
sentations by backpropagation
8

---

Published as a conference paper at ICLR 2023
7 DISCUSSION
Implications and scope We have shown that contrastive learning can block-identify shared factors in the multimodal setting. Numerical simulations (Section 5.1) verify our main theoretical result (The- orem 1), showing that contrastive learning block-identifies content information (Definition 1), when the size of the encoding matches the number of content factors. Experiments on a complex dataset of image/text pairs corroborate that contrastive learning can isolate content in more realistic setting and even under some violations of the assumptions underlying Theorem 1. In Appendix B.2, we include further experiments that test violations with discrete factors and dimensionality ablations that examine the robustness and sample complexity. More generally, we observe that contrastive learning encodes invariant information (i.e., content) very well across all settings. When there is sufficient capacity, stochastically shared information (i.c., style) is encoded to a moderate degree, but without affecting the prediction of invariant information. For practice, our results suggest that contrastive learning without capacity constraints can encode any shared factor, irregardless of whether the factor truly invariant across modalities or if its effect on the observations is confounded by noise or other factors. This is in line with the information-theoretie view (Oord et al., 2018; Poole et al., 2019), i.e., that contrastive learning maximizes the mutual information between representations- measure of mutual dependence that quantifies any information that is shared. Our results demonstrate that the size of the encoding can be reduced to learn a representation that recovers invariant information, as captured by the notion of block-identifiablliyy In practice, this can be leveraged for representation learning in settings of content- preeettiees distribution shifts (Mitrovic al., 2021; Federici et al., 2021), where information relevant for downstream task remains unchanged.
Limitations and outlook First, Theorem 1 suggests that only invariant factors can be block- identified. However, in practice, there can be pairs of observations for which the invariance is inadvertently violated, e.g. due to measurement errors, occlusions, or other mistakes in the data collection. On the one hand, such violation can be viewed as mere artifact of the data collection and could be managed via interventions on the generative process, e.g., actions in reinforcement learning (Lippe et al.. 2022: Brehmer al., 2022; Ahuja al., 2022: Lachapelle et al., 2022). On the other hand, violations of the content- inteetttiivarinceee blur the line between content and style factors and it would be interesting to study identifiability a more general setting with only stochastically shared factors. Second, Theorem assumes that the number of content factors is known or that it can be estimated. In practice, this might not significant limitation, since the number of content factors can be viewed as single hyperparameter (c.g., Locatello et al., 2020), though the design of suitable heuristics is an interesting research direction. We explore the idea of estimating the number of content factors in Appendix B.2 Figure 7. Third, Theorem 1 assumes that all latent factors are continuous. While this assumption prevails in related work (Hyvärinen and Pajunen, 1999; Hyvärinen and Morioka, 2016; Hyvärinen et al., 2019; Gresele et al., 2019; Locatello et al., 2019; 2020; Zimmermann et al., 2021; von Kügelgen et al., 2021; Klindt et al., 2021), our results in Figure 3b indicate that in the presence of discrete factors, some style or modality-speiffe information can be encoded. In Appendix B.2 Figure 5, we provide numerical simulations that support these findings. Finally, our model can be extended more than two modalities a setting for which there are intriguing identifiability results (Gresele et al., 2019; Schölkopf et al., 2016) as well as suitable learning objectives (Tian al., 2019; Lyu et al., 2022). Summarizing, the described limitations mirror the assumptions on the generative process (Section 3), which may be relaxed in future work.
8 CONCLUSION
We addressed the problem of identifiability for multimodal representation learning and showed that contrastive learning can block-identify latent factors shared between heterogeneous modalities. We formalize the multimodal generative process novel latent variable model with mpdelity- generative mechanisms and nontrivial statistical and causal dependencies between latents. We prove that contrastive learning can identify shared latent factors up to block-wise indeterminacy and therefore isolate invariances between modalities from other changeable factors. Our theoretical results corroborated by numerical simulations and complex multimodal dataset of image/text pairs. More generally, we believe that our work will help in shaping theoretical foundation for multimodal representation learning and that further relaxations of the presented generative process offer rich opportunities for future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLED
ID was supported by the SNSF grant #200021-188466. EP was supported by the grant #2021-911 of the Strategic Focal Area "Personalized Health and Related Technologies (PHRT)" of the ETH Domain (Swiss Federal Institutes of Technology). Experiments were performed on the ETH Zurich Leonhard cluster. Special thanks Kieran Chin- Chin-Cheong for his support the early stages of the project as well as to Luigi Gresele and Julius von Kügelgen for helpful discussions.
REPRODUCIBILITY STATEMENT
For our theoretical statements, provide detailed derivations and state the necessary assumptions. The generative process is specified in Section and the assumptions for block-identiffabiiiyy are referenced in Theorem 1. We test violations of the key assumptions with suitable experiments (dimensionality ablations; discrete latent factors) and discuss the limitations of our work in Section Further, empirically verify theoretical results with numerical simulations and complex multimodal data. To ensure empirical reprodueibility, the results of every experiment were averaged multiple seeds and are reported with standard deviations. Information about implementation details, hyperparameter settings, and evaluation metrics are included in Appendix B. Additionally, we publish the code to reproduce the experiments.
REFERENCES
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu. H., Goyal, N., Okhonko, D.. Joshi. M.. Ghosh, G., Lewis, M., and Zettlemoyer, L. (2022). CM3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520.
Ahuja, K., Hartford, J., and Bengio, Y. (2022). Weakly supervised representation learning with sparse perturbations. In Advances Neural Information Processing Systems.
Akaho, S. (2001). A kernel method for canonical correlation analysis. arXiv preprint arxiv:cs/060907..
Andrew, G., Arora, R., Bilmes, J., and Livescu, K. (2013). Deep canonical correlation analysis. In International conference on machine learning.
Bach, F. R. and Jordan, M. I. (2002). Kernel independent component analysis. Journal of machine learning research, 3:1-48.
Bachman, P., Hjelm, R. D., and Buchwalter, W. (2019). Learning representations by maximizing mutual information across views. Advances in neural information processing systems.
Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A. (2022). MultMMEE: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision.
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. (2019). Multimodal Machine Learning: A survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423 443.
Bengio, Y., Courville, A. C., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1798)11798
Blender Online Community (2018). Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam.
Brehmer, J., de Haan, P., Lippe, P., and Cohen, T. (2022). Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems.
Chen, J. and Batmanghelich, K. (2020). Weakly supervised disentanglement by pairwise similarities. In AAAI Conference on Artificial Intelligence.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, E. (2020). A simple framework for contrastive learning of visual representations. In International Conference Machine Learning.
10

---

Published as a conference paper at ICLR 2023
Danmois, G. (1951). Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences.
Federici, M., Dutta, A., Forré, P., Kushman, N., and Akata, Z. (2020). Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations.
Federici, M., Tomioka, R., and Forré, P. (2021). An information-theorett approach to distribution shifts. In Advances Neural Information Processing Systems.
Geng, X., Liu, H., Lee, I.,, Schuurams, D., Levine, S., and Abbeel, P. (2022). Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205.142: 14204.
Gresele, L.,, Rubenstein, P. K. Mehrjou, A., Locatello, F,, and Schölkopf, B. (2019). The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ICA. In Conference on Uncertainty in Artificial Intelligence.
Guo, W., Wang. J., and Wang, S. (2019). Deep multimodal representation learning: A survey. IEEE Access, 7:63373-63394.
Gutmann, M. and Hyvärinen, A. (2010). Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition.
Higgins, I,, Amos, D., Pfau, D., Racanière, S., Matthey, L.,, Rezende, D. J., and Lerchner, A. (2018). Towards a definition of disentangled representations, arXiv preprint arXiv: 1812.02200
Huang, X., Liu, M., Belongie, S. J., and Kautz, J. (2018). Multimodal unsupervised Image-to-image translation. In European Conference on Computer Vision.
Hyvärinen, A. and Morioka, H. (2016). Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems.
Hyvärimen, A. and Morioka, H. (2017). Nonlinear ICA of temporally dependent stationary sources. In International Conference on Artificial Intelligence and Statistics.
Hyvärinen, and Pajunen, P. (1999). Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439.
Hyvärinen, A., Sasaki, H., and Turner, R. E. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning. International Conference on Artificial Intelligncce and Statistics.
James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An introduction to statistical learning. volume 112. Springer.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, L., and Girshick, R. (2017). CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Conference Computer Vision and Pattern Recognition.
Karami, and Schuurmans, (2021). Deep probabilistic canonical correlation analysis. In AAAI Conference on Artificial Intelligence.
Khemakhem, I., Kingma, D. P., Monti, R. P., and Hyvärinen, A. (2020). Variational autoencoders and nonlinear ICA: a unifying framework. In International Conference on Artificial Intelligence and Statistics.
Klindt, D. A., Schott, L., Sharma, Y., Ustyuzhaninov, I., Brendel, W., Bethge, M., and Paiton, D. M. (2021). Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations.
Kong, L., Xie, S., Yao. W., Zheng, Y. Chen, G., Stojanov, P., Akinwande, V., and Zhang, K. (2022).
Partial disentanglement for domain adaptation. In International Conference Machine Learning.

---

Published as a conference paper at ICLR 2023
Lachapelle, S., Rodriguez, P., Sharma, Y., Everett, K. E., Le Priol, R., Lacoste, A., and Lacoste- Julien, S. (2022). Disentanglement via mechanism sparsity regularization: a new principle for nonlinear ICA. In Conference Causal Learning and Reasoning.
Lehmann, E. L. and Casella, G. (2006). Theory of point estimation. Springer Science & Business Media.
Lippe, P., Sarah, M., S., L., M., A. Y., Taco, C., and S., G. (2022). CITRIS:
Causal identifiability from temporal intervened sequences. In International Conference Machine Learning.
Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference Machine Learning.
Locatello, F., Poole, B., Rätsch, G., Schölkopf, B., Bachem, O., and Tschannen, M. (2020). Weakly- supervised disentanglement without compromises. In International Conference on Machine Learning.
Lyu, Q. and Fu, X. (2020). Nonlinear multiview analysis: Identifiability and neural network-assisted implementation. IEEE Trans. Signal Process., 68:2697-2712.
Lyu, Q.. Fu, X., Wang, W., and Lu, S. (2022). Understanding latent correlation- based multiview learning and self-supervision: An identifiability perspective. In International Conference Learning Representations.
Mitrovic, J., McWilliams, B., Walker, J. C., Buesing, L. H., and Blundell, C. (2021). Representation learning via invariant causal mechanisms, In International Conference on Learning Representa- fions.
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Y. (2011). Multimodal deep learning. In International Conference on Machine Learning.
Oord, A. v. d., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint 1
Poklukar, P., Vasco, M., Yin, H., Melo, F. S., Paiva, A., and Kragic, D. (2022). Geometric multimodal
contrastive representation learning. In International Conference Machine Learning.
Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. (2019). On variational bounds of mutual information. In International Conference on Machine Learning.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C.,, and Chen. M. (2022). Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06/225.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021).
Zero-shot text-to-image generation. In International Conference on Machine Learning.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Conference Computer Vision and Pattern Recognition.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.,, Ghasemipour S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. (2022). Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems.
Salakhutdinov, R. and Hinton, G. E. (2009). Deep boltzmann machines. In International Conference Artificial Intelligence and Statistics.
Schölkopf, B., Hogg, D. W., Wang, D.,, Foreman-Mackey, D., Janzing, D., Simon- Gabriel, C.-J., and Peters, J. (2016). Modeling confounding by half-sibling regression. Proceedings of the National Academy of Sciences, 113(77)77311-7398
12

---

Published as a conference paper at ICLR 2023
Shi, Y., Siddharth, N., Paige, B., and Torr, P. (2019). Variational mixture--f-experrs autoencoders for mudti-modal deep generative models. In Advances Neural Information Processing Systems.
Shu, R., Chen, Y., Kumar, A., Ermon, S., and Poole, B. (2020). Weakly supervised disentanglement with guarantees. In International Conference on Learning Representations.
Song, L.., Anandkumar, A., Dai, B., and Xie, B. (2014). Nonparametric estimation of multi-view latent variable models. In International Conference on Machine Learning.
Suzuki, M., Nakayama, K., and Matsuo, Y. (2016). Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.018%
Tang, Q. Wang, W., and Livescu, K. (2017). Acoustic feature learning via deep variational canonical correlation analysis. In INTERSPEECH.
Tian, Y., Krishnan, D., and Isola, P. (2019). Contrastive multiview coding. arXiv preprint 19Xiv::906.05899.
Tsai, H., Bai, S., Liang, P., Kolter, J. Z., Morency, L., and Salakhutdinov, R. (2019). Multimodal transformer for unaligned multimodal language sequences. Conference of the Association for Computational Linguistics.
Tsai, Y. H., Wu, Y., Salakhutdinov, R., and Morency, L. (2021). Self-supervised learning from multi-view perspective. In International Conference on Learning Representations.
von Kügelgen, J., Sharma, Y., Gresele, L., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F. (2021). Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems.
Wang, T. and Isola, P. (2020). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning.
Wang, W., Lee, H., and Livescu, K. (2016). Deep variational canonical correlation analysis. arXiv preprint
Wu, M. and Goodman, N. (2018). Multimodal generative models for scalable weakly-supervised learning. In Advances in Neural Information Processing Systems.
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P. (2022). Contrastive learning of medical visual representations from paired images and text. Machine Learning for Healthcare.
Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. (2021). Contrastive learning inverts the data generating process. In International Conference Machine Learning.
13

---

Published as a conference paper at ICLR 2023
A THEORY
A.1 PROOF OF THEOREM 1
Theorem 1. Assume the data generating process described in Sec. 3.1, i.e. data pairs (X1,X)) generated from Equation (6) with Pz, = (x),(ma) and Paala as defined in Assumptions and Further, ussume that Pv is a smooth and continuous density on Z with pa(z) > O almost everywhere. Let 81 X1 + (0, 1)ne and g2 X2 + (0,1)ac be smooth functions that minimize MyymllignMucEm as defined in Eq. (5). Then, gı and g2 block-identify the true content variables in the sense of Def.
Proof. To prove Theorem I, we follow the proof structure from von Kügelgen et al. (2021, The- orem 4.4) and divide the proof into three steps. First, we show that there exists pair of smooth functions gi: 82 that attain the global minimum of LsymAlignMaxEnt (Eq. 5). Further, Equations (13- 15), we derive invariance conditions that have to hold almost surely for any pair of smooth functions g1 82 attaining the global minimum of Eq. (5). In Step 2, we use the invariance conditions derived in Step show by contradiction that any pair of smooth functions g1: 82 that attain the global minimum in Eq. (5) can only depend on content and not on style or modality-speciff information. In the third and final step, for h =- gi and h2 == g2 f2, we show that both functions must be bijections and hence that с is block-identified by g1 and g2 respectively.
Step 1. Recall the asymptotic form of the objective, as defined in Equation (5):
SsymAlignMasiin g 82) B(n (x1) (xx))ll 1/2 (H(g (xx)) + (x)2(x)))) (5)
The global minimum of EsymAlignMaxEan is reached when the first term is minimized and the second term is maximized. The first term is minimized when the encoders g1 and g2 are perfectly aligned, i.e. when g1 (x1) = gg (x2) holds for all pairs (2), ~ Px1,22 The second term attains its maximum when g1 and g2 map to a uniformly distributed random variable on (0, 1) 1)dc respectively.
To show that there exists a pair of functions that minimize CsymAlignMasttins let gì := d1 o fT. and let g2 d2 f. where the subscript I,n, indexes the subset of content dimensions w.l.o.g. and where d1 and d2 will be defined using the Darmois construction (Darmois, 1951; Hyvärinen and Pajumen. 1999). First, recall that fT (x1) ):n. = c and that fy (xx)1mm = ē by definition. Second, f E {1, 2]. let us define d; C + (0. 1) using the Darmois construction, such that d, maps and ē to a uniform random variable respectively. It follows that gi gž are smooth functions, because any function d2 obtained via the Darmois construction is smooth and f f are smooth as well (each being the inverse of a smooth function).
Next, we show that the pair of functions gi: g2, as defined above, attains the global minimum of the objective EsymAlignMaxEntt We have that
CoymAigeMMatan( g;) = B(xi p
g (x1) 1/ ((
(9)
B(xi P
|| (c) - dz(2) [2] +4
H
(10)
0
(11)
where by Assumption c = almost surely, which implies that the first term is zero almost surely. Further, d; maps c, é to uniformly distributed random variables (0, 1)"e, which implies that the differential entropy of dl (c) and dz(ā) is zero, as well. Consequently, there exists a pair of functions g7.g2 that minimizes LsymAlignMaxEntt
Next, let gi : 21 + (0. 1) and g2 22 + (0, Te be any pair of smooth functions that attains the global minimum of Eq. (5), i.e.,
CSymAlignManken 82) (2)
||g (x1) g (x2) 1/2 (H (x1)) + (x,))) 0
(12) Let h, == gi o f and h, == g2 o fz. and notice that both are smooth functions since all involved functions are smooth by definition. Since Equation (12) is a global minimum, it implies the following
'Note that we restrict the range of g1 and g2 to (0, 1) by definition merely to simplify the notation. Generally, the uniform distribution L/ h) is the maximum entropy distribution on the interval (a
14

---

Published as a conference paper at ICLR 2023
invariance conditions for the individual terms:
E (2) [l]ba hz(z))lla)
(13)
((h,
(14)
((2)(z)))
(15)
Hence, h (2)) = h2(22) must hold almost surely w.r.t. Xx Additionally, Equation (14) (resp.
Equation (15)) implies that č₁ hi (21) (resp. C22 (2(22)) must be uniform on (0, 1)
Step 2. Next, we show that any pair of functions that minimize LsymAlignMaxEat depend only on content information. Since style is independent of m, and m2, we first show that h1 (z1) does not depend on m1, and that h2 (Z2) does not depend on m2. We then show that h and h also cannot depend on style, based on a result from previous work.
First note, that we can exclude all degenerate solutions where gi maps a component of m, to a constant, since would not be invertible anymore and such solution would violate the invariance in Eq. (14). To prove contradiction, suppose that, w.l.ogg. h1 (c, m. )mm h1 (21 ). depends on some component in m, in the sense that the partial derivative of h1 11)1:m. w.r.t. some modality- specific variable 11. is non-zero for some point (c" , mf) E Z1. Specifically, it implies that the partial derivative 8h, (me//m// is positive in neighborhood around (c', ss s""""" which is z2 and mi, the fact that (g(22) (22) cannot not depend on mi, and that p(z) > almost everywhere, we come to a contradiction. That is, there exists an open set of points with positive measure, namely the neighbourhood around (c", s" .m which
a non-empty open set, since h1 is smooth. On the other hand, due to the independence
[h (2)()))ss) im. (222))
(16)
almost surely, which contradicts the invariance in Equation (13). The statement does not change, if we add further dependencies of h on components of mi, for h2 on components of m2, because mi and Z2 are independent, and m2 and z are independent as well. Hence, we show that any encoder that minimizes the objective in Equation (5) cannot depend on modality-speciff information.
Having established that neither h1 (Z1)1.m nor he (222) can depend on modality-specific informa- tion, it remains show that style information not encoded, as well. Leveraging Assumption we can show that the strict inequality in Equation (13) has a positve density if h, (Z1) )ine or hy (Zz) )mme was dependent on a dimension in s respectively which would again lead to a violation of the invariance derived in Equation (13), as shown in von Kügelgen et al. (2021, Proof of Theorem 4.2).
Step 3. remains to show that h, h2 are bijections. We know that and (0, 1)de are simply connected and oriented C' manifolds, and we have established in Step I that hJ and h2 are smooth and hence differentiable functions. Since Pe is a regular density, the uniform distributions w.r.t. the pushthrough functions h, and h are regular densities. Thus, h1 and h are bijections (Zimmermann et al., 2021, Proposition 5)
Step 3 concludes the proof. We have shown that for any pair of smooth functions g g2 that attain the global minimum of Eq. (5), we have that e is block-identifee (Def. 1) by gı and g2-
A.2 SYMMETRIC GENERATIVE PROCESS
Throughout the main body of the paper, we described an asymmetric generating mechanism, where Z2 is a perturbed version of 1₁. Here, we will briefly sketch out how our model and results can be adapted symmetric setting, where both z and Z2 are generated as perturbations z.
Concretely, we would need to make small adjustments to Assumptions and 2 as follows. We start with the content invariance in Assumption 1, which specifies how z₁ = (01, 51 m;) and 22 (ō2, 52, mī²) are generated.
Let i E [1,2]. The conditional density Pz, over Z, x takes the form
Pz, 2 (z, 1) 8(ē. c) m m: )s. (S.ls)
(17)
where 5(.) is the Dirac delta function, i.e., ē; almost everywhere, well as m, m, almost everywhere. Note that since è = a.c. and c = č a.c., it follows that è = c2 almost everywhere, which is property that is needed in Step in the proof of Theorem 1. In addition, it still holds
15

---

Published as a conference paper at ICLR 2023
that m TT 23. for i,j {1,2) and * j. which is needed in Step 2 of the proof to show that modality-speeiffe information is not encoded.
Lastly, need to revisit Assumption 2, for which both i and 5 would be generated through perturbations of via the conditional distribution Ps,|s on S x S, as described in Assumption 2, for each individually. As small technical nuance, would need specify the conditional generation of the perturbed style variables 8 and $2 such that they are not perturbed in an identical manner w.r.t. s. This can be ensured by, e.g., constraining PA appropriately to exclude the degenerate case where dimensions in S and $2 are perfectly aligned-a case that needs to be excluded for Step 2 of the proof of Theorem 1.
B EXPERIMENTS
B.1 EXPERIMENTAL DETAILS
Numerical simulation The generative process is described in Section 5.1. Here, provide additional information about the experiment. The invertible MLP is constructed similar to previous work (Hyvärinen and Morioka, 2016; Hyvärinen and Morioka, 2017: Zimmermann et al., 2021; von Kügelgen et al., 2021) by resampling square weight matrices until their condition number surpasses a threshold value. For the original setting (f1 = f2), we use one encoder (g, = gg). whereas for the multimodal setting (f, & f2), we use distinct encoders (gı & g2) to mirror the assumption of distinct mixing functions and because, in practice, the dimensionality of the observations can differ across modalities. In Table 2a, we specify the main hyperparameters for the numerical simulation.
Multimodal3Dldent Our dataset of image/text pairs is based on the code used to generate the Cadsal3DIdent (von Kügelgen et al., 2021; Zimmermann et al., 2021) and CLEVR (Johnson et al., 2017) datasets. Images are generated using the Blender renderer (Blender Online Community, 2018). The rendering serves as a complex mixing function that generates the images from 11 different parameters (i.e., latent factors) that are listed in Table 3. To generate textual descriptions. we adapt the text rendering from CLEVR (Johnson et al., 2017) and use 5 different phrases to induce modality-specifft variation. The latent factors used to generate the text are also listed in Table 3. The dependence between the image and text modality is determined by three content factors (object shape, x-position, and y-position) and one style factor (object color). For the object color in the image, we use continuous hue value, whereas for the text we match the RGB value with the nearest color value from one of three different palettes? that is sampled uniformly at random for each observation. Further, ensure that there are overlapping color values across palettes by using prefix for the respective palette (e.g., "xkcd:black") when necessary. In Section 5.2, we use a version of the Mulimmodallidden dataset with causal dependence from content style. Specifically, the color of the object depends on its x-position. In particular, we split the range of hue values [o, 1] into three equally sized intervals and associate each of these intervals with a fixed x-position of the object. For instance, if x-position is "left", we sample the hue value from the interval [0. 1/s]. Consequently, the color of the object can be predicted to some degree from the position of the object. Samples of image/text pairs from the Multimodal 3DIdent dataset are shown in Figures 2 and 4. The hyperparameters for the experiment are listed Table 2b. In Appendix B.2, we provide additional results for a version of the dataset with mutually independent factors.
Hligh-dimensin image pairs In Appendix B.2, we provide additional results using dataset of high-dimensional pairs of images of size 224x224x3. Similar to Mullimodal3 images are generated using Blender (Blender Online Community, 2018) and code adapted from previous work (Zimmermann et al.. 2021; von Kügelgen et al., 2021). Each image depicts a scene with one type of object (a teapot, like in Zimmermann et al., 2021) in front of colored background and illuminated by colored spotlight (for examples, see Figure 9). The scene is defined by 9 continuous latent variables each of which is sampled from a uniform distribution. Object positions (x-, y- and z-coordinates) are content factors that are always shared between modalities, while object-, spotlight- and background-colors are style factors that are stochastically shared. object rotation (a and | angles) for one modality and spotlight position for the other. To simulate modality-speciffi mixing functions, we render the objects using distinct textures (i.e., rubber and metallic) for each modality. Further, generate two versions of this dataset, with and without
factors are
16

---

Published as a conference paper at ICLR 2023
causal dependencies. For the dataset with causal dependencies sample the latent factors according to a causal model, where background depends on z-position and spotlight-color depends on object-color. We use ResNet- 18 encoders and similar hyperparameter values to those used for the image/text experiment (Table 2b).
Parameter
Value
Parameter
Value
Generating function
3-layer MLP
Generating function
Image and text rendering
Encoder
7-layer MLP
Image encoder
ResNet-18
Optimizer
Adam
Text encoder
4-layer ConvNet
Cond. threshold ratio
le-3
Optimizer
Adam
Dimensionality d
15
Batch size
256
Batch size
6144
Learning rate
1c-5
Learning rate
Temperature
Temperature т # Seeds
1.0
# Seeds
3
# Iterations
100,000
# Iterations
300,000
# Samples (train / val/test)
125,000/ 05,,000/10,,00/ 10,000
Similarity metric
Euclidian
Similarity metric
Cosine similarity
Gradient clipping
2-norm; max value 2
Gradient clipping
2-norm; max value
(a) Parameters used for the numerical simulation.
(b) Parameters used for Diddem
Table 2: Experimental parameters and hyperparameters used for the two experiments in the main text.
Latent factor
Distribution
Details
Object shape
Categorical
unique values
Object x-position
Categorical
unique values
Object y-position
Categorical
unique values
Object color
Uniform
hue value in [0,
Object rotation a
Uniform
angle value in [0,
Object rotation /
Uniform
angle value in [0,
Object rotation Y
Uniform
angle value in 00,
Spotlight position
Uniform
angle value in 00,
Spotlight color
Uniform
hue value in 00,
Background color
Uniform
hue value in (0,
Object shape
Categorical
unique values
Object x- x-position
Categorical
unique values
Object y-position
Categorical
unique values
Object color
Categorical
color names (3 palettes)?
Text phrasing
Categorical
unique values
Table 3: Description of the latent factors used to generate Mulimodall3Dlden The first 10 factors are used to generate the images and the remaining 5 factors are used to generate the text. Object z- position is kept constant for all images, which why we do list it among the generative factors. Independent factors are drawn uniformly from the respective distribution. Content factors are denoted in bold and style factors italic; the remaining factors are modality-specifff
We use the following three palettes from the matplot col.orlrrs API: Tableau colors (10 values). CSS4 colors (148 values), and XKCD colors (949 values).
17

---

Published as a conference paper at ICLR 2023
On the bottom- left the tabcoyan object the form of dragon.
On the top-center the there tabxoyan object the form of a heed.
A green yellow teapot mid-right of image.
An horse of tab:pink color visible posttioned in the bottom- center of the image.
A lawn green cow is the bottom- right the image.
xkodipoison hare the top-left of image.
spring green armadillo bottom-left the image.
Figure Examples image/text pairs from the Multimodal3Dident dataset. Each sample shows one of the seven shapes or classes of objects included in the dataset.
Generative process
R² (nonlincar)
Generative process
R² (nonlincar)
p(chg.)
Stat.
Cau.
Content c
Style s
p(chg.)
Stat.
Cau.
Content
Style
10
x
1,00 0.00
0.00
0.00 1: 0.00
L.
1,00 0.00
0.00 +: 0.00
0.75
x
X
0.99 : 0.01
0.00 + 0.00
0.75
X
X
1.00 = 0.00
0.00 0.00
0.75
/
x
0.99 = 0.00
0.52 + 0.09
0.75
x
0.99 = 0.01
0.36
0.75
x
= 0.00
0.79 + 0.04
0.75
1.00 + 0.00
0.03
0.75
0.01
0.81
0.75
0.99
0.05
(a) Original setting
(b) Multimodal setting
Table 4: Results of the numerical simulations without modality-specinf latent variables. We compare the original setting (f, f2. left table) with the multimodal setting (f; & f2, right table). Each row presents the results of different setup with varying style-change probability p(chg.) and possible statistical (Stat.) and/or causal (Caus.) dependencies. Each value denotes the n² coefficient of determination (averaged across 3 seeds) for nonlinear regression model that predicts the respective ground truth factor (c, s, or m,) from the learned representation.
B.2 ADDITIONAL EXPERIMENTAL RESULTS
Numerical simulation without modality-specife latents Recall that the considered generative process (Section 3) has sources of modelity-specifii variation: modality-specific mixing functions and modality-specific latent variables. To decouple the effect of these two sources of variation, we conduct an ablation study without mopelity-specific latent variables. Table 4 presents the results, showing that content is block-identified in both the original setting (f, fz, Table 4a) and the multimodal setting (f, 7 f2, Table 4b), Compared to Table I, we observe the content prediction is improved slightly in the case without modalltt-specifie latent variables. Hence, our results suggest that contrastive learning can block-identify content factors in the multimodal setting with and without modality sppecifcc latent variables.
Numerical simulation with discrete latent factors Extending the numerical simulation from Section 5.1, we test block-identifiabiiit of content information when observations generated from a mixture continuous and discrete latent variables, thus violating one of the assumptions from Theorem In this setting, content, style and modality-specifie information with 5 components sampled from either continuous normal distribution or a discrete multinomial distribution with & classes, for which we experiment with different À {3, 4, ...., 10). For all settings, we train an encoder with the InfoNCE objective and set the encoding size to 5 dimensions. The other hyperparameters used in this of experiments are detailed Table 2a. To ensure convergence of the models, we extended the number of training iterations to 600,000 and 3,000,000 for experiments and discrete content variables respectively. With discrete style variables and continuous content (Figures 5a and 5b), the results suggest that
random variables
with discrete sty aaylle/modlltyysseeeeie or modality-speche content block block-identified, since the prediction style and modality-speciffe information is at chance level (i.e., accuracy 1/k) while content is consistently fully recovered ( > 0.99). In the opposite setting, with continuous style and modality-speciiff variables and discrete content (Figure 5c), the number of content classes appears be critical factor for blck-identifiabllitt of content: while content is always encoded well, style information also encoded to significant extent when the number of content classes is small, but significantly less style can be recovered when the number
18

---

Published as a conference paper at ICLR 2023
pediction performacc
predict per peeformeres
predection performance
8.8
10.8
Content Syfts
Contant Style Modallty
0.66
Style Moda
0.0
0.4
Modalty
60.4
0.4
(a) Only is discrete
(h) Only m1, ma discrete
(c) Only is discrete
Figure 5: Numerical simulations with discrete latent factors. The results show three settings in each of which one group of latent variables is discrete while the remaining groups are continuous. Continuous variables are normally distributed, whereas discrete variables are sampled from multinomial distribution with distinct classes. We measure the prediction performance with a nonlinear model in terms of the R2 coefficient of determination for continuous factors and classification accuracy for discrete factors respectively. Each point denotes average across three seeds and error bars show the standard deviation, which is relatively small.
of content classes increases. Through this set of experiments, we challenge the assumption that all generative factors should be continuous (c.f., Section and show that block-identifiabiiiy of content can still be satisfied when content is continuous while style or modality-speciff variables are discrete. significant extent when content is discrete, which might
the other hand, style encoded explain our observation for the image/text experiment, where we saw that, in the presence of discrete content factors, some style information can be encoded.
Dimensionality ablations for the numerical simulation To test the effect of latent dimensionality on identifiability Figure presents dimensionality ablations where we keep the number of content dimensions fixed and only vary the number of style or modality-specific dimensions, nı, and 7mm respectively. Figures 6a and 6c confirm that block-identiffabiiityy of content still holds when we significantly increase the number of style or codality-speciffe dimensions, as the representation consistently encodes only content and no style or modality-specific information. In Figures 6b and 6d, we can observe that the training loss decreases more slowly when we increase the dimensionality of ne and respectively, which provides an intuition that the sample complexity might increase with the number of style and modality-specific dimensions.
Estimating the number of content factors The estimation of the number of content factors is an important puzzle piece, since Theorem assumes that the number of content factors is known or that can be estimated. In practice, the number of content factors be viewed as single hyperparameter Locatello et al., 2020) that can tuned with respect to a suitable model selection metric. For instance, one could use the validation loss for model selection, which would be convenient since the validation loss only requires a holdout dataset and no additional supervision. In Figure 7, we plot the validation loss (averaged over 2,000 validation samples) as a function of the encoding size for both experiments used in our paper. Results for the numerical simulation are shown in Figure 7a and for the image/text experiment Figure 7b. For both datasets, we observe that the validation loss increases most significantly in the range around the true number of content factors. For the numerical simulation, the results look promising as they show a clear "elbow" (James et al., 2013) at the correct value of 5, which corresponds to the true number of content factors. The results are less clear for the image/text experiment, where the elbow method might suggest the range of 2-4 content factors, while the true value is 3. While these initial results look promising, believe that more work is required to investigate the estimation the number of content factors and the design of suitable heuristics, which are interesting directions for future research.
19

---

Published as a conference paper at ICLR 2023
1.0
0.3
R (nonlinear)
0.6
Styte Modally
loss
0.2
0
50000
100000
150000 20000 250000 30000
modality-specific dimensionality Rm
training step
(a) Prediction performance a function of 7km
(b) Learning curves for different 1lm
1.0
0.8
FP (nonlinear)
0.6
Cortent e
loss value
Style
0.4
Modally mi
0.0
50000
150000 200000 250000 300000
style dimensionality
training step
(c) Prediction performance as function nı,
(d) Learning curves for different vt.
Figure 6: Dimensionality ablation for the numerical simulation. We consider the multimodal setting with mutually independent factors and test the effect of latent dimensionality on identifiability by keeping the number of content dimensions fixed and only varying the number of style modality- specific dimensions (n, and 7m respectively). In Figures 6a and 6c we measure the nonlinear prediction performance in terms of the R² coefficient of determination of nonlinear regression model that predicts the respective ground truth factor (c, s, or m,) from the learned representation. In Figures 6b and 6d, we plot the learning curves (i.e., the training loss) of the respective models to compare how fast they converge.
8.5
800
loss value
loss value
7.5
7.0-
6.5
8-
3
11
13
15
8
10
encoding size
encoding size
(a) Validation loss for the numerical simulation
(b) Validation loss for the image/text experiment
Figure 7: An attempt at estimating of the number content factors using the validation loss. The validation loss corresponds to the value of the LSyminioNCE objective computed dataset. Since we are interested in estimating the true number of content factors to select the encoding size appropriately, we plot the validation loss as a function of the encoding size. We show the validation loss for the numerical simulation with independent factors (Figure 7a) and for the image/text experiment (Figure 7b) respectively.
holdout
20

---

Published as a conference paper at ICLR 2023
Generative process
R (nonlinear)
p(chg.)
Stat.
Cau.
Content
Content c'
Style
Modality m,
1.0
X
0.60 1 0.00
1,00 1 0.00
0.00 1 0.00
0.00 1 0.00
0.75
x
x
0.00 + 0.00
1.00 + 0.00
0.00 0.00
0.00 0.00
0.75
X
0.00 1 0.00
1,00 1 0.00
0.50 1 0.19
0.00 1 0.00
0.75
x
V
0.01
+ 0.00
0.98
+ 0.00
0.03 + 0.01
0.00 + 0.00
0.75
V
0.28 0.14
0.91
0.39 0.20
0.00 0.00
Figure 8: Evaluation with test-time interventions. We use the interventional setup that is illustrated on the left, i.e. perturbed samples x1, X2 that are generated from the intervened content c', which is copy of the original content с with intervention, i.e., batch-wise permutation (~*) that makes c' independent of s. Each row presents the results of different setup with varying style-change probability p(chg.) and possible statistical (Stat.) and/or causal (Caus.) dependencies. Each value denotes the Ri coefficient of determination (averaged across 3 seeds) for a nonlinear regression model that predicts the respective ground truth factor (c, c', S, or m.) from the learned representation.
Evaluation with test-time interventions Previously, we observed that style can be predicted to some degree when there are causal dependencies from content to style (Table 1), which can be attributed to style information being partially predictable from the encoded content information in the causal setup. To verify that the encoders only depend on content information (i.c., that content is block-identifiede we assess the trained models using a novel, more rigorous empirical evaluation for the numerical simulation. We test the effect of interventions e - c', which perturb the content information af test time via batch-wise permutations of content, before generating x/ = [c',', s, m,) and x²- = (c',', 5, mi). Hence, we break the causal dependence between content and style (see illustration in Figure 8), which allows to better assess whether the trained encoders depend on content or style information. Specifically, we train the encoders for 3,000,000 iterations to ensure convergence and then train nonlinear regression models predict both the original and the intervened content variables from the learned representations. Figure 8 presents our results using the interventional setup, showing that in most cases only content information can be recovered. We observe an exception (underlined values) in the two cases with statistical dependencies, where some style information can be recovered, which expected because statistical dependencies reduce the effective dimensionality of content (cp. von Kligelgen et al., 2021). Analogously, in the case of statistical and causal dependencies, some of the original content information can be recovered via the encoded style information. In summary, the evaluation with interventions provides a more rigorous assessment of information can be recovered when the encoding size matches the true number of content dimensions.
in the causal setup, showing that neither style nor modality-speciff
High-dimensional image pairs with continuous latents To bridge the gap between continuous and discrete data, we provide additional experiment that offers a realistic setup but uses only continuous latent variables to satisfy the assumptions of Theorem 1. Previously, in Section 5.2, we used a complex multimodal datasct of image/text pairs, which were generated from a combination of continuous and discrete latent factors. Now, we consider a different dataset that consists of pairs digneeiimesiooall images generated from set of continuous latents, which is more in line with our theoretical assumptions. Note that datasets with pairs of images are common in practice, for example, in medical imaging where patients are assessed using multiple views (e.g.. images from different angles) or multiple modalities (e.g., as in PET-CT imaging). To generate the data, we adapt the code from 3DIdent (Zimmermann et al., 2021) to render pairs of images, for which the object position is always shared (i.e., content), the object-, spotlight- and background-colo is stochastically shared (i.e., style), and modality-specific factors are object rotation for one modality and spotlight position for the other. Additionally, we render the objects using different textures to simulate modality-speciff mixing process. Samples of image pairs are shown in Figure 9 and further details about the dataset can be found in Appendix B.I. We train the encoders with the InfoNCE objective for 60,000 iterations using the same architectures and hyperparameters for Multimodal3Dlden (Table 2b), and again evaluate the R² coefficient of determination using a kernel ridge regression that predicts the respective ground truth factor from the learned representations.
2
Figure 9: Examples of high-
dimensional image pairs.
21

---

Published as a conference paper at ICLR 2023
Object seposiiinn
Object x-pusitiun
prediction performance
0.8
Object position
z-pusition
prediction perf performance
0.8
Objoct y-positoon
Object z-puition
Object colo
Object solr
Spotight
Spotight color
0.6
Background
0.6
Background
Object rocation
Object rotation
rotation
rotation
0.4
0.4
0.2
0.2
0.0
10
12
2
10
12
encoding size
encoding size
(a) mutually independent factors
(b) With causal dependencies
Figure 10: Result with pairs of high-dimensional images. As function of the encoding size of the model, we assess the nonlinear prediction of ground truth factors to quantify how well the learned representation encodes the respective factors. Content factors are denoted in bold, style factors in italic, and modality-speciffe factors in regular font. Each point denotes the average R²² score across three seeds and bands show one standard deviation.
Figure 10 present results for the dataset of image pairs, showing the prediction performance as a function of the encoding size for the setting with causal dependencies (Figure 10b) and the setting with mutually independent latent variables (Figure 10a) respectively. In both settings, content information (i.e. object position) is recovered when sufficient encoding capacity is available. Style and mppelity- information, size. In Figure 10b we observe the recovery of some style information, which is expected because style be predicted some degree from the encoded content information when there is causal dependence of style on content. Overall, these findings lend further support to our theoretical result from Theorem as we investigate realistic setting with only continuous latent factors, which is more in line with our assumptions. Notably, the results appear more consistent with our theory, e.g., showing that less style and modality-specifie information is encoded, compared to our results for the image/text experiment, where we used combination of continuous and discrete latent factors.
the other hand, discarded independent of the encoding
Muldem with mutually independent factors For the results of the image/text experi- ment in the main text (Section 5.2) used the Mulimmodal3Dldent dataset, which designed such that object color is causally dependent on the x-position of the object to impose a causal dependence of style content. In Figure 11, we provide similar analysis using version of the dataset without the causal dependence, i.c., with mutually independent factors. For both modalities, we observe that object color is only encoded when the encoding size larger than four, i.c., when there is excess capacity beyond the capacity needed to encode all content factors. Hence, these results corroborate that contrastive learning can block-identify content factors in a complex multimodal setting with heterogeneous image/text pairs.
22

---

Published as a conference paper at ICLR 2023
1.0
1.0-
prediction per formancce
0.8
Object position
prediction per formance
0.8
Object position
Object shape
0.6
0.6
*
Ohyec mestion
rotation
0.4
Ohyer mestion
0.4
Spollight pmition
Object x-position
Spoiligt color
Object y-position
Backeyound
Object
Object
Text phrasing
10
12
encoding size
encoding size
(a) Prediction of image factors
(b) Prediction of text factors
Figure 11: Result on Midden with mutually independent factors. As function of the encoding size of the model, we assess the nonlinear prediction of ground truth image factors (left subplot) and text factors (right subplot) to quantify how well the learned representation encodes the respective factors. Content factors are denoted in bold and style factors in italic. Along the x-axis, we vary the encoding size, i.c., the output dimensionality of the model. We measure the prediction performance in terms of the R2 coefficient of determination for continuous factors and classification accuracy for discrete factors respectively. Each point denotes the average across three seeds and bands show standard deviation.
23


Reviews:
[{"review_id": "rlSkPAImeg", "reviewer": "Reviewer_LEuf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addressed the problem of identifiability for multimodal representation learning and showed that contrastive learning can block-identify latent factors shared between heterogenous modalities. The authors first prove this theorem based on some assumptions. Afterwards, some experimental results are offered to corroborate their theoretical results.", "summary_of_the_review": "In general, this article is well organized and reasonable, and the conclusion is helpful. But I still have some concerns, see Strength And Weaknesses.", "strength_and_weaknesses": "This paper proves that contrastive learning can block-identify latent factors shared between heterogenous modalities from both theoretical and experimental aspects. The authors describe their settings and experiments in a specific way, which is understandable.\nHowever, I still have some concerns about this paper. \n1) Theorem 1 assumes that the number of content variables is known or that it can be estimated. It seems satisfied in the settings of Multimodal3DIdent. But what about more complex natural datasets like COCO? Or some modalities beyond image and text? It needs more in-depth discussions, although the authors have mentioned in the limitation part. \n2) In the experimental part, the authors use Convnet for text encoding, which is the same as the convolutional network used for image encoder. Why not use more general and classical sequential neural network, like LSTM or GRU? They are more widely used in multimodal tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YG3hWT7T4gv", "reviewer": "Reviewer_YHeS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel identifiability analysis of a certain class of multimodal contrastive learning algorithms. The result is centered around a model with a latent variable partitioned into content (shared across modalities), style and and modality-specific information. The main theoretical result guarantees a form of block identifiability of the shared latent content. This work distinguishes itself by allowing for certain types of dependencies between the latent factors and by allowing for two distinct decoders, corresponding to the two modalities. The theory is illustrated in both a controlled synthetic dataset and a more realistic one presenting image/text pairs.   ", "summary_of_the_review": "I very much enjoyed reading this manuscript. It is well written, easy to follow and its topic is very timely. The identifiability results are novel and very clearly presented. I found the experiments with the realistic image/text data to be very convincing, although I was expecting more experiments where the assumptions of the theory are violated. The main drawback is that the theory applies only to continuous data and that one of the main application of multimodal contrastive learning is image/text data, but this point is acknowledge by the authors. Still, this remains an important contribution and I am very happy to recommend acceptance.", "strength_and_weaknesses": "Strengths:\n- This paper is very well written and fun to read (even on a Saturday morning).\n- The theoretical contribution is significant.\n- The technical assumptions are very clearly presented, a quality which is sometimes lacking in this line of work.\n- The topic is timely, given how popular multimodal contrastive learning is.\n- Presents a realistic experiments with image/text data.\n\nWeaknesses:\n- I would have appreciated more experiments where theoretical assumptions are violated, for example, what happens if Assumption 2 doesn't hold?\n- The theory applies only to continuous data (as is made clear by the assumption that both encoders f_1 and f_2 are diffeomorphisms), but the main applications of multimodal contrastive learning is with image/text data.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XBxdTeX0Tfc", "reviewer": "Reviewer_VsMH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work extends the contrastive learning based nonlinear mixture identification framework in (von Kugelgen et al. 2021) and considers a case where the two modalities have two different generative functions. The major contribution is the models’ identifiability analysis. The setting and the analysis can be considered as extensions of those in (von Kugelgen et al. 2021). A multimodal nonlinear mixture model is adopted, and a contrastive learning-based criterion is analyzed. The major claimed contribution lies in analyzing the case where two views have different generative nonlinear functions, but in (von Kugelgen et al 2021) assumed that the two views share the same generative nonlinear system.", "summary_of_the_review": "This is a well-written and well-organized paper. The major concern lies in its novelty relative to an existing work. This should be properly acknowledged and discussed, which would have made the contributions of this work clearer.\n", "strength_and_weaknesses": "Strength\n\n- The considered problem is interesting, and meaningful. Compared to the work of (von Kugelgen et al. 2021), the two different generative functions for two views, respectively, can be applied to a wider range of applications, e.g., text v.s. image type multiview data.\n\n- The paper is well-written.\n\nWeakness\n\n- Some technical assumptions are not easy to follow. Although the paper is in general pleasant to read, the part around Assumptions 1-2 are a bit hard to digest. Some more explanations and relating to physical meaning may help.\n\n- There work only established block identifiability of the shared components, but did not consider identifying the private components.\n\n- The work's identifiability result was established by assuming that infinite data samples are available, which is not realistic.\n\n**A major concern is that discussion regarding a very relevant existing work is missing**. The submission claims that a major contribution is that the work established block-identifiability of a multiview nonlinear mixture model when different modalities have different generating functions. However, the reviewer hopes to draw attention of the authors to the following paper:\n\n(Lyu et al 2022) Lyu, Qi, Xiao Fu, Weiran Wang, and Songtao Lu. \"Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective.\" In ICLR 2022\n\nThis recent work (Lyu et al 2022) has already established a similar identifiability result under a similar multiview model with different view-generating mechanisms. The work was not discussed in this submission, but it is very relevant: \n\nSome similarities are listed as follows:\n\n- Using this work’s notation, the generative model with x_1 = f_1(c,m_1) and x_2=f_2(c,m_2) was considered in (Lyu et al 2022), and the identifiability of c and m_1 and m_2 were established using a nonlinear multimodal learning criterion. This is quite similar to the generative model considered in this submission, where the latents can be split to shared and view-specific components.\n\n- The identifiability in both works were established via latent shard component matching with invertibility regularization. The ideas share the same spirit. \n\n- The criterion in (Lyu et al 2022) was also related to a multimodal contrastive learning criterion in Appendix I.2. This makes the learning criteria of the two works even more similar to each other.\n\nI hope to mention that some additional contributions in (Lyu et al 2022):\n\n- Unlike this paper that only considers infinite-sample cases, (Lyu et al 2022) also has finite-sample identifiability analysis.  \n\n- Unlike this submission that only considers block-identifying of the shared components, (Lyu et al 2022) also proved that the private components can be block-identified under some conditions.\n\nThe reviewer also acknowledges that there are some differences:\n\n- The model considered in this submission is slightly different, where the private components are split into s and m_i for i=1,2. But the model in (Lyu et al 2022) does not have the s part.\n- The assumptions used to establish identifiability are different in the two works.\n- The proof techniques are not the same.\n- Due to the introduction of s in this submission, the framework in this submission is perhaps more suitable for causal learning.\n\nGiven the similarity in terms of the generative models and the learning criteria in both works, the fact that (Lyu et al 2022) is completely missing from the bibliography of this submission is unfortunate. Proper discussions regarding the very relevant prior work (Lyu et al 2022) and existing block-identifiability results may make the contributions of this submission clearer.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "UodH7P90AqH", "reviewer": "Reviewer_akaD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies contrastive learning (e.g., in representation learning with multiview data or multimodal image/caption pairs). This paper presents new identifiability results for multimodal contrastive learning by extending recent work in identifiability results for multiview contrastive learning. Their main contribution is to extend the analysis of multiview non-linear ICA with a single shared latent variable to a multimodal case where it is possible to have a disentangled set of latent variables across modality-shared and modality-specific parts, which all jointly generate the observed multimodal data. Their main result shows that it is still possible to recover shared factors in this new setup, specifically that one can block-identify latent factors shared between modalities. They empirically verify these results on synthetic data and multimodal image/text paired data.", "summary_of_the_review": "The paper makes important contributions to the understanding of contrastive, multiview and multimodal learning, with important insights for future work backed up by theory and experiments, but there can be deeper empirical experiments and concluded insights to be more impactful.", "strength_and_weaknesses": "Strengths:\n1. The paper is well motivated and described. The ideas are clear and experiments are on several large multimodal datasets.\n2. The paper is largely clear with clear figures and exposition. It was a joy reading this paper.\n3. The paper makes important contributions to the understanding of contrastive, multiview and multimodal learning, with important insights for future work.\n\nWeaknesses:\n1. The synthetic and real data experiments should vary the contribution across modality-general style and content and modality-specific to better determine how one can identify the important modality-specific factors across different settings. For example does identifiability get harder with more modality-general information? or more modality-specific information?\n2. More real-world multimodal datasets can be experimented with, see https://arxiv.org/abs/2107.07502, especially datasets with a broad range of ratios in modality-general vs modality-specific information.\n3. The theory, while seems to be right and justified, is rather high-level, and does not delve into the details of learning and optimization (e.g., guarantees on how to learn such disentangled latent variables, or to encode different modalities). A lot of the proof follows by extending prior work in multiview ICA combined with the newly made assumptions in the paper, which is okay, but seems rather high-level. What if the assumptions are not made? What are some directions to further improve this line of theoretical study?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "45qqWeOavLS", "reviewer": "Reviewer_VZnd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a contrastive learning method combined with a generative process where the latent space involves the disentangled latent variables \"content\", \"style\" and \"domain-specific\" elements. The authors provide theoretical analysis using AlignUniform loss, a variant of the InfoNCE loss and demonstrate that contrastive learning in a multi-modality setting could block-identify the common latent attributes across domains like contents. Empirically, the authors conduct simulations with InfoNCE loss on both synthetic and Causal3DIdent data for numerical justifications. ", "summary_of_the_review": "The topic of this paper is interesting in the study of contrastive learning in a multi-modal case. The proposed method is well supported with both theoretical and empirical evidence. Some important perspectives regarding the proposed method are either not clarified or comprehensively studied at this moment. ", "strength_and_weaknesses": "**Strength**\n\nThe paper is well-written and organized, making it easy to follow. The proposed generative model is well presented and justified in a mathematical way. Combining the generative model with the contrastive framework is fairly novel in the ICA context. The generative process is reasonable and can be naturally combined with the contrastive framework. The theoretical analysis is well-defined with the given hypothesis. From the empirical perspective, the authors have shown their main experiments reasonably, and the results support the claims of the paper. The details included should be sufficient for reproducibility. \n\n**Weakness**\n\nAlthough the math analysis is clearly elaborated, the training of the proposed method is not clarified and I need to figure it out by reading their code. Also, after reading through the code, I found some details are omitted in the paper. For example, the generative function f and the contrastive encoder g need to be separately trained (please correct me if I understand it in the wrong way). Maybe adding a paragraph or an algorithm box will better clarify this part.\n\nThe practical training loss and the loss used for analysis are not the same. Although these two loss functions have a close correlation, there exists an approximation gap in terms of the number of samples used in the calculation. \n\nThe generative process also introduces several variables/hyper-parameters that need to handle. For example, the generative functions $f_1$ and $f_2$ need to be invertible; the dimensionality and covariances of the latent variables need to be carefully chosen. Some additional ablation studies might be needed to address these concerns. \n\nThe latent variables, even the whole framework, should be affected by the performance of the generative functions. The authors should provide either theoretical or empirical studies to instruct the impacts. For example, how does the contrastive encoder perform in terms of the reconstruction error of the generative models? \n\nFor practical use, since the framework has some assumptions which can hardly be satisfied in real-world cases, it is hard to evaluate the contributions from this perspective. The experiments are conducted at the toy level, and it is unclear what would happen in bigger and more complex data. Some relaxation or approximation are in need in the future.\n\nSome math notations are not fully explained. Please see the clarity part.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster