Towards a Unified Framework of Contrastive Learning for Disentangled Representations
Authors: Stefan Matthes
Zhiwei Han
Hao Shen
Venue: NeurIPS 2023 poster (2023)
Keywords: Disentanglement
Contrastive Learning
Identifiability
Representation Learning
Nonlinear ICA
TL;DR: We provide unified theoretical guarantees for disentanglement for a broader family of contrastive methods and prove identifiability of the true latents for four contrastive losses, without imposing common independence assumptions.

Abstract: Contrastive learning has recently emerged as a promising approach for learning data representations that discover and disentangle the explanatory factors of the data.
Previous analyses of such approaches have largely focused on individual contrastive losses, such as noise-contrastive estimation (NCE) and InfoNCE, and rely on specific assumptions about the data generating process.
This paper extends the theoretical guarantees for disentanglement to a broader family of contrastive methods, while also relaxing the assumptions about the data distribution.
Specifically, we prove identifiability of the true latents for four contrastive losses studied in this paper, without imposing common independence assumptions.
The theoretical findings are validated on several benchmark datasets.
Finally, practical limitations of these methods are also investigated.

Towards a Unified Framework of Contrastive Learning for Disentangled Representations
Stefan Matthes, Zhiwei Han, Hao Shen
fortiss GmbH, Munich, Germany (matthes, ,matthes,han, shen
Abstract
Contrastive learning has recently emerged promising approach for learning data representations that discover and disentangle the explanatory factors of the data. Previous analyses of such approaches have largely focused individual contrastive losses, such as noise-contrastive estimation (NCE) and InfoNCE, and rely on specific assumptions about the data generating process. This paper extends the theoretical guarantees for disentanglement to a broader family of contrastive methods, while also relaxing the assumptions about the data distribution. Specifi- cally, we prove identifiability of the true latents for four contrastive losses studied in this paper, without imposing common independence assumptions. The theoretical findings are validated on several benchmark datasets. Finally, practical limitations of these methods are also investigated.
1 Introduction
Learning to disentangle the explanatory factors of the observed data is valuable for a variety of ma- chine learning (ML) applications, as it allows for more compact, interpretable, and manipulable data representation. Consequently, this can improve sampling efficiency [46] and predictive performance [T] for downstream tasks, and promote faimess 33 Among the many efforts develop a theoretically grounded approach to learning such representations, contrastive learning (CL) has emerged particularly promising technique.
Intuitively, contrastive methods learn a function that maps observations into a representation space, such that related examples (e.g., augmentations of the same sample) are mapped close to each other and negative pairs are mapped far apart 66 48). recent years, this intuition has been refined and extended by different theories. One of these theories is that contrastive methods (approximately) invert the data generating process [49] 24] and thus recover the generative factors. We build upon this line of work and extend their theoretical findings to wider range of contrastive objectives.
In this study, we aim to develop unified framework of statistical priors on the data generating process to improve our understanding of CL for disentangled representations. Two important issues we will discuss are how to deal with nonuniform marginal distributions of the latent factors and situations when these factors are conditionally dependent to some degree. We demonstrate how this could be accomplished with a simple modification of common contrastive objectives. We also show that without these modifications, contrastive objectives make implicit assumptions about the data generating process and investigate when these assumptions are justified. We empirically verify our theoretical claims and show practical limitations of the proposed framework.
In short, the contributions of our study can be summarized as follows:
We extend and unify theoretical guarantees of disentanglement for a family of contrastive losses under relaxed assumptions about the data generating process.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The theoretical findings are empirically validated several benchmark datasets and we quantitatively compare the disentanglement performance of four contrastive losses.
We analyze the impact of partially violated assumptions and investigate practical limitations of the proposed framework.
2 Related Work
Disentangled Representation Learning The concept of learning representations whose components correspond to the explanatory factors of the data can be traced back to the principles of blind source separation [5] and discovering factorial codes [3]. In contrast to earlier work, however, the research focus has gradually shifted to applications where the relationship between data and underlying factors is nonlinear.
The difficulty of this task is that for independent and identically distributed (i.i.d.) data that depend nonlinearly on the latent factors, it is fundamentally impossible to identify these factors without labels or assumptions about the data generating process 23 34 26
few studies showed how the underlying generative factors can be recovered if they are mutually independent (with at most one of them being Gaussian) and their relation to the observed data exhibits certain regularities, such as a linear mixture followed by an element- wise nonlinearity [42). and local isometries [20]. Gresele et al. [15 also showed that conformal maps rule out some spurious solutions.
Most approaches, however, depart from the i.i.d. assumption and utilize co-observed dependent variables, such as a time index [21] prior observations [22] in time-series, observations obtained from interventions [3). data augmentation [47] or different views [14]. These approaches mainly differ in how they model the statistical and causal dependencies between the latent factors and their
relation to the auxiliary variable.
Hyvärinen et al. (24 modeled the distribution of the latent factors with the exponential family, where the key assumptions are that, for given auxiliary variable, the latent factors are conditionally independent and that the effect of the auxiliary variable on their distributions is sufficiently diverse. Their theory was later extended to the Variational Autoencoder (VAE (28) framework (26 and to energy-based models (27) with further relaxed independence assumptions. Other conditional distributions that have been considered are the Laplace distribution (29 and the von Hises-Fishte distribution [49]. In this work, we extend these results to distance-based conditional distributions and arbitrary marginals of the latents.
Another research direction considers pairs of observations whose underlying latents differ only in some of the factors and share the others, for example, through interventions actions. In 32 the shared factors or intervention targets are assumed to be known. Locatello et al. [36] showed how the ground-truth latents can be recovered when the pairs of observations share least one latent factor, which was extended to multi-dimensional factors in (10), but their approaches additionally require that the latent factors are mutually independent. In contrast, (30 4 proved identifiability for more general causal dependencies between the latent factors by exploiting sparse transitions.
Contrastive Learning In recent years, contrastive methods have shown remarkable ability to learn useful representations for downstream tasks 337 39
18 17 A number of works attribute this suc- cess primarily their tendency maximize mutual information (MI) between latent representations However, it has been observed that contrastive objectives with tighter MI bounds do
33
necessarily enhance downstream performance 45 44
More recently, by exploiting the alignment and uniformity properties (48) of InfoNCE [39], Zimmer- mann al. [49] showed that InfoNCE approximately inverts the data generating process, leading to the identification of the true latent factors. We extend this theory to a broader family of contrastive
methods 39 16 17 38)
3 Contrastive Learning for Disentangled Representations
In this section, we present our theoretical framework of contrastive methods for learning disentangled representations (Figure We begin with the formalization of our assumptions on the underlying data generating process.

---

(Dis-)similarity Measure
Generator
Encoder
Loss
&-NCE
OgB
8-INCE
g
fo
d
8- G-SCL
X
2
Ok.
6-NWJ
Figure I: Overview of our CL framework for disentanglement. The unknown generator maps latent pairs (s, p(s,S) S) to observation pairs (x,x), which are encoded by fo- After computing the dissimilarity measure according to Eq. (2), the representation is optimized with one of the contrastive losses.
3.1 Data Generating Process
In this work, we adopt the common assumption that the observations are generated by a smooth and invertible generator g S - X, where S S IR" denotes the space of latent factors and X C R the space of observations. Here, we assume n S m. The goal to find a function f: + IR" that inverts y up element-wise transformations or other simple mappings such as linear functions.
We do not impose any constraints on the ground-trufe -tuth marginal distribution of the latents, p(s), except that the support must be connected. Let ~ p(s) and (s, $) be a latent positive pair. We model the dependency between s and with the following conditional distribution
p(ss) ((s)
Q(5) d(s.s)
with ((s) / Q(5) (
d(s.8) ds,
where Q and Z are scalar-valued functions and d a distance function, which we will specify below. It is worth noticing that here the latent factors s; are not assumed to be statistically independent or conditionally independent given auxiliary observations (Q cannot necessarily be factorized).
Two related observations (e.g., consecutive frames in a sequence, multiple views augmentations of the same data point) are then generated by x = g(s) and = g()). For the sake of simplicity, we assume that the related instances are of the same modality, though it is straightforward to extend this framework different modalities, such as image and audio coming from different generators.
3.2 Contrastive Learning Approaches
Contrastive methods generally have in common that they employ certain mechanisms to measure
similarity between data points. The most frequent choice is the cosine similarity (6, while f, distances [7 and even learnable functions [2 24] have also been used. Inspired by the work in [22 24).
which relies on specific models of learnable similarity measures, we propose the following generic dissimilarity measure
8(z, = + azz) + a(2),
where a and â are learnable scalar- scalar-valued valued functions, e.g., neural networks, and is fixed expression that describes the interaction between related examples. the ideal case, d matches in Eq. (
Let us now recall some common contrastive methods. The first approach we investigate is noise- contrastive estimation (NCE) [16), which has been used for disentanglement (22 24]. Using our dissimilarity measure, the loss can be formulated as follows
La-NCe(f.)) (x.
log (x)(() f())))
E log [1 si((- ()),, x.x'<x
(3)
where sig the sigmoid function, Ppos is the distribution of positive pairs, and p is the distribution of all observations.
For simplicity. we assume here that both marginal distributions are the same. We consider the more general case of different marginals in the supplementary material.

---

Another popular choice is InfoNCE. which has shown remarkable success in self-supervised learning The loss function has several related forms and has been referred by different In the following, we use
39
19
18
names 25
39
6
31
-((((x))(())
6)
E - log
(4)
(xp (x]]
-((())))()) (((())))
{x)) 1m Ppp
where K the number of negative examples. Since LSANCE is invariant to changes a, we simply set (z) = O in this case.
HaoChen al. [1 introduced the concept spectral contrastive learning (SCL), which has classifi- cation guarantees for downstream tasks in the context of self-supervised learning. To fit this method into our theoretical framework, we use slightly modified version in our analysis, where we substitute f(x) f()) with --(f(())f(()) i.e.,
E (,,,c,.(,,) (x.R)
-2e -2- (((()) +
E
26(0(0))
xxx
Finally, we examine the Nguyen- "guyen-Wainright Wainright- (NWJ) objective using the same dissimilarity measure as before
Ca-Nw (f.8)
E (xpe)
f(f(x),
+
(((())(
(6)
X.X ~p
Note, that we here use it as a loss instead of a lower bound on MI. To the best of our knowledge, neither the SCL nor the NWJ objective have been employed learn disentangled representations or for ICA.
3.3 Identifiability
In this section, we derive precise conditions the data generating process under which the relation- ship between the learned representation and true latent factors can be described by a simple function. In line with (27], we speak of weak identifiability when this function is an affine mapping, and of strong identifiability when it is generalized permutation matrix
As in previous research [29 49], we assume that the interaction between and ş is (approximately) known, i.e., d in the dissimilarity measure as defined in Eq. 2 matches d in the conditional dis- tribution defined in Eq. ( Although this is strong restriction on the latent factors, satisfactory performance is still observed in practice, even when the assumption is violated. Note also that if they only match up to other simple transformations (e.g., invertible clement-wise) the true or learned latents, the recovered relation can be extended by just those transformations.
First, we study the case of weak identifiability. Eq. essentially states that the joint probability density function of positive pairs decreases exponentially with the distance between the underlying latent factors, distorted only by Q(5) and the marginal distribution p(s). Neither Q(S) nor p(s) depends on both instances of pair. This allows us to learn distance-preserviin representation while accumulating factors that depend exclusively on the first or second element in a and o, respectively. We formalize this in the following theorem.
Theorem (Weak identifiability). Let S C R" be open and connected, X' C R", and g: + X invertible and differentiable. Let further assume that the observed data satisfy the generative model given in Eq. (1). If d has one of the following properties:
(i) there exists a function ; IR+ + RR,, such that & o is a norm-iniee corm-induced metrid
(ii) d(s. s) = E. 52)). where each d, continuous and strictly increasing.
then the optimal estimator of any of the contrastive losses presented above identifies the true latent factors up affine transformations, i.e., h fo g an affine mapping.
This property sometimes called explicitness 44 informativeness by a simple function. A norm- norm-inndue induced metric is a metric which is translation invariant, d(x, y) = d(x + b, y b), and absolutely homogeneous, d(ax, oy) y)
9 when it is left open what is meant

---

The central idea of the proof, which can be found in the supplementary material, is to show that the learned representation preserves the distance from the latent space, i.., d(s. s) = d((((s), h (s)). Further, by constraining the dimension and connectivity of the latent space, such a mapping must be affine. As byproduct of the derivation, we obtain for the global optimum that
Ô o h = logp log Q
and, except for Lo K),
a h = log Z.
(8)
We verify this with simple example in Figure 2 This means that when we optimise the InfoNCE objective with ā set to zero, we implicitly assume that Q p. This is true, for example, when the marginal distribution is uniform and Q is constant. Thus, our theorem is consistent with the perspective of uniformity and alignment [48] and can be seen
generalization of Theorem from [49. In particular, we extend their results to other contrastive losses, nonconvex latent spaces, nonuniform marginal distributions, and even to the case where the latent factors are not conditionally independent.
Next we show strong identifiability when d in the conditional distribution of the latent factors is further restricted. The proof can be found in the supplementary material.
Theorem 2 (Strong identifiability). Assume that all conditions in Theorem aree satisfied. Let the function d from Eq. (1 be defined by
d(s,8) = > (1s,-5.1/03))
(9)
with 3 E (0, (2, oo) and o; > for all i, then h = is a generalized permutation matrix, i.e, composition of a permutation and element-wise scaling and sign flips.
This result is similar to earlier theorems 29 49 but relaxes some conditions. In particular, also holds for nonuniform marginals and / (0, 1). For the case 3 2, the conditional distribution quasi-Gaussin [22] and, as with linear Independent Component Analysis (ICA) for normally distributed sources, strong identifiability is then provably impossible.
In contrast to we show identifiability for the generalized normal distribution (excluding 3= =) 2) and do not assume some regularity conditions, e.g., that the encoder is twice differentiable and invertible. However, we do not claim to generalize their results, since the exponential family contains distributions that we do not consider here.
Theorem also has an interesting connection some recently proposed disentanglement methods that rely on sparsity 30 I This situation is similar to the case when 8 - 0 in Eq. (9) and the conditional p(š(s) becomes sparse. However, this is topic for future work.
4 Experiments
In this section, we conduct comprehensive quantitative evaluation of our theoretical framework. We largely adopt the experimental setup and evaluation protocol of 449 with one major difference. During training, we additionally optimize for a and ā in Eq. (2). both of which are parameterized by three-layer neural networks, respectively. In practice, both functions are normalized to a mean of zero and are jointly trained with a learnable offset c, i.e. 8(z, Z) = d(z, ž) + (z))) + ā()) + c. the case of 8-INCE, we use 8(z, 2) d(zz 2) + a(Z) instead. Further details on the implementation can be found in the supplementary material.
As in previous research [2 22 49). we test for weak identifiability. i.c., up to affine transformations, by fitting linear regression model between the ground-truth and recovered latents and compute the coefficient of determination (R²). For strong identifiability, i.e., up to generalized permutations and element-wise transformations, report the mean correlation coefficient (MCC).
4.1 Validation of Theoretical Findings
To validate our theoretical claims, we adopt a generative process similar to [49]. However, to demonstrate that identifiability can be achieved for nonconvex latent spaces, nonuniform marginals,
5

---

Table 1: Identifiability on synthetic data. MCC [%] mean = standard deviation over 2 random seeds
Scenario
8
ô-NCE
6-INCE
6-SCL
6-NWJ
Box (simple)
1/2
99.68 + 0.04
99.56 + 0.18
87.08 = 1.40
99.80 + 0,02
Box (simple)
99.91 0.01
99.90
94.33 = 2.02
99.87 0.01
Box (simple)
3
99.66 + 0.20
96.97
98.29 = 0.14
09.71
Box (simple)
5
99.79 = 0.02
96.56
98.63 = 0.36
99.74 = 0.00
Box (complex)
99.13 + 0,41
99.87 +: 0.05
83.73 + 2.49
93.14 + 4.97
Box (complex)
99.90 0.00
99.84 = 0.00
93.64 = 0.22
99.84 = 0.00
Hollow ball
99.73 0.11
99.70 0.08
90.27 6.48
99.73 + 0.08
Hollow ball
3
95.82 + 4.65
98.05 + 0.08
97.65 = 0.00
99.03 = 0.06
Hollow ball
5
98.52 + 0.21
96.25 0.43
97.57 = 0.06
98.67 = 0.16
Cube grid
99.87 + 0.02
99.66 + 0.04
96.92 = 1.88
99.79 + 0.01
Cube grid
97.42 0.12
82.26 + 7.06
96.17 0.07
97.96 + 0.06
and without conditionally independent latents, we investigate additional configurations of these components.
In this section, we consider latent spaces with n = 10 dimensions. We generate pairs of source signals (s, $) by first sampling from marginal distribution p(s) and subsequently from conditional distribution
Q(3) E. (1=))))) (1)
p()|s)
(10)
Z(s)
The observations (x, x) are then generated by passing and 5 individually through an (approximately) invertible multilayer perceptron (MLP).
We analyze the following four scenarios where we vary S, p(s), Q and A9:
Box (simple): S = [0, 1]", p(s) is uniform, Q is constant. This is our baseline.
Box (complex): S = [0, 1]", p(s) is a normal distribution with block-diagonal correlation matrix such that the odd dimensions are correlated with the even dimensions, Q has checkerboard pattern (i.e., has value on "white squares" and 0.1 on "black squares"), so the latent factors are not conditionally independent.
Hollow ball: S (s R" |
||s|| R2O < R) with inner radius and outer radius R. p(s) is uniform over the radius and angle (but not uniform in Fuclidean coordinates). Q is constant. Note that here & and Ky are not conditionally independent given because the support is not rectangular.
Cube grid: S = {s E []1, V < -bvs > 1,00 < 1) (i.e., S is disconnected), p(s) is uniform, Q is constant. Although our current analysis makes predictions about disconnected spaces, we investigate this configuration to empirically probe potential limits of our framework.
To fit our hardware setup, residual connections and batch normalization, which has been shown to be more stable. We generally leave the output space unrestricted, i.e., 2 = R
smaller batch size of 5120. Also,
encoder network with
Table provides summary of the MCC scores. The corresponding R² values can be found in the supplementary material. In each scenario, all contrastive losses achieve near-optimal disentanglement, with the exception of a few outliers. In particular, 8-SCL appears to numerically less stable than its counterparts, which we will examine in more detail in later sections. Additional experiments in the supplementary material compare 8-SCL with the original SCL and show that 5-SCL generally yields higher disentanglemest scores. However, we would like to point out that should not draw direct conclusions from these results about the performance of the original SCL on other downstream tasks.
To verify that the learned functions ē and converge to the solution given in Eq. 7 and Eq. 8 we construct a simple example where the ground truth is known. We choose 2 = 00 T
with uniform marginal distribution. The positive examples are sampled from truncated Laplace

---

G-NCE
6-INCE
G-SCL
G-NWJ
GT
10
-5
-50
De!
23 60
22 22
oc
24
0 86
07 566
22 g2
22 22
Figure 2: Learned functions a (top row) and á (bottom row) for a simple example described in the text. The ground truth is in the right column. In the case of 6-INCE,
zero.
Table 2: MCC [%] scores on synthetic data for a(z) ā(z)
Scenario
8
5-NCE
8-INCE
8-SCL
6-NWJ
Box (simple)
1/2
99.57
99.81
82.69
89.22
Box (simple)
1
99.85
99.91
94.00
99.81
Box (simple)
99.77
99.40
99.78
99.76
Box (simple)
5
99.82
98.86
94.84
99.82
Box (complex)
99.70
99.82
76.63
99.68
Box (complex)
99.69 3
99.75
89.78
99.51
Hollow ball
1
98.49
99.37
87.31
97.94
Hollow ball
98.59
97.23
96.39
98.64
Hollow ball
5
98.52
96.26
98.49
98.52
Cube grid 1
99.82
99.39
98.21
99.75
Cube grid
5
96.53
84.73
97.50
97.42
distribution. Figure 2 shows the results. Details and additional experimental setups can be found in the supplementary material.
Since learn the of and o, they only be identified up offset. In Figure 2 have added the learned bias to to confirm that the total offset is correct. All losses reach the global optimum except for 8-SCL, which appears be stuck at local minimum.
4.2 Evaluation under Partially Violated Assumptions
We also report results for the case where a and â are set learnable constant (see Table 2}. Interestingly, we do not observe that this change significantly affects the outcome. However, we do observe drastic performance drop 6-NCE, 8-SCL and 6-NWJ when this offset is fixed a value which is far from the ground truth. Specifically, setting the bias to zero causes drop in the MCC score by more than 20% on average when / 1. For larger we do not observe different behaviour. This is likely due to the fact that the optimal bias is far from O when 3 is smaller, which can distort the learned representation.
4.3 Evaluation on Kitti Masks
The KITTI Masks dataset 229 consists of segmentation masks of pedestrians extracted from the autonomous driving benchmark KITTI-MOTS (12). We compare the presented loss functions with

---

Table 3: KITTI Masks. MCC [%] mean + standard deviation over 10 random seeds
Laplace
Normal
At
Loss
Unbounded
Box
Unbounded
Box
0.05s
Slow VAE
29
66.1
4.5
8-contr
49
77.1 1.0
74.1 = 4.4
58.3 5,4
59.9 + 5.5
Ó-INCE
77.3 = 1.5
75.9 = 1.7
68.1 5.0
68.3 + 4.8
8-NCE
77.1 711110 1.0
7
60.2
68.0 1.5
5-SCI.
75.8 3.9
71.3 + 7.5
66.2 10.7
66.5 + 4.0
S-NWJ
77.3 0.7
77.7 + 0.9
67.5
69.2 5.3
0.15s
Slow 29
5.8
8-contr
49
1.9
80.9 = 3.8
60.2 8.7
68.4
8-INCE
1.4
77.0 3.3
6.2
68.4
6-NCE
79.3 2.3
79.5 - 2.7
62.7 4,6
68.4
8-SCL
72.9 + 3.9
74.4 2.6
62.5
66.1 5.7
S-NWJ
79.3 4: 2.0
78.2 + 6.1
68.2
72.6 + 6.7
Table 4: 3DIdent. R [%] and MCC [%] mean = standard deviation over 2 random seeds
R
MCC
8
Loss
Unbounded
Box
Unbounded
Box
2
8-contr
96.43 0.03
96.73 0.10
54.94 0.02
98.31 + 0.04
2
8-INCE
97.87 + 0.07
98.08 + 0.03
56.09
+ 0.10
98.98 + 0.09
2
8-NCE
98.05 0.04
97.91 0.02
55.34 0.11
98.95 = 0.08
2
8-SCL
71.07 = 3.76
74.92 + 2.28
49.88 + 1.20
76.02 = 3.13
2
8-NWJ
94.03 1.35
97.67 0.02
53.82 + 0.46
98.79 + 0.02
1
8-contr 49
96.87 + 0.08
98.38 + 0.03
8-INCE
98.03
0.06
97.45 + 0.04
55.79 + 0.09
98.63 + 0.05
S-NCE
97.83
0.04
97.33 + 0.03
56.35 + 0.08
98.71 + 0.02
1
5-SCL
78.59
3.92
93.13 + 1.83
52.28 + 1.98
95.77 + 2.58
1
8-NW
94.02 + 1.28
97.64 + 0.09
54.87 = 0.33
98.82 = 0.03
the current state of the for this dataset 29 49
Except for the additional parameters in use the same settings and follow their evaluation protocol.
As noted by Klindt et al. 229, the transitions of the latents mostly sparse. It is therefore not surprising that all contrastive losses consistently perform better when a Laplace conditional is assumed opposed normal conditional. Considering the large variance across different random seeds, all contrastive losses perform similarly well, with the largest outlier being 8-SCL, which could already be observed the synthetic dataset in the previous section.
Overall, there are only small differences between the different embedding spaces, with about half of the methods performing better each setup. Consistent with previous observations [49] see for all models slightly better performance for larger time intervals, At, between consecutive frames. One possible reason that a larger concentration of the conditional leads larger range of 8 values. causing numerical instability. We investigate this phenomenon in more detail in Section 4.5
4.4 Evaluation on 3DIdent
We additionally evaluate our framework under partially violated assumptions on 3DIdent 449 recently proposed benchmark dataset. The dataset is composed of rendered images showing complex object in different positions, rotations and under different lighting conditions. Positive pairs (s, 3) are obtained by randomly selecting s and matching sampled š' from the conditional p(š'(s) the closest with rendered image. We consider a normal (0 2) and Laplace conditional
(3 1). both cases we set d(z, 2) ||z 2||² where we either leave the output space unbounded
(Z IR") restrict it to adjustable box (Z = [0, 6)"h where b is a Iearnable parameter). The latter

---

100
100
100
MCC [N6]
80
80
80
AINCE
& NCE
NCC
60
6IICEE
60
6-SCL
5C5
6-CLL
6/NM)
6-NW)
6
40
40
40
20
30
40
50
100
20
100
200
100
100
100
80
[
G-NCE
G-NCE
G-NCE
&
60
6-INCE
60
G-INCE
60
G-INCE
6-SCL
6-SCL
6-NW)
G-NW]
6NNN)
40
40
40
10
20
30
40
10
20
50
100
200
10
20
50
100 200
Number of factors
Concentration (1/o) p())s) for p-1
Concentration (1/0) of p(š(s) p-3
Figure 3: MCC (top) and R scores (bottom) for different numbers of latent factors (left) and for different concentrations (1/or) of the conditional distribution for (center) and (right).
is achieved by applying a sigmoid function in the last layer of the network and multiplying the result by b. Table shows the results for weak and strong identifiability.
With the exception 6-SCL, examined contrastive losses have R2 values close to the theoretical optimum. However, 8-NWJ shows less robustness when we do not restrict the latent space compared to 8-NCE and S-INCE. This demonstrates that incorporating knowledge about the latent space stabilize training and lead to strong identifiability, even in the case of (truncated) Gaussian conditional distribution.
4.5 Investigation of Limitations
In this section, we discuss practical and theoretical limitations of the proposed framework. We first examine how changes in the number of latent factors and the concentration of the conditional distribution affect disentanglement. We use the same setup as in Section 4.1 except that we choose slightly smaller batch size 4096 and train for 8 10 iterations to ensure convergence when we consider more dimensions. The results are shown in Figure 3
First, we that all methods have difficulties when the number of latent factors is increased (see Figure 3l left column), In particular, d-SCL shows little robustness for larger dimensions.
that the performance of all methods declines with increasing concentration (1/o) of the conditional and is exacerbated by a larger shape parameter (3). The work in 49 noted that InfoNCE has difficulty disentangling the latent factors when the conditional distribution becomes flat relative to the marginal one, which, they pointed out, makes positive examples indistinguishable from negative ones. We observe here the same problem in the opposite case.
Furthermore,
Consider the following example. Suppose = [0, 1 and p(s) is uniform, then the range of ó at the global optimum [0, n//0". For larger and smaller o, this quickly leads vanishing values when calculating the exponential in the contrastive losses. This could also be one of the reasons why we see worse performance for smaller A7 in KITTI Masks.
It should be noted that employing Iearnable functions in the similarity measure may entail certain drawbacks, such as the need to store additional parameters and a slight increase in training time. Furthermore, have seen in Section 4.2 that in some cases the underlying data distribution can be adequately approximated by constant a and å. Although not observed in this work, it is also plausible that the inclusion of these functions for more complex datasets may increase the risk of getting stuck poor local minima during training, as incorrect shapes of these functions may promote poor representations (the shape of a and ā affects the gradient of f and vice versa).
Finally, we want to emphasize that the mechanism that governs the observation pairs is highly dependent the application and that the relationship between the latents may not be adequately

---

described by distance- bissence-based conditional as assumed in this work (Eq. (T). However, we leave the exploration of methods suitable for other mechanisms to future work.
Conclusion
In this work, propose theoretical framework of contrastive methods for learning disentangled representations. In contrast to previous research, our framework accounts for nonuniform marginal distributions of the factors of variation, allows for nonconvex latent spaces, and does not assume that these factors are statistically independent or conditionally independent given an auxiliary variable.
Our theory unveils that contrastive methods implicitly encode assumptions about the data generating process. We show empirically that even when these assumptions are partially violated, these methods learn to recover the true latent factors. This study provides further evidence that contrastive methods learn to approximately invert the underlying generative process, which may explain their remarkable success in many applications.
6 Acknowledgements
This work has been carried out as part of the AuSeSol-AT Project which is funded by Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (BMUV) under the Grant 67K121007F.
References
[1] K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. arXiv preprint arXiv:2206.01 101, 2022
[2] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information views. Advances in neural information processing xystems, 32, 2019.
[3] H. B. Barlow, T. P. Kaushal, and G. J. Mitchison. Finding minimum entropy codes. Neural Computation, I (3):412-423, 1989.
[4] J. Brehmer, P. De Haan, Lippe, and T. S. Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319-38331, 2022.
[5] J.FF. Cardoso. Blind signal separation: statistical principles. Proceedings of the IEEE, 86(10):0009-0055 1998.
[6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607, PMLR, 2020.
[7] T. Chen, C. Luo. and L. Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34:118344 11845, 2021.
E. Creager, D. Madras, J.-H. Jacobsen, M. Weis, K. Swersky, T. Pitassi, and R. Zemel. Flexibly fair representation learning by disentanglement. International conference on machine learning, pages 1436-1445. PMLR, 2019.
[9] C. Eastwood K. Williams. framework for the quantitative evaluation of disentangled representa- tions. In International Conference on Learning Representations, 2018.
[10] Fumero, Cosmo, S. Melzi, and E. Rodolà. Learning disentangled representations via product manifold projection. In International conference on machine learning, pages 3530-3540. PMLR, 2021.
[11] L. Gao, Q Mao, M. Dong, Y. Jing, and R. Chinnam, On learning disentangled representation for acoustic event detection. In Proceedings of the 27th ACM International Conference on Multimedia, pages 2006- 2014, 2019.
[12] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pages 3354-330 1361. IEEE, 2012.
[13] R. Geirhos, J.HH. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and E. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11))665-66) 663, 2020.
10

---

[14] L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty Artificial Intelligence, pages 217-227. PMLR, 2020.
[15] I.. Gresele, J. Von Kügelgeen, Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, new concept? Advances in neural information processing systems, 34:28233-28248, 2021.
[16] M. Gutmann and A. Hyvärinen. Noise-contrastive cstimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on arrificial intelligence and statisticx, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[17] HaoChen, C. Wei, A. Gaidon, and Ma. Provable guarantees self supervised deep learning with spectral contrastive loss, Advances in Neural Information Processing Systems, 34:5000-501 I, 2021.
[18] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation lcarning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[19] R. D. Hjelm, A. Fedorov, S. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:I808. 06670, 2018.
K. Grewal, P. Bachman, A. Trischler, and Y. Bengio.
[20] D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? Advances in Neural Information Processing Systems, 34:51 34:5150-5161, 2021.
[21] A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
[22] A. Hyvärinen and H. Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMIR, 2017.
[23] A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999.
[24] A. Hyvärinen, H. Sasaki, and R. E. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pages 859-868. PMIR, 2019.
[25] R. Jozefowicz. O. Vinyals, M. Schuster, N. Shazeer, and Wu. Exploring the limits of language modeling. arXiv preprint arXiv: 1602.00210, 2016.
[26] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-22111 2217. PMLR, 2020.
[27] I. Khemakhem, R. Monti, D. Kingma, and A. Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Syxtems, 33:12768-12278 2020.
[28] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[29] D. Klindt, L. Schott, Y. Sharma, I. Ustyuzhaninov, W. Brendel, M. Bethge, and D. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding, arXiv preprint XXi:22007.10930 2020.
[30] S. Lachapelle, P. Rodriguez, Y. Sharma, K. F. Everett, R. Le Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pages 428-484. PMI.R 2022.
[31] P. H. Le-Khac, G. Healy, and A. F. Smeaton. Contrastive representation leaming: A framework and review. leee Access, 1:193907-19393 2020.
[32] P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 33557-13600 PMLR, 2022.

---

[33] F. Locatello, G. Abbati. T. Rainforth, S. Bauer, B. Schölkopf. and O. Bachem. On the fairness of disentangled representations. Advances neural information processing systems. 32, 2019.
[34] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 4124. PMLR, 2019.
[35] F. Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf. and O. Bachem. Disentangling factors of variation using few labels. arXiv preprint arXiv: 1.XXi::905.000 00258,, 2019.
[36] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf. O. Bachem, and M. Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348- 6359. PMLR, 2020.
[37] A. Mnih and K. Kavukcuoglu. Learning word embeddings efficiently with conte-coe estimation. Advances in neural information processing systems, 26, 2013.
[38] X. Nguyen, M. Wainwright, and Jordan, Estimating divergence functionals and likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(1 1):5817-7778)1 2010.
[39] A. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 100.00377, 2018.
[40] K. Ridgeway and M. Mozer. Learning deep disentangled embeddings with the f-statistic loss, Advances in neural information processing systems, 31, 2018.
[41] R. Shu, Chen, A. Kumar. Ermon, and B. Poole. Weakly supervised disentanglement with guarantees. arXiv preprint arXiv:1910.0977,, 2019.
[42] A. Taleh and C. Jutten. Source separation post-nonlinear mixtures. JEEE Transactions on signal Processing, 47(10):2807-2820, 1999.
[43] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[44] Y. Tian, C. Sun, B. Poolc, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning? Advunces in neural information processing systems, 33:6827-6839, 2020.
[45] M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: 1907. 13625, 2019.
[46] S. Van Steenkiste, F. Locatello, J. Schmidhuber, and O. Bachem. Are disentangled representations helpful for abstract visual reasoning? Advances in Neural Information Processing Systems, 32, 2019.
[47] J. Von Kügelgenn Y. Sharma, I.. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and Locatello. Self- supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 14:11651-11667, 2021.
[48] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference on Machine Learning, pages 9929-99939 PMLR, 2020.
[49] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference Machine Learning, pages 12979-12990. PMLR, 2021.
12


Reviews:
[{"review_id": "6nBpXR3mOI", "reviewer": "Reviewer_6CFv", "summary": "Authors propose a generalized contrastive learning framework extending the guarantees for disentanglement learning. The paper also links contrastive learning objective with identifiability results in both week and strong identifiability setting.", "questions": "Here are some question that makes the paper bit unclear, addressing them can possibly improve the quality of the paper:\n\nQ1.  To recover true latent (to establish identifiability) you need contrastive examples varying in only one particular generating factor, which might not be always possible in reality, are there any tricks to circumvent this issue\n\nQ2. In case of theorem 1, I believe there are some strict constraints on how \\tilde{s} is constructed using s, these assumptions are not explicitly mentioned. (if there are no constraints, it's unclear how do you account for confounded generating factors)\n\nQ3. In case of confounded datasets, how would your identifiability result still hold? Since, in proof sketch you do not have any assumption over sufficiency and confounding factors.\n\nQ4. homeomorphism condition implicitly assumes the function to be bijective, continuous, and invertible which I believe is stronger assumption that [27] (correct me if I misunderstood the contribution).\n\nQ5.  The experiments are not clearly detailed: how is the data generated? what are the data generating factors?\n\nQ6. why is R2 value >> MCC? : as higher R2 indicates higher correlation which should result in higher MCC, but opposite effect is observed in your experiments, can you provide justification for this behavior?\n\nQ7. More details on training procedure would be helpful (how is parameter b learned?)\n\nQ8. How does the bound on features affect correlation values? (based on table 4 results bounded MCC >> unbounded MCC, what could be possible reason for this? is it an artifact of clipping values?)\n", "limitations": "Few technical limitations are mentioned in the paper, limitations of contrastive data acquisition in real-world setting is not discussed.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is mostly well written and easy to follow (in few places it is hard to digest)\n- Authors do great job in converting all different contrastive learning objectives with unified notation\n- contrastive learning was empirically studied for it's effectiveness for disentanglement learning, authors to a great job in formally introducing these concepts", "weaknesses": "- Some assumptions may be missing in case of thm1 (ref. questions for details)\n- Results on causal3DIndent are hard to interpret (ref. questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and advice on how to improve the quality of the work. We will include the mentioned limitation in the revised version and are happy to provide clarification on the raised issues.\n\nThe reviewer assessed the present work from the perspective of causal representation learning, however, there seems to be a misunderstanding regarding the modeling of the generative process. We do not explicitly model causal relationships between the latent factors in this paper. Instead, we assume that the latent factors of positive pairs are related according to Eq. 1. This allows for correlations between latent factors (no assumption on $p(s)$ except the support must be connected), and they need not be conditionally independent because $Q(\\tilde{s})$ does not necessarily factorize. Nevertheless, the connection to causal representation learning is an interesting research direction.\n\nQ1. In our approach, it is not necessary that only one generating factor changes between instances in a positive pair. In Theorem 1, the generating factors may also not be one-dimensional, such as rotations in 3D.\n\nQ2. In Theorem 1 we assume that \\tilde{s} depends on s according to Eq. 1. Roughly speaking, by considering pairs, the correlations between different latent factors cancel each other out.\n\nQ3. It is not clear to us how confounders would invalidate our theoretical results. If our previous answers cannot clarify this question, perhaps the reviewer is willing to elaborate on the question or provide an illustrative counterexample?\n\nQ4. The learned encoder in [27] is assumed to be a smooth homeomorphism with smooth inverse, i.e., a diffeomorphism (condition 5 in Theorem 1). We lift these assumptions and only require the encoder to be differentiable.\nUnder the stated assumptions in Theorem 1, we can show that the optimal encoder must also be invertible. It is thus not necessary to restrict the function space.\n\nQ5. The latent factors for 3DIdent are position, rotation, spotlight position and color of the object, spotlight and background  (3+3+1+3=10). For KITTI Masks they are the object position in the image and scale (2+1=3). In the other synthetic experiments the latent factors don’t have a particular meaning. Please also see the general response.\n\nQ6. We use R2 to measure weak identifiability (up to affine transformations) and MCC for strong identifiability (up to permutations and element-wise transformations). If, for example, the mapping from the true to the learned latents is a rotation by 45° around some axis, the R2 value will be 1 since there exists a linear map between them, but the MCC score will be much lower because the axes are not aligned.\n\nQ7. The scaling parameter $b$ is only learned when the output space $Z$ is restricted to a scalable box, i.e., $Z=[0,b]^n$. We achieve this by multiplying the output of a sigmoid in the last layer with b. In this case b is treated like any other learnable parameter in the neural network.\n\nQ8. When the true latent space is unbounded and the conditional of the generating factors is (quasi-)Gaussian or the chosen distance metric in the contrastive loss is Euclidean, we cannot achieve strong identifiability because the loss is then rotationally invariant. In this case we can identify the generating factors only up to affine transformations (high R2 scores). However, if the true latent space is a box, the conditional distribution is truncated and thus not rotationally invariant anymore. When we also restrict the learned latent space to a box these boxes align under certain conditions, leading to higher MCC scores. This is only possible through additional knowledge about the latent space.\n"}, {"review_id": "UUsDRiJK6R", "reviewer": "Reviewer_CnBd", "summary": "The paper develops a new theoretical framework for learning disentangled representations with contrastive methods. Building on the theory that contrastive methods approximately invert the data generating process, the paper extend it by proposing a new dissimilarity measure. The proposed framework supports a broader family of contrastive methods and relaxes the assumption about the data distribution and latent independence. Detailed theoretical proof is provided. Experiments validate the effectiveness of the proposed framework.\n", "questions": "- Could the author provide more intuition about the design of the dissimilarity measure (L106)? Are the optimal $\\alpha$ and $\\bar{\\alpha}$ function unique? Why jointly optimize them with the network help?\n- For a more complex dataset in real task, what are the considerations when designing $\\alpha$ and $\\bar{\\alpha}$ . Do you think a simple three-layer neural networks are also enough? Can you explain more on L287? What does \"incorrect shapes\" mean here?\n- It would be better if qualitative visualizations/comparison on more complex dataset (e.g. KITTI Masks or 3DIdent) are added to show the improvements.\n", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n\nThe paper is an extension of previous work [56] and does provide a new framework which can apply to more CL approaches and requires weaken assumption about both the data distribution and latent independence. I think the differences are obvious and the paper seems novel to me.\n\nQuality\n\nThe paper is well supported by theoretical analysis. Codes are also provided.\n\nClarity\n\nOverall the paper is well-organized.\n", "weaknesses": "The author should provide more intuition and high level explanation of their designs (details in Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation. We appreciate the nice summary of our research and the hints on where we can further expand the intuition behind the design choices. Please find below a detailed response to the questions raised.\n\nQ1. Yes, the optimal $\\alpha$ and $\\tilde{\\alpha}$ are unique up to an additive constant and depend only on the distribution of the generating factors (see Eq. 7/8). We add these learnable functions to the similarity measure to ensure that the analytical optimum is reachable. If these functions are not correct, the learned representation could potentially be distorted to compensate for these errors. However, our experiments in Section 4.2 show that even when they do not match the ground truth, the latent factors can still be disentangled.\n\nQ2. More complex marginals (e.g. higher dimensions, stronger correlations) require networks with higher capacity to be learned correctly. A learnable bias (or zero for InfoNCE) corresponds to uniform marginals and is therefore often a reasonable choice in practice. Although this was not observed in our experiments, the addition of more complex $\\alpha$ and $\\tilde{\\alpha}$ can potentially introduce poor local minima, for example if they are initialized far from the ground truth and thus negatively affect the learned representation. This is what we mean with “incorrect shapes”. We speculate that this, the higher memory consumption, and the little gain when compared to assuming uniform marginals are the reasons why not more sophisticated similarity measures have found adoption in practice.\nOur personal advice is to start with a learnable constant and introduce more complex functions (e.g. NNs) in a fine-tuning stage. We would also like to point out that inspecting the learned \\alpha and \\tilde{\\alpha} can yield further insights into the distribution of the unknown latents.\n\nQ3. In our experiments we haven’t observed a clear improvement on disentanglement for the investigated datasets, but also no decreased disentanglement scores. This indicates that the optimal encoder in the constrained function space ($\\alpha = \\tilde{\\alpha} = c$) is very close to the optimal encoder in the unconstrained space. However, to fully understand how the optimal encoder is affected, further research is needed.\n"}, {"review_id": "GLqh4Mxy21", "reviewer": "Reviewer_iM6P", "summary": "This paper proves two identifiability theorems for a broad family of contrastive learning methods,  without imposing the commonly used independence constraint. The theoretical findings are validated on several benchmark datasets, and the theoretical and practical limitations are fully discussed.", "questions": "1. why $\\beta=2$ is excluded in the conditions of Theorem 2?", "limitations": " The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The considered contrastive model family is broad enough to include several widely-used contrastive learning losses, in which some are never employed for disentanglement (SCL and NWJ objectives). This provides the paper with adequate novelty. \n2. The proofs of the theorems are enlightening for me. In my opinion, the techniques used in the proofs will push this field forward.\n3. The theoretical and practical limitations of the theory and the considered models are fully discussed, which is appreciated because the limitations of a work are of great meaning to readers.  ", "weaknesses": "1. If the paper can discuss the physical or geometric picture of the conditions and theory, it will be much better for readers to understand the theoretical results.\n2. The are some typos in the paper, e.g. Equation (23) on Page 4 of the Appendix misses a right bracket.\n3. Some concepts are used in the paper but not introduced, like 'semi-metric' in Equation (1) and 'norm-induced metric' in Theorem 1. I will suggest the authors briefly introduce these mathematical concepts in the footnotes, which will enhance the readability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive assessment. We will discuss the intuition behind the theory in more detail in the revised version and add footnotes on the used mathematical concepts.\nWe also thank the reviewer for pointing out typos.\n\n**Why β=2 is excluded in the conditions of Theorem 2?**\n\nExcluding β=2 is analogous to the assumption of non-Gaussian sources in linear ICA and is also equivalent to the exclusion of quasi-Gaussian sources in [25]. Without further knowledge about the shape of the latent space, the marginals, or the generator, strong identifiability is in this case provably impossible.\n\n**References**\n\n[25] Nonlinear ica of temporally dependent stationary sources\n"}, {"review_id": "qTbr4sdwMl", "reviewer": "Reviewer_TFiH", "summary": "This paper studies a unified framework of contrastive learning. The authors extend the theoretical result of [1] to embrace more contrastive losses, adapt it to marginal distributions beyond uniform distribution, and further relax some conditions that are required in [1]. The experimental results support the theoretical claim, and the authors also study/discuss the limitation of the theoretical framework.\n\n[1]: Contrastive Learning Inverts the Data Generating Process", "questions": "-- Regarding batch size, why the authors use 5120 in line 205, and 4096 in line 268?\n\n-- In this paper, for all the experiments, are users using the original SCL or the modified SCL (line 118-121).\n\n-- It is interesting to the authors conduct ablation studies on the number of latent factors, I have a few questions:\n\n1. What scenario are you using for this experiment? \n2. Compare with section 4.1, are you sampling more data? Or, as the dimension increases, are you also exponentially increase the data?", "limitations": "No negative impact found", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the story telling is good: The authors first come to a theoretical guarantees under some relaxed assumptions, then empirically validate the theoretical findings, and further analyze the impact of the violated assumptions, and discussed the limitations. Compare with previous close works, there are two advantages of this work, include:\n\n1. A unified framework instead of focusing on InfoNCE family, and 2. More relaxed conditions of the theorems. \n\nIn order to justify their theorem, the experiments on the synthetic datasets are designed to embrace more contrastive losses, and many scenarios are designed for non convex latent spaces, nonuniform marginals and without conditionally independent assumptions on latents. \n\nThe studies on the limitations of the framework also bring some insights for future research.\n", "weaknesses": "This paper has a lot of similarities with [1] (the 56th reference in the draft) in the foundation. Though this work does relax the theoretical guarantee of [1], the authors of [1] also empirically show that the underlying factors of variations can be identified even if the theoretical assumptions are severely violated. In Table 2 of this work, the authors also try to examine their theoretical framework under partially violated assumptions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and are glad that the reviewer recognizes the novelty and benefit of our work.\n\nQ1. In Section 4.5 (line 268), we analyzed the impact of larger dimensions on disentanglement performance, which requires more GPU memory. We decided to reduce the batch size to accommodate larger dimensions in memory.\n\nQ2. We used the modified SCL loss throughout the entire paper. Please also see the general response for more details.\n\nQ3.1. The generative process in Section 4.5 is the same as “Box (simple)” in Section 4.1. It differs only in the particular parameter analyzed, for example, the number of dimensions.\n\nQ3.2. In Section 4.1, 4.2 and 4.5 we have basically an infinite amount of data since we sample in each epoch new data from the “unknown” generator. We will make this clearer in the text. For a study of the impact of the number of data in a similar context, we would like to refer to [25].\n\n**References**\n\n[25]: Nonlinear ica of temporally dependent stationary sources\n"}, {"review_id": "5XT9JE0UYs", "reviewer": "Reviewer_m4mq", "summary": "This paper presents an analysis focusing on the disentanglement of the characteristics of internal representations obtained by contrastive learning. The aim is to recover the internal representation of the true latent space up to some transformation by placing certain assumptions on the data generation process. Particularly, it generalizes Zimmerman's theoretical analysis so that it can be applied to some loss function variants and relaxes the data distribution assumptions. The paper successfully derives the results of identifiability without the common assumption of independence in the research of disentangled representation.", "questions": "- **In the numerical experiments using synthetic datasets, why were the values of $\\beta$ not set to be compositional for each scenario?** For example, $\\beta=1/2$ tested in the Box (simple) scenario does not seem to be tested in other scenarios like Hollow ball and Cube grid. Specifically, for the Cube grid scenario, only this condition seems to skip the intermediate value of $\\beta=3$. Is there a clear reason for this? The validity of the parameter range used in the experiments should be clearly indicated.\n- **Sampling of positive pairs seems to be a crucial assumption in theory, but I could not glean from the text what assumptions were made.** In the Weakness section above, I have raise some concerns based on my understanding that positive pairs are sampled from a probability distribution derived from the conditional distribution of the true latent variables defined in Equation 1. Is this understanding correct? If this interpretation is wrong, it would be helpful to explicitly mention in the text about the assumption of the probability distribution of the positive pair samples.", "limitations": "This paper analyzes the acquisition of disentangled representation using contrastive learning from the viewpoint of identifiability, successfully relaxing some of the assumptions required in previous research. This is a theoretically solid result, and the results have been verified in numerical experiments using several benchmark datasets. However, as mentioned at the beginning of the Weakness section, there are significant concerns about applying the claims of this paper to practical situations. By appropriately discussing the practical validity of the assumed premises, I believe the significance of this paper will become more firmly established.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "While I could not fully follow the entire proof, the theoretical claim appears technically sound and solid. Including the analysis of the identifiability of Nonlinear ICA, the theoretical contributions obtained in this study are meaningful for the community focusing on acquiring disentangled representation with theoretical guarantees. Significantly, they have removed the independence assumption typically used in the analysis of disentangled representation. As it is difficult to sample all disentanglement factors in real-world datasets uniformly, this is an important direction for constructing practical theories. However, there are already several preceding studies on the approach of removing the independence assumption, so the positioning of this paper should be properly discussed, which I will mention in the Weakness section.", "weaknesses": "**There are certain concerns about the gap between theoretical claims and realistic experimental settings.** The fundamental theoretical claims in this paper are based on the conditional distribution in the true latent variable space defined in Equation 1. As far as I understand the claims in the paper, the identifiability can be guaranteed by training with the contrastive learning objective by assuming that the positive pair samples can be obtained in the form following this latent conditional distribution. However, it is questionable how much validity can be claimed to assume that positive pair samples can be obtained in such a distribution form when considering practical situations. If the true joint distribution of positive pairs is considered known, supervised learning should be done by simply generating labels from the distribution, so the significance should be in the setting where samples can be implicitly obtained while the explicit form of distribution is unknown. This concern relates to the practical applicability of the theoretical claims concluded in the paper and is important in positioning the contributions and significance of this paper. For example, SimCLR, widely known for its practical effectiveness, uses two different views augmented from data as a positive pair, but can this be justified from the perspective of Equation 1? How meaningful is it to consider the corresponding latent variable space?\n\n**It is necessary to show the contributions of existing related studies appropriately.** While the paper claims to have successfully derived the results of identifiability without the typical independence assumption in the research of disentangled representation, there are already preceding studies that theoretically guarantee disentanglement without imposing independence assumption using weak supervision [1, 2]. In particular, while [1] adopts a GAN-based approach for training the model, it strongly relates to the problem setting of this study, as it uses pairs that share a part of the disentangled factors as supervision signals. What advantages can be recognized in the preceding studies compared to these approaches?\n\n**A fair comparison is needed between contrastive objectives.** In the paper, a slightly different objective function is adopted from the original spectral contrastive learning (SCL) objective in order to fit into the theoretical framework of the proposed method (Equation 5). Some numerical experiments have pointed out the numerical instability of SCL, but could this be due to the influence of modifying the original objective function? It should be verified whether the same problem occurs in the objective function presented in the original paper. If the numerical instability pointed out in this paper is due to the modification introduced, it should be explicitly mentioned, as it could potentially mislead readers about the effectiveness of SCL.\n\n**References**\n\n1. Shu, R., Chen, Y., Kumar, A., Ermon, S., & Poole, B. (2019, September). Weakly Supervised Disentanglement with Guarantees. In *International Conference on Learning Representations*.\n2. Locatello, F., Poole, B., Rätsch, G., Schölkopf, B., Bachem, O., & Tschannen, M. (2020, November). Weakly-supervised disentanglement without compromises. In *International Conference on Machine Learning* (pp. 6348-6359). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed assessment and for pointing out where further comparison and discussion may be needed.\n\n**Gap between theoretical claims and realistic experimental settings**\n\nIt is true that the assumed conditional (Eq. 1) is not fully consistent with data augmentation, and it is useful to point this out in the paper, but this is far from the only practical application.\nWe would also like to point out that there is a difference between knowing the distribution of observed pairs and latent pairs. The latter does not allow labels to be generated and is sometimes even partially known. This may motivate, for example, a partial labeling.\n\n**Contributions of existing related studies**\n\nWe will add a discussion on weakly supervised disentanglement methods in the revised version. Both references mentioned have similarities with the limit case of Eq. 1, when $\\beta \\to 0$ leads to an l0-\"norm\" or sparsity prior (i.e., only some latent factors differ in a positive pair). Unlike [1, 2], Eq. 1 also allows for dense transitions, but in the sparse case our approach may not be as numerically stable as methods designed specifically for this case. We think further research is needed to fully understand the theoretical relationships and practical issues.\n\n**A fair comparison is needed between contrastive objectives**\n\nTo address this point we have conducted further experiments with the original SCL loss. Please see the general response for details.\n\n**Why not compositional $\\beta$ values?**\n\nWe fully agree and understand the reviewer's concerns. The set of chosen $\\beta$ values were mainly due to computational constraints available for preparing the submission. Our experiments suggest that convergence in terms of elapsed time can vary significantly, depending on the support of data, distributions, loss functions, hyper-parameters, etc. Therefore, in order to draw conclusive statements from the experiments to support the theoretical findings with a fixed number of iterations, e.g. 3e5, we only reported the results with almost converged experiments, which we believe to be sufficient to support the theoretical claim about the identifiability of nonlinear disentangled representations.\n\nIn the last week, we ran additional experiments for different scenarios. Here are the results after 3e5 iterations for “cube grid” with $\\beta=3$:\n\nMCC: $\\delta$-NCE: 98.62 | $\\delta$-INCE: 68.33 | $\\delta$-SCL: 93.27 | $\\delta$-NWJ: 98.55\n\nR2:  $\\delta$-NCE: 98.34 | $\\delta$-INCE: 92.07 | $\\delta$-SCL: 93.06 | $\\delta$-NWJ: 98.33\n\nand “box (complex)” with $\\beta=5$:\n\nMCC: $\\delta$-NCE: 92.08 | $\\delta$-INCE: 99.92 | $\\delta$-SCL: 92.90 | $\\delta$-NWJ: 99.91\n\nR2: $\\delta$-NCE: 98.21 | $\\delta$-INCE: 99.87 | $\\delta$-SCL: 97.34 | $\\delta$-NWJ: 99.89\n\nIn the cases where the scores are a little lower, the optimization has not yet completely converged. To illustrate this, we have included a figure showing the convergence of $\\delta$-INCE for various $\\beta$ values on “cube grid”. Also the experiments with $\\beta<1$ take much longer than $\\beta>=1$ due to division by zero checks when computing the gradients.\n\n**Sampling positive pairs**\n\nThe understanding is correct. In the revised version, we will make the sampling procedure clearer.\n"}, {"review_id": "xVdzSSkiiM", "reviewer": "Reviewer_bVRB", "summary": "In this paper the authors propose a new dissimilarity measure function for contrastive representation learning. They provide the identifiability of the factors of variations in four different contrastive objectives. Specifically, they describe the connection between the learned representations and true factors using some function in the data generating process. They extend the property to cases where there are non-convex latent spaces or/and non-uniform marginal distributions. Empirically they validate their approach on several datasets.", "questions": "Please see my questions in the previous section.", "limitations": "I think there is no potential negative societal impact of their work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is clearly written; The authors make it clear what the position of their approach in the literature of contrastive learning. The similarity measure function is simple to implement. I find it useful to discuss the strong and weak identifiability under different conditions, and the derivation is mathematically sound in my opinion. Though contrastive learning have been studies for a few years, it is interesting to investigate and unify the theoretical guarantees of disentanglement for a family of contrastive objectives under different assumptions.", "weaknesses": "1. I am a bit confused about the problem definition. While the authors try to model the relationship between true factors and the learned representations, in most cases we don't really know the true factor. Instead what we typically do in contrastive learning is to measure the similarity between pairs of observations where there are slight differences from certain perspectives, i.e. one image is the rotated version of the other. So I would hope that the authors can talk a bit more about how the identifiability works in practical scenarios.\n\n2. I wonder whether the authors need to any data augmentations when training with the contrastive loss. There does not seem to be any discussion on that.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that our contribution is valued, and we are grateful for the comments on where we can provide further elaboration.\n\n**Identifiability in practical scenarios**\n\nThe scenario described by the reviewer comes closest to our experiments with 3DIdent. However, instead of using data augmentation, the positive pairs can be considered as coming from successive frames of a video (same for KITTI Masks) where the underlying factors (e.g., position, rotation) change with some probability. Data augmentation behaves similarly, but does not allow, for example, to rotate or translate an object in all directions.\nOur theory assumes that such a mechanism for creating positive pairs changes the latent factors according to Eq. 1. This equation says that the factors we want to disentangle must at least sometimes change. This property is fundamental in disentanglement theory and is sometimes referred to as variability. It is important to note that it generally does not apply to pure data augmentation. To our knowledge, separating changing from non-changing factors is the best that can be provably achieved without supervision in the case where some factors never change [1].\n\n**Do we use data augmentation?**\n\nWe do not use data augmentation in this study. Rather, we leave it open how positive pairs are chosen for the contrastive loss, as this is highly dependent on the application and data augmentation may not be an option or the only option.\n\n**References**\n\n[1] Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style"}]

Author rebuttal:
We thank all reviewers for their constructive feedback and overall positive evaluation. Below we describe the main concerns and how we intend to address them. Additional points are discussed in the individual responses to the reviewers.

**Confusion about the generative process**

We will add a clearer description of the generative process in the revised version. We generate positive pairs by first sampling $s$ from a marginal distribution $p(s)$ and then $\tilde{s}$ from a conditional distribution according to Eq. 1. The observations are then obtained by transforming the true latents with an invertible generator. The latent distribution and generator are typically unknown in practice.

**More intuition behind Theorems 1 & 2**

Our main assumption (Eq. 1) essentially states that the probability of observing a positive pair decreases exponentially with the distance between the corresponding latents, "contaminated" only by $Q(\tilde{s})$ and the marginal $p(s)$. Neither $Q(\tilde{s})$ nor $p(s)$ depends on both instances of the pair. This allows us to learn a representation that preserves this distance metric together with $p(s)$ and $Q(\tilde{s})$. In this way we can identify the latents up to isometries. By further restricting the dimension and connectivity of the space, such an isometry must be affine and even a generalized permutation matrix for certain metrics.

**Transferability of theory to practice**

There are several ways to create positive pairs for contrastive learning: data augmentation, adjacent points in sequences, multiple views or modalities, weak supervision, and more. Each of these mechanisms can be described in terms of the underlying latent distribution. Our theoretical findings provide identifiability guarantees of the generative factors for specific forms of this mechanism (Eq. 1). Although this equation applies only to some practical scenarios, we believe it contributes to our general understanding. The study of other forms is an interesting future research direction.

**Comparison between original and modified SCL**

Throughout the paper, we use our modified version of SCL instead of the original version. This modified version sometimes fails to identify the latent factors due to numerical instabilities that the original SCL does not have (e.g., less oscillation during training). These instabilities are indeed due to the additional exponential term. However, in further experiments (which we will include in the appendix), the original SCL performs worse in the selected scenarios based on R2 and MCC. This does not mean, however, that these learned representations are generally less useful, only that the relationship between the true and inferred latents is not affine.


Meta-review: The paper is well-written and easy to follow, with the main contribution (the unified framework) being accessible and relevant. The reviewers positively called out the proof technique, which they found interesting and likely to spark more contributions. Some concerns were raised about the gaps in assumptions between theory and practice. I would recommend the authors to more clearly point this out in the paper.
Decision: Accept (poster)