Title: Towards a Unified Framework of Contrastive Learning for Disentangled Representations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Disentanglement

Contrastive Learning

Identifiability

Representation Learning

Nonlinear ICA
Abstract: Contrastive learning has recently emerged as a promising approach for learning data representations that discover and disentangle the explanatory factors of the data.
Previous analyses of such approaches have largely focused on individual contrastive losses, such as noise-contrastive estimation (NCE) and InfoNCE, and rely on specific assumptions about the data generating process.
This paper extends the theoretical guarantees for disentanglement to a broader family of contrastive methods, while also relaxing the assumptions about the data distribution.
Specifically, we prove identifiability of the true latents for four contrastive losses studied in this paper, without imposing common independence assumptions.
The theoretical findings are validated on several benchmark datasets.
Finally, practical limitations of these methods are also investigated.

Towards a Unified Framework of Contrastive Learning for Disentangled Representations

Abstract

Contrastive learning has recently emerged as a promising approach for learning data representations that discover and disentangle the explanatory factors of the data. Previous analyses of such approaches have largely focused on individual contrastive losses, such as noise-contrastive estimation (NCE) and InfoNCE, and rely on specific assumptions about the data generating process. This paper extends the theoretical guarantees for disentanglement to a broader family of contrastive methods, while also relaxing the assumptions about the data distribution. Specifically, we prove identifiability of the true latents for four contrastive losses studied in this paper, without imposing common independence assumptions. The theoretical findings are validated on several benchmark datasets. Finally, practical limitations of these methods are also investigated.

1 Introduction

Learning to disentangle the explanatory factors of the observed data is valuable for a variety of machine learning (ML) applications, as it allows for a more compact, interpretable, and manipulable data representation. Consequently, this can improve sampling efficiency [46] and predictive performance [35, 11, 13] for downstream tasks, and promote fairness [33, 8]. Among the many efforts to develop a theoretically grounded approach to learning such representations, contrastive learning (CL) has emerged as a particularly promising technique.

Intuitively, contrastive methods learn a function that maps observations into a representation space, such that related examples (e.g., augmentations of the same sample) are mapped close to each other and negative pairs are mapped far apart [6, 48]. In recent years, this intuition has been refined and extended by different theories. One of these theories is that contrastive methods (approximately) invert the data generating process [49, 24] and thus recover the generative factors. We build upon this line of work and extend their theoretical findings to a wider range of contrastive objectives.

In this study, we aim to develop a unified framework of statistical priors on the data generating process to improve our understanding of CL for disentangled representations. Two important issues we will discuss are how to deal with nonuniform marginal distributions of the latent factors and situations when these factors are conditionally dependent to some degree. We demonstrate how this could be accomplished with a simple modification of common contrastive objectives. We also show that without these modifications, contrastive objectives make implicit assumptions about the data generating process and investigate when these assumptions are justified. We empirically verify our theoretical claims and show practical limitations of the proposed framework.

In short, the contributions of our study can be summarized as follows:

We extend and unify theoretical guarantees of disentanglement for a family of contrastive losses under relaxed assumptions about the data generating process.

The theoretical findings are empirically validated on several benchmark datasets and we quantitatively compare the disentanglement performance of four contrastive losses.

We analyze the impact of partially violated assumptions and investigate practical limitations of the proposed framework.

2 Related Work

Disentangled Representation Learning The concept of learning representations whose components correspond to the explanatory factors of the data can be traced back to the principles of blind source separation [5] and discovering factorial codes [3]. In contrast to earlier work, however, the research focus has gradually shifted to applications where the relationship between data and underlying factors is nonlinear.

The difficulty of this task is that for independent and identically distributed (i.i.d.) data that depend nonlinearly on the latent factors, it is fundamentally impossible to identify these factors without labels or assumptions about the data generating process [23, 34, 26].

A few studies showed how the underlying generative factors can be recovered if they are mutually independent (with at most one of them being Gaussian) and their relation to the observed data exhibits certain regularities, such as a linear mixture followed by an element-wise nonlinearity [42], and local isometries [20]. Gresele et al. [15] also showed that conformal maps rule out some spurious solutions.

Most approaches, however, depart from the i.i.d. assumption and utilize co-observed dependent variables, such as a time index [21] or prior observations [22] in a time-series, observations obtained from interventions [36], data augmentation [47] or different views [14]. These approaches mainly differ in how they model the statistical and causal dependencies between the latent factors and their relation to the auxiliary variable.

Hyvärinen et al. [24] modeled the distribution of the latent factors with the exponential family, where the key assumptions are that, for a given auxiliary variable, the latent factors are conditionally independent and that the effect of the auxiliary variable on their distributions is sufficiently diverse. Their theory was later extended to the Variational Autoencoder (VAE [28]) framework [26] and to energy-based models [27] with further relaxed independence assumptions. Other conditional distributions that have been considered are the Laplace distribution [29] and the von Mises-Fisher distribution [49]. In this work, we extend these results to distance-based conditional distributions and arbitrary marginals of the latents.

Another research direction considers pairs of observations whose underlying latents differ only in some of the factors and share the others, for example, through interventions or actions. In [41, 32] the shared factors or intervention targets are assumed to be known. Locatello et al. [36] showed how the ground-truth latents can be recovered when the pairs of observations share at least one latent factor, which was extended to multi-dimensional factors in [10], but their approaches additionally require that the latent factors are mutually independent. In contrast, [30, 1, 4] proved identifiability for more general causal dependencies between the latent factors by exploiting sparse transitions.

Contrastive Learning In recent years, contrastive methods have shown a remarkable ability to learn useful representations for downstream tasks [37, 39, 6, 18, 17]. A number of works attribute this success primarily to their tendency to maximize mutual information (MI) between latent representations [39, 19, 2, 43]. However, it has been observed that contrastive objectives with tighter MI bounds do not necessarily enhance downstream performance [45, 44].

More recently, by exploiting the alignment and uniformity properties [48] of InfoNCE [39], Zimmermann et al. [49] showed that InfoNCE approximately inverts the data generating process, leading to the identification of the true latent factors. We extend this theory to a broader family of contrastive methods [39, 16, 17, 38].

3 Contrastive Learning for Disentangled Representations

In this section, we present our theoretical framework of contrastive methods for learning disentangled representations (Figure 1). We begin with the formalization of our assumptions on the underlying data generating process.

Generator

Encoder

(Dis-)similarity

Measure

Loss

3.1 Data Generating Process

In this work, we adopt the common assumption that the observations are generated by a smooth and invertible generator g : S → X , where S ⊆ R n denotes the space of latent factors and X ⊆ R m the space of observations. Here, we assume n ≤ m . The goal is to find a function f : X → Z ⊆ R n that inverts g up to element-wise transformations or other simple mappings such as linear functions.

We do not impose any constraints on the ground-truth marginal distribution of the latents, p ( s ) , except that the support must be connected. Let s ∼ p ( s ) and ( s , ˜ s ) be a latent positive pair. We model the dependency between s and ˜ s with the following conditional distribution

where Q and Z are scalar-valued functions and d is a distance function, which we will specify below. It is worth noticing that here the latent factors s i are not assumed to be statistically independent or conditionally independent given auxiliary observations ( Q cannot necessarily be factorized).

Two related observations (e.g., consecutive frames in a sequence, multiple views or augmentations of the same data point) are then generated by x = g ( s ) and ˜ x = g (˜ s ) . For the sake of simplicity, we assume that the related instances are of the same modality, though it is straightforward to extend this framework to different modalities, such as image and audio coming from different generators.

3.2 Contrastive Learning Approaches

Contrastive methods generally have in common that they employ certain mechanisms to measure similarity between data points. The most frequent choice is the cosine similarity [6], while ℓ p distances [7, 49] and even learnable functions [22, 24] have also been used. Inspired by the work in [22, 24], which relies on specific models of learnable similarity measures, we propose the following generic dissimilarity measure

where α and ˜ α are learnable scalar-valued functions, e.g., neural networks, and ˆ d is a fixed expression that describes the interaction between related examples. In the ideal case, ˆ d matches d in Eq. (1).

Let us now recall some common contrastive methods. The first approach we investigate is noisecontrastive estimation (NCE) [16], which has been used for disentanglement in [22, 24]. Using our dissimilarity measure, the loss can be formulated as follows

where sig is the sigmoid function, p pos is the distribution of positive pairs, and p is the distribution of all observations. 1

Another popular choice is InfoNCE, which has shown remarkable success in self-supervised learning [39, 6, 19, 18]. The loss function has several related forms and has been referred to by different names [25, 39, 6, 31]. In the following, we use

where K ≥ 1 is the number of negative examples. Since L δ -INCE is invariant to changes in α , we simply set α ( z ) = 0 in this case.

HaoChen et al. [17] introduced the concept of spectral contrastive learning (SCL), which has classification guarantees for downstream tasks in the context of self-supervised learning. To fit this method into our theoretical framework, we use a slightly modified version in our analysis, where we substitute f ( x ) T f (˜ x ) with e -δ ( f ( x ) ,f (˜ x )) , i.e.,

Finally, we examine the Nguyen-Wainright-Jordan (NWJ) objective [38] using the same dissimilarity measure as before

Note, that we here use it as a loss instead of a lower bound on MI. To the best of our knowledge, neither the SCL nor the NWJ objective have been employed to learn disentangled representations or for ICA.

3.3 Identifiability

In this section, we derive precise conditions on the data generating process under which the relationship between the learned representation and true latent factors can be described by a simple function. In line with [27], we speak of weak identifiability when this function is an affine mapping, and of strong identifiability when it is a generalized permutation matrix. 2

As in previous research [29, 49], we assume that the interaction between s and ˜ s is (approximately) known, i.e., ˆ d in the dissimilarity measure as defined in Eq. (2) matches d in the conditional distribution defined in Eq. (1). Although this is a strong restriction on the latent factors, satisfactory performance is still observed in practice, even when the assumption is violated. Note also that if they only match up to other simple transformations (e.g., invertible element-wise) of the true or learned latents, the recovered relation can be extended by just those transformations.

First, we study the case of weak identifiability. Eq. (1) essentially states that the joint probability density function of positive pairs decreases exponentially with the distance between the underlying latent factors, distorted only by Q (˜ s ) and the marginal distribution p ( s ) . Neither Q (˜ s ) nor p ( s ) depends on both instances of a pair. This allows us to learn a distance-preserving representation while accumulating factors that depend exclusively on the first or second element in α and ˜ α , respectively. We formalize this in the following theorem.

Theorem 1 (Weak identifiability) . Let S ⊆ R n be open and connected, X ⊆ R m , and g : S → X invertible and differentiable. Let us further assume that the observed data satisfy the generative model given in Eq. (1) . If d = ˆ d has one of the following properties:

(i) there exists a function ξ : R + → R + , such that ξ ◦ d is a norm-induced metric 3 ,

(ii) d ( s , ˜ s ) = ∑ i d i ( | s i -˜ s i | ) , where each d i is continuous and strictly increasing,

then the optimal estimator of any of the contrastive losses presented above identifies the true latent factors up to affine transformations, i.e., h = f ◦ g is an affine mapping.

The central idea of the proof, which can be found in the supplementary material, is to show that the learned representation preserves the distance from the latent space, i.e., d ( s , ˜ s ) = d ( h ( s ) , h (˜ s )) . Further, by constraining the dimension and connectivity of the latent space, such a mapping must be affine. As a byproduct of the derivation, we obtain for the global optimum that

We verify this with a simple example in Figure 2. This means that when we optimise the InfoNCE objective with ˜ α set to zero, we implicitly assume that Q = p . This is true, for example, when the marginal distribution is uniform and Q is constant. Thus, our theorem is consistent with the perspective of uniformity and alignment [48] and can be seen as a generalization of Theorem 5 from [49]. In particular, we extend their results to other contrastive losses, nonconvex latent spaces, nonuniform marginal distributions, and even to the case where the latent factors are not conditionally independent.

Next we show strong identifiability when d in the conditional distribution of the latent factors is further restricted. The proof can be found in the supplementary material.

Theorem 2 (Strong identifiability) . Assume that all conditions in Theorem 1 are satisfied. Let the function d from Eq. (1) be defined by

with β ∈ (0 , 2) ∪ (2 , ∞ ) and σ i > 0 for all i , then h = f ◦ g is a generalized permutation matrix, i.e., a composition of a permutation and element-wise scaling and sign flips.

This result is similar to earlier theorems [29, 49], but relaxes some conditions. In particular, it also holds for nonuniform marginals and β ∈ (0 , 1) . For the case β = 2 , the conditional distribution is quasi-Gaussian [22] and, as with linear Independent Component Analysis (ICA) for normally distributed sources, strong identifiability is then provably impossible.

In contrast to [24], we show identifiability for the generalized normal distribution (excluding β = 2 ) and do not assume some regularity conditions, e.g., that the encoder is twice differentiable and invertible. However, we do not claim to generalize their results, since the exponential family contains distributions that we do not consider here.

Theorem 2 also has an interesting connection to some recently proposed disentanglement methods that rely on sparsity [30, 1]. This situation is similar to the case when β → 0 in Eq. (9) and the conditional p (˜ s | s ) becomes sparse. However, this is a topic for future work.

4 Experiments

In this section, we conduct a comprehensive quantitative evaluation of our theoretical framework. We largely adopt the experimental setup and evaluation protocol of [49] with one major difference. During training, we additionally optimize for α and ˜ α in Eq. (2), both of which are parameterized by three-layer neural networks, respectively. In practice, both functions are normalized to a mean of zero and are jointly trained with a learnable offset c , i.e., δ ( z , ˜ z ) = ˆ d ( z , ˜ z ) + α ( z ) + ˜ α (˜ z ) + c . In the case of δ -INCE, we use δ ( z , ˜ z ) = ˆ d ( z , ˜ z ) + ˜ α (˜ z ) instead. Further details on the implementation can be found in the supplementary material.

As in previous research [21, 22, 49], we test for weak identifiability, i.e., up to affine transformations, by fitting a linear regression model between the ground-truth and recovered latents and compute the coefficient of determination ( R 2 ). For strong identifiability, i.e., up to generalized permutations and element-wise transformations, we report the mean correlation coefficient (MCC).

4.1 Validation of Theoretical Findings

To validate our theoretical claims, we adopt a generative process similar to [49]. However, to demonstrate that identifiability can be achieved for nonconvex latent spaces, nonuniform marginals, and, except for L δ -INCE ( f, δ ; K ) ,

and without conditionally independent latents, we investigate additional configurations of these components.

In this section, we consider latent spaces with n = 10 dimensions. We generate pairs of source signals ( s , ˜ s ) by first sampling from a marginal distribution p ( s ) and subsequently from a conditional distribution

The observations ( x , ˜ x ) are then generated by passing s and ˜ s individually through an (approximately) invertible multilayer perceptron (MLP).

We analyze the following four scenarios where we vary S , p ( s ) , Q and β :

Box (simple): S = [0 , 1] n , p ( s ) is uniform, Q is constant. This is our baseline.

Box (complex): S = [0 , 1] n , p ( s ) is a normal distribution with block-diagonal correlation matrix such that the odd dimensions are correlated with the even dimensions, Q has a checkerboard pattern (i.e., has value 1 on 'white squares' and 0.1 on 'black squares'), so the latent factors are not conditionally independent.

Hollow ball: S = { s ∈ R n | r < ∥ s ∥ < R, 0 < r < R } with inner radius r and outer radius R , p ( s ) is uniform over the radius and angle (but not uniform in Euclidean coordinates), Q is constant. Note that here ˜ s i and ˜ s j are not conditionally independent given s , because the support is not rectangular.

Cube grid: S = { s ∈ [ -1 , 1] n | ∀ i : s i < -b ∨ s i > b, 0 < b < 1 } (i.e., S is disconnected), p ( s ) is uniform, Q is constant. Although our current analysis makes no predictions about disconnected spaces, we investigate this configuration to empirically probe potential limits of our framework.

To fit our hardware setup, we use a smaller batch size of 5120. Also, we use an encoder network with residual connections and batch normalization, which has been shown to be more stable. We generally leave the output space unrestricted, i.e., Z = R n .

Table 1 provides a summary of the MCC scores. The corresponding R 2 values can be found in the supplementary material. In each scenario, all contrastive losses achieve near-optimal disentanglement, with the exception of a few outliers. In particular, δ -SCL appears to be numerically less stable than its counterparts, which we will examine in more detail in later sections. Additional experiments in the supplementary material compare δ -SCL with the original SCL and show that δ -SCL generally yields higher disentanglement scores. However, we would like to point out that one should not draw direct conclusions from these results about the performance of the original SCL on other downstream tasks.

To verify that the learned functions ˜ α and α converge to the solution given in Eq. (7) and Eq. (8), we construct a simple example where the ground truth is known. We choose S = Z = [0 , 1] 2 with a uniform marginal distribution. The positive examples are sampled from a truncated Laplace

6-NCE

6-INCE

6-SCL

6-NWJ

GT

5.4

5.2

5.0

4.8

-0.02

0.00

0.04

0.02

5.2

5.0

8't-

4.6

5.2

5.4

5.0

4.8

5.0

5.2

5.4

5.6

4.8

5.6

0.04

5.4

5.6

5.8

0.8

1

0.8

1

0.8

1

0.8

1

0.8

1

0.6

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0.0

0.2

0.2

0.2

0.2

0.2

0.4

0.4

0.4

0.4

0.4

-0.6

0.6

0.6

0.6

0.6

0.8

1

1

1

1

1

0.6

0.8

00.20.4 0.6 0.8

0.6

0.8

00.20.40.6 0.8

0.6

0.8

00.20.40.6 0.8 1

0.6

0.8

0.6

0.8

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0

0

0

distribution. Figure 2 shows the results. Details and additional experimental setups can be found in the supplementary material.

Since we learn the sum of α and ˜ α , they can only be identified up to some offset. In Figure 2, we have added the learned bias to α to confirm that the total offset is correct. All losses reach the global optimum except for δ -SCL, which appears to be stuck at a local minimum.

4.2 Evaluation under Partially Violated Assumptions

We also report results for the case where α and ˜ α are set to a learnable constant (see Table 2). Interestingly, we do not observe that this change significantly affects the outcome. However, we do observe a drastic performance drop in δ -NCE, δ -SCL and δ -NWJ when this offset is fixed to a value which is far from the ground truth. Specifically, setting the bias to zero causes a drop in the MCC score by more than 20% on average when β = 1 . For larger β we do not observe different behaviour. This is likely due to the fact that the optimal bias is far from 0 when β is smaller, which can distort the learned representation.

4.3 Evaluation on Kitti Masks

The KITTI Masks dataset [29] consists of segmentation masks of pedestrians extracted from the autonomous driving benchmark KITTI-MOTS [12]. We compare the presented loss functions with

the current state of the art for this dataset [29, 49]. Except for the additional parameters in δ , we use the same settings and follow their evaluation protocol.

As noted by Klindt et al. [29], the transitions of the latents are mostly sparse. It is therefore not surprising that all contrastive losses consistently perform better when a Laplace conditional is assumed as opposed to a normal conditional. Considering the large variance across different random seeds, all contrastive losses perform similarly well, with the largest outlier being δ -SCL, which could already be observed on the synthetic dataset in the previous section.

Overall, there are only small differences between the different embedding spaces, with about half of the methods performing better on each setup. Consistent with previous observations [49], we see for all models slightly better performance for larger time intervals, ∆ t , between consecutive frames. One possible reason is that a larger concentration of the conditional leads to a larger range of δ values, causing numerical instability. We investigate this phenomenon in more detail in Section 4.5.

4.4 Evaluation on 3DIdent

We additionally evaluate our framework under partially violated assumptions on 3DIdent [49], a recently proposed benchmark dataset. The dataset is composed of rendered images showing a complex object in different positions, rotations and under different lighting conditions. Positive pairs ( s , ˜ s ) are obtained by randomly selecting s and matching sampled ˜ s ′ from the conditional p (˜ s ′ | s ) to the closest ˜ s with a rendered image. We consider a normal ( β = 2 ) and a Laplace conditional ( β = 1 ). In both cases we set ˆ d ( z , ˜ z ) = ∥ z -˜ z ∥ 2 2 where we either leave the output space unbounded ( Z = R n ) or restrict it to an adjustable box ( Z = [0 , b ] n , where b is a learnable parameter). The latter

100

100

100

[%]

80

80

80

1

6-NCE

1

6-NCE

1

6-NCE

M

60

+1

6-INCE

60

+1

6-INCE

60

+1

6-INCE

6-SCL

1

6-SCL

1

6-SCL

6-NWJ

6-NWJ

+

6-NWJ

40

40

40

0

10

20

30

40

10

20

50

100

200

10

20

50

100

200

100

100

100

80

80

80

[%]

召

1

6-NCE

1

6-NCE

6-NCE

60

+1

6-INCE

60

11

6-INCE

60

+1

6-INCE

6-SCL

6-SCL

1

6-SCL

6-NWJ

6-NWJ

6-NWJ

40

40

40

0

10

20

30

40

10

20

50

100

200

10

20

50

100

200

Numberoffactors

Concentration (1/a) of p(s|s) for β=1

Concentration(1/a)of p(s|s)forβ=3

is achieved by applying a sigmoid function in the last layer of the network and multiplying the result by b . Table 4 shows the results for weak and strong identifiability.

With the exception of δ -SCL, the examined contrastive losses have R 2 values close to the theoretical optimum. However, δ -NWJ shows less robustness when we do not restrict the latent space compared to δ -NCE and δ -INCE. This demonstrates that incorporating knowledge about the latent space can stabilize training and lead to strong identifiability, even in the case of a (truncated) Gaussian conditional distribution.

4.5 Investigation of Limitations

In this section, we discuss practical and theoretical limitations of the proposed framework. We first examine how changes in the number of latent factors and the concentration of the conditional distribution affect disentanglement. We use the same setup as in Section 4.1, except that we choose a slightly smaller batch size of 4096 and train for 8 × 10 5 iterations to ensure convergence when we consider more dimensions. The results are shown in Figure 3.

First, we note that all methods have difficulties when the number of latent factors is increased (see Figure 3, left column). In particular, δ -SCL shows little robustness for larger dimensions.

Furthermore, we see that the performance of all methods declines with increasing concentration ( 1 /σ ) of the conditional and is exacerbated by a larger shape parameter ( β ). The work in [49] noted that InfoNCE has difficulty disentangling the latent factors when the conditional distribution becomes too flat relative to the marginal one, which, they pointed out, makes positive examples indistinguishable from negative ones. We observe here the same problem in the opposite case.

Consider the following example. Suppose S = [0 , 1] n and p ( s ) is uniform, then the range of δ at the global optimum is [0 , n/σ β ] . For larger β and smaller σ , this quickly leads to vanishing values when calculating the exponential in the contrastive losses. This could also be one of the reasons why we see worse performance for smaller ∆ t in KITTI Masks.

It should be noted that employing learnable functions in the similarity measure may entail certain drawbacks, such as the need to store additional parameters and a slight increase in training time. Furthermore, we have seen in Section 4.2 that in some cases the underlying data distribution can be adequately approximated by constant α and ˜ α . Although not observed in this work, it is also plausible that the inclusion of these functions for more complex datasets may increase the risk of getting stuck in poor local minima during training, as incorrect shapes of these functions may promote poor representations (the shape of α and ˜ α affects the gradient of f and vice versa).

Finally, we want to emphasize that the mechanism that governs the observation pairs is highly dependent on the application and that the relationship between the latents may not be adequately

described by a distance-based conditional as assumed in this work (Eq. (1)). However, we leave the exploration of methods suitable for other mechanisms to future work.

5 Conclusion

In this work, we propose a new theoretical framework of contrastive methods for learning disentangled representations. In contrast to previous research, our framework accounts for nonuniform marginal distributions of the factors of variation, allows for nonconvex latent spaces, and does not assume that these factors are statistically independent or conditionally independent given an auxiliary variable.

Our theory unveils that contrastive methods implicitly encode assumptions about the data generating process. We show empirically that even when these assumptions are partially violated, these methods learn to recover the true latent factors. This study provides further evidence that contrastive methods learn to approximately invert the underlying generative process, which may explain their remarkable success in many applications.

References

K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. arXiv preprint arXiv:2206.01101 , 2022.

P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems , 32, 2019.

H. B. Barlow, T. P. Kaushal, and G. J. Mitchison. Finding minimum entropy codes. Neural Computation , 1 (3):412-423, 1989.

J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems , 35:38319-38331, 2022.

J.-F. Cardoso. Blind signal separation: statistical principles. Proceedings of the IEEE , 86(10):2009-2025, 1998.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

T. Chen, C. Luo, and L. Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems , 34:11834-11845, 2021.

E. Creager, D. Madras, J.-H. Jacobsen, M. Weis, K. Swersky, T. Pitassi, and R. Zemel. Flexibly fair representation learning by disentanglement. In International conference on machine learning , pages 1436-1445. PMLR, 2019.

C. Eastwood and C. K. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations , 2018.

M. Fumero, L. Cosmo, S. Melzi, and E. Rodolà. Learning disentangled representations via product manifold projection. In International conference on machine learning , pages 3530-3540. PMLR, 2021.

L. Gao, Q. Mao, M. Dong, Y. Jing, and R. Chinnam. On learning disentangled representation for acoustic event detection. In Proceedings of the 27th ACM International Conference on Multimedia , pages 20062014, 2019.

A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition , pages 3354-3361. IEEE, 2012.

R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence , pages 217-227. PMLR, 2020.

L. Gresele, J. V on Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, a new concept? Advances in neural information processing systems , 34:28233-28248, 2021.

M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 297-304. JMLR Workshop and Conference Proceedings, 2010.

J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems , 34:5000-5011, 2021.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018.

D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? Advances in Neural Information Processing Systems , 34:5150-5161, 2021.

A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems , 29, 2016.

A. Hyvärinen and H. Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics , pages 460-469. PMLR, 2017.

A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks , 12(3):429-439, 1999.

A. Hyvärinen, H. Sasaki, and R. E. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 859-868. PMLR, 2019.

R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 , 2016.

I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020.

I. Khemakhem, R. Monti, D. Kingma, and A. Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems , 33:12768-12778, 2020.

D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

D. Klindt, L. Schott, Y. Sharma, I. Ustyuzhaninov, W. Brendel, M. Bethge, and D. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.10930 , 2020.

S. Lachapelle, P. Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning , pages 428-484. PMLR, 2022.

P. H. Le-Khac, G. Healy, and A. F. Smeaton. Contrastive representation learning: A framework and review. Ieee Access , 8:193907-193934, 2020.

P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning , pages 13557-13603. PMLR, 2022.

F. Locatello, G. Abbati, T. Rainforth, S. Bauer, B. Schölkopf, and O. Bachem. On the fairness of disentangled representations. Advances in neural information processing systems , 32, 2019.

F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning , pages 4114-4124. PMLR, 2019.

F. Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf, and O. Bachem. Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.01258 , 2019.

F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning , pages 63486359. PMLR, 2020.

A. Mnih and K. Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. Advances in neural information processing systems , 26, 2013.

X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory , 56(11):5847-5861, 2010.

A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

K. Ridgeway and M. C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. Advances in neural information processing systems , 31, 2018.

R. Shu, Y. Chen, A. Kumar, S. Ermon, and B. Poole. Weakly supervised disentanglement with guarantees. arXiv preprint arXiv:1910.09772 , 2019.

A. Taleb and C. Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on signal Processing , 47(10):2807-2820, 1999.

Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 776-794. Springer, 2020.

Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning? Advances in neural information processing systems , 33:6827-6839, 2020.

M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625 , 2019.

S. Van Steenkiste, F. Locatello, J. Schmidhuber, and O. Bachem. Are disentangled representations helpful for abstract visual reasoning? Advances in Neural Information Processing Systems , 32, 2019.

J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Selfsupervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems , 34:16451-16467, 2021.

T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pages 9929-9939. PMLR, 2020.

R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning , pages 12979-12990. PMLR, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors propose a new dissimilarity measure function for contrastive representation learning. They provide the identifiability of the factors of variations in four different contrastive objectives. Specifically, they describe the connection between the learned representations and true factors using some function in the data generating process. They extend the property to cases where there are non-convex latent spaces or/and non-uniform marginal distributions. Empirically they validate their approach on several datasets.\n\nSTRENGTHS:\nThis paper is clearly written; The authors make it clear what the position of their approach in the literature of contrastive learning. The similarity measure function is simple to implement. I find it useful to discuss the strong and weak identifiability under different conditions, and the derivation is mathematically sound in my opinion. Though contrastive learning have been studies for a few years, it is interesting to investigate and unify the theoretical guarantees of disentanglement for a family of contrastive objectives under different assumptions.\n\nWEAKNESSES:\n1. I am a bit confused about the problem definition. While the authors try to model the relationship between true factors and the learned representations, in most cases we don't really know the true factor. Instead what we typically do in contrastive learning is to measure the similarity between pairs of observations where there are slight differences from certain perspectives, i.e. one image is the rotated version of the other. So I would hope that the authors can talk a bit more about how the identifiability works in practical scenarios.\n\n2. I wonder whether the authors need to any data augmentations when training with the contrastive loss. There does not seem to be any discussion on that.\n\nQUESTIONS:\nPlease see my questions in the previous section.\n\nLIMITATIONS:\nI think there is no potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an analysis focusing on the disentanglement of the characteristics of internal representations obtained by contrastive learning. The aim is to recover the internal representation of the true latent space up to some transformation by placing certain assumptions on the data generation process. Particularly, it generalizes Zimmerman's theoretical analysis so that it can be applied to some loss function variants and relaxes the data distribution assumptions. The paper successfully derives the results of identifiability without the common assumption of independence in the research of disentangled representation.\n\nSTRENGTHS:\nWhile I could not fully follow the entire proof, the theoretical claim appears technically sound and solid. Including the analysis of the identifiability of Nonlinear ICA, the theoretical contributions obtained in this study are meaningful for the community focusing on acquiring disentangled representation with theoretical guarantees. Significantly, they have removed the independence assumption typically used in the analysis of disentangled representation. As it is difficult to sample all disentanglement factors in real-world datasets uniformly, this is an important direction for constructing practical theories. However, there are already several preceding studies on the approach of removing the independence assumption, so the positioning of this paper should be properly discussed, which I will mention in the Weakness section.\n\nWEAKNESSES:\n**There are certain concerns about the gap between theoretical claims and realistic experimental settings.** The fundamental theoretical claims in this paper are based on the conditional distribution in the true latent variable space defined in Equation 1. As far as I understand the claims in the paper, the identifiability can be guaranteed by training with the contrastive learning objective by assuming that the positive pair samples can be obtained in the form following this latent conditional distribution. However, it is questionable how much validity can be claimed to assume that positive pair samples can be obtained in such a distribution form when considering practical situations. If the true joint distribution of positive pairs is considered known, supervised learning should be done by simply generating labels from the distribution, so the significance should be in the setting where samples can be implicitly obtained while the explicit form of distribution is unknown. This concern relates to the practical applicability of the theoretical claims concluded in the paper and is important in positioning the contributions and significance of this paper. For example, SimCLR, widely known for its practical effectiveness, uses two different views augmented from data as a positive pair, but can this be justified from the perspective of Equation 1? How meaningful is it to consider the corresponding latent variable space?\n\n**It is necessary to show the contributions of existing related studies appropriately.** While the paper claims to have successfully derived the results of identifiability without the typical independence assumption in the research of disentangled representation, there are already preceding studies that theoretically guarantee disentanglement without imposing independence assumption using weak supervision [1, 2]. In particular, while [1] adopts a GAN-based approach for training the model, it strongly relates to the problem setting of this study, as it uses pairs that share a part of the disentangled factors as supervision signals. What advantages can be recognized in the preceding studies compared to these approaches?\n\n**A fair comparison is needed between contrastive objectives.** In the paper, a slightly different objective function is adopted from the original spectral contrastive learning (SCL) objective in order to fit into the theoretical framework of the proposed method (Equation 5). Some numerical experiments have pointed out the numerical instability of SCL, but could this be due to the influence of modifying the original objective function? It should be verified whether the same problem occurs in the objective function presented in the original paper. If the numerical instability pointed out in this paper is due to the modification introduced, it should be explicitly mentioned, as it could potentially mislead readers about the effectiveness of SCL.\n\n**References**\n\n1. Shu, R., Chen, Y., Kumar, A., Ermon, S., & Poole, B. (2019, September). Weakly Supervised Disentanglement with Guarantees. In *International Conference on Learning Representations*.\n2. Locatello, F., Poole, B., Rätsch, G., Schölkopf, B., Bachem, O., & Tschannen, M. (2020, November). Weakly-supervised disentanglement without compromises. In *International Conference on Machine Learning* (pp. 6348-6359). PMLR.\n\nQUESTIONS:\n- **In the numerical experiments using synthetic datasets, why were the values of $\\beta$ not set to be compositional for each scenario?** For example, $\\beta=1/2$ tested in the Box (simple) scenario does not seem to be tested in other scenarios like Hollow ball and Cube grid. Specifically, for the Cube grid scenario, only this condition seems to skip the intermediate value of $\\beta=3$. Is there a clear reason for this? The validity of the parameter range used in the experiments should be clearly indicated.\n- **Sampling of positive pairs seems to be a crucial assumption in theory, but I could not glean from the text what assumptions were made.** In the Weakness section above, I have raise some concerns based on my understanding that positive pairs are sampled from a probability distribution derived from the conditional distribution of the true latent variables defined in Equation 1. Is this understanding correct? If this interpretation is wrong, it would be helpful to explicitly mention in the text about the assumption of the probability distribution of the positive pair samples.\n\nLIMITATIONS:\nThis paper analyzes the acquisition of disentangled representation using contrastive learning from the viewpoint of identifiability, successfully relaxing some of the assumptions required in previous research. This is a theoretically solid result, and the results have been verified in numerical experiments using several benchmark datasets. However, as mentioned at the beginning of the Weakness section, there are significant concerns about applying the claims of this paper to practical situations. By appropriately discussing the practical validity of the assumed premises, I believe the significance of this paper will become more firmly established.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a unified framework of contrastive learning. The authors extend the theoretical result of [1] to embrace more contrastive losses, adapt it to marginal distributions beyond uniform distribution, and further relax some conditions that are required in [1]. The experimental results support the theoretical claim, and the authors also study/discuss the limitation of the theoretical framework.\n\n[1]: Contrastive Learning Inverts the Data Generating Process\n\nSTRENGTHS:\nOverall, the story telling is good: The authors first come to a theoretical guarantees under some relaxed assumptions, then empirically validate the theoretical findings, and further analyze the impact of the violated assumptions, and discussed the limitations. Compare with previous close works, there are two advantages of this work, include:\n\n1. A unified framework instead of focusing on InfoNCE family, and 2. More relaxed conditions of the theorems. \n\nIn order to justify their theorem, the experiments on the synthetic datasets are designed to embrace more contrastive losses, and many scenarios are designed for non convex latent spaces, nonuniform marginals and without conditionally independent assumptions on latents. \n\nThe studies on the limitations of the framework also bring some insights for future research.\n\nWEAKNESSES:\nThis paper has a lot of similarities with [1] (the 56th reference in the draft) in the foundation. Though this work does relax the theoretical guarantee of [1], the authors of [1] also empirically show that the underlying factors of variations can be identified even if the theoretical assumptions are severely violated. In Table 2 of this work, the authors also try to examine their theoretical framework under partially violated assumptions.\n\nQUESTIONS:\n-- Regarding batch size, why the authors use 5120 in line 205, and 4096 in line 268?\n\n-- In this paper, for all the experiments, are users using the original SCL or the modified SCL (line 118-121).\n\n-- It is interesting to the authors conduct ablation studies on the number of latent factors, I have a few questions:\n\n1. What scenario are you using for this experiment? \n2. Compare with section 4.1, are you sampling more data? Or, as the dimension increases, are you also exponentially increase the data?\n\nLIMITATIONS:\nNo negative impact found", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
This paper proves two identifiability theorems for a broad family of contrastive learning methods,  without imposing the commonly used independence constraint. The theoretical findings are validated on several benchmark datasets, and the theoretical and practical limitations are fully discussed.

STRENGTHS:
1. The considered contrastive model family is broad enough to include several widely-used contrastive learning losses, in which some are never employed for disentanglement (SCL and NWJ objectives). This provides the paper with adequate novelty. 
2. The proofs of the theorems are enlightening for me. In my opinion, the techniques used in the proofs will push this field forward.
3. The theoretical and practical limitations of the theory and the considered models are fully discussed, which is appreciated because the limitations of a work are of great meaning to readers.

WEAKNESSES:
1. If the paper can discuss the physical or geometric picture of the conditions and theory, it will be much better for readers to understand the theoretical results.
2. The are some typos in the paper, e.g. Equation (23) on Page 4 of the Appendix misses a right bracket.
3. Some concepts are used in the paper but not introduced, like 'semi-metric' in Equation (1) and 'norm-induced metric' in Theorem 1. I will suggest the authors briefly introduce these mathematical concepts in the footnotes, which will enhance the readability of the paper.

QUESTIONS:
1. why $\beta=2$ is excluded in the conditions of Theorem 2?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-25

comments: SUMMARY:
The paper develops a new theoretical framework for learning disentangled representations with contrastive methods. Building on the theory that contrastive methods approximately invert the data generating process, the paper extend it by proposing a new dissimilarity measure. The proposed framework supports a broader family of contrastive methods and relaxes the assumption about the data distribution and latent independence. Detailed theoretical proof is provided. Experiments validate the effectiveness of the proposed framework.

STRENGTHS:
Originality

The paper is an extension of previous work [56] and does provide a new framework which can apply to more CL approaches and requires weaken assumption about both the data distribution and latent independence. I think the differences are obvious and the paper seems novel to me.

Quality

The paper is well supported by theoretical analysis. Codes are also provided.

Clarity

Overall the paper is well-organized.

WEAKNESSES:
The author should provide more intuition and high level explanation of their designs (details in Questions).

QUESTIONS:
- Could the author provide more intuition about the design of the dissimilarity measure (L106)? Are the optimal $\alpha$ and $\bar{\alpha}$ function unique? Why jointly optimize them with the network help?
- For a more complex dataset in real task, what are the considerations when designing $\alpha$ and $\bar{\alpha}$ . Do you think a simple three-layer neural networks are also enough? Can you explain more on L287? What does "incorrect shapes" mean here?
- It would be better if qualitative visualizations/comparison on more complex dataset (e.g. KITTI Masks or 3DIdent) are added to show the improvements.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a generalized contrastive learning framework extending the guarantees for disentanglement learning. The paper also links contrastive learning objective with identifiability results in both week and strong identifiability setting.\n\nSTRENGTHS:\n- The paper is mostly well written and easy to follow (in few places it is hard to digest)\n- Authors do great job in converting all different contrastive learning objectives with unified notation\n- contrastive learning was empirically studied for it's effectiveness for disentanglement learning, authors to a great job in formally introducing these concepts\n\nWEAKNESSES:\n- Some assumptions may be missing in case of thm1 (ref. questions for details)\n- Results on causal3DIndent are hard to interpret (ref. questions)\n\nQUESTIONS:\nHere are some question that makes the paper bit unclear, addressing them can possibly improve the quality of the paper:\n\nQ1.  To recover true latent (to establish identifiability) you need contrastive examples varying in only one particular generating factor, which might not be always possible in reality, are there any tricks to circumvent this issue\n\nQ2. In case of theorem 1, I believe there are some strict constraints on how \\tilde{s} is constructed using s, these assumptions are not explicitly mentioned. (if there are no constraints, it's unclear how do you account for confounded generating factors)\n\nQ3. In case of confounded datasets, how would your identifiability result still hold? Since, in proof sketch you do not have any assumption over sufficiency and confounding factors.\n\nQ4. homeomorphism condition implicitly assumes the function to be bijective, continuous, and invertible which I believe is stronger assumption that [27] (correct me if I misunderstood the contribution).\n\nQ5.  The experiments are not clearly detailed: how is the data generated? what are the data generating factors?\n\nQ6. why is R2 value >> MCC? : as higher R2 indicates higher correlation which should result in higher MCC, but opposite effect is observed in your experiments, can you provide justification for this behavior?\n\nQ7. More details on training procedure would be helpful (how is parameter b learned?)\n\nQ8. How does the bound on features affect correlation values? (based on table 4 results bounded MCC >> unbounded MCC, what could be possible reason for this? is it an artifact of clipping values?)\n\nLIMITATIONS:\nFew technical limitations are mentioned in the paper, limitations of contrastive data acquisition in real-world setting is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}