SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To fit our hardware setup, we use a smaller batch size of 5120. Also, we use an encoder network with residual connections and batch normalization, which has been shown to be more stable. We generally leave the output space unrestricted, i.e.,  $\mathcal{Z} = \mathbb{R}^n$ .
Table 1 provides a summary of the MCC scores. The corresponding  $R^2$  values can be found in the supplementary material. In each scenario, all contrastive losses achieve near-optimal disentanglement, with the exception of a few outliers. In particular,  $\delta$ -SCL appears to be numerically less stable than its counterparts, which we will examine in more detail in later sections. Additional experiments in the supplementary material compare  $\delta$ -SCL with the original SCL and show that  $\delta$ -SCL generally yields higher disentanglement scores. However, we would like to point out that one should not draw direct conclusions from these results about the performance of the original SCL on other downstream tasks.
To verify that the learned functions  $\tilde{\alpha}$  and  $\alpha$  converge to the solution given in Eq. (7) and Eq. (8), we construct a simple example where the ground truth is known. We choose  $S = \mathcal{Z} = [0,1]^2$  with a uniform marginal distribution. The positive examples are sampled from a truncated Laplace
Figure 2: Learned functions  $\alpha$  (top row) and  $\tilde{\alpha}$  (bottom row) for a simple example described in the text. The ground truth is in the right column. In the case of  $\delta$ -INCE,  $\alpha$  is set to zero.
Table 2: MCC [%] scores on synthetic data for  $\alpha (\mathbf{z}) = \tilde{\alpha} (\tilde{\mathbf{z}}) = c$
Scenario	β	δ-NCE	δ-INCE	δ-SCL	δ-NWJ
Box (simple)	1/2	99.57	99.81	82.69	89.22
Box (simple)	1	99.85	99.91	94.00	99.81
Box (simple)	3	99.77	99.40	99.78	99.76
Box (simple)	5	99.82	98.86	94.84	99.82
Box (complex)	1	99.70	99.82	76.63	99.68
Box (complex)	3	99.69	99.75	89.78	99.51
Hollow ball	1	98.49	99.37	87.31	97.94
Hollow ball	3	98.59	97.23	96.39	98.64
Hollow ball	5	98.52	96.26	98.49	98.52
Cube grid	1	99.82	99.39	98.21	99.75
Cube grid	5	96.53	84.73	97.50	97.42
distribution. Figure 2 shows the results. Details and additional experimental setups can be found in the supplementary material.
Since we learn the sum of  $\alpha$  and  $\tilde{\alpha}$ , they can only be identified up to some offset. In Figure 2, we have added the learned bias to  $\alpha$  to confirm that the total offset is correct. All losses reach the global optimum except for  $\delta$ -SCL, which appears to be stuck at a local minimum.
4.2 Evaluation under Partially Violated Assumptions
We also report results for the case where  $\alpha$  and  $\tilde{\alpha}$  are set to a learnable constant (see Table 2). Interestingly, we do not observe that this change significantly affects the outcome. However, we do observe a drastic performance drop in  $\delta$ -NCE,  $\delta$ -SCL and  $\delta$ -NWJ when this offset is fixed to a value which is far from the ground truth. Specifically, setting the bias to zero causes a drop in the MCC score by more than  $20\%$  on average when  $\beta = 1$ . For larger  $\beta$  we do not observe different behaviour. This is likely due to the fact that the optimal bias is far from 0 when  $\beta$  is smaller, which can distort the learned representation.
4.3 Evaluation on Kitti Masks
The KITTI Masks dataset [29] consists of segmentation masks of pedestrians extracted from the autonomous driving benchmark KITTI-MOTS [12]. We compare the presented loss functions with
Table 3: KITTI Masks. MCC [%] mean ± standard deviation over 10 random seeds
Δt	Loss	Laplace	Normal
Unbounded	Box	Unbounded	Box
0.05s	SlowVAE [29]	66.1 ± 4.5
- " -	δ-contr [49]	77.1 ± 1.0	74.1 ± 4.4	58.3 ± 5.4	59.9 ± 5.5
- " -	δ-INCE	77.3 ± 1.5	75.9 ± 1.7	68.1 ± 5.0	68.3 ± 4.8
- " -	δ-NCE	77.1 ± 1.0	75.4 ± 2.4	60.2 ± 4.9	68.0 ± 1.5
- " -	δ-SCL	75.8 ± 3.9	71.5 ± 7.5	66.2 ± 10.7	66.5 ± 4.0
- " -	δ-NWJ	77.3 ± 0.7	77.7 ± 0.9	67.5 ± 5.3	69.2 ± 5.3
0.15s	SlowVAE [29]	79.6 ± 5.8
- " -	δ-contr [49]	79.4 ± 1.9	80.9 ± 3.8	60.2 ± 8.7	68.4 ± 6.7
- " -	δ-INCE	80.5 ± 1.4	77.0 ± 3.3	68.9 ± 5.2	68.4 ± 4.2
- " -	δ-NCE	79.3 ± 2.3	79.5 ± 2.7	62.7 ± 4.6	68.4 ± 3.5
- " -	δ-SCL	72.9 ± 3.9	74.4 ± 2.6	62.5 ± 5.0	66.1 ± 5.7
- " -	δ-NWJ	79.3 ± 2.0	78.2 ± 6.1	68.2 ± 3.5	72.6 ± 6.7
Table 4: 3DIdent.  ${R}^{2}\left\lbrack  \% \right\rbrack$  and MCC [%] mean  $\pm$  standard deviation over 2 random seeds
β	Loss	R²	MCC
Unbounded	Box	Unbounded	Box
2	δ-contr [49]	96.43 ± 0.03	96.73 ± 0.10	54.94 ± 0.02	98.31 ± 0.04
2	δ-INCE	97.87 ± 0.07	98.08 ± 0.03	56.09 ± 0.10	98.98 ± 0.09
2	δ-NCE	98.05 ± 0.04	97.91 ± 0.02	55.34 ± 0.11	98.95 ± 0.08
2	δ-SCL	71.07 ± 3.76	74.92 ± 2.28	49.88 ± 1.20	76.02 ± 3.13
2	δ-NWJ	94.03 ± 1.35	97.67 ± 0.02	53.82 ± 0.46	98.79 ± 0.02
1	δ-contr [49]	96.87 ± 0.08	98.38 ± 0.03
1	δ-INCE	98.03 ± 0.06	97.45 ± 0.04	55.79 ± 0.09	98.63 ± 0.05
1	δ-NCE	97.83 ± 0.04	97.33 ± 0.03	56.35 ± 0.08	98.71 ± 0.02
1	δ-SCL	78.59 ± 3.92	93.13 ± 1.83	52.28 ± 1.98	95.77 ± 2.58
1	δ-NWJ	94.02 ± 1.28	97.64 ± 0.09	54.87 ± 0.33	98.82 ± 0.03
the current state of the art for this dataset [29, 49]. Except for the additional parameters in  $\delta$ , we use the same settings and follow their evaluation protocol.
As noted by Klindt et al. [29], the transitions of the latents are mostly sparse. It is therefore not surprising that all contrastive losses consistently perform better when a Laplace conditional is assumed as opposed to a normal conditional. Considering the large variance across different random seeds, all contrastive losses perform similarly well, with the largest outlier being  $\delta$ -SCL, which could already be observed on the synthetic dataset in the previous section.
Overall, there are only small differences between the different embedding spaces, with about half of the methods performing better on each setup. Consistent with previous observations [49], we see for all models slightly better performance for larger time intervals,  $\overline{\Delta t}$ , between consecutive frames. One possible reason is that a larger concentration of the conditional leads to a larger range of  $\delta$  values, causing numerical instability. We investigate this phenomenon in more detail in Section 4.5.
4.4 Evaluation on 3DIdent
We additionally evaluate our framework under partially violated assumptions on 3DIdent [49], a recently proposed benchmark dataset. The dataset is composed of rendered images showing a complex object in different positions, rotations and under different lighting conditions. Positive pairs  $(\mathbf{s},\tilde{\mathbf{s}})$  are obtained by randomly selecting  $\mathbf{s}$  and matching sampled  $\tilde{\mathbf{s}}^{\prime}$  from the conditional  $p(\tilde{\mathbf{s}}^{\prime}|\mathbf{s})$  to the closest  $\tilde{\mathbf{s}}$  with a rendered image. We consider a normal  $(\beta = 2)$  and a Laplace conditional  $(\beta = 1)$ . In both cases we set  $\hat{d} (\mathbf{z},\tilde{\mathbf{z}}) = \| \mathbf{z} - \tilde{\mathbf{z}}\| _2^2$  where we either leave the output space unbounded  $(\mathcal{Z} = \mathbb{R}^{n})$  or restrict it to an adjustable box  $(\mathcal{Z} = [0,b]^{n}$ , where  $b$  is a learnable parameter). The latter
Figure 3: MCC (top) and  $\mathbb{R}^2$  scores (bottom) for different numbers of latent factors (left) and for different concentrations  $(1 / \sigma)$  of the conditional distribution for  $\beta = 1$  (center) and  $\beta = 3$  (right).
is achieved by applying a sigmoid function in the last layer of the network and multiplying the result by  $b$ . Table 4 shows the results for weak and strong identifiability.
With the exception of  $\delta$ -SCL, the examined contrastive losses have  $R^2$  values close to the theoretical optimum. However,  $\delta$ -NWJ shows less robustness when we do not restrict the latent space compared to  $\delta$ -NCE and  $\delta$ -INCE. This demonstrates that incorporating knowledge about the latent space can stabilize training and lead to strong identifiability, even in the case of a (truncated) Gaussian conditional distribution.
4.5 Investigation of Limitations
In this section, we discuss practical and theoretical limitations of the proposed framework. We first examine how changes in the number of latent factors and the concentration of the conditional distribution affect disentanglement. We use the same setup as in Section 4.1, except that we choose a slightly smaller batch size of 4096 and train for  $8 \times 10^{5}$  iterations to ensure convergence when we consider more dimensions. The results are shown in Figure 3.
First, we note that all methods have difficulties when the number of latent factors is increased (see Figure 3, left column). In particular,  $\delta$ -SCL shows little robustness for larger dimensions.
Furthermore, we see that the performance of all methods declines with increasing concentration  $(1 / \sigma)$  of the conditional and is exacerbated by a larger shape parameter  $(\beta)$ . The work in [49] noted that InfoNCE has difficulty disentangling the latent factors when the conditional distribution becomes too flat relative to the marginal one, which, they pointed out, makes positive examples indistinguishable from negative ones. We observe here the same problem in the opposite case.
Consider the following example. Suppose  $S = [0,1]^n$  and  $p(\mathbf{s})$  is uniform, then the range of  $\delta$  at the global optimum is  $[0,n / \sigma^{\beta}]$ . For larger  $\beta$  and smaller  $\sigma$ , this quickly leads to vanishing values when calculating the exponential in the contrastive losses. This could also be one of the reasons why we see worse performance for smaller  $\overline{\Delta t}$  in KITTI Masks.
It should be noted that employing learnable functions in the similarity measure may entail certain drawbacks, such as the need to store additional parameters and a slight increase in training time. Furthermore, we have seen in Section 4.2 that in some cases the underlying data distribution can be adequately approximated by constant  $\alpha$  and  $\tilde{\alpha}$ . Although not observed in this work, it is also plausible that the inclusion of these functions for more complex datasets may increase the risk of getting stuck in poor local minima during training, as incorrect shapes of these functions may promote poor representations (the shape of  $\alpha$  and  $\tilde{\alpha}$  affects the gradient of  $f$  and vice versa).
Finally, we want to emphasize that the mechanism that governs the observation pairs is highly dependent on the application and that the relationship between the latents may not be adequately
described by a distance-based conditional as assumed in this work (Eq. (1)). However, we leave the exploration of methods suitable for other mechanisms to future work.
In this work, we propose a new theoretical framework of contrastive methods for learning disentangled representations. In contrast to previous research, our framework accounts for nonuniform marginal distributions of the factors of variation, allows for nonconvex latent spaces, and does not assume that these factors are statistically independent or conditionally independent given an auxiliary variable.
Our theory unveils that contrastive methods implicitly encode assumptions about the data generating process. We show empirically that even when these assumptions are partially violated, these methods learn to recover the true latent factors. This study provides further evidence that contrastive methods learn to approximately invert the underlying generative process, which may explain their remarkable success in many applications.
This work has been carried out as part of the AuSeSol-AI Project which is funded by Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (BMUV) under the Grant 67KI21007F.
[1] K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. arXiv preprint arXiv:2206.01101, 2022.
[2] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
[3] H. B. Barlow, T. P. Kaushal, and G. J. Mitchison. Finding minimum entropy codes. Neural Computation, 1 (3):412-423, 1989.
[4] J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319-38331, 2022.
[5] J.-F. Cardoso. Blind signal separation: statistical principles. Proceedings of the IEEE, 86(10):2009-2025, 1998.
[6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] T. Chen, C. Luo, and L. Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34:11834-11845, 2021.
[8] E. Creager, D. Madras, J.-H. Jacobsen, M. Weis, K. Swersky, T. Pitassi, and R. Zemel. Flexibly fair representation learning by disentanglement. In International conference on machine learning, pages 1436-1445. PMLR, 2019.
[9] C. Eastwood and C. K. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
[10] M. Fumero, L. Cosmo, S. Melzi, and E. Rodola. Learning disentangled representations via product manifold projection. In International conference on machine learning, pages 3530-3540. PMLR, 2021.
[11] L. Gao, Q. Mao, M. Dong, Y. Jing, and R. Chinnam. On learning disentangled representation for acoustic event detection. In Proceedings of the 27th ACM International Conference on Multimedia, pages 2006-2014, 2019.
[12] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.
[13] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.