Disentangling Properties of Contrastive Methods
Authors: Jinkun Cao
Qing Yang
Jialei Huang
Yang Gao
Venue: ICLR 2022 Submitted (2022)
Keywords: self-supervised learning
representation disentanglement

Abstract: Disentangled representation learning is an important topic in representation learning, since it not only allows the representation to be human interpretable, but it is also robust and benefits downstream task performance. Prior methods achieved initial successes on simplistic synthetic datasets but failed to scale to complex real-world datasets. Most of the previous methods adopt image generative models, such as GAN and VAE, to learn the disentangled representation. But we observe they are hard to learn disentangled representation on real-world images. Recently, self-supervised contrastive methods such as MoCo, SimCLR, and BYOL have achieved impressive performances on large-scale visual recognition tasks. In this paper, we explored the possibility of using contrastive methods to learn a disentangled representation, a discriminative approach that is drastically different from previous approaches. Surprisingly, we find that the contrastive method learns a disentangled representation with only minor modifications. The contrastively learned representation satisfies a ``group disentanglement'' property, which is a relaxed version of the original disentanglement property. This relaxation might be useful for scaling disentanglement learning to large and complex datasets. We further find contrastive methods achieve state-of-thet-art disentanglement performance on several widely used benchmarks, such as dSprites and Car3D. It also achieves significantly higher performance on the real-world dataset CelebA.

Under review as a conference paper at ICLR 2022
DISENTANGLING PROPERTIES OF CONTRASTIVE METHODS
Anonymous authors Paper under double-blind review
ABSTRACT
Disentangled representation learning is an important topic in representation learn- ing, since not only allows the representation be human interpretable, but it is also robust and benefits downstream task performance. Prior methods achieved initial successes on simplistic synthetic datasets but failed to scale to complex real-world datasets. Most of the previous methods adopt image generative mod- els, such as GAN and VAE, to learn the disentangled representation. But we ob- serve they are hard to learn disentangled representation on real-world images. Re- cently, self-supervised contrastive methods such as MoCo, SimCLR, and B BOL have achieved impressive performances on large-scale visual recognition tasks. In this paper, we explored the possibility of using contrastive methods to learn a disentangled representation, discriminative approach that is drastically different from previous approaches. Surprisingly, we find that the contrastive method learns a disentangled representation with only minor modifications. The contrastively Iearned representation satisfies a "group disentanglement" property. which is re- laxed version of the original disentanglement property. This relaxation might be useful for scaling disentanglement learning to large and complex datasets. We further find contrastive methods achieve state-of-thet disentanglement perfor- mance on several widely used benchmarks, such dSprites and Car3D, It also achieves significantly higher performance on the real-world dataset CelebA
INTRODUCTION
Learning disentangled representation is a long-desired goal in the deep learning community (Ben- gt al. 2013 Peters et al.]|2201 2017 Goodfellow et al.) 2016 Bengio et al. [0077] Schmidhuber) 1992
Lake et al. [001
al.20777 7 Tschannen et al. 2018). A disentangled representation matches how humans un- derstand the world and provides many other benefits besides model interpretability (Bengio al.,
2013 Chen et al.] 2016 Kulkarni et al. 2015). And usually needs much fewer labels to learn challenging downstream tasks (van Steenkiste et 2019). also generalizes much better even in face of examples generated by an unseen combination of the attribute values (Achille et al., 2018).
Given its importance and potential large impacts downstream applications, disentangled repre- sentation learning has recently attracted great attention. Previous research has proposed lot of methods, either built on variational auto-encoders (Kingma Welling 2013), such as 3-VAE (Hig- gins et al. 2016) and FactorVAE (Kim & Mnih 2018), or generative adversarial network (Goodiel-
low et al. 2014), such InfoGAN (Chen et al. 2016) and InfoGAN-CR (Lin et al. 2020). Those methods have achieved preliminary successes on synthetic datasets such as dSprites (Matthey et al.)
2017) and 3Dshapes (Burgess & Kim 2018).
Albeit the initial successes, those synthetic datasets are limited in many aspects, for example, the background is usually clean and composed of a single color, the number of objects small and the objects are mainly 2D without texture and occlusions. It is still open question to scale those models to real-world complex datasets. It is even non con-trivial to scale basic generative models to complex datasets learn disentangled features, such ImageNet (Deng al. 2009
In this paper, instead of studying disentanglement feature learning with a generative model, we in- vestigate whether the diseriminatively-triie -rained contrastive models have the disentanglement property. Contrastive learning is a class of self-supervised learning methods that pull two augmentations of the same image close. The recent contrastive methods have achieved sfate-of-ther performance on

---

Under review as a conference paper at ICLR 2022
the image pretraining tasks. Representative methods include MoCo (He al. 2020). BYOL Grrill jet al. 2020), SimCLR (Chen et al. 2020) and SwAV (Caron et al. 2020), etc. Contrastive learn- ing has been proven to learn good visual representations from large-scale datasets. We continue to investigate the disentanglement property of their learned representations in this work.
To our surprise, we find that the widely used BYOL algorithm without any auxiliary loss exhibits strong feature disentanglement property. However, the disentanglement of contrastive learning is a weaker form of disentanglement. It follows the pattern that representation dimension disentan- gled to correspond to single factor but a single ground truth factor might appear in multiple latent feature dimensions. We name this type of disentanglement as "group disentanglemen"" Although group disentanglement is a weaker form of disentanglemmnt, we hypothesize that directly learning compact and disentangled representation might be hard, due to the lottery ticket hypothesis (Frankle & Carbin 2018). Pursuing group disentanglement instead of full disentanglement might be nec- essary to achieve disentanglement on the complex real-world dataset. The reason is probably that real-world images usually contain more details and noise, and "factor" might be always correlated to some other visual existence due to dataset bias. A good example is a ship that usually comes together with a large area of blue background in existing datasets.
Further, we find that contrastively trained representation achieves the state-of the-th--rttttt Factor VAE dis- entanglement score when evaluated on established benchmarks, such as Car3D (Reed et al. 2015 dSprites (Matthey et al.) 2017) and SmalINORB (LeCun et al., (004). Besides the widely used syn- thetic benchmarks, also evaluate the contrastive method on the CelebA (Liu al. 2018 human faces real-world dataset. We find that it also achieves better or comparable performances than the other methods on five commonly used disentanglement metrics.
In summary, our contributions in this paper is mainly empirical and include the follows:
1. We show that contrastive method, in particular BYOL,, learns "group-disentangledd" rep- resentations, without any extra auxiliary losses.
2. The contrastive method achieves the pate-of-the-are performance on several widely used disentanglement learning benchmarks.
3. We propose to quantitatively evaluate disentanglements on real-world dataset, which avoids the biases of synthetic images. Our contrastive method also achieves state-of ahe-art performance on this benchmark.
2 RELATED WORKS
Disentangled Representation Learning Disentangled representation is desired it represents human interpretable pattern (Bengio et al. 2013 Chen et al., 2016 Kulkarni et al.] 2015). enabling the downstream tasks learned more easily (van Steenkiste et al. (2119) and generalizes better (Achille)
et al. 2018). In this paper, we consider the fully unsupervised disentangled representation learning setting, i.e. assume no annotations which factors should be learned.
We notice the recent study of disentanglement is promoted by two communities: Disentanglement in Deep Features and Independent Component Analysis. Their research previously lie on different assumptions, data patterns, and evaluation metrics.
One community is motivated by the newly raised deep learning for encouraging disentangled repre- sentation over independent factors. they have shown much empirical progress on this problem and they directly term their goal as "disentanglement" The related study usually based on deep gener- ative models. For instance, VAE-based methods have achieved successes on this task (Higgins et al.) 2016 Kim Mnih 20188, Chen al., 20188, Kumar al., 2017). Besides, Generative Adversarial Networks (GAN) (Goodfellow et al. 2014:(Chen et al., (016)) are also put into the discussion of en- couraging representations' disentanglement. More recently, people have shown that the G-N-based approach can achieve competitive performance the above VAE variants (Lin et al.) 2020 Jeon ett al. 2018 Lee et al., (020b).. A recent work (Locatello et al. 2019) summarizes the popular meth- ods and metrics in this community and proposes tool for evaluation called disentanglement_Jl
including popular metrics such as DCI (Eastwood & Williams 2018), SAP (Kumar et al.) 2017).
MIG (Chen et al. (018)) and
2

---

Under review as a conference paper at ICLR 2022
Besides this series of studies, exploring underlying factors of variation in data pattern is long- standing goal of the Independent Component Analysis (ICA) community (Hyvärinen & Oja 2000). They share many similarities, for example, generative models, e.g., VAEs, are recently popular in 2020 ICA usually has different assumptions with the both (Khemakhem et al., 2020a Klindt et al.
"purely unsupervised learning" (Hälvä et al. 2021p. For example, the pattern of noise (Hyvarinen & Morioka, 2016) Khemakhem et al. (2200a) or some additional auxiliary variables (Hyvarinen et al.) 2019 Khemakhem et al. (2200b) can be observed. Traditionally, ICA uses identifiability assess their desired representation pattern and the popular metric is Mean Correlation Coefficient (MCC). VAE (Klindt et al., 2020) recently makes great effort to connect the two branches of study but it still requires additional information such temporal transition pattern. As our study is for purely unsupervised learning and some assumptions of ICA methods can not be well matched, we mainly follow the settings and benchmark by disenttangeement_lle (Locatello et al. 2019).
Contrastive Learning Contrastive learning methods such as SimCLR (Chen et al. 2020), MoCo (He al. 2020) and BYOL (Grill et al. 2020) have achieved great successes to learn good visual representation from no label. They create "views" by applying augmentations over images. They treat two views of the same image as "positive pairs", and views of all the other images as negatives. This setup is also known as examplar classification (Dosovitskiy et al.) (0144), or instance discrimination (Wu et al.) 2018). Representation learned by contrastive learning has shown great transfer capabilities to downstream tasks, such as object detection and semantic segmentation.
More recently, there cally Wann & Isola 2020; Arora et al. [011 Tsai et al. 2020; Tosh et al. [202]] Tian et al. 2020bb
lot of works trying to understand contrastive learning either theoreti-
Lee al. (2020a) or empirically (Tian et al. 2020a Zhao et al. 2020 Purushwalkam & Gupta
2020). Zimmermann et al. (20TT) suggests that the contrastive method can invert the data gener- ation process. The conclusion is based on the analysis of Wang & Isola (2020) where negative samples are necessary and expected to be infinite. Zimmermann et al. (2021) ) make a good bridge between contrastive learning and independent analysis and study the model identifiability quantita- tively in terms of MCC score. Compared with that, we focus more on more direct disentanglement analysis. Our contribution is more empirical but suggests the good disentanglement property of contrastive learning even without negative samples.
3 METHOD
In this paper, we explore whether contrastive methods learn a disentangled feature representation. If yes, under what condition it Icarns a disentangled representation. There are quite a few contrastive learning algorithms proposed (Grill et al. 2020 He et al.] 2020; Chen et al., 2020 Caron et al.) 2.20). Although they differ on some specific aspects, they all aim to pull two augmentations of one image close. Without the uniformity property provided by negative samples (Wang & Isola, 2020). we find BYOL (Grill et al. 2020) still achieves unexpected good disentangled feature representation.
3.1 BYOL METHOD
BYOL is an unsupervised learning method that pulls two augmentation views of the same image close (Grill et al. (2000) to learn a high-level image representation. A significant difference of it against other contrastive learning is the absence of negative pairs during training. As shown in Figure D for each image T, we obtain two views of it: æ and Iz by data augmentations. One of them goes through the online network stream, and the other goes through the target network stream. The target network's 's parameter is not trained by the gradient descent algorithm but set as the exponential moving average (ema) of the online network. Both the online and the target stream have representation network (encoder) and projection network. The online network has an extra prediction network after the projection network. The online stream's 's output 21 and the target stream's output 22 are pulled close to each other by requiring the two vectors to have similar directions in the latent space. More specifically, the loss function is
22)
| 1|22112
In practice, the online representation network (without the projection or the prediction network) is usually the representation model for the downstream tasks. In this work, we follow the same convention, i.e. we study the disentanglement property of the output of the representation network.
3

---

Under review as a conference paper at ICLR 2022
representation
projection
prediction
online
ema
ema
loss:
|a blaale
representation
projection
22
target
Figure 1: The BYOL network architecture. has an online stream and a target stream. The online stream has ietieenttttioo-oojectiiio tion part. The target branch is updated by exponential moving average of the online branch. The model is trained by encouraging the output of two streams from positive pair of views to get close.
pipeline while the target stream lacks the predic-
3.2 IMPLEMENTATION DETAILS
We follow most of the implementation details in the original BYOL paper, but we find that several details can be changed to achieve better feature disentanglement. For full implementation details please refer the appendix A.I
Architecture To make fair comparison, we follow the encoder architecture used in Factor- VAE (Kim & Mnih 2018) for all synthetic datasets, except that the latent code dimension set to be 1000, since the contrastive learning requires large latent code dimension to work (Grill et al., 2020 On CelebA, encoders are unified to ResNet-50 (He et al., 2016).
Normalization By default, BYOL uses batch normalization throughout the network. However, we find that batch norm in encoder can reduce the feature disentanglement in some cases (see Sec- tion 5.5 for ablations). Thus we replace all batch norm with group normalization (Wu & He (018). Unless otherwise stated, the group number is set
our implementation by default.
Augmentation Augmentation is a crucial component for unsupervised learning methods as it pro- vides a feasible way to create positive pairs in contrastive learning. In this work, we follow the default data augmentation used in BYO,, i.e., the composition of color jitter, graying, horizontal flip, gaussian blur, and random resize crop. We recognize that the parameter random resize crop size is critical to learning disentangled features, which we will discuss in the ablation study.
4 MAJOR RESULTS
In this section, summarize our major empirical findings. Section 5 will then continue to present experimental evidence to support these claims.
4.1 THE GROUP DISENTANGLEMENT PROPERTY
We find that the contrastive feature representation exhibits some form of disentanglement property: different ground truth factors map to different sets of latent dimensions, but multiple latent dimen- sions are corresponding to single ground truth factor. name this type of disentanglement as "group disentanglement" See Section 5.2 for experimental evidence to support this characteristic.
Contrastive Representations
Ground Truth Factors
Figure 2: Illustration of group disentanglement. The lower vector denotes four ground truth factors and the upper vector is the group disentangled feature vector. Each ground truth factor may corre- spond to multiple feature dimensions, but each feature dimension only correspond to one factor.
While there is not single widely accepted definition of the original disentanglement concept, describe it intuitively as (1) Disentanglement one dimension in the representation vector only represents ground truth factor (2) Compactness: ground truth factor only corresponds to

---

Under review as a conference paper at ICLR 2022
one dimension in the representation (3) Completeness: ground truth factor is left out in the representation. Group disentanglement relaxes the compactness requirement, but only requires the disentanglement and the completeness. Figure shows how group disentanglement looks like.
Group disentanglement is a weaker form of disentanglement. but we hypothesis that it might be better proxy of disentanglement to scale to complex and realistic datasets. Directly learning the ideal compact disentanglement representation with neural network might be very hard because the lottery ticket hypothesis (Frankle & Carbin 2018). The lottery ticket hypothesis states that usually a well-trained neural network can be pruned to a much smaller network while keeping the same accuracy. However, it is much harder to directly train the same small network from scratch when randomly initialized. The hypothesis encourages a large randomly initialized network from start to contain subset of the network that is the "winning ticket". We suspect that a compact disentangled representation is also hard to obtain for the same reason. Instead, group disentanglement might be much more realistic goal to achieve.
4.2 COMPETITIVE PERFORMANCE ACROSS BENCHMARKS AND METRICS
To quantitatively evaluate the disentanglement properties of the contrastive methods, we evaluate with existing benchmarks (Cars3D (Reed et al., 2013), dSprites (Matthey et al., 2017). Small- NORB (LeCun et al.) 2004), Shapes3D (Burgess & Kim 2018) with various metrics. We find that although contrastive learning is not designed to learn disentangled feature representation, it still achieves better comparable disentangled features when compared to existing specialized methods. We even build benchmarks on the real-world CelebA dataset for the first time, and the performance of contrastive learning is still robust to be top. See Section 5.3 for experimental support.
4.3 BATCH NORM DISCOURAGES FEATURE DISENTANGLEMENT
We find that batch norm in encoder consistently decreases the feature disentanglement level com- pared to no normalization layer (Section (..5). By keeping the batch norm in the projector and the predictor, removing batch norm in the encoder will not cause model collapsing, which agrees with the observation in previous works (Richemond et al., 2020). On the contrary, replacing batch norm in encoder with group norm or layer norm will increase the feature disentanglement while achieving similar accuracy in downstream factor prediction. We notice that a similar phenomenon has been discovered before in supervised feature disentanglement. For example, Bau et al. (2017) discovered that a network trained with batch normalization layers has less interpretable (disentangled) neurons. We still do not fully understand this behavior, but hypothesize that it may be caused by the shared batch statistics that make hard for a feature to be aligned to the ground truth factor.
EXPERIMENTS
In this section, we show the quantitative results to support the described observations above. We first introduce the experiment setup. Then we provide qualitative analysis about the learned representa- tion pattern by contrastive learning. At last, we list a series of quantitative benchmark experiments prove the good disentanglement property of B BOL on both synthetic and real-world datasets. Be- sides the main experiments in this section, more ablation experiments are provided in the appendix.
In this section, we aim to empirically understand the disentanglement properties of the contrastive method following the four questions. (1) What does the contrastive method learn in its latent repre- sentation? Is it disentangled? Can we visualize the latent space? (2) Quantitatively, how disentan- gled is the representation of contrastive methods? (3) How about real-world datasets? (4) What set of
best promotes the learned representation to be disentangled?
5.1 EXPERIMENTS SETUP
In this section, describe the experimental setup, including the datasets we evaluate, the metrics for a quantitative study, and the previous disentangled feature learning algorithms for comparison.
Datasets Previous works evaluate representation disentanglement only on synthetic datasets, such as dSprites, Cars3D, SmalINORB, and Shape3D. Besides those datasets, we also include real- world dataset CelebA. CelebA contains human celebrity faces images with 40 binary attributes annotations. The attributes include finennnained properties of the human face such as whether wearing glasses or whether has wavy hair. Table 6 in Appendix explains the dataset details.
5

---

Under review as a conference paper at ICLR 2022
Mutual information of factors and latent dimensions
0.5
0.4
Factor
Positio
02
320
180
240
300
260
420
480
540
600
660
720
780
840
9.00
960
1020
latent dimension Index
Figure 3: The mutual information heatmap between factors and latent code dimensions. It explains the pattern of "group disentanglement" that a factor shares high mutual information with multiple dimensions of representation instead of only one dimension described by the perfectly compact disentanglement. On the other hand, a certain representation dimension will not have high mutual information with more than one factor.
Co-occurrence mutual infomation
Metrics We follow disent aetttaangeeeiiii (Lo-
Stape
-0.00225
catello et al. 2019) to five popular metrics for eval- uation, f.e. VAF, FactorVAR, MIG, SAP, and DCL. We leave out the Modularity metric since Locatello et al.) (2019) suggests that it is inconsistent with other metries. Please refer Appendix [A.4 for details including the cal- culation of mutual information.
0.00200
Saak:
0.00175
Factor
Poiliax Y Ptaliux X
0.00150
0000125
0,00100
Reference Methods Most of the previous state-of-art
disentangled feature representation learning methods are either built on VAE (Kingma & Welling 2013)
0.00075
Shep
Scdla
Poction_X
Postion_ Y
GAN GGodffllow et al.] /3-AEE (Higgins et al. 2016 introduces hyper-parameter to adjust the KL con- straint in VAE. FactorVAE (Kim & Mnih 2018) and B- TCVAE use adversarial training to reduce the correla- tions different dimensions of the latent code. DIP- VAE (Kumar et al.) (2117) pull the posterior to a factorized prior. Besides the VAE-based methods, also investi- gate Generative Adversarial Networks (GANs). InfoGAN (Chen et al. 2016) encourages the latent code to have high mutual information with the generated image. IB-GAN (Jeon et al. (013) and InforGAN-CR (Lin et al. 2020) add more constraints on InfoGAN to further promote disentangle- ment.
Factor
Figure 4: The visualization of normal- ized co-occurrence of mutual informa- tion on dSprites.
5.2 UNDERSTANDING THE LEARNT REPRESENTATION
In this part, we make the qualitative study of the disentanglement of representation learned by BYOL. Since contrastive methods are generative models, is hard to directly do factor- controlled pixel-wise reconstruction for visualization. We thus turn to measure the mutual infor- mation between learned features and ground truth factors study that. The model is analyzed on dSprites dataset. dSprites has in total five factors (shape, scale, orientation, position_x, and po- sition.y). But as explained Appendix A.33 the orientation is ill-defined with ambiguity. For example, it is impossible to distinguish if square rotates O degrees 180 degrees. Therefore, ignore this factor temporarily in the following discussion.
After encoding input image to representation vector, compute the mutual information be- tween each factor and each representation dimension. The mutual information between all dimen- sions and all four left factors included in Figure It shows evident pattern that a ground truth factor can correspond to multiple representation dimensions but a single dimension of representation only has the high response one factor.
To have more intuitive understanding of to what extent a representation dimension may respond more than one factor, define metries by the normalized co-occurrence of mutual information.

---

Under review as a conference paper at ICLR 2022
Given the mutual information between the representation vector and the 1th factor, noted as M,, Mí is the mutual information by the jth dimension. Then, the normalized co-occurrence of mutual information between the 7th factor and the hthh factor is
(M, MR) Ci |Mal2 || || 1|M&212
EE-MM
where L is the representation vector length. The definition conforms to the InfoMax (Linsker) 1988) principle of maximizing the mutual information of conceptually correlated pairs. We visualized the normalized co-occurrence of mutual information among the four factors by the learned representa- tion in Figure 4 It agrees with the pattern revealed in Figure that representation code on a certain dimension will not have high mutual information to more than one factor. Moreover, shows the independence degree of factor pairs. For example, the shape and scale of dSprites objects are not fully disentangled because objects with the same scale value but in different shapes have different pixel area. The co-occurrence of mutual information on them is thus slightly higher than other pairs.
341
0
Tatent Indlex
Figure 5: Representation variation when manipulating factor only in the dimension-reduuedd version. (a) and (b), position x and position y are manipulated respectively and only cause one dimension severely variate. While when manipulating the ill-defined factor orientation, two dimen- sions variate. The result shows a more compact disentanglement after reduction of dimension for those well-defined independent factors.
The analysis above recognizes the pattern of "group disentanglement". Then comes a natural question of whether can get a compact and interpretable representation. Here, reduce the representation dimension by PCA to 10. Figure(3jan shows the result of latent variation when changing only one factor at once. Given three images with only one factor's value different, we generate the 10-dim representation vectors from them. Then, we compute the variance across the three vectors. The Figure 5 a) and (b) show how reduced latent code changes when manipulating position_x and position y factor respectively. It shows good disentanglement that only one representation dimension high variation. However, in F]g we show a failure mode of the ill-posed factor orientation that change of factor causes both 6th and 9th dimension of reduced representation to have large variations. From the results, we observe that manipulating one well-defined independent factor causes evident variance on only one dimension. Compared with the original group disentanglement pattern, the dimension-reduced version becomes more compact and interpretable.
To summarize, we have found strong empirical evidence about the "group disentanglement" property of the representation learned by contrastive learning. Moreover, we have shown that the representa- tion be made more compact through proper unsupervised reduction.
5.3 COMPARISON EXISTING BENCHMARKS
To quantitatively evaluate the disentanglement of contrastive learning, we compare contrastive learn- ing with the previous line disentangled representation learning methods on commonly used bench- marks in this section. In the previous works, lot of datasets have been proposed evaluate the disentanglement learning method. Locatello et al. (2019) suggests that new method should be eval- uated a wide range of datasets minimize potential dataset biases.

---

Under review as a conference paper at ICLR 2022
In this part, follow the evaluation protocol of Locatello et al. (2019) to gain quantitative disen- tanglement measurement on dSprites dataset. For results on SmallNORB, Cars3D and Shapes3D, please refer to the results in Appendix. B The results of VAEs are reported based on the best con- figuration from Locatello et al. (2019). We calculate the mean performance and standard deviation from the released raw logs. The results of InfoGAN, IB-GAN and InfoGAN-CR
reported in Lin et al. (2020) and we use them its officially released checkpoint to evaluate under our protocol. The performance of Ada-GVAE and Ada-ML-VAE are reported in previous paper (Locatello et al., 2020). In the independent component analysis community, we can also find some methods to com- pare with but usually they are designed for different benchmarks and metrics. We adopt the method reported in ICE-BeeM [Khemakhem et al.) 2020b) and migrate it to the benchmarks we evaluate on. However, as we focus on unsupervised learning setting, no factor attribute is available for training data, so the conditional denoising score matching (CDSM) version of ICE--BeM can not be used here. We turn to use its unconditional version, and we term as EBM (energy-based model) in following content.
Model
Beta
FactorVAR
MIG
SAP
DCI
8-VAE
82.3 (7.6)
65.8 (9.2)
26.3 (11.0)
(2.7)
39.3 (13.2)
3-TCVAE
86.7 (2.4)
76.6 (7.8)
23.8 (6.8)
6.9 (0.9)
36.3 (7.1)
VAE
FactorVAE
84.9 (2.8)
75.3 (7.4)
18.4)
(0.8)
28.8 (10.6)
DIE-I
82.7 (3.3)
59.1 (4.8)
(5.1)
(2.))
14.6)
DIP-VAE-II
81.5 (4.9)
58.6
(3.4)
(.2)
12.3
AnnealedVAE
86.5 (0.1)
(0.0)
35.2 (1.3)
(0.))
$7.9(2.11
Ada-GVAE
88.0 (2.7)
(3.9)
17.3 (4.7)
6.6 (2.0)
(4.6)
SlowVAE
87.0 (5.1)
75.2 (11.1)
28.3 (11.5)
4.4 (2.0)
47.7 (8.5)
ICA
EBM
82.3 (2.0)
65.7 (12.5)
1.7
(1.2)
19.1 (1.8)
InfoGAN
59.0 (7.0)
GAN
IB-GAN
80.0 (7.0)
InfoGAN-CR
85.5 (1.0)
88.0 (1.0)
19.8 (3.2)
6.0 (1.0)
(5.2)
CL
BYOL (Ours)
93.2 (0.4)
91.6 (0.8)
29.3 (0.4)
8.0 (0.4)
66.9 (0.2)
Table 1:
Mean and standard deviation (s.d.) metric scores on dSprites dataset. The results of BYOL are averaged over three random seeds. The results of Ada-GVAE and SlowVAE refer to the Slow- VAE paper (Klindt et al. 2020). It shows strong and robust disentanglement property of BYO..
All evaluation settings follow the configuration of Locatello et al. (2019). The results strongly suggest that contrastive learning can well promote disentangled features. Moreover, shows the evidence that different metrics can not be precisely aligned as they use different perspectives to measure the disentanglement degree, which itself evens lacks a unified definition. Moreover, as the latent dimension of VAE-based methods is only 128, smaller than that of BYOL (1000-d), we increase the representation dimension of VAEs to 1000 for more fair comparison. The results are in Table 11 in Appendix, showing that increasing latent dimension does not boost the VAEs* disentanglement. Therefore, the performance gap between B YOL and VAE-based methods does not come from the change of representation dimension.
5.4 COMPARISON ON WORLD DATASETS
Although the four datasets above can establish how well each method performs, those datasets are all synthetic and are limited in various aspects. For example, dSprites only has one object per image on a black background; SmalINORB contains synthetic images of 3D objects viewing from different angles, but only contains 50 instances within 5 categories. High performance not necessarily transfer to real world images, where there might be lot more variations and unseen objects during test time. To get clearer sense of how each of the methods would perform the real world datasets, we further evaluate our method and several other methods on CelebA.
those datasets does
For evaluation of MIG and SAP metrics, to make fair comparison, the representation vector of all methods is reduced to 40 dimensions. The results are shown in Table 2 BYOL continues to show high disentanglement compared with other methods.
8

---

Under review as a conference paper at ICLR 2022
Model
Beta
FactorVAB
MIG
SAP
DCI
VAE
21.5
6.1
0.8
0.9
11.2
3-VAE
19.1
5.8
0.1
0.6
8.7
VAE
3-TCVAE
19.9
9.8
0.6
1.2
3.5
FactorVAE
25.3
12.0
0,4
0.6
7.1
DIP-VAE-I
21.0
9.3
0.2
0.9
13,8
GAN
InfoGAN-CR
16.8
11.3
1.6
2.8
22.0
CL
BYOL (Ours)
35.7
11.5
2.6
8.2
41.0
Table 2: Disentanglement evaluation on the CelebA dataset. The result shows great robustness of BYOL's learned representations show disentanglement on real-world datasets. Yet, the large gap between the score from that on synthetic datasets emphasises the difficulty of learning disentangled factors on real-world images.
normalization
w/o norm
BN
GN
LN
FactorVAL score
89.6
85.0
91.6
91.4
19.1
Table 3: Results of using different normalization strategies on dSprites. For group normalization, we set group number to 4. BYOL collapses with instance normalization (IN). For complete results, refer Table 13 and Table in the appendix.
5.5 EFFECTS OF HYPER-PARAMS
significantly affect the performance contrastive learning. this section, we systematically study the influence disentanglement property from recognized crucial in- ductive bias for contrastive learning.
The batch normalization in B BYOL has been considered the reason why BYOL does not collapse without negative pairs for long. However, a previous work (Richemond et al. 2020) finds B BYOL avoid collapsing still recognized as key variant B YOL. Thus we experiment with five normalization layers con- figuration in the encoder network the dSprites dataset. The results are shown in Table We find the commonly used BN decreases the disentanglement performances. No normalization already does well regarding feature disentanglement. Group norm (Wu & He 2018) and layer norm (Ba et al. 2016) are the two best normalization techniques. On dSprites, instance norm (Ulyanov et al.) 2017 completely breaks the contrastive learning process. We provide more study on normaliza- tion Table 13 and Table in Appendix where more variables, such as batch size, are taken into consideration. From all the studies, it remains unclear why batch norm could harm the feature dis- entanglement. However, this phenomenon was also empirically observed in the supervised learning literature as well. For example, in the Bau et al. (2017) paper, the author shows that CNN with batch norm layer has significantly fewer interpretable concepts than those without BN.
without batch statistics. But the normalization strategy between layers is
We also perform an ablation study on other tion trained model collapse with a big learning rate (Table 12,. More details are provided Appendix D
For example, we find the representa- dataset show transferrable disentanglement other datasets and BN causes
6 CONCLUSION
In this paper, show some empirical study the disentanglement property by contrastive learn- ing. We find with minor modification, the existing BYOL, framework can achieve state-of-the-art disentanglement performance multiple benchmarks and under diverse quantitative metrics. Be- sides, the qualitative study reveals the different "disentanglement" nature of contrastive learning that the representation actually shows "group disentanglement"" It is thus relaxed form of disen- tanglement due to weaker bound of compactness. To the best of our knowledge, we are the first build quantitative disentanglement benchmark with contrastive learning involved and the first to build benchmark of disentanglement measurement on real-world datasets (CelebA). Recently, the study of contratsive learning, or generally self-supervised learning, still motivated by empirical observations. We wish our work can reveal some clues on this line of study.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY
In this section, we provide the information required to reproduce our results reported in the main text. And we commit to making the code implementation and evaluating checkpoints public.
BYOL implementation For the implementation details of BYOL, please refer to Appendix A.1 The model architecture, training setups, and dataset preprocessing are all explained detail.
VAE methods implementation For evaluation synthetic datasets, i.., dSprites, Cars3D, SmalINORB and Shapes3D, the disentanglement score is from the original logs of di sentangg ement lib Locatello et al. (2019) [ In the released logs, each method has dif- ferent training configurations, and our reported result is from the configuration with the highest average performance overall provided random seeds. For evaluation
CelebA dataset, follow an open-sourced implementation in Pytorch 2 and align the encoder architecture of all methods to be the same as described Appendix A.I Parameters are kept as the default well-tuned version in the provided implementation.
hard to insert GAN methods' performance the benchmark as the training is not always stable and the discriminator weights usually not provided in many public codebases. When evaluating on synthetic datasets, the FactorVAE scores of InforGAN, IB- GAN, and InfoGAN-CR
GAN methods implementation
provided in the paper of Lin et al. (020 But the evaluation of other metrics in Lin et al. 2020) uses a not aligned settings with Locatello et al., (2019), so we check its officially release reevaluate the provided implementation and model weights under the unified evaluation setup. We perform the same evaluation process for results on CelebA dataset.
Energy-based Model (EBM) We refer to the implementation of ICE-BBeM (Khemakhem al, 2020b) for this method. We use the officially released codebase for it The encoder implementation has been aligned with our default already. The only modification we make is to use the uncondi- tional version instead of its default conditional version in loss computation. Please refer to the runners/real.dar wtw.reaaaaaaaaaaurrppppep runner.py file for details. Besides, we add the dataset definition for dSprites, SmalINORB, Cars3D, and Shapes3D it the implementation for our method. All hyperparame- ters are kept the default in its released version.
Evaluation Protocol Though our study is focused on disentanglement property, the evaluation of a representation model is two-step. In the first step, we perform factor prediction as a downstream task based on the learned representation model. Only when the accuracy of factor prediction is high enough to ensure the model weights "valid"" we would to the evaluation of disentanglement metric scores. For the implementation details of factor prediction, please refer to Appendix A.3 For disentanglement metrics evaluation, we use the official implementation of Locatello et al. (2019)
ie. disent anglement..i. .ib.. The settings of some important parameters in Appendix A.4
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Alessandro Achille, Tom Eccles, Loic Matthey, Christopher P Burgess, Nick Watters, Alexander Lerchner, and Irina Higgins. Life-long disentangled representation learning with cross-domain latent homologies. arXiv preprint arXiv: 06508, 2018.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint 19X:::9902.09229. 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 2016.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5):1-41, 2007.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE transactions on paltern analysis and machine intelligence, 1788)1778- 2013.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http:///ghhuu..oonnoppppm/////ls 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. 2020.
Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disen- tanglement in variational autocncoders. In Advances in Neural Information Processing Systems, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 1180-2188, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scabe scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances neural information processing systems, 27:766-779 2014.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In International Conference on Learning Representutions, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03633 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
11

---

Under review as a conference paper at ICLR 2022
Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. arXiv preprint arXiv:2106. 09620, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. arXiv preprint, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29:3765-3773, 2016.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411 1-300, 2000.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Proceedings of the 32nd International Con- ference Neural Information Processing Systems, pp. 4020-4031, 2018.
Insu Jeon, Wonkwang Lee, and Gunhee Kim. Ib-gan: Disentangled representation learning with information bottleneck gan. arXiv preprint, 2018.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference Artificial Intel- ligence and Statistics, pp. 2207-2217. PMLR, 2020a.
Ilyes Khemakhem, Ricardo Pio Monti, Diederik P Kingma, and Aapo Hyvärinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. arXiv preprint arXiv:2002. / 2020b.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pp. 2649-2658. PMLR, 2018.
Diederik Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114, 2013.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.109: 2020.
Tejas Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua Tenenbaum. Deep convolutional inverse graphics network. arXiv preprint 173.0 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.0084 2017.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
12

---

Under review as a conference paper at ICLR 2022
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. Proceedings of the 2004 IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition, 2:11-104 Vol.2, 2004.
Jason Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. arXiv preprini arXiv:2008.000 01064, 2020a.
Wonkwang Lee, Donggyun Kim, Seunghoon Hong, and Honglak Lee. High-fidelity synthesis with
disentangled representation. In European Conference on Computer Vision, pp. 157-174. Springer, 2020b.
Zinan Lin, Kiran Thekumparampil. Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In International Conference on Machine Learning, pp. 6127-6139. PMLR, 2020.
Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba)
dataset. Retrieved August, 11,
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly- -eeekiy-uupervieed disentanglement without compromises. In International Confer- ence on Muchine Learning, pp. 3348-6359 PMLR, 2020.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://gtthu..com/// http://hww...mmmmmeeeeddddd/dsss 2017.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: In- variances, augmentations and dataset biases. arXiv preprint 1xXh::0077.33116 2020.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv
preprint arXiv: 10520, 2019.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making Advances in neural information processing systems, 28: 1252-1260, 2015.
Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. arXiv preprint arXiv:2010. 10241, 2020.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural computation, 4(6):863-879, 1992.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint arXiv:2005.10243, 2020a.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint
2020b.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1179-1206. PMLR, 2021.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Demystifying sulf-super learning: An information-theoreeee framework. arXiv e-prints, pp. arXiv-2006, 2020.
13

---

Under review as a conference paper at ICLR 2022
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-baase representation learning. arXiv preprint arXiv:1812.05069, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 6924-6932, 2017.
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? arXiv preprint arXiv: 12506, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929 9939. PMLR, 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 7773--3742, 2018.
Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimina- tion good for transfer learning? arXiv preprint arXiv:2006.06600 2020.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. arXiv preprint arXiv:2102.08855 2021.
14

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 IMPLEMENTATION OF BYOL MODEL
Architecture We describe the implementation details of BYOL in this part. To make a fair com- parison with previous methods, for synthetic datasets, i.e. dSprites, Cars3D, SmalINORB, and Shapes3D, we follow the encoder architecture in Factor VAE (Kim & Mnih, 2018).. The pipeline details are shown in Table After each shown convolutional layer in the figure, there is ReLU activation layer. By default, there is also a group normalization (group number = 4) layer. So, the encoder is stack of (Cony-Rel.U-GN blocks. For CelebA dataset, the encoder is the commonly adopted ResNet-50 (He et al. 2016) backbone. By default, the final output channel number is 1000, i.c, D 1000. We note here that the group normalization use default has nothing necessarily connected with the group disentanglement property we study.
Besides the representation network (encoder), BYOL also has a projector network and a predictor network. Both of them consists of pipeline "Linear - BN - ReLU - Linear". The pro- jection dimension 256 and the hidden dimension of the projector is 4096. The predictor keeps a 256-dimensional feature vector in its pipeline.
Encoder of BYOL
64 64 images pipeline:
input:
4x4 conv, stride 2, 32-channel
4 4 conv, stride 2, 32-channel
4x4 conv, stride 2, 64-channel
4 conv, stride 2, 64-channel
4x4 conv, stride 2. 128-channel
1x1 conv, stride 1, D-channel
Table The encoder architecture for our implemented BYOL, on synthetic datasets. By default, we set D 1000 to be aligned with the commonly used ResNet-50 backbone network. Besides, there a ReLU activation layer and a possible normalization layer following each convolutional layer to create stack of Coon-RReL-NNome blocks.
Training settings We make minor modification on the training setting of BYOL. For training on all datasets, the images are resized to 64x64. For the data preprocessing, we copy 1-channel images of dSprites and SmalINORB 3-channel. During training stage, such pipeline of augmentation (in PyTorch-style):
RundomApphyttransfommss http://raaaooooooooooooooer.OO. ColorJitter(O.8, 0.8, 0.8, 0.2), p=0.3)
2. transforms. Irunsforws. RandmmoorrizzoollFlllyy
3. Ganddwaappyytransoooss.uuussaaad
(1.0, 2.0)), p=0.2)
4. wunffomw..aamomnsssstccoosssseee 64), (cale=(0.6,1.0))
5. normalization
For dSprites and SmalINORB, image pixel value uniformly normalized from [0,255] to [0,1.0]. For Cars3D, Shapes3D, and CelebA, adopt the commonly used Imagenet-statistic normalization for preprocessing the image values.
During training, we Adam optimizer by default, whose learning rate is 3e - without weight decay. The batch size is set to be 512 without exceptional notation. For evaluation on dSprites, select the weights after training for 15 epochs for evaluation. We select
Shapes3D and CelebA, the weights after training for 140 epochs for evaluation on Cars3D and weights the 200th epoch on SmalINORB considering the small scale of these datasets.
15

---

Under review as a conference paper at ICLR 2022
As explained in some ablation experiments, the default training setting is not the could get the highest disentanglement score. For example, using Group Normalization + Weight Standard- ization achieves higher disentanglement score than only using Group Normalization or not using normalization. But we aim to make minor modifications to the original BYOL implementation and follow previously commonly used setups.
To decrease the influence randomness, we train each model configuration multiple times with different random seeds. We report the results by the average and three random seeds. To be precise, as our implementation is based on Pytorch, we would set initialize the libraries of numpy, torch, torch. cuda and random with the same random seeds. We use 0, as the three values of random seeds in trails.
A.2 IMPLEMENTATION OF VAE MODELS ON CELEBA
The benchmark (Locatello et al. 2019) provides VAE-based methods' disentanglement performance on synthetics datasets only, e.g., dSprites, Shapes3D, Cars3D, and SmalINORB. For the real-world CelebA dataset, we need to train the models by ourselves to gain the results in Table 2 We use an open-sourced implementation for VanillaVAE, 3-VAE, Factor-VAE, 8-TCVAE and DIP-VAE-I. We follow the provided well-tuned hyper-parametert provided by them. Please refer to the reposi- tory for details.
Linear classifier for factor prediction
input: D-dim latent code pipeline:
linear layer (1000 - 256)
ReLU
linear layer (256 128)
ReLU
linear layer (128 100)
ReLU
K linear layers (in paraltel, 100 - nx, S kK
Table 5: The implementation of linear classifier for factor prediction. With the D-dim latent code from the encoder, the classifier has multiple fully connected layers to shrink the feature vector to 100-dimensional. Then, given the number factor types K for the target dataset, as each factor has mx(1 < k < K) values, we have K linear layers following the last shared layer in parallel. These layers predict the factor value on the corresponding K factors.
A.3 IMPLEMENTATION OF FACTOR PREDICTION
As mentioned in the main paper, for each gained representation model, i.e., encoder, we will first confirm it "valid" by training linear classifier for factor prediction. In the stage, the encoder weights will be frozen and only the attached classifier trained. For the task on all datasets, we use a unified classifier as shown in Table S The classifier has stacked fully connected layers (lincar layers) and multiple in in-parallel heads for factor prediction. On different target datasets, the number of factors and the possible values of factors vary. For the datasets accuracy threshold to confirm the quality of a representation acceptable are shown in Table 6 We well-trained BetaVAE model weights do the factor prediction first. Its prediction accuracy serves as the threshold in this sanity check stage. Usually, the accuracy threshold is set to be 0.8 or higher. But for the orientation factor on dSprites and Shapes3D, we relax the constraint 0.5. The reason is that the factors are ill-defined to have ambiguity. For example, when a square or ellipse object in dSprites has an orientation of 180 degrees 0 degrees, the image can be the same while the factors of orientation are regarded differently. For this reason, these factors bring noise into
evaluate on, the value range and the
16

---

Under review as a conference paper at ICLR 2022
study and validate the model weights. We, therefore, relax the accuracy requirement for them in practice. Besides, SmalINORB is the most difficult dataset by our observation. The factor prediction accuracy it is usually much lower, especially for azimuth and clevation, both of which harder to be predicted.
Dataset
Factor
Value Range
Accuracy Threshold
Shape
(square, ellipse, heart)
0,95
Scale
6
0.95
dSprites
Orientation
40
0.50
Position X
32
0.80
Position Y
32
0.80
Floor hue
10
0.80
Wall hue
10
0.80
Object hue
10
0.80
Shapes3D
Scale
8
0.80
Shape
4
0.80
Orientation
15
0.50
Elevation
4
0.80
Cars3D
Azimuth
24
0.80
Object id
183
0.80
Instacne category
10
0.70
Elevation
9
0.40
SmalINORB
Azimuth
18
0.40
Lighting condition
6
0.90
CelebA
40 human face factors
binary
0.80
Table 6: The factor of datasets we evaluate on. Some factors are originally continuous but discretized into all integers. Therefore, all factor prediction is classification task. Given a representation model trained on the training set of a dataset, the linear classifier should achieve accuracy higher than the theshold on all factors be recognized "valid". Only "valid" representation model would be put into the next step for disentanglement score evaluation. For the details of these factor definition, please refer original papers of these datasets.
A.4 EVALUATION METRICS
Five metrics are used to quantify the disentanglement our model.
Beta Vae Metrics Introduced in Higgins et al. (2016), beta vae metrics assumes each dimension corresponds to one category linear classifier. Representations are obtained after generated sam- ples with only one factor k fixed. Calculating the summation of the divergence between different representations and putting this result into linear classifier, we train a model that possibly outputs the corresponding k. The accuracy of this linear model is the value of beta vae metrics.
Factor Vae Metrics Kim & Mnih (2018) argues the beta vae metrics has the tendency to fail into linear classifier. Representations are obtained after generated samples with only factor k fixed. Normalizing each dimension in rep- resentations in terms of standard deviation. Index of dimension with lowest variances of normalized representation and the factor index is the input/output for the linear classifier. The accuracy of this linear model the value of factor metrics.
spurious disentanglement, and proposes new metrics based
17

---

Under review as a conference paper at ICLR 2022
Mutual Information Gap Chen et al. (2018) assumes the disentanglement model has the property that most information of one specific factor is contained in one dimension or a group of certain dimensions. The mutual information gap is the summation of the difference of highest and second- highest normalized mutual information between fixed k factor and dimensions in representations as shown below:
H (T(P)): 2x)
max Tug, z1:)) 3#12
(1)
K
k=1
Where is the overall number of ground truth factors, 2 is the factors of latent variables and JA arg max; [(1,, =x).
DCI disentanglement As Eastwood & Williams (018)) suggests, disentanglement the entropy of the relative importance, evaluated with Lasso or Random Forest classifier, of each dimension specific factors. Completeness is the entropy of the possibility that one factor captured by single code variable. Informativeness, overlapping with Disentanglement, is the prediction error of the model predicting factors.
SAP Kumar et al. (2017) proposes the Separated Attribute Predictability (SAP) score. A score metrics is computed with classification score of predicting jth factors on 1th dimension as the ijth entry. SAP is the mean of the difference of the highest and second-highes scores for each column.
For evaluation protocol, we follow the implementation provided by Locatello et al. (2019). De- spite exception, the evaluation batch size is 64, the prune dims. threshold is 0.06. If classifier is required to be trained during evaluation, num frain is 10000 and num eval is 5000. For Mutual information computation, the used discretizer function is the histogram discretizer and the number of bins in discretization is 20. For evaluation of MIG and SAP on dSprites, SmalINORB, Cars3D, and Shapes3D, BYOL representation vectors are reduced 10 dimensions by PCA be aligned with other methods. For evaluation of MIG and SAP on CelebA, to have a fair comparison, the representation vectors of all methods are reduced 40 dimensions.
Model
Beta
Factor VAE
MIG
SAP
DCI
V-VAE
100.0 (0.0)
89.3 (1.2)
11.7 (1.1)
1.4 (0.9)
38.7 (4.6)
3-TCVAE
100.0 (0.0)
92.2 (2.7)
15,5 (2.9)
1..3)
42.7 (3.5)
VAE
FactorVAE
100.0 (0.0)
91.7 (4.1)
(2.2)
2.0 (0.5)
29.0 (6.7)
DIP-VAE-I
100.0 (0.0)
90.5 (5.0)
(2.8)
(1.4)
22.6 (5.6)
DIP-VAE-II
100.0 (0.0)
85.0 (6.1)
5.1 (2.7)
1.3 (0.8)
20.8 (5.4)
AnnealedVAE
100.0 (0.0)
85.0 (4.3)
(1.0)
1.5 (0.5)
18.5 (4.3)
VAE
100.0 (0.0)
90.4 (0.5)
15.4 (2.2)
1.6 (0.5)
48.0 (2.4)
CL
BYOL (Ours)
100.0 (0.0)
95.8 (1.2)
7.6 (0.9)
1.8 (0.7)
48.5 (2.3)
Evaluation of disentanglement on Cars3D by different metrics. The results of BYOL are averaged over three random seeds. It shows strong and robust disentanglement property of BYOL.
Table 7:
B MORE BENCHMARK RESULTS
We report the benchmark results on dSprites in the main text (Table T). We continue to report the full benchmark results on other datasets the [Loatello al. (2019) evaluation protocol, i.e., Cars3D, SmalINORB, and Shapes3D.
As the large-scale benchmark of Locatello et al. (2019) provides the original logs on Cars3D, and SmalINORB datasets, we simply report the best configuration VAE-based methods trained by them. The original logs on Shapes3D are not available, so we train and evaluate on Shapes3D by ourselves and the output results well aligned with the reported result in Locatello et al. (2019). The results of other methods come from the same resources as those on the dSprites benchmark. As we find
the energy-bane based model (EMB) implemented from ICE-BeeM (Khemakhem al. 2020b) fails to
18

---

Under review as a conference paper at ICLR 2022
learn valid representation Cars3D with its default configuration and hyperparameter settings, do not report here. By default, we report mean scores averaged on three random seeds for our method. But Shapes3D, the mean scores for some methods not available, report the median performance instead as the default in Locatello et al. (2019). The evaluation protocol is the same that for dSprites described in Appexdix A.4
The full results are shown Table 7 Table 8 and Table respectively On SmalINORB and Cars3D datasets, our method still achieves the state-of-the-art performance in terms of BetaVAE score, Fac- VorVAE score, and DCI. But its MIG and SAP results are outperformed by other methods. This is because the BYOL approach learns a group-disentangled representation. Therefore, there are mul- tiple dimensions in the representation denoting the same ground truth factor. MIG and SAP score computes the mutual information difference between the most related and second related dimension to each ground truth factor. Thus group disentangled feature can have low MIG and SAP score, even after the dimension reduction. On the other hand, on Shapes3D,, our method fails to make the state-of-the-art performance on all metrics and the gap between its performance and others on MIG is even larger. Given the good performance of other methods on Shapes3D, we need more knowledge about method's relative failure on this benchmark.
Model
BetaVAE
Factor
MIG
SAP
DCI
3-VAE
84.1 (2.7)
60.1 (2.4)
25.0 (1.1)
(1.4)..1)
32.6 (0.6)
8-TCVAE
84.5 (2.7)
60.3 (2.3)
25.4 (0.9)
11.7
35.2 (0.7)
VAE
FactorVAE
80.8 (3.8)
62.5 (3.6)
23.9 (2.0)
10.9)
33.4 (1.1)
DIP-VAE-I
84.2 (3.2)
69.8 (4.6)
24.3 (2.7)
10.2
30.0 (2.1)
DIP-VAE-II
85.2 (1.3)
(2.1)
25,5 (1.5)
14.4 (0.4)
32.3 (0.7)
AnnealedVAE
60.8 (6.2)
(9.9)
9.1 (2.2)
6.8 (0.8)
15.7 (6.4)
VAE
78.2 (3.8)
(2.9)
23.8 (L.8)
7.8 (1.1)
28.7 (0.7)
ICA
EBM
79.0 (4.4)
17.9(33.)
1.7 (0.5)
1.9 (0.1)
13.9 (2.2)
CL
BYOL (Ours)
97.0 (0.8)
81.0 (0.5)
3.3 (0.9)
2.2 (0.3)
51.0 (1.0)
Table 8: Evaluation of disentanglement SmalINORB by different metrics. The results of BYOL are averaged over three random seeds. It shows strong and robust disentanglement property of BYOL.
To conclude, on the benchmarks and under the evaluation protocol provided by Locatello et al. (2019), our implemented contrastive learning shows SoTA-level performance in most cases. Its downsides MIG and SAP metries are probably related to its relaxed version of disentanglement property. The standard benchmark results help to provide a more accurate sense of the disentangling representation ability of contrastive learning.
C MORE QUALITATIVE STUDY
Limited by the main text length limitation, make qualitative study about the disentan- glement property shown by the contrastive learning here. In Figure 4 we show the co-occurrence of mutual information of factors on dSprites our model. We perform some qualitative studies on SmalINORB, Cars3D, and Shapes3D as well. The visualization results are shown in Figure 6 We can have some observation from them.
Figure and Figure I0
SmalINORB For the results on SmalINORB in Figure 6 though most non-diagonal entries have very low co-occurrence mutual information, two pairs factors show slightly higher co- occurrence. They are "azimuth-elevation" and "object/instance category-lighting" After investi- gating the dataset, we find the two pairs of factors are not fully independent. Figure sshw some samples with corresponding factors manipulated. We could see that the elevation and azimuth not fully independent. And the relation between the instance/object category and the lighting factor even more obvious because the lighting condition is sensible by the shadow and under the
19

---

Under review as a conference paper at ICLR 2022
Model
BetaVAE
Factor VAE
MIG
SAP
DCI
B-VAE
98.6
83.9
22.0
6.2
58.8
3-TCVAB
99.8
86.8
27.1
7.9
70.9
VAE
FactorVAE
94.2
82.5
27.0
6.1
67.2
DIP-VAE-I
95.6
79.7
15.2
4.0
55.9
DIP-VAE-II
97.8
88.4
18.1
6.3
41.9
Anneled
86.1
80.9
35.9
6.2
47.4
Ada-ML-VAE
100.0
100.0
50.9
12.7
94.0
Ada-GVAE
100.0
100.0
56.2
15.3
94.6
SAewVAE
100.0 (0.1)
97.3 (4.0)
64.4 (8.4)
5.8 (0.9)
82.6 (4.4)
ICA
EBM
75.9 (11.2)
53.2 (8.7)
5.2 (2.2)
2.8 (1.1)
21.8 (11.0)
CL
BYOL (Ours)
91.5 (3.9)
82.5 (2.4)
5.2 (1.7)
2.8 (0.3)
53.1 (1.5)
The results of BYOL averaged over three random seeds. It shows strong and robust disentanglement property of BYOL. Because the original experiment logs on Shapes3D by disentanglement..mm lib is not released, we can not get the average performance of baseline models. Instead we report median disentanglement scores in this table by referring to results reported Locatello et al. (2020) but the std error not available. The median performance of Slow VAE is reported in its original paper (Klindt et al., 2020
Table Evaluation of disentanglement on Shapes3D by different metrics.
Co-occurrence mutual infomation
SmalINORB
0..0014
azimuth
0.0013
object category
elevation
3 lighting
-..0122
-.0010
0.0009
2
Pactor
Figure 6: The visualization of co-occurrence of mutual information of the factors of SmalINORB.
shape Instance
lighting
>
elevation
azimuth
Figure 7: Samples from SmalINORB dataset. The variance is controlled by the factor indicated on axis. The image is from Jakab et al. 2018
20

---

Under review as a conference paper at ICLR 2022
Co-oceurrence of mutual infomation
Cars3D
0.0015
elevation
-0.0014
1 azimuth
2 object type
-0.0012
0.0011
-0.0000
0.0009
Factor
Figure 8: The visualization of co-occurrence of mutual information of the factors of Cars3D.
instance, whose distribution and shape is highly determined by the instance category. This reveals common observation that, in real-world data (SmalINORB is real-world yet but is compli- cated than 2D synthetic images), the factors are usually hard to define to be fully independent or disentangled.
Cars3D For the results from Cars3D shown in Figure g only one pair of factors show some co- occurrence, i.e. "elevation-object type". To study how this happen, randomly sampled some data from Cars3D by different object types and elevations shown in Figure 9 It shows that with the same value of elevation, samples of different object types help different visual elevation. So these two factors are not well disentangled. This might explain the slightly higher co-occurrence of mutual information between this pair factors.
Shapes3D The results on Shapes3D are shown in Figure I0| The result shows relatively bad disentangiement. To be precise, some factor pairs show low mutual information co-occurrence as expected, such as the color factors of floor, wall, and object and the pair of "object color azimuth" But the MI co-occurrence of "wall color object size" and "object color object size;" are higher than we expected as we did not recognize their high dependence. This result might relate to our model's s relatively poor performance on Shapes3D as well.
We hope the additional results shown above are helpful to provide more sense of the dataset con- figuration and what we should expect from a well-disentangled representation model. The full benchmark results show that contrastive learning can achieve SoTA-level performance datasets under some metrics while the performance is not that good for SAP/MIG in some cases. A study of this observation is important metrics. Besides, as the latent dimension our implemented BYOL model is relatively high and the group disentanglement property relaxes the requirement of compactness in disentanglement, mak- ing dimension-win wise disentanglement not distinguishable as expected in compact disentanglement. Because SAP and MIG are measured by comparing dimension-wer mutual information differences, the property of group disentanglement is the main potential reason for the observed disagreement of disentangiement metries from the reported benchmarking results.
standard
shows the disagreement of existing disentanglement
D MORE ABLATION EXPERIMENTS
Limited by the main content page length, we put some additional ablation study here help better understand the influence of important inductive bias when studying representation disentanglement.
In the original implementation of contrastive learning, the min-ratio of Landdom-ResizeCriie is usually very small, e.g, 0.08 for BYOL or 0.2 for MoCo/SimCLR, However, we find this hyper- parameter is critical. We thus perform study with different minimum ratios of random-resize-crog The results shown in Table I0 From the table, find the disentanglement score first increases when increasing the min-ratio of random- raadom-resize-cnn rew--e--siizippppeeeee but then drops. The default min-ratio set to be 0.6 in implementation.
21

---

Under review as a conference paper at ICLR 2022
elevation
object type
Figure 9: Some samples from the Cars3D dataset. The factor of azimuth is the same for all samples. The factors of samples vary along the axis of object type and the axis of elevation. We could find these two factors are actually not fully independent.
Co-occurrence of mutual infomation
Shapes3D
0 floor color
0.0012
1 wall color
2 object color
object size
Factor
-0,0011
object type
azimuth
-0,0100
0.0009
Factor
Figure 10: The visualization of co-occurrence of mutual information of the factors of Shapes3D.
min-scale
Beta VAE
FAC
0.1
93.4
86.6
0.2
95.4
87.0
0.3
94.4
84.8
0.4
96.0
86.8
0.5
94.8
88.3
0.6
93.8
88.0
0.7
93.5
89.0
0.8
91.6
81.4
0.9
87.7
77.8
Table 10: The influence minimum scale random resize-crop augmentation on the disentangle- ment of BYOL learned features. Here, the batch size is set to be 64.
22

---

Under review as a conference paper at ICLR 2022
method
V
FactorVAE
MIG
SAP
VAE
79.6
50.0
4.3
0.3
B- 3-VAE
20.3
24.5
7.4
0.2
3-TCVAE
62.1
54.7
1.1
0.3
Factor-VAE
63.2
54.0
0.2
0.8
DIP-VAE-I
82.0
67.3
0.7
0.2
Table 11: Disentanglement performance of VAE-based methods on dSprites, with an increased di- mension of latent code 1000-dimensional.
Learning rate is also another important hyper-parameter, we use Adam optimizer with learning rate to be 3e - 4 as default. But we are still interested in the influence of learning rate to the disentan- glement property of learned representation. At the same time, batch normalization in the encoder network is another critical component. In this part, we aim to study the transferability representation,
learned train BYOL under with different learning rates and batch normalization three datasets respectively. But the evaluation is always performed on the test set of dSprites. The results are included in Table 12 It shows, norm not just shows higher disentanglement score than BN but is also more robust with the variance of learning rate. When the learning rate is high, e.g., 3e-3 3e-2, the model with BN has collapsed already while the model without norm can still output reasonably high disentanglement score. On the other hand, on the transferability from one dataset another, the disentanglement score also shows high agreement.
dSprites
3dShapes
CelebA
learning rate
w/ BN
w/o BN
w/ BN
w/o BN
w/ BN
w/o BN
3e-5
75.6
83.0
60.5
77.0
59.6
78.3
6e-4
83.2
92.0
64.3
72.4
67.3
81.1
le-3
80.9
91.8
65.2
72.3
63.7
81.0
3e-3
42.3
91.4
23.1
71.0
65.1
80.5
3e-2
19.1
85.9
17.0
70.6
25.9
80.3
Table 12: We study the impact of learning rates over trained model's disentanglement. As all models are all tested on dSprites benchmark, BYOL shows its good generalization ability when trained on one dataset but transferring other datasets.
As the supplement of study on the normalization strategy, we provide a more detailed study in Table 13 where five normalization strategies evaluated with different batch sizes, which was found crucial choosing normalization strategy in visual tasks. The result of the table still suggests that Batch Normalization weakens BYOL's disentanglement property and no normalization, group normalization, and layer normalization show good disentanglement with different batch sizes for training. When using instance normalization in the representation network, BYOL goes collapse on dSprites.
(2020) shows that it is possible achieve similar results when replacing BN with group norm and weight standardization (Qiao et al. 2019). We continue to show the influence of involving Group Normalization (GN) and weight standardization (WS) Table [4] Here, we keep setting the group number to and batch size to 512. And we use the two weight decay values in the original BYOL and Richemond et al. (2020) respectively (3e-8 and I. 1.5e-6). The results show that compared with solely using GN, adding additional Weight Standardization can not just help learn features of higher quality but also be more disentangled.
Richemond
23

---

Under review as a conference paper at ICLR 2022
batch size
w/o norm
BN
GN
LN
64
88.5
80.0
90.4
89.2
19.1
128
89.6
83.3
91.6
91.0
256
92.0
85.6
93.2
93.8
19.1
512
89.6
85.0
91.6
91.4
19.1
1024
86.1
83.1
88.0
85.9
19.1
2048
84.5
79.8
82.5
82.8
19.1
4096
82.4
75.8
82.2
81.9
19.1
Table 13: Results of using different normalization strategy with different batch size during training on dSprites. We evaluate the FactorVAE score to indicate the disentanglement property of model weights from dSprites. For group normalization, we set group number 4. BYOL collapses with instance normalization (IN) only and the evaluated disentanglement score from also collapses to constant here.
normalization
BetaVAE
FactorVAE
MIG
SAP
w/o norm
92.9
89.6
16.4
5.6
GN
93.2
91.6
29.4
8.0
GN + WS
96.6
91.6
31.5
7.4
GN WS (wd 3e-8)
96.7
93.5
31.3
7.6
GN WS (wd 1.5e-6)
96.2
19.1
32.2
7.5
Table 14: The results to study the influence of Group Normalization (GN) and Weight Standardiza- tion (WS) on the representation disentanglement. The results prove the effectiveness both GN and WS to help promote representation disentanglement.
24


Reviews:
[{"review_id": "vi34_SdzbTO", "reviewer": "Reviewer_DZ9Z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The simplicity of the method: BYOL with group norm instead of batch norm in the encoder.\n- The proposed concept of group-disentanglement is a more relaxed but also more realistic property in real world scenarios. My feeling is that redundancies are needed in both model parameters (as shown in lottery ticket and dropout) and representations (group disentanglement).\n- There are comprehensive experiments and ablation studies to support the claims in the paper: on synthetic datasets, a real world dataset CelebA (which I think is a significant step to study disentanglement), and important hyperparameters.\n", "summary_of_the_paper": "This work explores the disentangling properties of contrastive methods. The authors discover that contrastive methods, particularly BYOL, learns disentangled representations with just a change of normalization method in the encoder. The work also proposes a new concept called \"group disentanglement\", which is a relaxed version of the original disentanglement. BYOL learns representations with group disentanglement and achieves SOTA on disentanglement benchmarks on not only synthetic image datasets but also a real world dataset.", "summary_of_the_review": "I am not very familiar with the field of disentanglement study, so I am not able to judge very well the significance of this work. However I like the simple and clear idea (to study disentanglement properties of contrastive methods) and the comprehensive experimental results. I especially like the concept of group disentanglement and the improvement of benchmarks on a real world dataset. Therefore I am recommending an accept.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tsrFV0jjJA1", "reviewer": "Reviewer_By54", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper makes the previously unreported finding that contrastive learning produces \"group disentangled\" representations and outperform other disentanglement models (as measured by standard disentanglement measures). It offers a nice literature review, exposition is clear and provides many implementation details that should help reproducibility.\n\nThough presented results are solid, it would be interesting to see the complete set of results (probably in Appendix), namely Fig. 4 and Tab. 2 (or Tab. 1) for other datasets. It might greatly benefit from some insight (perhaps preliminary) on why contrastive representations are disentangled when supervised/unsupervised representations would not. It is also unclear to me the effect of the contrastive learning pre-selection (sec 7, last paragraph) in the disentanglement performance, how are the thresholds in Tab. 7 selected, how many contrastive models are discarded in this process and would disentanglement performance change if the same threshold was used for all factors or if this selection was not performed at all---as would be in practice where this factors are unknown.\n\nQuestions:\n* It is not clear how the MI is computed for Fig. 3, I believe somehwere in the Appendix it said the distribution is approximated with a histogram, but some details (or a pointer to the details that are in Appendix) would be appreciated.\n* I noticed the best validation FactorVAE metric across normalizations (0.916 in Tab. 4) is the same as the one reported in Tab.1, can you confirm this is by chance and not because the same dataset split was used for both.\n\nMinor:\n* abs: explored->explore\n* Sec 1, p. 4: and Swav (Caron et al., 2020) etc -> , SWav (Caron et al., 2020), etc | and Swav (Caron et al., 2020).\n* Sec 2, p. 2: \"which maximize the information and the generated image\"  seems to be missing words.\n* Sec 2, last p.: drop the \"or not\"\n* Sec 3.2: without notion otherwise -> unless otherwised stated\n* Fig 2: add labels to the bars, e.g., \"Contrastive representation\" and \"Ground truth factors\"\n* Tab 1, caption: contratstive -> contrastive", "summary_of_the_paper": "Image representations learned using BYOL, a contrastive method, show better disentanglement than those learned with disentanglement-specific methods. Compares across models in toy datasets and one real-world dataset, CelebA.", "summary_of_the_review": "Overall a nice paper that makes some hefty claims and as such, I would like to see more results, have some questions answered and discuss with other reviewers before I can recommend acceptance. Showing results hold for other contrastive methods would also go a great deal in showing the validity of them and increase the impact of this work but it might be outside its scope.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Fq8r3s-Bo6O", "reviewer": "Reviewer_K9hY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Contributions: As your results are purely empirical I think statements like \"We show that a contrastive method learns group-disentangled representations\" might be troublesome since readers could expect theoretical guarantees.\n- Contributions: You say that your contrastive method (which is confusing since you just the usual BYOL method without modifying it) achieves state-of-the-art performance; however, I do not see a comparison with reasonable models, e.g., [1], [2], [3].\n- Abstract: The statement \"Prior methods achieved initial successes on simplistic synthetic datasets but failed to scale to complex real-world datasets.\" is unclear - to which prior methods are you referring?\n- Related work/Discussion: (1) Regarding the Zimmermann et al. 2021 paper, you say they didn't perform any quantitative analysis. However, this paper even proposes a new dataset of 3D objects and demonstrates the success of a contrastive method on it. (2) Furthermore, the comparison seems a bit skewed (please correct me if I'm missing something here), since your primary point of criticism of that work are the (strong) theoretical assumptions - yet, your paper does not contain any theoretical but just empirical results. (3) Finally, you say that you \"build a series of quantitative benchmarks\", yet, later in the paper, it seems like you only re-use existing datasets. If you mean that you tried evaluating disentanglement on real-world datasets, you should cite previous papers that did so, e.g. [2]. Can you please specify what you mean by \"build\" and make this more precise? \n- Major Results: How does your group disentanglement property differ from existing metrics in this field? I think a discussion should be added here. Your current description of it via Disentlement, Compactness, and Completeness sounds a lot like the metric suggested by [4] (i.e., the DCI metric) - what makes it special?\n- Major Results: I don't really see how the lottery ticket hypothesis is connected to disentanglement. At the heart of your argument, you say that you only say you suspect this to be the reason but do not give any reasoning as to why you think so - you only describe the lottery ticket hypothesis before.\n- Table 1, 2: Over how many random seeds did you average? What is the error of the score? That information is necessary to decide whether the results of BYOL are significantly better than those of the other methods.\n- Table 3, 4: Is it correct that these results correspond to just a single random seed?\n- Table 4, 5.5: You only discuss the result of BN; however, it remains unclear whether the BN results are really significantly worse than the other results. Furthermore, there is no discussion of the IN results, even though, that number is drastically lower than that of BN.\n- Introduction/Related work/Results: You did not include the best-working VAE-based disentanglement method, namely SlowVAE [1].\n- 5.4: (1) How is the reduction to 40 feature dimensions performed? Do you mean that you changed the architecture such that the output is only 40 dimensional or did you post-process the 1000 and 128-dimensional features, respectively? (2) Given that CelebA does not contain ground-truth latent variables, how did you compute the disentanglement metrics? This is a crucial aspect that is missing at the moment.\n\n[1]: Klindt, David, et al. \"Towards nonlinear disentanglement in natural data with temporal sparse coding.\" \\\n[2]: Khemakhem, Ilyes, et al. \"ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA.\" \\\n[3]: Zimmermann, Roland S., et al. \"Contrastive Learning Inverts the Data Generating Process.\" \\\n[4]: Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations.", "summary_of_the_paper": "The paper applies the contrastive BYOL method on a set of datasets, which are mostly well-established in the disentanglement community.  Here, they show empirical evidence that BYOL successfully learns disentangled representations. ", "summary_of_the_review": "Even though the idea of the paper to look further into the disentanglement properties of contrastive methods is interesting, I cannot recommend accepting it in its current form due to a lack of comparison with existing methods, impreciseness in the language/results, and too few explanations about the observed results.\n\n**Update**: After the rebuttal, I increased my score by one level from 3 to 5 - yet, there are still open issues that prevent me from recommending acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EdljUTNTP-I", "reviewer": "Reviewer_xhwR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Before knowing that we were assigned this paper for review, we were excited about the claims of this paper and replicated the results. We are therefore now familiar with the paper. Although the results in the paper can be replicated, the claims are a little misleading. We will explain below:\n\nStrengths:\n* The observation that BYOL features have better disentanglement properties than specialized methods is interesting.\n* The observation that group disentanglement appears for the dSprites and Shape3D datasets is interesting, but its importance might be overstated\n* A useful study of the effect of the normalization method on the disentanglement quality.\n* The paper for the most part is nicely written and has attractive figures\n\nWeaknesses:\n* The presented metrics are incomplete. DCI is not provided on most of the synthetic experiments. Beyond dSprites (which is an easy dataset) only factorVAE is reported which gives a limited comparison between the methods.\n* Misleading claim - representations are group disentangled - first of all, we did not find this to be correct for SmallNORB which is the hardest synthetic dataset. More importantly, we found that even for the easiest dataset, the only reason why it looks to experience group disentanglement is that mutual information is only computed at the single dimension level. However, if we take the 20 dimensions that are least imformative about an attribute, we could still classify the attribute with near perfect accuracy. So the claim of group disentanglement is misleading, there are just a group of dimensions that each individually seems to have information about a distinct factor, but the group is not disentangled at all.\n* I'm not sure if this is actually claimed but it might be confusing to the reader who might think that group normalization is related to group disentanglement . In fact, the experiments showed this is not the case. Choosing a single group is fine. So it's really about replacing batchnorm by layernorm type normalization. \n* No analysis is provided to explain why BYOL representations have these attributes. As the method here is not applicative by itself, its main utility is the theoretical understanding. We not think that such an understanding is provided here.", "summary_of_the_paper": "This paper makes an interesting empirical observation - BYOL representations have better disentanglement properties, according to some metrics, than current specialized methods. Furthermore, the authors found that the selection of the normalization function affects the results. The authors also claim the dimensions are \"group disentangled\" although this is only shown on one dataset.", "summary_of_the_review": "To summarize, the empirical observation that BYOL features are better disentangled than other methods is interesting. However, there are two important limitations: i) there is very little analysis trying to explain why this happens, and as no applications were shown, better understanding and analysis should be the main product of this research but they are lacking. ii) the experiments are incomplete - the group disentanglement is only shown for one dataset, our replication did not observe it for SmallNORB, such selection of results might make readers reach the wrong conclusions about the properties of the method. I suggest more complete experiments and more analysis.\n\n###################\n\nOur discussion with the authors and the poor experimental design reduced our confidence in this paper - we change our recommendation to rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides additional empirical evidence that self-supervised learning methods can help disentangling factors of variation in a dataset. That said, the paper can benefit from better framing and perhaps comparison with existing work (e.g., https://arxiv.org/abs/2102.08850 and https://arxiv.org/abs/2007.00810). Furthermore, the authors acknowledge that there was a bug in their code, which I believe should at least lead to softening the claims about group disentanglement. Accordingly, please consider revising the paper and re-submitting to other venues.
Decision: Reject