Representation Disentanglement in Generative Models with Contrastive Learning
Authors: Shentong Mo
Zhun Sun
Shumin Han
Venue: ICLR 2022 Submitted (2022)

Abstract: Contrastive learning has shown its effectiveness in image classification and generation. Recent works apply the contrastive learning on the discriminator of the Generative Adversarial Networks, and there exists little work on exploring if contrastive learning can be applied on encoders to learn disentangled representations. In this work, we propose a simple yet effective method via incorporating contrastive learning into latent optimization, where we name it $\textbf{\texttt{ContraLORD}}$. Specifically, we first use a generator to learn discriminative and disentangled embeddings via latent optimization. Then an encoder and two momentum encoders are applied to dynamically learn disentangled information across large amount of samples with content-level and residual-level contrastive loss. In the meanwhile, we tune the encoder with the learned embeddings in an amortized manner. We evaluate our approach on ten benchmarks in terms of representation disentanglement and linear classification. Extensive experiments demonstrate the effectiveness of our ContraLORD on learning both discriminative and generative representations. 

Under review as a conference paper at ICLR 2022
REPRESENTATION DISENTANGLEMENT IN GENERA- TIVE MODELS WITH CONTRASTIVE LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Contrastive learning has shown its effectiveness in image classification and gen- eration. Recent works apply contrastive learning on the discriminator the Gen- erative Adversarial Networks. However, there is little work exploring if con- trastive learning can be applied the encoders-decoder structure to learn dis- entangled representations, In this work, propose simple yet effective method via incorporating contrastive learning into latent optimization, where we name it ContraLORD. Specifically, we first and disentangled embeddings via latent optimization. Then an encoder and two momentum encoders applied to dynamically learn disentangled information across large number of samples with content-level and residual-level contrastive loss. In the meanwhile, we tune the encoder with the learned embeddings in amortized manner. We evaluate our approach on ten benchmarks in terms of representation disentanglement and linear classification. Extensive experiments demonstrate the effectiveness of our ContraL.ORD on learning both discrimina- tive and generative representations.
generator to learn discriminative
INTRODUCTION
In recent years, disentanglement of factors in images has attracted many researchers' attention, which mainly includes two folds: adversarial and non-adversarial methods. Adversarial meth- ods (Mathieu al. [016)] Denton Birodkar 2017 Hadad al.) [0188] Razavi et al.] [2019] Gab-
bay & Hoshen] 2021) often apply min-max optimization framework (Goodfellww et al. 2014) for disentanglemen of images, which costs much time on
tuning. In terms of non-adversarial models, several variational autoencoders (Higgins et al. 2017] Kim & Mnih] 2018)
variants have been proposed to disentangle the generative factors in an unsupervised manner without inductive biases, which did not achieve satisfactory results proven in empirical study (Locatello et al. 2019).
With the extra class supervision, semi-supervised methods achieve promising performance in disen- tanglement. Typically, comprehensive experiments in (Locatello et al. 2020) validate the effective- ness of limited amount of supervision in state-of-the-art unsupervised disentanglement models. LORD (Gabbay & Hoshen 2020) applies a latent optimization framework with a noise regularizer on content embeddings to achieve superior performance over amortized inference. Based LORD, OverLORD (Gabbay & Hoshen 2021) is proposed to disentangle class, correlated and uncorrelated attributes for image translation.
more recent work (Gabbay et al., 202T) adopts pre-trained CLIP (Radford et al. 2021) model to generate partial annotations for image manipulation. How- ever, there exist two main drawbacks among these methods: 1) using different separate encoders for different factors is resource- wasteful for real-world applications and requires expensive human design. 2) just learning the content embeddings inside each sample not sufficient learn the diversity existing in the dataset.
Driven by the shortcomings discussed above, we propose a simple yet effective method named ConDraLORD, where we incorporate contrastive learning into latent optimization for representation disentanglement. Recent works (Deng et al., 2020; Ojha et al.) 2020) apply the contrastive learning on the discriminator of the GAN (Goodfellow et al.) 2014) for disentangling representations. Typi- cally, the 3D generation by comparing pairs of generated images. However, in this work, focus applying
learning in (Deng et al. 2020) is used for controllable face image

---

Under review as a conference paper at ICLR 2022
contrastive learning on the encoders learn the discriminative and generative embeddings with dis- entangled information. Specifically, we first apply a generator to learn discriminative and generative embeddings via latent optimization. Then apply encoder and momentum encoder to dy- namically learn disentangled information across a large number of samples with content-level and residual-level contrastive loss. In the meanwhile, use the learned discriminative and generative embeddings to tune the encoder in an amortized manner.
We evaluate our ContraLORD on two main tasks: linear classification and disentanglement. Exten- sive experiments show the effectiveness of the learned discriminative embeddings on linear classi- fication and generative embeddings on the disentanglement of factors. We conduct comprehensive studies on three benchmarks on linear classification and seven benchmarks on disentanglement to investigate if contrastive eef-ssupeviied models can learn disentangled features. In the meantime, we achieve superior performance on linear classification compared to baselines. Our Contral.ORI) also achieves promising results over state-of-the-ar methods terms of disentanglement.
The main contributions of this work can be summarized follows:
We present simple yet effective method called ContraLORD by incorporating contrastive learning into latent optimization for representation disentanglement and linear classifica- tion.
We formally explore the disentangled features across large number of samples with content-level and residual-Ievel contrastive losses.
Extensive experiments ten benchmarks show the effectiveness of approach learn- ing disentangled representations.
RELATED WORK
Discriminative representations learning. Discriminative representations learning has addressed researchers' attention for long time since discriminative representations are significant for image classification. Most of the previous works adopt supervised (Lezama et al. (2018) and unsupervised
learning (Ji et al. 2017 Zhang et al. [018 Zhou et al. 2018 Zhang et al. 2019)) to learn embeddings that most discriminate between classes the dataset. Typically, the principle of maximal coding rate reduction (Yu et al. 2020) is applied to maximize the coding rate difference between the whole dataset and the sum of each separate class. However, there exists little work of contrastive learning to explore the discriminative representations for the pre-training stage. In this work, we mainly focus on learning discriminative embeddings for linear classification by incorporating contrastive learning into latent optimization to improve the performance of baselines.
Disentangled Representations Learning. Disentangled representation learning aims learning generative factors existing in the dataset, that is, disentanglement learning. A bunch of previous work focuses on unsupervised learning with variational autoencoders, such as 3- (Higgins et al.) 2017), Factor- VAE (Kim & Mnih| 2018). Following those work, DCI disentanglement (Eastwood & Williams 2018). SAP score (Kumar et al. 2018), and Mutual Information Gap (MIG) (Chen et al.
2018) are often utilized quantitative metries to measure the quality of disentangled representa- tions. In recent years, semi-supervised models have been used by many researchers in the literature.
Adding limited amount of supervision unsupervised models is proven in (Locatello
2020)
to be effective in learning disentangled representations for real-world scenarios. LORD (Gabbay) & Hoshen 2020) leverages the latent optimization framework with noise regularizer on content embeddings for class and content disentanglement. More recently, simple framework for disen- tangling labeled and unlabeled attributes is utilized in OverLORD (Gabbay & Hoshen 2021) for high-fidelity images synthesis. A study (Gabbay et al. 2021 uses a CL.IP (Radford et al.) (02T) pre-trained model to annotate a set of attributes for disentangled image manipulation. In this work, intend to learn disentangled embeddings via combining latent optimization and contrastive self- supervised learning.
Contrastive Learning. Recently, contrastive self-supervised learning (Tian et al. 2020 Chen al.
2020a by Grill et al., 2020; He et al., (2020) Chen et al. 2020c Chen & He| 2021) has been explored lot by many effective methods. SimCLR (Chen et al. 2020a). end-to-end structure, is proposed to pull away from the features of each instance from those of all other instances in the training set.
In the self-supervised setting, low-level image transformations such cropping, scaling, and color
2

---

Under review as a conference paper at ICLR 2022
jittering are utilized for encoding the in-variances from samples. The InfoNCE loss, that is, the normalized temperature-scaled cross-entropy loss, are often optimized to maximize the similarity between positive samples and minimize the similarity between negative samples. Large batch size is always used in this kind of end-to-end structure (Chen et al., 2020a b) to accumulate a large bunch of negative samples in the contrastive loss. PIRT. (Misra & Maaten 2020) without large batch size applies memory bank to store negative samples and update representations at specified stage. MoCo (He al., 2020) and MoCov2 (Chen et al. (220c) replace the memory bank with memory encoder to queue new batch samples and to dequeue the oldest batch. In this work, we leverage content-level and residual-level momentum encoders store queue of negative samples with disentangled information for learning generative embeddings, where content-level and residual- level contrastive losses are applied to capture content and residual representations.
METHODOLOGY
3.1 PROBLEM SETUP
In this part, we first begin with the problem setup, and formally define the notations for easy read- ing. In terms of the problem, our goal is to demystify the disentangled and diseriminative features learned by contrastive self-supervised learning. To address this problem, we propose simple yet ef- fective method by combining contrastive learning and latent optimization together for representation disentanglement. To explain it better, we define the notations below in unified manner.
Notations. Given set of training examples .... we need to learn one discriminative embeddings d and generative embeddings gi from pre-defined set of embeddings {g1, g2, ..... gm where m denotes the total number of generative factors in the training data. That is, d; E R lxd .: E R1xg, where d, denote the dimensionality of discriminative and generative embedding, separately. generative embedding g: into two folds: content embeddings gf and residual embeddings g The content embeddings contain the information that is related to the discriminative embedding, while the residual embedding includes the uncorrelated information.
X2. ... Xn For each image x, i
our setting, we split the
Overall, the objective of our work is to Iearn d,, gf g for each image x, from a training dataset. In the next part, present the technical details of our method. To learn di gf gf from set of training examples, we propose a simple yet effective approach called ContraLORD, as shown in Figure D Our ContraLORD mainly includes two parts: 1) embedding optimization: we first use generator G(.) to learn discriminative and disentangled embeddings via latent optimization. 2) encoder pre- training: we apply encoder (()) and momentum encoder fm (-) dynamically learn disentangled information across large amount of samples with content- and residual-level contrastive loss.
3.2 EMBEDDING OPTIMIZATION
In order learn the discriminative and disentangled embeddings, we are motivated by LORD (Gab bay & Hoshen 2020) to introduce the latent optimization in the first stage. Specifically, we apply a generator G(+) to reconstruct the original image x; by using the discriminative and disentangled embeddings of each sample. Instead of using the KL-divergence in variation auto-encoders (Kingma & Welling, 2014), we equally add regularizer with Gaussian noise a fixed variance to the disen- tangled embeddings of and g.. Thus, the objective of embeddings optimization is defined as
Copt s (Gidd of +2. 2:)), À (1 +
where E(-) denotes & loss for synthetic data and VGG perceptual loss for real images. À is the penalty weight of the capacity of the disentangled embeddings. z, ~ (0,,o2T) In this way, we can learn the disentangled embeddings d, gf. g. without any adversarial learning involved, that is, ae
d, g g.
arg min Lopt- For training sets with annotations, d, is given.
3

---

Under review as a conference paper at ICLR 2022
Discriminative Embedding
d
81
X1
Generator
X2
Generative Embedding
82
N(0, r²
Embedding Optimization
Encoder Pre-training
X1
X2
Encoder
1
Figure 1: The overall framework of our proposed Contral.ORD model.
3.3 ENCODER PRE-TRAINING
After learning the optimized embeddings, we need to train generalized encoder during the pre- training stage. order to optimize the encoder f()), we reconstruct the original image x; with the output embeddings from f(-), and the loss function is calculated
n Cree ((((
where f(-) denotes l2 loss for synthetic data and VGG perceptual loss for real images. ha(-), ((()),(()).))))) h)()), (-) denote the head for generating the discriminative, content-level, and residual-leve embeddings. To learn more disentangled information the discriminative and disentangled embed- dings, we use the learned embeddings in the first stage as the supervision and define the objective as:
Leup Σ [Ia([((x)) - -, (IN(((())) g -
g
where d,, g, g. denote the learned representations from the first optimization stage, respectively.
Content-level Contrastive Loss. To learn more disentangled content embeddings across examples in the dataset, we feed content momentum queue (x/+=y, key .... key ,.... ,xx key of one query sample x x.uory into a momentum encoder fm(()). The illustration of content-level contrastive learning is shown in Figure 2 (left). As be seen, we generate the content embeddings g & {1, ... , K} from the momentum queue, and & from the query sample x query Then we calculate the similarity between the original content embedding g and g g for content-level contrastive loss. Finally, the content-level contrastive loss is defined as
exp(gf //TT)
Leon
log exp(g. g/Tr) EK exp(gf g//r)
4)
where K denotes the number negative samples in the momentum queue. T is a temperature hyper-parameter. In the backward process, update the parameters of the encoder f(:) according
4

---

Under review as a conference paper at ICLR 2022
content-level contrastivel loss
residual-level contrastivel loss
gradient
gradient
similarity
similarity
gi g2 g5 ge
Eq
Bi B2 B3 8k
share
share
content head
content head
residual head
residual head
content momentum encoder
residual momentum encoder
encoder
encoder
guery
key
query
key X
x
Aay Eey Kety Jey
x1
x
x
content mometum queue
residual mometum queue
Figure 2: Content-level (left) and residual-Ievel (right) contrastive learning designed in our Con- traLORD.
to the gradient of this loss. The parameters of the momentum encoder fm (-) is updated by fm (-) & mfm(-) (1 (+) where E (0, 1] a momentum coefficient.
Residual-level Contrastive Loss. In order to further disentangle the residual part in the disentangled embeddings, we introduce residual momentum encoder ftm receive the residual momentum queue and generate set of residual embeddings g k E 21,2... .. K}. Thus, the residual- contrastive loss is defined with key embeddings g, query embedding g and the original embedding g
exp /T/TT)
Cres
log
(5)
exp(g, g/T)) Σ expíg gk/T)
where K denotes the number of negative samples in the momentum queue, and temperature hyper-parameter. The gradient of this loss is also used to update the parameters of the encoder f()). We update the parameters of the momentum encoder fm(() (.) by fm(() <- mfm() + m))()),
where E (0, 1] is momentum coefficient.
The overall objective of our ContraLORD is optimized in an end-to-end manner as L= (Crec + -Suu) + Aeoo Ccon + Ares Cree
(6)
where Acons Ares denote the weight of the content-level and residual-level contrastive loss, respec- tively. Extensive ablation studies are conducted to explore the effects of each loss on the final performance of our ContraL.ODD. We summarize the overall algorithm of our training approach in Algorithm
3.4 SMOOTHNESS OF EMBEDDINGS
In order measure the smoothness of embeddings pre-trained by the encoder, borrow the idea of uniformity property in the instance-wise contrastive learning, and introduce the Gaussian potential kernel (Bartók & Csányi 2015) calculate the average pairwise Gaussian potential as:
(7) where Pe, Pr denotes the distribution of content and residual embeddings in the hyper-sphere, and t is a positive factor to define the weight of the &, distance between embeddings g, and g,, In our experiments, follow previous work (Wang & Isola 2020) and set 1=2.
smoothness (( 1 l.. =, E x,),,,
[e [e- -1,1_
5

---

Under review as a conference paper at ICLR 2022
Algorithm ContraLOORD main learning algorithm
Input: generator G(.), encoder f(.), momentum encoders fom (-), (-), heads ha(.). kg(())
1: Initialize the parameters G(-), [() ( (.), f' (-), (a)))a(())))))) ha(-), (n(())
2: Initialize the embeddings d; g., E {1,2) ,
,n]
3: # Embedding Optimization
4: for each epoch do
5: Apply G(-) reconstruct original images
6: Calculate the optimization loss in Eq.
7: Update d g.
8: # Encoder Pre-training
9:
for each epoch do
10:
Apply .()), ha(-), (z(-)) to reconstruct original images and calculate the loss in Eq.
Apply embeddings df as supervision and calculate the loss in Eq.
12:
Apply f(-), fm (-) to encode content features By gi and calculate loss as in Eq. g% and calculate loss as in Eq.
13:
Apply f(-), (-) to residual features & Compute the total loss Eq.
15:
Update the parameters of f(.), (-). h.(.)
Update the momentum parameters Jm (.) ()) Update the content and residual momentum queue
17:
Output: f(-), ha)),a()). (-)-)
EXPERIMENTS
4.1 DATASETS & CONFIGURATIONS
Following previous methods (He et al.
2020 Chen et al. 2020a). evaluate the linear classifi- cation of the encoder pre-traine dreitrained by our ContraLORD on three widely-used benchmarks, including CIFAR-10, CIFAR 100, ImageNet-100 100 Deng et al. 2009 Tian et al.) 2020). In terms of disentangle-
ment Gabbay & Hoshen
2020
2021). we evaluate the disentangled embeddings on four synthetic:
Shapes3D (Kim & Mnih 2018). Cars3D (Reed al. 2013), dSprites (Higgins et al.) (007)) Small-
Norb (LeCun et al.
2004]; And three real datasets: CelebA (Liu et al., 2015), AFHQ (Choi et al.,
2020 CelebA-HQ Karras al. 2018).
Specifically, Shapes3I (Kim & Mnih 2018) contains shapes, scales, 15 orientations, 10 floor colors, 10 wall colors, and 10 object colors. Cars3D (Reed al. 2015) includes 183 car CAD mod- els with 24 different azimuth directions and elevations, where 163 models are for training and 20 for testing. dSprites (Higgins et al., 2017) contains shapes, scales, 40 orientations, 32 x posi- tions, and 32 positions. Norb (LeCun et al. 2004) consists of 50 toys with 5 generic classes, lighting conditions, elevations, and 18 azimuths. CelebA (Liu al.) 2015) includes 10.177 celebrities, in total 202,599 images, where we use 9,177 images for training and 1,000 images for testing. AFHQ (Choi al.) (020) an animal face dataset with 15,000 high-quality images of three categories: cat, dog, and wildlife. CelebA-HQ (Karras et al. 2018) contains 30,000 high-qualit images from CelebA with gender as the class, and masks are used for the hairstyle disentanglement.
For fair comparison, we follow the same setting as previous work (Gabbay & Hoshen 2020 202
During embedding optimization, we set d=256, g=128, K=12800, and 0=0001.. The generator is optimized by Adam (Kingma & Ba 2014) optimizer with learning rate of 0.0001. We train the encoder with a learning rate of 0.001. For the regularized Gaussian noise, we set a=1. For encoder pre-training, we closely follow MoCo (He al. 2020) and use the same data augmentation. For the encoder networks, we experiment with the commonly used encoder architecture, ResNet-50. We train batch size 256 for 200 epochs.
4.2 EVALUATION METRICS
For evaluation metrics, t,p-1, t accuracy for linear classification. In terms of evaluating the disentangled embeddings, we use three mainly-used metrics in the literature: DCI (Eastwood & Williams 2018), SAP score (Kumar al. 2018), and MIG (Chen et al. 2018). DCI measures

---

Under review as a conference paper at ICLR 2022
the disentanglement, completeness, and informativeness of the generative embeddings. SAP score refers to a separated attribute predictability score that captures one generative factor in only one disentangled dimension. MIG is the mutual information gap to calculate the difference between the top two latent factors with the highest mutual information. In the meanwhile, we follow previous works (Gabbay & Hoshen 2020 202T) and evaluate our ContraL.ODD on FID and LPIPS. FIDD measures how the disentangled embeddings are translated to the target domain, while LPIPS is used for calculating the quality of transferred content embeddings in terms of perceptual similarity.
4.3 EXPERIMENTAL RESULTS
In this part, we conduct extensive experiments to evaluate the discriminative and disentangled em- beddings learned by our ContraLORD, which demonstrates the advantage of approach against previous work (Gabbay & Hoshen 2020 202I) to learn discriminative and disentangled represen- tations via content-level and residual-lvve contrastive loss.
Evaluation of discriminative embeddings. We evaluate the quality of the discriminative embed- dings on linear classification. Specifically, we train the linear model on frozen features from var- ious self-supervised methods and report the experimental results in Table 1 Our Contral.ORD substantially outperforms baselines (Gabbay & Hoshen 2020; 202T) terms of top-1 and top-5 accuracy all benchmarks. Particularly, achieve the performance gain over LORD (Gabbay & Hoshen 2020) by 8.88% 9,35%, 8.91%. the meanwhile, we surpass a concurrent work (Gabbay & Hoshen 2021) using the class embeddings with higher dimension. This demonstrates the supe- riority of our ContraL.RD incorporating content-level and residual-level contrastive learning into latent optimization. Furthermore, our ContraLORD outperforms the pure contrastive self-supervised methods (He et al., 22200] Chen et al.) 2020a), which also validates the effectiveness of latent opti- mization in learning more generalized and discriminative embeddings for linear classification.
Table 1: Comparison results of linear classification on CIFAR-10, CIFAR- 100, ImageNet-100, and TinllmageNet2200 datasets.
Dataset
Method
Arch.
Epochs
Top-1
Top-5
MoCo
Re
200
93.30
99.85
SimCLR
Re
200
92.00
99.81
CIFAR-10
LORD
RcsNet-50
200
85.13
96.22
OverLORD
ResNet-50
200
91.62
98.61
ContraLORD (ours)
ResNet-50
200
94.01
99.89
MoCo
ResNet-50
200
71.70
90.23
SimCLR
ResNet-50
200
71.58
90.11
CIFAR-100
LORD
5e
200
63.32
87.05
OverLORD
ResNet-50
200
69.96
89.53
Contral .ORD (ours)
ResNet-50
200
72.67
90.97
CMC
ResNet-50
200
66.20
88.75
MoCo
ResNet-50
200
72.80
91.04
ImageNet-100 100
LORD
50
200
67.32
89.26
OverLOORD
ResNet-50
200
70.
90.45
ContraL-ORD (ours)
5esNet-50
200
76.23
92.52
Evaluation of disentangled embeddings. Following existing work (Gabbay & Hoshen 2020
2021) we evaluate the disentanglement performance of disentangled embeddings with 100 labels on four synthetic datasets. Table 2ablee2repooe reports the comparison results. As can be seen, our ContraLORD still outperforms existing methods in terms of all metrics, including DCI. SAP, and MIG. This shows that our Contral-ORD with the content-level and residual-level contrastive loss is superior to learn more disentangled information in the disentangled embeddings. In the meanwhile, we follow the setting in OverLORD (Gabbay & Hoshen 2021) and conduct experiments three real benchmarks in Table We can observe that our ContraLORD achieves the best performance in terms 5 out of 7 evaluation metrics. For the other two metries, we still achieve comparable results when compared to OverLORD (Gabbay & Hoshen 2021). These results further validate the effectiveness of our ContraLORD in learning disentangled representations with more disentangled information.
Smoothness of content and residual embeddings. We simultaneously measure the smoothness score of content, and residual embeddings pre-trained by the encoder and report the results in the
7

---

Under review as a conference paper at ICLR 2022
Table 2: Disentanglement performance Shapes3D, Cars3D, dSprites, and SmallNorb datasets.
Dataset
Method
D (1)
(()
I(t)
SAP (t)
MIG (*)
Smoothness (t)
I Locatello ef al.
0.03
0.03
0.22
0.01
0.02
0.15
LORD
0.54
0.54
0.54
0.15
0.43
0.448
Sh
Gabbuy et al.
1.00
1.00
1.00
0.30
0.96
0.82
ContraLORD (ours)
1.00
1.00
1.00
0.42
1.00
0.96
Locatello al.
0.11
0.17
0,06
0.16
LORD
0.26
0.48
0.13
0.27
Cars3D
Gabbay et al.
0.40
0.41
0.56
0.15
0.35
0.33
ContraLORD (ours)
0.51
0.56
0.71
0.25
0.41
0.45
Locatello et al.
0.01
0.01
0.16
0.01
0.01
0.12
LORD
0.16
0.17
0.43
0.03
0.06
0.18
dSprites
Gabbay et al.
0.75
0.75
0.68
0.13
0.48
0.52
ContraLORD (ours)
0.85
0.84
0.79
0.24
0.62
0.67
Locatello al.
0.01
LORD
0.01
0.17
SmallNorb
Gabbay et al.
0.27
0.39
0.45
0.14
0.27
0..29
Contral.ORi (ours)
0.36
0.51
0.56
0,26
0.42
0.48
Table 3: Disentanglement performance on CelebA, AFHQ, and CelebA-HQ datasets.
CelebA
AFHQ
CelehA-HQ
Method
Id (1)
Exp (4)
Pose (1)
HID (L)
I.PISS (f)
HID (F2M,.) HID (M2F.)
LORD
0.48
3.2
3.5
97.1
OverLORD
0.63
2.7
2.5
16.5
0,51
54.0
42,9
ContraLORD (ours)
0.61
2.6
2.3
15.8
0,53
54.2
42.6
last column of Table We can observe that our ContraLORD outperforms existing methods by a large margin (0.14, 12, 0.15, 0.19) on all four benchmarks in terms of the smoothness score, which shows the advantage our ContraLORD uniformly distributed on the hyper-sphere. In the meanwhile, our smoothness score is positively cor- related with the previous disentanglement metrics. This demonstrates the effectiveness of learning uniformly distributed embeddings of disentangled information for representations disentanglement.
learning disentangled embeddings that are more
ABLATION STUDY
In this section, we perform comprehensive ablation studies to explore the effect of each loss (Crees Loup: batch size, and the number of negative samples (K) on the final perfor- mance our ContraLORD. Unless specified, we conduct all ablation studies on the Shapes3D dataset.
Table 4: Ablation study on each loss.
Lvee
Csup
Leon
Lres
(
Ct)
(
SAP (1)
MIG (t)
Smoothness (t)
X
X
X
X
0.02
0.01
0.13
0.01
0.01
0.09
/
X
X
X
0.26
0.29
0.31
0.08
0.15
0.22
1
/
0.54
0.54
0.54
0.15
0.42
0.48
/
0.89
0.88
0.37
0.85
0.75
d
1.00
1.00
1.00
0.42
1.00
0.96
Effect of each loss. To explore how each proposed loss affects the final performance of our method, we ablate each module on the final loss and show the disentanglement results in Table 4 Without four losses in the encoder pre-training stage, we achieve the worst performance. Adding Loup to only Cree boosts the results by 0.24, 0.28, 0.18, 0,07, 0.14, and 0.13. By combining Lcon with Loup and Lrecs we achieve performance gain of 0.37, 0.35, 0.34, 0.22, 0.43, and 0.27. These results demonstrate the effectiveness of our content-level and residual-level loss in learning disentangled embeddings. Finally, ContraL.ORD) with all losses, achieves the best performance in terms of
8

---

Under review as a conference paper at ICLR 2022
all disentanglement metrics and the smoothness score, which validates the rationality of each loss on learning disentangled representations.
Effect of batch size. Table 5 reports the exploration study results of batch size. Specifically, we vary the batch size from 16, 32, 64, 128, 256, 512 during the encoder pre- pre-training stage. From the results, can observe that approach performs the best when the batch size is 512. With smaller batch size of 256, our ContraLOORD does not have a large performance decline (0.03, 0.01) in terms of SAP score and Smoothness score. When the batch size is set to 32, our method has obvious performance decrease, which shows the importance of suitable batch size in our content-level and residual-leve contrastive loss by introducing negative samples across the same batch. When we increased the batch size to 1024, the performance of our approach on all disentanglement metrics and the smoothness score is deteriorated by the confusion of too many negative samples in the same batch.
Table 5: Ablation study on the bath size.
Batch Size
D (f)
C (t)
(tt
SAP (1)
MIG (1)
Smoothness (1)
32
0.82
0.81
0.79
0.36
0.82
0.72
0.89
0.87
0.86
0.38
0.88
0.79
128
0.93
0.91
0.91
0.39
0.91
0.85
256
1.00
1.00
1.00
0.42
1.00
0.96
512
1.00
1.00
1.00
0.45
1.00
0.97
1024
0.99
0.98
0.98
0.41
0.99
0.93
Effect of negative samples. In order explore the effect of negative samples the final perfor- mance of our Contral.ORID, we vary the number of negative samples from 1600, 3200, 6400, 12800, 25600, 51200 for the content and residual momentum queue. We show the experimental results in Table 6 As can be seen, with the increase of the number of negative samples in the momentum queue, our ContraLORD achieves better performance negative samples, i.e., large number of negative samples, degrades the performance of our ap- proach since it hard for the content-level and residual-level contrastive loss discriminate hard negative samples in the momentum queue with many negative samples. This further demonstrates the importance of negative samples in learning generative embeddings with disentangled informa- tion.
terms of all metrics. However, too many
Table 6: Ablation study on the number of negative samples.
K
D (t)
(t)
((t)
SAP (?)
MIG (f)
Smoothness (1)
1600
0.62
0.63
0.61
0.26
0.61
0.52
3200
0.81
0.79
0.77
0.33
0.79
0.68
6400
0.88
0.86
0.87
0.37
0.87
0.77
12800
1.00
1.00
1.00
0.42
1.00
0.96
25600
0.96
0.95
0.95
0.41
0.95
0.89
51200
0.87
0.85
0.85
0.36
0.86
0.75
6 CONCLUSION
In this work, we propose the ContraLORD, a simple yet effective approach by incorporating con- trastive learning into latent optimization for representation disentanglement. Specifically, we first use generator to learn discriminative and disentangled embeddings via latent optimization. Then an encoder and two momentum encoders are applied to dynamically learn disentangled information across a large number of samples with content-level and residual-level contrastive losses. Finally, we tune the encoder with the learned embeddings in an amortized manner. We conduct extensive experiments on ten benchmarks to demonstrate the effectiveness of our ContraLORD on learning disentangled representations. Comprehensive ablation studies also validate the rationality of each contrastive loss proposed in approach. We also empirically observe the importance of nega- tive samples across a large number of samples in learning generative embeddings with disentangled information.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Albert P. Bartók and Gábor Csányi Gaussian Approximation Potentials: a brief tutorial introduction.
arXiv preprint arXiv: 1.X.v:500..11. 366, 2015.
Ricky Q. Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2018.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference Machine Learning (ICML.), 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVE
Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint NoXiv:2000.04229,, 2020c.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia. Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Seale Hierarchical Image Database. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrattiv learning. 2020.
Emily L. Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation disen- tangled representations. In Proceedings of International Conference on Learning Representations (ICLR), 2018
Aviv Gabbay and Yedid Hoshen. Demystifying inter-class disentanglement. In Proceedings of International Conference on Learning Representations (ICLR), 2020.
Aviv Gabbay and Yedid Hoshen. Scaling-up disentanglement for image translation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Aviv Gabbay, Niv Cohen, and Yedid Hoshen. An image is worth more than a thousand words: To- wards disentanglemenn in the wild. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of Advvacces in Neural Information Processing Systems (NeurIPS), 2014.
Baan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukeuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Proceedings of Advances in Newral Information Processing Systems (NeurIPS), 2020.
Naama Hadad, Lior Wolf, and Shimon Shahar. two-step disentanglement method. pp. 772-780, 2018.
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9733 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework, In Proceedings of International Conference on Learn- ing Representations (ICIR), 2017.
Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian Reid. Deep subspace clustering networks. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of International Conference Learning Representations (ICLR), 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In Proceedings of the 35th Interna- fional Conference on Machine Learning (ICML), 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6980 2014.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2014.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. In Proceedings of International Conference on Learning Representations (ICLR), 2018
Y. LeCun, Fu Jie Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. I1-104 Vol.2, 2004.
José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Olé: Orthogonal low-rank embedding, plug and play geometric loss for deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8109-8118, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face autributes in the wild. In Proceedings of the International Conference on Computer Vision (ICCV), 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. In Proceedings of International Conference on Machine Learning (ICML), 2019.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In Proceedings of Interna- tional Conference on Learning Representations (ICLR), 2020.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann André LeCun. Disentangling factors of variation in deep representation using adversarial training. In Proceed- ings of Advances in Neural Information Processing Systems (NeurIPS), 2016.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Utkarsh Ojha, Krishna Kumar Singh, Cho-Jui Hsieh, and Yong Jae Lee. Elastic-infogan: Unsuper- vised disentangled representation learning in imbalanced data. 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint 103i::2033000020 2021.
11

---

Under review as a conference paper at ICLR 2022
Ali Razavi, Aäron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint onXiv:I906 00446, 2019.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In Pro- ceedings of Advances in Neural Information Processing Systems (NeurIPS). 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In Proceedings of International Conference on Machine Learning (ICML), 2020.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 2020.
Junjian Zhang, Chun-Guang Li, Chong You, Xianbiao Qi, Honggang Zhang, Jun Guo, and Zhouchen Lin. Self-supervised convolutional subspace clustering network. In Proceedings of IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5468-5477, 2019.
Tong Zhang, Pan Ji, Mehrtash Harandi, Richard I. Hartley and lan D. Reid. Scalable deep k- subspace clustering. In Proceedings of the Asian Conference on Computer Vision (ACCV), pp- 466-481, 2018.
Pan Zhou, Yunqing Hou, and Jiashi Feng. Deep adversarial subspace clustering. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1596-1604, 2018.
12


Reviews:
[{"review_id": "yPZ9MIDv25", "reviewer": "Reviewer_kr5R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The experiment results are convincing. The proposed method outperforms strong baselines and obtains state-of-the-art performance. The ablation study is comprehensive and clearly shows the empirical improvement brought by the proposed techniques.\n\nWeaknesses:\n1. The proposed method is a simple combination of LORD followed by contrastive representation learning. The technical novelty is limited.\n2. The motivation of using contrastive learning to learn disentangled representation is unclear. The goal of contrastive learning is to learn embeddings that are invariant to certain data augmentations, while the goal of disentanglement is to learn embeddings that are invariant to label-unrelated generative factors. The data augmentations are not necessarily label-preserving. Also, there exist label-unrelated variations not captured by data augmentations. It is not clear to me why contrastive learning helps disentanglement.\n3. The presentation is not very clear and I find it hard to understand some parts of the algorithm. Section 3.3 does not mention any usage of data augmentation during encoder pre-training, while section 4 states the same data augmentation as in MoCo is used. Authors should clarify what data augmentations are applied to which inputs during the encoder pre-training. Besides, it seems the content embedding and the residual embedding are trained with the same objective. It is not clear to me how they learn different information. The purpose of having three embeddings instead of two (as in LORD) is also not well explained. \n\nMinor comments that do not affect the rating:\n1. Section 3.1 \"discriminative embeddings d\" should be \"d_i\"\n2. \"The InfoNCE loss, that is, the normalized temperature-scaled cross-entropy loss, are \" -> \"is\"", "summary_of_the_paper": "The paper proposes a new two-step approach for learning disentangled representation. In the first step, a generator is learned to map latent embeddings to reconstruct images. Similar to LORD, the embedding corresponding to each image is jointly optimized with the generator weights. The second step uses contrastive learning together with regression loss to learn an encoder that maps images to the latent space, which is the main contribution of the paper. Results show that the proposed approach achieves state-of-the-art performance on several datasets for disentangled representation learning.", "summary_of_the_review": "Despite strong empirical performance, I am incline towards rejection due to the limited novelty and unclear motivation. The presentation can be improved as well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nMa7zP1XapX", "reviewer": "Reviewer_L4fH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and describes an interesting idea of augmenting MSE training with contrastive losses to improve the distilled latent representations of an encoder.\n\nThe experiments are quite extensive and include both synthetic and real-world datasets.\n\nAdditionally, I have a minor concern regarding the design of the model. The authors provide two types of learned continuous embeddings: content and residual, but from the text of the paper, I could not deduce the difference between these two variables, since they are treated equally throughout the optimization process and in the model itself. Therefore I would like the authors to clarify, what is the difference between these two variables.\n\nThe idea of using the two types of embeddings is not new and comes from another paper, which the authors site, but in this previous work, the two groups of variables are treated distinctly differently (namely, content embeddings do not have noise added to them, which allows learning representations correlated to the class labels).\n\nNow to the main weakness.\n\nThe comparison with [4] is presented in Table 4 has mixed results and, to the best of my knowledge, no discussion within the paper. Does it seem like the proposed method is not directly superior to [4] when applied to real-world data? Why does the proposed method outperform [4] on synthetic benchmarks, but not on all real-world benchmarks?\n\nAlso, I consider the contribution of adding contrastive losses into the training of discriminative models to be quite minor. In the paper by Ojha et al. [1] the exact same idea was used to improve disentanglement using the InfoGAN base model. Yes, the authors of the paper under review use a different version of contrastive loss, but their exact realization is already a prior work and was described in a paper by He et al. [2].\n\nIt seems like the whole idea of the paper itself is the following: replacing disentangled learning part in paper [1] with [3, 4], and replacing the contrastive learning part with [2]. I would like the authors to describe their contributions more clearly and correct me if my impression is wrong.\n\n\n\n[1] Utkarsh Ojha et al., \"Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data\"\n[2] He et al., \"Momentum Contrast for Unsupervised Visual Representation Learning\"\n[3] Gabbay et al., \"DEMYSTIFYING INTER-CLASS DISENTANGLEMENT\"\n[4] Gabbay et al., \"Scaling-up Disentanglement for Image Translation\"", "summary_of_the_paper": "The authors describe a method for learning disentangled representations (categorical and continuous). It is based on the generative latent optimization scheme for learning disentangled latent codes for a given dataset, followed by training an encoder network to predict these latent codes given an arbitrary image.\n\nThe main contribution of the authors is a proposition to include contrastive losses into the encoder training to prevent it from converging to a suboptimal state where it could predict similar embedding to dissimilar input images. They validate the resulting model using standard image generation and disentanglement benchmarks across multiple datasets.", "summary_of_the_review": "While the idea describes in the paper is interesting, and the evaluation quite comprehensive, it seems like the essence of the authors' contribution itself is quite minor, and at the moment I am not convinced that it passes the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aorcXU4Ihiy", "reviewer": "Reviewer_Ex2Y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The experiments successfully demonstrated that contrastive learning is somehow very useful in improving disentangled and discriminative representation learning. The improvement in terms of disentanglement learning in both synthetic and real datasets seems significant.\n2. The ablation studies were helpful to convince that contrastive learning on encoders is critical in improving the disentanglement learning performance. Yet, it is still unclear how and why it improves the performance (see the comments below).\n\nWeakness:\n1. Despite the impressive results, the technical contribution of this work is still quite incremental. It basically follows exactly the same framework of (Over)LORD (latent optimization followed by the encoder training) but combines MoCo in the encoder training for disentangled factors. It is an interesting observation that this simple combination can lead to significant improvement in disentanglement learning, but the paper could be stretched in many ways instead of a naive combination (e.g., how latent optimization and contrastive learning can mutually improve each other). \n\n2. The writing is somewhat unclear and not self-contained. For instance, (1) I assume that the paper follows the same problem setting of (Over)LORD where the class supervision is given. However, it is never mentioned clearly in the paper. It could be even more confusing since the discriminative embedding is indexed by the data instance, not clearly showing that instances in the same class share the same discriminative embedding. (2) Section 3.2 is not self-contained. The authors can describe why Eq.(1) can lead to disentanglement learning. (3) The authors did not mention if the generator parameters are trained in the first and/or second stages end-to-end. (4) It is unclear how the authors choose the positive pair $g^c_i$ in Eq.(4) and (5). Overall, I felt that the paper needs significant work to make descriptions clear and self-contained. \n\n3. The objective of (Over)LORD and contrastive learning seem conflicting in some ways. (Over)LORD disentangles factors by constraining information within the latents (g) via Gaussian noise (Eq.(1)). On the other hand, contrastive learning encourages the same latent to be maximally informative for instance discrimination, which aids better generalization. A naive combination of two objectives seems to be invasive to each other, and it is unclear why it leads to improvement. It would be great to hear the authors’ thoughts on this.  \n\n4. It is not crystal clear the source of improvement in the encoders; the ablation study demonstrated that the improvement is mainly from the contrastive loss, but it is unclear if (1) the contrastive loss simply improves the generalization of the encoders in unseen images or (2) it also helps to discover disentangled factors in training data. My thought is that the latter is less likely since it shares nearly the same latent embeddings with (Over)LORD to provide supervision to the encoders (Eq.(3)). Evaluation of disentanglement learning in training/testing data could give us more insights. It would be great if the authors also share a more in-depth analysis of the source of improvement in the rebuttal.\n\n5. In Table 1, comparisons to MoCO and SinCLR are unfair since they are self-supervised while the proposed method is based on supervised learning (i.e., trained with class labels). It should be clearly mentioned in the paper. \n\n6. Qualitative analysis/comparisons could be very helpful but are missing in the paper. For instance, comparisons with baselines such as disentanglement performance, generation quality/diversity, etc. could be very useful.\n", "summary_of_the_paper": "This paper proposed to improve the generative disentangled representation learning of LORD and OverLORD with contrastive learning. The idea is simple: when learning the amortized encoders in LORD and OverLORD, regularize them using contrastive learning to improve the generalization. Despite the simplicity, the authors demonstrated that this leads to a significant improvement in the disentanglement of representation.", "summary_of_the_review": "Overall, I was impressed that simple contrastive learning can improve disentanglement learning. However, the paper did not go much deeper into why and how it improves the performance, and lack some technical novelty. The presentation should be improved in many ways too. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0wZEfDRU6dh", "reviewer": "Reviewer_Th2p", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper is well-written and is easy to follow and it addresses an important problem related to representation learning for generative models which is a popular topic in the community.\n\n- The results validate the efficacy of the proposed method on many datasets/metrics.\n\n- The contrastive learning method is interesting and is well-motivated in the sense that it is not surprising that it should help improve the latent representation, and I applaud the authors for making it work for the latent representation learning of a generative model.\n\nCons:\n- The algorithm has many moving components such as generators, encoders, momentum encoders, and heads interacting in ways that seem complicated. Complicated methods are more difficult to reproduce and to use for new problem domains or build on for future research.\n \n- Further, no code is provided to understand the implementation or help in reproducing/verifying the results. \n\n- The novelty is slightly limited since it is direct integration of contrastive learning to OverLORD, which already performs latent optimization with disentanglement.\n\n- Getting 100% performance in Table 4 by just adding the residual-level loss seems suspicious, can the authors explain that this result is not an anomaly?\n\n- No multiple runs are shown to display the standard deviation and the mean result across random initializations. This result is important as it allows us to see if the reported performance is significant and robust to different seeds.  \n\n- How long did the training take? and how did the authors perform early stopping? \n\n- No robustness test showing how the weight coefficients affect the residual and content-level loss terms.", "summary_of_the_paper": "The authors propose a new method called ContraLord for the task of representation disentanglement for generative models. Their method extends LORD, a latent optimization method for improving content embeddings, and OverLORD which performs disentanglement on embeddings, by incorporating contrastive learning. Specifically, their method includes content-level and residual-level contrastive loss that leverage negative examples in a batch.  Their experimental results show promising results on standard datasets like CIFAR and ImageNet in terms of classification performance and disentanglement. The authors also included interesting ablations to showcase the efficacy of each loss term like the residual-level contrastive loss.", "summary_of_the_review": "I recommend an accept since the method seems sound, well-motivated and reasonable and the results seem promising in difficult datasets. However, the novelty of this work is slightly lacking since it is a straightforward integration of a contrastive learning method (which have been a common theme recently) to OverLORD, the cited paper that performs latent optimization with disentanglement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents modifying latent optimization for representation disentanglement using contrastive learning, resulting in improved performance on disentanglement benchmarks. Despite the empirical success, the proposed algorithm has many moving parts and loss functions. Most reviewers agree that given the incremental and complex nature of the proposed technique, the empirical results are not sufficient for acceptance at ICLR, especially since the results do not present additional insights into the inner workings of the method. I encourage the authors to try to simplify the technique, or provide a convincing evidence that such complexity is necessary.

PS:
I didn't find much discussion of how the hyper-parameters are chosen (temperature, lambda terms, etc.).
A discussion of recent self-supervised disentanglement methods (e.g., https://arxiv.org/abs/2102.08850 and https://arxiv.org/abs/2007.00810) can be helpful.
Decision: Reject