Year: 2021

Abstract:
Contrastive learning has shown its effectiveness in image classification and generation. Recent works apply contrastive learning on the discriminator of the Generative Adversarial Networks. However, there is little work exploring if contrastive learning can be applied to the encoders-decoder structure to learn disentangled representations. In this work, we propose a simple yet effective method via incorporating contrastive learning into latent optimization, where we name it ContraLORD. Specifically, we first use a generator to learn discriminative and disentangled embeddings via latent optimization. Then an encoder and two momentum encoders are applied to dynamically learn disentangled information across a large number of samples with content-level and residual-level contrastive loss. In the meanwhile, we tune the encoder with the learned embeddings in an amortized manner. We evaluate our approach on ten benchmarks in terms of representation disentanglement and linear classification. Extensive experiments demonstrate the effectiveness of our ContraLORD on learning both discriminative and generative representations.

heading: 1 INTRODUCTION
text: In recent years, disentanglement of factors in images has attracted many researchers’ attention, which mainly includes two folds: adversarial and non-adversarial methods. Adversarial methods (Mathieu et al., 2016; Denton & Birodkar, 2017; Hadad et al., 2018; Razavi et al., 2019; Gabbay & Hoshen, 2021) often apply a min-max optimization framework (Goodfellow et al., 2014) for disentanglement of images, which costs much time on hyper-parameters tuning. In terms of non-adversarial models, several variational autoencoders (Higgins et al., 2017; Kim & Mnih, 2018) variants have been proposed to disentangle the generative factors in an unsupervised manner without inductive biases, which did not achieve satisfactory results as proven in an empirical study (Locatello et al., 2019).
With the extra class supervision, semi-supervised methods achieve promising performance in disentanglement. Typically, comprehensive experiments in (Locatello et al., 2020) validate the effectiveness of a limited amount of supervision in state-of-the-art unsupervised disentanglement models. LORD (Gabbay & Hoshen, 2020) applies a latent optimization framework with a noise regularizer on content embeddings to achieve superior performance over amortized inference. Based on LORD, OverLORD (Gabbay & Hoshen, 2021) is proposed to disentangle class, correlated and uncorrelated attributes for image translation. A more recent work (Gabbay et al., 2021) adopts a pre-trained CLIP (Radford et al., 2021) model to generate partial annotations for image manipulation. However, there exist two main drawbacks among these methods: 1) using different separate encoders for different factors is resource-wasteful for real-world applications and requires expensive human design. 2) just learning the content embeddings inside each sample is not sufficient to learn the diversity existing in the dataset.
Driven by the shortcomings discussed above, we propose a simple yet effective method named ContraLORD, where we incorporate contrastive learning into latent optimization for representation disentanglement. Recent works (Deng et al., 2020; Ojha et al., 2020) apply the contrastive learning on the discriminator of the GAN (Goodfellow et al., 2014) for disentangling representations. Typically, the 3D imitative-contrastive learning in (Deng et al., 2020) is used for controllable face image generation by comparing pairs of generated images. However, in this work, we focus on applying
contrastive learning on the encoders to learn the discriminative and generative embeddings with disentangled information. Specifically, we first apply a generator to learn discriminative and generative embeddings via latent optimization. Then we apply an encoder and a momentum encoder to dynamically learn disentangled information across a large number of samples with content-level and residual-level contrastive loss. In the meanwhile, we use the learned discriminative and generative embeddings to tune the encoder in an amortized manner.
We evaluate our ContraLORD on two main tasks: linear classification and disentanglement. Extensive experiments show the effectiveness of the learned discriminative embeddings on linear classification and generative embeddings on the disentanglement of factors. We conduct comprehensive studies on three benchmarks on linear classification and seven benchmarks on disentanglement to investigate if contrastive self-supervised models can learn disentangled features. In the meantime, we achieve superior performance on linear classification compared to baselines. Our ContraLORD also achieves promising results over state-of-the-art methods in terms of disentanglement.
The main contributions of this work can be summarized as follows:
• We present a simple yet effective method called ContraLORD by incorporating contrastive learning into latent optimization for representation disentanglement and linear classification.
• We formally explore the disentangled features across a large number of samples with content-level and residual-level contrastive losses.
• Extensive experiments on ten benchmarks show the effectiveness of our approach on learning disentangled representations.

heading: 2 RELATED WORK
text: Discriminative representations learning. Discriminative representations learning has addressed researchers’ attention for a long time since discriminative representations are significant for image classification. Most of the previous works adopt supervised (Lezama et al., 2018) and unsupervised learning (Ji et al., 2017; Zhang et al., 2018; Zhou et al., 2018; Zhang et al., 2019) to learn embeddings that most discriminate between classes in the dataset. Typically, the principle of maximal coding rate reduction (Yu et al., 2020) is applied to maximize the coding rate difference between the whole dataset and the sum of each separate class. However, there exists little work of contrastive learning to explore the discriminative representations for the pre-training stage. In this work, we mainly focus on learning discriminative embeddings for linear classification by incorporating contrastive learning into latent optimization to improve the performance of baselines.
Disentangled Representations Learning. Disentangled representation learning aims at learning generative factors existing in the dataset, that is, disentanglement learning. A bunch of previous work focuses on unsupervised learning with variational autoencoders, such as β-VAE (Higgins et al., 2017), Factor-VAE (Kim & Mnih, 2018). Following those work, DCI disentanglement (Eastwood & Williams, 2018), SAP score (Kumar et al., 2018), and Mutual Information Gap (MIG) (Chen et al., 2018) are often utilized as quantitative metrics to measure the quality of disentangled representations. In recent years, semi-supervised models have been used by many researchers in the literature. Adding a limited amount of supervision to unsupervised models is proven in (Locatello et al., 2020) to be effective in learning disentangled representations for real-world scenarios. LORD (Gabbay & Hoshen, 2020) leverages the latent optimization framework with a noise regularizer on content embeddings for class and content disentanglement. More recently, a simple framework for disentangling labeled and unlabeled attributes is utilized in OverLORD (Gabbay & Hoshen, 2021) for high-fidelity images synthesis. A study (Gabbay et al., 2021) uses a CLIP (Radford et al., 2021) pre-trained model to annotate a set of attributes for disentangled image manipulation. In this work, we intend to learn disentangled embeddings via combining latent optimization and contrastive selfsupervised learning.
Contrastive Learning. Recently, contrastive self-supervised learning (Tian et al., 2020; Chen et al., 2020a;b; Grill et al., 2020; He et al., 2020; Chen et al., 2020c; Chen & He, 2021) has been explored a lot by many effective methods. SimCLR (Chen et al., 2020a), an end-to-end structure, is proposed to pull away from the features of each instance from those of all other instances in the training set. In the self-supervised setting, low-level image transformations such as cropping, scaling, and color
jittering are utilized for encoding the in-variances from samples. The InfoNCE loss, that is, the normalized temperature-scaled cross-entropy loss, are often optimized to maximize the similarity between positive samples and minimize the similarity between negative samples. Large batch size is always used in this kind of end-to-end structure (Chen et al., 2020a;b) to accumulate a large bunch of negative samples in the contrastive loss. PIRL (Misra & Maaten, 2020) without a large batch size applies a memory bank to store negative samples and update representations at a specified stage. MoCo (He et al., 2020) and MoCov2 (Chen et al., 2020c) replace the memory bank with a memory encoder to queue new batch samples and to dequeue the oldest batch. In this work, we leverage content-level and residual-level momentum encoders to store a queue of negative samples with disentangled information for learning generative embeddings, where content-level and residuallevel contrastive losses are applied to capture content and residual representations.

heading: 3 METHODOLOGY

heading: 3.1 PROBLEM SETUP
text: In this part, we first begin with the problem setup, and formally define the notations for easy reading. In terms of the problem, our goal is to demystify the disentangled and discriminative features learned by contrastive self-supervised learning. To address this problem, we propose a simple yet effective method by combining contrastive learning and latent optimization together for representation disentanglement. To explain it better, we define the notations below in a unified manner.
Notations. Given a set of training examples X = {x1,x2, · · · ,xn}. For each image xi, i ∈ {1, 2, · · · , n}, we need to learn one discriminative embeddings d and a generative embeddings gi from a pre-defined set of embeddings {g1,g2, · · · ,gm}, where m denotes the total number of generative factors in the training data. That is, di ∈ R1×d,gi ∈ R1×g , where d, g denote the dimensionality of discriminative and generative embedding, separately. In our setting, we split the generative embedding gi into two folds: content embeddings gci and residual embeddings g r i . The content embeddings contain the information that is related to the discriminative embedding, while the residual embedding includes the uncorrelated information.
Overall, the objective of our work is to learn di,gci ,g r i for each image xi from a training dataset. In the next part, we present the technical details of our method. To learn di,gci ,g r i from a set of training examples, we propose a simple yet effective approach called ContraLORD, as shown in Figure 1. Our ContraLORD mainly includes two parts: 1) embedding optimization: we first use a generator G(·) to learn discriminative and disentangled embeddings via latent optimization. 2) encoder pretraining: we apply a encoder f(·) and a momentum encoder fm(·) to dynamically learn disentangled information across large amount of samples with content- and residual-level contrastive loss.

heading: 3.2 EMBEDDING OPTIMIZATION
text: In order to learn the discriminative and disentangled embeddings, we are motivated by LORD (Gabbay & Hoshen, 2020) to introduce the latent optimization in the first stage. Specifically, we apply a generator G(·) to reconstruct the original image xi by using the discriminative and disentangled embeddings of each sample. Instead of using the KL-divergence in variation auto-encoders (Kingma & Welling, 2014), we equally add a regularizer with Gaussian noise of a fixed variance to the disentangled embeddings gci and gri . Thus, the objective of embeddings optimization is defined as
Lopt = s n∑
i=1
`(G(di, gci + zi, g r i + zi)), xi) + λ · (||gci ||2 + ||gri ||2) (1)
where `(·) denotes `2 loss for synthetic data and VGG perceptual loss for real images. λ is the penalty weight of the capacity of the disentangled embeddings. zi ∼ N (0, σ2I). In this way, we can learn the disentangled embeddings di, gci , gri without any adversarial learning involved, that is, d̃i, g̃ c i , g̃ r i = argmin Lopt. For training sets with annotations, d̃i is given.

heading: 3.3 ENCODER PRE-TRAINING
text: After learning the optimized embeddings, we need to train a generalized encoder during the pretraining stage. In order to optimize the encoder f(·), we reconstruct the original image xi with the output embeddings from f(·), and the loss function is calculated as
Lrec = n∑
i=1
` ( G ( hd ( f(xi) ) , hcg ( f(xi) ) , hrg ( f(xi) )) , xi )
(2)
where `(·) denotes `2 loss for synthetic data and VGG perceptual loss for real images. hd(·), hcg(·), hrg(·) denote the head for generating the discriminative, content-level, and residual-level embeddings. To learn more disentangled information in the discriminative and disentangled embeddings, we use the learned embeddings in the first stage as the supervision and define the objective as:
Lsup = n∑
i=1
||hd(f(xi))− d̃i||2 + ||hcg(f(xi))− g̃ci || 2 + ||hrg(f(xi))− g̃ri || 2 (3)
where d̃i, g̃ci , g̃ri denote the learned representations from the first optimization stage, respectively.
Content-level Contrastive Loss. To learn more disentangled content embeddings across examples in the dataset, we feed a content momentum queue {xkey1 , x key 2 , · · · , x key k , · · · , x key K } of one query sample xqueryi into a momentum encoder fm(·). The illustration of the content-level contrastive learning is shown in Figure 2 (left). As can be seen, we generate the content embeddings gck, k ∈ {1, 2, · · · ,K} from the momentum queue, and gcq from the query sample x query i . Then we calculate the similarity between the original content embedding gci and gcq, gck for content-level contrastive loss. Finally, the content-level contrastive loss is defined as
Lcon = n∑
i=1
− log exp(gci · gcq/τ) exp(gci · gcq/τ) + ∑K k=1 exp(gci · gck/τ) (4)
where K denotes the number of negative samples in the momentum queue. τ is a temperature hyper-parameter. In the backward process, we update the parameters of the encoder f(·) according
to the gradient of this loss. The parameters of the momentum encoder f cm(·) is updated by fm(·)← mfm(·) + (1−m)f(·), where m ∈ (0, 1] is a momentum coefficient. Residual-level Contrastive Loss. In order to further disentangle the residual part in the disentangled embeddings, we introduce a residual momentum encoder frm(·) to receive the residual momentum queue and generate a set of residual embeddings grk, k ∈ {1, 2, · · · ,K}. Thus, the residual-level contrastive loss is defined with key embeddings grk, query embedding grq , and the original embedding gri as
Lres = n∑
i=1
− log exp(gri · grq/τ) exp(gri · grq/τ) + ∑K k=1 exp(gri · grk/τ) (5)
where K denotes the number of negative samples in the momentum queue, and τ is a temperature hyper-parameter. The gradient of this loss is also used to update the parameters of the encoder f(·). We update the parameters of the momentum encoder fm(·) by fm(·) ← mfm(·) + (1 − m)f(·), where m ∈ (0, 1] is a momentum coefficient. The overall objective of our ContraLORD is optimized in an end-to-end manner as
L = (Lrec + ·Lsup) + λcon · Lcon + λres · Lres (6) where λcon, λres denote the weight of the content-level and residual-level contrastive loss, respectively. Extensive ablation studies are conducted to explore the effects of each loss on the final performance of our ContraLORD. We summarize the overall algorithm of our training approach in Algorithm 1.

heading: 3.4 SMOOTHNESS OF EMBEDDINGS
text: In order to measure the smoothness of embeddings pre-trained by the encoder, we borrow the idea of uniformity property in the instance-wise contrastive learning, and introduce the Gaussian potential kernel (Bartók & Csányi, 2015) to calculate the average pairwise Gaussian potential as:
smoothness = E(gi,gj)∼pc [e −t||gi−gj ||
2
] + E(gi,gj)∼pr [e −t||gi−gj ||
2
] (7)
where pc, pr denotes the distribution of content and residual embeddings in the hyper-sphere, and t is a positive factor to define the weight of the `2 distance between embeddings gi and gj . In our experiments, we follow previous work (Wang & Isola, 2020) and set t=2.
Algorithm 1 ContraLORD main learning algorithm Input: generator G(·), encoder f(·), momentum encoders f cm(·), frm(·), heads hd(·), hg(·).
1: Initialize the parameters G(·), f(·), fcm(·), frm(·), hd(·), hg(·), 2: Initialize the embeddings di, gi, i ∈ {1, 2, · · · , n} 3: # Embedding Optimization 4: for each epoch do 5: Apply G(·) to reconstruct original images 6: Calculate the optimization loss in Eq. 1 7: Update di, gi 8: # Encoder Pre-training 9: for each epoch do
10: Apply f(·), hd(·), hg(·) to reconstruct original images and calculate the loss in Eq. 2 11: Apply embeddings d∗i , g∗i as supervision and calculate the loss in Eq. 3 12: Apply f(·), f cm(·) to encode content features gcq, gck and calculate loss as in Eq. 4 13: Apply f(·), frm(·) to residual features grq, grk and calculate loss as in Eq. 5 14: Compute the total loss in Eq. 6 15: Update the parameters of f(·), hd(·), hg(·) 16: Update the momentum parameters of f cm(·), frm(·) 17: Update the content and residual momentum queue
Output: f(·), hd(·), hg(·)

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS & CONFIGURATIONS
text: Following previous methods (He et al., 2020; Chen et al., 2020a), we evaluate the linear classification of the encoder pre-trained by our ContraLORD on three widely-used benchmarks, including CIFAR-10, CIFAR100, ImageNet-100 (Deng et al., 2009; Tian et al., 2020). In terms of disentanglement (Gabbay & Hoshen, 2020; 2021), we evaluate the disentangled embeddings on four synthetic: Shapes3D (Kim & Mnih, 2018), Cars3D (Reed et al., 2015), dSprites (Higgins et al., 2017), SmallNorb (LeCun et al., 2004); And three real datasets: CelebA (Liu et al., 2015), AFHQ (Choi et al., 2020), CelebA-HQ (Karras et al., 2018).
Specifically, Shapes3D (Kim & Mnih, 2018) contains 4 shapes, 8 scales, 15 orientations, 10 floor colors, 10 wall colors, and 10 object colors. Cars3D (Reed et al., 2015) includes 183 car CAD models with 24 different azimuth directions and 4 elevations, where 163 models are for training and 20 for testing. dSprites (Higgins et al., 2017) contains 3 shapes, 6 scales, 40 orientations, 32 x positions, and 32 y positions. SmallNorb (LeCun et al., 2004) consists of 50 toys with 5 generic classes, 6 lighting conditions, 9 elevations, and 18 azimuths. CelebA (Liu et al., 2015) includes 10,177 celebrities, in total 202,599 images, where we use 9,177 images for training and 1,000 images for testing. AFHQ (Choi et al., 2020) is an animal face dataset with 15,000 high-quality images of three categories: cat, dog, and wildlife. CelebA-HQ (Karras et al., 2018) contains 30,000 high-quality images from CelebA with gender as the class, and masks are used for the hairstyle disentanglement.
For a fair comparison, we follow the same setting as previous work (Gabbay & Hoshen, 2020; 2021). During embedding optimization, we set d=256, g=128, K=12800, and λ=0.001. The generator is optimized by Adam (Kingma & Ba, 2014) optimizer with a learning rate of 0.0001. We train the encoder with a learning rate of 0.001. For the regularized Gaussian noise, we set σ=1. For encoder pre-training, we closely follow MoCo (He et al., 2020) and use the same data augmentation. For the encoder networks, we experiment with the commonly used encoder architecture, ResNet-50. We train at batch size 256 for 200 epochs.

heading: 4.2 EVALUATION METRICS
text: For evaluation metrics, we use top-1, top-5 accuracy for linear classification. In terms of evaluating the disentangled embeddings, we use three mainly-used metrics in the literature: DCI (Eastwood & Williams, 2018), SAP score (Kumar et al., 2018), and MIG (Chen et al., 2018). DCI measures
the disentanglement, completeness, and informativeness of the generative embeddings. SAP score refers to a separated attribute predictability score that captures one generative factor in only one disentangled dimension. MIG is the mutual information gap to calculate the difference between the top two latent factors with the highest mutual information. In the meanwhile, we follow previous works (Gabbay & Hoshen, 2020; 2021) and evaluate our ContraLORD on FID and LPIPS. FID measures how the disentangled embeddings are translated to the target domain, while LPIPS is used for calculating the quality of transferred content embeddings in terms of perceptual similarity.

heading: 4.3 EXPERIMENTAL RESULTS
text: In this part, we conduct extensive experiments to evaluate the discriminative and disentangled embeddings learned by our ContraLORD, which demonstrates the advantage of our approach against previous work (Gabbay & Hoshen, 2020; 2021) to learn discriminative and disentangled representations via content-level and residual-level contrastive loss.
Evaluation of discriminative embeddings. We evaluate the quality of the discriminative embeddings on linear classification. Specifically, we train the linear model on frozen features from various self-supervised methods and report the experimental results in Table 1. Our ContraLORD substantially outperforms baselines (Gabbay & Hoshen, 2020; 2021) in terms of top-1 and top-5 accuracy on all benchmarks. Particularly, we achieve the performance gain over LORD (Gabbay & Hoshen, 2020) by 8.88%, 9.35%, 8.91%. In the meanwhile, we surpass a concurrent work (Gabbay & Hoshen, 2021) using the class embeddings with a higher dimension. This demonstrates the superiority of our ContraLORD incorporating content-level and residual-level contrastive learning into latent optimization. Furthermore, our ContraLORD outperforms the pure contrastive self-supervised methods (He et al., 2020; Chen et al., 2020a), which also validates the effectiveness of latent optimization in learning more generalized and discriminative embeddings for linear classification.
Evaluation of disentangled embeddings. Following existing work (Gabbay & Hoshen, 2020; 2021), we evaluate the disentanglement performance of disentangled embeddings with 100 labels on four synthetic datasets. Table 2 reports the comparison results. As can be seen, our ContraLORD still outperforms existing methods in terms of all metrics, including DCI, SAP, and MIG. This shows that our ContraLORD with the content-level and residual-level contrastive loss is superior to learn more disentangled information in the disentangled embeddings. In the meanwhile, we follow the setting in OverLORD (Gabbay & Hoshen, 2021) and conduct experiments on three real benchmarks in Table 3. We can observe that our ContraLORD achieves the best performance in terms of 5 out of 7 evaluation metrics. For the other two metrics, we still achieve comparable results when compared to OverLORD (Gabbay & Hoshen, 2021). These results further validate the effectiveness of our ContraLORD in learning disentangled representations with more disentangled information.
Smoothness of content and residual embeddings. We simultaneously measure the smoothness score of content, and residual embeddings pre-trained by the encoder and report the results in the
last column of Table 2. We can observe that our ContraLORD outperforms existing methods by a large margin (0.14, 0.12, 0.15, 0.19) on all four benchmarks in terms of the smoothness score, which shows the advantage of our ContraLORD on learning disentangled embeddings that are more uniformly distributed on the hyper-sphere. In the meanwhile, our smoothness score is positively correlated with the previous disentanglement metrics. This demonstrates the effectiveness of learning uniformly distributed embeddings of disentangled information for representations disentanglement.

heading: 5 ABLATION STUDY
text: In this section, we perform comprehensive ablation studies to explore the effect of each loss (Lrec,Lsup,Lcon,Lres), batch size, and the number of negative samples (K) on the final performance of our ContraLORD. Unless specified, we conduct all ablation studies on the Shapes3D dataset.
Effect of each loss. To explore how each proposed loss affects the final performance of our method, we ablate each module on the final loss and show the disentanglement results in Table 4. Without four losses in the encoder pre-training stage, we achieve the worst performance. Adding Lsup to only Lrec boosts the results by 0.24, 0.28, 0.18, 0.07, 0.14, and 0.13. By combining Lcon with Lsup and Lrec, we achieve a performance gain of 0.37, 0.35, 0.34, 0.22, 0.43, and 0.27. These results demonstrate the effectiveness of our content-level and residual-level loss in learning disentangled embeddings. Finally, our ContraLORD, with all losses, achieves the best performance in terms of
all disentanglement metrics and the smoothness score, which validates the rationality of each loss on learning disentangled representations.
Effect of batch size. Table 5 reports the exploration study results of batch size. Specifically, we vary the batch size from 16, 32, 64, 128, 256, 512 during the encoder pre-training stage. From the results, we can observe that our approach performs the best when the batch size is 512. With a smaller batch size of 256, our ContraLORD does not have a large performance decline (0.03, 0.01) in terms of SAP score and Smoothness score. When the batch size is set to 32, our method has an obvious performance decrease, which shows the importance of suitable batch size in our content-level and residual-level contrastive loss by introducing negative samples across the same batch. When we increased the batch size to 1024, the performance of our approach on all disentanglement metrics and the smoothness score is deteriorated by the confusion of too many negative samples in the same batch.
Effect of negative samples. In order to explore the effect of negative samples on the final performance of our ContraLORD, we vary the number of negative samples from 1600, 3200, 6400, 12800, 25600, 51200 for the content and residual momentum queue. We show the experimental results in Table 6. As can be seen, with the increase of the number of negative samples in the momentum queue, our ContraLORD achieves better performance in terms of all metrics. However, too many negative samples, i.e., a large number of negative samples, degrades the performance of our approach since it is hard for the content-level and residual-level contrastive loss to discriminate hard negative samples in the momentum queue with many negative samples. This further demonstrates the importance of negative samples in learning generative embeddings with disentangled information.

heading: 6 CONCLUSION
text: In this work, we propose the ContraLORD, a simple yet effective approach by incorporating contrastive learning into latent optimization for representation disentanglement. Specifically, we first use a generator to learn discriminative and disentangled embeddings via latent optimization. Then an encoder and two momentum encoders are applied to dynamically learn disentangled information across a large number of samples with content-level and residual-level contrastive losses. Finally, we tune the encoder with the learned embeddings in an amortized manner. We conduct extensive experiments on ten benchmarks to demonstrate the effectiveness of our ContraLORD on learning disentangled representations. Comprehensive ablation studies also validate the rationality of each contrastive loss proposed in our approach. We also empirically observe the importance of negative samples across a large number of samples in learning generative embeddings with disentangled information.

Review summaries:
The paper proposes a new two-step approach for learning disentangled representation. In the first step, a generator is learned to map latent embeddings to reconstruct images. Similar to LORD, the embedding corresponding to each image is jointly optimized with the generator weights. The second step uses contrastive learning together with regression loss to learn an encoder that maps images to the latent space, which is the main contribution of the paper. Results show that the proposed approach achieves state-of-the-art performance on several datasets for disentangled representation learning.

The authors describe a method for learning disentangled representations (categorical and continuous). It is based on the generative latent optimization scheme for learning disentangled latent codes for a given dataset, followed by training an encoder network to predict these latent codes given an arbitrary image.  The main contribution of the authors is a proposition to include contrastive losses into the encoder training to prevent it from converging to a suboptimal state where it could predict similar embedding to dissimilar input images. They validate the resulting model using standard image generation and disentanglement benchmarks across multiple datasets.

This paper proposed to improve the generative disentangled representation learning of LORD and OverLORD with contrastive learning. The idea is simple: when learning the amortized encoders in LORD and OverLORD, regularize them using contrastive learning to improve the generalization. Despite the simplicity, the authors demonstrated that this leads to a significant improvement in the disentanglement of representation.

The authors propose a new method called ContraLord for the task of representation disentanglement for generative models. Their method extends LORD, a latent optimization method for improving content embeddings, and OverLORD which performs disentanglement on embeddings, by incorporating contrastive learning. Specifically, their method includes content-level and residual-level contrastive loss that leverage negative examples in a batch.  Their experimental results show promising results on standard datasets like CIFAR and ImageNet in terms of classification performance and disentanglement. The authors also included interesting ablations to showcase the efficacy of each loss term like the residual-level contrastive loss.