Learning Disentangled Representation by Exploiting Pretrained Generative Models:  A Contrastive Learning View
Authors: Xuanchi Ren
Tao Yang
Yuwang Wang
Wenjun Zeng
Venue: ICLR 2022 Poster (2022)
Keywords: Latent space discovery
Disentangled representation learning
Generative models
Contrastive learning

Abstract: From the intuitive notion of disentanglement, the image variations corresponding to different generative factors should be distinct from each other, and the disentangled representation should reflect those variations with separate dimensions. To discover the generative factors and learn disentangled representation, previous methods typically leverage an extra regularization term when learning to generate realistic images. However, the term usually results in a trade-off between disentanglement and generation quality. For the generative models pretrained without any disentanglement term, the generated images show semantically meaningful variations when traversing along different directions in the latent space. Based on this observation, we argue that it is possible to mitigate the trade-off by (i) leveraging the pretrained generative models with high generation quality, (ii) focusing on discovering the traversal directions as generative factors for disentangled representation learning. To achieve this, we propose Disentaglement via Contrast (DisCo) as a framework to model the variations based on the target disentangled representations, and contrast the variations to jointly discover disentangled directions and learn disentangled representations. DisCo achieves the state-of-the-art disentangled representation learning and distinct direction discovering, given pretrained non-disentangled generative models including GAN, VAE, and Flow. Source code is at https://github.com/xrenaa/DisCo.

Published as a conference paper at ICLR 2022
LEARNING DISENTANGLED REPRESENTATION BY EXPLOITING PRETRAINED GENERATIVE MODELS: A CONTRASTIVE LEARNING VIEW
Xuanchi Ren' Yang Tao"; Yuwang Wang", Wenjun Zeng' !HKSST, 2Xi'an an Jiaotong University, Microsoft Research Asia, 'EIT 1 xrenaa@connect. .uttinneeetttttttt..... ust.hk vt14212@stu.xjtu. edu. cn yuwward@nccroooft.ccmm
3
werjuuzeengeeias.acccoo
ABSTRACT
From the intuitive notion of disentanglement, the image variations correspond- ing to different factors should be distinct from each other, and the disentangled representation should reflect those variations with separate dimensions. To dis- cover the factors and learn disentangled representation, previous methods typically leverage an extra regularization term when learning to generate realistic images. However, the term usually results in a trade-off between disentanglement and generation quality. For the generative models pretrained without any disentangle- ment term, the generated images show semantically meaningful variations when traversing along different directions in the latent space. Based on this observa- tion, we argue that it is possible mitigate the trade-off by (i) leveraging the pretrained generative models with high generation quality, (it) focusing on discov- ering the traversal directions as factors for disentangled representation learning. To achieve this, we propose Disentaglement via Contrast (DisCo) as a frame- work to model the variations based on the target disentangled representations, and contrast the variations to jointly discover disentangled directions and learn disentangled representations. DisCo achieves the state-of-the-arg arr disentangled representation learning and distinct direction discovering, given pretrained non- disentangled generative models including GAN, VAE, and Flow. Source code https: /ttps://github. wht.ub.ttuhuu.oggggo com/ http///eeaaaaiiaaaa
INTRODUCTION
Disentangled representation learning aims to identify and decompose the underlying explanatory factors hidden in the observed data, which is believed by many to be the only way understand the world for Al fundamentally (Bengio & LeCun 2007). To achieve the goal, as shown in Figure [ (a), need encoder and generator. The encoder to extract representations from images with each dimension corresponds to one factor individually. The generator (decoder) decodes the changing of each factor into different kinds of image variations.
With supervision, we can constrain each dimension of the representation only sensitive to one kind of image variation caused by changing one factor respectively. However, this kind of exhaustive supervision is often not available in real-world data. The typical unsupervised methods are based on generative model to build the above encoder and generator framework, e.g., VAE (Kingma & Welling 2014) provides encoder and generator, and GAN (Goodfellow et al. 20144] Miyato et al. 2018] Karras al. 2019) provides generator. During the training process of the encoder and generator, achieve disentangled representation, the typical methods rely on additional disentanglement regularization term, e.g., the total correlation for VAE-based methods (Higgins et al. 2017 Burgess et al. 2018
Kumar et al. 20177] Kim & Mnih, 2018 Chen et al. 2018) mutual information for InfoGAN-ban methods (Chen et al.) 2016; Lin et al.] 2020).
"Equal contribution. Work done during internships at Microsoft Research Asia.
Corresponning author

---

Published as a conference paper at ICLR 2022
Disentangled Represintation Space
Disentangled Represeriation Space
Image
Encoder
Generator
Image
Fisel Generator
Image
Encoder
Larent Space
Latens
(0)
(b)
Figure 1: (a) The encoder and generator framework for learning disentangled representation. (b) Our alternative route to learn disentangle representation with fixed generator.
However, the extra terms usually result in trade-off between disentanglement and generation quality (Burgess et al., 2018 Khrulkov et al.] 2021). Furthermore, those unsupervised methods have been proved have an infinite number of entangled solutions without introducing inductive
bias (Locatello et al., 2019). Recent works (Shen & Zhou 2021 Khrulkov et al. 2021]] Karras et al.
2019 Härkönen et 2020; Voynov & Babenko 2020) show that, for GANs purely trained for image generation, traversing along different directions in the latent space causes different variations of the generated image. This phenomenon indicates that there is some disentanglement property embedded in the latent space of the pretrained GAN. The above observations indicate that training the encoder and generator simultaneous may not be the best choice.
We provide an alternative route to learn disentangled representation: fix the pretrained generator, jointly discover the factors in the latent space of the generator and train the encoder to extract disentangled representation, as shown in Figure [b). From the intuitive notion of disentangled representation, similar image variations should be caused by changing the same factor, and different image variations should be caused by changing different factors. This provide novel contrastive learning view for disentangled representation learning and inspires us to propose a framework: Disentanglement via Contrast (Di sCo) for disentangled representation learning.
In DisCo, changing a factor is implemented by traversing one discovered direction in the latent space. For discovering the factors, DisCo adopts typical network module, Navigator, provides candidate traversal directions in the latent space (Voynov & Babenko 2020 Jahanian et al., 2020 Shen et al. 2020). For disentangled representation learning. model the various image variations, we propose novel A A-Contrastor to build a Variation Space where we apply the contrastive loss. In addition the above architecture innovations, we propose two key techniques for DisCo: (i) an entropy-based domination loss to encourage the encoded representations to be more disentangled, (ii) hard negatives flipping strategy for better optimization of Contrastive Loss.
We evaluate DisCo on three major generative models (GAN, VAE, and Flow) on three popular disentanglement datasets. Di SCoo achieves the state-of-the-arr (SOTA) disentanglement performance compared to all the previous discovering- -baied methods and typical (VAE/InfoGAl) AN AN-based) methods. Furthermore, we evaluate D DisCo on the real-world dataset FFHQ (Karras et al.] (019) to demonstrate that it can discover SOTA disentangled directions in the latent space of pretrained generative models.
Our main contributions can be summarized as: (i) To our best knowledge, DisCo is the first unified framework for jointly learning disentangled representation and discovering the latent space of pretrained generative models by contrasting the image variations. (ii) We propose a novel A A-Contrastor to model image variations based the disentangled representations for utilizing Contrastive Learning. (iid) DisCo is an unsupervised and model-agnostic method that endows non-disentangled VAE, GAN, Flow models with the SOTA disentangled representation learning and latent space discovering. domination loss and a hard negatives flipping strategy.
(iv) We propose two key techniques for DisCo: an entropy-basee
RELATED WORK
Typical unsupervised disentanglement. There have been a lot studies on unsupervised disen- tangled representation learning based on VAE (Higgins et al. 2017 Burgess et al. 2018 Kumar et al. 2017 Kim & Mnih 2018 Chen et al., 2018) or InfoGAN (Chen et al.) 2016: Lin et al. 2020 These methods achieve disentanglement via extra regularization, which often sacrifices the generation quality (Burgess et al. 2018 Khrulkov et al. 2021). VAE-based methods disentangle the variations by factorizing aggregated posterior, and InfoGAN-basee methods maximize the mutual
2

---

Published as a conference paper at ICLR 2022
information between latent factors and related observations. VAE-based methods achieve relatively good disentanglement performance but have low-quality generation. InfoGAN- based methods have a relatively high quality of generation but poor disentanglement performance. Our method supplements generative models pretrained without disentanglement regularization term with contrastive learning in the Variation Space to achieve both high-fidelity image generation and SOTA disentanglement.
Interpretable directions in the latent space. Recently, researchers have been interested in discover- ing the interpretable directions in the latent space of generative models without supervision, especially for GAN (Goodfellow et al. 2014 Miyato et al. 2018: Karras et al. 2020), Based on the fact that the
GAN latent space often possesses semantically meaningful directions (Radford et al., 2015 Shen method to explore interpretable directions in the latent space of pretrained GAN. The subsequent works focus on extracting the directions from specific layer of GANs. Härkööen et al. (2020) search for important and meaningful directions by performing PCA in the style space of StyleGAN (Karras et al.) 2019; 2020). Shen & Zhou (2021) propose to use the singular vectors of the first layer of a generator as the interpretable directions, and Khrulkov et al. (202T) extend this method to the intermediate layers by Jacobian matrix. All the above methods only discover the interpretable directions in the latent space, except for Khrulkov et al.] (202T) which also learns disentangled representation of generated images by training an extra encoder in extra stage. However, all these methods can not outperform the typical disentanglement methods. Our method is the first to jointly learn the disentangled representation and discover the directions in the latent spaces.
it al. 2020, Jahamian et al., (020), Voynov & Babenko (2020) propose regression-
Contrastive Learning. Contrastive Learning gains popularity due to its effectiveness in representation learning (He al. 2020 Grill al.) 2020: van den Oord et al. 2018 Hénaffi 2020 Li al. 2020 Chen et al.] 2020). Typically, contrastive approaches bring representations of different views of the same image (positive pairs) closer, and push representations of views from different images (negative pairs) apart using instance-level classification with Contrastive Loss. Recently, Contrastive Learning extended to various tasks, such as image translation (Liu et al. 2021 Park al. 2020) and controllable generation (Deng et al. (020). In this work, we focus on the variations of representations and achieve SOTA disentanglemenn with Contrastive Learning in the Variation Space. Contrastive Learning is suitable for disentanglement due to: (i) the actual number of disentangled directions is
usually unknown, which is similar to Contrastive Learning for retrieval (Le- Khac et al. 2020). (it) it works in the representation space directly without any extra layers for classification or regression.
3 DISENTANGLEEMENT VIA CONTRAST
3.1 OVERVIEW OF DTSCO
From the contrastive view of the intuitive notion of disentangled representation learning, we propose a DisCo to leverage pretrained generative models to jointly discover the factors embedded as directions in the latent space of the generative models and learn to extract disentangled representation. The benefits of leveraging pretrained generative model are two-fold: (i) the pretrained models with high-quality image generation are readily available, which is important for reflecting detailed image variations and downstream tasks like controllable generation: (ii) the factors are embedded in the pretrained model, severing as an inductive bias for unsupervised disentangled representation learning.
DisCo consists of Navigator to provides candidate traversal directions in the latent space and a A A-Contrastor to extract the representation of image variations and build Variation Space based on the target disentangled representations. More specifically, A A-Contrastor is composed of two shared- weight Disentangling Encoders. The variation between two images is modeled as the difference of their corresponding encoded representations extracted by the Disentangling Encoders.
In the Variation Space, by pulling together the variation samples resulted from traversing the same direction and pushing away the ones resulted from traversing different directions, the Navigator learns discover disentangled directions factors, and Disentangling Encoder learns to extract disentangled representations from images. Thus, traversing along the discovered directions causes distinct image variations, which causes separated dimensions of disentangled representations respond.
Different from baseb or InfoGAN- methods, our disentangled representations and factors in two separate spaces, which actually does not affect the applications. Similar to the typical
3

---

Published as a conference paper at ICLR 2022
Latent Space
a-Contrastor
Variation Space
Navigator
Encoder
Fixed Generative Model
Encoder
Discovered Diroctions
Disentangled Representations
Samples from Variation Space
Contrustive Loss
Figure 2: Overview of DisCo. DisCo consists of: (7) Navigutor exploring traversal directions in the latent space of a given pretrained generative model, (it) a A--onntrastor encoding traversed images into the Variation Space, where utilize Contrastive Learning. Samples in the Variation Space correspond to the image variations along the directions provided by the Navigator labeled with different colors, respectively. A-C A-Contrastor includes two shared-weige weight Disentangling Encoders to extract disentangled representations respectively, and outputs the difference between the disentangled representations variation representation. The Generative Model is fixed, and the Navigator and Disentangling Encoders marked with grey color are learnable.
methods, the Disentangling Encoder can extract disentangled representations from images, and the pretrained generative model with discovered factors be applied controllable generation. Moreover, DisCo can be applied to different types generative models.
shows, given pretrained generative model G: Z - I, where Z E R² denotes the latent space, and I denotes the image space, the workflow is: 1) A Navigator A provides total of D candidate traversal directions in the latent space 2, e.g., in the linear case, A E LLx is a learnable matrix, and each column is regarded a candidate direction. 2) Image pairs G(z), G(z') are generated. is sampled from Z and z' = + A(d, e), where E {l,..... D} and & E R, and A(d, e) denotes the shift along the d- th direction with scalar. 3) The A S-Contrastor, composed of two -haghd-weight Disentangling Encoders E, encodes the image pair to a sample E V as
Here provide detailed workflow of DisCo. As Figure Segureeishar
v(2,d,E) E( A(d, (z)))
where V RR. denotes the Variation Space. Then we apply Contrastive Learning V to optimize the Disentangling Encoder E to extract disentangled representations and simultaneously enable Navigator A to find the disentangled directions the latent space Z.
3.2 DESIGN OF DISCO
We present the design details of DisCo, which include: (i) the collection of query set Q {q: J= positive key set K- (kx J. and negative key set K- {k; J.M which are three subsets of the Variation Space V. (ii) the formulation of the Contrastive Loss.
According to our goal of contrasting the variations, the samples from Q and KC share the same traversal direction and should be pulled together, while the samples from and X-- have different directions and should be pushed away. Recall that each sample v in V is determined v(z, d, e). To achieve the contrastive learning process, we construct the query sample qi = v(z,, E;), the key sample kn- = v²² d and the negative sample k = =(2 ,,,,,)) Specifically, we randomly sample direction index d from discrete uniform distribution (d(1, D} for { and (d) 1.2 to guarantee they are the same. We randomly sample {d, J:M. from the set of the rest of the directions (411,A D} \ {d} individually and independently to cover the rest of directions in Navigator A. Note that the discovered direction should be independent with the starting point and the scale of variation, which is in line with the disentangled factors. Therefore, { {2
1 {2, ) M all sampled from latent space 2, and {=,), ( y {-, MM are all sampled from shared continuous uniform distribution Ud[ -c, individually and independently. We normalize each sample in Q, K.+, and K. unit vector eliminate the impact caused by different shift scalars.
4

---

Published as a conference paper at ICLR 2022
For the design of Contrastive Loss, well-known form of Contrastive Loss is InfoNCE van den Oord et al. 2018):
B
N
exp(:: Kj /T)
LNCE = [
log
(2)
N+ a=1
exp(q: ka/T)
where т is a temperature hyper-pane parameter and {k, N+M originate from BCELoss (Gutmann & Hyvärinen 2010). BCELoss has been used to achieve contrastive learning (Wu et al. 2018 Le- Khac et al.
{ ) Uf Ji=1- The InfoNCE
2020; Mnih & Kavukcuoglu, 2013 Mnih &
Teh. 2012). We choose to follow them use BCELoss Liogits for reducing computational cost:
B +44)
1
Liogits
(1
(3)
B
1=1
N log (g. kg /T),
M
Σ
47
Σ
log(1 oq: km//T)>
(4)
m==
where denotes the sigmoid function, 1 denotes the part for positive samples, and 1,7 denotes the part for the negative Note that we use shared positive set for B different queries to reduce computational cost.
3.3 KEY TECHINIQUES FOR DISCO
Entropy-based domination loss. By optimizing the Contrastive Loss, Navigator A is optimized to find the disentangled directions in the latent space, and Disentangling Encoder E is optimized to extract disentangled representations from images. To further make the encoded representations more disentangled, i.e., when traversing along disentangled direction, only one dimension of the encoded representation should respond, we thus propose an entropy-based domination loss encourage the corresponding samples in the Variation Space be one-hot. To implement the cutropy-based domination loss, we first get the mean e of Q and K.+ as
B Σ
c B+N
qi
k
(5)
We then compute the probability as = exp (() exp e(i), where c(i) is the i-th element of e and is the number of dimensions of e. The entropy-based domination loss Les is calculated as
Led - Σ Pj log(p,).
(6)
Hard negatives flipping. Since the latent space of the generative models is high-dimension complex manifold, many different directions carry the same semantic meaning, These directions with the same semantic meaning result in hard negatives during the optimization of Contrastive Loss. The hard negatives here are different from the hard negatives in the works of self-super representation learning (He et al., 2020 Coskun al., 2018). where they have reliable annotations of the samples. Here, our hard negatives are more likely to be "false" negatives, and choose to flip these hard negatives into positives. Specifically, we use a threshold to identify hard negative samples, and use their similarity to the queries as the pseudo-labels for them:
i,-
log(1 - (aais)) + Σ au log(o(au)))
2T
where & denotes the modified 17, and aij qi kj /T. Therefore, the modified final BCELoss is:
B
Liogits-s
r+ + +1;)
(8)
B
5

---

Published as a conference paper at ICLR 2022
Cars3D
Shapes33
MPI3
Method
MIG
IXI
MIG
DCI
MIG
DXC
Typical disentanglement basellnes:
Factor
0.142 0.023
0.161 + 0.019
0.434 0.143
0.611 + 0.101
0.099 0.029
0.240 4: 0.051
6-TCVAE
0.080
0.023
0.140 0.019
0.106 = 0.175
0.613 = 0.114
0.114 0.012
0.237 + 0.056
InfoGAN-CR
0.011 1: 0.009
0,020 = 0.011
0.297 1 0.124
0.478 + 0.055
0.163 1: 0.076
0.241 + 0.075
Methods on pretrained GAN:
LD
0.086 0.029
0.216 + 0.072
0.168 0.056
0.380 +: 0.062
0.087 0.057
0.196 +: 0.038
CF
0.083 0.024
0.243 + 0.048
0.307
0.124
0.525 + 0.078
0.183 0.081
0.318 1: 0.014
GS
0.136 1: 0.006
0.209 +: 0.031
0.121 1: 0.048
0.284 + 0.034
0.163 1: 0.065
0.229 1 0.042
0.222
0.044
0.356
0.090
0.513 0.075
0.035
0.248 0.038
DisCo (ours)
0.179 0.037
0.271 + 0.037
0.512
0.068
0.708 + 0.048
0.222
0.027
0.292 + 0.024
Methods on protrained VAE:
LD
0,030 :: 0.025
0.068 + 0.030
0.010
0.035
0.068 + 0.075
0.024 + 0.026
0.035 + 0.014
DiaCo (ours)
0.103 0.028
0.211 + 0.041
0.161
0.033
0.068
0.030
Methods pretrained Flow:
LD
0.015 = 0.000
0.029 = 0.000
0.067
0.000
0.211 = 0.000
0.025 + 0.000
0.033 + 0.000 0.264 + 0.000
DiaCo (ours)
0.060 + 0.000
0.199 +: 0.000
0.1 1500000 + 0.000
0.525 + 0.000
0.076 + 0.000
Table 1: Comparisons of the MIG and DCI disentanglement metrics (mean + std). A higher mean indicates a better performance. Di SCo can extract disentangled representations from all three generative models, and Di sCo on GAN achieves the highest score almost all the cases, compared to all the baselines. All the cells except for Flow are results over 25 runs.
DC1
(1.00)
0.0
01.55
MIG
0.35
0.20-
0.30
0.10
0.05
0.0
0.00
Shapes3D
Car3D
MPI3D
Figure 3: Violin plots on three datasets (1: 3-TCVAE, 2: FactorVAE, 3: InfoGAN-CR, 4: CF, 5: LD. 6: GS, 7: DS, DisCo (ours)). DisCo on pretrained GAN consistently achieves the best performance, Each method has 25 runs, and the variance is due to randomness.
Full objective. With the above two techniques, the full objective is:
Llogits-f f + XLeds
(9)
where À is the weighting hyper-parameter for entropy-based domination loss Led-
EXPERIMENT
In this section, first follow the well-accepted protocol (Locatello et al. 2019 Khrulkov et al. 2021 to evaluate the learned disentangled representation, which also reflects the performance of discovered directions implicitly (Lin al.) 2020) (Section 4.1). Secondly, follow Li et al. (2021a) directly
evaluate the discovered directions (Section 4.2) Finally, we conduct ablation study (Section 4.3)
4.1 EVALUATIONS ON DISENTANGLED REPRESENTATION
4.1.1 EXPERIMENTAL SETUP
Datasets, We consider the following popular datasets in the disentanglement areas: Shapes3D (Kim & Mnih 2018 with ground truth factors, MPI3D (Gondal et al. 2019) with 7 ground truth factors,

---

Published as a conference paper at ICLR 2022
LD
CF
DisCo
StyleGAN2 FFHQ
Smile
StyleGAN2 FFHQ Bald
Figure 4: Comparison of discovered directions. DisCo can better manipulate desirable attributes while keeping others intact. Please refer to Appendix C for more qualitative results.
Variation
Convar
Variation
Contrust
Clasiification
MIG
DCI
0.023
0.225
0.562
0.736
0.012
0.002
0.452
(a) w/a Leed
(b) Ledd
Figure 5: Visualization of the variation of the encoded disentangled representations caused by the change of single ground truth factor.
Table Ablation on Contrast v.s. Classification
and Concatenation (Concat) v.s. Variation.
and Cars3D RRed al. 2015 with 3 ground truth factors. In the experiments of the above datasets, images are resized to the 64x64 resolution.
Pretrained generative models. For GAN, we use the StyleGAN2 model KKrrras et al. 2020). For VAE, we use a common structure with convolutions (Locatello et al. 2019). For Flow, we use Glow (Kingma & Dhariwal 2018).
Baseline. For the typical disentanglement baselines, we choose FactorVAE (Kim Mnih 2018), 3-TCVAE (Chen et al. 2018) and InfoGAN-CR (Lin et al. (2020). For discovering based methods, we consider serveral recent methods: GANspace (GS) (Härkönen et al. 2020). LatentDiscovery
(LD) (Voynov & Babenko 2020), ClosedForm (CF) (Shen & Zhou 2021 and DeepSpectral (DS) KKrulkov et al.) 2021). For these methods, we follow Khrulkov et al. (202T) to train an
additional encoder to extract disentangled representation. We are the first to extract disentangled representations from pretrained VAE and Flow, so we extend LD to VAE and Flow as a baseline.
Disentanglement metrics. We mainly consider two representative ones: the Mutual Information Gap (MIG) (Chen el al., 2018) and the Disentanglement metric (DCI) (Eastwood & Williams 2018). MIG requires each factor be only perturbed by changes single dimension representation. DCI requires each dimension only to encode the information of single dominant factor. We evaluate the disentanglement terms of both representation and factors. We also provide results for 3-VAE
score (Higgins et al. 2017) and FaciorVAE score (Kim & Mnih 2018 in Appendix B.3
Randomness. We consider the randomness caused by random seeds and the strength of the regular- ization term (Locatello et al. 2019). For random seeds, follow the same setting the baselines. Since DisCo does not have a regularization term, we consider the randomness of the pretrained generative models. For all methods, ensure there are 25 runs, except that Glow only has run, limited by GPU resources. More details are presented in Appendix
4.1.2 EXPERIMENTAL RESULTS
The quantitative results are summarized in Table and Figure 3 More details about the experimental settings and results presented in Appendix &
DisCo vs. typical baselines. Our DisCo achieves the SOTA performance consistently in terms of MIG and DCI scores. The variance due randomness of DisCo tends to be smaller than those typical baselines. We demonstrate that the method, which extracts disentangled representation from pretrained non-disentangled models, outperform typical disentanglement baselines.
7

---

Published as a conference paper at ICLR 2022
DisCo discovering- methods. Among the baselines based on discovering pretrained GAN, CF achieves the best performance. DisCo outperforms CF in almost all the cases by a large margin. Besides, these baselines need extra stage (Khrulkov et al. (02T))
I to get disentangled representation, while our Disentangling Encoder can directly extract disentangled representation.
4.2 EVALUATIONS ON DISCOVERED DIRECTIONS
To evaluate the discovered directions, we com- pare DisCo on StyleGAN2 with GS, LDD CF and DS on the real-world dataset FFHQ (Karras cct al. 2019 and adopt the comprehensive Ma-
MDS on CelehAHO-AAtriiiine
Method
Young
Smile
Bald
Blonde Hair
Overall
DS
0.518
0.570
0.524
0.511
0.531
CF
0.518
0.553
0.504
0.560
0.534
nipul ation Disentanglement Score (MDS) (Li et al., 2021a) as a metric. To calculate MDS,
GS
0.502
0.534
0.494
0.538
0.517
LD
0.627
0.531
0.521
0.514
0.549
DisCo
0.516
0.688
0.568
0.592
0.591
40 CelebaHQ-Attriuun leased by StyleGAN. Among them, we select Young, Smile, Bald and Blonde Hair, they are attributes with an available predictor and commonly found by all methods at the same time. The results are summarized in Table 3 DisCo has shown better overall performance compared to other baselines, which verifies our assumption that learning disentangled representation benefits latent space discovering. We also provide qualitative comparisons in Figure
predictors re-
Table 3: MDS comparison on facial attribute edit- ing. Our DisCo shows the best overall score for the latent discovering task FFHQ dataset.
Finally, we provide an intuitive analysis in Appendix D for why DisCo can find those disentangled directions.
4.3 ABLATION STUDY
In this section, perform ablation study of DisCo only on GAN, limited by the space. For the experiments, we use the Shapes3ID dataset, and the random seed is fixed.
Choice of latent space. For based GANs (Krrra al., 2019 2020), there is style space W, which is the output of style network (MLP) whose input is a random latent space Z. As demonstrated in Karras et al. (2019), W is more interpretable than 2. We conduct experiments on W and Z respectively to see how the latent space influences the performance. As shown in Table Di sCo on W is better, indicating that the better the latent space is organized, the better disentanglement DisCo can achieve.
Choices of A. Following the setting of Voynov & Babenko (2020), we mainly consider three options a linear operator with all matrix columns having unit length, a linear opera- tor with orthonormal matrix columns, or non-
Method
MIG
2 Unit length matrix
0.242
0.673
2 Orthonormal matrix
0.188
0.578
2 3 fully-connected layers
0..169
0.504
linear operator of 3 fully-connected layers.
W + Unit length matrix
0.547
0.730
W Orthonormal matrix
0.551
0.709
The results are shown in Table For latent spaces W and Z, A with unit-norm columns achieves nearly the best performance terms of MIG and DCI scores. Compared A with orthonormal matrix columns, using A with unit- norm columns is more expressive with less con- straints. Another possible reason is that A is global without conditioned on the latent code z. A non-linear operator is more suitable for local navigator A. For such much more complex local and non-linear setting, more inductive bias or supervision should be introduced.
W 3
layers
0.340
0.665
Slogits
0).134
0.632
Lingits + Led
0.296
0,627
Liogits-S
0.134
0.681
Ltogite + Led
0.547
0.730
Table 4: Ablation study of si on the latent spaces, types of A, and our proposed techniques.
Entropy-based domination loss. Here, we verify the effectiveness of entropy- -bsed domination loss Lnd for disentanglement. For desirable disentangled representation, one semantic meaning corresponds to one dimension. As shown in Table Led can improve the performance by a large
TThe above disentanglement metries (DCT and MIG) are available for FFHQ dataset.
8

---

Published as a conference paper at ICLR 2022
0.6
0.6
MiG
0.5
0.5
0.5
0.5
200r
0.3
04g
0.3
0.4
0.2
0.1
0.1
MIG
0.0
1.55
15.
80
24.72
32 64
45:48
04.32
72:24
0.
0.0
8:36
16:32
32.64
64.128
(a) Impact of N M with a fixed sum
(b) Impact of N + M with fixed ratio :
Figure 6: Study numbers of positive (N) and negative samples (M). The balance between positive and negative samples crucial for DisCo.
margin. We also visualize the Variation Space to further demonstrate the effectiveness of our proposed loss in Figure 5) Adding the domination loss makes the samples in the Variation Space to be one-hot, which is desirable for disentanglement.
Hard negatives flipping. We run our Di SCo with or without the hard negatives flipping strategy to study its influence. As shown in Table flipping hard negatives improve the disentanglement ability of DisCo. The reason is that the hard negatives have the same semantics as the positive samples. In this case, treating them the hard negatives does not make sense. Flipping them with pseudo-labels can make the optimization of Contrastive Learning easier.
Hyperparmeter N & M. We run DisCo with different ratios of N : M with a fixed sum of 96, and different sum of N + M with a fixed ratio 1 2 to study their impacts. As shown in Figure 6 (a). the best ratio N M 32: 64 = : 2, as the red line (MIG) and blue line (DCI) in the figure show that larger or smaller ratios will hurt Di DisCo, which indicates DisCo requires a balance between N and M. As shown in Figure 6 (b), the sum of N + M has slight impact on DisCo. For other hyperparameters,
set them empirically, and more details are presented in Appendix A
Contrast vs. Classification. To verify the effectiveness of Contrast, we substitute it with classification by adopting additional linear layer to recover the corresponding direction index and the shift along this direction. As Table shows, Contrastive Learning outperforms Classification significantly.
Concatenation vs. Variation. We further demonstrate that the Variation Space is crucial for DisCo. By replacing the difference operator with concatenation, the performance drops significantly (Table 2p. indicating that the encoded representation is not well disentangled. On the other hand, the disentangled representations of images are achieved by Contrastive Learning in the Variation Space.
4.4 ANALYSIS OF DIFFERENT GENERATIVE MODELS
As shown in Table D DisCo can be well generalized to different generative models (GAN, VAE, and Flow). i sCo GAN and VAE achieve relative good performance, while DisCo Flow is not as good. The possible reason is similar to the choice of latent space of GAN. We assume the disentangled directions we suspect that Flow may not conform to this assumption well. Furthermore, Flow has the problems of high GPU cost and unstable training, which limit us to do further exploration.
global linear and thus
linear navigator. In contrast to GAN and VAE,
5 CONCLUSION
In this paper, we present an unsupervised and model-agnostic method DisCo, which is Contrastive Learning framework to learn disentangled representation by exploiting pretrained generative models. We propose an entropy-based domination loss and a hard negatives flipping strategy to achieve better disentanglement. DisCo outperforms typical unsupervised disentanglement methods while maintaining high image quality. We pinpoint a new direction that Contrastive Learning can be well applied to extract disentangled representation from pretrained generative models. There may be some specific complex generative models, for which the global linear assumption of disentangled directions in the latent space could be a limitation. For future work, extending D sCo to the existing VAF- disentanglement framework an exciting direction.

---

Published as a conference paper at ICLR 2022
REFERENCES
Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards AL. In Large Scale Kernel Machines. MIT Press, 2007.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerehner. Understanding disentangling in beta-vae, arXiv:1804.03500 03599, 2018.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In NeurPIS, 2018.
Ting Chen, Simon Kormblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In NeurPIS, 2016.
Huseyin Coskun, David Joseph Tan, Sailesh Conjeti, Nassir Navab, and Federico Tombari. Human motion analysis with deep metric learning. In ECCV, 2018.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. CVPR, 2020.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. ICLR, 2018.
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volehkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentanglemen dataset. In NeurIPS, 2019.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1 406.2661,
2014. URLthttp::/aet http///axvvv///////v////////.avv www.vvorrgggoooo abs///////// 11..11106..6666
Jean-Basticn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent A new approach to self-supervised learning. In NeurPIS, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. AISTATS. JMLR Workshop and Conference Proceedings, 2010.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable GAN controls. In NeurIPS, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier Hénaff. Data-efficient image recognition with contrastive predictive coding. In ICML. 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In 2020.
10

---

Published as a conference paper at ICLR 2022
Y Jin, J Zhang, M Li, Y Tian, and H Zhu. Towards the high-quality anime characters generation with generative adversarial networks. In Proceedings of the Machine Learning for Creativity and Design Workshop at NeurIPS, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Samuli Laine, Miika A Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
Valentin Khrulkov, Leyla Mirvakhabova, Ivan V. Oseledets, and Artem Babenko. Disentangled representations from non-disentangled models. CoRR, abs/2102.06204, 2021.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICMI, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Neur/PS, 2018.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint (T1i::1711.000444 2017.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 2020.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com. cotpe//////xx//iiisit..ommm 2, 2010.
Wonkwang Lee, Donggyun Kim, Seunghoon Hong, and Honglak Lee. High-fidelity synthesis with disentangled representation. In ECCV, 2020.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. CoRR, 0b4/2000.00496, 2020. URL http::tt///stttt
org/ hts//bb//50 abaaaba...... 2005. 04966
Minjun Li, Yanghua Jin, and Huachun Zhu. Surrogate gradient field for latent space manipulation. In CVPR, 2021a.
Suichan Li, Dongdong Chen, Yinpeng Chen, Lu Yuan, Lei Zhang, Qi Chu, Bin Liu, and Nenghai Yu. Improve unsupervised pretraining for few-label transfer. ICCV, 2021b.
Zinan Lin, Kiran Thekumparampii, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In ICML, 2020.
Rui Liu, Yixiao Ge, Ching Lam Choi, Xiaogang Wang, and Hongsheng Li. Divco: Diverse conditional image synthesis via contrastive generative adversarial network. In CVPR, 2021.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In ICML, 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In NeurlPS, 2013.
Andriy Mnih and Yee Whye Tch. A fast and simple algorithm for training neural probabilistic language models. ICML, 2012.
Taesung Park, Alexei A. Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for unpaired image-to-image translation. In ECCV, 2020.
11

---

Published as a conference paper at ICLR 2022
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2015.
Scott E. Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In NeurIPS, 2015.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. CVPR, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
Akash Srivastava, Yamini Bansal, Yukun Ding, Cole L. Hurwitz, Kai Xu, Bernhard Egger, Prasanna Sattigeri, Josh Tenenbaum, David D. Cox, and Dan Gutfreund. Improving the reconstruction of disentangled representation learners via multi-stage modelling. CoRR, abs/2010. 2020.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/ 807.03748, 2018. URL hRtttt///axxix http://awxxvvv../bbbbaoba
hbtp:/b///1800 abg 03748
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In ICML, 2020.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICML., pp. 9929-9939. PMLR, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non con-paraettric instance discrimination. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
A.1 SETTING FOR DISCO
For the hyperparameters, we empirically set the temperature 7 to threshold T' to 0.95, batch size B to 32, the number of positives N to 32, the number of negatives K' to 64, the loss weight À for Led 1, the number of directions to and the dimension of the representation J to 32. We use an Adam optimizer (Kingma & Ba 2015) in the training process, as shown in Table Besides N and M, we empirically find that DisCo is not sensitive to threshold 2 0.9 and other hyperparameters.
For the randomness, there is no regularization term for Di sCo, thus the disentanglement performance is mainly influenced by the pretrained generative models. We follow Khrulkov et al. (202I) to run 5 random seeds to pretrain the GAN and 5 random seeds for training DisCo. We have the same setting for DisCo on GAN and VAE on all three datasets. For Flow, we only use one random seed to pretrain the Glow and one random seed for DisCo. Compare with the baselines, for DisCo on StyleGAN, our modification happens globally on all layers in the W space without any manual selection.
Parameter
Values
Optimizer
Adam
Adam: betal
0.9
Adam: beta2
0.999
Adam: epsilon
1.00e-08
Adam: learning rate
0.00001
Iteration:
100,000
Table 5: Optimizer for DisCo.
T
T 0.8
T 0.9
T = 0.95
T = 0.98
MIG
0.157
0.244
0.508
0.547
0.408
DCI
0.396
0.576
0.710
0.730
0.703
Table 6: Ablation study on hyperparameter T. DisCo not sensitive to T when T 2 0.9. For
T' < 0.9, we may flip true hard negative and thus lead the optimization of Contrastive Loss collapse.
13

---

Published as a conference paper at ICLR 2022
A.2 SETTING FOR BASELINES
In this section, we introduce the implementation setting for the baselines (including randomness).
VAE-based methods. We choose FactorVAF and 3-TCVAE the SOTA VAF-based methods, we follow Locatello et al. (2019) to use the same architecture of encoder and decoder. For the hyper-parameters, we use the the best settings by grid search. We set the latent dimension of representation to 10. For FactorVAE, we set the hyperparameter 7 to 10. For 3-TCVAE, we set the hyperparameter 6. For the random seeds, considering our method has 25 run, with different random seeds for each model to make the comparison fair.
25 times
InfoGAN-based methods. We choose InfoGAN -RR as a baseline. We use the official implementa- tion 2 with the best hyperparameter settings by grid search. For the random seeds, we run 25 times with different random seeds
Discovering-based methods. We follow Khrulkov et al. (202T) to use the same settings for the following four baselines: LD (GAN), CF, GS, and DS. Similar our method (DisCo), discovering- based methods do not have regularization term. Thus, for the randomness, strategy with Dis We take the top-10 directions for different random seeds for GAN and different random seeds for the additional encoder to learn disentangled representations.
adopt the same
LD (VAE) & LD (Flow). We follow LD (GAN) to use the same settings and substitute the GAN with VAB / Glow. The only exception is the randomness for LD (Flow). We only run one random seed to pretrain the Glow and use one random seed for the encoder.
A.3 MANIPULATION DISENTANGLEMENT SCORE
As claimed in Li al. (022aa) it is difficult evaluate the performance on discovering the latent space among different methods, which often use model-specific syper-parameters to control the editing strength. Thus, Li al. (021a) propose comprehensive metric called Manipulation Disen- tanglement Score (MDS), which takes both the accuracy and the disentanglement of manipulation into consideration. For more details, please refer to Li et al. (2021a).
A.4 DOMAIN GAP PROBLEM
Please note that there exists domain gap between the generated images of pretrained generative models and the real images. However, the good performance on disentanglement metrics shows that the domain gap has limited influence DisCo.
14

---

Published as a conference paper at ICLR 2022
A.5 ARCHITECTURE
Here, we provide the model architectures in our work. For the architecture of StyleGAN2, we follow Khrulkov al. (2021). For the architecture of Glow, we the open-source implementation
Conv 7 x x 64, stride
ReLu
Conv 4 x 4 64 x 128, stride =
ReLu
Conv 4 x 128 x 256, stride = 2
ReLu
Conv 4 x 4 256 x 256, stride = 2
ReLu
Conv 4 x 4 x 256 x 256, stride = 2
ReLu
FC 4096 x 256
ReLu
FC 256 x 256
ReLu
FC 256 x J
Table 7: Encoder E architecture used in DisCo. J is 32 for Shapes3D, MPI3D and Car3D.
FC J x 256
Rel.
FC 256 256
ReLu
FC 256 x 4096
ConvTranspor 4 x 4 x 256 x 256, stride = 2
ReLu
Conv ConyTranpoos 4 4 x 256 x 256, stride = 2
ReLu
Cony ConvTranspsseee 4 4 x 256 x 128, stride = 2
ReLu
Cony Transpose 4 4 x 128 x 64, stride = 2
ReLu
Conv TonyTrospe 7 7 x 64 3, stride = 1
Table 8: VAE's decoder architecture, Its encoder is the same the encoder sCo.
15

---

Published as a conference paper at ICLR 2022
B MORE EXPERIMENTS
B.1 MORE QUALITATIVE COMPARISON
We provide some examples for qualitative comparison. We first demonstrate the trade-off problem of the VAE-based methods. As shown in Figure 7 DisCo leverages the pretrained generative model and does not have the trade-off between disentanglement and generation quality.
B-TCVAE
DisCo
Figure 7: Demonstration the trade-off problem of the VAE-based method. 8-TCVAE has bad generation quality, especially on the real-world dataset. DisCo lerverages pretrained generative model that can synthesize high-quality images.
16

---

Published as a conference paper at ICLR 2022
Furthermore, shown in Figure and Figure 9 VAE-based methods suffer from poor image quality. When changing one attribute, the results of discovering-bet based methods tend to also change other attributes.
LD
CF
GS
DS
FactorVAI
3-TCVAE
DisCo (GAN)
Figure Comparison with baselines on Shapes3D dataset with Pose attribute.
17

---

Published as a conference paper at ICLR 2022
LD
CF
GS
DS
FactorVAE
B-TCVAE
DisCo (GAN)
Figure 9: Comparison with baselines Shapes3D dataset with Wall Color attribute. VAE-based methods suffer from poor image quality. Discovering-based methods tend to entangle Wall Color with other attributes.
18

---

Published as a conference paper at ICLR 2022
We also provide qualitative comparisons between DisCo and InfoGAN-CR. Note that the latent space InfoGAN-CR is not aligned with the pretrained StyleGAN2. InfoGAN-CR also suffers from the trade-off problem, and its disentanglement ability is worse than DisCo.
InfoGAN-CR
DisCo (GAN)
Figure 10: Comparison with baselines on Shapes3D dataset with Wall Color attribute. InfoGAN-CR entangles Wall Color with Object Color and Pose.
InfoGAN-CR
DisCo (GAN)
Figure 11: Comparison with baselines on Shapes3D dataset with Floor Color attribute. InfoGAN-CR entangles Floor Color with Object Color.
19

---

Published as a conference paper at ICLR 2022
We explain the comparison in the main paper and show manipulation comparisons here.
GS
LD
CF
DisCo
StyleGAN2 FFHQ Smile
Figure 12: Manipulation comparison with discovering based pipeline with Smile attribute. We explain the left column here. For GS. the manipulation also changes age. For LD, the manipulation also changes pose and skin tone. For CF, the manipulation also change identity.
GS
LD
CF
DisCo
StyleGAN2 FFHQ Bald
Figure 13: Manipulation comparison with discovering- pipeline with Bald attribute. We explain the left column here. For GS and LD, the manipulations also change age. For CF, the manipulation also changes skin tone.
GS
CF
DisCo
StyleGAN2 FTHQ Pose
Figure 14: Manipulation comparison with Sascovering-baser pipeline with Pose attribute. LD does not find the direction of pose attribute. GS, CF and DisCo manipulate pose successfully.
20

---

Published as a conference paper at ICLR 2022
B.2 ANALYSIS OF THE LEARNED DISENTANGLED REPRESENTATIONS
We feed the images traversing the three most significant factors (wall color, floor color, and object color) of Shapes3D into the Disentangling Encoders and plot the corresponding dimensions of the encoded representations to visualize the learned disentangled space. The location of each point is the disentangled representation of the corresponding image. An ideal result is that all the points form cube, and color variation is continuous. We consider three baselines that have relatively higher MIG and DCI: CF, DS, LD. As the figures below show, the points in the latent space of CF and DS are not well organized, and the latent space of all the three baselines are not well aligned with the axes, especially for LD. DisCo learns a well-aligned and well-organized latent space, which signifies a better disentanglement.
CF
DS
LD
Ours
B.3 MORE QUANTITATIVE COMPARISON
We provide additional quantitative comparisons in terms of B-VAF score and Factor VAE score. DisCo on pretrained GAN is comparable to discovering- based baselines in terms of B-VAE score and FactorVAE score, suggesting that some disagreement between these two scores and MIG/ DCL. However, note that the qualitative evaluation in Figure 8 Figure and Section B.2 shows that the disentanglement ability of DisCo is better than all the baselines on Shapes3D dataset.
Cars3D
Shapes3D
MPBD
Method B-VAF scote
VAc score
A-V score
VAF score
B-VAF score
VactorVAF
Typicall disentanglement baselines:
FactorVAE
1.00 = 0.00
0.906 0.062
0.892 + 0.064
0.840 + 0.086
0.339 + 0.029
0.152 = 0.025
3-TCVAE
0.999
0.855
0.082
0.978 0.036
0.873 1 0.074
0.318 0.012
0.179 0.017
InfoGAN-CE CR
0.450 =: 0.022
0.411
1 0.013
0.837 : 0.039
0.587 1 0.058
0.672 1 0.101
0.439 :: 0.061
Methods on pretrained GAN:
I.D
0.089 + 2.54c
0.039
0.913 0.063
0.805 0.064
0.057
0.391 0.039
CF
1.00 = 0.00
0.873
0.036
0.999 + 0.001
0.951 + 0.021
0.669 + 0.033
0.523 = 0.056
GS
1.00 = 0.00
0.932
0.018
0.914 = 0.041
0.758 + 0.091
0.605 : 0.061
0.165 = 0.036
DS
1.00 :: 0.00
0.871
0.047
0.991 0.022
0.929
1 0.065
0.651 0.043
0.502 0.042
DisCo (ours)
0.989 1: 6.86e 5
0.855
0.074
0.987 0.028
0.877 0.031
0.530 0.015
0.371 0.030
Methods on pretruined VAE:
LD
0.951 + 0.074
0.711 0.085
0.602
0.196
0.437 + 0.188
0.266
0.068
0.242 - 0.010
DisCo (ours)
0.999 = 5.1 5
0.761 = 0.114
0.999 + S S. 4
0.956 + 0.011
0.111 1:11::0 0.031
0.391 = 0.075
Methods on pretrained Flow:
LD
0.922 = 0.000
0.633 0.000
0,699 0.000
0.597 + 0.000
0.266 0.000
0.242 - 0.000 0.186 = 0.000
DisCo (ours)
1.00 =: 0.000
0.880 = 0.000
0. 0.860 = 0.000
0.851 + 0.000
0.538 = 0.000
Table 9: Comparisons of the 3-VAE and FactorVAE scores on the Shapes3D dataset (mean + variance). A higher mean indicates a better performance.
We also provide additional experiment Noisy-DSprites dataset. We compare DisCo with 3-TCVAE (the best typical method) and CF (the best discovering- based method) in terms of MIG and DCI metrics.
Method
8-TCVAE
CF
DisCo (GAN)
DCI
0.088 + 0.049
0.027 + 0.016
0.120 + 0.059
MIG
0.046 + 0.031
0.020 0.015
0.104 + 0.030
Table 10: Comparisons on Noisy-DSprites.
21

---

Published as a conference paper at ICLR 2022
C LATENT TRAVERSALS
In this section, we visualize the disentangled directions of the latent space discovered by Di sCo on each dataset. For Cars3D, Shapes3D, Anime and MNIST, the iamge resolution is 64 x 64. For FFHQ, LSUN cat and LSUN church, the image resolution is 256 x 256. Besides StyleGAN2 we also provide results of Spectral Norm GAN (Miyato 2018 on MNIST (LeCun and Anime Face (Jin et al. 2017) to demonstrate that DisCo can be well generalized to other types of GAN.
2010)
(a) StyleGAN2 Cars3l Azimuth
(b) StyleGAN2 Cars3D Yaw
(c) StyleGAN2 Cars3D Type
Figure 15: Examples of disentangled directions for StyleGAN2 on Cars3D discovered by DisCo.
22

---

Published as a conference paper at ICLR 2022
(a) StyleGAN2 Shapes3D Wall Color
(b) StyleGAN2 Shapes3D Floor Color
(c) StyleGAN2 Shapes3D Object Color
(d) StyleGAN2 Shapes3D Pose
Figure 16: Examples of disentangled directions for StyleGAN2 Shapes31) discovered by Di sCo. As shown in (b), the latent space has local semantic.
23

---

Published as a conference paper at ICLR 2022
(a) StyleGAN2 LSUN Cat Black
(b) StyleGAN2 LSUN Cat Eye
(c) StyleGAN2 LSUN Cat Hair
Figure 17: Examples of disentangled directions for StyleGAN2 LSUN Cat discovered by DisCo.
24

---

Published as a conference paper at ICLR 2022
(a) StyleGAN2 LSUN Church Hue
(b) StyleGAN2 LSUN Church Backgroud Removal
Au F
(c) StyleGAN2 LSUN Church - Sky
Figure 18: Examples of disentangled directions for StyleGAN2 on LSUN Church discovered by DisCo.
25

---

Published as a conference paper at ICLR 2022
(a) StyleGAN2 FFHQ Oldness
(b) StyleGAN2 FFHQ Hair
(c) StyleGAN2 FFHQ Race
Figure 19: Examples of disentangled directions for StyleGAN2 on FFHQ discovered by DisCo.
26

---

Published as a conference paper at ICLR 2022
(d) StyleGAN2 FFHQ Overexpose
(c) StyleGAN2 FFHQ
Pose
(I) StyleGAN2 FFHQ Smile
Figure 20: Examples of disentangled directions for StyleGAN2 on FFHQ discovered by DisCo.
27

---

Published as a conference paper at ICLR 2022
(a) SNGAN Anime Tone
(b) SNGAN Anime
Skin
(c) SNGAN Anime Pose
Figure 21: Examples of disentangled directions for SNGAN on Anime discoverd by D sCo.
28

---

Published as a conference paper at ICLR 2022
(d) SNGAN Anime
Naturalness
(c) SNGAN Anime
Glass
(f) SNGAN Anime Whiteness
Figure 22: Examples of disentangled directions for SNGAN on Anime discovered by DisCo.
29

---

Published as a conference paper at ICLR 2022
177770
7
7
7
7
7
333333333
4
4
(a) SNGAN MNIST Angle
9
9
99999999
9
9
1
I
5
(b) SNGAN MNIST Width
6
6
6
6
6
(c) SNGAN MNIST Thickness
Figure 23: Examples of disentangled directions for SNGAN on MNIST discovered by DisCo.
30

---

Published as a conference paper at ICLR 2022
(a) VAE Shapes3D Wall Color
(b) VAE Shapes3D Floor Color
(c) VAE Shapes3D Object Color
Figure 24: Examples of disentangled directions for VAE on Shapes3D discovered by DisCo.
31

---

Published as a conference paper at ICLR 2022
(d) VAE Shapes3D Pose
(c) VAE Shapes3D Height
(f) VAE Shapes3D Object Shape
Figure 25; Examples of disentangled directions for VAE on Shapes3D discovered by DisCo.
32

---

Published as a conference paper at ICLR 2022
(a) Glow Shapes3D Wall Color
(b) Glow Shapes33 Floor Color
(c) Glow Shapes3D Object Color
Figure 26: Examples of disentangled directions for Glow on Shapes3D discovered by DisCo.
33

---

Published as a conference paper at ICLR 2022
D AN INTUITIVE ANALYSIS FOR DISCO
DisCo works by contrasting the variations resulted from traversing along the directions provided by the Navigator. the method sufficient to converge to the disentangled solution? Note that is very challenging to answer this question. To our best knowledge, for unsupervised disentangled representation learning, there is no sufficient theoretical constraint to guarantee the convergence to disentangled solution Locatello et al. (2019). Here we provide intuitive analysis for Di DisCo and try to provide our thoughts on how D sisCo find the disentangled direction the latent space, pretrained GAN both quantitatively and qualitatively. The which is supported by observations
intuitive analysis consists of two part: (i) The directions that can be discovered by DisCo have different variation patterns compared random directions. (zi) DisCo hardly converges the an entangled solution.
D.1
WHAT KIND OF DIRECTIONS DIsCo CAN CONVERGE TO?
Discovered
-02
Discovered
-04
6.4
-03
0.2
-02
01
0.0
6.9
08
Random
6.7
-02
6A
0.0
Random
006
63
Traversal images
Floor color
Wall color
Figure 27: Visualization of the latent space of GAN Shapes3D with discovered directions from DisCo or random sampled directions. We traverse the latent space with range of [-25, 25] and a step of 0.5, which results in 10, 000 (100 x 100) samples.
First, we visualize the latent space and show that there are some variation patterns in the latent space for disentangled factors. We design the following visualization method. Given pretrained GAN and two directions in the latent space, we traverse along the plane expanded by the two directions to generate grid of images. The range is large enough to include all values of these disentangled factors, and the step is small enough to obtain a dense grid. Then, we input these images into an encoder that trained with ground truth factors labels. We get heatmap of each factor (the value is the response value corresponding dimension of the factor). In this way, we can observe the variation pattern that emerged in the latent space.
34

---

Published as a conference paper at ICLR 2022
We take the pretrained StyleGAN Shapes3D (synthetic) and FFHQ (real-world), For Shapes3D, we take background color and floor color as the two factors (since they refer to different areas the image, these two factors disentangled). For FFHQ, take smile (mouth) and bald (hair) the two factors (disentangled for referring to different areas). We then choose random directions and the directions discovered by Di sCo. The results are shown in Figure and
and Figure 28
We find a clear difference between random directions and directions discovered by DisCo. This is because DisCo can learn the directions by separating the variations resulted from traversing along with them. However, not all directions can be separated. For those directions in which the variations are not able to converge to them. Conversely, for those directions that can be easily recognized and clustered, Di sCo will converge to them with higher probability. From the following observations, we find that the variation patterns resulting from the directions corresponding to disentangled factors are easily recognized and clustered.
recognized or clustered by the encoder E, it is nearly impossible for DisCo
00
Discovered
06
35
-31
04
02
04
Discovered
67
006
04
-0a
-04
Random
06
04
0.000
022
Random
330
066
-385
04
Traversal images
Smile
Bald
Figure 28:
Visualization of the latent space of GAN on FFHQ with discovered directions from DisCo random sampled directions. We traverse the latent space with range of 15, 15) and step of 0.3, which results in 1, 0000 (100 x 100) samples. For better visualization, we only present the traversal results with step of 5 (10 x 10).
D.2 WHY DISCO HARDLY CONVERGES TO THE ENTANGLED CASES?
In the previous section, show that DisCo discover the directions with distinct variation patterns and exclude random directions. Here we discuss why DisCo can hardly converge to the following entangled case (trivial solution based disentangled one). For example, suppose there is an entangled direction of factors and B (A and change with the same rate when traversing along with in the latent space of generative models, and Di sCo separate the variations resulting
35

---

Published as a conference paper at ICLR 2022
Latent Space of Generative Models
Semantic directions
A
B
A+B
Isosemantic line
B
A
Note that: and is disentangled
Figure 29: Sketch map of latent space of generative models.
from the direction of A and the entangled direction. In that case, DisCo has no additional bias to update these directions to converge to disentangled ones.
In the following text, for case of referring to, denote the entangled direction of factors A and B (A and B change with the same rate when traversing along with it) as A+B direction, and direction of factor A (only A change when we traverse along with it). The reasons for why DisCo is hardly converged to the case of and A+B are two-fold:
(i) Our encoder is lightweight network (5 CNN layers 3 FC layers). It is nearly impossible for to separate the A and A+B directions.
(it) In the latent space of the pretrained generative models, the disentangled directions (A, B) consistent at different locations. In contrast, the entangled directions (A+B) are not, as shown in Figure 29
We conduct the following experiments to verify them. For (i), we replace our encoder Di sCo with ResNet-50 and train Di sCo from seratch on the Shapes31) dataset. The loss, MIG, and DCI are presented Table [1 The trivial solution possible when the encoder is powerful enough to fit the and A+B directions "become orthogonal". With this consideration, DisCC DisCo we adopt lightweight encoder to avoid this issue.
Our Encoder
ResNet-50
Param
4M
25.5M
Loss (L)
0.550
0.725
MIG (t)
0.562
0.03
DCI (1)
0.736
0.07
Table 11: Ablation study encoder of DisCo.
For (ii), the sketch Figure 29 demonstrates, the disentangled directions ("A"- color "B"- green color) are consistent, which is invariant to the location in the latent space, while the entangled directions ("A+B"- red color) is not consistent on different locations. The fundamental reason is that: the directions of the disentangled variations are invariant with the position in the latent space. However, the "rate" the variation is not. E.g., at any point in the latent space, going "up" constantly changes the camera's pose. However, at point going "up" with step means rotating 10 degrees. At point b, going "up" with step 1 means rotating degrees. When the variation "rate" of "A" and "B" are different, the "A+B" directions different locations not consistent.
Based on the different properties of disentangled and entangled directions in the latent space, DisCo discover the disentangled directions with contrastive loss. The contrastive loss be understood from the clustered view (Wang & Isola 2020 [i et al. (0216). The variations from the disentangled directions more consistent and be better clustered compared to the variations from the
36

---

Published as a conference paper at ICLR 2022
entangled ones. Thus, DisCo discover the disentangled directions in the latent space and learn disentangled representations from images. We further provide the following experiments to support above analysis.
D.2.1 QUANTITATIVE EXPERIMENT
We compare the losses of three different settings:
A: For navigator with disentangled directions, we fix the navigator and train the encoder until convergence.
A B: For navigator with entangled directions (we use the linear combination of the disentangled directions to initialize the navigator), we fix it and train the encoder until convergence.
A+ B + A: After A+B is convergent, we update both the encoder and the navigator until convergence.
The Contrastive loss after convergence is presented in Table 12
A
A A+1 B
A+ B A
Loss
0.5501
0.7252
0.5264
Table 12: Loss comparison on different settings.
The results show that: (i) The disentangled directions (A) can lead to lower loss and better perfor- mance than entangled directions (A+B), indicating no trivial solution. (ii) Even though the encoder with A+B converged, when we optimize the navigator, gradients will still backpropagate to navigator and converge to A.
D.2.2 QUALITATIVE EXPERIMENT
We visualize the latent space of GAN in Figure 30 to verify the variation "rate" the following way: in the latent space, select two ground truth disentangled directions: floor color (A) and background color (B) obtained by supervision with InterFaceGANN (Shen et al. 2020), we conduct equally spaced sampling along the two disentangled directions: A (labeled with green color variation), (labeled with gradient blue color) and composite direction (A+B, labeled with gradient red color) shown in Figure 30 (a).
Then we generate the images (also include other images on the grid as shown in Figure 30 (b) )
and feed the images in the bounding boxes into "ground truth" encoder (trained with ground truth disentangled factors) to regress the "ground truth" disentangled representations of the images.
3.5
2.5
2.5
0.0
2.0
-2.
-5.0
1.0-
-7.3
0.5
-10.0
-10
0.0
0.7
G.4
0.6
1.0
Tbbor color
Floor color
(a) Sampling points
(b) Generated images
(c) Representation of GT encoder
Figure 30: Visualization of GAN latent space.
In Figure 30 (c), the points labeled with green color are well aligned with the x-axis indicating only floor color change, points labeled with blue variation well aligned with the y-axis indicating only
37

---

Published as a conference paper at ICLR 2022
background color change. However, the points labeled with red color are NOT aligned with any line, which indicates the directions of A+B are not consistent. Further, the variation "rate" is relevant to the latent space locations for the two disentangled directions. This observation well supports our idea shown in Figure 29 The different properties between disentangled and entangled directions enable DisCo to discover the disentangled directions in the latent space.
E EXTENSION: BRIDGE THE PRETRAINED VAE AND PRETRAINED GAN
Researchers recently interested in improving image quality given the disentangled representation generated by typical disentanglement methods. Lee et al. (Lee et al. 2020) propose a post-processing stage using GAN based on disentangled representations learned by VAE-based disentanglement models. This method scarifies little generation ability due to an additional constraint. Similarly, Srivastava al. (Srivastava et al. 2020) propose to use a deep generative model with AdaIN (Huang & Belongiee 2017) as post-processing stage improve the reconstruction ability. Following this setting, we can replace the encoder disentangled baselines. In this way, we can bridge the pretrained disentangled VAE and pretrained GAN, as shown in Figure 31] Compared to previous methods, our method can fully utilize the state-of-the-art GAN and the state-of-the-art VAE-based method and does not need to train deep generative model from scratch.
sCo (GAN) with an encoder pretrained by VAF-based
B-TCVAE
DisCo (with a pretrained encoder)
Figure 31: Di SCo with a pretrained encoder allows synthesizing high-quality images by bridging
pretrained 3-TCVAE and pretrained StyleGAN2.
38

---

Published as a conference paper at ICLR 2022
F DISCUSSION ON RELATION BETWEEN BCELOSS AND NCELOSS
We would like to present a deep discussion on the relation between the BCELoss Liogite and NCELoss
LNCE. This discussion is related to the NCE paper Gutmann & Hyvärinen (2010), and InfoNCE paper van den Oord et al. (0018). The discussion is as following: (2) we first provide formulation of a general problem and get two objectives, L1 and L2, and L1 is the upper bound of L2. (ti) Following Gutmann & Hyvärinen (2010). we show that & is aligned with LACE under the setting of Gutmann & Hyvärinen (2010). (xii) Following van den Oord al. (2018), we prove Le is aligned with LNCE under the setting of van den Oord et al. (2018). (ait) We discuss the relation between these objectives and the loss in our paper.
Part I. Assume we have S observations {x: from a data distribution p(æ), each with label C. [0.1]. The we denote the posterior probabilities as p² (x) p(z C 1) and p (x) p(z|C = 0).
We define two objectives as follow:
L =
C, log P(C, = 1|x;)) + (1 - C,) log P(C, = 0/2,),
(10)
and
L2 Σ log P(C; = 1(2;))
(11)
Since - (1:) =0/x;) 2 0, we have
L1 2 L2.
(12)
L1 is the upper bound of Lz.
This a general formulation of a binary classification problem. In the context of our paper. we have a paired observation x; (q. kt), with q the query, and the key k is either from positive key set { N
negative key set (km Jmmm (i.e., {ke) N+M {k5 p U(km IM 1). where
N
M = S - N. And C2 is assigned as:
C. 1 {1 0, ha E M
KE ( 1,
(13)
In our paper, we have f(z)) = exp(q k/T).
Part II. In this part, following Gutmann & Hyvärinen (2010), show that L1 is aligned with
Llogits (Equation 3 in the main paper) under the setting of Gutmann & Hyvärinen (2010). Following
P -
Gutmann & Hyvärinen (2010),, we assume the prior distribution PTC = 0) PTC 1) = 1/2. according to the Bayes rule, we have
p(a 1)P(C=)
P(C = 1|x) = plx|C = =) P =0)
(14)
xx) (z)
And P(C = 0|.) = 1 P(C 1/x).
On the other hand, we have a general form of BCELoss, as
LBCE = Σ Cogoga kx/T) + (1 Ci) log(1 o(q kz/T)),
(15)
1=1
where (.) is the sigmoid function. We have
1
oqq k/T) = 1=exp(-1 k/T) 1 exply-k/7)
(16)
k(a)
39

---

Published as a conference paper at ICLR 2022
From Gutmann & Hyvärinen (2010) Theorem 1, know that when LECE is minimized, we have
p (x) p (x)
h(x)
(17)
Thus, we know the BCELoss LEC is approximation of the objective L1.
Part. III Following van Oord et (018), we prove Le is aligned with LNCe (Equation in the main paper) under the setting of van den Oord et al. (2018)
From the typical contrastive setting (one positive sample, others are negative samples, following van den Oord al. (2018)), we assume there is only one positive sample, others are negatives in { Then, the probability of z; sample from p² (x) rather then p (æ) is as follows,
p E0
p² P(C, = 1/xx) = Σj (x(x)))))))))))))) [ (x1)
(21)
(18)
p (2+)
Ej=1 P (17)
From van den Oord et al. (0188), we know that when minimize Equation p-(x) exp(y k/T) x p- In this case, we get the form of LNCF as
we have h(x)
exply k;/T)
LNCE = Σ C. log
(19)
1=1 Σ5 explq k;/T)
LNCE is approximate L2.
When generalize the contrastive loss into our setting (N positive samples, M negative samples). The BCELoss (Equation 15) can be reformulated as
Part. IV
The BCELoss (Equation 15 can be reformulated
LBC
Σ logolg 0lq kt /T)
Σ log(1 o(q km//T)).
(20)
Similarly, the NCEloss (Equation 19 can be reformulated as
exp(q kit /T)
LNCE= = Σ
log
(21)
EM+N explq ka/T)
j=1
LAGE is aligned with Liogits (Equation 3 in our main paper), and LNCE is aligned with Loee (Equation 2 in the main paper).
Now we have Li (approximated by LBCE) the upper bound of La (approximated by LNCE). However, as you may notice, the assumptions
made in Part II and Part III are different, one is P(C = 0) = P(C 1). the other one is only one positive sample, others are negative. Also the situation more general case (N positives, others negatives).
extent
However, they have the same objective, which is by contrasting positives and negatives, we can use h(x) = exp(g k//)) to estimate p /p- We can think the h(r) similarity score, i.c. if q and are from a positive pair (they have the same direction in our paper). h(x) should be as large as and vice versa. From this way, the image variation, i.e., similar variations have higher score h(x). , while different kinds of variation have lower score h(()). Then with this meaningful representation, in the latent space, can help to discover the directions carrying different kinds of image variation. This is an understanding, from contrastive learning view, of how our method works.
possible (p+ /p= >
learn the representations (q, k) reflect
40


Reviews:
[{"review_id": "zMRwq79V9Nf", "reviewer": "Reviewer_3Z9R", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Learning disentangled representations via contrastive learning on pretrained models is an interesting direction for this community.\n2. The introduced method is intuitive, simple, and effective.\n3. Extensive experiments on disentanglement learning and high-resolution datasets are conducted with satisfying results obtained.\n4. The introduced entropy-based domain loss and the hard negative flipping technique are effective and practical.\n5. The proposed method is general for multiple generative models.\n\nProblem:\nFor qualitative results on StyleGAN2 (Fig. 4 and Fig. 17 - 20), I wonder is there any manual selection on the layers to modify in the w space (like done in the methods GS and CF) or the modification happens globally on all layers in the w space?", "summary_of_the_paper": "This paper proposes to learn disentangled representations via contrastive learning on well-pretrained generative models. Extensive experiments are conducted on various datasets and the results validate the effectiveness of the method.", "summary_of_the_review": "The introduced method is simple, effective, and general for disentanglement learning. I recommend accept for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Tqx4jJUm12F", "reviewer": "Reviewer_j95X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The approach does not require any specific training.\n- There is no fixed generative model type: it can be applied to GANs, VAEs and Flow models.\n- The method significantly outperforms previous models in terms of disentanglement metrics.\n- The method is quite stable to random seeds.\n- The authors provide a thorough ablation study, report the model accuracy with std due to random seeds, check the model sensitivity to the values of hyperparameter T.\n\nWeaknesses:\n- The approach requires many 'tricks' and parts to work: Navigator, Contrastor consisting of two weight sharing encoders, contrastive approach, hard negatives flipping. Each component requires its own set of hyperparameters. The overall performance gain is significant, and the necessity is partially covered in the Ablation study section. But I wonder if it is needed to have two encoders with shared weights or it is possible to have only one? Is it required to tune hyperparameters for every component?", "summary_of_the_paper": "This paper presents a framework to model disentangled directions for pretrained models. Such an approach mitigates the problems with poor generation quality arising while training models with additional regularization terms to force disentanglement. The underlying idea is contrastive-based: similar image variations are caused by changing the same factors in contrast to the remaining image variations. The proposed framework is model-agnostic: it can be applied to GANs, VAEs and flow models.", "summary_of_the_review": "Overall, the proposed method is outperforming previous approaches in terms of disentanglement scores. My main concern is the general complexity of the model. \n\nUPD: I keep my score the same.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MrMTD50d_DT", "reviewer": "Reviewer_Go6R", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Good:\n\nThe idea is very simple and easy to implement.\n\nThe paper is very well written and easy to understand.\n\nThere are extensive ablations that show the effect of design choices and hyper-parameters.\n\nThe qualitative results look very good for disentangling, the proposed method preserves e.g. the identity much better when changing other attributes, like smile or baldness.\n\nQuantitatively the proposed method shows better performance than the baseline for many datasets and 3 different kinds of generative model: GAN, VAE and Flow. This is impressive and shows the methods generality.\n\nBad:\n\nAlthough the paper explained the method well from the perspective of reproducibility, it does not explain why the method should chose semantically meaningful directions. One can imagine a shortcut scenario, where the method learn 0.5a+0.5b and 0.5a-0.5b directions, where a and b are perfect semantically meaningful directions. In principle the training loss could be minimised with this solution as well (?). The reason why this does not happens is because of the implicit biases in the networks (?). But then why is this method performing better than prior works?\n\n\"(ii) the factors are embedded in the pretrained model, severing as an inductive bias for unsupervised disentangled representation learning.\"\nThis still allows for the mixed solution 0.5a+0.5b and 0.5a-0.5b.\n\nI think the following statement is incorrect, it should be removed:\n\"A composed of 3 fully-connected layers performs poorly, indicating the disentangled directions of the latent space W of StyleGAN is nearly linear.\"\n- W is nearly linear because there are good directions in it, and a linear method can perform well in it.\n- The 3 layer network fails for some other reason, in principle it should work at least as good as the linear model, as it has the preresentation capacity.\n\nThe method is very sensitive to the ratio between positive and negative samples. A very good tuning is needed, which is shown in the paper for most hyper-parameters. One might think that the gains come from the extensive tuning rather than the proposed idea itself.\n\nminor:\n- Although the images are resized to 64x64, it would be nice to see full resolution results with e.g. the StyleGAN2 generator. Or was the generator also retrained with reduced size images (for faster training I guess)?\n- some typos and grammar could be fixed, e.g. \"... generative model are two-ford: ...\"\n", "summary_of_the_paper": "The paper proposes a novel representation learning technique to disentangle the latent space of pre-trained generative models, by discovering semantically meaningful directions in them.\n\nThe method trains a navigator and a delta-contrastor network, which consists of 2 encoders sharing weights. First, random samples are perturbed along the directions obtained from the navigator. The perturbed vectors are then decoded with the pre-trained generator, then encoded and the difference between 2 samples are taken. The output is in the variation space, where a contrastive learning technique clusters together the samples that were perturbed with the same direction.\n", "summary_of_the_review": "I think the paper contains good practical ideas and a very extensive experimental evaluation.\n\nThe main weakness of the paper is the lack of deep insights. The reader learns a simple idea that is easy to implement and works well in practice, but does not get an answer to the \"why\" question.\n\nOverall the good outweighs the bad.\n\n--------- UPDATE ---------\n\nI have read the other reviews and the rebuttals. The authors have clarified some details in the experiments. I think the experiments are strong and give valuable data for future research. I raise my score to accept.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6Sk7Smv_omL", "reviewer": "Reviewer_sBQs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) The proposed method is novel and achieves SOTA results in disentanglement while ensuring good generation quality.\n2) Extensive experiments and ablation studies.\n3) In general, the paper is well written and easy to read. \n\nCons:\n1) There are still some flaws in the proposed method.\n2) Some details about how to compute MIG and DCI for discovering-based methods are missing.\n3) MIG and DCI metrics are out-of-date and may not well characterize disentanglement.\n", "summary_of_the_paper": "This paper proposes DisCo, a framework that learns disentangled representations from pretrained entangled generative models. Extensive experimental results show that DisCo outperforms many baselines in both quantitative and qualitative evaluations.", "summary_of_the_review": "1) Novelty:\n- The authors propose a novel and interesting idea of learning disentangled representations by reusing the decoder/generator of pretrained generative models and only learning the directions that lead to disentanglement. \n\n2) Correctness of the method:\n- Is there an absolute operator in Eq. 1?\n- $\\mathcal{V} \\in \\mathbb{R}^{J}$ is not mathematically correct, it should be $\\mathcal{V} \\subset \\mathbb{R}^J_+$ since $v$ is always >= 0.\n- Why are the sizes of the query key set and the positive key set different? They should be the same in contrastive learning. If the authors use multiple positive keys per query key, the size of the positive key set should be $B \\times N$ instead of $N$.\n- I think the formula of NCE in Eq. 2 is incorrect. The sum in the nominator should be outside the $\\log$ and should be treated as the mean over $N$. Which equation in (van de Oord et al., 2018) did the authors refer to? \n- I would like to see mathematical proof of why the BCELoss $\\mathcal{L}\\_{logit}$ (Eq. 3) is a lower bound of the NCELoss $\\mathcal{L}\\_{NCE}$ (Eq. 2). I have never seen such result in (van de Oord et al., 2018) or anywhere.\n- The loss Eq. 3 is indeed an upper bound of the negative mutual information (if the two sums in Eq. 4 is replaced by mean) although it has a different form from InfoNCE (van de Oord et al., 2018). I think the authors simply use the loss without really understanding what it is.\n- I do not really understand how contrastive learning help disentangle the right factors (or directions) if the latent code z and the shift $\\epsilon$ are sampled randomly for both query and positive samples. Could the authors elaborate more on this?\n\n3) Clarity in presentation:\n- It seems that the flipping of negative samples into positive samples in Eq. 7 plays an important role in the model’s performance but is not carefully analyzed in the paper. I am curious about the main cause of flipping. Is it due to the poor selection of positive and negative samples at the first step? How does the flipping rate change during learning? Could the authors show a curve of this? In Table 4, I see that negative flipping does not improve the performance if one-hot regularization is not enforced. Could the authors explain this?\n- How MIG and DCI are computed for discovering-based methods that only use GAN? It seems that these discovering-based methods use no encoder which is required to compute MIG and DCI.\n\n4) Choosing metrics:\n- I think the authors should use more up-to-date disentanglement metrics (e.g. JEMMIG [1]) when evaluating their method. The problem with MIG and DCI is that MIG was shown to capture only modularity, DCI consists of 3 separate sub-metrics and each of them only captures one aspect of disentanglement [2]. Could the authors tell which sub-metric of DCI you are using for evaluation? \n\n[1] Theory and Evaluation Metrics for Learning Disentangled Representations, Do & Tran, ICLR 2020\n\n[2] Measuring Disentanglement - A Review of Metrics, Zaidi et al., 2020\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework, named Disentaglement via Contrast (DisCo), to learn disentangled representations via contrastive learning on well-pretrained generative models. The method aims at simultaneously discovering semantically meaningful directions in pretrained generative models and training and encoder to extract them. The method uses contrastive learning where random samples perturbed along the discovered directions are regularised to be similar. The method is versatile and can be applied to various pretrained non-disentangled generative models including GAN, VAE, and Flow. Extensive experimental evaluation shows the benefits of the approach.

The authors provided a strong rebuttal addressing many of the concerns raised by the reviewers, including running new experiments (such as adding the JEMMIG metric to measure disentanglement as requested by Reviewer sBQs). This led to all reviewers recommending to accept the work.

The paper provides an exhaustive empirical evaluation testing several models and results are convincing. This was highlighted by all reviewers.

While the high level description of the method is clear, in practice the method is quite sophisticated requiring many heuristics (e.g. entropy-based domination loss or flipping hard negatives). This requires tuning several hyperparameters and complicates the message. This is mitigated by an ablation study presented by the authors highlighting the importance of each component. This was highlighted by Reviewer j95X and the AC agrees. The paper does provide implementation details, and reproducibility is not a concern.

Related to this point, Reviewer Go6R points out that the paper falls short in providing clear explanations on why the method is able to find meaningful semantic directions, and on where do the gains of the proposed model come from. While the paper could improve in this direction, the proposed empirical validation is convincing.

Overall the paper presents an interesting method that performs well in practice. All reviewers recommend accepting the work. The AC agrees with this recommendation.
Decision: Accept (Poster)