Demystifying Inter-Class Disentanglement
Authors: Aviv Gabbay
Yedid Hoshen
Venue:  (2020)
Keywords: disentanglement
latent optimization
domain translation

Abstract: Learning to disentangle the hidden factors of variations within a set of observations is a key task for artificial intelligence. We present a unified formulation for class and content disentanglement and use it to illustrate the limitations of current methods. We therefore introduce LORD, a novel method based on Latent Optimization for Representation Disentanglement. We find that latent optimization, along with an asymmetric noise regularization, is superior to amortized inference for achieving disentangled representations. In extensive experiments, our method is shown to achieve better disentanglement performance than both adversarial and non-adversarial methods that use the same level of supervision. We further introduce a clustering-based approach for extending our method for settings that exhibit in-class variation with promising results on the task of domain translation.

Published as a conference paper at ICLR 2020
DEMYSTIFYING INTER-CLASS DISENTANGLEMENT
Aviv Gabbay Yedid Hoshen
School of Computer Science and Engineering The Hebrew University of Jerusalem, Israel
ABSTRACT
Learning to disentangle the hidden factors of variations within set of observa- tions is a key task for artificial intelligence. We present unified formulation for class and content disentanglement and use it to illustrate the limitations of current methods. We therefore introduce LORD, a novel method based on Latent Opti- mization for Representation Disentanglement. We find that latent optimization, along with an asymmetric noise regularization, is superior to amortized inference for achieving disentangled representations. In extensive experiments, our method is shown to achieve better disentanglement performance than both adversarial and aon-aaveraarial methods that use the same level of supervision. We further in- troduce clustering based approach for extending our method for settings that exhibit in-class variation with promising results on the task of domain translation.
Project webpage: http://www.visionnnuj..cc il/ord
INTRODUCTION
Objects in the real world encompass many different attributes mixed together. Some of the attributes are permanent i.c. the class identity of the object, whereas others are transitory e.g. the pose of the object. Humans can often effectively separate between the class identity of the object, and the transitory pose of the object, even from a single observation. A key task for artificial intelligence is to empower computers learn separate between different attributes of observed data, often re- ferred to as disentanglement, In this paper, we present new method for achieving disentanglemen between the class of an object and the sample-specific content. We restrict our attention to images, to other modalities.
however some of our ideas may carry
There are multiple settings for disentanglement. The simplest is fully supervised for each training image both the class and content are given labels. A fully supervised scheme (e.g. deep encoders) may be trained to recover the class and content information from a single image. Conversely, a generative model can be trained to generate an image given input class and content information. On the other extreme, fully unsupervised disentanglement takes as input a set of images with no further information. A successful unsupervised disentanglement algorithm will be able to learn representation in which different factors of variation such as class and content will be represented separately. Fully unsupervised disentangiement is highly ambitious and is work in progress, current methods typically do not produce consistently good results in this setting (Locatello et al. 2019)
In this work, we deal with the class-supervised disentanglement task. In this setting, the class label for each image in the training set is given. Such supervision can be easily obtained in practice e.g. tracking an object in a video obtains multiple images in multiple poses of the same class (for example, person identity). The objective of the disentanglement task is to learn representation containing all the information not available in the class label, denoted as content. faces, this content includes: head pose, facial expression, etc. We begin by carefully analyzing the information contained in the class and content representations. We show that current methods allow information to leak between the representations leading to imperfect disentanglement, We therefore introduce LORD, a novel method which carefully ensures no information leakage between the class and content representations.
the case of
Our method differs from previous methods by several methodological improvements. i) We lever- age latent optimization to learn single representation for each class which is shared between all

---

Published as a conference paper at ICLR 2020
its samples. We show and discuss the benefits of this approach the amortized techniques. semiees ii) We introduce asymmetric regularization on the content latent codes to achieve -aaa-innva representations. We show the superiority of this technique over adversarial constraints and the KL- divergence. Latent optimization is very effective at learning disentangled representations at training time, however, it is not useful for obtaining class and content codes of unseen test images. Optimiz- ing over the latent codes at test time (without class supervision which exists at training time), leads to overfitting which results in entangled representations. We overcome this challenge by introducing second stage in which we use the class and content codes learned by our model in the first stage for training feed-forward class and content encoders. The encoders generalize well to unseen images and significantly reduce the inference time on new samples.
Our method is evaluated qualitatively and quantitatively in terms of generation of novel samples of observed classes. We also quantitatively evaluate the quality of disentanglement of learned features by classifying class labels from content codes and vice versa. Our method is shown to significantly outperform other adversarial and non-adversarial methods. Disentangling class and content rep- resentations assumes that intra-class variation is significantly lower than inter-class variation. We discover that this assumption can be relaxed by clustering in-class styles into separate classes. We demonstrate promising results of our approach on unsupervised domain mapping.
Our contributions in this work are follows: i) An insightful analysis of class-oonditionll disen- tanglement. ii) LORD: a new well-motivated non-adversarial method for disentanglement achieving SOTA results by shared latent optimization and asymmetric regularization. iii) Second stage amortization for single-shot class generalization. iv) The first effective method for disentanglement between 10k classes. v) A clustering based extension for style disentanglement.
1.1 RELATED WORK
Our work deals with flss--upervisee disentanglement. Several works based on variational autoen- coders (VAEs) (Kingma& & Welling 2014) have attempted disentanglement with no supervision e.g. 3-VAE (Higgins et al. (017)) and factor- VAE (Kim & Mnih 2018). In an extensive comparative study, [ocatello et al. (2019) show that none of the compared methods have been successful on all the datasets examined. It therefore seems likely that some supervision is required for effective dis- entanglement. Many works (including ours) provide only class supervision e.g. when the identity of face is given but not its transitory attributes. The disentanglement between the factors of variation is enforced by adversarial constraints (Mathieu et al., 20166] Szabó et al. 2018 Denton & Birodkar, 2017 Hadad al.] (2088) or by adveraarallial constraints e.g. cycle (Harsh Jha et al. (0188) or variational group codes (Bouchacourt et al. (2018). Differently from the above works, we learn per- class codes rather than per-image class. Similar design choice were taken by cGAN and eVAE, but for the application of image generation rather than disentanglement. cGAN and cVAE require the latent space to be Gaussian, which hurts disentanglement performance.
Many disentanglement methods use adversarial training (Goodfellow et al. 2014). Success was achieved on image generation (Brock et al.] 2019), image mapping (Isola et al. 2017) and domain alignment (Liu et al. (017). Adversarial methods are notoriously hard to optimize. require very careful architecture and hyper-parameter tuning due to their min-max nature. To overcome these issues, non-adversarial methods have been proposed to achieve better results tasks previously dominated by adversarial networks e.g. image synthesis (Bojanowski et af.] 2018 Razavi et al. 2019, smage-to-image mapping (Hoshen & Wolf 2018)) and word translation (Mukherjee et al. 2018 In this paper, we present non-adversarial method achieving state-of-the-art performance on disentangiement.
2 CLASS AND CONTENT DISENTANGLEMENT
Assume that we are given a collection of n images æ I2, Zn E X. For each image z;, we are given class label y, E [k). We assume that every image belongs to a single class, although this requirement can be relaxed. Note that many images may share the same class label (e.g. faces of the same person different poses). We denote the embedding of a given class y cy. We assume that the images can be disentangled into representations in two latent spaces y and C. Therefore, objective it to find class representation ey, y and content representations G E C. for each
2

---

Published as a conference paper at ICLR 2020
image 24- Let define the information that we wish each representation to contain. As there is some inconsistency in the notation used in the style-content, pose-content and domain translation literature, will define our terms precisely.
The image class representation Ey. needs to include all information that is shared by all images shar- ing the same class e.g. if classes correspond to different facial identities, then the class representation must include all the time-invariant facial information. The content representation Q includes all the information that is unchanged if the image is transferred between classes. This information must be independent of the class-ifformation. E.g for faces, content corresponds to time-varying facial information such as head pose and expression. Besides the class and content representations, im- ages may contain other image-specific information, which is not represented by the class-label and not expected to transferred across classes. The difference between content and style is seman- tic and requires careful design. In the facial identity example the style may include noise, lighting conditions or nuisance background features. We denote the style representation as 8, S.
We define a generator G, a neural network parameterized by 0, which transforms the disentangled representations into an image. Given our definitions above each image can be modeled by:
T3 = Go Si G.) Ij E&E& E cy. E Y S GEC
(1)
The content must be independent of the class and style, however the style may be class dependent. E.g. if the classes are shoe images and edge images, styles within the shoes class may include particular colors and textures, which are typical shoes but not edge images. More formally, the mutual information between and ey. should be zero:
I(e ey) =0 T(cs)= =0
In many cases, it can be assumed that inter-class variation is significantly larger that intra-class variation. Many approaches were devised to learn disentangled representations for this scenario, in which s; contains both class and style information of an image 24. We will critically review several representative methods.
Adversarial Methods: One way to ensure the independence between the content and class/style representations is using adversarial discriminators. We will summarize the ideas proposed in DrNet (Denton & Birodkar (017) as this approach has the best performance of all adversarial methods. These techniques do not learn class representation explicitly but instead strongly constrain a style encoding. The model of this method is described by:
T; = Go 8;, &;)
(3)
They attempt to ensure the similarity styles of images in the class using a similarity constraint
Csimilarity ||s; 3j if y/ V/- To ensure independence between s and c, adversarial discriminator Dy (c;, Cÿ) is trained to discover two images are from the same class. If the content representation is truly disentangled, then class information is available in the content code and the discriminator accuracy will not be greater than a random chance. This approach has two weaknesses: i) It does not directly prevent content information from leaking into the style representation (but only through a weak pairwise constraint). ii) Adversarial methods are notoriously hard to optimize and require careful
tuning due to the challenging saddle point optimization problem.
Non-Adversarial Methods: Due to the difficulty of adversarial training, non-adversarial methods have attracted attention. We will review the ideas in Multi-Level VAE (ML-VAE) (Bouchacour) et al. 2018). which performs the best of the non-adversarial methods and is most related to ours. ML-VAN VAE also does not learn reteesstprr.nnnnnoonn but style representation s; via amortized infer- ence. However, in order to limit the content information which flows to the generator from the style code, relies on the presence of samples from the same class in a mini-batch during training and accumulates their style encodings using product of normal densities before feeding the generator (the entire process is described in the original paper). To summarize, ML- VAE approximates the style representation M of group of observations M from the same class, and generates the image:
5; Go (0, 5M,
M Uilv =
3

---

Published as a conference paper at ICLR 2020
It limits the information in the content representation c, by constraining its distribution using KL- divergence with the standard normal distribution. This approach suffers from significant drawbacks: i) It uses grouped amortized encoding for inferring S As the size of mini-batch is limited, this either limits the group accumulation to be over a few samples which is biased, or limits batch- diversity by only including few classes which hurts optimization. ii) In experiments, the KL-divergence does not sufficiently constrain the information in C₁ i.c. in practice class information found in C₁.
LORD: LATENT OPTIMIZATION FOR REPRESENTATION DISENTANGLEMENT
In Sec. we analyzed the task of disentanglement between class and content. Our analysis high- lighted the issues faced by current state-of-the-art methods. In this section, introduce novel method motivated by the insights from the previous section.
3.1 LATENT OPTIMIZATION FOR CLASS SUPERVISION
We make explicit the assumption that inter-class variation is significantly larger than intra-class variation. This allows us to model images as a combination of class and content codes:
Ge 0,
(5)
Shared Latent Optimization: We model the class representation between all images belonging to the same class {a:ly; = y}. Instead of using amortized inference (learning mapping from the image the class codes using an encoder), we optimize over the class embeddings directly using latent optimization. This has several important benefits: i) As the code shared exactly between all images belonging to the same class (each having different content), it is impossible to include any content information in the class code. ii) As we learn per-class representations directly rather than using previous techniques as group averaging, each mini-bar niti-batch can contain images randomly sampled from all classes allowing maximal diversity.
embedding by that is shared
We learn the content representation by optimizing over per-sample content embeddings directly using latent optimization and not in an amortized fashion using image to content encoder. As we show in the experimental section, model trained with latent optimization preserves a very high degree of disentanglement along the training and is less sensitive to hyperparameter choices.
Asymmetric Noise Regularization: Latent optimization over the class embeddings ensures that no content information is present in the class representation. To ensure that class information does not leak into the content representation, we regularize the content code to enforce minimality of infor- mation. Previous approaches attempted to minimize content information by setting a bottleneck of small content code or by matching the content distribution to a prior normal distribution using KL-- divergence. Using small noiseless bottleneck, does not however reduce information significantly. A continuous variable may in fact store an infinite amount of information (although the amount of information the generator may extract is limited by other factors). our experiments, we found that regularizing with KL-divergence (as done by previous works) led to partial posterior collapse i.e. and respectively,
nearly all means and standard deviations learned by the encoder defaulted satisfying perfect standard normal distribution. For a few components, the encoder learned large means and very small standard deviations. The Kl--divereence therefore learned behavior similar to small-size bottleneck. This phenomenon implies that regularizing the distribution of the content codes with KL-divergence may require additional attention and a careful hyperparameter tuning. We present the experimental evidence in the Appendix A.3
In our approach, we regularize the content code with an additive Gaussian noise of a fixed variance, and activation decay penalty. In contrast to variational auto-encoder, do not learn the vari- ance, but rather keep it fixed. This prevents the possibility of the variance decreasing to a small value, ensuring that noise is applied equally on all components. Our objective function becomes:
4

---

Published as a conference paper at ICLR 2020
Latent Spares
Class Embedding
Content Embodding C1
Generator
Content Embeddiing C2
Figure 1: A sketch of the first stage: all class and content embeddings and the generator are jointly optimized. All images of the same class share single class embedding. The content embeddings regularized by gaussian noise. By the end of this stage, the latent space of the training disentangled. Note that the second stage is not shown.
C = Σ 0, xzll + Allc//1 x~0100,021)
(6)
The first loss terms uses a VGG perceptual loss as implemented by Hoshen Malik (219)). Un- less stated otherwise, we optimize over class and content codes (Cg, and c;) directly using latent optimization. All latent codes and the parameters the generator are learned end-to-end using stochastic gradient descent:
Ces Cn 0-) org min C
e,c,0
3.2 AMORTIZATION FOR ONE-SHOT INFERENCE
Latent optimization, which is used effectively for training, requires optimization for every image (including inference time). In the training set, a class embedding shared across multiple images, which prevents the embedding from including content information. However, inference time, single image from an unknown class observed. Optimizing over the latent codes for a single image leads to overfitting which results in entangled representations. Moreover, it requires iterative test-time inference since does not perform amortized inference.
To this end, introduce second stage which learns class and content encoders that directly in- fer class ey, and content e; representations from a single image I;, The second stage effectively amortizes the results of the first stage and generalizes well unseen classes and images. We train encoders Ey X' + y and E. X' + C, which take as input an image and output its class and content embeddings that were learned by our method in the first stage. We also use reconstruction loss. to ensure the representations learned in the second stage must reconstruct the original image dz=- The optimization objective is presented in Eq. & The optimization over the parameters of encoders E, and E, (which are randomly initialized) and the parameters of the generator G (which are initialized from first stage). Note that given Ey, and e; are the representations we have leamned during the previous stage.
LE (z.) - x; + ta [E,()))) Cy. 02 |((())))
cıll
(8)
After training, we can preserve the class of a new test image oi and transfer over the content from another image ii by decomposing the images into their disentangled class and content representa-
5

---

Published as a conference paper at ICLR 2020
MAE
DrNet
Ours
Figure 2: Comparison between our method and baselines on Cars3D (top) and SmallNorb (bottom).
tions and regenerating them as follows:
214-2 = (,
(9)
4 EXPERIMENTS
Our method is evaluated against SOTA techniques for class-supervised disentanglement. We do not compare to methods for fully-unse unsupervsee disentanglement as the results are not directly compara- ble and their performance is inferior the following benchmarks due to lower level of supervision. All the implementation details are provided the Appendix A.1
Datasets: We evaluate the performance of our method and the baselines several datasets (each with the appropriate class labels): Cars3D (car model as class label, azimuth and elevation as con- tent), SmallNorb (object type lighting elevation as class labels, azimuth as content), SmallNorb- Poses (object type x lighting as class labels, azimuth and elevation as content), CelebA (person identity class label, other unlabeled transitory facial attributes e.g. head pose and expression as content), KTH (person identity as class label, other unlabeled transitory attributes e.g skeleton po- sition as content), RaFD (facial expression class label, rest as varied content). more detailed description of each dataset and configuration can be found in the Appendix A.2
Baselines:
We compare our method against SOTA methods for class-supervised disentanglement. DrNet (Denton & Birodkar, 2017) and Szabó et al. (2018) encourage disentanglement by adversar- ial constraints, and ML-VAE (Bouchacourt
al., 2018) and Cycle- VAE (Harsh Jha al. 2018) use variants of VAE equipped with grouped class accumulation and cycle constraints discour- age degenerate solutions. We also compare against StarGAN (Choi et al.) 2018) Aulti-Domain translation. For faimess, evaluate each baseline with L1 and perceptual loss and report the best.
Quantitative Experiments:
Content transfer experiments: To test the quality of disentanglement, measure the quality of content transfer in terms of perceptual similarity by LPIPS (Zhang et al.) 2018). We use the content labels available in the Cars3D and SmallNorb datasets ground truth for content transfer. For given test images F; and Øj, we measure the similarity between Z1t-2 = Go GE (x;), and another image from the same class of x; matching the same content of æj. For CelebA, given
O, 0, Ex,)))

---

Published as a conference paper at ICLR 2020
ML-VAE
DrNet
Ours
Figure 3: Comparison between our method and baselines on KTH (top) and CelebA (bottom).
two images of the same person, infer the class (identity) representation from the first image and aim at reconstructing the second by extracting the content representation from an image of a different identity which has the most similar pose (nearest neighbour in the 68 fanial-landmarks space). Results are reported in Tab. [ It can be seen that we strongly outperform all the baselines.
Classification experiments: To assess the disentanglement of our learned representations, the protocol in Harsh Jha et al. (2018) and train classifier to classify class labels from content codes and vice versa. Results can be seen in Tab. 2 On all datasets, our model achieves near perfect disentanglement as the classifier could barely guess class labels from content codes by random chance (same for the other direction). All the baselines fail to zero out the mutual information between the two representations. To conclude, our method is able to learn the most disentangled features without introducing adversarial constraints. For CelebA, in order to test if the content of an image is predictable from the class code train linear regression model to regress the position of 68 facial landmarks. It can be seen that the linear regression results in the highest error on our class representations, indicating the highest degree of disentanglement of our method. It should be noted that all methods could classify class labels from class codes and content labels from content codes very accurately (not shown).
follow
Facial expression transfer experiment: We compare our method against StarGAN in the task of Multi-Domain translation. We follow the protocol in Choi et al. (2018) and compute the classifica- tion error of a facial expression classifier (trained on real images from RaFD) on synthesized images. We train both image translation models using the same training set and perform image translation on the same, unseen test set. As can be seen in Tab. g our model achieves lower classification error than StarGAN, indicating that our model produces more realistic facial expressions without using adversarial training.
Qualitative Experiments: We visually evaluate the results of our method against DrNet (strongest adversarial baseline) and ML-VAE (strongest non-adversarial baseline) in Fig. 2 and 3 In each experiment, we visualize switching between class (left column) and content (top row) codes for each pair within a set of 5 test images. On Cars3D, our method achieves excellent content transfer while keeping the class fixed. DrNet is mostly able to transfer the content, but it does not keep the model fixed. ML-VAE results are of lower fidelity. On SmallNorb, our method works well, whereas the baseline methods struggle with preserving the identity of the object in some rotations (e.g. bottom row). On KTH both our method well DrNet perform well, although our method
7

---

Published as a conference paper at ICLR 2020
Table Content transfer reconstruction error (LPIPS 4)
Cars3D
SmallNorb
SmallNorb-Poses
CelebA
Szabó et 2018)
0.137
0,417
0.214
0.331
Cycle- VAE Harsh Jha et al. 2018)
0.141
0.197
0.202
0.228
MALE (Bouchacourt et al. 2018)
0.132
0,210
0.173
0.222
DrNet DDntonn & Bırodkar 2017
0.095
0,166
0.152
0.229
Ours
0.078
0.117
0.106
0.197
Table 2: Classification accuracy of class labels from content codes (y + c) and of content labels from class codes (lower indicates better disentanglement), Note that the last right column presents the error of face landmark regression from the class codes (higher is better).
Cars3D
SmalINorb
CelebA
y +
-
+
f
R(y) 7
Szabó et al.
2018
0.91
0.82
0.36
0.37
0.09
3.59
Cycle- VAL
Harsh Jha et al. 2018
0.08
0.80
0.27
0.79
0.14
3.14
ML-VAE Bouchhcourt el al. 20188)
0.77
0.96
0.90
0.93
0.17
3.98
DrNet (Denton & Birodkar 2017)
0.26
0.68
< 0.01
0.78
0.03
3.23
Ours
0.01
0.01
0.01
0.05
0.01
4.75
Random chance
< 0.01
0.01
0.01
0.05
0.01
achieves more accurate transfer (e.g. last image in the top row). ML-VAE fails to transfer the skeleton on some identities (e.g. last row). CelebA, the baselines generally transfer head pose but do not preserve the person identity. Our method achieves better pose transfer than both baselines, and is able to maintain the identity.
Non-adversarial unsupervised domain translation: Our model can be used for the task of unsu- pervised domain translation by defining domain labels as class labels, as we demonstrate on RaFD dataset. We further extend our method for datasets in which classes (domains) exhibit in-class varia- tions by introducing preliminary step of clustering in-class styles (variations) into separate classes. For example, in the task of translating edge images into shoe images and vice-versa, style clusters by applying k-means on style features extracted from first layer of a pretrained VGG model (?). The shoe images are therefore separated into sub-classes instead single class which contains high variation. This step decreases the degree of uncertainty in translating images between classes which exhibit in-class variations (See Appendix Ao more details). We then apply LORD treating the obtained style clusters class labels on the Edges2Shoes dataset. Examples of transla- tion diversity along with style-guided edges shoe mapping are shown in Fig. More examples of unsupervised domain translation are provided in Fig. and Appendix A.9
first form
Table 3: An ablation study with several variants of LORD on Cars3D.
Transfer error (LPIPS) L
Classification accuracy |
+
-
Ours amortized (w/ KI.-divergence)
0.094
0.95
0.96
Ours amortized (w/ Asymmetric noise)
0.082
0.92
0.97
Ours semi amortized (w/ KL-divergence)
0.095
0.93
0.01
Ours semi amortized (w/ Asymmetric noise)
0.079
0.22
0.01
Ours (w/o second stage)
0.175
0.11
0.50
Ours (w/o regularization)
0.095
0.10
0.01
Ours
0.078
0.01
0.01
8

---

Published as a conference paper at ICLR 2020
Table Classification error (%) on transferred facial expressions from RaFD.
StarGAN Choi et al. 2018)
Ours
Real Images
2.2
1.8
0.8
Input
Angry
Contempt.
Disguste
Fearful
Happy
Sad
Surprised
Ours
StarGAN
Figure 4: A qualitative comparison between our method (upper row) and StarGAN (bottom row) in facial expression transfer on RaFD. See Appendix A.5 for more results.
5 ABLATION ANALYSIS
We perform careful ablation amalysis on the components is presented in Tab. 3 Shared latent optimization vs. amortized inference: We train amortized the
method, summary of this study
variants of our model with feed-forward class and content encoders instead of optimizing latent codes directly. Class representations of samples from the same class are averaged within mini-batch during training. It can be clearly observed from the results that class representations which are learned via amortized inference leak information about the actual content of each sample, resulting in entangled representations.
Moreover, we train semi aami-amortized variants of our model which leverages latent optimization for learning shared class representations, but uses a feed-forward encoder to infer the content code of an image. can be noticed that this variant achieves sub-optimal performance it leaks some class information into the content representation. order to assess the inductive bias conferred by latent opti- mization, we measure the accuracy of classifying class la- bels from content codes after every epoch. The change in the amount of class-dependent information contained in the content codes captured in Fig. 7 It can be observed that a randomly initialized content encoder (for amortiza- tion) encodes class-dependent information, which needs to be minimized as the training evolves. Initializing ran- dom content codes for latent optimization however pro- vides no information about a specific class. By the end of training, amortized models often do not succeed in distilling the class-invariant information and provide entangled representations, while model trained with latent optimization preserves a very
spach
Figure 7: Accuracy of classifying class labels from content codes as evidence for the inductive bias conferred by la- tent optimization on Cars3D.
Figure 5: Examples of the diversity in translating edges to shoes (upper row) and style-guided translation (bottom row). Triplet order in bottom row (left to right): edges, style, translation.

---

Published as a conference paper at ICLR 2020
(a) Anime to Faces
(h) Diversity translating Faces to Anime (using style clustering).
Figure 6: Examples of translations between anime and faces from CelebA using our method.
high degree of disentanglement. We hypothesize that achieving similar degree of disentanglement by amortization requires a more sophisticated objective and more careful hyperparameter tuning- An extended study is presented in Appendix [A.4. It should be noted that latent optimization requires more iterations than optimizing an amortized encoder and leads to a slower convergence (the num- ber of iterations increased by x2 in our experiments). In both the amortized and semi-amortized models, we find that the KL-divergence fails to regularize the information leakage from the class representation into the content representations. A visualization of the partial posterior collapse can be found in the Appendix A.3 We finally demonstrate the importance of our second stage by assess- ing the performance after the first stage only. This can be done by optimizing of a new test image while keeping the rest of the model frozen. As can be seen, this approach suf- fers from low performance in all metrics. The effect of the asymmetric noise regularization can be observed from the inferior performance of training our model without regularization. A qualitative visualization of this analysis is provided in Appendix A.6
the latent codes
DISCUSSION
Non-Adversarial training: Differently from most other previous works, we do not use adversarial training to enforce disentanglement between the class and content. Non-adversarial training has significant advantages in the ease of optimization. Interestingly, we achieve state-of-thh-ar perfor- mance without any adversarial constraints. We believe this should motivate researchers to further develop non-adversarial approaches.
Perceptual loss: For training our model, dataset. This is not extra supervision, as the imagenet dataset not strongly related to any of the tested datasets. In our experiments we found the perceptual loss was helpful to other method that did not use GANs on the output image (even if they used GANs on the intermediate features representations e.g. Denton Birodkar (2017)). In line with other work Hoshen & Malik (2019) we found that perceptual losses are very helpful for latent optimization.
a perceptual loss, originally trained the imagenet
7 CONCLUSION
We present effective approach for class-supervised image disentanglement, using shared latent optimization, an asymmetric regularization and a second amortization stage for single-shot gen- eralization. Our approach achieves state-of-the-an performance compared to both adversarial and non-adversarial disentanglement methods. We finally show how style clustering can extend our method for tackling domain translation inter-class disentanglement with promising results.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. ICML, 2018.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Thirty-Second AAAI Con- ference on Artificial Intelligence, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2019.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-imal translation. In CVPR, 2018.
Emily L Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In NIPS, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
Naama Hadad, Lior Wolf, and Moni Shahar. A two-step disentanglement method. In CVPR, 2018.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and VSR Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In ECCV, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Yedid Hoshen and Jitendra Malik. Non-adversarial image synthesis with generative latent nearest neighbors. CVPR, 2019.
Yedid Hoshen and Lior Wolf. Nam: Non-adversarial unsupervised domain mapping. In ECCV, 2018.
Phillip Isola, Jan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv:1802.05900 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Oliver Langner, Ron Dotsch, Gijsbert Bijlstra, Daniel HJ Wigholdus, Skyler T Hawk, and AD Van Knippenberg. Presentation and validation of the radboud faces database. Cognition and emotion, 277 2010.
Ivan Laptev, Barbara Caputo, et al. Recognizing human actions: local svm approach ICPR, 2004.
http://www.ndaa......ccaaaaaaaap
LeCun, Fu Jie Huang, and I. Bottou. Learning methods for generic object recognition with invariance to pose and lighting https://cs.nu..edu/ ylclab/dataanorb-vl... 100-smal// In CVPR, 2004.
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. arXiv
preprint arXiv: /001. 701.0/036. 2017.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In NIPS, 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. htp://mmlab.iee www.eeccc...cccceeeeee. www.k.tt....//pppesseeeeeeaaaaae In ICCV, 2015.

---

Published as a conference paper at ICLR 2020
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. ICML, 2019.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation deep representation using adversarial training. In NIPS, 2016.
Mckinsey. Anime face dataset, 2019. Available at:
Anime Prace-Datase
Tanmoy Mukherjee, Makoto Yamada, and Timothy Hospedales, Learning unsupervised word trans- lations without adversaries. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 627-632, 2018.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint
00446, 2019.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual mallogy-makinng http://iiwbb.oomm/////////////aa -al-aaaaytelsooofooooo In NIPS, 2015.
Attila Szabó. Qiyang Hu, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Challenges in disentangling independent factors of variation. ICLRW, 2018.
Aron Yu and Kristen Grauman. Fine- sine-grained visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 192-199, 2014.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
The architecture of the generator consists of 3 fully-connected layers followed by 6 convolutional layers (the first of them are preceded by an upsampling layer and followed by AdaIN normaliza- tion). We set the size of the content latent code to 128 and the size of the class code to 256 in all our experiments. We regularize the content embeddings with an additive gaussian noise with μ = and σ = 1 and an activation decay with À 0.001. We perform the latent optimization using SGD utilizing the ADAM method for 200 epochs, with learning rate of 0.0001 for the generator and 0.001 for the latent codes. For each mini-batch, we update the parameters of the generator and the latent codes with single gradient step each. For the second stage, the class and content encoders are CNNs with 5 convolutional layers and fully-connected layers.
A.2 DATASETS
Cars3D (Reed et al. spaced 24 azimuth directions and 4 elevations. We define the car model as the class and the rest as content. We use 163 car models for training and the other 20 are held for testing.
2015;: This dataset consists of 183 car CAD models, each rendered from equi-
SmaltNorb (LeCun et al. 2004): This dataset contains images of 50 toys belonging to 5 generic categories: four-legged animals, human figures, airplanes, trucks, and cars. The objects were imaged by two cameras under 6 lighting conditions, 9 elevations (30 to 70 degrees every 5 degrees), and 18 azimuths (0 to 340 every 20 degrees). We use this dataset in two configurations: i) SmallNorb: 25 separate identities for training and 25 for testing, treating lighting and elevations as part of the object class, and azimuth as the varied content. This configuration used for evaluating the generalization capability of the disentanglement methods from very limited set of seen classes. ii) SmallNorb- Poses: Using all the classes for training, holding out 10% of the images for testing. this case we treat the elevation part of the varied content as well.
12

---

Published as a conference paper at ICLR 2020
0.-
014
044
20
40
80
20
100
120
40
C0
80
100
120
coo8 index
code indea
Figure 8: Evidence for partial posterior collapse with KL.-divergence. 126 out of 128 components of the content code collapse to match a perfect standard normal distribution with zero mean and unit standard deviation. The remaining two components sustain much higher mean and much lower standard deviation. This prevents the regularization from acting as a tight bottleneck.
CelebA (Liu et al. 2015|: CelebA contains 202,599 facial images of 10,177 celebrities. The faces are aligned and cropped to contain only the facial region. We designate the person identity as the class, and transitory facial attributes such as head pose and expression as content. 9, 9,177 classes are used for training and the other 1,000 are held out for testing.
KTH (Laptev al. 2004y: KTH contains videos of 25 people, performing 6 different activities in different settings. We designate person identity as class, and transitory attributes (predominantly skeleton position) content. Due the very limited amount of subjects, for training, holding out 10% of the images for testing.
all the identities
RaFD (Langner et al. 2010): RaFD consists of 4,824 images collected from 67 participants making eight facial expressions in three different gaze directions, which are captured from three different angles. We treat the facial expression class and rest varied content, holding out 10% of the images for testing.
Edges2Shoes
YYu & Grauman 2014): A collection of 50,000 shoe images and their edge maps.
Anime Mckinsey 2019): A dataset consisting of 63,632 anime faces.
In all the experiments, images resized to 64x64 resolution fit the same architecture in LORD and the baselines. For evaluation on RaFD we follow the protocol in StarGAN (Choi et al. 2018 and crop the images to 128x 128.
A.3 KL-DIVERGENCE POSTERIOR COLLAPSE
We provide evidence for partial posterior collapse we experienced when regularizing the content codes with KL-divergence. Fig. 8 shows the mean and standard deviations of each of the 128 components the content code (averaged over all samples in the dataset) in a model trained on SmallNorb. It can be seen that 126 out of 128 components of the content code collapse to match a perfect standard normal distribution, while in the remaining 2 components the standard deviation reduced dramatically along with substantial increase in the mean. This phenomenon implies that regularizing the distribution of the content codes with KL-divergence may require additional attention and careful hyperparameter tuning. We find in our experiments that the asymmetric regularization introduced in our method results in better disentanglement.
A.4 INDUCTIVE BIAS LATENT OPTIMIZATION
We further provide the train and test losses (along with their decomposition into reconstruction and regularization terms) for the different variants LORD in Tab. S can be seen that the semi- amortized model (with asymmetric noise) achieves slightly lower reconstruction loss and lower
13

---

Published as a conference paper at ICLR 2020
Table A summary of train test losses of the different variants of LORD on Cars3D, Regulariza- tion measures the activation penalty of the content codes.
Reconstruction
Regularization
Total
Ours amortized (w/ KI.-divergence)
40.53 / 50.90
3.13 3.20
40.54 / 50.90
Ours amortized (w/ Asymmetric noise)
26.62 42.02
57.25
26.68 / 42.08
Ours semi amortized (w/ KL-divergence)
16.07 / 45.41
10.24/9.22
16.08 / 45,42
Ours semi amortized (w/ Asymmetric noise)
13.29 44.39
55.47 /1.995
13.34 / 44.44
Ours
13.88 / 44.78
56.17 /4.65
13.94 / 44.83
activation penalty (regularization of the content codes) than our latent optimization (fully unamor- tized) model. This emphasizes the effect of the inductive bias conferred by latent optimization which despite the higher losses results in better disentanglement performance, presented in Tab. and The second semi-amortized model, regularized with KL-divergence, achieves a much lower activation penalty in the content codes it collapses almost all means to zero. In all the experi- ments we set À = 0.001 (Eq. 6). In CelebA and SmalINORB, we failed to achieve better optima in both reconstruction and regularization losses, when using the semi-amortized model compared to our unllorraaddt model: [CelebA] ours: Rec = 100.38, Reg 42.99 semi-amortized (A = 0.001): Rec 82.07, Reg 128.84 semi-amortized (A 0.01): Rec 104.23, Reg 8.54. [SmalINORB] ours: Rec = 32.09, Reg 16.54 semi- amootizort (A 0.001): Rec 31.15,
Fig. 7
Reg 17.83. This behaviour emphasizes the difficulties in balancing the objectives for encouraging disentanglement using amortized inference.
A.5 QUALITATIVE RESULTS
We provide more qualitative results in Fig. 9
10
and
A.6 QUALITATIVE COMPARISON BASELINES
In addition to the quantitative assessment presented in the ablation study, we provide a qualitative comparison on CelebA of our method and 3 of baselines in Fig. 14
A.7 INTRA-CLASS VARIATION
Our image formation model, models images as being formed by class, content and residual (style) codes. The intra-class variation is formed by both the content and the residual information. The content is transferable between classes, the residual information codes, the residual information is small, reconstruction will be successful (as demonstrated in our experiments). If the residual information is very significant, it will not be possible to reconstruct images well only based on class and content leading to poor image formation models. For example, in the Cars3D experiment, the class labels represent the car model, content codes represent azimuth and elevation, and there is no residual information. In this case LORD performs well. We perform an exploratory experiment in which we aggregate similar car models into a single unified class (163 original car models are clustered into 50 super classes). In this case, the residual information contains the specification of the exact model within the super class. The residual information is therefore significantly larger. The class and content information not sufficient for reconstructing the original image perfectly. We demonstrate the degradation in the reconstruction in Fig. 15
not. Given class and content
14

---

Published as a conference paper at ICLR 2020
NA
20
Figure 9: More qualitative results of our method in transferring content between classes on CelebA
15

---

Published as a conference paper at ICLR 2020
2
Figure 10: More qualitative results of our method in transferring content between classes on CelebA
16

---

Published as a conference paper at ICLR 2020
Figure 11: More qualitative results of method in transferring content between classes Cars3D.
17

---

Published as a conference paper at ICLR 2020
Figure 12: More qualitative results of method in transferring content between classes on Small- Norb.
18

---

Published as a conference paper at ICLR 2020
Input
Angry
Contempt.
Disguste
Fearful
Happy
Sad
Surprised
Figure 13: More qualitative results of our method in facial expression transfer on RaFD.
A.8 STYLE CLUSTERING
We describe the preliminary step of per-class style clustering in Alg. 1
Algorithm 1: Style clustering
Input: images I1, .2,,.., In. E X and respective class labels E [k]
Number of styles per class E N
Feature extraction function ø X Rd
Output: Per class style labels th [n] + [k] x []]
Vi
In] fi <- ((x:))
// extract features from images
Vj
E 4 t k-means ((falys = j])
// cluster class j into 1 styles
Vi
m).
(b)) 4)
// assign joint class and style labels
We provide samples in Fig. 16 of clustering shoe images with k-means (k = 2, = 100) using style features et al. 2017) extracted from pretrained VGG model.
A.9 DOMAIN TRANSLATION RESULTS
We extend unsupervised domain translation approach support multi-attribute class labeling. We use the 40 annotated attributes in CelebA and use them to supervise 40 different shared class embeddings in addition single content embedding optimized per image. We demonstrate the effectiveness of this extended LORD approach in translating males to females and vice versa in Fig. 17
19

---

Published as a conference paper at ICLR 2020
(a) Amortized (w/ KL-divergence)
(b) Amortized (w/ Asymmetric noise)
(c) Semi-amortized (w/ Asymmetric noise)
(d) Ours
Figure 14: Qualitative comparison on CelebA between our method and 3 of our ablation base- lines. Fully-amortized models (a, b) fail preserve the class (person identity) across different content codes and introduce several artifacts, showing their lower degree of disentanglement. Semi- amortized model regularized with asymmetric noise (c) transfers over unreliable properties between identities (such as hair style). Our model (d) learns to disentangle the intrinsic characteristics of each identity and provides the best disentanglement and highest quality.
20

---

Published as a conference paper at ICLR 2020
Figure 15: A visualization of the degradation in reconstruction and disentanglement quality in cases where classes exhibit intra-class variations. It can be observed that the car model is not preserved well across different content codes.
Figure 16: Random samples from clusters of shoe images formed by k-means style features extracted from a pretrained VGG model.
21

---

Published as a conference paper at ICLR 2020
(a) Males to Females
(b) Females to Males
Figure 17: Examples of translations between genders faces from CelebA using our method.
22


Reviews:
[{"review_id": "rkeZVQEecr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: this paper proposes to basically combine class-conditional noisy autoencoding with GLO to achieve disentanglement while having only access to the content labels. They demonstrate that the method achieves impressive empirical results both in terms of disentanglement and a limited experiment on unsupervised domain translation.\n\nDecision: Weak Reject. I find the experimental results in this paper very appealing. However, I am weakly rejecting the paper because of 1) some problematic claims put forth in the paper, which I worry might mislead the reader and 2) lack of clarity in describing the procedure in the unsupervised domain translation setting.\n\nHere are some main comments:\n\n1.  KL-divergence v. asymmetric noise\nFirst, the authors claim that regularizing with KL-divergence leads to posterior collapse. But the particular experimental set up is tested on SmallNORB, which only has a small handful of factors of variation anyway). That KL-divergence “causes” posterior collapse is a claim that must be made very carefully. There are some very specific conditions under which this is known to be true empirically (for example, see the experiments in Burda’s IWAE paper and Hoffman’s DLGM paper), but in general, one should be careful with this claim. Can the authors please walk back on this statement?\n\nSecond, it is worth noting that asymmetric noise regularization is itself actually a special case of KL-divergence regularzation. When q(z|x) is forced to have a globally fixed variance, KL-divergence regularization becomes asymmetric noise regularization. \n\n2. Cost of training\nOne thing I feel should be made more clear in the paper is the training cost of GLO v. amortized models. How much slower is GLO compared to amortized models? How many iterations do you employ on a given minibatch of data when using GLO? \n\n3. Ablation study\nFirst, I think the authors should show us the actual visualizations for the amortized models. Without visual inspection, it’s hard to gauge the significance of the numbers in Table 3. \n\nSecond, the authors observed that the amortized models leak class information into the content representation. I find it fascinating that GLO does not. I would like the authors to dig deeper into what exactly is the inductive bias conferred by latent optimization. As of the moment, claim that “this variant is inferior to our fully unamortized model as a result of an inductive bias conferred by latent optimization” is a vacuously true statement since we know that amortized models and unamortized models should in theory have equivalent behavior in the infinite-capacity / oracle optimizer regime. I request that the authors show us the training and test losses (Eq 6 and its decomposition into reconstruction + regularization terms). Inspecting it may shed light on the inductive bias. \n\n4. Unsupervised Domain Translation\nThe result looks very good. However, the experimentation is too limited. I recommend that the authors try at least one other dataset.\n\nFurthermore, the description of how to apply LORD to unsupervised domain translation is uncomfortably vague. I am not sure if the provided code and description in the main text allows for reproduction of the UDT experiments. \n\nIf the authors are able to address the above questions and requests, then I am more than happy to raise my score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJxMpdc5KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes LORD, a novel non-adversarial method of class-supervised representation disentanglement. Based on the assumption that inter-class variation is larger than intra-class variation, LORD decomposes image representation into two parts: class and content representations, with the purpose of getting disentangled representation in those parts.  \nInspired by ML-VAE, authors try to: 1. learn to reconstruct the original image of disentangled representation. 2. eliminate the class information from content code by asymmetric noise regularization. The experimental results indicate that LORD succeeds to disentangle class information on content codes, while it outperforms style-content disentangled representations on style switching tasks (Figure 2 & 3).\nStrengths: \n1.LORD achieves significantly better performance than the state-of-the-art baseline on non-adversarial disentanglement methods.\n2., In terms of confusing the classifier in “Classification experiments” (Table2), disentangled content representation of LORD behaves like a random guess. This shows that LORD is indeed in preventing class information from leaking into content representations.\nWeaknesses:\n1. This paper is based on the assumption that “inter-class variation is larger than intra-class variation”. Authors should verify their assumption by quantitative results and illustrate the importance of inter/intra-class variation (e.g. how much information we may lose if ignoring the intra-class variation).\n2. Authors claim that no information leakage between class and content representation in Sec 1.1. However, the experiments only verify “no class information in content code”, but miss the inverse proposition (Is there any content information is class code?)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyeLAxM9FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework, called LORD, for better disentanglement of the class and content information in the latent space of image representations. The authors operate in a setting where the class labels are known. The authors perform \noptimization in the latent space to obtain these representations and argue that this is simpler and effective than adversarial and amortized inference approaches.\n\nThe main issue that the authors tackle is the information leakage between the representations for the class and content. The authors suggest several fixes for this. Firstly, the paper makes a distinction between content and style. Content is defined as the information that is unchanged when only the class labels are changes in the data generative process. The inherent randomness in the data generative process is defined as the style. \n\nTo disallow, the leakage from content/style code to class code, the authors suggest learning fixed codes for each class that does not vary across images. That is if two images have the same class by virtue of design they will have the same class codes. \n\nThe reverse, leakage from class codes to content codes is achieved by adding any asymmetric noise regularization term. This also seems to be aimed at reducing the total variability in the content codes. The authors claim that this is better than the bottleneck approach such as matching the code distribution to uniform prior and provide empirical evidence. Though in theory, it is not clear why one is better than the other. How was the sigma tuned for the regularization? Are the results dependent on this parameter?\n\nAfter learning the class and content embeddings for each sample in the training example, a pair of encoders are learned to predict these codes for unseen test images, without the need for optimization. \n\nOther comments:\n\nThe style code being 0 is not clear. Does the randomness in content code during the training account for the variations in the images not covered by class code and content code. \n\nThe methods seem heavily reliant on the imagenet trained VGG perceptual loss. This does not seem to be an issue in the datasets shown, do the authors anticipate any limitations generalizing to datasets such as in medical domains, etc. \n\nWhy is lighting chosen as the class label in the datasets? It will be interesting to see how the results change with different subsets of class labels and what is captured in the style codes. \n\nWhat are limitations from the assumption of low variability within a class?\n\nTypo: Page 4 - minimally -> minimality \n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for class-supervised disentangled representation learning. The method augments an autoencoder with asymmetric noise regularisation and is able to disentangled content (class) and style information from each other. The reviewers agree that the method achieves impressive empirical results and significantly outperforms the baselines. Furthermore, the authors were able to alleviate some of the initial concerns raised by the reviewers during the discussion stage by providing further experimental results and modifying the paper text. By the end of the discussion period some of the reviewers raised their scores and everyone agreed that the paper should be accepted. Hence, I am happy to recommend acceptance.
Decision: Accept (Poster)