Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary: this paper proposes to basically combine class-conditional noisy autoencoding with GLO to achieve disentanglement while having only access to the content labels. They demonstrate that the method achieves impressive empirical results both in terms of disentanglement and a limited experiment on unsupervised domain translation.

Decision: Weak Reject. I find the experimental results in this paper very appealing. However, I am weakly rejecting the paper because of 1) some problematic claims put forth in the paper, which I worry might mislead the reader and 2) lack of clarity in describing the procedure in the unsupervised domain translation setting.

Here are some main comments:

1.  KL-divergence v. asymmetric noise
First, the authors claim that regularizing with KL-divergence leads to posterior collapse. But the particular experimental set up is tested on SmallNORB, which only has a small handful of factors of variation anyway). That KL-divergence “causes” posterior collapse is a claim that must be made very carefully. There are some very specific conditions under which this is known to be true empirically (for example, see the experiments in Burda’s IWAE paper and Hoffman’s DLGM paper), but in general, one should be careful with this claim. Can the authors please walk back on this statement?

Second, it is worth noting that asymmetric noise regularization is itself actually a special case of KL-divergence regularzation. When q(z|x) is forced to have a globally fixed variance, KL-divergence regularization becomes asymmetric noise regularization. 

2. Cost of training
One thing I feel should be made more clear in the paper is the training cost of GLO v. amortized models. How much slower is GLO compared to amortized models? How many iterations do you employ on a given minibatch of data when using GLO? 

3. Ablation study
First, I think the authors should show us the actual visualizations for the amortized models. Without visual inspection, it’s hard to gauge the significance of the numbers in Table 3. 

Second, the authors observed that the amortized models leak class information into the content representation. I find it fascinating that GLO does not. I would like the authors to dig deeper into what exactly is the inductive bias conferred by latent optimization. As of the moment, claim that “this variant is inferior to our fully unamortized model as a result of an inductive bias conferred by latent optimization” is a vacuously true statement since we know that amortized models and unamortized models should in theory have equivalent behavior in the infinite-capacity / oracle optimizer regime. I request that the authors show us the training and test losses (Eq 6 and its decomposition into reconstruction + regularization terms). Inspecting it may shed light on the inductive bias. 

4. Unsupervised Domain Translation
The result looks very good. However, the experimentation is too limited. I recommend that the authors try at least one other dataset.

Furthermore, the description of how to apply LORD to unsupervised domain translation is uncomfortably vague. I am not sure if the provided code and description in the main text allows for reproduction of the UDT experiments. 

If the authors are able to address the above questions and requests, then I am more than happy to raise my score.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes LORD, a novel non-adversarial method of class-supervised representation disentanglement. Based on the assumption that inter-class variation is larger than intra-class variation, LORD decomposes image representation into two parts: class and content representations, with the purpose of getting disentangled representation in those parts.  
Inspired by ML-VAE, authors try to: 1. learn to reconstruct the original image of disentangled representation. 2. eliminate the class information from content code by asymmetric noise regularization. The experimental results indicate that LORD succeeds to disentangle class information on content codes, while it outperforms style-content disentangled representations on style switching tasks (Figure 2 & 3).
Strengths: 
1.LORD achieves significantly better performance than the state-of-the-art baseline on non-adversarial disentanglement methods.
2., In terms of confusing the classifier in “Classification experiments” (Table2), disentangled content representation of LORD behaves like a random guess. This shows that LORD is indeed in preventing class information from leaking into content representations.
Weaknesses:
1. This paper is based on the assumption that “inter-class variation is larger than intra-class variation”. Authors should verify their assumption by quantitative results and illustrate the importance of inter/intra-class variation (e.g. how much information we may lose if ignoring the intra-class variation).
2. Authors claim that no information leakage between class and content representation in Sec 1.1. However, the experiments only verify “no class information in content code”, but miss the inverse proposition (Is there any content information is class code?)

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a framework, called LORD, for better disentanglement of the class and content information in the latent space of image representations. The authors operate in a setting where the class labels are known. The authors perform 
optimization in the latent space to obtain these representations and argue that this is simpler and effective than adversarial and amortized inference approaches.

The main issue that the authors tackle is the information leakage between the representations for the class and content. The authors suggest several fixes for this. Firstly, the paper makes a distinction between content and style. Content is defined as the information that is unchanged when only the class labels are changes in the data generative process. The inherent randomness in the data generative process is defined as the style. 

To disallow, the leakage from content/style code to class code, the authors suggest learning fixed codes for each class that does not vary across images. That is if two images have the same class by virtue of design they will have the same class codes. 

The reverse, leakage from class codes to content codes is achieved by adding any asymmetric noise regularization term. This also seems to be aimed at reducing the total variability in the content codes. The authors claim that this is better than the bottleneck approach such as matching the code distribution to uniform prior and provide empirical evidence. Though in theory, it is not clear why one is better than the other. How was the sigma tuned for the regularization? Are the results dependent on this parameter?

After learning the class and content embeddings for each sample in the training example, a pair of encoders are learned to predict these codes for unseen test images, without the need for optimization. 

Other comments:

The style code being 0 is not clear. Does the randomness in content code during the training account for the variations in the images not covered by class code and content code. 

The methods seem heavily reliant on the imagenet trained VGG perceptual loss. This does not seem to be an issue in the datasets shown, do the authors anticipate any limitations generalizing to datasets such as in medical domains, etc. 

Why is lighting chosen as the class label in the datasets? It will be interesting to see how the results change with different subsets of class labels and what is captured in the style codes. 

What are limitations from the assumption of low variability within a class?

Typo: Page 4 - minimally -> minimality 





review_title: Official Blind Review #3

Meta-review: This paper proposes a novel method for class-supervised disentangled representation learning. The method augments an autoencoder with asymmetric noise regularisation and is able to disentangled content (class) and style information from each other. The reviewers agree that the method achieves impressive empirical results and significantly outperforms the baselines. Furthermore, the authors were able to alleviate some of the initial concerns raised by the reviewers during the discussion stage by providing further experimental results and modifying the paper text. By the end of the discussion period some of the reviewers raised their scores and everyone agreed that the paper should be accepted. Hence, I am happy to recommend acceptance.
Decision: Accept (Poster)