Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: # Summary
This paper proposes a new method called Disentangled Exploration Auto-Encoder (DEAE). This new method is based on “(Ge at al.) Zero-shot synthesis with group-supervised learning” to which a modified cyclic loss term is added. This method is trained on datasets with label supervision.

# Pros
1. Compared to “(Ge at al.) Zero-shot synthesis with group-supervised learning” the results seem to be more visually pleasing.

# Cons
1. The method is not clearly presented. In particular the loss terms are not mathematically expressed as equations. I don’t know if we’re expected to read the paper this method is based on to have detailed equations. In any case, the added cyclic loss is expressed as an equation either, so it’s really hard to say what the method really does. It’s also unspecified how the latent space is allocated to attributes.
2. The scope of the method is not clearly presented either: I was under the impression that the proposed method was comparable to other unsupervised auto-encoders while in fact it requires label supervision
3. It seems the paper claims to be more than it actually is: if I understand correctly, the sole contribution of this paper is a cyclic loss term, compared the (Ge et al) paper which can be seen as a regularizer.
4. Experiments are mostly focused on visual inspection of images and do not appear impressive to me (except for the toy dataset of colored letters, but then I don’t know what the SotA is like). Very few numerical results (save for dataset bias elimination) are presented. Results are not compared to other SotA methods, except the (Ge at al) paper which the method is based upon. Experimental setup is very incomplete.

# Questions and nits
1. It would have benefited my comprehension to mention in the abstract that the proposed method requires attribute/label supervision. In its current form, it seems to claim to solve disentanglement for unsupervised auto-encoders and I find it misleading.
2. The mention of the generative ability without GAN based training in the abstract is also confusing: the proposed method seems orthogonal to GANs and could be combined with them.
3. Several times I see the term “perfect disentanglement” to describe the improved disentanglement that this method offers compared to (Ge et al). What I don’t understand is what makes it “perfect”, can’t it be further improved?
4. “<mention unsupervised auto-encoders>. We propose a different solution to empower precise attribute controllable synthesis ability on autoencoders: DEAE”. To be fair, it’s also a solution to a different problem scope. The proposed method uses attributes while the cited methods don’t. So it’s really a solution to a different problem altogether.
5. Typo “whic h” => “which”
6. “Fig. 4 (d) shows that we can combine the UDVs to dicover new attribute values.” Aside from the typo on "discover", it’s unclear how you discover new attributes values (which I assume are centroid like the example you mentioned before for the blue color). Here instead, my understanding is that UDV just provides a vector along which values of interest may lie. It seems the eventual decision to make a value an attribute is manually decided by a human after inspecting the effects along an UDV axe.
7. Downstream task performance. It is a toy dataset and it’s hard to really tell the real power of the proposed method. A lot of information is missing, what are the sizes the $D_S$ and $D_L$? The only reported numbers are $D_S$ vs $D_{S+DEAE}$. What is the unreported accuracy gap between $D_{S+DEAE}$ and $D_{S+GSL-AE}$ that leads to the later conclusion that DEAE performs better? What is the accuracy for $D_L$? No information is known about the classifier network architecture(s), parameter sizes, tuning and whether or not they overfit or what other causes could be responsible for the observed results.

=====POST-REBUTTAL COMMENTS======== 
I thank the authors for the response and the efforts in the updated draft. Some of my queries were clarified, particularly concerning missing experimental details. However, unfortunately, I still think more needs to be clarified in the actual paper write up, notably on the points of non-adversarial as well as the method description.
review_title: Need to clearly state scope, method, contributions and to provide a more rigorous experimental setup.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: The paper proposes a new framework for adding a generative component to autoencoders without using adversarial or variational techniques. Further, the true motivation is to overcome issues due to naive interpolation (resulting in out of distribution samples) and to provide a semantic controllable synthesis in autoencoders. Empirical results show the proposed DEAE succeeds in both.

Strengths:
* I appreciate the intuition and examples throughout the paper, regarding the method itself and description of the experiments and desired outputs. This makes the paper easy to read and follow.
* The provided results support the claims (although they lack exhaustiveness, see below).


Weaknesses:
* Although I agree with the intuition of the method, no theoretical analysis is provided to justify the proposed solution. 
The empirical results seem to confirm the statements of contribution but in a very limited setup. A more exhaustive presentation would be more convincing.

Although the paper builds on previous work, I find it quite interesting and significant to have a method for controllable synthesis. If the authors address my concerns, I am willing to increase my score.  

Questions:

Methodology:
The main novelty in the proposed method is the L_{reg} loss term in the training of the DEAE, which was never explicitly written or explained beyond the intuitive interpretations. 


Is there any theoretical direction you can propose for justifying the claim that adding the regularisation loss helps to turn non-convex to convex latent spaces? If not, at least a toy experiment to validate this assumption might help.

Missing reference:
The technique proposed for novel attribute mining, UVD is quite similar to the concept vectors proposed in TCAV [1].

Experiments:
Only a few quantitative results were presented, and those are also missing standard deviations. Please add those to make the results complete.

The results for experiment 3.2, were the MSE losses normalized? If the latent spaces correspond to different models, how do we know if the numbers are comparable?

Other baselines:
- Generative models: at least some comparison to vanilla VAE, GANs as a sanity check (if the claim is generative AE)
- Why is GSL-AE not used in 3.4?
- Could you please provide more plots of the nice results (Figures 5 and 6) in the appendix just to confirm the presented results were not “cherry-picked”.


Minor:
- In Figure 5 the acronym for general AE is ‘AE’ instead of ‘GAE’
- I don’t see the need for  ‘non-adversarial’ in the title.

[1] Kim, Been, et al. "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)." International conference on machine learning. PMLR, 2018 
review_title: Good paper however the results can to be more exhaustive 

final_score: rating: 3: Clear rejection
review_content: Summary

This paper proposed to explore a given disentangled latent code and explore it through random interpolations. The idea is to ensure these interpolated codes rely inside data convex hull by enforcing latent code reconstruction by first decoding and then reencoding this explored codes. The authors proposed a way to manipulate the final latent space by using Unit Direction Vector (UDV) to generate unseen attribute values. The results are illustrated on 2 datasets (Fonts and RaFD) both qualitatively and quantitatively.

Reason for score

Overall I vote for reject, this paper has limited contribution and lack of consistency (disconnected ideas: DEAE and UDV), clarity (respective contributions of GSL-AE and DEAE) and justification (spurious and not convincing experimentations).


Pros
1. The general topic of having more controlable synthesis by having more disentangled representation if of clear interest for the field. 
2. The idea of reconstructing random interpolations of valid latent codes by decoding/reencoding is interesting. I don't think this is novel, but not widspread enough in the field so far, so the interest remains. 

Cons
1. While the topic is of interest for the field, the interest of the proposed approach is clearly limited since it relies on a preprocessing step (GSL-AE) of which it is already the role. A clearer distinction of the contributions of each step would help to illustrate the additional interest of DEAE. Unfortunatly the paper relies too much on GSL-AE which limits the potential scope of their main contribution which could be wider.
2. The title is misleading since it presents the approach to proposed a "disentangled exploration" while it is essentially provided by GSL-AE used as a preprocessing step. Thus, it would also be good to mention that the approach is supervised (like GSL-AE).
3. The paper lacks of clarity: UDV or "dataset bias elimination" seems to be  spurious addition and are not really well connected in the paper. It's hard to understand how it contributes to illustrate the interest of DEAE.
4. Results are not convincing.
- MSE loss between latent spaces from different methods cannot be compared at all. We can imagine spurious differences only due to scale differences for instance.
- Fig 2. is mentioned p.5 to illustrate the qualitative superiority of DEAE over GSL-AE: not really obvious of this state.
- Section 3.3 + fig 7.: interpolation is not smooth which contradicts the quality of the representation and the interest of UDV.
- Section 3.4: the idea of evaluating the performance of generative methods through their capacity of improving a classification task by augmenting the initial dataset, is very indirect and not really convincing.
- Gradcam results (Fig. 8) does not illustrate any improvement in favor of background information for the "unbiased model"
5. "Dataset bias elimination"
- This section seems spurious. Nothing in the proposed approach is really specific to such an application appart the global context to which it is attached. I don't see what this section brings to the paper.
- The way the unbias model is obtained is not realistic since you need to know what information has to be dropped, whereas in real applications you generally don't know the source of bias.
6. While having thinner way of exploring the latent space is interesting, the proposed UDV approach sounds spurious and ad hoc and does not seem to be connected to DEAE.

Questions
1. What is the context of application of your approaches since it relies on a supervised approach to create the initial disentangled embedding ? Coutrolable synthesis is generally understood in an unsupervised setting.
2. Is there a link between UDV exploration and the way DEAE is learnt ?
3. Why using "random interpolation" for inference and illustrating performance (in Fig. 5) ? It would be clearer with standard interpolation between left and right bounds. I also would expect that interpolations would reach (or at least get closer to) the targeted right bound.
4. How do you ensure that MSE (from section 3.2) can be compared between totally different latent spaces ?
5. Section 3.4: Why D_S is used rather than D_L for further analyses ? (to get augmented datasets)


Minor comments
1. Intro: Flows could be mentionned on top of VAE and GANs
2. Intro on GAN & VAE limits are too much like a caricature regarding recent results. VAE can be used to generate HD images now, and GAN are much easier to train these days with GP strategy or different learning rates between generator and discriminator for instance. Approaches like InfoGAN could be mentioned to illustrate standard strategies to control GANs.
3. Fig 2. Not clear which kind of interpolation has been used for GSL-AE and DEAE since neither font, font size, font color, letter nor background color are interpolated in the figures. This figure would be more powerful to stick to one (or a few) letter(s) and change only the considered attribute (background or font color). Here the comparison and the illustration of the effect is not easy.
4. Section 2. "whic h maps"  "which maps"
5. notation "*" has not been introduced (like in f_theta^* and g_phi^*)
6. L_reg has not been introduced or referred in the paper.
7. Fig 7. Choosing a fg color different from the targeted bg color would be better
review_title: This paper proposed to refine an encoding by latent code interpolation followed by decoding/reencoding for cycle reconstruction. A further exploration approach UDV is proposed to generate unseen combination of attributes.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

The paper presents an approach for controllable generation with auto-ecoders. It uses interpolation in a disentangled latent space using Group Supervised Learning. Generated samples are encoded back to the latent space as additional regularization to improve sample quality. The authors perform experiments on Fonts and RaFD datasets for controllable generation. They also show applications of their model in improving performance of classifiers and reducing bias.  
  
###########################################################

Strengths: 

The problem of controllable image generation is important and current approaches are far from perfect. Autoencoders with disentangled latent space are a promising candidate for this problem as they can both reconstruct the input and control the attributes. 

###########################################################

Weaknesses:
1. Literature review is inadequate and there are limited comparisons with related work. There are several other approaches that are not mentioned in the paper. Fader networks [A] also uses an encoder-decoder architecture trained to reconstruct images by disentangling salient information of the image and the values of attributes directly in the latent space. They provide a more comprehensive evaluation on CelebA and Flowers datasets while this paper use relatively simpler datasets, Fonts and RaFD. The authors need to clarify differences of their approach with [A] and provide comparisons with them. They need to include quantitative scores for their controllable generation. There are also other relevant papers such as Deep Feature Interpolation [B].
2. Experiments are mostly performed on the Fonts dataset which is relatively simple. It would be better to perform experiments on datasets such as CelebA which has attribute annotations. 
3. Novelty of the proposed approach is limited as the authors use an existing method (GSL) for disentangling the latent space and also encoding the image back to latent space is proposed in other works such as InfoGAN and BicycleGAN.
4. To obtain UDVs the authors train a binary classifier for each color value which is not efficient. 
5. It seems the bias experiment assumes that we know which attributes are entangled which might not be realistic. 
6. In figure 7(a) changes of colors are discrete while continuous changes are more desirable. 
7. In the results in Figure 2, it seems changing one attribute also affects the other.   

############################################################

Reason for Rating:

Overall, while autoencoders with disentangled latent space are a promising candidate for controllable generation, experimental results and novelty of the proposed approach are inadequate for publication. 

############################################################

References: 

[A] Fader Networks: Manipulating Images by Sliding Attributes, Lample et al., NIPS 2017 

[B] Deep Feature Interpolation for Image Content Changes, Upchurch et al., CVPR 2017

#############################################################

After author response: I thank the authors for their answers. However, as noted by other reviewers, experimental results and comparisons with related work are lacking, and I cannot increase my score without major changes to the paper. 
review_title: Interesting problem but novelty and evaluation are limited  

Meta-review: Overall, the paper makes some interesting and intuitive observations regarding the autoencoders with a cycle consistency, and aims at achieving controllable synthesis via a disentangled representation. However, the overall consensus was that the manuscript needs further iterations:

In particular:
The ideas should be made more precise using mathematical arguments, as it stands some ideas are (e.g. DEAE and UDV) disconnected.

The scope needs to be clarified, e.g. respective contributions of GSL-AE and DEAE, use of label information 

More numerical/quantitative evaluations, the current experimentation is not convincing enough, needed for better justification (spurious and not convincing experimentations)

The English of the manuscript could be improved as it occasionally hampers the flow.

Decision: Reject