Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on latent representations learning when some labels are provided. The authors propose a method called the characteristic capturing VAE (CCVAE). This method learns real-valued auxiliary variables that capture the label information. The proposed method is tested on a medical image and a face image dataset.

One of the major motivations of the proposed method is that it can be used to conditionally generate images based on desired characteristics. However, this paper proposes a VAE model, which usually generates lower-quality images compared to Generative Adversarial Networks (GAN). It is not clear to me why this paper focuses on a VAE model rather than a GAN model. There have been GAN-based methods [1, 2] also based on an auto-encoding framework. Note that if we remove the adversarial loss from the objective functions of these methods, and add the KL divergence penalty, these methods become VAE. Since these methods also introduce real-value variables, I believe the authors should compare to the VAE-version of these methods. 

In addition, in the VAE literature, [3] proposes a semi-supervised method.  Since [3] also involves a real-value latent variable, it looks like the difference between the proposed method and [3] is that the author extends [3] to multiple labels. Is this true? I suggest the authors better clarify the novelty of the proposed method compared to [3]. 

I do not suggest accepting this paper because the relationships to some existing methods are not sufficiently discussed.

Minor:
In the experiments, the paper reports the quantitative measures for classification and disentanglement. However, no quantitative measures for image quality are reported. I suggest the authors report some measures such as reconstruction error and FID score.

References
[1] Xiao, Taihong, Jiapeng Hong, and Jinwen Ma. "DNA-GAN: Learning disentangled representations from multi-attribute images." International Conference on Learning Representation Workshop. 2018.

[2]Xiao, Taihong, Jiapeng Hong, and Jinwen Ma. "Elegant: Exchanging latent encodings with gan for transferring multiple face attributes." Proceedings of the European conference on computer vision (ECCV). 2018.

[3] Li, Yang, et al. "Disentangled variational auto-encoder for semi-supervised learning." Information Sciences 482 (2019): 73-85.
review_title: The relationships to some existing methods are not sufficiently discussed.

final_score: rating: 7: Good paper, accept
review_content: **GENERAL**
The paper proposes to re-think the fashion of using label information in the VAE framework. The authors propose to disentangle information about the label (or, more generally, the context) in a "hard-coded" manner, namely, by using a separate set of variables for the label (context). The paper is written in a lucid manner, and the presented results are sound.

**Strengths:**
S1: The presented VAE framework (CCVAE) is interesting.

S2: The final objective in Eq. (6) that follows from applying Jensen's inequality two times, allows to efficiently train the model. Moreover, it allows the use of interventions to modify information about images.

S3: The demo is a nice fashion of presenting the idea.

S4: I highly appreciate that the authors used a dataset outside of standard benchmarks, and they focused on a medical application (the Chexpert dataset).

S5: The authors explained all implementation details that increases reproducibility of the paper. 

S6: I appreciate the close relation of the propose framework to other models, e.g., DIVA. Moreover, the idea of improving upon DIVA is very interesting.

**Deficiencies:**
D1: My only concern is rather average generative capabilities of the model. Nevertheless, it does not affect my overall good assessment of the paper.

**Remarks:**
R1: In Eq. 2, there is a fudge factor, \alpha. However, in the CCVAE objective (Eq. 4) it is no longer necessary. I would like to ask the authors whether they thought about adding this fudge factor, or it is simply unnecessary. (I am aware that q(y|x) follows naturally from the objective, nevertheless, it might further improve the classification accuracy).
review_title: A new VAE-based framework for utilizing context information

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper the authors have propsed a method to incorporate label information 
in variational autoencoders (VAEs) to captures the characteristics associated with 
those labels. Experiments show that using the label information helps them to better
intervation and conditional generation of images.

The paper is well written. However I have few concern regarding the model.

Firstly the paper has not cited a very important related work by
Adel, Tameem, Zoubin Ghahramani, and Adrian Weller. "Discovering interpretable 
representations for both deep generative and discriminative models." International 
Conference on Machine Learning. 2018.
where a similar hiererchical approach was proposed. 
The authors should provide a discussion related to this method or if possible should 
compare their performance by extending the previous model for multiple features.

The main concerns regarding the model perspective are as follows:

1. How the authors ensuring that there is no mutual information between the parts z_c and z_\c ?
There is no analysis regarding the same. They should report some quanitative measures as proposed by
betaVae to support the same.

2. How they are controlling the KLD between encoding and decoing distribution i.e 
KL(q(z|x) || p(z| z_c)) and KL(q(z_c|z) || p(z_c|y)) ? If the first kl is not significantly low, 
there will be information loss which might affect the quality of the overall generation? 
They should report the KL achieved in each layer.

3. As they are learning the reverse mapping from a given label to corresponding 
representation i.e. p_\theta(z_c | y) -- 
is it necessary to make z_c unidimensional for interpolation ? 
How are they learning the correspondance between fin grained values of y 
and z_c ? do they have such label information available? 
review_title: Capturing Label Characteristics in VAEs 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update after rebuttal: the authors have clarified some of my concerns, and I have therefore increased my score.

---------------
This paper introduces the characteristic-capturing VAE, an architecture that extends VAEs by modelling label-dependent characteristics in generative and classification tasks. In a multi-label setting, the CCVAE learns representations of individual characteristics, which are disentangled by design to the one of other labels.

The paper is well written in general, and is a good contribution to the important research direction on deep generative models focusing on using some labelled data to learn a disentangled latent space.
The experiments in the paper are well thought and show convincing performances in terms of label-dependent generations and interventions.

My biggest doubts are in terms of the positioning/novelty of the paper, which I found a bit confusing/misleading in the current form. When "rethinking supervision" the authors argue for the need of treating labels as auxiliary variables - and not as latent variables - in the model. This leads to the formulation called ALVAE, that introduces label-dependent information in latent space learning jointly a classifier from z to y. This formulation is novel as far as I know, but leads to some challenges mentioned by the authors in terms of conditional generation/intervention with this model. The authors then offer as a solution the CCVAE, which uses a conditional generative model p(z_c|y)p(y).
* Is y in a CCVAE still an auxiliary variable? To me it seems that the benefits of the CCVAE are not due to the auxiliary nature of y, but more from the fact that y is now a partially observed latent variable that is however only connected to x through continuous latent variables. If this is the case, why is it relevant to leave in the paper the whole ALVAE discussion which is purely theoretical and not even presented in the experiments? Perhaps the authors should focus more instead on clarifying even further on the benefits of the hierarchical generative architectures y->z->x.
* The discussions in section 3 are valid for 1-layer models such as M2 or MVAE models in which z_y=y. This discussion is not however considering hierarchical VAE architectures for semi-supervised learning, such as the one presented in "Semi-Supervised Generation with Cluster-aware Generative Models" by Maaløe et al, 2017. These architectures also learn a latent space that captures the information associated with a class, not just the class itself, and do so without the need for "auxiliary" variables. Is there any advantage of the CCVAE has with respect to this model?
review_title: Interesting paper, but there are some needed clarifications

Meta-review: This paper proposes a novel technique to learn a disentangled latent space using VAEs and semi-supervision. The technique is based on a careful specification of the joint distribution where the labels inform a factorisation of the distribution over continuous latent factors. The technique allows for inference, generation, and intervention in a tractable way.

The paper is well-written, the formulation is original, and the experiments convincing. There were some confusions that were mostly resolved during the discussion.  

In addition to the expert reviews attached, I would like to remark that I too find the formulation interesting and elegant. And if I may add to the discussion, oiVAE (output-interpretable VAEs) by Ainsworth et al presented at ICML18 is a related piece of work that did not occur to me earlier, but which the authors could still relate to (I'd certainly enjoy reading about the authors' views on that line of work). 
Decision: Accept (Poster)