Capturing Label Characteristics in VAEs
Authors: Tom Joy
Sebastian Schmon
Philip Torr
Siddharth N
Tom Rainforth
Venue: ICLR 2021 Poster (2021)
Keywords: variational autoencoder
representation learning
deep generative models

Abstract: We present a principled approach to incorporating labels in variational autoencoders (VAEs) that captures the rich characteristic information associated with those labels. While prior work has typically conflated these by learning latent variables that directly correspond to label values, we argue this is contrary to the intended effect of supervision in VAEs—capturing rich label characteristics with the latents. For example, we may want to capture the characteristics of a face that make it look young, rather than just the age of the person. To this end, we develop a novel VAE model, the characteristic capturing VAE (CCVAE), which “reparameterizes” supervision through auxiliary variables and a concomitant variational objective. Through judicious structuring of mappings between latent and auxiliary variables, we show that the CCVAE can effectively learn meaningful representations of the characteristics of interest across a variety of supervision schemes. In particular, we show that the CCVAE allows for more effective and more general interventions to be performed, such as smooth traversals within the characteristics for a given label, diverse conditional generation, and transferring characteristics across datapoints.

Published as a conference paper at ICLR 2021
CAPTURING LABEL CHARACTERISTICS IN VAEs
Tom Joy', Sebastian M. Schmon" 1.2, Philip H. S. Torr', N. Siddharth 1,3 UUniversity of Oxford *Improbable a University of Edinburgh & The Alan Turing Institute to tomjoyyrobote ox.oxxxoo wx.oc.x......
11,3 & Tom Rainforth 1
ABSTRACT
We present principled approach incorporating labels in variational autoencoders (VAEs) that captures the rich characteristic information associated with those labels. While prior work has typically conflated these by learning latent variables that directly correspond to label values, we argue this is contrary to the intended effect of supervision in VAEs VAEs-captuuing rich label characteristies with the latents. For example, we may want to capture the characteristics of a face that make it look young, rather than just the age of the person. To this end, we develop the characteristic capturing VAE (CCVAR), novel VAE model and concomitant variational objective which captures label characteristics explicitly in the latent space, eschewing direct correspondences between label values and latents. Through judicious structuring of mappings between such characteristic latents and labels, show that the CCVAE can effectively learn meaningful representations of the characteristics of interest across a variety of supervision schemes. In particular, we show that the CCVAE allows for more effective and more general interventions be performed, such as smooth traversals within the characteristics for a given label, diverse conditional generation, and transferring characteristics across datapoints.
INTRODUCTION
Learning the characteristic factors of perceptual observations has long been desired for effective machine intelligence (Brooks 1991 Bengio et al. 2013 Hinton & Salakhutdinov [0066] Tenenbaum
1998). In particular, the ability to learn meaningful factors capturing human-understandable charac- teristics from data has been of interest from the perspective of human-like learning (Tenenbaum & Freeman 2000; Lake al. (2115) and improving decision making and generalization across tasks (Bengio et al. Tonenbbennnnau
Freeman 2000).
At its heart, learning meaningful representations of data allows one to not only make predictions, but critically also to manipulate factors of datapoint. For example, might want to manipulate the age of a person in an image. Such manipulations allow for the expression of causal effects between the meaning of factors and their corresponding realizations in the data. They can be categorized into conditional generation-the ability to construct whole exemplar data instances with characteristics dictated by constraining relevant factors-and intervention-the ability manipulate just particular factors for given data point, and subsequently affect only the associated characteristics.
A particularly flexible framework within which to explore the learning of meaningful representations variational autoencoders (VAEs), a class of deep generative models where representations of data are captured in the underlying latent variables. A variety of methods have been proposed for inducing meaningful factors in this framework (Kim & Mnih 2018 Mathieu et al. 2019 Mao al.) 2019 Kingma et al., 2014 Siddharth et al. 2017 Vedantam et al. 2018). and it has been argued that the most effective generally exploit available labels to (partially) supervise the training process (Locatello al. 2019). Such approaches aim to associate certain factors of the representation (or equivalently factors of the generative model) with the labels, such that the former encapsulate the latter-providing mechanism for manipulation via targeted adjustments of relevant factors.
work done while at Oxford
cqual contribution

---

Published as a conference paper at ICLR 2021
Prior approaches have looked to achieve this by directly associating certain latent variables with labels (Kingma et al. Siddhhrth et al. 2017): Maaløe et al., 2016). Originally motivated by the desiderata of supe-supe classifcation, each label Is given corresponding latent variable of the same type (e.g. categorical), whose value is fixed to that of the label when the label is observed and imputed by the encoder when it is not.
Though natural, we argue that this assumption is not just unnecessary but actively harmful from representation-leamiie perspective, particularly in the context of performing manipulations. To allow manipulations, we want to learn latent factors that capture the characteristic information associated with a label, which is typically much richer than just the label value itself. For example, there are various visual characteristics of people's faces associated with the label "young," " but simply knowing the label is insufficient to reconstruct these characteristics for any particular instance. Learning a meaningful representation that captures these charac- teristics, and isolates them from others, requires encoding more than just just the label value itself, as illustrated in Figure
Hair color
Characteristics of hair color
Canncterritics Lstiessstess of srile
The key idea of our work is use labels help capture and isolate this related characteristic information in a VAE's repre- sentation. We do this by exploiting the interplay between the labels and inputs to capture more information than the labels
Smile
alone convey; information that will lost (or at least entangled) if we directly encode the label itself. Specifically, we introduce the characteristic capturing VAE (CCVAE) framework, which employs a novel VAE formulation which captures label char- acteristics explicitly in the latent space. For each label, we introduce set of characteristic latents that induced into capturing the characteristic information associated with that label. By coupling this with a principled variational objective and carefully structuring the characteristic-latent and label variables. we show that CCVAEs successfully capture meaningful representations, enabling better performance on manipulation tasks, while matching previous approaches for prediction tasks. In particular, they permit certain manipulation tasks that cannot be performed with conventional approaches, such as manipulating characteristics without changing the labels themselves and producing multiple distinct samples consistent with the desired intervention. We summarize our contributions as follows:
Figure 1: Manipulating label character- istics for "hoir color" and "amile".
i) showing how labels can used to capture and isolate rich churacteristic information; ii) formulating CCVAEs, a novel model class and objective for supervised and semi-supervised learning in VAEs that allows this information to be captured effectively; iii) demonstrating CCVAEs' ability successfully learn meaningful representations in practice.
BACKGROUND
VAEs (Kingma & Welling 2013 Rezende et al. 2014) are powerful and flexible class of model that combine the unsupervised representation-llarre capabilities of deep autoencoders (Hinton & Zemel 1994) with generative variabbe models popular tool to capture factored low-dimensional representations of higher-dimensional observations. In contrast to deep autoencoders, generative models capture representations of data not as distinct values corresponding to observations, but rather as distributions of values. A generative model defines joint distribution over observed data x and latent variables as pa (xx, 2) = p(2 (z)pe 2). Given model, learning representations of data can be viewed as performing inference learninng the posterior distribution po (z distribution of latent values for given observation.
x)) that constructs the
VAEs employ amortized variational inference (VI) (Wainwright & Jordan 2008 Kingma & Welling 2013) using the encoder and decoder of an autoencoder to transform this setup by i) taking the model likelihood pa(x =2 to be parameterized by neural network using the decoder, and ii) constructing an amortized variational approximation (x(x x) to the (intractable) posterior po(z x) using the encoder. The variational approximation of the posterior enables effective estimation of the objective-mmximiin maximizing the marginal likelihood seeeihooo--hrouuhhhhhh importance sampling. The objective is obtained through invoking Jensen's 's inequality to derive the evidence lower bound (EL.O) of the
2

---

Published as a conference paper at ICLR 2021
model which is given
Po pz(z (z. x))
log Po(z. x)
log po(x) (z) = log (z)o))
2 (x(m()
= [(x; Φ. 0).
(I)
(s(= a:)
(o x))
Given observations D { ................................ ........ .N taken be realizations of random variables generated
from an unknown distribution pD(a)), the overall objective is * Σn (( 0, Φ). Hierarchical VAEs(Sonderby et al. (016) impose a hierarchy of latent variables improving the flexibility of the approximate posterior, however we do not consider these models in this work.
Semi-supervised VAEs (SSVAEs) (Kingma et al. 2014 Maaløe et al. 2016 Siddharth et al. 2017)
consider the setting where subset of data S C DD is assumed to also have corresponding labels Denoting the (unlabeled) data as l S), the log- oog-magginal likelihood is decomposed as
log (D)
[(m.)) log y) + EmEu log (o)).
Σ (*.y)ES
wEl
where the individual lig-liielsooods are lower bounded by their ELBOs. Standard practice is then to treat y as a latent variable to marginalize
whenever the label is not provided. More specifically, most approaches consider splitting the latent space in 2 = {zy. ² and then directly fix my = y whenever the label is provided, such that each dimension of 2n explicitly represents predicted value of a label, with this value known exactly only for the labeled datapoints. Much of the original motivation for this (Kingma al. 2014) was based around performing semi-supervised classification of the labels, with the encoder being used to impute the values of žy for the unlabeled datapoints. However, the framework is also regularly used as a basis for learning meaningful representations and performing manipulations, exploiting the presence of the decoder to generate new datapoints after intervening on the labels via changes to žy. Our focus lies on the latter, for which we show this standard formulation leads to serious pathologies. Our primary goal is not to improve the fidelity of generations, but instead to demonstrate how label information can be used to structure the latent space such that it encapsulates and disentangles the characteristics associated with the labels.
3 RETHINKING SUPERVISION
As we explained in the last section, the de facto assumption for most approaches to supervision in VAEs that the labels correspond to a partially observed augmentation of the latent space, Zy. However, this can cause number of issues if we want the latent space to encapsulate not just the labels themselves, but also the characteristics associated with these labels. For example, encapsulating the youthful characteristics of face, not just the fact that it is "young" face. At an abstract level, such an approach fails to capture the relationship between the inputs and labels: it fails to isolate characteristic information associated with each label from the other information required to reconstruct data. More specifically, it fails to deal with the following issues.
Firstly, the information in datapoint associated with label is richer than stored by the (typically categorical) label itself. That is not to say such information is absent when we impose 2, y. but here it entangled with the other latent variables z Ag which simultaneously contain the associated information for all the labels. Moreover, when y is categorical, it can be difficult to ensure that the VAE actually uses zy. rather than just capturing information relevant to reconstruction in the higher-capacito continuous, Z\y" Overcoming this is challenging and generally requires additional heuristies and hyper-parametert
Second, we may wish to manipulate characteristics without fully changing the categorical label itself. For example, making CelebA image depict more or less 'smiling' without fully changing its " "smile" le" label. Here we do not know how to manipulate the latents to achieve this desired effect: we can only do the binary operation of changing the relevant variable in Zy. Also, we often wish to keep a level of diversity when carrying out conditional generation and, in particular, interventions. For example, if we want to add a smile, there is no single correct answer for how the smile would look, but taking Zy "smi le" only allows for single point estimate for the change.
Finally, taking the labels to be explicit latent variables can cause a mismatch between the VAE prior p(z) and the pushforward distribution of the data to the latent space q=2) E lgo(= =))- During training, latents are effectively generated according to g(z), but once learned, p(z) is used to make generations; variations between the two effectively corresponds to a train-test mismatch. As there is ground truth data distribution over the labels (which are typically not independent), taking the latents as the labels themselves implies that there will be ground truth q(z,). However, as this not generally known priori, will inevitably end up with mismatch.
3

---

Published as a conference paper at ICLR 2021
What do we want from supervision? Given these issues, it is natural to ask whether having latents directly correspond to labels is actually necessary. To answer this, we need to think about exactly what it is we are hoping to achieve through the supervision itself. Along with uses of VA more generally, the three most prevalent tasks are: a) Classification, predicting the labels of inputs where these are not known priori; b) Conditional Generation, generating new examples conditioned on those examples conforming to certain desired labels; and c) Intervention, manipulating certain desired characteristics of a data point before reconstructing
Inspecting these tasks, we see that for classification we need a classifier form to y. for conditional generation we need mechanism for sampling = given y, and for inventions we need to know how to manipulate to bring about desired change. None of these require to have the labels directly correspond to latent variables. Moreover, as we previously explained, this assumption can be actively harmful, such restricting the range of interventions that can be performed.
4 CHARACTERISTIC CAPTURING VARIATIONAL AUTOENCODERS
To correct the issues discussed in the last section, we suggest eschewing the treatment of labels as direct components of the latent space and instead employ them to condition latent variables which are designed to capture the characteristics. To this end, we similarly split the latent space into two components, x = ((c, but where ze. the characteristic latent, is now designed to capture the characteristics associated with labels, rather than directly encode the labels themselves. In this breakdown, ee is intended only to capture information not directly associated with any of the labels, unlike B which was still tasked with capturing the characteristic information.
For the purposes of exposition and purely to demonstrate how one might apply this schema, first consider a standard VAF, with latent space z = {Ze, x. The latent representation of the VAE will implicitly contain characteristic information required to perform classification, however the structure of the latent space will be arranged to optimize for reconstruction and characteristic information may be entangled between ze and e If we were now to jointly learn classifier-from Ze to with the VAE, resulting in the following objective:
J=Σ wel [vAE(x)+ +
(LvAE(z)) + (=)m) [log ( =c)])=
(2)
where a is hyperparameter, there will be pressure the encoder to place characteristic information in ze. which can be interpreted as a stochastic layer containing the information needed for classifica- tion and reconstruction!! The classifier thus acts as tool allowing y to influence the structure of it is this high level concept, i.e. using y to structure 2, that we utilize in this work.
However, in general, the characteristics of different labels will be entangled within ze. Though will contain the required information, the latents will typically be uninterpretable, and it is unclear how we could perform conditional generation or interventions. To disentangle the characteristies of different labels, we further partition the latent space, such that the classification of particular labels y² only has access to particular latents z and thus log (3 z.) = E, log 9p (y' =z). This has the critical effect of forcing the characteristic information needed to classify y² to stored only in the corresponding z/, providing means to encapsulate such information for each label separately. We further see that it addresses many of the prior issues: there are no measure- measure-theoretii thcoretic issues as x is not discrete, diversity in interventions is achieved by sampling different 2 for given label, z can be manipulated while remaining within class decision boundaries, and a mismatch between p(z.) and ((z)) does not manifest as there is no ground truth for g(2.).
How to conditionally generate or intervene when training with (2) is not immediately obvious though. However, the classifier implicitly contains the requisite information do this via inference in an implied Bayesian model. For example, conditional generation needs samples from p(z,) that classify to the desired labels, e.g. through rejection sampling. See Appendix A for further details.
4.1 THE CHARACTERISTIC CAPTURING VAE
One way to address the need for inference is to introduce a conditional generative model py (ze y). simultaneously learned alongside the classifier introduced in (2). along with prior ply). This
'Tooug,, for convenience. we implicitly assume here, and through the rest of the paper, that the labels are categorical such that the mapping ze y classifier, we note that the ideas apply equally well if some labels actually continuous, such that this mapping now probabilistic regression.
4

---

Published as a conference paper at ICLR 2021
approach, which we term the CCVAE, allows the required sampling for conditional generations and interventions directly. Further, by persisting with the latent partitioning above, we can introduce factorized set of generative models p(z, y) II. p(x" y²), enabling easy generation and manipulation of 2 individually. CCVAE ensures that labels remain part of the model for unlabeled datapoints, which transpires to be important for effective learning in practice.
To address the issue of learning, we perform variational inference, treating y as a partially observed auxiliary variable. The final graphical model is illustrated in Figure 2. The CCVAE be seen as a way of combining top-down and bottom-up information obtain a structured latent representation. However, it is important to highlight that CCVAE does not contain hierarchy of latent variables. Unlike a hierarchical VAE, reconstruction is performed only from 2 ~ (=(2 | æ) without going through the "deeper" y. doing so would lead loss of information due to the bottleneck of y. By enforcing each label variable to link to different characteristic-- latent dimensions, we are able to isolate the generative factors Figure 2: CCVAR graphical model. corresponding to different label characteristics.
pola x)
[o(s a;)
x
4.2 MODEL OBJECTIVE
We now construct an objective function that encapsulates the model described above, by deriving lower bound on the full model log- lig-likel which factors over the supervised and unsupervised subsets as discussed in $ 2 The supervised objective can be defined as
log P0. v) ( log (o 2)P v)p(y) LeevAE( v),
(3)
(0.0(z v)
with (e(= y) = p(zro)pp(z y). Here, we avoid directly modeling (p.o(= a, y); instead leveraging the conditional independence æ 1 y z, along with Bayes rule, to give
(3 Z.)4a(= ax) 22.0 x))
where Pp (y x)) -f 9p =e)go(> x)dz.
f
(2.0(2 xx v)
Using this equivalence in yields (see Appendix B.I for derivation and numerical details) ( xe)
log p= z ( 3)) + +og (p logg))))(y). (4)
LeevAE(2, y) - E (x(ox)
Lo x)) Qxp (1 Zo)
x:]]
(y ar) falls out naturally from the derivation, unlike previous models (e.g. Kingma et al. (2014); Siddharth et al. (0177). Not placing the labels directly in the latent space is crucial for this feature. When defining latents to directly correspond to labels, observing both æ and y detaches the mapping ( x)) between them, resulting in the parameters (p, d) not being motiivatinn addition of an explicit (weighted) classifier. Here, however, observing both and y does detach any mapping, since they are always connected via an unobserved random variable z., and hence do not need additional terms. From an implementation perspective, this classifier strength can be increased, we experimented with this, but found that adjusting the strength had little effect on the overall classification accuracies. We consider this insensitivity to be significant strength of this approach, as the model able to apply enough pressure to the latent space to obtain high classification accuracies without having to hand tune parameter values. We find that the gradient norm of the classifier parameters suffers from high variance during training, we find that not reparameterizing through ze in 4 zx) reduces this affect and aides training, see Appendix C.3. for details.
Note that classifier term log 9.
For the datapoints without labels, we can again perform variational inference, treating the labels as random variables. Specifically, the unsupervised objective, LCCVAE(x)) derives as the standard (unsupervised) ELBO. However, it requires marginalising over labels p(z) ((z,) p(z.e) Σn p( P(D). This can be computed exactly, but doing so can be prohibitively expensive if the number of possible label combinations is large. In such cases, we apply Jensen's 's inequality a second time to the expectation over y (see Appendix B.2) to produce a looser, but cheaper to calculate, ELBOO given
LCOVAB(x) = (=a() log Po(x 2)pu(z B)P(y))
(5)
( (y 2 1z))0(= x))
5

---

Published as a conference paper at ICLR 2021
Combining 4 and ( we get the following lower bound on the log probability of the data
log (D)
Σ
(a.y)ES (a, + Σ well LecvAr(#))
(6)
that unlike prior approaches faithfully captures the variational free energy of the model. As shown in S 6 this enables range of new capabilities and behaviors to encapsulate label characteristics.
5
RELATED WORK
The seminal work of Kingma et al. (2014) was the first to consider supervision in the VAEs setting, introducing the M2 model for Semi-supervise classification which was also approach to place labels directly in the latent space. The related approach of Maalee al. (2016) augments the encoding dis- tribution with an additional, unobserved latent variable, enabling better semi-supervised classification accuracies. Siddharth al. (2017) extended the above work to automatically derive the regularised objective for models with arbitrary (pre-defined) latent dependency structures. The approach of placing labels directly in the latent space was also adopted in Li et al. (2019). Regarding the disparity between continuous and discrete latent variables in the typical semi cumii-upereised VAEs, Dupont (2018) provide an approach to enable effective wnsupervised learning in this setting.
From a purely modeling perspective, there also exists prior work VAEs involving hierarchies of latent variables, exploring richer higher-order inference and issues with redundancy among latent variables both in unsupervised (Ranganath et al. [016 Zhao et al. (017)) and semi- supervisee (Maaløe) et al. 2017| 2019) settings. In the unsupervised case, these hierarchical variables do not have a direct interpretation, but exist merely to improve the flexibility of the encoder. The semi-supervised approaches extend the basic M2 model to hierarchical VAEs by incorporating the labels as an additional latent (see Appendix F in Maaløe al.. 2019) for example), and hence must incorporate additional regularisers in the form of classifiers as in the case of M2. Moreover, by virtue of the typical dependencies assumed between labels and latents, it is difficult to disentangle the characteristies just associated with the label from the characteristics associated with the rest of the data something we capture using our simpler split latents (Ze: 2)c).
From a more conceptual standpoint, Mueller al. 2017) introduces interventions (called revisions) on VAFs for text data, regressing to auxiliary sentiment scores as a means of influencing the latent variables. This formulation similar to (2) in spirit, although in practice they employ a range of additional factoring and regularizations particular to their domain of interest, in addition to training models in stages, involving different objective terms. Nonetheless, they share our desire enforce meaningfulness in the latent representations through auxiliary supervision.
Another related approach involves explicitly treating labels as another data modality (Vedantam et al. 2018 Suzuki et al. 2017 Wu & Goodman 2018) (Shi et al., 2019). This work is motivated by the need to learn latent representations that jointly encode data from different modalities. Looking back to (3)
by refactoring p(z p() as p(y z))(z), and taking q(> x. y) g(((2 x)) q( y)). one
derives multi-mmdal VAEs, where G can construct product (Wu & Goodman, 2018) or mixture (Shi
al. (219)) of experts. Of these, the MVAE (Wu & Goodman (0118) is more closely related to our setup here, as it explicitly targets cases where alternate data modalities are labels. However, they differ that the latent representations are not structured explicitly map to distinct classifiers, and do not explore the question of explicitly capturing the label characteristics. The JLVM model Afel jet al. (2018) similar to the MVAE, but motivated from an interpretability perspective- with labels providing "side-channel" information to constrain latents. They adopt flexible normalising- fiww posterior from data x, along with a multi-component objective that is additionally regularised with the information bottleneck between data latent z, and label y.
DIVA llse et al., 2019) introduces a similar graphical model to ours, but is motivated to learn a generalized classifier for different domains. The objective is formed of a classifier which regularized by a variational term, requiring additional byper--arameters and preventing the ability to disentangle the representations. In Appenddx C.4 we propose some modifications DIVA that allow it to be applied in our problem domain.
In terms of interoperability, the work of Ainsworth et al. (2018) is closely related to ours, but they focus primarily on group data and not introducing labels. Here the authors employ sparsity in the multiple linear transforms for each decoder (one for each group) to encourage certain latent dimensions encapsulate certain factors in the sample, thus introducing interoperability into the

---

Published as a conference paper at ICLR 2021
model. Tangentially to VAEs, similar objectives of structuring the latent space using GANs also exist Xiao et al. (2017] 2018), although they focus purely on interventions and cannot perform conditional generations, classification, or estimate likelihoods.
6 EXPERIMENTS
Following our reasoning 3 we now showcase the efficacy of CCVAE for the three broad aims of (a) intervention, (b) conditional generation and (c) classification for a variety of supervision rates, denoted by J. Specifically, we demonstrate that CCVAE able to: encapsulate characteristics for each label in an isolated manner, introduce diversity in the conditional generations; permit finer control on interventions; and match traditional metrics of baseline models. Furthermore, we demonstrate that existing method is able perform all of the above 2 highlighting its sophistication over existing methods. We compare against: M2 (Kingma et al. 2014); MVAE (Wu & Goodman 2018; and our modified version of DIVA (lise al. 2019). See Appendix C.4 for details.
To demonstrate the capture of label characteristics, we consider the multi-label setting and utilise the Chexpert (Irvin et al. 2019) and CelebA (L.iu et al. 2015) datasets For CelebA, we re- strict ourselves to the 18 labels which are distinguishable in reconstructions; see [Appendix C.T for details. We use the architectures from Higgins ct al. (2016) for the encoder and decoder. The label-predictive distribution ( ze) is defined Ber (ze)) with diagonal transforma- tion (-) enforcing (p( Ze)= = II: 2x- (Bt zz²).. The conditional prior (= y) is then defined as N (z diag ( (1)) with appropriate factorization, and has its parameters also derived through MLPs. See Appendix C.3 for further details.
6.1 INTERVENTIONS
If CCVAE encapsulates characteristics of a label in a single latent (or small set of latents), then it should be able smoothly manipulate these characteristics without severely affecting others. This allows for finer control during interventions, which is not possible when the latent variables directly correspond labels. To demonstrate this, we traverse two dimensions of the latent space and display the reconstructions in Figure 3 These examples indicate that CCVAE is indeed able to smoothly manipulate characteristics. For example, b) we are able to induce varying skin tones rather than have this be binary intervention on pale skin, unlike DIVA in a). In c). the x². associated with the neckt ie label has also managed to encapsulate information about whether someone is wearing a shirt is bare-necked. No such traversals possible for M2 and it is not clear how one would do them for MVAE; additional results, including traversals for DIVA, are given in in/ppendix D.2
Figure 3: Contimuous interventions through traversal of ze. From left to right, a) DIVA pale skin and young; b) CCVAE pale skin and young; c) CCVAE smi ling and necktie; d) CCVAE Pleural Effusion and Cardiomegaly.
6.2 DIVERSITY OF GENERATIONS
Label characteristics naturally encapsulate diversity (e.g. there are many ways to smile) which should be present the learned representations. By virtue of the structured mappings between labels and characteristic latents, and since z. is parameterized by continuous distributions, CCVAE is able to capture diversity in representations, allowing exploration for an attribute (c.g. smile) while
DIVA can perform the same tasks as CCVAF but only with the modifications ourselves suggest and still not to comparable quality.
CCCAAE is well-suited to multi-label problems, but also works multi-class problems. See Appendix D.6 for results and analyses MNIST and FNshionMNIST
7

---

Published as a conference paper at ICLR 2021
Figure 4: Diverse conditional generations for CCVAE, y is held constant along each row and each column represents different sample for ze ~ p(*ely). a/o is held constant over the entire figure.
Figure 5: Variance in reconstructions when intervening on a single label. [TTp two] CelebA, from left to right: reconstruction, bangs, syeglasses pale skin, smiling, necktie.. [Bottom] Chexpert: reconstruction, cardicmegaly, edema, consolidation. atelectasis, pleural effusion.
preserving other characteristics. This is not possible with labels directly defined as latents, as only discrete choices can be made diversity can only be introduced here by sampling from the unlabeled latent space-which necessarily affects all other characteristics. demonstrate this, we reconstruct multiple times with {ze (= y). Z for fixed Zie- We provide qualitative results Figure 4
If several samples taken from (z (3) when intervening only single characteristic, the resulting variations in pixel values should be focused around the locations relevant to that characteristic, e.g. pixel variations should be focused around the neck when intervening neckt To demonstrate this, we perform single interventions on each class, and take multiple samples of Ze ~ (ze y). We then display the variance of each pixel in the reconstruction in green in Figure 5 where it can be seen that generally there is only variance in the spatial locations expected. Interestingly. for the class smile (2nd from right), there variance in the jaw line, suggesting that the model is able capture more subtle components of variation that just the mouth.
6.3 CLASSIFICATION
To demonstrate that reparameterizing the labels in the latent space does not hinder classification accuracy, we inspect the predictive ability of CCVAE across a range of supervision rates, given in Table I It can be observed that CCVAE generally obtains prediction accuracies slightly superior to other models. We emphasize here that CCVAE's primary purpose not to achieve better classification accuracies; we simply checking that it does not harm them, which it most clearly does not.
Table 1: Classification accuracies.
CelebA
Chexpert
Model
0.004
0.06
f = 0.2
f = 1.0
f 0.004
f == 0.06
f = 0.2
f
CCVAE
0.832
0.862
0.878
0.900
0.809
0.792
0.794
0.826
M2
0.794
0.862
0.877
0.893
0.799
0.779
0.777
0.774
DIVA 0.807
0,860
0.867
0.877
0.747
0.786
0.781
0.775
MVAE
0.793
0.828
0.847
0.864
0.759
0.787
0.767
0.715
6.4 DISENTANGLEMENT OF LABELED AND UNLABELED LATENTS
If model can correctly disentangle the label characteristies from other generative factors, then manipulating ee should not change the label characteristics of the reconstruction. To demonstrate this, perform "characteristic swaps," where first obtain {*c, 2 for given image, then swap in the characteristics x, to another image before reconstructing. This should apply the exact characteristics, not just the label, the scene/hackground of the other image (ef. Figure 6).
8

---

Published as a conference paper at ICLR 2021
+
(se;#te)
(#o,*te)
(#e, #le)
Figure Characteristic swap, where the characteristics of the first image (blond hair, smi smiling, heavy makeup, female, no necktie, ne glasses ctc.) are transfered to the unlabeled char- acteristics of the second (red background etc.).
Comparing CCVAE our baselines in Figure we see that CCVAE able to transfer the exact characteristics to greater extent than other models. Particular attention is drawn to the preservation of labeled characteristics each row, where CCVAE is able to preserve characteristics, like the precise skin tone and hair color of the pictures on the left. We see that M2 is only able to preserve the label and not the exact characteristic, while MVAE performs very poorly, effectively ignoring the attributes entirely. Our modified DIVA variant performs reasonably well, but less reliably and at the cost of reconstruction fidelity compared to CCVAE.
unlabeled contextual attributes, A/e
CCVAF
lerrsstisssss Ze
label charac characteristics,
ME
DIVA
MVAE
Figure 7: Characteristic swaps. Characteristics (smi 1 Ling, brown hair, skin tone, etc) of the left image should be preserved along the row while background information should be preserved along the column.
An ideal characteristic swap should not change the probability assigned by a pre-trained classifier between the original image and a swapped one. We employ this as a quantitative measure, reporting the average difference in log probabilities for multiple swaps in Table 2 CCVAE is able to preserve the characteristics to a greater extent than other models. DIVA's performance is largely due to its heavier weighting the classifier, which adversely affects reconstructions,
earlier.
Table 2: Difference pogbrbbbaiiliitiss pre-trained classifier from denotation swaps, lower is better.
CelebA
Chexpert
Model
0.004
0.06
f 0.2
f 1.0
f 0.004
f 0.06
01.2
f 1.0
CCVAE
1.177
0.890
0.790
0.758
1.142
1.221
1.078
1.084
M2
2.118
1.194
1.179
1.143
1.624
1.43
1.41
1.415
DIVA
1.489
0.976
0.996
0.941
1.36
1.25
1.199
1.259
MVAE
2.114
2.113
2.088
2.121
1.618
1.624
1.618
1.601
7 DISCUSSION
We have presented a novel mechanism for faithfully capturing label characteristics in VAEs, the characteristic capturing VAE (CCVAE), which captures label characteristics explicitly in the latent space while eschewing direct correspondences between label values and latents. This has allowed us to encapsulate and disentangle the characteristics associated with labels, rather than just the label values. We are able to so without affecting the ability to perform the tasks one typically does in the (semi-)supervised particular, we have shown that, not only does this lead to more effective conventional label-switch interventions, it also allows for more fine-gra grained interventions to be performed, such as producing diverse sets of samples consistent with an intervened label value, or performing characteristic swaps between datapoints that retain relevant features.
classification, conditional generation, and intervention. In

---

Published as a conference paper at ICLR 2021
8 ACKNOWLEDGMENTS
TJ. PHST, and NS supported by the ERC grant ERC-2012-AdG 321 1-11662-HLOOS EPSRC grant Seebibyte EP/M013774/1 and EPSRC/MURI grant EP/N019474/1. Toshiba Research Europe also support TJ. TJ would also like thank Dr. M. Stoddart. PHST would also like acknowledge the Royal Academy of Engineering and FiveAL.
SMS was partially supported by the Engineering and Physical Sciences Research Council (EPSRC) grant EP/K503 11/1.
TR's research leading to these results has received funding from a Christ Church Oxford Junior Research Fellowship and from Tencent AI Labs.
REFERENCES
Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering interpretable representations for both deep generative and discriminative models. In International Conference on Machine Learning, pp. 50-59, 2018.
Samuel K. Ainsworth, Nicholas J. Foti, Adrian K.C. Lee, and Emily B. Fox. Interpretable VAEs for nonlinear group factor analysis. ICML, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8): 1988-188,, August 2013. ISSN 0162-8828.
Rodney A Brooks. Intelligence without representation. Artificial intelligence, 47(1-3):139-15, 1991.
Emilien Dupont. Learning disentangled joint continuous and discrete representations, In Advances in Neural Information Processing Systems, pp. 710-720, 2018.
Yarin Gal. Uncertainty in deep learning. PhD thesis, University Cambridge, 2016. Unpublished doctoral dissertation.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta-VAF, Learning basic visual concepts with constrained variational framework. Proceedings of the International Conference on Learning Representations, 2016.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504 2006.
Geoffrey E Hinton and Richard S Zemel. Autoencoders, minimum description length and helmholtz free energy. In Advances in neural information processing systems, pp. 3-10, 1994.
Maximilian Ilse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. arXiv preprint (tt::1905.1144 2019.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 590-597, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.611 2013.
10

---

Published as a conference paper at ICLR 2021
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):13322 1332-1338, 2015.
Yang Li, Quan Pan, Suhang Wang, Haiyun Peng, Tao Yang. and Erik Cambria. Disentangled variational auto-encoder for semi-supervised learning. Information Sciences, 482:73-85, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of dis- entangled representations. In International Conference Machine Learning, pp. 41 11244,24, 2019.
Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv preprint arXiv:1602.05447 2016.
Lars Maaløe, Marco Fraccaro, and Ole Winther. Semi- supervisee generation with cluster-aware generative models. arXiv preprint
2017.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In Advances in Neural Information Processing Systems, volume 32, pp. 5551-6562. Curran Associates, Inc., 2019.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint
2019.
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in 2019.
variational autoencoders. In International Conference on Machine Learning, pp.
Jonas Mueller, David Gifford, and Tommi Jaakkola. Sequence to better sequence: continuous revision of combinatorial structures. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 5536-2544.15 JMLR. org, 2017.
Searning-Volume
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International Conference on Machine Learning, pp. 324-333, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pp. 1278-1288 2014.
Yuge Shi, N. Siddharth, Brooks Paige, and Philip H. S. Torr. Variational mixture-of-expprts autoen- coders for multi-modal deep generative models. In Advances in Neural Information Processing Systems (NeurIPS), pp. 15692-15703 December 2019.
N. Siddharth, T Brooks Paige, Jan- Aillillemm Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi- supervised deep generative models. Advances in Neural Information Processing Systems, pp. 5925-5935, 2017.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. arXiv preprint
088333, 2018
Casper Kaae Sønderby, Tapani Raiko, Lars Maalge, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738-3746. 2016.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. In International Conference Learning Representations Workshop, 2017.
11

---

Published as a conference paper at ICLR 2021
Joshua B Tenenbaum. Mapping manifold of perceptual observations. In Advances in neural information processing systems, pp. 382-688, 1998.
Joshua B Tenenbaum and William T Freeman. Separating style and content with bilinear models. Neural computation, (26))1247-1255 2000.
Ramakrishna
Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. In Proceedings of the International Conference on Learning Representations, 2018.
Martin Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning, 1(1-2):1-305, 2008
Mike Wu and Noah Goodman. Multimodal generative models for scalable beakly-supervised learning. In Advances in Neural Information Processing Systems, pp. 5580-5590, 2018.
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Dna-gan: Learning disentangled representations from multi-attribute images. arXiv preprint arXiv: 1.05415, 2017.
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. In Proceedings of the European conference computer vision (ECCV), pp. 168-184, 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep genera- tive models. In International Conference on Machine Learning, pp. 4091-4099, 2017.
12

---

Published as a conference paper at ICLR 2021
A CONDITIONAL GENERATION AND INTERVENTION FOR EQUATION (2)
For the model trained using 2 as the objective to be usable, we must consider whether it can carry out the classification, conditional generation, and intervention tasks outlined previously. Of these, classification is straightforward, but it is less apparent how the others could be performed. The key here is to realize that the classifier itself implicitly contains the information required to perform these tasks.
Consider first conditional generation and note that we still have access to the prior p(z) as per a standard VAE. One simple way of performing conditional generation would be conduct a rejection sampling where we draw samples & ~ p(z) and then accept these if and only if they lead to the classifier predicting the desired labels up to desired level of confidence, i.e. 4s 2g) À where O < À < 1 is some chosen confidence threshold. Though such an approach is likely to be highly inefficient for any general p(=) due to the curse of dimensionality, in the standard setting where each dimension of z is independent, this rejection sampling can be performed separately for each =/ making it relatively efficient. More generally, we have that conditional generation becomes an inference problem where we wish to draw samples from
p (z (g ze) > {}) x p(z)I (g0 xc) > A)
Interventions can also be performed in analogous manner. Namely, for conventional intervention where we change one or more labels, we can simply resample the z associated we those labels, thereby sampling new characteristics match the new labels. Further, unlike prior approaches, we can perform alternative interventions too. For example, we might attempt to find the closest z to the original that leads to the class label changing: this can be done in a manner akin to how adversarial attacks are performed. Alternatively, we might look to manipulate the z without actually changing the class itself to see what other characteristics are consistent with the labels.
To summarize, yields objective which provides way of learning semi-supervised VA Es that avoids the pitfalls of directly fixing the latents to correspond to labels. It still allows us to perform all the tasks usually associated with semi-supervised VAEs and in fact allows more general form of interventions to performed. However, this comes at the cost requiring inference to perform conditional generation interventions. Further, the label variables y are absent when the labels are unobserved, there may be empirical complications with forcing all the denotational information to be encoded to the appropriate characteristic latent 2 In particular, hyperparameter a that must be carefully tuned to ensure the appropriate balance between classification and reconstruction.
still have
B MODEL FORMULATION
B. VARIATIONAL LOWER BOUND
In this section we provide the mathematical details of our objective functions. We show how to derive it lower bound the marginal model likelihood and show how we estimate the model components.
The variational lower bound for the generative model in Figure 2 given as
LeevAE = Σ (x) +
Σ
LeevAE(æ, v)
sEHl
(2.
Co =z) log
po 2 2)) (z y))
LeevAE(xt, y) = (x
+ log ( at) + logg(y),
App x))
(p (z) x:)
(x
x) x)p (ze ())(())
LCCVAE(2)+ = Egg (1l1)))
log
(3 2 2e)g0(2 xr)
The overall likelihood in the sumi-supervised case is given as
Po v) = II
po(a. v)
II (m(),, aEM/
(m.
To derive a lower bound for the overall objective, we need to obtain lower bounds on log (=) and log (a( y). When the labels are unobserved the latent state will consist of and y. Using the
13

---

Published as a conference paper at ICLR 2021
factorization according to the graph in Figure yields
poo z 2) ())(()))
log porr) 2 (aol log
4p(3 Zc) m))
where (z(2 y) = likelihood pa (x, y)
y). For supervised data points we consider a lower bound on the
po(æ x)pp(z y)p(y)
log v)
log
(o (z| ær y)dz,
(p x y)
in order to make sense of the term lpp0(2 consider the inference model
ae, y). which usually different from go(= a:) we
( (z) x))
Pp.0 xz y)
where (p a)) = 10 =0)g0(z m)dz.
a (
x))
Returning to the lower bound on log Pa (x, y) we obtain
log (a( y) 2 / log
pa
2 2)pp(2 y)p(y)
g(> æ. 2)
gx= x, 3)
= =flog
Pa(æ
pp (z B
y)p 9p x))
go 2.)9a(2 x)) dz
(ply (y Zc) x)
zo)
log 1p zc)90(z (mm) )
=)p v) p(x
Ea (# m))
+ log (p x)) + log p(U),
( x))
where (p(y) zc)/ /0. (y a)) denotes the Radon-Nikodym derivative of gp.o x. y) with respect to [2 x.). B.2 ALTERNATIVE DERIVATION OF UNSUPERVISED BOUND
The bound for the unsupervised case can alternatively be derived by applying Jensen's inequality twice. First, use the standard (unsupervised) ELBO
logpe(a) (x(()) log Po( x)p(z)]
(= x))
Now, since calculating p(z) = p(z))p((zz))) = p(z,e) En p(zc p)p(3) can be expensive we can apply Jensen's 's incquality second time to the expectation over 2. to obtain
log Pv v)p(8))
log p(z.) 2 (Ml=))
( =,)
Substituting this bound into the unsupervised ELBO yields again our bound
log Po ( 2)p(z 8)
((x) 2 (ul())))).
(0(z x) zc) + logp(y)
C IMPLEMENTATION
C.I CELEBA
We chose to use only a subset of the labels present in CelebA, since not all attributes are vi- sually distinguishable in the reconstructions e.g. (earrings). As such we limited ourselves to the following labels: arched eyebrows, bags under eyes, bangs, black hair, r, blond hair, brown hair, bushy eyebrows, chubby, eyeglasses, heavy makeup, male, beard, pale skin, receding hairline, smiling, wavy hair, wearing necktie, young. No images were omitted or cropped, the only modifications were keeping the aforementioned labels and resizing the images to be 64 x 64 in dimension.
C.2 CHEXPERT
The Chexpert dataset comprises of chest X-rays taken from variety of patients. We down-sampled each image to be 64 x 64 and used the same networks from the CelebA experiments. The five main stelectase is,
attributes for Chexpert are: cardiomegall
edema, consolidation pleurale ef fon. Which for non medical experts can be interpreted as: enlargement of the heart; fluid in the alveoli; fluid in the lungs; collapsed lung; fluid in the corners of the lungs.
14

---

Published as a conference paper at ICLR 2021
C.3 IMPLEMENTATION DETAILS
For our experiments we define the generative and inference networks follows. The approxi-
mate posterior represented as (0(2 x) = [Zcc 2ie (z) diag(o with He (a) and diag(o)) (x)) being the architecture from Higgins et al. (2016). The generative model po( 2)
represented by a Laplace distribution, again parametrized using the architecture from Higgins et al. 2016 The label predictive distribution (3 ze) is represented as Ber (T() with
T being diagonal transformation forcing the factorisation 4- 4 2x) = I1, 90- ze¹).
The conditional prior is given (= y) [ze (1) diag(o (3))), with the appropriate factorisation, where the parameters are represented by an MLP. Finally, the prior placed on the portion
of the latent space reserved for unlabelled latent variables is p(z)e) (z 0, I)). For the latent
space ze E R and e E IR (,",, where m
my + me with m. = 18 and mic 27 for CelebA.
The architectures given in and Table
Encoder
Decoder
Input 32 x 32 channel image
Inpul R"
32 x 4 x Conv2d stride 2 ReLU
372. 256 Linear layer
32 32
Conv2d stride & ReLU
128 256 4
Conv Transpaassp stride 1 & ReLU
64 32 X
Conv2d stride 2 & ReLU
64 x 128
4 Conv TonnSsaappsee2 stride 2 ReLU
128 x 64 X 4 x Cony2d stride 2 & ReLU
32 x 64
x 4 Conv Transsaaspose2 stride & ReLU
256 x 128 x 4 X 4 Conv2d stride 1 & ReLU
32 x 32 4 x 4 Cony Tranppaanspo stride & ReLU
256 x (2xm) Linear layer
3 x 32 x 4 x Conv Transpose2d stride 2 & Sigmoid
Classifier
Conditional Prior
Input e R m. x m, Diagonal layer
Input E I
m. x m, Diagonal layer
Table 3: Architectures for CelebA and Chexpert.
Optimization We trained the models on a GeForce GTX Titan GPU. Training consumed 2Gb for CelebA and Chexpert, taking around 2 hours to complete 100 epochs respectively. Both models were optimized using Adam with a learning rate of 10-4 for CelebA respectively.
C.3.1 HIGH VARTANCE OF CLASSIFIER GRADIENTS
The gradients of the classifier parameters yp suffer from high variance during training. We find that not reparameterizing for (y ze) reduces this issue:
(3 =c)
z z)p
(= y)
LeeVAE(2; (xr y) = (x(o))
log
+log (p. xx) log p(y)-
(8)
( x)) 4p
Z0) x))
where ze indicates that we do not reparameter- ize the sample. This significantly reduces the variance of the magnitude of the gradient norn Ve, allowing the classifier to learn appropriate weights and structure the latent space. This be seen in Figure 8) where we plot the gradient norm of yp for when we do reparameterize ze (blue) and when we do not (orange). Clearly not reparameterizing leads gradient norm of the classifier, which aides learn- ing. To certain extent these gradients can be viewed as redundant, as there is already gradients to update the predictive distribution due to the log ( (y| x)) term anyway.
Gradiants Norms Classifier Paramatees
230
150
lower variance in the
60
80
100
120
300
Dterston
Figure 8: Gradient norms of classifier.
15

---

Published as a conference paper at ICLR 2021
2d e
e
Aie
kc
N
y
yec
Figure 9: Left: Generative model for DIVA, Right: Inference model where dashed line indicates auxiliary classifier.
C.4 MODIFIED DIVA
The primary goal of DIVA is domain invariant classification and not obtaín representations of individual characteristics like we do here. The objective is essentially classifier which is regularized by a variational objective. However, to achieve domain generalization, the authors aim to disentangle the domain, class and other generative factors. This motivation leads to graphical model that is similar in spirit to ours (Figure 9). in that the latent variables are used to predict labels, and the introduction of the inductive bias to partition the latent space. As such, DIVA can be modified to suit our problem of encapsulating characteristics. The first modification we need to consider is the removal of zd. we not considering multi-domain problems. Secondly, factorization present in CCVAE namely (3 ze) = II gat( Izc'). ). With these two modifications an alternative objective can now be constructed, with the supervised given as
introduce the
LSDIV y) = (2)=)) (x z) - (x
- (=L 1)))
and the unsupervised as
LUDIVA(Z)) Bas(=/ log BKL (KL
ga (#c (elm)) (z)=)
flog Py (z v) - log (z(2)(=)
GE (v)= p(y) log ( zc)l;
where y has to be imputed. The final objective for DIVA then given as
CSDIVA log Po (D) N E
+ Ensu ( + alB m) log (3 =2)].
the differences to the objective of CCVAE, namely, there is no emergence of a natural classifier in the supervised case, and has to be imputed in the unsupervised case instead of relying on variational inference in CCVAE. Clearly such differences have a significant impact on performance as demonstrated by the main results of this paper.
is interesting
D ADDITIONAL RESULTS
D.1 SINGLE INTERVENTIONS
Here we demonstrate single interventions where we change the binary value for the desired attributes. To quantitatively evaluate the single interventions, intervene on single label and report the changes log-probabilities assigned by a pre-trained classifier. If the single intervention only affects the characteristics of the chosen label, then there should be change in other classes and only change on the chosen label. Intervening on all possible labels yields a confusion matrix, with the optimal results being diagonal matrix with zero off- off-diagonal elements. We also report the condition number for the confusion matrices, given in the titles.
It is interesting note that the interventions for CCVAE subtle, this is due to the latent p(z))))), which will be centered around the mean. More striking intervention can be achieved by traversing along z
16

---

Published as a conference paper at ICLR 2021
Confusion Matrix for ReVAE interventiones, 25226 635.
Confusion Matrix for ReViE intervenntions
789.
aeteed Sratrees
saae Hurlin
Musten
sesee Chlll
accied tyetrovo
100
acched tedlaws
10
tegs, theer Lyss
dagu Lhoker Eyss
Sango
tango
0.8
0.0
skorl
Duly Sydrerr
0.6
Jolly Sydrron
066
Chuctty
Outty
0.4
0.4
Mal
soort
sent
role
0.2
Pilc
0.2
Wery,
Weanig senig
Weanin Nectto
Young
Confusion Matrix ReVAE Interventions, 29.257.
Confusion Matrix
ReVAE Interventions, 35.978.
Maleop
Sies
Aword
your
Sydrooy
1.0
Archol Eydaoos
Diçe. Under Low
Daca Unider Lyne
Tangs
0.6
0.8
Dusly Syebroan
0.6
Juity Syebronn
0.6
chittyy
Luitty
tycglesses
Mels
0.4
Mole
0.4
pcom
Joom
Polc 1br
0.2
Palc Ski
022
Yeetnn Nwdd0w
88
Teemiigg Necdoe
Yuun
Classefed Labst
Clacsreed Lobdl
Figure 10: Confusion matrices for CCVAE for (from top left clockwise) f = 0.004, 0.06, 0.2, 1.0
17

---

Published as a conference paper at ICLR 2021
0.004
0.06
= 0.2
1.0
Figure 11: CCVAE From left to right: original, reconstruction, then interventions from switching on the following labels: arched eyebrows, bags under eyes, bangs, black haic, bload heir, brown haix, buahy eyebrows, chubby, eyeglasses, heavy makeup, male, ne beard, pale skin, receding haicline, smiling, wavy had r, wearing necktie, young.
18

---

Published as a conference paper at ICLR 2021
Cor fusion Metrix for M2 interventions. 3468237.55 500.
Confesson Motrix for M2 intervention k 72 697
Artot pettones
Spea
Maling
starion
Clate
Spee
herrisa
ackidd tyclrows
10
Giled tyelioee
10
sogu Uokk 2y45
Hege jyss
Binga
Danga
0.0
33
Drew Her
Drenn
Eas Sydtrows
24
Buin bycb
34
Cludds
Halc
04
Mac
24
worrd
no woard
Nlc
0.
Ric
Marri te Nirie
Young
Young
clwcafed
Contusion Matrix for M2 interventiors, 78.855.
Confuston Metric
Interventions, 72 22.49".
Spes
Gatee Sies
Achiod Eysiross
Deg. Usder cree
Degy Under Lyee
Eance
farcn
0.8
Grown
Eatty Cyetrowe
0.6
Bashy Eyecoowe
-36
chibbyy
tycplesess
inteeeee
Hale
0.4
Mac
04
moant
my maatt
Nic Skin
-22
Pecodingraal fnc
Nic Sin
22
Theiin Nechin
Neart FC. hechsw Young
Young
Cassrhes Land
Codsrice Label
Figure 12: Confusion matrices for M2 for (from top left clockwise) f = 0.004, 0.06, 0.2, 1.0
19

---

Published as a conference paper at ICLR 2021
0.004
0.06
= 0.2
1.0
Figure 13: M2. From left to right: original, reconstruction, then interventions from switching on the following labels: arched eyebrows. bags under eyes, bangs, bleck heit, blond hair, brown hair, bushy eyebrows, chubby, eyeglasses, heavy makeup, male, ne beard, pale skin, zeceding haicline, smiling, wavy has wearing necktie, 1o,,tt,,,oun,,, young.
20

---

Published as a conference paper at ICLR 2021
Oo trusssion Motrix for ReVNE interventions, 473,648.
Confesion Motrix for ReVAE intervent k. 662.
Spe
anai station
Clate Spe.
ackidd tyclrows
10
Giled tyeliowe
10
sogu Uokk 2y45
Hege jyss
Binga
Danga
0.0
33
Dre Her
Drown
tas Sydtrows
24
Buin Eydorwo
34
Clidds
Hal
04
Mac
24
worrd
no woard
Nic skin
0.2
Ric
Marri ai Niri
Young
Young
cleaafed
Contusion Matrix
ReVAE Interventions, 46,731.
Contusion Matrix
ReVAE Interventions 100.763.
Maee Spes
Gatee Saee
Aoe plaat Aadeop
Achiod Eysirous
4eles Eyetroo Degy Under Lyee
Deg. Usder cree
Dance
Darcn
0.8
Brown
Eatty Cystrowe
0.6
Bashy Eyecowwe
-36
chibbyy
tycplesess
Inteeeeee
Hale
0.4
Mac
04
moart
my maatt
Nic Skin
022
Pccodingsaal fnc
Nic San
22
Theiin Nechoe
Neart FC. hechsw Young
Young
Cadorees Land
Codsriie Label
Figure 14: Confusion matrices for DIVA for (from top left clockwise) 0.004, 0.06, 0.2, 1.0
21

---

Published as a conference paper at ICLR 2021
0.004
0.06
= 0.2
1.0
Figure 15: DIVA. From left to right: original, reconstruction, then interventions from switching on the following labels:
arched eyebrows. bags under eyes, bangs, bleck heir, blond hair, brown hair, bushy eyebrows, chubby, eyeglasses, heavy makeup, male, ne beard, pale skin, zeceding haicline, smiling, wavy has i, wearing necktie, io, young.
22

---

Published as a conference paper at ICLR 2021
Oo trulssion Motrix for ReVNE interventions, K 598 463
Confesion Matrix for ReVAE interventions, k 172.975
Spe
station
Clate Spee
herrise
ackidd tyclrows
10
Giled tyelioee
10
sogu Uokk 2ys5
Hege jyss
Binga
Danga
0.0
38
Drew Her
Drown
Eas Sydtrowo
24
Buin Eydbrwo
34
Cludds
Halc
04
Mac
24
woord
no woard
Nlc
0.2
Ric
Marr Pi Norie
Young
Young
clwcafed
Co rtusion Matrix for ReVAE Interventions, 200.770.
Confusion Matrix for ReVAE ineeventtioss 1140
actiee prefeccee
Spes
starcen
Suote Siee
Aoe plann Aadeup
yiar
your
deved
Achiod Eyciross
Deg. Usder cree
Degy Under Lyee
Eance
Darcn
0.8
arewn
Grown
Eatty Cystrowe
0.6
Bashy Eyecoowe
-3
chibbyy
tycplesess
Hale
0.4
Mac
04
moart
my maatt
Resocingrattin
Nic Skin
-22
Pecodingraal fnc
Nic Sann
22
Theiin Necron
Neart FC. hechsw
Young
Young
Casorees
Codsriee Label
Figure 16: Confusion matrices for MVAE for (from top left clockwise) f = 0.004, 0.06, 0.2, 1.0
23

---

Published as a conference paper at ICLR 2021
0.004
0.06
f = 0.2
1.0
7
7
7
6
3
Figure 17: MVAE. From left to right: original, reconstruction, then interventions from switching on the following labels: arched eyebrows. bags under eyes, bangs, bleck heir, blond hair, brown hair, bushy eyebrows, chubby, eyeglasses, heavy makeup, male, ne beard, pale skin, zeceding haicline, smiling, wavy has i, wearing necktie, 1o,,te,,,,on,,, young.
24

---

Published as a conference paper at ICLR 2021
D.2 LATENT TRAVERSALS
Here provide more latent traversals for CCVAB in Figure 18 and for DIVA in Figure 19 CCVAB able to smoothly alter characteristics, indicating that is able to encapsulate characteristics in a single dimension, unlike DIVA which is unable to alter the characteristics effectively, suggesting cannot encapsulate the characteristics.
D.3 GENERATION
We provide results for the fidelity of image generation on CelebA. To do this we use the FID metric Heusel et al. (2017), we omitted results for Chexpert as the inception model used in FID has not been trained on the typical features associated with X-Rays. The results are given in Table 4 interestingly for low supervision rates MVAE obtains the best performance but for higher supervision rates M2 outperforms MVAE. We posit that this is due to MVAE having little structure imposed on the latent space, as such the POE can structure the representation purely for reconstruction without considering the labels, something which obtains competitive results with respect to M2. It is important to note that generative fidelity is not the focus of this work as we focus purely on how structure the latent space using labels. It is no surprise then that the generations are bad as structuring the latent space will potentially be at odds with the reconstruction term in the loss.
possible the supervision rate is increased. CCVAE
Table 4: CelebA FID scores.
Model
f= 0.004
0.06
ʃ = 0.2
f = 1.0
CCVAB
127.956
121.84
121.751
120.457
M2
127.719
122.521
120.406
119,228
DIVA
192.448
230.522
218.774
201.484
MVAE
118.308
115.947
128.867
137.461
D.4 CONDITIONAL GENERATION
To asses conditional generation, we first train an independent classifier for both datasets. We then conditionally generate samples given labels and evaluate them using this pre-trained classifier: Results provided in Table 5. CCVAE and M2 are comparable in generative abilities, but DIVA and MVAE perform poorly, indicated by random guessing.
Table 5: Generations accuracies.
CalebA
Chexpert
Model
= 0.004
= 0.06
f f= 0.2 f = L.0
f = 0.004
f f= 0.06
f = 0.2
f /= 1.0
CCVAE
0,513
0.605
0.612
0.596
0.516
0.563
0.549
0.542
M2
0.499
0.61
0.612
0.611
0.503
0.547
0.547
0.558
DIVA
0.501
0.501
0.501
0.501
0.499
0.503
0.503
0.503
MVAE
0.501
0.501
0.501
0.501
0.499
0.499
0.499
0.499
D.5
DIVERSITY OF CONDITIONAL GENERATIONS
We also report more examples for diversity, in Figure
Figure 20
D.6 MULTI-CLASS SETTING
Here we provide results for the multi-class setting of MNIST and FashionMNIST. The multi-class setting is somewhat tangential some flexibility over the size of the latent space. Trying to encapsulate representations for each label not well suited for this setting, it's not clear how you could alter the representation of an image being a 6, whilst preserving the representation of it being an 8. In fact, there is really only one label for this setting, but takes multiple values. With this in mind. about how the latent space will be structured, we can set ze E IR or ze E N or conversely, store all of the representation ze, i.e. 2 0. Furthermore, we not need to enforee the factorization (y xx) = II, q (z)), and instead can be parameterized by a function F E + M where M the number of possible classes.
work, but we include it for completeness. For CCVAE, we have
can now make an explicit choice
Classification We provide the classification results in Table 6
25

---

Published as a conference paper at ICLR 2021
Figure 18: Various latent traversals for CCVAE
26

---

Published as a conference paper at ICLR 2021
Figure 19: Various latent traversals for DIVA.
27

---

Published as a conference paper at ICLR 2021
Figure 20: CCVAE, variance in reconstructions when intervening on single label. From left to right: reconstruction, then interventions from switching on the following labels: arched eyebrows, bags under eyes, bangs, black hair, blond hair, brown hais, bushy eyebrows, chubby, eyeglasses, heavy makeup, male, beard, pale skin, receding hairlins, ami amiling, vavy hair, wearing nocktic, young.
Table 6: Additional classification accuracies.
MNIST
FaSSiOnMNIST
Model 0.004
f 0.06
f 0.2
f= 1.0
0.004
f 0.06
f-2
f 1.0
CCVAE
0.927
0.974
0.979
0.988
0.741 0.756
0.865
0.879
0.901
M2
0.918
0.962
0.968
0.981
0.848
0.860
0.892
Conditional Generation We provide classification accuracies for pre-trained classifier using conditional generated samples as input and the condition as label. We also report the mutual information to give an indication of how out-o--dittribution the samples are. In order to estimate the uncertainty, we transform a fixed pre-trained classifier into a Bayesian predictive classifier that integrates over the posterior distribution of parameters as p(y x. D) p(y x xr,))p(w> D)dw. The utility of classifier uncertainties for out-of-distribution detection has previously been explored [Smith & Gal (018)). where dropout is also used at test time to estimate the mutual information (MI) between the predicted label y and parameters w (Gall 2016 [mmth & Gal 2018
1 wl xc, D) H pp(y x, D) Epp(()) [M [p(( a.w)]].
However, the Monte Carlo (MC) dropout approach has the disadvantage of requiring ensembling over multiple instances of the classifier for a robust estimate and repeated forward passes through the classifier to estimate MI. To mitigate this, we instead employ a sparse variational GP (with 200 inducing points) as a replacement for the last linear layer of the classifier, fitting just the GP to the data and labels while holding the rest of the classifier fixed. This, in our experience, provides a more robust and cheaper alternative to MC-dropout for estimating MI. Results are provided in
Table 7]
We can also perform latent traversals for the multi-class setting. Here, perform linear interpolation on the polytope where the corners are obtained from the network (z) for four different classes. We provide the reconstructions in Figure 21
Latent Traversals
28

---

Published as a conference paper at ICLR 2021
Table 7: Pre-trained classifier accuracies and MI for MNIST (top) and FashionMNIST (bottom).
f
0.004
0.06
0.2
1.0
Model
Acc
MI
Acc
MI
Acc
MI
Acc
MI
CCVAE
0.910
0.020
0.954
0.014
0.961
0.013
0.973
0.010
M
M2
0.883
0.035
0.929
0,026
0.934
0.024
0.948
0.020
CCVAE
0.734
0.025
0,806
0.024
0.801
0.028
0.798
0.029
F
M2
0.750
0.032
0.792
0.032
0.787
0.032
0.789
0.031
5
5
5
8
8
8
8
8
8
5
5
5 5
8
8
8
8
8
8
5
5
5 5
8
8
8
8
8
8
5
5
9
9
8
8
8
8
8
8
7
7
7
9
9
9
8
8
9
8
7
7
7
7
9
9
9
9
9
9
7
7
7
9
9
9
9
9
9
7
7
7
7
9
9
9
9
9
9
7
7
7
7
8
9
9
9
9
9
7
7
7
7
7
9
9
9
9
Figure 21: CCVAF latent traversals for MNIST and FashionMNIST: interesting to see how one class transforms into another, c.g. for MNIST we see the end of the curling around to form an 8 and steady elongation of the torso when traversing from t-shirt to dress.
Diversity in Conditional Generations Here show how
introduce diversity in the condi-
tional generations whilst keeping attributes such as pen-stroke and orientation constant. Inspecting the M2 results Figure 22 and FFigure 23] where have to sample from to introduce diversity, indicates that we are unable to introduce diversity without affecting other attributes.
Interventions We can also perform interventions on individual classes, as showed Fig ure 24
1
/
2
2
2
2
3
3
3
3
3
4
4
4
4
/
2
2
2
2
2
3
3
3
3
3
4
4
4
4
2
2
2
2
2
3
3
3
3
3
4
4
4
4
4
2
2
0
2
2
3
3
3
4
144444
2
2
2
2
2
3
3
3
4
4
H
4
5
5
5
6
6
6
6
6
7
7
7
9
7
8 8
8
8
8
9
9
9
9
9
5
5
5
5
5
6
6
6
6
6
7
7
7
7
6
8
8
8
8
9
9
9
9
5
5
5
5
5
6
6
6
6
77
7
7
7
6
8
8
8
9
9
9
9
9
5
5
5
5
5
6
6
6
7
7
7
7
8
8
8
8
8
9
9
6
9
9
5
5
5
5
5
6
6
6
6
7
7
7
7
8 8 8
8
9
6
9
9
9
Figure 22: CCVAE conditional generations with a fixed. Here we can see CCVAE able introduce diversity whilst preserving the "style" of the digit, e.g. pen width and tilt.
29

---

Published as a conference paper at ICLR 2021
/
2
2
2
2
2
3
3
3
3
3
4
4
4
4
1 Y.
2
2
2
2
3
3
3
3
3
4
4
4
4
4
1/11
| 111
2
2
2
2
3
3
3
3
3
4
4
4 4 4
4 4
4
2
2
2
2
2
3
3
3
3
3
4
4
4
2
2
2
3
2
3
3
3
3
3
4
4
5 5 5 5 5
6 9
6
6
6
7
7
7
7
7
8
8
8
8
9
9
9
9
5 5
5 5
6
6
6
6
6
7
7
7
7
7
8
8
8
8
8
9
9
9
9
5 5 5 5
6
6
6
6
6
7
7
7
7
f
8
8
{
9
9
9
9
5
5 5
5
6
6
6
6
7
7
7
7
7
8
8
8
8
8
Q
9
9
9
9
5 5 5
5
6
6
7
7
7
7
7
8
8
8
8
9
9
9
7
7
Figure 23: M2 conditional generations. Here we can see that M2 unable to introduce diversity without altering the "style" of the digit, e.g. pen width and tilt.
0
2
3
5
6
7
8
9
0
2
3
4
5
6
7 7 7
8 9 8
9
9
0
\
2
3
4
5
6
9
8
9
a 9 è
( 2
3
4
3
9
O
0
O
2
3
4
5
6
7
8 9
O 0 0
) 2
4
5
6
8 9
4
4 0
2
4
5
6
8 9
4 1
4
8 9
000120
4
5
8
2001234561
8
1
1
2
5
6
) 9
) 11010
3 4 5
{ 9
\
1
0
\
3
3
4
8
5
9
\ 110123444 4
2
$ 9
4
4
0
1
2
3
4
5
6
89
440 1
2 4344667
5
8
5
5
0
2
3
5
6
7
89
5 5 0
2
3
4
5
6
7
8
9
4
9
/
2
3
4
6
7
8
9
9 0
2
3
4
6
6
7
8
9
4
0
1
2
3
5
6
7
8 9
4 4 0
2
3
4
5
6
7 8 9
Figure 24: Left: CCVAE, right: M2. As with other approaches, can also perform wholesale interventions each class whilst preserving the style.
30


Reviews:
[{"review_id": "Sr9oDkkTJP8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on latent representations learning when some labels are provided. The authors propose a method called the characteristic capturing VAE (CCVAE). This method learns real-valued auxiliary variables that capture the label information. The proposed method is tested on a medical image and a face image dataset.\n\nOne of the major motivations of the proposed method is that it can be used to conditionally generate images based on desired characteristics. However, this paper proposes a VAE model, which usually generates lower-quality images compared to Generative Adversarial Networks (GAN). It is not clear to me why this paper focuses on a VAE model rather than a GAN model. There have been GAN-based methods [1, 2] also based on an auto-encoding framework. Note that if we remove the adversarial loss from the objective functions of these methods, and add the KL divergence penalty, these methods become VAE. Since these methods also introduce real-value variables, I believe the authors should compare to the VAE-version of these methods. \n\nIn addition, in the VAE literature, [3] proposes a semi-supervised method.  Since [3] also involves a real-value latent variable, it looks like the difference between the proposed method and [3] is that the author extends [3] to multiple labels. Is this true? I suggest the authors better clarify the novelty of the proposed method compared to [3]. \n\nI do not suggest accepting this paper because the relationships to some existing methods are not sufficiently discussed.\n\nMinor:\nIn the experiments, the paper reports the quantitative measures for classification and disentanglement. However, no quantitative measures for image quality are reported. I suggest the authors report some measures such as reconstruction error and FID score.\n\nReferences\n[1] Xiao, Taihong, Jiapeng Hong, and Jinwen Ma. \"DNA-GAN: Learning disentangled representations from multi-attribute images.\" International Conference on Learning Representation Workshop. 2018.\n\n[2]Xiao, Taihong, Jiapeng Hong, and Jinwen Ma. \"Elegant: Exchanging latent encodings with gan for transferring multiple face attributes.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\n[3] Li, Yang, et al. \"Disentangled variational auto-encoder for semi-supervised learning.\" Information Sciences 482 (2019): 73-85.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R1STqlx5NLA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**GENERAL**\nThe paper proposes to re-think the fashion of using label information in the VAE framework. The authors propose to disentangle information about the label (or, more generally, the context) in a \"hard-coded\" manner, namely, by using a separate set of variables for the label (context). The paper is written in a lucid manner, and the presented results are sound.\n\n**Strengths:**\nS1: The presented VAE framework (CCVAE) is interesting.\n\nS2: The final objective in Eq. (6) that follows from applying Jensen's inequality two times, allows to efficiently train the model. Moreover, it allows the use of interventions to modify information about images.\n\nS3: The demo is a nice fashion of presenting the idea.\n\nS4: I highly appreciate that the authors used a dataset outside of standard benchmarks, and they focused on a medical application (the Chexpert dataset).\n\nS5: The authors explained all implementation details that increases reproducibility of the paper. \n\nS6: I appreciate the close relation of the propose framework to other models, e.g., DIVA. Moreover, the idea of improving upon DIVA is very interesting.\n\n**Deficiencies:**\nD1: My only concern is rather average generative capabilities of the model. Nevertheless, it does not affect my overall good assessment of the paper.\n\n**Remarks:**\nR1: In Eq. 2, there is a fudge factor, \\alpha. However, in the CCVAE objective (Eq. 4) it is no longer necessary. I would like to ask the authors whether they thought about adding this fudge factor, or it is simply unnecessary. (I am aware that q(y|x) follows naturally from the objective, nevertheless, it might further improve the classification accuracy).", "extra_scores": {}, "extra_text": {}}, {"review_id": "GSzPClNuIOW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors have propsed a method to incorporate label information \nin variational autoencoders (VAEs) to captures the characteristics associated with \nthose labels. Experiments show that using the label information helps them to better\nintervation and conditional generation of images.\n\nThe paper is well written. However I have few concern regarding the model.\n\nFirstly the paper has not cited a very important related work by\nAdel, Tameem, Zoubin Ghahramani, and Adrian Weller. \"Discovering interpretable \nrepresentations for both deep generative and discriminative models.\" International \nConference on Machine Learning. 2018.\nwhere a similar hiererchical approach was proposed. \nThe authors should provide a discussion related to this method or if possible should \ncompare their performance by extending the previous model for multiple features.\n\nThe main concerns regarding the model perspective are as follows:\n\n1. How the authors ensuring that there is no mutual information between the parts z_c and z_\\c ?\nThere is no analysis regarding the same. They should report some quanitative measures as proposed by\nbetaVae to support the same.\n\n2. How they are controlling the KLD between encoding and decoing distribution i.e \nKL(q(z|x) || p(z| z_c)) and KL(q(z_c|z) || p(z_c|y)) ? If the first kl is not significantly low, \nthere will be information loss which might affect the quality of the overall generation? \nThey should report the KL achieved in each layer.\n\n3. As they are learning the reverse mapping from a given label to corresponding \nrepresentation i.e. p_\\theta(z_c | y) -- \nis it necessary to make z_c unidimensional for interpolation ? \nHow are they learning the correspondance between fin grained values of y \nand z_c ? do they have such label information available? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DSzfTbd_J6t", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after rebuttal: the authors have clarified some of my concerns, and I have therefore increased my score.\n\n---------------\nThis paper introduces the characteristic-capturing VAE, an architecture that extends VAEs by modelling label-dependent characteristics in generative and classification tasks. In a multi-label setting, the CCVAE learns representations of individual characteristics, which are disentangled by design to the one of other labels.\n\nThe paper is well written in general, and is a good contribution to the important research direction on deep generative models focusing on using some labelled data to learn a disentangled latent space.\nThe experiments in the paper are well thought and show convincing performances in terms of label-dependent generations and interventions.\n\nMy biggest doubts are in terms of the positioning/novelty of the paper, which I found a bit confusing/misleading in the current form. When \"rethinking supervision\" the authors argue for the need of treating labels as auxiliary variables - and not as latent variables - in the model. This leads to the formulation called ALVAE, that introduces label-dependent information in latent space learning jointly a classifier from z to y. This formulation is novel as far as I know, but leads to some challenges mentioned by the authors in terms of conditional generation/intervention with this model. The authors then offer as a solution the CCVAE, which uses a conditional generative model p(z_c|y)p(y).\n* Is y in a CCVAE still an auxiliary variable? To me it seems that the benefits of the CCVAE are not due to the auxiliary nature of y, but more from the fact that y is now a partially observed latent variable that is however only connected to x through continuous latent variables. If this is the case, why is it relevant to leave in the paper the whole ALVAE discussion which is purely theoretical and not even presented in the experiments? Perhaps the authors should focus more instead on clarifying even further on the benefits of the hierarchical generative architectures y->z->x.\n* The discussions in section 3 are valid for 1-layer models such as M2 or MVAE models in which z_y=y. This discussion is not however considering hierarchical VAE architectures for semi-supervised learning, such as the one presented in \"Semi-Supervised Generation with Cluster-aware Generative Models\" by Maaløe et al, 2017. These architectures also learn a latent space that captures the information associated with a class, not just the class itself, and do so without the need for \"auxiliary\" variables. Is there any advantage of the CCVAE has with respect to this model?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel technique to learn a disentangled latent space using VAEs and semi-supervision. The technique is based on a careful specification of the joint distribution where the labels inform a factorisation of the distribution over continuous latent factors. The technique allows for inference, generation, and intervention in a tractable way.

The paper is well-written, the formulation is original, and the experiments convincing. There were some confusions that were mostly resolved during the discussion.  

In addition to the expert reviews attached, I would like to remark that I too find the formulation interesting and elegant. And if I may add to the discussion, oiVAE (output-interpretable VAEs) by Ainsworth et al presented at ICML18 is a related piece of work that did not occur to me earlier, but which the authors could still relate to (I'd certainly enjoy reading about the authors' views on that line of work). 
Decision: Accept (Poster)