Capturing Label Characteristics in VAEs

1 INTRODUCTION . Learning the characteristic factors of perceptual observations has long been desired for effective machine intelligence ( Brooks , 1991 ; Bengio et al. , 2013 ; Hinton & Salakhutdinov , 2006 ; Tenenbaum , 1998 ) . In particular , the ability to learn meaningful factors—capturing human-understandable characteristics from data—has been of interest from the perspective of human-like learning ( Tenenbaum & Freeman , 2000 ; Lake et al. , 2015 ) and improving decision making and generalization across tasks ( Bengio et al. , 2013 ; Tenenbaum & Freeman , 2000 ) . At its heart , learning meaningful representations of data allows one to not only make predictions , but critically also to manipulate factors of a datapoint . For example , we might want to manipulate the age of a person in an image . Such manipulations allow for the expression of causal effects between the meaning of factors and their corresponding realizations in the data . They can be categorized into conditional generation—the ability to construct whole exemplar data instances with characteristics dictated by constraining relevant factors—and intervention—the ability to manipulate just particular factors for a given data point , and subsequently affect only the associated characteristics . A particularly flexible framework within which to explore the learning of meaningful representations are variational autoencoders ( VAEs ) , a class of deep generative models where representations of data are captured in the underlying latent variables . A variety of methods have been proposed for inducing meaningful factors in this framework ( Kim & Mnih , 2018 ; Mathieu et al. , 2019 ; Mao et al. , 2019 ; Kingma et al. , 2014 ; Siddharth et al. , 2017 ; Vedantam et al. , 2018 ) , and it has been argued that the most effective generally exploit available labels to ( partially ) supervise the training process ( Locatello et al. , 2019 ) . Such approaches aim to associate certain factors of the representation ( or equivalently factors of the generative model ) with the labels , such that the former encapsulate the latter—providing a mechanism for manipulation via targeted adjustments of relevant factors . ∗work done while at Oxford †equal contribution Prior approaches have looked to achieve this by directly associating certain latent variables with labels ( Kingma et al. , 2014 ; Siddharth et al. , 2017 ; Maaløe et al. , 2016 ) . Originally motivated by the desiderata of semi–supervised classification , each label is given a corresponding latent variable of the same type ( e.g . categorical ) , whose value is fixed to that of the label when the label is observed and imputed by the encoder when it is not . Though natural , we argue that this assumption is not just unnecessary but actively harmful from a representation-learning perspective , particularly in the context of performing manipulations . To allow manipulations , we want to learn latent factors that capture the characteristic information associated with a label , which is typically much richer than just the label value itself . For example , there are various visual characteristics of people ’ s faces associated with the label “ young , ” but simply knowing the label is insufficient to reconstruct these characteristics for any particular instance . Learning a meaningful representation that captures these characteristics , and isolates them from others , requires encoding more than just the label value itself , as illustrated in Figure 1 . The key idea of our work is to use labels to help capture and isolate this related characteristic information in a VAE ’ s representation . We do this by exploiting the interplay between the labels and inputs to capture more information than the labels alone convey ; information that will be lost ( or at least entangled ) if we directly encode the label itself . Specifically , we introduce the characteristic capturing VAE ( CCVAE ) framework , which employs a novel VAE formulation which captures label characteristics explicitly in the latent space . For each label , we introduce a set of characteristic latents that are induced into capturing the characteristic information associated with that label . By coupling this with a principled variational objective and carefully structuring the characteristic-latent and label variables , we show that CCVAEs successfully capture meaningful representations , enabling better performance on manipulation tasks , while matching previous approaches for prediction tasks . In particular , they permit certain manipulation tasks that can not be performed with conventional approaches , such as manipulating characteristics without changing the labels themselves and producing multiple distinct samples consistent with the desired intervention . We summarize our contributions as follows : i ) showing how labels can be used to capture and isolate rich characteristic information ; ii ) formulating CCVAEs , a novel model class and objective for supervised and semi-supervised learning in VAEs that allows this information to be captured effectively ; iii ) demonstrating CCVAEs ’ ability to successfully learn meaningful representations in practice . 2 BACKGROUND . VAEs ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) are a powerful and flexible class of model that combine the unsupervised representation-learning capabilities of deep autoencoders ( Hinton & Zemel , 1994 ) with generative latent-variable models—a popular tool to capture factored low-dimensional representations of higher-dimensional observations . In contrast to deep autoencoders , generative models capture representations of data not as distinct values corresponding to observations , but rather as distributions of values . A generative model defines a joint distribution over observed data x and latent variables z as pθ ( x , z ) = p ( z ) pθ ( x | z ) . Given a model , learning representations of data can be viewed as performing inference—learning the posterior distribution pθ ( z | x ) that constructs the distribution of latent values for a given observation . VAEs employ amortized variational inference ( VI ) ( Wainwright & Jordan , 2008 ; Kingma & Welling , 2013 ) using the encoder and decoder of an autoencoder to transform this setup by i ) taking the model likelihood pθ ( x | z ) to be parameterized by a neural network using the decoder , and ii ) constructing an amortized variational approximation qφ ( z | x ) to the ( intractable ) posterior pθ ( z | x ) using the encoder . The variational approximation of the posterior enables effective estimation of the objective—maximizing the marginal likelihood—through importance sampling . The objective is obtained through invoking Jensen ’ s inequality to derive the evidence lower bound ( ELBO ) of the model which is given as : log pθ ( x ) = logEqφ ( z|x ) [ pθ ( z , x ) qφ ( z | x ) ] ≥ Eqφ ( z|x ) [ log pθ ( z , x ) qφ ( z | x ) ] ≡ L ( x ; φ , θ ) . ( 1 ) Given observations D = { x1 , . . . , xN } taken to be realizations of random variables generated from an unknown distribution pD ( x ) , the overall objective is 1N ∑ n L ( xn ; θ , φ ) . Hierarchical VAEs Sønderby et al . ( 2016 ) impose a hierarchy of latent variables improving the flexibility of the approximate posterior , however we do not consider these models in this work . Semi-supervised VAEs ( SSVAEs ) ( Kingma et al. , 2014 ; Maaløe et al. , 2016 ; Siddharth et al. , 2017 ) consider the setting where a subset of data S ⊂ D is assumed to also have corresponding labels y. Denoting the ( unlabeled ) data as U = D\S , the log-marginal likelihood is decomposed as log p ( D ) = ∑ ( x , y ) ∈S log pθ ( x , y ) + ∑ x∈U log pθ ( x ) , where the individual log-likelihoods are lower bounded by their ELBOs . Standard practice is then to treat y as a latent variable to marginalize over whenever the label is not provided . More specifically , most approaches consider splitting the latent space in z = { zy , z\y } and then directly fix zy = y whenever the label is provided , such that each dimension of zy explicitly represents a predicted value of a label , with this value known exactly only for the labeled datapoints . Much of the original motivation for this ( Kingma et al. , 2014 ) was based around performing semi–supervised classification of the labels , with the encoder being used to impute the values of zy for the unlabeled datapoints . However , the framework is also regularly used as a basis for learning meaningful representations and performing manipulations , exploiting the presence of the decoder to generate new datapoints after intervening on the labels via changes to zy . Our focus lies on the latter , for which we show this standard formulation leads to serious pathologies . Our primary goal is not to improve the fidelity of generations , but instead to demonstrate how label information can be used to structure the latent space such that it encapsulates and disentangles the characteristics associated with the labels . 3 RETHINKING SUPERVISION . As we explained in the last section , the de facto assumption for most approaches to supervision in VAEs is that the labels correspond to a partially observed augmentation of the latent space , zy . However , this can cause a number of issues if we want the latent space to encapsulate not just the labels themselves , but also the characteristics associated with these labels . For example , encapsulating the youthful characteristics of a face , not just the fact that it is a “ young ” face . At an abstract level , such an approach fails to capture the relationship between the inputs and labels : it fails to isolate characteristic information associated with each label from the other information required to reconstruct data . More specifically , it fails to deal with the following issues . Firstly , the information in a datapoint associated with a label is richer than stored by the ( typically categorical ) label itself . That is not to say such information is absent when we impose zy = y , but here it is entangled with the other latent variables z\y , which simultaneously contain the associated information for all the labels . Moreover , when y is categorical , it can be difficult to ensure that the VAE actually uses zy , rather than just capturing information relevant to reconstruction in the higher-capacity , continuous , z\y . Overcoming this is challenging and generally requires additional heuristics and hyper-parameters . Second , we may wish to manipulate characteristics without fully changing the categorical label itself . For example , making a CelebA image depict more or less ‘ smiling ’ without fully changing its “ smile ” label . Here we do not know how to manipulate the latents to achieve this desired effect : we can only do the binary operation of changing the relevant variable in zy . Also , we often wish to keep a level of diversity when carrying out conditional generation and , in particular , interventions . For example , if we want to add a smile , there is no single correct answer for how the smile would look , but taking zy = `` smile '' only allows for a single point estimate for the change . Finally , taking the labels to be explicit latent variables can cause a mismatch between the VAE prior p ( z ) and the pushforward distribution of the data to the latent space q ( z ) = EpD ( x ) [ qφ ( z | x ) ] . During training , latents are effectively generated according to q ( z ) , but once learned , p ( z ) is used to make generations ; variations between the two effectively corresponds to a train-test mismatch . As there is a ground truth data distribution over the labels ( which are typically not independent ) , taking the latents as the labels themselves implies that there will be a ground truth q ( zy ) . However , as this is not generally known a priori , we will inevitably end up with a mismatch . What do we want from supervision ? Given these issues , it is natural to ask whether having latents directly correspond to labels is actually necessary . To answer this , we need to think about exactly what it is we are hoping to achieve through the supervision itself . Along with uses of VAEs more generally , the three most prevalent tasks are : a ) Classification , predicting the labels of inputs where these are not known a priori ; b ) Conditional Generation , generating new examples conditioned on those examples conforming to certain desired labels ; and c ) Intervention , manipulating certain desired characteristics of a data point before reconstructing it . Inspecting these tasks , we see that for classification we need a classifier form z to y , for conditional generation we need a mechanism for sampling z given y , and for inventions we need to know how to manipulate z to bring about a desired change . None of these require us to have the labels directly correspond to latent variables . Moreover , as we previously explained , this assumption can be actively harmful , such as restricting the range of interventions that can be performed .

Summary: The paper proposes to re-think the fashion of using label information in the VAE framework. The authors propose to disentangle information about the label (or, more generally, the context) in a "hard-coded" manner, namely, by using a separate set of variables for the label (context). The paper is written in a lucid manner, and the presented results are sound.