Probabilistic Semantic Embedding

1 INTRODUCTION . In the last few years one of the most exciting developments in machine learning has been in the development of unsupervised learning models to learn latent or embedding spaces as exemplified by variational auto-encoders ( VAEs ) ( Kingma & Welling , 2013 ) for images and word2vec ( Mikolov et al. , 2013 ) for words . In this paper we propose a novel framework for coupling together such spaces allowing us to associate an image with a natural language description through a powerful generative probabilistic-semantic embedding in which both images and text are represented as probability distributions in a common latent space . Our Probabilistic Semantic Embedding model , PSE , is an extension of a VAE , where , rather than use a Kullback-Leibler ( KL ) divergence term between the image distributions and a single prior , we use a KL divergence between the image distribution and a distribution learned from text . We show this schematically in Figure 1 . Both image and text inputs are fed to networks that output the parameters describing normal distributions in the same vector space . Throughout the paper we use images and text as our working examples of the data modalities , although our model is itself generic and could be applied to any modality . Furthermore , although not the focus of this paper , it can also be trivially extended to an arbitrary number of additional input modalities ( for example audio ) . Despite its deceptive simplicity , this model creates a powerful latent space representation where both images and textual descriptors are represented as probability distributions . The geometry of the latent space captures both visual and semantic similarity between classes of object . Unlike current state-of-the-art models such as the Conditional VAE ( CVAE ) , our model is not handicapped by an assumption of independence between textual terms or labels . We demonstrate that the model allows for modes of operation in which images can be generated directly from labels , or from other images , or both , and , labels predicted from images . We also further demonstrate how , by encoding the text using pre-trained word embedding vectors , we can also exploit the semantics of the word embedding space , allowing us to generate images using words that are outside of the vocabulary of labels used to train our model , whilst at the same time allowing the extended model to produce better classification accuracy on unlabelled images . We compare our proposed PSE model against a range of state-of-the-art models for conditional and joint generative modelling of images and labels , and demonstrate that our model is capable of lower reconstruction error and higher classification accuracy . We show how prior knowledge from a pre-trained word2vec embedding improves both classification and reconstruction performance . The remainder of the paper is structured as follows : In section 2 we describe our method in detail . Related work is described in Section 3 . In Section 4 we demonstrate our method using one-hot encoded labels and compare with the state-of-the-art . In section 5 we explore how the model can be extended by using pre-trained word embeddings to capture relationships that are not explicit from the labelled image data used to train the model . Conclusions and ideas for future research directions can be found in Section 6 . 2 PROPOSED METHOD . Our method can be seen as an extension of the standard VAE . Recall that in a VAE we have a data set , D = { xi } Ni=1 , that maps an input image xi , through an encoder network with parameters φ , to a distribution , qφ ( z|xi ) , in a latent space . During training we sample a vector z from qφ ( z|xi ) that is passed to a decoder network with parameters θ to generate a reconstruction x̂ = Dec ( z|θ ) . We can use the reconstruction to generate image vectors x′ from a distribution p ( x′|x̂ ) , often taken to be N ( x′|x̂ , σ2 I ) . As derived in the original VAE paper ( Kingma & Welling , 2013 ) , we can write down the Evidence Lower BOund ( ELBO ) as : elbo ( x ) = Eqφ ( z|x ) [ log ( pθ ( x|z ) ) ] −DKL ( qφ ( z|x ) ∥∥ p ( z ) ) ( 1 ) where pθ ( x|z ) = p ( x′|x̂ ) and p ( z ) = N ( z ; 0 , I ) is a “ prior ” distribution . To train the network we maximise elbo ( x ) for all x sampled from D. To use the VAE as a generative model we would sample z from the prior p ( z ) and feed this to the decoder . In the PSE model , we consider a data set made up of image-label pairs D = { ( xi , li ) } Ni=1 . The images are given to an encoder that outputs the parameters of a distribution qφx ( z|x ) , while the labels are fed to a second encoder network with weights φl that output the parameters for a distribution qφl ( z|l ) . In our experiments , we choose the parameters output by both encoder networks to be the means and variances of a multivariate normal distribution with a diagonal covariance matrix , but other reparameterisable distributions are possible . In training we sample a random latent vector z from qφx ( z|x ) and feed it to a decoder network to obtain a reconstruction image x̂ = Dec ( z|θx ) . The only modification to the objective function of a standard VAE is that we replace p ( z ) by qφl ( z|l ) . That is , we maximise LPSE ( x , l ) = Eqφ ( z|x ) [ log ( pθx ( x|z ) ) ] −DKL ( qφx ( z|x ) ∥∥ qφl ( z|l ) ) ( 2 ) averaged over all pairs ( xi , li ) drawn from the new data set . Image Generation . To use the PSE model as an image generator we would put in a label l and sample a latent vector z from qφl ( z|l ) . This is given to the decoder to obtain an image x̂ = Dec ( z , θx ) . Image Classification . To use the model for image classification we can put in an image x and encode this to a ( Normal ) probability distribution qφx ( z|x ) . We choose to use the mean of that distribution to represent the image , and then compute the most likely label on the basis of the likelihood of the mean vector belonging to one of the labels . To get good classification performance , the model must learn a latent space representation that correctly separates the labels . 2.1 ENHANCING THE DECODER WITH LABEL INFORMATION . In the basic PSE model , the decoder takes as input a sampled latent variable z and produces an image x̂ . There is however no explicit discriminative supervisory signal given to the decoder during training relating to the target label of the image being generated . Inspired by the approach taken in InfoGAN ( Chen et al. , 2016 ) , we can extend our model to enable label information to aid the training of the image generator . As the entire model is trained end-to-end , the hope is that this addition will both reduce reconstruction loss , and also produce improved semantic alignment ( as evidenced by improved classification accuracy ) within the latent space . In our extended model , PSE∗ , we add an additional classification network to predict a label , l̂ , from the reconstructed image . That is , l̂ = Cla ( x̂|θl ) , where Cla is the classifier network that takes input x̂ with weights θl . In a similar manner as for an image , we can define pθl ( l|z ) as the probability of generating a label vector l given a latent vector z . Following the same rationale we used for images , we choose pθl ( l|z ) = p ( l|l̂ ) to be the probability of generating the correct label l from the predicted label l̂ . If the labels are encoded as one-hot vectors then we can take log ( p ( l|l̂ ) ) to be the cross-entropy , while if we are using word2vec embeddings we could for example use p ( l|l̂ ) = N ( l|l̂ , σ2 I ) . We can then define a modified objective function LPSE∗ ( x , l ) = Eqφx ( z|x ) [ log ( pθx ( x|z ) ) + log ( pθl ( l|z ) ) ] −DKL ( qφx ( z|x ) ∥∥ qφl ( z|l ) ) . ( 3 ) This extended model provides a further layer of auto-encoding ( i.e . we generate an image x̂ and a label l̂ ) which , as we see in Section 4 , provides a further improvement in performance . The new term Eqφx ( z|x ) [ log ( pθl ( l|z ) ) ] can be considered as the mutual information between labels and images , as introduced by Chen et al . ( 2016 ) . In these models the labels need to be encoded in vector form . This can be a simple one-hot encoded vector that the model embeds or it could be encoded using a standard pre-trained word embedding such as word2vec . For short text description and a suitable embedding we can average the embedding vectors for each word . As most word embeddings are high dimensional the average of a few words usually corresponds to a vector whose nearest neighbours are the words being averaged . As an example , if we take the phrase “ white horse ” and calculate the mean of the word2vec embeddings for “ white ” and “ horse ” we obtain a vector in the word2vec space whose two nearest neighbour words ( measured using cosine similarity ) are “ white ” and “ horse ” . 3 RELATED WORK . The area of building visual-semantic encodings that link visual images with labels has a long history in the fields of image retrieval and annotation ( e.g . Hare et al. , 2006 ; 2008 ) . Recently , as a result of advances in deep learning , it has become a highly active area of research ( e.g . Frome et al. , 2013 ; Akata et al. , 2013 ; Karpathy et al. , 2014 ; Akata et al. , 2016 ; Reed et al. , 2016 ; Zhang et al. , 2017 ) . These models have been applied to zero-shot attribute prediction ( Frome et al. , 2013 ; Akata et al. , 2013 ) , image annotation and text-based image retrieval ( Karpathy et al. , 2014 ; Karpathy & Fei-Fei , 2015 ; Reed et al. , 2016 ) . An exciting advance in some of the recent models is that they are fully generative ; that is they model a distribution in the latent embedding space that can be sampled and used to generate novel images . Our proposed PSE and PSE∗ models fall in the latter category of generative models . We argue that our models are both simpler than existing models , but also more flexible , allowing us to trivially couple to a word2vec embedding rather than one-hot encoded labels . At the same time , our models do not impose independence constraints on the textual labels , and actually represent embedded text or labels by probability distributions in the same latent space as the image distributions . The remainder of this section explores the state-of-the-art in visual-semantic embedding models . VISUAL-SEMANTIC EMBEDDING . The common approaches for visual-semantic embedding models learn semantic relationship between images and annotated text descriptions into a vector space by maximising visual and semantic similarity . The Deep visual-semantic embedding model ( DeViSE ) , in which the core visual model and language model are pre-trained , uses a combination of dot-product similarity and hinge rank loss ( Frome et al. , 2013 ) . DeViSE has shown the ability to make correct predictions across unseen classes by leveraging semantic knowledge . Karpathy et al . ( 2014 ) introduced a structured max-margin objective to calculate the similarity between images and sentences . Reed et al . ( 2016 ) propose a structured joint embedding model for end-to-end training to improve the performance on fine-grained visual recognition . To represent text descriptions as abstract concepts , some works use Gaussian distributions to model words . Ren et al . ( 2016 ) use Mahalanobis distance to measure the distance between visual features and text distributions . Wang et al . ( 2017 ) propose a VAE-based visual-semantic models to learn label distributions from visual features , in which a marginal regulariser is introduced to separate one label distribution to its nearest neighbour distribution maximally . However , these studies all aim at performing image classification ; they are not capable of generating visual contexts after training . VAE-BASED CONDITIONAL GENERATIVE MODELS AND JOINT GENERATIVE MODELS . There are a number of models that are similar in design to ours , and that we can directly compare against . In most cases the labels in these models are taken to be independent and thus these models can not capture label dependence or similarity , or be easily extended to using free text descriptions . Nevertheless we can compare the performance of our model against these using one-hot encoded labels . CVAE . Conditional Variational Auto-Encoders ( CVAEs ) extend the original ELBO to a conditional bound elbo ( x|l ) to approximate the conditional log-likelihood , when the encoder and decoder are conditioned with label information ( Kingma et al. , 2014 ; Sohn et al. , 2015 ) . The standard CVAEs assume that all the prior label embeddings for different classes share the same Gaussian prior . We can view the CVAE latent space as being extended by a one-hot vector encoding the actual label . The model thus directly assumes independence of the labels as they are orthogonal in the latent space . The objective of the standard CVAE has the form LCVAE ( x , l ) = Eqφ ( z|x , l ) [ log ( pθ ( x|z , l ) ) ] +DKL ( qφ ( z|x , l ) ∥∥ p ( z ) ) . ( 4 ) where p ( z ) is a Gaussian prior for all latent distributions . When a latent code z is combined with different labels represented as one-hot vectors , images with various labels can be generated . Therefore , the learned part of the embedding space ( excluding the one-hot label dimensions ) , does not contain any label-specific semantic information . For CVAEs , modality information can not be exchanged bidirectionally , because learnt latent vectors can not be used to do image annotation . JMVAE and TrELBO . The Joint Multimodal Variational Auto-Encoder ( JMVAE , Suzuki et al . ( 2016 ) ) and the Triple ELBO model ( TrELBO , Vedantam et al . ( 2018 ) ) learn the joint representation from multiple inputs with complicated training objectives . Both models just concatenate images and labels represented by one-hot vectors as the input of image and label encoder q ( z|x , l ) . However , these models still keep a Gaussian prior . Therefore , joint embedding distributions are forced to be closed to the prior , which causes more overlap in the embedding space which can be disadvantageous as dissimilar concepts might be forced closer together . The objective of JMVAE has the form LJMVAE ( x , l ) =Eqφ ( z|x , l ) [ log ( pθx ( x|z ) ) ] + Eqφ ( z|x , l ) [ log ( pθl ( l|z ) ) ] −DKL ( qφ ( z|x , l ) ∥∥ p ( z ) ) − α ( DKL ( qφ ( z|x , l ) ∥∥ qφx ( z|x ) ) +DKL ( qφ ( z|x , l ) ∥∥ qφl ( z|l ) ) ) . ( 5 ) This objective contains two reconstruction terms and three regularisation terms . DKL ( qφ ( z|x , l ) ‖qφl ( z|l ) ) makes the JMVAE learn a joint visual-semantic embedding space . However , this model does not embed labels to a latent space entirely based their visual contexts . There is still a potential intersection between different labels which is not caused by visual similarity . The TrELBO introduces two unpaired inference networks to fit a joint distribution to learn distributions of visual concepts . The objective is LTrELBO ( x , l ) =Eqφ ( z|x , l ) [ log ( pθx ( x|z ) ) ] + λ1 Eqφ ( z|x , l ) [ log ( pθl ( l|z ) ) ] + Eqφx ( z|x ) [ log ( pθx ( x|z ) ) ] + λ2Eqφl ( z|l ) [ log ( pθl ( l|z ) ) ] −DKL ( qφ ( z|x , l ) ∥∥ p ( z ) ) −DKL ( qφx ( z|x ) ∥∥ p ( z ) ) −DKL ( qφl ( z|l ) ∥∥ p ( z ) ) . ( 6 ) This objective is more complicated , but it can not learn a compositional latent embedding space , because it lacks a regulariser to learn the relationship between label distributions and image distributions . SCAN . Symbol-Concept Association Network ( SCAN , Higgins et al . ( 2018 ) ) is capable of learning representation of concepts from visual content . SCAN extends β-VAE ( Higgins et al. , 2017 ) , which can learn disentangled latent representations . Concept distributions are learnt from these disentangled representations . Therefore , SCAN does not learn a visual-semantic embedding space directly . In addition , SCAN does not generate high-quality image samples from symbolic concepts , because of the trade off between reconstruction quality and disentanglement in β-VAE ( the controlled capacity term is not applied in the training process of SCAN ( Higgins et al. , 2018 ; Burgess et al. , 2018 ) ) . SCAN is not an end-to-end model ; it uses a β-VAE-based model ( Higgins et al. , 2017 ) to learn disentangled latent representations , and then it uses the KL divergence DKL ( qφx ( z|x ) ‖qφl ( z|l ) ) to learn average label distributions to fit disentangled image distributions with the same labels . However , learning disentangled representations in this way punishes reconstruction quality .

Summary: This paper presents a VAE model that jointly models images and their labels. Specifically, the following the VAE framework, the proposed model encodes an image into a latent variable, whose prior is conditioned on the labels of the image, and the latent variable is used to reconstruct the image. The paper also presents a variant which also reconstructs the labels