Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling

1 INTRODUCTION . Images and texts commonly occur together in the real world . There exists a wide variety of deep neural network based unidirectional methods that model images ( texts ) given texts ( images ) ( Gomez et al. , 2017 ; Kiros & Szepesvari , 2012 ; Reed et al. , 2016 ; Xu et al. , 2018 ; Zhang et al. , 2017a ) . There also exist probabilistic graphic model based bidirectional methods ( Srivastava & Salakhutdinov , 2012b ; a ; Wang et al. , 2018 ) that capture the joint distribution of images and texts . These bidirectional methods , however , often make restrictive parametric assumptions that limit their image generation ability . Exploiting recent progress on deep probabilistic models and variational inference ( Kingma & Welling , 2014 ; Zhou et al. , 2016 ; Zhang et al. , 2018a ; Goodfellow et al. , 2014 ; Zhang et al. , 2017b ) , we propose an end-to-end learning framework to construct multi-modality deep generative models that can not only generate vivid image-text pairs , but also achieve state-of-the-art results on various unidirectional tasks ( Srivastava & Salakhutdinov , 2012b ; a ; Wang et al. , 2018 ; Gomez et al. , 2017 ; Xu et al. , 2018 ; Zhang et al. , 2017a ; b ; Verma et al. , 2018 ; Zhang et al. , 2018b ) , such as generating photo-realistic images given texts and performing text-based zero-shot learning . To extract and relate semantic and visual concepts , we first introduce variational hetero-encoder ( VHE ) that encodes an image to decode its textual description ( e.g. , tags , sentences , binary attributes , and long documents ) , where the probabilistic encoder and decoder are jointly optimized using variational inference ( Blei et al. , 2017 ; Hoffman et al. , 2013 ; Kingma & Welling , 2014 ; Rezende et al. , 2014 ) . The latent representation of VHE can be sampled from either the variational posterior provided ∗Corresponding author by the image encoder given an image input , or the posterior of the text decoder via MCMC given a text input . VHE by construction has the ability to generate texts given images . To further enhance its text generation performance and allow synthesizing photo-realistic images given an image , text , or random noise , we feed the variational posterior of VHE in lieu of random noise as the source of randomness into the image generator of a generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) . We refer to this new modeling framework as VHE randomized GAN ( VHE-GAN ) . Off-the-shelf text decoders , image encoders , and GANs can be directly plugged into the VHE-GAN framework for end-to-end multi-modality learning . To begin with , as shown in Figs . 1 ( a ) and 1 ( b ) , we construct VHE-StackGAN++ by using the Poisson gamma belief network ( PGBN ) ( Zhou et al. , 2016 ) as the VHE text decoder , using the Weibull upward-downward variational encoder ( Zhang et al. , 2018a ) as the VHE image encoder , and feeding the concatenation of the multi-stochastic-layer latent representation of the VHE as the source of randomness into the image generator of StackGAN++ ( Zhang et al. , 2017b ) . While VHE-StackGAN++ already achieves very attractive performance , we find that its performance can be clearly boosted by better exploiting the multi-stochastic-layer semantically meaningful hierarchical latent structure of the PGBN text decoder . To this end , as shown in Figs . 1 ( a ) and 1 ( c ) , we develop VHE-raster-scan-GAN to perform image generation in not only a multi-scale low-to-high-resolution manner in each layer , as done by StackGAN++ , but also a hierarchical-semantic coarse-to-fine fashion across layers , a unique feature distinguishing it from existing methods . Consequently , not only can VHE-raster-scan-GAN generate vivid high-resolution images with better details , but also build interpretable hierarchical semantic-visual relationships between the generated images and texts . Our main contributions include : 1 ) VHE-GAN that provides a plug-and-play framework to integrate off-the-shelf probabilistic decoders , variational encoders , and GANs for end-to-end bidirectional multi-modality learning ; the shared latent space can be inferred either by image encoder q ( z |x ) , if given images , or by Gibbs sampling from the conditional posterior of text decoder p ( t | z ) , if given texts ; 2 ) VHE-raster-scan-GAN that captures and relates hierarchical semantic and visual concepts to achieve state-of-the-art results in various unidirectional and bidirectional image-text modeling tasks . 2 VARIATIONAL HETERO-ENCODER RANDOMIZED GANS . VAEs and GANs are two distinct types of deep generative models . Consisting of a generator ( decoder ) p ( x | z ) , a prior p ( z ) , and an inference network ( encoder ) q ( z |x ) that is used to approximate the posterior p ( z |x ) , VAEs ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) are optimized by maximizing the evidence lower bound ( ELBO ) as ELBO = Ex∼pdata ( x ) [ L ( x ) ] , L ( x ) : = Ez∼q ( z |x ) [ ln p ( x | z ) ] − KL [ q ( z |x ) ||p ( z ) ] , ( 1 ) where pdata ( x ) = ∑N i=1 1 N δxi represents the empirical data distribution . Distinct from VAEs that make parametric assumptions on data distribution and perform posterior inference , GANs in general use implicit data distribution and do not provide meaningful latent representations ( Goodfellow et al. , 2014 ) ; they learn both a generator G and a discriminator D by optimizing a mini-max objective as minG maxD { Ex∼pdata ( x ) [ lnD ( x ) ] + Ez∼p ( z ) [ ln ( 1−D ( G ( z ) ) ) ] } , ( 2 ) where p ( z ) is a random noise distribution that acts as the source of randomness for data generation . 2.1 VHE-GAN OBJECTIVE FUNCTION FOR END-TO-END MULTI-MODALITY LEARNING . Below we show how to construct VHE-GAN to jointly model images x and their associated texts t , capturing and relating hierarchical semantic and visual concepts . First , we modify the usual VAE into VHE , optimizing a lower bound of the text log-marginal-likelihood Et∼pdata ( t ) [ ln p ( t ) ] as ELBOvhe = Epdata ( t , x ) [ Lvhe ( t , x ) ] , Lvhe ( t , x ) : = Ez∼q ( z |x ) [ ln p ( t | z ) ] −KL [ q ( z |x ) ||p ( z ) ] , ( 3 ) where p ( t | z ) is the text decoder , p ( z ) is the prior , p ( t ) = Ez∼p ( z ) [ p ( t | z ) ] , and Lvhe ( t , x ) ≤ lnEz∼q ( z |x ) [ p ( t | z ) p ( z ) q ( z |x ) ] = ln p ( t ) . Second , the image encoder q ( z |x ) , which encodes image x into its latent representation z , is used to approximate the posterior p ( z | t ) = p ( t | z ) p ( z ) /p ( t ) . Third , variational posterior q ( z |x ) in lieu of random noise p ( z ) is fed as the source of randomness into the GAN image generator . Combing these three steps , with the parameters of the image encoder q ( z |x ) , text decoder p ( t | z ) , and GAN generator denoted by E , Gvae , and Ggan , respectively , we express the objective function of VHE-GAN for joint image-text end-to-end learning as min E , Gvae , Ggan max D Epdata ( t , x ) [ L ( t , x ) ] , L ( t , x ) : = lnD ( x ) + KL [ q ( z |x ) ||p ( z ) ] + Ez∼q ( z |x ) [ ln ( 1−D ( Ggan ( z ) ) ) − ln p ( t | z ) ] . ( 4 ) Note the objective function in ( 4 ) implies a data-triple-reuse training strategy , which uses the same data mini-batch in each stochastic gradient update iteration to jointly train the VHE , GAN discriminator , and GAN generator ; see a related objective function , shown in ( 10 ) of Appendix A , that is resulted from naively combining the VHE and GAN training objectives . In VHE-GAN , the optimization of the encoder parameter E is related to not only the VHE ’ s ELBO , but also the GAN mini-max objective function , forcing the variational posterior q ( z |x ) to serve as a bridge between VHE and GAN , allowing them to help each other . Although there are some models ( Mescheder et al. , 2017 ; Makhzani et al. , 2015 ; Tolstikhin et al. , 2018 ; Dumoulin et al. , 2017 ; Donahue et al. , 2017 ; Che et al. , 2017 ; Srivastava et al. , 2017 ; Grover et al. , 2018 ; Larsen et al. , 2016 ; Huang et al. , 2018 ) combining VAEs and GANs in various ways , they focus on single-modality tasks while the VHE-GAN on two different modalities . In Appendix A , we analyze the properties of the VHE-GAN objective function and discuss related works . Below we develop two different VHE-GANs , one integrates off-the-shelf modules , while the other introduces new interpretable hierarchical latent structure . 2.2 VHE-STACKGAN++ WITH OFF-THE-SHELF MODULES . As shown in Figs . 1 ( a ) and 1 ( b ) , we first construct VHE-StackGAN++ by plugging into VHE-GAN three off-the-shelf modules , including a deep topic model ( Zhou et al. , 2016 ) , a ladder-structured encoder ( Zhang et al. , 2018a ) , and StackGAN++ ( Zhang et al. , 2017b ) . For text analysis , both sequence models and topic models are widely used . Sequence models ( Bengio et al. , 2003 ) often represent each document as a sequence of word embedding vectors , capturing local dependency structures with some type of recurrent neural networks ( RNNs ) , such as long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) . Topic models such as latent Dirichlet allocation ( LDA ) ( Blei et al. , 2003 ) often represent each document as a bag of words ( BoW ) , capturing global word cooccurrence patterns into latent topics . Suitable for capturing local dependency structure , existing sequence models often have difficulty in capturing long-range word dependencies and hence macrolevel information , such as global word cooccurrence patterns ( i.e. , topics ) , especially for long documents . By contrast , while topic models ignore word order , they are very effective in capturing latent topics , which are often directly related to macro-level visual information ( Gomez et al. , 2017 ; Dieng et al. , 2017 ; Lau et al. , 2017 ) . Moreover , topic models can be applied to not only sequential texts , such as few sentences ( Wang et al. , 2009 ; Jin et al. , 2015 ) and long documents ( Zhou et al. , 2016 ) , but also non-sequential ones , such as textual tags ( Srivastava & Salakhutdinov , 2012a ; 2014 ; Wang et al. , 2018 ) and binary attributes ( Elhoseiny et al. , 2017b ; Zhu et al. , 2018 ) . For this reason , for the VHE text decoder , we choose PGBN ( Zhou et al. , 2016 ) , a state-of-the-art topic model that can also be represented as a multi-stochastic-layer deep generalization of LDA ( Cong et al. , 2017 ) . We complete VHE-StackGAN++ by choosing the Weibull upward-downward variational encoder ( Zhang et al. , 2018a ) as the VHE image encoder , and feeding the concatenation of all the hidden layers of PGBN as the source of randomness to the image generator of StackGAN++ ( Zhang et al. , 2017b ) . As in Fig . 1 , we use a VHE that encodes an image into a deterministic-upward–stochastic-downward ladder-structured latent representation , which is used to decode the corresponding text . Specifically , we represent each document as a BoW high-dimensional sparse count vector tn ∈ ZK0 , where Z = { 0 , 1 , · · · } and K0 is the vocabulary size . For the VHE text decoder , we choose to use PGBN to extract hierarchical latent representation from tn . PGBN consists of multiple gamma distributed stochastic hidden layers , generalizing the “ shallow ” Poisson factor analysis ( Zhou et al. , 2012 ; Zhou & Carin , 2015 ) into a deep setting . PGBN with L hidden layers , from top to bottom , is expressed as θ ( L ) n ∼ Gam ( r , 1/s ( L+1 ) n ) , r ∼ Gam ( γ0/KL , 1/s0 ) , θ ( l ) n ∼ Gam ( Φ ( l+1 ) θ ( l+1 ) n , 1/s ( l+1 ) n ) , l = L− 1 , · · · , 2 , 1 , tn ∼ Pois ( Φ ( 1 ) θ ( 1 ) n ) , ( 5 ) where the hidden units θ ( l ) n ∈ R Kl + of layer l are factorized under the gamma likelihood into the product of topics Φ ( l ) ∈ RKl−1×Kl+ and hidden units of the next layer , R+ = { x , x ≥ 0 } , s ( l ) n > 0 , and Kl is the number of topics of layer l. If the texts are represented as binary attribute vectors bn , we can add a Bernoulli-Poisson link layer as bn = 1 ( tn ≥ 1 ) ( Zhou , 2015 ; Zhou et al. , 2016 ) . We place a Dirichlet prior on each column of Φ ( l ) . The topics can be organized into a directed acyclic graph ( DAG ) , whose node k at layer l can be visualized with the top words of [ ∏l−1 t=1 Φ ( t ) ] φ ( l ) k ; the topics tend to be very general in the top layer and become increasingly more specific when moving downwards . This semantically meaningful latent hierarchy provides unique opportunities to build a better image generator by coupling the semantic hierarchical structures with visual ones . Let us denote Φ = { Φ ( 1 ) , . . . , Φ ( L ) , r } as the set of global parameters of PGBN shown in ( 5 ) . Given Φ , we adopt the inference in Zhang et al . ( 2018a ) to build an Weibull upward-downward variational image encoder as ∏N n=1 ∏L l=1 q ( θ ( l ) n |xn , Φ ( l+1 ) , θ ( l+1 ) n ) , where Φ ( L+1 ) : = r , θ ( L+1 ) n : = ∅ , and q ( θ ( l ) n |xn , Φ ( l+1 ) , θ ( l+1 ) n ) = Weibull ( k ( l ) n + Φ ( l+1 ) θ ( l+1 ) n , λ ( l ) n ) . ( 6 ) The Weibull distribution is used to approximate the gamma distributed conditional posterior , and its parameters k ( l ) n , λ ( l ) n ∈ RKl are deterministically transformed from the convolutional neural network ( CNN ) image features f ( xn ) ( Szegedy et al. , 2016 ) , as shown in Fig . 1 ( a ) and described in Appendix D.1 . We denote Ω as the set of encoder parameters . We refer to Zhang et al . ( 2018a ) for more details about this deterministic-upward–stochastic-downward ladder-structured inference network , which is distinct from a usual inference network that has a pure bottom-up structure and only interacts with the generative model via the ELBO ( Kingma & Welling , 2014 ; Gulrajani et al. , 2017 ) . The multi-stochastic-layer latent representation z = { θ ( l ) } Ll=1 is the bridge between two modalities . As shown in Fig . 1 ( b ) , VHE-StackGAN++ simply randomizes the image generator of StackGAN++ ( Zhang et al. , 2017b ) with the concatenated vector θ = [ θ ( 1 ) , · · · , θ ( L ) ] . We provide the overall objective function in ( 15 ) of Appendix D.2 . Note that existing neural-network-based models ( Gomez et al. , 2017 ; Xu et al. , 2018 ; Zhang et al. , 2017a ; b ; Verma et al. , 2018 ; Zhang et al. , 2018b ) are often able to perform unidirectional but not bidirectional transforms between images x and texts t. However , bidirectional transforms are straightforward for the proposed model , as z can be either drawn from the image encoder q ( z |x ) in ( 6 ) , or drawn with an upward-downward Gibbs sampler ( Zhou et al. , 2016 ) from the conditional posteriors p ( z | t ) of the PGBN text decoder p ( t | z ) in ( 5 ) .

Summary: This paper proposes a combined architecture for image-text modeling. Though the proposed architecture is extremely detailed, the authors explain clearly the overarching concepts and methods used, within limited space. The experimental results are extremely strong, especially on sub-domains where conditional generative models have historically struggled such as images with angular, global features - often mechanical or human constructed objects. "Computers" and "cars" images in Figure 2 show this quite clearly. The model also functions for tagging and annotating images - performing well compared to models designed *only* for this task.