VHEGAN: Variational Hetero-Encoder Randomized GAN for Zero-Shot Learning

1 INTRODUCTION . There has been significant recent interest in zero-shot learning ( ZSL ) that leverages auxiliary semantic information to transfer the knowledge learned from the training categories ( seen classes ) to testing ones ( unseen classes ) ( Fu et al. , 2018 ; Elhoseiny et al. , 2017a ) . A particularly challenging task is text-based ZSL ( Elhoseiny et al. , 2013 ; 2017a ; Ba et al. , 2015 ; Elhoseiny et al. , 2017b ; Qiao et al. , 2016 ; Zhu et al. , 2018 ) , which assumes that there are S seen image classes and U unseen ones , and image class c ∈ { 1 , . . . , S + U } is associated with both a set of Nc images { xcn } n=1 , Nc and a textual description tc . In the training stage , all or a subset of the images and textual descriptions of the S seen classes , denoted as { { ( xcn , tc ) } n=1 , Nc } c=1 , S , are used to learn the model . While in the testing stage , one removes the class label of an image from the unseen classes , and then maps that image to the text description of one of the U unseen classes ; if the true mapping is the same as the most likely mapping ( or among the top-five mappings ) ranked by a ZSL algorithm , it is considered as a correct classification in terms of Top-1 accuracy ( or Top-5 accuracy ) . The average Top-1 ( or Top-5 ) accuracy of these mappings is used to evaluate the performance of a ZSL algorithm . One may also consider a generalized ZSL ( GZSL ) setting , at the testing stage of which each held-out image , from a seen/unseen class that is not used for training , needs to be mapped to one of the S + U classes ( Socher et al. , 2013 ; Chao et al. , 2016 ; Verma et al. , 2018 ) . Given the success of deep generative models in unraveling complex data structure ( Kingma & Welling , 2014 ; Kingma et al. , 2014 ; Zhou et al. , 2016 ) , we are motivated to apply them to ZSL , especially when there are not many labeled data for the seen classes ( Wang et al. , 2018b ; Verma et al. , 2018 ) . Although some image caption models ( Wang et al. , 2017 ; Xu et al. , 2015 ) are able to exploit sequential information to related an image to a simple sentence , and a sequential description could be excellent at defining a specific individual image , the key words could be more effective to define a class of images ( i.e. , not a single image ) . In addition , the text description for a class of images could vary from just a few words ( e.g. , a sentence or several tags ) to hundreds of sentences ( e.g. , a document ) , whose key words relevant for ZSL could be more robustly identified under the bag-of-words ( BOW ) text representation . For these reasons , to address the challenges of text-based ZSL , we first introduce a variational hetero-encoder ( VHE ) that encodes an image to decode its BOW textual description . The proposed VHE is related to a usual variational auto-encoder ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) in using variational inference ( Jordan et al. , 1999 ; Blei et al. , 2017 ; Hoffman et al. , 2013 ) to jointly optimize its encoder and decoder , but differs from a VAE in making the input of the encoder , which is an image , different from the output of the decoder , which is a textual description associated with that image . Generative adversarial nets ( GANs ) ( Goodfellow et al. , 2014 ) and their various extensions , such as deep convolutional GANs Radford et al . ( 2016 ) , have made significant progresses in generating high-quality synthetic images . To further explore tighter relationships between images and texts to improve the performance of VHE for ZSL and add the capability to generate random images conditioning on an image , a document , or random noise , we further propose Variational Hetero-Encoder randomized Generative Adversarial Net ( VHEGAN ) that exploits the synergy between VHE and GAN . VHEGAN is distinct in several ways from the original GAN ( Goodfellow et al. , 2014 ) , which does not have an encoder , has little control over its latent space and hence the characteristics of generated images , and imposes no dependency relationship between the real images fed to its discriminator and the random noise fed to its generator . By contrast , VHEGAN inherits both the VHE image encoder and text decoder , makes the random noise fed into the GAN image generator be dependent on the image fed into the GAN discriminator during training , and can condition its latent space on either a document or an image during testing . Consequently , VHEGAN is capable of performing four different tasks : 1 ) text to image : generating images with the GAN image generator , or retrieving images , given the conditional posterior of a document under the VHE text decoder ; 2 ) image to text : generating documents with the VHE text decoder , or retrieving documents , given the variational posterior produced by the VHE image encoder fed with a natural image ( text-ZSL is such an task ) ; 3 ) image to image : generating images with the GAN image generator , or retrieving images , given the variational posterior produced by the VHE image encoder fed with a natural image , with the similarity between a generated image and the input one measurable by the GAN discriminator ; and 4 ) noise to text-image pair : generating a document using the VHE text decoder and a corresponding image by feeding the latent variable of the VHE text decoder into the GAN image generator . 2 JOINT TEXT AND IMAGE GENERATION FOR ZERO-SHOT LEARNING . The proposed VHE uses the Poisson gamma belief network ( GBN ) of Zhou et al . ( 2015 ) as its text decoder , whose output is a bag-of-words count vector , as shown in Fig . 1 ( a ) , and a Weibull distribution based deterministic-upward–stochastic-downward inference network ( Zhang et al. , 2018 ) as its image encoder , whose input is a natural image , as shown in Fig . 1 ( b ) . Integrating VHE and GAN , the proposed VHEGAN , as shown in Fig . 1 ( c ) , tightly couples the VHE text generator , VHE image encoder , and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document , under the Poisson GBN text decoder , be the distribution of the random noise fed into the GAN image generator ; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder ; and the GAN discriminator tries to discriminate between that selected image and the generated one . All the three components that are structurally coupled with each other will be jointly optimized . 2.1 VHE TEXT DECODER ( GENERATOR ) : POISSON GAMMA BELIEF NETWORK . Two methods are popular in representing texts : word embedding ( Bengio et al. , 2003 ; Mikolov et al. , 2013 ) and BOW count vectors . Considering the need of semantically relate a class of images to the key words in the corresponding class-specific textual description , we represent the text descriptions as BOW count vectors as { tc ∈ ZK0 } S+U c=1 , where Z = { 0 , 1 , · · · } and K0 is the vocabulary size . To extract hierarchical latent representation from these high-dimensional sparse count vectors tc , we choose the Poisson GBN ( Zhou et al. , 2015 ) , which can be considered as a member of deep exponential families ( Ranganath et al. , 2015 ) . It consists of multiple gamma distributed stochastic hidden layers , generalizing the “ shallow ” Poisson factor analysis ( Zhou et al. , 2012 ; Zhou & Carin , 2015 ) that is closely related to latent Dirichlet allocation ( LDA ) ( Blei et al. , 2003 ) into a deep setting . More specifically , modeling tc under the Poisson likelihood , its generative model with L hidden layers , from top to bottom , is expressed as θ ( L ) c ∼ Gam ( r , 1/s ( L+1 ) c ) , . . . , θ ( l ) c ∼ Gam ( Φ ( l+1 ) θ ( l+1 ) c , 1/s ( l+1 ) c ) , . . . , θ ( 1 ) c ∼ Gam ( Φ ( 2 ) θ ( 2 ) c , 1/s ( 2 ) c ) , tc∼Pois ( Φ ( 1 ) θ ( 1 ) c ) , ( 1 ) where the hidden units θ ( l ) c ∈ R Kl + of layer l are factorized under the gamma likelihood into the product of the factor loading Φ ( l ) ∈ RKl−1×Kl+ and hidden units of the next layer , R+ = { x , x ≥ 0 } , s ( l ) c > 0 , and Kl is the number of topics ( factors ) of layer l. We further restrict that the sum of each column of Φ ( l ) is equal to one via a Dirichlet prior . The topics Φ ( l ) at hidden layer l can be visualized as [ ∏l−1 t=1 Φ ( t ) ] φ ( l ) k , which are found to be very specific in the bottom layer and becoming increasingly more general when moving upwards . More details about the Poisson GBN can be found in Zhou et al . ( 2016 ) . Note as shown in Cong et al . ( 2017 ) , the Poisson GBN can also be represented as deep LDA , an alternative representation that facilitates the derivation of a scalable stochastic-gradient MCMC algorithm that is well suited for deep topic modeling . For simplicity , below we use the Poisson GBN to refer to both the Poisson GBN and deep LDA representations of the same underlying deep generative model . Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al . ( 2017 ) that uses LDA ( Blei et al. , 2003 ) , a “ shallow ” probabilistic model , and Ba et al . ( 2015 ) that uses non-probabilistic black-box DNNs to learn class-specific document features . 2.2 VHE IMAGE ENCODER : WEIBULL UPWARD-DOWNWARD VARIATIONAL ENCODER . The Poisson GBN is equipped with both a closed-form upward-downward Gibbs sampler that is suitable for a moderate-sized corpus ( Zhou et al. , 2016 ) , and a topic-layer-adaptive stochastic gradient Riemannian ( TLASGR ) MCMC that is scalable to a big corpus ( Cong et al. , 2017 ) . To avoid using iterations to sample the latent representation of a testing document from its posterior , Zhang et al . ( 2018 ) develop Weibull upward-downward variational encoder ( WUDVE ) that directly projects an observed bag-of-words vector into its posterior distribution given a random sample of the global parameters , leading to extremely fast inference for out-of-sample prediction . Let us denote Φ = { Φ ( 1 ) , . . . , Φ ( L ) , r } , where Φ ( l ) = { φ ( l ) k } k , l , as the set of global parameters of the Poisson GBN shown in ( 1 ) . In this paper , we leverage WUDVE as the variational inference network to approximate the true posterior distribution p ( θ | t , Φ ) , where θ = { θ ( l ) } 1 , L denotes the set of local parameters , but change the input of WUDVE from the bag-of-words vector t into an image x associated with t. More specifically , as illustrated in Fig . 1b , we construct it as qΩ ( θ |Φ , x ) : = [ L−1∏ l=1 qΩ ( θ ( l ) |Φ ( l+1 ) , θ ( l+1 ) , x ) ] qΩ ( θ ( L ) | r , x ) , ( 2 ) where , with f ( x ) representing a feature vector deterministically projected from x via an off-theshelf pre-trained deep neural network f ( · ) , we define qΩ ( θ ( l ) |Φ ( l+1 ) , θ ( l+1 ) , x ) = Weibull ( k ( l ) Ω ( f ( x ) ) +Φ ( l+1 ) θ ( l+1 ) , λ ( l ) Ω ( f ( x ) ) ) , qΩ ( θ ( L ) | r , x ) = Weibull ( k ( L ) Ω ( f ( x ) ) + r , λ ( L ) Ω ( f ( x ) ) ) , ( 3 ) where k ( l ) Ω ( f ( x ) ) ∈ RKl and λ ( l ) Ω ( f ( x ) ) ∈ RKl are deterministically transformed from f ( x ) . We use Ω to denote all the parameters of the inference network . Let us denote pdata ( x , t ) = 1∑C c=1 ∑Nc n=1 ∑C c=1 ∑Nc n=1 δ ( xcn , tc ) as the empirical data distribution for the image-text pairs . Given Φ of the Poisson GBN , to approximate the posterior distribution of the local parameters θ given text description t , we use WUVDE qΩ ( θ |Φ , x ) , whose input is an image x , and learn its parameters Ω by maximizing the evidence lower bound ( ELBO ) as max Ω LVHE ( Ω ) = E ( x , t ) ∼pdata ( x , t ) { EqΩ ( θ |Φ , x ) [ ln p ( t |Φ ( 1 ) , θ ( 1 ) ) − ln qΩ ( θ |Φ , x ) p ( θ |Φ ) ] } , ( 4 ) where p ( θ |Φ ) = [ ∏L−1 l=1 p ( θ ( l ) |Φ ( l+1 ) , θ ( l+1 ) , s ( l+1 ) ) ] p ( θ ( L ) | r , sL+1 ) is the prior distribution . We refer to Zhang et al . ( 2018 ) for more details on how WUDVE can be combined with TLASGRMCMC ( Cong et al. , 2017 ) to develop Weibull hybrid autoencoding inference ( WHAI ) , a hybrid SG-MCMC/variational inference that represents the posterior of Φ using MCMC samples , and the posterior of θ given a sample of Φ using WUDVE . To further improve the proposed VHE , below we show how to integrate it with a modified GAN to construct VHEGAN .

Summary: Paper Summary: This paper studies the zero-shot learning problem with deep generative models. More specifically, it proposed a hybrid framework that combines VAEs (more precisely, the variational hetero-encoder or VHE) and GANs all together. The entire model is composed of an image encoder (Weibull upward-downward variational encoder), a text decoder (Poisson Gamma belief network), and an image generator (generative adversarial network). Once learned, the generative models can be directly used for zero-shot classification and various image generation applications. In the experiments, two benchmark datasets CUB and Oxford-Flowers are used.