VHEGAN: Variational Hetero-Encoder Randomized GAN for Zero-Shot Learning

1 INTRODUCTION . There has been significant recent interest in zero-shot learning ( ZSL ) that leverages auxiliary semantic information to transfer the knowledge learned from the training categories ( seen classes ) to testing ones ( unseen classes ) ( Fu et al. , 2018 ; Elhoseiny et al. , 2017a ) . A particularly challenging task is text-based ZSL ( Elhoseiny et al. , 2013 ; 2017a ; Ba et al. , 2015 ; Elhoseiny et al. , 2017b ; Qiao et al. , 2016 ; Zhu et al. , 2018 ) , which assumes that there are S seen image classes and U unseen ones , and image class c ∈ { 1 , . . . , S + U } is associated with both a set of Nc images { xcn } n=1 , Nc and a textual description tc . In the training stage , all or a subset of the images and textual descriptions of the S seen classes , denoted as { { ( xcn , tc ) } n=1 , Nc } c=1 , S , are used to learn the model . While in the testing stage , one removes the class label of an image from the unseen classes , and then maps that image to the text description of one of the U unseen classes ; if the true mapping is the same as the most likely mapping ( or among the top-five mappings ) ranked by a ZSL algorithm , it is considered as a correct classification in terms of Top-1 accuracy ( or Top-5 accuracy ) . The average Top-1 ( or Top-5 ) accuracy of these mappings is used to evaluate the performance of a ZSL algorithm . One may also consider a generalized ZSL ( GZSL ) setting , at the testing stage of which each held-out image , from a seen/unseen class that is not used for training , needs to be mapped to one of the S + U classes ( Socher et al. , 2013 ; Chao et al. , 2016 ; Verma et al. , 2018 ) . Given the success of deep generative models in unraveling complex data structure ( Kingma & Welling , 2014 ; Kingma et al. , 2014 ; Zhou et al. , 2016 ) , we are motivated to apply them to ZSL , especially when there are not many labeled data for the seen classes ( Wang et al. , 2018b ; Verma et al. , 2018 ) . Although some image caption models ( Wang et al. , 2017 ; Xu et al. , 2015 ) are able to exploit sequential information to related an image to a simple sentence , and a sequential description could be excellent at defining a specific individual image , the key words could be more effective to define a class of images ( i.e. , not a single image ) . In addition , the text description for a class of images could vary from just a few words ( e.g. , a sentence or several tags ) to hundreds of sentences ( e.g. , a document ) , whose key words relevant for ZSL could be more robustly identified under the bag-of-words ( BOW ) text representation . For these reasons , to address the challenges of text-based ZSL , we first introduce a variational hetero-encoder ( VHE ) that encodes an image to decode its BOW textual description . The proposed VHE is related to a usual variational auto-encoder ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) in using variational inference ( Jordan et al. , 1999 ; Blei et al. , 2017 ; Hoffman et al. , 2013 ) to jointly optimize its encoder and decoder , but differs from a VAE in making the input of the encoder , which is an image , different from the output of the decoder , which is a textual description associated with that image . Generative adversarial nets ( GANs ) ( Goodfellow et al. , 2014 ) and their various extensions , such as deep convolutional GANs Radford et al . ( 2016 ) , have made significant progresses in generating high-quality synthetic images . To further explore tighter relationships between images and texts to improve the performance of VHE for ZSL and add the capability to generate random images conditioning on an image , a document , or random noise , we further propose Variational Hetero-Encoder randomized Generative Adversarial Net ( VHEGAN ) that exploits the synergy between VHE and GAN . VHEGAN is distinct in several ways from the original GAN ( Goodfellow et al. , 2014 ) , which does not have an encoder , has little control over its latent space and hence the characteristics of generated images , and imposes no dependency relationship between the real images fed to its discriminator and the random noise fed to its generator . By contrast , VHEGAN inherits both the VHE image encoder and text decoder , makes the random noise fed into the GAN image generator be dependent on the image fed into the GAN discriminator during training , and can condition its latent space on either a document or an image during testing . Consequently , VHEGAN is capable of performing four different tasks : 1 ) text to image : generating images with the GAN image generator , or retrieving images , given the conditional posterior of a document under the VHE text decoder ; 2 ) image to text : generating documents with the VHE text decoder , or retrieving documents , given the variational posterior produced by the VHE image encoder fed with a natural image ( text-ZSL is such an task ) ; 3 ) image to image : generating images with the GAN image generator , or retrieving images , given the variational posterior produced by the VHE image encoder fed with a natural image , with the similarity between a generated image and the input one measurable by the GAN discriminator ; and 4 ) noise to text-image pair : generating a document using the VHE text decoder and a corresponding image by feeding the latent variable of the VHE text decoder into the GAN image generator . 2 JOINT TEXT AND IMAGE GENERATION FOR ZERO-SHOT LEARNING . The proposed VHE uses the Poisson gamma belief network ( GBN ) of Zhou et al . ( 2015 ) as its text decoder , whose output is a bag-of-words count vector , as shown in Fig . 1 ( a ) , and a Weibull distribution based deterministic-upward–stochastic-downward inference network ( Zhang et al. , 2018 ) as its image encoder , whose input is a natural image , as shown in Fig . 1 ( b ) . Integrating VHE and GAN , the proposed VHEGAN , as shown in Fig . 1 ( c ) , tightly couples the VHE text generator , VHE image encoder , and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document , under the Poisson GBN text decoder , be the distribution of the random noise fed into the GAN image generator ; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder ; and the GAN discriminator tries to discriminate between that selected image and the generated one . All the three components that are structurally coupled with each other will be jointly optimized . 2.1 VHE TEXT DECODER ( GENERATOR ) : POISSON GAMMA BELIEF NETWORK . Two methods are popular in representing texts : word embedding ( Bengio et al. , 2003 ; Mikolov et al. , 2013 ) and BOW count vectors . Considering the need of semantically relate a class of images to the key words in the corresponding class-specific textual description , we represent the text descriptions as BOW count vectors as { tc ∈ ZK0 } S+U c=1 , where Z = { 0 , 1 , · · · } and K0 is the vocabulary size . To extract hierarchical latent representation from these high-dimensional sparse count vectors tc , we choose the Poisson GBN ( Zhou et al. , 2015 ) , which can be considered as a member of deep exponential families ( Ranganath et al. , 2015 ) . It consists of multiple gamma distributed stochastic hidden layers , generalizing the “ shallow ” Poisson factor analysis ( Zhou et al. , 2012 ; Zhou & Carin , 2015 ) that is closely related to latent Dirichlet allocation ( LDA ) ( Blei et al. , 2003 ) into a deep setting . More specifically , modeling tc under the Poisson likelihood , its generative model with L hidden layers , from top to bottom , is expressed as θ ( L ) c ∼ Gam ( r , 1/s ( L+1 ) c ) , . . . , θ ( l ) c ∼ Gam ( Φ ( l+1 ) θ ( l+1 ) c , 1/s ( l+1 ) c ) , . . . , θ ( 1 ) c ∼ Gam ( Φ ( 2 ) θ ( 2 ) c , 1/s ( 2 ) c ) , tc∼Pois ( Φ ( 1 ) θ ( 1 ) c ) , ( 1 ) where the hidden units θ ( l ) c ∈ R Kl + of layer l are factorized under the gamma likelihood into the product of the factor loading Φ ( l ) ∈ RKl−1×Kl+ and hidden units of the next layer , R+ = { x , x ≥ 0 } , s ( l ) c > 0 , and Kl is the number of topics ( factors ) of layer l. We further restrict that the sum of each column of Φ ( l ) is equal to one via a Dirichlet prior . The topics Φ ( l ) at hidden layer l can be visualized as [ ∏l−1 t=1 Φ ( t ) ] φ ( l ) k , which are found to be very specific in the bottom layer and becoming increasingly more general when moving upwards . More details about the Poisson GBN can be found in Zhou et al . ( 2016 ) . Note as shown in Cong et al . ( 2017 ) , the Poisson GBN can also be represented as deep LDA , an alternative representation that facilitates the derivation of a scalable stochastic-gradient MCMC algorithm that is well suited for deep topic modeling . For simplicity , below we use the Poisson GBN to refer to both the Poisson GBN and deep LDA representations of the same underlying deep generative model . Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al . ( 2017 ) that uses LDA ( Blei et al. , 2003 ) , a “ shallow ” probabilistic model , and Ba et al . ( 2015 ) that uses non-probabilistic black-box DNNs to learn class-specific document features . 2.2 VHE IMAGE ENCODER : WEIBULL UPWARD-DOWNWARD VARIATIONAL ENCODER . The Poisson GBN is equipped with both a closed-form upward-downward Gibbs sampler that is suitable for a moderate-sized corpus ( Zhou et al. , 2016 ) , and a topic-layer-adaptive stochastic gradient Riemannian ( TLASGR ) MCMC that is scalable to a big corpus ( Cong et al. , 2017 ) . To avoid using iterations to sample the latent representation of a testing document from its posterior , Zhang et al . ( 2018 ) develop Weibull upward-downward variational encoder ( WUDVE ) that directly projects an observed bag-of-words vector into its posterior distribution given a random sample of the global parameters , leading to extremely fast inference for out-of-sample prediction . Let us denote Φ = { Φ ( 1 ) , . . . , Φ ( L ) , r } , where Φ ( l ) = { φ ( l ) k } k , l , as the set of global parameters of the Poisson GBN shown in ( 1 ) . In this paper , we leverage WUDVE as the variational inference network to approximate the true posterior distribution p ( θ | t , Φ ) , where θ = { θ ( l ) } 1 , L denotes the set of local parameters , but change the input of WUDVE from the bag-of-words vector t into an image x associated with t. More specifically , as illustrated in Fig . 1b , we construct it as qΩ ( θ |Φ , x ) : = [ L−1∏ l=1 qΩ ( θ ( l ) |Φ ( l+1 ) , θ ( l+1 ) , x ) ] qΩ ( θ ( L ) | r , x ) , ( 2 ) where , with f ( x ) representing a feature vector deterministically projected from x via an off-theshelf pre-trained deep neural network f ( · ) , we define qΩ ( θ ( l ) |Φ ( l+1 ) , θ ( l+1 ) , x ) = Weibull ( k ( l ) Ω ( f ( x ) ) +Φ ( l+1 ) θ ( l+1 ) , λ ( l ) Ω ( f ( x ) ) ) , qΩ ( θ ( L ) | r , x ) = Weibull ( k ( L ) Ω ( f ( x ) ) + r , λ ( L ) Ω ( f ( x ) ) ) , ( 3 ) where k ( l ) Ω ( f ( x ) ) ∈ RKl and λ ( l ) Ω ( f ( x ) ) ∈ RKl are deterministically transformed from f ( x ) . We use Ω to denote all the parameters of the inference network . Let us denote pdata ( x , t ) = 1∑C c=1 ∑Nc n=1 ∑C c=1 ∑Nc n=1 δ ( xcn , tc ) as the empirical data distribution for the image-text pairs . Given Φ of the Poisson GBN , to approximate the posterior distribution of the local parameters θ given text description t , we use WUVDE qΩ ( θ |Φ , x ) , whose input is an image x , and learn its parameters Ω by maximizing the evidence lower bound ( ELBO ) as max Ω LVHE ( Ω ) = E ( x , t ) ∼pdata ( x , t ) { EqΩ ( θ |Φ , x ) [ ln p ( t |Φ ( 1 ) , θ ( 1 ) ) − ln qΩ ( θ |Φ , x ) p ( θ |Φ ) ] } , ( 4 ) where p ( θ |Φ ) = [ ∏L−1 l=1 p ( θ ( l ) |Φ ( l+1 ) , θ ( l+1 ) , s ( l+1 ) ) ] p ( θ ( L ) | r , sL+1 ) is the prior distribution . We refer to Zhang et al . ( 2018 ) for more details on how WUDVE can be combined with TLASGRMCMC ( Cong et al. , 2017 ) to develop Weibull hybrid autoencoding inference ( WHAI ) , a hybrid SG-MCMC/variational inference that represents the posterior of Φ using MCMC samples , and the posterior of θ given a sample of Φ using WUDVE . To further improve the proposed VHE , below we show how to integrate it with a modified GAN to construct VHEGAN .

Summary: This work suggests a new model incorporating deep topic model (text decoder),  VHE (image encoder), and GAN. The topic model and the VHE shares the topic parameters, and the GAN generate an image regarding the topic. Then, for ZSL, the image is encoded to corresponding topic parameters, and the parameter can tell which text description (unseen) is matched with the highest probability. GAN model is used to generate an image given the topic distribution. During the training of the GAN, the VHE and topic model is jointly trained and can enhance the ZSL performance marginally.