Learning shared manifold representation of images and attributes for generalized zero-shot learning

1 INTRODUCTION . The recent high performance of deep neural networks on image classification and object recognition depends greatly on whether we can obtain sufficiently labeled images of classes to predict . However , it is difficult to operate such a AI system as-is in the real world because the number of existing classes is huge and , as long as human beings create or develop new objects , their number can be expected to continue to increase day by day , resulting in difficulty obtaining labeled data of all these classes . Human beings have the excellent ability to infer objects that they have seen only a few by exploiting and transferring the knowledge learned in the past so as to survive in the harsh real world . In the machine learning context , this problem is formulated as few-shot learning . The extreme problem setting of few-shot learning is especially known as zero-shot learning ( ZSL ) , which is to train by labeled set from certain classes called seen classes , and then to predict completely unseen classes not included in the training set . It is usually accomplished by preparing pre-defined semantic representations of all classes such as attributes , and learning the relations between images and class-attributes , which we call the relation-based method in this paper ( Lampert et al. , 2009 ; 2014 ; Frome et al. , 2013 ; Akata et al. , 2013 ; 2015 ; Romera-Paredes & Torr , 2015 ; Kodirov et al. , 2017 ) . However , this conventional setting is unnatural in application to the real world because the objects we encounter are not always new but are rather likely to have been experienced in the past . Therefore , a more natural setting that the test set includes examples of both seen and unseen classes has been proposed , which is called generalized zero-shot learning ( GZSL ) ( Chao et al. , 2016 ) . It is known that GZSL is a more difficult setting than conventional ZSL because seen classes are also candidates for prediction . Especially , it is pointed out that prediction to unseen classes becomes fail severely in the case of relation-based methods ( Chao et al. , 2016 ; Xian et al. , 2018 ) . The main reason is that they can not obtain the generalized representation space such that examples of unseen classes are properly mapped because this information is not included in the training set . For example , if unseen classes are arranged to overlap with the seen classes in the representation space , which means this space is not generalized for unseen classes , all examples of the unseen classes might be biased to be predicted as one of the seen classes . Therefore , in this study , we propose a new solution of GZSL called shared manifold learning . At first , we aim to learn a mapping that embeds each image and attribute in the same representation space , i.e. , shared representation learning of images and attributes . This contributes to the prediction of the relations in shared representation that is robust against the representation of the inputs . In addition , we make sure that this mapping is also generalized , which means that the inputs of the unseen classes are embedded in appropriate places in shared representation by interpolating from the information of seen classes . That is , it can be regarded as manifold learning in shared representation . If such shared manifold learning can be performed properly , we can estimate the correct classattributes from the test image by embedding it in the shared manifold representation , regardless of whether this image is seen or unseen . To realize this , we propose modality invariant variational autoencoders ( MIVAE ) , which are based on variational autoencoders ( VAE ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . In this method , first of all , to obtain shared representations from inputs of images x and class-attributes ay , we define mappings from each input to the latent space z as the deep probabilistic distributions , qλ ( z|x ) and qλ ( z|ay ) . Next , to learn manifold representation from these two inputs , we prepare a VAE with both images and attributes as inputs . In learning of MIVAE , we not only maximize the lower bound of this VAE but also learn so that each mapping approximates the encoder of VAE , qφ ( z|x , a ) , and so that each mapping also approaches another . This makes it possible to obtain the mappings that embed each input in the shared manifold representation . See figure 1 as the outline of shared manifold learning with MIVAE . If we consider images and attributes as different modalities , then MIVAE can be regarded as an extension of works examining multimodal VAE ( Suzuki et al. , 2016 ; Vedantam et al. , 2017 ) . The contribution of this research is as follows . • We discuss that the biased problem is related to the reason why the relation-based methods in GZSL fails and propose shared manifold learning as a concept to solve it . • We propose MIVAE , which extended VAE as a method to realize shared manifold learning . • We confirm that MIVAE can perform shared manifold learning properly and that it can contribute to the higher performance of GZSL compared to existing relation-based GZSL methods . 2 PROBLEM FORMULATION . We assume that the dataset Dtr = { xi , yi } Ntri=1 is given as the training set , where xi ∈ X is the input data , e.g . an image , and yi ∈ Ys = { 1 ... , S } is the corresponding label data . The objective Training Test Training Test of ZSL is to train the classifier using Dtr and to predict labels ŷj ∈ Y from the example xj ∈ X in the test set Dts = { xj } Ntsj=1 . In the original ZSL , it is assumed that the classes of the test set are completely unseen in the training set , which means Y = Yu = { S + 1 , ... , U } . On the other hand , our goal is to train in the setting of GZSL , which includes both seen and unseen classes in the test set ( Y = Ys ∩ Yu ) . We also assume that we have the class-attribute matrixA ∈ RM× ( S+U ) as the semantic information of classes , where each column represents the M -dimensional attribute vector ac ∈ A = RM of each class c = 1 , ... , S + U . Using this attribute vector , the training set Dtr can be replaced as { xi , ayi } Ntri=1 . 3 PROPOSED METHOD . 3.1 BIASED PROBLEM IN RELATION-BASED GZSL AND SHARED MANIFOLD LEARNING . In this section , we discuss the reason why the prediction of test data fails in relation-based GZSL and how to solve it . The relation-based methods learn the relations between the image and the attribute by mapping them to the representation space . Depending on each method , this representation space is considered as attribute space ( image-attribute projection ) or another shared space ( shared representation learning ) . For details , see the section 4 . What is important in learning both ZSL and GZSL is to obtain the relations between images and attributes generalized to test data , that is , to map each pair of examples to the same place in the representation space . Furthermore , in the setting of GZSL , since the test set includes the seen classes , it is necessary to learn a generalized representation where the unseen classes do not overlap with the seen classes . However , this is clearly difficult because information of the unseen classes does not exist in the training set . If the generalization between classes fails and the representations of the unseen classes overlap the seen classes , all unseen examples may be classified biased toward the seen classes ( see figure 2 ( a ) ) . Therefore , in GZSL , we have to consider means to learn generalized representations not only between images and attributes but also between classes . Note that ZSL does not include a biased problem like GZSL because the test set does not include the seen classes ( see figure 2 ( b ) ) . In this paper , we propose the framework of learning to solve this problem as follows . First , we consider mapping from each image and attribute to the same space . At this time , we enforce the images and attributes to be the same representations in the embedding space ( shared representation learning ) . One of the advantages of mapping to the shared space is that we can think of relations between examples in a space with more information by integrating images and attributes . It is thought that we can estimate the location of unseen data better by performing manifold learning as described later in such a space . Also , Chao et al . ( 2016 ) argues that improvement of the representation of attributes is vital for GZSL , which can be realized by the shared representation integrated with the images . Another advantage is that since shared space does not depend on input dimensions or representation , we can make robust inference to the complexity of inputs . Next , we perform manifold learning to obtain a shared representation generalized in the unseen classes . In manifold learning , continuous mapping from input space to shared space can be learned . Therefore , unseen data also can be mapped to an appropriate place in the shared space by interpolating from the prior knowledge of the seen classes . We designate these series of learning as shared manifold learning of images and attributes . We claim that we can deal with the bias-related shortcoming of GZSL by learning shared manifold representation . 3.2 MULTIMODAL INVARIANT VARIATIONAL AUTOENCODER . Next , we propose modality invariant VAE ( MIVAE ) , which is a method of learning mapping from two inputs to a space to be shared and manifold representations . Here , we consider that the dataset of ZSL { xi , ayi } Ntri=1 has different modalities , which means that x and ay are different in their spaces and distributions , i.e. , X 6= A and p ( X ) 6= p ( A ) , where X = { x1 , ... , xNtr } and A = { ay1 , ... , ayNtr } . At first , we consider two Gaussian distributions parameterized with deep neural networks qλ ( z|x ) and qλ ( z|ay ) as mappings from each of X andA to Z , where λ is the parameter of the distribution . See appendix A for the way of parameterization with deep neural networks . In MIVAE , we aim to learn so that these mappings embed inputs in shared manifold representation . Note that it is important to set these mappings to the deep probabilistic distributions in obtaining shared representations . For a detailed discussion on this , see appendix C. Next , we use variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , which has been widely used as a model of representation learning and manifold learning . Moreover , we consider VAE with two modalities as input , which is called joint VAE ( Vedantam et al. , 2017 ) . Joint VAE has a latent concept z that is common to the two modalities x , ay , each of which is generated conditionally and independently from the concept . That is , their generative processes are z ∼ p ( z ) and x , ay ∼ pθ ( x , ay|z ) = pθ ( x|z ) pθ ( ay|z ) , where θ corresponds to model parameters . These processes indicate that the latent variable z can be regarded as a shared representation containing different modality information . To train this , we should maximize the joint distribution p ( x , ay ) = ∫ pθ ( x , ay|z ) p ( z ) dz over the training set . However , it is difficult to perform this maximization directly because this joint distribution is intractable . Therefore , we instead maximize the following evidence lower bound ( ELBO ) . Lvae = −DKL ( qφ ( z|x , ay ) ||p ( z ) ) + Eqφ ( z|x , ay ) [ log pθ ( x|z ) ] + Eqφ ( z|x , ay ) [ log pθ ( ay|z ) ] , ( 1 ) where qφ ( z|x , ay ) is an approximate distribution of posterior and φ represents parameters . We call qφ ( z|x , ay ) as encoder and pθ ( x|z ) and pθ ( ay|z ) as decoder . These models are also parameterized by deep neural networks . The distribution of the encoder is Gaussian , and that of the decoder depends on the distribution of the input data . From the encoder of trained VAE , we can extract shared representations that integrate the information of different modalities . In addition , the VAE can obtain the manifold representation of the input in the latent space and appropriately place the unseen input on the latent space by interpolating the seen information . Because we use images and attributes as inputs to joint VAE here , we can obtain the encoder that integrates and maps them into manifold representation , i.e. , we can perform shared manifold learning . Therefore , if we train each modality ’ s distribution to minimize the distance between the encoder , then we can obtain such representations from the distribution of each modality . We define this distance using Kullback-Liebler ( KL ) divergence as Ldist = DKL ( qφ ( z|x , ay ) ||qλ ( z|x ) ) +DKL ( qφ ( z|x , ay ) ||qλ ( z|ay ) ) . ( 2 ) Intuitively , because the encoder has more information to be conditioned , it seems that the mapping distributions , especially qλ ( z|ay ) , might degenerate if each is brought closer to the encoder . Actually , Vedantam et al . ( 2017 ) showed that minimization of Epdata ( x , ay ) [ DKL ( qφ ( z|x , ay ) ||qλ ( z|ay ) ) ] is equivalent to minimization of Epdata ( x , a ) [ DKL ( q avg φ ( z|ay ) ||qλ ( z|ay ) ) ] , where q avg φ ( z|ay ) = Epdata ( x|ay ) [ qφ ( z|x , ay ) ] , i.e. , this is the average of the encoder over all x corresponding to a . Therefore , the minimization of equation 2 encourages each mapping to close to the distribution complementing the other modality of the encoder , thereby resulting in stabilization . When equation 2 becomes minimum , both qλ ( z|x ) = ∫ qφ ( z|x , a ) da and qλ ( z|a ) =∫ qφ ( z|x , a ) dx are satisfied . However , if the objective function of VAE is also optimized simultaneously , then these minimization may not be performed properly due to the influence of the VAE terms during training . Therefore , to force the two distributions to be explicitly close at the training time , we also minimize the following equation : Lmap = DKL ( qλ ( z|x ) ||qλ ( z|ay ) ) . ( 3 ) Therefore , the objective function of MIVAE is L = −Lvae + λdistLdist + λmapLmap , ( 4 ) where λdist and λmap are hyper-parameters . When λmap = 0 , this objective function is the same as JMVAE ( Suzuki et al. , 2016 ) , which means that this model can be regarded as an extension of JMVAE . After training MIVAE , a label of test image can be simply predicted as ŷj = argminy∈Y DKL ( qλ ( z|xj ) ||qλ ( z|ay ) ) .

Summary: This paper proposes a multimodal VAE model for the problem of generalized zero shot learning (GZSL). In GZSL, the test classes can contain examples from both seen as well as unseen classes, and due to the bias of the model towards the seen classes, the standard GZSL approaches tend to predict the majority of the inputs to belong to seen classes. The paper proposes a multimodal VAE model to mitigate this issue where a shared manifold learning learn for the inputs and the class attribute vectors.