Relation-based Generalized Zero-shot Classification with the Domain Discriminator on the shared representation

1 INTRODUCTION . The recent high performance of deep neural networks on image classification and object recognition depends greatly on whether one can obtain sufficiently labeled images of classes to predict . Nevertheless , it is difficult to do this in the real world because the number of existing classes is enormous . As long as human beings create or develop new objects , their number might continue to increase daily , thereby creating difficulty in obtaining labeled data of all classes to predict . In recent years , this difficulty led to great interest in zero-shot learning ( ZSL ) ( Farhadi et al. , 2009 ; Frome et al. , 2013 ; Lampert et al. , 2014 ; Xian et al. , 2018a ) , which is training by a labeled set from certain classes called seen classes and then predicting completely unseen classes that are not included in the training set . Usually , ZSL is accomplished by preparing pre-defined semantic representations of all classes , such as attributes , and learning the relation between images and the class-attributes ( relation-based approach ) . Once it is learned from the training set , we can predict the labels of test examples from unseen classes by selecting the most relevant attributes on this relationship . However , it has been pointed out that this approach does not work for generalized zero-shot learning ( GZSL ) , which is a more general setting where test samples can be from both seen and unseen classes ( Chao et al. , 2016 ) . This is because all examples of the training set are obtained from the seen classes , so the data to be predicted as one of the unseen classes also has a strong relationship with the seen class attributes , resulting most of them assigned to one of the seen classes . In this paper , we refer to a domain as belonging to either seen or unseen , and call the problem that class prediction is biased to the seen domain as the domain bias problem . To address this difficulty , recent works have taken an approach of learning a generative model that generates images from corresponding attributes , and of then training a classifier to predict classes from the generated synthesis images ( Mishra et al. , 2017 ; Verma & Rai , 2017 ; Xian et al. , 2018b ; Felix et al. , 2018 ) . The advantage of this synthesis-based approach is that samples of both domains are obtainable by generation , which contributes to alleviation of the domain bias problem . However , these synthesis-based methods require the generative model to generate numerous diverse and highquality images for each class , including unseen ones , sufficient to classify with high-performance , which can be difficult and costly . In addition , this requires a classifier that classifies all classes from the generated images . On the other hand , conventional relation-based methods can make class predictions using only the learned image–attribute relation , not requiring enormous image generation and additional classifier training after training the model . Therefore , we address the following question in this paper : Can we mitigate the domain bias problem of GZSL in the relation-based manner and achieve high performance ? We first discuss the importance of the following requirements for good relation-based ZSL performance when embedding images and attributes in a shared representation space that satisfies the following requirements : images and attributes belonging to the same class must be in the same place ( modality invariance ) ; and different classes of samples must be separated in the shared space ( class separability ) . To achieve such embedding , we propose Modality-invariant and Classseparable Multimodal AutoEncoder ( MCMAE ) , which is an extension of variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . The objective of MCMAE is designed based on the two requirements presented above . Next , we hypothesize that the domain bias problem results from a situation in which the unseen domain overlaps that of the seen one in the shared space . To address this point , we explicitly introduce a discriminator for separation of these two domains . This discriminator is trained jointly with MCMAE . After training , it gives the probability of a test image being in a given domain . In other words , it gives prior knowledge of the domain in the shared space . Based on this insight , we consider the class prediction probability as a soft combination of MCMAE classification and consider the domain discriminator as a mixture model ( see Figure 1 ) . Such combination-based classification has been proposed as a “ gating ” approach ( Atzmon & Chechik , 2019 ) . However , unlike this work , our method is able to train the entire model while retaining an end-to-end manner . We call this proposed approach as MCMAE with the Domain discriminator ( MCMAE-D ) . The contribution of this research is the following . ( 1 ) We consider the requirements of the shared representation space to perform in the ZSL setting and introduce MCMAE as a model to learn the embedding of images and attributes into that space . ( 2 ) We also propose MCMAE-D , which combines MCMAE with the domain discriminator . The experiment results demonstrate that it greatly reduces the domain bias problem , thereby contributing to exceed the performances of the existing relation-based models greatly and to be equivalent to the state-of-the-art synthesis-based method . 2 PROBLEM FORMULATION : GENERALIZED ZERO-SHOT LEARNING . We assume that the dataset Dtr = { xi , yi } Ntri=1 is given as the training set , where xi ∈ X is the input data , e.g . an image , and where yi ∈ Ys = { 1 ... , S } is the corresponding label data . The objective of ZSL is to learn the classifier using Dtr and to predict labels ŷj ∈ Y from the example xj ∈ X in the test set Dts = { xj , yj } Ntsj=1 . For the standard ZSL , it is assumed that the classes of the test set are completely unseen in the training set , which means that Y = Yu = { S+1 , ... , S+U } . Our goal is to train in the setting of GZSL , which includes both seen and unseen classes in the test set ( Y = Ys ∪ Yu ) . Furthermore , we assume that we have the class-attribute matrix A ∈ RM× ( S+U ) as the semantic information of classes , where each column represents the M -dimensional attribute vector ac ∈ A = RM of each class c = 1 , ... , S + U . Using this attribute vector , the training set Dtr can be replaced as { xi , ayi } Ntr i=1 . Using the relation-based approach , the objective is changed to train the compatibility function of the input and attribute F ( x , ay ) , which represents how the input and attribute are related : stronger relations have greater values . Once this function is learned from the training set , the classification probability can be expressed as p ( y = c|x ) = exp ( F ( x , ac ) ) ∑ ŷ∈Y exp ( F ( x , aŷ ) ) . ( 1 ) One can predict the class labels by choosing the one that maximizes this probability , i.e. , ŷ = argmaxy∈Y p ( y|x ) . Additionally , we call seen and unseen ones as different domains and express it as a binary variable d ∈ { 0 , 1 } , where d = 0 represents the seen domain and d = 1 represents the unseen one . 3 PROPOSED METHOD . 3.1 RELATION-BASED CLASSIFICATION OF THE SHARED REPRESENTATION . In this study , the image x and the attribute ay are regarded as different modalities . Also , the mapping qϕx ( z|x ) and qϕa ( z|ay ) embed them into the same space , i.e. , the shared representation space z . In this approach , the compatibility function can be expressed with the Kullback–Leibler ( KL ) divergence as Fϕx , a ( x , ay ) = −DKL ( qϕx ( z|x ) ||qϕa ( z|ay ) ) , ( 2 ) where ϕx , a is a shorthand notation for ϕx and ϕa . Therefore , learning the compatibility function corresponds to learning embeddings into the shared representation . Then , what kind of space embedding engenders good performance of ZSL and GZSL ? First , images and attributes belonging to the same class must be embedded in a nearby place in terms of KL divergence . It is clear that this embedding is a requirement for ZSL because , if not satisfied , the assumption on the relation-based methods is violated , rendering it impossible to predict corresponding classes from an input . Note that this requirement must also be generalized to unknown data , i.e. , examples of unseen classes . That is , the shared representation must be modality invariant ( see Figure 2 ( a ) ) . Second , examples embedded in the shared space by the mapping must be grouped by the same class and separated from other classes . If not satisfied ( e.g. , an area of one class overlaps with that of other classes ) , they may be misclassified as other classes . This can be rephrased that the embedded representation needs to be easily separated by some classifier , i.e. , the shared representation must have class separability ( Figure 2 ( a ) ) . Class separability between domains is particularly important for GZSL performance . Because only seen data are given during GZSL training , there is no clue to properly embed the unseen classes . Consequently , embedding of the unseen domain might overlap with those of the seen domain , and any input might be only to assigned to one of the more relevant seen classes ( Figure 2 ( b ) ) . We hypothesize that the failure of class separability between domains is the underlying cause of the domain bias problem . 3.2 MODALITY-INVARIANT AND CLASS-SEPARABLE MULTIMODAL AUTOENCODER . Given training data ( x , ay ) , the simplest way to satisfy modality invariant is to maximize Eq . 2 over these data directly , but it is difficult to generalize to test data . For this study , we first introduce a shared representation learning method based on variational autoencoders ( VAE ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . In VAE , the data x is assumed to be generated from a generative model pθx ( x ) = ∫ pθx ( x|z ) p ( z ) dz ( θx is a learnable parameter ) , and learning is performed by maximizing the lower bound of the log marginal likelihood over the given data : LV AEθx , ϕx ( x ) = Eqϕx ( z|x ) [ log pθx ( x|z ) ] −DKL ( qϕx ( z|x ) ||p ( z ) ) , ( 3 ) where qϕx ( z|x ) represents an approximate distribution of the true posterior pθx ( z|x ) and ϕx is a parameter . This approximate distribution , also called the inference model , can be regarded as an embedding from x into z . Although VAE learns to maximize Eq . 3 not under an exact generative distribution but under a finite training set , its representation is known to generalize well to unseen inputs . This capability is suitable for shared representation learning in ZSL , which needs to be generalized to the examples of the unseen classes . To extend Eq . 3 to multimodal input of images and attributes , we first replace the prior in the regularization term from p ( z ) to qϕa ( z|ay ) : LMAEθx , ϕx , a ( x , ay ) = Eqϕx ( z|x ) [ log pθx ( x|z ) ] −DKL ( qϕx ( z|x ) ||qϕa ( z|ay ) ) . ( 4 ) This makes regularization of the shared representation more relaxed because the replaced prior is learnable . In addition , maximizing this lower bound engenders learning to bring the two embeddings closer together explicitly . Next , we introduce a model pθa ( ay|z ) that discriminates attributes from the representation . By learning this model together with Eq . 4 , one can add a constraint to the representation from which attributes can be successfully predicted : LCMAEθx , a , ϕx , a ( x , ay ) = L MAE θx , ϕx , a ( x , ay ) + Eqϕx ( z|x ) [ log pθa ( ay|z ) ] . ( 5 ) Attributes are semantically distributed representations of classes , which engenders the classseparable representation . In addition , this equation can be interpreted as introducing an attribute generative model pθ ( ay|z ) . From such a perspective , this equation considers not only the reconstruction of an image , but also the “ cross ” reconstruction of the corresponding attribute from the image . This model is actually the same as the multimodal model known as PSE ( Jiao et al. , 2019 ) , as described in Sec 4.2 . Furthermore , as discussed in Sec . 3.1 , the shared representation must be modality-invariant . Therefore , we add an expected term at qϕa ( z|ay ) in Eq . 5 , which is a constraint by which the embedded representation is the same as long as it represents the same thing , irrespective of which modality inference model is used . Therefore , our objective is finally LMCMAEθx , a , ϕx , a ( x , ay ) = L CMAE θx , a , ϕx , a ( x , ay ) + Eqϕa ( z|ay ) [ log pθx , a ( x , ay|z ) ] = Eqϕx ( z|x ) [ log pθx , a ( x , ay|z ) ] + Eqϕa ( z|ay ) [ log pθx , a ( x , ay|z ) ] −DKL ( qϕx ( z|x ) ||qϕa ( z|ay ) ) , ( 6 ) where log pθx , a ( x , ay|z ) = log pθx ( x|z ) + log pθa ( ay|z ) . All distributions are parameterized by deep neural networks . Moreover , we regard that the inference models as Gaussian and generative models as a Laplace distribution with a constant scale parameter . Therefore , the log-likelihood of the generative models is obtained by the negative absolute-difference loss . This model considers all the requirements that the shared representation should satisfy . For this study , we call this Modality-invariant and Class-separable Multimodal AutoEncoder ( MCMAE ) .

Summary: The main topic of this paper is generalized zero-shot learning. This paper modifies traditional VAE method with attribute matching prior to release the hidden features from original regularization. This paper also proposes a domain discriminator to enhance class-separability of learned features to avoid unseen classes to be covered by seen classes. Experiment results show their efficiency under relation-based setting.