Relation-based Generalized Zero-shot Classification with the Domain Discriminator on the shared representation

1 INTRODUCTION . The recent high performance of deep neural networks on image classification and object recognition depends greatly on whether one can obtain sufficiently labeled images of classes to predict . Nevertheless , it is difficult to do this in the real world because the number of existing classes is enormous . As long as human beings create or develop new objects , their number might continue to increase daily , thereby creating difficulty in obtaining labeled data of all classes to predict . In recent years , this difficulty led to great interest in zero-shot learning ( ZSL ) ( Farhadi et al. , 2009 ; Frome et al. , 2013 ; Lampert et al. , 2014 ; Xian et al. , 2018a ) , which is training by a labeled set from certain classes called seen classes and then predicting completely unseen classes that are not included in the training set . Usually , ZSL is accomplished by preparing pre-defined semantic representations of all classes , such as attributes , and learning the relation between images and the class-attributes ( relation-based approach ) . Once it is learned from the training set , we can predict the labels of test examples from unseen classes by selecting the most relevant attributes on this relationship . However , it has been pointed out that this approach does not work for generalized zero-shot learning ( GZSL ) , which is a more general setting where test samples can be from both seen and unseen classes ( Chao et al. , 2016 ) . This is because all examples of the training set are obtained from the seen classes , so the data to be predicted as one of the unseen classes also has a strong relationship with the seen class attributes , resulting most of them assigned to one of the seen classes . In this paper , we refer to a domain as belonging to either seen or unseen , and call the problem that class prediction is biased to the seen domain as the domain bias problem . To address this difficulty , recent works have taken an approach of learning a generative model that generates images from corresponding attributes , and of then training a classifier to predict classes from the generated synthesis images ( Mishra et al. , 2017 ; Verma & Rai , 2017 ; Xian et al. , 2018b ; Felix et al. , 2018 ) . The advantage of this synthesis-based approach is that samples of both domains are obtainable by generation , which contributes to alleviation of the domain bias problem . However , these synthesis-based methods require the generative model to generate numerous diverse and highquality images for each class , including unseen ones , sufficient to classify with high-performance , which can be difficult and costly . In addition , this requires a classifier that classifies all classes from the generated images . On the other hand , conventional relation-based methods can make class predictions using only the learned image–attribute relation , not requiring enormous image generation and additional classifier training after training the model . Therefore , we address the following question in this paper : Can we mitigate the domain bias problem of GZSL in the relation-based manner and achieve high performance ? We first discuss the importance of the following requirements for good relation-based ZSL performance when embedding images and attributes in a shared representation space that satisfies the following requirements : images and attributes belonging to the same class must be in the same place ( modality invariance ) ; and different classes of samples must be separated in the shared space ( class separability ) . To achieve such embedding , we propose Modality-invariant and Classseparable Multimodal AutoEncoder ( MCMAE ) , which is an extension of variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . The objective of MCMAE is designed based on the two requirements presented above . Next , we hypothesize that the domain bias problem results from a situation in which the unseen domain overlaps that of the seen one in the shared space . To address this point , we explicitly introduce a discriminator for separation of these two domains . This discriminator is trained jointly with MCMAE . After training , it gives the probability of a test image being in a given domain . In other words , it gives prior knowledge of the domain in the shared space . Based on this insight , we consider the class prediction probability as a soft combination of MCMAE classification and consider the domain discriminator as a mixture model ( see Figure 1 ) . Such combination-based classification has been proposed as a “ gating ” approach ( Atzmon & Chechik , 2019 ) . However , unlike this work , our method is able to train the entire model while retaining an end-to-end manner . We call this proposed approach as MCMAE with the Domain discriminator ( MCMAE-D ) . The contribution of this research is the following . ( 1 ) We consider the requirements of the shared representation space to perform in the ZSL setting and introduce MCMAE as a model to learn the embedding of images and attributes into that space . ( 2 ) We also propose MCMAE-D , which combines MCMAE with the domain discriminator . The experiment results demonstrate that it greatly reduces the domain bias problem , thereby contributing to exceed the performances of the existing relation-based models greatly and to be equivalent to the state-of-the-art synthesis-based method . 2 PROBLEM FORMULATION : GENERALIZED ZERO-SHOT LEARNING . We assume that the dataset Dtr = { xi , yi } Ntri=1 is given as the training set , where xi ∈ X is the input data , e.g . an image , and where yi ∈ Ys = { 1 ... , S } is the corresponding label data . The objective of ZSL is to learn the classifier using Dtr and to predict labels ŷj ∈ Y from the example xj ∈ X in the test set Dts = { xj , yj } Ntsj=1 . For the standard ZSL , it is assumed that the classes of the test set are completely unseen in the training set , which means that Y = Yu = { S+1 , ... , S+U } . Our goal is to train in the setting of GZSL , which includes both seen and unseen classes in the test set ( Y = Ys ∪ Yu ) . Furthermore , we assume that we have the class-attribute matrix A ∈ RM× ( S+U ) as the semantic information of classes , where each column represents the M -dimensional attribute vector ac ∈ A = RM of each class c = 1 , ... , S + U . Using this attribute vector , the training set Dtr can be replaced as { xi , ayi } Ntr i=1 . Using the relation-based approach , the objective is changed to train the compatibility function of the input and attribute F ( x , ay ) , which represents how the input and attribute are related : stronger relations have greater values . Once this function is learned from the training set , the classification probability can be expressed as p ( y = c|x ) = exp ( F ( x , ac ) ) ∑ ŷ∈Y exp ( F ( x , aŷ ) ) . ( 1 ) One can predict the class labels by choosing the one that maximizes this probability , i.e. , ŷ = argmaxy∈Y p ( y|x ) . Additionally , we call seen and unseen ones as different domains and express it as a binary variable d ∈ { 0 , 1 } , where d = 0 represents the seen domain and d = 1 represents the unseen one . 3 PROPOSED METHOD . 3.1 RELATION-BASED CLASSIFICATION OF THE SHARED REPRESENTATION . In this study , the image x and the attribute ay are regarded as different modalities . Also , the mapping qϕx ( z|x ) and qϕa ( z|ay ) embed them into the same space , i.e. , the shared representation space z . In this approach , the compatibility function can be expressed with the Kullback–Leibler ( KL ) divergence as Fϕx , a ( x , ay ) = −DKL ( qϕx ( z|x ) ||qϕa ( z|ay ) ) , ( 2 ) where ϕx , a is a shorthand notation for ϕx and ϕa . Therefore , learning the compatibility function corresponds to learning embeddings into the shared representation . Then , what kind of space embedding engenders good performance of ZSL and GZSL ? First , images and attributes belonging to the same class must be embedded in a nearby place in terms of KL divergence . It is clear that this embedding is a requirement for ZSL because , if not satisfied , the assumption on the relation-based methods is violated , rendering it impossible to predict corresponding classes from an input . Note that this requirement must also be generalized to unknown data , i.e. , examples of unseen classes . That is , the shared representation must be modality invariant ( see Figure 2 ( a ) ) . Second , examples embedded in the shared space by the mapping must be grouped by the same class and separated from other classes . If not satisfied ( e.g. , an area of one class overlaps with that of other classes ) , they may be misclassified as other classes . This can be rephrased that the embedded representation needs to be easily separated by some classifier , i.e. , the shared representation must have class separability ( Figure 2 ( a ) ) . Class separability between domains is particularly important for GZSL performance . Because only seen data are given during GZSL training , there is no clue to properly embed the unseen classes . Consequently , embedding of the unseen domain might overlap with those of the seen domain , and any input might be only to assigned to one of the more relevant seen classes ( Figure 2 ( b ) ) . We hypothesize that the failure of class separability between domains is the underlying cause of the domain bias problem . 3.2 MODALITY-INVARIANT AND CLASS-SEPARABLE MULTIMODAL AUTOENCODER . Given training data ( x , ay ) , the simplest way to satisfy modality invariant is to maximize Eq . 2 over these data directly , but it is difficult to generalize to test data . For this study , we first introduce a shared representation learning method based on variational autoencoders ( VAE ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . In VAE , the data x is assumed to be generated from a generative model pθx ( x ) = ∫ pθx ( x|z ) p ( z ) dz ( θx is a learnable parameter ) , and learning is performed by maximizing the lower bound of the log marginal likelihood over the given data : LV AEθx , ϕx ( x ) = Eqϕx ( z|x ) [ log pθx ( x|z ) ] −DKL ( qϕx ( z|x ) ||p ( z ) ) , ( 3 ) where qϕx ( z|x ) represents an approximate distribution of the true posterior pθx ( z|x ) and ϕx is a parameter . This approximate distribution , also called the inference model , can be regarded as an embedding from x into z . Although VAE learns to maximize Eq . 3 not under an exact generative distribution but under a finite training set , its representation is known to generalize well to unseen inputs . This capability is suitable for shared representation learning in ZSL , which needs to be generalized to the examples of the unseen classes . To extend Eq . 3 to multimodal input of images and attributes , we first replace the prior in the regularization term from p ( z ) to qϕa ( z|ay ) : LMAEθx , ϕx , a ( x , ay ) = Eqϕx ( z|x ) [ log pθx ( x|z ) ] −DKL ( qϕx ( z|x ) ||qϕa ( z|ay ) ) . ( 4 ) This makes regularization of the shared representation more relaxed because the replaced prior is learnable . In addition , maximizing this lower bound engenders learning to bring the two embeddings closer together explicitly . Next , we introduce a model pθa ( ay|z ) that discriminates attributes from the representation . By learning this model together with Eq . 4 , one can add a constraint to the representation from which attributes can be successfully predicted : LCMAEθx , a , ϕx , a ( x , ay ) = L MAE θx , ϕx , a ( x , ay ) + Eqϕx ( z|x ) [ log pθa ( ay|z ) ] . ( 5 ) Attributes are semantically distributed representations of classes , which engenders the classseparable representation . In addition , this equation can be interpreted as introducing an attribute generative model pθ ( ay|z ) . From such a perspective , this equation considers not only the reconstruction of an image , but also the “ cross ” reconstruction of the corresponding attribute from the image . This model is actually the same as the multimodal model known as PSE ( Jiao et al. , 2019 ) , as described in Sec 4.2 . Furthermore , as discussed in Sec . 3.1 , the shared representation must be modality-invariant . Therefore , we add an expected term at qϕa ( z|ay ) in Eq . 5 , which is a constraint by which the embedded representation is the same as long as it represents the same thing , irrespective of which modality inference model is used . Therefore , our objective is finally LMCMAEθx , a , ϕx , a ( x , ay ) = L CMAE θx , a , ϕx , a ( x , ay ) + Eqϕa ( z|ay ) [ log pθx , a ( x , ay|z ) ] = Eqϕx ( z|x ) [ log pθx , a ( x , ay|z ) ] + Eqϕa ( z|ay ) [ log pθx , a ( x , ay|z ) ] −DKL ( qϕx ( z|x ) ||qϕa ( z|ay ) ) , ( 6 ) where log pθx , a ( x , ay|z ) = log pθx ( x|z ) + log pθa ( ay|z ) . All distributions are parameterized by deep neural networks . Moreover , we regard that the inference models as Gaussian and generative models as a Laplace distribution with a constant scale parameter . Therefore , the log-likelihood of the generative models is obtained by the negative absolute-difference loss . This model considers all the requirements that the shared representation should satisfy . For this study , we call this Modality-invariant and Class-separable Multimodal AutoEncoder ( MCMAE ) .

Summary: This paper proposes a relation-based ZSL model which can effectively alleviate the domain bias problem. To this end, first, the paper claims that a good relation-based ZSL model should consider two requirements -- modality invariance and class separability. And the paper designed Modality-invariant and Class-separable Multimodal VAE (MCMVAE) based on VAEs to meet the two aforementioned requirements. Next, the paper hypothesizes that the domain bias problem is due to the overlap between seen and unseen classes in the shared space, and explicitly introduced a discriminator to separate the two domains. The paper performs experiments on ZSL benchmark datasets and shows that the proposed method outperforms other relation-based methods. Besides, the domain discriminator which can be applied to other models demonstrates its effectiveness in reducing domain bias given the experimental results.