Relation-based Generalized Zero-shot Classification with the Domain Discriminator on the shared representation
Authors: Masahiro Suzuki
Yutaka Matsuo
Venue:  (2020)

Abstract: Generalized zero-shot learning (GZSL) is the task of predicting a test image from seen or unseen classes using pre-defined class-attributes and images from the seen classes.  Typical ZSL models assign the class corresponding to the most relevant attribute as the predicted label of the test image based on the learned relation between the attribute and the image. However, this relation-based approach presents a difficulty: many of the test images are predicted as biased to the seen domain, i.e., the \emph{domain bias problem}. Recently, many methods have addressed this difficulty using a synthesis-based approach that, however, requires generation of large amounts of high-quality unseen images after training and the additional training of classifier given them.  Therefore, for this study, we aim at alleviating this difficulty in the manner of the relation-based approach. First, we consider the requirements for good performance in a ZSL setting and introduce a new model based on a variational autoencoder that learns to embed attributes and images into the shared representation space which satisfies those requirements. Next, we assume that the domain bias problem in GZSL derives from a situation in which embedding of the unseen domain overlaps that of the seen one. We introduce a discriminator that distinguishes domains in a shared space and learns jointly with the above embedding model to prevent this situation.  After training, we can obtain prior knowledge from the discriminator of which domain is more likely to be embedded anywhere in the shared space. We propose combination of this knowledge and the relation-based classification on the embedded shared space as a mixture model to compensate class prediction. Experimentally obtained results confirm that the proposed method significantly improves the domain bias problem in relation-based settings and achieves almost equal accuracy to that of high-cost synthesis-based methods.


Under review as a conference paper at ICLR 2020
RELATION-BASED GENERALIZED ZERO-SHOT CLASSIFICATION WITH THE DOMAIN DISCRIMINATOR ON THE SHARED REPRESENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Generalized zero-shot learning (GZSL) the task of predicting a test image from seen or unseen classes using pre-defined class-attribuees and images from the seen classes. Typical ZSL models assign the class corresponding to the most relevant attribute as the predicted label of the test image based the learned relation be- tween the attribute and the image. However, this relation- based approach presents difficulty: many of the test images are predicted as biased the seen domain, i.e., the domain bias problem. Recently, many methods have addressed this diffi- culty using a synthesis-basee amounts of high-qualite unseen images after training and the additional training of classifier given them. Therefore, for this study, we aim alleviating this difficulty in the manner of the Ilaation-baed approach. First, we consider the requirements for good performance in ZSL. setting and introduce a new model based on a variational autoencoder that learns to embed attributes and images into the shared representation space which satisfies those requirements. Next, we assume that the domain bias problem in GZSL derives from situation in which embedding of the unseen domain overlaps that of the seen one. We introduce a discriminator that distinguishes domains in shared space and learns jointly with the above embed- ding model to prevent this situation. After training, we can obtain prior knowledge from the discriminator of which domain is more likely be embedded anywhere in the shared space. We propose combination of this knowledge and the relation- based classification on the embedded shared space mixture model compen- sate class prediction. Experimentally obtained results confirm that the proposed method significantly improves the domain bias problem in relation- based settings and achieves almost equal accuracy to that of high-cost synthesis-based methods.
approach that, however, requires generation of large
INTRODUCTION
The recent high performance of deep neural networks image classification and object recogni- tion depends greatly on whether one can obtain sufficiently labeled images of classes to predict. Nevertheless, it is difficult do this in the real world because the number of existing classes is enormous. As long as human beings create or develop new objects, their number might continue increase daily, thereby creating difficulty in obtaining labeled data of all classes to predict. In recent years, this difficulty led to great interest in zero-shot learning (ZSL) (Farhadi et al., 2009; Frome et al., 2013; Lampert al., 2014; Xian al., 2018a), which training by a labeled set from certain classes called seen classes and then predicting completely unseen classes that are not in- cluded in the training set.
Usually, ZSL is accomplished by preparing pre-defined semantic representations of all classes, such as attributes, and learning the relation between images and the class-attributes (relation-basedd ap- proach). Once it is learned from the training set, we can predict the labels of test examples from unseen classes by selecting the most relevant attributes on this relationship. However, it has been pointed out that this approach does not work for generalized zero-shot learning (GZSL), which is a more general setting where test samples can be from both seen and unseen classes (Chao et al., 2016). This is because all examples of the training set obtained from the seen classes, so the

---

Under review as a conference paper at ICLR 2020
Shared representation space
leopard white 0.06 blue 0
horse Back 0.14
Seen
white 0.13
Unseen classes
Clussification prohubility
attributes
p(clasejimagg)
attributes
pt imugee, soca] 3 unseeu imuge))
Domain
(luneiinagy worn
+p imoppaded unseen)pt umsernfimage))
Discriminatoe
peoopardjuzggg. unsons)
test image
Figure I: An overview of MCMAE-D. Images and attributes are embedded in the shared represen- lation space by MCMAE inference models learned given the training set. We propose a model to distinguish different domains (seen or unseen) in space and to learn jointly with MCMAE. After learning, we can perform class prediction with reduced bias toward the seen classes by combining relation-based classifier and domain discriminator as a mixture model, as shown in the equation on the right side.
data to be predicted one of the unseen classes also has a strong relationship with the seen class attributes, resulting most of them assigned to one of the seen classes. In this paper, we refer to a domain as belonging either seen unseen, and call the problem that class prediction is biased to the seen domain as the domain bias problem.
To address this difficulty, recent works have taken an approach of learning a generative model that generates images from corresponding attributes, and of then training a classifier to predict classes from the generated synthesis images (Mishra al., 2017: Verma & Rai. 2017: Xian al., 2018b; Felix et al., 2018). The advantage of this synthesis- approach is that samples of both domains are obtainable by generation, which contributes to alleviation of the domain bias problem. However, these synthesis-based methods require the generative model to generate numerous diverse and high- quality images for each class, including unseen ones, sufficient to classify with high-performance, which can be difficult and costly. In addition, this requires classifier that classifies all classes from the generated images. On the other hand, conventional relation- based methods can make class pre- dictions using only the learned image- antt-bbt relation, not requiring enormous image generation and additional classifier training after training the model. Therefore, we address the following ques- tion in this paper: Can mitigate the domain bias problem of GZSL. in the relation-based manner and achieve high performance?
We first discuss the importance of the following requirements for good relation-based ZSL per- formance when embedding images and attributes in a shared representation space that satisfies the following requirements: images and attributes belonging to the same class must be in the same place (modality invariance); and different classes of samples must be separated in the shared space (class separability). To achieve such embedding, we propose -odality-invrrian invariant and Class- separable Multimodal AutoEncoder (MCMAE), which is an extension of variational autoencoders (VAEs) (Kingma& & Welling, 2013; Rezende et al., 2014). The objective of MCMAE is designed based on the two requirements presented above.
Next, we hypothesize that the domain bias problem results from a situation in which the unseen domain overlaps that of the seen one in the shared space. To address this point, explicitly intro- duce a discriminator for separation of these two domains. This discriminator is trained jointly with MCMAE. After training, it gives the probability of test image being in given domain. In other words, it gives prior knowledge of the domain in the shared space. Based on this insight, we con- sider the class prediction probability as a soft combination of MCMAE classification and consider the domain discriminator as a mixture model (see Figure 1). Such combination based classification "gating" approach (Atzmon & Chechik, 2019). However, unlike this work,
has been proposed our method is able to train the entire model while retaining an end-to-end manner. We call this proposed approach as MCMAE with the Domain discriminator (MCMAE-D).
The contribution of this research is the following. (1) We consider the requirements of the shared representation space to perform in the ZSL setting and introduce MCMAE as a model to learn the embedding of images and attributes into that space. (2) We also propose MCMAE-D, which com- bines MCMAE with the domain discriminator. The experiment results demonstrate that it greatly reduces the domain bias problem, thereby contributing to exceed the performances of the existing relation-bast based models greatly and to be equivalent to the state-of-the-art synthesis-based method.
2

---

Under review as a conference paper at ICLR 2020
Medalty trostiet Cluo squzaiiltyy
9a (=/22)
(a(a) a
Unseen classes
Ps(d(z)
-
y
5/ (2)aa)
pa(a, (s)
dy
hocse
hocss Medk
685
1813 AUE
MCMAE-D MCMAE Domain Discriminator
(3)
(b)
Figure 2: (a) Two requirements exist for achieving good performance in ZSL: modality invariant and class separability. (b) Failure of class separation between domains. The unseen domain overlaps with the seen domain. Therefore, all examples of the unseen classes might be predicted of the seen ones. (c) The network architecture of our proposed model, MCMAE-D.
2 PROBLEM FORMULATION: GENERALIZED ZERO-SHOT LEARNING
Ner is given as the training set, where æ; E X' is the input We assume that the dataset Drr fa;, V/ Vi [= data, e.g. an image, and where y/ E V = [I...,5 S} is the corresponding label data.
The objective of 7SL is to learn the classifier using Der and to predict labels új E ) from the example 2 X' in the test set Dts {æ,, yソ Nov. For the standard ZSL, it is assumed that the classes of the test set are completely unseen in the training set, which means that y = V = (S+1,..., S + U}. Our goal is train in the setting of GZSL, which includes both and unseen classes in the test set () yd U.W)
Furthermore, assume that have the class-attribute matrix A E Mx(+U) the semantic information of classes, where each column represents the M M-dimensional attribute vector a, E A = RM of each class I,... S + U. Using this attribute vector, the training set Der can be replaced as {Iz, a. 1im1.
Using the relation-based approach, the objective is changed to train the compatibility function of the input and attribute Fxx ay). which represents how the input and attribute are related: stronger relations have greater values. Once this function is learned from the training set, the classification probability can be expressed as
exp (F(x, ac))
ply = c/z) Evey exp (F(x,a)))
(1)
One can predict the class labels by choosing the one that maximizes this probability, i.e., f = arg maxyey p(y az).
Additionally, we call seen and unseen ones as different domains and express it as a binary variable d E {0,1), 1}, where d = represents the seen domain and represents the unseen one.
3 PROPOSED METHOD
3.1 RELATION-BASEE CLASSIFICATION OF THE SHARED REPRESENTATION
In this study, the image and the attribute a regarded as different modalities. Also, the mapping lo. (2/ar) and do (z|a,, embed them into the same space, i.e., the shared representation space
2. In this approach, the compatibility function can be expressed with the Kullback-Leillrr (KL) divergence as
Fo.. (x, az) a.) - DKL (90= (x|x)] (zla,)),
(2)
where &z is shorthand notation for ob and &a. Therefore, learning the compatibility function corresponds to learning embeddings into the shared representation. Then, what kind of space em- bedding engenders good performance of 7SL and GZSL.?

---

Under review as a conference paper at ICLR 2020
First, images and attributes belonging the same class must be embedded in nearby place in terms of KL divergence. It is clear that this embedding is a requirement for ZSL because, if not satisfied, the assumption the relation- based methods is violated, rendering it impossible to predict corre- sponding classes from an input. Note that this requirement must also be generalized to unknown data, i.e., examples of unseen classes. That is, the shared representation must be modality invariant (see Figure 2(a)).
Second, examples embedded the shared space by the mapping must be grouped by the same class and separated from other classes. If not satisfied (e.g., an area of one class overlaps with that of other classes), they may be misclassified as other classes. This can be rephrased that the embedded representation needs to be easily separated by some classifier, i.e., the shared representation must have class separability (Figure 2(a)).
Class separability between domains is particularly important for GZSL performance. Because only seen data are given during GZSL training. there no clue to properly embed the unseen classes. Consequently, embedding of the unseen domain might overlap with those of the seen domain, and any input might be only to assigned to one of the more relevant seen classes (Figure 2(b)). hypothesize that the failure of class separability between domains is the underlying cause of the domain bias problem.
3.2 MODALITY-NVVRIINN AND CLASS-SEPARABLE MULTIMODAL AUTOENCODER
Given training data (at, ay), the simplest way to satisfy modality invariant to maximize Eq. 2 over these data directly, but it is difficult to generalize to test data. For this study, we first introduce a shared representation learning method based on variational autoencoders (VAE) (Kingma & Welling, 2013; Rezende et al., 2014).
In VAE, the data æ is assumed to be generated from a generative model Po_ (œ) = - Po. (=)=))(2))) (02 is learnable parameter), and learning is performed by maximizing the lower bound of the log marginal likelihood over the given data:
V AE (x) Ess. (=|=) (log Po., (ar]²) DEL (gp, (2)6. (z(x))lp(()) R
where go. (z(zr represents an approximate distribution of the true posterior pe, (z|æx) and de is parameter. This approximate distribution, also called the inference model, can be regarded as an embedding from x into
Although VAE learns to maximize Eq. 3 not under an exact generative distribution but under finite training set, its representation is known to generalize well unseen inputs. This capability is suitable for shared representation learning in 7SL., which needs to be generalized to the examples of the unseen classes.
To extend Eq. 3 to multimodal input of images and attributes, we first replace the prior in the regu- larization term from p(z) to Qo.. a.)):
CMAE (x, ay) = Eas. (=|m flo Po., (m/2²) DEL (2|x (zla,))
(4)
This makes regularization of the shared representation more relaxed because the replaced prior is learnable. In addition, maximizing this lower bound engenders learning to bring the two embeddings closer together explicitly.
Next, we introduce a model pa. (a.1æ lz) that discriminates attributes from the representation. By learning this model together with Eq. 4, one can add constraint to the representation from which attributes can be successfully predicted:
L (I. a.y L (z, a.y) + Eger (z/m) [log Pe., (a,l2)).
Attributes are semantically distributed representations separable representation. In addition, this equation can be interpreted as introducing an attribute generative model (a,.l)). From such a perspective, this equation considers not only the recon- struction of an image, but also the "cross" reconstruction of the corresponding attribute from the image. This model is actually the same as the multimodal model known as PSE (Jiao et al., 2019), as described in Sec 4.2.
classes, which engenders the class-
4

---

Under review as a conference paper at ICLR 2020
Furthermore, discussed in Sec. 3.1, the shared representation must be modality inveriit......iant.... There- fore, we add an expected term at go. (2ay) in Eq. 5, which is a constraint by which the embedded representation is the same as long it represents the same thing, inrespective of which modality inference model is used. Therefore, our objective is finally
LCMAR CMCMAR AE (xx, ay) Lz..ii (x,ay) + Ege. (2lay) [log po... .. (x, [2]
= Fug, (z)(ee (x/e)) Po (2e, ay|=)] + Ras, (=la,) (log po... (xc, aylz)]
-DI (2)ar)llus, (zlay)):
(6)
where log Poz.. (ar, ay 2) log (ar 1)) log po. (a,) All distributions are parameterized by deep neural networks. Moreover, we regard that the inference models as Gaussian and generative models as a Laplace distribution with a constant scale parameter. Therefore, the log- ligliiklihood of the generative models is obtained by the negative absolute-diffcrence loss.
This model considers all the requirements that the shared representation should satisfy. For this study, we call this Modality- invariant and Class-separute Multimodal AutoEncoder (MCMAE).
3.3 DOMAIN DISCRIMINATOR
MCMAE includes domain invariance and class separability. However, class separability between domains is not fully considered because no knowledge related to the unseen domain is given during training. meaning that the domain bias problem cannot be avoided. Therefore, we propose addition of another approach to address this problem.
We introduce model p=(d]=) that discriminates a domain from the shared representation. If this discriminator can be trained together with MCMAE, then the representation is separable between domains, leading to alleviation of the domain bias problem.
For training pe pe(d2z (d]=), we create dataset with all class-attributes as inputs and corresponding domain variables as labels Dir. {(a ds)) Since the input of this discriminator is x, we should sample it from attributes a, using the inference model. Here, we use a reconstructed representation not only via the inference model of attributes but also via the generative and inference model of images. Therefore, the objective of the domain discriminator given (a,, d) is
(7) We will explain the reason for using the reconstructed representation and the details of Eq. 7 in Appendix B. When testing, the domain discriminator must be able to correctly discriminate the domain given an image. It is noteworthy that, for this to work properly, different modalities need to be embedded in the same place the shared space by each inference model.
Co (ay, d) -zr~ (=l=) (=)=),===q__ (=)a) [log pa (d]=)))
By training the objective of the domain discriminator together with that of MCMAE, the embedded shared representation prompted be separated by domain. The resulting objective, given D,, and Dtr.. becomes
Σ DEr (E..,))DEL
L (æ, ay) +
Σ
C. (aa.
[Deral
(ay.d)EDire
where a is a hyper-parameter representing the degree to which domain separation is enforced in the representation. We learn the model end-to-end by maximizing Eq.8 in terms of all parameters.
After training, this discriminator can predict the possibility of assigning a domain from any input z. In addition, if modality invariant embedding is possible, then one can predict the domain for unseen images and can thereby obtain "prior knowledge of the domain" from this classifier. We propose combination of this with an attribute-based classifier (Eq. to compensate the class prediction as mixture model:
p(y(x) ply|ar, d 0)p(d= 0|z) plyle, d 1)p(d 11x),
where ply x d = 0) Σee>, exp (F p(ylæ,
exp (F
exp (F(m.a,))) Eeer exp (F().
and p(d
1 [x) = exp(EE-eat)) (=]=) [log Ps 1|2)])
We expect that the domain bias problem can be alleviated by selecting class that maximizes this probability as the prediction result. We call this approach as MCMAE with the Domain discriminator (MCMAF-D). Figure 2(c) shows the overall architecture of MCMAE-D.
5

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
4.1 ZERO-SHOT LEARNING/ / GENERALIZED ZERO-SHOT LEARNING
The main approach for tackling ZSL is to learn a compatibility function that represents the relation between images and class-attributes from the seen data'. Some early studies used classifiers such as SVM to learn a mapping from images to class-attributes (Lampert et al., 2009; 2014), but many other methods use linear embedding into attributes (or semantic representations) and typically set a ranking loss as the objective function to train such embedding (Frome et al., 2013; Akata et al., 2015; Romera-Paredes & Torr, 2015; Kodirov al., 2017). Romera-Paredes & Torr (2015) add the regularization term of the mapping to this objective function, and Kodirov et al. (2017) min- imizes the embedding eтror in both directions with the objective. The nonlinear model is also used (Socher et al., 2013) to take non-linearity between images and attributes into consideration. However, with this mapping direction, a hubness problem occurs in which all images are assigned to one type of attributes. To avoid this, some methods measure the similarity in the image space by obtaining the reverse mapping of attributes to images (Zhang et al., 2017; Verma & Rai, 2017).
Another approach is to map images and attributes to another shared representation space and mea- sure the similarity in that space (Zhang & Saligrama, 2015; 2016), which adopted in our paper. The advantage of this is that it can obtain a representation that does not depend on the dimension or representation of the input. Moreover, arbitrary representation can be learned by adding constraints during training. Recently, several methods use VAEs to learn this, which we will discuss in Sec. 4.2.
These relation-based methods have the advantage of being able to perform classification just by learning the compatibility function, but they do not work well in GZSL due the domain bias prob- lem (Chao et al., 2016). One way to alleviate this problem is to calibrate the compatibility func- tion (Chao al., 2016; Liu et al., 2018). In recent years, synthesis-based methods" have become mainstream because of their significant improvement in GZSL performance (Mishra et al., 2017: Verma et al., 2018; Xian al., 2018b; Felix et al., 2018). In particular, Zhang & Koniusz (2018a) propose model selection mechanism that improves synthesis-based performance by distinguish- ing seen and unseen domains from the generated images. However, these synthesis-bast methods require costly image generation and additional classifier learning. In this study, we aim to achieve high performance with the end-to-end relation- based method.
4.2 MULTIMODAL DEEP GENERATIVE MODELS FOR SHARED REPRESENTATION LEARNING
It has been discussed that the importance of learning appropriate representation in ZSL. Jiang et al. (2017) propose to learn an attribute representation that discriminative between classes while main- taining their semantic information. In this study, we discuss how to learn a good shared representa- tion that integrates different modalities of images and attributes in ZSL.
In recent years, several methods have been proposed to learn the shared representation with VAEs. JMVAE (Suzuki et al., 2016) and TrELBO (Vedantam et al., 2017) are aimed at obtaining modality-invarian representation by learning the inference model of each modality close to that of taking two modality inputs. However, because an additional inference model is required, it is expensive in terms of the number of parameters. In addition, regularization that forces the repre- sentation to be close to the standard Gaussian prior makes it difficult to obtain a class-separable representation (Jiao et al., 2019).
PSE (Jiao et al., 2019) and CADA-VAE (Schonfeld et al., 2019) are very similar to MCMAE in terms of shared representation learning. PSE not for 7SL but is equivalent to Eq. 5 by considering label information as attributes a 3. Their mutual difference is therefore whether they contain the reconstruction term given attributes, imposing a modality invariant representation. CADA-VAF,
I 'Athhough Xian et al. (2018a) categorizes these methods into four groups, we call all of them relation- based approaches, because all methods learn the relation between images and classes (or attributes) and use it directly to evaluate test data.
IIn this study, we refer to a method that requires training a classifier given samples generated from learned model as synthesis-based.
In Jiao al. (2019), this model called PSE*, and the model corresponding Eq. is called PSE.

---

Under review as a conference paper at ICLR 2020
synthesis-basee model the shared representation, has objective that consists of three parts*: VAE losses for each modality, cross-alignment losses that reconstruct one modality from the other, and distribution-alignmeet loss that brings the inference models of each modality closer. The main difference between MCMAF is that the inference model is forced to stay close to the prior. Similarly to JMVAE and TrELBO, this might be too restrictive obtain good shared representation.
EXPERIMENTS
5.1 DATASET AND SETTING
For experimentation, we use the following four datasets, which are commonly used for ZSL: Animals with Attributes (AWA) (Lampert et al., 2014), CUB-200-2011 Bird (CUB) (Wah al.. 2011), SUN Attribute (SUN) (Patterson & Hays, 2012), and Attribute Pascal and Yahoo (aPY) (Farhadi et al., 2009). Each dataset includes images each class, where each class is rep- resented by semantic attributes. For fair comparison, we the 101-layered ResNet (He et al., 2016) provided by Xian et al. (2018a) as the image vector. Fur- thermore, for the class-attribute representation, we use the attributes valued continuously between and I provided with each dataset. We followed the split proposed in Xian et al. (2018a) for splitting each dataset into train, validation, and test. The hyper-parameter selection of our model was based on this crain-valdattinn split. In training of GZSL, we used both as training data.
2048-dim top-layer embedding of
As the metric of evaluation, we use the average per-class top-1 accuracy on both the seen and unseen classes (referred as acc, and acc,). In addition, to evaluate the performance on GZSL, we calculate the harmonic mean of acc, and acca, which is accи = (2 acc, acc.)/ (acr, + acc.).
The architecture of each deep probabilistic model is listed in Appendix A. We set the dimension of the latent variable 512 and a = 0.01. We used the Adam optimization algorithm (Kingma & Ba, 2014) with learning rate of 10-3, In all experiments, we trained for 100 epochs². All models in this paper were implemented using PyTorch (Paszke et al., 2017).
5.2 ANALYSIS OF OUR PROPOSED METHOD
This section presents analyses of the proposed method from various perspectives. We use AWA for this analysis because the number of data per class is greater than that for other datasets.
First, we confirm the effectiveness of the proposed method by comparing it with a similar shared representation learning model: PSE and CADA-VAE. To align the conditions, we used the same network structure for the inference and generative models of each modality throughout all methods. The original CADA-VAB paper takes synthesis- bbased approach in which samples are generated from the shared space and use for learning a classifier. Here, to compare the effectiveness of shared representation learning, we evaluate CADA-VAE with a relation- based approach similar to MC- MAE. In all methods, the classification probability was obtained by Eq. 1.
Table presents results in GZSL, First, compared with PSE and MCMAE, acc, is higher in the MCMAF, but acc, is higher in the PSE. One might infer that this result means that MCMAE ob- tains shared representation that is not generalized. However, when learning these models and domain classifiers simultancously and when classifying them with the mixture prediction model (shown as PSE-D and MCMAE-D), MCMAE-D has markedly higher performance. Furthermore, the performance of the domain discriminator by AUROC is the highest in MCMAE-D. As described in Sec. 3.3, this domain discriminator approach does not work properly unless the modality invari- ance is generalized. This suggests that MCMAE can obtain modality-invariant representation. On the other hand, because PSE does not consider modality invariance, the domain discriminator does not generalize well the image modality. In addition, the result that acc, is greatly improved by MCMAE-D implies that MCMAE can acquire the class separability representation but its perfor- mance is degraded due to the domain bias problem.
*See Appendix for details of TADA-VAF.
"This was determined based on performance when learning with the training-test split in AWA. For detail of learning progress in our method, see Appendix D.
7

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison with shared representation learning models in GZSL setting. We include the results of combining each model with domain discriminator (denoted as X-D). AUROC shows the evaluation of domain prediction by the domain discriminator using the area under the curve of the receiver operating characteristic.
Models
AUROC
PSE
86.4
49.6
VAE (relatioo-based)
21.4
73.9
33.1
MCMAF
25.3
39.3
PSE-D
36.6
58.4
45.0
0.78
CADA- VAE-D (relation-hased)
50.6
43.6
46.8
0.77
MCMAE-D
60.4
67.9
63.9
0.89
PSF
MCMAE-D
antelope (seen)
horse (seen)
german-shepperr Iseen)
blue- whueewhalee (seen)
mole (seen)
humpbackewhi whale (sean) bot
pig lunsean)
(unseon)
collie (unseen)
Figure 3: 2-D representation of PSE, MCMAE, and MCMAE-D. These were obtained by learning by setting the z dimension of the model to 2. Circle plots show embedding of the test images by Po. [)) Triangles represent embedding of the class-attributes A denotes seen class and v is an unseen one. The contour lines in MCMAE-D represent the domain prediction probability p(d 1|2) obtained by the domain discriminator.
To confirm this consideration qualitatively, we visualize the shared representation of PSE, MCMAE, and MCMAE-D in two dimensions. Figure 3 portrays the visualization results. In PSE, the test images of the seen classes are well embedded around the corresponding attribute. However, for the unseen class, the data of some classes such as "collie", are embedded in location that differs from the corresponding attribute, which indicates that the modality invariance is generalized insufficiently. By contrast, in MCMAE, i... we can obtain a generalized modality-invariin representation. However, it is apparent that some class attributes overlap, which can cause the domain bias problem. Then, moving to the MCMAE- D representation, we confirm that embedding of seen and unseen is separated slightly because of learn embedding with the domain discriminator jointly. Furthermore, the domain classifier has prior knowledge to predict which domain is the location before the test images are given. Then we confirm that the test images are actually embedded almost in accordance with it. This demonstrates that the domain discriminator has the ability to alleviate the domain bias problem.
can confirm that the unseen images are embedded almost appropriately,
Let us go back to Table and see the CADA-VAE results. Schonfeld et al. (2019) reports that this model performs well with a synthesis-based approach in latent space. However, this result demonstrates that it is worse than PSE and MCMAE in the relation-based case, This is probably true because the representation is over-conttrained by the regularization term of the inference model included in CADA- VAE.
MDMACO
65
40-
68
32
64
22.
25.6
$12
1024
6.03
23
The dimeniionnnooommm usann ablc
(A)
Figure 4: Transition of GZSL performance (the harmonic mean) when each parameter is changed: (a) the dimension of the shared representation and (b) the coefficient of domain discriminator a in MCMAE-D.
8

---

Under review as a conference paper at ICLR 2020
Table 2: Comparison with GZSL state- of-the--rt models. Bold typeface denotes the best perfor- mance among the relation-based models.
CUB
AWA
SUN
aPY
based
GFZSI. (Verma & Rai, 2017)
45.7
0.0
80.3
0.0
39.6
0.0
0.0
83.3
0.0
CONSE (Seegedy al. 2015)
1,6
72.2
3.1
0.4
88.6
0.8
6.8
39.9
11.6
0.0
91.2
0.0
(Socher
2013)
19.5
SYNC (Chingpinya et al., 2016)
11.5
70.9
19.8
8.9
87.3
16.2
7.9
43.3
13.4
7.4
66.3
13.3
ESZSL (Romera Paredes PRoeneru-Puredes& & Torr, 2015)
12.6
63.8
21.0
6.6
75.6
12.1
11.0
27.9
15.8
2.4
70.1
4.6
SJE [Akain 2015)
23.5
59.2
33.6
74.6
19.6
14.7
30.5
3.7
DEVISE (Frome et al., 2013)
23.8
53.0
32.8
13.4
68.7
22.4
16.9
27.4
20.9
4.9
76.9
9.2
ZSKL (Zhang Koniusz, 20186)
21.6
52.8
30.6
17.9
82.2
29.4
20.1
31.4
24.5
10.5
76.2
18.5
MCMAE
30.9
64.9
41.9
25.3
88.4
39.3
21.1
39.8
27.6
5.2
87.9
9.8
MCMAE-D
38.3
43.7
60..4
67.9
47.1
28.8
35.8
20.8
52.7
29.8
Syniheiis-ased
CVAE ASA (Mishru et al., 2017)
34,5
47.2
26.7
SP.GZSI (Verma et al., 2018)
41.5
63.3
46.7
$6.3
67.8
61.5
40.9
30.5
345,9
K-CLSWGAN
20186)
Cycle-UU)WGANN (Felix al., 2018)
47.9
59.3
53.0
63.4
59.8
47.2
33.8
39.4
CADA-VAE (Schonfeld et al., 2019)
51.6
53.5
52.4
57,3
72.8
64.1
47.2
35.7
40.6
ModelSel-JWay (Zhang & Koniusz, 201
52.6
76.7
62.4
28.4
75.5
41.2
Next, analyze the parameter sensitivity of our proposed models: The dimension of the shared representation (in both MCMAE and MCMAE-D) and the coefficient domain discriminator a (in MCMAE-D). Figure 4 presents the results. First, from Figure 4(a), mance decreases slightly as the dimension of latent variables increases. This seems to be because in large dimension, embedding of the unseen domain becomes more difficult. On the other hand, we find that MCMAF-D is robust to the dimension of latent variables. This result shows that the domain classification probability contributes to the compensation of the performance significantly. Next, from Figure 4(b), we find that increasing the value of the coefficient a decreases the GZSL performance because over-enforced separation might cause imperfect embedding from the input.
that MCMAE perfor-
5.3 COMPARISON WITH GZSL STATE-OF-THE-AR MODELS
Table 2 presents the respective performance results obtained for the proposed method and GZSL tfte--of-the-art methods. First, MCMAE can be seen to have the same performance as existing relation-basee models. These models have low accuracy for the unseen domain, indicating that the domain bias problem occurs. Next, in synthetic-based models, the accuracy of the unseen domain is almost identical to that of the seen, indicating that the domain bias problem is mitigated. However, as described above, this approach must generate many images for even unseen classes after training their generative model and must prepare and learn additional classifier.
Finally, the results obtained with MCMAE-D, which learns domain discriminator jointly, clarify that the domain bias problem has been relaxed greatly and MCMAE-D achieves the best performance among the relation-based models. Additionally, it is apparent that it performs as well as state-of- the-art synthesis- based models.
These results revealed that the domain bias problem in GZSL can be alleviated and that high per- formance can be achieved using the domain discriminator the shared representation, even in relation-based approach.
CONCLUSION
This study addressed the domain bias problem in GZSL. First, considered domain invariance and class separability as requirements necessary for high performance in ZSL, and introduced MC- MAE that learns embedding in a shared representation that satisfy these requirements. Next, we assumed that the domain bias problem occurs when the unseen domain overlaps with the seen do- main and, to address this, proposed MCMAF-D learn the discriminator that distinguishes two domains from its representation. This discriminator not only encourages domain embedding to be separated in the representation space during training, but it also gives prior knowledge of which domain point in the representation belongs after training. Therefore, we combined this discrim- classification on MCMAE shared representation as a mixture model to
inator and relation-basse ascertain the classification probability. Through experimentation, mitigates the domain bias problem considerably. Future studies will assess application of our model to larger amounts of data while taking advantage of the delation-based method.
confirmed that this approach

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2927-2936, 2015.
Yuval Atzmon and Gal Chechik. Adaptive confidence smoothing for generalized zero-shot learning- In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11671 1- 11680, 2019.
Soravit Changpinyo, Wei-Lum Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero- shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- fion, pp. 5327-5336, 2016.
Wei-Lun Chao, Soravit Changpinyo. Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In European Conference on Computer Vision, pp. 52-68. Springer, 2016.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In Computer Vision and Pattern Recognition, 2009. CVPR 2009, IEEE Conference on, pp. 1778- 1785. IEEE, 2009.
Rafael Felix, Vijay BG Kumar, Ian Reid, and Gustavo Carneiro. Multi-modal cycle-consistent generalized zero-shot learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 21-37, 2018.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A deep visual-semantic embedding model. In Advances neural information processing systems, pp. 2121-2129, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Huajie Jiang, Ruiping Wang, Shiguang Shan, Yi Yang, and Xilin Chen. Learning discriminative latent attributes for zero-shot classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4223 4232, 2017.
Yue Jiao, Jonathon Hare, and Adam Prügel--Bnneet. Probabilistic semantic embedding, 2019. URL .nt httt://ormmmdddwwwseeeeee
attps://cppntee
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
Elyor Kodirov, Tao Xiang, and Shaogang Gong. Semantic autoencoder for zero-shot learning. arXiv preprint arXiv: arXivv1704..000 170400834.. 2017.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning detect unseen object classes by between-class attribute transfer. In Computer Vision and Pattern Recognition, 2009.
CVPR 2009. IEEE Conference on, pp. 951-9591118 IEEE, 2009.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine In- telligence, 36(3):45333-) 465, 2014.
Shichen Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Generalized zero-shot learning with deep calibration network. In Advances in Neural Information Processing Systems, pp. 2006- 2016, 2018
Ashish Mishra, M Reddy, Anurag Mittal, and Hema A Murthy. A generative model for zero shot learning using conditional variational autoencoders. arXiv preprint arXiv: 2017.
10

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recogniz- ing scene attributes. In Computer Vision and Puttern Recognition (CVPR), 2012 IEEE Conference on, pp. 2751-275555 IEEE, 2012.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: /40 2014.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learn- 2015.
ing. In International Conference on Machine Learning, pp.
Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8247-8255, 2019.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross transfer. In Advances in neural information processing systems, pp. 935- 943, 2013.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint 161 1.01891, 2016.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. arXiv preprint
2017.
Vinay Kumnr Verma and Piyush Rai. A simple exponential family framework for zero-shot learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 792- 808. Springer, 2017.
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. Generalized zero-shot learn- ing via synthesized examples. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition (CVPR), 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-uesd
birds-200-2011 dataset. 2011.
Yonggin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a com- prehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 2018a
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 5542-5551, 2018b.
Hongguang Zhang and Piotr Koniusz. Model selection for generalized zero-shot learning. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pp. 0-0, 201 8a.
Hongguang Zhang and Piotr Koniusz. Zero-shot kernel learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 7670-7679, 2018b.
Li Zhang, Tao Xiang, Shaogang Gong, et al. Learning a deep embedding model for zero-shot learning. 2017.
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via semantic similarity embedding. In Proceedings of the IEEE international conference on computer vision, pp. 4166-4174 2015.
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via joint latent similarity embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6034-
6042, 2016.
11

---

follows (DdimA is the dimension of Under review as a conference paper at ICLR 2020
A
NETWORK ARCHITECTURES
The Gaussian distribution is parameterized as
p==|2) = (2; o") - [=(Mmp((z)) a ((onp
where fx represents neural network and Softplus is softplus function.
Moreover, the Laplace distribution with constant scale is parameterized as follows.
p(= z) L(:; μ. ), = (fMLL(())))
In this paper, is assumed that only the mean parameter μs is output deterministically when sampling from the Laplace distribution with a constant scale:
L(x; μ. 2c =
For the notation of model structures, denote linear fully-connected layer with A units as k, batch normalization, and ReLU as DkBR. Also, we denote DkBR without batch normalization and ReLU Dk. In addition, the process of applying after is denoted I-J, and the process of concatenating the last layers of the two networks I, J into one layer is denoted as (I,
The network structures of distributions of MCMAE attributes and DdimZ is that of latent variable):
po. (xrl=) (Laplace)
.u: D2048
fMLP: 2-D1024BR-D1024BB
pa_ (alz) (Laplace)
- -fa: DdimA
-MMIP: z D-D1024BR-D1024BB
2x (2|2: (Gaussian)
- Su and faa: DdimZ
fMLP: x x-D1 11-D0024BR-D102000
1o. (zla) (Gaussian)
- -f and so=: DdimZ
fMLP: a-D1024BR-11022400
Moreover, the structure of the domain discriminator px B(d: Sigmoid ((MLP()))
(where B means Bernoulli distribution and Sigmoid is sigmoid function) is follows.
(d(22) (Bernoulli)
fú: D1
fMLP: 2-D1024BB
B LEARNING THE DOMAIN DISCRIMINATOR
straightforward way to learn the domain discriminator to use an inference model Lo., (z a.,)) to maximize the following objective function:
(10)
By learning both the inference model and the discriminator together, expect obtain a represen- tation that can be separated by the domain. However, only 9o. (2(a,,) will be affected strongly by the learning of the discriminator and might not be stable during training. In addition, the representation that we want to discriminate the test time embedded from the images, not from the attributes. Therefore, take method of generating image data in pseudo and learning the discriminator
12

---

Under review as a conference paper at ICLR 2020
60
50
acCN
40
CUB
30
aPY
20
20
40
60
80
100
Epoch
Figure 5: Learning curves of MCMAE-D for each dataset.
the representation inferred from them, which is why the reconstructed representation in discriminator learning.
Because attributes have less variation than images, the variance of representation sampled from To. (zlay) becomes small, and as a result, that of the reconstructed representation can also be small, which may lead to overfitting of the domain discriminator alleviate this, we consider increasing the variance of the inference model when calculating Eq.7, that is, we consider the inference model as 9e. (zlay) = (2; μt, a² d²a²), where a 1. this study, we set = 2. Note that this change is only for 7 and does not apply to other calculations such as the objective function of MCMAE (Eq.6).
the representation during training. To
C CADA-VAE
The objective of CAAAA-VAE (Schonfeld al., 2019) follows:
CCADA-VAF (x. a.) = CVAE (x) + Da.0a (ay) Bz.nidr.a
AV
+y (z(a) [log pe, (x|=)] + Egs, (=1m) pe. (a/z)])
-8W2 (x|z), 96. (= (=)a,)),
(11)
where W2(p.4) ④) is the 2-Wasserstein distance between p and and where 5 and y are weighting factors. Moreover, AE, is the objective of VAE that takes attributes inputs and uses inference and generative models for attributes.
In experiment Table all coefficients in Eq. 11 MCMAF. Moreover, in the relation-based setup, we used the Wasserstein distance instead of KI. divergence calculate the compatibility function (Bq. 2).
1 for fair comparison with
D LEARNING PROGRESS OF MCMAE-D
One advantage of relation-based methods including MCMAE-D is that GZSL performance be verified during model learning. On the other hand, the synthesis-based method cannot be confirmed it unless learning of the generative model completed and the synthesis data are generated. Figure shows learning curves of MCMAE-D for each dataset. From this result, it can be seen that the learning progress of this model almost stable, although the performance of aPY drops before 100 epochs.
13


Reviews:
[{"review_id": "rJgIEz285r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe main topic of this paper is generalized zero-shot learning. This paper modifies traditional VAE method with attribute matching prior to release the hidden features from original regularization. This paper also proposes a domain discriminator to enhance class-separability of learned features to avoid unseen classes to be covered by seen classes. Experiment results show their efficiency under relation-based setting.\n\nPros:\n1.This paper proposes an important insight that in generalized ZSL, the unseen classes may be dominated by seen classes in the feature space.\n2.An easy but efficient domain discriminator method is proposed to separate different classes to avoid domination. \n3.Even without large synthetic learning architecture, the proposed method gets comparable results.\n\nComments:\n1.The proposed MCMVAE is No-longer a VAE but an AE with attribute matching loss. Except that a new theory of MCMVAE is proposed, it is not rigorous to relate MCMVAE to VAE.\n2.Add results using synthetic architecture to get a better result will make this method more reliable.\n3.Why discriminator is harmful for PSE method?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rJg9dq0kqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a relation-based ZSL model which can effectively alleviate the domain bias problem. To this end, first, the paper claims that a good relation-based ZSL model should consider two requirements -- modality invariance and class separability. And the paper designed Modality-invariant and Class-separable Multimodal VAE (MCMVAE) based on VAEs to meet the two aforementioned requirements. Next, the paper hypothesizes that the domain bias problem is due to the overlap between seen and unseen classes in the shared space, and explicitly introduced a discriminator to separate the two domains. The paper performs experiments on ZSL benchmark datasets and shows that the proposed method outperforms other relation-based methods. Besides, the domain discriminator which can be applied to other models demonstrates its effectiveness in reducing domain bias given the experimental results.\n\n+Strengths:\n1. Clear writing logic. The author clearly depicts how to get the final loss of the method step-to-step and the relationship with existing methods.\n2. The version without the domain discriminator (i.e. MCMVAE) is similar to PSE and CADA-VAE as the author acknowledges. However, the domain discriminator has certain novelty and can be applied to other methods. The overlap among seen and unseen classes is an important problem (domain bias problem named by the author) and the add of the domain discriminator to distinguish whether a sample is from seen classes or unseen classes is reasonable, which can provide better class separability (among seen and unseen classes).\n\n-Weaknesses:\n1. Although the author claims that the proposed method is a relation-based method, it is strange that the proposed method is called xxVAE but in Table 2 it doesn't fall into synthesis-based methods (as CVAE-ZSL and CADA-VAE do). Although it is derived from VAE, the current method doesn't seem to be called a VAE any more (some of the regularizations of the VAE are relaxed). Also, are the two terms -- relation-based and synthesis-based -- first proposed by the author? Is there a clear boundary between those two groups of methods?\n2. It is recommended that an additional figure that depicts the framework is added (similar to Figure 2 in CADA-VAE) to promote better understanding. Currently, the method part only contains formulas with many parameters, making it difficult to grasp the idea of the whole framework at first glance.\n3. The novelty of this paper is somewhat limited while missing some relevant works, e.g.[r1, r2]. [r1] learns a latent space where the compactness within class and separateness between classes are considered. [r2] uses a two-stage prediction for GZSL.\n[r1] Jiang et al. Learning Discriminative Latent Attributes for Zero-Shot Classification. In IEEE ICCV 2017.\n[r2] Zhang et al. Model Selection for Generalized Zero-shot Learning. In arXiv 2018.\n4. It is a question whether the seen and unseen classes can be separated (Whether a two stage process is correct?). The key for ZSL is knowledge transfer and the base is that seen and unseen classes are related [r3]. If they are separated, can one use the model trained on seen classes to recognize the unseen classes? This is quite problematic. Besides, in Tab.2 there lacks of necessary comparisons with recent relation-based approaches e.g.[r3][r4], which makes the evaluation less sufficient.\n[r3] Jiang et al. Transferable Contrastive Network for Generalized Zero-Shot Learning. In IEEE ICCV 2019.\n[r4] Li et al. Discriminative Learning of Latent Features For Zero-Shot Recognition. In IEEE CVPR 2018.\n5. Some unclear/incorrect descriptions of the method:\n5.1) The formulation of GZSL is incorrect. Y= union(y_s  y_u), but not intersection(y_s  y_u)\n5.2) How is the class separation formulated in the framework?\n5.3) In Sec.3.2, why is the log-likelihood of the generative models can be obtained by the L1 loss?\n\nMinor issues:\n1. Better use vectorgraphs for clear view (especially for Figure 3 and 4). \n2. Incomplete reference: for Probabilistic semantic embedding (PSE), the reference should add the conference information.\n3. Grammar and spelling mistakes: \n[1] Content in Figure 2 (not caption): unseen class -> unseen classes\n[2] Last line in 4.1: MCVAE-D -> MCMVAE-D\n[3] Last paragraph in 4.2: close -> stay close\n[4] Last model name in Table 1: MCMVAE -> MCMVAE-D\n4. The color bar for the contours at the rightmost of Figure 3 is not clear (not the standard way to draw a color bar, better refer to what a color bar is usually drawn).\n5. If possible, better reduce the main text to 8 pages as recommended by the submission instructions (e.g. some content of the method part can be moved to the appendix?).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lrWRQvtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel approach for (generalized) Zero-shot learning (GZSL). As showing in the numerical experiments on some real data, the method demonstrates the significant improvement on the accuracy of prediction comparing to some state-of-the-art methods. \nThe main key of the method is using Variational Inference, variational autoencoders. The authors have taken into account the modality of the data through reparametrize the distributions, especially the inside class invariant modality and class separability. Moreover, the authors also propose to take into account a kind of biasness domain into the learning procedure, which details in adding a regularization of the domain discriminator into the objective function.\n\nThe paper is nicely written, espcially with a clear formal introduction to the problem of GZSL.\n\nHowever, I have some questions:\n1) Does the test set has some labels? How do you know your method works well? I can not find where you have defined a kind of loss so that we can compare the predicted labels \\hat{y}_j ?  (In Section 2.)\n2) How do you learn the \"replaced prior\" in equation (4) ?\n3) It is not enough detail on how do you optimize the objective (8) ? a detail explain algorithm would make the paper significant, indeed.\n4) In Table1, would MCMVAE in the last row be MCMVAE-D ?\n\nFinal, I expect the authors will make their codes available for the readers.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a relation-based model that extends VAE to explicitly alleviate the domain bias problem between seen and unseen classes in the setting of generalized zero-shot learning.

Reviewers and AC think that the studied problem is interesting, the reported experimental results are strong, and the writing is clear, but the proposed model and its scientific reasoning for convincing why the proposed method is valuable is somewhat limited. Thus the authors are encouraged to further improve in these directions. In particular:

- The idea of using a variant of the widely-used domain discriminator to make seen and unseen classes distinguishable is somewhat contradicted to the basic principle of zero-shot learning. How to trade off the balance between seen and unseen classes has been an important problem in generalized ZSL. These problems need further elaboration.

- The proposed model itself is not a real "VAE", making the value of an extensive derivation based on variational inference less prominent. 

- There is also the need to compare with the baselines mentioned by the reviewers. 

Overall, this is a borderline paper. Since the above concerns were not addressed convincingly in the rebuttal, I am leaning towards rejection.
Decision: Reject