Adaptive and Generative Zero-Shot Learning
Authors: Yu-Ying Chou
Hsuan-Tien Lin
Tyng-Luh Liu
Venue: ICLR 2021 Poster (2021)
Keywords: Generalized zero-shot learning
mixup

Abstract: We address the problem of generalized zero-shot learning (GZSL) where the task is to predict the class label of a target image whether its label belongs to the seen or unseen category. Similar to ZSL, the learning setting assumes that all class-level semantic features are given, while only the images of seen classes are available for training. By exploring the correlation between image features and the corresponding semantic features, the main idea of the proposed approach is to enrich the semantic-to-visual (S2V) embeddings via a seamless fusion of adaptive and generative learning. To this end, we extend the semantic features of each class by supplementing image-adaptive attention so that the learned S2V embedding can account for not only inter-class but also intra-class variations. In addition, to break the limit of training with images only from seen classes, we design a generative scheme to simultaneously generate virtual class labels and their visual features by sampling and interpolating over seen counterparts. In inference, a testing image will give rise to two different S2V embeddings, seen and virtual. The former is used to decide whether the underlying label is of the unseen category or otherwise a specific seen class; the latter is to predict an unseen class label. To demonstrate the effectiveness of our method, we report state-of-the-art results on four standard GZSL datasets, including an ablation study of the proposed modules. 

Published as a conference paper at ICLR 2021
ADAPTIVE AND GENERATIVE ZERO-SHOT LEARNING
Yu- Ying Chou Hsuan-Tien Lin & Tyng-Luh Liu
Institute of Information Science, Academia Sinica, Taiwan Taiwan Al Labs, Taiwan DDepartment of Computer Science, National Taiwan University, Taiwan
[c0792200) hciin .tt.tttuu edu. tww liutyngeiiss sinica. edu. tw
ABSTRACT
We address the problem of generalized zero-shot learning (GZSL) where the task is to predict the class label of target image whether its label belongs to the seen or unseen category. Similar to ZSI., the learning setting assumes that all class-level semantic features are given, while only the images of seen classes are available for training. By exploring the correlation between image features and the corresponding semantic features, the main idea of the proposed approach is enrich the cemantic-to-visuie (S2V) embeddings via a seamless fusion of adaptive and generative learning. To this end, we extend the semantic features of each class by supplementing image-adaptive attention so that the learned S2V emhedding can account for only inter-class but also intra-class variations. In addition, to break the limit of training with images only from seen classes, we design generative scheme simultaneously generate virtual class labels and their visual features by sampling and interpolating over seen counterparts. In inference, testing image will give rise two different S2V embeddings. seen and virtual. The former used to decide whether the underlying label is of the unseen category or otherwise a specific seen class; the latter is to predict an unseen class label. demonstrate the effectiveness of our method, we report state-of-the-art results on four standard GZSL datasets, including an ablation study of the proposed modules.
INTRODUCTION
Different from conventional learning tasks, zero-shot learning (ZSL) by Lampert et al. (200)); Palatucei et al. (2009): Akkaa et al.) (2013) explores the extreme case of performing inference only over samples of unseen classes, To make the scenario more realistic, generalized zero-shot learning (GZSL) (Chao et al. 2016 Xian et al. 2017) subsequently proposed so that inference can concern samples of both seen and unseen classes. Nevertheless. the learning setting in ZSL/GZSL is essentially the same where sample classes are divided into two categories, seen and unseen, but only those samples of seen classes accessible training. In addition, each of all the classes under consideration is characterized by semantic features such as attributes (Xian et al.) 2018b) or text descriptions (Zhu et al. 2018) specify and relate seen and unseen classes.
The lack of training samples from unseen classes has prompted generative approaches (Chen et al. 2018 Felix al., 2018 Kumar Verma al. 2018 Mishra al. 2018) creating synthetic data from semantic features of unseen classes. The strategy could enable learning semantic-visual alignment on unseen classes implicitly, and thus improves the ability to classify unseen classes. However, such generative models are indeed trained on seen samples, and the quality of synthesized unseen samples predominantly influenced by seen classes. If the number of training samples of each seen class small, it is hard for generative models to adequately synthesize samples of unseen classes, leading to unsatisfactory zero-shot learning. To better address the issue, we propose to synthesize visual and semantic features of virtual classes rather than those of the unseen classes. An interesting analogy is that childhood experience and relevant study (Greene 1995) suggest the behavior of using human imagination to produce new object concepts could assist cognitive capability. To mimic people utilizing imagination for exploring new knowledge, we create virtual classes by the integration of past "experience" (seen classes). In detail, extend the mixup technique by /Zhang al. (2018) to

---

Published as a conference paper at ICLR 2021
generate virtual classes, with subtle difference that mixing is conducted on the semantic features (in addition to the visual ones), instead of the class label vectors.
In ZSL/GZSL, each seen unseen class is typically described by single semantic feature vector. The practice is useful in differentiating different classes in a principled way, but may not be sufficient to reflect the inter-class and intra-class visual discrepancies, not to mention the ambiguities caused by different backgrounds, view orientations, or occlusion in images. The concern of inefficient class-level representation can also be observed from how the semantic feature vectors are constructed. Take, for example, the Attribute Pascal and Yahoo (aPY) dataset (Farhadi et al. 2009), where each instance is annotated by 64 attributes. The semantic features of each class in aPY are obtained by averaging the attribute vectors of all its instances. We are thus motivated to introduce an image- adaptive class representation, integrating the original semantic features for inter-class discrimination with an image-specific attention vector for intra-class variations.
With the addition of virtual training data and the andgge-d class representation, our method is designed to learn two classification experts: one for seen classes and the other for unseen classes. Both experts project the image-adaptive semantic feature vectors to the visual space and use cosine similarity to find the class label most similar to the given visual feature vector. The seen expert is trained with the provided training (seen) data, while the class prediction is over all possible classes, including seen and unseen. In inference, if its predicted class is not within the seen category. The testing sample is deemed to be from the unseen category, whose label is then decided by the unseen expert. The unseen expert is trained with the virtual data only, and the process indeed resembles meta-learning, However, the effectiveness of meta-lcarning is boosted by the design of the image- adaptive mechanism in that fine-tuning is not needed in performing zero-shot classification over unseen classes. We characterize the main contributions of this work as follows.
Instead of generating synthetic data of unseen classes, we propose to yield virtual classes and data by mixup interpolations. The virtual classes of synthetic data can then be seamlessly coupled with meta-learning to improve the inference on unseen testing samples.
We introduce the concept of representing each class with image-adaptive semantic features that could vary among intra-class samples. While the adaptive mechanism improves clas- sifying the seen classes, it manifests the advantage in boosting the effect of meta-learning over virtual data to zero-shot inference over unseen classes.
We demonstrate -f-teoof-hhe-art results of zero-shot learning over four popular benchmark datasets and justify the design of our method with a thorough ablation study.
RELATED WORK
We review relevant literature in this section. First, we describe generative approaches for ZSL/GZSL that synthesize unseen images for training. To improve GZSL performance, we propose to couple virtual class generation with meta-learning for mimicking the inference scenario. Next, we discuss attention approaches that extract discriminating features from images to help classification.
2.1 GENERATIVE APPROACHES FOR ZsL/GZSL
Arguably one of the most important problems in ZSL/GZSL is to prevent models from being biased to
seen classes. Generative approaches (Chen et al., 2018 Felix et al. 2018 Kumar Verma et al. 2018 Mishra et al. 2018 Schonfeld et al. 2219 Paul et al. 2019 Xian et al. 2019) tackle the problem by synthesizing visual features of unseen classes from their semantic features with generative models like Generative Adversarial Networks (GAN; Goodfellow et al., 2014) or Variational Autoencoders (VAE) (VAE: Kingma& & Welling 2014). The synthetic visual features act as pseudo-examples of unseen classes, and effectively reduce the ZSL/GZSL problem to supervised learning one.
One key issue behind generative approaches comes from the insufficient amount of data to learn good generative model. As consequence, some semantic features that seem important during training may cause overfitting, and others that seem less important may be completely dropped. Therefore, several prior techniques propose new constraints or losses to preserve semantic features and regularize the generative model. For instance, (Chen et al.) 2018) avoids the dropping of semantic information by disentangling the semantic space into two subspaces, one for classification and the
2

---

Published as a conference paper at ICLR 2021
other for reconstruction; Felix al. (2018) enforces visual-semantee feature consistency by requiring synthesized visual features to reconstruct semantic features accurately.
However, even with the new constraints losses, the success of generative approaches still highly depends on whether there is enough data/variation in the seen classes to synthesize diverse visual features. Otherwise, the yielded features will be too close those few seen classes and cannot help ZSL/GZLL much. For example, in AWA2 dataset (Xian et al. 2018a), the unseen samples of "rat" are easily misclassified as seen classes of "mouse" "rat" are doomed to be confused with seen visual features of those similar classes. The phenomenon inspires us generate synthetic visual features for not only the given unseen classes but also the virtual unseen classes. These virtual classes provide wider spectrum of support for unseen scenarios to improve 7SL/G performance.
"hamster" because synthetic visual features for
Our other focal effort is to connect the concept of meta-learning /Vinyals et al. 2016 Finn et al. 2017) with virtual classes the training phase of the unseen expert. The most closely related approaches in ZSL are Li et al. (2019b); Yu et al. (2020); Vermma et al. (2020); [Sung al. (201 8); Hu et al. (201 8). Verma et al. (2020) combines meta-learning and generative models to conquer limits in generative models. Li et al. (2019b) mimics the inference scenario by randomly selecting seen classes as "fake" new classes in each episode. Yu et al. (2020) randomly splits seen classes into two sets to train and simulate the ZSL inference scenario in each episode by creating virtual classes from seen classes using mixup. Owing to the training scenario more resembling the inference setting in ZSL., our model achieves better $2V embedding on unseen classes and obtains state-of-ther thrtthrrrttt ZSL performance on most datasets. Besides, |Sung et al. (2018) learns a deep distance metric classify samples. Hu et al. (2018) utilizes the correction module assist classification. On the contrary, our model does not need to learn complex relationship between classes and additional assistant module to achieve good performance.
refine the model. In contrast,
2.2 ATTENTION
Attention mechanism is widely used in ZSL (Ji et 2018 Xie al. 2019 Huynh & Elhamifar Min et al.] 2020; Liu et al. (019). Highlighting important local features and reducing noisy feature influence generate more effective mapping between visual and semantic domains. Ji et al. (2018) proposes SaGA which utilizes semantic features to emphasize most informative visual local features. (Xie al. (019) employs AREN and ACSE focus the most important region in images. Huynh & Elhamifar (2020) applies dense attribute-attention to find the most discriminating image parts and embed them to semantic features individually. DVBE (Min et al. 2020) utilizes spatial and channel attention to maximize inter-class margin. While the attention to visual features is research efforts, relatively few attempts assess the importance of attention semantic features. In the ZSL setting, each class contains only one semantic feature vector such that intra-class variations are neglected from the single semantic representation. For example, the object of interest can be occluded partly in the image in the front of different backgrounds, and a unique class-wise semantic feature vector is hard to reflect such variances. Thus, attention to semantic features is worth further exploring in ZSL. Along this line, the most relevant work to ours is LFGAA (Liu et al. 2019), which considers semantic prediction of samples. Different from LFGAA, our method puts attention maps them to the visual domain. The effect of the proposed attention mechanism is to attract the visual representation of ground-truth class and repel others. Besides, our other main difference from LFGAA is the shortcut design in the attention model. We add a shortcut to prevent the model from overfitting on training data and retain discriminating dimensions for unseen class classification.
proved useful by the above--entiones
discriminating dimensions of semantic features and
METHOD
We coin the proposed method as AGZSL in that its main idea is the fusion of Adaptive and Generative mechanisms in solving 7SI/GZSI. (See Figure T) We describe below the details of AGZSI..
3.1 NOTATIONS AND FORMULATION
We denote training data as D = Ds U Dr where Dş = [xx, E X, E As. ys E {.)) comprises seen data and Dú (a.,, E An, Yu E )u)) includes unseen class labels yu and the corresponding
3

---

Published as a conference paper at ICLR 2021
Visual space
Semantic space
Visual space
Semantic space
X'
B(ar, 0]
a'
ag
attention
X,
a
X +
Bgy a
vieual reproseration withou ISS
vitual apprsssstation with
with WA
celocted clas
Vinaal coos
(a) Adaptive learning
(b) Generative learning
Figure 1: Overview of AGZSL. The same color reflects the same class association between visual features and attributes. (a) Image-adaptive mechanism (marked as purple) based on x yields proper attention adapt the semantic features of and a' so that their S2V embedding (marked orange) displays the desired effects in & (attracted to x) and in X (repelled from x). (b) We generate virtual classes from pair of classes of semantic features a and a' with selected weights via beta distribution ((a,3) B). X and a,/al, represent visual and semantic features of virtual classes.
semantic/attribute feature vectors a The dimensions of visual features and semantic features are respectively expressed as x E IRd and R while the number of total classes is n. We also assume that the seen category Vd, and the unseen category Vi, are disjoint, i.e., J', n 0 and } U. = y. A E k as semantic features of all classes. Finally, the virtual data via mixup interpolations is denoted as Dy {(x, X a, E Au, Yo ).) As explain later, Yo is simply a new label of virtual class, rather than from label interpolation as in Zhang et al. (2018).
Besides, we define A, U.A.A
To perform the classification task, we explore the similarity correlations in the visual domain and consider semantic-to- visuall (S2V) embeddings. Analogous to COSMO (Atzmon & Chechik 2019). method includes seen and unseen experts, fa and fu. to classify testing samples as shown in Figure 2 however, the algorithmic designs are fundamentally different. The novel component of our method is the introduction of Image-Adaptive Semanties (IAS) in both experts to expand the spectrum of S2V embeddings. Specifically, in training, IAS diversifies the total mappings of S2V embedding from the number of different semantic vectors/classes to the number of samples. Its effect is twofold. First, it boosts the classification performance of the seen expert. Second, it generalizes the learnin of the unseen expert with virtual data and yields good performance in the inference of samples from unseen classes. Given testing sample x in inference, we apply the two learned experts by
9. = arg max (x) and iu arg max (x( Au). yEY
vEY.
To decide the class label 9 of x, we use the seen expert to discriminate whether the sample is from the seen category. If it is the case then ÿ is decided by fs, and otherwise by fu. That is, have
i = {
Ú. if g. V... Du otherwise.
(2)
3.2 IMAGE ADAPTIVE SEMANTICS (IAS)
As ZSL///G 7L7// assumes one semantic feature vector for each class, the conventional semantic-to- visual architecture leads one single class-level visual feature vector, which is common to all samples of the same class. To learn such single embedding vector with the constraint of being simultancously similar to all visual features of samples from the same class tends to yield unsatisfactory classification outcomes in that simply ignores the intra-class variations in visual features.
Diversifying semantic-to- to-visual embeddings with IAS establishes the core of our model as shown in Figure 3 To realize the concept, the network module learns to map image- adaptive semantic features to the corresponding visual features, according to the cosine similarity. To this effect, IAS leverages the given visual features focus discriminative dimensions of the respective semantic features. Specifically, we apply IAS to obtain modified semantic features Á(x) k by
A
A c softmax(g(x)),
4

---

Published as a conference paper at ICLR 2021
Training phase
Inference phase
y
+
A
X
Seen Expert
CELoss
A
Seen Expert
arg max ycy
Ds
9. E >>
f.
yv
fx
X
arg MAX NEV.
X
Unseen Expert
CELoss
Au
Unseen Expert
9u
g. E V
A
fu
fu
Figure 2: The proposed seen and unseen experts. In the learning phase, the seen expert fa is trained with seen visual features X,, seen semantic features As, and corresponding class labels ye while the unseen expert fa is trained via meta-learning with virtual data. In the inference phase, given testing sample x, fs additionally fed with all semantic features A, and fa only with unseen semantic features A.. The decision rule leading to the class label is based on (2) Note that in both training and inference stages, the image-adaptive mechanism to semantic features carried out.
where softmax(g(x)) E I represents using linear layer g predict semantic attention and @ denotes Hadamard product to combine attention and semantic features. Note that the shortcut implementation in (3) pivotal as the softmax function often puts high attention weights on sparse dimensions and draws others near to zero. Without the shortcut, the modified A would miss some part of semantic features and degenerate the classification performance. Also, it is worthy to point out that compared to LFGAA (Liu et al. 2019), our method pays attention to the semantic features of classes, not the semantic prediction of the model. With (3), the process to drive the image-adaptive class-level visual features (xx can expressed by
R
XER
A(x)
E TR kxn
v(x)
Rdxn
(4)
LAS
S2V
Once we have obtained the class-level visual features, we then use the cosine similarity and softmax to calculate the probability of each class. We have the class probability of prediction on x:
p(x) = f(x,(xxxxxx A) = softmax(o x cos(V(x)), x)),
(5)
where a is the learnable scale and f is indeed the general form of a classification expert. We conclude by stating that the Cross-Fnntroyy(EE) loss is adopted to update the model:
log (x()),
where (x) is the predicted probability of ground-truth label.
3.3 SEEN EXPERT
The objective of seen expert fa is to recognize seen classes and otherwise decide whether a testing sample belongs to unseen classes in inference phase. In zero-shot learning, especially GZSL, the capability to single out testing samples of unseen classes is crucial to the classification performance. one can establish reasonable split mechanisms such as in Min et al. (2020) and Chen et al. (2020), the more challenging GZSL would be simplified into ZSL. For example, COSMO (Atzmon & Chechik 2019) utilizes Confidence Base Gating to discriminate between seen and unseen classes, while DVBE MMin et al. (020)) considers the entropy threshold. In our formulation, the seen expert learns to predict all possible classes, including seen and unseen, although the training data assume only seen class labels. That is, f. is trained to map (image adaptive) seen semantic features to their corresponding visual features. Such optimization can be thought of as one-category anomaly detection over only classes. Thus, in inference, if the highest cosine similarity value by the expert fa is not yielded by one of the seen classes, the testing sample is considered abnormal from the unseen category and its classification should be decided by the unseen expert discussed next.
5

---

Published as a conference paper at ICLR 2021
L2
softmax > p(xx)
X
FC
softmax
g
A
A
FC
FC
L2
IAS
S2V
Hadamard product
Add
Matrix multiplication
Figure 3: The architectures of IAS and S2V (FC: linear layer, L2: Ln-normalization).
3.4 UNSEEN EXPERT
So far we have stated the objective of the unseen expert Su is to classify the labels of those that cannot be decided by the seen expert, as in ② To resolve the dilemma zero-shot setting, we apply meta-learning to train fu over virtual classes by mixup interpolations. The exact number of virtual classes in each episode is decided by the pilot study (Li et al. 2019b), and an ablation study in appendix is provided to analyze its effect the performance of zero-shot classification. To begin with, in each episode we randomly select m pairs of seen classes. Then, we create m virtual classes of these pairs by mixup. The semantic and visual features for these virtual classes are defined as
a, = ha, + a) aj A) xj:
X, Ax;
where a; and aj semantic features of paired classes, and x, and x, are randomly chosen from visual samples of the paired classes. Following Zhang et al. (2018). E [0, 1] sampled from the beta distribution. We then define a new virtual class Ve. with a, as semantic features, be associated with virtual visual features X Previous generative models (Schonfeld et al. 2019 Xian et al. 2019) synthesize unseen samples by GAN/VAE their variants. In comparison, the proposed meta-learning has at least three advantages for ZSL. First, the small number of classes in some datasets, such as 50 AWA2, may fall short provide sufficient support set such that the generative model is hard be optimized, while episodic meta-learning over virtual classes can overcome this problem. Second, some part semantic features which are important to classify unseen classes do not have enough training classes in the datasets. With mixup semantic features as virtual classes, model can learn underlying discriminating semantic features well. Third, the fusion image-adaptive mechanism and meta-learning is flexible. As result, do not need online fine-tuning in applying the unseen expert fa yielded by meta-learning to predict the label unseen class for testing samples.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTING
Datasets. In GZSL, there are four widely used benchmark datasets for evaluation, which are AWA2
(Xian et al. 2018a) Sa). USCD Birds-200-2011 (CUB) (Welinder et al.) 2010), SUN (Patterson & Hays 2012), and aPY (Farhadi al. 2009). AWA2 contains 37K images and 50 kinds of animals with 85- dimension attributes. CUB consists of 11K images and 200 bird species described with 3 312-dimension attributes. SUN is composed of 14K images 717 classes labeled with 102-dimension attributes. aPY includes 15K images and 32 different classes with 64-dimension attributes. The attributes of class in datasets will be taken as the semantic features in our work. We follow existing GZSL approaches (Min et al., 2020 Li et al. 2019b) and extract 2048-dimensional visual features for each image by the ResNet backbone (He et al. 2016) pretrained on ImageNet- We then split the data into seen and unseen classes according the benchmark procedure from Xian al. (007))

---

Published as a conference paper at ICLR 2021
Table I: GZSL results four datasets. All methods in comparison utilize ResNer101 the backbone for fairness. Notation "*" means the method fine-tunes the backbone to match the characteristic of datasets.
ARAZ
CUB
SUN
BPY
G7ST.
7ST.
GZST.
G7ST.
T1
H
T1
T1
S
H
T1
H
SP-EEN
58.5
23.3
90.9
37.1
55.4
34.7
70.6
46.6
59.2
24.9
38.6
30.3
24.1
13.7
63.4
22.6
DLFZRL
60.9
45.1
51.9
37.1
42.5
24.6
38.5
31.0
PSR
63.8
20.7
73.8
32.3
56.0
24.6
54.3
33.9
61.4
20.8
37.2
26.7
38.4
13.5
51.4
21.4
CDL.
54.5
23.5
55.2
32.9
63.6
21.5
34.7
26.5
43.0
19.8
48.6
27.1
POZSL
53.2
51.4
46.9
35.1
35.3
35.2
27.9
64.1
64.1
38.8
EVARGAN-D2
70.3
57.1
76.1
65.2
72.9
63.2
75.6
68.9
65.6
50.1
37.8
43.1
LsrGAN
54.6
74.6
63.0
48.1
59.1
53.0
44.8
37.7
40.9
TA-VAGANN
73.4
55.5
83.6
66.7
74.3
63.8
79.3
70.7
66.7
41.8
51.9
46.3
OCD-CVAE
71.3
59.5
73.4
65.7
60.9
44,8
59.9
51.3
62.1
44.8
42.9
43.8
ZSML Softmax
76.1
58.9
74.6
65.8
69.6
60.0
52.1
$5.7
60.2
64.1
36.3
46.6
40.9
GXE
71.1
56.4
81.4
66.7
54.4
47.4
47.6
47.5
62.6
36.3
42.8
39.3
38.0
26.5
74.0
39.0
E-PON
73.4
52.6
83.5
64.6
72.4
52.0
61.1
56.2
Relation
38.1
Correlation Net
45.8
41.9
LFGAA+Hibrd
68.1
36.2
80.9
40.0
AREN
66.9
54.7
79.1
64.7
72.5
63.2
69.0
66.0
60.6
40.3
32.1
35.9
39.2
30.0
47.9
36.9
COSMO
44.4
37.7
DVBE*
62.7
77.5
69.4
64.4
73.2
68.5
44.1
41.6
42.8
37.9
55.9
45.2
ours
75.8
65.1
78.9
71.3
57.2
41.4
49.7
65.3
29.9
40.2
54.3
41.0
35.1
65.5
45.7
ours
76.4
69.0
86.5
76.8
77.2
69.2
76.4
72.6
66.2
50.5
43.1
46.5
43.7
36.2
58.6
44.8
We evaluate our approach for both ZSL and GZSL settings. For the ZSL setting, the average per-class top-1 accuracy (T) on unseen classes is taken the evaluation metric. For GZSL (Xian et al., (017), the evaluation metrics include seen classes accuracy S, unseen class accuracy U, and their harmonic mean H (2U-S)((U) (U
Evaluation.
Implementation. As suggested in Li al. 2019b), normalize the visual and semantic features into [0, 1]. The architecture of semantic-to-vistee visual embedding contains a two-layer linear model with 1,600 hidden units and utilizes ReLU the hidden and output layer. The seen and unseen experts trained by Adam optimizer with a learning rate x 10-5 and 5 x 10-4 respectively for all datasets. We apply 200,000 episodes train the unseen expert. In each episode, we randomly generate 16 or 20 (based on the dataset) virtual classes and 4 samples for each class. We follow the setting in (Min al. 2020) to fine-tune the backbone.
4.2 COMPARISON WITH STATE-OF-THE-ART APPROACIES
We compare our method with 13 recent G7SL. approaches. These include SP-AEN (Chen et al. 2018
DLFZRL Tong et al. 2009), PSR (Annadani & Biswas 2018), CDL (Jiang et al. 2018). PQZSL
Li al. 2019a), E-AAGA---2 (Xian al. 22019), LsrGAN (Vyas et al. 2020). TF- TAEGAN
Narayan et al. 2020), OCD-CVAE (Keshari et al. 2020). ZSML Softmax (Verma et al.
2020).
EGG Yu et al. 2020), Relation Net (Sung et al.) 2018), Correlation Net (Hu al. 2018
GXE
Li et al. 2019b). LFGAA+Hybrid (Liu et al. 2019). AREN (Xie et al.) 2019), COSMO (Atzmon
Chechik 2019). DVBE (Min al.] 2020 Table T shows that the proposed AGZSL achieves
the best GZSL performance on all four datasets. It improves harmonic mean by 7.4% on AWA2, 1.9% on CUB, 0.2% on SUN, and 0.5% on aPY, respectively. Particularly, among those methods in comparison, DVBE needs to search the entropy threshold 7 and the embedding model for the best harmonic means according to each dataset. contrast DVBE, without entropy threshold and specific model searching, our model achieves over 4% improvement on most datasets. The main reason for better performance is that AGZSL provides more discriminating visual representation for both seen and unseen classes. Therefore, our model can effectively separate seen and unseen samples and achieve better performance in the harmonic mean. Notably, AGZSL drops performance on aPY after backbone fine-tuning. It could be due to that its number of seen classes and training images are fewest among the four datasets. fine-tuning the backbone causes the visual features to overfit training samples and become less discriminating for correct classification.
7

---

Published as a conference paper at ICLR 2021
Table 2: The effect of virtual classes and IAS for G7SL., We remove these components baseline (b). In the ablation study, we add virtual classes (v) and IAS (I) step by step to show their effect on GZSL.
AMRZ
CUB
SUN
aBY
ZSL
GZSL
ZSL
GZSL
ZSL
GZSL
ZSL
GZSL
b
72.2
63.2
86.1
72.9
73.7
68.6
71.3
69.9
63.9
47.6
42.1
44.7
40.0
33.4
57.2
42.2
b+v
75.8
66.9
75.3
76.1
70.3
71.3
70.8
65.6
48.6
42.1
45.1
42.8
57.2
43.6
b+v41
76,4
69.0
86.5
76.8
77.2
69.2
76.4
72.6
66.2
50.5
43.1
46.5
43.7
58.6
44.8
Furthermore, generative models achieve higher performance CUB and SUN, because in these two datasets there are many seen classes for synthesizing unseen samples better quality. In comparison, model overcomes the influence of few classes and also achieves the best performance AWA2 and aPY. Our results empirically support that the introduction of virtual classes and samples indeed helpful classify unseen classes. Finally, we remark that since the backbone is pretrained on ImageNet-1K, which rather different from the ZSL fine-grained databases, e.g., CUB and SUN. Thus, the harmonic mean improves after fine-tuni tin-ttnnin the backbone on these datasets.
4.3 ABLATION STUDY
To evaluate the benefits of tackling ZSL/GZSL with virtual classes and IAS, we carry out an ablation study on the four datasets. We incrementally include each key component to assess their effect. As shown in Table 2 the virtual classes improve ZSL accuracy significantly. The ZSL performance improves 4.6%
AWA2, 2.4% on CUB, 1.7% on SUN, and 2.8% aPY, respectively. The result implics that the virtual classes are advantageous to ZSL. After adding IAS to our model, both seen and unseen accuracy increases, and the harmonic mean H improves about 1.5% average. The consistent gain exemplifies that IAS can provide better semantic-to-visuut embedding and achieve higher performance on GZSL.
4.4 COSINE SIMILARITY MARGIN
Figure 4 fillustrates the effect of the proposed AGZSL model. The cosine similarity margin means the inner product value between the correct class and the other nearest class. A negative margin indicates that the model predicts a wrong class. the other hand, a large positive margin implies that model results in proper embedding distinguishing the correct class from others effectively. Thus, secking a positive and larger margin is a crucial and important objective in GZSL. As shown in the first row of Figure 4 the margins of unseen samples shift right on all four datasets by method. shows that AGZSL indeed generates better embedding on the visual space and makes the model easier to predict the correct class. Moreover, from the second row in Figure the plots show that method improves the margins of seen samples dramatically on most datasets. The above experiment explains that AGZSL can achieve higher margin for most samples and achieves better classification performance on the four testing datasets.
4.5 THE DISTRIBUTION OF VIRTUAL CLASSES
To explicitly exhibit the distributions of seen, unseen and virtual classes, apply PCA to map the class-wise semantic features to a two-dimensional space, as shown in Figure The features of virtual classes scatter around seen classes, because we have used B(5,1) to generate virtual classes. Advantageously, some virtual classes are very close to unseen classes the semantic domain and turn out to be helpful for enabling the unseen expert to classify samples of unseen classes. With meta-learning over these numerous virtual classes and samples, the proposed S2V can better learn the embedding between semantic and visual domains, and results in an effective unseen expert.
8

---

Published as a conference paper at ICLR 2021
sumplees
103
100
350633-2 33
-3--------44-
similaty
coinne simlarty
siileitt
(a) AMA2 unseen samples
(b) CUB unseen samples
(c) SUN unseen samples
aPY unseen samples
100
333
335
106
200
Skrbedd P
-82 -53
-22 -23
63 as similerity
similiity
(c) ANA2 seen samples
(f) CUB seen samples
SUN seen samples
(h) aPY seen samples
Figure 4: The cosine similarity margin between the correct class and other nearest class. The blue area depicts the cosine similarity margin of baseline without IAS and generative learning. On the other hand, the orange area represents the cosine similarity margin of ours. Most of the samples improve their margin by method, suggesting the proposed techniques are beneficial for GZSL.
34
4
42
(a) ANA2
(b) CUB
(c)
SUN
(d)
aPY
Figure 5: The semantic features of virtual/secn/unseen classes over the four datasets.
CONCLUSION
In this paper, we propose Image Adaptive Semanties (IAS) and meta-learning with virtual classes and samples to solve the 7SL/G7SL problem. To deal with the intra-class visual discrepancies, IAS adaptively emphasizes the most discriminating dimensions in semantic features with respect to the underlying visual features. To better classify samples of the unseen classes in inference, we propose a novel formulation of generative meta-learning. Different from previous generative models that focus on synthesizing unseen samples for training the model, we create virtual classes and their respective virtual samples in the training phase. Further, to imitate the ZSL inference scenario, carry out meta-learning with these virtual data to extend model. In experiment, demonstrate that AGZSL heneficial to tackle the challenging ZSL/GZSL problem and achieves significant advantages those in comparison the four 7SL/G7SS benchmark datasets.
ACKNOWLEDGEMENTS
This work was supported in part by the MOST, Taiwan under Grants 110-2634-F-001-099 and 107-2628- 22888--00-000--M-- We are grateful to the National Center for High-performance Computing for providing computational resources and facilities.
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for attribute-baue based classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 819. 2013.

---

Published as a conference paper at ICLR 2021
Yashas Annadani and Soma Biswas. Preserving semantic relations for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7603-7612, 2018.
Yuval Atzmon and Gal Chechik. Adaptive confidence smoothing for generalized zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1167
11680, 2019.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In European Conference Computer Vision, pp. 52-68. Springer, 2016.
Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recogni- tion using semantics-preserving adversarial embedding networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1043-1052 2018.
Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng. A boundary based out-of-distribution classifier for generalized zero-shot learning. In Proceedings of the European conference on computer vision (ECCV), 2020.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1785, IEEE, 2009.
Rafael Felix, Vijay BG Kumar, Ian Reid, and Gustavo Carneiro. Multi-modal cycle-consistent generalized zero-shot learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 21-37, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1126-112 1135. PMLR, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems. pp.
2014.
Maxine Greene. Releasing the imagination: Essays on education, the arts, and social change. Jossey-Bass, 1995.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Lily Hu, Caiming Xiong, and Richard Socher. Correction networks: Meta-learning for zero-shot learning. 2018.
Dat Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute- based attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4483 4493,4433,44 2020.
Zhong Ji, Yanwei Fu, Jichang Guo, Yanwei Pang, Zhongfei Mark Zhang, et al. Stacked semantics- guided attention model for fine-grained zero-shot learning. In Advances in Neural Information Processing Systems, pp. 5995-6004, 2018.
Huajie Jiang, Ruiping Wang. Shiguang Shan, and Xilin Chen. Learning class prototypes via structure alignment for zero-shot recognition. In Proceedings of the European conference on computer vision (ECCV), pp. 118-134, 2018
Rohit Keshari, Richa Singh, and Mayank Vatsa. Generalized zero-shot learning via over-complete distribution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrnn Recognition, pp. 13300-13308, 2020.
10

---

Published as a conference paper at ICLR 2021
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.). 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14- 46, 2014, Conference Track Proceedings. 2014.
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. Generalized zero-shot learning via synthesized examples. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4281 4-289, 2018.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning detect unseen object classes by between-class attribute transfer. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 951-958. IEEE, 2009.
Jin Li, Xuguang Lan, Yang Liu, Le Wang, and Nanning Zheng. Compressing unknown images with product quantizer for efficient zero-shot classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5463-5472, 2019a.
Kai Li, Martin Renqiang Min, and Yun Fu. Rethinking zero-shot learning: A conditional visual classification perspective. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3583-3592 2019b.
Yang Liu, Jishun Guo, Deng Cai, and Xiaofei He. Attribute attention for semantic disambiguation in 7ero-shot learning. In Proceedings of the IEEE International Conference Computer Vision, pp. 6698-6707, 2019.
Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zheng-Jun Zha, and Yongdong Zhang Domain-aware visual bias eliminating for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1664-12673, 2020.
Ashish Mishra, Shiva Krishna Reddy, Anurag Mittal, and Hema A Murthy. A generative model for zero shot learning using conditional variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2188-21966 2018.
Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees G. M. Snoek, and Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. In Proceedings of the European conference computer vision (ECCV), 2020.
Mark Palatucci, Dean Pomerleau, Geoffrey Hinton, and Tom M Mitchell. Zero-shot learning with semantic output codes. In Advances in neural information processing systems, pp. 1410-144 1418, 2009.
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recogniz- ing scene attributes. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2751-2758. IEEE, 2012.
Akanksha Paul, Narayanan Krishnan, and Prateek Munjal. Semantically aligned bias reducing zero shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7056-7065, 2019.
Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 8247-8255, 2019.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 1199-1208, 2018.
Bin Tong, Chao Wang, Martin Klinkigt. Yoshiyuki Kobayashi, and Yuuichi Nonaka. Hierarchical disentanglement discriminative latent features for zero-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 11467-11476, 2019.
Vinay Kumar Verma, Dhanajit Brahma. and Piyush Rai. Meta-learning for generalized zero-shot learning. In AAAI, pp. 6062-6069 2020.

---

Published as a conference paper at ICLR 2021
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Daniel D. Lee, Masashi Sugiyama, Ulike von Luxburg. Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3338,-63,,--- 2016.
Maunil Vyas, Hemanth Venkateswara, and Sethuraman Panchanathan. Leveraging seen and unseen semantic relationships for generative zero-shot learning. In Proceedings of the European conference on computer vision (ECCV), 2020.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4582-4591, 2017.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-226; 2018a.
Yongqin Xian, Tobias Lorenz, Bernt Schicle, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5542-5551, 2018b.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generating framework for any-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10275-10284 2019.
Guo-Sen Xie, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang. Jie Qin, Yazhou Yao, and Ling Shao. Attentive region embedding network for zero-shot learning. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9384-9393, 2019.
Yunlong Yu, Zhong Ji, Jungong Han, and Zhongfei Zhang. Episode-based prototype generating network for zero-shot learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14035-14044, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zero-shot learning from noisy texts. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1004-1013, 2018.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
Table
The impact of shortcut in IAS on four datasets.
AWAZ
aBY
shortcut
7SL.
GZSL.
ZSL.
GZSI.
7SL.
GZSI.
7SL.
CZZSI
S
S
S
S
23.9
13.0
30.0
18.2
23.3
17.0
45.8
24.8
42.1
22.5
20.3
21.4
30.5
14,9
4.7
7.1
76.4
69.0
86.5
76.8
77.2
69.2
76.4
72.6
66.2
50.5
43.1
46.5
43.7
36.2
58.6
44.8
100
number zomposs
399
aifforoncc of cosine simiorty
dfference of todnee simitarty
afference of costne smiofity
Smforty aPY samples
dfferencce of codlnc cmmrrm
AMA2 samples
(b) CUB samples
(c) SUN samples
345
335
1:0
11h difwence aimilerity
difference similerly
difeeeece costne amilnit
diference comtre simientty
(c) ANA2 seen samples
(f) CUB samples
SUN samples
(h) aPY seen samples
Figure 6: The difference of cosine similarity between our model and baseline. In our experiment, most of the samples improve their embedding with IAS and virtual classes
Shorteut in IAS Table 3 shows that shortcut in IAS improves performance lot. The unseen class accuracy increase around 50% on AWA2 and SUN. Besides, the harmonic mean also enhances 20% 60% four datasets with the shortcut. The main reason is that the softmax usually puts high attention weight on few dimensions and decreases others near to zero. Thus, without shortcut, the model prone overfitting on training data.
Cosine similarity enhancement Figure @ illlstraaes better embedding for seen and unseen classes in our model. Since apply cosine similarity classify samples, the higher representation of each class is closer to their visual samples. The first row in shows effect of our model on unseen classes. With virtual classes and IAS, cosine similarity increase averaged 0.05 to 0.1 on different datasets. The increasing amount shows our model provides better visual-to-semantic embedding for unseen classes and samples are easier to classify correctly. Therefore, our model can achieve good ZSL performance. Besides, the second row in iillustrate the effect of IAS on seen classes. Most of the testing seen samples increase their cosine similarity around 0.02 on four datasets. With the IAS and virtual classes, the model can achieve better visual-to-semantie embedding on all classes and obtain better the generalized seen accuracy on four datasets.
means visual
virtual classes number per episode To search the class number in each episode for best ZSL, we search different numbers, i.e, 4,8,12 12, and per episode. 4able/[show that the best class number setting is related to the class number of classes. For example, the AWA2 and aPY which include fewer unseen classes obtain the best ZSL accuracy with 16 virtual classes. On the other hand, the CUB and SUN which include more unseen classes achieve the best performance with 20 virtual classes. The reason is that the advantage meta-learnigg is to simulate inference scenarios while training (Li et al. 2019b). The virtual class number closer to the unseen class number is favorable in the meta-learning setting. Therefore, the best virtual class number different according to the unseen number of the dataset.
13

---

Published as a conference paper at ICLR 2021
5
B(5,5)
B(1,1)
B(5,1)
3)
3
(
2
0.0
0.2
0.4
0.6
0.8
1.0
Random Variable
Figure 7: Beta distribution.
Table episode.
ZSL accuracy w.r.t virtual classes per
Table 5: ZSL accuracy w.r.t beta distribution.
AWA2
CUB
SUN
aPY
B(a,3)
AWA2
CUB
SUN
aPY
75.7
76.2
64.6
40.8
(5,5)
74.8
75.6
65.4
38.2
76.0
76.3
65.1
42.8
(1,1)
75.3
76.4
65.8
40.2
16
76.4
76.3
65.1
43.7
(5.1)
76.4
77.2
66.2
43.7
75.5
77.2
66.2
41.1
Effect of beta distribution on ZSL accuracy explore the best beta distribution setting for ZSL, our experiments apply three different a and f sets. As shown in Figure 7 B(1,1) means virtual classes distribute uniformly between selected class pairs. Then, B(5 means most of the virtual classes distribute among the middle of selected class pairs. On the other hand, B(5,1) means most virtual classes distribute closer to one the classes in selected class pairs. Table 5 demonstrates that B(5,1) achieves best performance based on one class are closer to real-world classes. For example, the zebra is very similar to the horse in shape. Thus, the virtual classes distributing among classes enhance ZSL. accuracy very beneficial for training ZSL.
each dataset. In speculation, the virtual classes created
14


Reviews:
[{"review_id": "yABHy-V4BAq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I. Summary\n\nThe authors consider (generative) zero-shot classification. Their approach, combines two main aspects: (1) generating \"virtual\" classes by mixup interpolation and (2) they introduce an attention mechanism, dubbed \"image-attentive attention\" to allow their approach to model both intra- and extra- class variations.\n\nII. Strong and weak points.\n\nPositive: \n- Strong results: overall the results are good. Yes, the paper may not account for the latest crop of papers from e.g. CVPR but I feel that is besides the point as these methods are structurally different.\n- Relatively extensive evaluation: there is an ablation study, and a distinction over whether the image encoder is fine-tuned.\n- Writing is clear, paper is reasonably easy to follow.\n\nNegative:\n- A large part of the improvement stems from the virtual classes, which is essentially mixup applied to ZSL. This is somewhat derivative.\n\nIII. Rating and questions to authors\nDespite the negative comment I mentioned, I feel the paper should be accepted. How does the paper work with actual generated samples as in the GAN/VAE-based models? Do these extra samples work well with the virtual samples? Is the information redundant/complementary?", "extra_scores": {}, "extra_text": {}}, {"review_id": "f1F0sD9ejJy", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework for the GZSL using the meta-learning and attention mechanism. The image-guided attention on the semantic space helps to adapt the better class specific semantic information. The modified semantic space projected to the visual space and in the visual space, cosine similarity is measured. The paper learns separate expert for the seen and unseen classes. The unseen class expert is trained with the pseudo negative samples with pseudo negative labels. Meta-learning based training helps to learn the model when only a few examples per class are available.\n\nComment: \n1: The adaptive modification of the semantic space is novel in the ZSL setup with clear intuition. Also, the mixup idea in the ZSL setup seems to work well, it helps to separate the seen and unseen class data hence choosing the seen/unseen expert is easy. \n2: The paper shows a strong result on the standard ZSL dataset. The ablation over the various component shows the efficacy of the proposed component. \n\n3: How meta-learning is used in the paper is not clear also in the implementation details it's not mentioned. What is the task definition? How meta-train and meta-test are defined? The paper mention they use meta-learning, but nothing is clear about this. Therefore reproducibility is the main issue, and I think using the provided information the reported result can not be reproduced. Please update the experimental setup and provide the detail so that result can be easily reproduced.\n\n4: There are a few recent works [a,b,c,d] that explore the meta-learning framework for the ZSL setup. The author should mention these paper and add the advantage of the proposed model over the other meta-learning approach also these result should be compared. Note: [d] is mentioned in the paper. Also please add an ablation if the model has advantage using the meta-learning, i.e. what is the result of the proposed component if we use the simple network without meta-learning.\n\n[a] Meta-Learning for Generalized Zero-Shot Learning, AAAI-20\n[d] Episode-based prototype generating network for zero-shot learning, CVPR-20\n[b] Learning to Compare: Relation Network for Few-Shot Learning, CVPR-18\n[c] Correction Networks: Meta-Learning for Zero-Shot Learning, ArXiv\n\n5: Reproducibility is the main challenge here I request the author, please provide the code for the CUB dataset (since its look too high, I suspect the result in the inductive setting).\n\n6: In the table-1, what is T for the ZSL result? Is it transductive setting?\n\nOverall I like the idea, and paper shows a good result. Reproducibility is the main challenge, and CUB results look too high, I like to verify the result, therefore requesting the author to submit the code for the CUB dataset. Also, please discuss the suggested paper that is also using meta-learning based training. I will update my score on the successful verification of the code and result and the updated paper quality.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "d956RA2nEAL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Due to time shortage this will be a short review. I have gone through the paper carefully.\nMotivation\nThe authors motivate their work well and have a thorough literature survey.\nMethod\nThe authors essentially create clusters that can anticipate unseen classes. That idea is not new in and of itself but the authors' realization of the idea through mix-ups and image-adaptive semantics is new and interesting. The overall approach is technically sound and each component is well motivated and described.\n\nResults\nThe results are convincing. The authors get an across the board improvement over the state of the art.\nClarity\nThe paper is clearly written and has a good logical flow. I would recommend not using adjectives like elegant and insightful for one's own work. Perhaps such assessments are best left to the reviewers.\nQuality, originality and significance\nThe paper presents moderate innovation in my view. It is thorough and hence does deserve consideration. In short, good quality, reasonable originality and decent significance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CEeF3wtaskH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper solves ZSL and GZSL problem by an elegant fusion of adaptive and generative learning. Different from previous generative models that synthesize unseen samples for training the model, authors create virtual classes as unseen classes in the training phase. In addition, four standard GZSL datasets demonstrate the effectiveness of the proposed method. \nStrengths:\n1. The motivation and the contribution are clearly presented in this manuscript. In general, this paper is well-written.\n2. Instead of generating synthetic data of unseen classes, authors propose to yield virtual classes and data by mixup interpolations.\n3. The architectures of IAS and S2V are simple and effective in terms of experimental results.\nWeaknesses:\n1. Parameter settings for the network are unclear.\n2. More generative ZSL methods from CVPR’20 or ECCV’20 should be compared.\n3. More experimental results such as a tsne illustration for generated virtual classes should be given.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Hwqf6mgXl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors proposed an interesting method for zero-shot learning. In particular, the authors adopted an attention mechanism from the input feature in the semantic to visual mapping, to introduce intra-class variations in the visual space. They also propose a process to synthesize \"fake\" class representations such that a classifier for unseen classes can be trained.  Combining these two the authors demonstrated significant results on benchmark zero-shot learning datasets.\n\nReason for score: I think this paper is marginally below the acceptance threshold. It proposed an elegant method which produces state-of-the-art results on benchmark datasets. But more details need to be revealed to help readers understand the method (discussed below), and I hope the authors can address my concerns during the rebuttal.\n\nPros:\n1. The paper proposed an interesting attention-based method to introduce intra-class variations to the representations for each class. \n\n2. The authors proposed to synthesize virtual classes to enrich the data for training an unseen classifier.\n\n3. The extensive experiments on the benchmark datasets illustrate the superiority of the proposed method.\n\nCons:\n1. The overall training scheme is not very clear and I hope the authors can clarify them. For example, what is the order of training the seen/unseen experts? Do they share any component, like the IAS or S2V module?\n\n2. Baseline explanation is missing. In Sec. 4.3, the authors demonstrate the ablation study where the baseline model does not have IAS or the virtual classes training scheme. How is this baseline implemented?  Without the virtual classes how would you train the unseen expert? My understanding is that without the IAS the model reduces to pure S2V and without the virtual classes the model can only train on seen classes, in which the entire model reduce to a single cosine similarity classifier in the visual feature space (given the seen features and semantics). Then how do you leverage the seen/unseen expert inference strategy? I am curious about it because this baseline reaches considerably high performance in Table 2, in fact already outperforms many state-of-the-art methods in Table 1. It is kind of shocking to me that such a naive baseline contributes this level of accuracy improvement, if I didn't misunderstand it. I would be appreciated if the authors can deliberate this part, so readers can understand the contribution of each module in the proposed method.\n\n3. Why is seen expert detecting the abnormal? In Sec. 3.3, the authors claim that \"Such optimization (the cross-entropy) can be thought of as one-category anomaly detection over only seen classes\". During training, since the seen expert has no data for unseen classes, the classifier is easily biased to the seen classes, even the new data is from an unseen class in the inference time. That's one of the reasons why many embedding based zsl methods have very poor generalized zero-shot learning performance [Xian et al, cvpr 2017]. Could authors deliberate more on why this specific optimization (the cross-entropy on the seen) can address this issue, such that the model can work as a anomaly detector and predict higher score for unseen classes when unseen example is input? It would also be helpful if the authors can demonstrate the predicted scores of the unseen examples empirically.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission combines meta-learning and attention mechanism for generalised zero-shot learning. The image-guided attention on the semantic space helps to adapt the better class specific semantic information while separate experts operate on the seen and unseen classes. The unseen class expert is trained with the pseudo negative samples with pseudo negative labels. Meta-learning based training adapts the model to few-shot learning scenario. The submission has received two accept, two weak accept and one weak reject reviews. All reviewers found the methodology interesting but they found it moderately novel. The experimental evaluation has been found strong. The rebuttal addressed all the reviewers' concerns and during the discussion phase all reviewers recommended acceptance. The meta reviewer follows the consensus of all the reviewers and recommends acceptance.
Decision: Accept (Poster)