Isometric Propagation Network for Generalized Zero-shot Learning
Authors: Lu Liu
Tianyi Zhou
Guodong Long
Jing Jiang
Xuanyi Dong
Chengqi Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: Zero-shot learning
isometric
prototype propagation
alignment of semantic and visual space

Abstract: Zero-shot learning (ZSL) aims to classify images of an unseen class only based on a few attributes describing that class but no access to any training sample. A popular strategy is to learn a mapping between the semantic space of class attributes and the visual space of images based on the seen classes and their data. Thus, an unseen class image can be ideally mapped to its corresponding class attributes. The key challenge is how to align the representations in the two spaces. For most ZSL settings, the attributes for each seen/unseen class are only represented by a vector while the seen-class data provide much more information. Thus, the imbalanced supervision from the semantic and the visual space can make the learned mapping easily overfitting to the seen classes. To resolve this problem, we propose Isometric Propagation Network (IPN), which learns to strengthen the relation between classes within each space and align the class dependency in the two spaces. Specifically, IPN learns to propagate the class representations on an auto-generated graph within each space. In contrast to only aligning the resulted static representation, we regularize the two dynamic propagation procedures to be isometric in terms of the two graphs' edge weights per step by minimizing a consistency loss between them. IPN achieves state-of-the-art performance on three popular ZSL benchmarks. To evaluate the generalization capability of IPN, we further build two larger benchmarks with more diverse unseen classes and demonstrate the advantages of IPN on them.

Published as a conference paper at ICLR 2021
ISOMETRIC PROPAGATION NETWORK FOR GENERALIZED ZERO-SHOT LEARNING
Lu Liu', Tianyi Zhou', Guodong Long', Jing Jiang', Xuanyi Dong', Chengqi Zhang?
i University of Technology Sydney, & University of Washington
Corresponding to: ww.liu.cs@iclou..o @icloud.com
ABSTRACT
Zero-shot learning (ZSL) aims to classify images of an unseen class only based on a few attributes describing that class but no access to any training sample. A pop- ular strategy is to learn mapping between the semantic space class attributes and the visual space of images based on the seen classes and their data. Thus, an unseen class image can be ideally mapped to its corresponding class attributes. The key challenge is how to align the representations in the two spaces. For most ZSL settings, the attributes for each seen/unseen class are only represented by a vector while the seen-class data provide much more information. Thus, the im- balanced supervision from the semantic and the visual space can make the learned mapping easily overfitting the classes. To resolve this problem, we pro- pose Isometric Propagation Network (IPN), which learns to strengthen the re- lation between classes within each space and align the class dependency in the two spaces. Specifically, IPN learns to propagate the class representations on an auto-generated graph within each space. In contrast only aligning the resulted static representation, we regularize the two dynamic propagation procedures to be isometric in terms of the two graphs' edge weights per step by minimizing a consistency loss between them. IPN achieves state-of-the-art performance on three popular ZSL benchmarks. To evaluate the generalization capability of IPN, we further build two larger benchmarks with more diverse unseen classes, and demonstrate the advantages of IPN on them.
INTRODUCTION
One primary challenge on the track from artificial intelligence to human-level intelligence is to improve the generalization capability of machine learning models unseen problems. While most supervised learning methods focus on generalization to unseen data from training task/classes, zero- shot learning (ZSL) (Larochelle et al. [0088] Lampert et al.) 2014 Xian et al.) 2019a) has a more ambitious goal targeting the generalization to new tasks and unseen classes. In the context of image classification, given images from some training classes, 7SL aims classify new images from unseen classes with zero training image available. Without training data, it is impossible to directly learn mapping from the input space the unseen classes. Hence, recent works introduced learning to map between the semantic space and visual space (Zhu et al. 2019a] Li et al. 2017] Jiang et al. 2018) so that the query image representation and the class representation can be mapped to shared space for comparison.
However, learning to align the representation usually leads to overfitting on seen classes (Liu al., 2018). One of the reasons is that in most zero-shot learning settings, the number of images per class is hundreds while the semantic information provided for one class is only limited to one vector. Thus, the mapping function is casily overfitting to the seen classes when trained using the small set of attributes. The learned model typically has imbalanced performance on samples from the seen classes and unseen classes, i.e., strong when predicting samples from the seen classes but struggles in the prediction for samples in the unseen classes.
In this paper, propose "Isometric Propagation Network (IPN)" which learns to dynamically interact between the visual space and the semantic space. Within each space, IPN uses an attention module to generate category graph, and then perform multiple steps of propagation on the graph.

---

Published as a conference paper at ICLR 2021
bone
hot
cette
kypeer
valia
Epecheree
der
baiver
1008
potphis
dirpang
word
skap
paila
visual class prototypes and the generated graph
(b) semaniic class prototypes the genersted graph
concatenatid dual prototypes
Figure I: I-SNE (Matten & Hinton 2008 of the prototypes produced by IPN on AWA2. Blue/red
points represent unseen/seen classes.
In every step, the distribution of the attention scores generated in each space are regularized to be isometric that implicitly aligns the relationships between classes in two spaces. Due to different motifs of the generated graphs, the regularizer for the isometry between the two distributions are provided with sufficient training supervisions and can potentially generalize better on unseen classes. To get more diverse graph motifs, we also apply episodic training which samples different subset of classes the nodes for every training episode rather than the whole of seen classes all the time,
To evaluate IPN, we compared it with several state-of-the-art ZSL, methods on three popular ZSL. benchmarks, i.e., AWA2 (Xian al. 2019a), CUB (Welinder et al. 20T0) and aPY (Farhadi et al.) 2009]. To test the generalization ability on more diverse unseen classes rather than similar classes to the seen classes, e.g., all animals for AWA2 and all birds for CUB, we also evaluate IPN on two new large-scale datasets extracted from fieredlmageNet (Ren et al. 2018), i.e., Mereeedl-meeeee-Meeeeeee and derrellmageNet-Segregtte
IPN consistently achieves state-of-the-an performance in terms of the harmonic mean of the accuracy on unseen and seen classes. We show the ablation study of each component in our model and visualize the final learned representation in each space in Figure] [ shows the class representations in each space learned by IPN are evenly distributed and connected smoothly between the seen and unseen classes. The two spaces can also reflect each other.
2 RELATED WORKS
Most ZSL. methods can be categorized as either discriminative or generative. Discriminative meth- ods generate the classifiers for unseen classes from its attributes by learning a mapping from input image's 's visual features to the associated class's attributes on training data of seen classes (Frome
ctt al.
2013
Akata et al.] 2015a by Romera-Paredes & Tom 2015 Kodirov et al. 2017: Socher
et al.
2013
Cacheux et al., 2019: Li et al.) 2019b). or by relating similar classes according to
their shared properties e.g.. semantic features (Noruzii al.) 2013 Chi et al.) 2021) or phantom
classes (Changpinyo et al. 2016), etc. In contrast, generative methods employ the recently popular image generation models (Goodfellow al., 2014) synthesize images of unseen classes, which reduces ZSL to supervised learning (Huang et 2019) but usually increases the training cost and model complexity.
addition, the study of unsuperviscd learning (Fan et al. 202 I Liu al. 202I), 2019b a Ding et al. 2020: 2021).
)) or architecture de- sign (Dong et al. | |021 Dong & Yang 2019) can also be used improve the representation ability of visual or semantic features.
meta learning (Santoro et 20,,6016 Liu et al.
Alignment between two spaces in ZSL Frome et al. (2013): Socher et al. (2013); Zhang et al. 2017) proposed to transform representations either to the visual space or the semantic space for distance for comparison. Li et al. (2017) introduces how to optimize the semantic space so that the two spaces can be better aligned. /Jiang et al.] (2018); Liu et al. (2020) tries to map the class representations from two spaces to another common space. Comparatively, our IPN does not align two spaces by feature transformation but by regularizing the isometry of the intermediate output from two propagation model.
Graph in ZSL Graph structures have already been explored for their benefits to some discriminative ZSL models, e.g., [ang et al. 20 Kampfimeyer et al. (2019) learn to generate the weights of fully-connected classifier for a given class based on its semantic features computed pre-
2

---

Published as a conference paper at ICLR 2021
Visual propugation
Selected Edge between strongly related classes
Alention scure between visual prototypes
Dropped Edge between wealll relsted classes
Attention score between semaatic protoly
Distance-hased elassilter
Visual prototype
Interaction fin training)
Concul
shape: large
head: heavy
Querry
beak: heavy
leg: moseular
Semantic propegation
Final
feature
image
Interaction
Semantic
prototype
Visual protutype
Consistency loes
Semantic protoly pe propogation graph
prototype
propagation gruph
The pipeline of IPN. First, we initialize the visual and semantic prototypes from seen classes's s data and class attributes. Then, prototype propagation in each space can be per- formed over multiple steps. Finally, the concatenated prototypes form the support set of distance based classifier. RIGHT: Isometric propagation. we propagate prototypes of each class to other connected classes on the graph using attention module. The attention score is regularized to be isometric.
Figure 2: LEFT:
defined semantic graph using graph convolutional networks (GCN). They require (1) a pre-trained CNN whose last fully-connected layer provides the ground truth of the classifiers ZSL aims to generate; and (2) an existing knowledge graph as the input of GCN. By contrast, our method learns to automatically generate the graph and the class prototypes composing the support set of distance- based classifier. Hence, our method does not require a pre-trained CNN to provide a ground truth for the classifier. In addition, train modified graph attention network (GAN) (rather than GCN) to generate the prototypes, so it learns both the class adjacency and edges by itself and can provide more accurate modeling of class dependency in scenarios when knowledge graph is provided. Tong et al. (2019) and Xie et al. (2020) also leverage the idea of graph to transform the feature extracted from the CNN while our graph serves the paths for propagation. It requires an off-the-shed graph generated by ConceptNet (Speer et al. [016], while IPN applies prototype propagation and learns to construct the graph an end-to-end manner.
Attention in ZSL The attention module has been applied for sero learning for localization.
Liu et al. (2019c) introduces localization on the semantic information and Xie et al. (2019) proposes localization on the visual feature. Zhu et al. (2019b) further extends it to multi-localization guided by semantic. Different from them, our attention serves as generating the weights for propagation.
Our episodic training strategy is inspired by the meta-learning method from (Santoro et al. 2016). Similar methods have been broadly studied in meta-learning and few-shot learning approaches (Finn) et al., 2017, (nell et al., 2017| (ung al. 2018) but rarely studied for ZSL in previous works. In this paper, we show that training ZSL model as a meta-learner can improve the generalization performance and mitigate overfitting.
PROBLEM SETTING
We consider the following setting of 7SL. The training set is composed of data-label pairs (ær, y), where æ E X is a sample from class y, and y must be one of the seen classes in yseen, During the test stage, given test sample æ of class y the ZSL model achieved in training expected to produce the correct prediction of y. In order to relate the unseen classes in test to the seen classes in training, further assume that seen and unseen classes are disjoint, i.e., yreen yunssen = 0. The test samples come from both yeeen and yunseen in generalized zero-shot learning.
also given the semantic embedding sy of each class y yseen U Junseen We
ISOMETRIC PROPAGATION NETWORK
In this paper, bridge the two spaces via class prototypes in both the semantic and visual spaces, ie., a semantic prototype ps and visual prototype P for each class y. In order to generate high-quality prototypes, develop learnable scheme called "Isometric Propagation Network
3

---

Published as a conference paper at ICLR 2021
(IPN)" to iteratively refine the prototypes by propagating them on graph of classes each prototype is dynamically determined by the samples and attributes of its related classes.
In this section, we will first introduce the architecture of IPN. An illustration of IPN is given in Figure - Then, we will elaborate the episodic training strategy, which a consistency loss is minimized for isometric alignment between the two spaces.
4.1 PROTOTYPE INITIALIZATION
Given a task T defined on a subset of classes y² with training data D",, IPN first initializes the semantic and visual prototype for each class y E y²" The semantic prototype of class y is trans- formed from its attributes Sy. Specifically, we have drawn on the expert modules from (Zhang & Shi 2019) to initialize the semantic prototypes Po[0]. previous works (Snell al. 2017 2017:/Sntorro al. 2016), the visual prototype of a class is usually computed by averaging the Teatures of images from the class. IPN, for each seen class y E y² nysee, we initialize its visual prototype Py
by
p 0
W
(x.), DT & ((x)))) D 91 = :yy)) (WL.BIED
Σ
(I)
D
where (.) a backbone convolutional neural nets (CNN) extracting features from raw images, and W linear transformation aiming change the dimension of the visual prototype to be the same as semantic prototypes. We argue that W is necessary because (1) the visual feature produced by the CNN is usually in high dimension, and reducing its dimension can save significant computations during propagation; (2) keeping the dimensions of the visual and semantic prototypes the same can avoid the dominance of either of them in the model.
During test, we initialize the visual prototype of each unseen class )" Numsseen as the weighted sum of its K -eeeeesst classes from yeeen the semantic space
i.e.
pa 0o W. E a (P P.) P. =EN
where W the same transformation from Eq. I and Ng is the set of the top-K (K hyper- parameter) seen classes" with the largest similarities cº (P, [0], P (0]) class y in semantic space.
4.2 CATEGORY GRAPH GENERATION
Although the propagation needs to be conducted category graph, IPN does not rely on any given graph structure as many previous works (Kampffmeyer et al. 2019 Wang al. (018) so it has less limitations when applied to more general scenarios. In IPN, we inference the category graph of yr by applying thresholding to the attention scores between the initialized prototypes. In the visual space, the set of edges is defined by
T. {1,,2) 2) y, DT, (P" 2 2
(3)
where is the threshold. Thereby, the category graph generated as GT:" "TT,ETTT Similarly, in the semantic space, we generate another graph GT.s (yT gT.s).
4.3 ISOMETRIC PROPAGATION
Given the initialized prototypes and generated category graphs in the two spaces, we apply the following prototype propagation in each space for 7 steps in order to refine the prototype of each class. In the visual space, each propagation step updates
p +
Σ
a" ((Pj) PP
1 'This only happens test since we do not use any information unseen classes during training. Here the notation N2 needs little overloading since it denotes the set of neighbor classes of elsewhere the paper.
4

---

Published as a conference paper at ICLR 2021
In the semantic space, apply similar propagation steps classes graph G by using the attention module associated with a a'(.)). Intuitively, the attention module computes the similarity between the two prototypes. In the visual space, given the prototypes Pa and P for class y and 2, the attention module first computes the following cosine similarity:
(P), c¹ c' P.) = [1 | (P))l
(5)
where (-) learnable transformation and || 222 represents the fg norm. For class y on category graph, we can use the attention module to aggregate messages from its neighbor classes Ny. In this case, we need to further normalize the similarities of y to each 2 E Ng as follows.
explyc" (P, .P..P.)) a" (Py P.) ELEN, expl70= (Py P.).
(6)
where y a temperature parameter controlling the smoothness of the normalization. In the semantic space, we have similar definitions of h h-).,, (-.-) and (-;-)) In the following, we will use P and Py |t] represent the visual and semantic prototype of class y the th step of propagation. to represent the size of a set and ReLU(.)) - max
We will use |
After steps of propagation, we concatenate the prototypes achieved in the spaces for each class to form the final prototypes:
py (.... VyEyE.
The formulation of a² (-)) and c* (+.-) follows the same way a (",,)) and c"(s)). To be simple, we don't rewrite the formulations here.
4.4 CLASSIFIER GENERATED FROM PROTOTYPES
Given the final prototypes and a query/test image æ, we apply fully- fully-connected layer to predict the similarity between æ and each class's prototype. For each class E y² it computes
f(x P.y w ReLU(W W + 60)) b
Intuitively, the above classifier be interpreted
distance classifier with distance metric f(; ). When applied to a query image æ, the predicted class is the most probable class, i.e.,
④ arg max Pr(y]æ:; P), where Pr denotes the probability.
4.5 TRAINING STRATEGIES
Episodic training on subgraphs. In each episode, we sample subset of training classes yT C yeeen and then build a training (or support) set DDT train and a validation (or query) set DJ Pealid from the training samples of yT They are two disjoint sets of samples from the same subset of train- ing classes. We only use Dirain to generate the prototypes P by the aforementioned procedures, and then apply backpropagation to minimize the loss of DT vaiid This equals to an empirical risk minimization, i.c.,
min Σ Σ
- log Pr(ylæ;
(9)
yd yyaen (D.N)~ UaIGe
This follows the same idea of meta meta-learning Ravi & Larochelle (016)): it aims to minimize the validation loss instead the training loss and thus can further improve generalization and restrain overfitting. Moreover, since we train the IPN model on multiple random tasks T and optimize the propagation scheme over different subgraphs (each generated for a random subset of classes), the generalization capability of IPN to unseen classes and new tasks can be substantially enhanced by this meta-traine meta-training strategy. Moreover, effectively reduces the propagation costs since computing the attention needs 0(D)"1 operations. In addition, shown in the experiments, episodic train- ing is effective in mitigating the problem of data imbalance and overfitting to seen classes, which harasses many ZSL. methods.
5

---

Published as a conference paper at ICLR 2021
Consistency loss for alignment of visual and semantic space. In ZSL, the interaction and align- ment between different spaces can provide substantial information about class dependency and is critical to the generation of accurate class prototypes. In IPN, we apply prototype propagation in two separate spaces on their own category graphs by using two untied attention modules. In order to spaces consistent, add extra consistency loss to the training objective in Eq. (9) It computes the KL-divergence between two distributions defined in the two spaces for each class y y" at every propagation step t:
keep the propagation in the
log p514= py pi
D P = Σ SEY
(10)
where
explc (P PT[t]]
p][1]= A
(11)
EEYT explc" 1c P P
explc (P: [4 P1(
pf -
(12)
Eveyr explc" (appcc(PPP))
P!(t)] (P
The consistency loss enforces the two attention modules in the two spaces to generate the same at- tention scores over classes, even when the prototypes and attention module's parameter are different in the two spaces. In this way, they share similar class dependency the generated dual prototypes are well aligned and can provide complementary information when generating the classifier. The final optimization we used train IPN becomes:
the propagation steps. So
min Σ
Σ
- log Pr(y|æ: P)+ +
Σ DRL(p](ilp]]
(13)
yT-yoe
(2.g)) unttd
1=1 VEYT
where weight for the consistency loss, and P. pf[t], and pt(t] are computed only based on Draiin the training (or support) set of task-T for classes in yT.
EXPERIMENTS
We evaluate IPN and compare it with several state-of-the-art ZSL models in both ZSL setting (where the test set is only composed of unseen classes) and generalized ZSL setting (where the test set is composed of both seen and unseen classes). We report three standard ZSL evaluation metrics, and provide ablation study of several variants of IPN in the generalized ZSL setting. The visualization of the learned class representation is shown in Figure [
5.1 DATASETS AND EVALUATION CRITERIA
Small and Medium benchmarks. Our evaluation includes experiments on three standard bench- mark datasets widely used in previous works, i.e., AWA2 (Xian al. (019a). CUB (Welinder et al.) 2010) and aPY (Farhadi et al.] 2009). The former two are extracted from ImageNet-IK, where AWA2 contains 50 animals classes and CUB contains 200 bird species. aPY includes 32 classes. Detailed statistics of them can be found in Table
Two new large-scale benchmarks. The three benchmark datasets above might be too small for modern DNN models since the training may suffers from large variance and end-to-end training usually fails. Previous ZSSL methods therefore fine-tune a pre-trained CNN achieve satisfactory performance on these datasets, which is costly and less practical. Given these issues, we proposed two new large-scale datasets extracted from tieredImageNet (Ren et al. 2018). i.e., tieredImageNeet Segregated and fieredImageNet- Mixed, whose statistics are given in Table [ Comparing to the three benchmarks, they are over 10x larger, and training a ResNet101 (He al. 2016) from scratch on their training sets can achieve a validation accuracy of 65%, indicating that end-to-end training without fine-tuning is possible. These two benchmarks differ in that the "Mixed" version allows seen class and an unseen class to belong to the same super category, while the "Segregated" version does not. The class attributes are the word embeddings of the class names provided by GloVe (Pennington et al.) 2014). We removed the classes that do not have embeddings in Glo
Following the most popular setting of ZSL (Xian et al. 2019a),, we report the per-class accuracy and harmonic mean of the accuracy
and unseen classes.

---

Published as a conference paper at ICLR 2021
Dataset
#Attributes
#Seen
#Unseen
#Imgs(Tr-S) #Imgs(Te-S)
#Imgs(Te-U)
CUB (Welinder et al. 2010
312
150
50
7,057
1.764
2.967
AWA2 al. 201 1a
85
40
10
23,527
5,882
7,913
aPY (Farhadi et al. 2009
64
12
5,932
1,483
7,924
dieredlmageNeet-Speee Segregated
300
422
182
378,884
162,952
232,129
dieredlageNNet-Ming
300
157
399,741
171,915
202,309
Table I:
Datasets Statistics. CUB, AWA2 and aPY are three standard ZSL benchmarks, while are new large-scale ones proposed by with different dis-
crepancy between seen and unseen classes. "Tr-S", "Te-S" and "Te-U" denote seen classes in train- ing, seen classes in test and unseen classes in test.
CUB
AWA2
aPY
fiered-M U
tiered-S
Methods
S
U
H
S
U
H
S
U
H
H
S
U
H
GDAN
66.7
39.3
49.5
67.5
32.1
43.5
75.0
30.4
43.4
50.5
2.1
4.0
3,9
1.2
1.8
LisGAN
57.9
46.5 51.6
76.3
52.6
62.3
68.2
34.3
45.7
1.6
0.1
0.2
0.2
0.1
0.1
Generative
CAD
53.5
51.6 52.4
75.0
55.8 63.9
60.1
3.5
6.5
0.8
0.1
0.2
Models
E-VAGAN-D2
60.1
48.4 53.6
70.6
57.6 63.5
PIFT
54.8
47.0
50.6
72.6
55.3 62.6
Relation Net
61.1 38.1 47.0
93.4
30.0 45.3
31.4 3.4 6.1
45.8 1.2 2.3
GAFF
52.1
22.5 31.4
78.3
26.8 40.0
68.1
15.8
25.7
PQZSI.
51.4
46.9
64.1
27.9
38.8
Discriminative Models
MLSE
71.6 22.3 34.0
83.2 23.8 37.0
74.3
12.7 21.7
CRNet
56.8 45.5 50.5
78.8
52.6 63.1
68.4
32.4 44.0
58.0
2.5 4.9
57.6 1.0 2.0
TCN
52.0 52.6 52.3
65.8 61.2 63.4
64.0
24.1 35.1
RGEN
73.5 60.0
66.1
76.5 67.1 71.5
48.1
30.4
37.2
APN
55.9
51.7
83.9
54.8 66.4
74.7
32.7
45.5
IPN (ours)
73.8
60.2 66.3
79.2
67.5
72.9
66.0
37.2
47.6
50.1
6.1 11.0 54.4 2.5 4.7
Table 2: Performance of existing ZSL models and IPN on five datasets (generalized ZSL setting),
where "S" denotes the per-class accuracy (%) on seen classes, "U" denotes the per-class accuracy (%) unseen classes and "H" denotes their harmonic mean.
5.2 IMPLEMENTATION AND HYPERPARAMENE DETAILS
For fair comparison, we follow the setting in Xian et al. 2019a) and use a pre-trained ResNet- 101 (He et al. 2016) extract -i48-dinensional image features without fine-tuning. For the three standard benchmarks, the model is pre-trained on ImageNet-1K K. For the two new datasets, it is pre-trained on the training set used train 7SL. models.
All hyperparameters were chosen on the validation sets provided by Xian et al. (2019a). We use the same hyperparameters tuned on AWA2 for other datasets since they are of similar data type, except for aPY on which we choose learning rate 1.0 x 10-3 and weight decay 5.0 x 10-4 For the rest of the datasets, we train IPN by Adam (Kingma& & Ba 2015) for 360 epochs with a weight decay factor 1.0 x 10 The learning rate starts from 2.0 10 and decays by multiplicative factor 0.1 every 240 epochs. In each epoch, we update IPN on multiple N-way-K-shha samples per class), each corresponding the number of episodes each epoch is M/NK, where the total size of the training set, 30 and K 1 (a small K can mitigate the imbalance between seen and unseen classes); h² and h² linear transformations; we set temperature = 10, threshold = cos 40 for the cosine similarities between class prototypes, and weight for consistency loss 1. We use propagation steps 2.
tasks (N-classes with K episode mentioned in Section 4.5 In our experiments,
5.3
MAIN RESULTS
The results are reported in Table 2 Although most previous works (Xian et al., 2019b a Sariyildiz & Cinbis, 2019) show that the more complicated generative models usually outperforms the discrim- inative ones. Our experiments imply that IPN, as a simple discriminative model, can significantly outperform the generative models requiring much more training efforts. Most baselines achieve

---

Published as a conference paper at ICLR 2021
high ACCseen seen but much lower ACC unseen since they suffer from and more sensitive to the imbal- ance between seen and unseen classes. In contrast, IPN always achieves the highest H-metric among all methods, indicating less overfitting to seen classes. This is due to the more accurate prototypes achieved by iterative propagation in two interactive spaces across seen and unseen classes, and the episodic training strategy, which effectively improves the generalization.
In scaling the experiments to the two large datasets, discriminative models perform better than gen- erative models, which are easily prone to overfitting divergence. Moreover, during the test stage, generative models require to synthesize data for unseen classes, which may lead to expensive compu- tations when applied to tasks with many classes, In contrast, IPN scales well to these large datasets, outperforming state-of-ther thetthe-rt baselines by large margin. The performance of most models on -eweeaaeeeeeeeeeaaaallaaaa
is generally worse than aagre classes in the former do not share the same ancestor classes with the seen classes, while in the latter such sharing allowed.
because that the unseen
5.4 ABLATION STUDY
Are two prototypes necessary? We compare IPN with its two variants, each using only one out of the two types of prototypes from the two spaces when generating the classifier. The results are shown in Figure It shows that semantic prototype (only) performs much better than visual prototype (only), especially on the unseen classes. This is because that the unseen classes' semantic embedding is given while its visual information is missing (zero images) during the test stage. Although IPN is capable of generating a visual prototype based on other classes' images and class dependency, much less accurate than the semantic prototype. However, it also shows that the visual prototypes (even the ones of unseen classes) can provide extra information and further improve the performance of the semantic prototype, e.g., they improve the unseen classes' accuracy from 61. % to 67.5 57.5%. One possible reason is that the visual prototype bridges the query image's visual feature with the semantic features of classes, and thus provides extra "visual context" for the localization of the correct class. In addition, we observe that IPN with only the visual prototype converges faster than IPN with only semantic prototype (30 epochs vs. 200 epochs). This might be an artifact caused by the pre-trained CNN, which already produces stable visual prototypes since the early stages. However, raises a caveat that heavily relying on a pre-trained model might discourage the learning and results in a bad local optimum.
H
H
S
75
60
60
50
30
30
Semantic
Visual
Visual
Semantic
Semuntic
Visual
Visual
Semantic
Semantic
Visual
wlo Consistency
Effect of Prototypes
Effect of Propagation
Effect of Isometric Alignment
Figure 3: Ablation study of IPN on AWA2 (generalized ZSL setting). "S", "U" and "H" the same notations used in Table 2 (a) Only one of the dual prototypes are used to generate classifiers. (b) Only one or none of the propagation in the spaces are applied in IPN. (c) Different types of interaction between spaces. "visual/semantic attention only" means we use visual/semantic attention in both spaces, "two separate attentions" refers to IPN trained without consistency loss.
How's the benefit of propagation? We then evaluate the effects of the propagation scheme in the two spaces. In particular, we compare IPN with its variants that apply the propagation steps in only one of the two spaces, and the variant that does not apply any propagation after initialization. Their performance is reported in Figure 3 Without any propagation, the classification performance heav- ily relies on pre-trained CNN. Hence, the accuracy of seen classes can be much higher than that of the unseen classes. Compared to IPN without any propagation, applying propagation in either the visual space or the semantic space can improve the accuracy of unseen classes by 2~33% while the accuracy on the seen classes stays almost the same. When equipped with dual propagation,
8

---

Published as a conference paper at ICLR 2021
observe significant improvement of the accuracy of the unseen classes but degradation the seen classes. However, they lead to the highest H-metric, which indicates that the dual propaga- tion achieves desirable trade-off between overfitting seen classes and generalization to unseen classes
Effect of Isometric Alignment In IPN, minimize consistency loss during training for align- ment the visual and semantic space. Here, we consider two other options for alignment that exchange the attention modules used in the two spaces, i.e., using the visual attention module in the semantic space, or using the semantic attention module in the visual space. We also include the IPN trained without consistency loss or any alignment as a baseline. Their results are reported in Figure 3 Compared to IPN with/without consistency loss, sharing single attention module be- tween the two spaces suffers from the overfitting problem on the seen classes. This because that the visual features and semantic embeddings are two different types of features so they should not share the same similarity metric. contrast, the consistency loss only enforces the output similarity scores of the two attention modules to be close, while the models/functions computing the similarity can different. Therefore, to produce the same similarity scores, it does not require each class having the same features in the two spaces. Hence, it leads to much softer regularization.
SENSITIVITY OF HYPERPARAMETERS
73.0
70
harmonic mean
60
72.5
50
72.0
episodic
40
71.5
30
71.0
20
20
30
40
50
60
70
80
90
10
15
20
25
30
0
3
the threshold (e)
the temperature (y)
The value of K (30-way- k-soo)
73.0
72.5
harmonic meean
72.5
72.0
67.5
71
71.5
65.0
70
62.5
70.5
60.0
npi-apic
57.5
67
66
35
40
2
2.0e1 4.0e4 6.0e1 8. 8.0e4 10.0et 00. The welght decay
The value of (N-way-1-shot)
The propagation steps (+)
Figure 4: Performance analysis when using different values of hyperparameters AWA2.
The proposed IPN has some hyperparameters as mentioned in Section 5.2 To analyze the sensi- tivity of those hyperparameters, such threshold, temperature, and K for N-way-K-shot task, propagation steps, etc. We try different values and show the performance on the AWA2 dataset in Figure The temperature 7 is not sensitive for the accuracy. The threshold needs be bigger than (osine(30") to give IPN some space for the filtering. The propagation steps T is best at 2. Weight decay is sensitive hyperparameter and this is also found in previous ZSL literature (Zhang & Shi
2019
Liu et al. 2019d).
CONCLUSIONS
In this work, we propose Isometric Propagation Network, which learns to relate classes within each space and across two spaces. IPN achieves ofate-of-the-art performance and we show the importance of each component and visualize the interpretable relationship of the learned class rep- resentations. In future works, we will study other possibilities to initialize the visual prototypes of unseen classes in IPN.

---

Published as a conference paper at ICLR 2021
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. image classification. TPAMI, 2015a. doi: 10.1100/PPAM22215522279999
Label-embedding for
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR, 2015b.
Yannick Le Cacheux, Herve Le Borgne, and Michel Crucianu. Modeling inter and intra-class rela- tions in the triplet loss for zero-shot learning. In ICCV, 2019.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero- shot learning. In CVPR, 2016.
Soravit Changpinyo, Wei-Lun Chao, and Fei Sha. Predicting visual exemplars of unseen classes for zero-shot learning. In ICCV, 2017.
Haoang Chi, Feng Liu, Wenjing Yang, Long Lan, Tongliang Liu, Gang Niu, and Bo Han. Meta discovery: Learning to discover novel classes given very limited data, 2021.
Kaize Ding, Jianling Wang, Jundong Li, Kai Shu, Chenghao Liu, and Huan Liu. Graph proto-
typical networks for few-shot learning on attributed networks. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020.
Kaize Ding, Qinghai Zhou, Hanghang Tong, and huan Liu. Few-shot network anomaly detection via cross-network meta-learning. Proceedings of The Web Conference 2021, 2021.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In CVPR, 2019.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-Bench: Benchmarking nas algorithms for architecture topology and size. TPAMI, 2021. doi: 10. 20.1009//AAMM..21..05322
Hehe Fan, Ping Liu, Mingliang Xu, and Yi Yang. Unsupervised visual representation learning via dual-level progressive similar instance selection. TCYB, 2021. doi: 10.1109/TCYB.222.. 3054978.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML. 2017.
Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A deep visual-semantic embedding model. In NeurlPS, 2013.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
He Huang, Changhu Wang, Philip S Yu, and Chang-Dong Wang- Generative dual adversarial net- work for generalized zero-shot learning. In CVPR, 2019.
Huajie Jiang, Ruiping Wang. Shiguang Shan, and Xilin Chen. Learning class prototypes via structure alignment for zero-shot recognition. In ECCV, 2018.
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero-shot learning. In CVPR, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Elyor Kodirov, Tao Xiang, and Shaogang Gong. Semantic autoencoder for zero-shot learning. In CVPR, 2017.
10

---

Published as a conference paper at ICLR 2021
Christoph H Lampert. Hannes Nickisch, and Stefan Harmeling. Attribute-basel classification for zero-shot visual object categorization. TPAMI, 2014. doi: 10.1199/PPAM..2033.4444
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In IJCAI, 2008.
Jingjing Li, Mengmeng Jin, Ke Lu, Zhengming Ding, Lei Zhu, and Zi Huang. Leveraging the invariant side of generative zero-shot learning. In CVPR, 2019a.
Kai Li, Martin Renqiang Min, and Yun Fu. Rethinking zero-shot learning: A conditional visual classification perspective. In CVPR, 2019b.
Yanan Li, Donghui Wang. Huanhang Hu, Yuetan Lin, and Yueting Zhuang. Zero-shot recognition using dual visual-semantic mapping paths. In CVPR, 2017.
Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learn- ing deep kernels for non-parametric two-sample tests. In ICML, volume 119, pp. 6316-6326, 2020.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Prototype prop- agation networks (PPN) for weakly-supervised few-shot learning on category graph. In IJCAI, 2019a.
Lu Liu, Tianyi Zhou, Guodong Long. Jing Jiang, and Chengqi Zhang. Learning to propagate for graph meta-learning. In NeurlPS, 2019b.
Shichen Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Generalized rero-shot learning with deep calibration network. In NeurlPS, 2018.
Yang Liu, Jishun Guo, Deng Cai, and Xiaofei He. Attribute attention for semantic disambiguation in zero-shot learning. ICCV, 2019c.
Yang Liu, Deyan Xie, Quanxue Gao, Jungong Han, Shujian Wang, and Xinbo Gao. Graph and autoencoder based feature extraction for zero-shot learning. In IJCAI, 2019d.
Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu. Graph self-supervised learning: A survey, 2021.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 2008
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. In NeurIPS, 2013.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2016.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. In ICLR, 2018.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach zero-shot learn- ing. In ICML, 2015.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In JCML, 2016.
Mert Bulent Sariyildiz and Ramazan Gokberk Cinbis. Gradient matching generative networks for zero-shot learning. In CVPR, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NewrIPS, 2017.
11

---

Published as a conference paper at ICLR 2021
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In NeurlPS, 2013.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. arXiv preprint arXiv:1612.03975. 2016.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, 2018.
Bin Tong, Chao Wang, Martin Klinkigt, Yoshiyuki Kobayashi, and Yuuichi Nonaka. Hierarchical disentanglement of discriminative latent features for zero-shot learning. In CVPR, 2019.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. CVPR, 2018
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pictro Perona. Caltech-ucsd birds 200. Technical report, California Institute of Technology, 2010.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a com- prehensive evaluation of the good, the bad and the ugly. TPAMI, 2019a. doi: 10.1109/TPAMI 2018.2857768.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generat- ing framework for any-shot learning. CVPR, 2019b.
Guo-Sen Xie, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, and Ling Shao. Attentive region embedding network for zero-shot learning. In CVPR, 2019.
Guo-Sen Xie, Li Liu, Fan Zhu, Fang Zhao, Zheng Zhang, Yazhou Yao, Jie Qin, and Ling Shao. Region graph embedding network for zero-shot learning. CVPR, 2020.
Fei Zhang and Guangming Shi. Co-representation network for generalized zero-shot learning. In ICML, 2019.
Li Zhang, Tao Xiang, and Shaogang Gong. Learning deep embedding model for zero-shot learning. In CVPR, 2017.
Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Generalized zero-shot recognition based on visually semantic embedding. In CVPR, 2019a
Yizhe Zhu, Jianwen Xie, Zhiqiang Tang, Xi Peng, and Ahmed Elgammal. Semantic-guided multi- attention localization for zero-shot learning. In NeurlPS, 2019b.
A EVALUATION CRITERIA
Considering the imbalance between test classes, by following the most recent works (Xian et al. 2019a), given a test set D, we evaluate ZSL methods' performance based on the per-class accuracy averaged over targeted set of classes ), i.e.,
1 y
1
ACCy
1 y],
(14)
D2y
(x,))ED,
D,
( yi)
D yi =y}
Following Xian al. (2019a), we report ACCy when y Vxeen and y Vunseens and the harmonic mean of them, i.c.,
H = 2. ACCy re.n AAC
(15)
ACC) + ACC 20Cyanneen
12

---

Published as a conference paper at ICLR 2021
Methods
CUB
AWA2 aPY tiered-M tiered-S
Generative Models
LisGAN (Li et al.
2019a)
58.8
70.6
43.1
1.6
0.1
f- 1-VAHAA--D2 Xian et al.
2019b)
61.0
71.1
SAE Kodirov
2017
33.3
54.1
8.3
DEM ZZang et al. 2017
51.7
67.1
35.0
Discriminative Models
RN (Sung al. 20 (8)
55.6
64.2
30.1
6.1
2.5
GAFE (Liu al.
2019d)
52.6
67.4
44.3
IPN (ours)
59.6
74.4
42.3
14.0
5.0
Table 3: Per-class accuracy (%) unseen classes achieved by existing ZSL models and IPN on five datasets (the ZSL setting).
B EXPERIMENTS ON ZERO-SHOT LEARNING SETTING
The comparison between IPN to other baselines on the setting of Zero-shot Learning is shown in Table 3 On CUB, AWA2, aPU, tiered-M, and tiered-S, our IPN outperforms other algorithms by large margin.
C MORE EXPERIMENTS ON LARGE-SCALE DATASETS
In our work, we follow the train/test split suggested by Frome et al. (2013), who proposed to use the 21K ImageNet dataset for zero-shot evaluation. Here, 2-hops and 3-hops represent that the evalua- tion is over the classes which are 2/3 hops away from ImageNet- Ik. Different from experiments AWA2, we use N=20 and K=5 for the full ImageNet. Besides, since the edges between each class provided on the full ImageNet, directly use them instead of calculating by Eq.
Hit@k (%)
[2-hops]
Hit@k k (%)
[3-hops]
Methods
5
10
20
5
10
20
ConSE
Norouzi ct al. 2013)
8.3
21.8
30.9
41.7
2.6
7.3
11.1
16.4
SYNC
Changpinyo et al. 2016)
10.5
28.6
40.1
52.0
2.9
9.2
14.2
20.9
EXEM
Changpinyo al. 2017)
12.5
32.3
43.7
55.2
3.6
10.7
16.1
23.1
GCNZ (Wang et al. 2018
19.8
53.2
65.4
74.6
4.1
14.2
20.2
27.7
DGP
Kamptfmeyer et
2019)
26.6
60.3
72.3
81.3
6.3
19.3
27.7
37.7
IPN (ours)
27.1
61.1
73.8
82.9
6.8
20.1
28.9
39.6
Table 4:
Top-k accuracy for the different models on the ImageNet dataset. Accuracy when only testing on unseen classes. Results are taken from (Kampffmeyer et al. 2019).
13


Reviews:
[{"review_id": "qe842yvj7bQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an Isometric Propagation Network (IPN) for ZSL. The proposed IPN aims to learn to strengthen the relation between classes within each space and align the class dependency in the two spaces. IPN achieves the state-of-the-art performance on three popular ZSL benchmarks.\n\nStrengths:\n1. The explanation of the objective function and the main formula is clear.\n2. The introduction of related work is very organized.\n\nWeaknesses:\n1. The motivation and the advantages of IPN are unclear.\n2. Most compared methods are out of data. Some generative or discriminative ZSL methods proposed in CVPR’20 or ECCV’20 are not compared.\n3. More experimental results besides ZSL results and H value should be given to show the advantages of IPN.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mHPhYnmbUP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Paper Summary*\nThe authors propose a novel computational pipeline to tackle a well-known problem in zero-shot learning: although multiple visual instances are available for the classes and categories to be recognized, one and only one semantic embedding is available to describe the classes/categories while using side information like attributes or relevant textual information. To cope with that problem, authors learn visual and semantic prototypes which are then adopted to perform gradient descent over a graph in which the topological relationship among similar/dissimilar classes are preserved. In the experimental validation, the proposed method shows its superiority among a number of prior methods in zero-shot learning, including discriminative and generative methods. \n\n*Pros*\n* The computational pipeline is novel and very original with respect to prior work, which mainly focuses on feature generating mechanisms\n* Several state-of-the-art approaches are outperformed on 3 classical benchmark datasets for zero-shot learning (AWA2, CUB, aPY)\n\n*Cons*\n* **Sensitivity**. The method is depending upon a number of hyper-parameters (such as the threshold $\\varepsilon$ for the visual/semantic neighbor graphs or the softmax temperature $\\gamma$). \n* **About datasets**. Although tiered-ImageNet is a popular benchmark in few-shot learning, there are methods which provide experimental results on the full ImageNet, such as [Kampffmetyer et al., Rethinking Knowledge Graph Propagation for Zero-Shot Learning, CVPR 2019]. It is not fully clear to which extent the proposed variants of tiered-ImageNet, adapted for zero-shot learning, would improve upon the existing ImageNet specs. \n\n*Minor Comments*\nThe references (Xian et al. 2019a) and (Xian et al. 2019b) refer to the same paper. \n\n*Pre-Rebuttal Evaluation* I think the method is novel and original, and it is adopting a very original direction which is dissimilar for mainstream approaches in zero-shot learning. My only questions relate to the computational sensitivity of the proposed method, which needs some further experimental shreds of evidence. Also, the datasets selected by authors need some clarifications, in both why a new benchmark was introduced and why. It would be great to have more experimental pieces of evidence on other benchmark datasets which are usually considered in zero-shot learning (such as the ''standard'' ImageNet or classical datasets such as SUN -- see (Xian et al. 2019a) -- or FLO -- see (Xian et al. 2018). If the two prior aspects will be addressed by the Authors, I would be more than happy to call for a full acceptance. \n\n*Post-Rebuttal Evaluation [FINAL]*\n\n\nI have thoroughly inspected the revised manuscript and read the response provided by authors. I was pleased in registering that my two requests were taken into consideration by authors who provided a solid sensitivity analysis and additional experimental evidences on ImageNet. Therefore, taking this into account, I am now convinced in raising my original score (5: Marginally below acceptance threshold) to a full acceptance (7: Good paper, accept). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "hUu8PxcFHha", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on improving zero-shot classification by reducing the bias of the classifier towards seen classes. The bias occurs since the embedding is trained with visual examples from the seen classes, while using only the attribute information from unseen classes for testing. Authors propose an isometric propagation network that build a graph in both visual and semantic space, performs some steps of propagation, and then uses the updated prototypes for training a classifier. They use attention to construct the graph and also use attention to regularize the graph edges between the two spaces to be isometric. Authors also propose to use an episodic training method to improve learning. \n\nPros:\n- Paper is well motivated and targets an important problem in ZSL\n- Achieves SOTA results on standard benchmarks and proposed large scale benchmarks \n- Show ablation studies for the proposed components\n- Idea of using prototypes that are more informative than either semantic or visual attributes/features is interesting\n\nCons/Questions\n- The paper could have been better written. It was hard to follow at some points\n- Section 4: \"In this paper, we bridge the three spaces via class... \" Is it three or two spaces\n- I was not convinced with the results in the ablation studies.  For most cases the harmonic mean is close the best value- which raises questions about how much these additional contributions are helping and what exactly is resulting in improvement over the baseline. The mode without any propagation achieves H=70.7 in table2(b). How do the method achieves 70.7 (which is more than most SOTA methods) without any propagation. Similarly the consistency loss only seems to be improving performance by 2.9% over H=70.1. Either these ablation studies are not set properly or the baseline is already high. \n- How much is the episodic training helping? \n- Can the authors provide a baseline the only uses the semantic features and then learning the classifier in Eq (8). Also, did the authors try using the prototypes in an embedding based classifier\n- Why is the classifier in Eq (8) referred to as a KNN classifier.  \n\nOverall I find the idea interesting but I am giving a borderline rating due to several unanswered questions and lack of strong ablation studies highlighting the effectiveness of the proposed idea. \n\nFinal rating- I am satisfied with the author's response and updating my ratings. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4nqd6gouRN8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of paper**\nIn this paper Zero Shot Classification is studied using prototypes. Each class is represented with a visual and semantic prototype, and at test time compared to a visual example + prototype for a(n unseen) test class. The most similar test class is chosen. In this work a novel method is proposed to construct the prototypes, which are trained in an episode learning setting. On various benchmarks the proposed method performs better than existing method for the generalized zero-shot classification task (seen + unseen) test classes.\n\n\n**Strengths**\n1. This paper combines semantic and visual prototypes via a kind of attention method for (unseen) test classes (Eq 2), and at train time via a KL divergence, yielding more visual&semantic prototype neighbours. That is interesting way of combining the two spaces. Could the attention module directly use both spaces, and not only via a KL loss?\n\n2. The proposed method obtains good results on different benchmark tasks.\n\n**Weaknesses**\n1. Notation is not always clear, and partly undefined. Below a list of some examples:\n  * Eq 1: given that D is fixed for a class (at any iteration, not only at [0]), this seems to be the simple projection: W P, where W depends on the iteration, so may be W[0] should be used. Note that is unclear why the visual prototypes should be mapped to the semantic prototypes. The reverse is equally explainable. [Please make clear that W is shared among all prototypes].\n  * Eq 2 (and below): a^s and c^s are not defined. \n  * Eq 2: How is P^v_z defined? When similar to Eq 1, then there is a double projection with W.\n  * Eq 8: If w and b are not class dependent (ie depend on Py), then at least b do not add anything to the argmax. \n  * Pr (below Eq 8) is undefined.\n  * Section 4.5: unclear what parameters are trained here (W, W1, W2, b1, w, b)? Unclear how important the episode training is compared to 'standard' training. \n2. Relation to other methods are unclear. For example, the label/graph propagation is not related to other methods. The episodic learning is not related to other method (in ZSL), also its importance is only implicitly visible from the experimental results (by comparing the ablations with results from the main table).\n3. Unjustified claims and missing experiments:\n * A new large scale benchmark to train end-to-end is introduced [Section 5.1 paragraph 'Two new large-scale benchmarks'], but in 5.2 it is stated 'we use pre-trained classifiers without fine-tuning'. Besides many more papers on ZSL have conducted experiments on (subsets of) ImageNet.\n  * Claim in Sect 5.3 'For example IPN brings 10% improvement to H-metric on AWA2', according to Table 1: 72.9 vs 71.5. That is a 1% improvement. \n  *  'Initialisation of visual prototypes of unseen classes is important' [P7, Sect 5.3] (currently done using Eq 2). But it is left unclear how this is studied, concluded. \n * The method relies on a few hyper-parameters (temperature, threshold, episode learning N, K, learning weight (lambda), and the number of propagation steps), these are not studied, but for eg the propagation steps it is claimed that more steps perform better. This should be studied.\n * There are a few alternatives to the same idea (ie how to use the neighbour propagation in the two spaces) which are not explored.\n\n\n **Minor**\n- The anonymous GitHub repository is not available (404 error)\n- Figure 1: It seems that a nearest prototype classifier is used instead of the depicted nearest neighbour classifier \n- Typo: Page 3, Sect 4, first sentence: 'we bridge the three spaces' --> two?\n- Page 4, Sect 4.1: It is unclear what the geometry structure of  semantic prototypes are and how these are preserved.\n- Page 8, Sect 6, first sentence: 'each space ans across two' --> and\n\n\n**Conclusion**\nThis paper proposes a novel method to learn semantic and visual prototypes for zero-shot classification, which uses episodic learning. The prototypes are propagated over a visual/semantic neighbours graph. That is an interesting approach and experimentally the method obtains good results on various benchmarks. The influence of some of the key  parameters of the method are not studied, nor the disentanglement between the episodic training and the prototype propagation. This combined with the many (minor!) flaws and unclarities in the paper yields my rating of marginally below the acceptance threshold.\n\n** Post Rebuttal**\nThe paper has been improved based on the reviews, I think this paper is now ok to accept.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three experts in the field recommend accepting the paper (ratings 7,7,6) after the author response, appreciating the improvements the authors made.
[Note: The AC is mainly disregarding R3's rating, as R3 did not respond to the early request of the AC to clarify their review, did not respond to the authors request for clarification, and did not participate in any discussion past their initial short review.]

The solid experimental evaluation and an original methodology for zero-shot learning speak for accepting the paper.

[The area chair is certain about accepting the paper, but not fully confident if it should be Poster or Spotlight.]





Decision: Accept (Poster)