Zero-Shot Recognition through Image-Guided Semantic Classification
Authors: Mei-Chen Yeh
Fang Li
Bo-Heng Li
Venue:  (2021)
Keywords: zero-shot learning
visual-semantic embedding
deep learning

Abstract: We present a new visual-semantic embedding method for generalized zero-shot learning. Existing embedding-based methods aim to learn the correspondence between an image classifier (visual representation) and its class prototype (semantic representation) for each class. Inspired by the binary relevance method for multi-label classification, we learn the mapping between an image and its semantic classifier. Given an input image, the proposed Image-Guided Semantic Classification (IGSC) method creates a label classifier, being applied to all label embeddings to determine whether a label belongs to the input image. Therefore, a semantic classifier is image conditioned and is generated during inference. We also show that IGSC is a unifying framework for two state-of-the-art deep-embedding methods. We validate our approach with four standard benchmark datasets.



Under review as a conference paper at ICLR 2021
ZERO-SHOT RECOGNITION THROUGH IMAGE-GUDDED SEMANTIC CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
We present new visual-semantic embedding method for generalized zero-shot learning. Existing embedding- based methods aim to learn the correspondence be- tween an image classifier (visual representation) and its class prototype (semantic representation) for each class. Inspired by the binary relevance method for multi- label classification, we learn the mapping between image and its semantic clas- sifier. Given an input image, the proposed Image-Guided Semantic Classification (IGSC) method creates label classifier, being applied to all label embeddings determine whether a label belongs to the input image. Therefore, a semantic clas- sifier is image conditioned and is generated during inference. We also show that IGSC is a unifying framework for two state-of-the-art deep-embedding methods. We validate our approach with four standard benchmark datasets.
INTRODUCTION
As a feasible solution for addressing the limitations of supervised classification methods, zero- shot learning (ZSL) aims to recognize objects whose instances have not been seen during train- ing (Larochelle al., 2008 Palatucci et al.] 2009). Unseen classes are recognized by associating seen and unseen classes through some form of semantic space; therefore, the knowledge learned from seen classes is transferred to unseen classes. In the semantic space, each class has cor- responding vector representation called a class prototype. Class prototypes can be obtained from human-annotatee annotated attributes that describe visual properties of objects (Farhadi al. [009 Lampert et al.
2014) or from word embeddings learned in an unsupervised manner from text corpus Mikolov
ct al.
2013 Pennington et al. 2014 Devlin al. 2018).
A majority of ZSL methods can be viewed using the visual-semantic embedding framework, as dis- played in Figure (a). Images are mapped from the visual space to the semantic space in which all classes reside. images and labels are projected to a latent space (Yang & Hospedales 2015)
Liu et al. 2018). Then, the inference performed in this common space (Akata et al. 2013] Frome et al. 2013: Socher et al. 2013), typically using cosine similarity Euclidean distance. Another perspective of embedding- based methods is to construct an image classifier for each unseen class by learning the correspondence between binary one see-versus-eess image classifier (i.e., visual repre- sentation of a class) and its class prototype in the semantic space (i.e., semantic representation of a class) (Wang al. 2019). Once this correspondence function is learned, a binary one- ressusesss--est image classifier can be constructed for an unseen class with its prototype (Wang et al. (2019). For example, a commonly used choice for such correspondence is the bilinear function (Frome et al.
2013
Akata et al.) 2013 2015 |omera--Paaeeee Paredes & Torr 2015 Li et al. 2018). Considerable efforts have been made to extend the linear function nonlinear ones (Xian al. 2016 Wang al. 2017
Elhoseiny et al.] 2017: Qiao et al., 2016). Figure [ (b) illustrates this perspective.
Learning the correspondence between an image classifier and a class prototype has the following drawbacks. First, the assumption of using a single image classifier for each class is restrictive be- cause the manner for separating classes in both visual and semantic spaces would not be unique. We argue that semantic classification should be conducted dynamically conditioned on an input image. For example, the visual attribute wheel may be useful for classifying most car images. Nevertheless, cars with missing wheels should also be correctly recognized using other visual attributes. There- fore, instance-specifie semantic classifiers are more preferable than category-speciffe ones because the classifier weights can be adaptively determined based image content. Second, the scale of

---

Under review as a conference paper at ICLR 2021
(n)
(b)
(0)
Figure 1: Zero-shot learning paradigms. (a) Conventional visual-to-semantee mapping trained on classification loss. (b) Another interpretation of visual-to-semanti mapping between visual and semantic representations. (c) The proposed IGSC, aiming to learn the correspondence between an image and semantie classifier.
training data for learning the correspondence is constrained to be the number of class labels. In other words, a training set with C labels has only C' visual-semantic classifier pairs to build the correspon- dence. This may hinder the robustness of deep models that usually require large-scale training data. Finally, although class embedding has rich semantic meanings, each class is represented by only single class prototype determine where images of that class collapse inevitably (MarcoBaroni 2016 Fu et al. 2015). The mapped semantic representations from images may collapse to hubs, which are close to many other points in the semantic space, rather than being similar to the true class label [MarcoBaroni 2016).
In this paper, we present new method, named Image- Guided Semantic Classification (IGSC), to address these problems. IGSC aims to learn the correspondence between an image and its corre- sponding label classifier, as illustrated in Figure [T (c). In contrast to existing methods focusing on the learning of visual (or semantic) representations (Zhang et al. [0166] Frome et al.) 2013 Socher (ct al.) 2013). IGSC analyzes the input image and seeks for combinations of variables in the semantic space (e.g., combinations of attributes) that distinguish class (belonging to the input) from other classes. The proposed IGSC method has the following characteristics:
IGSC learns the correspondence between image in the visual space and classifier in the semantic space. The correspondence can be learned with training pairs in the scale of training images rather than that of classes.
IGSC performs learning to learn in end-to-end manner. Label classification is conducted by a semantic classifier whose weights are generated on the fly. This model is simple yet powerful because of its adaptive nature.
IGSC unifies visual attribute detection and label classification. This is achieved via the design of a conditional network (the proposed classifier learning method), in which label classification is the main task of interest and the conditional input image provides additional information a specific situation.
IGSC alleviates the hubness problem. The correspondence between an image and seman- tic classifier learned from seen classes can be transferred to recognize unseen concepts.
We evaluated IGSC with experiments conducted on four public benchmark datasets, including SUN (Patterson & Hays 2012), CUB (Patterson & Hays 2012), AWA2 (Lampert et al.) (014), and aPY (Farhadi et al. 2009). Experimental results demonstrated that the proposed method achieved promis- ing performance, compared with current state-of-the the-trrtttt methods. The remainder of the paper is ganized as follows: We briefly review related work in Section 2 Section 3 presents the proposed
framework. The experimental results and conclusions are provided in Sections and respectively.
2 RELATED WORK
Zero-shot learning has evolved rapidly during the last decade, and therefore documenting the exten- sive literature with limited pages is rarely possible. this section, we review a few representative zero-shot learning methods and refer readers to (Xian et al.) 2019a Wang et al. 2019) for a com- prehensive survey. One pioneering main stream of ZSL uses attributes to infer the label of an image belonging one of the unseen classes (Lampert al. 2014 Al-Halah al. 2016 Norouzi al.
2

---

Under review as a conference paper at ICLR 2021
2014 Jayaraman & Grauman 2014 Kankuekul al. 2012). The attributes of an image pre- dicted, then the class label is inferred by searching the class which attains the most similar set of attributes. For example, the Direct Attribute Prediction (DAP) model (Lampert et al. (009) esti- mates the posterior of each attribute for an image by learning probabilistic attribute classifiers. A test test sample is then classified by each attribute classifier alternately, and the class label is predicted by probabilistic estimation. Similar to the attribute-based methods, the proposed method has the merits of modeling the relationships among classes. However, IGSC unifies these two steps: attribute clas- sifier learning and inferring from detected attributes to the class. Furthermore, attribute classifiers are jointly learned in IGSC.
A broad family of ZSL methods apply an embedding framework that directly learns a mapping from the visual space to the semantic space (Palatucci et al., 2009 Akata et al. 2013; 2015 Romera- Paredes & Torr 2015). The visual-to-semantiy mapping can be linear (Frome et al. 2013) or non- linear (Socher et al. 2013). For example, DeViSE (Frome et al.) 2013) learns a linear mapping between the image and semantic spaces using an efficient ranking loss formulation. Cross-Modal Transfer (CMT) Socher et al. 2013) uses neural network with two hidden layers to learn a nonlin- projection from image feature space word vector space. More recently, deep neural network models are proposed to mirror learned semantic relations among classes in the visual domain from the image (Annadani & Biswas 2018) or from the part (Zhu et al. 2018a) levels. IGSC is also an embeddinng- ZSI. method. IGSC differs significantly from existing methods in that IGSC learns the correspondence between an image and its semantic classifier, enabling the possibility of using different classification manners to separate class prototypes in the semantic space.
Recent ZSL models adopt the generative adversarial network (GAN) (Goodfellow et al., 2014) or other generative models for synthesizing unseen examples (Bucher et al. Long et al.) 2017;
Jiang et al. 2018 Verma et al., 2018: Xian et al.) 2018 Zhu et al.) 2018b][Xian al. [019b]] Verma et al. 2020: Yu et al., 2020, Ma & Hu 2020) or for reconstructing training images (Chen et al.
2018). The synthesized images obtained at the training stage can fed to conventional classifiers
s0 that 7SI. is converted into the conventional supervised learning problem (Long et al.) 2017)
The transformation from attributes to image features require involving generative models such as denoising autoencoders (Bucher et al., 2017). GAN (Xian et al. 2018 Zhu et al. 2018b) or their variants (Verma et al., 20188 et al. 2018 Xian et al. 2019b; Yu et 2020) Ma & Hu 2020).
Despite outstanding performances reported in the papers, these works leverage some form of the unseen class information during training. In view real-world applications involving recognition in-the-wild novel classes including the image samples as well as the semantic representations may not be available in model learning. The proposed method is agnostic to all unseen class information during training. Furthermore, the proposed method is much simpler in the architecture design and has a much smaller model size, compared with the generative methods.
3 APPROACH
3.1 PROBLEM DESCRIPTION
Given training set S = (I Bn). n 1...N). N). with Un E Jr being class label in the seen class set, the goal of ZSL is to learn a classifier f: X + y which can generalize to predict any image to its correct label, which is not only in 3/ but also in the unseen class set V'. In the prevailing family of compatibility learning ZSL (Xian et al. 2019a; Ba et al. 2015)), the prediction is made via:
i f(x; = arg max F(I (I)
(1)
NEY
In particular, y = Yu. this is the conventional ZSL setting; y = V. U.... this is the generalized zero-shot learning (GZSL) setting, which is more practical for real-world applications. The compat- ibility function parammettterie by wis used to associate visual and semantic information.
In the visual space, each image x has vector representation, denoted by 6(z). Similarly, each class label y has a vector representation in the semantic space (called the class prototype), denoted by (ly). In short, (x) and Φdy) are the image and class embeddings, both of which are given.
3

---

Under review as a conference paper at ICLR 2021
label classifier g(+)
label
semantic embedding
M (x-adaptive)
compatibility score
F(x, y. W)
Reshape
FC 1024
image
visual embedding
e(x)
image-to-classifer mapping function h(+)
ReLU
FC 2048
an manage-conddtionnllodeee
Figure 2: The architecture of IGSC. This model receives an image and label, and it returns the compatibility score of this input pair. The score indicates the probability of the label belonging to the image. The score is calculated by label classifier g(-), whose weights M are stored in the output layer of a fully connected neural network. Therefore, weight values depend on the input image. The neural network characterized by the parameters W, which the only parameters required to learn from training data.
3.2 IMAGE-GUIDED SEMANTIC CLASSIFICATION MODEL
The compatibility function in this work achieved by implementing two functions, h(((r); W) and g(((v); M), illustrated in Figure The first function h(-) receives an image embedding input and returns parameters M characterizing label classifier:
M h(((z);; W).
In other words, h(-) learns the mapping between image representations and model (i.e., semantic classifier) representations. Each image has its own semantic classifier. Images of the same class may have different classifier weights. Different from existing methods where the classifier weights are part of model parameters and thereby being static after training, the classifier weights in IGSC are dynamically generated during test time.
The image-to- collassieel mapping can be either linear or nonlinear. Figure 2 shows an implementation of nonlinear model that involves two fully connected layers and an output layer. The dimension of the output layer is set to accommodate the label classifier weights. We emphasize again that W are the only model parameters required to learn from training data.
The second function g(-) is label classifier, characterized by the parameters outputted by h(.). This function takes a label vector as input, and returns a prediction score indicating the probability of the label belonging the input image:
g(p(y);
Let sy denote the prediction score for label j. In multi-class (single-label) image classification, the final compatibility score is obtained by normalizing the prediction scores probabilistic values with softmax:
F(x, W) exp(s;)
(4)
ED exp(**)
One image is assigned to the class with the highest compatibility score. In multi-label image classi- fication, we replace softmax by sigmoid activation function. The prediction is made by choosing labels whose compatibility score is greater than threshold.
is worth noting that the mechanism IGSC is similar to that Dynamic Filter Networks (Jia et al., 2016), in which the filters are generated dynamically conditioned also appears in (Zhao et al. 2018), which predicts a set of adaptive weights from conditional inputs to linearly combine the basis filters. The proposed method differs fundamentally in that both (Jia jet al., 2016) and (Zhao et al. 2018) focus on learning image representations, while our method aims learn model representations that applied to different modality (i.e., labels).
input. A similar mechanism
4

---

Under review as a conference paper at ICLR 2021
3.3 FORMS OF LABEL CLASSIFIESS
The image-guided label classifier can be either linear or nonlinear, which receives a label embedding and returns prediction score of the label. In this study experiment with two variations of the label classifier. The linear label classifier is represented as:
(5) where m E Rd is a weight vector, b is threshold and M = (m. b). The dimension d is set to that of the label vector (e.g., d= 300 if using 300-dim word2vec (Mikolov al., (013)). Alternatively, the nonlinear label classifier implemented using a two-layer neural network:
g(o(y); M) mø(y) b.
g M) m2 tanh (M
b1) b2,
where M1 E Raxd m2 E R and M = (M1, bj m2 b2). The nonlinear classifier characterizes the d-dim semantic space by using h perceptrons and performs the classification task. We set h to 30 in the experiments. As will be shown in Section the nonlinear label classifier outperforms a linear one.
For GZSL, it is beneficial to enable calibrated stacking (Chao et al. 2016), which reduces the scores for seen classes. This leads the following modification:
ÿ = arg max (g(:(v); M) y1 E V.)),
where 1ly E v.1= E {0,1) indicates whether or not y is seen class and y is a calibration factor.
3.4 LEARNING MODEL PARAMETERS
Recall that the objective of ZSL. is correctly assign the correct label image. This is typ- ical classification problem. For a training sample zi, Let yi = {ut v2 .... y ('dl) E {0, denote the one-hot encoding of the ground truth label and pa {pl, p² ..... pill denote the compatibil- ity scores of x; (Equ. 4p. That is, pf Fx,, yj; W). The model parameters W are learned by
minimizing the cross entropy loss:
L -= -= oe log(r!) + x/ log(1 -r()-
We had also experimented with the hinge loss and achieved similar performances. The model pa- rameters including W and those of the image/semantic embedding networks can be jointly learned end-to-end; however, the results reported in Section 4 were obtained by freezing the weights of fea- ture extractors for fair comparison. That is, all methods under comparison used the same image and semantic representations the experiments.
3.5 CONNECTION TO PREVIOUS METHODS
Finally we show how previous supervised visual-semantic embedding methods DeViSE (Frome al. 2013) and CMT (Socher et al. 2013) are special cases of our method.
DeViSE (Frome et al. 2013) uses projection layer (a linear transformation) that maps a visual vector to the semantic space, and compute a dot-product similarity between the projected visual vector and the vector representation of the correct label. The behavior is identical to special case of our method, where both h(-) and g(-) are linear. CMT (Socher et al. (013)) uses neural network with two hidden layers and the standard nonlinearity tanh to learn a nonlinear projection from image feature space to word vector space and compute Euclidean distances of the L.2 normed vectors. This is identical to the special case of using nonlinear h(-) and linear g(-), except that we use ReL.U instead of tanh in the nonlincar transformation.
EXPERIMENTS
4.1 DATASETS AND EXPERIMENTAL. SETTING
We used four popular benchmark datasets, including coarse- coarse-grained and fine-grained datasets, for evaluating the proposed method. The statistics of the datasets summarized in Table Please
5

---

Under review as a conference paper at ICLR 2021
Table I: Summary of the datasets used in the experiments
Number of classes
Number samples
Dataset
Embedding dim.
Seen
Unseen
Training
Test (seea)
Test (unseen)
Total
SUN
Patterson & Hays 2012
102
580 + 65
72
10,320
2,580
1,440
14,340
CUB
Welinder et al. 2010
312
100 +
7,057
2,967
1,764
11,788
AWA21
Lampert
27
23.527
5,882
7,913
37,322
aPY
Farhadi ef 2009
64
15
12
5,932
1,483
7,924
15,339
Xian et al. 2019a) for detailed descriptions. We followed the new split provided by (Xian et al. 2019a) because this split ensured that classes at test should be strictly unseen at training.
Visual and semantic embeddings. For fair comparison, we used the 2048- sis--dimensional ResNet- 101 features provided by (Xian et al., (019a) as image representations. For label representations, we used the semantic embeddings provided by (Xian et al.) 2019a), each of which an L2-normalized attribute vector. Note that IGSC is flexible in that the visual and semantic embeddings, h(-) and g(-) functions all be customized to meet specific needs.
Training details. We used Adaptive Moment Estimation (Adam) for optimizing the model. We augmented the data by random cropping and mirroring. The learning rate was set fixed 10-5 Training time for single epoch ranged from 91 seconds to 595 seconds (depending on which dataset was used). Training the models using four benchmark datasets roughly took 11 hours in total. The runtime was reported running on machine with an Intel Core i7-7700 3.6 3.6-GHz CPU, NVIDIA's GeForce GTX 1080Ti and 32 GB of RAM. The dimension in the nonlinear variant of the semantic classifier g(-) was set to 30 in the experiments.
Evaluation protocols. We followed the standard evaluation metrics used in the literature. For ZSL, used average per-class top-1 accuracy as the evaluation metric, where the prediction (Eq. [D is successful if the predicted class is the correct ground truth. For GZSL, we reported acc, (test images are from seen classes and the prediction labels are the union of seen and unseen classes) and accu (test images are from unseen classes and the prediction labels are the union of seen and unseen classes). We computed the harmonic mean (Xian et al. 2019a) of accuracy rates on seen classes acc, and unseen classes acca:
2 X acC, X acCu
H
acc, acc,
The harmonic mean offers comprehensive metric in evaluating G7SL methods. The harmonic mean value is high only when both accuracy rates are high. We reported the average results of three random trials for each ZSL and GZSL experiment.
Table 2: Ablation study effects of different design choices
ZSL
GZSL
acc
acc,
acc,
H
linear h + linear g
56.97
19.56
28.71
23.27
linear + nonlinear g
54.56
17.15
31.99
22.32
nonlinear h linear
58.01
19.68
31.08
24.10
nonlinear + nonlinear g
58.30
19.88
36.41
25.72
4.2 ABLATION STUDY
First, we investigate the effects of different designs of the and the label classifier g()). We reported the results on the SUN benchmark; however, similar findings can be found using other datasets.
mapping function It(-)
Table 2 shows the results of the ablation experiment. In both settings (ZSL and GZSL), using nonlinea image-to-classifee mapping (i.e., h(-)) is essential to the performance. A significant per- formance gain was observed when nonlinear h(-) was applied. The combination of linear h(-) and nonlinear g(.) performed the worst. A possible reason that a linear mapping does not have sufficient capacity model the relation between visual feature and its corresponding semantic

---

Under review as a conference paper at ICLR 2021
Table 3: Standard zero-shot learning results (top-1 accuracy) four benchmark datasets
Method
SUN
AWA2
aPY
DAP (Lampert ct al.
2009
39.9
40.0
46.1
33.8
IAP (Lampen el at. 2009
19.4
24.0
35.9
CONSE (Norouzi et al. 2014
38.8
44.5
26.9
CMT (Socher
2013
39.9
37.9
SSE (Zhang Saligram grama
2015
51.5
43.9
61.0
34.0
LATENT IXian
55.3
49.3
55.8
35.2
ALE (Akala a.
58.1
54.9
62.5
39.7
DeViSE (Frome
2013
56.5
52.0
59.7
39.8
SJE (Akata el 2013
53.7
53.9
61.9
32.9
ESZSL (Romera Paredies & Tom 2015
$4.5
53.9
58.6
38.3
SYNC
Changginyy
2016
56.3
55.6
46.6
23.9
SAE
Kodirov al. 2017
40.3
33.3
54.1
8.3
GFZSL
Verma Kal 2017
60.6
49.3
63.8
38.4
IGSC
58.3
56.9
62.1
35.2
Table Generalized zero-shot learning results (top-1 accuracy and on four benchmark datasets.
All methods are agnostic to both unseen images and unseen semantic vectors during training.
SUN
CUB
AWA2
aPY
Method
aCc,
GCC,
acC.
acc,
H
acc,
RCC,
acc.
acc.
DaPiLampert
2009
4.2
25.1
7.2
1.7
67.9
3.3
0.0
84.7
0.0
4.8
78.3
9.0
IAP Lammert mi
2009
1.0
37.8
1.8
0.2
72.8
0.4
0.9
87.6
1.8
5.7
65.6
10.4
CONSE Norouzz
2014
72.2
(aclir
8.1
21.8
11.8
7.2
49.8
12.6
0.5
90.0
1.0
1.4
85.2
2.8
Sncher ef ml 2013
8.7
28.0
13.3
4.7
60.1
8.7
8.7
89.0
15.9
10.9
74.2
19.0
SSE Zlinn Salprama
2015
4.0
46.9
14.4
14.8
78.9
LATEM Xlan el al 201b
14.7
28.8
19.5
15.2
$7.3
24.0
11.5
77.3
20.0
0.1
73.0
0.2
ALE(Akala dle 2015
21.8
33.1
26.3
23.7
62.8
34.4
14.0
81.8
23.9
73.7
8.7
DEVISE
2015
16.9
27.4
20.9
23.8
$3.0
32.8
17.1
74.7
27.8
76.9
9.2
Akata et du p013
14.7
30.5
19.8
23.5
59.2
33.6
8.0
73.9
14.4
3.7
$5.7
6.9
ESZST Komera
2015
12.6
21.0
SYNC Chnng pinvoroyoor 2010
7.9
43.3
13.4
11.5
70.9
19.8
10.0
90.5
18.0
7.4
66.3
13.3
SAE Kodirov el 2017
8.8
11.8
7.8
54.0
13.6
1.1
82.2
2.2
80.9
0.9
GFZSL
0.0
45.7
2.5
4.8
83.3
0.0
SP-AEN (Chen m 2018
24.9
38.6
30.3
34.7
70.6
46.6
23.3
90.9
37.1
13.7
63.4
22.6
Anninian & Biswax 2018
20.8
37.2
26.7
24.6
54.3
33.9
20.7
73.8
32.3
13.5
51.4
21.4
AREN (Xie
19.0
38.8
25.5
38.9
78.7
52.1
5.6
92.9
26.7
76.9
16.4
IGSC
19.8
36.4
25.7
27.8
66.8
39.3
19.8
84.9
32.1
13.4
69.5
22.5
IGSC+CS
39.4
31.3
34,9
40.8
60.2
48.7
25.7
83.6
39.3
23.1
58.9
33.2
classifier, and using nonlinear g(.) exacerbates the overfitting problem of learned semantic clas- sifiers to seen classes. While a nonlinear lt(-) successfully modeled the mapping between a visual feature and its label classifier, using a nonlinear 9(-) further improved the recognition performance, especially under the setting of GZSL.
4.3 COMPARISONS WITH STATE-OF-THEE ART EMBEDDING-BANE BASED APPROACHES
We compared the IGSC method with a variety standard and generalized ZSL methods reported
in [Xian et al. 2019a). These methods can be categorized into 1) attribute- based: DAP (Lampert
al.
2009p, TAP (Lampert al. 2009). CONSE (Norouzi al. 2014). SSE (Zhang Saligrama
2015
SYNC (Changpinyo et al. 2016); and 2) embedding- based: CMT/CMT* (Socher et al. LATEM Xian et al. 2016), ALE(Akata al. 2013). DeViSE(Frome et al., 2013). SJE(Akata
2013
et al.
Coomera-Paredes, & Torr 2015), SAE(Kodirov et al. 2017). GFZSI (FFSSL(Ves (Verma & 2017). Performances of the methods are directly reported from the paper (Xian et al. 2019a).
2015),
Rai
Please note that all methods under comparison-includee the proposed method are inductive to both unseen images and unseen semantic vectors. Only labeled training instances and class pro- totypes of seen classes transductive for unseen class prototypes and unlabeled unseen test instances can achieve better per- formances because more information is involved in model learning. Recent methods in the inductive setting are only inductive to samples (Jiang et 2018) Felix (2018] Xian et al. [2019b] Schon-
available in this experimental setting. Alternatively, methods that are
feld al. 2019 Verma al. 2020) Yu et al. 2020 Ma & Hul 2020
Huynh & Elhamifar 2020

---

Under review as a conference paper at ICLR 2021
Table 5: N₁ skewness SUN benchmark.
ZSL
GZSL
test (seen)
test (unseen)
test (seen)
test (unseen)
DeViSE (Frome et al. 2013
2.163
2.360
2.355
2.849
IGSC
1.046
0.380
0.111
2.452
These methods use unseen class labels during training, which is different to our setting and, there- fore, are not compared.
We reported the performance the proposed IGSC method with (or without) calibrated stacking 7): 1) IGSC uses the nonlinear-nonlinent combination; 2) IGSC+CS enables calibrated stack- ing. Table shows the conventional ZSL results. IGSC has superior performance to those of other methods on the CUB dataset and achieves comparable performances on the other datasets. Although GFZSL (Verma & Rai 2017) has the best performances on SUN and AWA2, this method performs poorly under the GZSL setting.
Table the generalized resulss. In this experiment, recent inductive methods (Chen al., 2018 Annadani & Biswas] 2018; Xie et al., (019) are included for comparison. The semantics- preserving adversarial embedding network (SP-AEN) (Chen et al., 2018) is a GAN-based method, which uses an adversarial objective to reconstruct images from semantic embeddings. The preserv- ing semantic relations (PSR) method (Annadani & Biswas 2018) is an embedding- approach utilizing the structure of the attribute space using set of relations. Finally, the attentive region embedding network (AREN) (Xie et al., (019) uses an attention mechanism to construct the embed- dings from the part level (i.e., local regions), which consists of two embedding streams to extract image regions for semantic transfer.
By examining the harmonic mean values, IGSC consistently outperformed other competitive meth- ods on three out of the four datasets. The performance gain validated the effectiveness of learning image-guided semantic classifiers. Compared with embedding based methods, this novel paradigm not only has more training pairs (in the scale of the training images) for learning the correspondence between an image and its corresponding label classifier but also allows different manners to separate classes based on the content of input image. In comparison with attribute based methods which take a two-step pipeline to detect attributes from one image and aggregate the detection results for label prediction, IGSC unifies the steps. Compared with recent methods (Chen et al., 2018 Annadani & Biswas 2018 Xie et al.. 2019). IGSC much simpler and therefore has a greater flexibility. We have not integrated powerful components for GZSL such as feature generators and attention models, yet it has achieved comparable (or superior) performance to existing sophisticated methods.
An additional reason that explains the improved performance is that the hubness may be alleviated IGSC, which avoids nearest neighbor searches of class prototypes in the semantic space. We con- ducted experiment to realize whether IGSC reduces hubness. To measure the degree of hubness, we used the skewness of the empirical N₁ distribution (Radovanović et al.) 2010 SSigetoo et al.) 2015]. We conducted this experiment on the SUN benchmark because it is the only dataset contain- ing an equal number of test images per class. As we hardly found the skewness analyses in recent literature, we implemented De ViSE (Frome et al. 2013) and compare it with the proposed method. The results are summarized in Table 5 IGSC produced smaller skewness values. One possible rea- son explaining why hubness alleviated is that the "matching" between a visual representation and a class prototype is more flexible in IGSC than that in nearest neighbor search. A label is considered correct long as its embedding is on the right side of the decision surface, learned conditioned on the input image embedding.
To better understand the strengths and weaknesses of the proposed method, we compare its perfor- mance with recent GAN-based methods. The notable performance of f-VAEGAN-D2 (Xian et al.) 2019b) achieved the harmonic mean values of 41.3 on SUN and 53.6 on CUB. This GAN-based approach and its variants focus the design of a data augmentation scheme where arbitrarily many synthetic features of both seen and unseen classes can be created to aid in improving the discrimina- tive power of classifiers. Linear softmax classifiers are typically used in such approaches. Altern-- tively, the proposed embedding- baasedde--beeedd method focuses on the design of classifiers. These techniques may complement each other to advance methods for zero-shot learning.
8

---

Under review as a conference paper at ICLR 2021
Table 6: Ablation study effects of different visual models
SUN
CUB
AWA2
aPY
Backbone
aCCC
acCC
H
acCu
acCs
H
aCC,
aCCs
H
acC,
aCC4
H
Res- 101
22.5
36.1
27.7
27.8
66.8
39.3
19.8
84.9
32.1
13.4
69.5
22.5
Res-152
23.7
36.1
28.6
28.6
68.0
40.3
22.7
83.9
35.7
14.9
67.6
24.4
Res- ces-101+CS
39.4
31.3
34.9
40.8
60.2
48.7
25.7
83.6
39.3
23.1
58.9
33.2
1es-152+CS
40.8
31.2
35.3
42.9
61.0
50.4
27.1
83.0
40.9
26.4
53.3
35.3
(a)
(b)
Figure 3: t-SNE visualization of the model space learned by IGSC: (a) seen classes, (b) unseen
classes. Best viewed in color.
4.4 DISCUSSIONS
We discuss the model flcxibility and visualize the classifier weights generated by IGSC.
Model extendibility. The proposed IGSC model is flexible that the visual and semantic embed- dings. the h(-) and g(-) functions all be customized to meet specific needs. We provide proof of concept analysis, in which we investigate the effect of replacing Res-101 with Res-152. Table 6 shows the result. Performance improvements
observed when By elaborating other components in IGSC, it seems reasonable to expect this approach should yield even better results.
deeper visual model.
Model visualization. Finally, we visualize the "model representations" of the label classifiers by using t-SNE (yan der Maaten Hinton 2008 of the dynamically generated classifier weights. Figure 3 displays the visualization results using the aPY dataset. Each point in the figure represents label classifier generated on-the-fly by feeding a test image to the IGSC model. Colors indicate class labels. Although each image has its own label classifier, the IGSC method tends to generate similar model representations for images of the same class (Fig. 3 (a)). Please note that this framework allows the learning of "single" label classifier for each class (class-level classifier), i.e., the model representations for images of the same class are identical. However, the results show that instance- level classifiers benefit the optimization of recognition accuracy (points are scattered but fall around).
CONCLUSION
We propose a unifying visual-semantic embedding model that transform an image into label clas- sifier, consequently used to predict the correct label in the semantic space. Modeling the corre- spondence between an image and its label classifier enables a powerful GZSL method that achieves promising performances on four benchmark datasets. One future research direction we pursuing to extend the method for multi-label zero-shot learning. in which images are assigned with multi- ple labels from open vocabulary. This would take full advantage of the semantic space. Another direction is to explore model learning with less restricted setting, which can transductive for specific unseen classes or test instances.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for attribute-ban baseee classification. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 119, 2013.
Zeynep Akata, Scott H. Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 2927-2936, 2015.
Z. Al-Halah, M. Tapaswi, and R. Stiefelhagen. Recovering the missing link: Predicting class- attribute associations for unsupervised zero-shot learning. In Proc. of IEEE International Confer- ence on Computer Vision and Pattern Recognition, 2016.
Yashas Annadani and Soma Bíswas. Preserving semantic relations for zero-shot learning. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 7603-7612, 2018.
Jimmy Lei Ba. Kevin Swersky, Sanja Fidler, and Ruslan Salakhutdinov. Predicting deep zero- shot convolutional neural networks using textual descriptions. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2015.
M. Bucher, S. Herbin, and F. Jurie. Generating visual representations for zero-shot classification. In arXiv preprint arXiv: 079750677 2017.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero- shot learning. In Proc. of TEEE International Conference on Computer Vision and Pattern Recog- nition, pp. 5327-5336, 2016.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong. and Fei Sha. An empirical study and analysis of generalized zero- shot learning for object recognition in the wild. In Proc. of European Conference on Computer Vision, 2016.
Long Chen, Hanwang Zhang, Wei Liu Jun Xiao, and Shih-Fu Chang. Zero-shot visual recogni- tion using sementiss-preserving adversarial embedding networks. Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2018.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Mohamed Elhoseiny. Yizhe Zhu, Han Zhang, and Ahmed Elgammal. Link the head the "beak": Zero shot learning from noisy text description at part precision. In Proc. of IEEE International Conference on Computer Vision and Puttern Recognition, pp. 6288-6297 2017.
Ali Farhadi, Ian Endres, Derek Hoiem, and David A. Forsyth. Describing objects by their attributes. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1778 1785, 2009.
R. Felix, B. G. V. Kumar, I. D. Reid, and G. Cameiro. Multimodal cycle-consistent generalized zero-shot learning. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2018.
Andrea Frome, Gregory Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In Proc. of Neural Information Processing Systems, 2013.
Yanwei Fu, Timothy M. Hospedales, T. Y. Xiang, and Shaogang Gong. Transductive multi-view zero-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37:2332- 2345, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. of Neural Information Processing Systems, 2014.
10

---

Under review as a conference paper at ICLR 2021
Dat Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute- based attention. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2020.
D. Jayaraman and K. Grauman. Zero-shot recognition with unreliable attributes. In Proc. of Neural Information Processing Systems, 2014.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V. Gool. Dynamic filter networks. In Proc. of Neural Information Processing Systems, 2016.
Huajie Jiang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Learning class prototypes via structure alignment for zero-shot recognition. In Proc. of European Conference Computer Vision, 2018 8.
Kankuckul, Kawewong, S. Tangruamsub, and O. Hasegawa. Online incremental attribute- based zero-shot learning. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2012.
Elyor Kodirov. Tao Xiang, and Shaogang Gong, Semantic autoencoder for zero-shot learning. In Proc. of IEEE International Conference Computer Vision and Pattern Recognition, pp. 4447- 4456, 2017.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In Proc. of IEEE International Conference on Com- puter Vision and Pattern Recognition, pp. 951-958, 2009.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-aseed classification for zero-shot visual object categorization. IEEE Transactions on Puttern Analysis and Machine telligence, 36:453 465, 2014.
H. Larochelle, D. Erhan, and Y. Bengio. Zero-data learning of new tasks. In Proc. of AAAI Confer- ence on Artificial Intelligence, 2008.
Yan Li, Zhen Jia, Junge Zhang, Kaiqi Huang, and Tieniu Tan. Deep semantic structural constraints for zero-shot learning. In Proc. of AAAI Conference on Artificial Intelligence, 2018.
Shichen Liu, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. Generalized zero-shot learn- ing with deep calibration network. In Proc. of Neural Information Processing Systems, 2018.
Yang Long, Li Liu, Ling Shao, Fumin Shen, Guiguang Ding, and Jungong Han. From zero-shot learning to conventional supervised classification: Unseen visual data synthesis. In Proc. of IEEE International Conference Computer Vision and Pattern Recognition, 2017.
Peirong Ma and Xiao Hu. A variational autoencoder with deep embedding model for generalized zero-shot learning. In Proc. of AAAI Conference Artificial Intelligence, 2020.
A. G. MarcoBaroni. Hubness and pollution: Delving into cross-space mapping for zero-shot learn- ing. In Proc. of the Association for Computational Linguistics, 2016.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Proc. of Neural Information Processing Systems, 2013.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Gregory S. Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. In Proc. of International Conference on Learning Representations, 2014.
Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton, and Tom M. Mitchell. Zero-shot learning with semantic output codes. In Proc. of Neural Information Processing Systems, 2009.
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and rec- ognizing scene attributes. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 2751-2758, 2012.
Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. In Proc. of Empirical Methods in Natural Language Processing, 2014.
11

---

Under review as a conference paper at ICLR 2021
Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton den Hengel. Less is more: Zero-shot learning from online textual documents with noise suppression. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 2249-22222222 22592 2016.
Miloš Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović. Hubs in space: Popular near- est neighbors in high-dimensional data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11:2487 2531, 2010.
Bernardino Romera-Paredes and Philip H. S. Torr. An embarrassingly simple approach to zero-shot learning. In Proc. of IEEE International Conference on Machine Learning. 2015.
E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, and Z. Akata. Generalized zero- and few-shot learning via aligned variational autoencoders. In Proc. of IEEE International Conference Computer Vision and Pattern Recognition, 2019.
Yutaro Shigeto, Ikumi Suzuki, Kousuke Hara, Masashi Shimbo, and Yuji Matsumoto. Ridge regres- sion, hubness, and shot learning. In Proc. of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2015.
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, and An- drew Y. Ng. Zero-shot learning through cross-modal transfer. In Proc. of Neural Information Processing Systems, 2013.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008.
Vinay Kumar Verma and Piyush Rai. simple exponential family framework for zero-shot learn- ing. In Proc. of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2017.
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. Generalized zero-shot learn- ing via synthesized examples. In Proc. of IEEE International Conference Computer Vision and Pattern Recognition, 2018
Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. Meta-learning for generalized zero-shot learning. Proc. of AAAI Conference on Artificial Intelligence, 2020.
Wei Wang, Chunyan Miao, and Shuji Hao. Zero-shot human activity recognition via nonlinear compatibility based method. In Proc. of International Conference Web Intelligence, pp. 322- 330, 2017.
Wei Wang, Vincent Wenchen Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology, 10:13:1-13:37. 2019.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge J. Belongie, and Pietro Perona. Caltech-ucsd birds 200. In Caltech, Tech. Rep. CNS-TR2010-001, 2010.
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh N. Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 69-77, 2016.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2018
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning- com- prehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:2251 - 2265, 2019a.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan- A feature gener- ating framework for shot learning. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2019b.
12

---

Under review as a conference paper at ICLR 2021
Guo-Sen Xie, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang. Jie Qin, Yazhou Yao, and L. M. Shao. Attentive region embedding network for zero-shot learning. In Proc. of IEEE International Con- ference Computer Vision and Pattern Recognition, 2019.
Yongxin Yang and Timothy M. Hospedales. A unified perspective on multi-domain and multi-task learning. In Proc. of International Conference on Learning Representations, 2015.
Yunlong Yu, Zhong ji, Jungong Han, and Zhongfei Zhang. Episode-basee prototype generating network for zero-shot learning. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2020.
Yang Zhang, Boqing Gong, and Mubarak Shah. Fast zero-shot image tagging. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2016.
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via semantic similarity embedding, In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 4166- 4174, 2015.
Fang Zhao, Jian Zhao, Shuicheng Yan, and Jiashi Feng. Dynamic conditional networks for few-shot learning. Proc. of European Conference on Computer Vision, 2018.
Pengkai Zhu, Hanxiao Wang. and Venkatesh Saligrama. Generalized zero-shot recognition based visually semantic embedding. In Proc. of IEEE International Conference Computer Vision and Puttern Recognition, 2018a. 8a.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zero-shot learning from noisy texts. Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, 2018b.
13


Reviews:
[{"review_id": "thCRuYL2mn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\nThe authors tackle the problem of zero-shot learning, that is, the recognition of classes and categories for which no visual data are available, but only semantic embedding, providing a description of the classes in terms of auxiliary textual descriptions. To this aim, authors propose a method dubbed  Image-Guided Semantic Classification in which a two-stream network (fed by either visual and semantic embedding) learns a compatibility function whose recognition performance is enhanced by means of calibrated stacking (Chao et al. 2016).  \n\n*Pros*\n* The method is simple and easy to understand. \n\n\n*Cons*\n* The computational pipeline is not novel and largely inspired by methods such as the ones proposed in [Yang et al., A Unified Perspective on Multi-Domain and Multi-Task Learning, ICLR 2015] or [Liu et al. Generalized Zero-Shot Learning with Deep\nCalibration Network, NeurIPS 2018] which are not even cited in the paper, unfortunately. Authors seem essential to add a calibration module to this kind of architectures: since the calibration module is inherited from prior works (Chao et al. 2016), I found the method quite incremental.\n\n* Within the experimental comparison, GAN-based methods are not reported although being a mainstream class of state-of-the-art methods. I am referring to works such as CLSWGAN [Xian et al. Features Generating Networks for Zero-Shot Learning, CVPR 2018], f-VAEGAN-D2 [Xian et al. A Feature Generating Framework for Any-Shot Learning, CVPR 2019],  CADA-VAE [Schonfeld et al., Generalized zero- and few-shot learning via aligned variational autoencoders, CVPR 2019], DLFZRL [Tong et al., Hierarchical disentanglement of discriminative latent features for zero-shot learning, CVPR 2019] or tf-VAEGAN [Narayan et al. Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification, ECCV 2020]. Authors justified this approach by reducing the methods in comparison within the ones which use unseen semantic embeddings only at test time. However, the knowledge of semantic embeddings also for the unseen classes is something which is necessary in a zero-shot recognition paradigm: the authors themselves take advantage of them during the nearest neighbor search. Thus, at this point, using the semantic embeddings for the unseen classes is therefore legit and I do not see any added value in constraining the experiments.\n\n* The reported performance is highly suboptimal with respect to the state-of-the-art: invertible zero-shot recognition flows, recently proposed at ECCV 2020, greatly outperformed the proposed approach by margin: H=49.8 on aPY, H=54.8 on SUN, H=59.4 on CUB and H=68.0 on AWA2.  \n\n*Pre-Rebuttal Evaluation*\nI regret to register a substantial overlap with prior methods, thus undermining the novelty impact of the present submission. On the experimental side, I found a sharply gapped performance which is highly inferior with respect to the state-of-the-art, caused by reducing the approaches included in the comparison on the basis of which methods exploit unseen semantic embeddings for training (this claim is not convincing in my opinion, unseen semantic embeddings are used in any cases, why not exploiting them for feature generation purposes?). For those reasons I am afraid to discourage the acceptance of the manuscript\n\n*Post-Rebuttal Evaluation [FINAL]*\nI would like to thank the authors for their response and for the updated version of the manuscript, I appreciate their efforts. Unfortunately, I still believe that the paper lacking about original contribution and I am not fully convinced by the authors' comments on the relationship with [Yang et al., A Unified Perspective on Multi-Domain and Multi-Task Learning, ICLR 2015] and [Liu et al. Generalized Zero-Shot Learning with Deep Calibration Network, NeurIPS 2018] which I still judge highly overlapping with the current methodology.\nFurthermore, although authors clarified on the avoidance of using semantic embeddings for the training methodology, I do not see a sharp point in pursuing this approach given the high gap in performance with prior art (GAN-based). \nFor all these reasons, I regret to confirm my initial rejection score.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lZW_eBwJ7NH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "------------\nSummary: This paper proposes a simple yet effective method for zero-shot learning. In the method, a network is learned to predict the compatibility function weight given the input of the image. The predicted weight is then applied to semantic attributes and the final class label is predicted by the maximum compatibility score. The method is evaluated on benchmark datasets and illustrates competitive performance.\n\n------------\n\nReason for score:\nThe paper is overall clear and easy to understand, and the proposed method is simple and achieves competitive results. But the paper lacks significance and contributions (detailed below), which makes me hard to judge its potential impact in the related area . So I don't find the paper meet the standard of ICLR. Hopefully the authors can address my concerns in the rebuttal period.\n\n------------\n\nPros:\n1. The authors propose a simple method for zero-shot learning motivated by learning instance-guided classifier weights.\n2. The method demonstrates competitive results on benchmark datasets with a very simple implementation.\n3. The paper is written clearly and easy to follow.\n\n------------\nCons:\n1. The paper fails to validate its motivation. The authors claim the method is proposed to learn an \"image-guided semantic classification model\" and the \"weight values depend on the input image\". Although the model architecture complies the motivation and the evaluation results are good, it is hard to tell if the results are achieved by the \"image-guided weight values\". The authors didn't provide either theoretical analysis or empirical evaluations to demonstrate that the weights learned by the model is image-sensitive and the overall performance benefits from it. For example, it will be helpful if the authors can visualize the \n2. Some implementation details are missing and some choices in the method lack explanations. For example, why the model is trained by a cross-entropy loss in (8)? Is it specifically beneficial to the proposed method in the zero-shot learning context? For the non-linear label classifier, what is the hidden layer dimension $h$ used in the experiments? What is the effect of extending the non-linear label classifier to deeper networks? Will it saturate? etc. The paper lacks this type of details or evaluations, which makes the readers hard to evaluate its potential effectiveness.\n3. The proposed method is a framework which can implemented in various ways, but the authors fail to provide enough analysis on its potential extensions. For example, in the experiments, for fair comparison the authors froze the feature extractor. But it will be helpful if the authors can also provide the performance with the feature extractor fine-tuned, so as to provide insight on how this framework will synergize with deep feature networks. In addition, I'd love to see the performance of the proposed method on large-scale recognition-in-the-wild dataset like ImageNet.\n4. Although the authors fix the feature extractor and only compare with the methods in the exact same setting (namely, feature extractor fixed, no unseen information in training), the baselines compared are rather weak. For example, the baseline models may not benefit from the Calibrated Stacking (CS),  and the proposed method without CS has less significant results. Some baseline models like SP-AEN have lower unseen and higher seen than the proposed method, so it is possible they can achieve better performance with CS. Also, like we discussed in 3, many feature-based methods achieve significantly better results and it will be great to show how the proposed method works if the entire model including the feature extractor is trained end-to-end. \n\nBased on the points above, I find the paper less significant and lack contributions, thus giving my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "keueJPowQXP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes zero-shot learning for image classification.  The proposed method is termed as Image-Guided Semantic Classification (IGSC) that learns image-specific label classifiers to achieve zero-shot classification. Experimental results on four standard datasets are reported.\n\nThe authors are suggested to address the following comments.\n1. The idea to learn image-specific classifiers (specified by the yielded parameters) resembles the conventional local learning, where a classifier is learned for each training sample. In the last paragraph of page 4, the authors mention that the proposed IGSC mechanism is similar to that of  Dynamic Filter Networks by Jia et al., but  also point out a fundamental difference that their method instead aims to learn model representations. However, similar techniques in this aspect have already been explored, e.g., in solving the VQA problem. Take, for example, the CVPR 2016 paper, entitled “Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction.” The novelty of the proposed image-guided formulation seems to be limited. \n2. The math notations in the sentence right after equation (1) are incorrect. Y \\in Y_u should be corrected into Y = Y_u. Also, Y in Y_s \\cup Y_u should be Y = Y_s \\cup Y_u. \n3. Besides the technical novelty, my main concern about this work is that the experimental results are not convincing. For example, the previous technique AREN (Xie et al., 2019) is included in Table 4, but not in Table 3. However, the classification results achieved by AREN are considerably better than those yielded by the proposed technique.  For the ZSL results in Table 3, AREN achieves 60.6/71.8/67.9/39.2, while those by the proposed method are 58.3/56.9/62.1/35.2. For the GZSL results in Table 4, the acc_u result of SUN by AREN is 19%, not 9%. Furthermore, as Calibrated Stacking (CS) is also used in AREN, it is not reasonable that the comparison excludes the GZSL results of AREN+CS, which are significantly better than those by the proposed IGSC+CS. In particular, the respective harmonic means derived by AREN+CS are 35.9/66.0/64.7/36.9, while those by the proposed technique (IGSC+AC) are 34.9/48.7/39.3/33.2.\n4. The comparisons skip more recent ZSL techniques in that the proposed IGSC formulation does not consider semantic vectors of unseen classes. However, the semantic vectors of both seen and unseen classes are often made available in the training stage of ZSL and most recent ZSL techniques do consider exploring such information. It would be useful if the proposed IGSC can be generalized to take account of all the available semantic information so that comprehensive comparisons to SOTA ZSL techniques can be carried out.\n\nAfter author response: \nMy main concerns about this paper are technical novelty and weak experimental results. The authors did not make efforts to address the two aspects properly. For example, I have listed several issues in my comment 3 about the specific experimental results, but the authors did not try to address them at all. Their responses to Reviewer 4 about the transductive ZSL are not relevant to the weak experimental results of concern.  As most of my concerns are not resolved in the response, I see no evidence to upgrade my rating.\n\nI appreciate that the authors have revised their responses. However, the added response still does not address the two specific questions (the reasons why AREN is not included in Table 3, and AREN+CS not in Table 4) in my comment 3. My final evaluation about the paper is on the negative side in that its technical novelty is moderate and the experimental results are not convincing. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "RUpNdrCWu4i", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a visual-semantic embedding model useful for generalized zero-shot learning. The proposed model transforms an image into a label classifier, which is then used to predict the correct label in the semantic space. \n\nThe paper is well constructed and easy to read. It provides a good presentation of some related work and identifies the contributions as compared to existing approaches.\n\nThe experimental validation is performed on four popular public datasets and compares the performance to several state of the art approaches. The obtained performance shows similar/promising results as compared to the state of the art.\n\nFrom my perspective, the paper is missing some experimental analysis/comparison to some recent methods that are inductive only to samples and also some methods that are transductive for unseen class prototypes and unlabeled unseen test instances \n(for instance, papers mentioned in Section 4.3). First, that comparison will allow evaluating the performance of the proposed approach to more recent papers than the ones used in Section 4.2. Second, it seems that these methods, specifically the ones that are transductive for unseen class prototypes, achieve a much higher performance and it's important to evaluate the performance of the proposed method in that setting or to report on the performance loss when someone decides to use this approach in that specific setting (inductive to both unseen images and unseen semantic vectors).  A discussion that addresses the above questions/concerns could do it too.\n\nPost-Rebuttal Evaluation [FINAL] I would like to thank the authors for their response and for updating their paper based on the reviewers feedback. Following these updates, I'm changing my recommendation to Rejection for the following 2 reasons: 1- the technical novelty of this paper is moderate and there's a significant gap in the model performance as compared to state of the art, 2- the authors failed to provide convincing answers to many of the reviewers concerns, including motivation for not using semantic embeddings during the training process, and not comparing their approach to  transductive ZSL ones which achieve a higher performance.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on zero-shot learning.  The reviewers appreciated the simplicity of the method and its clear exposition.  However, concerns were raised over novelty, motivation, and empirical validation.  After reading the authors' response, the reviewers remained of the opinion that these concerns have not yet been addressed sufficiently.  Based on these points, the paper is not yet ready for publication.
Decision: Reject