Refining Visual Representation for Generalized Zero-Shot Recognition through Implicit-Semantics-Guided Metric Learning
Authors: YanHe Chen
Mei-Chen Yeh
Venue: Submitted to ICLR 2023 (2023)
Keywords: generalized zero-shot learning
metric learning
multi-class classification

Abstract: Deep metric learning (DML) is effective to address the large intra- and the small inter-class variation problem in visual recognition; however, when applied for generalized zero-shot learning (GZSL) in which the label of a target image may belong to an unseen category, this technique can be easily biased towards seen classes. Alternatively in GZSL some form of semantic space is available, which plays an important role in relating seen and unseen classes and is widely used to guide the learning of visual representation. To take advantage of DML while avoiding overfitting to seen classes, we propose a novel representation learning framework$\textemdash$Metric Learning with Implicit Semantics (MLIS)$\textemdash$to refine discriminative and generalizable visual features for GZSL. Specifically, we disentangle the effects of semantics on feature extractor and image classification of the model, so that semantics only participate in feature learning, and classification only uses the refined visual features. We further relax the visual-semantic alignment requirement, avoiding performing pair-wise comparisons between the image and the class embeddings. Experimental results demonstrate that the proposed MLIS framework bridges DML and GZSL. It achieves state-of-the-art performance, and is robust and flexible to the integration with several metric learning based loss functions.


Under review as a conference paper at ICLR 2023
REFINING VISUAL REPRESENTATION FOR GENERAL- IZED ZERO-SHOT RECOGNITION THROUGH IMPLICIT- SEMANTICS-GUIDED METRIC LEARNING
Anonymous authors Paper under double-biee blind review
ABSTRACT
Deep metric learning (DML) effective to address the large intra- and the small inter-class variation problem in visual recognition; however, when applied for generalized zero-shot learning (GZSL) in which the label belong to an unseen category, this technique can be easily biased towards classes. Alternatively in GZSL some form of semantic space available, which plays important role in relating to guide the learning of visual representation. To take advantage of DML while avoiding overfitting to seen classes, we propose a novel representation learning framework--Metric Learning with Implicit Semantics (MLIS)-to refine discrim- inative and generalizable visual features for GZSL. Specifically, we disentangle the effects of semantics on feature extractor and image classification of the model, that semantics only participate in feature learning, and classification only uses the refined visual features. We further relax the visual-semantic alignment re- quirement, avoiding performing pair-wise comparisons between the image and the class embeddings. Experimental results demonstrate that the proposed MLIS framework bridges DML and GZSL. It achieves state-of-the-ar performance, and is robust and flexible the integration with several metric learning based loss functions.
target image may
and unseen classes and is widely used
INTRODUCTION
With the consideration of real-world recognition problems that may not have defined all classes dur- ing training, generalized zero-shot lcarningg (GZSL) aims to leverage third-party data (e.g., attributes, semantic descriptors) to recognize samples from both of the seen and unseen classes Socher et al. (2013); Chao et al. (2016); Pourpanah et al. (2022). Therefore, a typical dataset for studying this problem is divided into two class sets: seen and unseen, with intersection in between Xian et al. (2018a). Only samples of the seen classes are available for training the image recognition model; however, samples of both seen and unseen classes may appear during inference. The third-party data involving class-level semantic descriptors such as attributes are important in GZSL to relate seen and unseen classes. The knowledge learned from the seen classes must be generalized recognize an unseen class through semantic information, because the visual data of unseen classes are absent in the training stage.
Whether zero-shot setting is applied or not, an image recognition task can be categorized into fine-graines and moarse-griined based on the amount of inter-class variation in visual appearance. Fine-grained recognition is considered more difficult than coarse-grainet recognition due to subtle differences between classes. Nevertheless, the large intra-class variation fine-grained recognition, often neglected in current studies, poses additional challenges to the task. Figure I displays a few samples in the CUB benchmark Wah et al. (2011). Some samples look quite differently to other samples in the same class. Such large intra-class variation in appearance is inevitable because factors such as migration and molt may affect how birds change their colors.
Deep metric learning (DML) offers natural solution to address large intra-class and small inter- class variance problem Hoffer & Ailon (2015); Wang & Chen (2017): Wang et al. (2019); Sun al. (2020). It provides flexible similarity measurement of data points. Each sample can have

---

Under review as a conference paper at ICLR 2023
different penalty in updating the model. By optimizing the contrastive loss from positive pairs (intra-class) and negative pairs (inter-class), a model can leverage wise supervision to learn embeddings and give more penalties to hard samples. Furthermore, this technique can be applied on large scale dynamic and open-ended image datasets, and can allow extensions with limited efforts to classes.
The semantic information in GZSL has been used to generate the visual features for unseen classes Xian et al. (2018b; 2019), as well as to guide the learning of discriminative visual features Ji et al. (2018); Zhu et al. (2019); Li & Yeh (2021). However, when DML is applied, a model can be eas- ily overfitted to the seen classes despite the merits of DML mentioned above Bucher et al. (2016). Furthermore, a broad family of GZSL methods learn a joint embedding space, in which the clas- sification performed by directly comparing the embedded data points with the class prototypes Xian et al. (201 8a). Learning such embedding functions can be difficult, because image features are extracted by a visual model pre-trained on ImageNet and class prototypes are human annotated at- tributes or are from word embeddings learned from text corpus. The visual and the semantic feature vectors may reflect inconsistent inter-class and intra-class discrepancies, The difficulty is exacer- bated with the integration of generative methods to create visual features for unseen classes because the distribution of synthesized features is less predictable.
Delving into the image classification framework, it is composed of feature extractor and classifier Krizhevsky et al. (2012); He et al. (2016). Previous works typically use semantics in both feature extraction and classification Hu et al. (2020); Liu al. (2021); Chen al. (2021a); Chandhok & Balasubramanian (2021). As a result, the model is forced to align visual and semantic spaces, which may be difficult because of the modality gap mentioned above. Furthermore, semanties are used in both synthesizing visual features and learning embedding functions, which may introduce serious bias towards seen classes. To better leverage the semantic information, viable solution is to refine visual embeddings by semantics, while the classification is performed only based on visual features.
Therefore, we present novel representation learning framework, named Metric Learning with Im- plicit Semantics (MLIS), for GZSL. It takes advantage of metric learning to refine discriminative visual features from the original image features, while avoiding overfitting by making good (but not too extensive) use of semantics. MLIS decouples the effect of semantics on feature extractor and image classification, so that semantics only participate in feature learning, and classification only uses the refined visual features. This decoupling facilitates the training of both tasks. In feature learning we further relax the risual-semantic alignment requirement, avoiding performing pair-wise comparisons between the image and the class embeddings. To summarize, MLIS has the following characteristics that distinguish itself from existing methods:
Semantic descriptors are given and fixed; they cannot be trained or fine-tuned. A GZSL model will reply on semantics relate the seen and unseen classes; therefore fixing seman- tics reduces model complexity and thereby also reduces the chances of being overfitted.
Semantic descriptors are involved only in training the encoder; they are not used for down- stream lasks (e.g. classification, segmentation). The downstream model utilizes only the visual features to perform the task. In this work semantic information is agnostic to the classification task.
The entire framework learns only to refine visual features, and semantic descriptors implic- itly affect the learning of visual features. We only pair an input visual feature vector up with its semantic descriptor to compute the loss, not all semantic descriptors.
Visual-semantic alignment is not strictly enforced as we rely only on visual features to per- form classification. The MLIS model is optimized to refine visual features so that when they are concatenated with the semantic descriptor of the target class, the metric learn- ing based loss is minimized. We learn semantically meaningful visual features via metric learning from the semantics without aligning the visual and the semantic space.
We conduct extensive experiments on five benchmark datasets, including CUB Wah et al. (2011) 1), AWA2 Xian et al. (2018a). SUN Patterson & Hays (2012), FLO Nilsback & Zisserman (2008), and aPY Farhadi et al. (2009). We demonstrate the superiority of the proposed method with performance on par with the state of the art. The code will be available upon paper acceptance.
2

---

Under review as a conference paper at ICLR 2023
(a) Laysan Albatross
(b) Mallard
(c) Anna Hummingbird
Figure 1: Examples showing large intra-class variation fine fine-grained recognition task.
RELATED WORK
Zero-shot learning has evolved rapidly during the last decade and therefore documenting the exten- sive literature with limited pages is rarely possible. We review a few pioneering and recent zero-shot learning methods, with an emphasis on how the semantic information is utilized in these works.
broad family of GZSL methods apply an embedding framework that learns a mapping between visual and semantic spaces. Some methods learned the mapping function from the visual space to the semantic space Akata et al. (2013); Frome et al. (2013); Socher et al. (2013). To alleviate the hubness problem caused by mapping from the high-dimensional visual space to the low-dimensional semantic space, some methods reversely learned the mapping from the semantic space to the visual space Changpinyo et al. (2016); Atzmon & Chechik (2019). The strategy of learning the mappings from these spaces to a common space has also been explored in Wang & Chen (2017). All of the embedding strategies perform the direct alignment of the visual and the semantic space, which can be difficult because of the inherent gap between them.
Recent methods fused the visual and the semantic features using concatenation Hu et al. (2020); Liu et al. (2021). For example, the FREE model Chen et al. (2021a) incorporated semantic to visual map- ping into unified generative model refine the visual features. Both of the real and the fake visual features were used to reconstruct semantic embeddings and were converted into oow-dimeniional latent space. The visual features concatenated with latent embedding, together with reconstructed pseudo-semantics, were used as inputs to train a classifier. In two-level adversarial visual- coupling (TACO) Chandhok & Balasubramanian (2021), two generative models were applied to create visual and semantic features, both of which were used to train the classifier. Our approach similar in that it uses concatenation of visual and semantic features. However, semantics are not involved in learning the target task in our method. Decoupling the effect of semantics on fea- ture extractor and the downstream task is one of the main difference of our method to these works. Moreover, these approaches feed the concatenated features to many layers for computation, which in our view contradicts the goal of avoiding direct visual-semantic alignment.
Semanties have also been used guide the learning of visual attention maps. For example, in Dense Attention Zero-shot LEaring (DAZLE) Huynh & Elhamifar (2020), different image regions were considered correspond different attributes, and therefore dense attention mechanism was proposed to obtain attribute-based features from a local image region for each attribute. Semantic- guided multi-attention (SGMA) Zhu et al. (2019) also discovered the most discriminative parts of objects, which jointly learned cooperative global and local features from the whole object well as the detected parts to categorize objects based on semantic descriptions. The stacked semantics- guided attention GAA model Ji et al. (2018) obtained semantic relevant features by using individ- class semantic features to progressively guide the visual features to generate an attention map for weighting the importance of different local regions. TCN Jiang et al. (2019) performed an element- wise product operation on visual and semantic features to determine whether image and semantic information consistent. These methods align the visual and the semantic features harder through an attention mechanism and learn the visual features based on seen classes. Therefore, the methods may easily be overfitted to seen classes, leading to inferior generalization ability on unseen classes.
Finally, semantics are also utilized to be part of the input to generative models to alleviate the extreme data-imbalance problem by synthesizing unseen class features. The synthesized features
3

---

Under review as a conference paper at ICLR 2023
(a) Visual refinement
[c] Inference
Metric learning with implicit semantics
conesentaatin
CNN
L Metrie
(Freeze)
LCE
Clazaliia
Synthetic feature refinement
Semantic embedding
C: Generater
CWtk Bockbone
CLS
Dlectmirator
Label
(b) Classifier training
a
D
LWGAN-GP
E
CLS
Ace
Teature generation
(Freeze)
Overview of the proposed MLIS framework. (a) Visual refinement: this stage learns visual representations. An encoder trained to extract discriminative and generalizable visual fea- tures. It receives both of the real and the synthetic data. (b) Classifier training: when the encoder is trained until converge, we train a classifier based on visual features alone. (c) Inference: the inference is performed by using the encoder and the classifier trained in the previous stages.
Figure 2:
can be fed to conventional classifiers so that GZSL is converted into the conventional supervised learning problem Long et al. (2017). Generative models such as variational autoencoders Bucher et al. (2017); Verma et al. (2018); Schönfeld et al. (2019); Ma & Hu (2020), Generative Adversarial Networks (GANs) Chen al. (2018); Li et al. (2019) and their variants Xian et al. (2019) have been used. Following the trend, the proposed method also contains generative model to synthesize unseen class features. Recently, the generative models are used create not only visual features but also semantic features Chen et al. (2021a): Chandhok & Balasubramanian (2021). Our approach does not allow changes of semantic space to reduce the chance of overfitting.
Our approach has some similarities with CE-GZSL Han et al. (2021). This method learns two visual embeddings (class-level and instance-level) from an image, while ours learns single one. Their class-level contrastive embedding method concatenates the visual embedding with all seman- tie embeddings tte visual embedding an instance directly affected by all semantic descriptors. We pair up only with the corresponding semantic descriptor the visual embedding of an instance is indirectly affected by other semantic descriptors and the model complexity significantly reduced.
In summary, existing methods alter the semantic space Chen al. (2021a); Chandhok & Balasub- ramanian (2021), align hardly the visual and the semantic spaces Huynh & Elhamifar (2020); Zhu et al. (2019); Ji et al. (2018); Jiang al. (2019), and let semantics participate in learning down- stream models Chen et al. (2021a); Chandhok & Balasubramanian (2021); ) Hu et al. (2020): Liu al. (2021). We believe that those characteristics hinder the development of powerful GZSL. model. Generating fake semantic features alters the original semantic space and makes the repre- sentation learning task more difficult. Strietly aligning the visual and the semantic spaces difficult to achieve and the model must handle the gap between them. Involving semantics in downstream tasks makes the model less focused on the visual cues and also increases the chance of overfitting to seen classes. The proposed MLIS method strives to escape the negative influence caused by these factors.
METHOD
The overall architecture is displayed in Fig. 2, consisting of pre-trained Convolutional Neural Network (CNN), encoder (E) and classifier (CLS). Besides, it contains generator (G) and a discriminator (D) to create visual features for seen and unseen classes. The entire procedure is divided into three stages: visual refinement, classifier training and inference. The visual refinement

---

Under review as a conference paper at ICLR 2023
stage (Fig. 2(a)) focuses on refining discriminative and generalized visual representations to facil- itate classifier learning. Through the implicit guidance of semantics specified in this method, the visual representation learning task exploits the advantages of metric learning that accurately define the notion of similarity between samples. In the classifier training stage (Fig. 2(b)), seman- ties will not participate in this process, which can only alleviate the learning difficulty of the classifier but also improve the training efficiency. After the first two stages are accomplished, the inference trivial using the trained encoder and the classifier (Fig. 2(c)). We focus on the visual refnement stage and provide technical details in the following subsections.
3.1 METRIC LEARNING WITH IMPLICIT SEMANTICS
To deal with the large intra-class variation problem, in which hard samples can be very far from the optimal solution, we apply metric learning to manipulate the similarity relationship between samples. In metric learning each sample can have its own weight to influence model training in backward propagation based the similarity other samples. This process is similar that of mining because it finds the best samples to train on.
In order to put more penalty on the loss of hard samples so that they can move towards the correct feature cluster further in each update, we apply the circle loss Sun et al. (2020) as the loss function to optimize the models. In circle loss, different similarity scores should have different penalties. This loss aims to simultaneously maximize the within-class similarity Sp and minimize the between-class similarity 8n- The loss is defined as follows:
log[1 Ee expp(7(> + m)) Σ exp((>(s())1
Leirele
where the scale factor y and the relaxation margin are two hyperparameters of this loss. We will empirically analyze their effects in Section 4.5. I. and K are the numbers of between-class and within-class similarity scores, which may vary in different batches depending on the training instances sampled in a batch. The similarity between samples is calculated by the cosine similarity.
However samples of unseen classes are not available during training in GZSL. In the absence of training data for unseen classes, metric learning can be easily overfitted to the seen classes, leading the lack of generalization capability on unseen classes. Therefore, we propose Metric Learning with Implicit Semantics (MLIS), considering both of the semantic and the visual features in calcu- lating the cirele loss. Specifically, a training sample is represented by concatenated feature vector consisting of the refined visual features (outputted by F)) and the semantic embedding of the target class (given, fixed). Note that one class contains only a semantic feature vector. Therefore, images of the same class may have different visual features, but are concatenated with fixed semantic feature vector to calculate the sample similarity to other samples. Note that we do not compare a visual embedding h and semantic embedding a. Instead, concatenate them. The dimension of the visual and the semantic embedding can be different.
This indirect alignment between visual and semantic information is more flexible and achievable because the refined visual embeddings do not require to align perfectly to the semantic embeddings. We seek for refined visual features, when concatenated with corresponding semantic features, can be classified well. The semantics act as prior to guide the visual representation learning. Furthermore, although visual feature vector is only concatenated with the ground truth semantic feature vector, the representation learning is affected by the entire semantic distribution. The encoder must consider inter-class similarities between semantic embeddings in learning the visual representations, Suppose the encoder does nothing and returns the input vector (original features): in this extreme case, the classification difficult because of the gap between two spaces. Now, suppose the encoder does perfect alignment: sample is represented by its semantic descriptor, leading to an extremely easy classification task (samples of the same class has one unique representation; each class has its own representation). The tradeoff between representation learning and classification is considered in MLIS, enabling indirect -isual-semantic alignment.
As will be demonstrated in Sec. 4.4, we have experimented with the circle loss and a few other metric learning based loss functions in MLIS. All of them have improved performance, especially on the unseen classes.
5

---

Under review as a conference paper at ICLR 2023
3.2 FEATURE GENERATION
To compensate the lack of training samples for unseen classes, a generator (G) is trained in the visual refinement stage generate visual features for unseen classes. It can generate more features for seen classes as well. Therefore, the classifier can be trained with samples of both seen and unseen classes in the next stage.
We develop our generative model based on WGAN-GP Gulrajani et al. (2017), which is easier to train than Generative Adversarial Networks (GANs) Goodfellow et al. (2014). In particular, the gradient penalty (GP) has been shown in Gulrajani et al. (2017) to be more effective than the weight clipping used in Wasserstein GAN (WGAN) Arjovsky et (2017). We follow Xian et al. (2018b) to extend the improved WGAN Guirajani et al. (2017) to conditional WGAN by integrating the semantic embeddings. The loss defined in WGAN-GP is:
LWGAN-CP = ( a)] (x, a)] + V= 1
(2)
where J G(z,a), f or (1 a) E with U(0, 1), À is the penalty coefficient, a is semantic descriptor, and is a random noise.
3.3 SYNTHETIC FEATURE REFINEMENT
Now we pay attention to the input features to our encoder. The encoder is trained by minimizing the circle loss Sun et al. (2020), which helps extract effective visual representation differentiate between seen classes. We do not compute the similarity between real and synthesized features, because those supervisory signals are not reliable. To improve the quality of the synthesized features (T,, and T.), we add classification loss to further improve the discrimination between both of the real and the fake features.
Therefore, the encoder is optimized by using two losses: the circle loss considers semantic embed- dings and uses different penalty to improve the discrimination between seen classes and to ensure the stability of the model, while the classification loss is used to separate all classes (including seen and unseen and involving real and synthesized features). The conventional cross-entropy is used in this work:
M
1 LCE = N
Σ
Yie log(tc))
(3)
where M the number of classes, Vie indicates whether the i-th sample belongs to the class c, and Pie is the predicted probability.
3.4 CLASSIFIER TRAINING AND INFERENCE
Finally we describe the classifier training stage (Fig. 2(b)) and the inference stage (Fig, 2(c)). To train the image classifier, we use not only real features but also synthesized features (for both seen and unseen classes). The features are further refined by using the encoder. Finally, we use the reñned features of all samples to train a softmax classifier as the final GZSL classifier. During inference, the pre-trained CNN model, the encoder, and the classifier are used to predict the class label of test image. Note that the classification process involves only visual features.
EXPERIMENTS
4.1 DATASETS AND EVALUATION PROTOCOL.
We evaluated the proposed method on five GZSL benchmark datasets, including CUB Wah et al. (2011), AWA2 Xian et al. (2018), SUN Patterson & Hays (2012), FLO Nilsback Zisserman (2008), and aPY Farhadi al. (2009). We followed the experimental settings provided by Xian et al. (2018a) 8a) and summarized them in Table 1. We calculated the top-1 accuracy of seen classes (S) and unseen classes (U), and reported the harmonic mean: H 2=U+8
Each experiment was run U+S
five times and the accuracy scores were reported.

---

Under review as a conference paper at ICLR 2023
Table 1:
Statisties of the datasets used in our experiments. training set, test set, S: seen
classes, U: unseen classes)
Number of classes
Number of images
Dataset
Total
Seen
Unseen
Y
Y
Y
CUB Wah al. (2011)
200
150
50
7,057
1,764
2,967
AWA2 Xian et al. (2018a)
50
40
10
23,527
5,882
7,913
SUN Patterson & Hays (2012)
717
645
72
10,320
2,580
1,440
FLO Nilsback & Zisserman (2008)
102
82
20
5,631
1,403
1,155
aPY Farhadi al. (2009)
32
12
5,932
1,483
7,924
4.2 IMPLEMENTATION DETAILS
We used the visual features provided by Xian et al. (2018a) in the experiments. They are 2048- dimensional features extracted by ImageNet-1K Krizhevsky al. (2012) pre-trained ResNet-101 He et al. (2016). For semantic features, we used category-level attributes for AWA2, SUN, and aPY. For CUB and FLO, we followed Narayan al. (2020); Xian et al. (2019); Jiang et al. (2019); Li et al. (2019); Xian et al. (2018b) and used 1024-dimensional semantic descriptors extracted from textual descriptions of the images by character-based CNN-RNN Reed al. (2016).
The encoder contains one fully connected (fc) layer with ReLU. The classifier contains one fe layer with logSoftmax. The generator and the discriminator are both fc layers with LeakyReLU, The metric learning loss functions were developed based upon Musgrave et al. (2020).
Table 2: Comparison with state-of-the-an methods. U and S are the top-1 accuracy of unseen and classes. H is the harmonic mean of U and S. The best result is marked in red, and the second best result marked in blue.
Method
Venue
CUB S
AWA2
SUN
FLO S
aPY
H
U
S
U
H
U
H
TCN
ICCV'19
52.6
52.0
52.3
61.2
65.8
63.4
31.2
37.3
34.0
24.1
64.0
35.1
LisGAN
C9PR'19
46.5
57.9
51.6
52.6
76.3
62.3
42.6
36.6
39.4
57.7
83.8
68.3
34,3
68.2
45.7
E-VAGAN-D22
CVPR'19
70.6
DAZLE
CVPR'20
56.7
59.6
58.
60.3
75.7
67.1
52.3
24.3
33.2
DVBE
CVPR 20
53.2
60.2
56.5
63.6
70.8
67.0
45.0
37.2
40.7
32.6
58.3 41.8
Keshari
CVPR 20
44.8
59.9
51.3
59.5
73.4
65.7
44.8
42.9
43.8
7IC.LDM
Sensors" 21
40.3
62.9
49.1
31.9
92.5
47.4
23.5
33.9
27.6
WACV'21
Li et al.
AAAU21
51.1
58.2
54.4
56.9
80.2
66.6
36.6
47.6
41.4
CN-GZSI
ICLR'21
49.9
50.7
50.3
60.2
77.1
67.6
HSVA
NIPS'21
52.7
58.3
55.3
56.7
79,8
66.3
GCM-CF
CVPR'21
61.0
59.7
60.3
60.4
75.1
67.0
47.9
37.8
42.2
37.1
56.8
44.9
FREE
ICCV'21
VGSE-SMO
C2PRR222
24.1
45.7
31.5
45.7
66.7
$4.2
25.5
35.7
29.8
TDCSS
CVPR'22
44.2
62.8
51.9
59.2
74,9
66.1
54.1
85.1
66.2
MLIS GZSL
Ours
66.1
66.7
66.4
62.0
78.2
69.2
45.6
37.9
41.4
62.8
82,9
71.5
34.6
66.1
45.4
4.3
COMPARISON WITH STATE-OF-THE-ARTS
First, we compare the recognition performance of the proposed method with state-of-the of-the-art methods including Jiang al. (2019); ;); et al. (2019); Xian et al. (2019); Huynh Elhamifar (2020); Min et al. (2020); Keshari et al. (2020); Liu et al. (2021); Chandhok & Balasubramanian (2021): Li et al. Elhoseiny: Chen et (2021b); Yue et al. (2021): Chen al. (2021a); Xu
(2021); 1); Skorokhodov et al. (2022); Feng et al. (2022). Table 2 shows the comparison results.
The proposed method achieve the best results CUB and AWA2. In the CUB experiment, our performance is 4-5% higher in accuracy than the second best performance for both seen and unseen classes, and is 6.1% higher in h-mean. In the AWA2 experiment, we also have the best h-mean recognizing seen and unseen classes. We
accuracy and have more consistent performance achieve the second best results in FLO and aPY in all evaluation metrics. The accuracy on SUN is not good those other datasets. We observe that the accuracy rates of all methods SUN

---

Under review as a conference paper at ICLR 2023
relatively low. One possible reason is that this dataset is composed of scene categories such as "arena rodeo", "arena soccer", "art school", "art studio" rather than specific objects. Moreover, has more classes (717 classes) than other datasets and has fewer images (only 20) per category.
The proposed MLIS framework re-in re-investigates the role of semantics in GZSL. Although the training set contains samples only from the categories, encoder extract discriminative visual fea- tures that are also generalizable to unseen classes because we maintain the original semantic space. The learned visual representations are implicitly affected by the semantic embeddings well the relationship between them. The classification is performed solely based on visual features; therefore, image features are not required to be matched with their semantic features, avoiding the requirement of performing direct disual-semante alignment. However, the learned visual embedding space is still semantically meaningful, so that the classification performance in this space is satisfactory.
4.4 ABLATION EXPERIMENTS
We investigate the effect the critical components introduced in the proposed framework. This ablation experiment involves the comparison of methods including (1) the direct application of Met- ric Learning (MI.)-irche loss Sun et al. (2020) in this experiment- representation learning, (2) the proposed metric learning with Implicit Semantic (IS), and (3) the addition of synthetic feature refinement (SFR), in which classification loss further considered to improve the discrimination of visual representations between seen and unseen classes. Table 3 shows the results.
Table 3: Ablation study effects of components in the proposed framework
CUB S
AWA2
SUN $
FLO
aPY
ML
IS
SFR
U
H
U
H
C
H
U
S
H
U
S
X
54.7
45.0
49.3
53.7
73.6
62.1
46.0
36.1
411.5
61.8
81.1
70.1
29.9
60.9
40.1
65.0
67.0
66.0
78.0
69.4
45.6
36.2
40.4
60.8
79.8
69.0
35.1
62.2
44.9
66.6
66.7
66.6
62.6
79.7
70.1
47.3
38.0
42.2
62.1
85.6
72.0
35.2
68.0
46.4
We can observe that using metric learning alone does not achieve state-of-the-ard performance on all datasets. For example, the performance in CUB, AWA2, and aPY are not satisfactory. With the proposed implicit semanties, the performance those datasets can be significantly boosted, as displayed in the second row of Table 3. The proposed IS method designed to be compatible with metric learning based loss, taking advantage of how it learns discriminative visual features. Finally, SFR can used to further improve the discrimination between seen and unseen samples. This component helps improve the performance in all datasets. The results validate that metric learning with implicit semantics (IS) achieves a better discrimination between samples than using metric learning alone. Adding SFR can achieve stable performance of the (or the unseen) class and improve the accuracy of the other.
Table 4: Performance of MLIS with three different metric learning based loss functions
CUB
AWA2
Loss
U
S
H
U
S
H
Circle loss Sun et al.
54.7
45.0
49.3
53.7
73.6
62.1
(2020)
/
65.0
67.0
66.0
62.6
78.0
69.4
Contrastive loss Hadsell (2006)
46.4
51.2
48.7
54.0
73.3
62.2
58.7
62.5
60.5
53.8
76.9
63.3
Multi-similarity loss Wang et al. (2019)
51.3
25.2
33.8
45.7
77.0
57.4
65.1
67.9
66.5
56.5
80.4
66.4
To further demonstrate that MLIS can improve other metric learning based methods, replaced the circle loss with few different loss functions including contrastive loss Hadsell et al. (2006) and multi-similarity loss Wang al. (2019). As the goal of this experiment was not to compare the performance of these loss functions, we did not finettune the hyperparameters for each of them. We reported the experimental results on CUB (fineggrained) and AWA2 (coarse-grained). Table
8

---

Under review as a conference paper at ICLR 2023
shows the results. With the appropriate usage of semantics, all of the metric learning methods can be significantly boosted and are less biased toward seen classes.
4.5 HYPERPARAMETER ANALYSIS
In this section, we analyzed the effect of the number of synthesized features on the performance. In the MLIS framework, we can synthesize features in the visual refinement stage and the classifier training stage. Figure 3 shows the results on the CUB dataset (ine-grained) and the AWA2 dataset -raaieeg which Fig. 3())bb) shows the analysis results in the visual refinement stage and Fig. 3(c)(d) shows those in the classifier training stage. In the visual refinement stage, relatively great performance was obtained when the generator created five samples for each unseen class. Syn- thesizing more samples, somehow surprisingly, decreased the accuracy of unseen classes. However, in the classifier training stage, we needed a sufficient amount of synthesized features for each unseen class to achieve good performances. This is explainable because rely solely visual cues to perform classification.
200
6114
E
Withet: carale canner doser
lorthrt cavales becenter dosee
Soyter sarales dioee
Sprihii caralis for unirre dlosee
(a) CUB
(b) AWA2
(c) CUB
(d) AWA2
Figure 3: Effect of the number of synthesized samples the classification performance. (a)-(b):the
visual refinement stage, (c)-(d): the classifier training stage.
Finally, we analyzed the effect of the two hyperparameters m and y in circle loss on the classification performance. The relaxation factor m controls the radius of the circular decision boundary, and the scale factor 7 determines the largest scale of each similarity score. Figure 4(a)(b) show the effect of the hyperparameter m on CUB and AWA2. A small m value (below 0.4) favorable, which consistent with the finding reported in the original paper Sun et al. (2020). Figure ((c) show the effect of the hyperparameter y on CUB and AWA2. The classification performance is fairly stable with respect to this parameter.
Blvee
oileem
2045
CLO
78:55
AcCJOCY D80
753
NCC Arcome (M)
05:1
hyperporameter of Circle loss 'm!
hyperzarameter of Circle loss 'm'
Hyperparameter of Ordee loss
Hyperparameter of Circle loss
(a) CUB
(b) AWA2
(c) CUB
(d) AWA2
Figure 4: Effect of the hyperparameters m and in circle loss to the classification performance.
5 CONCLUSION
In this paper, we re-investigate the role of semantics in the generalized zero-shot learning task. We identify and address two major issues: overfitting to seen classes and large intra-class variation. We propose MLIS a visual representation framework to shine metric learning in the generalized zero- shot setting. The MLIS method disentangles the effects of semantics on feature extractor and image classification of the model, and prevents the direct alignment of the visual and the semantic spaces. In the experiments, we demonstrate the superiority the MLIS method with performance on par with the state of the art five benchmarks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for attribute-based classification. In Proc. of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 819-826, 2013.
Martin Arjovsky. Soumith Chintala, and Léon Bottou. Wasserstein gan. In 2017. doi: 10.4550ARXII..011..70
Yuval Atzmon and Gal Chechik. Adaptive confidence smoothing for generalized zero-shot learning. In IEEE Conference on Computer Vision, 2019.
Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Improving semantic embedding consistency by metric learning for zero-shot classification. In European Conference on Computer Vision, 2016.
Maxime Bucher, Stephane Herbin, and Fred' eric Jurie. Generating visual representations for zero- shot classification. In IEEE International Conference Computer Vision, 2017.
Shivam Chandhok and Vineeth N Balasubramanian. Two-level adversarial visual-semantic coupling for generalized zero-shot learning. In IEEE Winter Conference on Applications of Computer Vision, pp. 3100-3108, 2021.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero- shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition the wild. In European Conf. Computer Vision, 2016.
Long Chen, Hanwang Zhang, Wei Liu Jun Xiao, and Shih-Fu Chang. Zero-shot visual recognition using comantics-preserviinn adversarial embedding networks. In IEEE Int. Conf. Computer Vision and Pattern Recognition, 2018.
Shiming Chen, Wenjie Wang. Beihao Xia, Qinmu Peng, Xinge You, Feng Zheng, and Ling Shao. Free: Feature refinement for generalized zero-shot learning, In IEEE/CVF International Confer- ence on Computer Vision, pp. 122 131, 2021a.
Shiming Chen, GuoSen Xie, Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, and Ling Shao. Hsva: Hierarchical semantic-visual adaptation for zero-shot learning. Advances in Neural Information Processing Systems, 1121166221166644 2021b.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1785, IEEE, 2009.
Yaogong Feng, Xiaowen Huang, Pengbo Yang. Jian Yu, and Jitao Sang. Non-generative generalized zero-shot learning via task-correlated disentanglement and controllable samples synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9346-9355, 2022.
Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In Proc. of Neural Information Processing Systems, 2013.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Pro- cessing Systems, 2014.
2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in Neural Information Processing Systems, 30, 2017.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pp. 1735- 1742. IEEE, 2006.
10

---

Under review as a conference paper at ICLR 2023
Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. Contrastive embedding for generalized zero-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE International Conference on Computer Vision and Pattern Recognition, pp. 770- 778, 2016.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International Workshop on Similarity- Puttern Recognition, 2015.
Ping Hu, Stan Sclaroff, and Kate Saenko. Uncertainty-aware learning for zero-shot semantic seg-
mentation. Advances in Neural Information Processing Systems, 33:21713-21722 2020.
Dat Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute- based attention. In IEEE International Conference on Computer Vision and Pattern Recognition, pp. 4483 4493, 2020.
Zhong Ji, Yanwei Fu, Jichang Guo, Yanwei Pang. Zhongfei Mark Zhang, et al. Stacked semantics- guided attention model for fine-grained zero-shot learning. Advances in Neural Information Pro- cessing Systems, 31, 2018.
Huajie Jiang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Transferable contrastive network for generalized zero-shot learning. In IEEE International Conference on Computer Vision. pp. 9774, 2019.
Rohit Keshari, Richa Singh, and Mayank Vatsa. Generalized zero-shot learning via over-complete distribution. IEEE International Conference on Computer Vision and Puttern Recognition, pp- 15300-13308. 2020.
Alex Krizhevsky, Nya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Fang Li and Mei-Chen Yeh. Generalized zero-shot recognition through image-guided semantic classification. In IEEE Conference on Image Processing, 2021.
Jingjing Li, Mengmeng Jing, Ke Lu, Zhengming Ding, Lei Zhu, and Zi Huang. Leveraging the invariant side of generative zero-shot learning. In IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1402-7411, 2019.
Xiangyu Li, Zhe Xu, Kun Wei, and Cheng Deng. Generalized zero-shot learning via disentangled representation. In AAAI Conference on Artificial Intelligence, volume 35, pp. 1966-1974, 2021.
Jingyi Liu, Caijuan Shi, Dongjing Tu, Ze Shi, and Yazhi Liu. Zero-shot image classification based learnable deep metric. Sensors, 21(9):3241, 2021.
Yang Long, Li Liu, Ling Shao, Fumin Shen, Guiguang Ding, and Jungong Han. From zero-shot learning to conventional supervised classification: Unseen visual data synthesis. In IEEE Int. Conf. Computer Vision and Pattern Recognition, 2017.
Peirong Ma and Xiao Hu. A variational autoencoder with deep embedding model for generalized zero-shot learning. In AAAI Conference on Artificial Intelligence, 2020.
Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zueng-Jun Zha, and Yongdong Zhang. Domain-aware visual bias eliminating for generalized zero-shot learning. In IEEE International Conference on Computer Vision and Pattern Recognition, pp. 12664- 12673, 2020.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Pytorch metric learning, 2020.
Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snoek, and Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. In European Con- ference on Computer Vision, pp. 479-495. Springer, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, Dec 2008.
11

---

Under review as a conference paper at ICLR 2023
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recog- nizing scene attributes. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 2751-2758. IEEE, 2012.
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi- Zhao Wang, and Q. M. Jonathan Wu. A review of generalized zero-shot learning methods. In arXiv:2011.086411 2022.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine- grained visual descriptions. In IEEE International Conference Computer Vision and Pattern Recognition, pp. 49-58, 2016.
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero- and few-shot learning via aligned variational autoencoders. In IEEE Int. Conf. Computer Vision and Puttern Recognition, 2019.
Ivan Skorokhodov and Mohamed Elhoseiny. Class normalization for (continual)? generalized zero- shot learning. International Conference on Learning Representations.
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Oshert Bastani, Christopher D. Manning, and An- drew Y. Ng. Zero-shot learning through cross-modal transfer. In Neural Information Processing Systems, 2013.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Cirele loss: A unified perspective of pair similarity optimization. In IEEE International Conference on Computer Vision and Pattern Recognition, pp. 6398 2398-6407,20 2020.
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. Generalized zero-shot learn- ing synthesized examples. In IEFE Int. Conf. Computer Vision and Pattern Recognition, 2018.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The caltech-ucsd birds-200-200 dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology. 2011.
Qian Wang and Ke Chen. Zero-shot visual recognition via bidirectional latent embedding. Interna- tional Journal of Computer Vision, 2017.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5022-5030, 2019.
Yongqin Xian, H. Christoph Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning com- prehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018a
Yongqin Xian, Tobias Lorenz, Bern Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In IEEE International Conference on Computer Vision and Pattern Recogni- tion, pp. 5542-5551, 2018b.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generat- ing framework for any-shot learning. In IEEE International Conference Computer Vision and Pattern Recognition, pp. 10275-10284, 2019.
Wenjia Xu, Yonggin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Vgse: semantic embeddings for zero-shot learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9316-9325, 2022.
Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Counterfactual zero- shot and open-set visual recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15404-15414, 2021.
Yizhe Zhu, Jianwen Xie, Zhiqiang Tang, Xi Peng, and Ahmed Elgammal. Semantic-guiee guided multi- attention localization for zero-shot learning. Advances in Neural Information Processing Systems, 32, 2019.
12


Reviews:
[{"review_id": "opxRKwCZS-D", "reviewer": "Reviewer_wRXx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Metric Learning with Implicit Semantics (MLIS) to refine discriminative and generalizable visual features for GZSL. MLIS disentangles the effects of semantics on feature extractor and image classification of the model, so that semantics only participate in feature learning, and classification only uses the refined visual features. MLIS relaxes the visual-semantic alignment requirement, avoiding performing pair-wise comparisons between the image and the class embeddings.Experiments are conducted on golden GZSL datasets.", "summary_of_the_review": "Due to limited novelty, I must reject it!", "strength_and_weaknesses": "The organization is good. Nevertheless，\n\n\n1.The motivation for the usage of metric learning for improving the generalization is not new. In fact, using metric learning to solve ZSL have been explored in previous works (e.g., Yannick Le Cacheux et al. In ICCV19 ). The overall compared methods are limited, and more methods should be surveyed and compared.\n\n2.The framework is actually incremental improvements on existing methods, e.g., improved on Yongqin Xian et al. in CVPR 2018 by feeding additional unseen features as inputs. From this point, the novelty is limited to the community, since this kind of strategies have been explored in the community (e.g., in LsrGAN of ECCV20). Furthermore, the idea of disentangling feature extraction and classifier training is not new. All these aspects reduced the contributions of this paper to the community. \n\n3.The overall writing is poor. The formulas are also confusing. Please amend them accordingly.\n\n4.How to initialize the network weights, from scratch? What’s the specific network architecture? \n\n5.Since additional operation such as two-step decoupled training is used for the proposed method, I am doubted about the tradeoff between the running time and accuracy.\n\n6.The parameter analysis is shown, however, what’s the specific parameters for achieving these results? It seems the authors have choosed to compare with the methods worse than theirs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OAvLfX7LvW", "reviewer": "Reviewer_iKnN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper talks about ZSL. The authors ground their motivation on the fact that in-class samples could be still very different from each other, and they propose to tackle this problem in ZSL. The main contribution yields the concept of Metric Learning with Implicit Semantics (MLIS) and it is plugged into the training stage of a generative ZSL model.\n\nThe proposed model is basically extending a GAN in ZSL. MLIS classifies the visual samples by a circle loss when the generative model is being trained. An additional cross-entropy loss is also applied. A default GAN loss counts the last part of the model's training objective.", "summary_of_the_review": "I would say the quality of this paper is below the expectation of a regular ICLR paper. It lacks novelty, and some designs are not properly explained.", "strength_and_weaknesses": "## Pros\n---\n* Clear expression of the method\n* Good reference to the related articles (though it seems to be too much)\n\n## Cons\n---\n* This paper has several unclear or confusing parts. Some may be due to sloppy writing, but some show a lack of in-depth thinking into the problem.\n  * In Fig. 2, the MLIS loss is termed as $L_{Metric}$, but in the main body, it becomes $L_{circle}$. \n  * $L_{circle}$ is based on cosine similarity. So what about $L_\\text{CE}$? There is no clue of how the class-wise probability $p$ is obtained.\n  * The authors need to clarify whether the generated unseen samples are involved in computing $L_{circle}$ and the values of $L$ and $K$. This is important as otherwise the generative model would not be related with $L_{circle}$ and then could be removed from the entire framework.\n* $L_{circle}$ is built upon concatenated features ($h$ and $a$). Usually, they need to be calibrated before concatenation. Uncalibrated features could lead to problems in computing cosine similarity, including\n  * Bad magnitude normalizer\n  * Un-normalized similarity bias provided by $a$\n* $L_{circle}$ and $L_\\text{CE}$ are functionally overlapping, but their presence is different, i.e., circle loss vs cross-entropy. I think the authors need to explain this inconsistency in loss design.\n* The overall design is not very novel to me. GAN + classification loss just looks like InfoGAN with some add-ons, while the circle loss is also off-the-shelf. The concept of MLIS does not bring anything new as it is just a simple modification on the input side of the circle loss. This does not convince me to be a good ICLR paper.\n* Please be aware of the font difference in subscriptions between $L_{circle}$ and $L_\\text{CE}$.\n* Marginal performance gain", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LqVzdgWqa9Q", "reviewer": "Reviewer_WJ1v", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method, metric learning with implicit semantics (MLIS), for generalization zero-shot learning (GZSL). It leverages a metric learning loss during training to refine the visual feature and better align the semantic information. In addition, to alleviate the feature bias towards the seen classes, it also uses a generative model to simulate the unseen class features to improve the overall quality. The proposed MLIS was evaluated on multiple benchmark datasets for GZSL, and competitive performance has been observed.", "summary_of_the_review": "This paper proposes an interesting method that can improve the feature quality for GZSL through metric learning and feature synthesis. However, it does not have enough novelty, and the performance margin is not large enough to justify an acceptance.", "strength_and_weaknesses": "Strength:\n1. The paper is written clearly and easy to follow\n2. The proposed method is extensively evaluated on multiple benchmark datasets and shows competitive results.\n\nWeakness:\n1. This paper lacks novelty. Using metric learning to improve the feature quality and generative model to synthesize unseen features are well studied in the GZSL domain. The essential techniques used in this paper, including the WGAN-GP for feature generation, and the circle loss, are proposed in previous works.\n2. The empirical performance is not compelling enough to me, considering the method is mainly composed of existing techniques.\n3. It lacks some qualitative results to justify the motivation. The paper was motivated by the fact that fine-grained GZSL classification is more sensitive to intra-class variations. So it is better to have some visualization (like tSNE graphs) to demonstrate that the refined features have fewer intra-class divergences compared to the raw features and other methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9l2imadhO6l", "reviewer": "Reviewer_tWPT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces deep metric learning for generalized zero-shot learning, and proposes a novel framework named MLIS to avoid overfitting to seen classes. MLIS disentangles the effect of semantics on feature learning and classification, making the training of the two tasks more effective. Moreover, this paper concatenates semantic descriptors and visual features directly instead of aligning them, reducing the model’s complexity. Experimental results demonstrate the effectiveness of the proposed MLIS.", "summary_of_the_review": "This paper is clear and easy to follow. However, the contributions still need to be explained more clearly. A fair comparison is also needed on the CUB dataset. ", "strength_and_weaknesses": "Strengths\n1.\tThis paper is well written and easy to follow. \n2.\tThe proposed framework is simple and effective, and easy to reproduce. \n\nWeaknesses\n1.\tThe motivation is unclear. The authors consider that semantics used in both synthesizing visual features and learning embedding functions will introduce bias toward seen classes. However, some methods [1][2] using semantics seem to get better results. Please compare with them and give more explanations for the motivation of the proposed paper. Moreover, it would be more convincing to add comparative experiments that use semantics but do not decouple.\n2.\tThe comparison on the results of the CUB is unfair. Most of the methods, such as FREE and f-VAEGAN-D2, utilize 312-dimensional attributes as the auxiliary semantic information. You need to experiment with attributes rather than 1024-dimensional semantic descriptors if you want to compare with these methods.\n3.\tSome recent methods like [2] and [3] are ignored to be compared. \n4.\tI wonder why the results are so low using only ML in the ablation experiments. The results are even lower than some simple early methods like f-CLSWGAN [4] and f-VAEGAN-D2 [5]. More explanations can be given. \n5.\tA minor problem. In section 3.4, the authors said that synthesized features including both seen and unseen classes are used to train the final classifier. However, it seems that only the synthesized unseen features are used. \n\n[1] Generative Dual Adversarial Network for Generalized Zero-shot Learning. CVPR 2019.\n[2] Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification. ECCV 2020.\n[3] Contrastive Embedding for Generalized Zero-Shot Learning. CVPR 2021.\n[4] Feature Generating Networks for Zero-Shot Learning. CVPR 2018.\n[5] F-VAEGAN-D2: A feature generating framework for any-shot learning. CVPR 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject