Attribute Alignment and Enhancement for Generalized Zero-Shot Learning
Authors: Nannan Lu
MingKai Qiu
Venue: Submitted to ICLR 2023 (2023)
Keywords: zero-shot learning
image classification
attribute alignment
graph neural network
attention network

Abstract: Generalized zero-shot learning (GZSL) aims to recognize both seen and unseen classes, which challenges the generalization ability of a model. In this paper, we propose a novel approach to fully utilize attributes information, referred to as attribute alignment and enhancement (A3E) network. It contains two modules. First, attribute localization (AL) module utilizes the supervision of class attribute vectors to guide visual localization for attributes through the implicit localization capability within the feature extractor, and the visual features corresponding to the attributes (attribute-visual features) are obtained. Second, enhanced attribute scoring (EAS) module employs the supervision of the attribute word vectors (attribute semantics) to project input attribute visual features to attribute semantic space using Graph Attention Network (GAT). Based on the constructed attribute relation graph (ARG), EAS module generates enhanced representation of attributes. Experiments on standard datasets demonstrate that the enhanced attribute representation greatly improves the classification performance, which helps A3E to achieve state-of-the-art performances in both ZSL and GZSL tasks.

Under review as a conference paper at ICLR 2023
ATTRIBUTE ALIGNMENT AND ENHANCEMENT FOR GENERALIZED ZERO-SHOT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Generalized zero-shot learning (GZSL) aims to recognize both seen and unseen classes, which challenges the generalization ability of a model. In this paper, we propose novel approach to fully utilize attributes information, referred to as attribute alignment and enhancement (A3E) network. It contains two modules. First, attribute localization (AL) module utilizes the supervision of class attribute vectors to guide visual localization for attributes through the implicit localization capability within the feature extractor, and the visual features corresponding to the attributes (attribute-visuu features) are obtained. Second, enhanced attribute scor- ing (EAS) module employs the supervision of the attribute word vectors (attribute semantics) to project imput attribute visual features to attribute semantic space us- ing Graph Attention Network (GAT). Based on the constructed attribute relation graph (ARG), EAS module generates enhanced representation of attributes. Ex- periments on standard datasets demonstrate that the enhanced attribute representa- tion greatly improves the classification performance, which helps A3E to achieve state-of-the t--the--rt art performances in both ZSL and GZSL tasks.
INTRODUCTION
Zero-shot learning aims to recognize unseen classes that have not been appeared during training phase, common solution resort auxiliary information bridge the gap between seen and un- seen domains to achieve knowledge transfer from the seen to the unseen. Semantics are the most frequently used auxiliary information for ZSL, either by class descriptions, word vectors (Mikolov 2013) or attributes (Farhadi et al. 2009). A general paradigm (Xic et al. 2019 Zhu et al. 2019 Huynh & Elhamifar 2020b] Min et al. al.[22020)) Xie al. 2020 Ge et al. [021] Liu et al. [202 1b, Chen et al. [ 2022) is to learn mapping that projects visual features of seen samples into an embed-ding space to align with semantic attributes. With the assumption that seen and unseen do- mains share the same attribute space, the learned knowledge from seen classes is easily transferred to the unseen ones. And then, the subsequent classi-fication is accomplished by measuring compatibil- ity scores between the projected features and the attribute prototypes. Recent works embeddings turn to local features of image parts, i.c. part-based embedding meth-ods (Elhoseiny et al.) 2017). to learn discriminative features easy for classification. Comparatively, gener-ative methods (Xian al.; 2019b Huynh& & Elhamifar] 2020a Ma & Hu 2020 Han et al.) 2021] Chen et al., 2021 Chou al. (021) utilize semantic information of unseen classes synthesize unseen visual features by generative model, such as generative adversarial network (GAN) (Goodfellow et al. 2020) or vari- ational autoencoder (VAE) (Kingma & Welling 2013), so that convert zero-shot classification to the traditional supervised model learning that could be trainable with generated samples. However, the features inferred from semantic information mostly are high-level visual representation, which
et al.
are often non-discriminative to class recognition (Huynh & Elhamifar 2020b
Xian et al., 2019b
Huynh & Flhamifar 2020a).
Recently, generalized zero-shot learning (GZSL) for its rigorous and realistic nature has received increasing attention in this field, where seen classes and unseen classes constitute the testing space. Embedding methods are inherently inferior in GZSL since the model training merely relies sam- ples of seen classes, and thus inevitably biases towards the seen ones. Moreover, the sesnatcsemantic alignment in embedding models is just operated in seen domain, and the visual di- di-vergence between seen and unseen domains may strengthen the bias, namely domain shift (Fu et al.) 2014). Differ- methods have been explored to improve the model performance in GZSL. Some studies try to

---

Under review as a conference paper at ICLR 2023
Attributes: Head Eyes Book Neck
Book
Avcí
Bonk
Navh
PMI
Rahanzed Regreentotions
GA
GAT
GA
Output
Fjycs
Eyyo
AVFs
Head
Head
Input
I.
Figure 1: Attribute Localization.
mitigate the bias by introducing constraints on losses to calibrate output pre-diction probability, which usually require unseen semantics as side infor-mation (Huynh & Elhamifar [020b] Xie et al.) 2020j. The Parts Relation Rea-soning is used RGEN (Xie et al. (020)) to capture appearance relationships among image parts, which is believed to be a complementary cue for improving the performance. GCNZ (Veličković et al. 2018) utilizes class relationships to infer classifier parame- ters directly from knowledge graph. Relation learning is no novelty to ZSL, however, the semantic relationship between attributes is rarely explored in previous works. Huynh & Elhamifar (2020b) have informed by introduc-ing word vectors of attribute that there wealth of semantic infor- mation attributes beyond the commonly used class attribute vectors. There are also rich semantic relationships between attributes, which can be transferred to visual domain help mitigate visual- semantic gap. Once the relations between attributes are modeled, it is possible to enhance the fi-nal prediction of classes by the interplay of attributes. Existing methods tried capture the semantic relations in the at-tributes, such as using the entanglement of CNN and GCN based on knowledge graph about attrib-utes (Hu et al. 2022). However, despite the fact that nodes graph are explic-itly defined as attributes, those methods lack mechanism to accurately align nodes to the corresponding attributes. To the best of our knowledge, the fusion of relation learning and attention mechanism has not been studied in ZSL.
Accordingly, we propose an attribute alignment and enhancement (A3E) network for GZSL, which incorporates attribute alignment (AA) pipeline and attribute enhancement (AE) module. AA pipeline consists of attribute localization (AL) module and attribute scoring (AS) module, this novel ap- proach of attribute alignment allows model to subtly catch visual features corresponding attributes (namely attribute-visual features, AVFs) by fully utilization of attribute knowledge (both class at- tribute vectors and attribute word vectors). Compared to previous part-based methods that require complex accessories such as attention module and part detector, A3E simplifies its AA pipeline to single convolutional layer with single linear transformation, and still delivers competitive re- sults. Most importantly, the resulted AVFs serve as the carriers for attributes which support the subsequent attribute enhancement process. In order to model the relations of attributes, AE module first constructs an attribute-relatio graph (ARG), where relation-ships of attributes are quantified as graph edges, then, facilitated by graph neural networks, embeds the input AVFs into attribute semantics space. The enhanced attribute features are obtained through the outputs of graph nodes. Figure [ demonstrates the basic process of AE module. Experiments in three standard ZSL datasets show that A3E reaches the state-of-the- art results in both ZSL and GZSL without extra information from unseen classes or auxiliary constraints on output probabilities, verifying the advantages of our proposed method.
Our contributions can be summarized as:
A novel attribute enhancement (AE) module is created to explicitly model the relationship between attributes, and the enhanced attribute representation is generated with attribute-rclations modeled inside.
To align graph nodes with attributes, an efficient attribute alignment (AA) pipeline designed to generate visual fea-tures corresponding to attributes, namely attribute-visua features (AVFs).
We propose an attribute alignment and enhancement (A3E) network that based on the AA pipeline and AE module, innovative combination of attention mechanism and semantic-relation learning.
2

---

Under review as a conference paper at ICLR 2023
Extensive experiments on three bench-mate show that our design can significantly improve results in both ZSL and GZSL tasks.
RELATED WORK
There are two main paradigms for ZSL/GZSL: generative methods (Xian et al.; [019b] Huynh &
Elhamifar 2020a Ma & Hu| [220 Han et al., 2021; Chen et al.] 2021a la cj Chou et al., (022)) and
embedding methods (iie et al., 2019 Zhu et al. 2019;
Huynh & Elhaaiffrr 2020b, Min et al,)
2020 Xie et al., 2020: Ge et al. 2021 Liu et al.
2021b; Chen et al., b021b; 2022). Generative
methods covert ZSL problem into traditional supervised learning using visual features synthesized by generative models for unseen classes (Liu et al. 202 However, generative models such as
GAN or VAE are often difficult to generate high-quality synthetic samples for unseen classes to train classifiers (Pourpanah et al.] 2022). On the other hand, embedding methods learn mapping that aligns visual features with semantic prototypes, therefore achieve knowledge transfer from seen unseen classes via their sharable semantics. According the mapping space, embedding methods can be divided into three categories: visual space embedding (Zhang et al. 2017), semantic space embedding (Zhu et al. 2019 Huynh & Elhamifar 2020b Xie al. 2020 Liu al. 2021b) and common space embedding (Min et al., 2022), with their respective pros and cons.
As its name suggests, semantic space embedding projects visual features into semantic space. Re- cent studies further suggest that global visual features are detrimental to classification (Xie et al. 2019 Zhu et al., 2019; Huynh & Elhamifar [0200 Xie et al., 2020). Instead of using noisy global features, part-based embedding methods try improve classification performance by locating dis- criminative parts image. (Flhoseiy et (2017) deployed visual part detector to link text descrip- tions with corresponding image regions, which would be fed into the part-based visual classifiers. SGMA (Zhu et al. 2019) employed a multi nulti-attention module and DAZLE (Huynh & Elhamifar 2020b) constructed a hierarchical linear structure, all in order to focus the model discriminative regions in image. Whereas, the model with part detector attention module would become complex, so that make it difficult to train and optimize. SELAR (Yang et al. (021) proposed to localize part features by the implicit localization ability within feature extractor, where the complex attention module is replaced with a single convolution layer. Most of the above models use class attribute vectors as semantic information. However, since the attribute space spanned by class attribute vec- tors is inevitably suffered from hubness problem (Zhang et al. (2177). the choice of embedding space is still an issue that is worth to explore in subsequent study.
With the increasing attention paid to ZSL, various techniques from other fields were also incorpo- rated into ZSL models, such as knowledge distillation (Chen et al.) 2022). meta-learning (Verma al. 2020) and graph learning (Xie et al.] [2020] WWang et al.J 2018). Graph Neural Networks (GNNs) (Kipf & Welling 2017: Veličković et al. 2018) were proposed to model non-Euclidean data, especially for those with graph structure, Veličković et al. (2018) firstly introduced Graph Con- volutional Networks (GCN) (Kipf & Welling 2017) to explicitly model relations between classes in ZSL by knowledge graph. And RGEN (Xic et al. 2020) employed GCN to represent the rela- tions among local image regions. Whereas, none of them have explored the semantic relations that implied within attributes.
Inspired by the advantages and deficiencies of previous works, proposed A3E employs dual embedding strategy to fully utilize the rich semantics beneath attributes, and incorporates GAT (Veličković et al. 2018) dynamically model the semantic relations between attributes.
3 ATTRIBUTE ALIGNMENT AND ENHANCEMENT
In this section, we will specify how and why the A3E network is proposed. Here we follow the pipeline that A3E processes the samples, and present the whole structure and details of our model.
3.1 ATTRIBUTE LOCALIZATION
In order to explore those rich semantics and relations between attributes, we need to obtain the visual representations for attributes first. Instead of generating discriminative regions using various of attention modules, Yang al. (202T) innovated to utilize the implicit attribute localization ability
3

---

Under review as a conference paper at ICLR 2023
within feature extractor (CNNs) to get part location, here refer to it as attribute localization (AL). AL greatly reduces the complexity of the model by replacing the complicated attention module with single 1xl convolution:
cone (v)
where, v = up (x:)is the visual features with x W x C dimensions extracted by the backbone network yp (+). x; is the i-th imput image, and cone (-) is the 1x1 convolution with 1 x 1 C. x A parameters. E IR XxXX is the output features with attribute localization, referred to as attribute features.
Under the supervision of attribute vectors, this simple convolution could gather most important spacial information of attributes. The loss function of AL module is defined as follows:
CAL = CE (SoftMax ( (M (v))) (v))²)
where yi is the label of i-th input image, and GMP is the global maximum pooling function that employs spatial aggregation to the attribute features. A E TR xA is the seen attributes matrix where N is the number of seen classes. SoftMar' (-) is SoftMax activation function and CE (-) is the cross entropy loss commonly used in ZSL models.
Blue area the bottom of Figure 2 shows the layout of AL module.
3.2 ATTRIBUTE SCORING
As mentioned in the previous section, attribute space, despite being commonly used in ZSL models, suffers from several problems like hubness problem. We are aware of the rich semantic information beneath attributes. Inspired by Huynh & Elhamifar (2020b), we introduce attribute semantic space to collaborate with attribute space, which forms our attribute scoring (AS) module.
With the attribute features from AL module, the visual location of every attribute is encoded in each channel of â. Naturally, we thought of making it into a set of attention masks as a substitute for attention mechanism. The attribute masks are obtained through the sigmoid function that normalizes values of each channel in ä into a range from 0 to 1, where the value approaching to 1 stands for high confidence of having attribute-related visual features in the location, while that approaching to O is the opposite. Therefore, we can extract visual features for each attribute-related image region using attribute masks by performing the broadcasted Hadamard production between visual features Hx xC and each channel of normalized à which produces A masked visual features that correspond to A attributes, namely sitribute-visull features (AVFs) (a) E R HxXWX
(a E [1. A)).
To achieve zero-shot classification, the next step is to map AVFs into attribute semantic space with reference to DAZLE (Huynh & Elhamifar 2020b). The attribute semantic space is constructed using word vectors of attributes that are usually produced by word vector models like Word2Vec (Mikolov) et al. 2013). With these attribute word vectors, employ AS module to map the above AVFs into attribute semaaii space, and then calculate the class score of samples as follows:
é, = w (WAAP(vVO)
ww
[1
(3)
P. = e a [1,A]
(4)
8 = a p" CE [1
(5)
where GAP(() is the global average pooling function that aggregates AVFs into 1 x C' dimensions, and W (.) the linear mapping with C e parameters. vector, and e E R is the ground truth attribute word vector. Pa is the attribute score for the a a-th attribute. p E I is the concatenation of po. (a E [1, Al). a gIxA (c E [1, sN is the class attribute vector of the c-th class which is extracted from the c-th row of attribute matrix A And se is the class score of the current sample belonging the e-th class.
E Rixe is the predicted word
4

---

Under review as a conference paper at ICLR 2023
Thus, the attribute scoring loss based cross entropy is designed:
LAS CE (SoftMax (SAS) : (/b)
where SAS E RN is the concatenation of class score &f (c [1,
Ultimately, a novel attribute alignment (AA) pipeline is constructed, which consists of AL and AS modules. AA innovates attribute alignment approach through integrating attribute space and attribute semantic space into a unified pipeline of ZSL model. Subsequently, we will model the relations between attributes.
3.3 ATTRIBUTE ENHAACCEENTT
Studies on CNNs (or DNNs) have shown that feature extractor built up by convolutional neurons does well in extracting visual patterns from images. However, what they non-visual concepts from image samples. Unfortumately, the abstract concepts are common in the attribute sets of many ZSL datasets. For example, Animals with Attributes 2 (AwA2) (Xian al. 2019a) has 85 expert-defined attributes total. Roughly half of these attributes can be directly related visual representations (like "stripes" and "tail"), while more than half of them do not correspond directly to the visual representation (such as "fast" and "smart"), This is even more troublesome for part-based ZSL methods since it hard to locate a visual region for such non-visual attributes thus confusing the model.
good at is to extract
To address the issue, we turn to exploiting the implicit semantic relations between attributes. The idea is to dynamically build the associations between visual-related attributes and non-visual at- tributes by modeling the relations between them. With the assumption that all attributes share the same semantic space, it allows the model to enhance the usability of non-visual attributes using representations of visual-related attributes.
In order to model the relations of attributs, we construct attribute relation graph (ARG) G based on point-wise mutual information (PMI) (Bouma 2009) according to Hu et al. (2022). Let G has A vertices corresponding A attributes, the edges between attributes (vertices) defined based on their normalized PMI values referring to threshold 6:
{ 1, PMI (x, y) ā O, else
A., A₁,
where A, is the element in the i-th row and j-th column of A. We the undirected graph for ARG, so that its adjacency matrix is symmetric, that is Ar, = A See Appendix ( for the detailed formula of PMI. The selection of threshold will be discussed in detail in experiments section.
So far, ARG discussed above is still a static structure shared by all categories, which cannot be dynamically optimized with different classes and samples. Meanwhile, the PMI-based connections (graph edges) may not always represent the correct relations between attributes. Therefore, instead of using the well-known graph convolutional networks (GCN), we leverage graph attention networks (GAT) Veličković et al. 2018) to achieve dynamic modeling based on ARG.
GAT proposed by Veličković al. can dynamically adjust edge weights with self-attention mech- anism solve series of problems in spatial GNNs. According to AA pipeline in the previous subsection, A sets of global AVFs ((o) (a E [1, A]) belonging to the sample are obtained, which will be subsequently used as the input of nodes in G We use a two-layer GAT network in the pa- per, i.e. E {0, 1). The outputs of GAT are the predicted word vectors é. E Rxxe (a E [1, Al) from each node, where is the dimension of word vector. the process, GAT not only projects the input global AVFs into attribute semantic space (like the AS module), but also models attribute relations in the output, namely predicted word vectors é. With the help of ARG that connected attributes by their semantic relation, GAT enhance the expression of certain attribute-relate features (es- pecially those from non- nis-visual attributes), yielding more discriminative semantic representations. Hence, we name it attribute enhancement (AE) module.
As the output of AE module has the same form of predicted word vectors like AS module, an integrated enhanced attribute scoring (EAS) module is naturally formed. We can follow formula
5

---

Under review as a conference paper at ICLR 2023
Attribute Scoring
Hadamard Product
Mabri Numppicttion
A
Sum
Concolation
f
wdv
Wordbvex
LUTT
Cres Entrepyy
A
au
Lors
P
Coacot
Prodiction
pooludiity
Attribute Enhancement
GMM
Meximam Pooling
Loss
Output
GNF
Global Average Poeling
Prediction
Sigmoid
Lewss
P
1xTxC*A
Ha c
Conv
W
281
Attribute Localization
Figure 2: Attribute Alignment and Enhancement Network.
and 5 calculate the class vector SEAS, and the enhanced attribute scoring loss is constructed as:
LEAS = CE (SoftMaz (SEAS) ,i))
(8)
The dropout strategy often used improve the generalization GNN models by randomly "drop- ping" some nodes (sweeping node features) during training. The method is originally used in GNNs to suppress over-smoothing problem (Li et al. 2018), while we use here to perturb the original data (AVFs) to enhance the generalization of the model. Subsequent experiments will demonstrate the significant impact the dropout strategy on GZSL performance.
3.4 GENERALIZED ZERO-SHOT IMAGE CLASSIFICATION
Finally, by integrating the attribute localization (AL) module and enhanced attribute scoring (EAS) module, A3E network for GZSL is constructed (see Figure 2
3.4.1
OVERALL LOSS
The overall loss function of A3E is:
CAFE = ALAI + (1 A) LEAS
(9)
where is the weighting coefficient that balance the two modules.
3.4.2 INFERENCE
In the inference stage, A3E employs fusion prediction method as follows:
1 arg max (ASAL (1 (1-A) A)SEAS + (A(cENT)) CEN
(10)
where ÿ is the predicted label, and À is a coefficient which has identical function with the one in training loss. SAL is the output probability of AL module and SEAS is the output probability of EAS module. N = N n N the set of all classes labels. 3 is an adjustable bias, and A is an indicator which will take 1 for unseen classes, and -1 for seen classes.
To alleviate the inevitable bias caused by domain shift in GZSL, we set a calibration bias refer (Huynh & Elhamifar 2020b). However, it is worth notice that unlike (Huynh & Elhamifar

---

Under review as a conference paper at ICLR 2023
Table 1: Results of conventional 7SL. and G7SL classification AwA2, CUB and SUN datasets. The best and second- best results are marked in bold and underline, respectively. The symbol "." indicates no results. The symbol 4* represents models with 448x448 input size.
Methods
ZSL.
GZSL
ZSL.
GESL
ZSL.
GZSL
MCA
Uisean
MCA
Uhseen
Generative Methods
CVPR 2019 FVAEGAN-D2
71.1
57.6
70.6
61.0
48.4
60.1
53.6
64.7
45.1
38.0
41.3
NeuriPS 2020
Composer 71.
62.1
77.3
69.4
56.4
63.8
59.9
55.1
22.0
31.4
ICCV
FREE
55.7
57.7
41.7
Embedding Methods
NewiPS 2019
68.8
37.6*
52.5*
36.7*
71.3*
18.5*
60.3
75.7
67.1
65.9
56.7
59.6
58.1
52.3
24.3
33,2
ECCV2020
RGEN
73.6
67.1
76.5
71.5
76.1
60.0
73.5
66.1
63.8
44.0
31.7
36.8
SR2E
58*
80.7
67.5*
61.6*
70.6*
65,6 65
43.1
36,8*
39,7*
SPL 2021
SELAR
52.0
71.9
60.3
62-1
61.9
63.6
10.5
32.9
36.3
NewrPP 2021
ISVA
56.7
79.8
66.3
62.8
52.7
58.3
55.3
63.8
48.6
39.0
43.3
CVPR 2022
MSON
70.1*
62"
74.5*
67.7*
76,1*
68.7*
67.5*
68.1* 1*
65.8*
52.2
34,2*
41,3*
A3E
74.1
69.3
71.2
70.2
74.9*
66.3*
71.6*
68.8*
64.0"
46.8
30,0*
36,5*
2020b), the calibration in our method only includes a bias on the prediction probability, and does not include any additional loss term that usually requires unseen semantics adjust output predictions.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
4.1.1 DATASETS
We evaluate A3E network on three GZSL benchmarks: Animals with Attributes 2 (AwA2) (Xian ct al. 2019a). Caltech- UCSD Birds 200-2011 (CUB) (Wah et 2011) and SUN attribute database (SUN) Patterson et al. 2014). AwA2 is a coarse- -raiind dataset with 37,322 images from 50 animal classes, each of which has 85 attributes. While, CUB is fine-grained bird dataset containing 11, 7,888 images from 200 bird classes with 312 attributes. SUN is also fine-gradd areigra dataset which includes 14,340 images from 717 scene categories, and each class has 102-dimension attribute vector.
In order to fairly compare with the state-of-the-art ZSL models, we adopt the proposed split (PS) of datasets presented in (Xian, Schiele, and Akata 2017). Evaluation metries shown in Appendix B
4.1.2 IMPLEMENTATIO DETAILS
All the experiments in the paper are conducted on NVIDIA GeForce RTX 3090 with 24 GB video memory size. Software versions are Python 3.9, PyTorch 1.1 1.0,, NumPy 1.22.3 and CUDA 11.3.
For better reproducibility of experiment results, we manually fixed the random seed to 1024 for all tests. Detailed settings would be listed in Appendix C
4.2 COMPARISON WITH
We compare our A3E network with several state-of-the-art models both ZSL setting and GZSL setting. The results are presented in Table [ On the ZSL side, A3E shows very gratifying results in all three datasets. While it is competitive compared with the tfate-of-the-art models on CUB and SUN, the most compelling achievement that A3E outperforms the long-standing record by RGEN (Xie et al.) 2020) on AwA2 since 2020. On the GZSL side, A3E reaches the highest harmonic mean (6) on CUB dataset, which is the current best generalized model on CUB. It also gets a satisfactory performance on AwA2, though is inferior to RGEN. Whereas, harmonic mean of A3E on AwA2 still improved by at least 2.5% compared with the former state-of-the th--hh--rt art models, which verifies that A3E is strong competitor so far, compared with most of the models except for RGEN. In general, the results prove that the proposed attribute alignment and enhancement work effectively on AwA A2 and CUB, which are beneficial represent the rich semantic relations between attributes

---

Under review as a conference paper at ICLR 2023
Table 2: Ablation study under two datasets. The best results marked in bold.
AwA2
CUB
Method
ZSL
GZSL
ZSL
GZSL
MCA
Unseen
Seen
11
MCA
Unseen
Seen
II
AL+AS(AA pipeline)
60.4
$2.6
78.5
63.0
57.7
44.6
60.5
51.3
AL+EAS(A3E Network)
74.1
69.3
71.2
70.2
74,9
66.3
71.6
68.8
hidden in AwA2 and CUB. The data characteristic of Aw and CUB promotes A3E network more generalized. However, the performance SUN dataset declines sharply. The reason accounted for the phenomenon is that attributes of SUN are more abstract which are hard to capture their correspondent image regions. More importantly, the semantic relations are too sparser compared with the former datasets to affect the performance of attribute enhancement.
Among all comparison models, DAZLE, RGEN and MSDN are the noteworthy as well as our A3E network. They have some similarity in research motivation. DALZE and MSDN adopt at- tribute scoring mechanisms, as well A3E. RGEN uses GNN to perform region-based based relations on visual features. Whereas, A3E employs AE module to model semantic relations between attributes. From the GZSL. results on AwA2 and CUB, A3E successfully surpasses these models by the sta- ble performance which be expressed the average of harmonic means: A3E network reaches 69.5% in average, while DAZLE, RGEN and MSDN are 62.6%, 68.8% and 37.9% respectively(sec Appendix Dj. The more surprising fact is that when A3E generalizes well in GZSL settings, it does not resort to any probability tricks commonly used in above models, such as the balance loss used by RGEN and the calibration loss used in DALZE and MSDN models. The modified losses by the probability tricks require the supervision from unseen semantics during training, which are contrary to the original setting of ZSL to some extent. In contrast, A3E only relies seen samples and generalizes better than those models that require unseen semantics, which further demonstrates the superiority of the propose method.
4.3 ABLATION STUDY
4.3.1 EFFECTS OF ATTRIBUTE ENHANCEMENT
The proposed A3E network is composed of three interrelated modules, namely, AL, AS and AE modules. Among them, AE module based on GAT is the most innovative and representative com- ponent of work. To further evaluate the efficacy of attribute enhancement in actual task, conduct ablation studies on AwA2 and CUB datasets, by setting the baseline model with only AL. and AS modules, i.e. attribute alignment (AA) pipeline. Table shows the results of ablation study.
To fairly compare the baseline model with A3E network, we use the same parameter setting in both. Even so, the ZSL accuracy of A3E has improved by staggering 13.7% and 17.2% for AwA2 and CUB datasets, respectively. Similarly, drastic performance boost is also present in the GZSL set- ting. where A3E exceeds the baseline by up to 7.2% and 17.5% in harmonic mean metric on AwA2 and CUB, respectively. Thus, with the help of AE module. A3E exceeds the baseline with absolute superiority in all settings, with an even greater advantage in CUB. In effect, the characteristics of attributes such as semantics and relations vary with different datasets. The reason that the perfor- mance on CUB has dramatic improvement by AE module resulted from the stronger semantic relations between attributes of CUB, where attributes are more uniform to describe image contents.
4.3.2 EFFECTS OF DROPOUT
To investigate the effects of dropout in AE module, we conduct experiments with different dropout rates on CUB dataset. The results concluded in Appendix F
8

---

Under review as a conference paper at ICLR 2023
Seen CZSS)
Sozs G7SL.
Seen CISS
Soze GPSL.
-
80
90
70
70
50
50
50
40
02 033
05 96 6.7 688 0.9
0.1 0.2 033
0.4 0.3 0.0 0.7 008
02 0.3 0.4 06 0.7
0.000.2 1000020004000000
(a) results on AwA2
(b) results on CUB
(a) results on AwA2
(b) results on CUB
Figure 3: Effects of weighting coefficient À on Figure 4: Effects of threshold on AwA2 and CUB datasets.
AwA2 and CUB datasets.
4.4 HYPERPARAMETER ANALYSIS
4.4.1 EFFECTS OF WEIGHTING COEFFICIENT À
Figure is the summary of the experimental results on tuning weighting coefficient À between AL and EAS modules. We do not show experiments for À at O and since they are no longer considered in our A3E model. In CUB dataset (see Figure (()),, both ZSL and GZSL indicators increase with the rise of A. While in Aw dataset (see Figure 3()), the accuracies of unseen classes in ZSL and GZSL rise with the increase of X. But the accuracy of seen classes in GZSL shows the opposite trend, to cause harmonic means of GZSL slowly improving. The increasing of À represents that the model emphasizes more on AL module learning. Results show that despite the simpler structure and fewer parameters of the AI. module, its importance in the objective function is less than that of the EAS module. It is proven by the results that model performance improves as the effort invested in AL module (i.e., value of A) increases. Specially, attribute localization more demanding for the fine-grained datasets with more subtle features, such as the CUB dataset. Considering all the indicators, we set 0.6 and 0.9 for AwA2 and CUB, respectively.
4.4.2 EFFECTS OF THRESHOLD
Threshold 5 controls the generation of ARG. Smaller value of 8 indicates easier connection between nodes in ARG, which means more edges in the graph. When AE module does not work. 4 shows how different values of 5 affect model performance on the two datasets. As 8 increases in CUB dataset (see Figure 4b), all four metrics increase equally until 5 = 0.8. In AwA2 dataset (see Figure (gure((a)), [)), we can find general uptrend in the accuracy of unseen classes as 5 increase, and the best accuracy is found representing attribute relationships in ARG should gradually decrease. Obviously, edges created by PMI do not perfectly correspond with the semantic relationships of attributes. We believe that the graphs generated by lower values of 8 have model performance decline. When the structure of ARG is simplified with higher threshold, GAT is more likely to obtain robust information from ARG, thus enhancing the classification performance. To balance the accuracies seen classes and unseen classes, we set = 0.9 and = 0.8 for AwA2 and CUB, respectively.
there is no edge ARG, so that
0.9. With the increase in ò, the edges
noisy connections (edges), which leads to the
5 CONCLUSION
In the paper, we propose an attribute alignment and enhancement (A3E) network, which consist of attribute alignment (AA) pipeline and AE module. Therefore, A3E can align each attribute with corresponding image region and enhance their representations by the semantic relations between attributes through GNNs. At last, the experiments on three ZSL. datasets have demonstrated the superiority of A3E network on ZSL/GZSL classification.
REFERENCES
Gerlof Bouma. Normalized (Pointwise) Mutual Information in Collocation Extraction. pp. 11, 2009.

---

Under review as a conference paper at ICLR 2023
Shiming Chen, Wenjie Wang. Beihao Xia, Qinmu Peng, Xinge You, Feng Zheng, and Ling Shao. FREE: Feature Refinement for Generalized Zero-Shot Learning. In 2021 IEEE/CVE Interaational Conference on Computer Vision (ICCV), pp. 122-131, Montreal, QC, Canada, October 2021a. IEEE. ISBN 978-1-66552-812255 doi: 10. (0.1009/CCV48922.000 202.89922...2200000 0021990999.
Shiming Chen, Guo-Sen Xie, Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, and Ling Shao. HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning pp. 13, 2021b.
Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang, Qinmu Peng, Kai Wang, Jian Zhao, and Xinge You. MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning, April 2022.
Zhi Chen, Yadan Luo, Ruihong Qiu, Sen Wang. Zi Huang. Jingjing Li, and Zheng Zhang. Semantics Disentangling for Generalized Zero-Shot Learning. In 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8692-8700, Montreal, QC, Canada, October 2021c. IEEE. ISBN 978-1-66542-812-5. doi: 10. 10.100C99499922.200000 20..0009......00088. .00859.
Yu- Ying Chou, Hsuan- Tien Lin, and Tyng-Luh Liu. ADAPTIVE AND GENERATIVE ZERO- SHOT LEARNING. pp. 14, 2021.
Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal. Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precision, September 2017.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEFE Conference on Computer Vision and Pattern Recognition, pp. 1778-1785, Miami, FL. June 2009, IEEE, ISBN 978-1-4244-3992-8 doi: 10.1
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, Zhenyong Fu, and Shaogang Gong. Transductive Multi-view Embedding for Zero-Shot Recognition and Annotation. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, volume 8690, pp- 584-599. Springer International Publishing, Cham, 2014. ISBN 978-3-319-100045 978-3-319- 10605-2. doi: 10. 1007/978- 10007/97--331999 3-319- 10605-2 38.
Jiannan Ge, Hongtao Xie, Shaobo Min, and Yongdong Zhang. Semantic-guided Reinforced Re- gion Embedding for Generalized Zero-Shot Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2):1406-1414, May 2021. ISSN 2374-3468, 2159-5399.. doi: 10. 10.1669/aaai. w3w.2.15522333233
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 13(11):139-144, 2020.
Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. Contrastive Embedding for Generalized Zero-Shot Learning. In 202/ /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2371-2381, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-66544-5092 doi: 10.1109/CVPR44437.00 0111444437.221.002242 100240
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770-778, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-8851-1. doi: 109/ CVPR.2016.90.
Yang Hu. Guihua Wen, Adriane Chapman, Pei Yang. Mingnan Luo. Yingxue Xu, Dan Dai, and Wendy Hall. Graph-Based Visual-Semantic Entanglement Network for Zero-Shot Image Recog- nition. IEEE Transactions on Multimedia, 24887, 2022. ISSN 1520-9210, 1941-0077.
doi: 10. 10.11999TMM2202213322222
Dat Huynh and Ehsan Elhamifar. Compositional Zero-Shot Learning via Srne-Grained Dense Fea- ture Composition. pp. 12, 2020a.
Dat Huynh and Ehsan Elhamifar. Fine-Grained Generalized Zero-Shot Learning via Dense Attribute-Baset -ased Attention. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4482-448 4492, Seattle, WA, USA, June 2020b. IEEE, ISBN 978-1-72811 1-72817- 168-5. doi: 10.1 10.1109/CVPP422000 2020.00454.
10

---

Under review as a conference paper at ICLR 2023
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Thomas N. Kipf and Max Welling, Semi-Supervised Classification with Graph Convolutional Net- works, February 2017.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper Insights into Graph Convolutional Networks for Semi- Semi-Supervised Learning, January 2018.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Xuanyi Dong, and Chengqi Zhang. ISOMETRIC PROPAGATION NETWORK FOR GENERALIZED ZERO-SHOT LEARNING, pp. 13, 2021a.
Yang Liu, Lei Zhou, Xiao Bai, Yifei Huang, Lin Gu, Jun Zhou, and Tatsuya Harada. Goal-Oriented Gaze Estimation for Zero-Shov Shot Learning, March 2021b.
Peirong Ma and Xiao Hu. A Variational Autoencoder with Deep Embedding Model for General- ized Zero-Shot Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07); April 2020. ISSN 2374-3468, 2159-5399. doi: 1111////aa..34417..66....
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Repre- sentations Vector Space, September 2013.
Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang. Zueng-Jun Zha, and Yongdong Zhang. Domain-aware Visual Bias Eliminating for Generalized Zero-Shot Learning, April 2020.
Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding, International Journal of Computer Vision, 108(1-2): 59-81, May 2014. ISSN 0920-5691, 1573-1405. doi: 10. 10.1007/s1 12600133112--
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang. Chee Peng Lim, Xi-Zhao Wang, and Q M. Jonathan Wu. A Review of Generalized Zero-Shot Learning Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp- 1-20, 2022. ISSN 0162-8828,
2160-9292, 1939-3539, doi: 10.1 1111000/A...22222319966666
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting, pp. 30, 2014.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks, February 2018.
Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. Meta-Learning for Generalized Zero-Shot Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04))6062-6069, April 2020. ISSN 2374-3468, 2159-5399. doi: 10. 10.1669/aai.vv33 10.1609//aai..34444.6666666 v34904.6069
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. pp. 2011.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs, April 2018
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-Shot Learning-A Comprehensive Evaluation of the Good, the Bad and the Ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2109):2551-2266 September 2019a. ISSN 0162-8828. 2160- 9292, 1939-3539. doi: 10. 10.11099TPAMML.21182277777
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. F-VAEGAN-D2: Feature Generating Framework for Any-Shot Learning. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10267-10276, Long Beach, CA, USA, June 2019b. IEEE. ISBN 978-1-72813-293--8 doi: 10. 00.1109/CVPR.2019.000
11

---

Under review as a conference paper at ICLR 2023
Guo-Sen Xie, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, and Ling Shao. Attentive Region Embedding Network for Zero-Shot Learning. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9376-9385, Long Beach, CA, USA, June 2019. IEEE ISBN 978-1-72813-2991 doi: 10.1 D0.1009/CVP.22119.oo
Guo-Sen Xie, Li Liu, Fan Zhu, Fang Zhao, Zheng Zhang, Yazhou Yao, Jie Qin, and Ling Shao. Region Graph Embedding Network for Zero-Shot Learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12349, pp- 562-580. Springer International Publishing, Cham, 2020. ISBN 978-3-030-58547-1 978-3-030- 58548-8. doi: 10.1107/778-300) 030-5854888 33.
Shiqi Yang, Kai Wang, Luis Herranz, and Joost van de Weijer. On Implicit Attribute Localization for Generalized Zero-Shot Learning. IEEE Signal Processing Letters, 28:872-876, 2021. ISSN 1070-9908, 1558-2361. doi: 10.1 00.1000015P220023077765.
Li Zhang, Tao Xiang, and Shaogang Gong. Learning a Deep Embedding Model for Zero-Shot Learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3010-3019, Honolulu, HI, July 2017. IEEE. ISBN 978- 978-1-5386-0457-1. doi: 10. 109/PP... 2017.321.
Yizhe Zhu, Jianwen Xie, Zhiqiang Tang, Xi Peng, and Ahmed Elgammal. Semantic-Guided Multi- Attention Localization for Zero-Shot Learning. pp. 11, 2019.
A POINT-WISE MUTUAL INFORMATION
Point-wise mutual information (PMI) (Bouma 2009) is proposed measure the relationship of two points (objects), defined as:
p p v) p (v)
PMI (x, In
(II)
where p (xx, y) is the joint probability of and y. In practice, the normalized PMI is more commonly used as:
PMT (r,y) PMI, (x. v)
(12)
- -Ipp (z,
where its values are in the range of [-1,1], which give a good indication of the relevance between and y. Specifically, I means they are co-occurrence, while -1 shows the opposite case. But, 0 indicates they are not relevant at all.
B EVALUATION METRICS
We consider both conventional ZSL and GZSL settings in all three datasets. Mean Class Accuracy (MC.) is adopted the evaluation indicator for ZSL setting, which takes average value of Top-1 accuracies on unseen classes. And harmonic mean (/)) is employed to evaluate the performance of GZSL setting, which the most comprehensive metric reflect model performance by taking accuracies of seen and unseen classes into consideration. The specific formula is defined as follows:
2 MCAs x MCAB
H MCAs + MCAU
(13)
where MCAs and MCAp are the MCAs for classes and unseen classes, respectively.
12

---

Under review as a conference paper at ICLR 2023
C MODEL SETTINGS
We use the fixed ResNet101 (He et al. network, which is commonly used the backbone network in many models (Huynh & Elhamifar 20206; Xie et al.] 2020 Ge et al., 2021] Chen et al.] 2021b 2022). The input images of model reshaped as 224x224 pixels for AwA2 datasets and 448x448 pixels for CUB and SUN datasets since the finer details could significantly improve performance on fine-grained datasets. We use the Word2 model trained on Google News to generate attribute word vectors with 300 dimensions.
2016) pretrained on ImageNet as the feature extractor A3E
We adopt ADAM optimizer (Kingma & Ba 2017 in model training and set weigh decay to x 10-6. We empirically set the hidden layers and attention heads of GAT network in EAS module to (200, }} for AwA2, {200. 4] for CUB and { (1000, 5) for SUN, with dropout rate fixed to 0.2. For AwA2 dataset, we set the learning rate 10. Regarding to CUB dataset, the learning rate is set to 7.5 X 10-6, Batch size is set to 8, and maximum iteration number is 30. As to SUN dataset, we set the learning rate to x 10-6, batch size to 16 and maximum iteration number to 25. The learning rate for 1xl convolution in AL module is 10 times greater than the given values in datasets. The calibration bias 3 is set to 2.0, 0.4 and 0.4 for AwA2, CUB and SUN, respectively.
10-6, batch size 64 and maximum iteration number to
There are two hyperparameters in A3E network: weighting coefficient À and ARG threshold ò. We set À to 0.6, 0.8 and 0.9 for AwA2, CUB and SUN datasets. While 5 is set to 0.9, 0.8 and 0.1 for these three datasets, respectively. The influence of the hyperparameters is explored in the following experiments.
D AVERAGE HARMONIC MEANS ON AWA2 AND CUB
Table 3: Average harmonic means on AwA2 and CUB datasets.
AwA2
CUB
Average
Method
H
DAZLE
67.1
58.1
62.6
RGEN
71.5
66.1
68.8
MSDN
67.7
68.1
67.9
A3E
70.2
68.8
E ABLATION STUDY ON DROPOUT
ZSL + Unseen GZSL
Seen GZSL
70
65
55
50
40
0.1
0.2
0.3
0.4
0.5
Figure 5: Effects of dropout rate d on CUB dataset.
Dropout is commonly used strategy to prevent neural networks from overfitting (Srivastava al. 2014]. our work, we employ dropout on GAT of AE module by randomly dropping some node features during training stage. The dropping process is controlled by dropout rate d that determines the proportion dropped nodes. As shown in Figure with dropout rate d increasing from 0 to 0.2 (0 means dropout is inactive), almost all indicators
the rise, except slight fluctuation
13

---

Under review as a conference paper at ICLR 2023
for the class accuracy GZSL. The accuracy decreases gradually from 0.3. To sum up, both ZSL accuracy and GZSL harmonic mean reach the best values at 0.2 dropout rate, then begin to decline with the increase of d. The phenomenon verify functionality of dropout: preventing models from overfitting. Besides, dropout indispensable to AE module because it improves the generalization ability of model.
F
TERM EXPLANATION
Attribute semantics: The word vectors of attributes, which is the output of word2vee model by taking the attribute name as the input. The resulted representation codes semantic information of each input attribute, called the attribute semantics.
Class attribute vectors: set of vectors corresponds to the set of classes. Each vector encodes information about the attributes of the corresponding class, and the presence of attribute is indicated by a continuous or binary value at its corresponding element of the vector.
Attribute prototypes is actually the class attribute vectors in the attribute space, that has identical meanings with class attribute vectors.
14


Reviews:
[{"review_id": "_KKz0wMQ-wS", "reviewer": "Reviewer_a3kk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses Zero-Shot Learning problems by localisation and GAT based attribute enhancement. The method is evaluated on the three datasets and did not achieve state-of-the-art results.", "summary_of_the_review": "Overall, the paper is lack of theoretical contribution and the performance is not promising. Although the paper is well written and presented, it is still below the standard of ICLR expected.", "strength_and_weaknesses": "The paper is well-presented and well-designed.\n- Localisation, GAT are not new to ZSL. No clear motivation or novel insights for these designs are provided.\n- Ablation study is very rough and superficial.\n- Experimental results are not competitive.\n- Lack of theoretical contributions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_-BBqIWi0xi", "reviewer": "Reviewer_2RdH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to fully utilize attributes information for GZSL for simultaneously exploring attribute alignment and enhancement (A3E). A3E consists of an attribute localization (AL) module and enhanced attribute scoring (EAS) module. AL is used for localizing attribute regions, and EAS further enhances these features by GAT. Experiments are conducted on golden GZSL datasets.", "summary_of_the_review": "Combination of existing works, and limited novelty to the community. ", "strength_and_weaknesses": "The organization is fine. Experiments are sufficient.\n\nHowever,\n\n1.Many statements in this paper are not well motivated, e.g., ``implicit localization capability within the feature extractor’’ is not clear; ``exciting achievements for common categories’’ is also not clear. The story on comparisons among human and ZSL is commonly used in the community; you cannot say this because others have done this. The resulting facts of ZSL can not guarantee the reason of using auxiliary information for knowledge transferring. In fact, auxiliary information is one choice of conducting ZSL, there much exist other manners for performing ZSL,e.g., matching with external dataset. By saying embedding methods are inherently inferior in GZSL to generative methods, why? Also, it is fine to use unseen semantics for overcoming bias issue, yet not prone to unexpected effects. By saying unexpected effects, what do you mean? Overall, the challenges are not well defined. The writing is also not accurate.\n\n2.Terms of attribute semantics, class attribute vectors, attribute prototypes and attribute-visual features are confusing to the beginner of ZSL field. More explanation should be given. Generative adversarial network (GAN) or variational autoencoder (VAE) miss references. Typos exists: so that convert ?\n\n3.The motivation for the usage of GAT for improving the generalization is not clear. In fact, using graph neural network to model attribute relationships have been explored in previous works (e.g., LsrGAN in ECCV20). The overall compared methods are limited, and more methods should be surveyed and compared.\n\n4.The framework is actually incremental improvements on existing methods, e.g., improved on Yang et al. in CVPR 2021 by seeing each channel as attention mask. From this point, the novelty is limited to the community, since this kind of attention has been explored in the community (e.g., in AREN of CVPR2019). Furthermore, the idea of mapping AVFs into attribute semantic space by DAZLE is also not new. The idea of attribute scoring is just the combination of DAZLE with AREN for solving the same task in this paper. All these aspects reduced the contributions of this paper to the community. \n\n5.The overall writing is poor. The formulas are also confusing. Please amend them accordingly.\n\n6.How to initialize the network weights, by pre-trained ImageNet weights or from scratch? What’s the specific network architecture? \n\n7.Since additional operation such as attention is used for the proposed method, I am doubted about the tradeoff between the running time and accuracy.\n\n8.The parameter analysis is shown, however, what’s the specific parameters for achieving these results? E.g. \\beta. It seems the authors have reported adjusted results by varying many \\beta, this is thus not fair to compare with other methods without CS adjustments. Also, the authors used additional attribute features for model training which is also not fair compared with counterparts.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0I8dv4MPInH", "reviewer": "Reviewer_hgQc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an attribute alignment and enhancement (A3E) network for zero-shot learning. It uses the attribute location model to align attributes and utilized the attribute relation graph to enhance the attribute. Experiments on three datasets show the effectiveness of the proposed approach.", "summary_of_the_review": "The novelty of this paper is limited and the detailed implementations are unclear. The experiments are not sufficient, which lacks experiment analysis and qualitative results to show the effectiveness.", "strength_and_weaknesses": "Strength:\n1. The paper is well-organized, which is easy to read. However, some details are unclear.\n2. The performance seems good.\n3. The combination of attribute alignment and enhancement seems novel, but the novelty is limited.\n\nWeakness:\n1. The novelty of this paper is limited. It seems to be a combination of AL [Yang et al. (2021)] and attribute relation graph[Hu et al. (2022)]. As can be seen in the approach, most parts introduce the workflow without detailed formulation.\n2. Eq 2 aims to learn projection from the visual space to the word vector space. However, there is no supervision loss. How to guarantee the projections are properly learned?\n3. How does attribute alignment conducted? The supervision seems to be classification losses and no explicit alignment is conducted.\n4. What is the difference between Eq 8 and Eq 6? They use the same representations.\n5. How does the ‘triangle’ in Eq 10 obtained? It is strange to know whether a sample is seen or unseen.\n6. The results in Table 1 are unclear. Why the input size on AWA is different from the other two datasets? Do the authors use different networks?\n7. The authors only describe the phenomenon in Section 4.4 without detailed analysis. Moreover, no detailed qualitative analysis to show the effectiveness of the proposed approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject