Dual Progressive Prototype Network for Generalized Zero-Shot Learning
Authors: Chaoqun Wang
Shaobo Min
Xuejin Chen
Xiaoyan Sun
Houqiang Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: generalized zero-shot learning
progressive prototypes

Abstract: Generalized Zero-Shot Learning (GZSL) aims to recognize new categories with auxiliary semantic information, e.g., category attributes. In this paper, we handle the critical issue of domain shift problem, i.e., confusion between seen and unseen categories, by progressively improving cross-domain transferability and category discriminability of visual representations. Our approach, named Dual Progressive Prototype Network (DPPN), constructs two types of prototypes that record prototypical visual patterns for attributes and categories, respectively. With attribute prototypes, DPPN alternately searches attribute-related local regions and updates corresponding attribute prototypes to progressively explore accurate attribute-region correspondence. This enables DPPN to produce visual representations with accurate attribute localization ability, which benefits the semantic-visual alignment and representation transferability. Besides, along with progressive attribute localization, DPPN further projects category prototypes into multiple spaces to progressively repel visual representations from different categories, which boosts category discriminability. Both attribute and category prototypes are collaboratively learned in a unified framework, which makes visual representations of DPPN transferable and distinctive.	Experiments on four benchmarks prove that DPPN effectively alleviates the domain shift problem in GZSL.

Dual Progressive Prototype Network for Generalized Zero-Shot Learning
Chaoqun Wang's Shaobo Min', Xuejin Chen' Xiaoyan Sun', Houqiang Li!
'School of Data Science
The National Engineering Laboratory for Brain-inss iraiininspired Intelligence Technology and Application University of Science and Technology of China, Hefei, Anhui, China
Tencent Data Platform, Shenzhen, Guangdong, China cq14@mail ustc. edu. cn, beb.bobminnteeneni com, (xjchen99, ,sunxiaoyan, 1ihq] @ustc. edu. cn
Abstract
Generalized Zero-Shot Learning (GZSL) aims to recognize new categories with auxiliary semantic information, e.g., category attributes. this paper, we han- dle the critical issue of domain shift problem, i.e., confusion between seen and unseen categories, by progressively improving cross-domain transferability and category discriminability of visual representations. Our approach, named Dual Progressive Prototype Network (DPPN), constructs two types of prototypes that record prototypical visual patterns for attributes and categories, respectively. With attribute prototypes, DPPN alternately searches attribute-related local regions and updates corresponding attribute prototypes to progressively explore accurate attribute-region correspondence. This enables DPPN produce visual representa- tions with accurate attribute localization ability. which benefits the semantic visual alignment and representation transferability. Besides, along with progressive at- tribute localization, DPPN further projects category prototypes into multiple spaces to progressively repel visual representations from different categories, which boosts category discriminability. Both attribute and category prototypes collabora- tively learned in a unified framework, which makes visual representations of DPPN transferable and distinctive. Experiments four benchmarks prove that DPPN effectively alleviates the domain shift problem in GZSL.
Introduction
Deep learning methods depend heavily on enormous manually-labelled data, which limits their further applications [7 52
36 19 [T]. Therefore, Generalized Zero-Shot Learning (GZSL) recently attracts increasing attention, which aims to recognize images from novel categories with only seen domain training data. Due unavailable unseen domain data during training, GZSL methods introduce category descriptions, such as category attributes [5] or word embedding associate two domain categories.
[7 26
41
basic framework of embedding- sponding category descriptions in joint embedding space [16] 55
GZSL to align global image representations with corre- 2
6), as shown in Fig. (a). Due to the domain shift problem across two domain categories, unseen domain images tend to be misclassified as seen categories. To address this issue, recent methods focus discovering discrimi- native local regions to capture subtle differences between two domain categories. For example, AREN (45] and VSE (55] leverage attention mechanism to discover important part regions, which improves feature discrimination. DAZLE 20 and RGEN 46 introduce semantic guidance, e.g., category
38
"Corresponding Author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Localiziing
Updkling
Yellow
Align
Fmy
Paws
Visual Represantation
Updkting
Shariing Atribbtte
Progressie Attribute
Progressive Catrgory Prototypes
(a)
(b)
Protolypes
(c)
Protolypes
Figure I: The motivation of DPPN. (a) General GZSL methods directly align global image features with category attributes. (b) A typical but--based method, i.., APN [47], learns prototypes shared by all images for attribute localization. (c) DPPN progressively adjusts prototypes according different images and introduces category prototypes to enhance category discriminability.
attributes, into region localization to narrow the semantic-visual gap. Among existing methods, APN [47 is most related to our approach. As shown Fig. (b), APN constructs visual prototypes to indicate the typical visual patterns of each attribute, for example describing what attribute "Furry" visually refers to, and these prototypes are shared across all images to search attribute-matched local regions. However, due to image variances, the textures corresponding to the same attribute may vary seriously across images. Thus, sharing prototypes APN can not well depict the target image.
In this paper, we propose a novel Dual Progressive Prototype Network (DPPN), which constructs two types of progressive prototypes for respective attributes and categories to gradually improve cross-domain transferability and category discriminability of visual representations. Instead of sharing prototypes, DPPN dynamically adjusts attribute prototypes for each image to capture the vital visual differences of the same attribute in different images. This is achieved by alternately localizing attribute regions and updating attributes prototypes in turn, shown in Fig. [ (c). With image- specific prototypes, attribute localization, i.e., attribute-region correspondence, gets more accurate. To explicitly preserve such correspondence in the final representations, DPPN aggregates the attribute- related local features by concatenation, instead of widely-used Global Average Pooling (GAP) that attribute
will damage the attribute localization ability. Furthermore, along with poogressivelyuuddaeee prototypes, DPPN also builds category prototypes to record prototypical visual patterns for different categories. The category prototypes are projected into multiple spaces to progressively enlarge category margins, strengthening category discriminability of visual representations. Consequently, with cross-domain transferability and category discriminability, DPPN can effectively bridge the gap between seen and unseen domains.
Experiments on four benchmarks demonstrate that our DPPN alleviates the domain shift problem in GZSL and obtains new state-of of-the-art performance. Our contributions can be summarized as three- fold. a) We propose a novel Dual Progressive Prototype Network (DPPN) that constructs progressive prototypes for both attributes and categories to gradually improve cross-domain transferability and category diseriminability
visual representations. b) An alternation updating strategy is designed to dynamically adjust attribute prototypes according to target images. Besides, DPPN aggregates attribute-relate local features by concatenation to produce image representations, which explicitly preserves the attribute-region correspondence. c) DPPN projects category prototypes into multiple spaces to progressively enhance category discriminability.
Related Work
Generalized Zero-Shot Learning. GZSL aims recognize new categories using semantic knowl- edge transferred from seen categories. Early GZSL methods learn joint embedding space to align global image representations with corresponding category descriptions, e.g., attributes [[5] 35
or text descriptions [5] 37]. Since unseen and seen categories share common semantic space, the semantic-aligned image representations can be transferred from seen to unseen domain. Based on this paradigm, many works focus on improving the discrimination of embedding space by designing elaborate semantic-visual alignment functions [38
27 For example, some methods 33 use high-dimensional visual features to span the embedding space, which is
40 53 (5)
21

---

proved more discriminative than that spanned by category attributes. Other methods 23 9 42 32
utilize auto-encoders to preserve semantic relationships between categories in the embedding space.
Though effective, these methods suffer from the domain shift problem, Le,, two domains have different data distributions. Since only seen domain images are available during training, images from unseen categories tend to be recognized as seen categories. To this end, DVBE [31] and Boundary-based OOD
[0] explore out-of-distribution detection to treat seen and unseen domains separately. Some
works 28 20 suppress the seen category confidence when recognizing images to better distinguish two domain samples. These methods can effectively alleviate the domain shift problem via extra processing. but they ignore the discriminability of local attribute-related information in distinguishing two domains.
Part-Based GZSL. Since global image representations contain much noisy background information which is trivial for knowledge transfer, recent part-based methods 44 45 47 aim to localize part regions and capture important visual details to better understand the semantic-visur relationship. For example, SaGA [49), AREN(45), and VSE (55) leverage the attention mechanism to learn semantic- relevant representations by automatically discovering discriminative parts in images. RGEN [46] uses the region graph to introduce region-based relation reasoning to GZSL and learns complementary relationships between different region parts inside an image. GEM-ZSL [30 imitates human attention and predicts human gaze location to learn visual attention regions. Usually, the semantic guidance, e.g., category attributes, is used to guide the part localization [29]. Thus their generated local features can better match corresponding category attributes, which can alleviate the domain shift problem. Instead of treating category attributes as an all-in-one vector, DAZLE (20) proposes a dense attribute attention mechanism to produce local attention for each attribute separately. APN 47 constructs prototypes for separate attributes, which are shared by all images to localize attribute-related regions via region searching. In this paper, instead of using prototypes shared by all images, DPPN dynamically adjusts attribute prototypes according to different images, which learns visual representations with more accurate attribute localization and transferability. Besides, in DPPN, we also design progressive category prototypes to enhance category discriminability of visual representations.
3 Dual Progressive Prototype Network
3.1 Problem Formulation
The target of GZSL is to recognize images of novel categories trained with only seen domain data. In this paper, we denote S = {X, y, a, E Xe: y E V.. ay E } as seen domain data, where X E CRCN indicates image features extracted by the backbone network, and X,, RCx1 encodes the local information at the n-th region. y the corresponding category label, and ag E R N.x is the category description, such as the category-level vector with Na attributes. C' is the number of feature channels, and N W x H. The unseen domain data is similarly defined as U1 and V, n Va- = o. Given S during training. GZSL aims to recognize images from either X, or X, during inference. A basic framework is to learn image representation f(X) that is aligned with corresponding category attributes by minimizing:
Loss 10a=-a Σ log exp(f(X)Ta) a.) XEX. Ejey, exp(
where f(-) is a visual projection function, which is generally implemented via Global Average Pooling (GAP) and linear projection. v2s is the abbreviation of visual-to -o-emanto-emaatic projection.
Based on Li2s, APN 47 expects to improve the localization ability of attributes for intermediate feature X. Thus, APN constructs set attribute prototypes P = =Pp PNN J. where P, E Rx11 records visual patterns for the i-th attribute, e,g., depicting what attribute "Yellow Wing" looks like. P is learnable and shared by all images. With P, APN learns attribute localization by minimizing:
Lapn Σ d(((X,P),,,) FEX,
(2)
I(X,P) is an attribute localization function that searches the most related local feature X,, for P. E P, which regresses attributes â, E Nax1, d(c,-) is distance measurement function, e.g., l2 norm, that
3

---

Matri Mullipiiation
Update
Hadamant Product
g
Yowr jos
IYP
nTP
Features X
pk (X)
Attrbutes
Alribule
Prolulyge
Dimension Reduction
Localization
Redizement
Progrusiiv Caleguay Chesification
Prototype Selection
Ctll
CNN
wel.0poove Aod
ReLJ
poodiis
W
(X) C)) 00 XXX CT
Update X
Figure 2: Training pipeline of DPPN. DPPN progressively learns attribute prototypes Pk and category prototypes C* via Le and Ca- { ... K} consist visual representations of K iterations, which gradually capture attribute localization and category discrimination.
aligns the predicted attributes ay and ground truth attributes ag. Finally, the main insight of APN is to minimize Lapn Luss, and the inference function is:
arg
min d(f(X), VEYAO
However, due to instance variance, occlusion, and noise, the visual textures that correspond to the same attribute may vary severely across different images. Thus, it is unreasonable only rely sharing attribute prototypes P to accurately localize attribute-related regions for each individual image. Besides, Lapm applied to intermediate features X mere constraint. During inference, the final visual representation f(X) for recognition is directly aggregated by the intermediate features using GAP, which damages the attribute localization ability.
To this end, our Dual Progressive Prototype Network (DPPN) progressively explores attribute localization and category discrimination for different images with two modules, i.e., Progressive Attribute Localization and Progressive Category Classification, as shown in Fig. 2
3.2 Progressive Attribute Localization
The Progressive Attribute Localization (PAL) module aims to dynamically adjust attribute prototypes according to the target image to progressively capture local correspondence between different attributes and image regions.
Define pD [pf.-. p&, } as set of trainable attribute prototypes, which are randomly initialized and shared across all images. With pe initialized, PAL first localizes related local regions for each prototype p.. and then calculates specific visual features for all (pf.-. ...
.N,_ by:
p1 = (X-)) 1::55==(XXT0 pp)
(4)
where S E IR Nx.. is similarity matrix, and Sn. measures the similarity between the n-th local feature X,, and the i-th attribute prototype p.. fi(-) a Softmax normalization along each column. With 5, Eq. 4 aggregates related region features in X calculate attribute-specific features, which produces P [p]. Pk. )- far(.) (.) is prototype refinement function implemented by two fully-connected (FC) layers shown in Fig. 2
Compared to the original image feature X E RCXN, pl CxN explicitly captures specific visual patterns of the target image for each attribute, e.g., df aggregates related local features in X that correspond to the i-th attribute.

---

Considering instance variance, the sharing attribute prototypes po cannot well localize accurate attribute-related regions for all images. In Eq. (4), compared to po, the visual patterns of P!, that correspond to different attributes, are more specific to the given image feature X. Thus, PAL further regards pl as updated attribute prototypes from po for the target image. By replacing P with pk+1 and repeating Eq. ( PAL can progressively adjust attribute prototypes for a specific image by:
pk+1 for (orr(Xt (Xf(XTp())
(5)
where pk+1 leads to better attribute localization than Pt:
For the k-th iteration, since Ph IRC* contains specific visual patterns for different attributes, PAL concatenates all {p" ... . .Px,) J to produce the visual representation f*(X);
(6) where (ra(.) is dimension reduction layer and projects each p E Rex1 into RDx1, where D < C, to avoid excess calculation complexity. cal(.) concatenates all elements of the input. (*(X) RNxx1, where N, D Na. With multi-iterative (Pk]k- -, ,KK PAL gradually generates K visual representations {f (X))] -,.... ,K).
f²(X) cat (fra
Finally, (X) is aligned with corresponding attributes in joint embedding space by:
c Σ (((((x)) XEX,
where g(-) is a semantic projection function implemented by FC to project attribute vector into latent space, where the visual representations and projected attribute features can be well aligned, following (3I. The semantic alignment supervision Lea is applied all {fth (X) (X)]k 1, K} for training acceleration. As l: increases appropriately, f (X) localizes attributes more accurately. Thus, f (X) is used as the final visual representation for inference.
Consequently, by dynamically adjusting attribute prototypes according to the target image, PAL can progressively improve the attribute localization ability of visual representations, as shown in Fig. 3 PAL captures the attribute-reiion correspondence, which narrows the semantic-visual gap between category attributes and visual representations and boosts knowledge transfer between seen and unseen domains.
3.3 Progressive Category Classification
Besides exploring the correspondence between attributes and local image regions via PAL, we design a Progressive Category Classification (PCC) module to repel visual representations from different categories, which can enlarge category margins.
Similar to the sharing attribute prototypes in PAL, PCC defines a set of learnable category prototypes Co {cf, .cy. ), where cf E N.X. records the visual representation center for the j-th category. No = [D'a] is the number of seen categories. Since the attribute prototypes are progressively updated in PAL and K visual representations {X)(k = 1, K} } are accordingly generated for each image, sharing category prototypes Co cannot well model visual category differences for all iterations of f* (X). Thus, PCC is similarly designed to adjust category prototypes for different (*(X) by:
ck+1= = Scx + W
where f-)(-) is a prototype selection function as shown in Fig. Since f*+ derives from f*(X), the category center at the (I+11)- iteration should not deviate from that of the k-th iteration. Thus, for(1) actually serves as a gating function implemented by channel attention mechanism, which controls the information flow from Ch to c++1 This can ease the training difficulty of PCC by avoiding repetitive learning for Ck+1 W is learnable bias at the k-th iteration, which supplements some specific information for ck+1
At the k-th iteration, with the visual representation (x) and category prototypes Ck (ct, ... J. PCC repels different categories by:
Σ log exp(X* (X)Tx(t XEX,
C
(9)
EjEV, Ejey,epp(( exp(s*
5

---

Similar to L La is applied to all the K' iterations. Compared to Lozs in Eq. ① Let can better repel visual representations from different categories via progressive category prototypes {C* k 1, K.
With wetpsssiiiil--pdateee category prototypes, PCC improves the category discriminability of visual representations (X)] 1, K).
3.4 Overall Objective
Overall, the objective loss function of DPPN is:
Call E + XL2),
(10)
where À is the hyper-parameter balance Let. The attribute prototypes and category prototypes collaboratively trained a unified framework, which enables the final visual representation f (X)
to simultaneously capture attribute-region correspondence and category discrimination. During
inference, only visual representation f (X) at the K-th iteration used by:
j = arg min d(f (x))) NEY.Uy...
(11)
3.5 Discussion
Compared to APN 47 DPPN is a much different and novel method with three main differences: a) instead of sharing prototypes for all images in APN, DPPN dynamically adjusts attribute prototypes according to different images. Specifically, DPPN introduces attribute- clues from the target image feature into attribute prototypes. that the prototypes result in better attribute localization; b) different from APN's averagely pooling local visual features into a global one, DPPN concatenates local features represent an image, which better preserves attribute-region correspondence. The final representation of DPPN is made up of attribure-localized features during both training and inference, instead of regarding attribute localization mere supervision during training in APN; and c) DPPN further exploits progressive category prototypes to repel visual representations from different categories, which enhances category discrimination.
adapted to the target image and
4 Experiments
4.1 Experimental Settings
Datasets.
Four public GZSL benchmarks, i.e., Caltech-USCD Birds-200-2011 (CUB) 43 SUN
35). Animals with Attributes2 (AWA2) (24), and Attribute Pascal and Yahoo (aPY) 14 are adopted in this paper. CUB contains 11, 788 bird images in 200 species with 312 description attributes. SUN contains 14, 340 scene images in 717 classes with 102 attributes. AWA2 contains 37, 322 animal images in 50 classes with 85 attributes. aPY contains 15, 339 object images in 32 categories with 64 attributes.
Evaluation Metrics. The widely-used harmonic mean H = (2MCA, x (CAA)/MMAA + MCA,) used to evaluate GZSL performance. MCA, and MCA,, are the Mean Class Top-1 Accuracy for seen and unseen domains, respectively.
Implementation Details. The input images are resized to 448 x 448 following [55 45 Random cropping and flipping are used for data augmentation. ResNet-101 [18 pretrained on ImageNet [2 used the backbone. A two-step training strategy is adopted, which trains DPPN with the fixed backbone and then fine-tunes the whole network on two 1080ti GPUs. Adam optimizer [22) is used with batch size of 64 and (r 2e C' 512 since we use a conv. layer for dimension reduction after the backbone. N₂ is the number of attributes, which is 312, 85, 64, 102 for respective CUB, AWA2, aPY, and SUN. K 3 and À 1.0. N, will be discussed in the ablation study.
4.2 Ablation Study
Analysis of Attribute Localization and Category Discrinination. The core motivation of DPPN is to learn visual representations that simultancously explore category discrimination and attribute-

---

Method
La
Ca
Lu
K
MCA.
CDB
MCA.,
H
MCA.
aPY
MCA,
GFLOPs
Base-V2S
50.5
84.4
63.2
30.4
42.6
35.5
62.396
+FCC
70,2
+PAL
7
64.7
71.8
68.1
34.2
53.8
41.8
62,809
#POC&TAL
69.2
71.4
70.3
35.6
39.0
44.4
62,842
#FCC
10.2
77.1
13.3
40.0
61.2
62,500
Table I: Effect of PCC and PAL on CUB and aPY datasets. GFLOPs is calculated with input size 448 448 the CUB dataset.
13
ConoC Color
Throat Color
Tail Shape
Leg Color
Bill Color
Tail Color
Wing Shape
Figure 3: Visualization of attribute localization at different iterations. The localization gets more and
more accurate as k increases from 0 to 2.
region correspondence via the proposed PCC and PAL, thus analyse how PCC and PAL affect GZSL performance in this part. For simple comparison, we set K = in PCC and PAL. Results listed in Table [T] "Base-V2S" V2S" is the baseline method trained by Lu2s in Eq. ① "+PCC" adds PCC module to Base-V2S. which introduces stronger category discrimination constraint L..- We can observe that "+PCC" obtains 1.1% and 4.7% gains H over "Base-V20 V2S" CUB and aPY. This is because that PCC explicitly pushes representations away from different categories via category prototypes, thus enlarging category margins for more accurate category classification and boosting GZSL. The third model "+PAL" replaces f(-) in "Base-V25 with PAL, which enables visual representations with attribute localization ability. Compared with "Base- V2S". PAL module brings 4.9% and 6.3% gains on CUB and aPY. This derives from that PAL captures attribute-region correspondence by utilizing attribute prototypes localize attribute-relater local regions and produce serribute-specfiii visual representations. Finally, ++PCC&PAL simultaneously considers category discrimination and attribute localization by incorporating both PCC and PAL, which obtains 7.1% and 8.9% gains. This proves that both attribute-region correspondence and category discrimination are critical to GZSL and complementary to each other.
Notably, PAL. and PCC bring negligible additional computation, even when K = 3, because fra(-) in Eq. 66) controls the dimension of representation f²(X) to limit the computation burden. We visualize the difference of representation distribution between Base- V2S and our DPPN in the Appendix.
Effect of Progressive Prototype Updating. DPPN progressively updates attribute and category prototypes to learn more transferable and distinctive representations. Here, analyse how such progressive learning strategy impacts the attribute localization ability and category recognition by evaluating varying K in PAI. and PCC. The results are given in Fig.
As K rises from 1 3, on all the four datasets gradually increases. The best H 73.5%. 73.1%. 48.4%, and 41.0% on CUB, AWA2, aPY, and SUN is obtained when K = 3. This demonstrates that, with category and attribute prototypes updated, the visual representations become more discriminative and transferable to the unseen categories. Here, to intuitively present the attribute localization progressively learned by PAL, we visualize the attribute localization results of PAL at different iteration & when setting K = 3. As shown in Fig. 3 the localization gets more and more precise as attribute prototypes gradually update. With progressively updated attribute prototypes, the PAL module can finally accurately localize corresponding attribute-relater visual regions. Besides, with updating, the prototype for the same attribute gets more specifie to the target image the first four columns, reflecting that progressive updating can adapt prototypes according different images.
7

---

MCA MCA H
45
50
40
25
30
K
CUB
AWA2
aPY
SUN
Figure 4: Effect of progressive updating with varying K on four datasets.
PAL
PCC
CUB
aPY
farl
caf(-1
(um())
fea(-)
MCCA,
MCA,,
H
MCA.
ATCA,
H
70.2
77.1
73.5
40.0
61.2
48.4
67.4
78.2
72.4
36.3
60.9
45.5
67.0
70.3
68.7
35.6
62.0
45.2
70.4
73.0
71.7
37.3
59.4
45.8
68.3
76.0
71.9
38.4
57.1
45.9
Table Evaluation of components in DPPN.
This proves that attribute prototypes can capture the attribute-region correspondence, and progressive updating makes prototypes more specific and distinctive. When K > 3, H drops. The reason may be that the over-updated prototypes become unstable and hard to train. K = is a good trade-off between general knowledge of whole dataset and specific knowledge towards image. Thus, we set K = 3 for the rest experiments.
In summary, both quantitative and qualitative results demonstrate that progressive updating can improve attribute and category prototypes, which better captures attribute-region correspondence and category discrimination.
Evaluation of Components PAL. PAL aims to learn visual representations with accurate attribute localization. Thus, we evaluate two important components of PAL, ie., far(-) (-) in Eq. (5) and cal(.) in Eq. (6) The results are given in Table 2 improve prototype quality. As shown the second row of Table without far (.). H drops by 1.1% and 2.9% on CUB and aPY, respectively. This demonstrates that far(.) benefits attribute prototype updating, thereby boosting attribute localization ability.
far(-) is refinement function between Pk and pk+1 to
cat(.) is the aggregation function used to produce visual representation f*(X) by concatenating attribute prototypes P = {pf.... PN Compared to summing (pi. ... . } up or max pooling operation, cat(-) better preserves attribute-region correspondence. The third and fourth rows Table 2 show that H drops replacing eat(-) with either summing up max pooling. This proves that cat(-) benefits local correspondence preservation.
Effect of (.) in PCC. (.) in Eq. (8) serves as a gating function to ease the training of category prototypes. this part, we analyse the impact of f.) Eq. (8) f..()) (-) removes redundancy in Ck Without fee (.), the category prototype updating becomes C++ C W which passes all information in Ch to C++1 The results are listed in the last row of Table 2 Without f.) H drops by 1.6% and 2.5% on CUB and aPY. This proves that (-) helps to ease the training process of PCC, which learns more discriminative visual representations.
Effect of X. in Fig. 5 (a) and (b). As À rises from 0.0 to 1.0, i.e., category discrimination supervision La is introduced into DPPN, H increases on both CUB and aPY. The best H is obtained when À = 1.0. This proves the effectiveness of category discrimination brought by PCC. When À 1.0, H starts to drop. Thus, we set À = 1.0 for better results.
the hyper-parameter to balance L.,, Here, we evaluate the effect of as shown
Effect of N.. N, is the dimension of (X) and N, = D No. N₂ is the number of attributes, which are 312, 85, 64, 102 for CUB, AWA2, aPY, and SUN, respectively. Here, we set different D to evaluate the effect of N. on recognition performance and calculation addition. Fig. (c) shows values H as N, varies. 5 shows GLOPs addition over N, 512. When N,, around 2, 048, best performance is obtained with relatively small calculation complexity addition. Thus, we set N,, 2496, 21 2125, 2048, 2142 and 8, 25, 32, 21 for CUB, AWA2, aPY, and SUN, respectively.

---

MCAA MEA
MCA - -MCA
CUB aPY
[461
2025
05015
401
0
N
015
10
05 (b) Top- AAC. an
612
1004
2008
4036
312
1028
204
400
(a) Top-1 100.-AACC.---. CUB
(c)
H- on CUB and aPY
(c)
GFLOP&AAd
Figure Effect of hyper-parameters À and N
CUB
AWA2
aPY
SUN
Methods
MCA.
MCAA,
MCA.
MCA,
H
MCA,
MCA,
MCA.
MCA,
IZF-Softmax
52.7
68.0
59.4
60.6
77.5
68.0
42.3
60.5
49.8
52.7
57.0
54.8
TE-VAEGAN 44
63.8
79.3
70.7
41.8
51,9
46.3
E-PGN 50
52.0
61.1
56.2
52.6
83.5
64.6
60.4
75.1
67.0
37.1
56.8
47.9
CE-GZSL
63.9
66.8
65.3
63.1
78.6
70.0
48.8
38.6
43.1
MLSE
22.3
71.6
34.0
23.8
83.2
37.0
12.7
74.3
21.7
20.7
30.4
26.4
COSMO
57.8
50.2
37.7
41.0
PREN
32,5
55.8
43,1
32.4
88.6
47.4
35.4
27.2
30.8
VSE-S
33.4
87.5
48.4
41.6
91.3
57.2
24.5
72.0
36.6
29
43.4
79.6
56,2
50.0
90.3
64.4
20.8
34,9
26.1
AREN 45
63.2
69.0
66.0
54.7
79.1
64.7
30.0
47.9
36.9
40.3
32.3
35.9
CosineSoftmax 25
47.4
47.6
47.5
56.4
81.4
66.7
26.5
74.0
36.3
42.8
39.3
RGEN 46
73.5
60.0
66.1
76.5
67.1
71.5
48.1
30.4
37.2
31.7
44.0
36.8
DAZLE
56.7
59.6
58.1
60.3
75.7
67.1
52.3
24.3
33.2
56.5
78.0
65.5
37.6
GEM-7SL,
3D
64.8
77.1
70.4
64.8
77.5
70.6
38.
35.7
36.9
DPPN
70.2
77.1
73.5
63.1
86.8
73.1
40.0
61.2
48.4
47.9
35.8
41.0
Table 3: Results of GZSL on four classification benchmarks. Our DPPN belongs to embedding-
methods (EMB.). Generative methods (GEN.) utilize extra synthetic unseen domain data for training. The best result is bolded, and the second best is underlined.
4.3 Comparison with State-of-the-Art Methods
We compare DPPN with the state-of-ther of-the-art GZSL methods, of which the results given in Table
Among existing methods, APN 47 the most related method which also utilizes visual prototypes to localize visual parts. Different from APN that counts only sharing prototypes for all images, our DPPN adjusts attribute prototypes dynamically according to the target image and exploits category prototypes to enhance category discrimination. Thus, DPPN learns specific and distinctive visual representations and surpasses APN by a large margin, i.e., 6.3%. 7.6%, and 3.4% for H on CUB, AWA2, and SUN datasets, respectively. APN does not provide codes and is approximately similar to PAL with K 1. As shown in Fig. 3 it qualitatively implies that progressive updating attribute prototypes can learn visual representations with better attribute localization ability. Besides, DAZLE 20 uses dense attribute attention to focus on relevant regions, which is inferior to DPPN by 15.4%. 6.0%. and 7.8% CUB, AWA2, and SUN, respectively. This proves the effectiveness of designing progressive attribute and category prototypes in GZSL.
Compared to other embedding- based methods, our method surpasses the best one by respectively 3.1%. 1.6%, and 9.4% for H on CUB, AWA2, aPY datasets, and obtains comparable best H performance SUN dataset. The results demonstrate that progressively exploring attribute-region correspondence and category discrimination can effectively enhance cross-domain transfcrability and category discriminability of visual representations. The reason for relatively small improvement SUN may be that #categories is large while #images each category is small in SUN, leading to difficulties for DPPN to learn accurate attribute localization and category discrimination.
Compared to generative methods that utilize additional unseen category labels during training, DPPN can achieve comparable, even better results, especially on CUB and AWA2 datasets. This reveals that with the assistance progressive attribute localization and category discrimination, DPPN can surpass generative methods without complex GAN training.

---

Conclusion
In this paper, we propose a Dual Progressive Prototype Network (DPPN) to progressively explore both attribute-region correspondence and category discrimination for GZSL. Specifically, DPPN constructs progressive prototypes for both attributes and categories. DPPN alternatieeyy localizes attribute- related visual regions and adjusts attribute prototypes towards target images, which improves attribute localization ability and cross-domain transferability of visual representations. Along with progressive attribute prototypes, DPPN progressively projects category prototypes to multiple spaces to enforce visual representations away from different categories, thus enhancing category discriminability. Extensive experimental results on four public datasets demonstrate the effectiveness of our DPPN.
Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation of China (NSFC) under Grants 61632006 and 62076230.
References
[1] Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(7), 1425-1438 (2016)
[2] Akata, Z., Reed, S., Walter, D., Lee, H., Schiele, B.: Evaluation of output embeddings for fine-grate mne-grained image classification, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2927-2936 (2015)
[3] Annadani, Y., Biswas, S.: Preserving semantic relations for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7603-7612 (2018)
[4] Atzmon, Y., Chechik, G.: Adaptive confidence smoothing for generalized zero-shot learning. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
1 11671-1 1680 (2019)
[5] Ba, J.L., Swersky, K., Fidler, S., salakhutdinov, R.: Predicting deep zero-shot convolutional neural networks using textual descriptions. In: Proceedings of the IEEE International Conference on Computer Vision. pp.
(2015)
[6] Bucher, M., Herbin, S., Jurie, F:: Improving semantic embedding consistency by metric learning for zero-sho shot classiffication. In: Proceedings of the European Conference Computer Vision. pp. 730-746 (2016)
[7] Changpinyo, S., Chao, W.L., Gong, B., Sha, F: Synthesized classifiers for zero-shot learning. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5327-5336 (2016)
[8] Chao, W.L., Changpinyo, S., Gong, B., Sha, F: An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In: Proceedings of the European Conference on Computer Vision. pp. 62-68 (2016)
[9] Chen, L., Zhang, H., Xiao, J., Liu, W., Chang, S.F.: Zero-shot visual recognition using semantics preserving adversarial embedding network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1043-1052 (2018)
[10] Chen, X., Lan, X., Sun, F., Zheng, N.: A boundary based out-of-distribution classifier for generalized zero-shot learning. In: Proceedings of the European Conference Computer Vision. pp. 572-588 (2020)
[11] Chou, Y.Y., Lin, H.T., Liu, T.L.: Adaptive and generative zero-shot learning. In: International Conference Learning Representations (2021)
[12] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 248-255 (2009)
10

---

[13] Ding, Z., Liu, H.: Marginalized latent semantic encoder for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6191-6199 (2019)
[14] Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778- 1785 (2009)
[15] Ferrari, V., Zisserman, A.: Learning visual attributes. Advances Neural Information Process- ing Systems 20, 433 440 (2007)
[16] Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T., et al.: Devise: A deep visual-semantic embedding model. In: Advances in Neural Information Processing Systems. pp. 2121-2129 (2013)
[17] Han, 7., Fu, 7.,, Chen, S., Yang, J.: Contrastive embedding for generalized zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2371-2381 (2021)
[18] He, K., Zhang, X., Ren, S., Sun, J:: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 770-778 (2016)
[19] Huang, H., Wang, C., Yu, P.S., Wang, C.D.: Generative dual adversarial network for generalized zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 801-8 201-810(( (2019)
[20] Huynh, D., Elhamifar, E.: Fine-grained generalized zero-shot learning via dense sttribute-based attention. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 4483-44999 (2020)
[21] Jiang. H., Wang, R., Shan, S., Chen, X.: Transferable contrastive network for generalized zero-shot learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 9765-9774 (2019)
[22] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (2015)
[23] Kodirov, E., Xiang. T., Gong, S.: Semantic autoencoder for zero-shot learning. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3174-3183 (2017)
[24] Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 951-958 (2009)
[25] Li, K., Min, M.R., Fu, Y.: Rethinking zero-shot learning: A conditional visual classification perspective. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3583-3599 (2019)
[26] Li. Y., Wang, D., Hu, H., Lin, Y., Zhuang, Y.: Zero-shot recognition using dual visual-semantic mapping paths. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 3279-328 (2017)
[27] Liu, L., Zhou, T., Long, G., Jiang, J., Dong. X.. Zhang, C.: Isometric propagation network for generalized zero-shot learning. In: International Conference on Learning Representations (2021)
[28] Liu, S., Long, M., Wang, J., Jordan, M.I.: Generalized zero-shot learning with deep calibration network. In: Advances Neural Information Processing Systems. pp. 2005-2015 (2018)
[29] Liu, Y., Guo, J., Cai, D., He, X.: Attribute attention for semantic disambiguation in zero- shot learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp- 6698-6709 (2019)
11

---

[30] Liu, Y., Zhou, L., Bai, X., Huang. Y. Gu, L.. Zhou, J., Harada, T.: Gioal-oriette gaze estimation for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3794-3806 (2021)
[31] Min, S., Yao, H., Xie, H., Wang, C., Zha, ZJ., Zhang, Y.: Domain-aware visual bias eliminating for generalized zero-shot learning. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12664-1222 1264-1263322 (2020)
[32] Min, S., Yao, H.. Xie, H., Zha, Z.J., Zhang, Y.: Domain-specific embedding network for zero-shot recognition. In: Proceedings of ACM International Conference on Multimedia, pp. 2070-2078 (2019)
[33] Min, S., Yao, H., Xie, H., Zha, Z.J., Zhang, Y.: Domain-orienter semantic embedding for zero-shot learning. IEEE Transactions on Multimedia 23, 3919-39999999 --33 (2020)
[34] Narayan, S., Gupta, A., Khan, F.S., Snock, C.G., Shao. L.: Latent embedding feedback and discriminative features for zero-shot classification. In: Proceedings of the European Conference on Computer Vision. pp. 1-23 (2020)
[35] Patterson, G., Hays, J.: Sun attribute database: Discovering, annotating, and recognizing scene attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2751-2758 (2012)
[36] Rahman, S., Khan, S., Porikli, F.: A unified approach for conventional zero-shot, generalized zero-shot, and few-shot learning. IEEE Transactions on Image Processing 27, 5652-5667 (2018)
[37] Reed, S., Akata, Z., Lee, H., Schiele, B.: Learning deep representations of fine-grained visual descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 49-58 (2016)
[38] Romera-Paredes, B., Torr, P.: An embarrassingly simple approach to zero-shot learning. In: International Conference on Machine Learning. pp. 2152-2161 (2015)
[39] Shen, Y., Qin, J., Huang, L.: Invertible zero-shot recognition flows. In: Proceedings of the European Conference on Computer Vision. pp. 614-631 (2020)
[40] Shigeto, Y., Suzuki, I., Hara, K., Shimbo, M., Matsumoto, Y.: Ridge regression, hubness, and zero-shot learning. In: Joint European Conference on Machine Learning and Knowledge Discovery Databases. pp. 135-151 (2015)
[41] Socher, R., Ganjoo, M., Manning, C.D., Ng, A.: Zero-shot learning through cross- moosl transfer. In: Advances in Neural Information Processing Systems, pp. 935-943 (2013)
[42] Tong, B., Wang, C., Klinkigt, M., Kobayashi, Y., Nonaka, Y.: Hierarchical disentanglement of
discriminative latent features for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11467-11476 (2019)
[43] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-uest birds-200-22 dataset. In: California Institute of Technology. pp. 1-8 (2011)
[44] Xian, Y., Akata, Z., Sharma, G., Nguyen, Q. Hein, M., Schiele, B.: Latent embeddings for zero-shot classification. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition. pp. 69-77 (2016)
[45] Xie, G.S., Liu, L.,, Jin, X., Zhu, F., Zhang, Z., Qin, J., Yao, Y., Shao, L.: Attentive region bedding network for zero-shot learning. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9384 9393 (2019)
[46] Xie, G.S., Liu, L., Zhu, F., Zhao, F., Zhang, Z., Yao, Y., Qin, J., Shao, L.: Region graph embedding network for zero-shot learning. In: Proceedings of the European Conference on Computer Vision. pp. 562-580 (2020)
[47] Xu, W., Xian, Y., Wang, J., Schiele, B., Akata, Z.: Attribute prototype network for zero-shot learning, In: Advances in Neural Information Processing Systems. pp. 21969-2111111 22980 (2020)
12

---

[48] Ye, M., Guo, Y.: Progressive ensemble networks for zero-shot recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11728-11723 (2019)
[49] Yu, Y., Ji, Z., Fu, Y., Guo, J., Pang, Y., Zhang, Z. Stacked semantic-guided attention model for fine-grained zero-shot learning. In: Advances in Neural Information Processing Systems, 5998 6007 (2018)
[50] Yu, Y., Ji, Z., Han, J., Zhang, Z.: Episode-based prototype generating network for zero-shot learning. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 14035-14044 (2020)
[51] Yue, Z., Wang. T., Zhang, H., Sun, Q. Hua, X.S.: Counterfactual zero-shot and open-set visual recognition. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 16004-1414 (2021)
[52] Zhang, H., Long, Y., Guan, Y., Shao, L..: Triple verification network for generalized zero-shot learning. IEEE Transactions on Image Processing 28(1), 506-517 (2019)
[53] Zhang, H., Koniusz, P.: Zero-shot kernel learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7670-7679 (2018)
[54] Zhang, L., Xiang, T., Gong, S.: Learning a deep embedding model for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2021-2030 (2017)
[55] Zhu, P., Wang, H., Saligrama, V.: Generalized zero-shot recognition based on visually se- mantic embedding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2995-3003 (2019)
13


Reviews:
[{"review_id": "uURCUMgklVZ", "reviewer": "Reviewer_LSav", "summary": "This paper proposes a Dual Progressive Prototype Network that updates the prototypes for each attribute and categories progressively. To achieve this, an alternation updating strategy is adopted to adjust prototypes according to input images. The model is evaluated on 4 standard ZSL datasets and obtains the new state-of-the-art performance. The effect of each component is evaluated with extensive ablation study.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. The alternative updated prototype can better capture the local information of each attribute and result in better ZSL knowledge transfer. The extensive experiments and ablation study validate the effectiveness of the proposed method.\n\n[Post rebuttal] The authors have addressed my concern. After reading other reviewers' comments I still lean to accept the paper as the alternative updated prototype seems interesting to the community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "However, I have the following concerns towards the network architecture, loss function, and the training procedure.\n\n1. Line 215: is the architecture of the ResNet101 backbone changed? Why $C = 512$? Should it be $2048$?\n\n2. Why not presenting the ZSL performance as well? The ZSL performance can present knowledge transferability as well.\n\n3. Line 248: Any insights on why the localization ability will be more and more precise? Why the prototypes have self-correction ability? Apart from the qualitative results, a quantitative evaluation will be more convincing.\t\t\n\n4. Equation 9: it seems that the loss is an ordinary cross-entropy loss. It might be inappropriate to call it contrastive loss. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "snLpMPTk1j", "reviewer": "Reviewer_ZcLN", "summary": "The bias towards seen categories in GZSL is a main challenge, and the authors argue that shared prototypes across all images are partly responsible, as well as providing suboptimal discriminability. Thus, the paper proposes increasing the discriminability of attributes and categories prototypes in a per-image basis via progressive refinement through an iterative attention mechanism. The results show improvement over related works, such as AREN and APN.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe paper borrows from previous frameworks and the main novelty is the progressive adaptation of prototypes to each image, which is interesing. Overall, the novelty is moderate.\n\nQuality:\nThe submission is technically sound, with exhaustive experiments.\nYet widely used, I'm concerned about the harmonic mean (H) metric not being an accurate representation of the performance of a GZSL method, since it requires calibration, and thus can be unfair to compare non-calibrated approaches, as discussed in [Changpinyo2020]. The seen-unseen accuracies curve and AUSUC metric described in [Changpinyo2020] are arguably better metrics. \n\nClarity:\nThe submission is relatively clear and well organized. However, I have some concerns:\n- Not clear if the method uses a part extractor or just operates on the convolutional feature tensor. Regions refers to extracted bounding boxes, or implicit local regions corresponding to receptive fields of representations? \n- Not clear how local features are aggregated into the global representation. It usually uses GAP, but here is removed and not clarified afterward (or couldn't find it).\n- Please don't use comma to separate thousands when enumerating numebers, e.g. N_v=2, 496, 2, 125, 2, 048, 2, 142. It is very confusing.\n\nSignificance:\nThe authors propose essentially an improvement over a previous GZSL framework. The results provide a moderate improvement over the baseline (i.e. shared prototypes), but I think the significance may be low since I think the problem of discrimination lies less in the alignment of prototypes, and more in the aggregation mechanism. \n\nI have several concerns about the paper:\n- The paper focuses on improving the discriminability of attribute prototypes, but not on how they are aggregated in the global representation (which is not clear to me from the text). Recently, SELAR showed that an important bottleneck in GZSL to benefit from localized attribute representations is precisely the aggregation mechanism via GAP, showing a large improvement in discriminability for GZSL simply replacing GAP in ResNet by global max pooling. Thus, the gain of the propose approach may be only significant with a suboptimal aggregation mechanism (e.g. GAP or other).\n- Thus, the authors should compare with SELAR, which is a straightforward to implement baseline, and then implement their approach to evaluate whether progressive prototype improvement is still significant when a more discriminative aggregation mechanism such as GMP is used.\n- The comparison may not be fair to many methods. The proposed method requires a large image of 448 x 448 (as AREN), but most papers work on 224x224 pixels (e.g. SELAR). It also requires fine tuning (perhaps also a part extractor??), while other methods use non fine tuned feature extractors. For a clear and fair comparison, all these details should be explicitly mentioned. \n\n[Chang2020] Classifier and Exemplar Synthesis for Zero-Shot Learning, IJCV 2020\n[SELAR] On Implicit Attribute Localization for Generalized Zero-Shot Learning, SPL 2021", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors didn't discussed these topics in the manuscript.\nRegarding limitations of their work and suggestions, please refer to my comments above. About societal impact, I don't foresee any significant negative one.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "UYxC367Gv6", "reviewer": "Reviewer_UvLd", "summary": "This paper aims for generalised zero-shot learning with prototypes. Rather than using one set of attributes, two sets are used here: attribute localization prototypes and category-level prototypes. Experiments on four datasets show the effect of the proposed approach, with top scores for harmonic means.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe first strength of the paper is the notion of two levels of prototypes. Prototypes have recently shown to be effective for (generalised) zero-shot learning but mostly focus on attribute learning. Here, prototypes are learned both at the attribute-level and category-level, to both help with finding suitable attribute representations and class representations.\n\nSecond, the paper is accompanied by many ablation studies to better understand the method and its parameters. Moreover, the comparative evaluation shows the effectiveness of the proposed approach, with state-of-the-art harmonic means on all four datasets.\n\nThird, the figures in the paper help with understanding the method and the difference with related research. The discussion on the relation with priors works is also extensive, which helps with placing the paper into the proper context.\n\nWeaknesses:\n\nThe first weakness of the paper is that the method is not well explained. While the method is not overly complicated, the writing and explanation do not help with understanding what is going on. At a high-level, this paper follows APN with category-level prototypes on top. In multiple cases, the paper makes things harder to understand however. These include:\n\n- The paper states that “APN [47] constructs prototypes for separate attributes, which are shared by all images to localize attribute-related regions via region searching. In this paper, instead of using prototypes shared by all images, DPPN dynamically adjusts attribute prototypes according to different images” (lines 91-93) and “First, instead of using shared prototypes, DPPN progressively adjusts attribute prototypes according to different images via the novel PAL.” (Line 194-195). What does this even mean? According to Equation 7, the proposed loss is over all images, just like in APN. It is not clear what is precisely novel in the first part of the method based on the paper.\n- How is the progressive learning done? Is there a stage-wise training of the attribute prototypes first, then the same for the category prototypes, and finally the overall loss is optimised? Or is everything optimised end-to-end? Any explanation on how the actual training is done is missing in the paper.\n- “Similar to Lsa, the multi-stage supervision is used in Lcl.” What does multi-stage supervision mean?\n\nA conceptual difference with APN is the use of category attributes. The loss in Equation 9 pushes representations from different categories further away. While this makes sense in a supervised setting, it poses a problem in the zero-shot setting, as it becomes harder to have an unseen-class be the most likely category. This is also evident from the experiments. While the harmonic mean is high overall, this is clearly due to a better recognition of the seen categories. The accuracy of the unseen categories is not great across the datasets compared to other approaches. The harmonic mean is saved by the improved recognition for seen categories. As such, it seems that the approach is more tailored towards supervised learning than zero-shot learning. Table 1 makes things harder to understand, since the results between CUB and aPY are anything but consistent.For CUB the seen accuracy actually goes down, while it forms the biggest improvement on aPY.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The checklist mentions Section 4.3 for a discussion on limitations, but it only covers praise for the method, except for a footnote on SUN. The societal impact statement notes that fewer annotations due to zero-shot generalization will lead to unemployed annotators.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1ZfgWr_Jusx", "reviewer": "Reviewer_LFjT", "summary": "A Dual Progressive Prototype Network (DPPN) approach is proposed to progressively explore both attribute-region correspondence and category discrimination for GZSL. It alternatively locates attribute-related visual regions and updates attribute prototypes, which improves attribute localization ability and cross-domain transferability of visual representations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It's reasonable to alleviate domain shift problem by progressively adjusting the attribute prototypes and enlarging the category margins. The paper is logical and easy to understand. However, there are some details should be further stated and analyzed.\n1.\tSome of the descriptions in this paper is not accurate enough, e.g., for GZSL, the search space should be notified.\n2.\tIn the ablation study, the accuracy of model with PCC and PAL on the seen samples is descended, and it seems that this method restrains the performance of classification on seen samples. I wonder the reason of the phenomenon, and the authors should explain it. Besides, the authors only discuss the ablation study on CUB and aPY, and I am curious about the results on AWA and SUN. It would be better if the authors shows it more completely.\n3.\tBesides the APN, whether the other competitive methods are bound up with DPPN? If they are, the authors should state mean idea of them and the similarities and differences between them and DPPN. \n4.    The performance on the SUN dataset is not good actually, comparing with the recent work: \nZongyan Han, Zhenyong Fu, and Jian Yang. Learning the redundancy-free features for generalized zero-shot object recognition. In CVPR, pages 12862–12871, 2020.\nShaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zheng-Jun Zha, and Yongdong Zhang. Domain-aware visual bias eliminating for generalized zero-shot learning. In CVPR, pages 12661–12670, 2020\nWenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. In NeurIPS, pages 21969–21980, 2020.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses generalised ZSL via proposing a new prototype refinement-based scheme to reduce confusion between known and novel categories. Reviewers felt there were various sources of confusion and explanation quality. However they also agree that the methodology is solid enough and the evaluation is strong. I recommend accept, and encourage the authors to take reviewers comments on board in refining the quality and clarity of exposition in the final version. 
Decision: Accept (Poster)