Dual Progressive Prototype Network for Generalized Zero-Shot Learning
Authors: Chaoqun Wang
Shaobo Min
Xuejin Chen
Xiaoyan Sun
Houqiang Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: generalized zero-shot learning
progressive prototypes

Abstract: Generalized Zero-Shot Learning (GZSL) aims to recognize new categories with auxiliary semantic information, e.g., category attributes. In this paper, we handle the critical issue of domain shift problem, i.e., confusion between seen and unseen categories, by progressively improving cross-domain transferability and category discriminability of visual representations. Our approach, named Dual Progressive Prototype Network (DPPN), constructs two types of prototypes that record prototypical visual patterns for attributes and categories, respectively. With attribute prototypes, DPPN alternately searches attribute-related local regions and updates corresponding attribute prototypes to progressively explore accurate attribute-region correspondence. This enables DPPN to produce visual representations with accurate attribute localization ability, which benefits the semantic-visual alignment and representation transferability. Besides, along with progressive attribute localization, DPPN further projects category prototypes into multiple spaces to progressively repel visual representations from different categories, which boosts category discriminability. Both attribute and category prototypes are collaboratively learned in a unified framework, which makes visual representations of DPPN transferable and distinctive.	Experiments on four benchmarks prove that DPPN effectively alleviates the domain shift problem in GZSL.

Dual Progressive Prototype Network for Generalized Zero-Shot Learning
Chaoqun Wang's Shaobo Min', Xuejin Chen' Xiaoyan Sun', Houqiang Li!
'School of Data Science
The National Engineering Laboratory for Brain-inss iraiininspired Intelligence Technology and Application University of Science and Technology of China, Hefei, Anhui, China
Tencent Data Platform, Shenzhen, Guangdong, China cq14@mail ustc. edu. cn, beb.bobminnteeneni com, (xjchen99, ,sunxiaoyan, 1ihq] @ustc. adu. cn
Abstract
Generalized Zero-Shot Learning (GZSL) aims to recognize new categories with auxiliary semantic information, e.g., category attributes. this paper, we han- dle the critical issue of domain shift problem, i.e., confusion between seen and unseen categories, by progressively improving cross-domain transferability and category discriminability of visual representations. Our approach, named Dual Progressive Prototype Network (DPPN), constructs two types of prototypes that record prototypical visual patterns for attributes and categories, respectively. With attribute prototypes, DPPN alternately searches attribute-related local regions and updates corresponding attribute prototypes to progressively explore accurate attribute-region correspondence. This enables DPPN produce visual representa- tions with accurate attribute localization ability. which benefits the semantic visual alignment and representation transferability. Besides, along with progressive at- tribute localization, DPPN further projects category prototypes into multiple spaces to progressively repel visual representations from different categories, which boosts category discriminability. Both attribute and category prototypes collabora- tively learned in a unified framework, which makes visual representations of DPPN transferable and distinctive. Experiments four benchmarks prove that DPPN effectively alleviates the domain shift problem in GZSL.
Introduction
Deep learning methods depend heavily on enormous manually-labelled data, which limits their further applications [7 52
36 19 [T]. Therefore, Generalized Zero-Shot Learning (GZSL) recently attracts increasing attention, which aims to recognize images from novel categories with only seen domain training data. Due unavailable unseen domain data during training, GZSL methods introduce category descriptions, such as category attributes associate two domain categories.
[5] or word embedding
[7 26
41
basic framework of embedding- sponding category descriptions in joint embedding space [16] 55
GZSL to align global image representations with corre- 2
6), as shown in Fig. (a). Due to the domain shift problem across two domain categories, unseen domain images tend to be misclassified as seen categories. To address this issue, recent methods focus discovering discrimi- native local regions to capture subtle differences between two domain categories. For example, AREN (45] and VSE (55] leverage attention mechanism to discover important part regions, which improves feature discrimination. DAZLE 20 and RGEN 46 introduce semantic guidance, e.g., category
38
"Corresponding Author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Localiziing
Updkling
Yellow
Align
Fmy
Paws
Visual Represantation
Updkting
Shariing Atribbtte
Progressie Attribute
Progressive Catrgory Prototypes
(a)
(b)
Protolypes
(c)
Protolypes
Figure I: The motivation of DPPN. (a) General GZSL methods directly align global image features with category attributes. (b) A typical butt-based method, i.., APN [47], learns prototypes shared by all images for attribute localization. (c) DPPN progressively adjusts prototypes according different images and introduces category prototypes to enhance category discriminability.
attributes, into region localization to narrow the semantic-visual gap. Among existing methods, APN [47 is most related to our approach. As shown Fig. (b), APN constructs visual prototypes to indicate the typical visual patterns of each attribute, for example describing what attribute "Furry" visually refers to, and these prototypes are shared across all images to search attribute-matched local regions. However, due to image variances, the textures corresponding to the same attribute may vary seriously across images. Thus, sharing prototypes APN can not well depict the target image.
In this paper, we propose a novel Dual Progressive Prototype Network (DPPN), which constructs two types of progressive prototypes for respective attributes and categories to gradually improve cross-domain transferability and category discriminability of visual representations. Instead of sharing prototypes, DPPN dynamically adjusts attribute prototypes for each image to capture the vital visual differences of the same attribute in different images. This is achieved by alternately localizing attribute regions and updating attributes prototypes in turn, shown in Fig. [ (c). With image- specific prototypes, attribute localization, i.e., attribute-region correspondence, gets more accurate. To explicitly preserve such correspondence in the final representations, DPPN aggregates the attribute- related local features by concatenation, instead of widely-used Global Average Pooling (GAP) that attribute
will damage the attribute localization ability. Furthermore, along with poogressivelyuuddaeee prototypes, DPPN also builds category prototypes to record prototypical visual patterns for different categories. The category prototypes are projected into multiple spaces to progressively enlarge category margins, strengthening category discriminability of visual representations. Consequently, with cross-domain transferability and category discriminability, DPPN can effectively bridge the gap between seen and unseen domains.
Experiments on four benchmarks demonstrate that our DPPN alleviates the domain shift problem in GZSL and obtains new state-of of-the-art performance. Our contributions can be summarized as three- fold. a) We propose a novel Dual Progressive Prototype Network (DPPN) that constructs progressive prototypes for both attributes and categories to gradually improve cross-domain transferability and category diseriminability
visual representations. b) An alternation updating strategy is designed to dynamically adjust attribute prototypes according to target images. Besides, DPPN aggregates attribute-relate local features by concatenation to produce image representations, which explicitly preserves the attribute-region correspondence. c) DPPN projects category prototypes into multiple spaces to progressively enhance category discriminability.
Related Work
Generalized Zero-Shot Learning. GZSL aims recognize new categories using semantic knowl- edge transferred from seen categories. Early GZSL methods learn joint embedding space to align global image representations with corresponding category descriptions, e.g., attributes [[5] 35
or text descriptions [5] 37]. Since unseen and seen categories share common semantic space, the semantic-aligned image representations can be transferred from seen to unseen domain. Based on this paradigm, many works focus on improving the discrimination of embedding space by designing elaborate semantic-visual alignment functions [38
27 For example, some methods 33 use high-dimensional visual features to span the embedding space, which is
40 53 (5)
21

---

proved more discriminative than that spanned by category attributes. Other methods 23 9 42 32
utilize auto-encoders to preserve semantic relationships between categories in the embedding space.
Though effective, these methods suffer from the domain shift problem, Le,, two domains have different data distributions. Since only seen domain images are available during training, images from unseen categories tend to be recognized as seen categories. To this end, DVBE [31] and Boundary-based OOD
[0] explore out-of-distribution detection to treat seen and unseen domains separately. Some
works 28 20 suppress the seen category confidence when recognizing images to better distinguish two domain samples. These methods can effectively alleviate the domain shift problem via extra processing. but they ignore the discriminability of local attribute-related information in distinguishing two domains.
Part-Based GZSL. Since global image representations contain much noisy background information which is trivial for knowledge transfer, recent part-based methods 44 45 47 aim to localize part regions and capture important visual details to better understand the semantic-visue relationship. For example, SaGA [49), AREN(45), and VSE (55) leverage the attention mechanism to learn semantic- relevant representations by automatically discovering discriminative parts in images. RGEN [46] uses the region graph to introduce region-based relation reasoning to GZSL and learns complementary relationships between different region parts inside an image. GEM-ZSL [30 imitates human attention and predicts human gaze location to learn visual attention regions. Usually, the semantic guidance, e.g., category attributes, is used to guide the part localization [29]. Thus their generated local features can better match corresponding category attributes, which can alleviate the domain shift problem. Instead of treating category attributes as an all-in-one vector, DAZLE (20) proposes a dense attribute attention mechanism to produce local attention for each attribute separately. APN 47 constructs prototypes for separate attributes, which are shared by all images to localize attribute-related regions via region searching. In this paper, instead of using prototypes shared by all images, DPPN dynamically adjusts attribute prototypes according to different images, which learns visual representations with more accurate attribute localization and transferability. Besides, in DPPN, we also design progressive category prototypes to enhance category discriminability of visual representations.
3 Dual Progressive Prototype Network
3.1 Problem Formulation
The target of GZSL is to recognize images of novel categories trained with only seen domain data. In this paper, we denote S = {X, y, a, E Xe: y E V.. ay E } as seen domain data, where X E CRCN indicates image features extracted by the backbone network, and X,, RCx1 encodes the local information at the n-th region. y the corresponding category label, and ag E R N.x is the category description, such as the category-level vector with Na attributes. C' is the number of feature channels, and N W x H. The unseen domain data is similarly defined as U1 and V, n Va- = o. Given S during training. GZSL aims to recognize images from either X, or X, during inference. A basic framework is to learn image representation f(X) that is aligned with corresponding category attributes by minimizing:
Loss 10a=-a Σ log exp(f(X)Ta) a.) XEX. EjEy, exp(
where f(-) is a visual projection function, which is generally implemented via Global Average Pooling (GAP) and linear projection. v2s is the abbreviation of visual-to -o-emant--emaatic projection.
Based on Li2s, APN 47 expects to improve the localization ability of attributes for intermediate feature X. Thus, APN constructs set attribute prototypes P = =Pp PNN J. where P, E Rx11 records visual patterns for the i-th attribute, e,g., depicting what attribute "Yellow Wing" looks like. P is learnable and shared by all images. With P, APN learns attribute localization by minimizing:
Lapn Σ d(((X,P),,,) a.).
(2)
FEX,
I(X,P) is an attribute localization function that searches the most related local feature X,, for P. E P, which regresses attributes â, E Nax1, d(c,-) is distance measurement function, e.g., l2 norm, that
3

---

Matri Mullipiiation
Update
Hadamant Product
Yowr jos
IYP
nTP
Features X
pk (X)
Attrbutes
Alribule
Prolulyge
Dimension Reduction
Localization
Redizement
Progrusiiv Caleguay Chesification
Prototype Selection
Ctll
CNN
wel.0poove Aod
ReLJ
poodies
W
(X) C)) 00 XXX CT
Update X
Figure 2: Training pipeline of DPPN. DPPN progressively learns attribute prototypes Pk and category prototypes C* via Le and Ca- { ... K} consist visual representations of K iterations, which gradually capture attribute localization and category discrimination.
aligns the predicted attributes ay and ground truth attributes ag. Finally, the main insight of APN is to minimize Lapn Luss, and the inference function is:
arg
min d(f(X), VEYAO
However, due to instance variance, occlusion, and noise, the visual textures that correspond to the same attribute may vary severely across different images. Thus, it is unreasonable only rely sharing attribute prototypes P to accurately localize attribute-related regions for each individual image. Besides, Lapm applied to intermediate features X mere constraint. During inference, the final visual representation f(X) for recognition is directly aggregated by the intermediate features using GAP, which damages the attribute localization ability.
To this end, our Dual Progressive Prototype Network (DPPN) progressively explores attribute localization and category discrimination for different images with two modules, i.e., Progressive Attribute Localization and Progressive Category Classification, as shown in Fig. 2
3.2 Progressive Attribute Localization
The Progressive Attribute Localization (PAL) module aims to dynamically adjust attribute prototypes according to the target image to progressively capture local correspondence between different attributes and image regions.
Define pD [pf.-. p&, } as set of trainable attribute prototypes, which are randomly initialized and shared across all images. With pe initialized, PAL first localizes related local regions for each prototype p.. and then calculates specific visual features for all (pf.-. ...
.N,_ by:
p1 = (X-)) 1::55==(XXT00 pp)
(4)
where S E IR Nx.. is similarity matrix, and Sn. measures the similarity between the n-th local feature X,, and the i-th attribute prototype p.. fi(-) a Softmax normalization along each column. With 5, Eq. 4 aggregates related region features in X calculate attribute-specific features, which produces Pt [p]. Pk. )- far(.) (.) is prototype refinement function implemented by two fully-connected (FC) layers shown in Fig. 2
Compared to the original image feature X E RCXN, pl CxN explicitly captures specific visual patterns of the target image for each attribute, e.g., df aggregates related local features in X that correspond to the i-th attribute.

---

Considering instance variance, the sharing attribute prototypes po cannot well localize accurate attribute-related regions for all images. In Eq. (4), compared to po, the visual patterns of P!, that correspond to different attributes, are more specific to the given image feature X. Thus, PAL further regards pl as updated attribute prototypes from po for the target image. By replacing P with pk+1 and repeating Eq. ( PAL can progressively adjust attribute prototypes for a specific image by:
pk+1 for ( (Xf(XTp())
(5)
where pk+1 leads to better attribute localization than Pt:
For the k-th iteration, since Ph IRC* contains specific visual patterns for different attributes, PAL concatenates all {p" ... . .Px,) J to produce the visual representation f*(X);
(6) where (ra(.) is dimension reduction layer and projects each p E Rex1 into RDx1, where D < C, to avoid excess calculation complexity. cal(.) concatenates all elements of the input. (*(X) RNxx1, where N, D Na. With multi-iterative (Pk]k- -, ,KK PAL gradually generates K visual representations {f (X))] -,.... ,K).
f*(X) cat (fra
Finally, (X) is aligned with corresponding attributes in joint embedding space by:
c Σ (((((x)) XEX,
where g(-) is a semantic projection function implemented by FC to project attribute vector into latent space, where the visual representations and projected attribute features can be well aligned, following (3I. The semantic alignment supervision Lea is applied all {fth (X) (X)]k 1, K} for training acceleration. As l: increases appropriately, f (X) localizes attributes more accurately. Thus, f (X) is used as the final visual representation for inference.
Consequently, by dynamically adjusting attribute prototypes according to the target image, PAL can progressively improve the attribute localization ability of visual representations, as shown in Fig. 3 PAL captures the attribute-reiion correspondence, which narrows the semantic-visual gap between category attributes and visual representations and boosts knowledge transfer between seen and unseen domains.
3.3 Progressive Category Classification
Besides exploring the correspondence between attributes and local image regions via PAL, we design a Progressive Category Classification (PCC) module to repel visual representations from different categories, which can enlarge category margins.
Similar to the sharing attribute prototypes in PAL, PCC defines a set of learnable category prototypes Co {cf, .cy. ), where cf E N.X. records the visual representation center for the j-th category. No = [D'a] is the number of seen categories. Since the attribute prototypes are progressively updated in PAL and K visual representations {X)(k = 1, K} } are accordingly generated for each image, sharing category prototypes Co cannot well model visual category differences for all iterations of f* (X). Thus, PCC is similarly designed to adjust category prototypes for different (*(X) by:
ck+1= = Scx + W
where f-)(-) is a prototype selection function as shown in Fig. Since f*+ derives from f*(X), the category center at the (I+11)- iteration should not deviate from that of the k-th iteration. Thus, for(1) actually serves as a gating function implemented by channel attention mechanism, which controls the information flow from Ch to c++1 This can ease the training difficulty of PCC by avoiding repetitive learning for Ck+1 W is learnable bias at the k-th iteration, which supplements some specific information for ck+1
At the k-th iteration, with the visual representation (x) and category prototypes Ck (ct, ... J. PCC repels different categories by:
Σ log exp(X* (X)Tx(t XEX,
C
(9)
EjEV, Ejey,epp(( exp(s*
5

---

Similar to L La is applied to all the K' iterations. Compared to Lozs in Eq. ① Let can better repel visual representations from different categories via progressive category prototypes {C* k 1, K.
With wetpsssiiiil--pdateee category prototypes, PCC improves the category discriminability of visual representations (X)] 1, K).
3.4 Overall Objective
Overall, the objective loss function of DPPN is:
Call E + XL2),
(10)
where À is the hyper-parameter balance Let. The attribute prototypes and category prototypes collaboratively trained a unified framework, which enables the final visual representation f (X)
to simultaneously capture attribute-region correspondence and category discrimination. During
inference, only visual representation f (X) at the K-th iteration used by:
j = arg min d(f (x))) NEY.Uy...
(11)
3.5 Discussion
Compared to APN 47 DPPN is a much different and novel method with three main differences: a) instead of sharing prototypes for all images in APN, DPPN dynamically adjusts attribute prototypes according to different images. Specifically, DPPN introduces attribute- clues from the target image feature into attribute prototypes. that the prototypes result in better attribute localization; b) different from APN's averagely pooling local visual features into a global one, DPPN concatenates local features represent an image, which better preserves attribute-region correspondence. The final representation of DPPN is made up of attribure-localized features during both training and inference, instead of regarding attribute localization mere supervision during training in APN; and c) DPPN further exploits progressive category prototypes to repel visual representations from different categories, which enhances category discrimination.
adapted to the target image and
4 Experiments
4.1 Experimental Settings
Datasets.
Four public GZSL benchmarks, i.e., Caltech-USCD Birds-200-2011 (CUB) 43 SUN
35). Animals with Attributes2 (AWA2) (24), and Attribute Pascal and Yahoo (aPY) 14 are adopted in this paper. CUB contains 11, 788 bird images in 200 species with 312 description attributes. SUN contains 14, 340 scene images in 717 classes with 102 attributes. AWA2 contains 37, 322 animal images in 50 classes with 85 attributes. aPY contains 15, 339 object images in 32 categories with 64 attributes.
Evaluation Metrics. The widely-used harmonic mean H = (2MCA, x (CAA)/MMAA + MCA,) used to evaluate GZSL performance. MCA, and MCA,, are the Mean Class Top-1 Accuracy for seen and unseen domains, respectively.
Implementation Details. The input images are resized to 448 x 448 following [55 45 Random cropping and flipping are used for data augmentation. ResNet-101 [18 pretrained on ImageNet [2 used the backbone. A two-step training strategy is adopted, which trains DPPN with the fixed backbone and then fine-tunes the whole network on two 1080ti GPUs. Adam optimizer [22) is used with batch size of 64 and (r 2e C' 512 since we use a conv. layer for dimension reduction after the backbone. N₂ is the number of attributes, which is 312, 85, 64, 102 for respective CUB, AWA2, aPY, and SUN. K 3 and À 1.0. N, will be discussed in the ablation study.
4.2 Ablation Study
Analysis of Attribute Localization and Category Discrinination. The core motivation of DPPN is to learn visual representations that simultancously explore category discrimination and attribute-

---

Method
La
Ca
Lu
K
MCA.
CDB
MCA.,
H
MCA.
aPY
MCA,
GFLOPs
Base-V2S
50.5
84.4
63.2
30.4
42.6
35.5
62.396
+FCC
70,2
+PAL
7
64.7
71.8
68.1
34.2
53.8
41.8
62,809
#POC&TAL
69.2
71.4
70.3
35.6
39.0
44.4
62,842
#FCC
10.2
77.1
13.3
40.0
61.2
62,500
Table I: Effect of PCC and PAL on CUB and aPY datasets. GFLOPs is calculated with input size 448 448 the CUB dataset.
13
ConoC Color
Throat Color
Tail Shape
Leg Color
Bill Color
Tail Color
Wing Shape
Figure 3: Visualization of attribute localization at different iterations. The localization gets more and
more accurate as k increases from 0 to 2.
region correspondence via the proposed PCC and PAL, thus analyse how PCC and PAL affect GZSL performance in this part. For simple comparison, we set K = in PCC and PAL. Results listed in Table [T] "Base-V2S" V2S" is the baseline method trained by Lu2s in Eq. ① "+PCC" adds PCC module to Base-V2S. which introduces stronger category discrimination constraint L..- We can observe that "+PCC" obtains 1.1% and 4.7% gains H over "Base-V20 V2S" CUB and aPY. This is because that PCC explicitly pushes representations away from different categories via category prototypes, thus enlarging category margins for more accurate category classification and boosting GZSL. The third model "+PAL" replaces f(-) in "Base-V25 with PAL, which enables visual representations with attribute localization ability. Compared with "Base- V2S". PAL module brings 4.9% and 6.3% gains on CUB and aPY. This derives from that PAL captures attribute-region correspondence by utilizing attribute prototypes localize attribute-relater local regions and produce serribute-specfiii visual representations. Finally, ++PCC&PAL simultaneously considers category discrimination and attribute localization by incorporating both PCC and PAL, which obtains 7.1% and 8.9% gains. This proves that both attribute-region correspondence and category discrimination are critical to GZSL and complementary to each other.
Notably, PAL. and PCC bring negligible additional computation, even when K = 3, because fra(-) in Eq. 66) controls the dimension of representation f²(X) to limit the computation burden. We visualize the difference of representation distribution between Base- V2S and our DPPN in the Appendix.
Effect of Progressive Prototype Updating. DPPN progressively updates attribute and category prototypes to learn more transferable and distinctive representations. Here, analyse how such progressive learning strategy impacts the attribute localization ability and category recognition by evaluating varying K in PAI. and PCC. The results are given in Fig.
As K rises from 1 3, on all the four datasets gradually increases. The best H 73.5%. 73.1%. 48.4%, and 41.0% on CUB, AWA2, aPY, and SUN is obtained when K = 3. This demonstrates that, with category and attribute prototypes updated, the visual representations become more discriminative and transferable to the unseen categories. Here, to intuitively present the attribute localization progressively learned by PAL, we visualize the attribute localization results of PAL at different iteration & when setting K = 3. As shown in Fig. 3 the localization gets more and more precise as attribute prototypes gradually update. With progressively updated attribute prototypes, the PAL module can finally accurately localize corresponding attribute-relater visual regions. Besides, with updating, the prototype for the same attribute gets more specifie to the target image the first four columns, reflecting that progressive updating can adapt prototypes according different images.
7

---

MCA MCA H
45
50
40
25
30
K
CUB
AWA2
aPY
SUN
Figure 4: Effect of progressive updating with varying K on four datasets.
PAL
PCC
CUB
aPY
farl
caf(-1
(um())
fea(-)
MCCA,
MCA,,
H
MCA.
ATCA,
H
70.2
77.1
73.5
40.0
61.2
48.4
67.4
78.2
72.4
36.3
60.9
45.5
67.0
70.3
68.7
35.6
62.0
45.2
70.4
73.0
71.7
37.3
59.4
45.8
68.3
76.0
71.9
38.4
57.1
45.9
Table Evaluation of components in DPPN.
This proves that attribute prototypes can capture the attribute-region correspondence, and progressive updating makes prototypes more specific and distinctive. When K > 3, H drops. The reason may be that the over-updated prototypes become unstable and hard to train. K = is a good trade-off between general knowledge of whole dataset and specific knowledge towards image. Thus, we set K = 3 for the rest experiments.
In summary, both quantitative and qualitative results demonstrate that progressive updating can improve attribute and category prototypes, which better captures attribute-region correspondence and category discrimination.
Evaluation of Components PAL. PAL aims to learn visual representations with accurate attribute localization. Thus, we evaluate two important components of PAL, ie., far(-) (-) in Eq. (5) and cal(.) in Eq. (6) The results are given in Table 2 improve prototype quality. As shown the second row of Table without far (.). H drops by 1.1% and 2.9% on CUB and aPY, respectively. This demonstrates that far(.) benefits attribute prototype updating, thereby boosting attribute localization ability.
far(-) is refinement function between Pk and pk+1 to
cat(.) is the aggregation function used to produce visual representation f*(X) by concatenating attribute prototypes P = {pf.... PN Compared to summing (pi. ... . } up or max pooling operation, cat(-) better preserves attribute-region correspondence. The third and fourth rows Table 2 show that H drops replacing eat(-) with either summing up max pooling. This proves that cat(-) benefits local correspondence preservation.
Effect of (.) in PCC. (.) in Eq. (8) serves as a gating function to ease the training of category prototypes. this part, we analyse the impact of f.) Eq. (8) f..()) (-) removes redundancy in Ck Without fee (.), the category prototype updating becomes C++ C W which passes all information in Ch to C++1 The results are listed in the last row of Table 2 Without f.) H drops by 1.6% and 2.5% on CUB and aPY. This proves that (-) helps to ease the training process of PCC, which learns more discriminative visual representations.
Effect of X. in Fig. 5 (a) and (b). As À rises from 0.0 to 1.0, i.e., category discrimination supervision La is introduced into DPPN, H increases on both CUB and aPY. The best H is obtained when À = 1.0. This proves the effectiveness of category discrimination brought by PCC. When À 1.0, H starts to drop. Thus, we set À = 1.0 for better results.
the hyper-parameter to balance L.,, Here, we evaluate the effect of as shown
Effect of N.. N, is the dimension of (X) and N, = D No. N₂ is the number of attributes, which are 312, 85, 64, 102 for CUB, AWA2, aPY, and SUN, respectively. Here, we set different D to evaluate the effect of N. on recognition performance and calculation addition. Fig. (c) shows values H as N, varies. 5 shows GLOPs addition over N, 512. When N,, around 2, 048, best performance is obtained with relatively small calculation complexity addition. Thus, we set N,, 2496, 21 2125, 2048, 2142 and 8, 25, 32, 21 for CUB, AWA2, aPY, and SUN, respectively.

---

MCAA MEA
MCA - -MCA
CUB aPY
[461
2025
05015
401
0
N
015
10
05 (b) Top- AAC. an
612
1004
2008
4036
312
1028
204
400
(a) Top-1 100.-AACC.---.. CUB
(c)
H- on CUB and aPY
(c)
GFLOP&AAd
Figure Effect of hyper-parameters À and N
CUB
AWA2
aPY
SUN
Methods
MCA.
MCAA,
MCA.
MCA,
H
MCA,
MCA,
MCA.
MCA,
IZF-Softmax
52.7
68.0
59.4
60.6
77.5
68.0
42.3
60.5
49.8
52.7
57.0
54.8
TE-VAEGAN 44
63.8
79.3
70.7
41.8
51,9
46.3
E-PGN 50
52.0
61.1
56.2
52.6
83.5
64.6
60.4
75.1
67.0
37.1
56.8
47.9
CE-GZSL
63.9
66.8
65.3
63.1
78.6
70.0
48.8
38.6
43.1
MLSE
22.3
71.6
34.0
23.8
83.2
37.0
12.7
74.3
21.7
20.7
30.4
26.4
COSMO
57.8
50.2
37.7
41.0
PREN[48
32,5
55.8
43,1
32.4
88.6
47.4
35.4
27.2
30.8
VSE-S
33.4
87.5
48.4
41.6
91.3
57.2
24.5
72.0
36.6
29
43.4
79.6
56,2
50.0
90.3
64.4
20.8
34,9
26.1
AREN 45
63.2
69.0
66.0
54.7
79.1
64.7
30.0
47.9
36.9
40.3
32.3
35.9
CosineSoftmax 25
47.4
47.6
47.5
56.4
81.4
66.7
26.5
74.0
36.3
42.8
39.3
RGEN 46
73.5
60.0
66.1
76.5
67.1
71.5
48.1
30.4
37.2
31.7
44.0
36.8
DAZLE
56.7
59.6
58.1
60.3
75.7
67.1
52.3
24.3
33.2
56.5
78.0
65.5
37.6
GEM-7SL,
3D
64.8
77.1
70.4
64.8
77.5
70.6
38.
35.7
36.9
DPPN
70.2
77.1
73.5
63.1
86.8
73.1
40.0
61.2
48.4
47.9
35.8
41.0
Table 3: Results of GZSL on four classification benchmarks. Our DPPN belongs to embedding-
methods (EMB.). Generative methods (GEN.) utilize extra synthetic unseen domain data for training. The best result is bolded, and the second best is underlined.
4.3 Comparison with State-of-the-Art Methods
We compare DPPN with the state-of-ther of-the-art GZSL methods, of which the results given in Table
Among existing methods, APN 47 the most related method which also utilizes visual prototypes to localize visual parts. Different from APN that counts only sharing prototypes for all images, our DPPN adjusts attribute prototypes dynamically according to the target image and exploits category prototypes to enhance category discrimination. Thus, DPPN learns specific and distinctive visual representations and surpasses APN by a large margin, i.e., 6.3%. 7.6%, and 3.4% for H on CUB, AWA2, and SUN datasets, respectively. APN does not provide codes and is approximately similar to PAL with K 1. As shown in Fig. 3 it qualitatively implies that progressive updating attribute prototypes can learn visual representations with better attribute localization ability. Besides, DAZLE 20 uses dense attribute attention to focus on relevant regions, which is inferior to DPPN by 15.4%. 6.0%. and 7.8% CUB, AWA2, and SUN, respectively. This proves the effectiveness of designing progressive attribute and category prototypes in GZSL.
Compared to other embedding- based methods, our method surpasses the best one by respectively 3.1%. 1.6%, and 9.4% for H on CUB, AWA2, aPY datasets, and obtains comparable best H performance SUN dataset. The results demonstrate that progressively exploring attribute-region correspondence and category discrimination can effectively enhance cross-domain transfcrability and category discriminability of visual representations. The reason for relatively small improvement SUN may be that #categories is large while #images each category is small in SUN, leading to difficulties for DPPN to learn accurate attribute localization and category discrimination.
Compared to generative methods that utilize additional unseen category labels during training, DPPN can achieve comparable, even better results, especially on CUB and AWA2 datasets. This reveals that with the assistance progressive attribute localization and category discrimination, DPPN can surpass generative methods without complex GAN training.

---

Conclusion
In this paper, we propose a Dual Progressive Prototype Network (DPPN) to progressively explore both attribute-region correspondence and category discrimination for GZSL. Specifically, DPPN constructs progressive prototypes for both attributes and categories. DPPN alternativeyy localizes attribute- related visual regions and adjusts attribute prototypes towards target images, which improves attribute localization ability and cross-domain transferability of visual representations. Along with progressive attribute prototypes, DPPN progressively projects category prototypes to multiple spaces to enforce visual representations away from different categories, thus enhancing category discriminability. Extensive experimental results on four public datasets demonstrate the effectiveness of our DPPN.
Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation of China (NSFC) under Grants 61632006 and 62076230.
References
[1] Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(7), 1425-1438 (2016)
[2] Akata, Z., Reed, S., Walter, D., Lee, H., Schiele, B.: Evaluation of output embeddings for fine-grate mne-grained image classification, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2927-2936 (2015)
[3] Annadani, Y., Biswas, S.: Preserving semantic relations for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7603-7612 (2018)
[4] Atzmon, Y., Chechik, G.: Adaptive confidence smoothing for generalized zero-shot learning. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
1 11671-1 1680 (2019)
[5] Ba, J.L., Swersky, K., Fidler, S., salakhutdinov, R.: Predicting deep zero-shot convolutional neural networks using textual descriptions. In: Proceedings of the IEEE International Conference on Computer Vision. pp.
(2015)
[6] Bucher, M., Herbin, S., Jurie, F:: Improving semantic embedding consistency by metric learning for zero-sho shot classiffication. In: Proceedings of the European Conference Computer Vision. pp. 730-746 (2016)
[7] Changpinyo, S., Chao, W.L., Gong, B., Sha, F: Synthesized classifiers for zero-shot learning. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5327-5336 (2016)
[8] Chao, W.L., Changpinyo, S., Gong, B., Sha, F: An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In: Proceedings of the European Conference on Computer Vision. pp. 62-68 (2016)
[9] Chen, L., Zhang, H., Xiao, J., Liu, W., Chang, S.F.: Zero-shot visual recognition using semantics preserving adversarial embedding network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1043-1052 (2018)
[10] Chen, X., Lan, X., Sun, F., Zheng, N.: A boundary based out-of-distribution classifier for generalized zero-shot learning. In: Proceedings of the European Conference Computer Vision. pp. 572-588 (2020)
[11] Chou, Y.Y., Lin, H.T., Liu, T.L.: Adaptive and generative zero-shot learning. In: International Conference Learning Representations (2021)
[12] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 248-255 (2009)
10

---

[13] Ding, Z., Liu, H.: Marginalized latent semantic encoder for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6191-6199 (2019)
[14] Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778- 1785 (2009)
[15] Ferrari, V., Zisserman, A.: Learning visual attributes. Advances Neural Information Process- ing Systems 20, 433 440 (2007)
[16] Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T., et al.: Devise: A deep visual-semantic embedding model. In: Advances in Neural Information Processing Systems. pp. 2121-2129 (2013)
[17] Han, 7., Fu, 7.,, Chen, S., Yang, J.: Contrastive embedding for generalized zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2371-2381 (2021)
[18] He, K., Zhang, X., Ren, S., Sun, J:: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 770-778 (2016)
[19] Huang, H., Wang, C., Yu, P.S., Wang, C.D.: Generative dual adversarial network for generalized zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 801-8 201-810(( (2019)
[20] Huynh, D., Elhamifar, E.: Fine-grained generalized zero-shot learning via dense sttribute-based attention. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 4483-44999 (2020)
[21] Jiang. H., Wang, R., Shan, S., Chen, X.: Transferable contrastive network for generalized zero-shot learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 9765-9774 (2019)
[22] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (2015)
[23] Kodirov, E., Xiang. T., Gong, S.: Semantic autoencoder for zero-shot learning. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3174-3183 (2017)
[24] Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 951-958 (2009)
[25] Li, K., Min, M.R., Fu, Y.: Rethinking zero-shot learning: A conditional visual classification perspective. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3583-3599 (2019)
[26] Li. Y., Wang, D., Hu, H., Lin, Y., Zhuang, Y.: Zero-shot recognition using dual visual-semantic mapping paths. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 3279-328 (2017)
[27] Liu, L., Zhou, T., Long, G., Jiang, J., Dong. X.. Zhang, C.: Isometric propagation network for generalized zero-shot learning. In: International Conference on Learning Representations (2021)
[28] Liu, S., Long, M., Wang, J., Jordan, M.I.: Generalized zero-shot learning with deep calibration network. In: Advances Neural Information Processing Systems. pp. 2005-2015 (2018)
[29] Liu, Y., Guo, J., Cai, D., He, X.: Attribute attention for semantic disambiguation in zero- shot learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp- 6698-6709 (2019)
11

---

[30] Liu, Y., Zhou, L., Bai, X., Huang. Y. Gu, L.. Zhou, J., Harada, T.: Gioal-oriente gaze estimation for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3794-3806 (2021)
[31] Min, S., Yao, H., Xie, H., Wang, C., Zha, ZJ., Zhang, Y.: Domain-aware visual bias eliminating for generalized zero-shot learning. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12664-1222 1264-1263332 (2020)
[32] Min, S., Yao, H.. Xie, H., Zha, Z.J., Zhang, Y.: Domain-specific embedding network for zero-shot recognition. In: Proceedings of ACM International Conference on Multimedia, pp. 2070-2078 (2019)
[33] Min, S., Yao, H., Xie, H., Zha, Z.J., Zhang, Y.: Domain-orienter semantic embedding for zero-shot learning. IEEE Transactions on Multimedia 23, 3919-39999999 --3330 (2020)
[34] Narayan, S., Gupta, A., Khan, F.S., Snock, C.G., Shao. L.: Latent embedding feedback and discriminative features for zero-shot classification. In: Proceedings of the European Conference on Computer Vision. pp. 1-23 (2020)
[35] Patterson, G., Hays, J.: Sun attribute database: Discovering, annotating, and recognizing scene attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2751-2758 (2012)
[36] Rahman, S., Khan, S., Porikli, F.: A unified approach for conventional zero-shot, generalized zero-shot, and few-shot learning. IEEE Transactions on Image Processing 27, 5652-5667 (2018)
[37] Reed, S., Akata, Z., Lee, H., Schiele, B.: Learning deep representations of fine-grained visual descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 49-58 (2016)
[38] Romera-Paredes, B., Torr, P.: An embarrassingly simple approach to zero-shot learning. In: International Conference on Machine Learning. pp. 2152-2161 (2015)
[39] Shen, Y., Qin, J., Huang, L.: Invertible zero-shot recognition flows. In: Proceedings of the European Conference on Computer Vision. pp. 614-631 (2020)
[40] Shigeto, Y., Suzuki, I., Hara, K., Shimbo, M., Matsumoto, Y.: Ridge regression, hubness, and zero-shot learning. In: Joint European Conference on Machine Learning and Knowledge Discovery Databases. pp. 135-151 (2015)
[41] Socher, R., Ganjoo, M., Manning, C.D., Ng, A.: Zero-shot learning through cross- moosl transfer. In: Advances in Neural Information Processing Systems, pp. 935-943 (2013)
[42] Tong, B., Wang, C., Klinkigt, M., Kobayashi, Y., Nonaka, Y.: Hierarchical disentanglement of
discriminative latent features for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11467-11476 (2019)
[43] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-uest birds-200-22 dataset. In: California Institute of Technology. pp. 1-8 (2011)
[44] Xian, Y., Akata, Z., Sharma, G., Nguyen, Q. Hein, M., Schiele, B.: Latent embeddings for zero-shot classification. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition. pp. 69-77 (2016)
[45] Xie, G.S., Liu, L.,, Jin, X., Zhu, F., Zhang, Z., Qin, J., Yao, Y., Shao, L.: Attentive region bedding network for zero-shot learning. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9384 9393 (2019)
[46] Xie, G.S., Liu, L., Zhu, F., Zhao, F., Zhang, Z., Yao, Y., Qin, J., Shao, L.: Region graph embedding network for zero-shot learning. In: Proceedings of the European Conference on Computer Vision. pp. 562-580 (2020)
[47] Xu, W., Xian, Y., Wang, J., Schiele, B., Akata, Z.: Attribute prototype network for zero-shot learning, In: Advances in Neural Information Processing Systems. pp. 21969-2111111 22980 (2020)
12

---

[48] Ye, M., Guo, Y.: Progressive ensemble networks for zero-shot recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11728-11723 (2019)
[49] Yu, Y., Ji, Z., Fu, Y., Guo, J., Pang, Y., Zhang, Z. Stacked semantic-guided attention model for fine-grained zero-shot learning. In: Advances in Neural Information Processing Systems, 5998 6007 (2018)
[50] Yu, Y., Ji, Z., Han, J., Zhang, Z.: Episode-based prototype generating network for zero-shot learning. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 14035-14044 (2020)
[51] Yue, Z., Wang. T., Zhang, H., Sun, Q. Hua, X.S.: Counterfactual zero-shot and open-set visual recognition. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 16004-1414 (2021)
[52] Zhang, H., Long, Y., Guan, Y., Shao, L..: Triple verification network for generalized zero-shot learning. IEEE Transactions on Image Processing 28(1), 506-517 (2019)
[53] Zhang, H., Koniusz, P.: Zero-shot kernel learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7670-7679 (2018)
[54] Zhang, L., Xiang, T., Gong, S.: Learning a deep embedding model for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2021-2030 (2017)
[55] Zhu, P., Wang, H., Saligrama, V.: Generalized zero-shot recognition based on visually se- mantic embedding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2995-3003 (2019)
13


Reviews:
[{"review_id": "gzW3snEqZSb", "reviewer": "Reviewer_8tZA", "summary": "This paper presents a Dual Progressive Prototype Network,  which constructs two types of prototypes that record prototypical visual patterns for attributes and categories, respectively. It aims to search attribute-related local areas, and updates attribute prototypes to explore accurate attribute-region correspondence.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method does appear to bring some significant novelty with respect to the literature.\nThe classification experiments prove the efficiency of the proposed method.\nThe ablation study is sufficient.\nThe code is provided.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Evaluation:\nFig.2 is not quite clarifying for the method put forward in the paper.\nHow to set the hyper-parameters? What are the values of the parameters in section 4.3?\nIn Fig. 4,   the proposed method obtained the best results when k was set as 3 in some datasets. Is there any analysis of it? Why not the results get better with K increasing?\nAre the K in PAC and PCC the same?\nHow do you get the comparison results in Table 3? Did you run the code by yourself or just record them from the papers? \n\nPresentation:\nThe presentation of the proposed method requires significant improvements. Also, very long sentences should be avoided.\nThe description of the method is incomplete and hard to understand. If the available space is not sufficient for a complete presentation of the model, it is better to move some existing subsections to a supplementary material (e.g. ablation study part).\nAlso, the proposed additions to existing APN should be better motivated (what problem they solve? what is the insight?) and clearly described. \n\nMinor corrections (non-exhaustive list):\nLine 42 “dynamically adjust” should be “dynamically adjusts”\nLine85 “complementary” should be “complimentary”", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "RyWsyYTGdp", "reviewer": "Reviewer_fpHH", "summary": "This paper describes a dual progressive prototype network (DPPN) for generalized zero-shot learning.  The attribute prototypes are learned to yield visual representations with attribute localizations, while the resulting features can be used to improve the category prototypes. The two mechanisms are coupled to progressively carry out and collaboratively achieve good GZSL performance. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of the proposed method is reasonable and partly motivated by the related work, APN [47]. The proposed DPPN comprises two main modules. The first module (Progressive Attribute Localization, PAL) includes progressively trainable attribute prototypes, which are used to yield the visual representation for inference. The second is the Progressive Category Classification (PCC) module, which can be used to enhance the representation of the category prototypes. My detailed comments are as follows.\n\n• In comparison with APN [47], the proposed method enriches the technique with the PCC module and the generalization of the progressive mechanism of K iterations. From the experimental results and the brief discussion in line 255, the classification performance would degrade for K>3. Could the authors provide more comprehensive explanations to this phenomenon?\n\n• Suppose DPPN is trained with K=3. Then, (from line 156) it is expected to use f^3(X) as the visual representation in inference.  Does this strategy always give better classification performance than simply using f^1(X) as the representation in inference since the network parameters have already been updated in the “progressive” training process of PAL and PCC?\n\n• Some of the related work is not included. For example, the following paper from ICLR 2021, “Adaptive and Generative Zero-Shot Learning,” which should be included in the experiment comparisons.\n  \n• In line 154, it says that the semantic projection function g is implemented by FC for “redundancy reduction.” What are the justifications behind this statement? In particular, g is a mapping from R^{N_a} to R^{N_v}, where N_v > N_a and N_v = D \\times N_a. \n\n• Minor mistakes: line 148, the subscript should start from 1 (not 0). In Table 3, the results for the SUN dataset by TF-VAEGAN [34] are incorrect (those are based on transductive).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In the revision, the authors are suggested to concisely describe the limitations and potential negative social impacts, which may arise from the particular work.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not find anything serious, required the need of ethics review.\n\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "HHKP7oWHto2", "reviewer": "Reviewer_hqqu", "summary": "The authors propose a dual progressive prototype network (DPPN) for the generalized zero shot learning task, and they argue that the network can produce the attribute-based visual representation with category-specific features. The DPPN has two different modules. The Progressive Attribute Localization module was trained iteratively, and its final K-th output is mainly applied as adjusted visual representation. The Progressive Category Classification module is used to enhance the feature maps through contrastive learning.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is novel compared to previous works, and achieves sufficient improvements on four classification datasets and SOTA on three of four datasets.  The paper is clearly structured and well written in language. The experiments and ablation studies are clear, but some of them can be more complete and comparable.\n\n1. Compared to the most related APN method, the novelty of DPPN is located in the iterative learning (dynamic update for inference) of attribute prototypes, regarding the main difference between Eq.(2) and Eq.(7), also between Eq.(3) and Eq.(11).\n2. The descriptions of the update process of the L_sa and L_cl loss in L155 and L182 are not clear enough, how to integrate the loss of k-th iteration from K, since the progressive processing is claimed as a major contribution. \n3. The discussion in Sec.3.5 is not sufficient, since attribute prototypes in PAL are also defined as shared prototypes, though adjusted by K times iterations for specific input images. Or explain more about the contribution of â€œdynamically adjust attribute prototypes according to different imagesâ€. The training process in DPPN is declared as more compact than in APN, but PAL and PCC require multi-stage training (mentioned in L182). The end-to-end training of DPPN (in L60) is not consistent with the two-stage training (L213).\n4. Only the lambda settings of the CUB and APY datasets are discussed between L278-L282. Whether the default lambda values of AWA2 and SUN are set as 0.1, which was mentioned in L216 before, but 0.1 is excluded in Fig.5(a)(b). Or why are the other two datasets excluded from this discussion? The same question for the N_v discussion.\n5. The abbreviation definition of v2s is not mentioned in the text. There is less discussion (only ResNet 101 was mentioned in Sec.4.1 ) of the backbone in the CNN encoder. How about applying other backbones?\n6. All four datasets are from the classification task, how about an experiment on a dense prediction task, such as semantic segmentation or instance segmentation. \n7. The comparison of generative methods is insufficient, since the improvement on SUN is not exactly the same as the other three datasets in Table 3. It would be better to test how the features from DPPN can be applied in a generative method, and what results. For example, the setting of APN in VAEGAN or other GAN-based methods. The comparison can be more comprehensive, if the evaluation of DPPN in Zero-shot Learning Metrics is conducted.\n8. Adam as the optimizer mentioned in L214 should be cited as well.\n9. Visualization of visual features shown in Fig.1 of Supplementary has only unseen categories, what about the seen categories? How about the visualization comparison with previous works.\n\n- \nWe have carefully read the rebuttal by the authors and the reviews from the other reviewers.\n\nIn general, the author made clear responses to the unclear issues of the paper. In the discussion on the two hyperparameters (lambda and N_v), the test results of the other two benchmarks have been added, making the discussion relatively more comprehensive. However, two questions are unclear. The first one is about the training process, where the whole ResNet101 is retained between multiple stages, and how it will affect the batch size regarding various K settings in the same machine. Another problem is that the discussion about different backbones is insufficient. The results of adding other recent backbones are expected instead of a ResNet50.\n\nOverall, we will thus keep our initial rating.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitation is not included, and we suggest the authors can address the limitation and potential improvement of this work, or some future work for the next step.\nThe broader impact was mentioned in the supplementary.\n", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "ca 9 hours"}, "extra_text": {}}, {"review_id": "3ZRwNGyc_jl", "reviewer": "Reviewer_FH4n", "summary": "This paper proposes a dual prototype network for generalized zero-shot learning (GZSL). The basic idea is to extend the existing framework, i.e., APN, to update the attribute localization function progressively. Beside, an additional progressive category classification (PCC) module is developed to achieve more discriminative classification results by repelling visual representations of different categories. Experimental results on four popular benchmarks demonstrate the effectiveness of the proposed modules.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the idea/contribution of this paper is a bit incremental as it originates from a previous work that was the first to propose the concept of attribute prototypes. However, the extension to/the difference from [47] has been clearly stated and adequately justified, which can be found in the extensive ablation study. The presentation is also very clear and easy to follow.\n\nSome detailed comments are as follows:\n\n1. L154: g(.) is a semantic projection function for redundancy reduction. Does this function influence the performance significantly? What if we discard it? How will the final performance change?\n\n2. It seems that the proposed method is not specifically designed for GZSL and can also be used for the conventional zero-shot learning setting (CZSL). Have you tried to evaluate the method in terms of CZSL?\n\n3. L156-157: only the final K-th representation is used for inference. Have you tried to use the summation or concatenation of all the K representations as the final one?\n\n4. Since the PAL module learns/updates attribute prototypes based on each image, will the order of the images fed into the network affect the final performance? How will it affect or why won't it affect?\n\n5. The performance on aPY is exceptionally higher than the current state of the art. How to explain this?\n\nTypo:\nL148: $p_0^{k}$ should be $p_1^{k}$.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has four reviewers: two for borderline accept, two for borderline reject. Despite mixed scores, the reviewers have very thorough discussion and recommendation to further improve the papers, particularly, including,  futher clarifying the training process, experimental settings, and experimental analysis.  Some more ablation experiments are all essential to validate the contributions. 

The authours make very hard jobs and try to clarify the comments from reviewers. This is much appreciated. Unfortunately, there are still several things unclear, and the reviewers think they are very important for the acceptance. Specifically, (1) The traing stage is unclear. As ResNet-101 is retained between multiple stages; how it will affect the batch size regarding various K settings in the same machine. Additionally, note that ResNet101 is pre-trained on ImageNet, which may have overlapped images with the benchmark datasets, and potentially an issue of breaking the ZSL setting. I personally guess this is one reason why the improvement on aPY is exceptionally higher as commented by the reviewer; (2) Discussion about different backbones is insufficient; adding more recent backbones are expected. This can better validate the contributions claimed in this paper. Due to these reasons, we would expect the authours to integrate these comments, and further improve this paper for the future top  conference.
Decision: Accept (Poster)