DSP: Dynamic Semantic Prototype for Generative Zero-Shot Learning
Authors: Shiming Chen
Hou Wen Jin
Ziming Hong
Yibing Song
Tongliang Liu
Xinge You
Kun Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Zero-Shot Learning
Generative Model
Knowledge Transfer

Abstract: Generative models (e.g., generative adversarial network (GAN)) have advanced zero-shot learning (ZSL). Studies on the generative ZSL methods typically produce visual features of unseen classes to mitigate the issue of lacking unseen samples based on the predefined class semantic prototypes. As these empirically designed prototypes are not able to faithfully represent the actual semantic prototypes of visual features (i.e., visual prototypes), existing methods limit their ability to synthesize visual features that accurately represent real features and prototypes. We formulate this phenomenon as a visual-semantic domain shift problem. It prevents the generative models from further improving the ZSL performance. In this paper, we propose a dynamic semantic prototype learning (DSP) method to align the empirical and actual semantic prototypes for synthesizing accurate visual features. The alignment is conducted by jointly refining semantic prototypes and visual features so that the generator synthesizes visual features which are close to the real ones. We utilize a visual$\rightarrow$semantic mapping network (V2SM) to map both the synthesized and real features into the class semantic space. The V2SM benefits the generator to synthesize visual representations with rich semantics. The real/synthesized visual features supervise our visual-oriented semantic prototype evolving network (VOPE) where the predefined class semantic prototypes are iteratively evolved to become dynamic semantic prototypes. Such prototypes are then fed back to the generative network as conditional supervision. Finally, we enhance visual features by fusing the evolved semantic prototypes into their corresponding visual features. Our extensive experiments on three benchmark datasets show that our DSP improves existing generative ZSL methods, \textit{e.g.}, the average improvements of the harmonic mean over four baselines (e.g., CLSWGAN, f-VAEGAN, TF-VAEGAN and FREE) by 8.5\%, 8.0\% and 9.7\% on CUB, SUN and AWA2, respectively.

Under review as a conference paper at ICLR 2023
DSP: DYNAMIC SEMANTIC PROTOTYPE FOR GENERA- TIVE ZERO-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Conditional generative models (e.g., generative adversarial network (GAN)) have advanced zero-shot learning (ZSL). Studies on the generative ZSL methods typi- cally produce class-specific visual features of unseen classes to mitigate the issue of lacking unseen samples based on the predefined class semantic prototypes. As these empirically predefined prototypes are not able to faithfully represent the actual semantic prototypes of visual features (i.e., visual prototypes), existing methods limit their ability to synthesize visual features that accurately represent real features and prototypes. We formulate this phenomenon visual-semantic domain shift problem. It prevents the conditional generative models from further improving the ZSL performance. In this paper, propose dynamic seman- tic prototype learning (DSP) method to align the empirical and actual semantic prototypes for synthesizing accurate visual features. The alignment conducted by jointly refining semantic prototypes and visual features so that the conditional generator synthesizes visual features which are close the real ones. We utilize a visual->semante mapping network (V2SM) to map both the synthesized and real features into the class semantic space. The V2SM benefits the generator to synthe- size visual representations with rich semantics. The visual features supervise our visual-oriented semantic prototype evolving network (VOPE) where the predefined class semantic prototypes are iteratively evolved become dynamic semantic pro- totypes. Such prototypes are then fed back to the generative network as conditional supervision. Finally, enhance visual features by fusing the evolved semantic prototypes into their corresponding visual features. Our extensive experiments on three benchmark datasets show that DSP improves existing generative ZSL methods, e.g., the average improvements of the harmonic mean over four baselines (e.g.. CLSWGAN, f-VAEGAN, TF- VAEGAN and FREE) by 8.5%, 8.0% and 9.7% on CUB, SUN and AWA2, respectively.
INTRODUCTION
Zero-shot learning (ZSL) recognizes the unseen classes, by transferring semantic knowledge from some known classes unknown ones. Recently, conditional generative models (e.g., generative adversarial networks (GANs) Goodfellow et al. (2014) and variational autoencoder (VAE) Kingma & Welling (2014)) have been successfully applied ZSL and achieved promising performance. They synthesize the class-specific images or visual features of unseen classes to mitigate the lack of unseen samples based on the condition of the predefined class semantic prototypes Arora et al. (2018); Xian et al. (2018; 2019b); Chen et al. (2021a;b) Because of the high quality synthesis of GAN, studies arise to generate unseen samples the CNN features to benefit ZSL Yan al. (2021); Xian al. (2019b).
The class-specific unseen sample generations are conditioned on the semantic prototypes. However, there is limitation that the empirically predefined semantic prototypes are not able to faithfully represent the actual semantic prototypes of visual features (i.e., visual prototypes). As shown in Fig. 1, i) the predefined class semantic prototypes annotated by human, inevitably resulting in some inaccurate annotations (e.g., the attribute "bill color orange" in Sample-2 in Fig. 1(a)), and ii) the visual images are with multi-views, resulting in the phenomenon that some importantly annotated attributes do not appear in the visual representations (e.g., the attribute "bill color orange" in Sample-3 and attribute "leg color black" in S in Fig. (aa). That is, some visual representations of one

---

Under review as a conference paper at ICLR 2023
Real Sample of Class-l
Predefined Semantic Prototype
Bill Color Orange
Real Sample of Classs-2
0.8
Leg Color Black
Synthesized Sample of Class-2 + Predefined Semantic Prototype of Class-2
0.6
Upper Part Color Gray
(b) Visual Features Learned by Existing Generative ZSL Methods
0.2
Breast Color White
Visual Prototype of Class-2
0.1
Bill Shape Dagger
Dynamic Semantic Prototype of Class-2
(a) The Mapping Between Visual Representations and the Predefined Class Semantic Prototype
(c) Visual Features Learned by Our DSP
Decision Boundary
Figure Motivation illustration. (a) The fact that some visual representations of one class are incorrectly mapped with their common predefined semantic prototype. (b) Existing generative ZSL methods merely generate visual features guided by the predefined class semantic vectors that fail to accurately represent the semantic prototypes of visual representations (also denoted visual prototypes), resulting in the synthesized visual features far away from the corresponding real visual features and real semantic prototypes. (c) Our DSP dynamically evolves the predefined semantic prototypes as dynamic semantic prototypes that are closer to their visual prototype, enabling the generator to synthesize reliable visual features and enrich semantic information into visual features with the dynamic semantic prototypes. (Best viewed in color)
class are incorrectly mapped with their common predefined semantic prototype. As such, the visual features synthesized by existing generative ZSL methods are far away from their corresponding real visual features and visual prototypes (as shown in Fig. 1(b)), which heavily limits their classification problem of Visual-Semantic Domain Shift. There-
performance. We formulate this phenomenon fore, it is essential to refine the empirical defined semantic prototype. Using an accurate prototype benefits the generator supervision to improve the generative ZSL performance.
In light of these observations, we argue that the predefined semantic prototypes can be refined to become consistent to the actual semantic prototypes (i.e., visual prototypes) according to the visual information. As such, the refined semantic prototypes can capture the visual samples exactly and act as an accurate conditional signal for the generator, enabling generative ZSL methods learn a desirable semantic-vissuu mapping (i.e., generator). Targeting this goal, in this paper, we propose a dynamic semantic prototype learning (DSP) method to align the empirical and actual semantic prototypes for synthesizing reliable visual features. The alignment is conducted by jointly refining semantic prototypes and visual features so that the conditional generator synthesizes class-specific visual features which are close to the real ones, shown in Fig. (c)). Accordingly, our DSP well tackles the
domain shift problem and advances the generative ZSL methods.
Specifically, DSP consists of risual-rsemanti mapping network (V2SM) and visual-oriented semantic prototype evolving network (VOPE). Cooperated with the generator, V2SM maps visual features into the class semantic space, enabling the conditional generator synthesize class-specific visual samples with rich semantic information and convey visual information into the VOPE. Under the supervision of visual information, VOPE iteratively evolves and refines the predefined class semantic prototypes to become dynamic semantic prototypes, which are fed back to the generative network as a conditional supervision encourage the generator for reliable visual feature synthesis. Finally, DSP concatenates the evolved semantic prototypes into visual features for enhancement, enabling the visual features to be closer to their corresponding semantic prototypes and alleviate the cross-dataset bias problem Chen et al. (2021a). Extensive experimental results demonstrate consistent performance gains over the state-of-the-art generative ZSL methods on three challenging benchmark datasets, i.e., CUB Welinder et al. (2010), SUN Patterson & Hays (2012) and AWA2 Xian et al. (2019a). For example, the average improvements of harmonic mean over four baselines (e.g., CLSWGAN Xian et al. (2018), F-VAAGAN Xian et al. (2019b), VAEGAN Narayan et al. (2020) and FREE Chen et al. (2021a)) are 8.5 % 8.0 % and 9.7% on CUB, SUN and AWA2, respectively. Notably, our DSP is flexible and effective to be entailed in any generative ZSI. method.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Emmedddingbaasee Zero-Shot Learning. Embedding- -ased ZSL methods were popular early, they learn a visual -semantic projection on seen classes that is further transferred to unseen classes Akata et al. (2016; 2015); Huynh & Elhamifar (2020a); Chen et al. (2022): Huynh & Elhamifar (2020b), Considering the cross dataset bias between ImageNet and ZSL benchmarks Chen et al. (2021a), embedding- methods were recently proposed focus on learning the region-based visual features to enhance the holistic visual features' using attention mechanism Yu et al. (2018); Zhu et al. (2019); Xie al. (2019); Huynh & Elhamifar (2020a); Xu al. (2020). However, these methods depends on the attribute features for attribute localization. As such, Wang et al. (2021) introduces DPPN to iteratively enhance visual features using the category/visual prototype as supervision. Differently, we jointly and mutually refine the visual features and semantic prototype, enabling significant visual-semantic interactions in generative ZSL.
Generative Zero-Shot Learning. Since umbedding-baseed methods learn the 7SL. classifier only on seen classes, inevitably resulting in the models overfitting seen classes. To tackle this challenge, generative ZSL methods employ the generative models (e.g., VAE and GAN) to generate the unseen visual features for data augmentation Arora et al. (2018); Xian et al. (2018); Shen et al. (2020), ZSL is then converted to a supervised classification task further. As such, the generative ZSL methods have shown significant performance and become very popular recently. However, there are lots of challenges that need to be tackled in generative ZSL. To improve the optimization process of ZSL. methods, Skorokhodov & Elhoseiny (2021) introduces class normalization for the ZSL. task. Çetin al. (2022) introduces closed-form sample probing for generative ZSL. To avoid ZSL. models overfitting to seen classes, Liu et al. (2021) introduces an isometric propagation network to strengthen the relation between classes within the visual and semantic spaces. Chou et al. (2021) designs a generative scheme to simultaneously generate virtual class labels and their visual features. To enhance the visual feature generative ZSL methods, Felix et al. (2018), Narayan al. (2020) and Chen et al. (2021a) refine the holistic visual features by learning a visual -semanti projection, which encourages conditional generator to synthesize visual features with more semantic information. Orthogonal to these methods, we propose novel dynamic semantic prototype learning method to tackle the visual-semantic domain shift problem and advance generative ZSL further.
Visual-Semantic Domain Shift vs Projection Domain Shift. ZSL learns projection between visual and semantic feature space on seen classes, and then the projection is transferred to the unseen classes for classification. However, data samples of the seen classes (source domain) and unseen classes (target domain) are disjoint, unrelated for some classes, and their distributions can be different, resulting in a large domain shift Fu et al. (2015); Wan et al. (2019); Pourpanah et al. (2022). This challenge is the well-known problem of projection domain shift. To tackle this challenge, induetive-based methods incorporate additional constraints or information from the seen classes Zhang et al. (2019); Jia et al. (2020); Wang et al. (2021). Besides that, several transductive- methods have been developed
alleviate the projection domain shift problem Fu et al. (2015); Xu et al. (2017); Wan et al. (2019). We should note that the visual-semante the projection domain shift. Visual-semante domain shift denotes that the two domains (visual and semantic domains) share the same categories and only the joint distribution of input and output differs between the training and test stages. In contrast, the projection domain shift can be directly observed in terms of the projection shift, rather than the feature distribution shift Pourpanah et al. (2022). As shown in Fig. 1. we demonstrate that iisual-semantic domain shift is bottleneck challenge in generative ZSL and is neglected by existing methods.
domain shift is different from
DYNAMIC SEMANTIC PROTOTYPE LEARNING
Problem Setting: The problem setting of ZSL is formulated as follows. Assume that we have seen class data D = (xff, } with C* seen classes, where of E X denotes the i-th visual feature with 2048-dim extracted from a CNN backbone (e.g., ResNet101 He et al. (2016)), and yf e yds is the corresponding class label. Note that D' is divided into training set Di, and test set Di, according to Xian et al. (2019a). Another set of unseen classes C" is Di = {(x," v("),, where x" e X' are the visual features of unseen classes, and y!" )² are the corresponding labels. A set of class semantic vectors (semantic values annotated by humans according to attributes) of the class e E C C with
Holistic visual features directly extract from CNN backbone (e.g., ResNet101 He et al. (2016) pretrained on ImageNet.
3

---

Under review as a conference paper at ICLR 2023
T
SSeaat-Semantic Mapping Network
L Seye
Conditional Generative
V228
L
Gaussian Noise
(G)
L S25
Z k
Vector 0
Visual-Oriented Semantic Prototype Evolving Network
2
2x:
Training Phase
Evolved with 2x
C222 + (1 a) 2
T 8/u
Inference Phase
D 8/1
fioh crow X
coupion torn X
2 8/u
lasan X
Visual-Oriented Semantic Prototype Evolving Network
Classifier
elegant tern
T syn
eared eaebe X
Y 2
L 2 syna
Conditional Generative Model
(G)
Figure 2: Framework the dynamic semantic prototype learning (DSP). Traning Phase (Top): DSP includes visual- mapping network (V2SM) and visual-oriented semantic prototype evolving network (VOPE). V2SM maps visual features into the class semantic space, enabling the conditional generator synthesizes visual samples with rich semantic information and conducts visual information into the VOPE. VOPE iteratively evolves the predefined class semantic prototype under the supervision visual information. Our DSP is jointly trained with the generative model. Inference Phase (Bottom):DSP extracts the evolved semantic prototypes g*/M for seen/unseen classes using the pretrained VOPE, and take the dynamic semantic prototypes Tn = az" + (1 a)=² as input of the conditional generator synthesize class-specific visual feature samples # for unseen classes.
Then, DSP concatenates visual features with their corresponding z*/u to enhance visual feature representations, which are utilized for classification.
|A] attributes, denoted as = === - (CZSL) setting, aim to learn classifier for unseen classes, i.e., foZsl X y contrast, we aim to learn classifier for seen and unseen classes in the generalized zero-shot learning (GZSL)
(A)] E Rl In the conventional zero-shot learning
setting, i.e., faZst
Y Y.S.
In the following, we first briefly introduce the generative ZSL model. Then, we specifically demon- strate our dynamic semantic prototype learning method, its framework is shown in Fig. 2).
3.1 GENERATIVE ZSL MODEL
In our work, we focus on generative ZSL Xian et (2018; 2019b). The main goal of generative ZSL
learn conditional generator G: 2 x O - X, which takes the class semantic vectors/prototypes E 2 and Gaussian noise vectors o E O inputs, and synthesizes the class-specific visual feature samples f E X. Once such conditional generative model is learned, we can take it to synthesizes large number class-specific visual feature samples for unseen classes based on the semantic vector z", and the final classifier over y can be obtained using any standard supervised classifier (e.g., softmax).
3.2 VISUAL-SEMANTIC MAPPING NETWORK
To encourage the conditional generator (G) to synthesize class-specific visual samples with enriched semantic information and conduct visual information for evolving the semantic prototype. design a -Semanne Mapping Network (V2SM) that is entailed on the of the conditional generative model. V2SM is multilayer perceptron (MLP), which has residual block avoid too much
4

---

Under review as a conference paper at ICLR 2023
information loss when mapping visual features of 2048-dim to semantic features of |A|- The network details are presented in Appendix A. Specifically, V2SM maps the real/synthesized visual features z/2 as semantic vectors Eyeat Ssgns respectively, formulated as: &real (2) or 2ngn V2SM($),
where Sreal Sayn 2. and 2 E RIAI We can find that the conditional generator (i.e., semantic- mapping) and V2SM conduct a semantic cycle network, i.e., 2 - G(o,, & and è + (2)M(#) = 2, where o is a random Gaussian noise vector with |A -dim. As such, V2SM encourages the conditional generator to synthesize visual features with more semantic information. To enable V2SM to map visual features into semantic space effectively, we employ the semantic cycle-consistency loss (Lesyye) during training. This loss can be written as:
Sicyc E [|] &reat =* [I]l] + E [ll 2 12ll11
where 2 is the dynamic semantic prototype learned by VOPE at the k-th step, which will be introduced in the next subsection. This is different from TF-VAEGAN Narayan et al. (2020) and FREE Chen et al. (2021a) that use the predefined semantic prototypes as semantic supervision in their semantic decoders. Since the dynamic semantic prototype is more close to its corresponding visual prototype, provides more accurate supervision signals for the V2SM and generative models. As such, DSP can significantly improve the performance of TF- VAEGAN and FREE.
We find that 2, 8 and žg are closely affected by each other. = (2SM(z) is mapped from the visual features in V2SM and used to convey visual signal to supervise VOPE, enabling to be progressively evolved as dynamic semantic prototype 2 = az + (1 - a:) VOPE(= (2x = z when k=0). As such, 2g IS closer to its corresponding visual prototype, and serves the accurate condition information that encourages generator to synthesize reliable visual features.
3.3 VISUAL-ORIENTED SEMANTIC PROTOTYPE EVOLVING NETWORK
We further introduce visual-oriented semantic prototype evolving network (VOPE) to refine the semantic prototype under the supervision of visual information. As such, the dynamic semantic prototype can be closer to its corresponding visual prototype and the visual-semantic domain shift can be alleviated, as shown in Fig. I(c). Considering that the predefined semantic prototypes have rich accurate information annotated by humans, we progressively update it progressively based on the visual information guidance. Targeting this goal, VOPE is designed as an MLP network with a residual block that acts as a routing gate mechanism implemented by channel attention. which evolves the specific attribute value of semantic prototypes according to the corresponding visual information. The network details are presented in Appendix B.
Specifically, VOPE iteratively evolves the initialized semantic prototype 25 (2t is the predefined semantic prototype z annotated by humans when k=0) to another semantic prototype space ă, formulated as:
E VOPH(2k)
To encourages ž evolved with visual supervision information, we conduct the visual signals from V2SM, e.g. 2. We formulate & as supervision using visual-to-semantic loss:
Lv2s = E [1 cosine(2, 2, 1)]
Here, we use cosine similarity as a weak constraint for VOPE to conduct the information transfer from
the visual domain to the semantic domain. The reason is that we do not desire the visual information
to be over strong, helping the semantic prototype evolve progressively. Simultancously, we also use the semantic reconstruction loss to constrain VOPE, defined as:
Ls2s = E [|| +1 =x|]i],
where 2 is the dynamic semantic prototype at the & + 1-th step, formulated as follows:
Σk+1 az + (1 - -a)ž*+1
(6)
where a is a combination coefficient for fusing the dynamic semantic prototype evolved in the previous step and the mapped semantic vectors ž of the current step in VOPE. Thus, 24 is dynamically evolved for each training step and fed back to VOPE as input. Considering that the empirically predefined semantic prototype provides rich semantic priori, a is set to relatively large (i.e., 0.9) for preserving its original information. Such setting i) encourages the dynamic semantic prototype to evolve progressively under the supervision of visual information, and ii) facilitates model optimization.
5

---

Under review as a conference paper at ICLR 2023
3.4 OPTIMIZATION
Our DSP is jointly optimized with the generative model, e.g., CLSWGAN Xian et al. (2018), f- VAEGAN Xian et al. (2019b), TF. VAEGAN Narayan et (2020) and FREE Chen et al. (2021a), with the following objective function:
(7) where LG is the loss of generative model, Aseye. \v2s, As2s are the weights that control the importance of the related loss terms. Since Aseze and A525 are semantic reconstruction losses, we set Aays = A52s. With the similar measurement loss terms and model components designed for DSP, we empirically observe that our model is robust and easy to train on various 7SL. benchmarks when it is entailed on various generative models.
Liotal = La + Ssye + Av2s + As2s
3.5 INFERENCE PHASE
Synthesizing Visual Features for Unseen Classes: After training, we first take the pre-trained VOPE to update the predefined semantic prototypes to be dynamic semantic prototypes for unseen classes using Eq. 3 and Eq. 6, i.e. 2 az" + (1 - VOPE(=") Then, we take the dynamic semantic prototypes of unseen classes conditional supervision in generator (G) to synthesize class-specific visual features for unseen classes:
Esyn G(o, """,
where o a random Gaussian noise vector with |A| Al-dim.
Enhancing Visual Features for All Classes: Finally, we concatenate visual features (including the real visual features of seen classes and real/synthesized visual features of unseen classes) and their corresponding evolved semantic prototypes å VOPE(=) to enhance them, enabling the visual features to be closer to their corresponding semantic prototypes and alleviate the cross-dataset bias Chen et al. (202Ia). It is formulated as:
Seen Classes -ir = xir + 5";
The = =/ + 8",
Unseen Classes F Syg
1 3";
T/e = =x E.
(10)
where 2" U g" = -. -tr is the real visual feature in the training set Di,,, is the real visual feature in the test sets Die and Di. As such, the enhanced visual features are closer to their corresponding semantic prototypes, and they are more discriminative as the cross-dataset bias is alleviated. These enhanced visual features are used for classification later.
ZSL Classification: We employ Tfr and Tyyn to lean classifier (e.g., softmax), i.e., feast X
y UY" or fazst X + 77 yY". Once the classifier is trained, we use if, and žy, to test the model further. Note that DSP is inductive method as we do not the real visual features of unseen classes for model optimization.
4 EXPERIMENTS
Datasets. We evaluate the effectiveness of our DSP on three well-known 7SL benchmark datasets, ie., two fine-grained datasets ( CCB Welinder et al. (2010), SUN Patterson & Hays (2012) and FLO Nilsback & Zisserman (2008)) and one coarse-grained dataset (AWA2 Xian al. (2019a)). CUB consists of 11,788 images of 200 bird classes (seen/unseen classes = 150/50) captured by 312 attributes. SUN includes 14,340 images of 717 scene classes (seen/unseen classes 645/72) described by 102 attributes. AWA2 contains 37,322 images of 50 animal classes (seen/unseen classes 40/10) captured by 85 attributes. FLO inludes 8,189 images of 102 flower classes (seen/unseen classes = 82/20) captured by 1024 attributes.
Implementation Details. We use the training splits proposed in Xian et al. (2019a). Meanwhile, the visual features of images (input size 224 x 224) are extracted from the 1148-dimensional top-layer pooling units of CNN backbone (i.e., ResNet-101 He et al. (2016)) pre-trained on ImageNet. The V2SM and VOPE are MLPs with residual blocks, network details of which can be found in Appendix A and Appendix B. respectively. Since f- F-VAEGAN Xian et al. (2019b) is a popular generative ZSL method, we take it baseline and entail our DSP on it for conducting the ablation study, qualitative experiment and hhpehh-prramae parameter analysis. Following Xian et al. (2019a), we also use the same validation split each dataset to conduct the hyperparameter setting. We synthesize 150, 800

---

Under review as a conference paper at ICLR 2023
Table 1: Compared our DSP with the state-of-the-arts on CUB, SUN, AWA2 and FL.O benchmark datasets in the GZSL setting. + denotes embedding- based methods, while a denotes generative methods. The best and second-best results are marked in Red and Blue, respectively.
Methods
CUB
SUN
AWAZ
FLO
5
H
5
H
3
S
SOMA Zhu (2019)
36.7
71.3
48.5
37.6
87.1
AREN Xie al. (2019)
38.9
78.7
52.1
19.0
38.8
25.5
15.6
92.9
26.7
APN Xu (2020)
65.3
69.3
67.2
41.9
34.0
37.6
56.5
78.0
65.5
DAZLE Huynh Flhamifar (2020t)
56.7
59.6
52.3
24.3
60.3
75.7
CN Skorokhodov & Elhosciny (2021)
49.9
50.7
50.3
44.7
41.6
43.1
60.2
77.1
67.6
F2DFareer Nacem
(2022)
35.3
57.6
43.8
66.8
76.8
71.5
35.8
91.9
51.5
VALGAN Xian et al. (20195)
48.7
58.0
52.9
45.1
38.0
41.3
$7.6
70.6
63.5
56.8
74.9
64.6
TF-VAGAAN Narayan al. (2020)
53.7
61.9
57.5
48.5
37.2
42.1
58.7
76.1
66.3
62.5
$4.1
71.7
LarGAN Vyas ot (2020)
48.1
59.1
53.0
44.8
37.7
40.9
54.6
74,6
63.0
AGZSL Chou et al. (2021)
48.3
58.9
52.1
29.9
40.2
34.3
65.1
78.9
71.3
FREE Chen
54.9
47.4
37.2
75.4
84.5
GCM-CF Yue et al. (2021)
61.0
39.7
50.3
47.9
37.8
42.2
60.4
75.1
67.0
HSVA Chen al. (2021b)
52.7
58.3
55.3
48.6
39.0
43.3
59.3
76.6
56.8
FREE+ESZSI, Cotin et al. (2022)
51.6
60.4
55.7
48.2
36.5
41.5
51.
78.0
61.8
65.6
82.2
72.9
TF VE-VAGGN++ESZSL Çetin et al. (2022)
51.1
63.3
56.6
44.0
39.7
41.7
55.2
74.7
63.5
63.5
83.2
72.1
IVAEGAN
DSP
41.3
88.8
86,9
and 3400 visual features per unseen class to train the classifier for SUN, CUB and AWA2 datasets,
respectively. The combination coefficient for updating semantic prototypes is set 0.9 for all datasets. The loss weight of Aseye = As2s is set to 0.1, 1, 0.01 and 0.001 for CUB, SUN and AWA2, respectively. The loss weight of \v2s is set to 0.6/0.6 and 1.0 for CUB/AWA2 and SUN, respectively. The specific hyper-parameter settings of our DSP entailed on other baselines are in Appendix C.
Evaluation Protocols. During testing (ZSL classification), we follow the unified evaluation pro- tocols proposed in Xian et al. (2019a). We calculate the top-1 accuracy of the unseen class for the CZSL setting (denoted as Ace). In the GZSL setting, we measure the top-1 accuracy on seen and unseen classes, denoted as S and U, respectively. Their harmonic mean (defined as H = (2 x S US/(S + U)) are a better protocols the GZSL, setting. As such, we typically take the H to discuss the performance of 7SL. methods in the GZSL. setting.
4.1 EXPERIMENTAL RESULTS
Comparing with the SOTA. We first compare our f- f-VAEGAN+DSP with the state-of-the- art meth- ods in the GZSL setting. Table 1 shows the performance of various ZSL methods, including non- generative methods (e.g., SGMA Zhu et al. (2019), AREN Xie et al. (2019), APN Xu et al. (2020), DAZLE Huynh & Elhamifar (2020a), CN Skorokhodov & Elhoseiny (2021), 12DFormer Naeem et al. (2022)) and generative methods (e.g.. f-VAEGAN Xian et al. (2019b). TF-VAEGAN Narayan et al. (2020), AGZSL Chou et al. (2021), GCM-CF Chou et al. (2021), HSVA Chen et al. (2021b), ESZSL Çetin et al. (2022)). Compared to the popular non-gencrative methods, our f-VAEGAN+DSP achieves the best results on all datasets, i.e., CUB (H = 67.4), SUN (H = 48.1), AWA2 (H = 74.2) and FLO (H 75.2). Thus, our DSP effectively helps generative ZSL methods beat the embedding- based methods. Our method also achieves the best results with significant improvements compared to the other generative methods. For instance, our F-VAEAN+DSP outperforms the latest generative method (i.e., TF-VAEGAN+ESZSL Çetin et al. (2022)) by large-margin, resulting in the improve- ments of harmonic mean by 10.8%. 6.4% and 10.7% on CUB, SUN and AWA2, respectively. These results consistently demonstrate the superiority and great potential of our DSP in generative ZSL.
Furthermore, we also compare our DSP with the popular generative methods in the CZSL setting, as shown in Table 2. F-VAEGAN+DSP achieves competitive performances of harmonic mean with 62.8, 68.6 and 71.6 on CUB, SUN and AWA2, re- spectively. These results indicate that DSP also effectively tackles the semantic-visue domain shift problem of the generative ZSL in the CZSL setting.
Table 2: Compared our method with the gener- ative ZSL methods in the CZSL setting.
CUB
SUN
AWA2
Methods
acc
acc
acc
CLSWGAN Xian al. (2018)
57.3
60.8
68.2
EVARGAN et (2019h)
64.7
CADA Schinfelld et al. (2019)
59.8
61.7
63.0
Composer Narayan et (2020)
FREE Chen et al. (2021a)
64.8
65.0
68.9
HSVA Chen al. (202
62.8
63.8
70.6
FVAEGAN + DSP
62.8
68.6
71.6
Ablation Study. Based on the baseline (i.e., f-VAEGAN Xian et al. (2019b)), we conduct ablation studies to evaluate the effectiveness of our DPS in terms of the LSeyes Lv2s, Le2s, semantic prototype evolution (i.e., Eq. 6) and enhancing visual feature (i.e., Eq. 9 and Eq. 10). Our results are shown in Table 3. When DSP is without the constraints of Lseyer its performance is dropped
7

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation studies for different components of DSP. The baseline is the popular generative ZSL model, Le,, f-VAEGAN Xian al. (2019b).
CUB
SUN S
Method
S
H
0
H
bascline
48.7
58.0
52.9
45.1
38.0
41.3
baseline+DSP (w/o Seyyc)
60.0
63.9
61.9
57.4
38.4
46.0
bascline+DSP (w/o Ls2s)
61.9
69.6
65.5
58.1
37.2
45.4
baseline+DSP (w/o Lv2s)
58.8
61.0
59.9
55.1
34.8
42.7
baseline+DSP (w/o Eq. 6)
54.4
55.3
54.8
50.3
36.2
42.1
baseline+DSP (w/o Eq. and Eq. 10)
52.4
53.9
53.1
54.2
35.0
42.5
bascline+DSP (full)
62.5
73.1
67.4
57.7
41.3
48.1
as the generator in the generative model lacks semantic cycle-consisteney supervision. Meanwhile, the performance of DSP is dropped slightly without the optimization of Cs2s. In contrast, if DSP can not be evolved under the supervision of visual information using Lvss. DSP achieves very poor results compared to its full model, i.e., the harmonic mean drops by 7.5% and 5,4% on CUB and SUN. respectively. These results demonstrate that evolving semantic prototypes according to the visual information is effective for refining the predefined semantic prototypes aligned to the visual prototypes, helping DSP tackle the risual-semante domain shift problem in generative ZSL. Notably, if DSP does not dynamically evolve the predefined semantic prototypes for generative learning, ie., DSP w/o Eq. 6, it almost loses its effectiveness and only achieves little improvement for the baseline with additional constraints (i.e., Lseyes L22s and Ls2s). For instance, its performance of harmonic mean drops by 12.6% and 6.0% on CUB and SUN, respectively, compared to its full model. Enhancing visual feature also effectively improves the performance of DSP. These results show that the dynamic semantic prototype is effective to tackle the visual-semantic domain shift.
Table 4: Evaluation of DSP with multiple popular generative ZSL models on three benchmark datasets. Each row pair shows the effect of adding DSP to particular generative ZSL model.
CUB
SUN
AWA2
Methods
U
S
CLSWGAN Xian (2018)
43.7
S7.7
49.7
42.6
36.6
39.4
37.9
61.4
59.6
CLSWGAN Xian
(2018)DSP
51.4
6388
56.9
48.3
43.0
45,5 6.
60.0
86.0
70.7" uI.
LVAEGAN Xian el al. (2019h)
48.7
58.0
52.9
45.1
38.0
41.3
$7.6
70.6
63.5
EVAEGAN Xian DDS
62.5
73.1
67.4 1445
57.7
41.3
48.1
66
63.7
88.8
74,2
VAHGANN Narayan et al. (2020)
53.7
61.9
57.5
48.5
37.2
42.1
58.7
76.1
66.3
TA-GANAAN Narayan et al. (SP20)DDSP
58.7
67.4
62.8
60.3
45.3
51.7
65.6
87.1
74.8
FREE Chen et al. (2021a)
54.9
60.8
57.7
47.4
37.2
41.7
60.4
75.4
67.1
FREE Chen et al. DSSP
60,9
68.7
64.6
60.3
44.1
51.0
'9.3
65.3
89,2
75.4 (8.
Generative ZSL Methods with DSP. To evaluate our dynamic semantic prototype learning method as a general technique to improve generative model, we entail into four recent popular gen- erative ZSL methods, i.e., CLSWGAN Xian et al. (2018), f-VAEGAN Xian et al. (2019b). TF- VAEGAN Narayan et al. (2020) and FREE Chen et al. (2021a). We use the official repositories shared by respective authors of these works implement experimental results. Since TF-VAEGAN and FREE have the semantic decoder, we only add the VOPE with LV25 and $22 to enable them to learn dynamic semantic prototypes. The specific hyper-parameter settings of our DSP entailed on these baselines are in Appendix C. As shown in Table 4, DSP consistently improves the performances of all baselines on all benchmark datasets by large-margin. For example, the average performance gains of harmonic mean are 8.5%, 8.0% and 9.7% on CUB, SUN and AWA2, respectively. These results demonstrate that i) the and neglected by existing generative 7SL, and ii) our dynamic semantic prototype learning method effectively tackles this problem.
domain shift is an important bottleneck challenge
Qualitative Performance. To intuitively show the effectiveness of our DSP, we qualitatively inves- tigate the improvements over baseline (i.e., f-VAEGAN Xian et al. (2019b)). We conduct t-SNE visualization Maaten & Hinton (2008) of the real/synthesized visual features learned by the f- VAEGAN and E-VAEGAN+DSP on 10 classes randomly selected from CUB. As shown in Fig. 3(Left), the visual features synthesized by f-VAEGAN Xian et al. (2019b) are far away from their corresponding real visual features, caused by the sisual-semantic domain shift problem. Meanwhile, since the cross-dataset bias problem Chen et al. (2021a), the real visual features directly extracted
8

---

Under review as a conference paper at ICLR 2023
F-VEGAN
+
20
10
-20
Figure 3: t-SNE visualizations of visual features learned by the f-VAEGAN Xian et al. (2019b) (Left) and f- F-VAGAN+DSP (Right). The 10 colors denote 10 different classes randomly selected from CUB. The "o" and "x" indicate the real and synthesized visual features, respectively, The t-SNE visualizations on SUN and AWA2 are in Appendix D. (Best viewed color)
6
U
0
H
80.0
O
8
O
80.0
(Ng
72.5
72.5
72.5
Accumss
66.0
66.0
66.0
65.0
67.5
67.6
57.5
57.5
50.0
50.0
0.001
0.01
0.1
0.2
0.001
0.01
0.1
0.6
0.8
N
syn
ASyy
Ay25
(a)
(b)
(c)
(d)
Figure 4: The effects of (a) combination coefficient a, (b) synthesizing number of per unseen classes
Nagns (c) loss weights ASege and (d) loss weights Av2s. We take CUB example.
from the CNN backbone are poor quality. In contrast, our DSP dynamically evolves the predefined semantic prototypes accurately represent the visual representations, enabling the generator to synthesize reliable visual features for all classes (as shown in Fig. 3(Right)). Furthermore, DSP enhances the real/synthesize visual features by enriching semantic information into visual represen- tations. As such, DSP obtains significant performance gains over f-VAEGAN Xian et al. (2019b), i.e. considerable improvements of harmonic mean with 14.5% 6.8% and 10.7% on CUB, SUN and AWA2, respectively.
Hyper-Parameter Analysis. We conduct extensive experiments to investigate the effects of various hyper-parameter on CUB, iee, combination coefficient a in Eq. 6, synthesizing number of per unseen classes Nayn the loss weights ASeye and Av2s. Since Lseye and L52s are the semantic reconstruction loss, we set ÄSeye As2s and take one loss weight into the analysis. From the results in Fig. 4(a), DSP is not insensitive to the combination coefficient for dynamic semantic prototypes evolution, and DSP achieves better results with relatively large value of a (i.e., 0.9). The reason is that the dynamic semantic prototypes can adapt to the evolution according to the visual-semantic interactions during optimization. The results shown in Fig. 4(b) indicate our DSP is robust to the Nayn when N syn is not set too large. Results in Fig. 4(c) and Fig. 4(d) show the best effects of our f-VAEGAN+DSP can be achieved when ASeye and λv2s is set to 0.1 and 0.6, respectively. Additionally, we find that \v2s should be set to larger than ÄSeye and As2s This is because the dynamic semantic prototypes should mainly depend on the visual information during evolution. Overall, our model is robust to all tyyer-pprammer and easy to optimize.
5 CONCLUSION
In this work, we are the first to discuss that the predefined semantic prototype results in the visual- semantic domain shift problem in generative ZSL. To tackle this problem and advance generative ZSL, we propose dynamic semantic prototype learning method, which jointly and mutually refines the semantic prototypes and visual features enabling the generator to synthesize reliable visual features. Our DSP is plug-and-pla to be entailed various baselines and consistently achieves large- margin improvements on three ZSL benchmark datasets. We believe that our work will facilitate the development of stronger zero-shot systems and motivate their deployment in real-world applications.

---

Under review as a conference paper at ICLR 2023
Reproducibility statement. We discuss the problem of issul--smantic domain shift in generative ZSL and propose dynamic semantic prototype learning to tackle it. We intuitively demonstrate that is an important bottleneck problem in generative ZSL. We also clearly introduce model details, implementation details and hyper-parameter settings, which we use in our experiments. We will provide source code a public repository once our paper is accepted.
REFERENCES
Zeynep Akata, S. Reed, D. Walter, H. Lee, and B. Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR, pp. 2927-2936, 2015.
Zeynep Akata, F. Perronnin. Z. Harchaoui, and C. Schmid. Label-embedding for image classification.
IEFE Transactions on Pattern Analysis and Machine Intelligence, 38:1 1011142--444 1438, 2016.
Gundeep Arora, V. Verma, Ashish Mishra, and P. Rai. Generalized zero-shot learning via synthesized examples. In CVPR, pp. 4281-4289, 2018
Shiming Chen, Wenjie Wang, Beihao Xia, Qinmu Peng. Xinge You, Feng Zheng, and Ling Shao. Free: Feature refinement for generalized zero-shot learning. In ICCV, 2021a.
Shiming Chen, Guo-Sen Xie, Yang Yang Liu, Qinmu Peng. Baigui Sun, Hao Li, Xinge You, and Ling Shao. Hsva: Hierarchical semantic-visual adaptation for zero-shot learning. NeurIPS, 2021b.
Shiming Chen, Ziming Hong, Yang Liu, Guo-Sen Xie, Baigui Sun, Hao Li, Qinmu Peng, Ke Lu, and Xinge You. Transzero: Atrrbbue-guided transformer for zero-shot learning. In AAAI, 2022.
Yu- Ying Chou, Hsuan-Tier Lin, and Tyng-Luh Liu. Adaptive and generative zero-shot learning, In ICLR, 2021.
Rafael Felix, B. V. Kumar, Reid, and G. Carneiro. Multi-modal cycle-consistent generalized zero-shot learning. In ECCV, 2018.
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, and Shaogang Gong. Transductive multi-view zero- shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37:2332-2345 2015.
lan Goodfellow, Jean Pouget-Abadie, M. Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
D. Huynh and E. Elhamifar. Fine-grained generalized zero-shot learning via dense attribute-based attention. In CVPR, pp. 4482 4492, 2020a.
Dat T. Huynh and Elhamifar. Compositional zero-shot learning via fine-graing dense feature composition. In NeurIPS, 2020b.
Zhen Jia, Zhang Zhang, Liang Wang, Caifeng Shan, and Tieniu Tan. Deep unbiased embedding transfer for zero-shot learning. IEEE Transactions on Image Processing, 29:1958-1977 2020.
Diederik P. Kingma and M. Welling, Auto-encoding variational bayes. In 2014.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Xuanyi Dong, and C. Zhang. Isometric propagation network for generalized zero-shot learning. In ICLR, 2021.
L. V. D. Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2600 2008.
Muhammad Ferjad Nacem, Yongqin Xian, Luc Van Gool, and Federico Tombari. 12dformer: Learning image to document attention for zero-shot image classification. In NewrIPS, 2022.
Sanath Narayan, A. Gupta, F. Khan, Cees G. M. Snoek, and L. Shao. Latent embedding feedback and discriminative features for zhro classification. In ECCV, 2020.
10

---

Under review as a conference paper at ICLR 2023
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. ICVVIP,, pp. 722-729, 2008.
G. Patterson and J. Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR, pp. 2751-2758, 2012.
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, and Xizhao Wang. A review of generalized zero-shot learning methods. IEEE transactions on pattern analysis and machine intelligence, 2022.
Edgar Schönfeld, S. Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero- and few-shot learning via aligned variational autoencoders. In CVPR, pp. 8239-8247, 2019.
Yuming Shen, J. Qin, and L. Huang. Invertible zero-shot recognition flows. In ECCV, 2020.
Ivan Skorokhodov and Mohamed Elhoseiny. Class normalization for (continual)? generalized zero-shot learning. In ICLR, 2021.
M. R. Vyas, Hemanth Venkateswara, and S. Panchanathan. Leveraging seen and unseen semantic relationships for generative zero-shot learning. In ECCV, 2020.
Ziyu Wan, Dongdong Chen, Yan Li, Xingguang Yan, Junge Zhang, Yizhou Yu, and Jing Liao. Transductive zero-shot learning with visual structure constraint. In NeurIPS, 2019.
Chaoqun Wang, Shaobo Min, Xuejin Chen, Xiaoyan Sun, and Hougiang Li. Dual progressive prototype network for generalized zero-shot learning. In NeurlPS, 2021.
Welinder, S. Branson, T. Mita, C. Wah, Florian Schroff. Serge J. Belongie, and Perona. Caltech- uesd birds 200. Technical Report CNS-TR-2010-000 2010-001, Caltech,, 2010.
Yongqin Xian, T. Lorenz, B. Schiele, and Zeynep Akata, Feature generating networks for zero-shot learning. In CVPR, pp. 5542-5551, 2018.
Yongqin Xian, Christoph H. Lampert. Bernt Schiele, and Zeynep Akata. Zero-shot learning a comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions Putlern Analysis and Machine Intelligence, 41:225 1-2255125552. 2019a.
Yongqin Xian, Saurabh Sharma, Schiele, and Zeynep Akata, F-vaegan-d2: A feature generating framework for any-shot learning. In CVPR, pp. 10267-10276, 2019b.
Guo-Sen Xie, L.. Liu, Xiaobo Jin, Zhu, Zheng Zhang. J. Qin, Yazhou Yao, and L. Shao. Attentive region embedding network for zero-shot learning. In CVPR, pp. 9376-9385, 2019.
Wenjia Xu, Yongqin Xian, Jiuniu Wang, B. Schiele, and Zeynep Akata, Attribute prototype network for zero learning. In NeurlPS, 2020.
Xing Xu, Fumin Shen, Yang Yang, D. Zhang, Heng Tao Shen, and Jingkuan Song. Matrix tri- factorization with manifold regularizations for zero-shot learning. In CVPR, pp. 2007-2016, 2017.
Caixia Yan, Xiaojun Chang, Zhihui Li, Zong Yuan Ge, Weili Guan, Lei Zhu, and Qinghua Zheng. Zeronas: Differentiable generative adversarial networks search for zero-shot learning. IEEE transactions on pattern analysis and machine intelligence, 2021.
Yunlong Yu, Zhong Ji, Yanwei Fu, Jichang Guo, Yanwei Pang. and Zhongfei Zhang. Stacked remantics-guided attention model for fine-grained zero-shot learning. In NeurlPS, 2018.
Zhongqi Yue, Tan Wang, Hanwang Zhang, Qianru Sun, and Xiansheng Hua. Counterfactual zero-shot and open-set visual recognition. In CVPR, 2021.
Haofeng Zhang, Yang Long. Yu Guan, and Ling Shao. Triple verification network for generalized zero-shot learning. IEEE Transactions on Image Processing,
Yizhe Zhu, Jianwen Xie, Z. Tang, Xi Peng, and A. Elgammal. Semantic-guided multi-attention localization for zero-shot learning. In NeurIPS, 2019.
Samet Çetin, Orhun Bugra Baran, and Ramazan Gokberk Cinbis. Closed-form sample probing for learning generative models in zero-shot learning. In ICLR, 2022.
11

---

Under review as a conference paper at ICLR 2023
APPENDIX Appendix organization:
Appendix A: Network details of V2SM.
Appendix B: Network details of VOPE.
Appendix C: The hyper-parameterr settings of our DSP entailed on four baselines.
Appendix D: t-SNE Visualization on SUN and AWA2.
A
NEWORK DETAILS OF V2SM
As shown in Fig. 5, the network of V2SM is an MLP with a residual block, which avoids too much information loss when mapping the visual feature 2048-din into semantic space with |A| Aimm
2r5 R 2048
2C
ReLU
FC
ReLU
FC
xt's
ReLU
FC
ReLU
FC
RI4
Vector Addition
Figure 5: Network details of V2SM.
B NEWORK DETAILS OF VOPE
As shown in Fig. 6, the network of VOPE is MLP with residual block (fusing with Hardamad Product). Essentially, the network of VOPE a routing gate mechanism implemented by channel attention, which is effective in selectively evolving the specific attribute value of semantic prototypes according to the corresponding visual information. As such, the predefined semantic prototypes can be progressively evolved and aligned to the visual prototypes.
ReLU
W e RIA
Zk
FC
FC
Z k
de e R
ReLU
Z'ERI4
FC
Hadamard Product
Figure Network details VOPE.
C THE HYPER-PARAMETER SETTINGS OF OUR DSP ENTAILED ON VARIOUS BASELINES
We present the hyper-parameter settings our DSP entailed on various baselines (i.e.,
CLSWGAN Xian et al. (2018), f-VAEGAN Xian et al. (2019b), TF-VAEGAN Narayan et al. (2020) and FREE Chen et al. (2021a) on CUB, SUN and AWA2. Including the synthesizing number of per unseen classes N Neyn. the loss weights Aseyes \v2s and combination coefficient in Eq. 6. We empirically observe that our DSP is robust and easy to train when entailed on various generative models. Based on these hyper- parameerr settings, DSP achieves significant performance gains
12

---

Under review as a conference paper at ICLR 2023
Table 5: The hyper-parameter settings of our DSP entailed various baselines (iee, CLSWGAN Xian et al. (2018), 1-VAGGAN Xian et al. (2019b), TF-VAEGAN Narayan et al. (2020) and FREE Chen al. (2021a)) on CUB, SUN and AWA2. Including the synthesizing number per unseen classes N the loss weights Aseye. Av2s and combination coefficient a in Eq. 6.
Methods
CUB
SON
AWA1
Negn
Aseye
Xyag
a
Nay
ABrge
Avas
8
N.gn
ASege
Avas
tr
clsGAAN Xian al. (2018) DSP
300
0.15
1.0
0.9
300
0,005
1,0
0.9
3400
0.1
0.9
T-VAEGAN Xian (DD19))+DSP
0.1
0.6
0.9
0.01
0.
0.001
0.9
TF- VARGAN Narayan et al (2020) DSP
400
001
1.0
0.9
500
0.08
1.5
0.
5300
0.09
1.4
0.9
FREE Chen al, (2021a)
0.1
0.6
0.9
0.01
0.
4000
0.001
0.9
the various popular generative models all datasets. For instance, the average performance gains harmonic mean are 8.5%, 8.0% and 9.7% on CUB, SUN and AWA2, respectively, Please refer to Table 4.
D T-SNE VISUALIZATION ON SUN AND AWA2
As shown in Fig. 7, I-SNE visualizations of visual features learned by the f-VAEGAN Xian et al. (2019b) (Left) and f- VAEGAN+DSP (Right) on SUN and AWA2. Analogously, the visual features generated by f-VAEGAN are also far away from their corresponding real ones, and the discrimination of these real/synthesized visual features is undesirable. In contrast, our DSP help f-VAEGAN synthesize visual features close their corresponding real ones and enhance these real visual features. As such, our DSP significantly improves the performances of f- f-VAEGAN Xian et al. (2019b) on SUN and AWA2.
FVAEGAN
+
(a) SUN
0-
-10
-10-
-15
-15
FVAEGGN
DSP
20
5-
(b) AWA2
-10
-15
20-
-20
-20
-15
-10
10
20
-30
-20
-10
10
Figure t-SNE visualizations of visual features learned by the f-VAEGAN Xian al. (2019b) (Left) and E-VAEGAN+DSP (Right) on (a) SUN and (b) AWA2. The "o" and "*" indicate the real and synthesized visual features, respectively. (Best viewed color)
13


Reviews:
[{"review_id": "xlJLlm1lga", "reviewer": "Reviewer_gkTe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a dynamic semantic prototype learning method that aligns empirical semantic prototypes with actual semantic prototypes in order to synthesize accurate visual features. The method consists of a generative model that generates visual features from semantic prototypes, a network that maps visual features to semantic prototypes, and a network that modifies semantic prototypes by considering visual features. VOPE iteratively evolves predefined class semantic prototypes to become dynamic semantic prototypes. Using standard zero shot learning benchmark dataset, the proposed method improves the performance of the baseline generative approach ZSL methods.", "summary_of_the_review": "Although the proposed method shows some novelty and performance improvement from the baseline, it is not considered to have reached the stage of publishing at this time due to its inferior performance compared to embedding-based methods and the low quality of writing.", "strength_and_weaknesses": "- Strength\n\n-- This paper proposes a new method for modifying predefined semantic prototypes to actual semantic prototypes using visual features.\n\n-- Combining the proposed method with the ZSL method for a generative approach improves the performance of baselines methods.\n\n- Weaknesses\n\n-- The proposed method improves the performance of the generative ZSL methods, but is inferior to the embedding-based ZSL method.\n\n-- The writing quality of the paper is low, making the paper difficult to read. The same explanations appear repeatedly in the paper. In addition, there are some typos.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "u3OObNF_f6Q", "reviewer": "Reviewer_J6h5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a dynamic semantic prototype learning method, which jointly refines the semantic prototypes and visual features enabling the generator to synthesize reliable visual features. The authors design a visual-oriented semantic prototype evolving network to update the semantic prototypes iteratively. Experiments on three datasets show the effectiveness of the proposed approach. ", "summary_of_the_review": "This paper proposes a novel dynamic prototype network, which is a little complicated. Some errors may exist in the approach and experiments.", "strength_and_weaknesses": "Strength:\n1. This paper is well-organized and well-written, which is easy to read.\n2. The idea of dynamic prototypes for zero-shot learning is interesting and the prototype evolving is novel.\n3. The proposed approach achieves good performance on different baselines.\n\nWeakness:\n1. The V2SM network contains six fully connected layers, which introduce many parameters. How does the training efficient? Will it be easy to overfit?\n2. What is the motivation to design the structure of VOPE? It is a structure of self-attention, which is not influenced by visual samples.  \n3. It is unclear how the dynamic prototypes are obtained in the inference phase. The authors said ‘ take the pretrained VOPE to extract the dynamic semantic prototypes for all classes ’, which is strange. In the training stage, the prototypes for the seen classes are evolving with changing parameters. However, in the test stage, they are obtained with fixed parameters, which are not the same. Moreover, only the k+1th prototype is used, is it reasonable to be called a dynamic prototype?\n4. In the classification process, the authors concatenate the visual features and the semantic features to augment the features. Why concatenate the features? The influence is not shown. \n5. Figure 3 is strange. Why the real samples are scattered in the first picture while they are clustered in the second picture?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tCt_h4ziFf", "reviewer": "Reviewer_LoBd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a semantic class embedding refinement mechanism by introducing two networks, (1) V2SM to enable the generator to synthesize semantically rich visual features and (2) VOPE to iteratively evolve the quality of semantic class embeddings throughout the training phase in generative zero-shot learning setting. Proposed refinement mechanism can be easily integrated into any existing generative ZSL pipeline just by simply introducing the proposed loss terms.  \n", "summary_of_the_review": "Overall, the experimental improvements are impressive. However, the paper does not give a strong intuition why the model works / why is it desiged as this. It might have been good to show each module's / step's effect on a simple (synthetic) 2D dataset example. \n\nI have also concerns regarding the model selection procedure, given the sensitivity of the model to certain hyper-parameters.", "strength_and_weaknesses": "Pluses \n- The method seems to be clearly defined, though in a confusing way (see below).\n- Good experimental results: the experimental results show that the proposed method yields significant improvements based on multiple baselines. \n- A somewhat detailed ablation study is also presented.\n- Experimental results for various hyper-parameters are given.\n\nMinuses\n- While the method appears to be mostly well-defined, the paper does not give a clear and strong intuition why the method works (so well). The reasoning behind various modeling decisions are not fully clear. For example, it is not clear why z-tilde may really differ from z. In this sense, VOPE as a whole can also simply be seen as an extension of V2SM loss. Overall, despite understanding  Overall, I find it hard to understand the intuition behind the technical details of the model, i.e. there is a gap between the high level ideas mention in Section 1, and the way model is designed.\n- As a continuation of my previous comment, I wonder whether VOPE could have been replaced by a simple class embedding-to-z_{k+1} mapping possibly with strong yet simple regularization (dropout, l2 regularization, etc).\nThe decisions in Section 3.4 are also not very clear. Why do you use dynamic prototypres for G conditioning? (Why conditioning do you use for G during training?)\n- Similarly,  why do you do concatenation (Eq 9-10) ?  How does it affect the results? \n- There is a lack of hyper-parameter tuning section since the application of invalid hyper-parameter tuning is a common malpractice in zero-shot learning studies. Could you please elaborate on your final model/classifier selection criteria? Do you perform the final model selection (together with loss weights, combination coefficient etc.) based on a validation set. Are the reported results on paper obtained without any further test set based finetuning? \n- Minor: Performance evaluation of the proposed method on FLO dataset can be added since it is a widely used dataset among the works on generative ZSL and most of the state-of-the-art methods presented for comparison introduced FLO dataset in their evaluation scheme. (In this context: Figure 4 suggests that the details of certain HPs are highly sensitive.)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LCgHoDinM-", "reviewer": "Reviewer_qGfG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on refining the semantic prototypes in ZSL. The authors top a V2SM network to the generator, mapping visual features to a semantic-related space. On the other hand, they transfer the semantic embeddings by another VOPE module. The outputs of V2SM and VOPE are aligned by cosine similarity and l2 losses. The most important step then is updating the semantic prototypes by the output of VOPE using momentum update.\n\nThe authors claim this procedure helps train ZSL classifiers and the experiments are conducted on conventional ZSL datasets.", "summary_of_the_review": "As is discussed, this paper requires class labels as inputs for classification, which is a wrong design. I would reject this paper with no doubt. The authors might have made some mistakes in writing or it is just that they proposed something wrong. No matter what the cause is, this paper should not be considered for publication on any venue with its current form.", "strength_and_weaknesses": "## Pros\n---\n* Good reference to the related articles\n* The writing is easy to follow.\n\n## Cons\n---\nThis paper comes with tons of problems. I think there is no need to point out all of them because the design goes wrong from the very beginning.\n\nLet us go directly to the Inference Phase of DSP. Training a softmax classifier in this paper requires a concatenated input of:\n* The real/synthetic visual feature\n* The evolved semantic prototype\n\nHow can one do classification by feeding a class prototype? Or let me ask how can one know which class prototype to concatenate before classifying the datum?\n\nThis design is breaching almost all the fundamentals of a recognition task. I must say the entire design is technically wrong.\n\nHere are some notes on the other problems the authors need to be aware of:\n* Eqs. (4) and (5) apply different distance measurements on the same feature space. Please consider making them consistent.\n* Below Eq. (1), $z_{real} \\cup z_{syn}=z\\in R$ is totally a wrong expression.\n* The model actually does not require momentum update (Eq. (6)). Initializing with the original semantic feature and then letting the gradient update z would be fine. Using Eq. (6) here is just a waste of time.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject