HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning
Authors: Shiming Chen
Guosen Xie
Yang Liu
Qinmu Peng
Baigui Sun
Hao Li
Xinge You
Ling Shao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Zero-shot learning
Domain adaptation
Text-image matching
Heterogeneous feature representation

Abstract: Zero-shot learning (ZSL) tackles the unseen class recognition problem,  transferring semantic knowledge from seen classes to unseen ones. Typically, to guarantee desirable knowledge transfer, a common (latent) space is adopted for associating the visual and semantic domains in ZSL.  However, existing common space learning methods align the semantic and visual domains by merely mitigating distribution disagreement through one-step adaptation. This strategy is usually ineffective due to the heterogeneous nature of the feature representations in the two domains, which intrinsically contain both distribution and structure variations. To address this and advance ZSL, we propose a novel hierarchical semantic-visual adaptation (HSVA) framework. Specifically, HSVA aligns the semantic and visual domains by adopting a hierarchical two-step adaptation, i.e., structure adaptation and distribution adaptation. In the structure adaptation step, we take two task-specific encoders to encode the source data (visual domain) and the target data (semantic domain) into a structure-aligned common space. To this end, a  supervised adversarial discrepancy (SAD)  module is proposed to adversarially minimize the discrepancy between the predictions of two task-specific classifiers, thus making the visual and semantic feature manifolds more closely aligned. In the distribution adaptation step, we directly minimize the Wasserstein distance between the latent multivariate Gaussian distributions to align the visual and semantic distributions using a common encoder. Finally, the structure and distribution adaptation are derived in a unified framework under two partially-aligned variational autoencoders. Extensive experiments on four benchmark datasets demonstrate that HSVA achieves superior performance on both conventional and generalized ZSL. The code is available at \url{https://github.com/shiming-chen/HSVA}.

HSVA: Hierarchical Semantic- Visual Adaptation for Zero-Shot Learning
Shiming Chen', Guo-Sen Xie*, Yang Liu", Qinmu Peng', Baigui Sun
Hao Li Xinge You', Ling Shao
Huazhong University of Science and Technology (HUST), China
8 Alibaba Group, Hangzhou, China
Mohamed bin Zayed University of AI (MBZUAD, UAE
FInception Institute of Artificial Intelligence (IIAI), UAE
(shimingchen, youug)uuuuuee edu. en ling. sing.shaa@ieee.oo org
Abstract
Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. Typically, to guarantee desirable knowledge transfer, common (latent) space is adopted for associating the visual and semantic domains ZSL.. However, existing common space learning methods align the semantic and visual domains by merely mitigating distribution disagreement through one-step adaptation. This strategy is usually ineffective due the heterogeneous nature of the feature representations in the two domains, which intrinsically contain both distribution and structure variations. To address this and advance ZSL, we propose novel hierarchical semantic-visual adaptation (HSVA) framework. Specifically, HSVA aligns the semantic and visual domains by adopting a hierarchical two-step adaptation, i.e., structure adaptation and distribu- tion adaptation. In the structure adaptation step. we take two task-specifi encoders to encode the source data (visual domain) and the target data (semantic domain) into structure- crepancy (SAD) learning is proposed to adversarially minimize the discrepancy between the predictions of two task-specific classifiers, thus making the visual and semantic feature manifolds more closely aligned. In the distribution adaptation step, directly minimize the Wasserstein distance between the latent multivari- ate Gaussian distributions to align the visual and semantic distributions using a common encoder. Finally, the structure and distribution adaptation are derived in unified framework under two partially-alignee variational autoencoders. Extensive experiments four benchmark datasets demonstrate that HSVA achieves superior performance on both conventional and generalized 7SL. The code is available at httts://githuu: com/ swwming-cheennnn chen/HSVA)
common space. To this end, a supervised adversarial dis-
Introduction
Over the past decade, significant progress has been made in zero-shot learning (ZSL), which aims to recognize new classes during learning by exploiting the intrinsic semantic relatedness between seen and unseen categories [
3 Inspired by the way humans learn unknown concepts, side- information (e.g.. attributes word vectors (5). and sentences 6 related seen/unseen classes employed to guarantec knowledge transfer between the seen/unseen data. Common space learning is one typical method for representing the relationship between the visual and semantic domains for ZSL, which is key to dealing with the knowledge transfer 77. However, existing common space
*Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

Manifold
Manifold
Manifold
Class
Semantic
Visual features
Imuge
Mapping
Image
Mapping
Mapping
Side- Information
Mappine
Side- Information
Mapping
(a) One-step adaptation
(b) Hicrarchical adaptation
CADA VAE: clacses
CADAVAE: unseen classes
HSVA seen classes
HSUA: classes
(c) t-SNE visualization of features for CADA-VAL
(d) I-SNE visualization of features for HSVA
Figure Illustration of HSVA. Common space learning methods learn domain-aligner feature representations for semantic and visual domains in latent embedding space with semantic-visual adaptation. However, the heterogeneous feature representations of the semantic and visual domains vary in both distribution and structure. (a) One-step adaptation focuses on distribution alignment between visual and semantic domains, neglecting structure variation. This causes the semantic and visual distributions to be located in different manifolds, resulting in the misclassificatiio of some samples. (b) Hierarchical adaptation, contrast, can Iearn an intrinsic and discriminative common space for semantic and visual feature representations by adopting sequential structure adaptation and distribution adaptation. We provide t-SNE visualizations [14 of features learned by (c) CADA- CADA-VAE []] and (d) our HSVA on 10 classes from AWA2 (the "o" and "x" indicate visual and semantic features, respectively, and different colors denote different classes). Best viewed in color.
learning methods align the distributions of the semantic and visual domains with one-step adaptation ا, [0] [2] 13), while neglecting the fact that the heterogeneous feature representations of distinct semantic and visual domains have both distribution and structure variations. Thus, their performance is limited. In this work, we propose novel common space learning formulation that leverages a hierarchical semantic-visual adaptation to learn an intrinsic common space for better alignment of the two heterogeneous representation domains.
The heterogeneous feature representations of the semantic and visual domains are distinct [5 16) Although one-step adaptation learns Distribution-allined feature representations for the semantic and visual domains using two different mapping models (e.g., encoders) and distribution alignment constraints (e.g., maximum mean discrepancy (MMD)), the two distributions are located in different manifolds since the one-step adaptation does consider the manifold structure relationship between visual and semantic representations, shown in Figure I (a) and (c). If we take the Euclidean distance or manifold distance [17 [8] to measure the relationship between different classes, the classifier inevitably misclassifies some samples, thus leading to inferior 7SL performance. As such, mapping the semantic and visual domains into an intrinsic and desirable common space for conducting an effective ZSL is highly necessary.
To learn structure- and inttribution-alinnnn feature representations for distinct visual and semantic domains, we propose a hierarchical semantic-visua adaptation (HSVA) framework to learn an intrinsic common space that contains the essential multi-mi modal information associated with unseen classes (Figure and (d)). HSVA aligns the semantic and visual domains with structure adaptation (SA) and distribution adaptation (DA) unified framework, under two partially-aligned variational autoencoders. In the SA, we use two task-specific encoders to encode source data (visual domain) and target data (semantic domain), respectively, into a structure-aligned space, which learned through supervised adversarial discrepancy (SAD) learning to minimize the discrepancy between

---

the predictions of the two lask-specifi classifiers. This encourages the visual and semantic feature manifolds to be closer to each other, thus aligning the manifold structure variation. In the DA, we map the Arructrr-allined features into distribution encoder, which is optimized by minimizing the Wasserstein distance between the latent multivariate Gaussian distributions of the visual and semantic domains. The common encoder preserves the structure-aligned representations in the DA. Thus, HSVA can learn a structure and distribution- aligned common space for visual and semantic feature representations, which is better than the distribution-alignen
common space using common
feature representations learned by existing common space learning methods 8 9 7 TO II [9]. To the best of our knowledge, this is the first work that leverages hierarchical semantic- visual adaptation to address heterogeneous feature alignment in ZSL. Following [TI), we four challenging benchmark datasets, i.e., CUB, SUN, AWAI,
conduct extensive experiments and AWA2, under both conventional and generalized ZSSL. settings. The proposed HS VA achieves consistent improvement over the existing common space learning methods all datasets. We show qualitatively that the structural variation is important for the interaction between visual and semantic domains. Moreover, show significantly better common space for visual and semantic feature representations than the existing common space learning methods, e.g., CADA-VAB [.].
2 Related Work
Zero-Shot Learning. In practice, the relationship between visual and semantic domains is determined by learning an embedding space where semantic vectors and visual features interact 220 24
22 23
26 19]. There are three mapping methods for learning such an embedding space, including direct mapping, model parameter transfer, and common space learning 77.. Direct mapping learns a mapping function from visual features to semantic representations [4. 20 27 However, its generalization limited by the high intra-class variability of the visual domain. Meanwhile, the lower-dimenssonal semantic space shrinks the variance of the projected data points, resulting in the hubness problem (28). Model parameter transfer, in contrast, takes place in the visual space, where the model parameters for unseen classes are usually obtained [29] [30. In essence, generative ZSL follows this methodology, simultancously learning semantic-vvistee mapping and data augmentation B1
25
22 33 However, since the inter-class relationships among unseen classes are not taken into account, model parameter transfer is often limited 77. Finally, common space learning learns common representation space into which both visual features and semantic representations are projected for effective knowledge transfer [34] 9
22
32
By calibrating the common space using bidirectionally aligned knowledge of the visual and semantic representations, it can simultaneously avoid the issues of direct mapping and model parameter transfer. As such, common space learning a promising methodology for ZSL. However, existing common space learning methods align the semantic and visual distributions with one-step adaptation, which neglects the distinct heterogeneous feature representations of the semantic and visual domains that are characterized by structure and distribution variations. This causes the distribution-alige alligned common space to be located in a different manifold, and samples to thus inevitably misclassifidd. In contrast, we propose an effective framework with both structure adaptation and distribution adaptation to address these challenges.
Domain Adaptation. Domain adaptation aims to learn domain-invariant representations by min- imizing the discrepancy between the distributions of source and target via distribution alignment, domain adversarial learning, or task-specific methods. It can be divided into homogeneous and heterogeneous domain adaptation according to the characteristics of the source and target data. In the homogeneous domain adaptation setting, the feature spaces of the source and target domains are identical, with the same dimensions. Thus, one-step adaptation can be employed [35] [36] 37). In contrast, in the heterogeneous setting, the feature spaces of the source and target domains are distant, and their dimensions also typically differ. Since there is little overlap between the two domains, one-step domain adaptation is not effective in the heterogeneous domain adaptation [38] 39]. Thus, multi-step (or transitive) domain adaptation methods are used for heterogeneous domain adaptation. However, these methods cannot be directly used in Z.L. Since ZSL has four domains (i.e., seen, unseen, semantic, and visual), knowledge cannot be transferred between two domains only, i.e., the distributions of the source and target domains cannot be aligned. We argue that the structure variation should also be taken into account when aligning the visual and semantic domains in ZSL.
3

---

CNN Backbone
Language Model
Visual Encoder
E
Semantic Encoder
B
Commoa Encoder
CZ.S1
Classäfier 1
CLS
Classifier?
D
Visual Decader
Semuntic Decoder
Manifol
Backbone Net
Strcture-Adppaaaoon
Cays
Cire
E
CLS
D
pis poo
E
a
Red winged
Fly
E
Structure aligged space
CLS Lere
Distribution aligned spaces
D
Canc
The framework of the proposed hierarchical semantic-visur adaptation (HSVA). HSVA consists of two partially-aligned variational autoencoders, which simultaneously perform structure adaptation and distribution adaptation with various constraints. Our two-step adaptation formulation helps HSVA learn structure and distribution-alignne aligned feature representations for visual and semantic in unified framework, which avoids the adverse effects of heterogeneous feature representations between distinct visual and semantic domains.
Figure 2:
Hierarchical Semantic- Vennnttc-Visual Adaptation
The task of ZSL is formulated as follows. Let S {(x y a²) x E A} be training set consisting of image features x extracted by a convolutional neural network (CNN) Backbone (e.g., ResNet-101),, where seen class embeddings a are learned by a language model and seen class labels y". In addition, an auxiliary training set U {y" a²) y" E Y") from unseen classes is used, where y" denotes unseen classes from a Y which disjoint from YS are the unseen visual features in the test set. Note that x = x² U x and a a Ua". The goal of conventional ZSL (CZSL) is to learn classifier for unseen visual features, i.e., fczsr. : X - yv, while for generalized ZSL (GZSL) the goal is to learn classifier for seen and unseen visual features, ie., faZsL: X + Y UYS.
X
In the following, we introduce our HSVA, which learns an intrinsic common space for semantic and visual feature representations to conduct ZSL. As shown in Figure 2 our framework consists of two partially-aligned autoencoders, which conduct structure adaptation (SA) and distribution adaptation (DA). In the end of this section, we demonstrate how we perform zero-shot learning using the feature representations extracted from the intrinsic common manifold space.
3.1 -Aliiall-AAligedd Variational Autoencoders
Our HSVA model consists of partially-aligned variational autoencoders. The first includes two feature encoders (visual encoder Ex and common encoder E²) and one decoder (visual decoder D*). E and E² are used learn structure and Iestributionaalinnne visual embeddings, respectively, which share two common spaces (i.e., structure and distribution-aligned spaces) with the semantic embeddings. D aims decode the distribution-aligned embeddings from the visual and semantic features to be similar to the visual features. Analogously, the second variational autoencoder has similar structure the first (i.e., includes semantic encoder E the E and semantic decoder D*), but it is employed to learn semantic embeddings. The partially-aligned variational autoencoders are first optimized by two variational autoencoders (VAE) losses:
( E*) (.), Elog D* -KL (x M) Ix))),
2 AE", = Ellog (2 "LLEKKKEEE""""
(E" (E"( (a'))) la²)).
(2)
where nn and na the visual and semantic embeddings in the distribution distribution-aligned space, y is the weight of the KL-divergence, and p(2" la*) and p(="<(=>>> (x=*) are the prior distributions assumed to be (0, 1). To make the visual and semantic embeddings in the structure- and distribution-aligner spaces more consistent, we then use cross reconstruction loss to constrain the two partially- -aiignd

---

variational autoencoders:
LREC(FE F", E², D² D _ REE [F" F²", D²) L PREC [E", F², FF", D
(3)
where
CR ,E,, "E", "D) = - -2"
(4)
CRE E D", = | -
(5)
Finally, the two partially- aliggned variational autoencoders conduct structure and distribution adapta- tion for the visual and semantic features, which will be introduced in the following.
3.2 Structure Adaptation
Structure adaptation (SA) is used to guide the encoder E and E to learn a structure- alligndd space for the visual and semantic features. Previous methods 77 II directly align the semantic and visual features with distribution distance constraints (e.g., MMD), which neglects the structure variation between the heterogeneous feature representations of the two domains. Furthermore, these methods fail to learn discriminative features as they do not take category relationships into account, as shown in Figure I (a). Motivated by task-specific unsupervised domain adaptation (36 37,, we propose a supervised adversarial discrepancy (SAD) learning to minimize the discrepancy between the outputs of two task-specific classifiers (CLS*, CLSS). This enables us learn class decision boundary and embed semantic and visual features into a structure-ale allignd common space for SA. Thus, SA can learn discriminative structure-alige aligned representations for semantic and visual features. Different from 36 37, which only detect the target samples (e.g., semantic embeddings (a)) to learn a single encoder (generator), SAD simultaneously detects the source samples (e.g., visual embeddings E (x)) and target samples (e-g., semantic embeddings E (a)) to learn good structure-aligned common space with two task-specific encoders. Specifically, SAD includes three steps for optimization: 1) semantic and visual embedding classification for E² E", CT.S* and CT.52; 2) discrepancy maximization for CLS and CLS²; and 3) discrepancy minimization for E and E".
Semantic and Visual Embedding Classification. We train LT L", CLS and CLS by minimizing the softmax cross- cross-entropy to collectly classify the structure-aligned semantic and visual embeddings.
This is crucial for enabling classifiers and encoders (E", E²) to obtain task-specife discriminative features. The visual and semantic classification loss is formulated as:
LoLs(E E", CLS', , CLS (CcLs(E"(u (E" (x²), CoLSa (E"(x"),
LCLSi
(6)
- LCLs' (E" y²) + LoLs (L (a"),,))))
where
LOLS (x, (xLL2(xx)) -E Σ |mmy log (y/ l
Here K is the number of seen classes, I[x=>] is an indicator function (i.e., it is one when k = y. x. We use two classifiers to simultaneously classify the visual and semantic embeddings to initialize the classifiers, which are used for discrepancy learning later.
otherwise zero), and p (y/x) is the prediction x..
Discrepancy Maximization for Classifiers. In this step, we freeze the parameters of the encoders (F**, E²) and update the classifiers (CLS), CI.52) to maximize the discrepancy between the outputs of the two classifiers on the visual and semantic embeddings. Thus, the source and target samples outside the support of task-specific decisions are identified, providing the positive signals to separate the class decision boundaries. Since sliced Wasserstein discrepancy (SWD) (37] provides geometrically meaningful guidance for capturing the natural notion of dissimilarity, we use it to measure the discrepancy between two classifier predictions for visual embeddings Pe(y") E (x*) (x*))) and semantic embeddings (3)" EE E
E (x*)),
(a'))))))
CDIS: (CLS), CCLS²) Lsw (L" (x")) (d"" Ry (z)l)))
(8)
LSwD (x) = Σ
(9)
5

---

where Rem is the mith one mie-dimensional linear projection operation, and @ is uniform measure on the unit sphere S(d-1) in IR. The larger the LDIS, the larger the discrepancy.
Discrepancy Minimization for Encoders. To encourage the visual embedding (E"(x*) and semantic embedding (E" (a*)) to be aligned well in the structure-aligned space, we freeze the parameters of the two classifiers and update the two encoders (E", E*) minimize the discrepancy between the outputs of the two classifiers on (x(x") and E (a"):
DDI (E", ,E) = (S + (a"))
(10)
This encourages the visual and semantic feature manifolds be closer each other, and thus the manifold structure variation is circumvented.
3.3 Distribution Adaptation
After structure adaptation, HSVA can learn structure-aligned common space avoid manifold structure difference between visual and semantic embeddings. However, the structure-aligned common space is still limited by distribution variation. To further learn distribution-aligned common space for visual and semantic feature representations, we propose distribution adaptation. Existing common space learning methods | (encoders) to align the visual and semantic distributions to learn distribution-alignne In contrast, our main idea is protect the pruuctre-alignee visual and semantic embeddings in a iistributtio-alignne space using a common encoder E², which preserves the structure-aligne aligned representations the DA. Our DA is first optimized by minimizing the Wasserstein distance between the latent multivariate Gaussian distributions, formulated as follows:
I0
M
employ two different mapping functions embedding.
EA,EE,E",EY, (1 1
11=1|(00)) -(00)))
(1I)
where |1 denotes the squared matrix Frobenius norm.
In essence, our HSVA also a generative model, which different to the GAN/VAE based methods 40 33 (i.e., learning mechanism and optimization). Due to the bias problem that arises when using generative models for ZSL, the synthesized unseen samples (E*(E* ()"))) in the distribution- aligned common space might be unexpectedly close to the seen ones. We propose to explicitly tackle this seen-unseen bias problem by preventing the encoded unseen samples (F* (a*()) from colliding with the encoded seen ones (E* (E* (x*))")) Since correlation alignment (CORAL) [11 has been shown effective for asymmetric transformations in domain adaptation, we take the CORAL- based metric to measure the discrepancy between seen and unseen class samples. Different from 41,, which takes CORAL to decrease the discrepancy between two domains, we aim to increase the discrepancy. Hence, we employ the inverse CORAL (iCORAL), defined upon the numerical negation of CORAL:
(E E E²) (C
(12)
3.4 Optimization
Our full model optimizes the partially-aligned variational autoencoders, DA and SA, simultaneously with the following objective function:
CHsVA(E" E., E,, CLL' CULS, E D",
D D", LiAE + AV + M + Lels + > + Cpist) + >a( + CDA), (13)
where A1. 22. Ag are the weights that control the importance of the related loss terms. Similar to the alternative updating policy for generative adversarial networks (GANs), we alternately train E E", CLS" CLS when optimizing SA. Although, HSVA has three components in total, the whole training process simultaneous and loss weights of all terms in Eq. 13 are the same for all datasets. The consistently significant results on all datasets show that our model is robust and easy to train.
"Since E (E*(2z) and E² (E (a")) are used to train ZSL classifier later, we take the visual features of seen classes and semantic features of unseen classes into account.

---

3.5 Classification
Once our HSVA model is learned, the visual and semantic features are encoded as new feature representations in a distribution-alieen aligned common space, where structure variation is aslo considered for classification. Using the reparametrization trick [42], we take E and E to encode the visual features of seen classes (x:*) and unseen classes (x**) into structure- and distribution-aligned feature representations, i.e., ==" E (E"x**) and zz E (x")). Analogously, we take E and E to encode the semantic embeddings (x* from the training set) and 2" to train supervised classifier (e.g., softmax). Once the classifier is trained, we use z" (x* from the test set) and z" to test the model. Note that our HSVA is an inductive method as we do not use the visual features of unseen classes for training.
d (a)), where 2" z" Uza" We employ z²
4 Experiments
Datasets. We conduct extensive experiments four well-known ZSL benchmark datasets, includ- ing fine-graines datasets (e.g., CUB [43] and SUN (44) and coarse grauned datasets (c.g.. AWA1 4 and AWA2 45).. CUB includes 11,788 images of 200 bird classes (seen/unseen classes 150/50) with 312 attributes. SUN contains 14,340 images from 717 scene classes (scen/unseen classes 645/72) with 102 attributes. AWAI and AWA2 contain 30,475 and 37,322 images from 50 animal classes (seen/unseen classes = 40/10) with 85 attributes.
Implementation Details. We the training splits proposed in 46 Meanwhile, the visual features are extracted from the 2048-dimensional top-layer pooling units of a CNN backbone (i.e., ResNet-101) pre-trained ImageNet. The encoders and decoders are multilayer perceptrons (MLPs). We employ the Adam optimizer [47] with 81 = 0.5 and 82 = 0.999. We use an annealing scheme [48 to increase the weights 7, A1, Ag Ag with same setting for all datasets. Specifically, 7 is increased by a rate of 0.0026 per epoch until epoch 90, a is increased from epoch 21 to 75 by 0.044 per epoch, and X2. λ3 are increased from epoch 0 to epoch 22 by a rate of 0.54 per epoch. In the CZSL setting, we synthesize 800, 400 and 200 features per unseen class to train the classifier for AWAI, CUB and SUN datasets, respectively. In the GZSL setting, we take 400 synthesized features per unseen class and 200 synthesized features per seen class to train the classifier for all datasets. The dimensions of the structure- aligned and distribution-alieee aliggned spaces are set 2048 and 64 for three datasets (i.e.. CUB, AWAI, AWA2), respectively, and 2048 and 128 for SUN benchmark.
During testing, follow the unified evaluation protocols proposed in 45 Specifically, we measure the top-1 1 accuracy of unseen class (Acc) for the C7SL. setting. In the GZSL setting, we take the top-1 accuracy on seen classes (S) and unseen classes (U), as well their harmonic mean (defined as H = (2 x S x US/(S + U)).
Evaluation Protocols.
AWA1
CUB
SUN Acc
Method
Ace
Acc
DeVisF(NeurIPS'3
34
54.2
52.0
56.5
DCN(NeurlPS'15 18)
10
65.3
56.2
61.8
VADA-VAECCVPR 19)
11
63.0
59.8
61.7
Our HSVA
70.6
62.8
63.8
Table Results of common space learning methods under the CZSL setting on three datasets.
4.1 Experimental Results
Results of Conventional Zero-Shot Learning. [ the results of CZSL on various datasets. Our HSVA, with softmax classifier, significantly outperforms other common space learning methods [34 I0 II by at least 5.3%. 3.0% and 2.0% on AWAI, CUB and SUN, respectively. Our significant performance improvements demonstrate that hierarchical semantic-visual adaptation can effectively learn an intrinsic common space to represent visual and semantic features. This space is structure- and distribution-aligeed and alleviates the gap between heterogeneous feature representations.
7

---

AWA1
AWA2
CUB
SUN
Method
U
S
H
U
S
H
CRne((CCLL19)
49
58.1
74.7
65.4
45.5
56.8
50.5
34.1
36.5
35.3
PC/SLICCPR'1
50
31.7
70,9
43.8
43.2
51.4
46.9
35.1
35.3
35.2
1CNUCCV'19
51
49.4
76.5
60.0
61.2
65.8
63.4
52.6
52.0
52.3
31.2
37.3
34.0
Non-common space
SGMA 19)
52
37.6
87.1
52.5
36.7
71.3
48.5
DVBEICVPR 20)
53
63.6
70.8
67.0
53.2
60.2
56.5
45.0
37.2
40.7
DAZLEICVPR'20)
60.3
75.7
67.1
59.6
56.7
58.1
52.3
24.3
33.2
RGENCCCCV
55
67.1
76.5
71.5
60.0
73.5
66.1
44.0
31.7
36.8
CompserrNeurlPS" 20)
26
62.1
77.3
68.8
63.8
56.4
59.9
55.1
22.0
31.4
APN(NeurIPS 25
56.5
78.0
65.5
65.3
69.3
67.2
41.9
34.0
37.6
FCLSWGANICVPRR 18)
16
57,9
61.4
59.6
43.7
57.7
49.7
42.6
36.6
39.4
EVAEGANICVPR 19)
51
57.6
70.6
63.5
48.4
60.1
53.6
45.1
38.0
41.3
LisGAN(CVPR
56
52.6
76.3
62.3
46.5
57.9
51.6
42.9
37.8
40.2
LsCGAN((CCC2200
54.6
74.6
48.1
53.0
44.8
37.7
40.9
AGZSL(ICLR'2) 26
65.1
78.9
71.3
41.4
49.7
45.2
29.9
40.2
34.3
Comnnn space
DeViSE/NeurIPS 133
144
13.4
68.7
22.4
17.1
74.7
27.8
23,8
53.0
32.8
16.9
27.4
20.9
ReViSE(ICCV"
46.1
37.1
41.1
46.4
39.7
42.8
37.6
28.3
32.3
24.3
20.1
22.0
DCN(NeurIPSS 18)
10
25.5
84,2
39.1
28.4
60.7
38.7
25.5
37.0
30.2
CADA-VAEICVIE 19)
57.3
72.8
64.1
55.8
75.0
63.9
51.6
53.5
52.4
47.2
35.7
40.6
(GALLNNeuIPSS 19
52.7
74.0
61.5
52.5
86.3
65.3
40.9
55.3
47.0
35.5
34.4
34.9
Our HIVA
59.3
76.6
66.8
56.7
79.8
66.3
52.7
58.3
55.3
48.6
39.0
43.3
Table 2: State-of-the-art comparison under the GZSL setting on four datasets. The hest and second- best results are marked in red and blue, respectively. The "Common space" and "Non-common space" denote common space learning and non-common space learning methods, respectively. Meanwhile, the non-common space learning methods include the non-generative methods and generative methods.
AWAI
AWA2
CUB
SUN
Method
8
H
U
S
U
H
U
H
HSVA w/o LSa HSVA w/o LDA and LICORAL
55.2
71.5
62.3
53.9
71.2
63.5
49.3
57.9
53.2
49.7
37.2
42,6
31.3
70.6
43,4
26.3
78.4
39.3
41.9
56.1
47.9
42.2
35.5
38.6
HSVA wlo LICORAL
54.3
73.5
62.4
51.3
82.6
63.3
51.1
$7.0
53.9
45.1
38.9
41.8
HSVA (full)
59.3
76.6
66.8
56.7
79.8
66.3
52.7
58.3
55.3
48.6
39.0
43.3
Table 3: Ablation study of HSVA under the GZSL setting on four datasets. "HSVA w/o Lsa" denotes HSVA without the constraints in SA, "HSVA w/o LD and LicORai." denotes HSVA without the constraints in DA. "HSVA w/o LicORaL" denotes HSVA without LICORAL. and "HSVA (full)" denotes our full model HSVA.
Results of Generalized Zero-Shot Learning. Table shows the GZSL performances of various methods, including non-generative methods (direct mapping and model parameter transfer), generative methods, and common space learning methods. Compared to other common space learning models 94.9 IO 19, our HSVA achieves significant improvements of least 2.7%. 1.0%, 2.9% and 2.7% in harmonic mean on AWAI, AWA2, CUB and SUN, respectively. Especially, our HSVA outperforms the latest common space learning method (i.c., SGAL (19) by a large margin, resulting in harmonic mean improvements of 5.3%, 1.0%, 8.3% and 8.4% on AWAI, AWA2, CUB and SUN, respectively. Since our method can also act as a generative method, we compare it with other of the-art generative methods. The results show that our method performs significantly better on AWAI, CUB and SUN datasets. Our model also achieves competitive results compared to non non-generative methods. For example, HSVA performs best results (e.g., the harmonic mean of 43.3) on SUN that is the most challenging benchmark ZSL. These results consistently demonstrate the superiority and great potential of hierarchical semantic-vise visual adaptation.
Ablation Study. We conduct ablation studies to evaluate the effectiveness of our HSVA in terms of the structure adaptation constraints (denoted as Cs distribution adaptation constraints (i.c., LDA + L.CORAl), and only inverse CORAL LiCORAL- Our results are shown in Table HSVA performs poorer than its full model when no constraints are used during structure adaptation, i.e., the harmonic mean drops by 4.5% on AWA1 and 2.1% on CUB. constraints are not used during distribution adaptation. HSVA achieves very poor results compared to its full model, i.e., the harmonic mean drops by more than 23 23.4% on coarse-grained datasets and 4.7% on fine-grained datasets. These results show that distribution alignment is essential for common space learning, while structure adaptation is necessary for semantic-visual adaptation. This typically neglected by distribution alignment methods with one-step adaptation 8
10
19]. Morcover, LiCORAL pushes
8

---

CADA-NAE: classes
CADAVAE: unseen classes
HSVA: classes
HSVA: unseen classes
(a) CUB
CADA-VAE: seen classes
CADAVIE: unseen classes
HSV:: seen classes
HSWX unseen classes
(b) AWAI
Figure 3: t-SNE visualization 14] of visual/semantic features learned by our HSVA and CADA- VAE [T]] for the same seen and unseen classes. Different colors denote different classes, The "o" and "x" indicate visual and semantic features, respectively. We conduct experiments on 10 classes of (a) CUB and (b) AWAI. CADA- VAE clearly maps the visual and semantic features into various manifolds focusing on distribution alignment, while HSVA learns intrinsic common space for discriminative feature representations by adopting sequential structure and distribution adaptation.
unseen classes away from seen classes, which explicitly tackles the seen-unseen bias problem. HSVA cooperates with iCORAL AL to improve the accuracy of classification using softmax, achieving harmonic mean improvement of 4.4%, 3.0%, 1.4%, and 1.5% on AWA1, AWA2, CUB, and SUN, respectively.
Qualitative Results. We first qualitatively investigate the difference between our HSVA and one- step adaptation methods (e.g.. CADA-VAE (I]) for common space learning. We conduct t-SNE visualization 14) of visual/semantic features mapped by our HSVA and CADA-VAE on 10 classes from (a) CUB and (b) AWA1. As shown in Figure 3 CADA-VAR maps semantic and visual features into different manifolds, which confuses the visual and semantic features of various categories, resulting in misclassification for some samples. This intuitively shows that the existing common space learning methods neglect the different structures of the heterogeneous feature representations the visual and semantic domains. In contrast, HSVA learns an intrinsic common space that better bridges the gap between the heterogeneous feature representations for effective knowledge transfer, using structure and distribution adaptation. Thus, hierarchical semantic-visual adaptation is an effective solution for common space learning in ZSL.
In addition, unlike other methods 8 9 10 11] that can only align visual and semantic distribu- tions, our HSVA can take class relationships into account to learn discriminatively aligned feature representations. As shown in Figure 3 CADA-VAE maps visual and semantic features into compact manifolds, resulting in confused class relationships. In contrast, HSVA learns discriminative structure- and distribution-align -liinned features for the visual and semantic feature representations, which enables HSVA VA to achieve better ZSL classification.
5 Conclusion
In this work, we develop zero-shot learning framework, i.e. hierarchical semantic-visual adaptation (HSVA), to learn an intrinsic common space for visual and semantic feature representations. By conducting structure and distribution adaptation using two partially-aligned variational autoencoders, our model effectively tackles the problem of heterogeneous feature representations and bridges the gap between the visual and semantic domains. We demonstrate that HSVA achieves consistent improvement over the current state-of-the-ar methods on four ZSL benchmarks. We qualitatively verify that our hierarchical semantic-visual adaptation framework can align the visual and semantic domains in terms of both structure and distribution, while other common space learning methods only

---

take distribution alignment into account. This intuitively shows that the learned feature representations of HSVA are more discriminative and accurate than other methods. To the best of our knowledge, we are the first to investigate the different structural characteristics of the visual and semantic domains in Z.S.. We believe that our work will facilitate the development of stronger visual-and-language learning systems, including zero-shot learning, visual question answering [57], image captioning [58. natural language for visual reasoning [59], heterogeneous domain adaptation [60].
Acknowledgements
This work is partially supported by NSFC (61772220), Special projects for technological innovation in Hubei Province (2018ACA135) and Key R&D Plan of Hubei Province (2020BAB027).
References
[1] C. H. Lampert, H. Nickisch, and S. Harmeling, "Learning detect unseen object classes by between-class attribute transfer," in CVPR, 2009, pp. 951-958.
[2] H. Larochelle, D. Erhan, and Y. Bengio, "Zero-data learning of new tasks," in AAAI, 2008, pp. 646-651.
[3] M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell, "Zero-shot learning with semantic output codes," " in NeurIPS, 2009, pp. 1410-1418.
[4] C. H. Lampert, H. Nickisch, and S. Harmeling, "Attribute- based classification for zero-shot visual object categorization," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, pp. 453-465, 2014.
[5] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality." in NeurlPS, 2013, pp. 3111-3119.
[6] S. Reed, Z. Akata, H. Lee, and B. Schiele, "Learning deep representations of ane-grained visual descriptions," " in CVPR, 2016, pp. 49-58.
[7] Q Wang and K. Chen, "Zero-shot visual recognition via bidirectional latent embedding." International Journal of Computer Vision, vol. 124, pp. 356-383, 2017.
[8] Z. Zhang and V. Saligrama, "Zero-shot learning via via joint latent similarity embedding." in CVPR. 2016, pp. 6034-6042.
[9] Y.H.. H. Tsai, L.-K. Huang, and R. Salakhutdinov, "Learning robust visual-semantic embed- dings," in ICCV, 2017, pp. 3591-3600.
[10] S. Liu, M. Long, J. Wang, and M. I. Jordan, "Generalized zero-shot learning with deep calibra- tion network." " in NeurlPS, 2018.
[11] E. Schönfeld, S. Ebrahimi, Sinha, T. Darrell, and Akata, "Generalized zero- and few-shot learning via aligned variational autoencoders," in CVPR, 2019, pp. 8239-8247.
[12] J. Huang, S. Lu, D. Guan, and X. Zhang. "Contextuull-reaatoo consistent domain adaptation for semantic segmentation." " in ECCV, 2020.
[13] J. Huang, D. Guan, A. Xiao, and S. Lu, "Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data," arXiv preprint arXiv:21 10.03374, 2021.
[14] L. V. D. Maaten and E. Hinton, "Visualizing data using t-sne," Journal of Muchine Learning Research, vol. 9, pp. 2579-2605. 2008.
[15] H. Li, S. J. Pan, R. Wan, and A. Kot, "Heterogeneous transfer learning via deep matrix completion with adversarial kernel embedding." in AAAI, 2019.
[16) Y. Luo, Y. Wen, T. Liu, and D. Tao, "Transferring knowledge fragments for learning distance metric from a heterogeneous domain," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, pp. 1013-1026. 2019.
[17] Z. Fu, T. Xiang. E. Kodirov, and S. Gong, "Zero-shot learning semantic class prototype graph." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, pp. 2009-2022, 2018
[18] Z. Fu, F. Kodirov, Xiang, and S. Gong. "Zero-shot object recognition by semantic manifold distance," in CVPR, 2015, pp. 2635-2644.
10

---

[19] H. Yu and B. Lee, "Zero-shot learning via simultaneous generating and learning," in NeurlPS, 2019.
[20] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, "Label-embedding for image classifica- tion," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, pp. 1425-1438, 2016.
[21] R. Felix, B. V. Kumar, I. Reid, and G. Cameiro, "Multi-modal cycle-consistent generalized zero-shot learning," in ECCV, 2018.
[22] M. R. Vyas, H. Venkateswara, and S. Panchanathan, "Leveraging seen and unseen semantic relationships for generative zero-shot learning," in ECCV, 2020.
[23] D. T. Huynh and E. Elhamifar, "Compositional zero-shot learning via fine-grained dense feature composition," NeurIPS, 2020.
[24] G.-S. Xie, L. Liu, X. Jin, F. Zhu, Z. Zhang. J. Qin, Y. Yao, and L. Shao, "Attentive region embedding network for zero-shot learning," in CVPR, 2019, pp. 9376-9385.
[25] W. Xu, Y. Xian, J. Wang, B. Schiele, and 7. Akata, "Autribute prototype network for zero-shot learning," in NeurlPS, 2020.
[26] T.-L. L. Yu- Yunging Chou, Hsuan- Lin, "Adaptive and generative zero-shot learning," in ICLR, 2021.
[27] Z. Akata, S. Reed, D. Walter, Lee, and B. Schiele, "Evaluation of output embeddings for fine- fine-grained image classification," in CVPR, 2015, pp. 2927-2936.
[28] K. Li, M. R. Min, and Y. Fu, "Rethinking zero-shot learning: A conditional visual classification perspective," " in ICCV, 2019, pp. 3582-3591.
[29] C. Gan, M. Lin, Y. Yang. Zhuang, and Hauptmann, "Exploring semantic inter-class relationships (sir) for zero-shot action recognition," in AAAI, 2015.
[30] S. Changpinyo, W.-L. Chao, B. Gong, and F. Sha, "Synthesized classifiers for zero-shot learning," " in CVPR, 2016, pp. 5327-5336.
[31] Y. Xian, S. Sharma, B. Schicle, and Z. Akata, -F-vacgan-d2: A feature generating framework for any-shot learning," in CVPR, 2019, pp. 10267-10276.
[32] Y. Yu, Ji, Han, and Z. Zhang. "Episode- prototype generating network for zero-shot learning." in CVPR, 2020, pp. 14 (33211404)
[33] S. Chen, W. Wang, B. Xia, Q Peng. X. You, F. Zheng, and L. Shao, "Free: Feature refinement for generalized zero-shot learning," " in ICCV, 2021.
[34] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, "Devise: deep visual-semantic embedding model," NeurlPS, 2013.
[35] M. Long, Z. Cao, J. Wang, and M. I. Jordan, "Conditional adversarial domain adaptation,"" in NeurIPS, 2018.
[36] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada, "Maximum classifier discrepancy for unsu- pervised domain adaptation,"" " in CVPR, 2018 pp. 3723-3732.
[37] C.-Y. Lee, Batra, M. H. Baig, and Ulbricht, "Sliced wasserstein discrepancy for unsuper- vised domain adaptation," in CVPR, 2019, pp. 207
[38] B. Tan, Y. Song, F. Zhong, and Q Yang, "Transitive transfer learning." in KDD, 2015.
[39] B. Tan, Y. Zhang, S. J. Pan, and Q. Yang, "Distant domain transfer learning," in AAAI, 2017.
[40] Narayan, A. Gupta, Khan, C. G. M. Snoek, and I.. Shao, "Lateet embedding feedback and discriminative features for zero-shot classification," in ECCV, 2020.
[41] B. Sun, J. Feng, and K. Saenko, "Return of frustratingly easy domain adaptation," in AAAI, 2016.
[42] D. P. Kingma and M. Welling, "Auto-encoding variational bayes," in ICLR, 2014.
[43] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff. J. Belongie, and P. Perona, "Caltech-uesd birds 200," Technical Report CNS-TR-28 -NS-TR-2010-00,, Caltech. 2010.
[44] G. Patterson and J. Hays, "Sun attribute database: Discovering, annotating, and recognizing scene attributes." in CVPR, 2012, pp. 2751-2758.

---

[45] Y. Xian, B. Schiele, and Z. Akata, "Zero-shot learning - the good, the bad and the ugly," Proc. CVPR, pp. 3077-3086, 2017.
[46] Y. Xian, T. Lorenz, B. Schiele, and Z. Akata, "Feature generating networks for zero-shot learning," in CVPR, 2018, pp. 5542-5551.
[47] D. Kingma and J. Ba, "Adam: A method for stochastic optimization." in JCLR, 2015.
[48] S. Bowman, Vilnis, O. Vinyals, M. Dai, Józefowicz, and S. Bengio, "Generating sentences from continuous space," in CoNLL, 2016.
[49] F. Zhang and G. Shi, "Co-representation network for generalized zero-shot learning," in ICML,
2019.
[50] J. Li, X. Lan, Y. Liu, L. Wang, and N. Zheng, "Compressing unknown images with product quantizer for efficient zero-shot classification, in CVPR, 2019, pp. 5458-5467.
[51] H. Jiang, Wang. Shan, and X. Chen, "Transferable contrastive network for generalized zero-shot learning," in ICCV, 2019, pp. 9764-9773.
[52] Y. Zhu, J. Xie, Z. Tang, X. Peng, and A. Elgammal, "Semantic-guided multi-attention localiza- tion for zero-shot learning," in NewrIPS, 2019.
[53] S. Min, H. Yao, H. Xie, C. Wang, Z. Zha, and Y. Zhang, "Domain-aware visual bias eliminating for generalized zero-shot learning." in CVPR, 2020, pp. 12 661-12677
[54] D. Huynh and E. Elhamifar, FFine-grained generalized zero-shot learning via dense attribute- based attention," in CVPR, 2020, pp. 4482 4492.
[55] G.-S. Xie, L. Liu, X. Jin, F. Zhu, Z. Zhang, Y. Yao, J. Qin, and L. Shao, "Region graph embedding network for zero-shot learning," in ECCV, 2020.
[56] J. Li, M. Jing, K. Lu, Z. Ding, L. Zhu, and Z. Huang, "Leveraging the invariant side of generative zero-shot learning," in CVPR, 2019, pp. 7394-7403.
[57] S. Antol, A. Agrawal, Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, "Vqa: Visual question answering," in ICCV, 2015.
[58] S. Herdade, A. Kappeler, K. Boakye, and J. Soares, "Image captioning: Transforming objects into words," " in NeurIPS, 2019.
[59] X. Li, X. Yin, C. Li, X. Hu, P. Zhang, L. Zhang, L. Wang. H. Hu, L. Dong, F. Wei, Y. Choi, and J. Gao, "Oscar: Object-semantee aligned pre-training for vision-language tasks," in ECCV, 2020.
[60] S. Li, B. Xie, J. WI, Y. Zhao, C. Liu, and Z. Ding. "Simultaneous semantic alignment network for heterogeneous domain adaptation," in ACM MM, 2020.
12

---

Checklist
For all authors...
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See the abstract and introduction.
Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
Did you discuss any potential negative societal impacts of your work? [Yes]
Did you describe the limitations of your work? [No]
2. If you are including theoretical results...
Did you state the full set of assumptions of all theoretical results? [No]
Did you include complete proofs of all theoretical results? [No]
3. If you ran experiments..
Did you include the code, data, and instructions needed reproduce the main experimental results (either in the supplemental material URL)? [Yes] We anonymously released our pre-trained models and test code at (tthttp::/// ptstps://ge htt///gittuu com/ anonymonnnmmuuamm ssubmi tion/
HSVA-NeurIPS IPS-21
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
Did you report error bars (e.g., with respect the random seed after running experiments multiple times)? [No]
Did you include the amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
If your work uses existing assets, did you cite the creators? [Yes]
Did you mention the license of the assets? [No]
Did you include any new assets either in the supplemental material or URL? [Yes] We provided more experimental analyses in the supplemental material. We also
anonymously released our pre-trained models and test code at http//gtttuuu.... com/
ssonymou-ssuumi wttp:/oowwsssomtiiiiootto////t IPrr--2211
Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [No]
Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
If you used crowdsourcing or conducted research with human subjects...
Did you include the full text of instructions given to participants and screenshots, if applica-
ble? [No]
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [No]
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No]
Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [No]
13


Reviews:
[{"review_id": "kp8ZHk0I_fV", "reviewer": "Reviewer_Nr4Y", "summary": "This paper proposes an hierarchical semantic-visual adaptation framework to migrate the distribution and structure disagreement in generative zero-shot learning (ZSL) models. This network adopts hierarchical two-step adaptation. The task-specific adaptation is complemented by training two task-specific encoders with a supervised adversarial discrepancy module. The distribution adaptation is achieved by minimizing the Wasserstein distance between visual and semantic distributions. Quantitative results on four widely used benchmarks indicates the effectiveness of the proposed method. Qualitative visualization shows the network learns intrinsic common space that better bridges the gap between the heterogeneous feature representations. \t\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality] The proposed two-step adaptation is novel and seems interesting. The model deals with the “structure variation” problem in one-step adaptation. However, the related work part misses comparison with two important GAN and VAE based related work [1r,25]. The author is encouraged to fully discuss the distinction of the proposed method and state out the novelty. \n \n[Clarity] The paper is very well organized and easy to follow. However, the definition of “distribution” and “structure” variation between semantic and visual features is not well explained, which hinders the motivation of this paper.\nFor instance, in Figure 1 the author argues that one-step adaptation causes the semantic and visual distributions to be located in different manifolds, however, it seems like the two distributions are located on the same manifold from the t-SNE visualization in (c). And the main problem in (c) seems like high inner-class variance and low inter-class variance.\n\n[Quality] The effectiveness of the proposed components is verified by ablation study and qualitative results. However, I have one concern over the architecture and the experiment design the “structure adaptation” section.\nIn the structure adaptation section, two “task-specific” classifiers are proposed to classify feature and semantic features into the corresponding category and a SAD module is proposed to minimize the discrepancy between the classifiers’ outputs. However, there lack of explanation for this specific design since SA is the most distinct part of this paper compared to other models [29,30,11].\n1. What structure do the two classifiers represent? Why choose two classifiers, not three or five?\n2. It seems like the input and constraints of the two classifiers are exactly the same. If the two classifiers are initialized with the same parameter, will they turn out to be the same? Then why would we need two classifiers? Will one classifier also work? Several ablations for the SA section are missing. For instance, what is the performance of (1) using two classifiers without $L_{DIS}$, (2) using one or three classifiers?\n\n[Significance]\nQuantitative results on four widely used benchmarks indicate the effectiveness of the proposed method. However, there are two concerns towards the qualitative results shown in Figure 3.\nShowing qualitative ablation on the proposed loss functions will shed more insights on the contribution of this work, e.g., what’s the effect of removing SA or DA on Figure 3?\nHow does the author choose the 10 classes in Figure 3? It seems that CADA-VAE is much worse than the proposed HSVA qualitatively. However, from quantitative results, CADA-VAE is achieving comparable results (only 2-3% lower) with the proposed method. Why is this happening? Does it mean that the quality of feature distribution is not the main factor of high ZSL accuracy?\n\n[1r] Narayan, Sanath, et al. \"Latent embedding feedback and discriminative features for zero-shot classification.\" ECCV, 2020.\n\n\n[Post-rebuttal]  The authors have addressed my concerns. After reading other reviewers' comments, I agree to accept the paper as it is interesting and sound.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper deals with an interesting problem but some concerns are unsolved and make the contribution somehow weak (see detailed comments as before). I would like to raise my rating if the author’s feedback properly addresses the concerns.\n\nFollowing are minor questions:\n\nLine 113: “ $a^s$ are learned by a language model and …” is not accurate. In this paper, $a^s$ is collected from human annotation not learned from the language model.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "There is no negative societal impact of this work.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ciCP1Pdsl52", "reviewer": "Reviewer_JPqU", "summary": "In zero-shot learning (ZSL) problem, feature representations of distinct semantic and visual domains are intrinsically heterogeneous, thus having both distribution and structure variations. To this problem, this paper proposes a novel HSVA framework that aligns the semantic and visual domains by structure adaptation (SA) and distribution adaptation (DA). SA, consisting of two encoders, aims to encode the visual data and semantic data into a structure-aligned common space, by using a SAD module. Then, DA further explores a common encoder to align the visual and semantic distributions. This paper also conducts extensive experiments to show the effectiveness of the proposed framework for both conventional ZSL and generalized ZSL settings. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n- Significance and novelty: The main idea of this paper is to learn an intrinsic common space in which the visual and semantic features can be aligned from the perspective of both structure and distribution. The idea of two-step semantic-visual adaptation seems interesting and potentially inspirational for other ZSL works. \n- The authors conducted extensive experiments on benchmarks to show its effectiveness. \n- The t-SNE visualization of visual/semantic features in Figure 3 intuitively show the manifold of the feature space.  \n\nWeakness: \n- Model complexity: Though the idea seems novel, the model is somewhat cumbersome to me, e.g. the objective function has 8 terms and 4 hyper-parameters (include $\\gamma$). It may be very challenging to find the optimal hyper-parameters for each specific dataset. \n- Data sets and training: In addition, in benchmarks like AwA, each class has a unique attribute vector and multiple training samples. Whether the imbalance between visual domain and semantic domain affects the training process also requires discussion. \n- Experiments and performance: From the experimental results in Table 2, it can be seen that the proposed method has no advantage in performance comparison on the fine-grained data set, i.e. CUB. Some analysis of the above phenomena may contribute to a better understanding of the proposed method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "Xtn2eb2ihhe", "reviewer": "Reviewer_jKPX", "summary": "The paper proposes a new two-step, hierarchical adaptation approach for Zero-Shot Learning. The first step is structure adaptation where visual data and target/semantic data (eg attributes) are encoded into an aligned common space. This is similar to prior work, but is accomplished using a new module that adversarially minimizes the discrepancy between the visual and semantic-based classifiers. The second step is distribution adaptation which seeks to align the visual and semantic feature distributions using a common encoder.\n\nThe paper ablates the contribution of both structure and distribution alignment on several ZSL benchmarks, showing significant improvements. It also compares favorably against all prior common space ZSL methods on all benchmarks, and against all prior work (common and non-common space) on several benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern is with the complexity of the approach, and the lack of ablation studies to identify which components are most important.  However, the final loss function is quite complicated. Although all components seem to be conceptually sound, given the complexity of the overall method (and the introduction of several modifications to prior work), I believe the authors should have conducted more detailed ablation studies. For example, Ln 146 mentions that prior work uses MMD for structure alignment, but this paper experiments with a new module SAD. This change should have been empirically validated. Does the adversarial approach improve generalization?\n\nI hope the authors can provide more insights into additional components of the proposed method, as I will reassess my score based on it.\n\n-------\n\nPost rebuttal:\n\nAfter carefully considering the comments of all reviewers, as well as the rebuttal, I'm still leaning towards acceptance.\nThe paper introduces an interesting and well-motivated development over VAE-style approaches for ZSL, and achieves good results on standard ZSL benchmarks.\nMy main concern with respect to this paper was the lack of ablations, given the complexity of the training schema. I appreciate that the authors try to address this concern in rebuttal. The provided ablations make sense and cast light on the advantages of different components of the method. I strongly encourage the authors to add these ablations to the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper aims to improve zero-shot learning (an established research area). As such, it does not introduce new negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours split over 2 days"}, "extra_text": {}}, {"review_id": "SeuoslXgguc", "reviewer": "Reviewer_TyLe", "summary": "This paper proposes a new method for zero-shot learning. It enforces the classes are well aligned by using a domain-specific discrepancy on an intermediate space of the encoder. The method is evaluated on benchmark datasets and achieves good results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel method for ZSL. It integrates a two-step encoding process to align the representations from visual and semantic spaces. Compared to previous methods in the same category, this method has an additional encoder that maps the visual features and semantic attributes to a common space. In this common space, it leverages the sliced Wasserstein discrepancy to learn the decision boundary for each domain, such that the encoding can be better aligned. In the empirical evaluation, the proposed method shows superior performance to the corresponding one-step methods. \n\nThe originality of this paper is kind of weak to me because the proposed method is a combination of existing works. It integrates the sliced Wasserstein discrepancy into the CADA-VAE framework by introducing an additional encoding layer. It also integrates CORAL to improve the class-specific discrepancy in the distribution alignment stage. These integrations are rather empirical. It is good to see composing these ideas together can improve the CADA-VAE framework, but it makes the contribution of this paper less significant. \n\nMoreover, the proposed method has large performance gaps between state-of-the-art methods on popular benchmark datasets like AWA2 and CUB. This also makes an empirical method less significant.\n\nThe writing of this paper is not very good, and it is hard to follow for some parts.\n\n-----------------\nPost-rebuttal\nAfter reading the authors' response and other reviewers' comments, I agree that the proposed method in this paper is interesting and more than just a stack of existing works. And it is also different from previous works. So I raise my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper didn't discuss the limitation or potential negative social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a two-stage adaptation approach, i.e., structure adaptation (SA) and distribution adaptation (DA), for closing the gap between the visual (image) feature space and the semantic (e.g., attributes) feature space, resulting in a common space that not only captures the manifold structure but also induces the discriminative property. With the learned common space, (G)ZSL can be easily fulfilled, and very promising experimental results on four ZSL benchmarks are achieved. 

This work is motivated and posed on the foundation of CADA-VAE [11]. Through the authors’ rebuttal as well as the insightful discussion among the reviewers, the AC thinks that the improvement over [11] brought by the proposed HSVA is sufficient in terms of both methodology and performance. The authors proffered a good rebuttal, where more ablation studies are shown to demonstrate the effectiveness of HSVA. Taking together all the detailed comments, the AC votes for accepting the paper and would believe that it can advance the research of ZSL.  In the camera-ready version, the authors are suggested to: 1) strengthen the writing, 2) supplement the key ablation results in the main paper, and 3) highlight the differences from the existing works and clarify the necessity of the overall objective in Eq. (13).       

Decision: Accept (Poster)