HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning
Year: 2022

Abstract:
Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. Typically, to guarantee desirable knowledge transfer, a common (latent) space is adopted for associating the visual and semantic domains in ZSL. However, existing common space learning methods align the semantic and visual domains by merely mitigating distribution disagreement through one-step adaptation. This strategy is usually ineffective due to the heterogeneous nature of the feature representations in the two domains, which intrinsically contain both distribution and structure variations. To address this and advance ZSL, we propose a novel hierarchical semantic-visual adaptation (HSVA) framework. Specifically, HSVA aligns the semantic and visual domains by adopting a hierarchical two-step adaptation, i.e., structure adaptation and distribution adaptation. In the structure adaptation step, we take two task-specific encoders to encode the source data (visual domain) and the target data (semantic domain) into a structure-aligned common space. To this end, a supervised adversarial discrepancy (SAD) learning is proposed to adversarially minimize the discrepancy between the predictions of two task-specific classifiers, thus making the visual and semantic feature manifolds more closely aligned. In the distribution adaptation step, we directly minimize the Wasserstein distance between the latent multivariate Gaussian distributions to align the visual and semantic distributions using a common encoder. Finally, the structure and distribution adaptation are derived in a unified framework under two partially-aligned variational autoencoders. Extensive experiments on four benchmark datasets demonstrate that HSVA achieves superior performance on both conventional and generalized ZSL. The code is available at https://github.com/shiming-chen/HSVA .

heading: 1 Introduction
text: Over the past decade, significant progress has been made in zero-shot learning (ZSL), which aims to recognize new classes during learning by exploiting the intrinsic semantic relatedness between seen and unseen categories [1, 2, 3]. Inspired by the way humans learn unknown concepts, sideinformation (e.g., attributes [4], word vectors [5], and sentences [6]) related to seen/unseen classes is employed to guarantee knowledge transfer between the seen/unseen data. Common space learning is one typical method for representing the relationship between the visual and semantic domains for ZSL, which is key to dealing with the knowledge transfer [7]. However, existing common space
∗Corresponding author
35th Conference on Neural Information Processing Systems (NeurIPS 2021), virtual.
learning methods align the distributions of the semantic and visual domains with one-step adaptation [8, 9, 7, 10, 11, 12, 13], while neglecting the fact that the heterogeneous feature representations of distinct semantic and visual domains have both distribution and structure variations. Thus, their performance is limited. In this work, we propose a novel common space learning formulation that leverages a hierarchical semantic-visual adaptation to learn an intrinsic common space for a better alignment of the two heterogeneous representation domains.
The heterogeneous feature representations of the semantic and visual domains are distinct [15, 16]. Although one-step adaptation learns distribution-aligned feature representations for the semantic and visual domains using two different mapping models (e.g., encoders) and distribution alignment constraints (e.g., maximum mean discrepancy (MMD)), the two distributions are located in different manifolds since the one-step adaptation does not consider the manifold structure relationship between visual and semantic representations, as shown in Figure 1 (a) and (c). If we take the Euclidean distance or manifold distance [17, 18] to measure the relationship between different classes, the classifier inevitably misclassifies some samples, thus leading to inferior ZSL performance. As such, mapping the semantic and visual domains into an intrinsic and desirable common space for conducting an effective ZSL is highly necessary.
To learn structure- and distribution-aligned feature representations for distinct visual and semantic domains, we propose a hierarchical semantic-visual adaptation (HSVA) framework to learn an intrinsic common space that contains the essential multi-modal information associated with unseen classes (Figure 1 (b) and (d)). HSVA aligns the semantic and visual domains with structure adaptation (SA) and distribution adaptation (DA) in a unified framework, under two partially-aligned variational autoencoders. In the SA, we use two task-specific encoders to encode source data (visual domain) and target data (semantic domain), respectively, into a structure-aligned space, which is learned through supervised adversarial discrepancy (SAD) learning to minimize the discrepancy between
the predictions of the two task-specific classifiers. This encourages the visual and semantic feature manifolds to be closer to each other, thus aligning the manifold structure variation. In the DA, we map the structure-aligned features into a distribution-aligned common space using a common encoder, which is optimized by minimizing the Wasserstein distance between the latent multivariate Gaussian distributions of the visual and semantic domains. The common encoder preserves the structure-aligned representations in the DA. Thus, HSVA can learn a structure and distributionaligned common space for visual and semantic feature representations, which is better than the distribution-aligned feature representations learned by existing common space learning methods [8, 9, 7, 10, 11, 19]. To the best of our knowledge, this is the first work that leverages hierarchical semantic-visual adaptation to address heterogeneous feature alignment in ZSL. Following [11], we conduct extensive experiments on four challenging benchmark datasets, i.e., CUB, SUN, AWA1, and AWA2, under both conventional and generalized ZSL settings. The proposed HSVA achieves consistent improvement over the existing common space learning methods on all datasets. We show qualitatively that the structural variation is important for the interaction between visual and semantic domains. Moreover, we show significantly better common space for visual and semantic feature representations than the existing common space learning methods, e.g., CADA-VAE [11].

heading: 2 Related Work
text: Zero-Shot Learning. In practice, the relationship between visual and semantic domains is determined by learning an embedding space where semantic vectors and visual features interact [20, 21, 22, 23, 24, 25, 26, 19]. There are three mapping methods for learning such an embedding space, including direct mapping, model parameter transfer, and common space learning [7]. Direct mapping learns a mapping function from visual features to semantic representations [4, 20, 27]. However, its generalization is limited by the high intra-class variability of the visual domain. Meanwhile, the lower-dimensional semantic space shrinks the variance of the projected data points, resulting in the hubness problem [28]. Model parameter transfer, in contrast, takes place in the visual space, where the model parameters for unseen classes are usually obtained [29, 30]. In essence, generative ZSL follows this methodology, simultaneously learning semantic→visual mapping and data augmentation [22, 31, 32, 22, 33]. However, since the inter-class relationships among unseen classes are not taken into account, model parameter transfer is often limited [7]. Finally, common space learning learns a common representation space into which both visual features and semantic representations are projected for effective knowledge transfer [34, 9, 7, 10, 11]. By calibrating the common space using bidirectionally aligned knowledge of the visual and semantic representations, it can simultaneously avoid the issues of direct mapping and model parameter transfer. As such, common space learning is a promising methodology for ZSL. However, existing common space learning methods align the semantic and visual distributions with one-step adaptation, which neglects the distinct heterogeneous feature representations of the semantic and visual domains that are characterized by structure and distribution variations. This causes the distribution-aligned common space to be located in a different manifold, and samples to thus inevitably misclassified. In contrast, we propose an effective framework with both structure adaptation and distribution adaptation to address these challenges.
Domain Adaptation. Domain adaptation aims to learn domain-invariant representations by minimizing the discrepancy between the distributions of source and target via distribution alignment, domain adversarial learning, or task-specific methods. It can be divided into homogeneous and heterogeneous domain adaptation according to the characteristics of the source and target data. In the homogeneous domain adaptation setting, the feature spaces of the source and target domains are identical, with the same dimensions. Thus, one-step adaptation can be employed [35, 36, 37]. In contrast, in the heterogeneous setting, the feature spaces of the source and target domains are distant, and their dimensions also typically differ. Since there is little overlap between the two domains, one-step domain adaptation is not effective in the heterogeneous domain adaptation [38, 39]. Thus, multi-step (or transitive) domain adaptation methods are used for heterogeneous domain adaptation. However, these methods cannot be directly used in ZSL. Since ZSL has four domains (i.e., seen, unseen, semantic, and visual), knowledge cannot be transferred between two domains only, i.e., the distributions of the source and target domains cannot be aligned. We argue that the structure variation should also be taken into account when aligning the visual and semantic domains in ZSL.

heading: 3 Hierarchical Semantic-Visual Adaptation
text: The task of ZSL is formulated as follows. Let S = {(xs, ys, as) | xs ∈ X, ys ∈ Y s, as ∈ A} be a training set consisting of image features x extracted by a convolutional neural network (CNN) Backbone (e.g., ResNet-101), where seen class embeddings as are learned by a language model and seen class labels ys. In addition, an auxiliary training set U = {(yu, au) | yu ∈ Y u} from unseen classes is used, where yu denotes unseen classes from a set Y u, which is disjoint from Y S . xu ∈ X are the unseen visual features in the test set. Note that x = xs ∪ xu and a = as ∪ au. The goal of conventional ZSL (CZSL) is to learn a classifier for unseen visual features, i.e., fCZSL : X → Y U , while for generalized ZSL (GZSL) the goal is to learn a classifier for seen and unseen visual features, i.e., fGZSL : X → Y U ∪ Y S . In the following, we introduce our HSVA, which learns an intrinsic common space for semantic and visual feature representations to conduct ZSL. As shown in Figure 2, our framework consists of two partially-aligned autoencoders, which conduct structure adaptation (SA) and distribution adaptation (DA). In the end of this section, we demonstrate how we perform zero-shot learning using the feature representations extracted from the intrinsic common manifold space.

heading: 3.1 Partially-Aligned Variational Autoencoders
text: Our HSVA model consists of two partially-aligned variational autoencoders. The first includes two feature encoders (visual encoder Ex and common encoder Ez) and one decoder (visual decoder Dx). Ex and Ez are used to learn structure and distribution-aligned visual embeddings, respectively, which share two common spaces (i.e., structure and distribution-aligned spaces) with the semantic embeddings. Dx aims to decode the distribution-aligned embeddings from the visual and semantic features to be similar to the visual features. Analogously, the second variational autoencoder has a similar structure to the first (i.e., it includes a semantic encoder Ea, the Ez , and a semantic decoder Da), but it is employed to learn semantic embeddings. The partially-aligned variational autoencoders are first optimized by two variational autoencoders (VAE) losses:
LxV AE(Ex, Ez) = E[logDx(zx s )]− γKL(Ez(Ex(xs)))‖p(zx s |xs)), (1) LaV AE(Ea, Ez) = E[logDa(za s )]− γKL(Ez(Ea(as)))‖p(za s |as)), (2)
where zx s and za s
are the visual and semantic embeddings in the distribution-aligned space, γ is the weight of the KL-divergence, and p(za
s |as) and p(zxs |xs) are the prior distributions assumed to be N (0, 1). To make the visual and semantic embeddings in the structure- and distribution-aligned spaces more consistent, we then use a cross reconstruction loss to constrain the two partially-aligned
variational autoencoders:
LREC(Ex, Ea, Ez, Dx, Da) = LxREC(Ex, Ez, Dx) + LaREC(Ea, Ez, Da) (3) where
LxREC(Ex, Ez, Dx) = ∥∥∥xs −Dx(zas)∥∥∥
1 , (4) LaREC(Ea, Ez, Da) = ∥∥∥as −Da(zxs)∥∥∥
1 . (5)
Finally, the two partially-aligned variational autoencoders conduct structure and distribution adaptation for the visual and semantic features, which will be introduced in the following.

heading: 3.2 Structure Adaptation
text: Structure adaptation (SA) is used to guide the encoder Ex and Ea to learn a structure-aligned space for the visual and semantic features. Previous methods [7, 11] directly align the semantic and visual features with distribution distance constraints (e.g., MMD), which neglects the structure variation between the heterogeneous feature representations of the two domains. Furthermore, these methods fail to learn discriminative features as they do not take category relationships into account, as shown in Figure 1 (a). Motivated by task-specific unsupervised domain adaptation [36, 37], we propose a supervised adversarial discrepancy (SAD) learning to minimize the discrepancy between the outputs of two task-specific classifiers (CLS1, CLS2). This enables us to learn a class decision boundary and embed semantic and visual features into a structure-aligned common space for SA. Thus, SA can learn discriminative structure-aligned representations for semantic and visual features. Different from [36, 37], which only detect the target samples (e.g., semantic embeddings Ea(a)) to learn a single encoder (generator), SAD simultaneously detects the source samples (e.g., visual embeddings Ex(x)) and target samples (e.g., semantic embeddings Ea(a)) to learn a good structure-aligned common space with two task-specific encoders. Specifically, SAD includes three steps for optimization: 1) semantic and visual embedding classification for Ex, Ea, CLS1 and CLS2; 2) discrepancy maximization for CLS1 and CLS2; and 3) discrepancy minimization for Ex and Ea.
Semantic and Visual Embedding Classification. We trainEx,Ea,CLS1 andCLS2 by minimizing the softmax cross-entropy to collectly classify the structure-aligned semantic and visual embeddings. This is crucial for enabling classifiers and encoders (Ex, Ea) to obtain task-specific discriminative features. The visual and semantic classification loss is formulated as:
LCLS(Ex, Ea, CLS1, CLS2) = LCLS1(Ex(xs), ys) + LCLS2(Ex(xs), ys) + LCLS1(Ea(as), ys) + LCLS2(Ea(as), ys),
(6)
where
LCLS1(x,y) = LCLS2(x,y) = −E K∑ k=1 I[k=y] log p (y | x) . (7)
Here K is the number of seen classes, I[k=y] is an indicator function (i.e., it is one when k = y, otherwise zero), and p (y|x) is the prediction of x. We use two classifiers to simultaneously classify the visual and semantic embeddings to initialize the classifiers, which are used for discrepancy learning later.
Discrepancy Maximization for Classifiers. In this step, we freeze the parameters of the encoders (Ex, Ea) and update the classifiers (CLS1, CLS2) to maximize the discrepancy between the outputs of the two classifiers on the visual and semantic embeddings. Thus, the source and target samples outside the support of task-specific decisions are identified, providing the positive signals to separate the class decision boundaries. Since sliced Wasserstein discrepancy (SWD) [37] provides geometrically meaningful guidance for capturing the natural notion of dissimilarity, we use it to measure the discrepancy between two classifier predictions for visual embeddings (p1(ys|Ex(xs)), p2(y s|Ex(xs))) and semantic embeddings (p1(ys|Ea(as)), p2(ys|Ea(as))):
LDIS1(CLS1, CLS2) = −LSWD(Ex(xs))− LSWD(Ea(as)), (8) LSWD (x) = ∑ m (Rθmp1(y|x),Rθmp2(y|x)) , (9)
whereRθm is the mth one-dimensional linear projection operation, and θ is a uniform measure on the unit sphere S(d−1) in Rd. The larger the LDIS , the larger the discrepancy. Discrepancy Minimization for Encoders. To encourage the visual embedding (Ex(xs)) and semantic embedding (Ea(as)) to be aligned well in the structure-aligned space, we freeze the parameters of the two classifiers and update the two encoders (Ex, Ea) to minimize the discrepancy between the outputs of the two classifiers on Ex(xs) and Ea(as):
LDIS2(Ex, Ea) = LSWD(Ex(xs)) + LSWD(Ea(as)). (10)
This encourages the visual and semantic feature manifolds to be closer to each other, and thus the manifold structure variation is circumvented.

heading: 3.3 Distribution Adaptation
text: After structure adaptation, HSVA can learn a structure-aligned common space to avoid manifold structure difference between visual and semantic embeddings. However, the structure-aligned common space is still limited by distribution variation. To further learn a distribution-aligned common space for visual and semantic feature representations, we propose distribution adaptation. Existing common space learning methods [8, 9, 7, 10, 11] employ two different mapping functions (encoders) to align the visual and semantic distributions to learn distribution-aligned embedding. In contrast, our main idea is to protect the structure-aligned visual and semantic embeddings in a distribution-aligned space using a common encoder Ez , which preserves the structure-aligned representations in the DA. Our DA is first optimized by minimizing the Wasserstein distance between the latent multivariate Gaussian distributions, formulated as follows:
LDA(Ex, Ea, Ez) = (∥∥∥µxs − µas∥∥∥2 2 + ∥∥∥(δxs) 12 − (δas) 12 ∥∥∥2 F ) 1 2 , (11)
where ‖ · ‖2F denotes the squared matrix Frobenius norm. In essence, our HSVA is also a generative model, which is different to the GAN/VAE based methods [40, 31, 33] (i.e., learning mechanism and optimization). Due to the bias problem that arises when using generative models for ZSL, the synthesized unseen samples (Ez(Ea(au))) in the distributionaligned common space might be unexpectedly close to the seen ones. We propose to explicitly tackle this seen-unseen bias problem by preventing the encoded unseen samples (Ez(Ea(au))) from colliding with the encoded seen ones (Ez(Ex(xs)))2. Since correlation alignment (CORAL) [41] has been shown effective for asymmetric transformations in domain adaptation, we take the CORALbased metric to measure the discrepancy between seen and unseen class samples. Different from [41], which takes CORAL to decrease the discrepancy between two domains, we aim to increase the discrepancy. Hence, we employ the inverse CORAL (iCORAL), defined upon the numerical negation of CORAL:
LiCORAL(Ex, Ea, Ez) = −CORAL(Ez(Ex(xs)), Ez(Ea(au))). (12)

heading: 3.4 Optimization
text: Our full model optimizes the partially-aligned variational autoencoders, DA and SA, simultaneously with the following objective function:
LHSV A(Ex, Ea, CLS1, CLS2, Ez, Dx, Da) = LxV AE + LaV AE + λ1LREC + LCLS + λ2(LDIS1 + LDIS2) + λ3(LiCORAL + LDA),
(13)
where λ1, λ2, λ3 are the weights that control the importance of the related loss terms. Similar to the alternative updating policy for generative adversarial networks (GANs), we alternately train Ex, Ea, CLS1, CLS2 when optimizing SA. Although, HSVA has three components in total, the whole training process are simultaneous and loss weights of all terms in Eq. 13 are the same for all datasets. The consistently significant results on all datasets show that our model is robust and easy to train.
2Since Ez(Ex(xs)) and Ez(Ea(au)) are used to train a ZSL classifier later, we take the visual features of seen classes and semantic features of unseen classes into account.

heading: 3.5 Classification
text: Once our HSVA model is learned, the visual and semantic features are encoded as new feature representations in a distribution-aligned common space, where structure variation is aslo considered for classification. Using the reparametrization trick [42], we take Ex and Ez to encode the visual features of seen classes (xs) and unseen classes (xu) into structure- and distribution-aligned feature representations, i.e., zx s = Ez(Ex(xs)) and zx u = Ez(Ex(xu)). Analogously, we take Ea and Ez to encode the semantic embeddings a as za = Ez(Ea(a)), where za = za s ∪ zau . We employ zxs (xs from the training set) and za u
to train a supervised classifier (e.g., softmax). Once the classifier is trained, we use zx s (xs from the test set) and zx u
to test the model. Note that our HSVA is an inductive method as we do not use the visual features of unseen classes for training.

heading: 4 Experiments
text: Datasets. We conduct extensive experiments on four well-known ZSL benchmark datasets, including fine-grained datasets (e.g., CUB [43] and SUN [44]) and coarse-grained datasets (e.g., AWA1 [4] and AWA2 [45]). CUB includes 11,788 images of 200 bird classes (seen/unseen classes = 150/50) with 312 attributes. SUN contains 14,340 images from 717 scene classes (seen/unseen classes = 645/72) with 102 attributes. AWA1 and AWA2 contain 30,475 and 37,322 images from 50 animal classes (seen/unseen classes = 40/10) with 85 attributes.
Implementation Details. We use the training splits proposed in [46]. Meanwhile, the visual features are extracted from the 2048-dimensional top-layer pooling units of a CNN backbone (i.e., ResNet-101) pre-trained on ImageNet. The encoders and decoders are multilayer perceptrons (MLPs). We employ the Adam optimizer [47] with β1 = 0.5 and β2 = 0.999. We use an annealing scheme [48] to increase the weights γ, λ1, λ2, λ3 with same setting for all datasets. Specifically, γ is increased by a rate of 0.0026 per epoch until epoch 90, λ1 is increased from epoch 21 to 75 by 0.044 per epoch, and λ2, λ3 are increased from epoch 0 to epoch 22 by a rate of 0.54 per epoch. In the CZSL setting, we synthesize 800, 400 and 200 features per unseen class to train the classifier for AWA1, CUB and SUN datasets, respectively. In the GZSL setting, we take 400 synthesized features per unseen class and 200 synthesized features per seen class to train the classifier for all datasets. The dimensions of the structure-aligned and distribution-aligned spaces are set to 2048 and 64 for three datasets (i.e., CUB, AWA1, AWA2), respectively, and 2048 and 128 for SUN benchmark.
Evaluation Protocols. During testing, we follow the unified evaluation protocols proposed in [45]. Specifically, we measure the top-1 accuracy of unseen class (Acc) for the CZSL setting. In the GZSL setting, we take the top-1 accuracy on seen classes (S) and unseen classes (U), as well as their harmonic mean (defined as H = (2× S × U)/(S + U)).

heading: 4.1 Experimental Results
text: Results of Conventional Zero-Shot Learning. Table 1 presents the results of CZSL on various datasets. Our HSVA, with a softmax classifier, significantly outperforms other common space learning methods [34, 10, 11] by at least 5.3%, 3.0% and 2.0% on AWA1, CUB and SUN, respectively. Our significant performance improvements demonstrate that hierarchical semantic-visual adaptation can effectively learn an intrinsic common space to represent visual and semantic features. This space is structure- and distribution-aligned and alleviates the gap between heterogeneous feature representations.
Results of Generalized Zero-Shot Learning. Table 2 shows the GZSL performances of various methods, including non-generative methods (direct mapping and model parameter transfer), generative methods, and common space learning methods. Compared to other common space learning models [34, 9, 10, 11, 19], our HSVA achieves significant improvements of at least 2.7%, 1.0%, 2.9% and 2.7% in harmonic mean on AWA1, AWA2, CUB and SUN, respectively. Especially, our HSVA outperforms the latest common space learning method (i.e., SGAL [19]) by a large margin, resulting in harmonic mean improvements of 5.3%, 1.0%, 8.3% and 8.4% on AWA1, AWA2, CUB and SUN, respectively. Since our method can also act as a generative method, we compare it with other state-ofthe-art generative methods. The results show that our method performs significantly better on AWA1, CUB and SUN datasets. Our model also achieves competitive results compared to non-generative methods. For example, HSVA performs best results (e.g., the harmonic mean of 43.3) on SUN that is the most challenging benchmark in ZSL. These results consistently demonstrate the superiority and great potential of hierarchical semantic-visual adaptation.
Ablation Study. We conduct ablation studies to evaluate the effectiveness of our HSVA in terms of the structure adaptation constraints (denoted as LSA), distribution adaptation constraints (i.e., LDA + LiCORAL), and only inverse CORAL LiCORAL. Our results are shown in Table 3. HSVA performs poorer than its full model when no constraints are used during structure adaptation, i.e., the harmonic mean drops by 4.5% on AWA1 and 2.1% on CUB. If constraints are not used during distribution adaptation, HSVA achieves very poor results compared to its full model, i.e., the harmonic mean drops by more than 23.4% on coarse-grained datasets and 4.7% on fine-grained datasets. These results show that distribution alignment is essential for common space learning, while structure adaptation is necessary for semantic-visual adaptation. This is typically neglected by distribution alignment methods with one-step adaptation [8, 9, 7, 10, 11, 19]. Moreover, LiCORAL pushes
unseen classes away from seen classes, which explicitly tackles the seen-unseen bias problem. HSVA cooperates with iCORAL to improve the accuracy of classification using softmax, achieving harmonic mean improvement of 4.4%, 3.0%, 1.4%, and 1.5% on AWA1, AWA2, CUB, and SUN, respectively.
Qualitative Results. We first qualitatively investigate the difference between our HSVA and onestep adaptation methods (e.g., CADA-VAE [11]) for common space learning. We conduct t-SNE visualization [14] of visual/semantic features mapped by our HSVA and CADA-VAE on 10 classes from (a) CUB and (b) AWA1. As shown in Figure 3, CADA-VAE maps semantic and visual features into different manifolds, which confuses the visual and semantic features of various categories, resulting in misclassification for some samples. This intuitively shows that the existing common space learning methods neglect the different structures of the heterogeneous feature representations in the visual and semantic domains. In contrast, HSVA learns an intrinsic common space that better bridges the gap between the heterogeneous feature representations for effective knowledge transfer, using structure and distribution adaptation. Thus, hierarchical semantic-visual adaptation is an effective solution for common space learning in ZSL.
In addition, unlike other methods [8, 9, 7, 10, 11] that can only align visual and semantic distributions, our HSVA can take class relationships into account to learn discriminatively aligned feature representations. As shown in Figure 3, CADA-VAE maps visual and semantic features into compact manifolds, resulting in confused class relationships. In contrast, HSVA learns discriminative structureand distribution-aligned features for the visual and semantic feature representations, which enables HSVA to achieve better ZSL classification.

heading: 5 Conclusion
text: In this work, we develop a zero-shot learning framework, i.e. hierarchical semantic-visual adaptation (HSVA), to learn an intrinsic common space for visual and semantic feature representations. By conducting structure and distribution adaptation using two partially-aligned variational autoencoders, our model effectively tackles the problem of heterogeneous feature representations and bridges the gap between the visual and semantic domains. We demonstrate that HSVA achieves consistent improvement over the current state-of-the-art methods on four ZSL benchmarks. We qualitatively verify that our hierarchical semantic-visual adaptation framework can align the visual and semantic domains in terms of both structure and distribution, while other common space learning methods only
take distribution alignment into account. This intuitively shows that the learned feature representations of HSVA are more discriminative and accurate than other methods. To the best of our knowledge, we are the first to investigate the different structural characteristics of the visual and semantic domains in ZSL. We believe that our work will facilitate the development of stronger visual-and-language learning systems, including zero-shot learning, visual question answering [57], image captioning [58], natural language for visual reasoning [59], heterogeneous domain adaptation [60].

heading: Acknowledgements
text: This work is partially supported by NSFC (61772220), Special projects for technological innovation in Hubei Province (2018ACA135) and Key R&D Plan of Hubei Province (2020BAB027).

Review summaries:
This paper proposes an hierarchical semantic-visual adaptation framework to migrate the distribution and structure disagreement in generative zero-shot learning (ZSL) models. This network adopts hierarchical two-step adaptation. The task-specific adaptation is complemented by training two task-specific encoders with a supervised adversarial discrepancy module. The distribution adaptation is achieved by minimizing the Wasserstein distance between visual and semantic distributions. Quantitative results on four widely used benchmarks indicates the effectiveness of the proposed method. Qualitative visualization shows the network learns intrinsic common space that better bridges the gap between the heterogeneous feature representations. 	 

This paper proposes a new method for zero-shot learning. It enforces the classes are well aligned by using a domain-specific discrepancy on an intermediate space of the encoder. The method is evaluated on benchmark datasets and achieves good results.

The paper proposes a new two-step, hierarchical adaptation approach for Zero-Shot Learning. The first step is structure adaptation where visual data and target/semantic data (eg attributes) are encoded into an aligned common space. This is similar to prior work, but is accomplished using a new module that adversarially minimizes the discrepancy between the visual and semantic-based classifiers. The second step is distribution adaptation which seeks to align the visual and semantic feature distributions using a common encoder.  The paper ablates the contribution of both structure and distribution alignment on several ZSL benchmarks, showing significant improvements. It also compares favorably against all prior common space ZSL methods on all benchmarks, and against all prior work (common and non-common space) on several benchmarks.

In zero-shot learning (ZSL) problem, feature representations of distinct semantic and visual domains are intrinsically heterogeneous, thus having both distribution and structure variations. To this problem, this paper proposes a novel HSVA framework that aligns the semantic and visual domains by structure adaptation (SA) and distribution adaptation (DA). SA, consisting of two encoders, aims to encode the visual data and semantic data into a structure-aligned common space, by using a SAD module. Then, DA further explores a common encoder to align the visual and semantic distributions. This paper also conducts extensive experiments to show the effectiveness of the proposed framework for both conventional ZSL and generalized ZSL settings.