HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning

1 Introduction . Over the past decade , significant progress has been made in zero-shot learning ( ZSL ) , which aims to recognize new classes during learning by exploiting the intrinsic semantic relatedness between seen and unseen categories [ 1 , 2 , 3 ] . Inspired by the way humans learn unknown concepts , sideinformation ( e.g. , attributes [ 4 ] , word vectors [ 5 ] , and sentences [ 6 ] ) related to seen/unseen classes is employed to guarantee knowledge transfer between the seen/unseen data . Common space learning is one typical method for representing the relationship between the visual and semantic domains for ZSL , which is key to dealing with the knowledge transfer [ 7 ] . However , existing common space ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . learning methods align the distributions of the semantic and visual domains with one-step adaptation [ 8 , 9 , 7 , 10 , 11 , 12 , 13 ] , while neglecting the fact that the heterogeneous feature representations of distinct semantic and visual domains have both distribution and structure variations . Thus , their performance is limited . In this work , we propose a novel common space learning formulation that leverages a hierarchical semantic-visual adaptation to learn an intrinsic common space for a better alignment of the two heterogeneous representation domains . The heterogeneous feature representations of the semantic and visual domains are distinct [ 15 , 16 ] . Although one-step adaptation learns distribution-aligned feature representations for the semantic and visual domains using two different mapping models ( e.g. , encoders ) and distribution alignment constraints ( e.g. , maximum mean discrepancy ( MMD ) ) , the two distributions are located in different manifolds since the one-step adaptation does not consider the manifold structure relationship between visual and semantic representations , as shown in Figure 1 ( a ) and ( c ) . If we take the Euclidean distance or manifold distance [ 17 , 18 ] to measure the relationship between different classes , the classifier inevitably misclassifies some samples , thus leading to inferior ZSL performance . As such , mapping the semantic and visual domains into an intrinsic and desirable common space for conducting an effective ZSL is highly necessary . To learn structure- and distribution-aligned feature representations for distinct visual and semantic domains , we propose a hierarchical semantic-visual adaptation ( HSVA ) framework to learn an intrinsic common space that contains the essential multi-modal information associated with unseen classes ( Figure 1 ( b ) and ( d ) ) . HSVA aligns the semantic and visual domains with structure adaptation ( SA ) and distribution adaptation ( DA ) in a unified framework , under two partially-aligned variational autoencoders . In the SA , we use two task-specific encoders to encode source data ( visual domain ) and target data ( semantic domain ) , respectively , into a structure-aligned space , which is learned through supervised adversarial discrepancy ( SAD ) learning to minimize the discrepancy between the predictions of the two task-specific classifiers . This encourages the visual and semantic feature manifolds to be closer to each other , thus aligning the manifold structure variation . In the DA , we map the structure-aligned features into a distribution-aligned common space using a common encoder , which is optimized by minimizing the Wasserstein distance between the latent multivariate Gaussian distributions of the visual and semantic domains . The common encoder preserves the structure-aligned representations in the DA . Thus , HSVA can learn a structure and distributionaligned common space for visual and semantic feature representations , which is better than the distribution-aligned feature representations learned by existing common space learning methods [ 8 , 9 , 7 , 10 , 11 , 19 ] . To the best of our knowledge , this is the first work that leverages hierarchical semantic-visual adaptation to address heterogeneous feature alignment in ZSL . Following [ 11 ] , we conduct extensive experiments on four challenging benchmark datasets , i.e. , CUB , SUN , AWA1 , and AWA2 , under both conventional and generalized ZSL settings . The proposed HSVA achieves consistent improvement over the existing common space learning methods on all datasets . We show qualitatively that the structural variation is important for the interaction between visual and semantic domains . Moreover , we show significantly better common space for visual and semantic feature representations than the existing common space learning methods , e.g. , CADA-VAE [ 11 ] . 2 Related Work . Zero-Shot Learning . In practice , the relationship between visual and semantic domains is determined by learning an embedding space where semantic vectors and visual features interact [ 20 , 21 , 22 , 23 , 24 , 25 , 26 , 19 ] . There are three mapping methods for learning such an embedding space , including direct mapping , model parameter transfer , and common space learning [ 7 ] . Direct mapping learns a mapping function from visual features to semantic representations [ 4 , 20 , 27 ] . However , its generalization is limited by the high intra-class variability of the visual domain . Meanwhile , the lower-dimensional semantic space shrinks the variance of the projected data points , resulting in the hubness problem [ 28 ] . Model parameter transfer , in contrast , takes place in the visual space , where the model parameters for unseen classes are usually obtained [ 29 , 30 ] . In essence , generative ZSL follows this methodology , simultaneously learning semantic→visual mapping and data augmentation [ 22 , 31 , 32 , 22 , 33 ] . However , since the inter-class relationships among unseen classes are not taken into account , model parameter transfer is often limited [ 7 ] . Finally , common space learning learns a common representation space into which both visual features and semantic representations are projected for effective knowledge transfer [ 34 , 9 , 7 , 10 , 11 ] . By calibrating the common space using bidirectionally aligned knowledge of the visual and semantic representations , it can simultaneously avoid the issues of direct mapping and model parameter transfer . As such , common space learning is a promising methodology for ZSL . However , existing common space learning methods align the semantic and visual distributions with one-step adaptation , which neglects the distinct heterogeneous feature representations of the semantic and visual domains that are characterized by structure and distribution variations . This causes the distribution-aligned common space to be located in a different manifold , and samples to thus inevitably misclassified . In contrast , we propose an effective framework with both structure adaptation and distribution adaptation to address these challenges . Domain Adaptation . Domain adaptation aims to learn domain-invariant representations by minimizing the discrepancy between the distributions of source and target via distribution alignment , domain adversarial learning , or task-specific methods . It can be divided into homogeneous and heterogeneous domain adaptation according to the characteristics of the source and target data . In the homogeneous domain adaptation setting , the feature spaces of the source and target domains are identical , with the same dimensions . Thus , one-step adaptation can be employed [ 35 , 36 , 37 ] . In contrast , in the heterogeneous setting , the feature spaces of the source and target domains are distant , and their dimensions also typically differ . Since there is little overlap between the two domains , one-step domain adaptation is not effective in the heterogeneous domain adaptation [ 38 , 39 ] . Thus , multi-step ( or transitive ) domain adaptation methods are used for heterogeneous domain adaptation . However , these methods can not be directly used in ZSL . Since ZSL has four domains ( i.e. , seen , unseen , semantic , and visual ) , knowledge can not be transferred between two domains only , i.e. , the distributions of the source and target domains can not be aligned . We argue that the structure variation should also be taken into account when aligning the visual and semantic domains in ZSL . 3 Hierarchical Semantic-Visual Adaptation . The task of ZSL is formulated as follows . Let S = { ( xs , ys , as ) | xs ∈ X , ys ∈ Y s , as ∈ A } be a training set consisting of image features x extracted by a convolutional neural network ( CNN ) Backbone ( e.g. , ResNet-101 ) , where seen class embeddings as are learned by a language model and seen class labels ys . In addition , an auxiliary training set U = { ( yu , au ) | yu ∈ Y u } from unseen classes is used , where yu denotes unseen classes from a set Y u , which is disjoint from Y S . xu ∈ X are the unseen visual features in the test set . Note that x = xs ∪ xu and a = as ∪ au . The goal of conventional ZSL ( CZSL ) is to learn a classifier for unseen visual features , i.e. , fCZSL : X → Y U , while for generalized ZSL ( GZSL ) the goal is to learn a classifier for seen and unseen visual features , i.e. , fGZSL : X → Y U ∪ Y S . In the following , we introduce our HSVA , which learns an intrinsic common space for semantic and visual feature representations to conduct ZSL . As shown in Figure 2 , our framework consists of two partially-aligned autoencoders , which conduct structure adaptation ( SA ) and distribution adaptation ( DA ) . In the end of this section , we demonstrate how we perform zero-shot learning using the feature representations extracted from the intrinsic common manifold space . 3.1 Partially-Aligned Variational Autoencoders . Our HSVA model consists of two partially-aligned variational autoencoders . The first includes two feature encoders ( visual encoder Ex and common encoder Ez ) and one decoder ( visual decoder Dx ) . Ex and Ez are used to learn structure and distribution-aligned visual embeddings , respectively , which share two common spaces ( i.e. , structure and distribution-aligned spaces ) with the semantic embeddings . Dx aims to decode the distribution-aligned embeddings from the visual and semantic features to be similar to the visual features . Analogously , the second variational autoencoder has a similar structure to the first ( i.e. , it includes a semantic encoder Ea , the Ez , and a semantic decoder Da ) , but it is employed to learn semantic embeddings . The partially-aligned variational autoencoders are first optimized by two variational autoencoders ( VAE ) losses : LxV AE ( Ex , Ez ) = E [ logDx ( zx s ) ] − γKL ( Ez ( Ex ( xs ) ) ) ‖p ( zx s |xs ) ) , ( 1 ) LaV AE ( Ea , Ez ) = E [ logDa ( za s ) ] − γKL ( Ez ( Ea ( as ) ) ) ‖p ( za s |as ) ) , ( 2 ) where zx s and za s are the visual and semantic embeddings in the distribution-aligned space , γ is the weight of the KL-divergence , and p ( za s |as ) and p ( zxs |xs ) are the prior distributions assumed to be N ( 0 , 1 ) . To make the visual and semantic embeddings in the structure- and distribution-aligned spaces more consistent , we then use a cross reconstruction loss to constrain the two partially-aligned variational autoencoders : LREC ( Ex , Ea , Ez , Dx , Da ) = LxREC ( Ex , Ez , Dx ) + LaREC ( Ea , Ez , Da ) ( 3 ) where LxREC ( Ex , Ez , Dx ) = ∥∥∥xs −Dx ( zas ) ∥∥∥ 1 , ( 4 ) LaREC ( Ea , Ez , Da ) = ∥∥∥as −Da ( zxs ) ∥∥∥ 1 . ( 5 ) Finally , the two partially-aligned variational autoencoders conduct structure and distribution adaptation for the visual and semantic features , which will be introduced in the following .

Summary: This paper proposes an hierarchical semantic-visual adaptation framework to migrate the distribution and structure disagreement in generative zero-shot learning (ZSL) models. This network adopts hierarchical two-step adaptation. The task-specific adaptation is complemented by training two task-specific encoders with a supervised adversarial discrepancy module. The distribution adaptation is achieved by minimizing the Wasserstein distance between visual and semantic distributions. Quantitative results on four widely used benchmarks indicates the effectiveness of the proposed method. Qualitative visualization shows the network learns intrinsic common space that better bridges the gap between the heterogeneous feature representations.