HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning

1 Introduction . Over the past decade , significant progress has been made in zero-shot learning ( ZSL ) , which aims to recognize new classes during learning by exploiting the intrinsic semantic relatedness between seen and unseen categories [ 1 , 2 , 3 ] . Inspired by the way humans learn unknown concepts , sideinformation ( e.g. , attributes [ 4 ] , word vectors [ 5 ] , and sentences [ 6 ] ) related to seen/unseen classes is employed to guarantee knowledge transfer between the seen/unseen data . Common space learning is one typical method for representing the relationship between the visual and semantic domains for ZSL , which is key to dealing with the knowledge transfer [ 7 ] . However , existing common space ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . learning methods align the distributions of the semantic and visual domains with one-step adaptation [ 8 , 9 , 7 , 10 , 11 , 12 , 13 ] , while neglecting the fact that the heterogeneous feature representations of distinct semantic and visual domains have both distribution and structure variations . Thus , their performance is limited . In this work , we propose a novel common space learning formulation that leverages a hierarchical semantic-visual adaptation to learn an intrinsic common space for a better alignment of the two heterogeneous representation domains . The heterogeneous feature representations of the semantic and visual domains are distinct [ 15 , 16 ] . Although one-step adaptation learns distribution-aligned feature representations for the semantic and visual domains using two different mapping models ( e.g. , encoders ) and distribution alignment constraints ( e.g. , maximum mean discrepancy ( MMD ) ) , the two distributions are located in different manifolds since the one-step adaptation does not consider the manifold structure relationship between visual and semantic representations , as shown in Figure 1 ( a ) and ( c ) . If we take the Euclidean distance or manifold distance [ 17 , 18 ] to measure the relationship between different classes , the classifier inevitably misclassifies some samples , thus leading to inferior ZSL performance . As such , mapping the semantic and visual domains into an intrinsic and desirable common space for conducting an effective ZSL is highly necessary . To learn structure- and distribution-aligned feature representations for distinct visual and semantic domains , we propose a hierarchical semantic-visual adaptation ( HSVA ) framework to learn an intrinsic common space that contains the essential multi-modal information associated with unseen classes ( Figure 1 ( b ) and ( d ) ) . HSVA aligns the semantic and visual domains with structure adaptation ( SA ) and distribution adaptation ( DA ) in a unified framework , under two partially-aligned variational autoencoders . In the SA , we use two task-specific encoders to encode source data ( visual domain ) and target data ( semantic domain ) , respectively , into a structure-aligned space , which is learned through supervised adversarial discrepancy ( SAD ) learning to minimize the discrepancy between the predictions of the two task-specific classifiers . This encourages the visual and semantic feature manifolds to be closer to each other , thus aligning the manifold structure variation . In the DA , we map the structure-aligned features into a distribution-aligned common space using a common encoder , which is optimized by minimizing the Wasserstein distance between the latent multivariate Gaussian distributions of the visual and semantic domains . The common encoder preserves the structure-aligned representations in the DA . Thus , HSVA can learn a structure and distributionaligned common space for visual and semantic feature representations , which is better than the distribution-aligned feature representations learned by existing common space learning methods [ 8 , 9 , 7 , 10 , 11 , 19 ] . To the best of our knowledge , this is the first work that leverages hierarchical semantic-visual adaptation to address heterogeneous feature alignment in ZSL . Following [ 11 ] , we conduct extensive experiments on four challenging benchmark datasets , i.e. , CUB , SUN , AWA1 , and AWA2 , under both conventional and generalized ZSL settings . The proposed HSVA achieves consistent improvement over the existing common space learning methods on all datasets . We show qualitatively that the structural variation is important for the interaction between visual and semantic domains . Moreover , we show significantly better common space for visual and semantic feature representations than the existing common space learning methods , e.g. , CADA-VAE [ 11 ] . 2 Related Work . Zero-Shot Learning . In practice , the relationship between visual and semantic domains is determined by learning an embedding space where semantic vectors and visual features interact [ 20 , 21 , 22 , 23 , 24 , 25 , 26 , 19 ] . There are three mapping methods for learning such an embedding space , including direct mapping , model parameter transfer , and common space learning [ 7 ] . Direct mapping learns a mapping function from visual features to semantic representations [ 4 , 20 , 27 ] . However , its generalization is limited by the high intra-class variability of the visual domain . Meanwhile , the lower-dimensional semantic space shrinks the variance of the projected data points , resulting in the hubness problem [ 28 ] . Model parameter transfer , in contrast , takes place in the visual space , where the model parameters for unseen classes are usually obtained [ 29 , 30 ] . In essence , generative ZSL follows this methodology , simultaneously learning semantic→visual mapping and data augmentation [ 22 , 31 , 32 , 22 , 33 ] . However , since the inter-class relationships among unseen classes are not taken into account , model parameter transfer is often limited [ 7 ] . Finally , common space learning learns a common representation space into which both visual features and semantic representations are projected for effective knowledge transfer [ 34 , 9 , 7 , 10 , 11 ] . By calibrating the common space using bidirectionally aligned knowledge of the visual and semantic representations , it can simultaneously avoid the issues of direct mapping and model parameter transfer . As such , common space learning is a promising methodology for ZSL . However , existing common space learning methods align the semantic and visual distributions with one-step adaptation , which neglects the distinct heterogeneous feature representations of the semantic and visual domains that are characterized by structure and distribution variations . This causes the distribution-aligned common space to be located in a different manifold , and samples to thus inevitably misclassified . In contrast , we propose an effective framework with both structure adaptation and distribution adaptation to address these challenges . Domain Adaptation . Domain adaptation aims to learn domain-invariant representations by minimizing the discrepancy between the distributions of source and target via distribution alignment , domain adversarial learning , or task-specific methods . It can be divided into homogeneous and heterogeneous domain adaptation according to the characteristics of the source and target data . In the homogeneous domain adaptation setting , the feature spaces of the source and target domains are identical , with the same dimensions . Thus , one-step adaptation can be employed [ 35 , 36 , 37 ] . In contrast , in the heterogeneous setting , the feature spaces of the source and target domains are distant , and their dimensions also typically differ . Since there is little overlap between the two domains , one-step domain adaptation is not effective in the heterogeneous domain adaptation [ 38 , 39 ] . Thus , multi-step ( or transitive ) domain adaptation methods are used for heterogeneous domain adaptation . However , these methods can not be directly used in ZSL . Since ZSL has four domains ( i.e. , seen , unseen , semantic , and visual ) , knowledge can not be transferred between two domains only , i.e. , the distributions of the source and target domains can not be aligned . We argue that the structure variation should also be taken into account when aligning the visual and semantic domains in ZSL . 3 Hierarchical Semantic-Visual Adaptation . The task of ZSL is formulated as follows . Let S = { ( xs , ys , as ) | xs ∈ X , ys ∈ Y s , as ∈ A } be a training set consisting of image features x extracted by a convolutional neural network ( CNN ) Backbone ( e.g. , ResNet-101 ) , where seen class embeddings as are learned by a language model and seen class labels ys . In addition , an auxiliary training set U = { ( yu , au ) | yu ∈ Y u } from unseen classes is used , where yu denotes unseen classes from a set Y u , which is disjoint from Y S . xu ∈ X are the unseen visual features in the test set . Note that x = xs ∪ xu and a = as ∪ au . The goal of conventional ZSL ( CZSL ) is to learn a classifier for unseen visual features , i.e. , fCZSL : X → Y U , while for generalized ZSL ( GZSL ) the goal is to learn a classifier for seen and unseen visual features , i.e. , fGZSL : X → Y U ∪ Y S . In the following , we introduce our HSVA , which learns an intrinsic common space for semantic and visual feature representations to conduct ZSL . As shown in Figure 2 , our framework consists of two partially-aligned autoencoders , which conduct structure adaptation ( SA ) and distribution adaptation ( DA ) . In the end of this section , we demonstrate how we perform zero-shot learning using the feature representations extracted from the intrinsic common manifold space . 3.1 Partially-Aligned Variational Autoencoders . Our HSVA model consists of two partially-aligned variational autoencoders . The first includes two feature encoders ( visual encoder Ex and common encoder Ez ) and one decoder ( visual decoder Dx ) . Ex and Ez are used to learn structure and distribution-aligned visual embeddings , respectively , which share two common spaces ( i.e. , structure and distribution-aligned spaces ) with the semantic embeddings . Dx aims to decode the distribution-aligned embeddings from the visual and semantic features to be similar to the visual features . Analogously , the second variational autoencoder has a similar structure to the first ( i.e. , it includes a semantic encoder Ea , the Ez , and a semantic decoder Da ) , but it is employed to learn semantic embeddings . The partially-aligned variational autoencoders are first optimized by two variational autoencoders ( VAE ) losses : LxV AE ( Ex , Ez ) = E [ logDx ( zx s ) ] − γKL ( Ez ( Ex ( xs ) ) ) ‖p ( zx s |xs ) ) , ( 1 ) LaV AE ( Ea , Ez ) = E [ logDa ( za s ) ] − γKL ( Ez ( Ea ( as ) ) ) ‖p ( za s |as ) ) , ( 2 ) where zx s and za s are the visual and semantic embeddings in the distribution-aligned space , γ is the weight of the KL-divergence , and p ( za s |as ) and p ( zxs |xs ) are the prior distributions assumed to be N ( 0 , 1 ) . To make the visual and semantic embeddings in the structure- and distribution-aligned spaces more consistent , we then use a cross reconstruction loss to constrain the two partially-aligned variational autoencoders : LREC ( Ex , Ea , Ez , Dx , Da ) = LxREC ( Ex , Ez , Dx ) + LaREC ( Ea , Ez , Da ) ( 3 ) where LxREC ( Ex , Ez , Dx ) = ∥∥∥xs −Dx ( zas ) ∥∥∥ 1 , ( 4 ) LaREC ( Ea , Ez , Da ) = ∥∥∥as −Da ( zxs ) ∥∥∥ 1 . ( 5 ) Finally , the two partially-aligned variational autoencoders conduct structure and distribution adaptation for the visual and semantic features , which will be introduced in the following .

Summary: This paper proposes a new method for zero-shot learning. It enforces the classes are well aligned by using a domain-specific discrepancy on an intermediate space of the encoder. The method is evaluated on benchmark datasets and achieves good results.