Implicit Semantic Response Alignment for Partial Domain Adaptation

1 Introduction . Deep neural networks have achieved impressive results in various supervised learning applications such as object recognition [ 15 , 18 ] , semantic segmentation [ 51 , 12 , 48 ] and multi-modal learning [ 34 , 40 ] , but annotating a large-scale dataset for deep learning models training could be tremendously grueling and expensive . Thus , considerable efforts have been dedicated to domain adaptation ( DA ) , which attempts to circumvent labeling unfamiliar target data by transferring knowledge from a well-studied source domain data . Existing DA methods align two differently-distributed domains by finding invariant representations from the transferable feature structure [ 24 ] . Matching the source and target distributions with a discrepancy loss based on high-order statistics such as maximum mean discrepancy ( MMD ) [ 1 ] is commonly used along with iterative pseudo pseudo labeling strategy [ 16 , 37 , 22 ] . Recently efforts seek a domain-invariant feature space by adversarial learning , so that the source classifier be directly used for the target data prediction . Adding a weighting schema further improves the classification ability of adversarial methods . However , traditional domain adaptation 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . requires a related source domain that shares the same label space with the unlabelled target domain , which is not always available in real-life . Sometimes , target label space is only a subset of source labels space when people try to transfer knowledge from a more comprehensive domain , and partial domain adaption ( PDA ) is introduced because of this practical situation . The mismatched label space poses a difficult challenge where aligning all source domain with the small target domain could suffer from negative transfer problem . To this end , various PDA algorithms [ 2 , 50 , 3 , 14 , 19 , 4 , 17 , 37 , 23 ] extend the reweighting schema to alleviate negative transfer by reducing the influence of the irrelevant categories . Down-weighting the samples from the outlier source classes is one of the most commonly used strategies , while other recently works further assign weights on the instance level for both source and target samples to mitigate domain divergence . In general , these PDA methods aim to identify and discard all the irrelevant classes , in an attempt of aligning the marginal distributions only for the categories shared both domains . However , some extra classes are semantically correlated with the target classes , which could potentially contain valuable information for the PDA task . We believe that fully utilizing the relevant information hidden across different classes helps promote positive transfer . For example , cats and dogs have clear distinguished features for class separation ; however , they also share many common semantic topics including fur , four legs , and so on . Therefore , we expect to explore the relationships among different categories by implicit semantics and achieve the semantic level alignment . Contributions . In this paper , we propose the implicit semantic response alignment , as a plugin module for any existing PDA models , which extracts implicit semantics from all categories including the shared categories and source-only categories and reduce the distribution discrepancy on the semantic level . Specifically , every sample is similarly decomposed into an embedding vector representing diverse implicit semantic topics with a class2vec machine . Under the guidance of each implicit semantic topic , we employ an attention-based weighting schema on the features by the implicit semantic response . The semantically weighted feature masks of the source and target are calibrated together as our final alignment . The major contributions are summarized as follows : • We exploit the relationships of all available categories by extracting implicit semantics from visual features , which allows our method to utilize relevant information contained in every sample , including those from the long neglected unshared categories in the PDA problem . • With the help of a novel feature-level weighting strategy guided by implicit semantic responses , we align the source and target data distribution based on the implicit semantic topics shared between two domains to boost the positive transfer . • We demonstrate the effectiveness of our method by boosting the existing state-of-art PDA models on various benchmarks including Office31 [ 36 ] , Office-Home [ 44 ] and ImageNetCaltech [ 4 ] , and provide several detailed in-depth explorations of our purposed method . 2 Related Work . Unsupervised domain adaptation ( UDA ) has attracted increasing attention over the past decades due to the high cost of annotating a massive amount of new data for training deep convolutional neural networks . Various UDA models have been established to overcome the distribution disparities between the source and target domains , among which two types of strategies are widely adopted . The first type [ 42 , 28 , 25 , 8 , 27 , 30 ] matches two data distributions based on their high-order statistical properties to diminish domain discrepancy , such as maximum mean discrepancy ( MMD ) [ 1 ] . Inspired by Generative Adversarial Networks [ 9 ] , other UDA methods [ 6 , 7 , 41 ] resort to adversarial learning , which aims to capture a domain-invariant feature representation that is capable of confusing the domain discriminator . Some works [ 33 , 26 , 39 ] also incorporate reweighting schema derived from the interaction between discriminator and classifier to further align different domains . Additionally , iterative pseudo labeling is used by recent methods [ 5 , 22 , 46 ] in an attempt to align the label spaces of different domains . However , in a more realistic situation where the source and target label spaces are not identical , the prediction capability of these methods is seriously impaired by negative transfer due to the heterogeneous label distributions . A significant amount of research efforts [ 13 , 31 , 29 , 21 , 20 , 49 , 38 ] have been done on diverse aspects of transferring knowledge between domains with mismatched label space , while this work only focuses on a special case , partial domain adaptation ( PDA ) where the source label space subsumes target label space . Most conventional PDA methods [ 2 , 50 , 3 , 14 , 19 , 4 , 17 , 37 , 23 ] extend the deep adversarial adaptation introduced by Domain-Adversarial Neural Network [ 7 ] with varied reweighting approaches . Selective Adversarial Network ( SAN ) [ 2 ] employs multiple class-wise discriminators to select out the source samples in the outlier classes and down-weight their importance during transfer . Similarly , Importance Weighted Adversarial Nets ( IWAN ) [ 50 ] assesses the importance of source samples by one auxiliary domain classifier along with a fixed source feature extractor . Partial Adversarial Domain Adaptation ( PADA ) [ 3 ] assigns class-level weights for both source classifier and domain adversary according to the estimated target label distribution . Multi-Weight Partial Domain Adaptation [ 14 ] exploits a multi-weight mechanism consisting of shared-class weights and shared-sample weights calculated by a shared-sample classifier to distinguish outlier classes and samples . Deep Residual Correction Network ( DRCN ) [ 19 ] utilizes a weighted class-wise matching strategy with a plug-in residual block that automatically identifies the most relevant source subclasses and aligns with them target data . Example Transfer Network ( ETN ) [ 4 ] weights source samples based on their discriminative information extracted by a transferability quantifier . Adaptively-Accumulated Knowledge Transfer ( A2TK ) [ 17 ] optimizes both domain-wise distribution adaptation and class-wise distribution alignment by iteratively filtering out confident task-relevant target samples and their corresponding source categories . Select , Label , and Mix ( SLM ) [ 37 ] integrates pseudo labelling for the target domain to improve the discriminability of the invariant feature representation shared by the source and target domain . Lately , BA3US [ 23 ] incorporates weighting schema with augmenting target domain with source data , which achieves state-of-art on several PDA benchmark datasets . Generally , existing PDA methods involving adversarial training aim to transform the PDA task into an UDA-like problem by matching the source and target data on the sample level . On the other hand , our method tries to transform the problem on the semantic level , then align two different domains based on the implicit semantics . Aside from adversarial learning , recent UDA methods adopting other adaptation approaches also yield superior results for some PDA benchmarks . Larger Norm More Transferable ( AFN ) [ 47 ] accomplishes alignment by reducing a novel statistic distance designed by the authors that characterizes the mean-feature-norm discrepancy between different domains . Source Hypothesis Transfer ( SHOT ) [ 22 ] proposes a source data-free framework that learns target-specific features that can be accurately recognized by the frozen classifier trained on the source domain , with the help of self-supervised pseudo-labeling and information maximization . Different from the existing literature , we propose the implicit semantic response alignment , a plugin module to add on the existing PDA frameworks , to explore the relationships among different categories . Rather than the category level transfer , our semantic level alignment gathers all semantic response from multiple related categories across source and target domains to boost positive transfer . 3 Methodology . In this section , we elaborate on our method for partial domain adaptation . First , the overview of the proposed framework is introduced . Then we provide detailed technical formation and objective function for each component in our model . 3.1 Framework Overview . To achieve implicit semantic alignment , we propose a novel framework as shown in Figure 1 that consists of implicit semantic discovery and semantic alignment . The implicit semantic discovery module extracts diverse semantics from the backbone features with a class2vec machine , where every data point is represented by an embedding vector . Each dimension in this semantic space is considered as a implicit semantic topic , which guides the following source and target feature space alignment as an intermediate signal . Based on the discovered implicit semantics , we firstly calculate the response of every implicit topic to the feature space with an attention layer . Specifically , we obtain the attention signal of the backbone , such as ResNet , corresponding to every implicit semantic topic with a topic attention receptor , and assign attention weights to the features based on the signal strength . By taking the dot product of the attention weights and backbone features , we obtain a weighted feature mask to each implicit topic . The final semantic response alignment is achieved by forcing the source and target mask vectors corresponding to the same implicit topic to be as similar as possible . Implicit semantic guided features allow our model to transfer the relevant semantic information contained by the samples from the extra classes with similarity in certain semantics and boost the positive transfer . In the following section , we elaborate the implicit semantic response alignment details . 3.2 Implicit Semantic Discovery and Alignment . Given a source domain with ns labeled samples associated with Cs categories and a target domain of nt unlabeled samples associated with Ct categories , Partial Domain Adaptation ( PDA ) occurs when the source label space subsumes the target label space , i.e. , Ct ⊂ Cs . To tackle the partial domain adaptation , a PDA model B extracts visual features and reduces the feature dimension down to d with a bottleneck layer . On this d-dimensional space , B then matches the labeled source domain Ds = { Xs , Y s } and unlabelled target domain Dt = { Xt } . To further improve exsiting PDA methods , we propose the class2vec machine to seek the implicit semantics shared by various categories and align the semantic response between source and target data . Let f denote the class2vec machine and gj is the topic attention receptor for the j-th dimension of implicit semantic space Z . To be exact , f extracts the implicit semantic embedding from the feature space of a certain backbone PDA model B and gj calculates the response of X to the j-th implicit semantic topic , i.e. , f : X → Z ⊆ Rde , gj : X → Zj ⊆ R , for j ∈ ( 1 , ... , de ) , where de is the number of implicit semantic topics . The semantic topic extractor , as well as the topic attention receptors , are shared by both source and target domains . Implicit semantic discovery . We use an auto-encoder to achieve the implicit semantic topic extraction from the backbone feature space X . The auto-encoder consists of the class2vec machine , also known as encoder f and decoder h : Z → X ⊆ Rd which maps the hidden space back to a reconstruction of the original features . This implicit semantic discovery module is trained on the l2 reconstruction error term as follows : Lc2v = 1 ns ns∑ i=1 ||xsi − h ◦ f ( xsi ) ||22 + 1 nt nt∑ i=1 ||xti − h ◦ f ( xti ) ||22 , ( 1 ) where ◦ denotes the function composition . Topic attention reception . In order to capture the interaction between the backbone features and each implicit semantic topic , we employ a topic attention receptor gj to regress X on Zj , j ∈ ( 1 , ... , de ) . The absolute gradients of gi ’ s input layer are treated as the attention signals of the backbone features corresponding to the j-th semantic topic . We add l1 regularization on the first layer to encourage sparsity of the gradients . The total prediction loss for all de regression models is expressed as : Lreg = 1 de de∑ j=1 ( ns∑ i=1 ||f ( xsi ) j − gj ( xsi ) ||22 + nt∑ i=1 ||f ( xti ) j − gj ( xti ) ||22 + λreg P∑ p=1 ||wj , p||1 ) , ( 2 ) where f ( x ) j donates the j-th dimension of vector f ( x ) , and wj , p donates the p-th weight in the first layer of gj , which totally contains P activations in its first layer . Note that we only need the input gradients , so these multilayer perceptron regressors are trained independently from other parts of the model . Thus Lreg does not contribute to the total alignment loss . Semantic topic alignment . We encourage the semantics from source and target domains as similar as possible with semantic topic alignment . With the topic-specific attention , we further assign weights to the backbone features based on the semantic attention signals . For the j-th hidden topic , A ( j ) ∈ Rn×d , which is the absolute gradients of the i-th topic attention receptor gi with respect to the input feature X , is used as the attention weights of X . The attention feature mask for the j-th semantic topic is defined as Mj = N ( Aj ) ⊗X , where ⊗ denotes element-wise multiplication , and N ( · ) denotes l2 normalization operator . The source/target attention feature masks Ms/tj are the sub-matrices of Mj corresponding to the source/target features Xs/t , respectively . The final semantic alignment is guided by the loss term as follows : La = de∑ j=1 ||sum ( Msj , 1 ) − sum ( M tj , 1 ) ||22 , ( 3 ) where sum ( · , 1 ) returns a row vector containing the sum of each column . Overall Objective . Based on a backbone PDA model B , we propose our overall objective function as follows : min B , f , h LB + αLc2v + βLa , min gj Lreg , ( 4 ) where LB is the loss function of the backbone PDA model B , and j ∈ ( 1 , ... , de ) . We use hyperparameters α and β to balance between semantic discovery and alignment , respectively . The topic attention receptors gj are trained independently and do not propagate to other parts of the network .

Summary: This paper introduces a new partial domain adaptation (PDA) method. Different from previous methods that ignore the irrelevant source classes, this work argues that source data coming from extra classes also contributes to the adaptation. Motivated by this, the proposed method first identifies implicit semantic, then performs semantic alignment. The strength of this work is proved on common benchmarks.