ToAlign: Task-Oriented Alignment for Unsupervised Domain Adaptation

1 Introduction . Convolutional Neural Networks ( CNNs ) have made extraordinary progress in various computer vision tasks , with image classifcation as a most representative one . The trained models generally perform well on the testing data which shares similar data distribution to that of the training data . However , in many practical scenarios , drastic performance degradation is observed when applying such trained models to new domains with domain shift [ 57 ] , where the data distributions between the training and testing domains are different . Fine-tuning on labeled target data is a direct solution but is costly due to the requirement of target sample annotations . In contrast , unsupervised domain adaptation ( UDA ) requires only the labeled source data and unlabeled target data to enhance the model ’ s performance on the target domain , which has attracted increasing interest in both academia [ 3 , 2 , 72 , 58 , 25 , 31 ] and industry [ 63 , 27 ] . There has been a large spectrum of UDA methods . Supported by the theoretical analysis [ 3 ] , the overwhelming majority of methods tend to align the distributions of source and target domains . A line of works [ 6 , 67 , 43 , 54 , 55 ] explicitly align the distributions based on domain discrepancy measurements , e.g. , Maximum Mean Discrepancy ( MMD ) [ 6 ] . Another line of alignment-based UDAs borrow ideas from Generative Adversarial Networks [ 19 ] and use domain adversarial training ∗This work was done when Guoqiang Wei was an intern at MSRA . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to learn domain-aligned/invariant features , which dominate in the top performance methods . In the seminal work Domain Adversarial Neural Network ( DANN ) [ 17 , 18 ] , a domain discriminator is trained to distinguish the target features from source features while a feature extractor ( generator ) is trained to generate domain-invariant features to fool this discriminator . Following DANN , a plethora of variants have been proposed [ 58 , 40 , 51 , 11 , 50 , 61 , 38 , 41 , 14 , 10 , 64 ] . It is noteworthy that the goal of alignment in UDA is to alleviate the adverse effect of domain shift to improve the classifcation performance on unlabeled target data . Even though impressive progress has been made , there is a common intrinsic limitation , i.e. , alignment is still not deliberately designed to dedicatedly/proactively serve the fnal image classifcation task . In many previous UDAs , as shown in Figure 1 ( a ) , the alignment task is in parallel with the ultimate classifcation task . The assumption is that learning domain-invariant features ( via alignment ) reduces the domain gap and thus makes the image classifer trained on source readily applicable to target [ 3 ] . However , with alignment treated as a parallel task , there is a lack of mechanism to make it explicitly assist classifcation , where the alignment may contaminate the discriminative features for classifcation [ 28 ] . Previous works ( e.g. , CDAN [ 40 ] ) exploit class information ( e.g. , predicted class probability ) as a condition to the discriminator . MADA [ 42 ] implements class-level domain alignment by applying one discriminator per class . Their purpose is to provide additional helpful information to the discriminator [ 40 ] or perform class-level alignment [ 42 ] , but they are still short of explicitly making alignment assist classifcation . Some works move a step forward and investigate what features the networks should align for better adaptation . [ 62 , 32 ] focus on transferable local regions , which are selected based on the uncertainty or entropy of the domain discriminator , for alignment . However , such self-induced feature selection is still not specifc to the optimization of classifcation task ; instead , it is based on the alignment task itself . There is no guarantee that alignment positively serves the classifcation task . Hsu et al . [ 23 ] carry out object centerness-aware alignment by aligning the center part of the objects to exclude the background distraction/noise for domain adaptive object detection . However , the feature in object center position could be task-irrelevant and thus is not suited for alignment . Moreover , regarding such centerness feature as alignment objective is somewhat ad-hoc , which is still not designed directly from the perspective of assisting classifcation . We pinpoint that the selection of `` right '' features to achieve task-oriented alignment is important . For classifcation , the essence is to train the network to extract class-discriminative feature . Similarly , for UDA classifcation , it is also desired to assure strong discrimination of the target domain features without class label supervision . Thus , we intend to align target features to the task-discriminative source features while ignoring the task-irrelevant ones . Note that for the feature of a source sample , it contains both task/classifcation-discriminative and task-irrelevant information , because the network is in general not able to suppress non-discriminative feature responses ( e.g. , responses unrelated to image class or those related to other tasks such as alignment ) perfectly [ 52 , 9 ] . Aligning target features with task-irrelevant source features would prevent alignment from serving classifcation and lead to poor adaptation . Intuitively , for example , image style that is a non-causal factor for classifcation can be considered as task-irrelevant information and the bias towards such factor in alignment may hurt the classifcation task . We demonstrate this by conducting experiments where only the source task-irrelevant features are utilized to align with target i.e. , the scheme Baseline+TiAlign in Figure 3 . The performance of Baseline+TiAlign ( in purple ) on target test set drops drastically compared to the source-only method which dose not incorporate any alignment technique . This corroborates that aligning with task-irrelevant features is even harmful to the classifcation on target domain . Motivated by this , in this paper , we propose an effective UDA method named Task-oriented Alignment ( ToAlign ) to make the domain alignment explicitly serve classifcation . We achieve this by performing feature alignment guided by the meta-knowledge induced from the classifcation task to make the target features align with task-discriminative source features ( i.e. , `` positive '' features ) , to avoid the interference from task-irrelevant features ( i.e. , `` negative '' features ) . Figure 2 conceptually illustrates the comparison between our proposed alignment and previous one . Particularly , as illustrated in Figure 1 ( b ) , to obtain the suitable feature from a source sample for alignment with target samples , we leverage the classifcation task to guide the extraction/distillation of task-related/discriminative feature fp s , from original feature fs . Correspondingly , for the domain alignment task , we enforce aligning target features with the source positive features by domain adversarial training to achieve task-oriented alignment . In this way , the domain alignment will better assist the classifcation task . We summarize our main contributions as follows : • We pinpoint that the selection of `` right '' features to achieve task-orientated alignment is important for adaptation . • We propose an effective UDA approach named ToAlign which enables the alignment to explicitly serve classifcation . We decompose a source feature into a task-relevant/discriminative one and a task-irrelevant one under the guidance of classifcation-meta knowledge for performing classifcation-oriented alignment , which explicitly guides the network what features should be aligned . Extensive experimental results demonstrate the effectiveness of ToAlign . ToAlign is generic and can be applied to different adversarial learning based UDAs to enhance their adaption capability , which helps achieve the state-of-the-art performance with a negligible increase in training complexity and no increase in inference complexity . 2 Related Work . Unsupervised Domain Adaptation aims to transfer the knowledge from labeled source domain ( s ) to unlabeled target domain . Ben et al . [ 3 ] theoretically reveal that learning domain-invariant representations helps make the image classifer trained on source domain applicable to target domain . Various works learn domain-invariant features by aligning the source and target distributions measured by some metrics [ 6 , 67 , 43 , 54 , 55 , 45 ] , or by domain adversarial learning [ 58 , 40 , 51 , 11 , 50 , 61 , 38 , 69 , 56 , 41 , 14 , 10 , 64 , 8 , 29 ] . The latter is overwhelmingly popular in recent years owing to its superiority in dealing with distribution problems [ 19 ] . Note that our proposed method is designed to enhance the capability of the widely used domain adversarial learning based approaches . For domain adversarial learning based approach ( e.g. , DANN [ 17 , 18 ] ) , in general , a domain discriminator is trained to distinguish the source domain from the target domain , meanwhile a feature extractor is trained to learn domain-invariant features . Many variants of DANN have been proposed [ 40 , 14 , 56 , 10 , 12 , 56 , 69 , 35 , 5 ] . CDAN [ 40 ] further conditions the discriminator on the image class information conveyed in the classifer predictions . MADA [ 42 ] implements class-wise alignment with multi-discriminators . GSDA [ 24 ] performs class- , group- and domain-wise alignments simultaneously , where the three types of alignment are enforced to be consistent in their gradients for more precise alignment . HDA [ 12 ] leverages domain-specifc representations as heuristics to obtain domain-invariant representations from a heuristic search perspective . CMSS [ 66 ] exploits Curriculum Learning ( CL ) [ 4 ] to align target samples with the dynamically selected source samples to exploit the different transferability of the source samples . However , in these methods , the domain alignment is designed as a task in parallel with the image classifcation task . It does not explicitly take serving classifcation as its mission , where such alignment may result in loss of discriminative information . Jin et al . [ 28 ] remedy the loss of discriminative information caused by alignment via incorporating a restoration module . Wei et al . [ 64 ] pinpoint that alignment and classifcation are not well coordinated in optimization where they may contradict with each other . They thus propose to use meta-learning to coordinate their optimization directions . In this paper , to make alignment explicitly serve classifcation , we propose a task-oriented alignment . Guided by the classifcation meta-knowledge , task-discriminative sub-features are selected for alignment . Different from [ 64 ] , we investigate what features should be aligned to assist classifcation and intend to provide more interpretable alignment . We are the frst to perform task-oriented alignment by decomposing source feature into task-discriminative and task-irrelevant feature , and explicitly guides the network what sub-features should be aligned . Note that Huang et al . [ 26 ] propose to decouple features into domain-invariant and domain-specifc features , where the former ones are aligned for unsupervised person re-identifcation . [ 44 , 7 ] exploit the VAE framework with several complex losses to perform the disentanglement from the perspective of domain and semantics simultaneously , and only use domain-invariant semantics for inference , leaving domain-specifc but task-related information underexplored . In contrast to focusing on the domain-level , our decomposition strategy focuses on the task-level guided by image classifcation task , where we further enable domain alignment on the task-discriminative features to proactively serve image classifcation . 3 Task-Oriented Alignment for UDA . Unsupervised domain adaptation ( UDA ) for classifcation aims to train a classifcation model on labeled source domain image set Xs and unlabeled target domain image set Xt to obtain high classifcation accuracy on a target domain test set . Most popular adversarial learning based UDAs attempt to align the features of the source and target domains to alleviate the domain gap to improve the classifcation performance on target domain . As mentioned before , aligning based on holistic features is sub-optimal , where such alignment does not explicitly serve classifcation . To address this , as illustrated in Figure 1 ( b ) , we propose an effective task-oriented alignment to explicitly make the alignment serve classifcation . Particularly , we propose to decompose a source sample feature into a task-discriminative one that should be aligned , and a task-irrelevant one that should be ignored based on the classifcation meta-knowledge . Then , we perform alignment between the target features and the positive source features , which is consistent with the essence of the classifcation task , i.e. , focusing on discriminative features . In Sec . 3.1 , to be self-contained , we briefy introduce adversarial learning based UDAs . We answer the question of what feature should be aligned to better serve classifcation and introduce our task-oriented feature decomposition and alignment in Sec . 3.2 .

Summary: The authors propose ToAlign, a method which reweights the source features in an adversarial domain adaptation pipeline based on their relevance in predicting the source label. This method could be used in conjunction with any alignment-based domain adaptation approach and has shown the ability to improve performance by focusing alignment on relevant features. The authors show the results of experiments in multiple domain adaptation settings and consistently improve upon baseline performance.