ToAlign: Task-oriented Alignment for Unsupervised Domain Adaptation
Year: 2021

Abstract:
Unsupervised domain adaptive classifcation intends to improve the classifcation performance on unlabeled target domain. To alleviate the adverse effect of domain shift, many approaches align the source and target domains in the feature space. However, a feature is usually taken as a whole for alignment without explicitly making domain alignment proactively serve the classifcation task, leading to suboptimal solution. In this paper, we propose an effective Task-oriented Alignment (ToAlign) for unsupervised domain adaptation (UDA). We study what features should be aligned across domains and propose to make the domain alignment proactively serve classifcation by performing feature decomposition and alignment under the guidance of the prior knowledge induced from the classifcation task itself. Particularly, we explicitly decompose a feature in the source domain into a task-related/discriminative feature that should be aligned, and a task-irrelevant feature that should be avoided/ignored, based on the classifcation meta-knowledge. Extensive experimental results on various benchmarks (e.g., Offce-Home, Visda2017, and DomainNet) under different domain adaptation settings demonstrate the effectiveness of ToAlign which helps achieve the state-of-the-art performance. The code is publicly available at https://github.com/microsoft/UDA.

heading: 1 Introduction
text: Convolutional Neural Networks (CNNs) have made extraordinary progress in various computer vision tasks, with image classifcation as a most representative one. The trained models generally perform well on the testing data which shares similar data distribution to that of the training data. However, in many practical scenarios, drastic performance degradation is observed when applying such trained models to new domains with domain shift [57], where the data distributions between the training and testing domains are different. Fine-tuning on labeled target data is a direct solution but is costly due to the requirement of target sample annotations. In contrast, unsupervised domain adaptation (UDA) requires only the labeled source data and unlabeled target data to enhance the model’s performance on the target domain, which has attracted increasing interest in both academia [3, 2, 72, 58, 25, 31] and industry [63, 27].
There has been a large spectrum of UDA methods. Supported by the theoretical analysis [3], the overwhelming majority of methods tend to align the distributions of source and target domains. A line of works [6, 67, 43, 54, 55] explicitly align the distributions based on domain discrepancy measurements, e.g., Maximum Mean Discrepancy (MMD) [6]. Another line of alignment-based UDAs borrow ideas from Generative Adversarial Networks [19] and use domain adversarial training
∗This work was done when Guoqiang Wei was an intern at MSRA. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
to learn domain-aligned/invariant features, which dominate in the top performance methods. In the seminal work Domain Adversarial Neural Network (DANN) [17, 18], a domain discriminator is trained to distinguish the target features from source features while a feature extractor (generator) is trained to generate domain-invariant features to fool this discriminator. Following DANN, a plethora of variants have been proposed [58, 40, 51, 11, 50, 61, 38, 41, 14, 10, 64].
It is noteworthy that the goal of alignment in UDA is to alleviate the adverse effect of domain shift to improve the classifcation performance on unlabeled target data. Even though impressive progress has been made, there is a common intrinsic limitation, i.e., alignment is still not deliberately designed to dedicatedly/proactively serve the fnal image classifcation task. In many previous UDAs, as shown in Figure 1 (a), the alignment task is in parallel with the ultimate classifcation task. The assumption is that learning domain-invariant features (via alignment) reduces the domain gap and thus makes the image classifer trained on source readily applicable to target [3]. However, with alignment treated as a parallel task, there is a lack of mechanism to make it explicitly assist classifcation, where the alignment may contaminate the discriminative features for classifcation [28]. Previous works (e.g., CDAN [40]) exploit class information (e.g., predicted class probability) as a condition to the discriminator. MADA [42] implements class-level domain alignment by applying one discriminator per class. Their purpose is to provide additional helpful information to the discriminator [40] or perform class-level alignment [42], but they are still short of explicitly making alignment assist classifcation.
Some works move a step forward and investigate what features the networks should align for better adaptation. [62, 32] focus on transferable local regions, which are selected based on the uncertainty or entropy of the domain discriminator, for alignment. However, such self-induced feature selection is still not specifc to the optimization of classifcation task; instead, it is based on the alignment task itself. There is no guarantee that alignment positively serves the classifcation task. Hsu et al. [23] carry out object centerness-aware alignment by aligning the center part of the objects to exclude the background distraction/noise for domain adaptive object detection. However, the feature in object center position could be task-irrelevant and thus is not suited for alignment. Moreover, regarding such centerness feature as alignment objective is somewhat ad-hoc, which is still not designed directly from the perspective of assisting classifcation.
We pinpoint that the selection of "right" features to achieve task-oriented alignment is important. For classifcation, the essence is to train the network to extract class-discriminative feature. Similarly, for UDA classifcation, it is also desired to assure strong discrimination of the target domain features without class label supervision. Thus, we intend to align target features to the task-discriminative source features while ignoring the task-irrelevant ones. Note that for the feature of a source sample, it contains both task/classifcation-discriminative and task-irrelevant information, because the network is in general not able to suppress non-discriminative feature responses (e.g., responses unrelated to
image class or those related to other tasks such as alignment) perfectly [52, 9]. Aligning target features with task-irrelevant source features would prevent alignment from serving classifcation and lead to poor adaptation. Intuitively, for example, image style that is a non-causal factor for classifcation can be considered as task-irrelevant information and the bias towards such factor in alignment may hurt the classifcation task. We demonstrate this by conducting experiments where only the source task-irrelevant features are utilized to align with target i.e., the scheme Baseline+TiAlign in Figure 3. The performance of Baseline+TiAlign (in purple) on target test set drops drastically compared to the source-only method which dose not incorporate any alignment technique. This corroborates that aligning with task-irrelevant features is even harmful to the classifcation on target domain.
Motivated by this, in this paper, we propose an effective UDA method named Task-oriented Alignment (ToAlign) to make the domain alignment explicitly serve classifcation. We achieve this by performing feature alignment guided by the meta-knowledge induced from the classifcation task to make the target features align with task-discriminative source features (i.e., "positive" features), to avoid the interference from task-irrelevant features (i.e., "negative" features). Figure 2 conceptually illustrates the comparison between our proposed alignment and previous one. Particularly, as illustrated in Figure 1 (b), to obtain the suitable feature from a source sample for alignment with target samples, we leverage the classifcation task to guide the extraction/distillation of task-related/discriminative feature fp
s , from original feature fs . Correspondingly, for the domain alignment task, we enforce aligning target features with the source positive features by domain adversarial training to achieve task-oriented alignment. In this way, the domain alignment will better assist the classifcation task.
We summarize our main contributions as follows:
• We pinpoint that the selection of "right" features to achieve task-orientated alignment is important for adaptation. • We propose an effective UDA approach named ToAlign which enables the alignment to explicitly serve classifcation. We decompose a source feature into a task-relevant/discriminative one and a task-irrelevant one under the guidance of classifcation-meta knowledge for performing classifcation-oriented alignment, which explicitly guides the network what features should be aligned.
Extensive experimental results demonstrate the effectiveness of ToAlign. ToAlign is generic and can be applied to different adversarial learning based UDAs to enhance their adaption capability, which helps achieve the state-of-the-art performance with a negligible increase in training complexity and no increase in inference complexity.

heading: 2 Related Work
text: Unsupervised Domain Adaptation aims to transfer the knowledge from labeled source domain(s) to unlabeled target domain. Ben et al. [3] theoretically reveal that learning domain-invariant representations helps make the image classifer trained on source domain applicable to target domain. Various works learn domain-invariant features by aligning the source and target distributions measured by some metrics [6, 67, 43, 54, 55, 45], or by domain adversarial learning [58, 40, 51, 11, 50, 61, 38, 69, 56, 41, 14, 10, 64, 8, 29]. The latter is overwhelmingly popular in recent years owing to its
superiority in dealing with distribution problems [19]. Note that our proposed method is designed to enhance the capability of the widely used domain adversarial learning based approaches.
For domain adversarial learning based approach (e.g., DANN [17, 18]), in general, a domain discriminator is trained to distinguish the source domain from the target domain, meanwhile a feature extractor is trained to learn domain-invariant features. Many variants of DANN have been proposed [40, 14, 56, 10, 12, 56, 69, 35, 5]. CDAN [40] further conditions the discriminator on the image class information conveyed in the classifer predictions. MADA [42] implements class-wise alignment with multi-discriminators. GSDA [24] performs class-, group- and domain-wise alignments simultaneously, where the three types of alignment are enforced to be consistent in their gradients for more precise alignment. HDA [12] leverages domain-specifc representations as heuristics to obtain domain-invariant representations from a heuristic search perspective. CMSS [66] exploits Curriculum Learning (CL) [4] to align target samples with the dynamically selected source samples to exploit the different transferability of the source samples.
However, in these methods, the domain alignment is designed as a task in parallel with the image classifcation task. It does not explicitly take serving classifcation as its mission, where such alignment may result in loss of discriminative information. Jin et al. [28] remedy the loss of discriminative information caused by alignment via incorporating a restoration module. Wei et al. [64] pinpoint that alignment and classifcation are not well coordinated in optimization where they may contradict with each other. They thus propose to use meta-learning to coordinate their optimization directions.
In this paper, to make alignment explicitly serve classifcation, we propose a task-oriented alignment. Guided by the classifcation meta-knowledge, task-discriminative sub-features are selected for alignment. Different from [64], we investigate what features should be aligned to assist classifcation and intend to provide more interpretable alignment. We are the frst to perform task-oriented alignment by decomposing source feature into task-discriminative and task-irrelevant feature, and explicitly guides the network what sub-features should be aligned. Note that Huang et al. [26] propose to decouple features into domain-invariant and domain-specifc features, where the former ones are aligned for unsupervised person re-identifcation. [44, 7] exploit the VAE framework with several complex losses to perform the disentanglement from the perspective of domain and semantics simultaneously, and only use domain-invariant semantics for inference, leaving domain-specifc but task-related information underexplored. In contrast to focusing on the domain-level, our decomposition strategy focuses on the task-level guided by image classifcation task, where we further enable domain alignment on the task-discriminative features to proactively serve image classifcation.

heading: 3 Task-Oriented Alignment for UDA
text: Unsupervised domain adaptation (UDA) for classifcation aims to train a classifcation model on labeled source domain image set Xs and unlabeled target domain image set Xt to obtain high classifcation accuracy on a target domain test set.
Most popular adversarial learning based UDAs attempt to align the features of the source and target domains to alleviate the domain gap to improve the classifcation performance on target domain. As
mentioned before, aligning based on holistic features is sub-optimal, where such alignment does not explicitly serve classifcation. To address this, as illustrated in Figure 1 (b), we propose an effective task-oriented alignment to explicitly make the alignment serve classifcation. Particularly, we propose to decompose a source sample feature into a task-discriminative one that should be aligned, and a task-irrelevant one that should be ignored based on the classifcation meta-knowledge. Then, we perform alignment between the target features and the positive source features, which is consistent with the essence of the classifcation task, i.e., focusing on discriminative features.
In Sec. 3.1, to be self-contained, we briefy introduce adversarial learning based UDAs. We answer the question of what feature should be aligned to better serve classifcation and introduce our task-oriented feature decomposition and alignment in Sec. 3.2.

heading: 3.1 Recap of Domain Adversarial UDAs
text: Domain adversarial learning based UDAs typically train a domain discriminator D to distinguish which domain (i.e., source or target) a sample belongs to, and adversarially train a feature extractor G to fool the discriminator D in order to learn domain-invariant feature representations. The network is also trained under the supervision of image classifcation on the labeled source samples. Particularly, D is optimized to minimize the domain classifcation loss LD (i.e., binary cross entropy loss). Meanwhile, G is optimized to maximize the domain classifcation loss LD and minimize the image classifcation loss Lcls (i.e., cross entropy loss):
argmin LD, D (1) argmin Lcls − LD, G
To achieve adversarial training, usually, gradient reversal layer (GRL) [17, 18] which connects G and D is used via multiplying the gradient from D by a negative constant during the back-propagation to G. LD is typically defned as [18, 40, 14]:
LD(Xs, Xt) = −Exs∼Xs [log(D(G(xs)))] − Ext ∼Xt [log(1 − D(G(xt)))] , (2)

heading: 3.2 Task-oriented Feature Decomposition and Alignment
text: In adversarial learning based UDAs, a feature ingested by D as a holistic feature from a source or target sample, in general contains both task/classifcation-discriminative information and task-irrelevant information. Intuitively, aligning the task-irrelevant features would not effectively reduce the domain gap of the task-discriminative features and thus brings no obvious beneft for the classifcation task. Mistakenly aligning the target features with the source task-irrelevant features would hurt the discrimination power of the target features. We also experimentally confrm that in Figure 3, i.e., aligning with task-irrelevant features (TiAlign, line in purple) drastically reduces the classifcation accuracy on the target domain. Therefore, we propose to decompose a holistic feature of each source sample into a task-discriminative feature and a task-irrelevant feature to enable the task-oriented alignment with the target features.
Particularly, we softly select/re-weight (based on Grad-CAM [52]) the feature vector f s of a source sample to obtain task-discriminative feature fs that is discriminative for identifying the groundtruth p class, which we refer to as positive feature. Correspondingly, the task-irrelevant feature fs can ben obtained simultaneously, which we refer to as negative feature.
Task-Oriented Feature Decomposition. Grad-CAM [73, 52, 9] is a widely used technique to localize the most important features for classifcation in a convolutional neural network model. As analyzed in [73, 52, 9, 53], the gradients (w.r.t. the feature for classifcation) of the fnal predicted score corresponding to the ground-truth class convey the task-discriminative information, which identifes the relevant features to recognize the image class correctly. It is noteworthy that such task-discriminative information is, in general, highly related (but not limited) to the foreground object in the classifcation task. In this work, motivated by Grad-CAM, we propose to use the gradients of the predicted score corresponding to the ground-truth class as the attention weights to obtain the task-discriminative features.
As illustrated in Figure 1, we obtain a feature map F ∈ RH×W ×M (i.e., a tensor of non-negative real + numbers, with height H , width W , and M channels) from the fnal convolutional block (with ReLU
layer) of the feature extractor. After spatial-wise global average pooling (GAP), we have a feature vector f = pool(F ) ∈ RM . The logits for all classes are predicted via the classifer C(·). Based on the response C(f), we can derive the gradient wcls ∈ RM of yk w.r.t. f :
∂yk cls w = , (3) ∂f
where yk is the predicted score corresponding to the ground-truth class k. As analyzed in [52, 9, 53], cls the gradient w conveys the channel-wise importance information of feature f for classifying cls tothe sample into its groudtruth class k. We draw inspiration from Grad-CAM which uses w modulate the feature map in channel-wise to fnd the classifcation-discriminative features. Similarly, cls modulated with w , we can obtain the task-discriminative (i.e., positive) feature as: cls cls fp = w f = sw f , (4)p
cls where represents the Hadamard product, the attention weight vector w = swcls, where s ∈ R+p is an adaptive non-negative parameter to modulate the energy E(fp) = ||fp||2 of fp such that E(fp) = 2 E(f): vuut s PM s = ||f ||2 2
||wcls f2 mm=1 , (5)PM = f ||2 2 (wcls )2 m fmm=1
Motivated by the counterfactual analysis in [52], the task-irrelevant (i.e., negative) feature can be cls cls represented as fn = −w f , where −w delights the task-discriminative channels since the p p cls task-discriminative channels (with larger values in w ) correspond to ones with smaller values in p cls -w .p
To better understand and validate the discriminativeness of the positive and negative features, we visualize the spatial maps F with channels modulated by wcls and −wcls following [52, 73]. As shown in Figure 4, the positive information is more related to the foreground objects that provide the discriminative information for the classifcation task, while the negative one is more in connection with the non-discriminative background regions.
Task-oriented Domain Alignment. As discussed above, we expect the domain alignment to explicitly serve the fnal classifcation task. Given the source task-discriminative features obtained based on the classifcation meta-knowledge, we can guide the target features to be aligned with the source task-discriminative features fp through different domain adversarial learning based alignment methods [17, 18, 12]. The procedure is almost the same as that in UDAs discussed in Sec. 3.1, except that the input source feature fs to the fnal domain discriminator is replaced by the positive feature f s p of this source sample. Thus, the domain classifcation loss is defned with a small modifcation on Eq. (2):
LD(Xs, Xt) = −Exs∼Xs [log(D(Gp(xs)))] − Ext∼Xt [log(1 − D(G(xt)))] . (6)
where Gp(xs) = fs denotes the positive feature of source xs.p Understanding from the Meta-knowledge Perspective. To enable a better understanding of ToAlign on why it works well, here, we analyse ToAlign from the perspective of meta-learning with meta-knowledge.
In an adversarial UDA framework, the image classifcation task and domain alignment task can be considered to be a meta-train task T tr and a meta-test task T te , respectively. ToAlign actually introduces knowledge communication from T tr to T te . In the meta-training stage, we can obtain the prior/meta-knowledge φtr of T tr . Without effective communication between T tr and T te, the optimization of T te may contradict that of T tr, considering that they have different optimization goals. To improve the knowledge communication from T tr to T te, certain meaningful prior/meta-knowledge φtr is helpful for a more effective T te|φtr . A typical implementation of passing meta-knowledge from T tr to T te is based on gradients [39, 16, 34, 64, 33], i.e., rT tr, which provides knowledge of T tr . Other mechanisms e.g., leveraging the parameters regularizer in a way of weight decay, are also exploited [1, 71]. In our ToAlign, instead of encoding the meta-knowledge φtr into the gradients w.r.t. the parameters, we use T tr to learn/derive attention weights for identifying T tr-related sub-features in the feature space and then pass such prior/meta-knowledge φtr to T te to make meta-test task T te φtr adapt its optimization based on φtr .
In this work, actually, we are motivated by the reliable human prior knowledge on what should be aligned across domains to better assist classifcation task for UDA (i.e., task/classifcationdiscriminative features), while excluding the interference from task-irrelevant ones. Accordingly, in our design, we obtain the prior/meta-knowledge for identifying task-discriminative features from the classifcation task (meta-train) and apply it to the domain alignment task (meta-test) to achieve task-oriented alignment.

heading: 4 Experiments
text: To evaluate the effectiveness of ToAlign, we conduct comprehensive experiments under three domain adaptation settings, i.e., single source unsupervised domain adaptation (SUDA), multi-source unsupervised domain adaptation (MUDA) and semi-supervised domain adaptation (SSDA). For SSDA, domain adaptation is performed from labeled source domain to partially labeled target domain [15].

heading: 4.1 Datasets and Implementation Details
text: Datasets. We use two commonly used benchmark datasets (i.e., Offce-Home [60] and VisDA2017 [46]) for SUDA and a large-scale dataset DomainNet [43] for MUDA and SSDA. 1) OffceHome [60] consists of images from four different domains: Art (Ar), Clipart (Cl), Product (Pr), and Real-World (Rw). Each domain contains 65 object categories in offce and home environments. Following the typical settings [14, 12, 64, 40], we evaluate methods on one-source to one-target domain adaptation, resulting in 12 adaptation cases in total. 2) VisDA-2017 [46] is a synthetic-to-real dataset for domain adaptation with over 280,000 images across 12 categories, where the source images are synthetic and the target images are real collected from MS COCO dataset [37]. 3) DomainNet [43] is a large-scale dataset containing about 600,000 images across 345 categories, which span 6 domains with large domain gap: Clipart (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R), and Sketch (S). For MUDA, following the settings in [43, 66, 12, 33, 59], we evaluate methods on fve-sources to one-target domain adaptation, resulting in 6 MUDA cases in total. For SSDA, we take the typical protocal in [22, 48, 12], where there are 7 SSDA cases conducted on the 4 sub-domains (i.e., C, R, P and S) with 126 sub-categories selected from DomainNet. All methods are evaluated under the one-shot/three-shot setting respectively, where besides unlabeled samples, one/three sample(s) per class in the target domain are available during training.
Implementation Details. We apply our ToAlign on top of two different baseline schemes: DANNP [14, 64] and HDA [12]. DANNP is an improved variant of the classical adversarial learning based adaptation method DANN [17, 18], where the domain discrimination D is conditioned on the predicted class probabilities. HDA is a state-of-the-art adversarial training based method which leverages the domain-specifc representations as heuristics to obtain domain-invariant representations.
Method Time/ms GPU mem./MB Acc./%
DANNP 550 6,660 67.9 DANNP+ MetaAlign[64] 1,000 10,004 69.5 DANNP+ ToAlign 590 6,668 69.9
Table 3: Ablation study on the infu- one iteration) and GPU memory for a mini-batch ence of s in Eq. 5. with batch size 32.
We use the ResNet-50 [21] pre-trained on ImageNet [30] as the backbone for SUDA, while using ResNet-101 and ResNet-34 for MUDA and SSDA respectively. Following [64, 40, 12], the image classifer C is composed of one fully connected layer. The discriminator D consists of three fully connected layers with inserted dropout and ReLU layers. We follow [69] to take an annealing strategy η0to set the learning rate η, i.e., ηt = , where p indicates the progress of training that increases (1+γp)τ linearly from 0 to 1, γ = 10, and τ = 0.75. The initial learning rate η0 is set to 1e − 3, 3e − 4, 3e − 4, and 1e − 3 for SUDA on Offce-Home, SUDA on VisDA-2017, MSDA on DomainNet, and SSDA on DomainNet, respectively. All reported experimental results are the average of three runs with different seeds.

heading: 4.2 Ablation Study
text: Effectiveness of ToAlign on Different Baselines. Our proposed ToAlign is generic and applicable to different domain adversarial training based baselines, where we focus on what features to align instead of the alignment methods. The last four rows in Table 1 show the ablation comparisons on Offce-Home. Our ToAlign improves the accuracy of baseline DANNP and HDA by 2.0% and 1.1% respectively. As can be seen from the results in Table 1, Table 2, Table 5 and Table 6, our ToAlign can consistently bring signifcant improvement over the baseline schemes under different domain adaptation settings, i.e., SUDA, MUDA and SSDA. ToAlign enables the domain alignment task to proactively serve the classifcation task, resulting in more effective feature alignment for image classifcation.
Effectiveness of Different Ways to Obtain Positive Features. As mentioned in Sec. 3.2, we use cls cls w = sw as the attention weight (which conveys the classifcation prior/meta-knowledge) to p derive positive feature fp, where s is a parameter to modulate the energy of fp. We study the infuence of s under the setting of Rw→Cl on Offce-Home for our scheme DANNP+ToAlign and illustrate
the results in Table 3. As discussed around Eq. (5), we can use an adaptively calculated s, which achieves 2% improvement over the baseline on target test data. Moreover, we can treat s as a preset hyper-parameter. We found that the performance drops drastically if s is too small (e.g., s = 1). That is because the energy of the source positive feature will get too weak when s gets too small (e.g., the source feature f ’s average energy E(f) is about 800; if s = 1, the source positive feature’s average energy E(fp) is about 2). Then, it would be ineffective to align the target with the source positive features. When s is larger than 16, the performance signifcantly outperforms the baseline and approaches the result of using adaptive s. As an optional design choice, we could transform the weight wcls with certain activation function σ(·) such as Sigmoid or Softmax followed by a best selected cls scaling factor s, i.e., w = sσ(wcls). We found the results (i.e., 69.6/69.7 for Sigmoid/Softmax) p are close to that without activation function. We reckon that what is more important is the relative cls importance among the elements in w . For simplicity, we fnally take the adaptive s (cf. Eq. 5) for all experiments.

heading: 4.3 Comparison with the State-of-the-arts
text: Single Source Unsupervised Domain Adaptation (SUDA). We incorporate our ToAlign into the recent state-of-the-art UDA method HDA [12], denoted as HDA+ToAlign. Table 1 shows the comparisons with the previous state-of-the-art methods on Offce-Home. HDA+ToAlign outperforms all the previous methods and achieves the state-of-the-art performance. It is noteworthy that HDA+ToAlign achieves the best adaptation results on almost all the one-source to one-target adaptation cases thanks to the effective feature alignment for classifcation. The results on VisDA-2017 could be found in Appendix, where HDA+ToAlign outperforms HDA by 0.9%.
Multi-source Unsupervised Domain Adaptation (MUDA). Table 2 shows the results on DomainNet, where all the methods take ResNet-101 as the feature extractor. We build our Baseline based on HDA [12]. For simplicity, we replace the multi-class domain discriminator in the original HDA by a two-class one as in [58, 17, 66]. Note that CMSS [66] selects suitable source samples for alignment while our ToAlign selects task-discriminative sub-feature for each sample for task-oriented alignment. Compared with Baseline, ToAlign brings about 0.9% improvement and helps to achieve the best performance on this more challenging dataset.
Semi-supervised Domain Adaptation (SSDA). Table 5 and Table 6 show the results on one-shot and three-shot SSDA respectively, where all the methods use ResNet-34 as backbone. To compare with previous methods, we apply ToAlign on top of HDA. HDA+ToAlign outperforms HDA by 0.6%/1.2% for one-/three-shot settings, and surpasses all previous SSDA methods.

heading: 4.4 Complexity
text: In Table 4, we compare the training complexity and performance of ToAlign with baseline DANNP, and DANNP+MetaAlign [64] which incorporates meta-learning to coordinate the optimization of domain alignment and image classifcation. In contrast, inspired by the prior knowledge of what feature should be aligned to serve classifcation task, we distill such meta-knowledge from classifcation task and explicitly pass it to alignment task for classifcation-oriented alignment, eschewing complex optimization. Compared with baseline, ToAlign introduces negligible additional computational cost (only 7%) and occupies almost the same GPU memory as the baseline, which is much smaller than that of DANNP+MetaAlign, which almost doubles the computational cost due to its complex meta-optimization. Thanks to our explicit design which makes domain alignment effectively serve the classifcation task, our ToAlign achieves superior performance to MetaAlign.

heading: 4.5 Feature Visualization
text: We visualize the target feature response maps F (which will be pooled to be the input of the image classifer) of the Baseline (DANNP) and ToAlign in Figure 5. Baseline sometimes focuses on the background features which are useless to the image classifcation task, since it aligns the holistic features without considering the discriminativeness of different channels/sub-features. Thanks to our task-oriented alignment, in ToAlign, the features with higher responses are in general related to task-discriminative features, which is more consistent with human perception. More results can be found in the Appendix.

heading: 5 Conclusion
text: In this paper, we study what features should be aligned across domains for more effective unsupervised domain adaptive image classifcation. To make the domain alignment task proactively serve the classifcation task, we propose an effective task-oriented alignment (ToAlign). We explicitly decompose a feature in the source domain into a task-related feature that should be aligned and a task-irrelevant one that should be ignored, under the guidance of the meta-knowledge induced from the classifcation task itself. Extensive experiments on various datasets demonstrate the effectiveness of our ToAlign. In our future work, we will extend ToAlign to tasks beyond image classifcation, e.g., object detection and segmentation.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Program of China 2018AAA0101400 and NSFC under Grant U1908209, 61632001, and 62021001.

Review summaries:
The paper proposed a method "To-Align" for alignment of the domains in subspaces that have relevant information embedded for a given task. For the classification task, the paper demonstrates that using Grad-CAM for obtaining subspaces is effective. Results on the single source and multi-source benchmarks show consistent improvements in the performance of UDA.  The motivation behind the usage of attention for domain adaptation has been well studied in the existing literature. To fully understand the benefits of the proposed method, more experimental studies would be necessary. However, in the current form, the paper is marginally acceptable.

This paper address the problem of Unsupervised Domain Adaptation. In particular, the authors investigate if aligning only task-oriented features could help to improve the results. They propose an approach that is mainly based on Grad-CAM. They use Grad-CAM to decompose a holistic feature into a task-discriminative feature and a task-irrelevant feature by softly selecting weights for the features. Later these features are integrated with an adversarial approach for domain adaptation. The authors demonstrate improved performance on several datasets. Additionally, in the supplementary material, the authors demonstrate feature response maps for the baseline and proposed approach on the source and target datasets. The method seems to indeed pick task-relevant features even on the target dataset, which was not necessarily the case for the baseline.

The authors propose ToAlign, a method which reweights the source features in an adversarial domain adaptation pipeline based on their relevance in predicting the source label. This method could be used in conjunction with any alignment-based domain adaptation approach and has shown the ability to improve performance by focusing alignment on relevant features. The authors show the results of experiments in multiple domain adaptation settings and consistently improve upon baseline performance.