ToAlign: Task-Oriented Alignment for Unsupervised Domain Adaptation
Authors: Guoqiang Wei
Cuiling Lan
Wenjun Zeng
Zhizheng Zhang
Zhibo Chen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Domain adaptation
Image classification
Adversarial learning

Abstract: Unsupervised domain adaptive classifcation intends to improve the classifcation performance on unlabeled target domain. To alleviate the adverse effect of domain shift, many approaches align the source and target domains in the feature space. However, a feature is usually taken as a whole for alignment without explicitly making domain alignment proactively serve the classifcation task, leading to sub-optimal solution. In this paper, we propose an effective Task-oriented Alignment (ToAlign) for unsupervised domain adaptation (UDA). We study what features should be aligned across domains and propose to make the domain alignment proactively serve classifcation by performing feature decomposition and alignment under the guidance of the prior knowledge induced from the classifcation task itself. Particularly, we explicitly decompose a feature in the source domain into a task-related/discriminative feature that should be aligned, and a task-irrelevant feature that should be avoided/ignored, based on the classifcation meta-knowledge. Extensive experimental results on various benchmarks (e.g., Offce-Home, Visda-2017, and DomainNet) under different domain adaptation settings demonstrate the effectiveness of ToAlign which helps achieve the state-of-the-art performance. The code is publicly available at https://github.com/microsoft/UDA.

ToAlign: Task-oriented Alignment for Unsupervised Domain Adaptation
Guoqiang Wei!* Cuiling Lan
Wenjun Zeng²
Zhizheng Zhang?
Zhibo Chen
Microsoft Research Asia xgg7441email http:4mmmmml.usss...mm wsw. waste.edu..cm en (culan, wezeng, zhizzhang) @microsoft con.
University of Science and Technology of China
chenzhibo@uste.c eew.uastttteeecccmmmmmmmm cn
Abstract
Unsupervised domain adaptive classification intends improve the classification performance on unlabeled target domain. To alleviate the adverse effect of domain shift, many approaches align the source and target domains in the feature space. However, feature is usually taken as a whole for alignment without explicitly making domain alignment proactively serve the classification task, leading to sub- optimal solution. In this paper, we propose an effective mask--riented Aligmment (ToAlign) for unsupervised domain adaptation (UDA). We study what features should be aligned across domains and propose to make the domain alignment proactively serve classification by performing feature decomposition and alignment under the guidance of the prior knowledge induced from the classification task itself. Particularly, we explicitly decompose feature the source domain into leskerelatedddiseriiinaiie feature that should be aligned, and task-irrelevant feature that should be avoided/ignored, based on the classification meta-knowledge- Extensive experimental results various benchmarks (e.g.. Office-Home, Visda- 2017, and DomainNet) under different domain adaptation settings demonstrate the effectiivness of ToAlign which helps achieve the ofate-of-the the-art art performance. The code is publicly available at wittttt:................t / thub. com mi crosoft/Unn
1 Introduction
Convolutional Neural Networks (CNNs) have made extraordinary progress in various computer vision tasks, with image classification as most representative one. The trained models generally perform well on the testing data which shares similar data distribution to that of the training data. However, in many practical scenarios, drastic performance degradation is observed when applying such trained models to new domains with domain shift (57), where the data distributions between the training and testing domains to the requirement of target sample annotations. In contrast, unsupervised domain adaptation (UDA) requires only the labeled source data and unlabeled target data to enhance the model's performance on the target domain, which has attracted increasing interest in both academia and industry 663 27
different. Fine-tuning labeled target data is direct solution but is costly due
58
There has been a large spectrum of UDA methods. Supported by the theoretical analysis 3. the overwhelming majority of methods tend to align the distributions of source and target domains.
line of works 67 4 54 55 explicitly align the distributions based on domain discrepancy measurements, e.g., Maximum Mean Discrepancy (MMD) [6] Another line of alignment based
UDAs borrow ideas from Generative Adversarial Networks 19 and use domain adversarial training
This work done when Guoqiang Wei intern MSRA.
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

GRL layer
GRL loyer
Backbone
Rackbong
Target
Target
Shared
D
Lp
Shired
D
Lp
Backbone
C
Lackbona
Source
wp
C
fecss
Classification meta knowledge
Source
Baseline
(b) ToAlign
Figure 1: Illustration of adversarial learning based (a) Baseline and (b) our proposed ToAlign. D and C' denote domain discriminator and image classifier respectively. (a) Baseline (e.g.. DANN (8]) directly aligns the target feature f with the holistic source feature f*. Domain alignment and image classification tasks are optimized in parallel. (b) Our proposed ToAlign makes the domain alignment proactively serve the classification task, where target feature f is aligned with source task-discriminative "positive" feature f. which is obtained under the guidance of meta meta-knowledge induced from the classification task. denotes Hadamard product.
learn domain -aiaain-lllgned/ivariaan features, which dominate in the top performance methods. In the seminal work Domain Adversarial Neural Network (DANN) [17 [8], domain discriminator is trained to distinguish the target features from source features while a feature extractor (generator) invariant features to fool this discriminator. Following DANN, plethora
trained to generate of variants have been proposed [58 40 51
50 61
10 64).
It is noteworthy that the goal of alignment in UDA is to alleviate the adverse effect of domain shift to improve the classification performance on unlabeled target data. Even though impressive progress has been made, there is common intrinsic limitation, i.e., alignment is still not deliberately designed to redicatedly/proocttive
serve the final image classification fask. In many previous UDAs. as shown in Figure (a), the alignment task is in parallel with the ultimate classification task. The assumption that learning
features (via alignment) reduces the domain gap and thus makes the image classifier trained on source readily applicable to target [] However, with alignment treated as a parallel task, there a lack of mechanism to make it explicitly assist classification, where the alignment may contaminate the discriminative features for classification (8]. Previous works (e.g., CDAN (40]) exploit class information (e.g., predicted class probability) as a condition to the discriminator. MADA 42 implements class-level domain alignment by applying one discriminator per class. Their purpose is to provide additional helpful information to the discriminator 400 or perform class-level alignment classification.
42]. but they are still short of explicitly making alignment assist
Some works move a step forward and investigate what features the networks should align for better focus on transferable local regions, which are selected based on the uncertainty or entropy of the domain discriminator, for alignment. However, such self-induced feature selection still not specific the optimization of classification task; instead, is based on the alignment task itself. There is no guarantee that alignment positively serves the classification task. Hsu el al, [23 carry out object centerness-awas
adaptation. 62 32
alignment by aligning the center part of the objects to exclude the background distraction/noise for domain adaptive object detection. However, the feature in object center position could be task iis--irrleevant and thus is not suited for alignment. Morcover, regarding such centerness feature alignment objective is somewhat ad-hoc, which is still not designed directly from the perspective of assisting classification.
We pinpoint that the selection of "right" features to achieve task-oriented alignment is important. For classification, the essence is to train the network extract class- miiss-discriminative feature. Similarly, for UDA classification, it is also desired to assure strong discrimination of the target domain features without class label supervision. Thus, we intend to align target features to the disriiicrrmiieative source features while ignoring the task-irrelevant ones. Note that for the feature of source sample, it contains both saakillassffaaaoonddicciiiiianii is in general not able to suppress non-discriminative feature responses (e.g., responses unrelated to
and task-irrelevant information, because the network

---

Align
rr3
(a) Baseline
(b) ToAlign
Figure 2: Conceptual comparison between (a) previous alignment and (b) proposed task-oriented alignment. {f'} and {f* denote the sets target features and source features, respectively. (a) Previous methods take each source feature as a holistic one for alignment with target features. (b) We decompose each source feature f into a dask-discriminative positive feature f and task-irrelevant negative feature f. and make the target features to be aligned with the positive source features {f;;)
while avoiding aligning with the negative source features {
image class or those related to other tasks such as alignment) perfectly [52 Aligning target features
with task-irrelevant source features would prevent alignment from serving classification and lead to poor adaptation. Intuitively, for example, image style that is a non-causal factor for classification can be considered as task-irrelevant information and the bias towards such factor in alignment may hurt the classification task. We demonstrate this by conducting experiments where only the source irreeeeeel features are utilized align with target i.e., the scheme Baseline+ TiAlign in Figure 3 The performance of Baseline+TiAue TiAlign (in purple) on target test set drops drastically compared to the source-only method which dose not incorporate any alignment technique. This corroborates that aligning with task-irrelevan features is even harmful to the classification on target domain.
Motivated by this, in this paper, we propose effective UDA method named Task-oriented Aligmment (ToAlign) to make the domain alignment explicitly serve classification. We achieve this by performing feature alignment guided by the meta-knowledge induced from the classification task make the target features align with nask-disrrmminative source features (i.e., "positive" features), to avoid the interference from tusk-irreeevant features (i.e., "negative" features). Figure T conceptually illustrates the comparison between our proposed alignment and previous one. Particularly, as illustrated in [ (b), to obtain the suitable feature from source sample for alignment with target samples, we leverage the classification task to guide the extraction/distiilatiio of dusk-related/discriineed feature f,, from original feature f* Correspondingly, for the domain alignment task, we enforce aligning target features with the source positive features by domain adversarial training to achieve maik-triented alignment. In this way, the domain alignment will better assist the classification task.
Figure
We summarize our main contributions as follows:
We pinpoint that the selection of "right" features to achieve task-orientated alignment is important for adaptation.
We propose an effective UDA approach named ToAlign which enables the alignment to explic- itly serve classification. We decompose a source feature into a mivanrddeeaadiisiiiinati one and task-irrelevant one under the guidance of classification-ne meta knowledge for performing muasificaation-orintte alignment, which explicitly guides the network what features should be aligned.
Extensive experimental results demonstrate the effectiveness of ToAlign. ToAlign generic and can be applied to different adversarial learning based UDAs to enhance their adaption capability, which helps achieve the statc-of-the-att performance with a negligible increase training complexity and no increase in inference complexity.
Related Work
Unsupervised Domain Adaptation aims to transfer the knowledge from labeled source domain(s)
to unlabeled target domain. Ben et al. [3] theoretically reveal that learning domain-invariant representations helps make the image classifier trained on source domain applicable to target domain.
Various works learn domain-invariant features by aligning the source and target distributions measured
by some metrics
6.
67
4
54
55 45), or by domain adversarial learning [88 40 50 61
38 69 [66 41
0
64
29
The latter is overwhelmingly popular in recent years owing to its
3

---

Image
Fest
Meg: Feet
Image
Pos Feet
Keg. Feor
17.5
35.0
22.5
320
43.0
423
Source Only
Figure 4: Visualization of dask-disrimiinaivee and task-irrelvant features. The positive features gen- erally focus the foreground objects which pro- vide the most discriminative information for clas- sification, while the negative focus non- discriminative background regions. The images are sampled from Office-Homes
400
2000
2003
4000
Classification accuracy on target (Rw-+CI in Office-Home) for different meth- ods. TiAlign denotes aligning target features with task-irrelevant source features.
Figure 3:
superiority in dealing with distribution problems 19 enhance the capability of the widely used domain adversarial learning based approaches.
Note that our proposed method is designed to
in general, domain dis- criminator is trained to distinguish the source domain from the target domain, meanwhile a feature extractor is trained learn domain-invarian features. Many variants of DANN have been proposed [40 [14 (66 [0] [12 (6) 669 335 55. CDAN [40] further conditions the discriminator on the image class information conveyed in the classifier predictions. MADA 422) implements class-wir wise align- ment with multi-discriminator. GSDA [24] performs class-, group- and domain-wise alignments simultaneously, where the three types of alignment are enforced to be consistent in their gradients for more precise alignment. HDA 12 leverages domain-specific representations as heuristics to obtain domain-invariant representations from heuristic search perspective. CMSS (66) exploits Curriculum Learning (CL) 4 to align target samples with the dynamically selected source samples exploit the different transferability of the source samples.
For domain adversarial learning based approach (e.g., DANN I7 18
However, in these methods, the domain alignment is designed as a task in parallel with the image classification task. It does not explicitly take serving classification its mission, where such alignment may result in loss discriminative information. Jin et al. [28] remedy the loss of discriminative information caused by alignment via incorporating restoration module. Wei al. (64) pinpoint that alignment and classification are not well coordinated in optimization where they may contradict with each other. They thus propose optimization directions.
meta-leaming coordinate their
In this paper, to make alignment explicitly serve classification, propose task-oriented alignment. Guided by the classification meta-knowledge, diseiiiccrmiinatiee sub-features are selected for align- ment. Different from (64). investigate what features should be aligned assist classification and intend to provide more interpretable alignment. We are the first to perform task-oriented alignment by decomposing source feature into -is-discriminative and task-irrelevant feature, and explicitly guides the network what sub-features should be aligned. Note that Huang et al. [26] propose to decouple features into domain-invarian and domain-specinf features, where the former ones aligned for unsupervised person re-identification.
exploit the VAE framework with several complex losses to perform the disentanglement from the perspective of domain and semantics simultaneously, and only use domain-invariant semantics for inference, leaving domain-specific but task-related informa- tion underexplored. In contrast to focusing the donain-level, our decomposition strategy focuses on the task-level guided by image classification task, where we further enable domain alignment on the task diss-iicrrmiiaaive features to proactively serve image classification.
3 Task-Oriented Alignment for UDA
Unsupervised domain adaptation (UDA) for classification aims to train classification model on labeled source domain image set X, and unlabeled target domain image set X, to obtain high classification accuracy a target domain test set.
Most popular adversarial learning based UDAs attempt to align the features of the source and target domains to alleviate the domain gap to improve the classification performance on target domain. As

---

mentioned before, aligning based on holistic features is sub-optimal, where such alignment does not explicitly serve classification. To address this, as illustrated in (igure([(b),, we propose an effective task-oriented alignment to explicitly make the alignment serve classification. Particularly, we propose to decompose a source sample feature into a diskimscrmminttvve one that should be aligned, and a lask-irrelevant one that should be ignored based on the classification meta- meta-knowledge Then, we perform alignment between the target features and the positive source features, which consistent with the essence of the classification task, i.e., focusing discriminative features.
In Sec. 3.1 question of what feature should be aligned to better serve classification and introduce our task-oriented feature decomposition and alignment in Sec. 3.2
be self- conffcontained, we briefly introduce adversarial learning based UDAs. We answer the
3.1 Recap of Domain Adversarial UDAS
Domain adversarial learning based UDAs typically train a domain discriminator D to distinguish which domain (i.e., source target) sample belongs to, and adversarially train a feature extractor G to fool the discriminator in order to learn comain-invariian feature representations. The network is also trained under the supervision of image classification on the labeled source samples. Particularly, D is optimized to minimize the domain classification loss Ln (i.e., binary cross entropy loss). Meanwhile, G is optimized to maximize the domain classification loss Ln and minimize the image classification loss Lete (i.e., cross entropy loss):
argmin LD.
D
argmin Lote - LD:
G
To achieve adversarial training, usually, gradient reversal layer (GRL) [17 [8] which connects G' and D is used via multiplying the gradient from D by a negative constant during the propaaaaptntionn to G. Ln is typically defined as
[18 40 14]
LD(X,, X,) = -x x, [og(D(G(xx))) - Ex,xx, DDG()))(
3.2 Task-oriented Feature Decomposition and Alignment
In adversarial learning based UDAs, a feature ingested by / as a holistic feature from a source or target sample, in general contains both information. Intuitivelyy aligning the lask-irreeevant features would not effectively reduce the domain gap of the dikkiiicrimietevvee features and thus brings no obvious benefit for the classification task. Mistakenly aligning the target features with the source task-irrelevant features would hurt the discrimination power of the target features. We also experimentally confirm that in Figure 3 i.e.. aligning with task irrelevant features (TiAlign, line in purple) drastically reduces the classification accuracy on the target domain. Therefore, we propose to decompose a holistic feature of each source sample into disk-iiscriminative feature and a task-irrelevant feature to enable the task-oriented alignment with the target features.
information and task-irrelevant
Particularly, we softly select/re-weige (based on Grad-CAM (52)) the feature vector f* of source sample obtain fask-disrimmnnttve feature f that is discriminative for identifying the groundtruth class, which we refer to as positive feature. Correspondingly, the task-irrelevant feature f. can be obtained simultaneously, which we refer to as negative feature.
Task-Oriented Feature Decomposition. Grad-CAM [3 52 is a widely used technique to localize the most important features for classification in convolutional neural network model. As 53 the gradients (w.r.t the feature for classification) of the final predicted score corresponding the ground-truth class convey the task-discriminative information, which identifies the relevant features to recognize the image class correctly. It is noteworthy that such mas-discriminnttie information is, in general, highly related (but not limited) to the foreground object in the classification task. In this work, motivated by CAM, we propose to use the gradients of the predicted score corresponding the ground-truth class as the attention weights to obtain the musk-discriminativ features.
analyzed in 73
52 9
As illustrated in Figure 1 we obtain a feature map F E HxWx (i.e., a tensor of non-negative real numbers, with height H,, width W, and M channels) from the final convolutional block (with ReLU
5

---

layer) of the feature extractor. After spatial-wiss global average pooling (GAP), we have feature vector f = pool(F) E RM.. The logits for all classes are predicted via the classifier C()). Based on the response (f)), we can derive the gradient Wets E IN of y w.r.t. f:
W dy Of
(3)
where y is the predicted score corresponding to the ground truth class k. As analyzed in [2 9 59 the gradient wels conveys the channel-wise importance information of feature for classifying the sample into its groudtruth class k. We draw inspiration from Grad-CAM which uses w to modulate the feature map in channel-wise to find the dlasifiattionddicriiinaaat features. Similarly, modulated with wals, we can obtain the task-discring losk-discrimiativee (i.e., positive) feature as:
als f= sw >
where represents the Hadamard product, the attention weight vector w =swels where R+ an adaptive non-negative parameter modulate the energy E {fp) || 112 of f, such that (f,)) & (()):
2 fll
EMe m=1 Sh Emen (1otle J2m)
(5)
Motivated by the counterfactual analysis in (52). the task-irrelevant (i.e., negative) feature can be represented as -w where -wp delights the aas-discriminative channels since the disk-discrmiinative channels (with larger values in wilt) correspond to ones with smaller values in -wol..
To better understand and validate the discriminativeness of the positive and negative features, visualize the spatial maps F with channels modulated by w and -wols following [52 73]. As shown in Figure 4 the positive information is more related to the foreground objects that provide the discriminative information for the classification task, while the negative one is more in connection with the non-diseriminative background regions.
Task-orienter Domain Alignment. As discussed above, we expect the domain alignment to ex- plicitly serve the final classification task. Given the source dask-iisrrmminatie features obtained based on the classification meta- meta-knowledge, we can guide the target features to be aligned with the source diss-iiirrmmiaaive features f, through different domain adversarial learning based alignment methods [7 [8 [2). The procedure is almost the same as that in UDAs discussed in Sec. [.I] except that the imput source feature f to the final domain discriminator is replaced by the positive feature f of this source sample. Thus, the domain classification loss is defined with a small modification on Eq. (2)
LD (X, X.) = Ex,,x, [log(D(G"(xx (xx))) - Ex, [log(1 D(G(x))))
(6)
where GO = f.j denotes the positive feature of source X..
Understanding from the Meta-knowledge Perspective. To enable a better understanding of ToAlign why it works well, here, we analyse ToAlign from the perspective of meta-learning with meta-knowledge.
In an adversarial UDA framework, the image classification task and domain alignment task can be considered to be a meta-train task &t and a meta-test task T respectively. ToAlign actually introduces knowledge communication from T to T In the meta-training stage, we can obtain ø² of TTT, Without effective communication between Tht and T the optimization of ' may contradict that of T considering that they have different optimization goals. To improve the knowledge communication from T to TTx certain meaningful prior/meta-knont knwwlegee 6tr is helpful for more effective T Atr A typical implementation of passing meta-knowlest kndwledge from T to T is based on gradients [39 16, 34 64 33), i.e., V which provides knowledge of Ttr Other mechanisms e.g., leveraging the parameters regularizer in way of weight decay, also exploited 71]. In our ToAlign, instead of encoding the meta-knowledge der into the gradients w.t.t the parameters, use T to learn/derive attention weights for identifying reeatte sub-features in the feature space and then pass such prior//meaa-knowlede 6fr to T to make meta-test task The adapt its optimization based on "r
the http//mtaaknwwlldeee

---

Metiod
Ar-sCi
Ar-Rw
C1-RR
Pr-sAr
Pt-eCI
Po
Rw-sPt
Soll
34.9
50.0
58.0
37.4
38.5
31.2
60.4
59.9
MCDXCVER 18]
48.9
68.3
74.6
61.3
67.6
57.0
47.1
75.1
52.2
79.6
64.1
CDAN(NeurIPS 18)
50.7
70.6
76.0
57.6
70.0
70.0
574
50.9
77.3
70.9
56.7
81.6
65.8
ALDM/AANI'0)
10
53.7
70.1
76.4
60.2
72.6
71.5
56.8
51.9
77.1
70.2
56.3
82.1
66.6
SymNNarNeudPS 18) [9]
47.7
72.9
78.5
64.2
71.3
74.2
63.6
47.6
79.4
7388
50.8
82.6
67.2
19
53.1
72.3
77.2
59.7
53.1
78.4
72.4
82.9
67.6
MDDDCCME
78.1
72.5
68.1
BNMCCVPR
13.7
79.0
62.4
54.8
80.7
72.4
83.5
GSDAICVPR'20
61.3
79.4
73.3
74.3
65.0
53.2
80.0
72.3
60.6
83.1
70.3
GVB(CVPR'20)
57.0
14.7
79.8
64.6
24.1
74.6
65.2
55.1
81.0
74.6
59.7
54.3
70.4
E-MisiAAAT'3
$7.7
76.6
79.8
63.6
74.1
75.0
6344
56.4
79.7
72.8
62.4
85.5
70.6
MecuAliniCVRR331 bd
59.3
76.0
80.
65.7
74.7
75.1
65.7
56.5
81.6
74.1
85.2
71.3
DANNP [4
54.2
70.0
77.6
62.3
724
73.1
61.3
52.7
80.0
72.0
56.8
83.1
67.9
DANNP-ToAlign
56.8,
74.8,
79.9q
61.0g
73.9.
15.9,
65.8;
53.7,
81.1;
73.1,
58.2;
84.0q
69.9,
IDDMNeurIPS 20
56.8
75.2
79.8
65.1
73.9
75.2
66.3
56.7
81.8
7544
99.7
84.7
70.9
HDA-ToAlign
76.9;
75,6,
77,0,
75.1.
60.0-
84.9q
72.0
Table 1: Accuracy (%) of different UDAs on Office-Home with ResNer-50 as backbone. Best in bold.
In this work, actually, we are motivated by the reliable human prior knowledge on what should be aligned across domains to better assist classification task for UDA (i.e.. task/classificationn discriminative features), while excluding the interference from task-irrelevant ones. Accordingly, in our design, we obtain the prior/meta- -nwetttaa--wweeeeeeee for identifying task-discriminative features from the classification task (meta-train) and apply it to the domain alignment task (meta-test) achieve task-oriented alignment.
4 Experiments
To evaluate the effectiveness of ToAlign, we conduct comprehensive experiments under three domain adaptation settings, i.e., single source unsupervised domain adaptation (SUDA), multi-source unsu- pervised domain adaptation (MUDA) and semi suni-supee domain adaptation (SSDA). For SSDA, domain adaptation is performed from labeled source domain to partially labeled target domain [15
4.1 Datasets and Implementation Details
Datasets. We use two commonly used benchmark datasets (i.e., Office-Home (60 and VisDA- 2017 (46) for SUDA and a large-scale dataset DomainNet [43] for MUDA and SSDA. Office- Home (60) consists of images from four different domains: Art (Ar), Clipart (CI), Product (Pr), and Real-World (Rw). Each domain contains 65 object categories in office and home environments.
Following the typical settings [1, 12, 64 40). we evaluate methods on one-source to one-target domain adaptation, resulting in 12 adaptation cases in total. 2) VisDA-2017 46 is synthetic-to-ret dataset for domain adaptation with over 280,000 images across 12 categories. where the source images are synthetic and the target images are real collected from MS COCO dataset [37] 3) DomainNet 43 is large-scale dataset containing about 600,000 images across 345 categories, which span 6 domains with large domain gap: Clipart (C), Infograph (I), Painting (P), Quickdraw
(Q), Real (R), and Sketch (S). For MUDA, following the settings in [43 66 12
33 59) we evaluate
methods on five-sources to onc-target domain adaptation, resulting in 6 MUDA cases in total. For SSDA, we take the typical protocal in 22 48 [2). where there are SSDA cases conducted on the sub-domains (i.e., C, R, and S) with 126 sub-categories selected from DomainNet. All methods evaluated under the one-shot/three-she setting respectively, where besides unlabeled samples, one/three sample(s) per class in the target domain are available during training.
Implementation Details. We apply our ToAlign on top of two different baseline schemes: DANNP [4 (44) and HDA [12]. DANNP is an improved variant of the classical adversarial learning based adaptation method DANN [17 18, where the domain discrimination D is conditioned on the predicted class probabilities. HDA is otate-of-the-art adversarial training based method which representations.
leverages the domain-specific representations as heuristics to obtain
7

---

Methods
Clipari
Infograph
Painting
Quickdraw
Real
Sketch
Avg.
Source-Only 21
13.0g..
13.3 so
51.920
33.7
32.9
ADDAICVPR'T 17)
47.5
11.4
16.7
14.7
49. 1o.s
33.5
32.2
DANNICMM 17
45.51o.50
10.1
37.0
17.2
48.9 48.9
31.8
32.610.8
DCTN(CVPR 65
48. 48.6 Pa
23.5
48. 48.8
7.220.40
g3.5
47.3
38.2
MCDICVPR'1N)
24.3
22. 1a 70
45.7
7.6
58 45.4
43.5
38
M'SDA(ICCV119)
4
57.2
24.2
51 51.6
5.2.0.45
61 s9..9
49.610.50
41.510.74
M'SDA-9(CCCY19)
+
58.6,o.5a
26.0
$2.31o.55
6.31
62. 1o.5
19.5
42.610.44
MDAN(NeurIPS 18)
70
60.3
25.0
50.3
8.2
60.5
51.3
42.8
MLMSDA(Arsiv 20)
61.420.70 79
26.2 41
$1.920.00
19.1
57.02..04 1.04
50.3
44.3
GVRG/CVPR
21.5
23.9
46.210.46
16.42o
67.810.s
52.510.0
46.010.a
CMSS(ECCV'20)
66
64.2
28.0
53.61o.39
16.0
63.4
51.81o.35
16.5
IIDA(NerIPPS 20)
12
63.6.0.a5
25.9
56. 16.1
16.610
69.1
14.3
47.610.00
Baseline
66.420.24
24.7+0.10
57.3+0.10
11.5
69.2+0.21
55.2+0.18
47.3+0.19
Baseline-ToAlign
667.0.0.22
25.99
557,8
,22.240.1
70.7
56.0
.48.2
Table Accuracy (%) of different MUDA methods on DomainNet with ResNet-101 as backbone. Best in bold.
Method
Acc.
Method
Time/ms
GPU mem./MB
Acc./%
DANNP
67.9
DANNP
550
6,660
67.9
59.7
DANNP+
1,000
10,004
69.5
8
8
68.8
MetaAlign (64)
= 16
69.7
DANNP+
590
6,668
69.9
ToAlign
DANNp+ToAlign 64
70.0
128
69.8
Table 4: Training complexity comparison (on GTX TITAN X GPU) in terms of computational time (of one iteration) and GPU memory for mini-batch with batch size 32.
Adaptive
69.9
Table 3: Ablation study on the influ-
ence of in Eq.
We the ResNet-50 21 pre-trained ImageNet 30 the backbone for SUDA, while using ResNet-101 and ResNet-34 for MUDA and SSDA respectively. Following 64 40 12]. the image classifier C' is composed of one fully connected layer. The discriminator D consists of three fully connected layers with inserted dropout and ReLU layers. We follow [69] to take an annealing strategy the learning
7), ie.., ?) (I+>p)= where p indicates the progress of training that increases
4, 3e and le - 3 for SUDA on Office-Home SUDA on VisDA-2017, MSDA on DomainNet, and SSDA on DomainNet, respectively. All reported experimental results are the average of three runs with different seeds.
linearly from 1, 10, and 0.75. The initial learning rate no set le
4.2 Ablation Study
Effectiveness of ToAlign on Different Baselines. Our proposed ToAlign is generic and applicable to different domain adversarial training based baselines, where we focus on what features to align instead of the alignment methods. The last four rows in Table [ show the ablation comparisons on Office-Home. Our ToAlign improves the accuracy of bascline DANNP and HDA by 2.0 % and 1.1% % respectively. As can be seen from the results in Table I Table 2 Table 5 and Table 6 our ToAlign can consistently bring significant improvement over the baseline schemes under different domain adaptation settings, i.e., SUDA, MUDA and SSDA. ToAlign enables the domain alignment task to proactively serve the classification task, resulting in more effective feature alignment for image classification.
Effectiveness Different Ways to Obtain Positive Features. As mentioned Sec. 3.2 we use wel+ el as the attention weight (which conveys the classification aator/mmta-knowilede to derive positive feature fp. where is parameter to modulate the energy f.. We study the influence of under the setting of R on Office-Home for scheme MANNP+ToAAie and illustrate

---

Methods
R
.C
>P
P.
C-
8 >P
P. PR
Avg.
Sounce-Only
21
50.8
$6.9
DANNICMIP15)
58.2
56.3
52.8
57.4
70.3
ADR/CLRRT 18) 10
57.1
61.3
57.0
51.0
$6.0
49.0
72.0
57.6
CDAN(NeurP'S 18) 40
65.0
64.9
63.7
53.1
63.4
54.5
73.2
62.5
ENTiNeurIPS'O5) 20
65.2
65.9
65.4
54.6
59.7
52.1
75.0
62.6
MME(ICCV119) 88
70.0
67.7
69.0
56.3
64.8
61.0
76.1
66.4
CANNIAnii +
72.7
70.3
69.8
60.5
66.4
62.7
77.3
68.5
GVRGCCVPR 20)
14
70.8
65.9
71.1
62.4
65.1
76.8
68.4
HDM[NeurIPS 20)
12
72.4
71.0
71.0
63.6
68.8
64.2
79.9
70.0
HDA+ToAliga
73.0;
72,0;
71.7,
63.0,
69.3;
64.6;
80.8
70.6;
Table 5: Accuracy (%) of different one-shot SSDA methods on DomainNet with ResNet-34 as backbone. Best in bold.
Methods
R-+C
R-P
P-4C
C-48
R-S
Avg.
Source-Only
ADRICLR'
60.7
61.9
60.7
54.4
59.9
51.1
74.2
60.4
CDAN(NeudPS
10
69.0
67.3
68.4
57.8
65.3
59.0
78.5
66.5
ENT(Ncu:IPS'O5) A0
71.0
69.2
71.1
60.0
62.1
61.1
78.6
67.6
MME/ЛCCV119 15
72.2
69.7
71.7
61.8
66.8
61.9
78.5
68.9
MetaMME/ECCV 20)
33
73.5
70.3
72.8
62.8
68.0
63.8
79.2
70.1
GVBG 20)
4
73,3
68.7
72.9
65.3
66.6
68.5
79.2
70.6
CANN(Arii 20)
17
75.4
71.5
73.2
64.1
69.4
61.2
80.8
71.2
HDAINeudPS 20)
12
74.5
71.5
73.9
65.9
70.1
65.9
81.9
71.8
HDA-ToAlign
75.7,
72.9;
75.6-
66.2;
71.1,
66.4;
83.0
73.0;
Table 6: Accuracy (%) of different three-shot SSDA methods on DomainNet with ResNet-34 as backbone. Best bold.
the results in Table 3 As discussed around Eq. we can adaptively calculated s, which achieves 2% improvement over the baseline on target test data. Moreover, we can treat as a preset hyper-parameter. We found that the performance drops drastically if is too small (e.g., = 1). That is because the energy of the source positive feature will get too weak when gets too small (e-g., the source feature f's average energy E(f) is about 800; if s 1, the source positive feature's average energy (ff,) is about 2). Then, it would be ineffective to align the target with the source positive features. When is larger than 16, the performance significantly outperforms the baseline and approaches the result of using adaptive s. As an optional design choice, we could transform the weight wal with certain activation function (.) such Sigmoid or Softmax followed by a best selected scaling factor s, i.e., w (.. We found the results (i.e., 69.6/69.7 for Sigmoid/Softmax) are close to that without activation function. We reckon that what is more important is the relative importance among the elements in was For simplicity, finally take the adaptive (ef. Eq. 5) for all experiments.
4.3 Comparison with the State-of-the-arts
Single Source Unsupervised Domain Adaptation (SUDA). We incorporate our ToAlign into the recent state-of-the-art UDA method HDA [12], denoted as +/DA+ToAligh Table the compar- isons with the previous state-of-the-art methods on Office-Home. +Toligg outperforms all the previous methods and achieves the state-of-the-art performance. It is noteworthy that HDA+ + ToAlign achieves the best adaptation results on almost all the one-source to one-target adaptation cases thanks to the effective feature alignment for classification. The results on VisDA-2017 could be found in Appendix, where HDA+ + ToAlign outperforms HDA by 0.9%.
Multi-source Unsupervised Domain Adaptation (MUDA). Table 2 shows the results on Domain- Net, where all the methods take ResNet-101 as the feature extractor. We build our Baseline based on HDA [2]. For simplicity, we replace the multi-class domain discriminator in the original HDA by two-class one as
58 17 66). Note that CMSS (66] selects suitable source samples for alignment while our ToAlign selects task-discriminative sub-feature for each sample for task-oriented alignment. Compared with Baseline, ToAlign brings about 0.9% improvement and helps to achieve the best performance on this more challenging dataset.

---

Image
Baseline
ToAlign
Image
Baseline
ToAlign
Image
Baseline
ToAlign
Bed
Chair
Figure Visualization of the feature response maps on target test images. First row: of Office-Home Second row: Painting of DomainNet.
Semi-supervised Domain Adaptation (SSDA). Table and Table 6 show the results on one-shot and three-shot SSDA respectively, where all the methods ResNet-34 backbone. To compare with previous methods, we apply ToAlign on top of HDA. HDA+ HoA+ToAlign outperforms HDA by 0.6%/1.2% for one-/three-shot settings, and surpasses all previous SSDA methods.
4.4 Complexity
In Table 4 we compare the training complexity and performance of ToAlign with baseline DANNP: and DANNP+ DANNP+MetaAlign [64] which incorporates meta-learning to coordinate the optimization of domain alignment and image classification. In contrast, inspired by the prior knowledge of what feature should be aligned to serve classification task, we distill such meta-knowledge from classification task and explicitly pass it alignment task for classification-orienee eschewing complex optimization. Compared with baseline, ToAlign introduces negligible additional computational cost (only 7%) and occupies almost the same GPU memory the baseline, which much smaller than that of its complex pettaaoptimization. Thanks our explicit design which makes domain alignment effectively serve the classification task, our ToAlign achieves superior performance to MetaAlign.
alignment,
which almost doubles the computational cost due
4.5 Feature Visualization
We visualize the target feature response maps F (which will be pooled to be the input of the image classifier) of the Baseline (DANNP) and ToAlign in Figure Baseline sometimes focuses on the background features which are useless to the image classification task, since it aligns the holistic features without considering the discriminativeness our task-oriented alignment, in ToAlign, the features with higher responses are in general related to task-dicrimminative features, which is more consistent with human perception. More results can be found in the Appendix.
different channels/sub-featun
Thanks to
5 Conclusion
In this paper, we study what features should be aligned across domains for more effective unsuper- vised domain adaptive image classification. To make the domain alignment task proactively serve the classification task, we propose an effective task-oriented alignment (ToAlign). We explicitly decompose feature in the source domain into task-related feature that should be aligned and task-irrelevant one that should be ignored, under guidance of the meta-knowledge induced from the classification task itself. Extensive experiments various datasets demonstrate the effectiveness of our ToAlign. In our future work, we will extend ToAlign to tasks beyond image classification, e.g., object detection and segmentation.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Program of China 201 8AAA0101400 and NSFC under Grant U1908209, 61632001, and 6002
10

---

References
[1] Y. Balaji, S. Sankaranarayanan, and R. Chellappa. Metareg: Towards domain generalization using reta-regularizatioon NeurIPS, pages 998-,008, 2018.
[2] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine learning, 79(1 (9111)):5111755 2010.
[3] S. Ben-David, J. Blitzer, K. Crammer, and F. Percira. Analysis of representations for domain adaptation. In NeurIPS, volume 19, page 137. MIT, 1998, 2007.
[4] Y. Bengio, J. Louradour, R. Collobert, and Weston. Curriculum learning. In ICML, pages 41 41-48, 2009.
[5] R. Bermúdez Chacón, M. Salzmann, and P. Fua. Domain-adaptive multibranch networks. ICLR, 2020.
[6] K. M. Borgwardt, A. Gretton, M. J. Rasch, H.-P. Kriegel, B. Schölkopf, and A. J. Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22 22144):49--57, 2006.
[7] R. Cai, Z. Li, P. Wei, J. Qiao, K. Zhang, and Z. Hao. Learning disentangled semantic representation for domain adaptation. In IJCAI, volume 2019, page 2060, 2019.
[8] J. Cao, O. Katzir, P. Jiang, D. Lischinski, D. Cohen-Or, C. Tu, and Y. Li. Dida: Disentangled synthesis for domain adaptation. arXiv preprint arXiv: 1805.08019, 2018.
[9] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, Grad-cam++:: Generalized gradient- based visual explanations for deep convolutional networks. In WACV, pages 839 847, 2018.
[10] M. Chen, S. Zhao, H. Liu, and D. Cai. Adversarial-Icamtn loss for domain adaptation. In AAAI, pages 3521 3521-3528, 2020.
[11] Q Chen, Liu, Z. Wang, Wassell, and Chetty. we-weighted adversarial adaptation network for unsupervised domain adaptation. In CVPR, pages 7976-7985, 2018.
[12] S. Cui, X. Jin, S. Wang. He, and Q Huang. Heuristic domain adaptation. In NewrIPS, 2020.
[13] S. Cui, S. Wang. J. Zhuo, L. Li, Q. Huang, and Q. Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In CVPR, pages 3941-3950. 2020.
[14] S. Cui, S. Wang. J. Zhuo, C. Su, Q. Huang, and Q Tian. Gradually vanishing bridge for adversarial domain adaptation. In CVPR, pages 12455-12464, 2020.
[15] J. Donahue, J. Hoffman, E. Rodner, K. Saenko, and T. Damell. Semi-supervised domain adaptation with instance constraints. In CVPR, pages 668-675, 2013.
[16] C. Finn, P. Abbecl, and Levine. Model-agnostic meta-leamiing for fast adaptation of deep networks. In JCML, page 1126-1135, 2017.
[17] Ganin and Lempitsky. Unsupervised domain adaptation by hackpropagation. ICMI,, pages 1180-1189, 2015.
[18] Y. Ganin, Ustinova, H. Ajakan, Germain, H. Larochelle, Laviolette, Marchand, Lempitsky, Domain adversaraa training of neural networks, Journal of Machine Learning Research, 17(1):2096 2030, 2016.
[19] Goodfellow, Pouget-Abadie, Mirza, Xu. D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NeurlPS, pages 2672-2680, 2014.
[20] Y. Grandvalet, Y. Bengio, et al. Semi-supervised learning by entropy minimization. In NeurlPS, pages 281-296, 2005.
[21] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[22] T. Hospedales, A. Antoniou, P. Micaclli, and A. Storkey, Meta-leaming in neural networks: A survey. arXiv preprint arXiv:2004.05433 05539, 2020.
[23] C.-C. Hsu. Y.H. Tsai, Y.-Y. Lin, and M.-H. Yang. Every pixel matters: Center-aware feature alignment for domain adaptive object detector. In ECCV, pages 733-748.
[24] L. Hu, M. Kan. S. Shan, and X. Chen. Unsupervised domain adaptation with hierarchical gradient synchronization. In CVPR, pages 4043 4052, 2020.

---

[25] J. Huang, D. Guan, A. Xiao, and S. Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data, In NewrlPS, 2021.
[26] Y. Huang, P. Peng. Y. Jin, Y. Li, and J. Xing. Domain adaptive attention learning for unsupervised person re-identification. pages 11069-10070 2020.
[27] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized www-miiiid-tt--aaan co-cannnical adaptation networks. In CVPR, pages 12627- 12637, 2019.
[28] X. Jin, C. Lan, W. Zeng, and Z. Chen, Feature alignment and restoration for domain generalization and adaptation. arXiv preprint arXiv:2066. 12009, 2020.
[29] G. Kang. I.. Zheng. Y. Yan, and Y. Yang. Deep adversarial attention alignment for unsupervised domain adaptation: the benefit of target expectation maximization. In ECCV, pages 401-416, 2018.
[30] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, page 1097-1105, 2012.
[31] N. Kundu, N. Venkat, A. Revanur. R. V. Babu, et al. Towards inheritable models for open-set domain adaptation. In CVPR, pages 12376-12385, 2020.
[32] V. K. Kurmi, S. Kumar, and V. P. Namboodiri. Attending to discriminative certainty for domain adaptation. In CVPR, pages 491-500, 2019.
[33] D. Li and T. Hospedales. Online meta-lcarning for multi-source and semi-supervised domain adaptation. In ECCV, pages 382-403, 2020.
[34] D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales. Learning to generalize: Meta-lcarning for domain generalization. AAAI, 2018.
[35] S. Li, C. H. Liu, Q. Lin, B. Xie, Z. Ding, G. Huang, and J. Tang. Domain conditioned adaptation network. In AAAI, pages 1 11386-1 18886-11399 1133, 2020.
[36] Z. Li, Z. Zhao, Y. Guo, H. Shen, and J. Ye. Mutual learning network for multi-source domain adaptation. arXiv preprint arXiv:2003. 129444 2020.
[37] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. T.. Zütnick. Microsoft Common objects in context. In ECCV, pages 740-755, 2014.
coco:
[38] H. Liu, M. Long, J. Wang, and M. Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In /CML, pages 4013-4022, 2019.
[39] P. Liu and X. Huang. Meta-learning multi-task communication. arXiv preprint arXiv:1810.09980 2018.
[40] Long. Z. Cao, Wang, and L. Jordan. Conditional adversarial domain adaptation. In NeurIPS, pages 1655,1655, 2018.
[41] Z. Lu, Y. Yang, X. Zhu, C. Liu, Y.-Z Song, and T. Xiang. Stochastic classifiers for unsupervised domain adaptation. In CVPR, pages 9111 19209, 2020.
[42] 7. Pei, Z. Cao, M. Long, and J. Wang. Multi-adversarial domain adaptation. In AAAI, volume 32, 2018.
[43] X. Peng, Q Bai, X. Xia, Z. Huang. K. Sacnko, and B. Wang. Moment matching for multi-source domain adaptation. In ICCV, pages 1406-1415,,0111 2019.
[44] X. Peng, Z. Huang, X. Sun, and K. Saenko. Domain agnostic learning with disentangled representations. In ICML, pages 5102-5112, 2019.
[45] X. Peng and K. Saenko. Synthetic to real adaptation with generative correlation alignment networks. In WACV, pages 1982-1991. 2018
[46] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain adaptation challenge. arXiv preprint 171.v.1710.06924, 2017.
[47] C. Qin. L. Wang. Q. Ma, Y. Yin. H. Wang, and Y. Fu. Opposite structure learning for semi-supervised domain adaptation. arXiv preprint arXiv:2002.02544 2020.
[48] K. Saito, D. Kim, S. Sclaroff. T. Darrell, and K. Saenko. Semi-supervised domain adaptation via minimax entropy. In ICCV, pages 8050-8054 8058, 2019.
12

---

[49] K. Saito, Y. Ushiku, T. Harada, and K. Saenko. Adversarial dropout regularization. In ICLR, 2018.
[50] K. Saito, K. Watanabe, Y. Ushiku, and Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, pages 3723-3732, 2018.
[51] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa. Generate adapt: Aligning domains using generative adversarial networks. CVPR, pages 8503 8512, 2018.
[52] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-base based localization. In ICCV, pages 618 626, 2017.
[53] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop. 2014.
[54] B. Sun, J. Feng, and K. Sacnko. Return of frustratingly easy domain adaptation. In AAAI, 2016.
[55] B. Sun and K. Saenko, Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pages 443 450, 2016.
[56] H. Tang and K. Jia, Discriminative adversarial domain adaptation. In AAAI, volume 34, pages 5940-5947 2020.
[57] A. Torralba and A. A. Efros. Unbiased look dataset bias. CVPR, pages 1521-1528, 201
[58] E. Tzeng, J. Hoffman, K. Sacnko, and T. Darrell. Adversarial discriminative domain adaptation. In CVPR, pages 7167-7176, 2017.
[59] N. Venkat, J. Kundu, D. K. Singh, A. Revanur, and R. Babu. Your classifier can secretly suffice multi-source domain adaptation. In NeurlPS, 2021.
[60] H. Venkateswara, J. Eusebio, S. Chakraborty, and Panchanathan. Deep hashing network for unsupervised domain adaptation. CVPR, pages 5018-5027, 2017.
[61] R. Volpi, P. Morerio, S. Savarese, and V. Murino. Adversarial feature augmentation for unsupervised domain adaptation. In CVPR, pages 5495-5504, 2018
[62] X. Wang, L. Li, W. Ye, M. Long, and J. Wang. Transferable attention for domain adaptation. In AAAI, volume 33, pages 5345-5352 2019.
[63] Z. Wang. Yu, Wei, Feris, Xiong, W.-m. Hwu, Huang, and Shi. Differential treatment for stuff' and things: simple unsupervised domain adaptation method for semantic segmentation. In CVPR, pages 12635-12644 2020.
[64] G. Wei, C. Lan, Zeng, and Z. Chen. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. CVPR, 2021.
[65] R. Xu, Z. Chen, W. Zuo, J. Yan, and L. Lin. Deep cocktail network: Multi-souree unsupervised domain adaptation with category shift. CVPR, pages 3964 3973, 2018
[66] L. Yang, Y. Balaji, S.-N. Lim, and A. Shrivastava. Curriculum manager for source selection in multi-source domain adaptation. In ECCV, volume 12359, pages 608 624, 2020.
[67] Zellinger, T. Grubinger, E. Lughofer, Natschläger, and Saminger-Platt Platz. Central moment discrep- ancy (cmd) for domain-invariant representation learning. In ICLR, 2017.
[68] Y. Zhang. T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. In ICMI., pages 7404-7413, 2019.
[69] Y. Zhang, H. Tang, K. Jia, and M. Tan. Domain-symmetric networks for adversarial domain adaptation. In CVPR, pages 5031-5040, 2019.
[70] H. Zhao, S. Zhang, G. Wu, J. M. Moura, J. P. Costeira, and G. J. Gordon. Adversarial multiple source domain adaptation. In NewrIPS, volume 31, pages 8559-8570, 2018.
[71] L. Zhao, Peng. Y. Chen, M. Kapadia, and D. Metaxas. Knowledge as priors: Cross-modal knowledge generalization for datasets without superior knowledge. In CVPR, pages 6528-6537, 2020.
[72] L. Zhong, Z. Fang. F. Liu, J. Lu, B. Yuan, and G. Zhang, How does the combined risk affect the performance of unsupervised domain adaptation approaches? In AAAI, 2021.
[73] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In CVPR, pages 2921-2929, 2929, 2016.
13


Reviews:
[{"review_id": "zmzfx5VgQ8", "reviewer": "Reviewer_uRc3", "summary": "The paper proposes use of Grad-CAM to identify positively aligned features for domain adversarial training to improve the downstream classification task in the target domain. ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I liked the paper overall. The idea is intuitive, it is well explained and compared against related work in literature. The evaluation is thorough in terms of comparisons with baselines, considering multiple scenarios and performing ablations. The visualization of the features are very useful to understand the concepts presented.\n\nA few suggestions to improve the paper:\n- Error bars have been shown in the appendix. It would be nice to have a pointer in the main paper noting that the proposed method is statistically better after considering variance across three runs with details in Appendix. \n- The Grad-CAM based methods proposed only applies to vision based models. The way the paper is written, it looks like the proposed solution is more broadly applicable. It will be good to call out this limitation and perhaps propose alternative approaches for other domains.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The broader impact section is left to Appendix instead of main paper", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "j77GvWpj3bo", "reviewer": "Reviewer_qjmz", "summary": "The paper proposed a method \"To-Align\" for alignment of the domains in subspaces that have relevant information embedded for a given task. For the classification task, the paper demonstrates that using Grad-CAM for obtaining subspaces is effective. Results on the single source and multi-source benchmarks show consistent improvements in the performance of UDA.\n\nThe motivation behind the usage of attention for domain adaptation has been well studied in the existing literature. To fully understand the benefits of the proposed method, more experimental studies would be necessary. However, in the current form, the paper is marginally acceptable.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is of good quality and clarity. However, it is not up to the mark with regard to originality and significance (idea & results). Please see the below full review below.\n\n1. Using attention for DA has already been explored in many papers [Ref 1,2]. The goal of this line of research is to identify the interesting aspects in the image for the given task. There have also been several methods that consider feature level manipulation (such as negative and positive features) for DA [Ref 3]. It is not clear why the proposed \"ToAlign\" performs better than any other attention-based works as the paper does not compare against them. \n\n2. Recent work on MUDA at NeurIPS [Ref 4] showed remarkable performance on DomainNet while being simpler than existing MUDA methods. Please comment on the performance of your method against this work. \n\n3. The experimental setup: The current set of experiments demonstrate improvement over a couple of baselines. How would the proposed \"ToAlign\" framework help with respect to other methods? Can any method exploit the benefits of \"ToAlign\"? What are the limitations? I would expect that 'ToAlign' will not improve over [Ref 1,2].\n\n4. The method demonstrates only a marginal overall improvement (0.7%) on the SSDA dataset. However, the method is consistently improving the baseline as shown in Table  1 and Table 2. \n\n5. Are authors planning to release a reference implementation? \n\n6. Minor comments\n - L193:: Classifier C and #channels C have the same notation. \n\n-----\nPlease cite these related work and mention how these methods are different from yours:-\n\n[Ref 1] Vinod Kumar Kurmi, Shanu Kumar, Vinay P. Namboodiri, \"Attending to Discriminative Certainty for Domain Adaptation,\" CVPR'19\n\n[Ref 2] Guoliang Kang, Liang Zheng, Yan Yan, and Yi Yang, \"Deep Adversarial Attention Alignment for Unsupervised Domain Adaptation: the Benefit of Target Expectation Maximization\", ECCV'18\n\n[Ref 3] Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, Rahul M V, R. Venkatesh Babu, \"Towards Inheritable Models for Open-Set Domain Adaptation\" CVPR'20\n\n[Ref 4] Naveen Venkat, Jogendra Nath Kundu, Durgesh Kumar Singh, Ambareesh Revanur, Venkatesh Babu R., \"Your Classifier can Secretly Suffice Multi-Source Domain Adaptation\", NeurIPS'20", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations:\n\n* The authors point out that this approach works only with in domain adversarial settings and not in a pseudo-labelling settings. The * authors are encouraged to inlude limitations w.r.t to the usage of Grad-CAM in the feature selection process and how it may/may not improve other baselines.\n\nSocietal Impact:\n* The authors should comment on any negative bias carried over by the Grad-CAM based feature selection.\n* Does \"ToAlign\" ensure fairness if the source dataset consists of imbalanced classes? \n* Would \"ToAlign\" carry over source dataset bias in medical applications such as tumor segmentation?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "f0O_A3dMOoC", "reviewer": "Reviewer_6xwo", "summary": "This paper address the problem of Unsupervised Domain Adaptation. In particular, the authors investigate if aligning only task-oriented features could help to improve the results. They propose an approach that is mainly based on Grad-CAM. They use Grad-CAM to decompose a holistic feature into a task-discriminative feature and a task-irrelevant feature by softly selecting weights for the features. Later these features are integrated with an adversarial approach for domain adaptation. The authors demonstrate improved performance on several datasets. Additionally, in the supplementary material, the authors demonstrate feature response maps for the baseline and proposed approach on the source and target datasets. The method seems to indeed pick task-relevant features even on the target dataset, which was not necessarily the case for the baseline.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the proposed approach very interesting and original. The proposed approach is rather a heuristic in nature, but it is well-aligned with the line of works focusing on only aligning features that serve classification problems. To me, it looks like a good attempt to distinguish between causal and non-causal features for classification and align only causal ones. There are some works showing that having causal features would serve generalization (e.g. https://arxiv.org/abs/1907.02893), therefore I find this method for domain adaptation interesting. Even this link be more on an intuitive level in this work and demonstrated theoretically, I find experimental results quite convincing. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors clearly discuss limitations in the main and in the supplementary text.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "7TdWWjcOjkZ", "reviewer": "Reviewer_9jPd", "summary": "The authors propose ToAlign, a method which reweights the source features in an adversarial domain adaptation pipeline based on their relevance in predicting the source label. This method could be used in conjunction with any alignment-based domain adaptation approach and has shown the ability to improve performance by focusing alignment on relevant features. The authors show the results of experiments in multiple domain adaptation settings and consistently improve upon baseline performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main ideas of the work are clearly presented; using ToAlign to reweight source features before aligning source and target distributions is shown to empirically improve performance on several actively explored domain adaptation tasks. The relevance of this proposed approach is demonstrated not just by its performance on challenging benchmarks, but also due to the fact that it can be easily combined with other state-of-the-art domain adaptation approaches to further improve performance. \n\nWhile the underlying math and experiments are clearly defined and discussed, elements of the motivation of this work are confusing or don't contribute to a better understanding of the work. Particularly the understanding from the meta-knowledge perspective, detracts from clearly understanding the work's motivations and intuitions. Is the meta-knowledge discussed in this section shared or accumulated across multiple examples or is instance-specific? If instance-specific is meta-knowledge the appropriate mechanism of conceptualizing this information?\n\nWhile to the best of my knowledge the approach proposed in this work is original, the treatment of prior work leaves much to be desired. The work presents its as one of the only approaches to focus on task oriented alignment, but ignores several works on disentangled representations for domain adaptation which attempt the same goal. Positioning the works improvements and motivations solely on the focusing on task-oriented features and ignoring how it relates to other work which attempt the same goal may lead to inaccurate takeaways from the paper and is a substantial concern with the quality of the work. \n(Domain Agnostic Learning with Disentangled Representation - Peng et al)\n(Learning Disentangled Semantic Representation for Domain Adaptation - Cai et al)\n(DiDA: Disentangled Synthesis for Domain Adaptation - Cao et al)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors don't discussion potential negative societal impacts of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses Grad-CAM to reweigh the latent features in order to align the source and target domains only with respect to task-oriented features. The method is simple and effective. The idea is clearly motivated in the context of relevant literature and the experimental results clearly demonstrate the effectiveness of the method. All reviewers, including myself, find the paper an interesting and solid contribution that is worth publication.
Decision: Accept (Poster)