Implicit Semantic Response Alignment for Partial Domain Adaptation
Authors: Wenxiao Xiao
Zhengming Ding
Hongfu Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Partial Domain Adaptation
Transfer Learning
Implicit Semantic Discovery
Feature-Level Weighting Schema
Semantic Alignment

Abstract: Partial Domain Adaptation (PDA) addresses the unsupervised domain adaptation problem where the target label space is a subset of the source label space. Most state-of-art PDA methods tackle the inconsistent label space by assigning weights to classes or individual samples, in an attempt to discard the source data that belongs to the irrelevant classes. However, we believe samples from those extra categories would still contain valuable information to promote positive transfer. In this paper, we propose the Implicit Semantic Response Alignment to explore the intrinsic relationships among different categories by applying a weighted schema on the feature level. Specifically, we design a class2vec module to extract the implicit semantic topics from the visual features. With an attention layer, we calculate the semantic response according to each implicit semantic topic. Then semantic responses of source and target data are aligned to retain the relevant information contained in multiple categories by weighting the features, instead of samples. Experiments on several cross-domain benchmark datasets demonstrate the effectiveness of our method over the state-of-the-art PDA methods. Moreover, we elaborate in-depth analyses to further explore implicit semantic alignment. 

Implicit Semantic Response Alignment for Partial Domain Adaptation
Wenxiao Xiao
Zhengming Ding
Department of Computer Science Brandeis University
Department of Computer Science
Tulane University
Waltham, MA 02451
New Orleans, LA 701 18
wenxiaoxiio@rraneiis.eee
Besingl@tulann edu
Hongfu Liu
Department of Computer Science
Brandeis University Waltham, MA 02451
hongfuliuubrandeis. edu
Abstract
Partial Domain Adaptation (PDA) addresses the unsupervised domain adaptation problem where the target label space is a subset of the source label space. Most state-of-art PDA methods tackle the inconsistent label space by assigning weights classes or individual samples, in an attempt to discard the source data that belongs to the irrelevant classes. However, we believe samples from those extra categories would still contain valuable information promote positive transfer. this paper, we propose the Implicit Semantic Response Alignment to explore the intrinsic relationships among different categories by applying a weighted schema on the feature level. Specifically, we design a class2vec module to extract the implicit semantic topies from the visual features. With an attention layer, we calculate the semantic response according to each implicit semantic topic. Then semantic responses of source and target data are aligned to retain the relevant information contained in multiple categories by weighting the features, instead of samples. Experiments on several cross- domain benchmark datasets demonstrate the effectiveness of our method over the state-of-the the-art PDA methods. Moreover, we claborate in-depth analyses to further explore implicit semantic alignment.
Introduction
Deep neural networks have achieved impressive results in various supervised learning applications such as object recognition [15] 18), semantic segmentation [51 12 48) and multi-modal learning 34 40) but annotating large-scale dataset for deep learning models training could be tremendously grueling and expensive. Thus, considerable efforts have been dedicated to domain adaptation (DA), which attempts to circumvent labeling unfamiliar target data by transferring knowledge from well-studied source domain data. Existing DA methods align two iiffereettl-ddistiibbe by finding invariant representations from transferable feature structure 24). Matching the source and target distributions with discrepancy loss based high-order statistics such maximum mean discrepancy (MMD) [ is commonly used along with iterative pscudo pscudo labeling strategy []6 37 22]. Recently efforts seek comaann-nvariial feature space by adversarial learning, that the source classifier directly used for the target data prediction. Adding a weighting schema further improves the classification ability of adversarial methods. However, traditional domain adaptation
domains
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

requires related source domain that shares the same label space with the unlabelled target domain, which is not always available in real-life. Sometimes, target label space is only a subset of source labels space when people try to transfer knowledge from more comprehensive domain, and partial domain adaption (PDA) is introduced because of this practical situation.
The mismatched label space poses a difficult challenge where aligning all source domain with the small target domain could suffer from negative transfer problem. To this end, various PDA algorithms 12] 50, 3
37 [ extend the reweighting schema to alleviate negative transfer by reducing the influence of the irrelevant categories. nown-weighting the samples from the outlier source classes is one of the most commonly used strategies, while other recently works further assign weights on the instance level for both source and target samples to mitigate domain divergence. In general, these PDA methods aim to identify and discard all the irrelevant classes, in an attempt of aligning the marginal distributions only for the categories shared both domains. However, some extra classes are semantically correlated with the target classes, which could potentially contain valuable information for the PDA task. We believe that fully utilizing the relevant information hidden across different classes helps promote positive transfer. For example, cats and dogs have clear distinguished features for class separation; however, they also share many common semantic topics including fur, four legs, and so on. Therefore, we expect to explore the relationships among different categories by implicit semantics and achieve the semantic level alignment.
Contributions. In this paper, we propose the implicit semantic response alignment, as a plug- in module for any existing PDA models, which extracts implicit semantics from all categories including the shared categories and source-only categories and reduce the distribution discrepancy on the semantic level. Specifically, every sample is similarly decomposed into an embedding vector representing diverse implicit semantic topics with a class2vec machine. Under the guidance of each implicit semantic topic, we employ an attention- bbase weighting schema on the features by the implicit semantic response. The semantically weighted feature masks of the source and target are calibrated together as our final alignment. The major contributions are summarized as follows:
We exploit the relationships of all available categories by extracting implicit semantics from visual features, which allows our method to utilize relevant information contained in every sample, including those from the long neglected unshared categories in the PDA problem.
With the help of novel feature-level weighting strategy guided by implicit semantic responses, we align the source and target data distribution based on the implicit semantic topics shared between two domains to boost the positive transfer.
We demonstrate the effectiveness of our method by boosting the existing sfate-of-art PDA models on various benchmarks including Office31 (36), Office-Home (44) and ImageNet- Caltech 4 and provide several detailed in-depth explorations of our purposed method.
2 Related Work
Unsupervised domain adaptation (UDA) has attracted increasing attention over the past decades due to the high cost of annotating a massive amount of new data for training deep convolutional neural networks. Various UDA models have been established to overcome the distribution disparities between the source and target domains, among which two types of strategies are widely adopted. The
first type 42 28 25 27 30 matches two data distributions based on their high-order statistical properties to diminish domain discrepancy, such as maximum mean discrepancy (MMD) [I] Inspired by Generative Adversarial Networks (9) other UDA methods 6 7 41 resort to adversarial learning, which aims to capture domain-invariant feature representation that is capable of confusing the domain discriminator. Some works [33] 26 39 also incorporate reweighting schema derived from the interaction between discriminator and classifier to further align different domains. Additionally, iterative pscudo labeling is used by recent methods (5] 22 46] in an attempt to align the label spaces of different domains. However, in more realistic situation where the source and target label spaces are not identical, the prediction capability of these methods is seriously impaired by negative transfer due to the heterogeneous label distributions.
have been done on diverse aspects of transferring knowledge between domains with mismatched label space, while this work only focuses on a special case, partial domain adaptation (PDA) where the source label space subsumes
significant amount of research efforts [3
31
29
21
20
49
38
target label space. Most conventional PDA methods 2 50
3
[4 19
37 23 extend the deep

---

adversarial adaptation introduced by Comain-Adversaan Neural Network (7] with varied reweighting approaches. Selective Adversarial Network (SAN) 22 employs multiple class-wise discriminators to select out the source samples in the outlier classes and down-weight their importance during
transfer. Similarly, Importance Weighted Adversarial Nets (TWAN) 50 assesses the importance of source samples by one auxiliary domain classifier along with fixed source feature extractor. Partial Adversarial Domain Adaptation (PADA) [3] assigns class-level weights for both source classifier and domain adversary according to the estimated target label distribution. MultiiWeight Partial Domain Adaptation [4 exploits a multi-weight mechanism consisting of shared-class weights and shared-samee sample weights calculated by a shapld classifier distinguish outlier classes and samples. Deep Residual Correction Network (DRCN) [19] utilizes a weighted class-wise matching
strategy with plug-in in residual block that automatically identifies the most relevant source subclasses and aligns with them target data. Example Transfer Network (ETN) [4] weights source samples based on their discriminative information extracted by transferability quantifier.
Knowledge Transfer (A TK) [17 optimizes both domain-wise distribution adaptation and class-wise distribution alignment by iteratively filtering out confident task-relevant target samples and their corresponding source categories. Select, Label, and Mix (SLM) (37) integrates pseudo labelling for the target domain improve the discriminability of the invariant feature representation shared by the source and target domain. Lately, BA (23 incorporates weighting schema with augmenting target domain with source data, which achieves state-of-art on several PDA benchmark datasets. Generally, existing PDA methods involving adversarial training aim to transform the PDA task into an UDA- likee problem by matching the source and target data on the sample level. On the other hand, our method tries to transform the problem on the semantic level, then align two different domains based on the implicit semantics. Aside from adversarial learning. recent UDA methods adopting other adaptation approaches also yield superior results for some PDA benchmarks. Larger Norm More Transferable (AFN) 47 accomplishes alignment by reducing novel statistic distance designed by the authors that characterizes the rrean-feature-omm discrepancy between different domains. Source Hypothesis Transfer (SHOT) (22) proposes a source data-free framework that learns target-specific features that can be accurately recognized by the frozen classifier trained on the source domain, with the help of self-supervised pseudo-labelingg and information maximization.
Different from the existing literature, we propose the implicit semantic response alignment, plug- in module to add on the existing PDA frameworks, to explore the relationships among different categories. Rather than the category level transfer, our semantic level alignment gathers all semantic response from multiple related categories across source and target domains to boost positive transfer.
3 Methodology
In this section, we elaborate on our method for partial domain adaptation. First, the overview of the proposed framework is introduced. Then we provide detailed technical formation and objective function for each component in our model.
3.1 Framework Overview
To achieve implicit semantic alignment, we propose novel framework shown in Figure thht consists of implicit semantic discovery and semantic alignment. The implicit semantic discovery module extracts diverse semantics from the backbone features with a class2vec machine, where every data point is represented by an embedding vector. Each dimension in this semantic space is considered implicit semantic topic, which guides the following source and target feature space alignment as an intermediate signal, Based on the discovered implicit semantics, we firstly calculate the response of every implicit topic to the feature space with an attention layer. Specifically, we obtain the attention signal of the backbone, such as ResNet, corresponding to every implicit semantic topic with a topic attention receptor, and assign attention weights to the features based on the signal strength. By taking the dot product of the attention weights and backbone features, we obtain a weighted feature mask to each implicit topic. The final semantic response alignment is achieved by forcing the source and target mask vectors corresponding to the same implicit topic to be as similar as possible. Implicit semantic guided features allow our model to transfer the relevant semantic information contained by the samples from the extra classes with similarity in certain semantics and boost the positive transfer. In the following section, we elaborate the implicit semantic response alignment details.
3

---

Implicit Semantic Discovery
Semsaiic Raspoaso
Notwrrk
veas22ve
PDA Module
Hidden Semantic. A Alggmment Between Source and Target
Attention W Weighted Feature
Source
Column
Mean
Topic Attcation Receptor
Semantic Topic lignment
Column
Mean
Figure 1: Framework of our proposed implicit semantic response alignment, where the upper parts demonstrate the class2vec layers discover implicit semantics and convert each sample to semantic vector, while the lower parts illustrate the topic attention receptors calculate the weighted feature masks and the semantic alignment layers align source and target semantic response based on each implicit semantic topic for backhone feature extractor update.
3.2 Implicit Semantic Discovery and Alignment
Given a source domain with labeled samples associated with C, categories and target domain of n, unlabeled samples associated with C, categories, Partial Domain Adaptation (PDA) occurs when the source label space subsumes the target label space, i.e., C. C C.. To tackle the partial domain adaptation, a PDA model B extracts visual features and reduces the feature dimension down to d with bottleneck layer. On this d-dimensional space, B then matches the labeled source domain D = (X*, Y and unlabelled target domain D' = {X*). To further improve exsiting PDA methods, we propose the class2vec machine to seek the implicit semantics shared by various categories and align the semantic response between source and target data. Let f denote the class2vec machine and gj is the topic attention receptor for the j-th dimension of implicit semantic space Z. To be exact, f extracts the implicit semantic embedding from the feature space a certain backbone PDA model B and 9j calculates the response of X to the j-th implicit semantic topic, i.e., X Z C Rd 9, X' 2j S R, for j (1, de), where de is the number of implicit semantic topics. The semantic topic extractor, as well as the topic attention receptors, are shared by both source and target domains.
Implicit semantic discovery. We use an auto-encoder to achieve the implicit semantic topic ex- traction from the backbone feature space X. The auto-encoder consists of the class2vec machine, also known as encoder f and decoder h + X S RRd which maps the hidden space back to reconstruction of the original features. This implicit semantic discovery module is trained on the l2 reconstruction error term as follows:
1
Rhe 1 h f(x))1+++
Re || h f(z))
L.z0
The
where denotes the function composition.
Topic attention reception. In order to capture interaction between backbone features and each implicit semantic topic, we employ topic attention receptor yj to regress X on Zj. (1, de). The absolute gradients of g input layer are treated as the attention signals the backbone features corresponding to the j-th semantic topic. We add 4 regularization the first layer to encourage

---

sparsity of the gradients. The total prediction loss for all de regression models is expressed as:
1 Crey de
E Σ
The
- - (2))) > IIf(a))) 9n g Xreg Σ |1 11w.ell1) (2)
where f(r) donates the j-th dimension of vector [(x), and Wijp donates the p-th weight in the first layer of gj. which totally contains P activations in its first layer. Note that only need the input gradients, so these multilayer perceptron regressors are trained independently from other parts of the model. Thus Creg does not contribute to the total alignment loss.
Semantic topic alignment. We encourage the semantics from source and target domains as similar possible with semantic topic alignment. With the topie-specific attention, we further assign weights to the backbone features based on the semantic attention signals. For the j-th hidden topic, AG) E Raxdd which the absolute gradients of the i-th topic attention receptor gi with respect to the input feature X, is used as the attention weights of X. The attention feature mask for the j-th semantic topic is defined as Mj = N (A₂) @ X, where @ denotes element- multiplication, and N-) denotes 12 normalization operator. The source/target attention feature masks M the sub-matrices of Mj corresponding to the source/target features X respectively, The final semantic alignment is guided by the loss term as follows:
de
L. = Σ [ssm Mj, 1) - sum 1)1
(3)
where sum () returns a row vector containing the sum of each column.
Overall Objective. Based on a backbone PDA model B, we propose our overall objective function as follows:
min LB + aLe2v + BLa, min Lrey: B.f.h
(4)
95
where Lg is the loss function of the backbone PDA model B, and j (1,...,de) de). We use hyper- parameters and B to balance between semantic discovery and alignment, respectively. The topic attention receptors are trained independently and do not propagate to other parts of the network.
4 Experiments
In this section, we demonstrate the performance our proposed implicit semantic response alignment by comparing with several the experimental setup, report the algorithmic performance and finally provide several in-depth exploration of our method at various perspectives.
art partial domain adaptation methods. We first introduce
4.1 Experimental Setup
Datasets. We include three domain adaptation benchmark datasets for performance evaluation. (1) Office-Honne Home dataset [44] is a challenging benchmark that includes four domains: Artistic images (Ar), Clip Art images (CI), Product images (Pr) and Real World image (Rw). All four domains contain totally 65 object categories, and we likewise follow PADA and select the first 25 categories
(in alphabetic order) within each domain as partial target domain. (2) Office31 dataset (36 contains images of 31 object categories in three domains: Amazon (A), DSLR (D) and Webcam (W). These
31 categories are commonly encountered in office environment. Following the same data protocol as PADA (3). pick 10 categories shared by Office31 and Caltech256 [0] target domains. (3) ImageNet- Caltecch 4 is a large-scale object classification dataset that consists ImageNer-IK (I) B5 and Caltech256 (C), which contains 1,000 classes and 256 classes respectively
target domains.
Task donated as S-T means that S is the source domain and T is the target domain. For task I-C. the training of ImageNet-IA K is used source domain and subset of 84 classes from Caltech256 used as target domain. While for task C-11 the Caltech256 is used as target domain and we choose the same 84 classes from the validation set of ImageNet-1K K the target domain.
Baseline models. We compare our results with non-transfer ResNet-50 [T], UDA method
CDAN+E (26) and 9 state-of-the-art PDA methods including IWAN [50]. SAN 22. PADA [3],
5

---

Table 1: Accuracy for Partial Domain Adaptation on Office- Home
Method
Ar-CI
Ar-+Py
Cl+AAr Ar-Rw
Cl-+Pr
Ci-Rw
Pr-Ar
Pr-+C1
Pr-Rw
Rw-+Ar
Ru-CII
Rw-PPr
Avg.
RoNNa-50 [1
41.79
CDAN
26
47.52
57.07
54.12
63.42
72.39
66.02
611.73
IWAN 50
53.94
54.45
78.12
61.31
47.95
63.32
54. 17
52.02
81.28
76.46
56.75
82,90
63.56
SAN 2
44,42
68.68
14.60
67.49
64.99
77.80
99.77 73
44.72
50:07
72.18
50.21
78.66
65.30
PADA 5
$1.95
67.00
78.74
52,16
53.78
59.03
52.61
43.22
78.79
73.73
56.60
77.09
62.06
[4
55,39
76.70
ETN
59.20
77.03
79.51
62.92
65.73
75.01
68.29
53.57
84.37
75.72
57.56
81,50
70.15
DRCK [1
$4,00
76.40
K3,00
62.10
64.50
71.00
70,80
49.50
80.50
77.50
59.10
79,90
69.00
AFN 47
58.93
76.25
K1,42
70.43
72.97
77.78
72. 36
55. 34
K01-00
75.81
60.42
79.90
71.81
56.54
82.55
75.29
BAPUS [X3
60.62
83.16
88.39
71.75
72.79
83.40
73.45
61.59
86.55
79.25
62.80
86 16.05
75.98
Ours US
64,66
82.97
89,12
75.67
75,52
85.36
78.51
64.24
88.07
51.27
65.31
86.67
78.20
185
90
85
80
75
20
80
85
50
Cl-ikw
Pe-Ar
Ho-LI OurseHAUU
H-
Rw-AA
Rw-CI
Rem+P
Ourse&"TK
BA
SEOOT
OurasSHOT
Figure 2: Performance of our implicit semantic response alignment different backbones.
SLM 37 and BA We use the classifi- cation accuracy for evaluating the performance. The classification accuracy is computed for every partial domain adaptation task and the average accuracy across all tasks is reported for each dataset.
MWPDA 14], ETN A. DRCN 19. AFN
Implementation details. We implement our model in PyTorch [32] using one NVIDIA Titan GPU card. Our implementation on the purposed implicit semantic alignment module is added on the state-of-art PDA model BS (23. For all three datasets, we adopt the pre-trained ResNet-50 (TT] network the backbone feature extractor. ASn auto-encoder with one hidden layer that consists of encoder f and decoder is used as the class2vec machine. For each implicit semantic topic, we employ multi-layer perceptron regressor with | regulation the attention receptor 9j. We train the network with the standard stochastic gradient descent optimizer and the learning rate is set to 1e-3 initially and decay exponentially during training. The learning rate of the backbone feature extractor is 0.1 of other layers and the topic attention receptors do not back paok-prop to other parts of the model. a and |3 are both set to for Office- -ome and MmageeNet-Calleche while for Office3 I we set and B to 0.1 and 0.5. Areg is set to 0.5 in all experiments. For Office-Hoome, Office31 and NeteeeeeeeCalllccc the maximum iterations for training is to 8,000, 4,000 and 40,000, respectively. The numbers of implicit semantic topics are set to 256, 64 and 16 separately for Office-Home, Office3/ Calleech--Catehhh and Imagener- Caltech. We run our model five times with different random seeds and report the
average classification accuracy in the following section. We also add our method on two additional PDA models, SHOT 22 and A TK [77 to evaluate our method's generalization ability on different backbone PDA frameworks. Our code will be publicly available at: httpss//github..co//my reman-align/lmllict--e Seniiiiii--eem-ee--eeei Semantic-Responseeeee
SeanrppleeteSSenatte-eeeeooonane Response-Aligment
4.2 Algorithmic Performance
Here we comprehensively evaluate proposed method with several state-of-art PDA models. Table [ and Sind[ssoww the classification accuracy of 12 models on three benchmarks, where the best and second best results are bold and underline highlighted in the table. Some results are directly reported from BA'US (33 with the same protocol.
On the Office-Home dataset, ours method added with BA US achieves the best or second-best results on 11 out of 12 transfer tasks. Regarding the average accuracy, our method advances the state-of-art results obtained by the original BS by 2.22% on this dataset. Compared with BA BA'US, our method makes more than 2% improvement on 9 tasks and the accuracy only decreases on one specific task Pr. Notice that for Offece-Hmme, the state-of-art art vanilla unsupervised domain adaptation

---

Table 2: Accuracy for Partial Domain Adaptation on Office31 and Saaggeeer-Caltech
Method
Office3J
Inagehier Caltech
A-D
A-W
D-
D-W W
W-A
W-D
Avg.
I-+C
C-+1
Avg.
ResNet-50 11
83.44
75.59
83.92
96.27
84,97
98.09
$7.05
69.69
71.29
70.49
93.58
98,
72.45
72.24
TWAN
50
90.45
89.15
95.62
99.
94.26
99.36
94,69
78.06
73.33
75.70
94.27
94.15
99.32
88.73
99.36
94.96
77:75
76.51
PADA
82.17
86.54
92. 69
99.32
95.41
100.00
92.69
77.03
70.48
73.76
MWPDA 14
95.12
96.61
95.02
100.00
95.51
100.00
97.04
DRCN
86.00
88.05
95.00
100.00
95.80
100.00
94, 24
75.30
78.90
77.10
SIM 37
96.73
99.77
96.1
100.00
95.89
99.79
98.38
82.31
81.41
81.86
BA US 23
99.36
98.99
94.82
100.00
94.99
98.73
80
44.00
83.68
Ours BA'US
98.73
99.32
95.41
100.00
95.41
100.00
98.15
85.28
83.73
$4.50
0.060
Source
Source
-0.055
0.033
-0.045
0.050
-0.022
Taiget
larget
0.025
000011
0.000
0.000
0.000
(a) Entropy conditioning weight
(b) Source confidence weight
(c) Our attention weight
Source
Source
Target
Target
Target
(d) ResNet-50
(e) BA US
( Ours BA'US
Figure 3: (a-b) Entropy conditioning weight and source confidence weight of BA'US US in mini- batch task the Ar-CCI (c) Ours attention map for the same mini-batch; (d-f) t-SNE visualization of features from Resnet-50, BS and ours of task Ar-
method CDAN+E performs even worse than ResNet-50, which demonstrates that the inconsistent label space, if not controlled with specific algorithmic design, inevitably brings in negative transfer by conventional domain adaptation algorithm. Some PDA methods including IWAN and PADA only slightly outperform ResNet-50. This implies the challenge posed by the negative transfer for this dataset and the improvement made by our method indicates the efficiency of implicit semantic alignment in this difficult setting.
Our method obtains the second-best average accuracy on the Office31 dataset, but still slight improves the results of BS in 5 out of tasks. Office31 is small-scale dataset with 31 categories, which makes it difficult to extract implicit semantic topics shared across different categories. Moreover, the high accuracy of BA US also indicates that the shared categories already provide enough information for partial domain adaptation. On the other hand, our method achieves the best results for average SrageeNet-Calteed dataset. Specifically, adding
accuracy and both transfer tasks the large-scale our model to US boosts the accuracy by 1 1.28% for transfer task IIc, where the source domain contains large number of irrelevant samples.
Our proposed implicit semantic response alignment a plug-in module, which can be easily adapted to other domain adaptation backbones. Besides BS we further evaluate our method with two additional state-of-art PDA models SHOT 22 and PTT [17 on Office- Home. SHOT a source- data-free domain adaptation framework that performs well on Office- Home in the partial setting. To
7

---

adapt SHOT in our scenario, we saved the -i4--dimnssionll source features after training on the source domain. During the training, we made the target model to learn semantics that are shared with the fixed source features. T is PDA framework that directly trains on the fine-tuned ResNet-50 features. We run our experiment using the authors' code with fine-tuned features provided by this GitHub repository 45 and report our best results of ^ As shown in Figure 2 adding our method on SHOT slightly improves the results. We conjecture that SHOT forces its target feature extractor focusing only on features that are similar to the source domain without adversarial domain alignment, which makes implicit semantic alignment less effective in the original space. Yet for A our method significantly advances the average prediction accuracy by 3%.
Source Cilly Classes
Snated Classes
Table 3: Partial domain adaptation on individual class for task Ar CCI on Office-Home Home
TV
Laptop
0.6
Keyboard
Class
BAPUS
Ours
Improv. (%6
Monitor
Computer
99
44
12.12
59.60
47.48
04
Cliphoards
67.50
Eraser
40
18
0.00
0.00
0.00
Penci
Candles
99
79.80
70.71
Notebook
022
All classes
2427
1675
60.62
64.66
4.04
Postit Notes
Computer
Clipboards
0.0
Erase
Candles
Laptop
Keytteard
Montor
Printer
Pancil
Nosteook
Posti Conpuoo Computer
Captingtses
Candles
MAB
(a) Eraser
(b) Fraser
(c) Eraser
Figure 4: Category similarity
Figure 5: Example of confusing annotations
4.3 In-depth Exploration
In this part, we hereby discuss some properties of our purposed method. Specifically, we look into our weighting schema on the feature level and its effect on the feature space. Furthermore, we explore the relationship between classes to help understand how implicit semantic alignment impacts classification results in the scenario of partial domain adaptation. Finally, we elaborate the key hyperparameter analyses within our method.
Topic attention weighting. In Figure 3 we first showcase the feature-level attention weights with the weighting strategy of BA US (b) show the entropy-aware weights and source confidence weights of BA BA'US on the features of one mini-batch during training. By assigning weights samples and classes, the model suppresses the influence of negative transfer caused by irrelevant source samples. In Figure (c), we visualize our attention weights corresponding to one implicit semantic topic on features of the same mini-batch. As expected, our weighting schema discoveries the information that responds to the same implicit topic source and target samples on the feature level, which enables us coordinate two domains based on the semantic topics, and transfers the effective information from multiple classes both the partial instance and feature levels. Moreover, we provide the t-SNE 4 visualizations of the final features of ResNet-50, US and ours in Figure 3 (d-e). The different distribution of feature embeddings demonstrates that BA'US aligns the source and target samples with respect to classes, while our purposed method further divides the class-related clusters into smaller and well-separated clusters corresponding to the implicit semantic topics.
23 and on task Ar-CI on Office-Home. Figure (a) and
Cross-class interaction. We further explore the relationship between the target classes C, and the source only classes C,/C,, well as the impact of our purposed method on the cross-class interaction. For the task Ar-CI on Office- Home, our method improves the accuracy in 11 out of 25 classes, where only classes suffer from the performance drop. To check the relationship between classes, we draw the similarity matrix among the classes in the source domain with ResNet-50 features, then demonstrate shared classes from C, and irrelevant classes from C./C, in Figure 4 where the relationship between shared classes and irrelevant source classes are highlighted with blue dashed lines the bottom left corner. The per-class accuracy differences between our method and BA BA'US for these 4 shared target classes are shown in Table

---

00.3
AccumayyN)
8 2
po.2
57
52
67.5
8
200900- 888
oof & P 2 20
93
WW-
(a) Task A A-D
(b) Task D-
(c) Number of implicit topics
Figure 6: Parameter analysis on Office31, where (a-b) Performance with different a and task A- and D~>; (c) Performance with different implicit semantic topics on task A-W and W->A
Our method significantly elevates the performance of categories Computer and Clipboards by 47.48% and 20.00%, respectively. The heatmap shows that these two classes are highly correlated to some of the extra source classes that are also semantically similar to Computer and Clipboards in real-world scenario. This huge performance boost illustrates that our method indeed extracts useful information for irrelevant source samples guide by implicit semantic alignment, promoting positive transfer in PDA setring. On the other hand, the prediction accuracy of Candles class decreases by 9.09%, because there is valuable information in the unrelated samples for this class. Thus, forcing semantic alignment on this class could cause more severe negative transfer. The class Eraser is a failure case that we highlight here. It is very similar Marker and Pencil classes in the feature space, but does not benefit from our method. We look into the original images and notice that the annotations for some Eraser samples could be quite confusing in this dataset. As shown in Figure [] all three samples containing multiple objects are only labeled as Eraser; however, the main objects in these images should belong to Marker Figure a) or Pencil Figure (5b-c). In this case, our attention mechanism might be ineffective due to the confusing annotations in the dataset, since the feature extractor mainly focuses on the salient object instead eraser itself.
Figure[5(--c)
Hyperparameter analysis. We explore our model's 's sensitivity to hyperparameters on four different tasks Office31. Our loss function contains and / to balance the reconstruction loss and the alignment loss. As shown in Tigure[[(a) and (b), the performance decreases significantly when the ratio between a and grows. In this case, our model focuses only on extracting implicit topics, while ignoring the alignment. Another hyperparameter is the number of implicit topics de, which potentially controls the amount of semantic information extracted by our model. Figure(((c) stipulates that small number works best for this uncomplicated dataset, because increasing the number of topics may force our model to align irrelevant noise. Notice that the results with different numbers of implicit topics in task A- are inconsistent. After in-depth exam of the dataset, we believe the inconsistency are caused by the nature of the source data. The Amazon domain in Office3/ has more heterogeneous images than the Webcam domain. In each category of the Amazon domain, images from the same category are of different objects. For example, the bike category in Amazon domain contains 82 images of 82 different bikes, while Webcam domain only contains 21 images of 6 unique bikes. Figure 6 (c) shows that the results of task W are consistent, when Webcam is used as source domain. As for more challenging datasets, experiment results indicates that larger number improves the performance by including more semantic information. For X,, As we discussed in sectior3.2] it's only used for controlling the sparsity of the gradients in topic attention receptors, which are trained independently from other components in our model. In practice, this hyper-parameter does not have significant impact on the performance and empirically set Xyeg to 0.5 in our experiments.
5 Conclusion
In this paper, we presented Implicit Semantic Response Alignment, novel approach to boost the existing partial domain adaptation models by exploring inherent class relationship across both source and target domains. It uncovered implicit semantic topics from features extracted by backbone model, and exploited weighting strategy on the feature level with the help of topic-specific attention responses. The semantic-basee based alignment on the weighted features retained the relevant cross-domain

---

information contained in multiple categories. Extensive experiments on several partial domain adaptation benchmarks evinced the effectiveness of method over state-of-art methods.
Methodological Limitations. The basic assumption of our implicit semantic response alignment is that different categories have partial in common in the implicit semantic space. By transferring the knowledge in the semantic level, rather than the category level, the relevant information from multiple categories are utilized to boost the performance. However, if there little in common among different categories, such as cat and computer, our method might not bring in the improvements.
Broader Impact. Partial domain adaptation is practical setting in unsupervised domain adaptation, where the source label space subsumes the target label space. In this paper, we explore the relationship among different categories and present it by implicit semantics. Based this, propose the implicit semantic response alignment to boost the positive transfer. The methodological philosophy on category relationship exploration be adapted tackle other research problems, including information retrieval, recommendation systems, user modeling and so on.
Potential Negative Societal Impact. We address the well-defined partial domain adaptation problem and conduct experiments on benchmark dataset. It does not involve sensitive attributes and we do not notice any societal issues.
Funding This paper partially supported by NSF OAC 1920147.
10

---

References
[1] K. M. Borgwardt, A. Gretton, M. J. Rasch, H.-P. Kriegel, B. Schölkopf, and A. J. Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14)xe9--557 2006.
[2] Z. Cao, M. Long, J. Wang, and M. L. Jordan. Partial transfer learning with selective adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[3] Z. Cao, L. Ma, M. Long, and Wang. Partial adversarial domain adaptation. In Proceedings of the European Conference on Computer Vision, pages 135-150, 2018.
[4] Z. Cao, You, M. Long. J. Wang. and Q Yang. Learning to transfer examples for partial domain adaptation. In The IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[5] C. Chen, W. Xie, W. Huang, Y. Rong, Ding, Y. Huang, Xu, and Huang. Progressive feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[6] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pages 11 180-1189. PMLR, 2015.
[7] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain adversarial training of neural networks. The journal of machine learning research, 17(1):2096- 2030, 2016.
[8] M. Gong, K. Zhang, T. Liu, D. Tao, C. Glymour, and B. Schölkopf. Domain adaptation with conditional transfcrable components. In International conference on machine learning, pages 2839-2848. PMLR, 2016.
[9] Goodfellow, J. Pouget-Abadis M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014.
[10] G. Griffin, A. Holub, and P. Perona. Caltech-255 object category dataset. 2007.
[11] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell. Cycada: Cycle- consistent adversarial domain adaptation. In International conference on machine learning, pages 1989- 1998, PMLR, 2018
[13] T. M. H. Hsu, W. Y. Chen, C.-A. Hou, Y.-H. H. Tsai, Y.-. Ych, and Y.-.. F. Wang. Unsupervised domain adaptation with imbalanced cross-domain data. In Proceedings of the IEEE International Conference on Computer Vision, pages 4121-4129, 2015.
[14] J. Hu, H. Tuo, C. Wang, L. Qiao, H. Zhong, and Z. Jing. Multi- partial domain adaptation. In The British Machine Vision Conference, page 5, 2019.
[15] Y. Jin, X. Wang, M. Long, and J. Wang. Minimum class confusion for versatile domain adaptation. In European Conference on Computer Vision, pages 464-480, Springer, 2020.
[16] T. Jing, M. Shao, and 7. Ding. Discriminative cross-domain feature learning for partial domain adaptation. arXiv preprint arXiv:2008.1 00008.11360.... 2020.
[17] T. Jing, H. Xia, and Z. Ding. Adaptively accumulated knowledge transfer for partial domain adaptation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1606-166 1614, 2020.
[18] S. Li. C. Liu, Q Lin, B. Xie. Z. Ding, G. Huang, and J. Tang. Domain conditioned adaptation network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11386-11393, 2020.
[19] S. Li, C. H. Liu, Q. Lin, Q. Wen, L. Su, Huang, and Ding. Deep residual correction network for partial domain adaptation. JEEE Transactions on Pattern Analysis and Machine Intelligence, page 1-1, 2020.
[20] Y. Li, M. Gong, X. Tian, T. Liu, and D. Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[21] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference Computer Vision, 2018.
11

---

[22] J. Liang, D. Hu, and J. Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. 2020.
[23] J. Liang, Y. Wang, D. Hu, R. He, and J. Feng. A balanced and uncertainly-aware approach for partial domain adaptation. In Europaan Conference on Computer Vision, 2020.
[24] T. Liu, Q. Yang, and D. Tao. Understanding how feature structure transfers in transfer learning. In Proceedings of the 26th International Joint Conference Artificial Intelligence, page Press, 2017.
AAAI
[25] M. Long, Y. Cao. J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105, PMLR, 2015.
[26] M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pages 1645-1655 2018.
[27] M. Long. H. Zhu, J. Wang. and M. Jordan. Deep transfer learning with joint adaptation networks. In International conference machine learning. pages 2208-2217. PMLR, 2017.
[28] Y. Luo, T. Liu, D. Tao, and C. Xu. Decomposition based transfer distance metric learning for image classification, IEEE Transactions on Image Processing, 3789:388--38 2014.
[29] R. Müller, Kornblith, and Hinton. Subclass distillation. arXiv preprint arXiv:2002.039888 2020.
[30] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Nenworks, 22(2):19--10,, 2010.
[31] P. Panareda Busto and J. Gall. Open set domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 754 703,-663,222 2017.
[32] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bue, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, pages 8024 8035. 2019.
[33] Z. Pci, Z. Cao, M. Long, and J. Wang. Multi-adversarial domain adaptation, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[34] N. Rasiwasia, J. Costa Pereira, E. Coviello, G. Doyle, G. Lanckriet, R. Levy, and N. Vasconcelos. A new approach cross- modal multimedia retrieval. In Proceedings of the 18th ACM international conference on Multimedia, pages 251 251-260, 2010.
[35] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 25111122222222 2015.
[36] K. Saenko, Kulis, M. Fritz, and T. Darrell. Adapting visual category models new domains. In European Conference on Computer Vision, 2010.
[37] A. Sahoo, R. Panda, R. Feris, K. Saenko. and A. Das. Select, label, and mix: Learning discriminative invariant feature representations for partial domain adaptation. arXiv preprint wwXiv:2012.03355 2020.
[38] K. Saito, D. Kim, S. Sclaroff, and K. Saenko. Universal domain adaptation through self supervision. arXiv preprint arXiv:2002.07900 2020.
[39] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3723-3732, 2018.
[40] X. Shu, G.-J. Qi, J. Tang, and Wang. Weakly-shared deep transfer networks for heterogeneouu-domain knowledge propagation. In Proceedings of the 23rd ACM international conference on Multimedia, pages 35 44, 2015.
[41] F. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7167-7176, 2017.
[42] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 2014.
12

---

[43] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[44] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages
2017.
[45] J. Wang al. Everything about transfer learning and domain adapation. http://brannsferlearning
xyz
[46] Q. Wang and T. Breckon. Unsupervised domain adaptation via structured prediction based selective pseudo-labeling. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6243-6250, 2020.
[47] R. Xu,
Yang, and L. Lin. Larger norm more transferable: adaptive feature norm approach for unsupervised domain adaptation. In The IEEE International Conference on Computer Vision, 2019.
[48] Y. Yang and S. Soatto. Fourier domain adaptation for semantic segmentation. In Proceedings of IEEE Conference on Compuder Vision and Pottern Recognition, pages 4085-4095, 2020.
[49] You. M. Long. Z. Cao, J. Wang. and Jordan. Universal domain adaptation. In Proceedings of IEEE Conference on Compuder Vision and Pattern Recognition, pages 2720-2729, 2019.
[50] Zhang, Z. Ding, Li, Ogunbona. Importance weighted adversarial nets partial domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8156 8164, 2018.
[51] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference computer vision, pages 2223 2223-2232, 2017.
13

---

Checklist
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Contributions in Section [
(b) Did you describe the limitations of your work? [Yes] See Methodological Limitations in Section
(C) Did you discuss any potential negative societal impacts of your work? [N/A] See Potential Negative Societal Impact in Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Implemen- tation details in Section
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Implementation details in Section
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Figure 2 and Figure
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Implementation details in Section 4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [No] All datasets are collected and made public by the creators. We cannot find the license information for these datasets. All datasets are cited Datasets, Section 4
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "niK8Q8j8i5r", "reviewer": "Reviewer_9Z8X", "summary": "This paper studies the partial domain adaptation problem. The authors propose an implicit semantic response alignment method where the source domain has more categories than the target domain. The motivation is to further utilize the extra source categories to boost the adaptation performance. To achieve this, anew module class2vec is proposed to extract the implicit semantics and explore the relationship among different categories, and align the semantic response of source and target data. Experiments on three benchmark datasets demonstrate the effectiveness of the proposed methods on different backbone networks. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points\n1.\tThe existing PDA methods design different strategies to lower the influence of source data that belongs to irrelevant classes. Despite the effectiveness of this practice, they inevitably overlook the potential” shared” knowledge among different categories, and treat the irrelevant class data as noise. Instead, the proposed implicit semantics idea investigated the “shared” knowledge across categories, which can be regarded as a concept of subclass, where different categories might share the same subclass.\nI like this idea, since it is intuitive for partial domain adaptation. I think this idea might be also applied to other problems, like class-incremental continual learning, etc.\n2. The presentation is clear and easy to follow. \n3. The technical part is solid, although the semantic extraction is a simple auto-encoder. Considering this is the first paper to explore this direction, I am ok with it.\n4. The proposed technique is an add-on module, which can be applied to the existing partial domain adaptation methods. The authors demonstrate the improvements with three partial domain adaptation methods, where the improvements on A^2TK and BA^3US are significant.\n5. Experiments well validate the idea, including the standard algorithmic performance comparison and several in-depth explorations. The topic attention weighting and cross-class interaction are helpful to readers.\n\nWeak points\n1. Section 3.1 talks about motivation, which is duplicated with some content in the introductory part. It can be more concise.\n2. Figure 4 is a good indicator to show why the proposed method works. It would be better if the authors can employ the category similarity to predict which category can be boosted. Some categories are dramatically enhanced while the candles category suffers from the negative transfer. \n\nTypo: figure 1 caption, “purposed” -> “proposed”\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No major concerns on limitations or potential negative social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "kNWqaOp1a_V", "reviewer": "Reviewer_SLrJ", "summary": "This paper introduces a new partial domain adaptation (PDA) method. Different from previous methods that ignore the irrelevant source classes, this work argues that source data coming from extra classes also contributes to the adaptation. Motivated by this, the proposed method first identifies implicit semantic, then performs semantic alignment. The strength of this work is proved on common benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Although the method is novel, its motivation still needs more clarification. The difference with previous contributions is clearly claimed. \n\nQuality: This is a complete work but the proposed method is not technically sound (see details in Limitation).\n\nClarity: The writing of this paper needs further improvement. Furthermore, some points in the method section are not clearly explained (see Limitation).\n\nI raise my rating to 6 based on the author's response and other reviewer's comments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "(1)\tThe motivation is not solid. Although the source examples from extra classes may share similar semantic information with target examples, these source examples also introduce noises and uncertainty.\n\n(2)\tIs that necessary to introduce another network (i.e., topic attention receptor) to get the attention by using its gradients? What is the motivation of this design? Why do we need this attention?\n\n(3)\tThe section “Semantic topic alignment” is confusing. For example, which normalization method is used here? What is the formal definition of A?\n\nMinor comments: In Line 155, it is more common to use Ct \\subset Cs\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "WQA61P4gimE", "reviewer": "Reviewer_BdsE", "summary": "This paper considers the problem of partial domain adaptation. To address this problem, the authors propose the Implicit Semantic Response Alignment (ISRA) method, which attempts to align the source and target domains on the semantic topic level instead of class level. Experiments on several datasets show the benefit of the proposed ISRA.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ This paper is well-written and easy to follow. The motivation and the overall organization are clear.\n\n+ The proposed method is novel. As far as I know, this is the first work that tries to align the source and target domains in the semantic topic level. Also, the proposed method achieves good results on several datasets, and, it can be injected to different methods, such as SHOT, BA3US.\n\n+ The authors provide good experiments to verify the effectiveness of the proposed method and also provide some analyses to reflect why the proposed method can work well. \n\nCons:\n\n- Although the proposed method considers the partial domain adaptation, other settings, such as closed-set and open-set, also share the same semantic topics between the source and target domains. Therefore, I think the proposed method can also be applied to other settings. I would like to see whether the proposed method can improve the results on other settings. If not, please explain the probable reason.\n\n- In hyperparameter analysis, the authors use office-31 to validate the parameters. However, since (1) the improvement on the office-home is not consistent and (2) the number of classes is limited, the results on office-31 can not well-reflect the parameter sensitivity of the proposed. Please provide the experiments on the office-home and the avg results on all directions should be reported.\n\n- In addition, the proposed method seems sensitive to the number of implicit topics. We can find that, even using a small value, the results are not consistent for A->W setting.\n\n- SHOT is a source-free method, how can the proposed method learn the semantic embedding for both the source and target domains?\n\n- Why not evaluate the proposed method on the VisDa and DomainNet? DomainNet has a lot of classes and the classes will share more semantic topics, which is more suitable for the proposed method. Because the authors argue that the poor results on the office-31 are caused by the limited number of classes. Thus, when there are more classes (such as DomainNet), the proposed method will introduce more improvements.\n\n- There are a lot of duplicate words between the Introduction and the motivation of the Method. Please modify one of them.\n\n- In Fig~1, it would be better to show the details of the implicit semantic response alignment, including the structure of Auto-Encoder, the alignment process. Also, for better understand the proposed method, the author can provide an algorithm part.\n\nOverall:\n\nIn summary, this paper proposes a novel method for partial domain adaptation and achieves promising results. However, some important experiments and explanations are missed in the current version. Therefore, I will give borderline reject this time and would like to see the response of the authors.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "3f4Vl-R7ve", "reviewer": "Reviewer_BpRc", "summary": "In this paper, the authors consider the partial unsupervised domain adaptation (PDA) problem. Different from the existing methods that assign lower weights to the source-specific classes and adapt one-to-one shared class mapping, the authors explore the relationship among different classes and expect to extract useful/similar information from other classes in a finer granularity for further boosting the positive transfer. The authors design a class2vec module to extract the implicit semantic topics from the visual features. Semantic responses of source and target data are aligned to retain the relevant information contained in multiple categories by weighting the features, instead of samples or classes. Extensive experiments on three benchmark datasets demonstrate the effectiveness of the proposed implicit semantic discovery and alignment.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The idea of implicit semantic discovery and alignment seems interesting to me. The illustrative example in the Introduction helps to understand the common character among different classes, which can be utilized for further boosting knowledge transfer. \n2. Inspired by the word2vec, the authors propose a similar method, class2vec, which presents each class into several implicit semantic topics. This module is an easy plug-in component, which can work with different PDA backbones to further improve their performance.\n3. The technical details are clear and easy to follow. The authors propose the source code for easy use, which will benefit the community.\n4. Experiments seem sufficient. The authors demonstrate the proposed technique on BA^3US has significant improvements compared with several SOTA baselines. The authors also demonstrate the effectiveness of implicit semantic discovery and alignment on different backbones. \n5. The paper explains the results well. For example, Figure 3(c) provides how the proposed method works at the semantic level. It shows that the weighting schema discoveries the information that responds to the same implicit topic in source and target samples on the feature level, where only partial instances and partial visual features are activated to a certain implicit semantic topic. Figures 4&5 and Table 3 show when the implicit semantic discovery and alignment works or does not work. It helps when selecting a method for real-world applications.\n\nWeaknesses:\n1. For Figure 3(c), it is necessary to add the class category of each sample. By this means, we can tell whether some instances/categories share common semantics or not.\n2. There are some presentation issues to improve the paper. For example, lambda_reg seems not discussed. The second best is not highlighted in Table 2 (W->D). Imagenet in line 229 should be ImageNet.\n\nConcerns:\nIn Eq. (1), the authors extract the semantics at the instance level and aggerate them together to present the class level semantics. I am not sure whether adding another constraint that the instances from the same class share the same semantic would contribute to the positive transfer. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact are included in the main paper. \nPlease refer to the main review for suggestions in detail.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on the partial domain adaptation problem. The proposal is an implicit semantic response alignment method where the source domain has more categories than the target domain. The philosophy behind sounds quite interesting to me, namely, utilize the extra source categories to boost the adaptation performance. This philosophy leads to a novel algorithm design I have never seen.

The clarity and novelty are clearly above the bar of NeurIPS. While the reviewers had some concerns on the significance, especially from Reviewer SLrJ and BdsE, the authors did a particularly good job in their rebuttal. Thus, most of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)