A Simple Unified Information Regularization Framework for Multi-Source Domain Adaptation
Authors: Geon Yeong Park
Sang wan Lee
Venue:  (2021)
Keywords: Multi-source Domain Adaptation
Transfer learning
Adversarial learning
Information theory

Abstract: Adversarial learning strategy has demonstrated remarkable performance in dealing with single-source unsupervised Domain Adaptation (DA) problems, and it has recently been applied to multi-source DA problems. While most of the existing DA methods use multiple domain discriminators, the effect of using multiple discriminators on the quality of latent space representations has been poorly understood. Here we provide theoretical insights into potential pitfalls of using multiple domain discriminators: First, domain-discriminative information is inevitably distributed across multiple discriminators. Second, it is not scalable in terms of computational resources. Third, the variance of stochastic gradients from multiple discriminators may increase, which significantly undermines training stability. To fully address these issues, we situate adversarial DA in the context of information regularization. First, we present a unified information regularization framework for multi-source DA. It provides a theoretical justification for using a single and unified domain discriminator to encourage the synergistic integration of the information gleaned from each domain. Second, this motivates us to implement a novel neural architecture called a Multi-source Information-regularized Adaptation Networks (MIAN). The proposed model significantly reduces the variance of stochastic gradients and increases computational-efficiency. Large-scale simulations on various multi-source DA scenarios demonstrate that MIAN, despite its structural simplicity, reliably outperforms other state-of-the-art methods by a large margin especially for difficult target domains.

Under review conference paper ICLR 2021
A SIMPLE UNIFIED INFORMATION REGULARIZATION FRAMEWORK FOR MULTI-SOURCE DOMAIN ADAPTA- TION
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial learning strategy has demonstrated remarkable performance in dealing with single-source unsupervised Domain Adaptation (DA) problems, and it has recently been applied to multi-source DA problems. Although most existing DA methods use multiple domain discriminators, the effect using multiple discrimi- nators on the quality of latent space representations has been poorly understood. Here provide theoretical insights into potential pitfalls of using multiple domain discriminators: First, comaan-discrimmnattee across multiple discriminators. Second, it not scalable in terms of computational resources. Third, the variance of stochastic gradients from multiple discriminators may increase, which significantly undermines training stability. To fully address these issues, we situate adversarial DA in the context of information regularization. First, we present unified information regularization framework for multi-source DA. It provides a theoretical justification using a single and unified domain dis- criminator to encourage the synergistic integration of the information gleaned from each domain. Second, this motivates implement a novel neural architecture called a Multi-source proposed model significantly reduces the variance of stochastic gradients and in- creases cimputational-efficinnn Large-scale simulations on various multi-source DA scenarios demonstrate that MIAN, despite its structural simplicity, reliably outperforms other state-of-the-arr methods by a large margin especially for difficult target domains.
information is inevitably distributed
Adaptation Networks (MIAN). The
INTRODUCTION
Although large number of studies have demonstrated the ability of deep neural networks solve challenging tasks, the tasks solved by networks are mostly confined a similar type or a single domain. One remaining challenge is the problem known as domain shift (Gretton et al. (2009)). ), where a direct transfer of information gleaned from single source domain to unseen target domains may lead to sigmificant performance impairment. Domain adaptation (DA) approaches aim to mitigate this problem by learning to map data of both domains onto common feature space. Whereas several theoretical results (Ben-David et al. (2007): Bliizer al. (2008): Zhao al. (2019a)) and algorithms for DA (Long et al. (2015) 2017); Ganin et al. (2016)) have focused on the case which only a single-source domain dataset is given, we consider a more challenging and generalized problem of knowledge transfer, referred to as Multi-source unsupervised DA (MDA). Following seminal theoretical result on MDA (Blitzer al. (2008): Ben-David et al. (2010)) technical advances have been made, mainly on the adversarial methods. et (2018) Zhao et al. (2019c)).
While most of adversarial MDA methods use multiple independent domain discriminators (Xu et al. 2018); Zhao al. (2188) al. (018); Zhao al. 20T9c/b), the potential pitfalls of this setting have not been fully explored. The existing works do not provide a theoretical guarantee that the unnecessary domain-sponn -ppccffc information is fully filtered out, because the mamaiindiscriminatie information is inevitably distributed across multiple diseriminators. For example, the multiple domain discriminators focus only on estimating the domain shift between source domains and the target, while the discrepancies between the source domains are neglected, making it hard to align all the given domains. This necessitates gameriig the domaan-diseriminative information with

---

Under review as conference paper ICLR 2021
unified discriminator. Moreover, the multiple domain discriminator setting is not scalable in terms of computational resources especially when large number of source domains are given, e.g., medical reports from multiple patients. Finally, it may undermine the stability of training, earlier works solve multiple independent adversarial minimax problems.
To overcome such limitations, we propose novel MDA method, called Multi-source Information- regularized Adaptation Networks (MIAN), that constrains the mutual information between latent representations and domain labels. First, we show that such mutual information regularization is closely related to the explicit optimization of the H-divergence between the source and target domains. This affords the theoretical insight that the conventional adversarial DA can be translated into an information -eeemmatio--hooettii---o regularization problem. Second, based on findings, we propose new optimization problem for MDA: minimizing adversarial loss over multiple domains with a single domain discriminator. We show that the domain shift between each source domain can be indirectly penalized, which is known to be beneficial in MDA (Li al. (2018); Peng et al. (2019)) with single domain discriminator, Moreover, by analyzing existing studies terms of information regularization, we found that the variance of the stochastic gradients increases when using multiple discriminators.
Despite its structural simplicity, we found that MIAN works efficiently across a wide variety of MDA scenarios, including the DIGITS-Five (Peng al. (2099), Office-31 (Saenko al. (2010), and Office-Home datasets /Venkateswara et al. (2017)) Intriguingly. MIAN reliably and significantly outperformed several ofae--o--thearr methods that either employ domain discriminator separately for each source domain (Xu al. (2018) or align the moments of deep feature distribution for every pairwise domain (Peng et al. (2019)).
2 RELATED WORKS
Several DA methods have been used in attempt to learn domain-invariinn representations. Along with the increasing use of deep neural networks, contemporary work focuses on matching deep latent representations from the source domain with those from the target domain. Several measures have been introduced to handle domain shift, such as maximum mean discrepancy (MMD) (Long (et al. 2014;22015 correlation distance (Sun et al. (2016); Sun & Saenko (20 ), and Wasserstein distance (Courty et al. (2017h) Recently, adversarial DA methods (Ganin al. (2016); (Tzeng (et al. (2017); Hoffiman et al. (2017): [Saito et al.) (20188 2017h) have become mainstream approaches owing to the development of generative adversarial networks (Goodfellow et al. (2014)). However, the abovementioned single-source DA approaches inevitably sacrifice performance for the sake of multi-source DA.
Some MDA studies (Blitzer et al. 20088: Ben--avid et al. (2010); Mansour al. (2009) y moffmman et al. (20T8) have provided the theoretical background for algorithm-level solutions. (Blizer et al. (2008) Ben-David et al. (2010)) explore the extended upper bound of true risk unlabeled samples from the target domain with respect to weighted combination of multiple source domains. Following these theoretical studies, MDA studies with shallow models (Duan et al. (2012b a); Chattopadhyay al. )2012) as well as with deep neural networks (Mancini et al. (2018); Peng et al. (2019): Li al. (2018) have heen proposed. Recently, some adversarial MDA methods have also been proposed. ct (2018) implemented a k-way domain discriminator and classifier to battle both domain and category shifts. Zhao et al. (2018) also used multiple discriminators to optimize the average case generalization bounds. Zhao et al. (2019c) chose relevant source training samples for the DA by minimizing the empirical Wasserstein distance between the source and target domains. Instead of using separate encoders, domain discriminators classifiers for each source domain in earlier works, our approach uses unified networks, thereby improving essouce--efficienn and scalability.
Several existing MDA works have proposed methods to estimate the source domain weights following (Blitzer et al. (2008); BBen-Dvvid et al.) (2010)). MMnsour et al. (2009) assumed that the target hypothesis be approximated by convex combination of the source hypotheses. (Peng et al. (2099) Zhao et al. (2018)) suggested ad-hoc schemes for domain weights based on the empirical risk of each source domain. Li et al. (2018) computed softmax-tranne transformed weight vector using the empirical Wasserstein-like measure instead of the empirical risks. Compared to the proposed methods without robust theoretical justifications, our analysis does not require any assumption or estimation for the domain coefficients. In our framework. the representations are distilled to be independent of the domain, thereby rendering the performance relatively insensitive to explicit weighting strategies.

---

Under review conference paper ICLR 2021
3 THEORETICAL INSIGHTS
We first introduce the notations for the MDA problem in classification. set of source domains m
and and the target domain are denoted by ( and Dy, respectively, Let Xs, =xx,,) (x j=1
Ys. = { be a set m i.dd. samples from Ds. Let Xy = {xx²' } ~ (D7) m be the set of m i.i.d. samples generated from the marginal distribution Dix- The domain label and its probability distribution are denoted by V and (v))), where v V and V the set of domain labels. In line with prior works (Hoffman et al. (2012): ; Gong et al. (213); : Mancini et al. (2018); (Gog al.] (2019)), domain label can be generally treated as a stochastic latent random variable in our framework. However, for simplicity, we take the empirical version of the true distributions with given samples assuming that the domain labels for all samples are known. The latent representation of the sample is given by Z, and the encoder is defined as F X - Z, with X' and Z representing data space and latent space, respectively. Accordingly. Zg, and Zy refer the outputs of the encoder F(Xs,) and F (. respectively. For notational simplicity, we will omit the index from Ds., Xs. and Zs, when N = 1. A classifier is defined as C: Z - where y is the class label space.
3.1 PROBLEM FORMULATION
For comparison with our formulation, we recast single-source DA as a constrained optimization problem. The truc risk (x))) unlabeled samples from the target domain is bounded above the sum of three terms (Ben-David et al. 2010):: : (I) true risk (s())) of hypothesis & on the source domain: (2)
H-divergence dz(15 DT) between a source and target domain distribution; and (3) the optimal joint risk X*
Theorem (Ben-David et al. (2010). Let hypothesis class H be a set of binary classifiers h {0, 1}. Then for the given domain distributions Ds and Dr,
vh H. (h)) cs(h) dn dD Dr) +
(1)
where du( Ds, DT) = 2sup I. [[[M(X] = 1)]
x-D [1(A(x)= = 1)] and ((a) is an indicator
NEW x=Dz
function whose value is if true, and otherwise.
The empirical H H-divergence dr (Xs, can be computed as follows (Ben-Davic et al. 2010
Lemma 1.
du(Xs. = -2(1 min REM
Σ X
[[(()) = 1| 1 Σ XEXT
[[((x) - =]])
(2)
Following Lemma domain classifier : Z + V can be used to compute the empirical H- divergence. Suppose the optimal joint risk X is sufficiently small as assumed in most adversarial DA studies (Saito al. (2017); Chen et al. (2019)). Thus, one can obtain the ideal encoder and classifier minimizing the upper bound of (h) by solving the following min-max problem:
F²CC== arg min L(F,C)+ /d Zr)
F.C
(3)
arg arg min max L(F,C) + 8-
Σ iz
1[b(z)) = 1]+
Σ -
F.C. KER
where L(F, C) is the loss function samples from the source domain, 3 is Lagrangian multiplier, {0. 1} such that each source instance and target instance are labeled as and O, respectively, and h is the binary domain classifier. Note that the latter min-max problem is obtained by converting -min into max and removing the constant term from Lemma 1.
3.2 INFORMATION REEIONNNEEGLARIEEED MIN MAX PROBLEM FOR MDA
Intuitively, it is not highly desirable to adapt the learned representation in the given domain to the other domains, particularly when the representation itself not sufficiently domain-indeepnnden.

---

Under review conference paper ICLR 2021
This motivates to explore ways to learn representations independent of domains. Inspired by a contemporary fair model training study (Roh et al. (2020)), the mutual information between the latent representation and the domain label / (2;V) can be expressed follows:
Theorem 2.
Let Pz (z(z) be the distribution of Z where E Z. Let h be domain classifier h : 2 - V. where the feature space and V is set of domain labels. Let ly (z) be a conditional probability
of V where v E V given Z = z, defined by h. Then the following holds:
max I(Z;V)= (n ver (2)==1
Σ VEV
Pv(())Ba-Paa [og by (z)] -(V)
(4)
The detailed proof is provided in the Roh et al (2020) and Supplementary Material. As done in Roh ct al. 2020), we can derive the empirical version Theorem
2 als follows:
((2VV)=
max hy M Σ
Σ log lv, (2,) + H(V).
where M is the number of total representation samples, is the sample index, and v; is the correspond- ing domain label of the ith sample. Using this equation, we combine our
objective function and the results of Lemma[] [ For binary classification V {0, 1) with Zs and Z of equal size M/2, we propose the following information-regulariiee minimax problem:
F C
arg min P.O
L(F,C) ()(
(6)
arg min max ((F,C)+3- M
Σ iz,
log h(z.) +
Σ juzeZr
log(1 - h(z,))),
P.C. KEH
where B is Lagrangian multiplier, hz;)) (1,) and h(2;) lx,)) with l(2))
representing the probability that 2, belongs to source domain. This setting automatically dismisses the condition Evey (z) = 1, Vz. Note that we have accommodated simple situation in which the entropy H() remains constant.
3.3 ADVANTAGES OVER OTHER MDA METHODS
The relationship between and (6) provides theoretical insights that the problem of minimizing mutual information between the latent representation and the domain label is closely related minimizing the Hi-divergence using the adversarial learning scheme. This relationship clearly underlines the significance of information regularization for MDA. Compared to the existing MDA approaches (Xu al. (2018); Zhao al. (2018)) which inevitably distribute comain-discrimmnativ knowledge over N different domain classifiess, the above objective function (6) enables us to scamlessly integrate such information with the single-domain classifier h.
Using single domain discriminator also helps reduce the variance of gradient. Large variances in the stochastic gradients slow down the convergence, which leads to poor performance (Johnson & Zhang (2013)) Herein, we analyze the variances of the stochastic gradients of existing optimization constraints. By excluding the weighted source combination strategy, we can simplify the optimization constraint existing adversarial MDA methods as sum of the information constraints:
I(Zx:Un)
Σ 1-1 (E( (x(x) (x)xx1111
[log Σ Pu (u)Eur [log Logg(((z)) (((z)) + Σ H(U)),
DEZ
(7)
where Us is the Ath domain label with 24 = (0, 1}. Pz. ju)()= = P2|v=N+1(-) corresponding to the target domain, Pz. Pz)ju=z(-) = PZ(v=z()) (-) corresponding to the kth source domain, and h (zz) being the conditional probability of E U given z, defined by the kth discriminator indicating that the sample is generated from the kth source domain. Again, we treat the entropy (Ux) as a constant. Note that the interaction information cannot be measured with (7).
Given M m(N + 1) samples with representing the number of samples per domain, an empirical
version of is:
N
i(Ze: Ux) =
1
max Eee AE
Σ
Σ
log K (2 + Σ H(U,).
(8)
M
M E
1x)=

---

Under review conference paper ICLR 2021
Let be a shorthand the kth term inside the first summation. Without loss of generality we make simplifying assumptions that all Var[4k] are the same for all k and so are CooeLe. 14. for all pairs. Then the variance of (8) is given by:
N V ar
N Var
N N +2
Ub)
+2Σ
Σ
CoulIa. Li)
=
M
k=15-k
(9)
N
N(N Coolle: 41)
m (N+ VarlAd]+
(N + 1)2
As earlier works solve N adversarial minimax problems, the covariance term is additionally included and its contribution to the variance does not decrease with increasing N. In other words, the covariance term may dominate the variance of the gradients as the number of domain increases. In contrast, the variance of our constraint (5 is inversely proportional to (N + 1)". Let Im be a shorthand for the maximization term except 37 in (5). Then the variance of (5) given by:
1 Var [(2:v)] m2(N + 1)2 (Varlful).
(10)
It imiples that our framework can significantly improve the stability of stochastic gradient optimization compared to existing approaches, especially when the model is deemed to learn from many domains.
3.4 SITUATING DOMAIN ADAPTATION IN CONTEXT OF INFORMATION BOTTLENECK THEORY
In this Section, we bridge the gap between the existing adversarial DA method and the information bottleneck (IB) theory (Tisbyy al. (2000); Tishby & Zaslavsky 20155; Alemi al. (2016)). Tishby (el al. (2000) examined the problem of learning an encoding such that is maximally informative about the class Y while being minimally informative about the sample X:
min B1 -1ZZ;))(((Z7)) Panc(8)x)
(11)
where 3 is a Lagrangian multiplier. Indeed, the role of the bottleneck term X(2:X) matches our mutual information (Z; V) between the latent representation and the domain label. We foster close collaboration between two information bottleneck terms by incorporating those into ((;; X, V).
Theorem 3. Let Petx (z) be a conditional probabilistic distribution of Z where Z, defined by the encoder F. given a sample x E X and the domain label v E V. Let Rz (z) denotes a prior marginal distribution of Z. Then the following inequality holds:
I(2;X.V)
mag Av( ko(x)=1,1z VEV
Pvv(y)Ep (ogg)e/) [log
H(V)
(12)
+ Ex. [DrI || Rz]]
The proof of Theorem uses the chain rule: proof is provided in the Supplementary Material. Whereas the role of /(Z; X V) is to purify the latent representation generated from the given domain, V(Z serves proxy for regularization that aligns the purified representations across different domains. Thus, the existing DA approaches (Luo et al. (2019); [Song et al. (2019)) using variational information bottleneck (Alemi et al. (2016)) can be reviewed as special cases for Theorem with single-source domain.
+ I(Z; TheThe detailed
4 MULTI-SOURCE
ADAPTATION NETWORKS
In this Section, we provide the details of our proposed architecture, referred multi-source information information-rogullrizeed adaptation network (MIAN). MIAN addresses the information-coostraiieed min-max problem for MDA (Section 3.2) using the three subcomponents depicted in Figure information regularization, source classihcation, and Decaying Batch Spectral Penalization (DBSP).
Information regularization. To estimate the empirical mutual information i(2; V) in 5) the
domain classifier h should be trained minimize softmax cross enropy. Let V {1,2,... N+

---

Under review conference paper ICLR 2021
Encoder
Domain Discriminator At
h(zs)
4 7 K
Mullodummii
Advorsorial brining
FON
Independast represertation
1 35
V(Z.V)
Pignald fanctors
Classifier
Decxying DSP
L(P.C)
Figure 1: Proposed neural architecture for multi-source domain adaptation: Multi-source Information- regularized Adaptation Network (MIAN). Multi- source and target domain imput data fed into the encoder. We denote arbitrary source domains as S; and Sy without loss of generality. The domain discriminator outputs a logit vector, where each dimension corresponds to each domain. CNN and FCN refers to convolutional neural networks and fully connected neural networks, respectively.
and denote hzz) as N + dimensional vector of the conditional probability for each domain given the sample Let be N dimensional vector of all ones, and [-=v] be vector whose vth value is 1 and O otherwise. Given M m(N - 1) samples, the objective is:
dimensional
min 1 M
XΣ>
Σ VEV =V
1 [kvv.] log h(2,)].
(13)
In this study, we slightly modify the objective (33 Specifically, we explicitly minimized the conditional probability of the remaining domains excepting the vth domain. Let 1 kyvl be the flipped version [|=v] Then the objective function for the domain discriminator is:
min h
M Σ Σ [v.) log h(z.)
1
(Ryp. log(1 h(z:))]
(14)
VEV
where the objective function for encoder training is to maximize 14. Our objective function is also closely related to that of GAN (Goodfellow al. (2014)) and we experimentally found that using the variant objective function of GAN (Mao al. (017)) works slightly better.
The above objective is closely related to optimizing every pairwise domain discrepancy between the given domain and the mixture of the others. Let each Dv and Dve represent the vth domain and the mixture the remaining N domains with the same mixture weight 1 Ñ relatively. Then
we can define H-divergence as dz (Dv. Dv and an average of such H-divergence for every v as dx (V). Assume that the samples of size m, Zv and Zve, generated from each D, and Dve, where Zuo = Uv'rv Zv with ZZvl m/N for all v' V. Thus the domain label v, for every jth sample in Zve. Then the average of empirical H-divergencee dz<(V) is defined as follows:
dz((V)
dxl I
N W+1
VEV
(15)
Σ 2 VEV
min
Σ [Z,))())) 1]+ CV.-v
Σ [[b((z,) 0))
N
bEH
#v
where (z) represents the vth value of h(z). Note that h(z) corresponds to N + dimensional one-hot classification vector (1). unlike in (14.. Then, let [[{[2)] := [x) = [)]vey be the
N + dimensional one-hot indicator vector. Given the unified domain discriminator in the inner minimization for every v in (15). we train to approximate the lower bound of (vx) as follows:
ht = arg max
Σ
Σ
[hy(Zz)) = =|+ Σ (())(z,) = o])
RCH
M
veV
tv.=v
j.vsyv
(16)
= arg min
Σ Σ
1 I((2))) + +1 1a (1 -[(((z)))
NER
VCV RvY=v

---

Under review conference paper ICLR 2021
where the latter equality obtained by rearranging the summation terms in the first equality.
Based on the close relationship between and (16), we can make the link between information regularization and H ---ivergencc optimization given multi-source domain; minimizing dy (V) closely related implicit regularization of the mutual information between latent representations and domain labels. Because the output vector h(z) in (5)) often comes from the arg operation, (15) not differentiable w.r.t z. However, our framework has differentiable objective as in (14).
There are two benefits minimizing (V)... First, includes H-ddivergence hetween target a mixture of sources, which directly affects the upper bound of the empirical risk on target samples (Theorem in BBen-David et al. (2010)) Second, (x()) owwer-bounds the average of every pairwise H-divergence between domains. The detailed proof is provided the appendix (Lemma 2 Note that unlike our single domain classifier setting, existing methods (Li ct al. (2018) require a number of about (N22) domain classifiers approximate all pairwise combinations of domain discrepancy.
Source classification. Along with learning domain domain-indepeneeee independent latent representations illustrated the above, we train the classifier with the labeled source domain datasets that can be directly applied to the target domain representations in practice. To minimize the empirical risk on source domain, we use generie softmax cross-entropy loss function with labeled source domain samples as [(F, C).
Decaying batch spectral penalization. Applying above information teeeortto---hooretiic insights, we further describe a potential side effect of existing adversarial DA methods. Information regularization may lead to overriding implicit entropy minimization, particularly in the early stages of the training, impairing the richness of latent feature representations. To prevent such a pathological phenomenon, we introduce a new technique called Decaying Batch Spectral Penalization (DBSP), which is intended control the SVD entropy of the feature space. Our version improves training efficiency compared original Batch Spectral Penalization (Chen al. (2019)). We refer to this version of our model as MIAN-y. As vanila MIAN is sufficient to outperform other state-of-the-an methods (Section 5). MIAN-y is further discussed in the Supplementary Material.
5
EXPERIMENTS
Table I: Accuracy (%) Digits-Five dataset. SYNTH denotes Synthetic Digits (Ganin & Lempitsky
2014)). The baseline results for the Digits-Five dataset were taken from (Peng al. (2019)).
Standards
Models
MNIST-M
MNIST
USPS
SVHN
SYNTH
Avg
Source Only
63.70
92.30
90.71
71.51
83.44
80.33
Source-
DAN
67.87
97.50
93.49
67.80
86.93
82.72
combined
DANN
70.81
97.90
93.47
68.50
87.37
83.61
Source Only
63.37
90.50
88.71
63.54
82.44
77.71
DAN
63.78
96.31
94.24
62.45
85.43
80.44
CORAL
62.53
97.21
93.45
64.40
82.77
80.07
DANN
71.30
97.60
92.33
63.48
85.34
82.01
Single-best
JAN
65.88
97.21
95.42
75.27
86.55
84.07
ADDA
71.57
97.89
92.83
75.48
86.45
84.84
MEDA
71.31
96.47
97.01
78.45
84.62
85.60
MCD
72.50
96.21
95.33
78.89
87.47
86.10
DCTN
70.53
96.23
92.81
77.61
86.77
84.79
Multi-
SDA
69.76
98.58
95.23
78.56
87.56
86.13
source
3
72.82
98.43
96.14
81.32
89.58
87.65
MIAN
84.36
97.91
96.49
88.18
93.23
92.03
To assess the performance of MIAN, we ran a large-scale simulation using the following benchmark datasets: Digits-Five, Office-31 and Office-Home. For fair comparison, we reproduced all the other baseline results using the same backbone architecture and optimizer settings as the proposed method. For the source-only and single-source DA standards, we introduce two MDA approaches (Xu et al. (2018); Peng al. (001 (1) source-combined, i.e., all source-domains incorporated into

---

Under review conference paper ICLR 2021
single source domain; (2) single-best, i.e. the best adaptation performance on the target domain reported. Owing to limited space, details about simulation settings, used baseline models and datasets are presented in the Supplementary Material.
Table Accuracy (%) on Office-31 dataset.
Standards
Models
Amazon
DSLR
Webcam
Avg
Source Only
55.23::0.72
95.59+ 199--1.37 1.37
87.06+t 1.50
79.29
Single-best
DAN
64.19=0.556
100.00=10.00
97.45=00.44
87.21
JAN
69.57:10.27
99.80-10.00
97.4:10.26
88.92
Source Only
60.80=12.00
92.68
86,91
80.13
JAN
70.15=#0.19
95.20:10.36
95.15=0..23
86.83
Source-
DANN
68.15:10.42
97.59+00.0
96.77=0.26
87.50
combined
DAN
65.77=0.74
99.26++0.3
97.51=0.41
87.51
DANN+BSP
71.13+10.4
96.65+L0.30
98.22--0.26
88.70
MCD
68.57++1.06
99.49+0.25
10.30+0.38
89.12
DCTN
02.54:10.50
99.44-10.25
97.92=0.29
86.70
Multi- source
SDA
67.19:10.22
99.3410.11
98.04:10.21
88.19
SDA-
09.41
99.6440.11
99.30=
89.45
MIAN
7..65
99.48-10.35
98.49
90.87
Table 3: Accuracy (%) Office-Homee dataset.
Standards
Models
Art
Clipart
Product
Realworld
Avg
Source Only
64.58100.68
52.32-+0.63
77.63+:0.2
80.70:10.81
68.81
DANN
64.26:10.59
58.01
76.44:10.47
78,80 18.80:10.49
69.38
Source- combined
DANN+BSP
66.10+0..2
61.03+00.39
78.13+0..1
79.92:10.13
71.29
DAN
68.28:00455
57.92=0.65
78.45=00.55
81.93:10.35
71.64
MCD
67.844+0.38
59.91+0.55
79.21+40.61
80.93+10.18
71.97
M
06.22:+0.52
58.55+00.2
79.45-10.52
81.35++0.19
71.39
Multi-
DCTN
66.92+ł0.60
61.82:00.46
79.20+0.588
77.78+0.59
71.43
source
MIAN
69.39-10.50
63.05
79.62.10.16
80.44.10.24
73.12
5.1 SIMULATION RESULTS
The classification accuracy for Digits-Five. Five, Office-31, and Office-Home summarized in Tables 2 and 3 respectively. We found that MIAN outperforms most other state of tate-offthe-aae ththharrt single-source and multi-source DA methods by a large margin. Note that our method demonstrated significant improvement in challenging domains, such MNIST-M, Amazon Clipart.
QUALITATIVE: AND QUANTITATIVE ANALYSES
Design of domain discriminator. To quantify the extent to which performance improvement is achieved by unifying the domain discriminators,
compared the performances of the four different versions MIAN (Figure 2a 2b No S-S align is same MIAN with the exception that only the target and each source domains are aligned. No LS uses the objective function as in (14), and unlike (Mao et al. (2017)) Multi D employs many discriminators the number of source domains which is analogous to existing approaches. For a fair comparison, all other experimental settings are fixed. The results illustrate that all the versions with the unified discriminator reliably outperform Multi D in terms of both accuracy and reliability. This suggests that unification of the domain discriminators can substantially improves the task performance.
Variance of stochastic gradients. With respect to the above analysis, we compared the variance the stochastic gradients computed with different available domain discriminators. We trained MIAN and Multi D using mini-hatches of samples. The number of samples in batch fixed 128 per

---

Under review conference paper ICLR 2021
PAUTH
20W
000
(a)
(b)
(d)
Figure (b)- Test accuracies for (a) MNIST-M and (b) SVHN as target domains. (c)~(d):
Variance of stochastic gradients after 1000 steps for (c) MNIST-M and (d) SVHN as target domains in log scale. Less is better.
CBR
CLR
75
3.3
03
0.3
Target donall
Arrazon
Terge: doman
(a) Discrepancy
(b) Accuracy
(c) (V)
(d) 1(2;; V)
Figure 3: (a) Proxy A-distance. (b)~(c) Ablation study on the objective of domain discriminator.
CEN stands for multi-class cross entropy loss in (13). while BCE stands for binary-class cross entropy
losses in 14). (d) Empirical information I(2 V). We treat H(V) log D/
domain. After the early stages of training, we computed the gradients for the weights and biases of both the top and bottom layers of the encoder the full training set. Figures 2c 2d show that MIAN with the unified discriminator yields exponentially lower variance D. Thus it is more feasible to use the unified discriminator when a large number of domains are given.
gradients compared to Multi
Proxy A-distance. To analyze the performance improvement in depth. measured Proxy A- Distance (PAD) an empirical approximation domain discrepancy (Ganin et al. (0166). Given the generalization erro on discriminating hetween the target and source samples, PAD is defined as d = 2(1 - 2e). Figure 3a shows that MIAN yields lower PAD between the source and target domain on average, potentially associated with optimizing (V).. To test this conjecture, we conducted an ablation study on the objective of domain discriminator (Figure (3b 3cp. All the other experimental settings were fixed except for using the objective of the unified domain discriminator (13). (14 While both cases help the adaptation, using
[44 yields lower (V)) and higher test accuracy.
Estimation of mutual information. We measure the empirical mutual information i(z V) with assuming H(V) constant. For the measurement, trained the domain discriminator minimize the softmax eross entropy (33) with sufficient iterations. Figure shows that MIAN yields lowest f(Z; V). guarantecing that the obtained representation achieves low-level of domain dependence.
6 CONCLUSION
this paper, we have presented a unified iioommttion-regularizitte framework for MDA. The proposed framework allows to examine the existing adversarial DA methods and also motivates implement novel neural architecture for MDA. We provided both theoretical arguments and empirical evidence to fully justify three potential pitfalls of using multiple discriminators: dispersed domain- -isrrmiiniisriiiaaiveee knowledge, lack of scalability and high variance in the objective. Our framework also establishes bridge between adversarial DA and Information Bottleneck theory. The proposed model does not require complicated settings such as image generation, pretraining. multiple discriminators, multiple encoders classifiers, which are often adopted in the existing MDA methods (Zhao al. (2019b C): Wang al. (019)).

---

Under review conference paper ICLR 2021
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy, Deep variational information bottleneck. arXiv preprint arXiv:611..00410. 2016.
Orly Alter, Patrick O Brown, and David Botstein. Singular value decomposition for genome-wide expression data processing and modeling. Proceedings of the National Academy of Sciences, 97 (100:101011 2000.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances neural information processing systems, pp- 137-144, 2007.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 19(1-2)1111-775, 2010,
John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. In Advances in neural information processing systems, pp. 129-136, 2008.
Rita Chattopadhyay, Qian Sun, Wei Fan, Ian Davidson, Sethuraman Panchanathan, and Jieping Ye. Multisource domain adaptation and its application to early detection of fatigue. ACM Transactions on Knowledge Discovery from Data (TKDD), 6(4):1-26 2012.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang- Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. International Conference on Machine Learning, pp. 1081-1090, 2019.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systemx, pp. 3730-3739, 2017.
Lixin Duan, Dong Xu, and Shih-Fu Chang. Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1338-1345 IEEE, 2012a.
Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang. Domain adaptation from multiple sources: domain-dependent regularization approach. IEEE Transactions on neural networks and learning systems, 23(3):504-5)) -18, 2012b.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.744: 740999977499222 2014.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17()):009-20330, 2016.
Boqing Gong, Kristen Grauman, and Fei Sha. Reshaping visual datasets for domain adaptation. In Advances in Neural Information Processing Systems, pp. 1286-1294, 2013.
Rui Gong, Wen Li. Yuhua Chen, and Luc Van Gool. Dlow: Domain flow for adaptation and general- ization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2477-2486, 2019.
Tan Goodfellow, Jean Pouget-Ahadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680 2014.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009.
Judy Hoffman, Brian Kulis, Trevor Darrell, and Kate Saenko. Discovering latent domains for multisource domain adaptation. In European Conference on Computer Vision, pp. 702-715. Springer, 2012.
10

---

Under review conference paper ICLR 2021
Judy Hoffiman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprini arXiv: 1.002113, 2017.
Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. Algorithms and theory for multiple-source adaptation. In Advances in Neural Information Processing Systems, pp. 8246-8256, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1rXiv:1502.03167, 2015.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information processing systems, pp- 315-323, 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(1 1):2278-2324, 1998.
Yitong Li, David E Carlson, al. Extracting relationships by multi-domain matching. In Advances in Neural Information Processing Systems, pp. 6798-6809, 2018.
Hong Liu, Mingsheng Long. Jianmin Wang. and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning, pp. 4013 4022, 2019.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer joint matching for unsupervised domain adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1410-1417, 2014.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. arXiv preprint arXiv:1502.0277 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 2217-2217. JMLR. org, 2017.
Yawei Luo, Ping Liu, Tao Guan, Junqing Yu, and Yang. Significance- information bottleneck for domain adaptive semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6778-6787, 2019.
Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Buld, Barbara Caputo, and Elisa Ricci. Boosting domain adaptation by discovering latent domains. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3771-3780, 2018
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. In Advances in neural information processing systems, pp. 1041-1048, 2009.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference computer vision, pp. 2794-2802, 2017.
Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image image translation for domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4500-4509, 2018.
Paul K Newton and Stephen A DeSalvo. The shannon entropy of sudoku matrices. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 466(2119)11957-11975 2010.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1415-1415, 2019.

---

Under review conference paper ICLR 2021
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Fr-train: mutual information- based approach to fair and robust training. arXiv preprini arXiv:2002. 10234, 2020.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regulariza- tion. arXiv preprint arXiv: 2017.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3723-3732, 2018.
Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8503-8512, 2018a.
Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa. Learning from synthetic data: Addressing domain shift for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3752-3761, 2018b.
Yuxuan Song, Lantao Yu, Zhangjie Cao, Zhiming Zhou, Jian Shen, Shuo Shao, Weinan Zhang, and Yong Yu. Improving unsupervised domain adaptation with variational information bottleneck. arXiv preprint
2019.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Eurepean conference on computer vision. pp. 443 450. Springer, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0000057, 2000.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7472-7481, 2018.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pp. 5018-5027,200 2017.
Haotian Wang, Wenjing Yang. Zhipeng Lin, and Yue Yu. Tmda; Task specifc multi-source domain adaptation via clustering embedded adversarial training. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 1372-1377. IEEE, 2019.
Jindong Wang. Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu. Visual domain adaptation with manifold embedded distribution alignment. In Proceedings of the 26th ACM international conference on Multimedia, pp. 402 410, 2018
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEE Conference on Computer Vision and Pattern Recognition, pp. 3964-3977 2018.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In Advances in neural information processing systems, pp. 8559-8570, 2018
12

---

Under review conference paper ICLR 2021
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representation for domain adaptation. arXiv preprini 1rXiv:190..09453 2019a.
Sicheng Zhao, Li, Xiangyu Yue, Yang Gu, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source domain adaptation for semantic segmentation. In Advances in Neural Information Processing Systems, pp. 7285-7298, 2019b.
Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu. Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-soure distilling domain adaptation. arXiv preprint arXiv: 11544, 2019c.

---

Under review conference paper ICLR 2021
A PROOFS
this Section, we present the detailed proof' for Theorems 2 and explained in the main paper. We also present Lemma 2 as mentioned in Section Following (Roh et al. (2020)), we provide a proof of Theorem 2 below for the sake of completeness.
A.1 PROOF OF THEOREM 2
Theorem 2. Let Pz(z) (z) be the distribution of Z where E Z. Let h be domain classifier h : 2 - V. where 2 is the feature space and V is the set of domain labels. Let (Z) be a conditional probability of where v given = z, defined by h. Then the following holds:
I(ZV)
max h. (x h1(2)=1,va VEV
(v [log (z)) H()
(17)
Proof. By definition,
1(z = DKI, P(Z)P(V))
log Pz.((z,)) H(V
(18)
PV
Pz(2)
VEV
Let us constrain the term inside the log by 1 = (x.((a)) where (2) represents the conditional Px(x)
probability of V v for any E V given Z Then have: Evey (z) 1 for all possible values of z according to the law of total probability. Let h denote the collection of (z) for all possible values of and and X be the collection of > for all values of Then, we construct the Lagrangian function by incorporating the constraint Ever hz(z) 1 as follows:
1(h, A)
[vV(v) (lo((h((z))) + H(V) +
Σ No (1
Σ (w(()) VEV
(19)
VEV
2E
We can use the following KKT conditions:
@l.h, A)
(v(v) (2)v(z) x² v(z,v)
(20)
Thy (z)
(z)
Σ (z) 0, V
(21)
Solving the two equations, we have for all the possible values of
Σ
Pr(())2/v(x) such that A; = P2(2) for all z. Then
A2
vEV
(z,((z,v)
(z)Z))
Pz(z)
(22)
Pv)a(v).
where the given (, is same as the term inside log in 18). Thus, the optimal solution of con-
cave Lagrangian function (19) obtained by (z) is equal to the mutual information in (8)). The
substitution of h (z) into (T8) completes the proof.
Our framework can further be applied to segmentation problems because provides new perspective on pixel space (Sankaranarayaaan et al. (2018 8a by; Murez et al. (2018) and segmentation space (Tai al. (2188) ) adaptation. The generator pixel space and segmentation space adaptation learns transform images or segmentation results from one domain to another. In the context information
regularization, we can view these approaches as limiting information I(X; V) between the generated output X and the domain label V. which is accomplished by involving the encoder for pixel-level generation. This alleviates the domain shift in a raw pixel level. Note that one can choose between limiting the feature-level or pixel-level mutual information. These different regularization terms may be complementary each other depending on the given task.

---

Under review conference paper ICLR 2021
A.2 PROOF THEOREM
Theorem 3. Let (z)x,(() be a conditional probabilistic distribution of Z where & 2, defined by the encoder F, given sample X and the domain label V. Let Rz Rz(z) denotes prior marginal distribution of Then the following inequality holds:
I(Z; V.
max by (x) Ro(2)-1,V2 1,82 VEV
H(V)
(23)
Px.vv D Rz]
Proof. Based on the chain rule for mutual information,
I(2 I(: I(2:XIV)
Σ
Pv(v) ( l (2) H(V)+ IV).
max hw(2) hu
VEV
(24)
where the latter equality is given by Theorem Considering I(Z;X V).
log (z.xlv(z,x)
Ev Pr I(2:X V)
Ex.xx
I'z (x)(())(x/(x))
12 (2)
Ex
B Be ParPa
log
Pz v (z)
Ex.vxxx.vv
Be Pa xvv
log P2/ (z)]
Ev Pe [log Pz/v [zi]
(25)
x Ex.v-Px.v Px.v
B Pan
log xx (x)]
By Ex- log Rz
Pz)xxv(2))
Px.v
Ber log
Rz(z)
Px,
DKL PZix.
Rz)
The second equality is obtained by using Pz, Pz,xju(z,x) (x) The inequality is
obtained by using D v Rz] Ex-Pai log (z)((z) logg R2(z)) O. where Rz is variational approximation prior marginal distribution of The last equality is obtained from the definition of KL-divergence. The substitution of (25) into (24) completes the proof.
The existing DA work on semantic segmentation tasks (Luo et al. (2019); Song et al. (2019)
) can be explained the process of fostering close collaboration between the aforementioned information bottleneck terms. The only difference between Theorem for V {0. 1} and the objective function (Luo et al. (2019) is that (Luo et al. (2019) ) employed the shared encoding P2 (z) instead of Pz Z). whereas some adversarial DA approaches use the unshared one (Teng et al. (017)).
A.3 PROOF OF LEMMA 2
dxe Lemma 2. Let dx( N+T Eveydx(D (Dv, D. Let H. be hypothesis class. Then,
dx(V) S N(N + 1))
Σ
dn(DD. Du)
(26)
v.uEV

---

Under review conference paper ICLR 2021
Proof. Let
represents the uniform domain weight for the mixture of domain Dye. Then,
dx(V)=
Σ
Dv..
N +
VEV
1 N +1
Σ
2 sup hEft
Ex-Poy [[(1(( = 1))] ExnPnx [1(1(x = 1))
VEV
Σ 2 sup
Σ uE Vinsév
a (BxxPP [1(x(x = 1))] ExwPag [1(x = 1))1)
N +
VEV
Σ
a 2 sup LEM
ExmP Dz [T(x(x = 1))] Pog [T< = 1))]
N +1
VEVnEVaaVv
1
Σ
dn
Du).
N(N 11)
V.VV
(27)
where the inequality follows from the triangluar inequality and jensen's 's inequality.
B EXPERIMENTAL SETUP
this Section, we describe datasets, network architecture and hyperparameter configuration.
B.1 DATASETS
We validate the Multi-source Information-rrgullriieed Adaptation Networks (MIAN) with the follow- ing benchmark datasets: Digits-Five. Office-31 and Office-Home. Every experiment is repeated four times and the average accuracy in target domain is reported.
Digits-Fiv Five (Peng al. (2019)) dataset is unified dataset including five different digit datasets: MNIST (LeCun et al. (1998)). MNIST-M (Ganin Lempitsky (2014)) Synthetic Digits (Ganin & Lempitsky (2014)), SVHN and USPS. Following the standard protocols of unsupervised MDA (Xu al. (2018); Peng al. (2019)), we used 25000 training images and 9000 test images sampled from a training and testing subset for each of MNIST, MNIST-M. SVHN, and Synthetic Digits. For USPS, all the data is used owing to the small sample size. All the images are bilinearly interpolated to 32 32.
Office-31 (Saenko et al. (0100) is popular benchmark dataset including 31 categories of objects an office environment. Note that is more difficult problem than Digits-Five, which includes 4652 images in total from the three domains: Amazon, DSLR, and Webcam. All the images are interpolated to 224 x 224 using bicubic filters.
Office-Home (Venkateswara of objects in office and home environments. It includes 15,500 images in total from the four domains: Artistic images (Art), Clip An(CCipaart, Product images (Product), and Real-World images (Realworld), All the images are interpolated to 224 x 224 using bicubic filters.
al. (217)) challenging dataset that includes 65 categories
B.2 ARCHITECTURES
For the Digits-Five Five dataset,
the same network architecture and optimizer setting in (Peng al. (2019)). For all the other experiments, results are based on ResNet-50, which is pre-trained on ImtgeNet. The domain discriminator is implemented as three-layer neural network. Detailed architecture shown in Figure 5
We compare our method with the following state-of-the-art domain adaptation methods: Deep Adaptation Network (DAN, Long et al. (013)). Joint Adaptation Network (JAN, Long et al. (2017). Manifold Embedded Distribution Alignment (MEDA, Waan et al. (2018)) Correlation Alignment (CORAL, Sun al. 2016)), Domain Adversarial Neural Network (DANN, Ganin et al. (201
16

---

Under review conference paper ICLR 2021
Colie Cendie
(a) Existing works
(b) Proposed model
Figure 4: Comparison of existing and proposed MDA models. (a) Existing multiple-diseriminatoo based methods align each pairwise source and target domain but may fail by neglecting the domain shift between source domains. also may suffer from unstable optimization and lack of resource- efficiency. (b) Our proposed model mitigates suggested problems by unifying domain discriminators.
Plic
Cliba
paatre Deginss
Prcpent
Pebi
Derre Beiin Spurnel (MBAhh))
(a) Encoder, domain discriminator, and classifier used (b) Encoder, domain discriminator, and classifier used in Digits-Five experiments
in Office-31 and Office-Home experiments
Figure 5: Network architectures. BN denotes Batch Normalization (loff & Szegedy (2015) and SVD denotes differentiable SVD in PyTorch for MIAN-y (Section
Batch Spectral Penalization (BSP, Chen (2099 ) Adversarial Discriminative Domain Adaptation (ADDA, Tzeng et (2017))) 7h), Maximum Classifier Discrepancy (MCD, /Satoo et al. (2018)). Deep Cocktail Network (DCTN. Xu et al. (2018)), and Moment Matching for Multi-Source Domain
Adaptation SDA, Peng et al. (2019)).
Table 4: Experimental setup. The batch size for each domain is reported.
Dataset
Optimization method
Learning rate
Momentum
Batch size
Iteration
Digits-Five
Adam mini-batch SGD
2e 4
(0.9, 0.99)
128
50000
Office-31
0.9
16
25000
Office-Home
mini-batch SGD
1e
0.9
16
25000
Hyperparameters Details of the experimental setup are summarized in Table 4 Other state-of-the- art adaptation models are trained based on the same setup except for these cases: DCTN show poor performance with the learning rate shown Table 4 for both Office-31 and Offic-Homee datasets. Following the suggestion of the original authors, le is used as learning rate with the Adam optimizer (Kingma & Ba (2014) MCD show poor performance for the Office-Home dataset with the learning rate shown Table le is selected learning rate. For both the proposed and other baseline models, the learning rate the classifier or domain discriminator trained from the scratch is set to be 10 times of those of Mmtageee-pretraiiin weights. in Office-31 and Office-Homm datasets. More hyperparameter configurations are summarized in Table (Section
C PSEUDOCODE
Due to the limited space, we provide the algorithm of MIAN in this Section. Details about training- dependent scaling of B, in Section E.

---

Under review conference paper ICLR 2021
Algorithm 1: Multi-source Information-regulliizzze Adaptation Networks (MIAN)
mini- bani-bacch size for each domain=m, Number of source domains=N, Training iteration 1}.
M=m([N + 1). Set of domain labels V = (I.....N.. N
for to 7' do
X {x+
union of samples : XsN XT)
Y [y: 1m a union samples (Ys,:...,,s,,
i= Let z, F(x.), and y. = ((F(x.)), vx
logh(z)) Backpropagate gradient of [(h), the variant (Mao et al. (2017)). to h.
[(h) - Evey Eiv,ey [1 (kvva log h(z;) + 1 A
log( h(z,))
[(F,C)= -mN Eyey Enyrey [k=y,, 8o 2(1 1+eep(-aa//1
// See Appendix
(()=)(,,))- (,1(h)
Backpropagate gradient of L(F) to F.
Backpropagate gradient of C(F,C)toC.
D ADDITIONAL RESULTS
Visualization of learned latent representations. We visualized domain--independen representations extracted by the input layer of the classifier with t-SNE (Figure (6) Before the adaptation process, the representations from the target domain were isolated from the representations from each source domain. However, after adaptation, the representations digits, as opposed to the domain.
well-aligned with respect the class of
Hy Hyperparameter sensitivity. We conducted the analysis on hyperparameter sensitivity with degree of regularization B. The target domain is set Amazon or Art, where the value 3g changes from 0.1 0.5. The accuracy high when 3g approximately between 0.1 and 0.3. We thus choose Sn 0.2 for Office-31, and Be 0.3 for Office-Home.
80
Art
Amauzn
Accuracy
70
0.1
(c)
(a)
(h)
Figure 6: (b)~ ISNE visualization (a) before and (b) after adaptation. Representations from target domain (SVHN) are shown red. Digit class labels are shown with corresponding numbers. (c) Analysis on hyperparameter sensitivity.
DECAYING BATCH SPECTRAL PENALIZATION
In this Section, we provides details on the Decaying Batch Spectral Penalization (DBSP) which expands MIAN into MIAN-Y.

---

Under review conference paper ICLR 2021
E.1 BACKGROUNDS
There is little motivation for models to control the complex mutual dependence to domains if reducing the entropy of representations is sufficient to optimize the value of [ = H(Z) H(2 V). If so, such implicit entropy minimization substantially reduce the upper bound of ((2 Y), leading to a increase in optimal joint risk X", In other words. the decrease in the entropy of representations may occur as the side effect of I(Z; V) regularization. Such unexpected side effect of information regularization is highly intertwined with the hidden deterioration of discriminability through adversarial training (Chen et al. 2219); y) Liu et al. (2019))
Based on these insights, we employ the SVD-entropy Hsv (Alter et al. 20000) of a representa- tion matrix Z to assess the richness of the latent representations during adaptation, since it is difficult
compute ZZ). Note that while Hsv D (Z) is not precisely equivalent to H ) H&v p (Z) can be used as a proxy of the level of disorder of the given matrix (Newton & DeSalvo (20T0). In future works, it would be interesting to evaluate the temporal change in entropy with other metrics. We found that Hsvpl p (Z) indeed decreases significantly during adversarial adaptation, suggesting that some eigenfeatures (or eigensamples) become redundant and, thus, the inherent feature-richness diminishes. (Figure 7ap To preclude such deterioration, we employ Batch Spectral Penalization (BSP) (Chen al. (2019) ) which imposes a constraint on the largest singular value to solicit the contribution of other eigenfeatures. The overall objective function in the multi-domain setting is defined as:
N+1 &
min [(F.C)+ (i + F,C
(28)
where 3 and y are Lagrangian multipliers and 8i,j is the jth singular value from the ith domain. We found that SVD entropy of representations is severely deteriorated especially in the early stages of training. suggesting the possibility of over-regularization The noisy domain discriminative signals in the initial phase (Ganin et al. (2016)) may distort and simplify the representations. To circumvent the impaired discriminability
the early stages of the training. the discriminability should be prioritized first with high y and low 3. followed by a gradual decaying and annealing in y and 88, respectively, so that sufficient level of domain transferability is guaranteed. Based on our temporal analysis, we introduce the training-dependent scaling of and y by modifying the progressive training schedule (Ganin et al. (206))):
1
8p = Bo 2012111
1 1 expp 1oopp(oo1 o
(29)
2 7p (I+ (-tepppoo-) 1),
where 3o and "o are initial values, a is a decaying parameter, and p is the training progress from 0 to 1. We refer to this version of our model MIAN-y. Note that MIAN only includes annealing-$,
excluding DBSP. For the proposed method, 9o is chosen from {0.1, 0.2, 0.3. 0.4, 0.5) for Office-31 and Office-Home dataset, while 3o 1.0 is fixed in Digits-Five. Too is fixed to (le-t} following
Chen et al. (2019).
Table 5: Hyper parameters configuration. Annealing-B 3 is not adopted in the Digits-Five experiment. Decaying batch spectral penalization is not adopted in the MIAN
Dataset(Model)
Po
70
k
Digits-Five (MIAN)
1.0
N/A
N/A
N/A
Office-31 (MIAN)
0.1
N/A
10.0
N/A
Office-31 (MIAN-7)
0.2
0.0001
10.0
Office-Home (MIAN)
0.3
N/A
10.0
N/A
Office-Home (MIAN-7)
0.3
0.0001
10.0
E.2 EXPERIMENTS
SVD-entropy. We evaluated the degree of compromise of SVD SVD-entropy owing to transfer learning. For this, DSLR was fixed as the source domain, and each Webcam and Amazon target domain
19

---

Under review conference paper ICLR 2021
was used simulate low (DSLR Webcam; DW) and high domain (DSLR DOSLR=AAmazon >Amazon; DA) shift conditions, respectively. SVD-entropy was applied to the representation matrix extracted from ResNet- 50 and MIAN (denoted as Adapt in Figure 7a) with constant 3 0.1. For accurate assessment, we avoided using spectral penalization. As depicted the Figure 7a adversarial adaptation, or information regularization, significantly decreases the SVD-entropy of both the source and target domain representations, especially in the early stages of training, indicating that the representations are simplified in of feature-richness, Moreover, when comparing the Adapt_ Souuoe and Adapt_ WW_oource conditions, we found that SVD-entropy decreases significantly as the degree of domain shift increases.
We additionally conducted analyses temporal changes of SVD entropy by comparing BSP and decaying BSP (Figure(7b) SVD entropy gradually decreases as the degree of compensation decreases in DBSP which Icads to improved transferability and accuracy. Thus DBSP can control the tradc-off between the richness of the feature representations and adversarial adaptation the training proceeds.
Ablation study. We performed an ablation study assess the contribution of the decaying spectral penalization and annealing information regularization to DA performance (Table 6 7 We found that the prioritization of reature-richneee in early stages (by controlling 3 and 7) significantly improves the performance. We also found that the constant penalization schedule (Chen al. (2019) is not reliable and sometimes impedes transferability in the low domain shift condition (Webeam, DSLR Table 6). This implies that the conventional BSP may over-regularize the transferability when the degree of domain shift and SVD-etrrppy decline are relatively small.
0.9
0.8
500 esemey
0.000
0875
0.5
9.0-
0.62
0.00
8875
0375
Number of Steps (xLe3)
25
8890
10
23
Masticn Gcludl
(b)
Figure 7: (a): SVD-entropy analysis. (Office-31. 31: Source domain: DSLR) (b): Comparisons between BSP and DBSP. (Office-31. 31; DSLR Amazon)
Table 6: Ablation study of decaying batch spectral penalization and annealing information regulariza- tion (Office-31). For accurate assessment of extent which performance improvement caused by each strategies, y is fixed as in Annealing- and | is fixed as 0.1 in Decaying-y, Results from A are reported in main paper.
Standards
Hyper parameters
Amazon
DSLR
Webcam
Avg
Baseline
B 0.1 as constant
69.98
99.48
98.13
89.20
Annealing-8
80 0.1, 10
74.65
99.48
98.49
90.87
7 le as constant
74.73
98.65
96.24
89.87
Decaying-y
70 le l 10
75.01
99.68
98.10
90.93
Full version
Bo = 0.1, 70 le 4 1
76.15
99.22
98.39
91.26
20

---

Under review conference paper ICLR 2021
Table Accuracy (%) on Office-Home dataset. Results from MIAN are reported in main paper.
Standards
Art
Clipart
Product
Realworld
Avg
MIAN
69.3911..00
63.05+0.61
79.62
80.4410.24
73.12
MIAN-3
09.88:10.35
64.20:10.68
80.87:0.37
0.24
74.11
21


Reviews:
[{"review_id": "ZV-It1oLfZv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple unified information regularization framework for multi-source domain adaptation. Experimental results verify the effectiveness of the proposed method. \n\nThe paper is well organized. The motivation is clear and the idea of reducing multiple discriminators to single discriminator is very reasonable. The experiments are somewhat convincing.\n\nThe main weaknesses of this paper lie in the following aspects:\n\n- The authors claim that domain-discriminative knowledge from other domains is not fully leveraged by multiple individual discriminators, while their proposed MIAN can filter out domain-specific information while preserve domain-shared information. However, such a claim is not supported by solid theoretical or empirical evidence but only the authors intuition.  Since the motivation of the whole paper is based on this claim, it greatly undermines the basis of this paper. Hence, it is critical for the authors to provide sufficient evidence to demonstrate this claim.\n\n- The overall contribution is not that large, as  the proposed information regularization method is simply based on recently proposed related work.  And some aspects of the proposed method are not clearly explained. For example, why Eqn. (3) and Eqn. (6) are equivalent? It is not very obvious to me. \n\n- The experimental results are not good enough. The proposed MIAN exhibits good performance on small datasets like Digits-Five and Office-31. But its performance (an average of 73.12) on larger dataset like Office-Home is worse than other competitors like MFSAN (an average of 74.1). Furthermore, why not conduct experiments on DomainNet, the largest benchmark created specifically for DA?\n\n- The experimental analysis seems to be inadequate. The paper lacks hyperparameter sensitivity analysis, which is standard practice in UDA area. Furthermore, although there are some analysis on variance of stochastic gradients, there is no evidence to support that this will lead to better performance, or  increased computational-efficiency or improved stability.\n\n- The presentation can be greatly improved. Some typos exist and some mathematical equations are not formal enough. E.g., $\\Xcal$, $\\Zcal$ and $M$ are used without or before definition. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UQFmBP_k1hl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Multi-source Information-regularized Adaptation Networks (MIAN) for multi-source domain adaptation. The presentation has a strong theoretical flavour. MIAN is evaluated on three benchmark datasets against other methods to show its superiority. At times, the paper is difficult to follow and lacks clarity. Please see my detailed comments below.\n\n1. The paper is organised with three pages of theoretical insights (Section 3) after Section 2 on related works, which could be difficult for many readers. It could be better to convey the intuition, high-level idea, or big picture with some visual illustration that can help readers appreciate the proposed idea(s).\n\n2. The paper reproduced quite some existing theories (and also a proof in the  appendix). I am wondering whether these materials can be presented in a more accessible and compact way. \n\n3. Although source code has been provided (which is good), providing pseudo-code can help readers better understand the proposed method and differentiate it from other existing ones. \n\n4. It will be better to perform some computational complexity analysis to give a fuller picture including the efficiency of the proposed method. \n\n5. At the top of Page 7, it says \"we reproduced all the other baseline results using the same backbone architecture and optimizer settings as the proposed method\". Have you tune the optimizer settings for the proposed method and baselines? How did you determine the hyperparameters reported in the appendix? Grid search?\n\n6. It is not clear why standard deviations are not reported for Digits-Five.\n\n7. On the three datasets studied, the models evaluated seem to be different for different datasets. For example, there is no result on Single-best at all for Office-Home. For \"single-best\", seven methods (besides source only) were reported for Digits-Five but only DAN and JAN were reported for Office-31. \n\n8. Repeating each experiment for only four times may not give a good estimation of the variance. In addition, the 100.00+-0.00 result of DAN on DSLR is impressive. I am wondering whether this perfect result will hold if you have more repetitions. \n\n9. On Office-31, MIAN actually only outperforms JAN's single-best by 2%, and the second best by 1.4%, which is far from a large margin claimed. \n\n10. Figure 2a and 2b (similarly 2c and 2d) are hardly readable on print.\n\n11. Minor issues.\nFigure 1 could be put at the top rather than in the middle of text. \nAcronyms are not always defined, e.g. FCN in Figure 1, although knowledgeable readers know what it means.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GG9neA-iTG2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summarize what the paper claims to contribute.**\nThe paper introduces a multi-source information-regularized adaptation network (MIAN). MIAN consists of three parts; information regularization, source classification and batch spectral penalization. The information regularization in MIAN provides an information-theoretic interpretation of multi-source domain adaptation. This interpretation seems to have lower information variance compared to the other multi-source domain adaptation technique relying on multiple binary domain classifiers. The performance of MIAN is evaluated on three benchmarks and it achieves competitive performance comparable to state-of-the-art methods.\n \n**List strong and weak points of the paper.**\n\n*strong points*\n\nThe proposed information-theoretic regularization is well-motivated and profound compared to the previous works on multi-source domain adaptation.\nGive a first attempt for theoretical explanation on multi-source domain adaptation by extending previous approaches dealing with only source and target domain cases.\n\n*weak points*\n\nA derivation, an ablation study and an analysis on the proposed method are missing.\nThe paper seems to be not finished yet.\n \n**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice**\n\nI give “Ok but not good enough - rejection (4)” to this paper. The motivation of the paper is plausible to multi-source domain adaptation and the proposed regularization technique seems to be profound. But the supporting evidence of the proposed method, such as appropriate derivation and proof, ablation study and analysis, are missing in the paper. I conjecture that the paper is not finished yet.\n \n**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**\n\n*variance of information*\n\n“In contrast, the variance of our constraint (5) is inversely proportional to $(N+1)^2$” I can’t find the derivation of this term, which is a crucial statement for the proposed method. In the supplementary materials of this paper and (Roh et al., 2020), i could not find the related covariance terms. Could you explain the variance of $\\hat{I}(Z; V)$ in eq (5) and $\\sum_{k=1}^{N}{\\hat{I} (Z_k ; U_k) }$ in terms of the number of domains $N$?\n \n*ablation studies on MIAN*\n\nWhich one is the counterpart of MIAN? I want to know the performance of the baseline architecture without MIAN regularization, which is an empirical evidence to check whether MIAN is working or not.\n \n**Provide additional feedback with the aim to improve the paper.**\n\n*paper editing issues*\n\nThe methods in the table are not cited in both the table and the main text. This makes it hard to read the experiment table.\n \n*Missing related works.*\n\nRefer [R1, R2] for multi-source domain adaptation. The paper [R2] achieves similar performance with MIAN on multi-source domain adaptation on office-31 benchmark, although it doesn't report standard deviation.\nThe major works on multi-source domain settings are dealing with domain generalization and larger scale benchmarks, such as DomainNet.\n \n*analysis on the method*\n\nI think that the motivation of this paper can be intuitively explained by using a figure description of the Venn diagram of information-theoretic measures. Refer to the Venn diagram figures on the page [R3].\nIs it possible to quantify the information shared among/between domains? I think that kind of measurement is very effective for visualizing multi-source domain adaptation & domain generalization.\n \n[R1] Boosting Domain Adaptation by Discovering Latent Domain, CVPR 18\n\n[R2] Domain Specific Batch Normalization for Unsupervised Domain Adaptation, CVPR 2019\n\n[R3] https://en.wikipedia.org/wiki/Information_theory_and_measure_theory", "extra_scores": {}, "extra_text": {}}, {"review_id": "-5KoqPJ7kW9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the multi-source domain adaptation problem. The authors examine the existing MDA solutions, i.e. using a domain discriminator for each source-target pair, and argue that the existing ones are likely to distribute the domain-discriminative information across multiple discriminators. By theoretically analyzing from the information regularization point, the authors present a simple yet powerful architecture called multi-source information-regularized adaptation network, MIAN.\n\nI vote to accept the paper.\n-This paper has a clear motivation, is well written, and establishes the final objective step by step with the theoretical supports. \n\n-The proposed objective is simple but powerful. I enjoy reading the analysis of advantages over the existing solutions, which is well reflected in the experiments. The reported performance is competitive, even compared to the missing reference ECCV’20 (see below).\n\n-The quantitative analyses validate the effectiveness of the model design. Particularly, the analysis on variance of stochastic gradients validates the technical benefits on optimization stability. \n\nA few comments/suggestions.\n-A few recent MDA works are missing, including but not limited to [ref-1, ref-2, ref-3]. Although not all of them using image datasets as testbed, but I would encourage the authors to include and discuss them under the same structure. \n[ref-1] Hang Wang et al., Learning to Combine: Knowledge Aggregation for Multi-Source Domain Adaptation, ECCV 2020\n[ref-2] Chuang Lin et al., Multi-source Domain Adaptation for Visual Sentiment Classification, https://arxiv.org/abs/2001.03886\n[ref-3] Haotian Wang et al., Tmda: Task-specific multi-source domain adaptation via clustering embedded adversarial training. ICDM 2019.\n\n-In section 3.3, the authors argue that their frameworks filtering out domain-specific information while preserving the amount of domain-shared information. The statement and the earlier discussion are intuitively correct to me. However, it would be great to see a quantitative or qualitive study on the effectiveness of the preserved domain-shared information and domain-specific separately.\n\n-Minors. In page 5, “It bias the representation towards …” -> “biases”\n\n=====\nUpdates: Thanks for the authors' response. I carefully read other reviewers' comments and responses. My concern on the missing study of empirical or theoretical support of claim \"framework can filter out domain-specific information while preserving the amount of domain-shared information\" was also raised by other reviewer. Overall, I still believe this paper provides new insights for this field.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although all reviewers agree that the work is interesting and has potential, several issues in the presentation and the experimental section (especially regarding the ablation) need to be worked on before granting acceptance to the paper. 
Decision: Reject