A Closer Look at Smoothness in Domain Adversarial Training
Authors: Harsh Rangwani
Sumukh K Aithal
Arihant Jain
Venkatesh Babu Radhakrishnan
Venue: ICLR 2022 Submitted (2022)
Keywords: Domain Adaptation
Optimization

Abstract: Domain adversarial training has been ubiquitous for achieving invariant representations and is used widely for various domain adaptation tasks. In recent times methods converging to smooth optima have shown improved generalization for supervised learning tasks like classification.  In this work, we analyze the effect of smoothness enhancing formulations on domain adversarial training, the objective of which is a combination of classification and adversarial terms. In contrast to classification loss, our analysis shows that \textit{converging to smooth minima w.r.t. adversarial loss leads to sub-optimal generalization on the target domain}. Based on the analysis, we introduce the Smooth Domain Adversarial training (SDAT) procedure, which effectively enhances the performance of existing domain adversarial methods for both classification and object detection tasks. Our smoothness analysis also provides insight into the extensive usage of SGD over Adam in domain adversarial training.   

Under review as a conference paper at ICLR 2022
A CLOSER LOOK AT SMOOTHNESS IN DOMAIN ADVERSARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Domain adversarial training has been ubiquitous for achieving invariant represen- tations and is used widely for various domain adaptation tasks. In recent times, methods converging to smooth optima have shown improved generalization for supervised learning tasks like classification. In this work, we analyze the effect of smoothness enhancing formulations on domain adversarial training, the objective of which is a combination of task loss (eg. classification, regression etc.) and adversarial terms. In contrast to task loss, our analysis shows that converging to smooth minima w.r.f. adversarial loss leads to sub-optimal generalization on the target domain. Based on the analysis, we introduce the Smooth Domain Adver- sarial training (SDAT) procedure, which effectively enhances the performance of existing domain adversarial methods for both classification and object detection tasks. Our smoothness analysis also provides insight into the extensive usage of SGD over Adam in domain adversarial training.
INTRODUCTION
Domain Adversarial Training (Ganin & Lempitsky, 2015) (DAT) refers to adversarial learning of neural network based feature representations that are invariant to the domain. For example, car images from the clipart domain have similar feature representations domain. DAT has been widely useful in diverse areas (cited 3540 times) such as fairness (Adel et al.. 2019), object detection (Saito et al., 2019), domain generalization (Li et al., 2018), image- to-image translation (Liu et al., 2017) etc. The prime driver of research on DAT is its application in unsupervised Domain Adaptation (DA), which aims to learn classifier using labeled source data and unlabeled target data, such that it generalizes well on target data. Various enhancements like superior objectives (Acuna al., 2021: Zhang al., 2019), architectures (Long al., 2018) etc. have been proposed to improve its effectiveness. However, as DAT objective is combination of Generative Adversarial Network (GAN) (Goodfellow et al., 2014) and Empirical Risk Minimization (ERM) (Vapnik, 2013) objectives, there has not been much focus on explicitly analyzing the nature of optimization in DAT. One direction of work aiming to improve generalization of ERM on unseen data focuses on developing algorithms that converge to a smooth (or a flat) minima (Foret et al., 2021: Keskar & Socher, 2017). However, we find that these techniques, when directly applied for DAT, do not significantly improve the generalization In this work, we analyze the loss landscape near the optimal point obtained by DAT, to gain insights into curvature. We first focus the eigen-spectrum of Hessian of the task loss (ERM term for classification) where we find that using Stochastic Gradient Descent (SGD) as optimizer converges smoother minima in comparison Adam (Kingma & Ba, 2014). Further we find that smoother minima w. w.r.k.ask loss leads to better generalization on the targel domain. Contrary to task loss, we find that smoothness enhancing formulation for adversarial components worsen performance, rendering ERM-based techniques which enhance smoothness for all loss components ineffective. Hence we introduce Smooth Domain Adversarial Training (SDAT), which aims only to reach a smooth minima w.r.t. task loss, and helps in generalizing better on the target domain. SDAT requires an additional gradient computation step and can be combined with existing methods with few lines of code. We show the soundness of the SDAT method theoretically by proving generalization bound (Sec. 4) on target error. We extensively verify the empirical efficacy of SDAT across various datasets for classification (i.e.. DomainNet, VisDA-2017 and Office-Home) along with showing a prototypical application in DA for object detection, demonstrating it's diverse applicability. In summary, we make the following contributions:
car images from the web
the target domain (Sec. 4 and 7).

---

Under review as a conference paper at ICLR 2022
Classffiction Lass Rg
Classifer fe
Fealure Extractor g
DAT shorp
Smooth Classifiation Loss HusS R [ho-..)
argett Data
SDAT lcads to (el manime
Formard Propegation
GRL
Badcowr Propagatiin
D Disciiinnttrr
Acversar al Loss
D3
d
8,7
Figure 1:
Overview of Smooth Domain Adversarial Training. Conventional approaches of smooth- ing loss do not discriminate between adversarial loss and task loss. Based on our theoretical analysis we propose SDAT which only focuses on smoothing task loss, leading to effective generalization on target domain.
We analyze the optimization procedure of DAT, establishing the correlation between the smoothness near optima w.r.t. task loss and generalization on the target domain.
Contrary to ERM, we show through our theoretical and empirical analysis that smoothness enhancing adversarial formulation leads to sub-optimal performance.
For enhancing the smoothness w.r.t. task loss near optima in DAT, we propose novel and theoretically motivated SDAT that improves the generalization on the target domain. SDAT effectively increases the average performance of even state-of-the-ar adversarial adaptation methods.
2 RELATED WORK
Unsupervised Domain Adaptation: It refers to a class of methods that aim to adapt models to work in target domain distinet from what it was trained One of the most prominent lines of work is based on DAT (Ganin & Lempitsky, 2015). This involves using an additional discriminator distinguish between samples of source and target domain. The goal of the model is to learn fea- tures that can not be distinguished between source and target. The follow-up works have improved this basic idea by introducing class information based discriminator (CDAN (Long et al., 2018)), introducing a transferable normalization function (Wang et al., 2019) etc. In this work, we focus on analyzing and improving such methods. Another line of work involves DA by using self-training on target domain (Kundu et al., 2020b;a; Prabhu et al., 2020) which will not be the focus of this work.
Smoothness of Loss Landscape: As neural networks operate in the regime of over parameter- ized models, low error on training data does not always lead to better generalization (Keskar al., 2017). Often it has been stated (He al., 2019; Dziugaite & Roy, 2017) that smoother minima does generalize better on unseen data. But until recently, this was practically expensive as smoothing required additional costly computations. Recently, method called Sharpness Aware Minimization (SAM) (Foret et al., 2021) has been proposed to find a smoother minima with an additional gradient computation step. SAM also improves the ImageNet model performance (Chen et al., 2021) on ImageNet-C and ImageNet-R (which are out of distribution). It has also been observed that smooth- ness w.r.t. input (image) beneficial for domain adaptation (Shu et al., 2018: Cai al., 2021), which motivates us to explore smoothness w.r.t weights (W) in case of DAT. However, the earlier work has focused on achieving a smoother minima w.r.t. W for ERM. Currently, no study has been done if the loss function is composed of both ERM and adversarial objectives (as present in DAT).
BACKGROUND
3.1 PRELIMINARIES
We will primarily focus on Unsupervised DA where we have labeled source data S {(x) b()) and unlabeled target data T {<()). The source samples are assumed to be sampled i.i.d. from source
Figures for the smooth minima and sharp minima from (Foret et al., 2021) and used for illustration purposes only.
2

---

Under review as a conference paper at ICLR 2022
distribution P3 defined on input space X' similarly target samples sampled i..d. from Pr. D is used for denoting the label set which is {1, 2........ k} in our case as we perform multiclass (k) clas- sification. We denote y: X' + J a mapping from images to labels. Our task is to find a hypothesis function hg that has a low risk on the target distribution. The source risk (a.k.a expected error) of the hypothesis hg is defined with respect to loss function l as: (ho E (2)) y(x))]]- The target risk Ryy (ha) is defined analogously. The empirical versions of source and target risk will be denoted by R'' (ho) and Rty(hh)) All notations used paper are summarized in Table In this work we build on the domain adaption theory of (Acuna et al., 2021) which is generalization of Ben-David et al. (2010). We first define the discrepancy between the two domains.
Definition 3.1 (Dha.e discrepancy). The discrepancy between two domains Ps and Pr is defined as followin::
Do ho H. (Ps Pr) = sup Pe~Ps WEH
-
(1)
Here 6 is a frenchel conjugate of a lower semi-continuous convex function & that satisfies ((1) = 0, and H is the set of all possible hypothesis (i.e. Hypothesis Space).
This discrepancy distance D is based on variational formulation of f-divergence (Nguyen al., 2010) for the convex function Φ. The Dio. H. the lower bound estimate of the f-divergence function (Pss (r)) See Lemma 4 in (Acuna et al., 2021) for additional details. We state a bound on target risk Ry (ha) based on Dh Dh. discrepancy (Acuna et al., 2021):
Theorem (Generalization bound). Suppose 1 x y + [0,1] C dom o- Let ht be the ideal joint classifier with least X R'shh") + (h") (i.e. joint risk) in H. We have the following
relation between source and target risk:
(ha) S Rlg(ho) + He (Ps|IPT)+ X
The above generalization bound shows that the target risk Riy (ho) is upper bounded by the source risk (ha) and the discrepancy term D Dhan2 along with an irreducible constant error A Hence, this infers that reducing source risk and discrepancy lead a to reduction in target risk. Based on this, we concretely define the unsupervised adversarial adaptation procedure in the next section.
3.2 UNSUPERVISED DOMAIN ADAPTATION
In this section we first define the components of the framework we use for our purpose: he = feoge gg where 9v is the feature extractor and fe is the classifier. The domain discriminator Dø, used for estimating the discrepancy between Ps and Pr is classifier whose goal is to distinguish between the features of two domains. For minimizing the target risk (Th. 1), the optimization problem can be written as:
min Ez-ry [1)) + D he 2 (Ps((Pr) 8
The discrepancy term under some assumptions (refer App. B) can be upper bounded by a tractable term:
D% th.. (Pss max dst
(4)
where - Ep [log(Da ( (x))) Epwp, log[1 Da (x)) This leads to the final opti-
mization objective of:
min max E (() (z). + de
(5)
The first term in practice is empirically approximated by using finite samples it's (ha) and used as task loss (classification) for minimization. The empirical estimate of the second term is adversarial loss which is optimized using gradient reversal layer (GRL) as it has min-max form. (Overview in Fig. 1) The above procedure composes DAT, and we use CDAN (Long et al., 2018) as our default DAT method.
3

---

Under review as a conference paper at ICLR 2022
DAT w/ Adam
DAT w/ SGD
SDAT
101
Amar: 2225,22 23
Aear 68
237
Arau 87.63
14-11
Ti(M): 8281511
Dersity (og Scale)
20
Tritf:: 2084.13
Pri 761 25
18-11
Large Gulker
[lng
10-1
28-
10
10-1
Elgervalues
Eigansaluss
Figure 2: Eigen Spectral Density plots of Hessian (V² 1V2R's (ho)) for Adam (left), SGD (middle) and SDAT (right) on Art > Clipart. Each plot contains the maximum eigenvalue (Amar) and the trace of the Hessian (Tr(H)), which are indicators of the smoothness (Lower Tr(H) and Amaz indicate the presence of smoother loss surface). Low range of eigenvalues (x-axis), Tr(H) and Xmaz for SGD indicates that it reaches a smoother minima compared to Adam. SDAT reaches a smoother minima compared to DAT with either SGD and Adam.
4 ANALYSIS OF SMOOTHNESS
In this section, we analyze the curvature properties of the loss with respect to the parameters. Specif- ically, we focus on analyzing the Hessian of empirical source risk H = V2 (ill'g(ho) which is the Hessian of classification (task) loss term. For quantifying the smoothness, we measure the trace Tr(H) and maximum eigenvalue of Hessian (Amaz) as a proxy for quantifying smoothness. This is motivated by analysis of which states that the high value of Amna and Tr(H) are indicative of low smoothness (Jastrzebski et al., 2020). We articulate our conjecture informally below:
Conjecture 1. Smoothing of empirical source risk (i.e. task loss) Rts (ha) leads to efficient DAT. In other words, decreasing Amaz of V R' leads to reduced error on target domain (ho).
For verifying conjecture, analyze the eigen spectrum of the Hessian (ha) where we find that in contrast to standard ERM (Ghorbani et al., 2019) the negative eigenvalues do not disappear as the training progresses. We show the Amaz. Tr(H) (H) and eigen spectrum for different algorithms, namely DAT w/ Adam, DAT w/ SGD and our proposed SDAT (which is described in detail in later sections) in Fig. 2. We find that high smoothness leads to better generalization on the target domain. We also provide additional results in Fig. 3 for empirical verification of the conjecture. Our conjecture also explains the reason for widespread usage of SGD for DAT as SGD converges to smoother minima (Ganin& & Lempitsky. 2015; Long et al., 2018; Saito et al., 2018a) which leads to efficient DAT, even though Adam has shown to be effective for min-max optimization (Gemp & McWilliams, 2019). More details regarding the Hessian analysis are provided in App. D.
4.1 SMOOTHING LOss LANDSCAPE
In this section we first introduce the losses which are based on Sharpness Aware Minimization (Foret et al., 2021) (SAM). The basic idea of SAM is to find smoother minima (i.e. low loss in neighborhood of 0) by using the following objective given formally below:
min max (0 + +) 0 SA
(6)
Here Loby any objective function to be minimized and p 2 0 is a hyperparameter which defines the maximum norm of the Since finding the exact solution of inner maximization is hard, SAM maximizes the first order approximation:
{(0) RJ arg max Lasg (0) + t VoLor (0) = (0)ll (0)l22
[el|Sp
The z(0) is added to the weights 0. The gradient update for 0 is then computed as V Lobo (0) (++2()))
The above procedure can be seen generic smoothness enhancing formulation for any Lobj. We now analogously introduce the sharpness aware source risk for finding a smooth minima:
max (ho 1
= max Ex-Ps (()) fx))
(8)
Sp
4

---

Under review as a conference paper at ICLR 2022
A) Sharpmess Correlation
B) Art - Clipart
C) SNGAN
GRAt
Ghandroot
Domein Accuracy (in %6
FID Score
350
225
200
SOAT (Tit Acc: 56.0)
Ada 5047
SOAT ml ade (Tig Acc: $2.8)
50
275
DAT SGD (Tgt Acc: 54.3)
30'
20002 40002
82000
120002
Avec
Iteration
Step
Figure 3: A) Error on Target Domain (y-axis) for Office-Home dataset against maximum eigenvalue Amaz of classification loss in DAT. When compared to SGD, Adam converges to a non-smooth minima (high Amax). leading to high error on target. B) Domain Accuracy (vs iterations) is lower when discriminator is smooth (i.e. SDAT w/ adv), which indicates suboptimal discrepancy estimation d=,, C) SNGAN performance on different datasets, smoothing discriminator GAN also leads to inferior GAN performance (higher FID) across both datasets.
We also now define the sharpness aware discrepancy estimation objective below:
max
++ + |el|
min
(9)
As d is be maximized the sharpness aware objective will have min instead of max, as |el|S
|ells
needs to find smoother maxima. We now theoretically analyse the difference in discrepancy estima- tion for smooth version dg (Eq. 9) in comparison to non-smooth version d'e'r (Eq. 4). Assuming D& is a L-smooth (common assumption for non-convex optimization (Carmon et al., 2020)), 7j is a small constant and d',r the optimal discrepancy, the theorem states:
Theorem 2. For a given classifier hg and one step of (steepest) gradient ascent iee. क' = $ + /V and T" = +
d "VVVdd,.rle++ || d ++((0)]))
S ST S n - co 2T(d's (g.r)
(10)
where a is the angle between Vd and Tds.rlo+400
The d's (non-smooth version) exceed d. (smooth diserepancy) significantly, as the term d'sr- -or--d5 dg.r A O, as the he objective is to oppose the convergence of d3 to optima dš.r (min-max training in Eq. 11). Thus dž:r be better estimate of discrepancy in comparison to dg"" A better estimate of d',, helps in effectively reducing the discrepancy between Ps and Pr, hence leads to reduced R2y (he). This is also observed in practice that smoothing the discriminator (SDAT w/ adv in Fig. 3) leads to low domain classification accuracy (proxy measure for d in comparison to DAT. Due ineffective discrepancy estimation, SDAT w/ adv results in sub-optimal generalization on target domain i.e. high target eтror (ha) (Fig. 3). For further establishing the generality of sub-optimality of smooth adversarial loss, we also perform experiments on Spectral Normalised Generative Adversarial Networks (SNGAN) (Miyato et al., 2018). In case of SNGAN we also find that smoothing discriminator through SAM leads to suboptimal performance (higher FID) as in Fig- 3. The above evidences indicates that smoothing the adversarial loss leads to sub-optimality, hence it should not be done in practice. The proof of the above theorem and additional experimental details are provided in the supplementary (refer App. C and App. E).
4.2 SMOOTH DOMAIN ADVERSARIAL TRAINING (SDAT)
We propose smooth domain adversarial training which only focuses on converging to smooth min- ima w.r.t. task loss (i.e. empirical source risk), whereas does no change for the discrepancy term. We define the optimization objective of our smooth domain adversarial training below:
5

---

Under review as a conference paper at ICLR 2022
min max max Rz ( ((:))] + +dg.r
(11)
1kl|S
The first term is the sharpness aware risk, and the second term the discrepancy term which is smooth in our procedure. The term d5 estimates D%, (Ps Pr) discrepancy. We empirically find that this optimization procedure effectively reduces the generalization error on the target domain compared to all other alternatives. We now show that optimizing Eq. 11 reduces Ry (ha) through a generalization bound. This bound establishes that our procedure is also consistent (i.e. in case of infinite data the upper bound is tight), similar to the DAT (Ganin et al., 2016) baseline.
Theorem 3. Suppose is the loss function, we denote X == R'ghh") ) + (h") and let ht be the ideal joint hypothesis:
By (ho) S max Rig(hote) + D (Psl < (110 +
(12)
Iklls
where 7 IR+ R+ a strictly increasing function.
The bound is similar generalization bounds for domain adaptation (Ben-David al., 2010; Acuna al., 2021). The main difference is the sharpness aware risk term max]jellsp (ho in place of source risk R's (ho), and an additional term that depends on the norm of the weights >(10)1//p²) The first is minimized by decreasing the empirical sharpness aware source risk by using SAM loss shown in Sec. 4. The second term is reduced by decreasing the discrepancy between source and target domains. The third term, it is function of norm of weights ||012² be reduced by using either L2 regularization or weight decay. Since we assume that the H. hypothesis class we have is rich, the A term is small. We now show the improvements due SDAT empirically in the following sections.
ADAPTATION FOR CLASSIFICATION
We evaluate proposed method three datasets: Office-Home, VisDA-2017, and Net,in as well as by combining SDAT with two DAT based DA techniques: CDAN and CDAN+MCC.
5.1 DATASETS
Home (Venkateswara et al., 2017): Office-Home consists of around 15,500 images from 65 classes and four distinct domains: Art (Ar), Clipart (CI), Product (Pr) and Real World (Rw).
-2017 (Peng et al., 2017): VisDA is dataset that focuses on the transition from simulation real world and contains approximately 280K images across classes.
DomainNet (Peng et al., 2019): DomainNet consists of 0.6 million images across 345 classes be- longing to six domains. The domains are infograph (inf), clipart (clp), painting (pnt), sketch (skt), real and quickdraw.
5.2 DOMAIN ADAPTATION METHODS
CDAN (Long et al., 2018): Conditional Domain Adversarial network is popular DA algorithm that improves the performance of the DANN algorithm. CDAN introduces the idea of multi-linear conditioning to align the source and target distributions better. CDAN* in Table and 4 refers to our implementation of CDAN method.
CDAN MCC (Jin et al., 2020): In this method, the minimum class confusion loss term is added as a regularizer to CDAN. Minimum class confusion is a non-adversarial term that minimizes the pair- wise class confusion on the target domain. This achieves state of the art accuracy among adversarial adaptation methods.
5.3 IMPLEMENTATION DETAILS
We implement our proposed method the Transfer- Transier-Learingg. (Junguang Jiang & Long, 2020) toolkit developed in Py PyTorch (Paszke al., 2019). The main difference between the perfor- mance reported the CDAN and our implementation (CDAN*) is the batch normalization layer in the domain classifier, which enhances performance.

---

Under review as a conference paper at ICLR 2022
Table 1: Accuracy (%) on Office-Home for unsupervised domain adaptation (ResNet-50). CDAN+MCC w/ SDAT outperforms other sophisticated state-of-the-an art DA techniques. CDAN w/ SDAT improves over performance of CDAN by 1.1 %.
Method
Ar-CI
Ar-Pr
Ar-Rw
Cl-Ar
Cl-Pr
CI-Rw
Pr-Ar
Pr-C1
Pr-Rw
Rw-Ar
Rw-CI
Rw-Pr
ResNet 30 [lle at al, 1016)
31,9
30,0
38.0
37.4
11.9
46.2
38.5
31.2
60.4
53.9
11.2
59.3
46.1
DAN (Long 2015)
43.6
57.0
67.9
25.8
56.5
600.4
44.0
43.6
67.7
63.1
51.5
74.3
56.3
(Gasin
2016)
76.8
JAN (Long et al., 2017)
15.9
61.2
68.9
50.4
59.7
61.0
45.8
43.6
70.3
63.9
52.4
76.8
58.3
CDAN (Long al., 2018)
49.0
69.3
74,5
54.4
66.0
68.4
53.6
48.3
75.9
68.4
55.4
80.5
63.8
MDD (Zhang at., 2019)
54,9
73.7
77.8
60.0
71.4
71.8
61.2
53.6
78.1
72.5
6012
82.3
68.11
FDAL pearson alignment
2021) 2021)
584
83.7
SRDC
(Tag al., 2020)
52.3
76.3
81.0
69.5
76.2
78.0
68.7
53.8
81.2
76.3
57.1
85.0
71.3
CDAN
54.3
70.6
76.8
61.3
09.5
71.3
61.7
55.3
80.5
74.8
00.1
84.2
68.4
CDAN w/ SDAT
56.0
72.2
78.6
62.5
73.2
71.8
62.1
55.9
80.3
75.0
61.4
84.5
69.5
CDAN+ MCC ala. 2001
57.0
76.0
81.6
64.9
75.9
75.4
63.7
56.1
81.2
14.2
61.
854
71.3
CDAN MCC w/ SDAT
58.2
77.1
82.2
66.3
77.6
26.8
63.3
57.0
82.2
74.9
64.7
86.0
72.2
We use a ResNet-50 backbone for 2017 and DomainNet experiments. The backbone is initialized with ImageNet weights. We use a learning rate of 0.01 with batch size 32 in all of our experiments. We tune value in SDAT for a particular dataset and use the same value across domains. The value is set to 0.02 for the Office-Home experiments, 0.005 for the VisDA-2017 experiments and 0.05 for the DomainNet experiments. More details are present in supplementary (refer App. F).
experiments and ResNet- 101 backbone for VisDA-
5.4 RESULTS
Table 2 shows the results on the large and chal- lenging DomainNet dataset across five domains as done in (Junguang Jiang & Long, 2020). The proposed method improves the performance of CDAN significantly across all source-target pairs. On specific source-target pairs like info- graph real, the performance increase is 4.5%.
Table 2: Results on DomainNet with CDAN w/
SDAT. The number in the parenthesis refers to the increase in accuracy with respect to CDAN.
Target (-)
Source (L)
clp
inf
pnt
real
skt
Avg
22.0
41.5
57.5
47.2
42,
clp
(+1..4
(+2.6)
(+1.5)
(+2.3)
(+2.0)
The overall performance of CDAN is improved by nearly 1.8% which is significant considering the large number of classes and images present in DomainNet.
inf
33,9
30.3
48.1
27.9
35.0
(+2.3)
(+1.0)
(+4.5)
(1.5)
(+2.3)
47,5
20.7
58.0
41.8
42.0
pnt
(+3.4)
(+0.9)
(+0.8)
(+1.8)
[+1.7)
56.7
25.1
53.6
43.9
44,8
For the Office-Home Home dataset, compare methods with other domain adaptation algo- rithms including DANN, SRDC, MDD and f- DAL. The results for the Office-Home dataset shown in Table 1. We can that adding
real
(+0.9)
(+0.7)
(+0.4)
(+1.6)
(+1.0)
58.7
21.8
48.1
57.1
46.4
skt
(+2.7)
(+1.1)
(+2.8)
(+2.2)
(+2.2)
49.2
22.4
43.4
55.2
40.2
42.1
Avg
(42.3)
(+1.0)
(+1.7)
(42.2)
(+1.8)
(+1.8)
SDAT improves the performance on both CDAN and CDAN+MCC across all the transfer tasks.
CDAN+MCC w/ SDAT achieves state-of of-tee-f-eeeeeee adversarial adaptation performance on the Office- Home dataset.
The class-wise wiss accuracy on VisDA-2017 are reported in Table 4. CDAN w/ SDAT improves the overall performance of CDAN by more than 1.5%. CDAN w/ SDAT improves the performance of underperforming minority classes like bicycle and car. Additional baselines and results are reported in supplementary (refer App. G) along with discussion statistical significance (App. J).
ADAPTATION FOR OBJECT DETECTION
To further validate our approach's 's generality and extensibility, we did experiments on DA for object detection. We use the same setting as proposed in DA-Faster (Chen et al., 2018) with all domain adaptation components and use it as our baseline. We use the mean Average Precision at 0.5 loU (mAP) our evaluation metric. In object detection, the smoothness enhancement can be achieved in two ways (empirical comparison in Sec. 6.2):
a) DA-Faster w/ SDAT-Clasiffictioon: Smoothness enhancement for classification loss.
b) DA-Faster w/ SDAT: Smoothness enhancment for the combined classification and regression loss.
7

---

Under review as a conference paper at ICLR 2022
Table 4: Accuracy (%) on VisDA-2017 for unsupervised domain adaptation (ResNet-100). The
mean column contains mean across all classes. SDAT particularly improves the accuracy in classes that have comparatively low CDAN performance.
Method
plane
beybl
bus
car
horse
knife
meyle
persn
plant
sktb
train
truck
mean
ResNet (He et al., 2016)
55.1
53.3
61.9
59.1
80.6
17.9
79.7
31.2
81.0
26.5
73.5
8.5
52.4
DANN (Ganin et al. 2)16)
81.9
77.7
82.8
44.3
81.2
29.5
65.1
28.6
$1.9
54.6
8288
7.8
$7.4
DAN (Long al., 2015)
87.1
63.0
76.5
42.0
90.3
42.9
85.9
53.1
49.7
36.3
85.8
20.7
61.1
MCD (Saito al., 2018h)
87.0
60.9
83.7
64.0
88.9
79.6
84.7
76.9
88.6
40.3
83.0
25.8
71.9
CDAN oLng et al., 2018)
85.2
66.9
83.0
50.8
84.2
74.9
88.1
74.5
83.4
76.0
81.9
38.0
73.9
AFN (Xu ct al., 2019)
93.6
61.3
84.1
70.6
94.1
79.0
91.8
79.6
89.9
55.6
89.0
24.4
76.1
MCC (Jin et al., 2020)
88.1
80.3
80.5
71.5
90.1
93.2
85.0
71.6
89.4
73.8
85.0
36.9
78.8
CDAN
94.9
72.0
83.0
57.3
91.6
95.2
91.6
79.5
85.8
88.8
87.0
40.5
80.6
CDAN SDAT
94.8
77.1
82.8
60.9
92.3
95.2
91.7
79.9
89.9
91.2
88.5
41.2
82.1
CDAN+MCC (Jin et al., 2020)
95.0
84.2
75.0
66.9
94.4
97.1
90.5
79.8
89.4
89.5
86.9
$4.4
83.6
CDAN+MCC w/ SDAT
95.8
85.5
76.9
69.0
93.5
97.4
88.5
78.2
93.1
91.6
86.3
55.3
84.3
6.1 EXPERIMENTAL SETUP
We evaluate our proposed approach on object detection on two different domain shifts:
Pascal to Clipart (P - C): Pascal (Everingham et al., 2010) is a real-world image dataset which consists images with 20 different object categories. Clipart (Inoue et al., 2018) is a graphical image dataset with complex backgrounds and has the same 20 categories as Pascal. We (He et al., 2016) backbone for Faster R-CNN (Ren et al., 2015) following Saito et al. (2019).
Resnet-101
Cityscapes to Foggy Cityscapes (C' Fc): Cityscapes (Cordts et al., 2016) is a street scene dataset for driving, whose images are collected in clear weather. Foggy Cityscapes (Sakaridis et al., 2018) dataset is synthesized from Cityscapes for the foggy weather. We use Resnet-50 (He al., 2016) as the backbone for Faster R- R-CNN for experiments on this task. Both domains have the same 8 object categories with instance labels.
The training is done via SGD with momentum 0.9 for 70k iterations with the learning rate of 10 10-3, and then dropped to 10-4 after 50k iterations. We split the target data into train and validation sets and report the best mAP on validation data. Additional experimental and implementation details are present in supplementary (refer App. F).
6.2 RESULTS
Table shows the results two domain shifts with varying batch size (bs) during training. We find that only smoothing w.r.t. classification loss is much more effective (SDATCClasiffictionn) than smoothing w.r.t. combined classification and regression loss (SDAT). On average, SDAT- Classification produces an mAP gain of 2.0% compared to SDAT, and 2.8% compared to DA-Faster baseline.
The proposed cantly outperforms DA-Faster baseline and im- proves MAP by 1.3% on P + C and by 2.8% on -
signifi-
Table 3: Results on DA for object detection.
C- Fe
P- C
P
Method
(bx=2)
(bs=2)
(br=8)
noteworthy that increase in performance of SDAT-Classificttion is consis- tent even after training with higher batch size (bs = 8) achieving improvement of 4.3% in mAP.
Faster (Chen et al., 2018)
35.21
29.96
26.00
DA Faster w/ SDAT
37.47
29.04
27.64
DA-Faster w/ SDAII-lassttiction
38,00
31.23
30.74
Table 3 also shows that even DA-Faster w/ SDAT (i.c. smoothing both classification and regression) outperforms DA-Faster by 0.9 % on average across all experiments. The improvement due to SDAT on adaptation for object detection shows the generality of SDAT across techniques that have some form of adversarial component present in the loss formulation.
DISCUSSION
How much smoothing is optimal?: Figure 4 (A) shows the ablation on value (higher p value corresponds to more smoothing) on the Ar+CI and Cl-Pr from Office-Home dataset with CDAN backbone. The performance of the different values of p is higher than the baseline with p 0. It
"Our Implementation of CDAN. Refer to Section 5.3.
8

---

Under review as a conference paper at ICLR 2022
A
B
55.00
0343
55.73
Smooth Smooth dise Accuracy
Accuracy (In
Accuracy (In
55.25
34.3
51.0
55.00
30
55.7
54.75
23
54,9
54.00
54.23
0.01
0.82
0.03
0.25
Label Noise
Figure 4: Analysis of SDAT for Ar - CI split of Office-Home dataset. A) Variation of target ac- curacy with maximum perturbation p. B) Comparison of accuracy of SDAT with DAT for different ratio of label noise. C) Comparison of accuracy when smoothing is applied to various loss compo- nents.
can be seen that p= 0.02 works best among all the different values and outperforms the baseline by at least 1.5%. We found that the same p value usually worked well across domains a dataset, but different P was optimal for different datasets.
Which components benefit from smooth optima?: Figure 4 (C) shows the effect of introducing smoothness enbancement for different components in DAT. For this use SAM on classifier (SDAT) b) discriminator (SDAT w/ adv) c) both classifier and discriminator (SDAT-all). It can be seen that smoothing the adversarial component (SDAT w/ adv) reduces the performance to 51.0%, which is significantly lower than even the DAT baseline.
Is it Robust to Label Noise?: In practical, real-world scenarios, the labeled datasets are often corrupted with some amount of label noise. Due to this, performing domain adaptation with such data is challenging. We find that smoother minima through SDAT lead robust models which generalize well on the target domain. Figure 4 (B) provides the comparison of SGD vs. SDAT for different percentages of label noise injected into training data (by flipping the labels).
Is it better than other smoothing techniques? To answer this question, we compare SDAT with different smoothing techniques originally proposed for ERM. We specifically compare our method against DAT, Label Smoothing (LS) (Szegedy et al., 2016), and VAT (Miy- ato et al., 2019). Stutz et al. (2021) recently showed that these techniques produce a signif- icantly smooth loss landscape in comparison to SGD. We also compare with a very recent SWAD (Cha et al., 2021) technique which is shown effective for domain generalization. For this, we run our experiments on four different splits of the Office-Home dataset and summarize our re- sults in Table We find that techniques for ERM (LS and VAT) fail to provide significant consistent gain in performance which also confirms the requirement of specific smoothing strategies for DAT. We find that SDAT even outperforms SWAD on average by significant margin of 1. 1.2%. Additional details regarding the specific methods are provided in the supplementary (refer App. H).
Table 5: Performance comparison across differ- ent loss smoothing techniques on Office- Home. SDAT outperforms other smoothing techniques in each case consistently.
Method Ar-CI
Cl-Pr
Rw-CI
Pi-CI
Avg
DAI
34.3
69.3
60.1
55.3
59.2
VAT
54.6
70.7
60.8
54.4
60.1
(+0.9)
SWAD
54.6
71.0
60.9
55.2
60.4
(+1.2)
LS
53.6
71.6
59.9
53.4
59.6
(+0.4)
SDAT
56.0
73,2
61.4
55.9
61.6
(42.4)
CONCLUSION
In this work, we analyse the curvature of loss surface of DAT used extensively for Unsupervised Domain Adaptation. We find that converging to smooth minima w.r.t. task loss (i.e., empirical source risk) leads to better generalization on the target domain. We also theoretically and empirically show that smoothness enhancing for adversarial components of loss lead to sub-optimal results, hence should be avoided in practice. We then introduce our practical and effective method, SDAT, the target
which only increases the smoothness w.r.t. task loss, leading to better generalization domain. SDAT leads to an effective increase even for the state of the art methods for adversarial domain adaptation and can be incorporated with just few lines of code change. One limitation of SDAT is presence of no automatic way of selecting p (determines extent of smoothness) which is a good future direction to explore.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Acuna, Guojun Zhang, Mare T Law, and Sanja Fidler. f-domain-adversarie learning: Theory and algorithms. arXiv preprint arXiv:2106. 1344, 2021. 1, 3, 6, 7, 14, 16
Tameem Adel, Isabel Valera, Zoubin Ghahramani, and Adrian Weller. One-network adversarial fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2412- 2420, 2019. 1
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010. 3, 6
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint arXiv:2106.04732, 2021. 20
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://wwk.
wandb. com/ Software available from wandb.com. 17
Guanyu Cai, Lianghua He, Mengchu Zhou, Hesham Alhumade, and Die Hu. Learning smooth representation for unsupervised domain adaptation, 2021. 2
Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points I. Mathematical Programming, 184(1):71-120, 2020. 5,
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. arXiv preprint arXiv:2102.08604 2021. 9, 19
Xiangning Chen, Cho-Jui Hsich, and Boqing Gong. When vision transformers outperform resnets without pretraining strong data augmentations. arXiv preprint arXiv:2106.01548, 2021. 2, 16
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-enn for object detection in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. -338, 2018. 7, 8, 18
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016. 8
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint 17X. 2017. 2
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338 2010. 8
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In International Conference on Learning Repre- sentations, 2021. URL https: h/tp::::////prreiiie//// n forumooruiiioo 6Tm1mpos LrM. LrM. 2, 4,
16
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015. 1, 2, 4,
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1):206--200 2016. 6, 7, 8, 17
Ian Gemp and Brian McWilliass The unreasonable effectiveness of adam on cycles. In NeurIPS Workshop Bridging Game Theory and Deep Learning, 2019. 4
10

---

Under review as a conference paper at ICLR 2022
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pp. 2232- 2241. PMLR. 2019. 4, 16
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: beyond sharp and flat local minima. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp.
2019. 2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016. 7,8
Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly- supervised object detection through progressive domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5001-5009, 2018. 8
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018. 19
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho and Krzysztof Geras". The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020. URL https:// openreview net/ wtre//foummmm ftttt/foommiddrrffoee rid=rgg774KKW
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In European Conference Computer Vision, pp. 464-480. Springer, 2020. 6, 7, 8, 17, 18
Bo Fu Junguang Jiang, Baixu Chen and Mingsheng Long. Transfer //gi wwt.hh......tttooomomeeeeeeeeeea thuml Learnnnng-
https:
2020. 6, 7, 18,
Minguk Kang and Jaesik Park. ContraGAN: Contrastive Learning for Conditional Image Genera- tion. 2020. 17
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv: 2017.
Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, 2017. 2
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXi::1412.6980, 2014.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 17
Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, Rahul M V, and R. Venkatesh Babu. Towards inheritable models for open-set domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020a. 2
Jogendra Nath Kundu, Naveen Venkat, Rahul M V, and R. Venkatesh Babu. Universal source- -free domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020b.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 2018.
11

---

Under review as a conference paper at ICLR 2022
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-ina -ineaage translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. 97-105,PMLR,, 2015. 7, 8
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference machine learning, pp. 2208 2208-2217.222 PMLR, 2017. 7
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1645-1655, 2018. 1,2,3, 4, 6, 7, 8, 17, 18
T. Miyato, S. Maeda, M. Koyama, and S. Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 41(8):1979-199; 2019. doi: 10. 109/TPAM1.20110 201...88811888888111 9, 19
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. 5, 17
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-8811 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http:: neurips httppppprr...rrrrrrppoopppeee//p an- ia-imperatives sevre-ssylee strssstyssseeee wwwpraaiiveesttlee-ii------eeeee www.ahaaeeeooaaaeeeeeeeeeee-eeee
wwtppp/pppppppppccc 901 pyttoo
wwtp---eeannnnn--raaaae add. 6
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.
Visda: The visual domain adaptation challenge, 2017. 6
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019. 6
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. arXiv preprint arXiv:2012.11466 2020. 2
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015. 8, 18
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regular- ization. In International Conference on Learning Representations, 2018a. 4
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 3723-3732, 2018b. 8, 17
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong-weak distribution align- ment for adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6956-6965, 2019. 1,
Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 19799)9973-99,, 2018. 8
12

---

Under review as a conference paper at ICLR 2022
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In International Conference on Learning Representations, 2018. 2
David Stutz, Matthias Hein, and Bernt Schiele. Relating adversarially robust generalization to flat minima. arXiv preprint arXiv:2104. 04448, 2021. 9, 19
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016. 9,
Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8725-8735, 2020. 7
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013. 1
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In (IEEE) Conference Computer Vision and Puttern Recognition (CVPR), 2017.6 6
Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable nor- malization: towards improving transferability of deep neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 1953-1963, 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2.
https: / ptttps://gitiii //ghtubb.... http///aacooooeerrthh.o. hetprccccdeeettttrre.o 2019. 18
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1426-1435, 2019.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data),
pp. 581-590. IEEE, 2020. 16
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation, In International Conference on Machine Learning, pp. 7404-7413. PMLR, 2019. 1, 7
13

---

Under review as a conference paper at ICLR 2022
APPENDICES
A NOTATION TABLE
Table SI contains all the notations used in the paper and the proofs of theorems.
Table S1: The notations used in the paper and the corresponding meaning.
Notation
Meaning
S
Labeled Source Data
T
Unlabelled Target Data
Ps (or Pr)
Source (or Target) Distribution
X
Input space
y
Label space
v-))
Maps image to labels
hp
Hypothesis function
(ha) (or (ha))
Source (or Target) risk
(ho) (or Rho))))
Empirical Source (or Target) risk
Pl
Hypothesis space
D H (PsllPr)
Discrepancy between two domains Ps and PT
Feature extractor
fe
Classifier
D+
Domain Discriminator
ds.r
Tractable Discrepancy Estimate
V R (ha) (or H)
Hessian of classification loss
Tr(H)
Trace of Hessian
Amaz
Maximum eigenvalue of Hessian Perturbation
p
Maximum norm of
B CONNECTION OF DISCREPANCY TO d&,r (EQ. 4) IN MAIN PAPER
We refer reader to Appendix C.2 of Acuna et al. (2021) for relation of d-.r- The d term defined Eq. 4 given as:
d Eew Logg
Be-r,, - (ox(())))
(S1)
The above term is exactly the Eq. C.I in Acuna et al. (2021) where they show that optimal S,T i.e.:
max d D (Pss(P)(PP)) -
- log(22
(S2)
Hence can say from result in Eq. 4 is consequence of Lemma I and Proposition in (Acuna et al., 2021), assuming that D3 satisfies the constraints in Proposition 1.
C PROOF OF THEOREMS
In this section provide proofs for the theoretical results present in the paper:
Theorem (Generalization bound). Suppose 1 : ) x y - 00 dom o- Let he be the ideal joint classifier with error X It's (h²) + R2y (h*). We have the following relation between source and target risk:
(ha) S Rt. (ha) + D ho, (Ps((PT) +
(S3)
Proof. We refer the reader to Theorem 2 in Appendix B of Acuna et al. (2021) for the detailed proof theorem.
We now introduce Lemma for smooth functions which will use in the proofs subsequently:
14

---

Under review as a conference paper at ICLR 2022
Lemma 1. For L-smooth function fw) the following holds where w is the optimal minima:
f(w) - f(w")((w"))))))))rr" 2 2L [V
Proof. The L-smooth function by definition satisfies the following:
I / =11e wil
f(w") S f(v) f(w) ff w) +
Now we minimize the upper bound wrt get a tight bound on f f(w").
L
D f(w) (f w)
110 w|
2
after doing V.D(v) = we get:
w V f(w)
By substituting the value of the upper bound we get:
1 1
fww)) < [(w)
[
2T.
Hence rearranging the above term gives the desired result:
1
f(w) - f(w") z 2T. [V
Theorem 2. For given classifier hg and one step of (steepest) gradient ascent i.e. 4' /VVd' and th = $ + "VVdV. V +++(0)]]) for maximizing
+
Rb S n - cos 2L(d'sr dz.T)
(S4)
where a is the angle between V and
Proof of Theorem 2. We assume that the function is L-smooth (the assumption of L-smoothness the basis of many results in non-convex optimization (Carmon et al., 2020)) in terms of input x. As for a fixed hg as we use reverse gradient procedure for measuring the discrepancy, only step analysis is shown. This is because only single step of gradient is used for estimating discrepancy dš,r i.e. one step of each min and max optimization is performed alternatively for optimization. After this the he is updated to decrease the discrepancy. Any differential function can be approximated by the linear approximation in case of small y/:
A+No so d + V T
(S5)
The dot product between two vectors can be written the following function of norms and angle between those:
vd T / |||||
cost
(S6)
V (x) The steepest value will be achieved when cos which is actually I|va (x)l
Now we
V rlutte) from the gradient descent. The II pletttll
compare the descent in another direction v2 difference in value can be characterized by:
dsT
0+ 0+112
1 cos a)
(S7)
As is an angle between V wl (v2) and T (o). The suboptimality dependent on the gradient magnitude. We use the following result to show that when optimality gap d'sr- -d3,(())) large the difference between two directions is also large.
15

---

Under review as a conference paper at ICLR 2022
For an L-smooth function the following holds according to Lemma 1:
f(w) - f(w=((())))))))))>))) z 2L [V](()]
As we are performing gradient ascent f(w) -ddgr- we get the following result:
(d's dg,T)) 2 2L ( (x)]l2
(d
n+nuz dg g,T
21(d'ST d 2 (n(1 COs )))
771 cos a) 2L(d's. ds,T) 2 (d d5.
This shows that difference in value of by taking a step in direction of gradient v vs taking the step in different direction 22 is upper bounded by the d'sr d (z), hence if we are far from minima
the difference can potentially large. As we are only doing one step of gradient ascent d dg will be potentially large, hence can lead to suboptimal measure of discrepancy.
Theorem 3. Suppose I is the loss function, we denote X == (lg(h") + (h"( and let h be the ideal joint hypothesis:
(ho) S max R's(ho+)) Do Dho (Psl + 7( 12 p²) +
(S8)
IK]|SP
where y IR R a strictly increasing function.
Proof of Theorem 3: In this case we make use of Theorem 2 in the paper sharpness aware mini- mization (Foret et al., 2021) which states the following: The source risk Rg(h) is bounded using the following PAC-Bayes generalization bound for any e with probability 1 1-6 6:
2
k klog
1
|01l
log(n)) k
+ 4 log z + Ō(1)
(S9)
Rs(ha)= S max Aho lellsp
T
here is the training set size used for calculation of empirical risk (s),), k is the number of parameters and ||0|| is the norm of the weight parameters. The second term in equation can be abbreviated 7(11012). Hence,
(hgh)) max (ho) + > (1101//pp)
(S10)
||<|S
From the generalization bound for domain adaptation for any f-divergence (Acuna al., 2021) (Theorem 2) we have the following result.
(ha) S R's + D (Ps Pr) + X
(S11)
Combining the above two inequalities gives us the required result we wanted to prove i.e.
(ho))))) S R2g(ho) + D (Psl[Pr) + (110)//p²)) x
(S12)
D HESSIAN ANALYSIS
We use the PyHessian library (Yao al., 2020) calculate the Hessian eigenvalues and the Hessian Eigen Spectral Density. All the calculations are performed using 50% of the source data at the last checkpoint. Only the source class loss is used for calculating to clearly illustrate our point. The partition was selected randomly, and the same partition was used across all the runs. We also made sure to use the same environment to run all the Hessian experiments. subset the data was used for Hessian calculation mainly because the hessian calculation is computationally expensive (Yao et al., 2020). This is commonly done hessian experiments. For example, (Chen et al., 2021) (refer Appendix D) uses 10% of training data for Hessian Eigenvalue calculation The PyHessian library uses Lanczos algorithm (Ghorbani et al., 2019) for calculating the Eigen Spectral density of the Hessian and uses the Hutchinson method calculate the trace of the Hessian efficiently.
16

---

Under review as a conference paper at ICLR 2022
Table S2: Architecture used for feature classifier and Domain classifier. C' is the number of classes. Both classifiers will take input from feature generator (ga).
Table S3: Accuracy (%) on VisDA- 2017 (ResNet-101).
Layer
Output Shape
Method
Synthetic - Real
Feature Classifier (fe)
DANN (Ganin ct al., 2016)
57.4
Bottleneck Dimension
MCD (Saito et al., 2018b)
71.4
Linear
CDAN (Long et al., 2018)
73.7
Domain Classifier (Dx)
CDAN
76.6
Bottleneck Dimension
CDAN w/ SDAT
78.3
Linear
1024
CDAN+MCC (Jin et al., 2020)
80.4
BatchNorm
1024
CDAN+MCC w/ SDAT
81.2
ReLUU
1024
Linear
1024
"Our implementation of CDAN. tefer Section F for more details.
BatchNorm
1024
ReLU
1024
Linear
E SMOOTHNESS OF DISCRIMINATOR IN SNGAN
We also did the similar experiment of smoothing discriminator in DAT (Sec. 4.1) for SNGAN Miyato et al. (2018) as the adversarial objective in GAN is similar to DAT. We use the same con- figuration for SNGAN as described in Pyr (Kang & Park, 2020) for both CIFARIO (Krizhevsky et al., 2009) and TinylmageNet with batch size of 256 in both cases. We then smooth the discriminator while discriminator trained by using the same formulation as in Eq. 9. We find that smoothing discriminator leads to higher (suboptimal) Fréchet Inception Distance in case of GANs as well, shown in Fig. 3.
F
EXPERIMENTAL DETAILS
F.1 IMAGE CLASSIFICATION
OfficeHHome: For CDAN methods, we train the models using mini-batch stochastic gradient descent (SGD) with batch size of 32 and learning rate of 0.01. The learning rate schedule is the same as (Ganin et al., 2016). We train for total of epochs with 1000 iterations per epoch. The momentum parameter in SGD is set to 0.9 and weight decay of 0.001 is used. For CDAN+MCC experiments, we use a temperature parameter (Jin et al., 2020) 2.5. The bottleneck dimension for the features is set to 2048.
VisDA-2017: We use a ResNet- 101 backbone initialized with ImageNet weights for VisDA-2017 experiments. Center Crop is also used as an augmentation during training. We use bottleneck dimension of 256 for both algorithms.
For CDAN runs, train the model for 30 epochs with optimizer setting that of Office- Home. For CDAN+MCC runs, we use temperature parameter of 3.0 and learning rate of 0.002. DomainNet: We use ResNet-101 backbone initialized with ImageNet weights for DomainNet experiments. We run all the experiments for 30 epochs with 2500 iterations per epoch. The other parameters are the same as that of Office-Home.
To show the effectiveness of SDAT fairly and promote reproducibility, we run with and without SDAT on the same GPU and environment and with the same seed. All the above experiments were run on Nvidia V100 and RTX 2080 GPUs. We used Wandb (Biewald, 2020) to track our experiments. We will be releasing the code to promote reproducible research.
F.1.1 ARCHITECTURE DOMAIN DISCRIMINATOR
One of the major reasons for increased accuracy in Office-Home baseline CDAN compared re- ported numbers in the paper is the architecture of domain classifier. The main difference is the use
17

---

Under review as a conference paper at ICLR 2022
Table S5: Accuracy(%) on DomainNet dataset for unsupervised domain adaptation (ResNet-101) across five distinct domains. The row indicates the source domain and the columns indicate the target domain.
ADDA
cip
inf
pnt
rel
skt
Avg
MCD
clp
ini
pnt
rel
skt
Avg
clp
11.2
24.1
41.9
30.7
27.0
clp
14.2
26.1
45.0
33.8
29.8
inf
19.1
16.4
26.9
14.6
19.2
23.6
21.2
36.7
18.0
24.9
pml
31.2
39.1
25.4
26.3
pnt
34.4
14.8
39.5
14.5
29.1
25.7
27.2
rel
42.6
19.6
42.6
29.3
33.5
ski
35.3
8.9
25.2
37.6
26.7
skt
41.2
13.7
27.6
34.8
29.3
Avg
31.3
11.0
23.7
36.4
24.1
25.3
Avg
35.4
15.6
29.4
41.7
27.4
29.9
CDAN
clp
inl
pnl
rel
ski
Avg
CDAN w/ SDAT
clp
ini
rel
skt
Avg
clp
20.6
38.9
56.0
44.9
40.1
clp
22.0
41.5
57.5
47.2
42.1
inf
31.5
29.3
43.6
26.3
32.7
inf
33.9
30.3
48.
27.9
35.0
pni
44.1
19.8
57.2
39.9
40.2
pnt
47.5
20.7
58.0
41.8
42.0
rel
55.8
24.4
53.2
42.3
43.9
rel
56.7
25.1
53.6
43.9
56.0
20.7
45.3
54.9
44.2
skt
58.7
21.8
Avg
46.9
21.4
41.7
52.9
38.3
40.2
Avg
49.2
22.4
43.4
55.2
40.2
42.1
of batch normalization layer in domain classifier, which was done in the library (Junguang Jiang & Long, 2020). Table S2 shows the architecture of the feature classifier and domain classifier.
F.2 ADDITIONAL IMPLEMENTATIONS DETAILS FOR DA FOR OBJECT DETECTION
In SDAT, we modified the loss function present in Chen et al. (2018) by adding classification loss smoothing, i.c. smoothing classification loss of RPN and ROI, used in Faster R-CNN (Ren al., 2015), by training with source data. Similarly. we applied smoothing to regression loss and found it be less effective. We implemented SDAT for object detection using Detectron2 (Wu al., 2019). We fixed to 0.15 for object detection experiments.
G ADDITIONAL RESULTS
VisDA-2017: Table S3 shows the overall accuracy on the VisDA-2017 with ResNet-101 backbone. The accuracy reported in this table is the overall accuracy of the dataset, whereas the accuracy re- ported in the Table 5 of the main paper refers to the mean of the accuracy across classes. CDAN w/ SDAT outperforms CDAN by 1.7%. showing the effectiveness of SDAT in large scale Synthetic + Real shifts. With CDAN+MCC as the backbone, adding SDAT improves the performance of the method to 81.2%.
DomainNet: Table S5 shows the results the proposed method on DomainNet across five domains. We compare results with ADDA and MCD and show that CDAN achieves much higher perfor- mance on DomainNet compared to other techniques. It can be seen that CDAN w/ SDAT further improves the overall accuracy DomainNet by 1.8%.
Results with DANN (Ganin & Lempitsky, 2015): Domain Adversarial neural networks introduced the concept of adversarial training in domain adaptation and seminal paper in the field of domain adaptation. Table $4 shows the results on some splits
OfficeHHone with DANN and DANN w/ SDAT. DANN w/ SDAT improves upon the performance on DANN specifically in challenging splits like Clipart Art where DANN w/ SDAT gets a 1% increase over DANN.
We have shown results with three different domain adaptation algorithms namely DANN (Ganin & Lempitsky, 2015), CDAN (Long et al., 2018) and CDAN+MCC (Jin et al., 2020). SDAT has shown to improve the performance of all the three DA methods. This shows that SDAT a generic method that can applied top of any domain adversaria training based method get better performance.
Table S4: Results on Office-Home dataset with DANN (Ganin Lempitsky, 2015). DANN w/ SDAT improves the performance over DANN across the four splits of Office-Home dataset showing the adaptability of the proposed method.
Method
Ar~CI
Cl-Pr
Rw-CI
Pr-Cl
Average
DANN (Ganin & Lempitsky, 2015)
52.6
65.4
60.4
52.3
57.7
DANN w/ SDAT
53.4
66.4
61.3
53.8
58.7
18

---

Under review as a conference paper at ICLR 2022
H DIFFERENT SMOOTHING TECHNIQUES
Stochastic Weight Averaging (SWA) (lzmailov et al., 2018): SWA is a widely popular technique to reach flatter minima. The idea behind SWA is that averaging weights across epochs leads to better generalization because it reaches a wider optima. 'he recently proposed SWA-Densely (SWAD) (Cha al., 2021) takes this step further and proposes average the weights across iterations instead of epochs. SWAD shows improved performance on domain generalization tasks. We average every 400 iterations in the SWA instead of averaging per epochs. We tried averaging across 800 iterations as well and the performance was comparable.
Virtual Adversarial Training (VAT) (Miyato al., 2019): VAT regularization technique which makes use of adversarial perturbations. Adversarial perturbations are created using Algo. present (Miyato et al., 2019). We added VAT by optimizing the following objective:
min En Ps
max DrL (ho + +r)]
(S13)
||rls
This value acts as negative measure of smoothness and minimizing this will make the model smooth. For training, set hyperparameters to 15.0, & to 1e-6, and as 0.1.
Label Smoothing (LS) (Szegedy et al., 2016): The idea behind label smoothing is to have a distri- bution outputs instead of one hot vectors. Assuming that there are k classes, the correct class gets a probability mention that label smoothing tends avoid sharper minima during training. We
a and the other classes gets a probability of a / (k-1). (Stutz et al., 2021)
smoothing
parameter (o) of 1 in all the experiments in Table S6. We also show results with smoothing param- eter of 0.2 and observe comparable performance. We observe that label smoothing slightly improves the performance over DAT.
Table S6: Different Smoothing techniques. We refer to (Stutz et al., 2021) to compare the proposed SDAT with other techniques show the efficacy of SDAT. It be seen that SDAT outperforms the other smoothing techniques significantly. Other smoothing techniques improve upon the perfor- mance of DAT showing that smoothing is indeed necessary for better adaptation.
Method
Ar-CI
Cl-Pr
Rw-CI
Pr-CI
DAT
54.3
69.5
60.1
55.3
VAT
54.6
70.7
60.8
54.4
SWAD-400
54.6
71.0
60.9
55.2
LS (a = 0.1)
53.6
71.6
59.9
53.4
LS (a = 0.2)
53.5
71.2
60.5
53.2
SDAT
55.9
73.2
61.4
55.9
OPTIMUM RHO VALUE
Table S7 and S8 show that p = 0.02 works robustly across experiments providing an increase in performance (although it does not achieve the best result each time) and can be used as a rule of thumb.
Table S7: P value for DomainNet
Split
DAT
SDAT 0.02) SDAT Reported (p 0.05)
clp-skt
44.9
46.7
47.2
skt-clp
56.0
59.0
58.7
skt-pnt
45.3
47.8
48.1
inf-rel
43.6
47.3
48.1
Table S8: value for VisDA-2017 Synthetic - Real
Backbone
DAT
SDAT (p= 0.02)
SDAT Reported( = 0.005)
CDAN
76.6
78.2
78.3
CDAN+MCC
80.4
80.9
81.2
19

---

Under review as a conference paper at ICLR 2022
Clipart- Painting
Clipart-+ Sketch
Infograph- Clipart
46
Validation Accuracy (in
Accuracy
32
44
30
28
Validation
40
26
34
24
32
4348 i SEAT
CONY
38
(DAM vt SOAT
C24A
22
CDAN SOAT CDAM
10
20
30
10
20
30
0
10
20
30
Epoch
Epoch
Epoch
Painting- Clipart
Real- Sketch
Sketch- Real
Validation Accuracy (in %0
44
Valldation Accuracy %0
46
56
44
Validation Accuracy
54
42
40
52
38
50
-38
36
48
96
34
26
46
SUATT
Epoch
Epoch
Epoch
Figure S1: Validation Accuracy across epochs on different splits DomainNet. We run on three different random seeds and plot the error bar indicating standard deviation across runs. CDAN w/ SDAT consistently outperforms CDAN across different splits of DomainNet.
J SIGNIFICANCE AND STABILITY OF EMPIRICAL RESULTS
To establish the empirical results' soundness and reliability, we run a subset of experiments (rep- resentative of each different source domain) on DomainNet. The experiments are repeated with three different random seeds leading to overall 36 experimental runs (18 for CDAN w/ SDAT (Our proposed method) and 18 for CDAN baseline). Due to the large computational complexity of each experiment (R20 hrs each), we have presented results for multiple trials on subset of splits. We find (in Table S9) that our method can outperform the baseline average each of tablishing significant improvement across all splits. However, DomainNet. the average increase (across three different trials) is close to the reported increase in all cases (Table $9), which also serves evidence of the soundness of reported results (for remaining splits). We also present additional statistics below for establishing soundness.
cases, es- found that due to the large size of
If the proposed method is unstable, there is large variance in the validation accuracy across epochs. For analyzing the stability of SDAT, we show the validation accuracy plots Figure S1 on six different splits of Nem We find that our proposed SDAT improves over baselines consistently across epochs without overlap in confidence intervals in later epochs. This also provides evidence for the authenticity and stability of our results. We also find that in some cases, like when using the Infographic domain as source, our proposed SDAT also significantly stabilizes the training (Figure S1 Infographic Clipart).
One of the other ways of reporting results reliably proposed by the concurrent work (Berthelot et al., 2021) (Section 4.4) involves reporting the median of accuracy across the last few checkpoints. The median is measure of central tendency which ignores outlier results. We also report the median of validation accuracy for our method across all splits for the last five epochs. observed that we observe similar gains for median accuracy (in Table S10) as reported in Table 2.
20

---

Under review as a conference paper at ICLR 2022
Table S9: DomainNet experiments over 3 different seeds. We report the mean, standard deviation,
reported increase and average increase in the accuracy (in %6).
Split
CDAN
CDAN w/ SDAT Reported Increase (Table 2)
Average Increase
clp-pnt
38.9 + 0.1
41.5 + 0.3
+2.6
+2.6
skt-rel
55.1
57.1 + 0.1
+2.2
+2.0
pnt-clp
4.3
47.1 + 0.3
+3,4
+2.6
rel:skt
0.4
43,9 0.1
+1.6
+1.5
clp-skt
0.2
47.3
+2.3
+2.4
inf-clp
31.4 0.5
34.2 +: 0.3
+2.3
+2.7
Table S10:
Median accuracy of last 5 epochs DomainNet dataset with CDAN w/ SDAT. The
number in the parenthesis indicates the increase in accuracy with respect to CDAN.
Target (-)
clp
inf
pnt
real
skt
Avg
Source ())
21.9
41.6
56.5
46.4
41.6
clp
(+1.7)
(+3.0)
(+1.3)
(+2.0)
(+2.0)
32.4
29.8
46.7
25.6
33.6
inf
(+7.9)
(+7.0)
(+12.7)
(+5.4)
(+8.2)
47.2
21.0
57.6
41.5
41,8
pnt
(+2.9)
(+1.1)
(+1.0)
(+2.4)
(+1.8)
56.5
25.5
53.9
43.5
44.8
real
(+0.7)
(+0.9)
(+0.5)
(+1.3)
(+0.8)
59.1
22.1
48.2
56.6
46.5
skt
(+3.0)
(+1.7)
(+3.1)
(+2.9)
(+2.7)
48.8
22.6
43.4
54.3
39.2
41.7
Avg
(+3.6)
(+1.3)
(43.4)
(+4.5)
(+2.8)
(+3.1)
As the Office-Home dataset is smaller (i.e., 44 images per class) in comparison to DomainNet find that there exists some variance in baseline CDAN results (This also reported in the well- known benchmark for DA (Junguang Jiang & Long, 2020)). For establishing the empirical sound- ness, we report results of 4 different dataset splits on three different random seeds. It can be seen in Table $11 that even though there is variance in baseline results, our combination of CDAN w/ SDAT can produce consistent improvement across different random seeds. This further establishes the empirical soundness of our procedure.
Table SII: Office- Home experiments over 3 different seeds. We report the mean, standard deviation, reported increase and average increase in the accuracy (in %)..
Split
CDAN
CDAN w/ SDAT Reported Increase (Table 1)
Average Increase
Ar-CI
53.9 = 0.2
55.5 + 0.2
+1.7
+1.6
Ar-Pr
70.6 : 0.4
72.1 + 0.4
+1.6
+1.5
Rw:CI
60.7 = 0.5
61.8 + 0.4
+1.3
+1.1
Pr:Cl
54.7 0.4
55.5 0.4
+0.6
+0.8
21


Reviews:
[{"review_id": "bAXv5WYu5r", "reviewer": "Reviewer_C1yh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n  * Theoretical foundation for smoothness\n  * Results sections include improvements on both image classification and object detection. \n  * Ablation experiments provide insights to the effect of a smoothness parameter. \n\nWeaknesses:\n\n  * Figure 4: not clear why and how the optimum for rho differs per dataset. Does this depend on the dataset size or diversity of objects? Moreover, the vanilla ResNets have a batchnorm layer, so it is not clear why figure 4a has such a sharp optimum at all. \n* What is the final recommendation of the paper? If the paper recommends using smoothness penalties, then what are the limits? For example, how does the smoothness of the classifier (focus of this study) interplay with smoothness of the domain discriminator? I could imagine that when the discriminator has sharp decision boundaries, then enforcing smoothness on the classifier would be less useful. (But in practise people employ many tricks to get a smooth discriminator [1][2]).\n* Table 1: most improvements occur when combining SDAT with MCC. This combination is not well explained in the paper. What makes these approaches complementary and why are improvements small (or non-existent) for vanilla CDAN?\n* Theorem 2: I don’t see how this theorem relates to the point of the paper. From equation 11 it seems smoothness is applied to h_\\theta. \n\n[1] Gulrajani et al. \"Improved training of wasserstein gans.\" arXiv 2017.\n\n[2] Arjovsky et al. \"Towards principled methods for training generative adversarial networks.\" arXiv 2017.\n\n[3] Miyato et al. \"Spectral normalization for generative adversarial networks.\" arXiv 2018.\n", "summary_of_the_paper": "This paper introduces a smoothness penalty for domain adversarial training. The penalty encourages smoothness on the parameter space of a discriminative model. This smoothness penalty is motivated by results in Sharpness Aware Minimization. Results are included on the popular DA datasets such as Office-home and VisDA, where superior results are shown. Ablation experiments show how different smoothness penalties change the observed improvements. ", "summary_of_the_review": "Paper with theoretical motivation and results on both classification and object detection. However, the limitations of the proposed method are not clear. When would smoothness not be useful and how do the improvements depend on the other tricks for training Domain Adaptation models. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VOrOOdMyoF2", "reviewer": "Reviewer_nwci", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n- The paper is very well-written and introduces all used concepts appropriately. This makes the paper easy to understand even for readers that are not familiar with domain adaption.\n- The motivational experiments performed in Section 4 (Analysis of Smoothness) are well-explained and convincing.\n- The author(s) provide(s) a generalization bound (Theorem 2) for their proposed method which justifies it from a theoretical perspective.\n- The experimental evaluation is very extensive: the method is evaluated on three different domain adaption datasets for classification (Office-Home, VisDA-2017, and DomainNet) and compared with different existing domain adaption methods. It is shown that SDAT can be combined with the existing methods CDAN and CDAN + MCC to increase generalization performance.\n- Furthermore, improved performance on domain adaption datasets for object detection is reported.\n- Additional experiments show that SDAT also increases the robustness to label noise and that SAM outperforms other smoothing techniques. \n\nWEAKNESSES:\n- I see not major issues with this submission.\n\nMINOR REMAKS:\n- Some references are not correctly capitalized (e.g., \"Lower bounds for finding stationary points i\" or \"Faster r-cnn\") also conference names are sometimes capitalized and sometimes not.", "summary_of_the_paper": "The paper analyzes the role of smoothness in domain adversarial training (DAT). The main insight of the paper is that smoother minima of the classification loss improve generalization in the target domain. This explains why SGD is usually preferred over Adam when optimizing the training objective of DAT. To further improve the generalization performance by leveraging this phenomenon the author(s) introduce(s) smooth domain adversarial training (SDAT) for classification and object detection. SDAT applies sharpness aware minimization (SAM) to find smoother minimia of the classification loss used during DAT.", "summary_of_the_review": "Overall, I would highly recommend to accept this submission to ICLR 2022. The theoretical and empirical results are convincing and the paper is of high quality.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FVwisolwsz", "reviewer": "Reviewer_zPwA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points\n\n1. The paper takes issue with an interesting point of the reason for SGD optimizer outperforming Adam on DA tasks for reaching a smoother minima, develops it theoretically, and proposed a novel loss function focusing on smoothing classifier loss to improve generalization on target domain.\n\n2. It fully discussed why we should not apply the smoothing method **on the discrepancy term**, and verify the theory empirically through experiment, further developing smooth theory with regard to **adversarial objectives.**\n\n## Weak points\n\n1. The method for smoothing ERM is more directly adopted from SAM rather than a novel idea, and the relation **between** smoothness **and** better generalization has also been discussed by previous papers (SWAD), therefore renders this paper not insightful enough. \n\n2. The proposed smoothing method w.r.t discrepancy term tend to \"lead to a suboptimal solution\", therefore not significantly helpful when tackling DA tasks (and I think it is hard to interpret). What actually makes such difference on the finally result is still, smoothness on ERM term.", "summary_of_the_paper": "The paper discussed a correlation of smoothness on **classification loss** and **generalization ability on target domain** and further develop a method to enhance such smoothness to achieve better performance on DA tasks. To do so, it adopted the losses based on Sharpness Aware Minimization, a minmax game of finding a smoother neighborhood of $\\theta$ , by computing first order deviation. Empirical studies are implemented to verify the theory and show the soundness of the proposed method.", "summary_of_the_review": "Overall, the author has well established a positive correlation between smoothness of classification loss and generalization capability on target domain, and a negative correlation between adversarial counterparts. The method of acquiring a flat minima also brings promising results on multiple DA classification and detection datasets. However, as I can see the experimental studies of the proposed smoothing method are well-designed and theories are rigorously proved, I am not completely convinced by the novelty of the work. Hope the author can make further analysis and explanation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AsLmszM_d8k", "reviewer": "Reviewer_rxW2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this work is to propose applying Sharpness Aware Minimization (SAM) to smooth only the task loss term of domain adversarial training. From an empirical standpoint the paper is interesting and points to potentially relevant investigation directions, however, I have concerns regarding the significance of this study since there is no clear motivation as to why previous findings on the iid setting showing that smoother minima generalize better would transfer to the unsupervised domain adaptation setting. Moreover, the empirical validation of the proposed approach is not extensive enough to provide sufficient evidence for the soundness of the reported findings. In the following, I provide more details about my concerns along with questions and suggestions.\n\n- Motivation:\n  - Although the authors (to some extent) empirically show that SDAT can improve the performance on the target domain on the considered cases, there is no justification on why one would expect better out-of-distribution generalization for smoother minimum. Would the authors clarify that? More specifically, what is the motivation/context for investigating Conjecture 1? As of now, it seems to me that this work is extending an analysis for the iid setting to unsupervised domain adaptation without a clear reason. \n  - It is not clear to me why the authors decided to provide results that rely on Acuna et al. 2021 rather than consider the classic results from Ben-David et al. 2010. There is no clear motivation for this choice in the manuscript and I believe the authors should better justify this choice.\n\n\n- Results:\n  - The relevance of both theoretical results are not clear to me. Theorem 2, for example, requires unrealistic assumptions such as L-smoothness. Theorem 3, in turn, states a bound that is looser than the known results in the literature and which are already vacuous, so it is not clear to me what is the relevance of this result in the context of unsupervised domain adaptation.\n  - It is hard to assess the relevance of the reported results as no measures of statistical significance were reported. More specifically, the authors should include in the paper information regarding the number of runs considered for each experiment and report at least the average and standard deviation for all the quantities reported. Otherwise, I don’t think it is possible to rely on the current experimental set-up to support the claims of this contribution.\n  - Please clarify how the 50% partition of the source data was selected to estimate the Hessian for the three compared methods and if the same partition was used for all, otherwise results on Figure 3 could be noisy.\n  - SDAT requires an extra gradient step in order to estimate the smoothing penalty in Eq. 7. How does this affect the computational cost of SDAT and how SDAT stands in comparison with the considered baselines?\n\n\n- Other concerns/questions:\n  - Some statements of the manuscript are not rigorous and require rephrasing:\n     - Abstract: “combination of classification and adversarial terms”. In the considered setting, the adversarial term of the loss also stems from a classification task (domain discrimination), therefore, it is confusing to the denominate both component losses in such a way. I suggest referring to the loss term related to classifying labels regarding a specific task as task loss.\n     - Section 3.1: please specify what “works well on the dataset” means (i.e. low risk on the target distribution).\n     - Theorems 1/3: please precisely define what is the ideal classifier $h^*$.\n     - Section 4: “where we find that in contrast to supervised learning” notice that, in practice, unsupervised domain adaptation also corresponds to supervised learning since all losses require either task or domain labels, so it shouldn’t be referred to as a setting opposed to supervised learning.\n  - Table 1 caption: what exactly “sophisticated” means here? \n\n\n\n- Related work:\n  - Foundational literature on domain adaptation is missing in the related work section, for example, Ben-David et al. (2007) and Long et al. (2016). \n\nBen-David et al., Analysis of representations for domain adaptation, 2007.\nLong et al., Unsupervised domain adaptation with residual transfer networks, 2016.\n", "summary_of_the_paper": "This work studies the loss landscape of domain adversarial neural networks for domain adaptation. The authors claim that similarly to the iid setting, smooth minima might yield better generalization on the unsupervised domain adaptation framework via domain adversarial training but in case smoothness is enforced only with respect to the task loss. Experiments on the OfficeHome dataset indicated this hypothesis might be true. The authors then proposed to apply a previously proposed approach to enforce smoothness (Sharpness Aware Minimization), so that only smoothness with respect to the task loss would be enforced. Moreover, a theorem showing that the non-smooth domain discrepancy estimation is better than the smoother counterpart was introduced, along with a generalization bound for the risk on the target domain in terms of the smooth risk on the source domain. Finally, the authors showed that the proposed approach, SDAT, improved the performance on target domains of three domain adaptation datasets in comparison to the considered baselines. Further experiments showed that SDAT performed better than other smoothing techniques, as well as yields improved robustness to label noise in comparison to not enforcing smooth minima. ", "summary_of_the_review": "This work studies the loss landscape of adversarial domain adaptation in terms of the smoothness of the minima. Despite showing somewhat empirically promising results, I found this work lacks a solid motivation for the presented analysis (i.e. why should we expect that strategies that improve generalization in the iid case would help in a non-iid setting as well?) and a sound empirical validation of the proposed approach. Moreover, the relevance of the presented generalization bound is also unclear to me at this point. All in all, I think this contribution seems promising but due to the concerns I raised in my review, I believe it is not yet ready for publication. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the loss landscape of domain adversarial neural networks for domain adaptation. First, the authors show that smooth minima with respect to adversarial loss leads to sub-optimal generalization on the target domain. Then, they suggest to enforce smoothness only with respect to the task loss. 3 reviewers are on negative sides, and 1 reviewer is on a positive side. All negative reviewers interacted with authors in the discussion period. After the discussion period, even the positive reviewer agreed negative comments of other reviewers and declined to champion this paper.  

AC thinks that this is a borderline paper; the proposed claims (both theoretic and empirical) are interesting and there is no critical weakness of this paper. However, the results are not super excited, as evidenced by 3 negative reviewers. In particular, AC agrees with negative comments of reviewers on limited novelty and lack of strong theoretical motivation for the proposed scheme. AC also thinks the performance improvements in experiments are not that significant. Furthermore, the problem scope is narrow, i.e., the authors study a certain property (smoothness) of a special algorithm (adversarial training) for domain adaptation that is a particular way for domain generalization. Hence, the impact to the community can be not significant as well. Considering all aspects, AC is a bit toward to suggesting rejection.
Decision: Reject