Year: 2021

Abstract:
Domain adversarial training has been ubiquitous for achieving invariant representations and is used widely for various domain adaptation tasks. In recent times, methods converging to smooth optima have shown improved generalization for supervised learning tasks like classification. In this work, we analyze the effect of smoothness enhancing formulations on domain adversarial training, the objective of which is a combination of task loss (eg. classification, regression etc.) and adversarial terms. In contrast to task loss, our analysis shows that converging to smooth minima w.r.t. adversarial loss leads to sub-optimal generalization on the target domain. Based on the analysis, we introduce the Smooth Domain Adversarial training (SDAT) procedure, which effectively enhances the performance of existing domain adversarial methods for both classification and object detection tasks. Our smoothness analysis also provides insight into the extensive usage of SGD over Adam in domain adversarial training.

heading: 1 INTRODUCTION
text: Domain Adversarial Training (Ganin & Lempitsky, 2015) (DAT) refers to adversarial learning of neural network based feature representations that are invariant to the domain. For example, car images from the clipart domain have similar feature representations as car images from the web domain. DAT has been widely useful in diverse areas (cited 3540 times) such as fairness (Adel et al., 2019), object detection (Saito et al., 2019), domain generalization (Li et al., 2018), imageto-image translation (Liu et al., 2017) etc. The prime driver of research on DAT is its application in unsupervised Domain Adaptation (DA), which aims to learn a classifier using labeled source data and unlabeled target data, such that it generalizes well on target data. Various enhancements like superior objectives (Acuna et al., 2021; Zhang et al., 2019), architectures (Long et al., 2018) etc. have been proposed to improve its effectiveness. However, as DAT objective is combination of Generative Adversarial Network (GAN) (Goodfellow et al., 2014) and Empirical Risk Minimization (ERM) (Vapnik, 2013) objectives, there has not been much focus on explicitly analyzing the nature of optimization in DAT. One direction of work aiming to improve generalization of ERM on unseen data focuses on developing algorithms that converge to a smooth (or a flat) minima (Foret et al., 2021; Keskar & Socher, 2017). However, we find that these techniques, when directly applied for DAT, do not significantly improve the generalization on the target domain (Sec. 4 and 7). In this work, we analyze the loss landscape near the optimal point obtained by DAT, to gain insights into curvature. We first focus on the eigen-spectrum of Hessian of the task loss (ERM term for classification) where we find that using Stochastic Gradient Descent (SGD) as optimizer converges to a smoother minima in comparison to Adam (Kingma & Ba, 2014). Further we find that smoother minima w.r.t.task loss leads to better generalization on the target domain. Contrary to task loss, we find that smoothness enhancing formulation for adversarial components worsen performance, rendering ERM-based techniques which enhance smoothness for all loss components ineffective. Hence we introduce Smooth Domain Adversarial Training (SDAT), which aims only to reach a smooth minima w.r.t. task loss, and helps in generalizing better on the target domain. SDAT requires an additional gradient computation step and can be combined with existing methods with a few lines of code. We show the soundness of the SDAT method theoretically by proving a generalization bound (Sec. 4) on target error. We extensively verify the empirical efficacy of SDAT across various datasets for classification (i.e., DomainNet, VisDA-2017 and Office-Home), along with showing a prototypical application in DA for object detection, demonstrating it’s diverse applicability. In summary, we make the following contributions:
• We analyze the optimization procedure of DAT, establishing the correlation between the smoothness near optima w.r.t. task loss and generalization on the target domain.
• Contrary to ERM, we show through our theoretical and empirical analysis that smoothness enhancing adversarial formulation leads to sub-optimal performance.
• For enhancing the smoothness w.r.t. task loss near optima in DAT, we propose a novel and theoretically motivated SDAT that improves the generalization on the target domain. SDAT effectively increases the average performance of even state-of-the-art adversarial adaptation methods.

heading: 2 RELATED WORK
text: Unsupervised Domain Adaptation: It refers to a class of methods that aim to adapt models to work in a target domain distinct from what it was trained on. One of the most prominent lines of work is based on DAT (Ganin & Lempitsky, 2015). This involves using an additional discriminator to distinguish between samples of source and target domain. The goal of the model is to learn features that can not be distinguished between source and target. The follow-up works have improved this basic idea by introducing a class information based discriminator (CDAN (Long et al., 2018)), introducing a transferable normalization function (Wang et al., 2019) etc. In this work, we focus on analyzing and improving such methods. Another line of work involves DA by using self-training on target domain (Kundu et al., 2020b;a; Prabhu et al., 2020) which will not be the focus of this work.
Smoothness of Loss Landscape: As neural networks operate in the regime of over parameterized models, low error on training data does not always lead to better generalization (Keskar et al., 2017). Often it has been stated (He et al., 2019; Dziugaite & Roy, 2017) that smoother minima does generalize better on unseen data. But until recently, this was practically expensive as smoothing required additional costly computations. Recently, a method called Sharpness Aware Minimization (SAM) (Foret et al., 2021) has been proposed to find a smoother minima with an additional gradient computation step. SAM also improves the ImageNet model performance (Chen et al., 2021) on ImageNet-C and ImageNet-R (which are out of distribution). It has also been observed that smoothness w.r.t. input (image) is beneficial for domain adaptation (Shu et al., 2018; Cai et al., 2021), which motivates us to explore smoothness w.r.t weights (W) in case of DAT. However, the earlier work has focused on achieving a smoother minima w.r.t. W for ERM. Currently, no study has been done if the loss function is composed of both ERM and adversarial objectives (as present in DAT).

heading: 3 BACKGROUND

heading: 3.1 PRELIMINARIES
text: We will primarily focus on Unsupervised DA where we have labeled source data S = {(xsi , ysi )} and unlabeled target data T = {(xti)}. The source samples are assumed to be sampled i.i.d. from source
1Figures for the smooth minima and sharp minima are from (Foret et al., 2021) and used for illustration purposes only.
distribution PS defined on input space X , similarly target samples are sampled i.i.d. from PT . Y is used for denoting the label set which is {1, 2, . . . , k} in our case as we perform multiclass (k) classification. We denote y : X → Y a mapping from images to labels. Our task is to find a hypothesis function hθ that has a low risk on the target distribution. The source risk (a.k.a expected error) of the hypothesis hθ is defined with respect to loss function l as: RlS(hθ) = Ex∼PS [l(hθ(x), y(x))]. The target risk RlT (hθ) is defined analogously. The empirical versions of source and target risk will be denoted by R̂lS(hθ) and R̂ l T (hθ). All notations used in paper are summarized in Table F. In this work we build on the domain adaption theory of (Acuna et al., 2021) which is a generalization of Ben-David et al. (2010). We first define the discrepancy between the two domains.
Definition 3.1 (Dφhθ,H discrepancy). The discrepancy between two domains PS and PT is defined as following:
Dφhθ,H(PS ||PT ) := sup h′∈H [Ex∼PS [l(hθ(x), h′(x))]]− [Ex∼PT [φ∗(l(hθ(x), h′(x)))]] (1)
Here φ∗ is a frenchel conjugate of a lower semi-continuous convex function φ that satisfies φ(1) = 0, andH is the set of all possible hypothesis (i.e. Hypothesis Space).
This discrepancy distance Dφhθ,H is based on variational formulation of f-divergence (Nguyen et al., 2010) for the convex function φ. TheDφhθ,H is the lower bound estimate of the f-divergence function Dφ(PS ||PT ). See Lemma 4 in (Acuna et al., 2021) for additional details. We state a bound on target risk RlT (hθ) based on D φ hθ,H discrepancy (Acuna et al., 2021): Theorem 1 (Generalization bound). Suppose l : Y × Y → [0, 1] ⊂ dom φ∗. Let h∗ be the ideal joint classifier with least λ∗ = RlS(h ∗) + RlT (h ∗) (i.e. joint risk) in H. We have the following relation between source and target risk:
RlT (hθ) ≤ RlS(hθ) +D φ hθ,H(PS ||PT ) + λ ∗ (2)
The above generalization bound shows that the target risk RlT (hθ) is upper bounded by the source risk RlS(hθ) and the discrepancy term D φ hθ,H along with an irreducible constant error λ
∗. Hence, this infers that reducing source risk and discrepancy lead a to reduction in target risk. Based on this, we concretely define the unsupervised adversarial adaptation procedure in the next section.

heading: 3.2 UNSUPERVISED DOMAIN ADAPTATION
text: In this section we first define the components of the framework we use for our purpose: hθ = fΘ◦gψ where gψ is the feature extractor and fΘ is the classifier. The domain discriminator DΦ, used for estimating the discrepancy between PS and PT is a classifier whose goal is to distinguish between the features of two domains. For minimizing the target risk (Th. 1), the optimization problem can be written as:
min θ
Ex∼PS [l(hθ(x), y(x))] +D φ hθ,H(PS ||PT ) (3)
The discrepancy term under some assumptions (refer App. B) can be upper bounded by a tractable term:
Dφhθ,H(PS ||PT ) ≤ maxΦ d Φ S,T (4)
where dΦS,T = Ex∼PS [log(DΦ(gψ(x)))] + Ex∼PT log[1−DΦ(gψ(x))]. This leads to the final optimization objective of:
min θ max Φ
Ex∼PS [l(hθ(x), y(x))] + dΦS,T (5)
The first term in practice is empirically approximated by using finite samples R̂lS(hθ) and used as task loss (classification) for minimization. The empirical estimate of the second term is adversarial loss which is optimized using a gradient reversal layer (GRL) as it has a min-max form. (Overview in Fig. 1) The above procedure composes DAT, and we use CDAN (Long et al., 2018) as our default DAT method.

heading: 4 ANALYSIS OF SMOOTHNESS
text: In this section, we analyze the curvature properties of the loss with respect to the parameters. Specifically, we focus on analyzing the Hessian of empirical source risk H = ∇2θR̂lS(hθ) which is the Hessian of classification (task) loss term. For quantifying the smoothness, we measure the trace Tr(H) and maximum eigenvalue of Hessian (λmax) as a proxy for quantifying smoothness. This is motivated by analysis of which states that the high value of λmax and Tr(H) are indicative of low smoothness (Jastrzebski et al., 2020). We articulate our conjecture informally below:
Conjecture 1. Smoothing of empirical source risk (i.e. task loss) R̂lS(hθ) leads to efficient DAT. In other words, decreasing λmax of ∇2θR̂lS(hθ) leads to reduced error on target domain R̂lT (hθ).
For verifying our conjecture, we analyze the eigen spectrum of the Hessian R̂lT (hθ) where we find that in contrast to standard ERM (Ghorbani et al., 2019) the negative eigenvalues do not disappear as the training progresses. We show the λmax, Tr(H) and eigen spectrum for different algorithms, namely DAT w/ Adam, DAT w/ SGD and our proposed SDAT (which is described in detail in later sections) in Fig. 2. We find that high smoothness leads to better generalization on the target domain. We also provide additional results in Fig. 3 for empirical verification of the conjecture. Our conjecture also explains the reason for widespread usage of SGD for DAT as SGD converges to smoother minima (Ganin & Lempitsky, 2015; Long et al., 2018; Saito et al., 2018a) which leads to efficient DAT, even though Adam has shown to be effective for min-max optimization (Gemp & McWilliams, 2019). More details regarding the Hessian analysis are provided in App. D.

heading: 4.1 SMOOTHING LOSS LANDSCAPE
text: In this section we first introduce the losses which are based on Sharpness Aware Minimization (Foret et al., 2021) (SAM). The basic idea of SAM is to find a smoother minima (i.e. low loss in neighborhood of θ) by using the following objective given formally below:
min θ max || ||≤ρ Lobj(θ + ) (6)
Here Lobj is any objective function to be minimized and ρ ≥ 0 is a hyperparameter which defines the maximum norm of the . Since finding the exact solution of inner maximization is hard, SAM maximizes the first order approximation:
̂(θ) ≈ arg max || ||≤ρ Lobj(θ) + T∇θLobj(θ) = ρ∇θLobj(θ)/||∇θLobj(θ)||2 (7)
The ̂(θ) is added to the weights θ. The gradient update for θ is then computed as∇θLobj(θ)|θ+̂(θ). The above procedure can be seen as a generic smoothness enhancing formulation for any Lobj . We now analogously introduce the sharpness aware source risk for finding a smooth minima:
max || ||≤ρ RlS(hθ+ ) = max|| ||≤ρ Ex∼PS [ l(hθ+ (x), f(x))] (8)
We also now define the sharpness aware discrepancy estimation objective below:
max Φ min || ||≤ρ
dΦ+ S,T (9)
As dΦS,T is to be maximized the sharpness aware objective will have min|| ||≤ρ instead of max || ||≤ρ , as it needs to find smoother maxima. We now theoretically analyse the difference in discrepancy estimation for smooth version dΦ ′′
S,T (Eq. 9) in comparison to non-smooth version d Φ′
S,T (Eq. 4). Assuming DΦ is a L-smooth (common assumption for non-convex optimization (Carmon et al., 2020)), η is a small constant and d∗S,T the optimal discrepancy, the theorem states:
Theorem 2. For a given classifier hθ and one step of (steepest) gradient ascent i.e. Φ′ = Φ + η(∇dΦS,T /||∇dΦS,T ||) and Φ′′ = Φ + η(∇dΦS,T |Φ+̂(Φ)/||∇dΦS,T |Φ+̂(Φ)||)
dΦ ′ S,T − dΦ ′′ S,T ≤ η(1− cosα) √ 2L(d∗S,T − dΦS,T ) (10)
where α is the angle between∇dΦS,T and∇dΦS,T |Φ+̂(Φ).
The dΦ ′ S,T (non-smooth version) can exceed d Φ′′
S,T (smooth discrepancy) significantly, as the term d∗S,T −dΦS,T 6→ 0, as the hθ objective is to oppose the convergence of dΦS,T to optima d∗S,T (min-max training in Eq. 11). Thus dΦ ′
S,T can be a better estimate of discrepancy in comparison to d Φ′′
S,T . A better estimate of dΦs,t helps in effectively reducing the discrepancy between PS and PT , hence leads to reduced RlT (hθ). This is also observed in practice that smoothing the discriminator (SDAT w/ adv in Fig. 3) leads to low domain classification accuracy (proxy measure for dΦs,t) in comparison to DAT. Due to ineffective discrepancy estimation, SDAT w/ adv results in sub-optimal generalization on target domain i.e. high target error RlT (hθ) (Fig. 3). For further establishing the generality of sub-optimality of smooth adversarial loss, we also perform experiments on Spectral Normalised Generative Adversarial Networks (SNGAN) (Miyato et al., 2018). In case of SNGAN we also find that smoothing discriminator through SAM leads to suboptimal performance (higher FID) as in Fig. 3. The above evidences indicates that smoothing the adversarial loss leads to sub-optimality, hence it should not be done in practice. The proof of the above theorem and additional experimental details are provided in the supplementary (refer App. C and App. E).

heading: 4.2 SMOOTH DOMAIN ADVERSARIAL TRAINING (SDAT)
text: We propose smooth domain adversarial training which only focuses on converging to smooth minima w.r.t. task loss (i.e. empirical source risk), whereas does no change for the discrepancy term. We define the optimization objective of our smooth domain adversarial training below:
min θ max Φ max || ||≤ρ
Ex∼PS [l(hθ+ (x), y(x))] + dΦS,T (11)
The first term is the sharpness aware risk, and the second term is the discrepancy term which is not smooth in our procedure. The term dΦS,T estimatesD φ hθ,H
(PS ||PT ) discrepancy. We empirically find that this optimization procedure effectively reduces the generalization error on the target domain compared to all other alternatives. We now show that optimizing Eq. 11 reduces RlT (hθ) through a generalization bound. This bound establishes that our procedure is also consistent (i.e. in case of infinite data the upper bound is tight), similar to the DAT (Ganin et al., 2016) baseline.
Theorem 3. Suppose l is the loss function, we denote λ∗ := RlS(h∗) + RlT (h∗) and let h∗ be the ideal joint hypothesis:
RlT (hθ) ≤ max|| ||≤ρ R̂lS(hθ+ ) +D φ hθ,H (PS ||PT ) + γ(||θ||22/ρ2) + λ∗. (12)
where γ : R+ → R+ is a strictly increasing function.
The bound is similar to generalization bounds for domain adaptation (Ben-David et al., 2010; Acuna et al., 2021). The main difference is the sharpness aware risk term max|| ||≤ρ R̂lS(hθ) in place of source risk RlS(hθ), and an additional term that depends on the norm of the weights γ(||θ||22/ρ2). The first is minimized by decreasing the empirical sharpness aware source risk by using SAM loss shown in Sec. 4. The second term is reduced by decreasing the discrepancy between source and target domains. The third term, as it is a function of norm of weights ||θ||22, can be reduced by using either L2 regularization or weight decay. Since we assume that the H hypothesis class we have is rich, the λ∗ term is small. We now show the improvements due to SDAT empirically in the following sections.

heading: 5 ADAPTATION FOR CLASSIFICATION
text: We evaluate our proposed method on three datasets: Office-Home, VisDA-2017, and DomainNet, as well as by combining SDAT with two DAT based DA techniques: CDAN and CDAN+MCC.

heading: 5.1 DATASETS
text: Office-Home (Venkateswara et al., 2017): Office-Home consists of around 15,500 images from 65 classes and four distinct domains: Art (Ar), Clipart (Cl), Product (Pr) and Real World (Rw). VisDA-2017 (Peng et al., 2017): VisDA is a dataset that focuses on the transition from simulation to real world and contains approximately 280K images across 12 classes. DomainNet (Peng et al., 2019): DomainNet consists of 0.6 million images across 345 classes belonging to six domains. The domains are infograph (inf), clipart (clp), painting (pnt), sketch (skt), real and quickdraw.

heading: 5.2 DOMAIN ADAPTATION METHODS
text: CDAN (Long et al., 2018): Conditional Domain Adversarial network is a popular DA algorithm that improves the performance of the DANN algorithm. CDAN introduces the idea of multi-linear conditioning to align the source and target distributions better. CDAN* in Table 1 and 4 refers to our implementation of CDAN method. CDAN + MCC (Jin et al., 2020): In this method, the minimum class confusion loss term is added as a regularizer to CDAN. Minimum class confusion is a non-adversarial term that minimizes the pairwise class confusion on the target domain. This achieves state of the art accuracy among adversarial adaptation methods.

heading: 5.3 IMPLEMENTATION DETAILS
text: We implement our proposed method in the Transfer-Learning-Library (Junguang Jiang & Long, 2020) toolkit developed in PyTorch (Paszke et al., 2019). The main difference between the performance reported in the CDAN and our implementation (CDAN*) is the batch normalization layer in the domain classifier, which enhances performance.
We use a ResNet-50 backbone for Office-Home experiments and a ResNet-101 backbone for VisDA2017 and DomainNet experiments. The backbone is initialized with ImageNet weights. We use a learning rate of 0.01 with batch size 32 in all of our experiments. We tune ρ value in SDAT for a particular dataset and use the same value across domains. The ρ value is set to 0.02 for the Office-Home experiments, 0.005 for the VisDA-2017 experiments and 0.05 for the DomainNet experiments. More details are present in supplementary (refer App. F).
5.4 RESULTS
Table 2 shows the results on the large and challenging DomainNet dataset across five domains as done in (Junguang Jiang & Long, 2020). The proposed method improves the performance of CDAN significantly across all source-target pairs. On specific source-target pairs like infograph ) real, the performance increase is 4.5%. The overall performance of CDAN is improved by nearly 1.8% which is significant considering the large number of classes and images present in DomainNet. For the Office-Home dataset, we compare our methods with other domain adaptation algorithms including DANN, SRDC, MDD and fDAL. The results for the Office-Home dataset are shown in Table 1. We can see that adding
SDAT improves the performance on both CDAN and CDAN+MCC across all the transfer tasks. CDAN+MCC w/ SDAT achieves state-of-the-art adversarial adaptation performance on the OfficeHome dataset. The class-wise accuracy on VisDA-2017 are reported in Table 4. CDAN w/ SDAT improves the overall performance of CDAN by more than 1.5%. CDAN w/ SDAT improves the performance of underperforming minority classes like bicycle and car. Additional baselines and results are reported in supplementary (refer App. G) along with a discussion on statistical significance (App. J) .

heading: 6 ADAPTATION FOR OBJECT DETECTION
text: To further validate our approach’s generality and extensibility, we did experiments on DA for object detection. We use the same setting as proposed in DA-Faster (Chen et al., 2018) with all domain adaptation components and use it as our baseline. We use the mean Average Precision at 0.5 IoU (mAP) as our evaluation metric. In object detection, the smoothness enhancement can be achieved in two ways (empirical comparison in Sec. 6.2) :
a) DA-Faster w/ SDAT-Classification: Smoothness enhancement for classification loss. b) DA-Faster w/ SDAT: Smoothness enhancment for the combined classification and regression loss.

heading: 6.1 EXPERIMENTAL SETUP
text: We evaluate our proposed approach on object detection on two different domain shifts:
Pascal to Clipart (P → C): Pascal (Everingham et al., 2010) is a real-world image dataset which consists images with 20 different object categories. Clipart (Inoue et al., 2018) is a graphical image dataset with complex backgrounds and has the same 20 categories as Pascal. We use Resnet-101 (He et al., 2016) backbone for Faster R-CNN (Ren et al., 2015) following Saito et al. (2019). Cityscapes to Foggy Cityscapes (C → Fc): Cityscapes (Cordts et al., 2016) is a street scene dataset for driving, whose images are collected in clear weather. Foggy Cityscapes (Sakaridis et al., 2018) dataset is synthesized from Cityscapes for the foggy weather. We use Resnet-50 (He et al., 2016) as the backbone for Faster R-CNN for experiments on this task. Both domains have the same 8 object categories with instance labels.
The training is done via SGD with momentum 0.9 for 70k iterations with the learning rate of 10−3, and then dropped to 10−4 after 50k iterations. We split the target data into train and validation sets and report the best mAP on validation data. Additional experimental and implementation details are present in supplementary (refer App. F).

heading: 6.2 RESULTS
text: Table 3 shows the results on two domain shifts with varying batch size (bs) during training. We find that only smoothing w.r.t. classification loss is much more effective (SDAT-Classification) than smoothing w.r.t. combined classification and regression loss (SDAT). On average, SDATClassification produces an mAP gain of 2.0% compared to SDAT, and 2.8% compared to DA-Faster baseline.
The proposed SDAT-Classification significantly outperforms DA-Faster baseline and improves mAP by 1.3% on P → C and by 2.8% on C → Fc. It is noteworthy that increase in performance of SDAT-Classification is consis-
tent even after training with higher batch size (bs = 8) achieving improvement of 4.3% in mAP.
Table 3 also shows that even DA-Faster w/ SDAT (i.e. smoothing both classification and regression) outperforms DA-Faster by 0.9 % on average across all experiments. The improvement due to SDAT on adaptation for object detection shows the generality of SDAT across techniques that have some form of adversarial component present in the loss formulation.

heading: 7 DISCUSSION
text: How much smoothing is optimal?: Figure 4 (A) shows the ablation on ρ value (higher ρ value corresponds to more smoothing) on the Ar)Cl and Cl)Pr from Office-Home dataset with CDAN backbone. The performance of the different values of ρ is higher than the baseline with ρ = 0. It
2Our Implementation of CDAN. Refer to Section 5.3.
Which components benefit from smooth optima?: Figure 4 (C) shows the effect of introducing smoothness enhancement for different components in DAT. For this we use SAM on a) classifier (SDAT) b) discriminator (SDAT w/ adv) c) both classifier and discriminator (SDAT-all). It can be seen that smoothing the adversarial component (SDAT w/ adv) reduces the performance to 51.0%, which is significantly lower than even the DAT baseline.
Is it Robust to Label Noise?: In practical, real-world scenarios, the labeled datasets are often corrupted with some amount of label noise. Due to this, performing domain adaptation with such data is challenging. We find that smoother minima through SDAT lead to robust models which generalize well on the target domain. Figure 4 (B) provides the comparison of SGD vs. SDAT for different percentages of label noise injected into training data (by flipping the labels).
Is it better than other smoothing techniques? To answer this question, we compare SDAT with different smoothing techniques originally proposed for ERM. We specifically compare our method against DAT, Label Smoothing (LS) (Szegedy et al., 2016), and VAT (Miyato et al., 2019). Stutz et al. (2021) recently showed that these techniques produce a significantly smooth loss landscape in comparison to SGD. We also compare with a very recent
SWAD (Cha et al., 2021) technique which is shown effective for domain generalization. For this, we run our experiments on four different splits of the Office-Home dataset and summarize our results in Table 5. We find that techniques for ERM (LS and VAT) fail to provide significant consistent gain in performance which also confirms the requirement of specific smoothing strategies for DAT. We find that SDAT even outperforms SWAD on average by a significant margin of 1.2%. Additional details regarding the specific methods are provided in the supplementary (refer App. H).

heading: 8 CONCLUSION
text: In this work, we analyse the curvature of loss surface of DAT used extensively for Unsupervised Domain Adaptation. We find that converging to a smooth minima w.r.t. task loss (i.e., empirical source risk) leads to better generalization on the target domain. We also theoretically and empirically show that smoothness enhancing for adversarial components of loss lead to sub-optimal results, hence should be avoided in practice. We then introduce our practical and effective method, SDAT, which only increases the smoothness w.r.t. task loss, leading to better generalization on the target domain. SDAT leads to an effective increase even for the state of the art methods for adversarial domain adaptation and can be incorporated with just a few lines of code change. One limitation of SDAT is presence of no automatic way of selecting ρ (determines extent of smoothness) which is a good future direction to explore.

heading: A NOTATION TABLE
text: Table S1 contains all the notations used in the paper and the proofs of theorems.
Table S1: The notations used in the paper and the corresponding meaning. Notation Meaning S Labeled Source Data T Unlabelled Target Data PS (or PT ) Source (or Target) Distribution X Input space Y Label space y(·) Maps image to labels hθ Hypothesis function RlS(hθ) (or R l T (hθ)) Source (or Target) risk
R̂lS(hθ) (or R̂ l T (hθ)) Empirical Source (or Target) risk H Hypothesis space Dφhθ,H(PS ||PT ) Discrepancy between two domains PS and PT gψ Feature extractor fΘ Classifier DΦ Domain Discriminator dΦS,T Tractable Discrepancy Estimate ∇2θR̂lS(hθ) (or H) Hessian of classification loss Tr(H) Trace of Hessian λmax Maximum eigenvalue of Hessian Perturbation ρ Maximum norm of
B CONNECTION OF DISCREPANCY TO dΦS,T (EQ. 4) IN MAIN PAPER
We refer reader to Appendix C.2 of Acuna et al. (2021) for relation of dΦS,T . The d Φ S,T term defined in Eq. 4 given as:
dΦS,T = Ex∼PS [log(DΦ(gψ(x)))] + Ex∼PT [log(1−DΦ(gψ(x)))] (S1)
The above term is exactly the Eq. C.1 in Acuna et al. (2021) where they show that optimal dΦS,T i.e.:
max Φ
dΦS,T = DJS(PS ||PT )− 2 log(2) (S2)
Hence we can say from result in Eq. 4 is a consequence of Lemma 1 and Proposition 1 in (Acuna et al., 2021), assuming that DΦ satisfies the constraints in Proposition 1.

heading: C PROOF OF THEOREMS
text: In this section we provide proofs for the theoretical results present in the paper:
Theorem 1 (Generalization bound). Suppose l : Y × Y → [0, 1] ⊂ dom φ∗. Let h∗ be the ideal joint classifier with error λ∗ = RlS(h ∗) + RlT (h ∗). We have the following relation between source and target risk: RlT (hθ) ≤ RlS(hθ) +D φ hθ,H(PS ||PT ) + λ ∗ (S3)
Proof. We refer the reader to Theorem 2 in Appendix B of Acuna et al. (2021) for the detailed proof the theorem.
We now introduce a Lemma for smooth functions which we will use in the proofs subsequently:
Lemma 1. For an L-smooth function f(w) the following holds where w∗ is the optimal minima:
f(w)− f(w∗) ≥ 1 2L ||∇f(w)||2
Proof. The L-smooth function by definition satisfies the following:
f(w∗) ≤ f(v) ≤ f(w) +∇f(w)(v − w) + L 2 ||v − w||2
Now we minimize the upper bound wrt v to get a tight bound on f(w∗).
D(v) = f(w) +∇f(w)(v − w) + L 2 ||v − w||2
after doing∇vD(v) = 0 we get:
v = w − 1 L ∇f(w)
By substituting the value of v in the upper bound we get:
f(w∗) ≤ f(w)− 1 2L ||∇f(w)||2
Hence rearranging the above term gives the desired result:
f(w)− f(w∗) ≥ 1 2L ||∇f(w)||2
Theorem 2. For a given classifier hθ and one step of (steepest) gradient ascent i.e. Φ′ = Φ + η(∇dΦS,T /||∇dΦS,T ||) and Φ′′ = Φ + η(∇dΦS,T |Φ+̂(Φ)/||∇dΦS,T |Φ+̂(Φ)||) for maximizing
dΦ ′ S,T − dΦ ′′ S,T ≤ η(1− cosα) √ 2L(d∗S,T − dΦS,T ) (S4)
where α is the angle between∇dΦS,T and∇dΦS,T |Φ+̂(Φ).
Proof of Theorem 2. We assume that the function is L-smooth (the assumption of L-smoothness is the basis of many results in non-convex optimization (Carmon et al., 2020)) in terms of input x. As for a fixed hθ as we use a reverse gradient procedure for measuring the discrepancy, only one step analysis is shown. This is because only a single step of gradient is used for estimating discrepancy dΦS,T i.e. one step of each min and max optimization is performed alternatively for optimization. After this the hθ is updated to decrease the discrepancy. Any differential function can be approximated by the linear approximation in case of small η:
dΦ+ηvS,T ≈ d Φ S,T + η∇dΦS,T T v (S5)
The dot product between two vectors can be written as the following function of norms and angle θ between those:
∇dΦS,T T v = ||∇dΦS,T || ||v|| cosθ (S6)
The steepest value will be achieved when cos θ = 1 which is actually v = ∇dΦS,T (x) ||∇dΦS,T (x)|| . Now we compare the descent in another direction v2 = ∇dΦS,T |w+ (w) ||∇dΦS,T |w+ (w)|| from the gradient descent. The difference in value can be characterized by:
dΦ+ηvS,T − d Φ+ηv2 S,T = η||∇d Φ S,T ||(1− cosα) (S7)
As α is an angle between∇dΦS,T |w+ (w) (v2) and∇dΦS,T (X) (v). The suboptimality is dependent on the gradient magnitude. We use the following result to show that when optimality gap d∗S,T−dΦS,T (x) is large the difference between two directions is also large.
For an L-smooth function the following holds according to Lemma 1:
f(w)− f(w∗) ≥ 1 2L ||∇f(w)||2
As we are performing gradient ascent f(w) = −dΦs,t, we get the following result:
(d∗S,T − dΦS,T ) ≥ 1
2L ||∇dΦS,T (x)||2
2L(d∗S,T − dΦS,T ) ≥ (dΦ+ηv2S,T − d Φ+ηv S,T ) 2
(η(1− cosα))2 η(1− cosα) √
2L(d∗S,T − dΦS,T ) ≥ (d Φ′ S,T − dΦ ′′ S,T )
This shows that difference in value of by taking a step in direction of gradient v vs taking the step in a different direction v2 is upper bounded by the d∗S,T − dΦS,T (x), hence if we are far from minima the difference can be potentially large. As we are only doing one step of gradient ascent d∗S,T −dΦS,T will be potentially large, hence can lead to suboptimal measure of discrepancy.
Theorem 3. Suppose l is the loss function, we denote λ∗ := RlS(h∗) + RlT (h∗) and let h∗ be the ideal joint hypothesis:
RlT (hθ) ≤ max|| ||≤ρ R̂lS(hθ+ ) +D φ hθ,H (PS ||PT ) + γ(||θ||22/ρ2) + λ∗. (S8)
where γ : R+ → R+ is a strictly increasing function.
Proof of Theorem 3: In this case we make use of Theorem 2 in the paper sharpness aware minimization (Foret et al., 2021) which states the following: The source riskRS(h) is bounded using the following PAC-Bayes generalization bound for any ρ with probability 1− δ:
RS(hθ) ≤ max || ||≤ρ R̂S(hθ) +
√√√√√√k log ( 1 + ‖θ‖22 ρ2 ( 1 + √ log(n) k )2) + 4 log nδ + Õ(1)
n− 1
(S9)
here n is the training set size used for calculation of empirical risk R̂S(h), k is the number of parameters and ||θ||2 is the norm of the weight parameters. The second term in equation can be abbreviated as γ(||θ||2). Hence,
RS(hθ) ≤ max || ||≤ρ R̂S(hθ) + γ(||θ||22/ρ2) (S10)
From the generalization bound for domain adaptation for any f-divergence (Acuna et al., 2021) (Theorem 2) we have the following result.
RlT (hθ) ≤ RlS(hθ) +D φ hθ,H (PS ||PT ) + λ∗ (S11) Combining the above two inequalities gives us the required result we wanted to prove i.e.
RlT (hθ) ≤ R̃lS(hθ) +D φ hθ,H (PS ||PT ) + γ(||θ||22/ρ2) + λ∗. (S12)

heading: D HESSIAN ANALYSIS
text: We use the PyHessian library (Yao et al., 2020) to calculate the Hessian eigenvalues and the Hessian Eigen Spectral Density. All the calculations are performed using 50% of the source data at the last checkpoint. Only the source class loss is used for calculating to clearly illustrate our point. The partition was selected randomly, and the same partition was used across all the runs. We also made sure to use the same environment to run all the Hessian experiments. A subset of the data was used for Hessian calculation mainly because the hessian calculation is computationally expensive (Yao et al., 2020). This is commonly done in hessian experiments. For example, (Chen et al., 2021) (refer Appendix D) uses 10% of training data for Hessian Eigenvalue calculation The PyHessian library uses Lanczos algorithm (Ghorbani et al., 2019) for calculating the Eigen Spectral density of the Hessian and uses the Hutchinson method to calculate the trace of the Hessian efficiently.
Table S2: Architecture used for feature classifier and Domain classifier. C is the number of classes. Both classifiers will take input from feature generator (gθ).
Layer Output Shape Feature Classifier (fΘ)
- Bottleneck Dimension Linear C
Domain Classifier (DΦ) - Bottleneck Dimension
Linear 1024 BatchNorm 1024
ReLU 1024 Linear 1024 BatchNorm 1024 ReLU 1024 Linear 1
Table S3: Accuracy (%) on VisDA2017 (ResNet-101).
Method Synthetic→ Real DANN (Ganin et al., 2016) 57.4 MCD (Saito et al., 2018b) 71.4 CDAN (Long et al., 2018) 73.7 CDAN*a 76.6 CDAN w/ SDAT 78.3 CDAN+MCC (Jin et al., 2020) 80.4 CDAN+MCC w/ SDAT 81.2
aOur implementation of CDAN. Refer to Section F for more details.

heading: E SMOOTHNESS OF DISCRIMINATOR IN SNGAN
text: We also did the similar experiment of smoothing discriminator in DAT (Sec. 4.1) for SNGAN Miyato et al. (2018) as the adversarial objective in GAN is similar to DAT. We use the same configuration for SNGAN as described in PyTorchStudioGAN (Kang & Park, 2020) for both CIFAR10 (Krizhevsky et al., 2009) and TinyImageNet 3 with batch size of 256 in both cases. We then smooth the discriminator while discriminator is trained by using the same formulation as in Eq. 9. We find that smoothing discriminator leads to higher (suboptimal) Fréchet Inception Distance in case of GANs as well, shown in Fig. 3.

heading: F EXPERIMENTAL DETAILS
text: F.1 IMAGE CLASSIFICATION
Office-Home: For CDAN methods, we train the models using mini-batch stochastic gradient descent (SGD) with a batch size of 32 and a learning rate of 0.01. The learning rate schedule is the same as (Ganin et al., 2016). We train it for a total of 30 epochs with 1000 iterations per epoch. The momentum parameter in SGD is set to 0.9 and a weight decay of 0.001 is used. For CDAN+MCC experiments, we use a temperature parameter (Jin et al., 2020) of 2.5. The bottleneck dimension for the features is set to 2048. VisDA-2017: We use a ResNet-101 backbone initialized with ImageNet weights for VisDA-2017 experiments. Center Crop is also used as an augmentation during training. We use a bottleneck dimension of 256 for both algorithms. For CDAN runs, we train the model for 30 epochs with same optimizer setting as that of OfficeHome. For CDAN+MCC runs, we use a temperature parameter of 3.0 and a learning rate of 0.002. DomainNet: We use a ResNet-101 backbone initialized with ImageNet weights for DomainNet experiments. We run all the experiments for 30 epochs with 2500 iterations per epoch. The other parameters are the same as that of Office-Home.
To show the effectiveness of SDAT fairly and promote reproducibility, we run with and without SDAT on the same GPU and environment and with the same seed. All the above experiments were run on Nvidia V100 and RTX 2080 GPUs. We used Wandb (Biewald, 2020) to track our experiments. We will be releasing the code to promote reproducible research.
F.1.1 ARCHITECTURE OF DOMAIN DISCRIMINATOR
One of the major reasons for increased accuracy in Office-Home baseline CDAN compared to reported numbers in the paper is the architecture of domain classifier. The main difference is the use
3https://www.kaggle.com/c/tiny-imagenet
Table S5: Accuracy(%) on DomainNet dataset for unsupervised domain adaptation (ResNet-101) across five distinct domains. The row indicates the source domain and the columns indicate the target domain.
ADDA clp inf pnt rel skt Avg MCD clp inf pnt rel skt Avg clp - 11.2 24.1 41.9 30.7 27.0 clp - 14.2 26.1 45.0 33.8 29.8 inf 19.1 - 16.4 26.9 14.6 19.2 inf 23.6 - 21.2 36.7 18.0 24.9 pnt 31.2 9.5 - 39.1 25.4 26.3 pnt 34.4 14.8 - 50.5 28.4 32.0 rel 39.5 14.5 29.1 - 25.7 27.2 rel 42.6 19.6 42.6 - 29.3 33.5 skt 35.3 8.9 25.2 37.6 - 26.7 skt 41.2 13.7 27.6 34.8 - 29.3 Avg 31.3 11.0 23.7 36.4 24.1 25.3 Avg 35.4 15.6 29.4 41.7 27.4 29.9 CDAN clp inf pnt rel skt Avg CDAN w/ SDAT clp inf pnt rel skt Avg clp - 20.6 38.9 56.0 44.9 40.1 clp - 22.0 41.5 57.5 47.2 42.1 inf 31.5 - 29.3 43.6 26.3 32.7 inf 33.9 - 30.3 48.1 27.9 35.0 pnt 44.1 19.8 - 57.2 39.9 40.2 pnt 47.5 20.7 - 58.0 41.8 42.0 rel 55.8 24.4 53.2 - 42.3 43.9 rel 56.7 25.1 53.6 - 43.9 44.8 skt 56.0 20.7 45.3 54.9 - 44.2 skt 58.7 21.8 48.1 57.1 - 46.4 Avg 46.9 21.4 41.7 52.9 38.3 40.2 Avg 49.2 22.4 43.4 55.2 40.2 42.1
of batch normalization layer in domain classifier, which was done in the library (Junguang Jiang & Long, 2020). Table S2 shows the architecture of the feature classifier and domain classifier.
F.2 ADDITIONAL IMPLEMENTATIONS DETAILS FOR DA FOR OBJECT DETECTION
In SDAT, we modified the loss function present in Chen et al. (2018) by adding classification loss smoothing, i.e. smoothing classification loss of RPN and ROI, used in Faster R-CNN (Ren et al., 2015), by training with source data. Similarly, we applied smoothing to regression loss and found it to be less effective. We implemented SDAT for object detection using Detectron2 (Wu et al., 2019). We fixed ρ to 0.15 for object detection experiments.

heading: G ADDITIONAL RESULTS
text: VisDA-2017: Table S3 shows the overall accuracy on the VisDA-2017 with ResNet-101 backbone. The accuracy reported in this table is the overall accuracy of the dataset, whereas the accuracy reported in the Table 5 of the main paper refers to the mean of the accuracy across classes. CDAN w/ SDAT outperforms CDAN by 1.7%, showing the effectiveness of SDAT in large scale Synthetic → Real shifts. With CDAN+MCC as the backbone, adding SDAT improves the performance of the method to 81.2%. DomainNet: Table S5 shows the results of the proposed method on DomainNet across five domains. We compare our results with ADDA and MCD and show that CDAN achieves much higher performance on DomainNet compared to other techniques. It can be seen that CDAN w/ SDAT further improves the overall accuracy on DomainNet by 1.8%. Results with DANN (Ganin & Lempitsky, 2015): Domain Adversarial neural networks introduced the concept of adversarial training in domain adaptation and is a seminal paper in the field of domain adaptation. Table S4 shows the results on some splits on Office-Home with DANN and DANN w/ SDAT. DANN w/ SDAT improves upon the performance on DANN specifically in challenging splits like Clipart→ Art where DANN w/ SDAT gets a 1% increase over DANN. We have shown results with three different domain adaptation algorithms namely DANN (Ganin & Lempitsky, 2015), CDAN (Long et al., 2018) and CDAN+MCC (Jin et al., 2020). SDAT has shown to improve the performance of all the three DA methods. This shows that SDAT is a generic method that can applied on top of any domain adversarial training based method to get better performance.
Table S4: Results on Office-Home dataset with DANN (Ganin & Lempitsky, 2015). DANN w/ SDAT improves the performance over DANN across the four splits of Office-Home dataset showing the adaptability of the proposed method.
Method Ar)Cl Cl)Pr Rw)Cl Pr)Cl Average DANN (Ganin & Lempitsky, 2015) 52.6 65.4 60.4 52.3 57.7 DANN w/ SDAT 53.4 66.4 61.3 53.8 58.7

heading: H DIFFERENT SMOOTHING TECHNIQUES
text: Stochastic Weight Averaging (SWA) (Izmailov et al., 2018): SWA is a widely popular technique to reach a flatter minima. The idea behind SWA is that averaging weights across epochs leads to better generalization because it reaches a wider optima. The recently proposed SWA-Densely (SWAD) (Cha et al., 2021) takes this a step further and proposes to average the weights across iterations instead of epochs. SWAD shows improved performance on domain generalization tasks. We average every 400 iterations in the SWA instead of averaging per epochs. We tried averaging across 800 iterations as well and the performance was comparable. Virtual Adversarial Training (VAT) (Miyato et al., 2019): VAT is regularization technique which makes use of adversarial perturbations. Adversarial perturbations are created using Algo. 1 present in (Miyato et al., 2019). We added VAT by optimizing the following objective:
min θ Ex∼PS [ max||r||≤ DKL(hθ(x)||hθ(x+ r))] (S13)
This value acts as a negative measure of smoothness and minimizing this will make the model smooth. For training, we set hyperparameters to 15.0, ξ to 1e-6, and α as 0.1. Label Smoothing (LS) (Szegedy et al., 2016): The idea behind label smoothing is to have a distribution over outputs instead of one hot vectors. Assuming that there are k classes, the correct class gets a probability of 1 - α and the other classes gets a probability of α / (k-1). (Stutz et al., 2021) mention that label smoothing tends to avoid sharper minima during training. We use a smoothing parameter (α) of 0.1 in all the experiments in Table S6. We also show results with smoothing parameter of 0.2 and observe comparable performance. We observe that label smoothing slightly improves the performance over DAT.
Table S6: Different Smoothing techniques. We refer to (Stutz et al., 2021) to compare the proposed SDAT with other techniques to show the efficacy of SDAT. It can be seen that SDAT outperforms the other smoothing techniques significantly. Other smoothing techniques improve upon the performance of DAT showing that smoothing is indeed necessary for better adaptation.
Method Ar)Cl Cl)Pr Rw)Cl Pr)Cl DAT 54.3 69.5 60.1 55.3 VAT 54.6 70.7 60.8 54.4 SWAD-400 54.6 71.0 60.9 55.2 LS (α = 0.1) 53.6 71.6 59.9 53.4 LS (α = 0.2) 53.5 71.2 60.5 53.2 SDAT 55.9 73.2 61.4 55.9

heading: I OPTIMUM RHO VALUE
text: Table S7 and S8 show that ρ = 0.02 works robustly across experiments providing an increase in performance (although it does not achieve the best result each time) and can be used as a rule of thumb.
Table S7: ρ value for DomainNet Split DAT SDAT(ρ = 0.02) SDAT - Reported (ρ = 0.05) clp)skt 44.9 46.7 47.2 skt)clp 56.0 59.0 58.7 skt)pnt 45.3 47.8 48.1 inf)rel 43.6 47.3 48.1
Table S8: ρ value for VisDA-2017 Synthetic ) Real Backbone DAT SDAT (ρ = 0.02) SDAT Reported(ρ = 0.005) CDAN 76.6 78.2 78.3 CDAN+MCC 80.4 80.9 81.2
0 10 20 30 Epoch
32
34
36
38
40
42
Va lid
at io
n Ac
cu ra
cy (
in %
)
Clipart Painting
CDAN w/ SDAT CDAN
0 10 20 30 Epoch
38
40
42
44
46
Va lid
at io
n Ac
cu ra
cy (
in %
)
Clipart Sketch
CDAN w/ SDAT CDAN
0 10 20 30 Epoch
22
24
26
28
30
32
34
Va lid
at io
n Ac
cu ra
cy (
in %
)
Infograph Clipart
CDAN w/ SDAT CDAN
0 10 20 30 Epoch
34
36
38
40
42
44
46
Va lid
at io
n Ac
cu ra
cy (
in %
)
Painting Clipart
CDAN w/ SDAT CDAN
0 10 20 30 Epoch
34
36
38
40
42
44
Va lid
at io
n Ac
cu ra
cy (
in %
)
Real Sketch
CDAN w/ SDAT CDAN
0 10 20 30 Epoch
46
48
50
52
54
56
Va lid
at io
n Ac
cu ra
cy (
in %
)
Sketch Real
CDAN w/ SDAT CDAN
Figure S1: Validation Accuracy across epochs on different splits of DomainNet. We run on three different random seeds and plot the error bar indicating standard deviation across runs. CDAN w/ SDAT consistently outperforms CDAN across different splits of DomainNet.

heading: J SIGNIFICANCE AND STABILITY OF EMPIRICAL RESULTS
text: To establish the empirical results’ soundness and reliability, we run a subset of experiments (representative of each different source domain) on DomainNet. The experiments are repeated with three different random seeds leading to overall 36 experimental runs (18 for CDAN w/ SDAT (Our proposed method) and 18 for CDAN baseline). Due to the large computational complexity of each experiment (≈20 hrs each), we have presented results for multiple trials on a subset of splits. We find (in Table S9) that our method can outperform the baseline average in each of the 6 cases, establishing significant improvement across all splits. However, we found that due to the large size of DomainNet, the average increase (across three different trials) is close to the reported increase in all cases (Table S9), which also serves as evidence of the soundness of reported results (for remaining splits). We also present additional statistics below for establishing soundness.
If the proposed method is unstable, there is a large variance in the validation accuracy across epochs. For analyzing the stability of SDAT, we show the validation accuracy plots in Figure S1 on six different splits of DomainNet. We find that our proposed SDAT improves over baselines consistently across epochs without overlap in confidence intervals in later epochs. This also provides evidence for the authenticity and stability of our results. We also find that in some cases, like when using the Infographic domain as a source, our proposed SDAT also significantly stabilizes the training (Figure S1 Infographic ) Clipart).
One of the other ways of reporting results reliably proposed by the concurrent work (Berthelot et al., 2021) (Section 4.4) involves reporting the median of accuracy across the last few checkpoints. The median is a measure of central tendency which ignores outlier results. We also report the median of validation accuracy for our method across all splits for the last five epochs. It is observed that we observe similar gains for median accuracy (in Table S10) as reported in Table 2.
Table S9: DomainNet experiments over 3 different seeds. We report the mean, standard deviation, reported increase and average increase in the accuracy (in %).
Split CDAN CDAN w/ SDAT Reported Increase (Table 2) Average Increase clp)pnt 38.9 ± 0.1 41.5 ± 0.3 +2.6 +2.6 skt)rel 55.1 ± 0.2 57.1 ± 0.1 +2.2 +2.0 pnt)clp 44.5 ± 0.3 47.1 ± 0.3 +3.4 +2.6 rel)skt 42.4 ± 0.4 43.9 ± 0.1 +1.6 +1.5 clp)skt 44.9 ± 0.2 47.3 ± 0.1 +2.3 +2.4 inf)clp 31.4 ± 0.5 34.2 ± 0.3 +2.3 +2.7
Table S10: Median accuracy of last 5 epochs on DomainNet dataset with CDAN w/ SDAT. The number in the parenthesis indicates the increase in accuracy with respect to CDAN.
Target (→) clp inf pnt real skt AvgSource (↓)
clp - 21.9 41.6 56.5 46.4 41.6(+1.7) (+3.0) (+1.3) (+2.0) (+2.0)
inf 32.4 - 29.8 46.7 25.6 33.6(+7.9) (+7.0) (+12.7) (+5.4) (+8.2)
pnt 47.2 21.0 - 57.6 41.5 41.8(+2.9) (+1.1) (+1.0) (+2.4) (+1.8)
real 56.5 25.5 53.9 - 43.5 44.8(+0.7) (+0.9) (+0.5) (+1.3) (+0.8)
skt 59.1 22.1 48.2 56.6 - 46.5(+3.0) (+1.7) (+3.1) (+2.9) (+2.7)
Avg 48.8 22.6 43.4 54.3 39.2 41.7(+3.6) (+1.3) (+3.4) (+4.5) (+2.8) (+3.1)
As the Office-Home dataset is smaller (i.e., 44 images per class) in comparison to DomainNet we find that there exists some variance in baseline CDAN results (This is also reported in the wellknown benchmark for DA (Junguang Jiang & Long, 2020)). For establishing the empirical soundness, we report results of 4 different dataset splits on three different random seeds. It can be seen in Table S11 that even though there is variance in baseline results, our combination of CDAN w/ SDAT can produce consistent improvement across different random seeds. This further establishes the empirical soundness of our procedure.
Table S11: Office-Home experiments over 3 different seeds. We report the mean, standard deviation, reported increase and average increase in the accuracy (in %).
Split CDAN CDAN w/ SDAT Reported Increase (Table 1) Average Increase Ar)Cl 53.9 ± 0.2 55.5 ± 0.2 +1.7 +1.6 Ar)Pr 70.6 ± 0.4 72.1 ± 0.4 +1.6 +1.5 Rw)Cl 60.7 ± 0.5 61.8 ± 0.4 +1.3 +1.1 Pr)Cl 54.7 ± 0.4 55.5 ± 0.4 +0.6 +0.8

Review summaries:
This work studies the loss landscape of domain adversarial neural networks for domain adaptation. The authors claim that similarly to the iid setting, smooth minima might yield better generalization on the unsupervised domain adaptation framework via domain adversarial training but in case smoothness is enforced only with respect to the task loss. Experiments on the OfficeHome dataset indicated this hypothesis might be true. The authors then proposed to apply a previously proposed approach to enforce smoothness (Sharpness Aware Minimization), so that only smoothness with respect to the task loss would be enforced. Moreover, a theorem showing that the non-smooth domain discrepancy estimation is better than the smoother counterpart was introduced, along with a generalization bound for the risk on the target domain in terms of the smooth risk on the source domain. Finally, the authors showed that the proposed approach, SDAT, improved the performance on target domains of three domain adaptation datasets in comparison to the considered baselines. Further experiments showed that SDAT performed better than other smoothing techniques, as well as yields improved robustness to label noise in comparison to not enforcing smooth minima. 

This paper introduces a smoothness penalty for domain adversarial training. The penalty encourages smoothness on the parameter space of a discriminative model. This smoothness penalty is motivated by results in Sharpness Aware Minimization. Results are included on the popular DA datasets such as Office-home and VisDA, where superior results are shown. Ablation experiments show how different smoothness penalties change the observed improvements. 

The paper discussed a correlation of smoothness on **classification loss** and **generalization ability on target domain** and further develop a method to enhance such smoothness to achieve better performance on DA tasks. To do so, it adopted the losses based on Sharpness Aware Minimization, a minmax game of finding a smoother neighborhood of $\theta$ , by computing first order deviation. Empirical studies are implemented to verify the theory and show the soundness of the proposed method.

The paper analyzes the role of smoothness in domain adversarial training (DAT). The main insight of the paper is that smoother minima of the classification loss improve generalization in the target domain. This explains why SGD is usually preferred over Adam when optimizing the training objective of DAT. To further improve the generalization performance by leveraging this phenomenon the author(s) introduce(s) smooth domain adversarial training (SDAT) for classification and object detection. SDAT applies sharpness aware minimization (SAM) to find smoother minimia of the classification loss used during DAT.