Free Lunch for Domain Adversarial Training: Environment Label Smoothing
Authors: YiFan Zhang
xue wang
Jian Liang
Zhang Zhang
Liang Wang
Rong Jin
Tieniu Tan
Venue: ICLR 2023 poster (2023)
Keywords: Out-of-Distribution Generalization
Domain adaptation/generalization
Domain adversarial training
environmnt label noise
non-asymptotic convergence

Abstract: A fundamental challenge for machine learning models is how to generalize learned models for out-of-distribution (OOD) data. Among various approaches, exploiting invariant features by Domain Adversarial Training (DAT) received widespread attention. Despite its success, we observe training instability from DAT, mostly due to over-confident domain discriminator and environment label noise. To address this issue, we proposed Environment Label Smoothing (ELS), which encourages the discriminator to output soft probability, which thus reduces the confidence of the discriminator and alleviates the impact of noisy environment labels. We demonstrate, both experimentally and theoretically, that ELS can improve training stability, local convergence, and robustness to noisy environment labels. By incorporating ELS with DAT methods, we are able to yield state-of-art results on a wide range of domain generalization/adaptation tasks, particularly when the environment labels are highly noisy. 


FREE LUNCH FOR DOMAIN ADVERSARIAL TRAINING: ENVIRONMENT LABEL SMOOTHING
Yi-Fan Zhang' 1 Xue Wang?, Jian Liang"
Zhang Zhang"? 1.2 .iiang Wang',, Rong Jin Tieniu Tan'
1 NNational Laboratory of Pattern Recognition (NLPR), Institute Automation
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)
3 Machine Intelligence Technology, Alibaba Group.
ABSTRACT
A fundamental challenge for machine learning models is how to generalize learned models for out-of-distribution (OOD) data. Among various approaches, exploiting invariant features by Domain Adversarial Training (DAT) received widespread attention. Despite its success, we observe training instability from DAT, mostly due over-confident domain discriminator and environment label noise. To address this issue, we proposed Environment Label Smoothing (ELS), which encourages the discriminator output soft probability, which thus reduces the confidence of the discriminator and alleviates the impact of noisy environment labels. We demonstrate, both experimentally and theoretically, that ELS can improve training stability, local convergence, and robustness to noisy environ- ment labels. By incorporating ELS with DAT methods, we are able to yield the state-of-art results wide range of domain gemeralization/adaptttio tasks, par- ticularly when the environment labels are highly noisy. The code is avaliable at https://github.com//ffi http://////iaaaaa///iiiooooooot 144/e/4rromeett--aieee Lawee-Saoolhlhiiiiiii
INTRODUCTION
Despite being empirically effective visual recognition benchmarks (Russakovsky al., (001)). modern neural networks are prone to learning shortcuts that stem from spurious correlations (Geirhos) a 2020), resulting in poor generalization for out-of-distribution (OOD) data. A popular thread of methods, minimizing domain divergence by Domain Adversarial Training (DAT) (Ganin et al. 2016), has shown better domain transfer performance, suggesting that it is potential to an effective candidate to extract domain-invariamt features. Despite its power for domain adaptation and domain generalization, DAT is known be difficult to train and converge (Roth et al. 2017 Jenni Favaro 2019 Arjovsky & Bottou] 2017 Sønderby et al. 2016).
The main difficulty for stable training to maintain healthy com- petition between the encoder and the domain discriminator. Recent work seeks to attain this goal by designing novel optimization meth-
ods (Acuna et al.) 2022 Rangwani et al., 2022), however, most of them require additional optimization steps and slow the convergence. In this work, aim tackle the challenge from totally different aspect from previous works, i.e., the environment label design.
Oseimi Essiormas Midd &e Varila DAT
T |o
0
Teomnrment Seecetiing (y-0i))
Two important observations that lead to the training instability of DAT motivate this work: (i) The environment label noise from en- vironment partition (Creager al. 2021) and training (Thanh-Tung let al. 2019). As shown in Figure 1 different domains of the VI.CS benchmark have no significant difference in image style and some images are indistinguisaable for which domain they belong. Besides, when the encoder gets better, the generated features from different domains are more similar. However, regardless of their quality, features are still labeled differently. As shown in ('hanh-Tung et al. 2019), discriminators will
Figure 1: A motivating exam- ple of ELS with 3 domains on the VLCS dataset.
Work done during an internship at Alibaba Group. Work done A Alibaba Group, and now affiliated with Twitter.

---

overfit these mislabelled examples and then has poor generalization capability. (ii) To our best knowledge, DAT methods all assign one-hot environment labels to each data sample for domain discrimination, where the output probabilities will be highly confident. For DAT, very confident domain discriminator leads to highly oscillatory gradients (Arjovsky & Bottou 2017). which is harmful training stability. The first observation inspires to force the training process to be robust with regard to environment-label noise, and the second observation encourages the discriminator to estimate soft probabilities rather than confident classification. To this end, we propose Environment Label Smoothing (ELS), which is a simple method to tackle the mentioned obstacles for DAT. Next, we summarize the main methodological, theoretical, and experimental contributions.
Methodology: To our best knowledge, this is the first work to smooth environment labels for DAT. The proposed ELS yields three main advantages: (i) does not require any extra parameters and optimization steps and yields faster convergence speed, better training stability, and more robustness label noise theoretically and empirically; (ii) despite its efficiency, ELS also easily to implement. People can easily incorporate ELS with any DAT methods in very few lines of code; (iii) ELS equipped DAT methods attain superior generalization performance compared to their native counterparts;
Theories:
The benefit of ELS is theoretically verified in the following aspects. (i) Training stability. divergence minimization, where ELS is
We first connect DAT to shown able extend the support of training distributions and relieve both the oscillatory gradients and gradient vanishing phenomenons, which results in stable and well-behaved training. (ii) Robustness noisy labels. We theoretically verify that the negative effect caused by noisy labels be reduced or even eliminated by ELS with proper smooth parameter. (iii) Faster non-asymptotic convergence speed. We analyze the non-asymptotic convergence properties of DANN. The results indicate that incorporating with ELS can further speed up the convergence process. In addition, we also provide the empirical gap and analyze some commonly used DAT tricks.
Experiments: (i) Experiments are carried out on various benchmarks with different backbones, including image classification, image retrieval, neural language processing, genomics data, graph, and sequential data. ELS brings consistent improvement when incorporated with different DAT methods and achieves competitive or SOTA performance on various benchmarks, e.g., average accuracy on Rotating MNIST (52.1% - 62.1%), worst group accuracy CivilComments (61.7% 65.9%), test ID accuracy on RxRx1 (22.9% + 26. 76),, average accuracy on Spurious-Fourier dataset (11.1% + 15.6%). (ii) Even if the environment labels of ELS + DANN will not degrade much and is superior to native DANN. (iii) Abundant analyzes on training dynamics ablations on hyper-parameter for ELS and some useful suggestions about choosing the best smooth parameter considering the dataset information are given.
random or partially known, the performance
conducted verify the benefit of ELS empirically. (iv) We conduct thorough
2
METHODOLOGY
For domain generalization tasks, there are M source domains (D.) i-1- Let the hypothesis h be the composition of h = h o g, where g g pushes forward the data samples to a representation space Z and h = (6.())...,ndd softmax activation function. The classifier is defined h E H'' Z - [0, 1] (E, where C' is the number of classes. The cost used for the discriminator can be defined as:
H : Z + [0. M] MM f..()) 1 is the domain discriminator with
max n (DI.... Das)= max EXED, log o g((x)) + ..... EXED log has g(x), h
(1)
heat
where h2 g(x) is the prediction probability that x is belonged to D,. Denote the class label, then the overall objective of DAT is
1 M E M ExED, g(x)) v)1 (DI.., Das).
min max W.g
h
where & is the cross-entropy loss for classification tasks and MSE for regression tasks, and is the tradeoff weight. We call the first term empirical risk minimization (ERM) part and the second term
2

---

adversarial training (AT) part. Applying ELS, the target in Equ. [ can be reformulated
max d heil
(D1, (D Das) max RX he7l
y ylog g(x) + (1-?) Σ
log (he a ((x)) +..+
M M-11 j=1:f=1
Ex lloghss g(x) (1-2)
M
Σ j=1:jaM
log (h2 g((x)
M
THEORETICAL VALIDATION
In this section, we first assume the discriminator is optimized with no constraint, providing theoretical interpretation of applying ELS. Then how ELS makes the training process more stable is discussed based on the interpretation and some analysis of the gradients. We next theoretically show that with ELS, the effect of label noise can be eliminated. Finally, to mitigate the impact of the no constraint assumption, the empirical gap. parameterization gap, and con-asymptotic convergence property are analyzed respectively. All omitted proof' can be found in the Appendix.
3.1 DIVERGENCE MINIMIZATION INTERPRETATION
In this subsection, the connection between ELS/one-sided ELS and divergence minimization is studied. The advantages brought by ELS and why GANs prefer one-sided ELS are theoretically claimed. We begin with the two-domain setting, which is used in domain adaptation and generative adversarial networks. Then the result in the multi-domain setting is further developed.
Proposition 1. Given two domain distributions Ds, Dr over X, and a hypothesis class H. We suppose h H the optimal discriminator with no constraint, denote the mixed distributions with DDx = >Ds + (1 ()r Dr >D + (1-7)Ds Then minimizing domain divergence by adversarial training with ELS is equal minimizing 2D (Dss((D)))))) 22ogg 2, where Das is the Jensen-Shanon (JS) divergence.
hyper-parameter E [0.5, 1] as
Compared to Proposition 2 in (Acuna et al. 2021) that adversarial training in DANN is equal to minimize 2DJst (D>>>[(Dr) 2 log 2. The only difference here is the mixed distributions Dsr Dro, which allows more flexible control
divergence minimization. For example, when 7 Ds Ds, DT = Dr which is the same as the original adversarial training; when 7 0.5, Ds Dr 0.5(Ds + Dr) and DJs (D (Drr) = 0, which means that this term will not supply gradients during training and the training process will convergence like ERM. In other words, controls the tradeoff between algorithm convergence and adversarial divergence minimization. One main argue that adjusting the tradeoff weight À can also balance AT and ERM, however, À can only adjust the gradient contribution of AT part, i.e., 2XV DJs (Dss, Dr) and cannot affect the training dynamic of DJs DDs(Ds, Dr). For example, when Ds, Dr have disjoint support, DDss Dr) is always zero no matter what À is given. On the contrary, the proposed technique smooths the optimization distribution Ds. Dr of AT, making the whole training process more stable, but controlling À cannot do. In the experimental section, we show that in some benchmarks, the model cannot converge even if the tradeoff weight is small enough, however, when ELS is applied, DANN+ELS attains superior results and without the need for small tradeoff weights or small learning rate.
As shown in (Goodfellow) 2016), GANs always which is a simple modification of the label smoothing technique and only replaces the target for real examples with value slightly less than one, such as 0.9. Here we connect one-sided label smoothing to JS divergence and seek the difference between native and one-sided label smoothing techniques. See Appendix(A.2] to multi-domain settings, e.g., domain generalization, and multi-source GANs (Trung Le et al. 2019) (See Proposition Appendix A.3 for detailed proof and analysis.). We find that with ELS, flexible control on algorithm convergence and divergence minimization tradeoffs can be attained.
technique called one-sided label smoothing,
proof and analysis. We further extend the above theoretical analysis
3.2 TRAINING STABILITY
Noise injection for extending distribution supports. The main source of training instability of GANs is the real and the generated distributions have disjoint supports or lie on low dimensional
3

---

manifolds (Arjovsky & Bottou 2017: Roth al. 2017). 7. Adding noise from arbitrary distribution
to the data is shown to be able to extend the support of both distributions (Jenni & Favaro 2019)
Arjovsky & Bottou 2017 Sønderby et al. 2016) and will protect the discriminator against measure () adversarial examples (Jenni & Favaro 2019), which result in stable and well-behaved training, Environment label smoothing can be viewed
kind of noise injection, egg., in Proposition [ Der Dr 7Ds Dr) where the noise is y(ss Dr) and the two distributions will be more
likely to have joint supports.
ELS relieves the gradient vanishing phenomenon. As shown in Section 3.1 the adversarial target is approximating KL or JS divergence, and when the discriminator not optimal, such approximation is inaccurate. We show that in vanilla DANN, as the discriminator gets better, the gradient passed from discriminator to the encoder vanishes (Proposition f and Proposition (5) Namely, either the approximation is inaccurate, or the gradient vanishes, which will make adversarial training extremely hard (Arjovsky Bottou, 2017). Incorporating ELS shown able to relieve the gradient vanishing phenomenon when the discriminator is close to the optimal one and stabilizes the training process.
DANN
1250
DANX-ELS
ELS serves as data-driven regularization and stabilizes the oscillatory gradients. Gradients of the encoder with respect to ad- versarial loss remain highly oscillatory in native DANN, which is an important reason for the instability of adversarial training (Mescheder et al. 2018). Figure shows shows training process, where the PACS dataset is used as an example. With ELS, the gradient brought by the adversarial loss smoother and more stable. The benefit is theoretically supported in Section [A.6] where applying ELS is shown similar to adding a regularization term on discriminator parameters, which stabilizes the supplied gradients compared to the vanilla adversarial loss.
1000
150
500
250
the gradient dynamics throughout the
2200
Figure The sum of gradients provided to the encoder by the adversarial loss.
3.3
ELS MEETS NOISY LABELS
To analyze the benefits of ELS when noisy labels exist, we adopt the symmetric noise model (Kim al. 2019). Specifically, given two environments with a high-dimensional feature x and environment label y {0, 1). assume that noisy labels ÿ are generated by random noise transition with noise rate e Pgğ = 1ly 0)- P = 0ly- 1). Denote f := h g, l the cross-entropy loss and j the smoothed noisy label, then minimizing the smoothed loss with noisy labels can be converted to
min B (a (x((f())
- (1-
5)]
min mijn - 2Te)) -X) (((()),)))
7
where " is the optimal smooth parameter that makes the classifier return the best performance on unseen clean data (Wei al. 2022). The first term in Equ. (4) is the risk under the clean label. The influence of both noisy labels and ELS are reflected in the last term of the Equ. (4). B(c.))-l(((((()))) y) (<)))),))) [<)),))] y)] is the opposite of the optimization process as we expect. Without label smoothing, the weight will be 7" 1 and high noisy rate e will let this harmful term contributes more to our optimization. On the contrary, by choosing smooth parameter -- the second term will be removed. For example, if e 0, the best smooth parameter is just 7- 1-2e*
3.4 EMPIRICAL GAP AND PARAMETERIZATION GAP
Propositions in Section 3.1 and Section 3.2 are based on two unrealistic assumptions. (i) Infinite data samples, and (ii) the discriminator is optimized without a constraint, namely, the discriminator optimized over definiteddimnnsnn samples are observed and the discriminator is always constrained smaller classes such as neural networks (Goodfellow et al. 2014) or reproducing kernel Hilbert spaces (RKHS) (Li et al. 7a). Besides, as shown in (Arora al. [017]] Schääee et al.) 2019), JS divergence has large empirical gap, e.g., let Dr, D. be uniform Gaussian distributions N f1). and D 1, D,, be empirical versions of Dj,, D. with n examples. Then we have [djs(D) (D,,, D.) d (D,,. D.)l= = log 2 with high probability. Namely, the empirical divergence cannot reflect the true distribution divergence.
space. In practice, only empirical distributions with finite
4

---

A natural question arises: "Given finite samples multi-domain AT over mitte-dmmensioal parame- terized space, whether the expectation over the empirical distribution converges to the expectation over the true distribution?". In this subsection, seek to answer this question by analyzing the empirical gap and parameterization gap, which is kdi. (PI.... Dar> di.g 1, I..... ...... where Dj is the empirical distribution of D; and h is constrained. We first show that, let H be hypothesis class of VC dimension d, then for any d' e (0, 1), with probability at least - 5, the gap is less than 4v ///dlol(222 (n*) + log 2/8)/n", where n' min(111-- .-... naa) and n, is the number of samples in D; (Appendix A.8). The above analysis is based on H divergence and the VC dimension; we further ana- lyze the gap when the discriminator is constrained to the Lipschitz continuous and build a connection between the gap and the model parameters. Specifically, suppose that each h; is L-Lipschitz with respect the parameters and p to denote the number of parameters of hŞ. Then given universal constant such that when n² 2 cpM log( Lp/e)/e, we have with probability at least 1 - exp(-p), the gap is less than e (Appendix A.9 Although the analysis cannot support the benefits of ELS, as far as we know, it is the first attempt to study the empirical and parameterization gap of multi-domain AT.
3.5 NON-ASYMPTOTI CONVERGENCE
As mentioned in Section 3.4 the analysis in Section 3.1 and Section 3.2 assume the optimal discrimi- nator can be obtained, which implies that both the hypothesis set has infinite modeling capacity and the training process can converge to the optimal result. If the objective of AT is convex convex-coneave concave. then many works can support the global convergence behaviors (Nowozin et al. 2016 Yadav et al. 2017p. However, the convex-concave assumption is too unrealistic to hold true Nie & Patel 2020 Nagarajan & Kolter 2017), namely, the updates of DAT are longer guaranteed to converge. In this section, we focus on the local convergence behaviors of DAT of points near the equilibrium. Specifically, focus the non-asymptotic convergence, which is shown able to more precisely reveal the convergence of the dynamic system than the asymptotic analysis (Nie & Patel 2020).
We build toy example help us understand the convergence of DAT. Denote the learning rate, ? the parameter for ELS, and c constant. We conclude our theoretical results (which are detailed in Appendix AO): (I) Simultaneous Gradient Descent (GD) DANN, which trains the discriminator and encoder simultaneously, has no guarantee of the non-asymptotic convergence. (2) If we train the discriminator n. times once we train the encoder n. times, the resulting alternating Gradient Descent (GD) DANN could converge with a sublinear convergence rate only when the 1/ S "namae Such results support the importance of alternating GD training, which is commonly used during DANN implementation (Gulrajani & Lopez- Paz 2021 (3) Incorporate ELS into alternating GD speeds up the convergence rate by a factor 2--1 that is, when 1) S Vhanne 2 the model could converge.
Remark. In the above analysis, we made some assumptions e.g., in Section 3.5 we assume the algorithms are initialized in neighborhood of unique equilibrium point, and in Section 3.4 we assume that the NN is L-Lipschitz. These assumptions may not hold in practice, and they are computationally hard verify. To this end, we empirically support our theoretical results, namely, verifying the benefits to convergence, training stability, and generalization results in the next section.
4 EXPERIMENTS
To demonstrate the effectiveness of our ELS, in this section, we select a broad range of tasks (in Table D.. which are image classification, image retrieval. neural language processing, genomics, graph, and sequential prediction tasks. Our target is to include benchmarks with (i) various numbers of domains (from to 120, 084); (ii) various numbers of classes (from 2 to 18, 530); (iii) various dataset sizes (from 3, 200 to 448, 000); (iv) various dimensionalities and backbones (Transformer, ResNet, MobileNet, GIN, RNN). See Appendix C for full details of all experimental settings, including dataset details, hyper-parameters, implementation details, and model structures. We conduct all the experiments on a machine with 17-8700K, 32G RAM, and four GTX2080ti. All experiments are repeated 3 times with different seeds and the full experimental results can be found in the appendix.
4.1 NUMERICAL RESULTS ON DIFFERENT SETTINGS AND BENCHMARKS
Domain Generalization and Domain Adaptation on Image Classification Tasks. We first in- corporate ELS into SDAT, which is variant of the DAT method and achieves the state-of-the-art
5

---

Table 1: A summary on evaluation benchmarks. Wg. acc. denotes worst group accuracy, 10 %/ acc. denotes 10th percentile accuracy. GIN (Xu et al. 2018) denotes Graph Isomorphism Networks, and CRNN (Gagnon-Audet Audet et al. 2022) denotes convolutional recurrent neural networks.
Task
Dataset
Domains
Classes
Metric
Backchone
Data Examples
rotated angles
Aug. acc.
70,000
PACS
4 image styles
7
Ang. acc.
ResNet50
9,991
Images Chesifiiatiin
Office-31
VLCS
3
4 image styles
31
5
Avg. sec.
Arg.
ReNNe500ReNN118
ResNet50
10,729
4,110
image styles
Aug. Bcc.
ResNec5oVViT
15,500
Rotaling MNIST
rotated angles
10
Arg. acc.
ETcoderSTN
60,000
Image Retrivaal
MS
locations
18,530
mAP Kank
MobileNetx 1.4
121,738
Neural Language Processing
CiviCCormentss
Amazon
demographic groups
7676
%s/AggWW acc.
Avg/W
DistilBERT
DistilBERT
448,000
Genomies and Graph
OGB MelPCBA
RxRx]
120.044 molecular scaffold
51 experimental basch
11399
128
WggAwgg/est Test acc.
Avg. acc.
ResNer-50
GIN
437,929
125,510
Sequential
Spurious-Foune Fourier
spurious comelations
Aug.
LSTM
12,0000
HHAR
Aug. scc.
Deep
13,674
Table 2: The domain adaptation accuracies (%) on Office-31. 1 denotes improvement of method with ELS compared to that wo/ ELS.
A W
D. W
W.D
A
D A
W-A
Avg
ResNet18
ERM Vapmik 1999
72.2
97.7
100.0
72.3
61.0
59.9
77.2
DANN
2016
84.1
98.1
99.8
81.3
60.8
63.5
81.3
DANN+ELS
85.5
99.1
100.0
82.7
62.1
64.5
82.4
0.2
SDAT Rangwani et al.
2022
87.8
98.7
100.0
82.5
73.0
72.7
85.8
SDAT-ELS
88.9
99.3
100.0
83,9
74.1
73.9
86.7
1.1
0.5
0.0
1.4
1.1
0.9
ResNet50
ERM Vapmik
1999
75.8
95.5
99.0
79.3
63.6
63.8
79.5
ADDA
Izeng al
94.6
97.5
99.7
90.0
69.6
72.5
87.3
CDAN
Loogg
93.8
98.5
100.0
73.4
70.4
87.7
MCC
JiB ct
2020
94.1
98.4
99.8
95.6
75.5
74.2
89.6
DANN
100.0
DANNFELS
92.2
98.5
100.0
85.9
74.3
75.3
87.7
0.9
1.3
0.0
1.8
1.4
1.7
1.2
SDAT Rangwani
2022
92.7
98.9
100.0
93.0
78.5
75.7
89.8
SDATFELS
93.6
99.0
100.0
93.4
78.7
77.5
90.4
0.0
performance on the Office- Home dataset. Table Table ssow that with the simple smoothing trick, the performance of SDAT is consistently improved, and many of the domain pairs, the improvement is greater than 1 1% Besides, the ELS can also bring consistent improvement both with ResNet-18. ResNet-50, and ViT backbones. The average domain generalization results on other benchmarks are shown in Table We observe consistent improvements achieved by DANN+ELS compared to DANN and the average accuracy VLCS achieved by DANN+ELS (81.5%) clearly out- performs all other methods. See Appendix [ Multi-Source Domain Generalization performance, DG performance Rotated MNIST and Image Retrieval benchmarks.
Domain Generalization with Partial Environment labels. One of the main advantages brought by ELS is the robustness to environment label noise. As shown in Figure (3a] when all environment labels are known (GT), DANN+ELS is slightly better than DANN. When partial environment labels known, for example, 30% means the environment labels of 30% training data are known and others are annotated differently than the ground truth annotations, DANN+ELS outperform DANN by large margin (more than 5% accuracy when only 20% correct environment labels are given). Besides, we further assume the total number of environments is also unknown and the environment number is generated randomly. M=2 in Figure ((a) means we partition the training data randomly into two domains, which are used for training then. With random environment partitions, DANN+ELS consistently beats DANN by a large margin, which verifies that the smoothness of the discrimination loss brings significant robustness to environment label noise for DAT.
Continuously Indexed Domain Adaptation. We compare DANN+ELS with state-of of-the-art continu- ously indexed domain adaptation methods. cooperss the accuracy of various methods. DANN shows an inferior performance to CIDA. However, with ELS, DANN+ELS boosts the generalization performance by large margin and beats the SOTA method CIDA (Wang al. 2020). We also

---

Table The domain generalization accuracies (%) on VLCS, and PACS. DANN+ELS compared to DANN.
denotes improvement of
Algorithm
PACS
VLCS
Avg
V
IRM
ARROWSKY BI A... 2019
79.3
97.6204
75.9
84.6
97.605
64.7
09.7
766
77.2
DANN DANEX Au.
2010
85.4 1.2
83.1=08
9.
0.8
$6.1
98. 0.8
7322
22
7
80.8
ARM
85.0 1.2
8.4
a.
0.5
85.8
9
665
02.7
7..
77.8
Kew
DDG
Zhang & RUAM
8.
85.0 1.9
97.2
84.3
0.7
88.9
99.1
66.5
7.
80.9
0.6
80.0
DANNFFLS
87.8 0.8
83.8 LB
97.1
=
0.1
8144 1.3
87.5
99.1 0.3
7322
11
0.9
79.9 0.9
81.5
2
0.7
188
0.5
1.1
0.7
Table Accuracy (%) on Office-Home for unsupervised DA (with ResNet-50 and ViT backbone). SDAT+ELS outperforms other SOTA DA techniques and improves SDAT consistently.
Method
Backhone
A-C
A-P
A-R
C-A
C-P
C-R
P-
P-C
P-R
R-A
R-C
R-P
Avg
ResNet-50 (He ml
2016
34,9
50.0
58.0
37.4
41.9
46.2
38.5
31.2
60.4
53.9
41.2
59.9
46.1
43.7
CDAN
Long et a.
2018
49.0
69.3
74.5
54.4
66.0
68.4
$5.6
48.3
75.9
68.4
55.4
80.5
63.8
71.8
f-DAL
Acana e 01
A021
RoaNet-50
56.7
77.0
81.1
63.1
72.2
75.9
64.5
54.4
81.0
72.3
58.4
83.7
70.0
SRIX
Taing
52.3
76.3
81.0
69.5
76.2
78.0
53.8
81.7
76.3
57.1
85.0
71.3
SDAT (Rangwan: el
2022
57.8
77.4
82.2
66.5
76.6
76.2
63.3
57.0
82.2
75.3
62.6
85.2
71.8
SDAI-EES
58.2
79.7
82,5
67.5
77.2
77.2
64.6
57.9
82,2
75.4
63.1
85.5
72.6
0.3
1.0
0.6
1.0
0.9
0.1
0.3
0.8
TVT
Yang
2
74.9
86.6
89.5
82.8
87.9
88.3
79.8
71.9
85.5
74.6
90.6
83.6
SDAT (Ringwain 2012
CDAN Long e al 2013
VIT
70.8
62.6
87.0
82.9
90.5
87.2
85.2
79.2
87.3
84.9
89.7
87.1
84.1
77.9
63.3
70.7
88.7
90.6
83.1
88.3
63.5
75.5
92.1
90.8
79.3
84.3
SDAT-ELS
72.1
90.6
88.1
89.7
70.7
88.4
76.5
1.3
0.3
0.1
0.0
0.8
0.0
0.0
0.0
0.2
0.1
1.0
0.0
0.3
visualize the classification results Circle Dataset (See Appendix C.1.1 for dataset details). Figure shows that the representative DA method (ADDA) performs poorly when asked to align domains with continuous indices. However, the proposed DANN+ELS can get near-optimal decision boundary.
Generalization results on other structural datasets and Sequential Datasets. Table shows the generalization results NLP datasets, and Table 7 14 show the results genomics datasets. DANN+ELS bring huge performance improvement on most of the evaluation metrics, e.g., 4.17% test worst-group accuracy on CivilComments, 3.79% test ID accuracy RxRx1, and 3. 3.13% test accuracy on Table 15 and Table 18 where DANN works poorly but DANN+ELS brings consistent improvement and beats all baselines on the Opurious-Fourice dataset.
Generalization results on sequential prediction tasks are shown in
4.2 INTERPRETATION AND ANALYSIS
To choose the best y. Figure 3(b) visualizes the best y values in our experiments. For datasets like PACS and VLCS, where each domain will be set as target domain respectively and has one best y, we calculate the mean and standard deviation of these y values. Our main observation is that, as the number of domains increases, the optimal will also decrease, which is intuitive because more domains mean that the discriminator more likely to overfit and thus needs a lower to solve the problem. An interesting thing is that in Figure 3(b) PACS and VLCS both have 4 domains,
Table 5: Rotating MNIST accuracy (%) at the source domain and each target domain. X denotes the domain whose images are Rotating by [X [x",x 45%].
Rotating MNIST
Algorithm
(Source)
45
90'
135
180
225'
270
315
Average
ERM [Vapnik
1999
26.8
31.6
35.1
37.0
28.6
76.2
45.0
ADDA
Tzeng el al
2017
97.6
70.7
22.2
32.6
38.2
31.5
20.9
65.8
40.3
DANN
Ganin et a
2016
98.4
81.4
38.9
35.4
40.0
43.4
48.8
77.3
52.1
CIDA
Wang
2020
99.5
80.0
33.2
49.3
50.2
51.7
54.6
81.0
57.1
DANN+ELS
98.4
81.4
55.0
39.9
43.7
45.9
53.7
78.7
62.1
16.1
4.5
3.7
2.5
4.9
1.4
10.0

---

Table 6: Domain generalization performance on neural language datasets. The backbone is and all results are reported over random seed runs.
Amazon- Wilds Val 10% Acc
Algorithm
Val Avg Ace
Test Avg Acc
Test 10% Ace
Val Worst-group
Test Worst-group
HRM Vapelk 1999
0.1
71.9
56.2 0.7
53.8
2
Group DRO
Saewwa e
2019
70.7 0.6
70.0 0.5
51.7 0.0
53.3 0.0
54.2 0.3
6,3 0.2
CORAL
San
=2.0 0.3
71.1 0.3
54.7 0.0
0.8
30.0
a.1 0.1
IRM
(Ajoekky et
70.5
32.2
Kewojelt
69.1
68.6 0.6
52.1
$2.0
34.9 1.2
9.1
DANN
Ganin al
2016
72.1=02
=
54.6
4.4 1.3
8.0 0.0
715
5388
0.2
0.2
0.1
0.9
0.5
14
CivllComments-Wlld
Algorithm
Val Avg Ace
Val Worst-Group Acc
Test Avg Acc
Test Worst-Group Acc
Group DRO Sagawa et al. 2019
90.4 = 0.4
65.0 = 3.8
90.2 = 0.3
69.1 1.8
Reweighted
90.0 = 0.7
63.7 2.7
89.8 a: 0.8
66.6 1.6
IRM (Arjovsky ct al 2019
89.0 = 0.7
65.9 2.8
88.8 = 0.7
66.3 : 2.1
ERM Vapmi 1999
92.3 +: 0.2
50.5 1.9
92.2 a 0.1
56.0 3.6
DANN (Ganin of al. 2016
87.0 0.3
64.0 2.0
87.0 0.3
61.7
DANN+ELS
= 0.4
66.0
1.9
1.4
4.3
Table 7: Domain generalization performance genomics dataset, RxRxI.
RxRxI-Wilds
Algocithm
Val Ace
Test ID Aec
Test Ace
Val Worst-Group Acc
Test ID Worst-Group Acc
Test Worst-Group Ace
ERM (appik 1998
19
0.4
2.4
1.12
103
1.3
IRM
Anursy al
06
1.
8.
19
1.2
DANK mann BI Bl. 2016
12.7 02
229+0.1
19.2+
1.0 6.1
4.6 -.6+004
3.600.0
DANNIFLS
14.1
26.7+0.1
21.2
7.2
4.2
0.1
0.6
but VLCS needs a higher Figure 6 shows that images from different domains in PACS are of great visual difference and can be easily discriminated. In contrast, domains in VLCS do not show significant visual differences, and it is hard to discriminate which domain one image belongs to. The discrimination difficulty caused by this inter-domain distinction another important factor affecting the selection of 7.
Annealing 7. To achieve better generalization performance and avoid troublesome parametric
searches, we propose to gradually decrease y training progresses, specifically, ? 1.0 M-1 where t. T' are the current training step and the total training steps. Figure B[c) shows that annealing y achieves comparable or even better generalization performance than fine-grained searched 7. Empirical Verification of theoretical results. We use the PACS dataset as an example to empirically support our thcoretical results, namely verifying the benefits to convergence, training stability, and generalization results. In Figure 4 ''' is set as the target domain and other domains
M T
82.5
Method
0.8
80.0
DANN-ELS
Optinnll
0.6
27.5
75.0
0.4
22.5
70.0
VLCS PACS MS Civil Raaxi OGB
DANN
DANN-ELS DANN-ELS
20%6 30%
50%
M-2 M-3
GT
120,04
(Opeinal (Amesling Method
Environmant Label Noise
(b)
(c) Figure 3: (a) Generalization performance DANN+ELS compared DANN with partial correct environment label on the PACS dataset (P as target domain). (b) The best r for each dataset. Civil the CivilComments dataset and OGB is the OGB-MoIPCBA dataset. (c) Average generalization accuracy on the PACS dataset with different smoothing policies.
8

---

1.0
DANNAELS
0.8
0.8
Clarchemto Lass
1.00
0.6
sarget
0.75
0.6
Avg
0.2
DANN
DANN
5000
(a) Classification loss.
(b) Avg accuracy of source domains.
(c) Acc on the target domain.
Figure 4: Training statistics on PACS datasets. Alternating GD with ni = 5, fg = is used. All other parameters setting the same and only the default hyperparameters and without the fine-grained parametric search.
as sources. Considering ELS, we can see that in all the experimental results, DANN+ELS with appropriate y attains high training stability, faster and stable convergence, and better performance compared to DANN. In comparison, the training dynamics of native DANN is highly oscillatory, especially in the middle and late stages of training.
5 RELATED WORKS
Label Smoothing and Analysis is technique from the 1980s, and independently re-discovered by (Szegedy et al. 2016). Recently, label smoothing is shown to reduce the vulnerability of neu- ral networks Caard-FFaree & Goodfellow) (016) and reduce the risk of adversarial examples in GANs (Salimans et al. 2016). Several works seek to theoretically or empirically study the effect of label smoothing. (Chen et al. 2020) focus on studying the minimizer of the training error and finding the optimal smoothing parameter. (Xu et al. 2020) analyzes the convergence behaviors of stochastic gradient descent with label smoothing. However,
we know, no study focuses on the effect of label smoothing on the convergence speed and training stability of DAT.
Domain Adversarial Training (Ganin al. 2016) using domain discriminator distinguish the source and target domains and the gradients of the discriminator to the encoder are reversed by the Gradient Reversal layer (GRL), which achieves the goal of learning domain invariant features. (Schoenauer-Sebag et al., 2019; Zhao et al.) 2018) extend generalization bounds in DANN (Ganin et 2016) to multi-source domains and propose multisource domain adversar- ial networks. (Hu et 202T incorporates the prototypical features into DAT to achieve semantic domain aligmment. (Acuna et al. 2022) interprets the DAT framework through the lens of game theory and proposes to replace gradient descent with high-order ODE solvers. (Rangwami et al. 2022) finds that enforcing the smoothness of the classifier leads to better generalization on the target domain and presents Smooth Domain Adversarial Training (SDAT). The proposed method is orthogonal to existing DAT methods and yields excellent optimization properties theoretically and empirically.
For space limit, the related works about domain adaptation, domain generalization, and adversarial Training in GANs are the appendix.
CONCLUSION
In this work, we propose a simple approach, iee, ELS, to optimize the training process of DAT methods from an environment label design perspective, which is orthogonal to most existing DAT methods. Incorporating ELS into DAT methods is empirically and theoretically shown to be capable of improving robustness noisy environment labels, converge faster, attain more stable training and better generalization performance. As far as we know, our work takes a first step towards utilizing and understanding label smoothing for environmental labels. Although ELS is designed for DAT methods, reducing the effect of environment label noise and a soft environment partition may benefit all DG/DA methods, which is promising future direction.

---

7 ACKNOWLEDGEMENT
This work was partially funded by the National Natural Science Foundation China (Grant No. 62276256, 62076078), the Beijing Nova Program under Grant Z21 7 10010010022211111 108, and the National Natural Science Foundation China (62236010, 61721004, and U1803261)
REFERENCES
David Acuna, Guojun Zhang, Marc T Law, and Sanja Fidler. f-domain adversarial learning: Theory and algorithms. In International Conference Machine Learning, pp. 66-75. PMLR, 2021.
David Acuna, Mare T Law, Guojun Zhang. and Sanja Fidler. Domain adversarial training: A game perspective. ICLR, 2022.
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv: /911. 1900004 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:101.00462. 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp.
PMLR, 2017.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02890 2019.
Sanjeev Arora, Rong Ge, Yingyu Liang. Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference Machine Learning, pp. 224-232. PMLR, 2017.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NIPS, 2006.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 2010.
Dimitri P Bertsekas. Nonlinear programming. In thena scientific Belmont, 1999.
Gilles Blanchard, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. J. Mach. Learn. Res., 2021.
Blair Chen, Liu Ziyin, Zihao Wang, and Paul Pu Liang. An investigation of how label smoothing affects generalization. arXiv preprint 2000:2000., 2020.
Peixian Chen, Pingyang Dai, Jianzhuang Liu, Feng Zheng, Qi Tian, and Rongrong Ji. Dual distribution alignment network for generalizable person re-identification. AAAI Conference on Artificial Intelligence, 2021.
Xu Chen, Jiang Wang, and Hao Ge. Training generative adversarial networks via primal-dual subgradient methods: a lagrangian perspective on gan. arXiv preprint vrXiiv/802.011 2018.
Seokeon Choi, Taekyung Kim, Minki Jeong, Hyoungseob Park, and Changick Kim. Meta batch- instance normalization for generalizable person re-identification. In Computer Vision and Pattern Recognition (CVPR), 2021.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. ICML, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Farzan Farnia and Asuman Ozdaglar. Do GANs always have Nash equilibria? In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 Proceedings of Machine Learning Research, pp. 3029 3039. PMLR, 13-18 Jul 2020. URL.tttea htttps:/pproceed: ngs ml www.spppww...... http://ppppoppoooooooooooee//aaa
html
10

---

Cean-Chrittophe Gagnon-Audet Kartik Ahuja, Mohammad- Marvishi-Bayart Guillaume Du- mas, and Irina Rish. Woods: Benchmarks for oitt---distribution generalization time series tasks. arXiv preprint (rXxv:220300977: 2022.
Zhe Gan, Liqun Chen, Weiyao Wang, Yuchen Pu, Yizhe Zhang, Hao Liu, Chunyuan Li, and Lawrence Carin. Triangle generative adversarial networks. Advances in neural information processing systems, 30, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 2016.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In ICCV, 2015.
Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. varia- tional inequality perspective on generative adversarial networks. arXiv preprint 1rXiv:1802.100 1005,1051, 2018.
Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks. WiXiv:1701.00100 2016.
arXiv preprint
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
D. Gray, S. Brennan, and H. Tao. Evaluating Appearance Models for Recognition, Reacquisition, and Tracking. Proc. IEEE International Workshop on Performance Evaluation for Tracking and Surveillance (PETS), 2007.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Hirzer, Csaba Beleznai, Peter M Roth, and Horst Bischof. Person re-identification by descriptive and discriminative classification. In Scandinavian Conference on Image Analysis, 2011.
Dapeng Hu, Jian Liang, Qibin Hou, Hanshu Yan, and Yunpeng Chen. Adversarial domain adaptation with prototype-based normalized output conditioner. IEEE Transactions on Image Processing, 30: 9359-9371, 2021.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Simon Jenni and Paolo Favaro. On stabilizing generative adversarial training with noise. In Proceed- ings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12145-12153, 2019.
Jieru Jia, Qiuqi Ruan, and Timothy M Hospedales. Frustratingly easy person re-identification: Generalizing person re-id in practice. arXiv preprint arXiv:/905.0:44 2019.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Less confusion more transferable: Minimum class confusion for versatile domain adaptation. In ECCV, 2020.
Hassan K Khalil. Non-linear Systems. Prentice-Hall, New Jersey, 1996.
Youngdong Kim, Junho Yim. Juseung Yun, and Junmo Kim. Niul: Negative learning for noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 101-110, 2019.
11

---

Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips. Irena Gao, Tony Lee, et al. Wilds: benchmark of in-the-wild distribution shifts. In ICML, 2021.
David Krueger, Ethan Caballero, Joern-Henrk Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In ICML, 2021.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017a.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017b.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI, 2018a.
Wei Li and Xiaogang Wang. Locally aligned feature transforms across views. In Computer Vision and Puttern Recognition (CVPR), June 2013.
Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. Deepreid: Deep filter pairing neural network for person re-identification. In Computer Vision and Pattern Recognition (CVPR), June 2014.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao.
Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018b.
Chunxiao Liu, Shaogang Gong, Chen Change Loy, and Xinggang Lin. Person re-identification: What features are important? In European Conference on Computer Vision (ECCV). Springer, 2012.
Mingsheng Long, Zhangjic Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. Advances in neural information processing systems, 30, 2017.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICMI,, 2018.
K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representa- tion. In ICML, 2013.
Vaishnavh Nagarajan and J Zico Kolter. Gradient descent gan optimization is locally stable. Advances in neural information processing systems, 30, 2017.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets. Advances in neural information processing systems, 30, 2017.
Weili Nie and Ankit B Patel. Towards a better understanding and regularization of gan training dynamics. In Uncertainty in Artificial Intelligence, pp. 281-291. PMLR, 2020.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Yunchen Pu, Shuyang Dai, Zhe Gan, Weiyao Wang, Guoyin Wang. Yizhe Zhang, Ricardo Henao, and Lawrence Carin Duke. Jointgan: Multi-domain joint distribution learning with generative adversarial nets. In International Conference on Machine Learning, pp. 4151-4160. PMLR, 2018.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. arXiv preprint 1rXiv:2109.02934, 2021.
12

---

Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and R Venkatesh Babu. A closer look at smoothness domain adversarial training. ICML, 2022.
Kevin Roth, Aurélien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In NIPS, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226 Springer, 2010.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.0873 2019.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Computer Vision and Pattern Recognition (CVPR), 2018.
Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping, 38(1 1):::39-352020 2017.
Alice Schoenauer- Sebag. Louise Heinrich, Mare Schoenauer, Michele Sebag. Lani Wu, and Steve Altschuler. Multi-domain adversarial learning. arXiv preprint arXiv::903.092: 2019.
Florian Schäfer, Hongkai Zheng, and Anima Anandkumar. Implicit competitive regularization in gans, 2019.
Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised map inference for image super-resolution. arXiv preprint
2016.
Jifei Song, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy M. Hospedales. Generalizable person re-identification by domain-invariant mapping network. In Computer Vision and Pattern Recognition (CVPR), June 2019.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Springer, 2016.
European conference on computer vision, pp.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818 18--8826, 2016.
Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8725-8735. 2020.
Chenyang Tao, Liqun Chen, Ricardo Henao, Jianfeng Feng, and Lawrence Carin Duke. Chi-square generative adversarial network. In International conference machine learning, pp. 1887-4896. PMLR, 2018.
Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh. Improving generalization and stability of generative adversarial networks. arXiv preprint /xXiv:1902.03984 2019.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011.
13

---

Quan Hoang Trung Le, Hung Vu, Tu Dinh Nguyen, Hung Bui, and Dinh Phung. Learning generative adversarial networks from multiple data sources. In Proceedings of the 28th International Joint Conference Artificial Intelligence, 2019.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7167-7176, 2017.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017.
Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. ICML, 2020.
Jindong Wang and Wenxin Hou. Deepda: Deep domain adaptation toolkit. https://github com/
DeepDA
David Warde-Farley and Ian Goodfellow. 11 adversarial perturbations of deep neural networks. Perturbations, Optimization, and Statistics, 311:5, 2016.
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. ICML, 2022.
Zheng Wei-Shi, Gong Shaogang, and Xiang Tao. Associating groups of people. In British Machine Vision Conference (BMVC), 2009.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et Huggingface's transformers: State-of-the-are natural language processing. arXiv preprint arXiv:/910.03771, 2019.
Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. End-to-end deep learning for person search. arXiv preprint arXiv: 1604. 160.001850 2(2), 2016.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? arXiv preprint arXiv:1810.00820 2018.
Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, and Rong Jin. Towards understanding label smoothing. arXiv preprint arXiv:2006. 1653, 2020.
Abhay Yadav, Sohil Shah, Zheng Xu, David Jacobs, and Tom Goldstein. Stabilizing adversarial nets with prediction methods. arXiv preprint arXiv:1705.07364 2017.
Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. Tvt: Transferable vision transformer for unsupervised domain adaptation. arXiv preprint 1.Xiv:2108.05980 2021.
Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain generalization. arXiv preprint 11. 2021a.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. NewrIPS, 2021b.
Yi-Fan Zhang, Zhang Zhang, Da Li, Zhen Jia, Liang Wang, and Tieniu Tan. Learning domain invariant representations for generalizable person re-identification. arXiv preprint arXiv:2103.15800 2021c
Yi-Fan Zhang, Hanlin Zhang, Jindong Wang, Zhang Zhang, Baosheng Yu, Liang Wang, Dacheng Tao, and Xing Xie. Domain-specific risk minimization. arXiv preprint M0Xiv:2208.08666 2022a.
YiFan Zhang, Feng Li, Zhang Zhang, Liang Wang, Dacheng Tao, and Tieniu Tan. Generalizable person re-identification without demographics, 2022b. URL https: http::///ppppeeee..eeoo net/ forum? d=VNdFPD5wqghh
14

---

YiFan Zhang, Hanlin Zhang, Zachary Chase Lipton, Li Erran Li, and Eric Xing. Exploring transformer backbones for heterogeneous treatment effect estimation. In NeurIPS ML Safery Workshop, 2022c.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-7413, PMLR, 2019.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In NeurIPS, 2018.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re- identification: benchmark. In International Conference Computer Vision (ICCV), December 2015.
Zhedong Zheng. Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In International Conference on Computer Vision (ICCV), Oct 2017.
15

---

Appendix
CONTENTS
1
Introduction
2
Methodology
3
Theoretical validation
3.1
Divergence Minimization Interpretation
3.2
Training Stability
3.3
ELS meets noisy labels
3.4
Empirical Gap and Parameterization Gap
3.5
Non-Asymptotic Convergence
Experiments
4.1
Numerical Results on Different Settings and Benchmarks
4.2
Interpretation and Analysis
5
Related Works
Conclusion
7
Acknowledgement
10
Proofs of Theoretical Statements
17
A.I Connect Environment Label Smoothing to JS Divergence Minimization
17
A.2 Connect One-sided Environment Label Smoothing to JS Divergence Minimization
19
A.3 Connect Multi-Domain Adversarial Training KL Divergence Minimization
20
A.4 Training Stability Brought by Environment Label Smoothing
21
A.5 Training Stability Analysis Multi-Domain settings
23
A.6 ELS stabilize the oscillatory gradient
24
A.7 Environment label smoothing meets noisy labels
A.8 Empirical Gap Analysis Adopted from Vapnik-Chervoneele framework
25
A.9 Empirical Gap Analysis Adopted from Neural Net Distance
26
A.10 Convergence theory
27
B Extended Related Works
30
C Additional Experimental Setups
31
C.1 Dataset Details and Experimental Settings
C.2 Backbone Structures
16

---

D Additional Experimental Results
34
D.I Additional Numerical Results
D.2 Additional Analysis and Interpretation
34
D.3 Ablation Studies
A PROOFS OF THEORETICAL STATEMENTS
The commonly used notations and their corresponding descriptions are concluded in Table
A.1 CONNECT ENVIRONMENT LABEL SMOOTHING TO JS DIVERGENCE MINIMIZATION
To complete the proofs, we begin by introducing some necessary definitions and assumptions.
Definition 1. (H "F--divereence (Ben-Davien el al. (006]). Given two domain distributions Ds, DT over X, and hypothesis class H. the H-divergence between Ds, DT is
dH (Ds. DT) = sup Dx [h(x)=xxx)]])]x))>( = 1] (x(x))
(5)
heMl
Definition 2. (Empirical --divergence (Ben-David et al. 2006 For an symmetrie hypothesis class
H. one can compute the empirical H-ddivereecce between two empirical distributions 'Dg and Dr by computing
1 m [[)()))
=0] 0]+ + [((()) (11(xx)=11] =
dx Dr) =2
min
(6)
where m,n is the number of data samples Dy and Di respectively and [[a] is the indicator function which is I predicate a is true, and otherwise.
Vanilla DANN estimating the "min" part of Equ. 66) by domain discriminator, that models the probability that given input is from the source domain or the target domain. Specially, let the hypothesis h be the composition of h å g. where H. is additional hypothesis and g G pushes forward the data samples to a representation space Z. DANN (Ben-David et al. 2006) seeks to approximate the H-divergence of Equ. (6)
max dh (Ds; max Ex log h g(x.) Exy-DY log h (xx)), heft
(7)
hetl
Table 8: Notations.
Symbol
Description
Ds, Dr, Dj
Distributions for source domain, target domain, and domain
Ds. Dz, Dj
Empirical distributions for source domain, target domain, and domain Density functions for source domain, target domain, and domain i. Data samples from source domain, target domain, and domain Feature distributions of Dss, Dr, DD; respectively.
Ps.Pe,Pi
X,, Nr.
Di. D..D;
which also termed g DDs, g Dr. g Di Density functions for D3 D D respectively. Data samples from D D;, D:
P..rt.P
2i, Z;
H. H.G.
Support sets for hypothesis. discriminator, and feature encoder. Hypothesis, discriminator, the optimal discriminator, and feature encoder. Number of training distributions, number of data samples in D,. plyper-parameter for the environment label smoothing. H-divergence and Empirical -ddivergence.
h. h,h,h"g
M,n:
7
dn. dr
17

---

where the sigmoid activate function is ignored for simplicity, h g(x) is the prediction probability that is belonged D3 and 1 - h g(x) is the prediction probability that x is belonged Dr. Applying environment label smoothing, the target can be reformulated to
maxdi.r. (Ds, Dr) he7l
max DD log g(x,) + (1 7)) log - g(xx))]
he
(8)
Ex [[1 7) log g(xx) + log h g(x:)))
When é {0, 11, Equ. 8 is equal to Equ. 7 and no environment label smoothing is applied. Then we prove the proposition
Proposition
1 Suppose h the optimal domain classifier with no constraint and mixed distri-
Dg = yDs+ +0s++11100 (1-
butions
Dr yDr (1-++17)) D) with hyper-parameter % then max di hhan (Ds; DT)
(Ds (rr)) 2log 2, where DJs the Jensen-Shanon (JS) divergence.
Proof. Denote the injected source/target density as P. == gop.. == gopt, where Ps: Pt is the density of Ds, Dr respectively. We can rewrite Equ. (8)
dign (Ds: [ 15 v5(2) log[>logh())+ (1 log (1-
(9)
(1=))(1 h(z) 7log [x))]
We first take derivatives and find the optimal :
Ødi,g, hgg (Ds,
7 {xx) +11 -1
(1 (1- log 1 -1
(z)
1 h())) (z)
7(2)) +7 7(z)
ah(z)
ri(zz) ((
(1- h(z)) -h(z))=
= p((z)) - 1
(z) (((0))+(((((==pp(()
(z pf (z)
(10)
For simplicity, use Pa. denote (2), pf(2) (z) respectively and ignore the fz. Plugging Equ. into Equ. 8 we can get
(00
[ Ps y yog
Pt + >p+ Pi)
(P: P.)
max h.g., (Ds, DT) heft
+ (1 - log
De + Pt
Ps + Dt
Pt + (ps - Pt)
Ps ( - Pa)
+pr (1 (1-7)log log
+ log
de
Ps + Pt
Pe + Pt
f ps log
Ps + Tpp - Ps)
Pt. + {ps - pt)
+ pt log
Do Dt.
Ps pt.
7pp PPt)
Ps + + Po) dz
PoT log
pr.
P. 7 - Ds)
Dt + 7P, - Pi)
(11)
18

---

pv Ps- + (1
Let
distribution densities that the convex combinations of Pa.Pr.
Pt + (1 p>
P3 2P, 2
have
Pt Pp (7-1)p2 2y-1
29-1
and Ps Pe Pa Pc. Then
in Equ.
(11 can be rearranged
Pe
Ps
9-1
Pr
Ps
Po p log
log
+
Prr log
Ps log
27 1
Ps pe
Ps
Pr
27
Ps pr
ps
pe
7
Py log
Pv
+ Ps log
ps'
Pe
log
Po'
+ Pe log
Ps'
7
Pr log
Pe
+ pd log
Ps'
24
Ps Pr
Pr
Pv
P + Pe
27
Pa + pv
Ps + P
Pv
Pd
7
Pd
Pv
= Pr log
+ Ps log
Pst log
+
Pr log
Po Pr
Pst pe
2 1
Pv
Pe
2 Pr log
2pr
ps log
2p
2log?
7
ps
log
Pet
+ pe log
Pe Ps
Pv + pe
Ps + Pr
27
Pe
= 2s (Dss((DsllDrg - log 2
(pr pv log
Ps'
27
Pr
(12)
2
in Equ.
can be rearranged to 11
Por
Ps
log
Pt log
Dr.
Ps'
y
log
Ps'
Yps (r
1)pr
YPP + (7 1)pp'
(13)
Pr
27 -
24 -
Pd
(por Pe)
log
27
Pe
By plugging the rearranged
1 and
2 into Equ.
11
we get
max d di.g.r (Ds,Dr) ( 2Dss her
- 2og
(14)
A.2 CONNECT ONE-SIDED ENVIRONMENT LABEL SMOOTHING TO JS DIVERGENCE MINIMIZATION
Proposition 2.
Given domain distributions Ds, Dr over X, where Ds is the read data distribu- tion and Dr is the generated data distribution. The cost used for the discriminator is:
max di (Ds, DT) REM
max Ex,-Ds log ((x(((x)) ) Ex log (1 -((x))),
(15)
he'l
where h H. X mixed distributions
[0. Suppose h E H. the optimal discriminator with no constraint and
D:s = 7Ds
'DT Dr (1 (1-7)Ds with hyper-parameter 7/ Then to minimize domain
divergence by adversarial training with one-sided environment label smoothing is equal to minimize (Dss(Ds>\(rr)) 2oog where Das the lensen-Shanon (JS) divergence.
Proof. Applying one-sided environment label smoothing, the target can be reformulated to
max (n, Dr) max Ex,-Dy ( (rlog/((x.) + +(1 7) log (1 (xxx))) Ex,-D, [log (1 l(xx))) helt
heil
[ px(x)) log [ log h -)) log h(x)) (x) log(1 h(x)) heht
max
(16)
where 7 is a value slightly less than one, Ps (x). Pr (x) is the density of Ds, Dr respectively. By taking derivatives and finding the optimal h get h² (x)= Plugging the optimal h
(x)(x)
19

---

into the original target can get:
-x px())) ( log
(x)
(11-)) log Pt(x) (1 (>)
+x):x) log pr(x) (1-77ps(xx) dx
(x(x) + px(xx)
(x(x) + pr (x(
p,(x) px(x)
= Jxx Pe(x)) log
YPP (x)
+ +[pa(1-9) px)) log Px(x) (1)))p(xx) dx
(x) + (x(
(x px)(x)
Per (x)
Pe (x)
= Px log
+ +pr(x) log (x( Pr (x)
dx
(x) + (x)
=2D,s (D (Dsr11DT.)
log 2,
(17)
DD yDs {
where
mixed distributions and
Ps = 7Pp Pv = Pt + (1-7)ps
are their
Dr - Dr (1 s)
densities.
Our result supplies an explanation to "why GANs only use one-sided label smoothing rather than native label smoothing". That is, if the density of real data in region is near zero (x 0, native environment label smoothing will be dominated by only the generated sample densities because P. = Pt (P: -Pt) (1-r)pt Namely, the discriminator will not align the distribution be- pr p. (P: ps) 7P:
tween generated samples and real samples, but enforce the generator to produce samples that follow the fake mode Dr. In contrast, one-sided label smoothing reserves the real distribution density as far as possible, that is, P,- = PP rp:, which avoids divergence minimization between fake mode to fake mode and relieves model collapse.
A.3 CONNECT MULTIIDOMAII ADVERSARIAL TRAINING TO KI DIVERGENCE MINIMIZATION
M Proposition 3. Given domain distributions (D i-1 over X, and hypothesis class H. Suppose h e H the optimal discriminator with no constraint and mixed distributions DD DMi = Di, and (Dę- 1-y MM
yD,+ M-1 Lj-lijei DYM with with hyper-parameter 7 [0.5, 1]. Then to minimize domain divergence by adversarial training w/wo environment label smoothing is equal to minimize ΣM D (Dil(DDi and ΣM DKL (Drl respectively, where DEL is the Kullback-Leibie (KL) divergence.
Proof. We restate corresponding notations and definitions as follows. Given M domains {D, M Let the hypothesis h be the composition of /t- = h g. where g & pushes forward the data samples to a representation space Z and the domain discriminator with softmax activation function is defined as = ()) ...., ..,,,,,,.,,... (-) E H Z [o, 1] M; EM (.) 1. Denote D; the feature distribution of D; which encoded by encoder g. The cost used for the discriminator be defined
M Σ
max di.g (DI.... .M)) = -max Ez-Do log hu (z) + +...+ Ex- log (2), s.t. heft
(1(z)=)
heHt
(18)
Denote pf (z) the density of feature distribution g o D;. For simplicity, we ignore Lg. Applying
lagrange multiplier and taking the first derivative with respect to each hn, we can get
0dp. = (z)) 1 0
{ hi(z) = pi(x)
[i(z))
pi(x)
th
h1(2)
((())-++Pi,(*)
{ ha)(2)= = Pi)(x)
adp.e
(x.(x)
(z) h>)(()
A=
ahst
([(x)+++p$c
(19)
where À is the lagrange variable and 1 is because the constraint ΣM hz(z) 1. Denote D M
Ee DD; is mixed distribution and PMix
ΣM Pi is the density. Then we have
20

---

max hh (D (DI...., DDs) = f2a pz)) log pi(z) M
Pisse (z) + +z)(x) log pá(z)
Pisie (z) + + +++ris(2 (2) log Pis (z) Pisce (z)
d2
he7t
= Σ DKL
(20)
where DRL is the KL divergence. With environment label smoothing, the target is
(1- Σ
max d (Di....,
= max go
y Io (())
log (h,( +...
he7t
he7l
M - j-1:41
ylog ylogher (ogghar()) + (1-7)
Dx-
Σ log (z)(
,.s Σ hz(z) =1
M- - jolueM
1=1
(21)
Take the same operation as Equ.
19 we can get
0d
8b1
(2) h1(=)
+
M-
1-2
1 - Σو (2)
h1(=)
hi(z)= (x)(2)) 1-y Ep (x)
pf (x)) (x)
>pps (2) h+s(2)
1-7 MM
2 Pj 2)) har(s) A= 0
(z) = (x) M-T Σ M
8hus
M-I
+ pi
(22)
Denote (Dv 7D, + 1-7
1 DM set of mixed distributions and (pr(z) yp (z) 1-7 Σ jj-lsjs (z) MM the corresponding densities. Plugging Equ. (22) to the target we can get M-1
M-1
M
Je TPt (z) log (2) 1-2 M
M-1 E
(1-?)
Σ (z) log TPE(2) + 1o ΣM Ei-1o+
M
dg
(z) + + Pir (z)
M
pī(z) +...
+ PPrs
k=1:kei
J mp (z) log (z) + 1- M
M-1 2j (z
(1 - 7)
M
(p((z) + 1-7 Σ Ej Pj (z(
Σ pt
M-1
dz
(z(z) + + pis
M - k-1:kei
(z( + ... p5s 2z)
M
f& (x) + (1-7)
M
(z)
M-1 2j=1.jei DJ (z)
1-y MM
Σ
pī(z) log
dr
M-1
p. (z) + ... PM
k=1:k=s
Σ DxL(Dx,(DMak))
(23)
A.4 TRAINING STABILITY BROUGHT BY ENVIRONMENT LABEL SMOOTHING
Let Ds. Dr two distributions and Dj. Djj, their induced distributions projected by encoder g : X + Z over feature space. We first show that if D. D are disjoint or lie in low dimensional manifolds, there is always a perfect discriminator between them.
Theorem 1. (Theorem 2. in (Arjovsky & Bottou 2017).) If two distribution D's, Dr. have support contained on two disjoint compact subsets M and P respectively, then there is smooth optimal discriminator h² : Z + [0, that has accuracy 1 and V vah²' (z) for all z ~ MUP.
Theorem 2. (Theorem 2.2. in (Arjovsky & Bottou 2017).) )) Assume two distribution Dis, Dj have support contained in two closed manifolds M and that don't perfectly align and don't have full dimension. Both Di, Dj- are assumed to be continuous in their respective manifolds. Then, there is smooth optimal discriminator h² Z > [0, 1] that has accuracy and for almost all -MPP, h smooth in neighbourhood of z and V 2, (z) O.
Namely, if the two distributions have supports that are disjoint or lie on low dimensional manifolds, the optimal discriminator will be accurate on all samples and its gradient will be zero almost everywhere. Then study the gradients we pass to the generator through discriminator.
21

---

Proposition 4. Denote g(f:-): x Z differentiable function that induces distributions D's. D with parameter and h a differentiable discriminator If Theorem
1 or holds, given a e-optimal assume the Jacobian matrix of
discriminator h, that is supzez ll v.h(z) ||2 +lh(z) (z) g(0;x) given is bounded by supxex [II ())((x))) [12] s C, then we have
lim voda, (Ds, 1|2=0 0
(24)
c-0
lim || go (Ds, Dr) 2| 2(1 7)
(25)
c
Proof. Theorem 1 or show that in Equ. (8). ht is locally one on the support of D and zero on the support of Dj. Then, using Jensen's 's inequality, triangle inequality, and the chain rule on these supports, the gradients we pass the generator through discriminator given x, Dg is
Vollx, - Ds
+ (1->) log (1 h g(0:xx))] ll2
Ex,-DD:
|l Vorlogho [1z]] Ex,-Ds [|] Vo -) Ve(1->)
log (1 h p(fxx.))) [|a]
Voho g(; 1|
-xx (1-7) V g(;xx,) 1|2
Ex
(0:))(:
|| h g(0 xx)
Vzh(z) 112ll Jo(g((;x.)) 112
(1-7) V Vzh(z) l12ll Ja(o((:x,)) ll2 || h g(0; x.)
$ Ex
+ EX.-DS
g x.)
6 (o(((xx)) 1l2
(0;00 (xxx)))) ll2 |1 - h g(00 x.) +
[o(
<DDBx,-Ds
lh+ ((0xx.) -c (1 (1-7) (xx -Ds
cC $ 1 (1-7)C,
(26)
where the fifth line is because we have h(x) 1ht((" (=) -e when is small enough and vzh(z) 11 e. Similarly we can get the gradients given X2 Dr is
|| v glEx Dr [[1 -1-))) 7) log h g(x:) + + log (1 - h gxx:))] ll2
E ll (o(g(()x))) ll2
(s( ll2
(1 (11-7) Ex,
+ +xBxx-Dr
Ih- g(0; x1) ec
|1 1 h o g 0 x,) -
(27)
EC s (11 (1-~) 7)- 1-c
Here h(z) hr (2) + because ht is locally zero on the support of Dz. Then we have
lim || Vod (Ds, Dr) ll2 c-0
lim || Volx,-Ds [ log h g(0;x,) (1 log
log goo llz
+ || Volx,-D; Dr [11 7) log h (xxx) + 7 (1 (1- h g(xx)) ll2
C
C
(28)
lim ạ
+ 1-6
+ (1- 21-
c-0
1-6
2(1 C,,
where 1 is equal to the gradient of native DANN in Equ. 7 times y, namely
lim |l Vod (Ds, Dr) || 0, -
(29)
'The constraint on vxl(() || is because the optimal discriminator has zero gradients almost everywhere, and [f))) h" (x)) constraint on the prediction accuracy.
22

---

which shows that discriminator gets better, the gradient of the encoder vanishes. With environment label smoothing, we have
lim ll Vadk (Ds, Dr) 1|2- -7)- c-0
(30)
which alleviates the problem of gradients vanishing.
A.5 TRAINING STABILITY ANALYSIS OF MULTI-DOMANE SETTINGS
Let (D:) a set of data distributions and {D M their induced distributions projected by encoder E=
g: Z over feature space. Recall that the domain discriminator with softmax activation function defined as h (hI.....hms e H Z [0. 1]M² where hz(z) denotes the probability that belongs to D. To verify the existence of each optimal discriminator hū, we can easily replace D5, Dj in Theorem 1 and Theorem P by DJ, DiE Dj respectively. Namely, if distribution Di and ΣM Ej=1:j4; Dj have supports that are disjoint or lie on low dimensional manifolds, hū can perfectly discriminate samples within and beyond D and its gradient will be zero almost everywhere.
Proposition 5. Denote g(f:-) :X + Z a differentiable function that induces distributions {D with parameter 0, and (ha corresponding differentiable discriminators. If optimal discriminators for induced distributions exist, given any e-optimal discriminator hú, we have supwez Vzh,(z) ll +lhx(z) hī (z)l < e, assume the Jacobian matrix of g(0; x) given x is bounded by supxex LIl Jo(g((:x)) l1a] s C, then we have
lim (od (DI, (DI,..,,DM) 112=0
(31)
lim Vod (DI...., Dwr) 11 M(1
(32)
Proof. Following the proof in Proposition 4. we have
(1-7) Σ
lim
|| VolxED
logh, o g(x) + M 1 j=l;jei
log (h, o g(x)) 1l2
-0
|| Vah, g(f:x) l12
+ (1 (1- 7)
M
Vohy g(0xx) l1a
s lim Ex-D, c+0
Ex-D, M j=l:ja
Σ
lh, o g(f:x)
[hy g(f:x))
< || [o({(()) 112
(1-?)
M
Jo ();)())))))))))) 112
(33)
lim (im=BxxD) c+0
Σ BB
|h; g(0xx) cc
M j- jolees
1h3 ((0;xxx)000) +
eC'
$ <im -00
7 11-
+ (1
= =(17)CC
where the second line is because for z Dj, hg (z) is locally one and other optimal discriminators hī (z)] # j.j E [M] are all locally zero, thus we have hg (z) a hद (z) c, and hy(2) s hī (z) + c.
lim =C 1- = 0 is the gradient that passed to the generator by native multi-domain DANN (Equ. I..
Environment label smoothing leads to another term, that is (1 C)C and avoid gradients vanishing. Consider all distributions, we have
lim || Vodi.g (DI...., Dar) ll c-0
(1-7) M Σ log (h, )(x(())) llz j-2
< lim || VOLEEDD, ? logg g(xx+ M-1
c00
(34)
(1-7)
log (h, o(x)) 112
lim || VellxeD +0
7 ylogh o g(x)
Σ
M 1 f=1
M(1-7)C,
23

---

A.6 ELS STABILIZE THE OSCILLATORY GRADIENT
For the clarity of our proof, the notations here is a little different compared to other sections. Let ec(i) be the cross-entropy loss for class i, we denote g is the encoder and (w; ) is the classification parameter for all domains, then the adversarial loss function for a given sample x with domain index i here is
F(x, i) -(1-7)ec()
Y Ee((i) M
=ec(1)
+ 7 Σ (ed(j) - ed(i)
M 1
Ex Exp((()0(x)) ) ) exp(u]g(())
exp(1e]((1))
=ec(i) +
M Σ
- -log
+log En exp(w])((=))
=ec(1) +
Σ
-w)g(1)) + log Σ [exp(w(o(x))) + +.]](z) log Σ {exp(wZo(a)))
M )
=ec(i) + M -1
7 Σ
(wi - myy) ((())))
=-w] + log
Σ [exp(wis(()))
M -111 Σ E((r: -wy) ww)) g(())
(35)
We compute the gradient:
0F(r,i) -g()) + exp(w]g(x) Ow;
((x) (x)( M M- g(x)) --11+ p(1) + M g(x), (36)
7_
En exp(1
where p(i) denotes E. exp((( When is small (e.g., S M(1 p(i))) the gradient will be
exp(w g(x)
further pullback towards O. Similarly, for wy and g(x), we have
OF(a, exp(w)]((x))
M - ((x)() = ((x()) - M 11-1 g(x)
7
7
Owl E cxp(w)](xx) g(x)
(F
-1;; + Σ exp(wfg(2) 0g(x)
E(ws - wo) [(w;- =-(1- 7
M M ww;+ + Σ ((()) M -
7
Σn exp(w]g(x) Wy
M -1
(37)
then with proper choice of y (c.g., 7 $ miny Mp(j)), the gradient w.r.t wy and g(x) will also shrink towards zero.
A.7 ENVIRONMENT LABEL SMOOTHING MEETS NOISY LABELS
In this subsection, we focus on binary classification settings and adopt the symmetric noise model (Kim et al. 2019). Some of our proofs follow (Wei al., 2022 but different results and analy- ses are given. The symmetric noise model is widely accepted in the literature on learning with noisy labels and generates the noisy labels by randomly flipping the clean label the other possible classes. Specifically, given two environment with high- nigh-dimensional feature environment label (0, 1}. denote noisy labels ÿ is generated by noise transition matrix T, where Ti denotes denotes the probability of flipping the clean label y i to the noisy label ÿ j. i.e., Tij = P(ū jly i). Let = P(ū = 1ly = 0) P(ğ 0ly- = 1) denote the noisy rate, the binary symmetric transition matrix
becomes:
1-e T-(1-e 1-e) e
T=
(38)
Suppose (x. v) are drawn from joint distribution D, but during training, only samples with noisy labels are accessible from (x, D. Denote f= o g and the cross-entropy loss, minimizing the smoothed loss with noisy labels can then be converted to
min E. (..))= ([(s--((())))))) [((f()))))))))) (x).0') ]²)] min E (O |
(39)
24

---

Let 10cc==10 = 1 - according to the law of total probability, we have Equ.
39 is equal
minE,
(40)
recall that E= ||ap::/::|0044400000000000000000
min (cf-g
+ +B (1
min E. y=u [[1-000000|1110000--------001110
+ E y-1 [L00=-0000b|((((00000--111111000
min R y-0 - [|1"0000000000000000000000111111
1 [|0|0-00000|1|100000000000111110
(41)
+ + |bgp0
min E o|gp0//||000011oorrr///////00111
min mji1((_000o0(++00(((00000)))
+ +0000-00000
/ /hhp::/::hhaalllll-----.--..../.
(((+))))
Assume 7 is the optimal smooth parameter that makes the corresponding classifier return the best performance on unseen clean data distribution (Wei et al. 2022). Then the above equation can be converted to
min ((
(42)
27 -
namely minimizing the smoothed loss with noisy labels is equal to optimizing two terms,
min
(7" -
(43)
Risk under clean label
Reverse optimization
where 1 is the risk under the clean label. The influence of both noisy labels and ELS are reflected the last term Equ. (43). Considering the reverse optimization term ② which is the opposite of the optimization process as we expect. Without label smoothing, the weight of 2 will be >" 1 +e and high noisy rate will let this harmful term contributes more our optimization. In contrast, will be removed. For example, if the noisy rate is
by choosing the smooth parameter = 1-2e" zero, the best smooth parameter is just 7
7"-e
A.8 EMPIRICAL GAP ANALYSIS ADOPTED FROM MONNI-CHERVONENKT FRAMEWORK
Theorem 3. (Lemma I in (Ben-Davic et al. 20100) Given Definition 7 Land Definition 2 let H be hypothesis class of VC dimension d. If empirical distributions Ds and Dr all have at least samples, then for any 8 (0, 1). with probability at least - ö,
dlog(2n) log
(Ds, s du (Ds, Dr) + 4V
(44)
25

---

Denote convex hull A the set of mixture distributions, A (Da DMi =Ee n. i, é A}. where is standard M - 1-mimlexx.. The convex hull assumption is commonly used in domain generalization setting (Zhang et al. 2021a Albuquerque et al., 2019 Zhang et al. 2022a), while none of them focus on the empirical gap. Note that dx ( D Mix. Dr) in domain genera generalization setting is intractable for the unseen target domain Dr is unavailable during training. We thus need to convert dx (D Mix Dr) tractable objective. Let D MMw Di=1 Hi Di (To...., Th)) A, where To..... M = arg mo..,.. dz Min, Dr), and Miz is the element within A which is closest the unseen target domain. Then we have
du Mias Dr) ssup ke7
Ex- [h(x) 1] - X [h(x)= = 1]]
2 sup Ex-Dart [1(xx 1] 1x Xx M Mis [x(x) = =1]
he7t
(45)
Ex-Din Mix [h(x) 1] Ex-D, [h(x) =
S dn (DMiz: Dr) + du (D Mizi Min)
The explanation follows (Zhang et al. the convex combination of the source domain approximate the target domain".. The minimization of the first term requires diverse data or strong data augmentation, such that the unseen distribution lies within the convex combination of source domains. We dismiss this term in the following because it includes Dr and cannot be optimized. Follows Lemma in (Albuquerque et al. 2019), the second term be bounded by,
2021a] that the first term corresponds "To what extent can
M dz (D (iMI DMw) S ΣΣ Mej (Di, max (je[M]
(46)
namely the second term can be bounded by the combination of pairwise H-divergence between source domains. The cost (Equ. (I) used for the multi-domain adversarial training can be seen as an approximation of such a target. Until now, we can bound the empirical gap with the help of Theorem 3
M Σ ΣΣ =1
M M Σ
dlog(2 (aiinninn n2)) + log 2
nz du (D,, (D,,
Σ =1
(D,,
+
min( mii,,,,, n2,))
(47)
M M
M
d 2n log 2
Σ
Σ
क, du D,)
Σ
Σ
k 3 du (D,, Dj $4\
where n; is the number of samples in Dj and nz* = min(m,,...,,..
A.9 EMPIRICAL GAP ANALYSIS ADOPTED FROM NEURAL NET DISTANCE
Proposition 6. (Adapted from Theorem A.2 in Arora et al. 2017 Let (Di) a set of distributions and (D:) be empirical versions with at least n² samples each. We assume that the set of discriminators with softmax activation function h(0;-) = (hi(01.)....,,h (.M.,).))) E H. Z + [o, 1, EM h,;(0;;) D are L-Lipschitz with respect to the parameters O and use p denote the number of parameter O,. There is universal constant e. such that when n² epM log Lppr)) with probability at least 1 - exp(-p) over the randommess of (D.) jmp
we have
di.g (DI,..., Dar) - dig (DI, DMr) I
(48)
Proof. For simplicity, we ignore the parameter 0, when using lx;+)). According to the following triangle inequality, below we focus on the term Ez-geD, log h (z) - -F B.ggo. log (z)) and other
"There might he some confusion here hecause in Section A.4 we use as the parameters of encoder h. The usage is just for simplicity but does not that h,g have the same parameters.
26

---

terms have the same results.
dig (D....., Das) dh.a (D), ID...... D
Rx,goD log in (2) ................................ D log frr - F gogoO, log (2)))(z) ... E rrg log logh he (z)
Ex-goD, log h (z) E angot, log h (z) +..++ go log (z) - E g.ggo log har
(49)
Let be a finite set such that every 01 E θ is within distance c of a 01 E $, which is also termed 4LM ALM -net. Standard construction given a & satisfying log |$| (pplog(pp/e) namely there aren't too many distinct discriminators in 4. By Chernoff bound, we have
Pr Ee- log (oghi(z) -E BagooD log hr iz) 2M exp( T"
n'c
(50)
2M
Therefore, when n* spM cp.log((p))) for large enough constant 6, we can union bound over all 0n E $ With probability least exp(-p), for all 0 4, we have Piz-gDD log (z) - B 2-90D, log hi (z) & 2M' Then for every 01 E O, we can find a 0f e + such that ||0] Bill s c/4LM. Therefore
ExzgoD log h (01:z) z-goD, log hi (01:z))
Ea D log hn (0);2) E z.goo, log hi (01;2))
<
Ex-goD, log i (0);z) Ex-goD, log hi (01;z)
(51)
B RugoD, log (8' E Bo log (01:2))
6
$
2M
4M
4M
M
Namely we have
d/ (D)
DM
/ (D (DI...
DAS 3 M
(52)
M
The result verifies that for the multi-domain adversarial training, the expectation over the empirical distribution converges the expectation over the true distribution for all discriminators given enough data samples.
A.10 CONVERGENCE THEORY
In this subsection, we first provide some preliminaries before domain adversarial training convergence analysis. We then show simultaneous gradient descent DANN is not stable near the equilibrium but alternating gradient descent DANN could converge with sublinear convergence rate, which support the importance of training encoder and discriminator separately. Finally, when incorporated with environment label smoothing, alternating gradient descent DANN is shown able attain faster convergence speed.
A.10.1 PRELIMINARIES
The asymptotic convergence analysis is defined as applying the "ordinary differential equation (ODE) method" to analyze the convergence properties of dynamic systems. Given discrete-time system characterized by the gradient descent:
F,((') g²+1 8 + r)h(8").
(53)
where M(:): :R- - IR is the gradient and 1 is the learning rate. The important technique for analyzing asymptotic convergence analysis is Hurwitz condition (Kaliil. 1996): if the Jacobian of the dynamic system A = h' ((0) at stationary point 0* is Hurwitz, namely the real part of every eigenvalue of is positive then the continuous gradient dynamics are asymptotically stable.
Given the same discrete-time system and Jacobian A, to ensure the non-asymptotic convergence, we need to provide an appropriate range of 1 by solving [1 + (x)) < 1, V e Sp(A), where (A) is the spectrum of A. Namely, we can get constraint of the learning rate, which thus is able to evaluate the minimum number of iterations for an e-error solution and could more precisely reveal the convergence performance of the dynamic system than the asymptotic analysis (Nie & Patel 2020
27

---

Theorem 4. (Proposition 4.4.1 in (Bertsekas) 1999).) Let F 22 - 52 be continuously differential function on an open subset Q in R and let 8 E Q be so that
1. F((0") = and
2. the absolute values of the eigenvalues of the Jacobian [da] <1, E (p(F,((0))
Then there is an open neighborhood U of 0" so that for all 6 E U, the iterates gh+1 = F, (0*) is locally converge to . The rate of convergence is at least linear. More precisely, the error |I H 80 in O(Amazi () for k
where Xmox the eigenvalue of (e)))) with the largest absolute value. When [Ax] 1, F will not converge and when Paa 1, F is either converge with a sublinear convergence rate or cannot converge.
Finding fixed points of (0)= 0 mp((0) is equivalent to finding solutions to the nonlinear equation 1(0)= 0 and the Jacobian is given by: (0) = I n))((),
(54)
where both (O), (O) are not symmetric and can therefore have complex eigenvalues. The following Theorem shows when a fixed point of F satisfies the conditions of Theorem +
Theorem 5. (Lemma 4 in (Mescheder et al. 2017(.) Assume A = h 1'(0))o=0- only has eigenvalues
with negative real-part and let 1) > U, then the eigenvalues of the matrix I + A lie in the unit ball if and only if
2a a2 12
2 1 + ( ( -a Sp(A)
1
(55)
Namely, both the maximum value of a and b/a determine the maximum possible learning rate. Although (Acuna et al. 202T) shows domain adversarial training is indeed a three-player game among classifier, feature encoder, and domain discriminator, it also indicates that the complex eigenvalues with a large imaginary component are originated from dicoder-discrmminato adversarial training. Hence here we only focus on the two-player zero-sum game between the feature encoder, and domain discriminator. One interesting thing is that, from non-asymptotic convergence analysis, we can get result (Theorem 5) that is very similar to that from the Hurwitz condition (Corollary 1
in (Acuma et al. 2021 : 17 8- -V= = a E Sp(A) and |a| < (6))
A.10.2 A SIMPLE ADVERSARIAL TRAINING EXAMPLE
According Ali Rahimi's test of times award speech at NIPS 17, simple experiments, simple theorems are the building blocks that help us understand more complicated systems. Along this line, we propose this toy example understand the convergence of domain adversarial training. Denote Ds 2s. Dr- It two Dirac distribution where both 21 and 22 are float number. In this setting, both the encoder and discriminator have exactly one parameter, which is fg, fa respectively The DANN training objective in Equ. 7 is given by
dg f(0.0.x.)) f(-020,0:),
(56)
where f(t)= = log (1/(1+ exx-)))))))) and the unique equilibrium point of the training objective in Equ. (56) is given by 6 6% We then recall the update operators of simultancous and alternating Gradient Descent, for the former, we have
0. VVa da
(,)
(57)
OJ + vVo.o
For the latter, we have F. (0) Fr, (0), and F Fn,2 are defined as (0) 0- nVo do 0a
) F 0. +
0. (e++nVo,d a,
(58)
If we update the discriminator nhe times after update the encoder ne times, then the update operator will be F,, = 1 (0) FT To understand convergence of simultaneous and alternating gradient descent, we have to understand when the Jacobian of the corresponding update operator has only eigenvalues with absolute value smaller than 1.
'One may argue that neural networks are non- non-linear, but Theorem 4.5 from Khalit, 1996 shows that one "lincarize" any non-linear system near equilibrium and analyze the stability of the lincarized system to comment the local stability of the original system.
28

---

A.10.3 SIMULTANEOUS GRADIENT DESCENT DANN
Proposition 7. The unique equilibrium point of the training objective in Equ. (56) is given by
8% -==2=00 Moreover, the Jacobian of (0) - 0g- -e-n/V at the equilibrium point has the two eigenvalues
=0.
0a n/a
11/2 1 " 111.- xelit
(59)
namely (0) will never satisfies the second conditions of Theorem whatever n is, which shows that this continuous system is generally not linearly convergent to the equilibrium point.
0. Vo do
)
Proof. The Jacobian of F,)(0) = 0, no
0. 1/ 10--1(0a20 (0at.) (0200.))) O2 ((.I5(() (0.0,x,) ( 01 + (
VoF,(0) Vo
o. ((abez)))
-n (z xxf(( )
(x (0200.t))
xt (0g00 1 (xx= Itt) xx)
(60)
The derivation result of Vo, 0e (0 ( (lg0 000 (0g00())) should have been 1 (0 (00l:l"""(00aaa. (0.0.x.)- f (0.00.z)
(61)
Since the equilibrium point (1) = (0,0). for points near the equilibrium, we ignore high-order infinitesimal terms egg, 82, 82 0.0a. We can thus obtain the derivation of the second line. The
X a=day(a-d)==-add-oo and then the eigenvalues of V (0) f - zz/i. Obviously [A] > and the proposition is completed. O
eigenvalues of the second-order matrix A
A.10.4 ALTERNATING GRADIENT DESCENT DANN
Proposition 8. The unique equilibrium point of the training objective in Equ. (6)) is given by 8% = 0 If we update the discriminator na times after we update the encoder ne times. Moreover, the Jacobian of Fy (0)) F (0) (Equ. (58) has eigenvalues
a 2
)
a2 2
11/2 =
1,
(62)
where a= --immaa - 2zl 1412 = for 7 S -nendls--al and [A1p2 otherwise. Such
result indicates that although alternating gradient descent does not converge linearly to the Nush- equilibrium, it could converge with a sublinear convergence rate.
Proof. The Jacobians of alternating gradient descent DANN operators (Equ. (58) near the equilib- rium are given by:
(00 ( 1 -n (x, ((,0...)))
xz )=
[2, (x,-2z) 21) ) (63)
(x 1 As result, the Jacobian the combined
similarly we can get (0)) update operator V (f), is
1
me 2
(1g Iz (Iz xt +
VoF, (0) VoF V
(64)
gra (x, xz)
An easy calculation shows that the cigenvalues of this matrix are
V(1 ReRd 12 - -2)22)
2 100axx))')
11/2 8 11 (x,
Reld
x1)²
-1
(65)
29

---

Let 2/ /naenlarr mzl, we can get 1122
a² o
V(1 a²
- If 1- )²²²² namely
a > 2, then 11122 = V 12 2 - To satisfy [>| < 1, we have
(1
< <,, which conflicts
2
with the assumption. That is s 2, and in this case A1/2l 1.
A.10.5 ALTERNATING GRADIENT DESCENT DANN+ELS
Incorporate environment label smoothing to Equ. (6)). the target is revised into:
de.- = = - (1-r)f(-0.8=.=) + (1-7))((00...) (1-)
(66)
Proposition 9. The unique equilibrium point of the training objective in Equ. (66) is given by 8% = 0 we update the discriminator na times after we update the encoder ne times. Moreover, the Jacobian of Fg (0) F,,1 (0) (Equ. (8) has eigenvalues
11/2 1 a²
a
V
1
1,
(67)
2
-/nanill.- 2 and P21/2 > otherwise. Such result indicates that alternating gradient descent DANN+ELS could converge faster than alternating gradient descent DANN.
where a 2
2xy-1 / xel. 141/2== = 1) 1
Proof. The operator for alternating gradient descent DANN+ELS is F,, F, (0), and F, F,2 near the equilibrium are given by:
0g 17Vo, do.7 Os
0. (0)
(0) 0.
700 (1))))))
(0) =
On
0.
(0)
04 + n/ dadd
Oa
x1 + (1)+(1))))+1)) f(())
(68)
The Jacobians of alternating gradient descent DANN+ELS operators near the equilibrium are given by:
(0) (
m((2-1) (Ix sit) ) voFo, -
1 )
1(23-1) (I. xz)
(69)
As result, the Jacobian of the combined update operator VoF is
(27-1) (xr, - Fzi)
VoF,(() VoF (ONV
2
m2y-2-11 (x. xz))
mmm. (3. - 3zg +1
2
(70)
An easy calculation shows that the eigenvalues of this matrix
11/2 1 nena 17 ?2 1x) V( nend 12(2- (x,)))=1)))))))) -1
(71)
8
Similarly to the proof of Proposition 8 let a 27-1 Ta x, xzl, we can get \1/2 1
a
2
V1--))-1- -1 Only when a S 2, \1/2 are on the unit circle, namely 7) S Vhatt x. -2 2
Compared to the result in Proposition g which is 1) $ - the additional 2--1 enables us to choose more large learning rate and could converge to an small error solution by fewer iterations.
B EXTENDED RELATED WORKS
Domain adaptation and domain generalization MMannet et al. 2013 Sagawa et al. 2019 Li et al.
2018a Blanchard et al. 2021 Li et al. 2018b Zhang al. 202 2a 2022c) aims learn model
30

---

that extrapolate well in unseen environments. Representative methods like AT method (Ganin et al. 2016) proposed the idea of learning domain-invarianl representations as an adversarial game. This approach led to a plethora of methods including state-of-the art approaches (Zhang et al. 2019 Acuna et al. 2021 2022). In this paper, we propose a simple but effective trick, ELS, which benefits the generalization performance of methods by using soft environment labels.
Adversarial Training in GANs is well studied and many theoretical results of GANs motivate the analysis in this paper. e.g,, divergence minimization interpretation (Goodfellow al.
2014 Nguyen
et al. 2017), generalization of the discriminator (Arora et al. 2017; Thann-Tahhhhhh el al. 2019), training
stability (Thanh- Thanh-Tung et al., 2019 Schäfer al. 2019 | Arjovsky & Bottou 2017 Arjovsky et al.
2017), nash equilibrium (Farnia & Ozdaglar 22200; Nagarajan & [olter|[017] and gradient descent
in GAN optimization (Nagarajan & Kolter] 2017 Gidel et al. 2018 Chen et al. 2018). Multi-domain image generation is also related to this work, generalization to the JSD metric has been explored to
address this challenge (Gan et al. 2017; Pu et al., 2018 (Trung Le et al. 2019). However, most of
them have to build M pairwise critics, which is expensive when M is large. x² GAN Tao
al. 2018 firstly attempts to tackle the challenge and only needs M 1 critics.
C ADDITIONAL EXPERIMENTAL SETUPS
C.I DATASET DETAILS AND EXPERIMENTAL SETTINGS
In this subsection, we introduce all the used datasets and the hyper-parameters for reproducing the experimental results in this work. We have uploaded the codes for all experiments in the supplementary materials to make sure that all the results are reproducible. All the main hyper- parameters for reproducing the experimental results in this work are shown in Table
C.1.1 IMAGES CLASSIFICATION DATASETS
Experimental settings. For DG and multi-souree DG tasks, all the baselines are implemented using the codebase of Domainbed (Gulrajani & Lopez- Paz 2222] and we use as encoders ConvNet for RotatedMNIST (detailed in Appdendix D.1 (Gulrajani & Lopez-Paz (021)) ) and ResNet-50 for the remaining datasets. The model selection that
of the three selection methods in (Gulrajani & Lapez-Paz 2021). That is, we choose the model maximizing the accuracy on validation set that follows the same distribution of the test domain. For DA tasks, all baselines implementation and perraeprrameters follows (Wang & Hou). For Continuously Indexed Domain Adaptation tasks, all baselines are implemented using PyTorch with the same architecture as (Wang et al.) 2020). Note that although our theoretical analysis on non-asynn asymptotic convergence is based on alternating Gradient Descent, current DA methods mainly build on Gradient Reverse Layer. For fair comparison, in our experiments considering domain adaptation benchmarks, we also use GRL as default and let the analysis in future work.
is test-domain validation,
Rotated MNIST (Ghifary al. 2015) consists of 70,000 digits in MNIST with different rotated angles where domain is determined by the degrees d {0, 15, 30, 45, 60, 75).
PACS Li et al. 2017b) includes 9, 991 images with 7 classes y { dog, elephant, giraffe, guitar, horse, house, person } from 4 domains d e (art, cartoons, photos, sketches).
VLCS Torralba & Efros 2011) is composed of 10,729 images. 5 classes y e { bird, car, chair, dog, CCaaaansschhCCtchhhhhhhhhh
person from domains d E Caltecch10, MabelMe, SUN09, VOC2007).
Office-31 (Saenko al. 2010) contains contains 4, 110 images, 31 object categories in three domains: de { Amazon, DSLR, and Webcam].
Office-Home (Venkateswara et al. 2017): consists of 15,500 images from 65 classes and domains: de { Art (Ar), Clipart (CI), Product (Pr) and Real World (Rw) }.
Rotating MNIST (Wang et al. 2020) is adapted from regular MNIST digits with mild rotation to significantly Rotating MNIST digits. In our experiments, 45%) is set as the source domain and others are unlabeled target domains. The chosen baselines include Adversarial Discriminative Domain Adaptation (ADDA (Tzeng et al. 2017)), and CIDA (Wang et al. 2020). ADDA merges data with different domain indices into one source and one target domain. DANN divides the continuous
31

---

domain spectrum into several separate domains and performs adaptation between multiple source and target domains. For Rotating MNIST, the seven target domains contain images rotating by d <{[0p, 45%), [45", [45), [90%, 135"). ..... [315 .309)) degrees, respectively.
Circle Dataset (Wang et al. 2020) includes 30 domains indexed from to 30 and Figure 5(a) shows the 30 domains in different colors (from right to left is 1,..., ,330 respectively). Each domain contains data on a circle and the task is binary classification. Figure ( positive samples as red dots and negative samples as blue crosses. In our experiments, We use domains 6 as source domains and the rest as target domains.
C.1.2 IMAGE RETRIEVAL DATASETS
Experimental settings. Following previous generalizable person RelD methods, we use Mo- bVleNerV2 (Sandler et al. (2018) with multiplier of 1.4 as the backbone network, which is pretrained on ImageNet (Deng et al. 2009). Images are resized to 256 x 128 and the training batch size N is set to 80. The SGD optimizer used to train all the components with a learning rate of 0.01, momentum of 0.9 and a weight decay of 5 10"4, The learning rate is warmed up in the first 10 epochs and decayed to its 0.1x and 0.01x at 40 and 70 epochs.
We evaluate the proposed method by Person re-identification (ReID) tasks, which aims to find the correspondences between person images from the same identity across multiple camera views. The training datasets include CUHK02 (Li & Wang 2013). CUHK03 eLi al. 2 Market 501 (Zheng
al. 2015), DukeMTMC-ReII (Zeng al. (017), and CUHK-SYSU PersonSearch (Xiao et al. 2016 The unseen test domains are VIPeR (Gray et al. 2007). PRID (Hirzer et al., 1), QMUI. GRID [Lu al. 2012). and i-LIDS (Wei-Shi al., 2009). Details of the training datasets are summarized in [O the test datasets are summarized in Table [T] All the assets (i.e., datasets and the codes for baselines) we use include a MIT license containing a copyright notice and this permission notice shall be included in all copies or substantial portions of the software.
GRID (iu al. 2012) contains 250 probe images and 250 true match images of the probes in the gallery. Besides, there are a total of 775 additional images that do not belong to any of the probes. We randomly take out 125 probe images. The remaining 1 25 probe images and 1025 775 250) images in the gallery are used for testing.
i-LIDS (Wei-Shi et al. 2009) has two versions, images and sequences. The former is used in our experiments. It involves 300 different pedestrian pairs observed across two disjoint camera views 1 and in public open space. We randomly select 60 pedestrian pairs, two images per pair are randomly selected as probe image and gallery image respectively.
PRID2011 (Hirzer et al., 2011) has single-shot and multi-shor versions. We use the former in our experiments. The single-sho- version has two camera views A and B, which capture 385 and 7449 pedestrians respectively. Only 200 pedestrians appear in both views. During the evaluation, 100 randomly identities presented in both views selected, the remaining 100 identities in view A constitute probe set and the remaining 649 identities in view B constitute gallery set.
VIPeR (Gray al. 2007 contains 632 pedestrian image pairs. Each pair contains two images of the same individual seen from different camera views 1 and 2. Each image pair was taken from an arbitrary viewpoint under varying illumination conditions. To compare other methods, we randomly select half of these identities from camera view 1 as probe images and their matched images in view as gallery images.
We follow the single-shot shot setting. The average rank-k (R-k) accuracy and mean Average Precision (mAP) over 10 random splits are reported based on the evaluation protocol
1.3 NEURAL LANGUAGE DATASETS
CivilComments-We Wilds (Koh et al. 2021 contains 448, 000 comments online articles taken from the Civil Comments platform. The input is a text comment and the task is to predicate whether the comment rated as toxic, e.g... the comment Maybe you should learn to write coherent sentence so we can understand WTF your point is is rated as toxic and applaud your father. He was a good man! We need more like him. is not. Domain in CivilComments dataset is an 8-dimensional
32

---

binary vector where each component corresponds to whether the comment mentions one of the 8 demographic identities (male, female, LGBTQ, Christian, Muslim, other religions, Black, White).
Amazon-Wildh (Koh al. 2021> contains 539, 520 reviews from disjoint sets of users. The imput the review text and the task is to predict the corresponding 1-to-5 star rating from reviews of Amazon products. Domain d identifies the user who wrote the review and the training set has 3, 3,920 domains. The 10-th percentile of per-user accuracies metric is used for evaluation, which is standard measure model performance on devices and users at various percentiles in effort to encourage good performance across many devices.
C.1.4 GENOMICS AND GRAPH DATASETS
RxRxI-wilds (Koh et al. 2021 comprises images of cells that have been genetically perturbed by siRNA, which comprises 125, 510 images of cells obtained by fluorescent microscopy. The output indicates which of the 139 genetic treatments (including no treatment) the cells received, and d specifies 51 batches in which the imaging experiment was run.
OGB-MoIPCBA (Koh et al. 2021) is multi-label classification dataset, which comprises 437,920 molecules with 120, 084 different structural scaffolds. The input is a molecular graph, the label is a 28-dimensional binary vector where each component corresponds to a biochemical assay result, and the domain d specifies the scaffold (i.e., cluster of molecules with similar structure). The training and test sets contain molecules with disjoint scaffolds; The training set has molecules from over 40, 000 scaffolds. We evaluate models by averaging the Average Precision (AP) across each of the 128 assays.
C.1.5 SEQUENTIAL DATA
Spurious-Fouries Gagnon- Audet et al. 2022) is a binary classification dataset (y E (low-frequeney peak (L) and high-frequency peak (H).), which is composed of one-dimensional signal. Domains de (10%, 80%, 90%} contain signal-label pairs, where the label is a noisy function of the low- and fighffrequencies such that low-frequency peaks bear varying correlation of d with the label and high-freyuency peaks bear an invariant correlation of 75% with the label.
HHAR (Gagnon-Auder et al. 2022 6 activities classification dataset (y e (Stand, Sit, Walk, Bike, Stairs up, and Stairs Down F), which is composed of recordings of 3-axis accelerometer and 3-axis gyroscope data. Specifically, the input x is recordings of 500 time-steps of a 6-dimensional signal sampled at 100Hz. Domain d consist of five smart device models: d {Nexus 4, Galaxy S3, Galaxy S3 Mini, LG Watch, and Samsung Galaxy Gears).
C.2 BACKBONE STRUCTURES
Most of the backbones are ResNet-50/ReeNett118 and we follow the same setting as the reference works. Here we briefly introduce some special backbones used in our experiments,i.e, ConvNet for Rotated MNIST, EncoderSTN for Rotating MNIST, DistilIBERT for Neural Language datasets, and GIN for OGB-MoIPCBA.
MNIST ConvNet. is detailed in Table. 12
DiT. We use the implementation from (Wolf et al. 2019) and finetune BERT-base-uncee uncased models for neural language datasets. EncoderSTN use a four-layer convolutional neural network for the encoder and a three-layer MLP to make the prediction. The domain discriminator a four-layer MLP. The encoder is incorporated with a Spacial Transfer Network (STN) (Jaderberg et al. 2015), which takes the image and the domain index as input and outputs set of rotation parameters which then applied to rotate the given image.
Graph Isomorphism Networks (GIN) (u et al. 2018) combined with virtual nodes is used for OGB-MoIPCBA dataset, this is currently the model with the highest performance in the Open Graph Benchmark.
Deep ConvNets (Schirrmeister et al. 2017) for HHAR combines temporal and spatial convo- luich fits this data well and we use the implementation the BrainDecode Schimmeis- ter (Schirreeister et al., 2017) Toolbox.
33

---

(a) Domains.
(b) Ground Truth.
(c) ADDA.
(d) DANN+ELS
Figure 5: Results on the Circle dataset with 30 domains. (a) shows domain index by color, (b) shows label index by color, where red dots and blue crosses are positive and negative data sample. Source domains contain the first 6 domains and others are target domains.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 ADDITIONAL NUMERICAL RESULTS
Multi-Source Domain Generalization. IRM (Arjovsky et al. 2019) introduces specific conditions for an upper bound on the number of training environments required such that an invariant optimal model can be obtained, which stresses the importance of the number of training environments. In this paper, we reduce the training environments on the Rotated MNIST from five to three. As shown in Table 17] as the number of training environment decreases, the performance of TRM fall sharply (e.g., the averaged accuracy from 97.5% to 91.8%), and the performance on the most challenging domains d (0,5) 5) decline the (94.9% 80.9% and 95.2% 91. 1..1% 1%). In contrast, both ERM and DANN+ELS retain high generalization performances and DANN+ELS outperforms ERM in most domains.
Image Retrieval. We compare the proposed DANN+ELS with methods on typical DG-Rell setting. As shown in Table T6] we implement DANN with various hyper-parameters while DANN always fails to converge on RelD benchmarks. As illustrated in Appendix Figure 8 the training statistics with the baseline, where DANN highly unstable and attains inferior results. However, equipped with ELS and following the same hyper-parameter DANN, DANN+ELS attains well-training stability and achieves either comparable or better performance when compared with recent state-of-the-art DG-RelD methods. See Appendix [.2 for t-sne visualization and comparison.
we compare
Table 14: Domain generalization performance Table 15: Domain generalization performance the OGB MGB-MolPCBA dataset.
on the Apurious-Fourice dataset.
OGB-McIPCBA
Spurious-Fourie
Algoritium
Val
Test
Algorithm
Train validaaion
Test validation
ERM Virnik
27,8 a: 0.1
27.2
ERM Viraik
9,7 0.3
9.1
Group DRO
Samnen df
2019
23.1 0.6
22
IRM Arinsss a
X119
9.3 + 0.1
$7.6
CORA1 5in Samnk
Auro
184 082
179+00.
SD Pezzctie
Alr
10.2 0.1
9.2
VRES
Camnn 3u1
14.1+0.5
(Camm
DANI-ELS
18.
17.21003
DANIVELLS
10.7 0.6
15.6
1.0
4.5
D.2 ADDITIONAL ANALYSIS AND INTERPRETATION
T-sne visualization. We compare the proposed DANN+ELS with MetaBIN and ERM through 1-SNE visualization. We observe distinct division different domains in Figure 7 and Figure 7 which indicates that domain-specific feature space is learned by the ERM. MIN perform better than ERM and the proposed DANN+ELS can learn more domain-invarian representations while keeping discriminative capability for ReID tasks.
D.3 ABLATION STUDIES
34

---

PACS
Art Painting
Cartoon
Photo
Sketch
Dog
Elephant
VLCS
10
LabelMe
SUN09
VOC2007
Bird
Car
Figure 6: Data examples from the PACS and the VLCS datasets.
(a) ERM (All).
(b) MetaBIN (All).
(c) DANN+ELS (AlI).
(d) ERM (Test)
(c) MetaBIN (Test).
(f) DANN+ELS (Test).
Figure 7: Visualization of the embeddings on training and test datasets. Query and gallery samples of these unseen datasets are shown using different types of mark. Best viewed in color.
35

---

Models DANN
Models
Classificaton Loss
30
DANN
Dicctmiaation
20
DANS
DANS
Average m.P
Models
20
ERM
-10
DANN
10
01
ERM
Iterations
Berraiins
Derations
(a) Domain discrimination loss.
(b) Identity classification loss.
(c) Average mAP on the test sct.
Figure 8: Training statistics on RelD datasets.
Table 9: Hyper-parameters for different benchmarks. Lrg, Decayy: learning rate and weight decay for the encoder and classifier; Lra, Decaya: learning rate and weight decay for the domain discriminator; bsz: batch size during training; dsteps: the discriminator is trained dsteps times once the encoder and classifier
trained; W Wegg: tradeoff weight for the gradient penalty; A: tradeoff weight for the adversarial loss. The default 32 for Adam and Adam W optimizer 0.99 and the momentum for SGD) optimizer is 0.9. / means domain discriminators trained on the dataset GRL but not alternating gradient descent.
Task
Dutsets
Lrg
Lrdd
Decay,
Decayd
bsz
datepa
W reg
Rotated MNIST
IE-03
1E-03
0.5
0E+00
0.0
64
0.5
PACS
SE-05
SE-05
0.5
0E400
0.0
32
0.5
Images
Classification
VLCS
SE-05
5E-05
0E+00
0.0
32
(fesse)3(
1E-02
1E-02
SGD
1E-3
1E-3
32
1,00
Office Home (ResNet50)
Office-Home
1E-02
1E-02
SGD
IE-3
1E-3
32
0
1.00
Office-31 (VTT)
2F:03
2F:03
SGD
1F-3
1F-3
24
1.00
Office-Home Home (VIT)
2E:03
2E-03
SGD
IE-3
IE-3
24
1,00
Rotating MNIST
2F:04
2F-04
0.9
SE-04
SB-04
100
2,00
Image
Retrieval
MS
IE-02
IE-02
SGD
SE-04
SE-04
80
1.00
Neural Language
CiviCoomments
IE-05
IE-05
SGD
1E-02
0
16
1.00
Processing
Amazon
1E-04
2F-04
Adam
1E-02
0.11
Genomics
RxRx1
IC-44
2C-04
0.9
IEE05
0
72
0.1T
and Gruph
OGB-MoIPCBA
8E-04
1E-02
0.9
IE-05
0
32
0.11
Sequential
Spurious-Found
4E-04
4E-04
IE-03
78
Prediction
HHAR
3E-03
IE-03
0.5
0E+00
13
3.5
Table 10: Training Datasets Statistics.
Dataset
IDs
Images
CUHK02
1,816
7,264
CUHK03
1,467
14,097
DukeMTMM-Re--U
1,812
36,411
Market-1501
1,501
29,419
CUHK-SYSU
11,934
34,547
Table 11: Testing Datasets statistics.
Probe
Gallery
Dataset
Pr. IDs
Pr. Imgs
Ga. IDs
Ga. imgs
PRID
100
100
649
649
GRID
125
125
1025
1,025
VIPeR
316
316
316
316
i-LIDS
60
60
60
60
36

---

Table 12: Details of MNIST ConvNet architecture. All convolutions 3x3 kernels and "same" padding
Layer
Conv2D (in=d, out=64)
ReLU
3
GroupNorm (groups=8)
Conv2D (in=64, out=128, stride=2)
5
ReLU
6
GroupNorm (8 groups)
7
Conv2D (in=128, out= 128)
8
ReLU
9
GroupNorm (8 groups)
10
Conv2D (in=128, out= 128)
11
ReLU
12
GroupNorm (8 groups)
13
Global average- poogee--ooiigg
Table 13: The domain genernalization/adaaatttiii accuracy Rotated MNIST.
Rotated MNIST
Algorithm
45
75
Avg
ERM Vapnik 1999
95.3 0.2
98.7 = 0.1
98.9 0.1
98.7 0.2
98.9 0.0
96.2
97.8
IRM (Ariovsky
94.9
98.7 0.2
98.6
98.6
9.1
95.2
97.5
DANN (Ganm
95.9
98.6 0.1
98.7
99.0
98.7 9..7
96.5
97.9
ARM Zhang et 20216
95.9 0.4
99.0 0.1
98.8 0.1
98.9 0.1
99.1 0.1
96.7 0.2
98.1
DANN+ELS
96.3 1.310.. 0.1
98.7 0.1
98.9 0.3
99.1 0.1
98.7 0.0
96,9 0.5
98.1
0.4
0.1
0.2
0.1
0.0
0.4
0.2
Table 16: Comparison with recent state-of-the-art DG-ReID methods.
denotes DANN cannot
converge and attains infinite loss.
Average R-1
Methods
mAP
R-1
R-5
R-10
mAP
R-1
R-5
R-10
mAP
R-1
R-5
R-10
mAP
R-1
R-5
R-10
mPA
DIMN Supe
2019
47.5
37.9
51.2
70.2
76.0
60.1
39.2
67.0
76.7
320
29.3
53.3
65.3
41.1
70.2
89.7
$4.35
78.4
DDAN Chen 2 au
2024
63.1
60.5
71.8
56.1
54.5
62.7
71.9
58.9
50.6
62.1
73.8
78.5
85.3
92.5
DIR DIR-ReID
Zrang Rt
2021
63.8
71.2
58.5
76.9
83.3
67.0
69.7
85.8
91.0
77.1
48.2
67.1
76.3
57.6
79.0
94.8
97.2
83.4
MetaBIN
64.2
71.9
76.8
81.9
67.6
70.6
$6.5
91.5
78.2
47.3
66.0
74.0
56.4
79.5
93.0
97.5
85,5
Group
Uni
Zhane m.. 2izza
65.4
72.8
60.0
78.2
82.8
68.4
73.5
85.3
91.7
79.4
47.5
69.3
77.4
57.2
80.7
94.0
97.0
$6.2
DANN
Ganii et al. 2013)
DANN+ELS
61.2
72.1
59.3
76.4
82.7
67.1
87.7
91.7
77.7
67.5
57.2
79.8
91.7
97.2
86.1
Table 17: Generalization performance on multiple unseen target domains. f denotes improvement of DANN+ELS compared to DANN, and y is the hyper-parameter for environment label smoothing.
Rotated MNIST
Target domains (0', "0".00"00
Target domains (15", ,5",
75")
Method
30
60
15"
45
75"
Avg
ERM Vapnik, 1999
96.0
0.4
0.1
0.3
99.1
96.7 0.3
98.0
IRM (Arjovsky
19
80.9 3.2
94.7 0.9
94.3 1.3
94.3
0.8
95.5
91.1 3.1
91.8
DANN Gaann al 2016
96.6 0.2
98.8 0.3
98.7 = 0.1
98.6 0.4
98.8 0.2
96.9 + 0.1
98.1
DANN=ELS
96.7 = 0.4
98.9 0.2
98.8 0.1
98.8 + 0.1
99.0 0.2
97.0 + 0.4
98.2
0.1
0.1
0.1
0.2
0.2
0.1
0.1
37

---

Table 18: Generalization performance sequential benchmarks. denotes improvement of DANN+ELS compared to DANN.
HHAR
Train- domain validation
Algorithm ID ERM
Nexus
Galazy S3
Galaxy Mini
LG watch
Sam. Gear
Average
98.91
98. 14.....115
98.68=0.15
90.08+0.28
80.63+1.33
93.35
ERM
97.6440. 15
97.64+0.00
92.51=0.46
71.69+0. 14
61. 1..4411.0 1.04
84.28
IRM
96.0240.17
95.75=0.22
89.46=0.50
66.49+0.94
57.6620.37
81.08
SD
98. 14+0.01
98.3240. 19
92.71=0.09
75.12+0.18 18
63.85+0.28
85.63
VREx
95.81+0.50
95.92+0.23
90.72+0.10 10
69.04+0.23 23
56.42+1.57
81.58
DANN
94.45 0.44
95.05 0.10
88.70 0.56
68.33 0.49
1.24 58.45
80.99
DANN+ELS
95.95 # 0.39
95.65 0.42
90.50 = 0.39
69.55 + 0.36
58.45 # 0.24
82.02
1.5
0.6
1.8
1.22
0.0
1.03
Oracle train-domain validation
Algorithm ID ERM
Nexus 98.91
Galazy $3
Galaxy S3 Mini
LG watch
Sam. Gear
Average
98.44+0.15
98,68=0.15
90,08=0.28
80.63+1. .33
93.35
ERM
97.98=0.02
97.9240.05
93.09=0.15
71.9610.04
64.08+0.66
85.01
IRM
96.0220.17
95.75:0.22
89.91=0.25
68.00=0.34
57.77+0.42
81.49
SD
98.48+0.01
98.67+0.11
94.36+0.24
75.12+0.18
64.86+0.28
86.3
VREx
96.6520.18
96.3020.05
90.98=0.16
69.3910.27
59.1240.80
82.49
DANN
95.95 +: 0.21
96.20 0.07
89.91 = 0.73
72.70 4: 0.63
58.45 1.77
82.64
DANN+ELS
96.79 + 0.13
06.9440.1 0.13
91.57 0.22
72.70 + 0.63
59.80 + 0.84
83.56
0.84
0.74
1.66
0.0
1.35
0.92
38


Reviews:
[{"review_id": "inkfs7C0QI", "reviewer": "Reviewer_hUFN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Poor generalization on out-of-distribution data is a common problem among machine learning models. In classification tasks, one recently proposed solution is Domain Adversarial Training, which asks the model to predict the domain of the data sample in addition to its class, and encourages the model to learn domain-agnostic class-relevant features through an adversarial objective. However, this method is known to suffer from training instability. To that end, the authors proposed an extremely simple modification — label smoothing on the environment (i.e., domain) label — that improves not only the training stability but also robustness to label noise, convergence speed, and performance.", "summary_of_the_review": "This paper introduced a seemingly trivial modification, label smoothing on environment label, to a popular approach for out-of-distribution generalization. Notably, the authors showcased the advantages of the label smoothing in three different aspects through a combination of theorical and empirical analyses. Overall, I find this a very solid piece of work and would recommend for acceptance of this submission.", "strength_and_weaknesses": "Strengths\n1. In general, I am in favor of simple-but-effective improvements.\n2. The authors demonstrated via theoretical validations three advantages of the proposed method: (1) training stability, (2) robustness to label noise, and (3) non-asymptotic convergence speed. Further, the authors used empirical studies to complement the theoretical results. (1) is supported by Figure 2 and 5; (2) by Figure 4(a); and (3) by Figure 5.\n3. I particularly like section 3.4, where the authors explicitly faced the unrealistic assumptions in their theoretical validation and analyzed the “empirical gap and parametrization gap”.\n4. I am also glad about the fact that the authors intentionally organize the proofs and non-central results to the appendix, leaving the logic flow in the main text fluent and engaging.\n\nWeaknesses\n1. On the flip side of the first strong point I listed, one potential concern may be the lack of the so-called “novelty”. I fully acknowledge the thoroughness and comprehensiveness of this work, yet the technical contribution (apart from the analyses and insights) is among the least “innovative” I have seen. Afterall, label smoothing has been around for such a long time.\n\nMinor Things\n1. Would it be slightly more intuitive to write $D_S’ = D_T + \\lambda(D_S - D_T)$ as $D_S’ = \\lambda D_S + (1 - \\lambda) D_T$ instead?\n2. Typo. Page 2. “phenomena” instead of “phenomenons”.\n3. Minor grammar thing. Page 3, under Proposition 2. “One **may** argue that adjusting the tradeoff weight λ can also balance AT and ERM**.** **H**owever, λ can only adjust the gradient contribution of AT part, …”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Uo0ALdTTKC", "reviewer": "Reviewer_kFo4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the domain adversarial training problem and propose to use label smoothing for domain discrimination. Specifically, motivated by the observation that different domains from VLCS dataset show small difference, thus a soft domain label should be applied to domain adversarial training instead of hard label. By simply rewriting the cross-entropy loss, the proposed ELS can be easily implemented. Extensive theoretical and empirical analyses are provided to support the proposed method.", "summary_of_the_review": "I have carefully read the whole paper. This paper is quite solid and has excellent completeness, however, there are still some concerns (see weaknesses). If the authors can address my concerns, I will consider raising my score.", "strength_and_weaknesses": "Strength:\n- This paper is well-organized and easy to follow. The general framework is pretty clear and follows rigorous logic.\n- The theoretical analysis shows many helpful properties of ELS, which are further validated by effective experimental results. To my opinion, this is a very solid paper.\n- The ELS method is very easy to implement. The details as well as codes are provided to ensure reproducibility.\n\nWeakness:\n- Although this paper is quite solid, I am still concerned about the motivation for introducing label smoothing. Indeed, the domains from the VLCS dataset are similar to each other. However, in other domain adaptation datasets containing significantly different domains (such as the Office-Home dataset), using smoothed domain labels might not improve the performance any further. As shown in the continuously indexed domain adaptation in Table 5, the improvement is 10%. Compared to tables 2 and 4, the improvement of ELS is less than 1%, especially in Table 4, ELS only surpasses the backbone method by 0.3%. So, is it possible that the proposed method is only effective on a dataset that has a small domain gap, and as the domain gap becomes smaller, the improvement would decrease?\n- When the encoder gets better, the generated features from different domains are more similar. However, a good encoder is also based on the quality of the discriminator. By using label smoothing, the discriminator can not be as discriminative as vanilla DAT which leverage hard labels, thus the performance of the encoder could doubtful. Is there any analysis of the learned discriminator?\n- Moreover, the improvements on the dataset Office-Home can be achieved by tuning the Hyper-parameter $\\gamma$, whose optimal value is not provided in the parameter analysis in Figure 4 (b). In my opinion, by tuning $\\gamma$, there is always a better classifier than the vanilla DAT.\n- In the introduction, the authors are motivated by the observation that VLCS has similar domains, thus label smoothing (smaller $\\gamma$) should be introduced, instead of the vanilla DAT ($\\gamma$=1). However, in Section 4.2, “PACS can be easily discriminated”, thus the optimal $\\gamma$ is smaller than VLCS. Is this a contradiction? Please give an explanation.\n- Additionally, the selection of $\\gamma$ is unclear to me. Do the authors conduct a grid search in the experiments? How to obtain the mean and std of $\\gamma$ to produce Figure 4 (b)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EBe-5FLnaUa", "reviewer": "Reviewer_LfGU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a DAT label smoothing approach. The results are backed by the theoretical evidences. However, the main novelty is minimalistic.", "summary_of_the_review": "The papers proposes a label smoothing idea for DAT. The novelty is limited as the idea of label smoothing is already established in so many previous papers.  ", "strength_and_weaknesses": "Strength:\n- The paper is well written.\n- The empirical results are backed by theoretical evidences\n- Simple solution\n\nWeakness\n- The paper used an already established idea with minor modifications and that is the major drawback of the work\n- The paper showed label smoothing approach but the question is wouldn't it attribute to the catastrophic forgetting of the discriminator. \nQuestions:\n1) The contribution seems marginal as most of the proofs (Theorem 1 and 2) and  can be directly inferred from previous GAN based works. \n2) Proposition 1, do we need both the $\\gamma(D_{S}-D_{T})$ and $\\gamma(D_{T}-D_{S})$ \n3) Proposition 2, would $p_{s^{'}}$ be $p_{t}$? If proposition 2 have issues for $p_{s}\\rightarrow 0$ then that would be similar for proposition 1, right?\n4) Proposition 3, for multilabel scenario would the $D_{Mix}$ provide any meaningful information? Though theoretically we could show the use of it but I am curious whether there would be any practical implication of that.\n5) Sec A.5, would the mixture provide any meaningful information or it would do only catastrophic forgetting to the discriminator?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster