When Source-Free Domain Adaptation Meets Learning with Noisy Labels
Authors: Li Yi
Gezheng Xu
Pengcheng Xu
Jiaqi Li
Ruizhi Pu
Charles Ling
Ian McLeod
Boyu Wang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Source-Free Domain Adaptation
Unsupervised Domain Adaptation
Noisy Label Learning

Abstract: Recent state-of-the-art source-free domain adaptation (SFDA) methods have focused on learning meaningful cluster structures in the feature space, which have succeeded in adapting the knowledge from source domain to unlabeled target domain without accessing the private source data. However, existing methods rely on the pseudo-labels generated by source models that can be noisy due to domain shift. In this paper, we study SFDA from the perspective of learning with label noise (LLN). Unlike the label noise in the conventional LLN scenario, we prove that the label noise in SFDA follows a different distribution assumption. We also prove that such a difference makes existing LLN methods that rely on their distribution assumptions unable to address the label noise in SFDA. Empirical evidence suggests that only marginal improvements are achieved when applying the existing LLN methods to solve the SFDA problem. On the other hand, although there exists a fundamental difference between the label noise in the two scenarios, we demonstrate theoretically that the early-time training phenomenon (ETP), which has been previously observed in conventional label noise settings, can also be observed in the SFDA problem. Extensive experiments demonstrate significant improvements to existing SFDA algorithms by leveraging ETP to address the label noise in SFDA.

Published as a conference paper at ICLR 2023
WHEN SOURCE-FREE DOMAIN ADAPTATION MEETS LEARNING WITH NOISY LABELS
Li Yi!
Gezheng Xu Pengcheng Xu²
Jiaqi Li Boyu Wang 122
Ruizhi Pu²
Charles Ling* A. Ian McLeod
'Department of Statistical and Actuarial Sciences University of Western Ontario gxu86, pxu67, j113779, rpu2, charles ling, aimcleod) @uwo. ca bwangecsd. ca
Department of Computer Science
ABSTRACT
Recent state-of-the-arr source-free domain adaptation (SFDA) methods have fo- cused on learning meaningful cluster structures in the feature space, which have succeeded in adapting the knowledge from source domain to unlabeled target domain without accessing the private source data. However, existing methods rely on the pseudo-labels generated by source models that can be noisy due domain shift. In this paper, we study SFDA from the perspective of learning with label noise (LLN).. Unlike the label noise in the conventional L.LN scenario, we prove that the label noise in SFDA follows a different distribution assumption. We also prove that such a difference makes existing LLN methods that rely on their distribution assumptions unable address the label noise in SFDA. Empirical evidence suggests that only marginal improvements are achieved when applying the existing LLN methods to solve the SFDA problem. On the other hand, although there exists fundamental difference between the label noise in the two scenar- ios, demonstrate theoretically that the early-time training phenomenon (ETP), which has been previously observed in conventional label noise settings, can also observed in the SFDA problem. Extensive experiments demonstrate significant improvements to existing SFDA algorithms by leveraging ETP to address the label noise in SFDA.
INTRODUCTION
Deep learning demonstrates strong performance on various tasks across different fields. However, is limited by the requirement of large-scale labeled and independent, and identically distributed (i.i.d.) data. Unsupervised domain adaptation (UDA) is thus proposed to mitigate the distribution shift between the labeled source and unlabeled target domain. In view of the importance data privacy, it is crucial to be able to adapt a pre-trained source model to the unlabeled target domain without accessing the private source data, which is known as Source Free Domain Adaptation (SFDA).
The current offteoofthh-art art SFDA methods (Liang al. 2020; Yang et al. 202 la b) mainly focus on learning meaningful cluster structures in the feature space, and the quality the learned cluster structures hinges the reliability of pseudo labels generated by the source model. Among these
methods, SHOT (Liang et al. 2020) purifies pseudo labels of target data based on nearest centroids, and then the purified pseudo labels are used guide the self-training. G-SFDA (Yang et al. 202 Ib)
and NRC (Yang et al. (021a) further refine pseudo labels by encouraging similar predictions to the data point and its neighbors. For a single target data point, when most of its neighbors predicted, these methods can provide an accurate pseudo label to the data point. However, as we illustrate the problem in Figure [[i a-b), when the majority its neighbors are incorrectly predicted to category, it will be assigned with an incorrect pseudo label, misleading the learning of cluster structures. The experimental result on VisDA (Peng et al.) (017). shown in Figure Titl further verifies this phenomenon. By directly applying the pre-trained source model each target domain instance
correctly
"quual contribution Corresponding author

---

Published as a conference paper at ICLR 2023
(6)
Scurcdstodel
bb existing SFDA
Feadd
(c): ours
Sourca/Urrae ed target data
Label
Mistaeledd target cata
Correctt predicted cata
(i) Overview of the SFDA problem and our method
Neighboes Label Noise Andlysis On VisDA
Figure 1: (i) (a) The SFDA problem can be formulated LLN problem. (b) The existing SFDA algorithms using the local cluster information cannot address label noise due to the unbounded label noise (Section 3). (c) We prove that ETP exists in SFDA, which can be leveraged address the unbounded label noise (Section T. (ii) Observed Label Noise Phenomena on VisDA dataset.
(central instance), we collect its neighbors and evaluate their quality. We observed that for cach class a large proportion of the neighbors are misleading (i.e., the neighbors' pseudo labels are different from the central instance's true label), some even with high confidence (e.g., the over-confident misleading neighbors whose prediction score is larger than 0.75). Based on this observation, we can conclude that: (1) the pseudo labels leveraged in current SFDA methods can be heavily noisy; (2) some pseudo-label purification methods utilized in SFDA, which severely rely on the quality of the pseudo label itself, will be affected by such label noise, and the prediction error will accumulate as the training progresses. More details can be found in Appendix
In this paper, we address the aforementioned problem by formulating SFDA learning with label noise (LLN). Unlike existing studies that heuristically rely on cluster structures or neighbors, we investigate the properties of label noise in SFDA and show that there is intrinsic discrepancy between the SFDA and the LLN problems. Specifically, in conventional LLN scenarios, the label noise generated by human annotators or image search engines (Patrini al.) 2017: Xiao al. 2015 Xia et al., (2200a). where the underlying distribution assumption is that the mislabeling rate for sample is bounded. However, in the SFDA scenarios, the label noise is generated by the source model due to the distribution shift, where we prove that the mislabeling rate for sample is much higher, and can approach 1. We term the former label noise in L.NN as bounded label noise and the latter label noise in SFDA as unbounded label noise. Moreover, we theoretically show that most existing LLN methods, which rely on bounded label noise assumption, are unable to address the label noise in SFDA due to the fundamental difference (Section 3).
To this end, we leverage early-time training phenomenon (ETP) in LLN to address the unbounded label noise and to improve the efficiency of existing SFDA algorithms. Specifically, ETP indicates that classifiers can predict mislabeled samples with relatively high accuracy during the early learning phase before they start to memorize the mislabeled data (Liu et al.) 2020). Although ETP has been previously observed in, it has only been studied in the bounded random label noise in the conventional LLN scenarios. In this work, we theoretically and empirically show that ETP still exists in the unbounded label noise scenario of SFDA. Moreover, we also empirically justify that existing SFDA algorithms can be substantially improved by leveraging ETP, which opens up a new avenue for SFDA. As an instantiation, we incorporate simple early learning regularization (ELR) term (Liu et al. 2020) with existing SFDA objective functions, achieving consistent improvements on four different SFDA benchmark datasets. As comparison, we also apply other existing LIN methods, including Generalized Cross Entropy (GCE) (Zhang & Sabuncu 2018). Symmetric Cross Entropy Learning (SL) (Wang | (2019b). Generalized Jensen-Shannon Divergence (GJS) (Englesson & Azizpour 2021 and Progressive Label Correction (PLC) (Zhang et al. 2021), to SFDA. Our empirical evidence shows that they are inappropriate for addressing the label noise in SFDA. This is also consistent with our theoretical results (Section 4).
Our main contribution can be summarized as: (1) We establish the connection between the SFDA and the LLN. Compared with the conventional LLN problem that assumes bounded label noise, the problem in SFDA can be viewed as the problem of LLN with the unbounded label noise. (2)
2

---

Published as a conference paper at ICLR 2023
We theoretically and empirically justify that ETP exists in the unbounded label noise scenario. On the algorithmic side, we instantiate our analysis by simply adding a regularization term into the SFDA objective functions. (3) We conduct extensive experiments to show that ETP can be utilized to improve many existing SFDA algorithms by a large margin across multiple SFDA benchmarks.
2 RELATED WORK
Source-free domain adaptation. Recently, SFDA are studied for data privacy. The first branch of research is to leverage the target pscudo labels to conduct self-training to implicitly achieve adaptation (Liang et al. 2021 Tanwisuth et al. 2021 Ahmed et al. 2021 Yang et al. 2021b). SHOT (Liang et 20.22 introduces k-means clustering and mutual information maximization strategy for self-training. NRC (Yang et al. 2021a) further investigates the neighbors of target clusters to improve the accuracy of pseudo labels. These studies more or less involve pseudo-label purification processes, but they are primarily heuristic algorithms and suffer from the previously mentioned label noise accumulation problem. The other branch is to utilize the generative model to synthesize target-style training data (Qiu et al.) 2021; Liu et al., (021b). Some methods also explore the SFDA algorithms in various settings. USFDA (Kundu et al., 2020a) and FS (Kundu et al. (020b) design methods for universal and open-set UDA. In this paper, we regard SFDA as the LLN problem. We aim to explore what category of noisy labels exists in SFDA and to ameliorate such label noise to improve the performance of current SFDA algorithms.
Learning with label noise. Existing methods for training neural networks with label noise focus on symmetric, asymmetric, and mnstance-dependen label noise. For example, a branch of research focuses on leveraging noise-robust loss functions to cope with the symmetric and asymmetric noise, including GCE (Zhang& & Sabuncu 2018). SL (Wang et al. 2019b), NCE (Ma et al. 2020), and GIS (Englesson & Azizpour 2021 I), which have been proven effective in bounded label noise. On the other hand, CORES (Cheng et al. 2020) and CAL (Zhu al.
2021) are shown useful in mitigating instance-
dependent label noise. These methods are only tailed to conventional LLN settings. Recently, Liu al. (020)) has studied early-time training phenomenon (ETP) in conventional label noise scenarios and proposes a regularization term ELR to exploit the benefits of ETP. PCL (Zhang et al.) 202I) is another conventional LLN algorithm utilizing ETP, but it cannot maintain the exploit of ETP SFDA as memorizing noisy labels is much faster in SFDA. Our contributions are: (1) We theoretically and empirically study ETP in the SFDA scenario. (2) Based on an in depth analysis of many existing LLN methods (Zhang & Sabuncu 2018 Wang et al. 2019b] Englesson & Azizpour 2021 Zhang
ct al. 202I), we demonstrate that ELR is useful for many SFDA problems.
LABEL NOISE IN SFDA
The presence of label noise on training datasets has been shown to degrade the model performance 2017 Han et al. 2018). In SFDA, existing algorithms rely on pseudo- labels produced by the source model, which are inevitably noisy due to the domain shift. The SFDA methods such as Liang et al. 20200 Yang et al. (2021a b) cannot tackle the situation when some target samples and their neighbors are all incorrectly predicted by the source model. In this section, we formulate the SFDA the problem of L.LN to address this issue. We assume that the source domain Ds and the target domain Dr follow two different underlying distributions over X' x ), where X' and ) are respectively the input and label spaces. In the SFDA setting, we aim learn a target classifier f(x; 0) X - y only with pre-trained model fx( on Ds and set of unlabeled target domain observations drawn from Dr. We regard the incorrectly assigned pseudo-labels as noisy labels. Unlike the "bounded label noise" assumption in the conventional LLN domain, we will show that the label noise in SFDA is unbounded. We further prove that most existing LLN methods that rely on the bounded assumption cannot address the label noise in SFDA due to the difference.
(Malach &
Label noise in conventional L.LN settings: In conventional label noise settings, the injected noisy labels are collected by either human annotators or image search engines (Lee et al. 20018] Li ct al. 2017 Xiao et al. 2015). The label noise is usually assumed to be either independent of instances (i.c.. symmetric label noise asymmetric label noise) (Patrini et al. 2017 Liu & Tao 2015 Xu et al.
2019b) or dependent of instances (i.e., instance restance-dependen label noise) (Berthon et al.
2021 Xia et al.
2020b). The underling assumption for them is that a sample has the highest probability of being
in the correct class y, i.e., Pr(ý i|Y i, X x] Pr(ý = Y i, X =x), Va X j.
3

---

Published as a conference paper at ICLR 2023
where Y is the noisy label and Y is the ground-truth label for input X. Equivalently, it assumes bounded noise rate. For example, given an image to annotate, the mislabeling rate for the image is bounded by small number, which is realistic in conventional LLN settings (Xia et al. 2020b [Cheng et al. 2020). When the label noise is generated by the source model, the underlying assumption of these types of label noise does not hold.
Label noise in SFDA: As for the label noise generated by the source model, mislabeling rate for an image can approach 1, that is, Pr[ý = j|Y i, X = - 1, 3S C X, Ve E E j. To understand that the label noise in SFDA is unbounded, Gaussian mixture distribution with equal priors for both domains. Let the first component (y 1) of the source domain distribution D be N ((1,o²La)) L'),, and the second component (y -1) of Ds be (M22 I'),), where μ1, P22 E Rd and Ia E Rdxd__ For the target domain distribution Dy. let the first component (y = 1) of Dr be N(p1 + A, da))), and the second component (y 11) of Dr be .V ((22 + A, o La),, where A E IR is the shift of the two domains. Notice that the domain shift considered a general shift and has been studied in [Stojanov et al. (202T); (Zhao et al. (2019). where we also illustrate the domain shift in Figure in supplementary material.
consider two-component Multivariate
Let Is be the optimal source classifier. First, we build the relationship between the mislabeling rate for target data and the domain shift:
Pr [fs(x) * y) = _ 0 - + 2 1 (x.9)~DT
d1
1
d2 o
(I)
where d1 = 112 m1 cl| sign(
H2 ا
||c]]] d2 |1 mn + cll. c a p1),
m1) is the magnitude of domain shift, and + is the standard normal cumulative distribution function. Eq. (1) shows that the magnitude of the domain shift inherently controls the mislabeling error for target data. This mislabeling rate increases as the magnitude of the domain shift increases. We defer the proof and details to Appendix B
A ("a |
More importantly, we characterize that the label noise is unbounded among these mislabeled samples. Theorem 3.1. Without loss of generality, we assume that the A is positively correlated with the vector μ22 fA,, Le., A (122 m1)= > O. For (x, v) Dr, if x E R, then
Pr(fs(x) + vl 2 1 5, where 5 E (0, 1) (i.e., = 0.01). = R. n R2 R. = (x x p1 11 A
(2)
a Vd
log 1-8
)},
/d
and R2 {x: : 1d > (ad 2u 22). Meanwhile, R is non-empty when a >> (log
5 )/d,
where a= A (#2-#1) > 0 is the magnitude of the domain shift along the direction μ12 111.
1l 1122-1111" |=
Conventional LLN methods assume that the label noise is bounded: Pr[ff(() Fy] < m, Y(x, y) ~
Dr. where JH is the labeling function, and m 0.5 if the number of clean samples of each component are the same (Cheng et al.) 2020). However, Theorem 3.1 indicates that the label noise generated by the source model is unbounded for any x E R. In practice, region R is non-empty as neural networks are usually trained on high dimensional data such that d > 1. easy to satisfy. The probability measure on = R, R2 (i.e., Pr(x.y)>>r x R)) increases as the magnitude of the domain shift a increases, meaning more data points contradict the conventional LLN assumption. More details can be found in Appendix C
80 a > (log I- )/d -
Given that the unbounded label noise exists in SFDA, the following Lemma establishes that many existing LLN methods (Wang et al. 2019b) Ghosh et al.) 2017; Englesson & Azizpour 2021 Ma
et al. 2020), which rely on the bounded assumption, are not noise tolerant in SFDA.
Lemma 3.2. Let the risk of the function h X + y under the clean data be R(h) Ex.y (exx)))))((())......)))))))))))) y)], and the risk of h under the noisy data be R(h) = (x,(()un(((x)), 7) where the noisy data follows the unbounded assumption, i.e.. Prlğ # y R = 1 8 for subset
R X and ó 10. Then the global minimizer h² of R(h) disagrees with the global minimizer ht of R(h) on data points x E R with high probability at least 1 5. 8.
We denote fIN by the existing noise-robust loss based LLN methods in Wang et al. (2019b); Ghosh et al. 2. Englesson & Azizpour (202T): Ma et al. (2020). When the noisy data follows the bounded ussumption, these methods are noise tolerant as the minimizer h* converges to the minimizer h with high probability. We defer the details and proof of the related L.N methods to Appendix D
4

---

Published as a conference paper at ICLR 2023
4 LEARNING WITH LABEL NOISE IN SFDA
Given a fundamental difference between the label noise in SFDA and the label noise in conventional LLN scenarios, existing LLN methods, whose underlying assumption is bounded label noise, cannot be applied to solve the label noise in SFDA. This section focuses on investigating how to address the unbounded label noise in SFDA.
Motivated by the recent studies Liu et al. 2020); Arpit et al. (017)). which observed an early-time training phenomenon (ETP) on noisy datasets with bounded random label noise, we find that ETP does not rely on the bounded random label noise assumption, and it can be generalized to the unbounded label noise in SFDA. ETP describes the training dynamics of the classifier that preferentially fits the clean samples and therefore has higher prediction accuracy for mislabeled samples during the early-trainine stage. Such training characteristics can be very beneficial for SFDA problems in which we only have access to the source model and the highly noisy target data. To theoretically prove ETP in the presence of unbounded label noise, we first describe the problem setup.
We still consider two-component Gaussian mixture distribution with equal priors. We denote y by the true label for x, and assume it a balanced sample from (-1++1). The instance x is sampled from the distribution (yj,, o'aa). where || = 1. We denote by the noisy label for x. We observe that the label noise generated by the source model is close to the decision boundary revealed in Theorem 3.1 So, to assign the noisy labels, we let y3 y). where ((x, y) sign (1( f4 > r} 0.5) is the label flipping function, and r controls the mislabeling rate. If (xx, y) < 1, then the data point is mislabeled. Meanwhile, the label noise is unbounded by adopting the label flipping function ((x, y): Pr(ğ & v|yx / S r] = 1, where R = [x yxx J S r).
We study the early-time training dynamics of gradient descent on the linear classifier. The parameter 0 is learned over the unbounded label noise data (Iz. Øe )= with the following logistic loss function:
[(1++1) = Σ log (1 + exp (-0:0
where 01+1 02 VoC())) and is the learning rate. Then the following theorem builds the connection between the prediction accuracy for mislabeled samples at an sarly-training time T.
Theorem 4.1. Let B = {x: + y) be set of mislabeled samples. Let 1o (B: O) be the prediction accuracy calculated by the ground-trut ter for mislabeled samples. If at most half of the samples are mislabeled (r 1), then there exists proper time T and constant > 0 such that for any O < a < co and n + co, with probability
labels and the predicted labels by the classifier with parame-
1 (1))
(B:0T)= 2 1- exp( exexp
200 (
(3)
Erfl ا-r exp( ( 2(1+2o)a
where g(0)
0 is monotone decreasing function that g(o) - oo as
(2+(1+20)
o + 0. and Erf[x] =
2
let
e - dt.
Vi
The proof is provided in Appendix E Compared to ETP found in Liu et al. 2020)), where the label noise is assumed to be bounded, Theorem presents that ETP also exists even though the label noise is unbounded. At proper time T. the classifier trained by the gradient descent algorithm can provide accurate predictions for mislabeled samples, where its accuracy is lower bounded by function of the variance of clusters When a + O. the predictions of all mislabeled samples equal to their ground-truth labels (i.e., x(B 07) + 1). When the classifier trained for a sufficiently long time, it will gradually memorize mislabeled data. The predictions of mislabeled samples are equivalent to their incorrect labels instead of their ground -tutt labels (Liu et al. 2020 Maennel et al.) 2020). Based on these insights, the memorization of mislabeled data can be alleviated by leveraging their predicted labels during the early-training time.
To leverage the predictions during the early-training time, adopt a recently established method, early learning regularization (ELR) (Liu et al.) (2020). which encourages model predictions to stick to the early-time predictions for Since ETP exists in the scenarios of the unbounded label noise, ELR can be applied to solve the label noise in SFDA. The regularization is given by:
LELR (0:) log(1 v. [xx 0t)),
5

---

Published as a conference paper at ICLR 2023
Tainna ACC.
Tham PPamnng ACE
Traiiing Arc.
5
80
Traiiing
70
ELR
2
61 ELR
OCE
20 Synthoric
60
80
100
140
25
50
100
125
150
175
40 00
320 140
Rcal
amazon
webcam
(i VisDA-C
(ii) DomainNet
(iii)
(iv) Office-31
Figure 2: Training accuracy on various target domains. The source models initialize the classifiers and annotate unlabeled target data. As the classifiers memorize the unbounded label noise very fast, for the first 90 steps, we evaluate the prediction accuracy on target data every batch, and one step represents one training batch. After the 90 steps, we evaluate the prediction accuracy for every 0.3 epoch, shown as one step. We use the CE, GCE, and ELR to train the classifiers on the labeled target data, shown in solid green lines, solid orange lines, and solid blue lines, respectively. The dotted red line represents the accuracy of labeling target data. Eventually, the classifiers memorize the label noise, and the prediction accuracy equals the labeling accuracy (shown in (iii-iv)). Additional results on transfer pairs can be found in Appendix F
where we overload f(x 0c) to be the probabilistic output for the sample x, and Gr = -ih-1 1 + (1 (x; 9.) is the moving average prediction for x, where B3 is hyperparameter. To see how ELR prevents the model from memorizing the label noise, we calculate the gradient of Eq. (4) with respect to [xx 0z), which given by:
(LLLR((:)) df(x;; 0() 1 D. [xx 0z)
Note that minimizing Eq. (4) forces f(x; 02) to close to Tt. When ÿe is aligned better with fx; 0z). the magnitude of the gradient becomes larger. It makes the gradient of aligning fx; 0.) with yr overwhelm the gradient of other loss terms that align f(x; with noisy labels. As the training progresses, the moving averaged predictions Ve for target samples gradually approach their ground- truth labels till the time T. Therefore, Eq. (4) prevents the model from memorizing the label noise by forcing the model predictions to stay close to these moving averaged predictions Dt. which are very likely to be ground-trutt labels.
Some existing LLN methods propose to assign pseudo labels to data or require two-stage training for label noise (Cheng et al., (2200) Zhu et al., [021] Zhang al., (021). Unlike these LLN methods, Eq. (4) can be easily embedded into any existing SFDA algorithms without conflict. The overall objective function is given by:
L= LSFDA + XLELR,
(5)
where LSEDA is any SFDA objective function, and À is hyperparameter.
Empirical Observations on Real- World Datasets. We empirically verify that target classifiers
have higher prediction accuracy for target data during the early training and adaptation stage. We propose leveraging this benefit to prevent the classifier from memorizing the noisy labels. The observations are shown in Figure 2 The parameters of classifiers are initialized by source models. Labels of target data annotated by the initialized classifiers. We train the target classifiers target data with the standard cross-entropy (CE) loss and the generalized cross-entropy (GCE) loss, a well-known noise-robust loss widely leveraged in bounded LLN scenarios. The solid green, orange and blue lines represent the training accuracy of optimizing the classifiers with CE loss, GCE loss, and ELR loss, respectively. The dotted red lines represent the labeling accuracy of the initialized classifiers. Considering that the classifiers memorize the unbounded label noise very fast, we evaluate the prediction accuracy on target data every batch for the first 90 steps. After 90 steps, we evaluate the prediction accuracy for every 0.33 epoch. The green lines show that ETP exists in SFDA, which is consistent with our theoretical result. Meanwhile, in all scenarios, green and orange lines show that classifiers provide higher prediction accuracy during the first a few iterations. After few iterations, they start to memorize the label noise even with noise-robust loss (e.g.. GCE). Eventually, the classifiers are expected to memorize the whole datasets. For conventional LLN settings, it has been empirically verified that it takes a much longer time before classifiers start memorizing the label noise (Liu al. 2020 Xia et al. 2020a). We provide further analysis in Appendix H We highlight

---

Published as a conference paper at ICLR 2023
Table I: Accuracies (%) on Office-Home for ResNet50-based methods
Method
SHAr
+CTAr Ar CI-AAIC---HH -Rw
Pt-PP-
-CIP PirPrt--www RRw
AArRw---- MCC.R---H Avg
MCD
Saito et all
80
X
48.9
68.3
74.6
61.3
67.6
68.8
$7.0
47.1
75.1
69.1
52.2
79.6
64.1
CDAN
2011
X 50.7
70.6
76.0
57.6
70.0
70.0
57.4
50.9
77.3
70.9
56.7
81.6
65.8
SAFN
(X mi 20119
X 52.0
71.7
76.3
64.2
69.9
71.9
63.7
51
77.1
70.9
57.1
81.5
67.3
Symnets
Zhang
20
47.7
72.9
78.5
64.2
71.3
74.2
64.2
48.8
79.5
74.5
52.6
82.7
67.6
MDD
Zhang cu. 20196
X
54.9
73.7
77.8
60.0
71.4
718
61.2
53.6
78.1
72.5
60.2
82.3
68.1
TADA
Wang
53.1
72.3
77.2
59.1
71.2
72.1
59.7
53.1
78.4
72.4
60.0
82.9
67.6
BNM
[Cui et
2020
X
52.3
73.9
80.0
63.3
72.9
74.9
61.7
49.5
79.7
70.5
53.6
82.2
67.9
BDG
Lang el
2020
51.5
73.4
65.3
71.5
65.1
49.7
81.1
74.6
55.1
84.8
68.7
SRDC
Tang et 2020
X
52.3
76.3
81.0
69.5
76.2
78.0
68.7
53.8
81.7
76.3
57.1
85.0
71.3
RSDA MSTN
53.2
77.7
66.4
74.0
76.5
67.9
53.0
82.0
75.8
57.8
85.4
70.9
Source Only
44.6
67.3
74.8
52.7
62.7
53.0
40.6
73.2
65.3
45.4
78.0
60.2
+ELR
52.4
73.5
77.3
62.5
70.6
71.0
61.1
50.8
78.9
71.7
56.7
81.6
67.3
SHOT [Liang al. 2020
57.1
78.1
81.5
68.0
78.2
78.1
67.4
54.9
82.2
73.3
58.8
84.3
71.8
+ELK
/
58.7
78.9
82.1
68.5
79.0
77.5
68.2
57.1
81.9
74.2
59.5
84.9
72.6
G-SEDA (Yang al.
2b
55.8
77.1
80.5
66.4
74.9
77.3
66.5
53.9
80.8
72.4
59.7
83.2
70.7
+ELR
56.4
77.6
81.1
67.1
75.2
77.9
65.9
55.0
81.2
72.1
60.0
83.6
71.1
NRC
Yang et al. 2021a
56.3
77.6
81.0
65.3
78.3
77.5
64.5
56.0
82.4
70.0
57.1
82.9
70.8
+ELK
58.4
78.7
81.5
69.2
79.5
79.3
66.3
58.0
82.6
73.4
59.8
85.1
72.6
that PCL Zhang et al. 2021) leverages ETP at every epoch, so it cannot capture the benefits of ETP and is inappropriate for unbounded label noise due to the fast memorization speed in SFDA. As comparison, we choose ELR since it leverages ETP at every batch. The blue lines show that leveraging ETP via ELR can address the memorization of noisy labels in SFDA.
5 EXPERIMENTS
We aim to improve the efficiency of existing SFDA algorithms by using ELR to leverage ETP. We evaluate the performance four different SFDA benchmark datasets: Office-31 (Saenko et al. 2010). Office-Home (Venkateswara et al., (017). VisDA (Peng et al., (017) and Domain (Peng et al.) 2019). Due to the limited space, the results on the dataset Office-31 and additional experimental details are provided in Appendix G
Evaluation. We incorporate ELR into three existing baseline methods: SHOT (Liang al. 2020), G-SFDA (Zhang & Sabuncu 2018). and NRC (Yang et al. (021a)) SHOT uses k-means clustering and mutual information maximization strategy to train the representation network while freezing the final linear layer. G-SFDA aims to cluster target data with similar neighbors and attempts to maintain the source domain performance. NRC also explores the neighbors of target data by graph-based methods. ELR can be easily embedded into these methods by simply adding the regularization term into the loss function to optimize without affecting existing SFDA frameworks. We average the results based on three random runs.
Results. Tables 4 show the results before/after leveraging the early-time training phenomenon, where Table 4 is shown in Appendix G Among these tables, the top part shows the results of conventional UDA methods, and the bottom part shows the results of SFDA methods. In the tables, SF indicate whether the method is source free or not. We use Source Only ELR to indicate ELR with self-training. The results show that ELR itself can boost the performances. As existing SFDA methods are not able address unbounded label noise, incorporating ELR into these SFDA methods can further boost the performance. The four datasets, including all 31 pairs (e.g., A + D) of tasks, show better performance after solving the unbounded label noise problem using the early-time training phenomenon. Meanwhile, solving the unbounded label noise on existing SFDA methods achieves sfate-of-ther art on all benchmark datasets. These SFDA methods also outperform most methods that need to access source data.
Analysis about hyperparameters B and X. The hyperparameter ,3 is chosen from (0.5. 0.6, 0.7, 0.8, 0.9, 0.99), and À is chosen from (1, 3, 7, 12, 25). We conduct the sensitivity study on hyperparameters of ELR on the DomainNet dataset, which is shown in Figure Ba-b). In each Figure, the study is conducted by fixing the other hyperparameter to the optimal one. The performance is robust to the hyperparameter 3 except B = 0.99. When 3 = 0.99, classifiers are sensitive to changes in learning curves. Thus, the performance degrades since the learning curves change quickly in the unbounded label noise scenarios. Meanwhile, the performance is also robust to the hyperparameter except when À becomes too large. The hyperparameter X is to balance the effects of existing SFDA
7

---

Published as a conference paper at ICLR 2023
barely using ELR to address the SFDA problem is not comparable to these SFDA methods. Hence, a large value of makes neural networks neglect the effects of these SFDA methods, leading to degraded performance.
algorithms and the effects of ELR. As indicated in Tables 14
Table 2: Accuracies (%) on DomainNet for ResNet50- based methods.
Method
SFR
+CR
-PR
+SC
RCC
sPC
>SP
>R
-CP
>SS
RRS
CCS
-CCSSPPA
Avg
MCD
(Sito et al.
2018b)
X
61.9
69.3
56.2
79.7
56.6
53.6
83.3
58.3
60.9
81.7
56.2
66.7 65.4
DANN
Ganin ct al.
2016)
X
63.4
73.6
72.6
86.5
65.7
70.6
86.9
73.2
70.2
85.7
75.2
70.0 74.5
DAN
Long al.
201.5
x
64.3
70.6
58.4
79.4
56.7
60.0
84.5
61.6
62.2
79.7
65.0
62.0 67.0
COAL
Tan
2020
73.9
75.4
70.5
89.6
70.0
71.3
89.8
68.0
70.5
88.0
73.2
70.5 75.9
MDD Zhang al.
2019b)
x
77.6
75.7
74.2
89.5
74.2
75.6
90.2
76.0
74.6
86.7
72.9
73.2 78.4
Source Only
53.7
71.6
52.9
70.8
49.5
58.3
85.2
59.6
59.1
30.6
74.8
65.7 61.0
+ELR
70.2
81.7
61.7
79.9
63.8
67.0
90.0
72.1
66.8
85.1
78.5
68.8 73.8
SHOT [Liang al.
2020)
73.3
80.1
65.8
91.4
74.3
69.2
91.9
77.0
66.2
87.4
81.3
75.0 77.7
+ELR
78.0
81.9
67.4
91.1
75.9
71.0
92.6
79.3
68.0
88.7
84.8
77.0 79.7
G-SFDA YYng et al.
2021b) /
65.8
78.9
60.2
80.5
64.7
64.6
89.3
69.9
63.6
86,4
78.8
71.1 72.8
+ELR
69.4
80.9
60.6
81.3
67.2
66.4
90.2
73.2
64.9
87.6
82.1
71.0 74.6
NRC Yang et a0
69.8
81.1
62.9
83.4
74.4
66.3
90.3
73.4
65.2
88.2
82.2
75.8 76.4
+ELR
75.6
82.2
65.7
91.2
77.2
68.5
92.7
79.8
67.5
89.3
85.1
77.6
79.4
80
80
SHOT
NRC
8 Accuracy
79
Test Accuracy
Test Accures)
78:
78
78-
77
77
NRC
NRC
76-
SHOT
76
SHOI
0.5
0.6
0.7
0.3
0.9
0.99
12
25
Vanilla
GCE
SL
GJS
ELR
(c) embedding differrnt LLN
(a) hyperparameter
(b) hyperparameter
methods into SFDA algorithms
Figure 3: (b)-(b) show the test accuracy on the DomainNet dataset with respect to hyperparameters of ELR. (c) shows the test accuracy of incorporating various existing LLN methods into the SFDA methods on the DomainNet dataset.
5.1 DISCUSSION ON EXISTING LLN METHODS
As we formulate the SFDA as the problem of LLN, it is interest discuss some existing LLN methods. We mainly discuss existing LLN methods that can be easily embedded into the current SFDA algorithms. Based on this principle, we choose GCE (Zhang & Sabuncu 2018), SL (Wang al. 2019b) and GIS (Englesson & Azizpour 2021) that have been theoretically proved to be robust to symmetric and asymmetric label noise, which are bounded label noise. We highlight that a more recent method GJS outperforms ELR in real-world noisy datasets. However, will show that GJS is inferior to ELR SFDA scenarios, because the underlying assumption for GIS does not hold in SFDA. Besides ELR, which leverages ETP, PCL is another method leverage the same phenomenon, but we will show that it is also inappropriate for SFDA.
To show the effects of the existing LLN methods under the unbounded label noise, we test these LLN methods on various SFDA datasets with target data whose labels are generated by source models. As shown in Figure GCE, SL. GJS, and PCL are better than CE but still not comparable to ELR. Our analysis indicates that ELR follows the principle of ETP, which is theoretically justified in SFDA scenarios by our Theorem 3.1 Methods GCE, SI. and GJS follow the bounded label noise assumption, which does not hold in SFDA. Hence, they perform worse than ELR in SFDA, even though GJS outperforms ELR conventional LLN scenarios. PCL (Zhang et al. 2021) utilizes ETP to purify noisy labels of target data, but it performs significantly worse than ELR. As the memorization speed the unbounded label noise is very fast, and classifiers memorize noisy labels within few iterations (shown in Figure 2) purifying noisy labels every epoch is inappropriate for SFDA. However, we notice that PCL performs relatively better on DomainNet than on other datasets. The reason behind it is that the memorization speed in the DomainNet dataset is relatively slow than
8

---

Published as a conference paper at ICLR 2023
Table Accuracies (%) on VisDA-C (Synthesis + Real) for R methods.
Method
SFilanebeyel bus car
Per-class
DANN Ganin
2016
81.9 77.7 82.8 81.2
29.5
65.1
28.6
51.9
54.6
82.8
7.8
57.4
DAN
Long cl al
2015)
87.1 63.0 76.542.0 90.3 42.9
85.9
53.1
49.7
36.3
85.8
20.7
61.1
ADR
Saito ct al.
2018a
94.2 48.5 84.072.9 90.1 74.2
92.6
72.5
80.8
61.8
82.2 28.8
73.5
CDAN
Long ct
2018
85.2 66.9 83.050.8 84.2
74.9
88.1
74.5
83.4
76.0
81.9 38.0
73.9
SAFN
Xu el al.
2019a>
93.6 61.3 84.1 94.1
79.0
91.8
79.6
89.9
55.6
89.0 24.4
76.1
SWD
Lee el al.
2019)
90.8 82.5 81.770.5 91.7
69.5
86.3
77.5
87.4
63.6
85.6 29.2
76.4
MDD
Zhang el
al.
20199b
74.6
MCC
jn et al.
2020)
88.7 80.3 80.571.5 90.1 93.2
85.0
71.6
89,4
73.8
85.0 36.9
78.8
STAR
Lu at
2020
95.0 84.0 84.673.0
91.6
91.8
85.9
78.4
94.4
84.7
87.0 42.2
82.7
RWOT
2020)
95.1 80.3 83.790.0 92.4
68.0
92.5
78.4
90.4
68.2
84.0
Source Only
60.9 21.6 50.967.6 65.8
6.3
82.2
23.2
57.3
30.6
84.6
8.0
46.6
+ELR
95.4 45.7 89.7 69.8 94.1
97.1
92.9
80.1
89.7
52.8
83.3
4.3
74.6
SHOT (Liang al.
2020
94.3 88.5 80. 57.3 93.1
94.9
80.7
80.3
91.5
89.1
58.2 86.3
82.9
+ELR
95.8 84.1 83.366.9 93.9
97.6
89.2
90.6
90.4
87.2
48.2
84.1
G-SFDA (Yagg et al.
2021b
96.0 87.6 85.372.8 95.9
94.7
88.4
79.0
92.7
93.9
87.2
43.7
84.8
+ELR
97.3 89.1 89.879.2 96.9
97.5
92.2
82.5
95.8
94.5
87.3 34.5
86.4
NRC Yang et al. 2021a
96.9 89.7 84.0
96.6
86.5
80.9
92.8
92.6
90.2
60.2
85.4
+ELK
97.1 89.7 82.7
62.0
96.2
97.0
87.6
81.2
93.7
94.1
90.2
58.6
85.8
80.0
64.5
64.0
64
83.0
80.3
12.0
Q5
(i) Of
(ii) Office-Home
(iii) VisDA
(iv) DomainNet
Figure Evaluation of label noise methods on SFDA problems. We use source models as an initialization of classifiers trained on target data and also source models annotate unlabeled target data. Then we treat the target datasets as noisy datasets and use different label noise methods to solve the memorization issue.
other datasets, which is shown in Figure 2 In conventional LLN scenarios, PCL does not suffer from the issue since the memorization speed is much lower than the conventional LLN scenarios.
In Figure BJc, also evaluate the performance by incorporating the existing LLN methods into the SFDA algorithms SHOT and NRC. Since PCL and SHOT assign pscudo labels to target data, PCL is incompatible with some existing SFDA methods and cannot be easily embedded into some SFDA algorithms. Hence, we only embed GCE, SL., GJS, and ELR into the SFDA algorithms. The figure illustrates that ELR still performs better than other L.NN methods when incorporated into SHOT and NRC. We also notice that GCE, SL, and GJS provide marginal improvement to the vanilla SHOT and NRC methods. We think the label noise in SFDA datasets the hybrid noise that consists of both bounded label noise and unbounded label noise due to the non-linearity of neural networks. The GCE. SL, and GJS can address the bounded label noise, while ELR can address both bounded and unbounded label noise. Therefore, these experiments demonstrate that using ELR to leverage ETP can successfully address the unbounded label noise SFDA.
6 CONCLUSION
In this paper, we study SFDA from a new perspective of LLN by theoretically showing that SFDA can be viewed as the problem of LLN with the unbounded label noise. Under this assumption, we rigorously justify that robust loss functions are not able to address the memorization issues of unbounded label noise. Meanwhile, based this assumption, we further theoretically and empirically analyze the learning behavior of models during the early-time training stage and find that ETP can benifit the SFDA problems. Through extensive experiments across multiple datasets, show that ETP can be exploited by ELR to improve prediction performance, and can also be used to enhance existing SFDA algorithms.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by Natural Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program.
REFERENCES
Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Koy-Chowdhury. Unsupervised multi-source domain adaptation without access to source data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10103-101 2021.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233-242.
PMLR, 2017.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In International Conference Machine Learning, pp. 825-836. PMLR, 2021.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: sample sieve approach. arXiv preprint arXiv:2010.02347, 2020.
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang. and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. CVPR, 2020.
Erik Englesson and Hossein Azizpour. Generalized jensen jensen-shannen shannon divergence loss for learning with noisy labels. arXiv preprint arXiv:2105.04522, 2021.
Keinosuke Fukunaga. Introduction statistical paltern recognition. Elsevier, 2013.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain aevesaril training of neural networks. The Journal of Machine Learning Research, 17(1):2096-200 2016.
Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo-label loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9101-9110, 2020.
Bo Han, Quanming Yao, Xingrui Yu, Gang NIu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing numan-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. (026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. ECCV, 2020.
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free free domain adap- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4544 4553, 2020a.
10

---

Published as a conference paper at ICLR 2023
Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, R Venkatesh Babu, et al. Towards inheritable models for open-set domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Viston and Pattern Recognition, pp. 12376-12383 2020b.
Chen-Yu Yu Lee, Tanmay Batra, Mohammad Haris Baig. and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10285-10295 2019.
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5447-5456, 2018.
Wen Li, Limin Wang. Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint / 2017.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028-6039, PMLR, 2020.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. Advances in Neural Information Processing Systems, 34, 2021a.
Lydia T. Liu, Max Simchowitz, and Moritz Hardt. The implicit faimess criterion of unconstrained learning. In Proceedings of the 36th International Conference on Machine Learning, pp. 4051- 4060, 2019.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. arXiv preprint arXiv:2007.0015// 2020,00011,,......,,,,22.2222..0
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-461, 2015.
Yuang Liu, Wei Zhang, and Jun Wang. Cource-free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1224, 2021b.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. ICML, 2015.
Mingsheng Long. Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1647-1657, 2018.
Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu, Yi-Zhe Song, and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9111 111110000 2020.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normal- ized loss functions for deep learning with noisy labels. In International Conference on Machine Learning, pp. 6543-6553, PMLR, 2020.
Hartmut Maennel, Ibrahim M Alabdulmohsin, Ilya O Tolstikhin, Robert Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? Advances Neural Information Processing Systems, 33:19693-19704, 2020.
Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". Advances in Neural Information Processing Systems, 30, 2017.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1944-1952, 2017.
11

---

Published as a conference paper at ICLR 2023
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Zhen Qiu, Yifan Zhang, Hongbin Lin, Shuaicheng Niu, Yanxia Liu, Qing Du, and Mingkui Tan. Source-free domain adaptation via avatar prototype generation and adaptation. arXiv preprini arXiv:2106. 15326, 2021.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models new domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regulariza- tion. ICLR, 2018a.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723-3732, 2018b.
Changjian Shui, Qi Chen, Jiaqi Li, Boyu Wang, and Christian Gagné. Fair representation learning through implicit path alignment. In Proceedings of the 39th International Conference on Machine Learning, pp. 20156-201 20175, 2022a.
Changjian Shui, Gezheng Xu, Qi CHEN, Jiaqi Li, Charles Ling, Tal Arbel, Boyu Wang, and Christian Gagné. On learning fairness and accuracy on multiple subgroups. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-19, 2022. doi: 10. 109/TNNLS 2022.3352220....55552
Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime Carbonell, and Kun Zhang. Domain adaptation with invariant representation learning: What transformations learn? Advances in Neural Information Processing Systems, 34:24791-24803, 2021.
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: an empirical odyssey. In European Conference Computer Vision, pp. 585-602. Springer, 2020.
Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8725-8735, 2020.
Korawat Tanwisuth, Xinjie Fan, Huangjie Zheng, Shujian Zhang, Hao Zhang, Bo Chen, and Mingyuan Zhou. A prototype-oriented framework for unsupervised domain adaptation. Advances Neural Information Processing Systems, 34, 2021.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5018-5027, 202,502222007, 2017.
Roman Vershynin. High-ddienssooaal probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
Boyu Wang, Jorge Mendez, Changjian Shui, Fan Zhou, Di Wu, Gezheng Xu, Christian Gagné, and Eric Eaton. Gap minimization for knowledge sharing and transfer, 2022.
Ximei Wang, Liang Li, Weirui Ye, Mingsheng Long, and Jianmin Wang. Transferable attention for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5345-5352, 2019a.
12

---

Published as a conference paper at ICLR 2023
Yisen Wang, Xingjun Ma. Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 322-330, 2019b.
Yuan Wu, Diana Inkpen, and Ahmed El-Roby. Dual mixup regularized learning for adversarial
domain adaptation. ECCV, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In International Conference on Learning Representations, 2020a.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7610, 2020b.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-2699, 2015.
Renjun Xu, Pelen Liu, Liyan Wang. Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4394 4403, 2020.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In The IEEE International Conference on Computer Vision (ICCV), October 2019a.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. I_dmi: An information-theorreie noise-robust loss function. arXiv preprint 1XXix:9909003380 2019b.
Guanglei Yang, Haifeng Xia, Mingli Ding, and Zhengming Ding. Bi-directional generation for unsupervised domain adaptation. In AAAI, pp. 6615-6622 2020.
Shiqi Yang. Joost van de Weijer, Luis Herranz, Shangling Jui, al. Exploiting the intrinsic neighbor- hood structure for source-free domain adaptation. Advances in Neural Information Processing Systems, 34, 2021a.
Shiqi Yang. Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on 2021b.
Computer Vision, pp.
Li Yi, Sheng Liu, Qi She, A. Ian McLeod, and Boyu Wang. On learning contrastive representations for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16682-16691, June 2022.
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain Nomain-symmetric networks for adversarial domain adaptation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5031-5040, 2019a.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: progressive approach. arXiv preprint arXiv:2103.07756 2021.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-7413, 2019b.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp. 7523-7532 PMLR, 2019.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10113-10123, 2021.
13

---

Published as a conference paper at ICLR 2023
A NEIGHBORS LABEL NOISE OBSERVATIONS ON REAL- WORLD DATASETS
This section provides more observed results and explanations of Neighbors' label noise during the Source-Free Domain Adaptation process real-world datasets.
6.40
Faue Seitton lero
0.90
6.30
Figure 5: True/False Neighbors on Office-Home
0.64
4.5
044
912
0.0
Flam 9p 9a
New
Figure 6: True/False Neighbors on VisDA
680
6:0
Cas Fdw
Figure 7: Neighbors Label Noise Analysis Office-Home
Currently, most SFDA methods inevitably leverage the pseudo-labels for self supervieee learning to learn the cluster structure of the target data in the feature space, in order realize the domain adaptation goal. However, the pseudo labels generated by the source domain are usually noisy and of poor quality due to the domain distribution shift. Some neighborhood-baid
heuristic methods (Yang et al. 2021a b) have been proposed to purify these target domain pseudo labels, which use the pseudo label of neighbors in the feature space to correct and reassign the central data's pseudo label. In fact, such methods rely on a strong assumption: a relatively high quality of the neighbors' pseudo label. However, in our experimental observations, we find that at the very beginning of the adaptation process, the similarity of two data points in the feature space can not fully represent their label space's is connection. Furthermore, such methods are easy to provide useless and noisy prediction information for the central data. We will show some statistical results on VisDA and Office-Honne Home, these two real-world datasets.
Following the neighborhood construction method in Yang et al. 1202 we use the pre-trained source model to infer the target data, extract the feature space outputs and get the prediction results. We use the cosine similarity on the feature space to find the top & similar neighbors (e.g.. k 2) for each data point (named as the central data point). Then, we collect the neighbors regarding the ground truth label of central data points and study the neighbor's quality for each class.
14

---

Published as a conference paper at ICLR 2023
Neighbors who do not belong to the correct category We define the neighbors who do not belong to the same category as its central data point as False Neighbor, which means their ground-truth labels are not the same: Yneighbor # Yeentrat- And the results of VisDA (train - validation) and
Office-Home (Pr -> CI) datasets are shown in Figure and Figure S
Neighbors who can not provide useful prediction information We further study the prediction information provided by such neighbors. Regardless of their true category properties, we consider neighbors whose Predicted Label the same as the Ground Truth Label of the central data point to be Useful Neighbors; otherwise, they are Misleading Neighbors, as they can not provide the expected useful prediction information. We denote the Misleading Neighbors Ratio as the proportion of noisy neighbors among all neighbors for each class. Besides, as some methods heuristically utilize the predicted logits as the predicted probability or confidence score in the pseudo label purification process, we further study the Over-Confident Misleading Neighbors Ratio for each class. We defined the over-confident misleading neighbors ratio as the number of over-confident misleading neighbors (misleading neighbors with high predicted logit, larger than 0.75) divided by the number of all neighbors per class. The results on VisDA and Office-Home are shown in Figure Li and Figure 7
We want to clarify that the above exploratory experiment results can only reflect the phenomenon of unbounded noise in SFDA to some extent: the set of over-confidence misleading neighbors is non-empty can correspond, to some extent, to the fact that R is non-empty proved in Theorem 3.1 but the definition of misleading neighbors does not rigorously satisfies the definition of unbounded label noise.
B RELATIONSHIP BETWEEN MISLABELING ERROR AND DOMAIN SHIFT
In this part, we focus on explaining the relationship between the label noise and the domain shift, as illustrated in Figure The following theorem characterizes the relationship between the labeling error and the domain shift.
Theorem B.1. Without loss of generality, we assume that the A is positively correlated with the vector 122 pt, iee., A (u2 - p1) > Let Is be the Bayes optimal classifter for the source domain S. Then
Pr [fs(x) + fy = D dg (x.y)~-T
1
d2
2
(6)
2
where d1 = 122 cll sign(
M2 1 ||=|)) d2 = m cll. c= (12 11)
A (Wa mn)
1|42 14111
and & is the standard normal cumulative distribution function.
Theorem B.I indicates that the labeling error for the target domain can be represented by a function of the domain shift A, which can be shown numerically in Figure g The projection of the domain shift A on the vector f42 f4 is given by Since e is on the direction of μ1 441 can also be represented by ax(2 - μ ). where a E R characterizes the magnitude of the domain shift. More specifically, in Figure 8 we present the relationship between the mislabeling rate and a for all possible A. When À is positively correlated with μ22 - Pr (assumption in Theorem B.J.. we have a > O, and when A is negatively correlated with μ2 141, we obtain a < O. In both situations, we can observe that the labeling error increases with the absolute value of a increasing, which implies that the more severe the domain shift is, the greater the mislabeling error will be obtained. Besides, note that when the source and target domains are the same, the mislabeling error in Eq. (6) is minimized and degraded the Bayes error, which cannot be reduced (Fukunaga) 2013). This corresponds to the situation when is perpendicular to /22 11, = 0, and a = O shown in Figure g
B. PROOFS FOR THEBRRE B.
Proof. The Bayes classifier fs predicts x to the first component when
Pryy = 1 = x] 0.
log Pr(y = -1
Since the distributions of the two components with the same priors for the source domain are given by N(1, oala) and Nu o2 respectively. Based on Bayes' rule, Eq. (7) is equivalent to
Pr[X = xly -
log
Pr[X x > 0
(8)
15

---

Published as a conference paper at ICLR 2023
04-
Target
6.4
03
-15
-10
03
0.0
03
15
sign(e
1el
Deg -Ill
1ps-
Figure 8: Plot of Mislabeling Rate with different a. We define e as the projection of the domain shift A the vector μl f11, and represents the magnitude of domain shift projected on f22
AA1-
Solving the left hand side of Eq. 8 by using the knowledge of two multivariate Gaussian distributions, we get
1] x (M1 u2) Pr = x|y==1
Pr[X xly
|
|22
his = log
x²
2a
So fs predicts x to the first component when h5(x) > 0 and fs predicts x to the second component when hx(x) I 0 The decision boundary such that hg (z) O. When there is domain shift
0, we have Dg 5 = Dr and the mislabeling rate is the Bayes error, which given by:
1
Pr Pr [Js(x) # vl x-N(Mn (xs(x) < Oly 1] + 2 x~N(uz.0
Pr
[1s(() > 0ly -1]
(x,y)~D3
(10)
We first study the first term Eq.
Pr xl
(xs(x) < 0ly=
[||x - u1 11l12 1|
[x|x (M1 -22) x 12-112111212 (2T02) $ exp
dz dra
20²
Ei= x?
Fd
cxp
dridz2 ... ..d
(x] Ex|-oe -00-00- 20 10 (2ma2)
20²
x3
cxp
dxd
2To²
20
du
① do
where the second equality is because of the rotationally symmetric property for isotropic Gaussian random vectors, & is the cumulative distribution function of the standard Gaussian distribution, and du = (14 f1) /21]. Applying the similar mathematical steps for the second term in Eq. (10). and take them into Eq. (T0):
122 11
Pr [fs(x) + y] = 40 (x.))~Ds
(11)
2a
When there is no domain shift, the labeling error is the Bayes error, which expressed by Eq.
Then consider the case when A + O. The distributions of the first and the second component N(p1 + A, La) and N(42 + A, I respectively. Notice that the decision boundary z is the affine hyperplane. Any shift paralleled to this affine hyperplane will not affect the final component predictions. The domain shift A can be decomposed into the sum of two vectors: the one is paralleled to this affine hyperplane, and another is perpendicular the hyperplane. It is straightforward to
16

---

Published as a conference paper at ICLR 2023
Mislabaling Area: with very high probabillty
Target Domain
Source Domain
Domain Shift
+
Mislabeled Target Data Ground Truth Classifier For Target Domain
Source Model
Figure Illustration of noisy labels generated by the domain shift.
verify that 142 M4 is perpendicular to the hyperplane. Thus, we project the domain shift A onto the vector μ22 - 14 to get the component of that perpendicular to the hyperplane, which given by:
(12 - 11) AT ( 11)
(12)
11222 141
A (1وي-ي1
Since we assume A is positively correlated to the vector #22 11, 12-- can be regarded as the magnitude of the domain shift along the direction μ22 fl. Note that the results also hold for the case where A is negatively correlated to /22 141. The whole proof can be obtained by following the very similar proof steps for the positively correlated case.
The mislabeling rate of the optimal source classifier fs on target data is:
1
1
Pr [fs(x) + y] Nim
Pr
[ns(x) Oly 1]+ 2 N ( o
Pr
[hs(x) Oly 1]
(x.))>Dr
(13)
We first calculate the first term of Eq. 13 Following the same tricks discussed above:
Pr x-N(M++A Pr xoN(
[xs(x) < 0ly 1]
LL)
(xs(x) < 0ly = 1]
||x - #1 -11
(2TO2) exp
dæ ... dea
20
(xix (ui = 10112 Iagl?
Ei 22
(2T02)$ cxp
202
ddi .rd
(x Coctii 22.. Koo,di <ma)
100
x2
2TT2 di
exp
2
dxd
= dj
(14)
where di
u2 cl| sign(
p2 ||c|]]
17

---

Published as a conference paper at ICLR 2023
Similarly, the second term is given by:
d2
Pr
(ns(x) > 0ly -1]
=
(15)
x~ x oaL)
0
where d2 || #
cll-
Taking Eq.
14 and Eq.
15
into Eq. 13 we have
d1
d2
Pr
[fs(x) + vl = 4p
(16)
(x.y)~>T
2
2
C PROOFS FOR THEOREM 3.1
Proof. Without loss of generality, choose assume μ2 = μı + old the convenient way to present our results. From the proof for Theorem B.1 we know that Xp = 11 decision boundary such that hr (xx) 0, where
A is at the
x (11 u2)
A
112
A
hx
g²
20
Let fr be the optimal Bayes classifier for the target domain, which can be obtained the same way as fs mentioned in B.1 The equation hr (xo) implies that
Pr by = 1 = xol. Pr ly 1|X = xol (x.y)~~r
(x.g)>>Dr
Note that X₂ is the affine hyperplane where hz)( O. Any data points this hyperplane will have the equal probabilities to be correctly classified. We start from this hyperplane and calculate another point X1, where Pr(x,))>>, [y 1|X x1] is at least 8 Pr(x.)>D, [y - x1). Thus, for any points that mislabeled and far away from x1 will result in Pr(x.a)>D, [y 1|X
x1] 2 1 - 8. We first aim to find such data point x Let X1 = mmo ld, where mn is the scalar measures the distance between the point x1 the hyperplane We need to find mg such that
Pr(x1 = 1) Pr(x -1) 21 - 8,
(17)
where
(x = 1)
Pr(xily- = =1)
||X| - 11
A
[x]
f12 - A
exp (
202
202
H2 Moold
#1 + moold
exp
20
20
exp (mnd)
(18)
get m (log 1-6)/d. Since the isotropic Gaussian random vectors has the rotationally symmetric property, we can transform the integration of multivariate normal distribution to standard normal distribution with different intervals of integration. Then any data points from a region that have at most ||x f4 All distance to its mean μ + A will have least 0.99 probability coming from the first component. Let the region R, be:
Taking Eq. 118) into Eq. 17).
R {x: ||x - p1
All
S ||x - #1 A
Equivalently, taking R, can be simplified:
Vd
log
R
[x:
x
S a
}}
2
dd
18

---

Published as a conference paper at ICLR 2023
The region R₁ is valid when data dimension d is large. This is realistic in practice. Since neural networks are usually dealing with high dimension data, for example > (1), the region R2 is valid.
On the other hand, aim to find region R2 where all data points mislabeled. From the proof for Theorem 1, the source classifier hg is given by
hx(x) x (11 H22
|| ||
(19)
x²
20²
Any data points are classified to the second component if lx(()) < Hence
R2 = (x:x od + 2p 1.
2
We take the intersection of R1 and Rg, all data points from this intersection are (1) having at least 1 probability coming from the first component, and (2) being classified to the second component. Formally, for (x, ~ Dr, if x E R1 n R2, then
Pr(Ss(x) + vl 211 2
(20)
We note that X E R1RRs R2 is non-empty when (log 1/0//d < a, where a magnitude of the domain shift along with the direction μ42 fr1 Since x1 is chosen from R1, verify that R, n R2 is non-empty, we only need to verify that xı also belongs to R2.
(19-H1) is the
P2 1
X1 R2 if and only if:
od 2p 1
x l
2 1. mgo 1a) T 1. ad + 2 1a
(11 + c 0
2
2
od 24 1s
(M) aold 1a moola) _ 1.
2
(a - mo)od >0,
A (12)
where c = (t22 - 11)
1112-1111
Therefore, a > mg 2 (log 2-20 )/d, R. RR2 is non-empty,
Next, we show Pr(x.))D,, [x R| increases increases.
Let event Ag be a set of x such that they are mislabeled by fs (i.c. fs(x) # y). Let event A₁ be set of such that they are from the first component but are mislabeled the second component with a probability Pr[fs(x * y)] < 1 - 5. Let event A, be set of x such that they are from the second component but are mislabeled to the first component with a probability (r[fs(x + v)]
5. Thus
Pr (x.y)~DT
[x R]
Pr (x.u)~DD
[Ao]
Pr (x.))>DT
[Ai]
Pr
[A2]
(21)
(x.y)~Dr
event A3 a of such that they are from the first component such that Pr[Js(x + v)] < 1-8 or Pr[fs(x= = y)] < 1 - 8. Let event A, be set of x such that they are from the second component but are mislabeled to the first component. For Pr[Aa].
Pr 1يا
Pr As] (x.))~
RD]
(x.g)~NV(
A
which does not change as the domain shift A varies. Meanwhile.
Pr
[A4] = +
(x.g)>~N(p 11
which is given by Eq. (5). By our assumption, the domain shift is positively correlated with the vector laz f41- So when increases, Pr A [A4] decreases.
19

---

Published as a conference paper at ICLR 2023
Since A1 S Ay and AzA1, the probability measure R. is given by:
Pr
[x E R]
Pr
[Ao]
Pr
[Ai]
Pr [A2]
(x.))>Dr
(x
(x.y)~Dr
(x.))>Dr
Pr
A
Pr [Ag]
Pr
[A4],
(22)
(x
(x.))~Dr
Dr
(x.))~Dr
where the first term is the mislabeling rate that increases second term is a constant; the third term decreases as as a increases. The equality in Eq. 22 holds
increases (given by Theorem B. : the
when - co. Therefore, when the magnitude of the domain shift increases, the lower bound of
Pr(x.y)-D x E R] increases, which forces more points to break the conventional LLN assumption.
D BACKGROUND INTRODUCTION AND PROOFS FOR LEMMA 3.2
Learning with label noise is an important task and topic in deep learning and modern artificial intelligence research. The main idea behind it is robust training, which can be further divided into fine-grained categories, such as robust architecture, robust regularization, robust loss design, and simple selection (Song al., 2022). For example, for the robust architecture- beaee methods, they propose to modify the deep model's architecture, including adding an adaptation layer or leveraging a dedicated module, to learn the label transition process and tackle the noisy label. In addition, the robust regularization approaches usually enforce the DNN to overfit less to false-labeled examples by adopting regularizer, explicitly or implicitly. For instance, Yi et al. (2022) proposed to utilize contrastive regularization term to learn noisy-label robust representation. Recently, with the widespread implementation of technologies, the topics of trustworthiness and fairness have drawn lot of interest (Liu et al. 2019; Shui et al. [2022a b). How to provide trustworthy and fair learning in LLN problems is a significant research direction. this paper, we will, however, develop our discussion based the robust loss methods in LLN.
In this section, we will first introduce the concepts and technical details of some noise-robust loss based LLN methods, including GCE (Zhang & Sabuncu 2018). SL (Wang al.. 2019b). NCE (Ma) et al. 2020), and GJS (Englesson & Azizpour 2021). Then, we will present the proof details of Lemma 3.2
D.1
LOSs FUNCTIONS IN LLN METHODS
Among the numerous studies of LLN methods, loss correction is a major branch of research. The main idea of loss correction is to modify the loss function and make it robust noisy labels.
As indicated in Ma et al. 2020), the loss function is defined to be noise robust if ΣK k) C, where C is a positive constant and K is the overall class number of label space. For example, the most widely utilized Cross- -ooss-y (CE) loss is unbounded and therefore not robust to the label noise. Some LLN studies show that existing loss functions such as mean absolute error (MAE) (Ghosh et al., (017), reverse cross entropy (RCE) (Wang et al.) (009b), normalized cross entropy (NCE) Ma et al. 20000, and normalized focal loss (NFL) are noise-robust and that combining them with CE can help mitigate the sensitivity of the model to noisy labels.
More specifically, for given data (x,y) and classifier h(x). GCE (Zhang & Sabuncu, 2018 leverages the negative Box-Cox transformation as loss function, which can exploit the benefits of both the noise-robus provided by MAE and the implicit weighting scheme of CE:
LGCE (eCrE(((e) (h(x), ex) (1 (x)")
q
where q (0, 1] is a hyperparameter be decided.
Another noise-robust loss based method SL (Wang et al. 2019b) proposes combining the reverse cross entropy (RCE) loss, which is noise tolerant, with CE loss and obtain the LSI.
1ST. = alCE + BRRCE
K
K
+ -a [k))))oggp({)))
Σ p(k(x)loogg(kx))
20

---

Published as a conference paper at ICLR 2023
where p(k|x) is the predicted distribution over labels by classifier h(x) and (k)x)) is the ground truth class distribution conditioned on sample x.
GJS (Zhan & Sabuncu 2018) utilizes the multi-distribution generalization of Divergence as loss function, which has been proven noisc-robust and fact generalization of CE and MAE. Concretely, the generalized JS divergence and GJS loss are defined as:
DEJS. Σ n PKL (1))
Σ n myD(i)
(GJS(X, y, DGJS, (y, h(xx())x)))))))))))))())) .... h(x(N))
Z
where π, p(x) are categorical distributions over K classes, x ~ A(x), a random perturbation of sample x, and Z = -(1 T (1))og(11 T1)
Further, Ma et al. (020)) shows simple loss normalization scheme which can be applied for any loss C:
{(h(x),y)
NORM
Σ ((h(x),
The study found that the normalized loss can indeed satisfy the robustness condition. However, it will also cause an underfitting problem some situations.
Note that generalized cross entropy (GCE (Zhang & Sabuncu 2018 extends MAE and symmetric loss (SL (Wang et al. 2019b)) extends RCE. So we study GCE and SL in our experiments instead studying MAE and RCE. Besides, GIS (Englesson & Azizpour 2021) is shown to be tightly bounded around Σ K {(h(x), k). All these methods have shown to be noise tolerant under either bounded random label noise or bounded class-conditional label noise with additional assumption that R(h*)
We show that under the same assumption with unbounded label noise datasets, these methods are
not noise tolerant in section D.2
D.2 PROOFS FOR LEMMA
Proof. Let Myg (x) be the Pr[ý = kYY y, X x] probability of observing a noisy label given the ground-truth label y and sample Let (x) "kzy Tyyk (x). The risk of h under noisy data is given by
R(h) =Ex (LLLN (( 5)]
= Ex Eylx Eg (LLx)/))))
= Ex.,
- Ty (x) (x)) +
Σ Tox (x)
K (1 Th (x))
= -Ex.y
(Lu((((),(k) Σ (LI()(()) + Σ
nun (LL
k
-Ixx,
(x)(
Σ ((L((h(x) + Σ (x) k:)
kØy
Exx.
(1 (1-17 (x,,(())))
Ex.yy
- (
(23)
Since Eq. (23 holds for both ht and h², we have
R(h²) Ex.y (1- (x()))
Ex.y
Σ (1 (1-7)(x)) - Tox (LNN)))))
(24)
and
R((h) Ex.
(1-ng(x))(
Ex.y
Σ (1 (1-17 (x(x)x Myk
7)x((x)) (x)) (LIN (xx),
(25)
21

---

Published as a conference paper at ICLR 2023
As h" is the minimizer of R(h), R(h*) = R(h"). Then combine Eq. (4 and Eq. 25
have
Ex.y
Σ (1 (1-7y(x) - (x)((x)) (fLLN(h" L (xx), (xx,, - 0. kty
(26)
We note that (x),, 2 (h implies (x( = 0 and (x) = for l * y. where P& (x) is the probability output by h- for predicting the sample x to be the class k. This argument proved given by Wang et af. (001b)); Ghosh et al. (2017); Yang et al. (202Tb): Ma et al. (2020)
(Theorem 1&2 in Ghosh et al. (017). Theerem 1 in Wang et al. (011b)) Lemma 1&2 in Ma et al. (2020) and Theorem 1&2 Englesson & Azizpour 2021)).
To let ( ( (x). k) 2 ( ( k) holds for all inputs x, previous studies assume the bounded label noise, which is given by
1 my (x) - (x) > Vx s.t. P(X = x) >0.
(27)
For random label noise which assumes that the mislabeling probability from the ground-truth label to any other label is the same for all inputs, i.e. 7)j(x) = ao vi + j. where do is a constant. Let
? (K 1) then Eq. [2 is degraded to
7
1
> 0
K
K
1>
K - 1
7
7 < 1
K
This bounded assumption is commonly assumed by Wang et al.
(019b): Ghosh et al. (201);; Yang
al. (202Tb) Ma et al. (2020) (Theorem in Ghosh et al. (2017) Theorem in Wang et (2019b),
Lemma I in Ma et al. 2020) and Theorem 1 in Englesson & Azizpour (2021).
For class-conditional label noise, which assumes the yá (xi = (x2) for any inputs x and X2.
Let Mija (x) = Mjj. Then the bounded assumption Eq. (27) is degraded to
nyk < 17y-
This bounded assumption is also commonly assumed, and can be found in Theorem 2 in Ghosh et al. (2017). Theorem in Wang et al. (2019)), 2 in Ma et al. (2020) and Theorem in Englesson & Azizpour (202T.
However, in SFDA, we proved that the following event B holds with probability at least - 6: my (x) Myk (x) Vx E R.
(28)
Indeed, we first denote B, (0 + y/x E R by the event that R mislabeled. Then
Pr Pr B B Pr
Pr[B[] |
Pr B, Pr[B.]
8
Given the result in Eq. 288 and combined it with the Eq. 26) we have
( (Lx)()((x),)) (x), < luuN(h" (x), When the event B holds, the condition GIIN(h
(x), (x),
(x),)=(xx),k holds.
Note that only LI (x), 2 LLL (h=((, means (x( 0 for + y and Py 1 for * y. It means that the optimal classifier h² from noisy data can make correct predictions on any inputs, which is consistent with the optimal classifier h* obtained from clean data.
As for the condition (uLN(h* (x), k) < ELL (h+ (x), k), we can get (x) for a k + y, which means that the optimal classifier h² from noisy data cannot make correct predictions samples
22

---

Published as a conference paper at ICLR 2023
R. To verify this, the robust loss function RCE ÉRCE example, and it can be easily generalized to other robust los functions mentioned above. Based on the definition of the RCE loss Wann et al. 2019b), we have
RRCE (h" (x); k) CRCEE Pk (x))
RCE (h' (x), k) =CRCE.
where CRCE > a constant. The above equations show that any S px(x) S 1 can make the condition (xxx(h((x)) (x), S (xxx(h k) hold. Meanwhile, h+ is the global minimizer of the risk over the noisy data, which makes ht* memorize the noisy dataset.
Therefore, h* makes incorrect predictions for x R such that p& (x) = 1 for a k * y, and h* is the global optimal over clean data, which gives correct predictions for E such that ps((x) 1 for y. That completes the proof h* makes different predictions on x R. compared to h²
E PROOFS FOR THEOREM
The proof for Theorem 4.1 partially adopted from Liu et al. (2020). Note that dealing with unbounded label noise, whereas the bounded label noise is considered in Liu et al. (2020). As indicated in Liu et al. 2020)). T' is set as the smallest positive integer such that 0 0.I, and == (2(11)) with high probability. Parameters is initialized by Kaiming initialization (He et al. 2015) that fg NO, It), and 16, μu| converges in probability to O. For simplicity, we assume 80 without loss of generality. The proof consists of two parts. The first part is to show that 0r-1 highly positively correlated with the ground truth classifier. The second part to show that prediction accuracy mislabeled samples can be represented as the correlation between the learned classifier and the ground truth classifier.
Proof. To begin with, we show the first part. Let samples = yi(u oz.), where z (a), The gradient of the logistic loss function with respect to the parameter is given by:
VoL(O)) =
x (ta x;) 71)
2n
1=1
TiX; +
x, tA, x;)
(29)
2n
2n
Then we will show that -1² T VoL(O,) is lower bounded by positive number. We first show the bound on Din Eq. (29). Since is sampled from standard normal distribution, Σ Dif x; has limited variance. By the law of large number, Σ. 8ipe" x; converges in probability to its mean. Therefore,
jx 11] =E(pp" x1 {yx # S rr] Elij x1 (yx p p2
[E x1l {yx م (}]b))
+ E[E x1 (yx 4> (>]l]
=Ei- p x1 1 r)/y
E/ x1 #> rHly 1]
23

---

Published as a conference paper at ICLR 2023
Note that x = is a Gaussian random vector with independent entries, we have x T f w + 1, where (0, o²). Therefore, the above expectation is equivalent to
jj 14]
(w + 1)
(w 1) dlP..
100
+oc
w dlp, m
+
dF m
dF
+
dF
r-1
dIF
w dP +
dPP
20
rr-1
Erf
r
2 0
exp (
(r
1)2
(30)
V20
22T
2x²
where Erf[æ] mislabeled. Thus
2 v So + dt.
Note that
which means that most half of samples are
Elu
T x =
Erf
r
+
exp ( - (r - 1)2
20 > 0.
2
2
22
Now we deal with the @in Eq.
29..
1
1
Lanh(0, r;))]
1 pl
2n
2n
1=1
$
||ل||
|Pll
(31)
2n
(u x1, ,....., TTx xn)
R", and p (tanh 0, 21), tanh(0, x2), ..., tanh(0, xn))
I
By triangle inequality of the norm,
|qll = ||q + < | 1 + |||||| = Vn || 1||,
where q random vector with Gaussian coordinates. By Lemma E.1
||q / s 20
(32)
with probability 5 when n 2 ai log 1/6, where C1 a constant.
On the other hand,
||p - tanh(O, p)1,, + tanh(0, p)1,,1| - tann((,) (a,nh(0,)))) 1|1n,|| + ||p - tanh(0, 1)1m
< ||tnnh(0 ||]tanh(0, 11) + |10:1| ||q 1 1|1
=tanh(0, p) 20 n ||,11,
(33)
where the second inequality is by Lemma 9 from Liu et al. (2020), the last inequality by Lemma E.1
Then we take Eq. 31 and Eq. 33 together, and then take them and Eq.30) 30 into -p VoL(A.).
which gives
exp (- (r (r-1)² 1)2
VoL(O)) P Erfl V20
+
(tanh μ4) + 20 [10.l))
(34)
V 2m
202
By Lemma 8 from Liu et al. be rewritten as:
2020
we have supnErd
VoL(O)) < + 2a. Therefore, Eq. (34 can
-V [o,((1)]
Erfl 1-
/20
+ x exp
/2= + +2
(r=1)²²
20
a (tanh(0) (4) +2r ((0(1l)
1 1 2a
b0
σ (tanh (0, ()) + 20 110.11)
(35)
1 20
1 + 2a
24

---

Published as a conference paper at ICLR 2023
where we let by }/rl( V20
+
V2T
exp
20
Then prove -V.L(0.) [o.10,)1
TO 1+27 by mathematical induction, which can help get rid of the dependence on 0, for the lower bound in Eq. (35)
by
For = 0, the inequality holds trivially. By the gradient descent algorithm, oc+1
E == VeL(O,), where -u voL(O.)/ (o) z 10 1426
bo
0,
Eii H VoC(O,)
|10++11
17 EL (oC(
bo (Ei (oL()))
10 1+20
Ei Vo((0.)
1 bo
10 1 + 20
Ast + < T, we have 10e+1ll= S 10
101+20 2+ (+ S 1+ Taking it into Eq. (35 we have
bo
VoL(O,) VoC(())
bp
o +
1+20
2
b0
+ 20
1 +
To show -VoC(0,) is lower bounded by [V
bo 1+27.
we need to have
10
9 bo
1 20
h(o) 10 1 20
00.1
0
bp
is straightforward to verify that h(o = 0) > 0 and can be verified that when 0 < a < co. we
have h'(o) 0. Therefore, for 0 < < co and any t < T -
voC(O,)
1 bu
[
10 1014 11+20 20
Hence by gradient descent algorithm 0r = -1 T-1 VoL(O,) and the same proof above, we have
0T |
60
(36)
10
For the second part: the prediction accuracy mislabeled sample set B converges in probability to its mean. Therefore, the expectation of the prediction accuracy on mislabeled samples is given by
E| (sign((()) = yRl =E[1 (sign(y0) (μ - oz)) = v}1
=E (sign 0J (μ ([o))) = 1}]
Prip00, 0T]
(37)
Note that z is a standard Gaussian vector, 0 is distributed as (00 [|0r| Thus, Eq. (37 is
cquivalent to (-_ o (1))))) 1
By the inequality 4(x) I exxpl=²2/1 /2} for 0, then we have
e 1 <
T #
bp
2 1 e
2 1
-epp 200 (1 + 20 )
|
2
We denote g(o) by:
Erf 1-r
exp (r-1) 1)
g(0) 2(1 + 20
20²
2T (1
20
where g( > 0 for any 0. Note that g(a) + when 0, and g(ar) is monotone decreasing
as a increases since g' (g)) < for a > 0.
25

---

Published as a conference paper at ICLR 2023
Lemma E.1. Let X (2, X...... ....... X R be random vector with independent. Gaussian
coordinates X₁ with E[X,] = 0 and E[X;]
= co. Then
Pri 1 vnl Vnl - 2 exp ( an))
where a >> 0 is a constant.
Proof. The Gaussian concentration result is taken from Proposition 5.34 in Vershynin (2018). which will be used here for proving Theorem 4.1
F ADDITIONAL LEARNING CURVES
We provide additional learning curves on DomainNet dataset, shown in Figure T0 The dataset contains 12 pairs of tasks showing: (1) target classifiers have higher prediction accuracy during the early-training time; (2) leverage ETP by using ELR can alleviate the memorization of unbounded noisy labels generated by source models.
ELR
EUR
ELA
Acurnryy
ELR
ELR
R
ELR
EUR
30
130
100
100
10
300
100
30
Accumey
12
CE
120
120
300
Figure 10: The source models are used to initialize the classifiers and annotate unlabeled target data. As the classifiers memorize the unbounded label noise very fast, we evaluate the prediction accuracy on target data every batch for the first 90 steps. After the 90 steps, we evaluate the prediction accuracy for every 0.3 epoch. We use the CE and ELR to train the classifiers on the labeled target data, shown in solid green lines and solid blue lines, respectively.
G EXPERIMENTAL DETAILS
In this section, we additionally show the overall training process of our method, illustrated in Figure II and in Algorithm [ Besides, provide more experimental information of our paper in details.
Datasets. We use four benchmark datasets, which have been widely utilized the Unsupervised
Domain Adaptation (UDA) (Long al. 2015 Tan et al. 2020 Wang et al.) 2022) and Source-Free
26

---

Published as a conference paper at ICLR 2023
Ft
Stopgrad
f(æ; 0+);0t)
LELR
Data
Lseda
Target Classifier
Figure 11: Overview of the SFDA problem and our method.
Algorithm 1: SFDA ELR Source Free Domain Adaptation with ELR
Input: Source Pre-Trained Model: f(r; 0o). Target Data: (z,),,, Training Epochs: T Initialize prediction bank y with To =
for epoch=/ to T do
for iterations 1,,,,,,... do
Compute the SFDA objective LSFDA (depends on concrete SFDA algorithms)
Update the prediction bank V: He 38h-1 + (1 - (x, Compute the ELR regularization LHR: LEIR log(1 5. (xx))
Compute the total loss: L LSEDA + XLEIR Update the parameters of f(0)) via C
end end
Output: Target Adapted Model f(x;0
27

---

Published as a conference paper at ICLR 2023
Table 4: Accuracies (%) on Office-31 for ResNet50-based methods
Method
SF
A-D
A-W W
D- W
W-D
D-*A
W-+
Avg
MCD
Saito et al
2018b
X
92.2
88.6
98.5
100.0
69.5
69.7
86.5
CDAN
Long et al
2018
X
92.9
94.1
98.6
100.0
71.0
69.3
87.7
MDD
Zhang el
20196
X
90.4
90.4
98.7
99.9
75.0
73.7
88.0
BNM
Ui al. 2020
90.3
91.5
98.5
100.0
70.9
71.6
87.1
DMRL
Wu et ul. 2020
93.4
90.8
99.0
100.0
73.0
71.2
87.9
BDG
Yang et al. 2020
93.6
93.6
99.0
100.0
73.2
72.0
88.5
MCC
ji et al.
2020
X
95.6
95.4
98.6
100.0
72.6
73.9
89,4
SRDC
Lang el at. 2020
95.8
95.7
99.2
100.0
76.7
77.1
90.8
RWOT [uu a.. 2020
94.5
95.1
99.5
100.0
77.5
77.9
90.8
RSDA-MSS MSTN (Gu al.
2020
95.8
96,1
99.3
100.0
77.4
78.9
91.1
Source Only
80.8
76.9
95.3
98.7
60.3
63.6
79.3
+ELR
90.9
89.0
98.2
100.0
67.1
64.1
84.9
SHOT (Liang et al. 2020
94.0
90.1
98.4
99.9
74.7
74.3
88.6
+ELR
94.9
91.6
98.7
100.0
75.2
74.5
89.3
G-SFDA Yang et al. 2021b
85.9
87.3
98.6
99.8
71.4
72.1
85.8
+ELR
86.9
87.8
98.7
99.8
71.4
72.9
86.2
NRC (Yang et al. 2021a
93.7
93.8
97.8
100.0
75.5
75.6
89
+ELK
93.8
93.3
98.0
100.0
76.2
76.9
89.6
Table 5: Optimal Hypermaraters (3/X) on various datasets.
Hyperparameters: BIX
Office-31
Office- Home
VisDA
DomainNet
ELR only
0.9/-
0.99/
0.99/-
0.9/-
ELR SHOT
0.7/1.0
0.6/3.0
0.6/25
0.7/7.0
ELR G-SFDA
0.8/1.0
0.9/1.0
0.5/7.0
0.8/12.0
ELR + NRC
0.5/1.0
0.6/3.0
0.5/3.0
0.8/3.0
Domain Adaptation (SFDA) (Liang al. 2020) scenarios, verify the effectiveness of leveraging
the early-time training phenomenon to address unbounded label noise. Office-31 (Saenko et al. 2010) contains 4, 652 images in three domains (Amazon, DSLR, and Webcam), and each domain consists of 31 classes. Office-Home (Venkateswara al. 2017) contains 15, 550 images in four domains (Real, Clipart, Art, and Product), and each domain consists of 65 classes. VisDA (Peng et al.) 2017) contains 152K synthetic images and 55K real object images with 12 classes. DomainNet (Peng al. 2019) contains around 600K images in six different domains (Clipart, Infograph, Painting, Quickdraw, Real and Sketch). Following previous work Tan et al.) (2020); Liu et al. (2021a), we select 40 the most commonly-seen classes from four domains: Real, Clipart, Painting, and Sketch.
Implementation. We use ResNet-50 (He al. 2016) for Office-31, Office-Home and DomainNet, and ResNet-101 (He et al.] 2016) for VisDA as backbones. We adopt fully connected (FC) layer the feature extractor batch normalization layer is put between the two FC layers and the weight normalization layer is implemented on the last FC layer. We the learning rate 1e-4 for all layers except for the last two FC layers, where we apply 1e-3 for the learning rate for all datasets. The training for source models are set be consistent with the SHOT (Liang et al. 2020). The hyperparameters for ELR with self-training, ELR with SHOT, ELR with G-SFDA, and ELR with NRC on four different datasets are shown in Table 5 We note that for ELR with self-training. there is only one hyperparameter 3 to tune. The hyperparameters for existing SFDA algorithms values for Office-31, Office- Home, and VisDA datasets. As these SFDA algorithms have not reported their performance for DomainNet dataset. We follow the hyperparameter search strategy from their work (Liang et al. 2020 Yang et al. 202 b), and choose the optimal hyperparameters 0.3 for SHOT, K 5 and M 5 for NRC. and
the backbone and another FC layer as the classifier head. The
set to be consistent with their reported
for G-SFDA
28

---

Published as a conference paper at ICLR 2023
H MEMORIZATION SPEED BETWEEN LABEL NOISE IN SFDA AND IN CONVENTIONAL LLN SETTINGS
Although ETP exists in both SFDA and conventional LLN scenarios, the memorization speed for them still different. Specifically, the target classifiers memorize noisy labels much faster in the SFDA scenario. It has already been shown that it takes many epochs before classifiers start memorizing noisy labels in conventional LLN scenario (Liu et al. 2020; Xia et al.) 2020a). We highlight that the main factor causing the difference is the label noise. To show it, we replace the unbounded label noise in SFDA with bounded random label noise, and we keep the other settings unchanged as introduced in 4 To replace the unbounded label noise with bounded random label noise, we use the source model to identify mislabeled target samples, then we assign random labels to these mislabeled samples. Figure 13 and Figure show the learning curves on Office- Home and Office-31 datasets with unbounded label noise and random bounded label noise. To better visualize the learning curves with unbounded label noise, we re-plot Figures 13 I4 with different y scale in Figures 13 16 These figures demonstrate that target classifiers memorizing noisy labels with unbounded label noise is much faster than noisy labels with random bounded label noise. The classifiers with bounded label noise (colored in red) expected to memorize all noisy labels eventually. As illustrated in Figures 15 16 the classifiers with unbounded label noise (colored in green) show that the noisy labels are already memorized. We note that for the first 90 steps, the prediction accuracy is evaluated every batch, while the prediction accuracy is evaluated every 0.3 epoch after that time. Therefore, for unbounded label noise, target classifiers start memorizing the noisy labels within the first epoch (consisting of more than 90 batches).
There are some existing LLN methods such PCL (Zhang al. 202T) to purify noisy labels every epoch based on ETP. Due to this difference, these LLN methods are not helpful to solving label noise SFDA as they are not able to capture the benefits of ETP. Our empirical results Section 5.. can support this argument. We also note that PCL does not suffer from the fast memorization speed and able to capture the benefits of ETP in conventional LLN settings. As we indicated Figures] [3 it takes much longer time (more than few epochs) for target classifiers to start memorizing bounded noisy labels. We hope these insights can motivate the researcher to consider memorization speed and design algorithms better for SFDA.
1 ADDITIONAL ANALYSIS OF ELR AND STANDARD SFDA METHOD NRC
In this section, we will theoretically and empirically compare ELR and NRC in detail. Specifically, NRC (Yang al. [021a] well-known SFDA method that explores the neighbors of target data by graph-based methods and utilizes these neighbors' information to correct the target data's s pseudo-label, in order to boost the SFDA performance. The proposed NRC loss has the following form:
NRC = Ldiv + LN + LE + Leelf
with:
Lave
Σ LL
the diversity loss where pa is the empirical label distribution and is uniform distribution; and
LN - 1 ----I T4 mEN},
A hx(:)
the neighbors loss, where m is the index of the m-th nearest neighbors of x;, S,, is the m-th item in memory bank S, Aim is the affinity value of m-thh nearest neighbors of input x; in the feature space.
T I((x)) LE = --=-1
MENT JEE
the expanded neighbors loss, where Ex contain the N-nearest neighbors of neighbor in NM.
Lsef = 1 Σ s(,))
29

---

Published as a conference paper at ICLR 2023
the self-regu regulaaiaaiiinn loss, where S; means the stored prediction in the memory bank, constant vector and is identical to the hxx,) as in NRC they update the memory banks before the training.
1.1 THEORETICAL ANALYSIS OF NRC's SELF-REGULARIZATION TERM COMPARED TO ELR
To emphasize the novelty of our proposed ELR in SFDA problems, will compare the original formulas and also the gradients of ELR and NRC's seff-regularization (SR) term in detail. And then, we will explain why NRC can benefit from the ETP only by adopting the SR term. As we formulate in the main paper, we can represent the ELR loss and the SR loss as follows:
(o, = logg1 - 0. f(x
(38)
and
CaR (B.) - f(x:
(39)
where Dt- - 0,) in ELR is the moving average prediction for x, and f(; 0z) in SR is the constant vector copied from the current training step's prediction. Besides, the gradients of ELR and SR are:
(O.) df(x;
D1
(40)
1 Ur f(x; 0₂)
and
dLsn ytc
(41)
df( 0.)
The motivation of the SR term is to emphasize the ego feature of current prediction and, therefore, to reduce the potential impact of noisy neighbors, whereas the ELR proposed in this paper considers the changes of prediction quality during the training process and aims encourage the model prediction to stick to the early-time predictions for each data point.
directly observe that ELR involves the previous training step's 's prediction information in loss (included in T.)). however, SR leverages only the prediction result of current step.
As shown in Eq. (38) and Eq.
339
Besides, if we further look at the gradient formulas of these two losses and analyze the back- df,ss(0))
find that the gradient af(x:0,) increases as the model prediction closes to the target St. which will further force the prediction f fxx 0.) close to ý thanks to the large magnitude of the gradient. And this will help with the utilization of early-time predictions and ETP. However, the gradient of LSR is a constant vector with values of prediction logits, which could be very small. So when dCan(0.) is small, SR term can be easily overwhelmed by the other loss terms that favour ]f(x;,)
propagation process,
fitting incorrect pseudo labels, leading to poor performance.
The above analysis shows a fundamentally different difference between SR and ELR. Specifically, SR does not utilize ETP and cannot handle the unbounded label noise either.
1.2 EMPIRICAL ANALYSIS OF ELR AND NRC IN TERMS OF THE UTILIZATION OF ETP
In addition to the above theoretical analysis for the loss functions, we also observed the same conclusion through experiments. As shown in Figure [2] we observe that thanks to the update of the pseudo-label with the process of adaptation in the SFDA method, overall, NRC can obtain a model with relatively high accuracy on the target domain. However, the performance drop still exists when using the NRC method alone, which can be effectively avoided by adding the ELR term. This confirms that ELR can effectively leverage ETP and avoid the problem of noisy label memorization.
J ADDITIONAL DISCUSSION OF PSEUDO-LABEL PURIFICATIONS IN SFDA AND LLN APPROACHES
In this section, we will further discuss the similarities and the differences between the LLN approaches and the pseudo-label purification processes proposed in current SFDA methods.
The main similarity between the existing SFDA approaches and the LLN methods is that both research fields have to deal with data with noise, aiming to get model with promising performance. As
30

---

Published as a conference paper at ICLR 2023
0.6
0.70
Tranivo Accsooy
034
Paining
570
NRC
078
NRC
NRC
NACHELL
MRCHELR
344
129
022
0575
Tamiie Koursss
4.500
Thaneo Acunnyy
0432
0135
6.000
0022
4375
0177
NRC
NRC
NRC
300
300
129
0.15
0319
0775
Thiring Focoreo
0703
8.25
0123
Thites
0200
NRC
0435
NRC
NRC
0353
MRC-FI
344
100
120
332
Ncurayy
0.0
NRC
0.00
NRC
NRC
3/4
100
L20
Figure 12: Srne-Grained Training Accuracy of NRC and NRC ELR Office-Home Home dataset. The solid green lines represent the training process of NRC, whereas the solid blue lines represent the training process of NRC with ELR term. The colored bands represent the performance drop.
for the differences, they can be mainly divided into the following aspects. From the perspective of motivations, most of the existing SFDA approaches are developed under the domain adaptation setting. They study how to best exploit the distribution relationship between the source and target domains the absence of source data, so as to achieve domain adaptation better. Their motivation to investigate how to better assign the pscudo-label. In contrast, LLN is an independent field that mainly studies, given a set of noisy data, how to deal with the label noise, conduct the model training, and obtain noise-robust model with better performance. Traditionally, the study of LLN does not involve assumptions about the data domain or source model. Meanwhile, there are more in-depth and rigorous studies (theoretical and methodological) on the types of noises, and how handle and exploit them. From the perspective of the methodology, in order to obtain a higher quality pscudo-label, many SFDA methods heuristically use clustering or neighbor features correct the pseudo-labels. and use the corrected labels to perform normal supervised learning. The current SFDA methods focus on the explicit pseudo-label purification process, which can be summarized as noisy label correction. However, for LLN, the noisy label correction is just research sub-branch. LLN also includes many other research directions, such as studies of different label noise types, research about how to utilize and even benefit from label noise in the training process, and how to train the model more robustly. Many noise-robust loss functions and related theoretical analyses have been developed.
We would like to emphasize that the motivation our paper is to investigate how to study SFDA from the perspective of learning with label noise. We combine the characteristics of SFDA with the LLN approaches and discover the unbounded nature of label noise in SFDA. Further, we rigorously distinguish which LLN methods use in SFDA. We believe that the studies of LLN can open new avenues for the research of SFDA and bring more ideas and inspiration the design of the SFDA algorithm.
help SFDA problems and which approaches limited in their
31

---

Published as a conference paper at ICLR 2023
166
un unbounded bounded
unbo undec
unbounded
Thertiy
bounded
bounded
150
150
125
160
Tarern 12
Tax
unhounded
Towwi Accumary
unbounded
unbo undec
50
150
70- A7
PV- RW
unbounded
Tanning RECLENoy
25
bounded
8
bouuudded bounded
unboundec
bounded
A7-110 1M
100
hounded bounded
unbounded bounded
bounded
80
Figure 13: Training accuracy on Office-Home dataset. The solid green lines represent the unbounded label noise in SFDA, whereas the solid red lines represent the bounded label noise.
unbounded bounded
unbounded
Training Accuracy
Accuracy
bounded
75
20
60
120
140
20
60
30
100
120
R
100
100
unbounded
unbounded bounded
Training Accuracy
Training Accuracy
50
bounded
09
100
140
100
unbounded
100
Training Accuracy
bounded
Training Accuracy
99.8
99
unbounded bounded
99.4
70
20
60
100
120
140
60
80
100
120
P.
Figure 14: Training accuracy on Office-31 dataset. The solid green lines represent the unbounded label noise in SFDA, whereas the solid red lines represent the bounded label noise.
32

---

Published as a conference paper at ICLR 2023
bouncied bounded
boundeed
houraat
08.23
65.75
12.
unbounded bounded
180
195
150
154
12
155
r6a
196
10.0
Ru
21
unbounded bounded
bounded
Ttrm Accuness
8
bounded
unbourdee bounded
Pi
0.55
Rr
unbouncied bounded
unbounded bounded
urbounded
Tainng Rocueee
bounded
2035-
2422
unboundied bounded
unbounded bounded
bouudddd bounded
Tyrnoo Acanno
1iiingg sccurscy
8
Figure 15: Figure 13
with different y-scale to better show learning details of the unbounded label
noise.
83.5
unbounded bounded
unbounded bounded
Training Accuracy
Training Accuracy
82.5
77
08.5
60
80
100
120
140
30
150
120
140
67
98.5
unbounded bounded
66
Training Accuracy
65
Training
87.0
unbounded
bounded
63
96.0
67
unbounded bounded
unbounded bounded
99.8
66
99.6
65
99
982
40
60
80
100
120
140
60
00
120
120
140
5
Figure 16: Figure 14 with different y-scale to better show learning details of the unbounded label noise.
33


Reviews:
[{"review_id": "tTB3V2Td_H", "reviewer": "Reviewer_nSg6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers source-free domain adaptation, where a learner has access to only a source model and unlabeled target data due to privacy concerns regarding source data. Existing practices in source-free domain adaptation is to use the source model to pseudo-label target data, which may cause mislabeling occasionally, and moreover, the mislabeling probability is much higher than the noise probability that is usually assumed in learning from label noises. Indeed, the authors show a theoretical statement that the mislabeling probability can be arbitrarily large and approaching 1 under a simple setting. Then, the authors reveal that ETP (early-time training phenomenon), which observes that a learner with fewer training epochs can avoid overfitting and memorizing the mislabeled data perfectly, can be observed in the unbounded noise case as well. By leveraging this fact, the authors propose to use early learning regularization (ELR), which uses the moving average of the label prediction over training epochs to mitigate the aforementioned overfitting issue, with existing source-free domain adaptation methods. As a result, the existing methods can be improved with ELR.", "summary_of_the_review": "The paper contributes to source-free domain adaptation from a new viewpoint, learning with label noises, which should be acknowledged. The overall quality of the manuscript is above the border. A few points I mentioned in \"Clarity\" section can be addressed, which I believe makes the paper much better.", "strength_and_weaknesses": "### Strengths\n\n- **Nice theory to demonstrate the issue of unbounded noises.** Although the setup is limited to a simple Gaussian case, the authors provide a nice theoretical illustration of how the labeling error of the pseudo-labeling in source-free domain adaptation can be arbitrarily large.\n- **Simple remedy to existing source-free domain adaptation.** The proposed remedy to the unbounded noise is to use the moving average of the label prediction over training epochs, which is practically very simple to use. So far, this method has been used in the context of learning from label noises, but its applicability and effectiveness in source-free domain adaptation has not been witnessed. The authors contribute in this line to show ETP can be observed in the unbounded noise case as well.\n\n### Weaknesses\n\nBasically, I do not see any weaknesses of this paper except for small unclear points. I point out them in the subsequent \"Clarity\" section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pySwfA16Azh", "reviewer": "Reviewer_CRh4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors have proved that the “label noise” in source-free domain adaptation is unbounded. Therefore, existing LLN methods that rely on their distribution assumptions are unable to address the “label noise” in source-free domain adaptation (SFDA). The authors have also demonstrated theoretically that the early-time training phenomenon can also be observed in the SFDA problem.", "summary_of_the_review": "The major contribution of this paper is the theoretical analysis. The practical novelty may not be that significant. I would like to weakly accept this paper.", "strength_and_weaknesses": "Strength:\n+ The theoretical analysis that the label noise is unbound when there is a domain shift is novel to me.\n+ The authors have demonstrated theoretically the early-time training phenomenon under certain conditions. Although the assumption can be strong, it is still interesting to see such an analysis.\n+ This paper is well-motivated. It is important to understand the difference between two different but similar problem settings. It can be helpful for the designing of new algorithms for source-free domain adaptation.\n\nWeakness:\n+ The assumption for Theorem 4.1 that at most half of the samples are mislabelled could be a little bit strong. The proposed method based on the early-time training phenomenon may not be that practical.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bv-fvo_Rv6", "reviewer": "Reviewer_3S6g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of source-free domain adaptation (SFDA), where a pre-trained source model is adapted using unlabeled target domain data without accessing any source domain data. While previous works mainly focused on cluster assumption in the feature space, the authors propose a different perspective. They formulate the SFDA problem as learning with label noise (LLN). In addition, since label noise in SFDA is unbounded unlike the noise in conventional LLN scenarios, they leverage the early-time training phenomenon (ETP) with a simple early learning regularization (ELR) term. The authors demonstrate that it consistently improves existing SFDA algorithms by a large margin on four different SFDA benchmark datasets.", "summary_of_the_review": "While the paper has its merits, unfortunately, it also has several issues which need to be addressed: (1) clarification of the novelty of the proposed method, (2) comparative explanations between the proposed method and self-regularization loss in NRC, and (3) detailed experiment protocols.", "strength_and_weaknesses": "Strengths\n- I think the idea of formulating the SFDA problem as LLN is interesting. The authors also theoretically showed why the existing LLN methods cannot address the noise in SFDA due to the fundamental differences.\n- The theoretical and empirical justification of ETP in the SFDA problem with the unbounded label noise seems to provide solid grounds for the proposed method.\n- Leveraging ETP with ELR term seems simple but effective across a wide range of SFDA algorithms and benchmark datasets.\n\nWeaknesses\n- While the previous SFDA methods do not explicitly formulate the SFDA problem as LLN, I think they are still partially based on the LLN methods. As the authors explained in the related work section, the SFDA methods leverage the target “noisy” pseudo labels and also some type of pseudo-label purification processes as in the LLN methods. It would be great if the authors can provide more clear explanations of the similarities and differences between their motivations. \n- The proposed early learning regularization (ELR) term encourages the model prediction to stick to the early-time predictions for each data point. Meanwhile, NRC contains a loss term, called self-regularization loss, which has a similar purpose of reducing the potential impact of noisy pseudo-labels and not ignoring the current prediction. Please provide comparative explanations between the two and how the ELR still improves the NRC (Table1). \n- In Empirical Observations on Real-World Dataset, it seems the authors trained classifiers using the fixed noisy labels with cross-entropy loss. However, SFDA algorithms (e.g. NRC) gradually update pseudo-labels rather than using the fixed pseudo-labels. Please provide clarifications. It would be also nice if you can also show the training accuracy of SFDA algorithms in Figure 2. \n- How did the authors get the baseline results? There seem to be some discrepancies compared to the results reported in their original papers. For example, NRC results seem lower than those reported in the original paper. The reported average results are 72.2 for Office-Home and 85.9 for VisDA-C. Then, the performance difference between NRC+ELR in Table1 (72.6) and the NRC in the original paper (72.2) seems marginal.\n- Please provide a more detailed protocol for the experiments. For example, assuming that no labeled target data is available, how do you decide (1) when to stop the training and (2) hyperparameters for each dataset? If the authors hand-picked the hyperparameters for each dataset, it might have caused over-optimistic results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FhHUkicR6-", "reviewer": "Reviewer_RMmn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Source-free domain adaptation (SFDA) methods usually use the pseudo-labels generated by source models which can be very noisy due to domain shifts. This paper studied and proposed an early learning regularization (ELR) based (Liu 2020) SFDA method to tackle the problem of learning with label noise (LLN). They show that pseudo-label noise generated by source models in SFDA is unbounded label noise. They theoretically and empirically proved that early-time training phenomenon exists in the unbounded label noise. Extensive expreiments were conducted to show that the proposed ELR based method can improve existing SFDA algorithms by around 1~2% on multiple DA benchmark datasets.      ", "summary_of_the_review": "In summary, the work theoretically and empirically studied the LLN in SFDA and the $\\textit{early-time training phenomenon}$ in unbounded LLN which hadn't been studied before. It proposed a ELR based solution for the unbounded LLN.  The work has enough novelty for the practical problem of training with unbounded label noise. The paper is basically well-written with fair experimental results. ", "strength_and_weaknesses": "Strength:\n\nThis work theoretically and empirically justifies that the early-time training phenomenon exists in the unbounded label noise scenario, and proposed an extended ELR scheme for SFDA methods. The work has enough novelty for a practical problem. The paper provides reasonable experimental results and fair discussions. And the paper basically is well-written and easy to follow. \n\nWeakness:\n\nThe paper lacks explanations and details in some places for readers who are not familiar with the topic. The paper should be more self-contained. According to the experiment results, the improvement by the proposed technique is marginal 1~2%.  \n\nIn EQ1, should $d_1$ and $d_2$ be $d_1 = \\frac{\\mu_2 - \\mu_1}{2} - C$ and $d_2 = \\frac{\\mu_2 - \\mu_1}{2} + C$  ?\nPlease add a brief explanation of $l_{LLN} $ in Lemma 3.2 in the paper. \n\n\n\n\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%