Dirichlet-based Per-Sample Weighting by Transition Matrix for Noisy Label Learning
Authors: HeeSun Bae
Seungjae Shin
Byeonghu Na
Il-chul Moon
Venue: ICLR 2024 poster (2024)
Keywords: learning with noisy label
noisy label classification
Transition matrix
Dirichlet distribution
Importance sampling
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We suggest Dirichlet based per-sample Weight Sampling which can explain both reweighting and resampling in a unified framework and propose a new transition matrix utilization method for learning with noisy label.

Abstract: For learning with noisy labels, the transition matrix, which explicitly models the relation between noisy label distribution and clean label distribution, has been utilized to achieve the statistical consistency of either the classifier or the risk. Previous researches have focused more on how to estimate this transition matrix well, rather than how to utilize it. We propose good utilization of the transition matrix is crucial and suggest a new utilization method based on resampling, coined RENT. Specifically, we first demonstrate current utilizations can have potential limitations for implementation. As an extension to Reweighting, we suggest the Dirichlet distribution-based per-sample Weight Sampling (DWS) framework, and compare reweighting and resampling under DWS framework. With the analyses from DWS, we propose RENT, a REsampling method with Noise Transition matrix. Empirically, RENT consistently outperforms existing transition matrix utilization methods, which includes reweighting, on various benchmark datasets. Our code is available at https://github.com/BaeHeeSun/RENT.

Published as a conference paper at ICLR 2024
DIRICHLET-BASED PER-SAMPLE WEIGHTING BY TRANSITION MATRIX FOR NOISY LABEL LEARNING
HeeSun Bae', Seungjae Shin', Byeonghu Na' & II-Chul Moon Department of Industrial and Systems Engineering, KAIST, 'summary.ai fcat2207, tmdwo0910, wp03052, cmoon) @kai www..t..........................
ABSTRACT
For learning with noisy labels, the transition matrix, which explicitly models the relation between noisy label distribution and clean label distribution, has been uti- lized to achieve the statistical consistency of either the classifier or the risk. Previ- ous researches have focused more on how to estimate this transition matrix well, rather than how to utilize it. We propose good utilization of the transition ma- trix is crucial and suggest a new utilization method based on resampling, coined RENT. Specifically, we first demonstrate current utilizations can have potential limitations for implementation. As an extension to Reweighting, we suggest the Dirichlet distribution-basedd per-sample Weight Sampling (DWS) framework, and compare reweighting and resampling under DWS framework. With the analy- ses from DWS, we propose RENT, REsampling method with Noise Transition matrix. Empirically, RENT consistently outperforms existing transition matrix utilization methods, which includes reweighting, on various benchmark datasets. Our code is available at https: thub. com/BBeeeeeSSnnRRR.
INTRODUCTION
The success of deep neural networks heavily depends on a large-sized dataset with accurate anno- tations (Daniely & Granot 2019 |erthhn et al. (021). However, creating such a large dataset is arduous and inevitably affected by human errors in annotations, referred to as noisy labels. It causes model performance degradation (Arpit et al. [017 Zhang et al.] 2222 la b), and studies have been suggested to solve this degradation (Zhang & Sabuncu 2018 Li et al. 2020 Wang et | 202T Wei eet al. [222bb] [ac et al.) 2022, Na al.) 2024). Among various treatments,
prominent approach
is to estimate the transition matrix from true labels to noisy labels (Patrini et al. 2017).
Transition matrix explicitly models the relation between noisy labels and the latent clean labels Yao et al. 2020) Li et al., 202T). It means that the transition matrix provides probability that a given true label is transitioned to another noisy label, where the true label is unknown in our setting. With this transition matrix, a trainer can ensure statistical consistency either to the true classifier (Patrini al. (017) to the true risk (Liu & Tao 2015) ideally. Since this information is unknown when learning with noisy label, previous transition matrix related studies have focused on the accurate estimation of transition matrix (Li et al. 2021 Cheng al. 2022).
Even if we assume that the transition matrix is accurately estimated, how we utilize the transition matrix can also impact the performance. Forward (Patrini et al. (017)) is one of the general risk structures for utilizing the transition matrix (Zhu et al. [021] Yang et al. 2022). It trains classifier by minimizing the divergence between the noisy label distribution and the classifier output weighted by the transition matrix. Other ways of transition matrix utilization include Reweighting (Liu & Tao 2015; Xia et al., 2019). Reweighting employs an importance-sampling technique, ensuring statistical consistency of the empirical risk to the true risk. However, in practice, the empirical risk of Reweighting can also deviate from the true risk because the estimation of per-sample weights relies on the imperfect classifier's output. In other words, when the classifier's output cannot accurately estimate per-sample weights, it can lead to potential mismatch between the estimated empirical risk and the true risk, compromising the effectiveness of reweighting transition matrix utilization.
Recently, An et al. (2020) suggested that resampling outperforms reweighting for correcting dataset sampling bias. Motivated by the potential benefit of resampling over reweighting, introduce

---

Published as a conference paper at ICLR 2024
Dirichlet based per-sample Weight Sampling (DWS) df -0
(x,
jr [0.7 00.7.0.2.0.11
(x, P)
Dastoret
wit from Dir (c)
witi from Dar(a (M)
Darsent
W [Mrg. wPs]
[1,0,0]
Closs
W1
Refined
Label
Wg
wrl [0.7,02,0033
[1,00]
"Re-wiight from T
[Existing method)
R(wlligg 3
(3.00
"Re-sample from T (RENT)
Figure I: Dirichlet distribution-basee per-sample Weight Sampling with shape parameter a and the mean vector μ.. Image the vertices of yellow triangles represents data instance. Blocks above the images represent true Class, noisy Label. Sides are implementation example of sampled un. wn(1) assigns weights to all data (Reweighting), while w²(2) simulates resampling refined dataset (RENT).
extended framework, Dirichlet distribution-based per-sample Weights Sampling (DWS), to encom- pass both resampling and reweighting. Figure [ shows an implementation example of our frame- work. (1) and (2) are sampled weight vectors from different Dirichlet distributions with the same mean. They represent different properties by the shape parameter, a. These weights are represented as reweighting and resampling in implementation, as each side of the figure.
We then analyze the impact of on DWS. First, affects the variance of risk and smaller means larger variance. When variance of the risk increases, it showed performance improvement accord- ing to Lin et al. (2022) empirically, so we expect better performance with small Second, the Mahalanobis distance between the true weight vector and the mean of per-sample weights sampling distribution is proportional to the square root of a, emphasizing the merit of small a. Finally, is related to label perturbation. This label perturbation aligns with Chen et al. (2020), who proposes label perturbation during training can reproduce better performance for learning with noisy label.
This analysis on the impact of a under DWS framework finally explains the differences between resampling and reweighting. provides theoretical rationale behind the superior performance of resampling over reweighting for learning with noisy label. It leads us to introduce RENT, the first REsampling method to utilize the Noise Transition matrix. RENT empirically shows better perfor- mance for 7 utilization when combined with various T estimation methods.
In summary, the contributions of this paper are as follows.
1) We suggest DWS, which samples per-sample weight vectors from the Dirichlet distribution.
2) With DWS, we can express reweighting and resampling in a unified framework and demonstrate resampling can be better than reweighting for learning with noisy label.
3) Under this situation, we suggest RENT, which resamples dataset with the transition matrix. RENT empirically shows good performance, suggesting a new transition matrix utilization method.
2
TRANSITION MATRIX FOR LEARNING WITH NOISY LABEL
2.1 PROBLEM DEFINITION: LEARNING WITH NOISY LABEL
This paper considers a classification task with classes. Let the uppercase, e.g. (X, Y), be random variable and the corresponding lowercase, e.g. (r, y). denote realized instance. We define an input as X E X' and true label Y E y, where
Rdd and y = (1.... C). respectively. Y E y represents noisy label. D = { [.) i= is the dataset with noisy labels. The model is fe(x) = a (g (z; f)), parameterized by 6, Rd - R2C is mapping function and o--) is softmax function. The objective is find the optimal classifier fū which minimizes the true risk as:
f argmin, Ri (fo), with R((f))= ((x.))~((x,)) [ (fe(z), v)]
I(..)) is a loss function that measures the prediction quality of the label. For simplicity, we denote Ri as R, omitting fo unless otherwise specified. Since D contains noisy labels, the empirical risk function using D. denoted as Rpmp : - Ei'1 (Jo (arx), ;:). does not converge to the true
2

---

Published as a conference paper at ICLR 2024
risk RR. It implies that fü cannot be accurately approximated by minimizing R, Therefore, our objective is to minimize Rú, or training fe approximate p(Y z), by learning with D. Please refer to Appendix B for more studies related to learning with noisy label task.
2.2 TRANSITION MATRIX FOR LEARNING WITH NOISY LABEL
When there are C' classes, the noisy label generation process can be explained with a matrix T e [0, _jcxe, whose entry Ti is the probability of clean label k being flipped to noisy label j. In other words, a noisy label ii is the sampling result from clean label y: with its flip probability as p(Y = J -.). Here, this matrix. T. has been referred to as the transition matrix in noisy label community (Patrini et al., 2017 Yao al. 2021 Then, the noisy label distribution, from which the noisy labelled dataset are sampled, can be expressed as Eq. 2
p( (Y with (zx(2) p(Y j|Y
=
Here, p(-I-) is vector and p(-1-) is scalar. Either p({(2) or p({[x) can be calculated from the other if T is given With this property. previous studies utilizing the transition matrix have been able to explain the statistical consistency of their classifier (Li et al. [021] Cheng et al.) 2022) or risk (Liu
& Tao 2015 Patrini al.
2017 Liu et al.
2023) their true counterpart. The problem is that true
T is unknown, and previous studies have focused on estimating the good transition matrix (Patrini
al. 2017 Xia et al., 2020 Zhang et al.
2021b Zhu et al. 2021 2022).
While we acknowledge the importance of estimating T', this paper emphasizes the utilization phase of T in its research scope. Modifying fz.Tmp is essential for training Sa to minimize Eq. when only T and D are available. We explicitly refer to this T utilization, which means, we divide the transition matrix based fe training into two phases: 1) T estimation and 2) T utilization. Empirically, T utilization impacts the model performance significantly, underscoring the importance of utilization phase. The following section analyzes the previous researches on T utilization, and we demonstrate that current practices on T utilization inherits significant drawbacks in actual deployments.
2.3 UTILIZING TRANSITION MATRIX FOR LEARNING WITH NOISY LABEL
In this section, assume that true T is accessible for the sake of analyzing T utilization. Until now, three directions have been proposed for T utilization (Patrini et al. 2017 Liu & Tao 2015). Each methodology claims that it guarantees specific type of statistical consistency, either to the classifier or the risk. However, there are cases when this ideal consistency is empirically hard to be achieved, and this section analyzes such practical situations without ideal consistency. We consider as Cross Entropy loss, which is generally used for classification. R... denotes the empirical risk of Ri..
1) Forward (Patrini al. 2017) risk minimizes the gap between Y and (x( RíF B (Tfo(X), 8)
The learned So is statistically consistent to the optimal classifier fā- However, fo trained with REF can be different from fÿ. According to Zhang et al. (0211), the gap between [8|z)) and the noisy label probability distribution approximated from D can be high. We specify the impact of this gap. <(> 0). to the classifier as fğ (z) fo(x) T- * 0. It means that (x|[ is not estimated accurately, the deviation of fo from fā is inevitable for classifier consistency. Please check Appendix also for more discussions regarding this issue.
C
2) Backward Patrini et al. 2017) risk is RI = E BxxxY) [(X)T=1] with ((X)
=
[1 (fo(X), 1) I.., (fo(X), C)). As p(Y]X) T-1 T-"p(Ý|x), Rt, is statistically consistent to Ri-
However, optimizing R can lead to unstable performances reported in Patrini et al. (0117).
3) Reweighting (Liu & Tao, 2015 Xia et al. 2019) risk computes per-sample weights based on the 1953). for the noisy label classification as:
likelihood ratio (Kahn & Marshall
RLRw E ()(x,))
-(Y=Y[X) (Tp(F1X))
(x)(X), } Here, (-)c means the -h cell value of the vector.
R would be expressed E (T
fo(2,)s ((fo(T)).
We define transition matrix as Fq. 2 for mathematical correctness. Appendix B.2 for more explanations. Sinne it is natural assuming the dependency of T on the input, we consider the transition matrix of x as T(x). In this paper, we omit æ from T(x), denoting T for convenience.
3

---

Published as a conference paper at ICLR 2024
By applying importance sampling (Kahn & Marshall 1953:
Katharopoulos & Fleuret 2018) to
T utilization, Reweighting does not suffer from unstable optimization as Backward. However, the problem is that pX is required as a component for per-sample weight, where estimating ((Y serves as the final objective. While previous studies (Liu & Tao 2015 |erthhn et al. 2021) have estimated p()]x) from the on-training classifier's ss output, this estimation can be inaccurate.
DWS: DIRICHLET-BASE
PER-SAMPLE WEIGHT SAMPLING
In this section, suggest a new framework, DWS, which incorporates per-sample Weight Sampling based on the Dirichlet distribution. Through this incorporation, we interpret sample reweighting and resampling by a single framework, facilitating the direct comparison of both methods. Then, analyze the impact of the shape parameter in the Dirichlet distribution, from which the per-sample weights are generated, to empirical risk. With these analyses, we propose resampling can be better choice than reweighting based on two characteristics: the closeness between the mean of the per- sample weight distribution and the true weight; and the label perturbation nature resampling when it is explained by DWS. Finally, we introduce our method, RENT, which becomes resampling- based approach for learning with noisy label.
3.1 DIRICHLET-BASED WEIGHT SAMPLING
Let Dir(aqu) be the Dirichlet distribution with parameters a E IR and l E RR, or is a scalar concentration parameter and ş is a base mea- sure with Σ H4s is equal to 1. a, Pi..., UN positive values by the definition. Figure 2 illustrates properties of the Dirichlet distribu- tion. As depicted in the figure, instances drawn from the distribution with small a tend to clus- ter around the vertices of simplex (the left- most). The sampling frequencies for each di- mension converge to the mean value of that di- mension. In contrast, as a increases, a greater proportion of samples from the Dirichlet distribution will exhibit vectors that are in proximity to the mean vector (the rightmost).
2=1.1
a=10 a=170
Figure 2: Density plot of Dir(aup) with different a. l is set as [0.7, 0.2, 0.1] for this illustration. Star (+) denotes the mean (μ)). Note that this value is invariant to a. Yellow denotes lower density, while it becomes denser progressively with violet.
Now, consider the property of reweighting and resampling. Reweighting multiplies pre-defined per- sample weight values the loss of each sample, indicating the importance of each sample. On the other hand, resampling alters the composition of the dataset by assigning an importance value to each sample via its sampling ratio. By interpreting not selecting as assigning weight value to a specific sample, resampling can be perceived as a process of per-sample weight vector sampling that exhibits concentration toward a specific dimension. Consequently, we suggest that both reweighting and resampling can be interpreted in the Dirichlet-based per-sample weight sampling (DWS) framework.
M
N
REDW == M
m w/l((o(x;) 7i), with wō ~ Dir(afu)
(3)
j=1
Here, w E RN is j-th sample following Dir(au and w = [w] .... wyyl M is the number of sampling w. With Eq. 3 per-sample weight parameters of existing Reweighting can be considered as the sampled weights from the Dirichlet distribution whose a - + Also, resampling can be interpreted as the reweighting method with sampled weights from the distribution with its a - O. In other words, we integrate both reweighting and resampling as Eq. 3 providing way compare these two distinct importance sampling based techniques comprehensively.
3.2 ANALYZING DWS FOR LEARNING WITH NOISY LABEL
Following Section 3.1
a controls the property of w. Therefore, analyzing the impact of a to R. necessary for comparing several DWS cases with different (i.e. reweighting, resampling).
((w)) and V (RRm))))) We start from the variance of wi as V(we) = #=11 (1-1) for
Σ1- a#t+l
a+1
all i 1,... N by definition. Here, V ((we) converges to -> 400 (reweighting) and becomes
4

---

Published as a conference paper at ICLR 2024
larger with smaller (if 0. it represents resampling). According to Lin et al. (022), increasing the variance of the risk function can improve robustness when learning with noisy label empirically. This study supports the robustness improvement of our framework, DWS with smaller a (or with larger V(w()) as Eq. Let = [(fo(=z). Mb),
Vi 1,... N for convenience.
M
V ((Ws)
1
Σ
(V(w)) + EddcCoo(uf w/) keyli
(4)
M
Note that our study is different from Lin et (202)) in that they focused on the impact of increasing the variance of the risk function regularization overfitting to noisy labels, while our objective is to suggest a unified framework to explain reweighting and resampling.
Distance from the true weight Mean of per-sample weights vectors, μ4, is approximated from the on-training classifier. Therefore, it may be different from the true per-sample weight vector.
Riv Note that the mean of per-sample weights distribution can be approximated as the Gaussian distribution following central limit theorem (Dou- glas C. Montgomery 2013), since w are i.i.d. sampled. means w 1/4
Let jo ,(Y= ( and Jr" Eg= A:'
p(V=ū.lzz
M7 2j=1 w will follow [p,, E/MM Σ is the covariance matrix of Dir(apu). We decompose Σ = S/(a + 1) as a and a-invariant term S. Then we get Mahalanobis distance (Mahalanobis 1936) between fr and v as:
das (p" 26 = (10 11²
Σ M
(1 μ) = v M(a + (1² 4) s- 1 (1+ 1))
(5)
The distance between μ1" and per-sample weight distribution is proportional to the square root of a. It means if μz (true) and μr (estimated) is not the same, d (p² . becomes smaller as a - 0.
Noise injection of Ri,DWS Recent researches including Neelakantan et (2015) suggest that in- jecting random noise the training procedure can improve generalization, and also works for learning with noisy label (Chen et al. 2020; Wei et al. (02Ia)). Following the concept, we interpret per-sample weights sampling as normally distributed noise injection during training, as Eq. 6 The intensity of noise can be controlled with a (Full derivation of Eq. i in Appendix D.Ib.
Σ pul(fo(xs)) (e) + Σ z.(ff((ss) Bc), with 2, N (0. Ma(1 - (1-))
N
lim REDW N++00
(6)
M(a +1)
Rowss with N - co can be similar to the risk of SNL (Chen et al.] 2020), who suggested label perturbation enhances the robustness for noisy label learning. We compare details of DWS and SNL. Specifically, the empirical risk function of SNL. is:
Comparison to Previous Work
Eq. 6 denotes hemp
Σ (Jo(Z)))))
zirl (Jo(w:),k), Zik 1(
In the above, a is hyperparameter and 2 is perturbation noise from standard Normal distribution. Eq. and Eq. [ are similar in three points. First, risks are decomposed into two parts: the static risk (the former) and the stochastic noise (the latter). Second, the mean of noise (E[z)) are O; and third, considering the noise parts (the latter part), they are composed as the multiplication of I(.,.) and 2.
On the other hand, Ra and Ri,SN, differ follows. First, R reflects distribution shift be- tween noisy and true label through p4, while R does not. This means that E can differ from RG. Second, the distributions of 24 for R are not identical, unlike R Since the vari- ance term in Eq. 6 increases with an increase in p:(5 0.5), introduces instance-wise adaptive perturbations. This results in larger perturbations applied confident samples, mitigating overfit- ting, while smaller perturbations to less confident samples, thereby preserving the training process. To empirically assess the difference between Rypss and R we compare the performance of the two risks and the noise injection to Reweighting in Section 4.4 and Appendix [44
Analyzing the impact of a to R smaller a can be beneficial. This also aligns with the experi- mental results in Section 4.3 and Appendix F. showing good performance with a - empirically. Based these analyses, suggest resampling utilize transition matrix in the following section.
5

---

Published as a conference paper at ICLR 2024
3.3 RENT: RESAMPLE FROM NOISE TRANSITION
This section proposes REsampling method to utilize Noise Transition matrix in this section. To our knowledge, this is the first resampling study on noisy label classifications. Inspired by the concept of
Resamllnn (SIR) (Rubin) 1988; Smith & Gelfand 1992), RENT involves resampling each data instance from the noisy-labelled dataset based on the calculated importance. Algorithm outlines the process of RENT
Algorithm 1: REsampling utilizing the Noise Transition matrix (RENT)
Input: Dataset D {æ4, 9r 1 classifier fa. Transition matrix T, Resampling budget M
Output: Updated fo
while fo not converge do
Get jı; = [a(())))) i. (T (oxx))) for all i
Construct Categorical distribution TN = Cat( An
UN "i
Eim
Independently sample W1). ... (IM, (MM. from TN
end Update fo by 0 + 0 Vo M Σ/= (((o(1)))
With the number of sampling as hyperparameter, we fix M .N for experiments unless specified otherwise. We provide the ablation study on M in Section Also, we conducted resampling based on mini-batch for implementation. We provide ablation for this sampling strategy in Appendix F.6 The empirical risk function of the resampled dataset is expressed as Eq. 8
1 Σ
A E
RIENNT = M ndl(fo(r)).0)) where [m1, (M1,-,, nx] Multi(M;
(8)
Σ
21=1
i. = fo(z;)s.// (TT Σn wa of Eq. with + O can be interpreted n. in Eq. other words, this multinomial distribution can be interpreted as distribution instance sampled from the Dirichlet distribution with a shape parameter, a, according Dirichlet-based based Weight Sampling.
In
Next, we focus on the property of the dataset sampled with RENT. With proposition 3.1 we demon- strate that R.RENT satisfies statistical consistency to the true risk. means that a dataset sampled from RENT can be regarded as i.i.d. sampled instances from the true clean label distribution, im- plying the possibility of RENT to build the noise-filtered dataset from the noisy-labelled dataset.
Proposition 3.1. If μ= is accessible, Ri is statistically consistent to R3 (Proof: Appendix D.3
4 EXPERIMENT
4.1 IMPLEMENTATION
Datasets and Training Details We evaluate our method, RENT, on CIFARIO and CIFARI00 Krizhevsky & Hinton 2009) with synthetic label noise and two real-world noisy dataset, CIFAR-
ION (Wei et al., 2022) and Clothing IM (Xiao et al.; 2015). The label noise in our experiments
include 1) Symmetric flipping (Yoo et al. [2020] Li al. 2021 Bae et al. (022) and 2) Asymmetric
flipping (Li et al. 2020 Liu et al. 2020 Bae et al. (022), marked as SN and ASN, respectively. CIFAR-10N is real-world nabsl-labelled dataset, with its label from Amazon M-turk. Clothing 1M another real-world noisy-labelled dataset with 1M images. For training, we report results with times replications unless specified. See appendix F.I for more implementation details.
T Estimation Baselines To check the adaptability of RENT over the different estimation of T, we apply Forward (Patrini et al. 2017), DualT Yao et al. 2020), TV (Zhang et al. 2021b), VolMinNet (Li et al. 202 and Cycle (Cheng et al. 2022) as estimation methods on the experiments. For real-world label noise, added PDN [Xia et al. 2020) and BLTM (Yang et al. 2022) as baselines for instance dependent
As RENT is method for 7 utilization, estimating T is prerequisite.
We show the process of DWS Appendix

---

Published as a conference paper at ICLR 2024
Table 1: Test accuracies on CIFARIO and CIFARI00 with various label noise settings. represents the training failure case. Bold is the best accuracy for each setting.
CIFARIO
CIFARI00
Base
Risk
SN 20%
SN 50%
ASN 20%
ASN 40%
SN 20%
SN 50%
ASN 20%
ASN 40%
CE
X
73,4
46.61997
78.4102
69.7
33.7
18.5
36.9111
27.3104
w/ FL
73.8
58.8
79.2:.0
74.2:05
30.7
15,5
34.2411
25.8
Forward
w/ RW
74.5
62.6411.
79.6
73.1
37.24226
23,5
27.2+
27.3:1133
RENT
78.7
69.0.a..
82.0100
77.8100
38.9
28.9
38.4107
30.4100
w/ FL
a9.9
71.8=a
82.9=02
77.7+06
35.2+00
23.4
38.3+04
28.4
DualT
RW
80.6
74.1
82.5+t02
77.9
38.5+110
12.0:t133
38.5+:1
24.0
RENT
02.
74.6
83.3
80,0
39,8
27.1
39.8
34,0
w/ FL
74.0.0.
50.41.0.
78.
71,61003
34,5
21.0114
33.9136
28,7100
TV
w/ RW
73. 7,09
48 48.5
77.3+:20
70,24:10
32.3:110
17.8:22
32.0::15
23.2:b0.9
RENT
78.8
62.5+118
81.0=04
74.0=0.
34.0=0.
20.0=a.
34,0aa2
25.5+00
FL
24.1
46.1
78.8 78.8
69.5
29.1
25.4
22.6+13
14.0
VolMinNet
RW
74.2:20.5
50.61b64
78.6220.
70.41:s
36.9111.2
24.41:30
34.91113
26.51b09
w/ RENT
79.4
62.611.
80.8100
74.0104
35.8
29.3
36.1
31,0108
w/ FL
81.6
82.81.04
54.3100
39.9128
39.4
31,3112
Cycle
w/ RW
80.2 2:100.2
57 37.0
78.19:n9
70.6::1.
37.8:127
30.2:1o6
38, 38.11:16
29.3::06
RENT
82.5
70.4
81.5
70.24:07
40.7
32.4:0.
40.7
32.2:00
FL 76.7.0o2
57.411..
75.0
70.7186
34.3
22.011.5
35.8105
31.9110
True T
RW
76.2+to.
58.6++1.2
35.0++08
21.8+00
21,3++66
21.6+104
w/ RENT
79.8
66.8
82.41:04
78.4
36.11:11
24.0
34,4109
27.2106
Test accuracies on CIFAR-ION and Clothing Due the space issue, we report perfor- mances only from parts of the baselines. Please refer to Appendix F2 for more results.
Table 2:
CIFAR-ION
Clothing
Risk
Aggre
Ranl
Ran2
Ran3
Worse
CE
80.8
75.610.
75.3
75.6106
60.4
66.9
w/ FL
79.611.
76.1
76.4104
76.0.0.2
64.5110
67.1
Forward
w/ RW
80.7
75.810.
76.0105
75.8
63.910.
66.811.
w/ RENT
80.8
77.7=0.
77.5=04
77.2+0.
68.0=09
68,2+00
FL
79.8
74.510.
74.5
74.3
57.5
64,910.
PDN
w/ RW
80.6*08
74.9+07
73.9=a7
74.4+00
58.7+0.5
w/ RENT
80.210.6
75.2
75.0
75.7
61.61116
67.2.00.
FL
81,5+u7
78.1
77.5+00
77.8+00.
65,8+1.0
67,2+08
BLTM
RW
54.0=339
64.4=27.0
50.9+529
38.0+324
43.5+2011
67.0=0.4
RENT
80.8.2.1
79.110.
18.
79.6.00.6
69.7120
70.0
transition matrix. To avoid the confusion, we denote Forward utilization explained in Section 2.3 as FL and the T estimation method from Patrinii et al. (2017) as Forward from now on. Also, we denote Reweighting utilization RW for convenience. Please check Appendix F.I for more explanations.
4.2 CLASSIFICATION ACCURACY
We compare FL, RW and RENT by applying each method to the various T estimation methods. Table shows the test accuracies of the classifiers trained with noisy-labelled CIFARIO and CI- FARI00 Experiments are conducted with 1) estimated T and 2) true T. First, RENT consistently outperforms FL in 42 out of 48 cases, as shown in the table. This demonstrates that RENT can improve performances of transition- based methods. is noteworthy that the performance gaps be- tween RENT and FL become larger in settings with higher noise ratios. Next, RENT outperforms RW in all cases except for two cases. Note that the performance gap between RW and RENT is marginal in the cases where RW is better, while the gap is significant when RENT exceeds RW.
Table 2 shows test accuracies for CIFAR-ION and Clothing M.. Again, RENT shows consistent improvement over the baselines for T utilization for real-world noisy labelled dataset. Please check Appendix F22 also for more experimental outputs including results over diverse T estimations.
'Some reported performances on this paper are different from those of the original paper. We unified experimental settings, e.g. network structures, epochs, etc, that were varied in previous studies. This setting discrepancy resulted in the changes, and we provide more details Appendix F2
7

---

Published as a conference paper at ICLR 2024
4.3
IMPACT OF TO DWS
As we demonstrated in Section would be able explain from [R R with a adjustment. Also, we explained the im- pact of a to DWS in Section [.2] Here, we report the model performances with various a empirically [ along with RW and RENT. As we can see in figure 3 there is an increas- ing trend of test accuracy with smaller a, and RENT shows the best performance for all cases consistently. It certainly aligns with the supe- rior performances of RENT over RW, explain- ing the benefits of introducing the variance to per-sample weights term empirically. Please re- fer to Appendix F3 for more results.
3.1 R Liws
TarVEr
Duaff
Cydla
16-1
10
(a) SN 20%
(b) SN 50%
Figure 3: Test accuracy with regard to various at for CIFARIO. (Star (x) is RENT and Cross (x) means RW, respectively.)
4.4 NOISE INJECTION IMPACT OF RENT
As in section 3.2
R.,Ds and Ras shares similar 80
70-
form in their structures, and compare the perfor- mance of DWS and SNL. in this section. Specifically, we report the performance of RENT, a sion of DWS. Since there is difference in the static risk term between Ri,SNL and RipDws we addition- ally implement the label perturbation technique sug- gested in the SNL paper (Chen et al. 2020) to RW for fair comparison. In figure 4 RENT consistently outperforms SNL,, implying label perturbation alone may be insufficient for managing noisy label. Fur- thermore, RENT performs better comparably with RW+e, indicating that RENT implicitly injects ade- quate noise during training process. Note that RW+e is highly sensitive to the value of o, so a simple combination of RW and label perturbation technique may not be enough for wide adaptation. Please check Appendix F4 also for more results.
70
60
ver-
60
50-
50
40
40
SNL
RW--
30
RENT
20-
0.0
0.5
1.0
0.0
0.5
1.0
(a) SN 20%
(b) SN 50%
Figure 4: Test accuracies over various for CIFARIO. RW+e denotes the integration of RW and the label perturbation technique.
4.5 OUTCOME ANALYSES OF RENT
wi value Samples with noisy labels should have weight values close to zero ideally, indicating their exclusion. We analyze the statistics of w; after train- ing in figure 5] It shows more than 80% of noisy labelled samples will be excluded.
80:0
2000
13300
1230
Clas (80
286-
(1.90
We also provide statistics on the number of clean and noisy samples whose categorical distribution param- eter is greater or smaller than 1/ BB respectively. For i..dd sampling within mini-batch (B represents the number of samples in the batch), all samples have the same weight value of 1/B. the normalized wi for (z,, 5.) is smaller than 1/B, it indicates that the sample is less likely to be selected compared to the i.i.d. sampling. The presence of large number of clean samples at the oversampling region and the concentration of noisy samples near zero support that fe trained with RENT effectively resamples clean samples when trained on noisy label.
Categieaal painee mhe
Categrrel mae
(a) SN 20%
(b) SN 50%
Figure 5: Histogram of w. of RENT CI- FAR 10. Cycle for T estimation. Blue and or- ange represents samples with clean and noisy labels, respectively. Vertical dotted line de- notes 1/B.
'We tested over E (0.1, 1.0, 10.0, 100.0, 1000.0].
8

---

Published as a conference paper at ICLR 2024
Confidence of wrong labelled samples Next, focus on the confidence value of samples with incor- rect labels. We compare the number of samples with incorrect labels based on the confidence of the mod- els trained using RW and RENT in figure [6] Two sets in the figure are distinguished by the threshold shows larger propor- tion of noisy samples is in the Uncertain group when the model is trained with RENT. implies that RW is more prone to fitting to noisy label, meaning the model memorizes incorrect labels during training.
IW RENT
30003
0000
Uneeram (1) Cestain
Unrestazn (f) Cenain (4)
(0.5 in our experiment).
(a) SN 20%
(b) SN 50%
Figure 6: Training data with incorrect labels divided by the confidence (threshold=0.5) Cycle for T estimation, on CIFARIO.
Resampled dataset quality We then analyze the quality of resampled instances by measuring their precision, recall and Fl score. Here, precision and recall can be recognized as the clarity and cover- age of the resampled dataset, respectively. Fig- ure 7 shows the efficacy of RENT over the base- lines (FINE (Kim et al. 2021) and MCD (Lee) et al., 2019)) considering the quality of resampled instances. RENT consistently surpasses the base- lines in Fl score, meaning RENT resamples clean yet diverse samples well.
FINE
0.8
0.8-
RENT
0.4
0.2
0.2
Prssision Rosall
Pancadux
Rocall
(a) SN 20%
(b) SN 50%
Figure Resampled dataset evaluation. Cy- cle for T estimation, on CIFAR 10.
Please check Appendix F.5 this section (Section 4.5)
also for more results of
4.6 ABLATION STUDY ON SAMPLING BUDGET
The size of the resampled dataset (M) be mod- ified as hyper-parameter, In practice, the sampling size can be adjusted based on the user's need, e.g. computer memory is not enough to accommodate the huge entire noisy dataset. As part of ab- lation study, we checked the classifier accuracy of fe when trained with different resampling budgets. Intuitively, the model performance could be either 1) higher, as it reduces the probability to resample noisy samples, or 2) lower, it restricts the chance for model to learn various data samples. Figure g illustrates the classification accuracy with different resampling budget of M. The results tend to show higher model performance for RENT compared FL, and indicate that RENT performs well even with smaller resampling budgets, suggesting for the further improvement of RENT.
Bounzphaa ct
Houcesbte badget 15
(a) SN 20%
(b) SN 50%
Figure 8: Ablation of M on CIFARIO. Re- sults from FL and RENT are denoted as dot- ted line and bold line, respectively.
For the ablation study regarding the sampling strategy, we report some results in Appendix F.6
CONCLUSION
In this paper, we first decompose the training procedure for noisy label classification with the label transition matrix T' as estimation and utilization, underscoring the importance of adequate utiliza- tion. Next, we present an alternative utilization of the label transition matrix by resampling, RENT. RENT ensures the statistical consistency of risk function to the true risk for data resampling by utilizing T. yet it supports more robustness to learning with noisy label with the uncertainty on per-sample weights terms. By interpreting resampling and reweighting in one framework through Dirichlet distribution- basedd per-sample Weight Sampling (DWS), we integrated both techniques and analyzed the success of resampling over reweighting in learning with noisy label. Our benchmark experiments with synthetic and real-world label noises show consistent improvements over the ex- isting T utilization methods well the reweighting methods.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was supported by Research and Development on Key Supply Network Identification, Threat Analyses, Alternative Recommendation from Natural Language Data (NRF) funded by the Ministry of Education (201RRAACC20086113)
REFERENCES
Jing An, Lexing Ying, and Yuhua Zhu. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. arXiv preprint 20tv:20009.344477 2020.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233-242. PMLR, 2017.
HeeSun Bac, Seungjae Shin, Byeonghu Na, JoonHo Jang, Kyungwoo Song. and Il-Chul Moon. From noisy prediction to true label: Noisy prediction calibration via generative model. In Inter- national Conference Machine Learning, pp. 1277-1297. PMLR, 2022.
Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Confer- ence on Machine Learning, pp. 540-550. PMLR, 2020.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependen label-noise learning possible. In International Conference on Machine Learning. pp. 825- 836.. PMLR, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution--aarr margin loss. Advances in neural information processing systems, 32, 2019.
Baiyun Chen, Shuyin Xia, Zizhong Chen, Binggui Wang, and Guoyin Wang. Rsmote: self- adaptive robust smote for imbalanced problems with label noise. Information Sciences, 553: 397-428, 2021.
Pengfei Chen, Guangyong Chen, Junjie Ye, and Pheng-Ann Heng. Noise against noise: stochastic label noise helps combat inherent label noise. In International Conference Learning Repre- sentations, 2020.
De Cheng, Yixiong Ning, Nannan Wang. Xinbo Gao, Heng Yang, Yuxuan Du, Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization. In Advances in Neural Information Processing Systems, 2022.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Rep- resentations, 2020.
Amit Daniely and Elad Granot. Generalization bounds for neural networks via approximate descrip- tion length. Advances in Neural Information Processing Systems, 32, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
George C. Runger Douglas C. Montgomery. Applied statistics and probability for engineers. John Wiley & Sons, 2013. ISBN 9781118539712.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2024
Yingsong Huang, Bing Bai, Shengwei Zhao, Kun Bai, and Fei Wang. Uncertainty aware learning against label noise on imbalanced datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.
2022.
Herman Kahn and Andy W Marshall. Methods of reducing sample size in monte carlo computations. Journal of the Operations Research Society of America, 1(5):263-278, 1953.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-2534, PMLR,
2018.
Tachyeon Kim, Jongwoo Ko, JinHwan Choi, and Se- Young Yun. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24137-24149, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Michat Koziarski, Michal Wožniak, and Bartosz Krawezyk. Combined cleaning and resampling al- gorithm for multi-class imbalanced data with label noise. Knowledge-Bass Systems, 204:106223, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning, PMLR, 2019.
pp.
Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels
as semi-supervised learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:
/ //operreview. wese:/forrmiidiieeeeee wtw.:mmm/dggg..........
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label- noise learning without anchor points. In International Conference on Machine Learning, pp. 6403-6413. PMLR, 2021.
Yexiong Lin, Yao, Yuxuan Du, Jun Yu, Bo Han, Mingming Gong, and Tongliang Liu. Do we need to penalize variance of losses for learning with label noise? arXiv preprint 12X:v:2201.12739, 2022.
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez- Granda. Early-learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
Granana
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-465 2015.
Yang Liu, Hao Cheng, and Kun Zhang. Identifiability of label noise transition matrix. In Interna- tional Conference on Machine Learning, pp. 21475-21496. PMLR, 2023.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Nor- malized loss functions for deep learning with noisy labels. In International conference on machine learning, pp. 6543-6553. PMLR, 2020.
PC Mahalanobis. Mahalanobis distance. In Proceedings National Institute of Science of India, volume 49, pp. 234-256, 1936.
Byeonghu Na. Ycongmin Kim, HeeSun Bae, Jung Hyun Lee, Se Jung Kwon, Wanmo Kang, and II-Chul Moon. Label-noise robust diffusion models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL tittps://openreee
ew. net. forumorrmm? i==
HXWTXXtHNI
Arvind Neelakantan, Luke Vilnis, Quoe V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv: 15II1060070 2015.
11

---

Published as a conference paper at ICLR 2024
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1944-1952, 2017.
Donald B Rubin. Using the sir algorithm to simulate posterior distributions. Bayesian statistics, 3: 395-402, 1988.
Adrian FM Smith and Alan E Gelfand. Bayesian statistics without tears: a sampling-resampling perspective. The American Statistician, 46(2): 88, 1992.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552-5560, 2018.
Xinshao Wang, Yang Hua, Elyor Kodirov, David A Clifton, and Neil M Robertson. Proselfic: Progressive self label correction for training robust deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 752-761, 2021.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 322-330, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13726-13735, 2020.
Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. Advances Neural Information Processing Systems, 34:7978-7992, 2021a.
Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. Advances in Neural Information Processing Systems, 34:7978-799 2021b.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Confer- ence on Learning Representations, 2022. URL TBWA6PLJZQM.
net/ forum?id-
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32:6838-6849, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Newral Information Processing Systems, 33, 2020.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2015.
Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent bayes-label transition matrix using deep neural network. In International Conference on Machine Learning, pp. 25302-25312 PMLR, 2022.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learn- ing. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 7260-7271. Curran Asso- ciates, Inc., 2020. URL /tspptttps///proeeeiiiggg http://ppooeeddaag..eeieeeeeeaee cttpppscc//ceee httpp//ccccpppppr///////eee 512c5cad6c3 7ecb
Paperr..--pppppppppp pd
12

---

Published as a conference paper at ICLR 2024
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependen label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34, 2021.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International Conference on Machine Learning. pp. 7164-7173. PMLR, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021a.
Manyi Zhang, Xuyang Zhao, Jun Yao, Chun Yuan, and Weiran Huang. When noisy labels meet long tail dilemmas: A representation calibration method. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pp. 15890-15900, October 2023.
Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In Marina Meila and Tong Zhang (eds.), Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 12501-12512. PMLR, 18-24 Jul 2021b. URL https: //procedd ml press/ 39/ zhang2 1n. html
Zhilu Zhang and Mert R Sabuneu. Generalized eross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learn- ing. pp. 11447-11457. PMLR, 2020.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. In International Conference on Machine Learning, pp. 12912-12923. PMLR, 2021.
Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training a model to predict. In International Conference on Machine Learning, pp. 27412-27427. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
A ILLUSTRATION OF DWS IMPLEMENTATION EXAMPLE
Figure shows an implementation example of our algorithm, DWS, as RW (previous) and RENT (ours).
Dirichlet based per-sample Weight Sampling (DWS)
a μ = [0. 0,0,2,00.11
a 0
w from Dir(a
from (ir(a(()p)
R] - (0.7
R(w*) ) (1.0
0.0
RW
RENT
times of w Sampling
W
[0.7,0.2.0.]]
w [1.0,]]
M times of Sampling
M-2
[0.7,0.2 [0.7,0.2,0.1]
1n [0.0.1]
M-1
[0.7,0.2.0.3]
MM -1 [0.1.0]
M [0.7.02.00.]]
WM [1.0,]]
Figure 9: Dirichlet distribution-bart based per-sample Weight Sampling (DWS) with shape parameter and the mean vector μa. Similar to Figure image at the vertices of yellow triangles represents data instance. Blocks above the images represent Class (upper), Label (lower). Green means labels are same as true class (clean) and Red means labels are different from true class (noisy). R(w!) represents the risk function with w per-sample weight vector. Each colored box represents one sampled w.
In DWS framework, sample per-sample weights vectors, for M times. Each sampled w becomes per-sample weight vector. From the property of the Dirichlet distribution, if the shape parameter & + oo, sampled w will be near the mean vector. However, a sampled w will be clustered to vertices. Due to the space issue, we showed only one sampled w in figure I while we illustrate more times of per-sample weights sampling here.
B PREVIOUS STUDIES
We first explain previous studies for learning with noisy labels. Next, focusing the studies utiliz- ing the transition matrix, we summarize previous studies considering the estimation the transition matrix, which were not included in the main paper.
B. LEARNING WITH NOISY LABEL
Considering learning with noisy label. several directions have been suggested, including sample
selection (Han al. 2018 Yu et al. 2019 Wei al. 2020: Cheng al. 2020), label correction
14

---

Published as a conference paper at ICLR 2024
Tanaka et al., 2018 Li et al. 2020 Zheng al. 2020 Wang al. 2021) and robust loss (Zang
& Sabuncu| 2018: Wang et al., 2019). Sample selection based methods manage noisy-labelled in- stances by setting the objective
removing them during training. For example, Han et al. (2018) assumed that samples whose losses are small during training have clean labels. However, this will cause learning bias toward early learned easy samples because the deep neural network will easily overfit to those samples. As a solution, Han et al. (2018) proposes using two same-structured net- works with different initialization point, and utilize the output of the other classifier as the metric to decide whether to select or not each sample for training classifier. However, since two different classifiers have still finally converged to the same output, Yu et al. (2019) has proposed to select samples when the outputs of the sample from the two different networks are different. It may solve the problem of two different networks' alignment considering the output of a sample after enough time, it may not work better even than Han et al. (018)) since it selects too small portion of samples, especially when the noise ratio is high. Wei et al. (2020) focused on this problem and they relieve this limitation by updating two networks together with introducing the KL divergence between the output of the two networks as regularization, making the output of two networks become closer to true labels and that of their peer network's. s. Cheng et al. (2020) selects samples with dynamic threshold and regularize confidences of samples.
On the other hand, Label correction-based methods do not waste samples with recycling noisy- labelled instances by relabelling them. For example, [Tnaka et al. (2018) optimizes both the classi- fier parameters and labels of data instances jointly, initialize the network parameters and train with the modified labels again. Li et al. (2020) considers samples with large loss as unlabeled samples and solve the nobsl-labelled dataset problems utilizing cemi-supervised learning techniques, giving the noisy-labelled samples pseudo-labels. Zheng et al. (2020) calculates the likelihood ratio between the classifier's 's confidence on noisy label and its confidence on its (assumed) true label prediction as its threshold to configure clean labeled dataset, and corrects the label into the prediction for samples with low likelihood ratio iteratively. Waang et al. (202T) analyzes several types of label modification approaches and suggests label correction regularization hyperparameter depending both learning time stage and confidence of a sample.
Studies modeling the loss function which is more robust to noisy label include Zhang & Sabuncu (2018), which combine the advantages of the mean absolute loss (MAE) and the cross entropy loss (CE) and Wang et al. (2019), which adds the original cross entropy and reverse cross entropy. Apart from that, studies like Liu et al.] (2020) relies on the fact that the deep neural networks memorize the noisy labels slowly, and they regularize the network not to memorize the noisy labels by maximizing the inner product between the model output and the weighted outputs of previous epochs.
Although these studies would show good performances empirically, these studies cannot ensure statistical consistency of (I) classifier or (2) the risk function to the true one Yao et al. 2020)) Therefore, we now focus on the studies which utilize T.
B.2 PREVIOUS RESEARCHES ON T ESTIMATION
In this section, we explain more details of the previous studies considering T.
T estimation under CaasiiCondiiinnal Noise (CCN) setting stems from Patrini et al. (2017). It de- fines T as the matrix of the transition probability from clean label noisy label. It also suggest two loss structures, Forward loss and Backward loss, which ensures statistical consistency of classifier and statistical consistency of risk, respectively. Since these two loss structures ensure theoretical success of learning with noisy label utilizing T, studies have focused on the estimation process of T. since T' is actually unknown information.
Patrini et al. (2017) estimate T with anchor points, yet finding the explicit anchor points in dataset may be unrealistic and difficult. Therefore, Yao et al. (2020) decomposes the objective of estimation as 2 easily learnable matrices: 1) a transition matrix from noisy label to bayes optimal label and 2) transition matrix from bayes label to true label. Since bayes optimal label is one-hot label, 1) can be estimated by summing up the samples with specific noisy label and the specific bayes optimal label, and 2) estimation would be easier than the original T estimation. Zhang et al. (021b) points out the overeonfidence issue for T estimation, and find way to estimate optimal T by maximizing the total variation distance between clean label posterior probabilities. However, it has theoretical assumption of having anchor points and requires ensuring the multiplication of two matrices (V and U) should
15

---

Published as a conference paper at ICLR 2024
be equal to true T. Li et al. (2021) is research studied at similar time as Zhang et al. (2021b) and it finds the optimal 7 by minimizing the volume of simplex enclosed by the columns of T. It relieves the anchor point assumption, yet it is still vulnerable to overconfidence issue reported at Zhang al. (2021b). Motivated by this error gap of 7 estimation, Cheng et al. (2022) suggests a comprehensive loss of utilizing both Forward and Backward loss structure. It shows good performance empirically, but the optimal T learned by Cheng et al. (0022) would be an identity matrix by its modeling, since both I" and T', which is an approximation of T are modeled as diagonally dominant and all cells are nonnegative.
Apart from these directions, Zhu et al. (2021) calculates T by solving the optimization problem with constraints. These constraints stems from Clusterability, which assumes samples with similar features would have same true label. As an effort to reflect the feature information to transition probability, T estimation processes under IDN condition have been studied. For example. Xia et al.) (020) assumes weighted sum of transition matrices of parts can explain instance-dependent transi- tion matrix. Berthon et al. 2202)) assumes accessibility to p(y = - = i, for every sample and calculates YY Yang et al. (2022) parameterize T(X) and trains a new deep neural network which gives instance-dependen transition matrix its output using distilled dataset, which is the subset of the original training dataset with its maximum softmax output is large enough. Although modeling instance dependent transition matrix may be more realistic, estimating T(X) is far more difficult because its domain space becomes much bigger than T modeling. Therefore, its estima- tion impossible without additional assumptions or information (Liu et al.) 2023), since the true C' x C' matrix cell values per every single samples would have unbounded solutions per each sample, making it harder to analyze the properties of estimated T(X).
Please note that to write as p(((2)) (YY)), the definition of T(x) notation should be as (x(()) = p(V = J = k, x)) Vj.k = 1,.., C. Take a 2-dimension example. should be as:
r 11x)] [p( 2|x)) Lo(y 2 = 1,))
1, f 1 1,
N = 2)) p(Ý 2 x]
p(Y -1(r)) p(Y = 2|xr
(9)
meaning that the (j, element of (x)) should be p(Ý j|Y k,
C DISCUSSIONS FOR PREVIOUS TRANSITION MATRIX UTILIZATION
We assume that is invertible, which has been generally assumed in the previous researches.
First we discuss Forward utilization. As we defined in the main paper, let e be p0 (x) - p( (xx), where 50 [2) is noisy label probability vector estimated from the classifier trained with noisy labels. Then, p( (|])) becomes p\ [ær) e, which means, T'p(Y (x) = p( (x) p [z) e. Therefore, T E.
p(Y|>) (z) becomes T (p [x) - = T T->T->((})>)
(p(Y|=)
Following Eq. 2 on the main paper, fo( will be equal to p(Y [z) if and only if To( p()) for all (I,, T.) M- It means that fo(x) = p(([z)) < R. = 0 and p( (z)) is required for train- ing for However, p(*(2z) should approximated by the noisy labels from the dataset since the
act value of (Y))))) is unknown and it makes the gap. In other words, we get So by minimizing
ReF = 1/4 DM (ox;), ūi), and if we minimize R so(x)) will be T-1 p(Y(x) so that the gap between fā (z) and fo( will become T-
The difficulty of estimating p({|=)) from learning model (() with δ has already been denoted before. Remark C. is one of those proposals.
Remark C.1. (Zhang et al.) 2021b) The estimation error of the noisy label posterior distribution, p(((.), from neural networks trained with 5 could be high. This confidence calibration might be more difficult than XX) estimation, because p(Y should be within the convex hull of transition matrix T, Cony(T), i.e., p() E Conv(T) C C-1, and Conv(T) + C-1 if + T.
Remark C.1 claims the difficulty of estimating p{| from D, because the value of Y (x) does not achieve the full range of [0, 1). Intuitively, p(|(x) would not be represented one-hot because
16

---

Published as a conference paper at ICLR 2024
the noisy label generation process would not be deterministic, which is also claimed by previous works Yao et al. 22020; Zhang et al. (2021b); Yao et al. (2021).
This failure of ( estimation a crucial issue considering the statistical consistency fa to fg since fo(x) = p({[z) [x) if and only To())) = p(Y [z). Therefore, the gap between p(Y (x:) and p(Y (r) make the inevitable gap for estimating (x) It means that classifier trained with empirical Forward risk may not be able estimate p(Y Y (x) accurately even with true T.
We also propose fo trained with R, can memorize all noisy label under the following assumption.
Assumption C.2. Given noisy training dataset D {z ū:) let æ; + xj for all # j. and T3 > Tix for all j + k of and fo has enough capacity to memorize all labels.
For the each term of the risk function R we have / (TT) T..) = - log 2 1 (a.s 2
log Ja (Ta)) It is from Ej=1 E (z)fs(z))) I Ej=1 T fz(x;) = Tz. Note that (x) 2 for all k and Ee fa(x) = Also (max:T = Tg. by the assumption C.2
Ej=1 (xa) To: when fz(2;) = 1 for j = 0: and 0 otherwise. Therefore, fo that mini-
mizes R can result in fo ( = ÿ: for all i 1, ,,.., Pl,
Figure TO supports the above proposal empirically. It de- picts the train accuracies with noisy labels (left), train accu- racies with the original true la- bels (center) and test accuracies (right) of various methods with Forward, where each method is trained on the noisy-labelled dataset. As training iterations progress, all methods utilizing Forward eventually memorize the noisy label, which leads to the degradation on the test ac- curacy. Note that the learning with true T (denoted as True T) also memorizes the noisy label. In contrast, methods equipped with RENT alleviate noisy label memorization; they also show the correct inference capabili- ties for the clean labels of noisy training instances, which an evidence of non-memorization.
90
Accuray
70-
CE
Forward
68
70-
Valldin
66
True 7
200
400
200
400
200
400
Trsining Epoch
Figure 10: Training accuracies with regard to the noisy labels (left), the clean labels (center), and test accuracies (right) of var- ious methods on CIFAR10 (symmetric 20% noise). We applied Forward and our method, RENT, to various T estimation Patrini et al.] (2017); Zhang et al. 2021b); Li et al. (202T). We express the result of Forward and the result of RENT as dotted line and bold line, for each estimation respectively. CE the result of the classifier trained with Cross Entropy loss.
We also discuss Backward briefly. Please note that (with as Cross Entropy loss)
C
(1x)TT))
Σ 4TT. ET log(ff(()) log(f
(10)
T = I, then since 7-1 I, Eq. becomes loo fo()).
+ T, we first think of C case, when T-1 has a simple form.
As fo(x)<* = - fo(z)g the derivative of the empirical backward risk over fo(x); becomes
T-1 T-1 To Note that T-1 Tin Tia Ja(a)s
T22
-Ti2]
Assuming T1 > Tai and T22 > Ti2. both Tip1 and T become negative. It means that 1 T is negative, making the derivative when fx(x) near goes to -o0.
+ 1-fo(2];
-T21
Tn
17

---

Published as a conference paper at ICLR 2024
For C >>> 2 case, the direct analysis is impossible because there is explicit form for the inverse matrix. However, since E [o, 1] the inverse of the transition matrix can easily be negative it is not identity matrix so the same issue may happen.
D DERIVATION AND PROOF
D.1 DERIVATION OF EQ. 6
Here, show the derivation of Eq. fuully. Notations same as the main paper, which means,
we RN is j-th sample following Dir(au).. Let W;
Σ
wi
4
1(fo(x)); (i) for all
M
i I,..., N for convenience.
M
N
N
R.DS
1
ΣΣ
Σ
4
Σ
Wel
(11)
M
M
j=1==1
Since not ... ia is i.i.d. for same i, apply Central Limit Theorem: W; N (wi pill-p))
M(t+))
Remp
W p4 + 14)4
(12)
N
(w] (4) 4, with (wo - 14) ~ N (0, 14(1-11)
p44
M(a+1)
D.2 EXPLANATION FOR THE CATEGORICAL DISTRIBUTION PARAMETER OF RENT
p(==Bla) p(Y=ū/=) Here, explain how the per-sample weights can be formulated (T)(Y(=)) p(}=ū|z))
Following the derivation from Liu & Tao (015)). The likelihood ratio between p(X, Y) and p(X, V) will be same as the likelihood ratio between XX) and p()()). change for noisy label classification task.
because p( ((X) does not
(f(f)) = ((x,))x(xx,)) [Iff(I)) w)1 = E ((xx)))( ((Fo(w)) pla, Y 1))
pl v 17)
((Jo(z). 1) p(Y = -))))(())
((Fo1z), 1) p(Y = 5/
E (..
B ((x
p(Ý ((1))(())
p(Y 5)x)
p(Y -](=r ((fo(z).
p(Ý -(lr)
(13)
Following the concept of 1992). this importance sampling is transformed algorithm |
(SIR) Rubin 1988 Smith & Gelfand
D.3 PROOF OF PROPOSITION 3.1
Proposition D.1. !f μz" is accessible, Ri is statistically consistent to Ri.
Note that the true mean of weight vectors, μ², is assumed be accessible for this part. In other words, we assume
R1
BN is equal to fa", with j4 (Tfo(::))
fa(ms)) for all i.
Ei-1 Rr
p and μr ΣN -(
as in the main paper.
Proof. We first start by rewriting the risk function of RENT (Same as the main paper). Let Dx {(0,, N. is the training dataset, which comes from x(X, Ý) = Tp(X, Y). with size N.
1 N RERRT = M
Σ nl(ff((xs), ūz), where [ni...., ny Multi(M;
uN
) (14)
y...
Σ i4
18

---

Published as a conference paper at ICLR 2024
Then,
Remp E LRENT
= E
E
RTRENT
[NN
=E
E
Σ
n.((fo(zs). 8i) DN
M
1=1
((f((z)). -
E
E
E
(15)
kml jix
By the assumption above,
N
74
((fo(z:).
Σ
(fo( (t)
A=1
i=1
k=
p(Y = ūr 21)
(7l(fo(Z.)..)
((fe(z;). (i)
Pk
Σ P i=1
p(Y = Et xz))
pYY (.())p(())) 11 ((fo(z)). U1)
N
p(Y = Di)
Σ
([f((x)), V.)
p(Y (((x;)p(a..)
Ek=1
N
p(Y Úí xza)
N
p(Y = 24) p( = Dr x..)
(x:).
(16)
N 2
pY=iha p(Y= (..
N
Ek=1 Me works
constant term with regard to Then, RERENT converges to R. p) strong law of large number N goes to infinity. (Note that DN comes from p(X, Y).) Therefore, RERENT is statistically consistent to the true risk.
] by the
E ALGORITHM FOR DIRICHLET BASED SAMPLING
Here, we show the algorithm of dirichlet based per-sample weight sampling process.
Algorithm 2: Dirichlet distributton-basedd per-sample Weight Sampling (DWS)
Input: Noisy dataset D {œx, de > classifier fo. Transition matrix T, concentration
parameter a, the number of sampling M
Output: Updated fe while fo not converge do N Get μ4 f.:/ Σÿ= Kÿ. where ju: [o(=:)) for all for j 1,.., M do
Independently sample w from Dir(a, 11)
end
Update fo by 1 0 Vo EN (f)))( 1
end
F EXPERIMENT
F.1 IMPLEMENTATION DETAILS
Network Architecture and Optimization We utilized ResNet34(He al. 2016) for CIFARIO and ResNet50 et al. 2016) for CIFAR100. We used Adam Optimizer (Kingma & Ba with learning rate 0.001 for training. No learning rate decay applied. We trained total 200
2014)
epochs for all benchmark datasets and no validation dataset was utilized for early stopping. We
utilized batch size of 128 and augmentation, HorizontalFlip and RandomCrop were applied.
19

---

Published as a conference paper at ICLR 2024
Considering Ilothing we used ResNet50 pretrained with ImageNet (Degg al. 2009). As a same condition with benchmark dataset setting, we utilized Adam Optimizer with learning rate 0.001 with no learning rate decay. We trained 10 epochs and set batch size as 100. RandomCrop, RandomHorizonaaffig and Normalization was applied during training, and only Centercrop and Normalization was applied at testing. For experiments over Cl do not use clean validation dataset in training.
Unless being specified, we keep this experimental settings for other experiments.
Synthetic noisy label generation Considering CIFARIO and CIFAR100, all samples from these benchmark dataset are assumed to have clean labels. Therefore, we arbitrarily inject noisy labels following the rules below. Let 7% a noisy label ratio.
(1) Symmetric flipping (SN) Yao al. (2020): Zhang et al. (2021b); al. (2021); Bae et al.
2022 flips labels uniformly to all other classes.
We set (1 % samples of each class unchanged.
(2) Asymmetric flipping (ASN) Li et al. (2020): Liu al. (2020): Cheng et al. (2022): Bae et al. 2022) flips labels to pre-defined similar class. For CIFAR TO, we flipped label class as Truck = Automobile, Bird = Airplane, Deer = Horse, Cat < Dog following the previous researches. For CIFARI00, we flipped between sub-classes within each super-class.
Dataset description CIFAR-ION (Wei et al. 2022) contains real-world noisy-labels of CIFARIO images from Amazon Mechanical Turk. The label noise includes 5 types; Aggregate (Aggre), Ran- doml (Ran1). Random2 (Ran2), Random3 (Ran3) and Worse. For detailed descriptions of how the noisy labels of each noise type are created, please refer the original paper. According to the original paper, the noise ratio is 9.03%, 17.23% 18.12%, 17.64% and 40.21% %, respectively.
Clothing IM (Xiao al. 2015) real-world noisy-labelles dataset collected from online shopping websites. The dataset includes million images with 14 classes. Noise ratio is estimated as 38%.
Baseline description Here, we explain methods that we used in experiments to estimate T..
Forward Patrini et al. (2017) identifies anchor points based on the calculated noisy class posterior probabilities. Following the customs of the original paper, we chose the top 3% confident sample for each class as anchor points.
DualT Yao et al. (2020) decomposes T into 1) transition from noisy label to intermediate class; and 2) transition from intermediate to true class to reduce the estimation gap of T'.
TV Zhang et al. (2021b) estimates by maximizing the total variation distance between clean label probabilities of samples. Although it requires an anchor point assumption theoretically, it does not need to find anchor points explicitly.
Ne al. (202 estimates the optimal T by minimizing the volume of the simplex formed by the column vectors of T. The volume measured as log of the determinant o as original paper.
Cycle Cheng al. 2002) develops an alternative method, which minimizes volume of T, without estimating the noisy class posterior probabilities as VolMinNet. minimizes the comprehensive loss which takes the form of forward plus backward plus the regularization term which obligates the
multiplication of and I' to be an identity.
PDN (Xia et al. (020)) assumes weighted sum of transition matrices of parts can explain instance- dependent transition matrix. Therefore it first calculates the transition matrix for anchor parts and get instance- wise weights to multiply to each part matrix.
BLTM Yang al. 2022 parameterize T TXX) and trains a new deep neural network which gives instance- dependent transition matrix as its output using distilled dataset, which is the subset of the original training dataset with its maximum softmax output is large enough.
We did not consider CSIDN (Berthon et al. 202 as our baselines because they requires the infor-
mation of p(Y = T Y = T.). although we do not have those kind of information basically.
20

---

Published as a conference paper at ICLR 2024
Techniques used to hinder memorization in previous researches niques in previous researches used to hinder memorization as table CIFAR1O experiment conditions.
We report regularization tech-
Settings are reported based on
Table Regularizations to hinder noisy label memorization used in previous researches 10)
Method
Dropout
Early Stopping
Lr decay
W decay
Augmentation
Forward Patrini al.
2017
X
O
10
Horizuntalllip, RandomCrop
DualT Yao et 2020
X
10-4
HorizontalFlip. RandomCrop. Normalization
Zhing et al.
2021b
X
10-4
HorizontalFip, RundomCrop, Normulization
VolMinNet Li
2021
10 3
HorizontalFlip, RandomCrop
Cycle Cheng 2022
X
0
10
Not reported
Ours
X X
X X
HorizontalFlip. RandomCrop
We marked the asterisk on the early stopping of TV Zhang et al. (2021b) since the number of epoch is different (40) from setting (200). As reported in table 3 conducted experiments removing several techniques without augmentation. We show the difference between the reported performance in the original paper and the reproduced performance following their condition, the reproduced performance under our experiment settings in the following section.
F.2 MORE RESULTS FOR CLASSIFICATION ACCURACY
Performance with regard to the noise ratio Figure shows the performance comparison of FL, RW and RENT changing the noise ratio for several estimation methods. As we can see the figure, the gaps between red lines and others become larger with higher noise ratio.
200
120
100
FL
RV
RW
RiN
RENT
RENT
26
20-
40
200
23
42
120
20
49
62
102
Sakie (30009)
Nocc 1st och
(a) Forward
(b) DualT
(c) TV
130
102
WWW
10
50-
MENT
20
20
21
Molte (oth
Malon miN5022
Note INT
(d) VolMinNet
(0) Cycle
(f) True
Performance comparison when using different T utilization methods over several T estimation methods. Subscripts of each figure represents estimation baselines and colored regions mean standard deviation. X-axis and Y-axis of each figure represents noisy label ratio and the test accuracy, respectively. CE means training with Cross Entropy loss.
Figure 11:
Experimental results are based on CIFARIO datasets with symmetric noise, and replicated over times. Please note that when the noise ratio becomes 90%, it means nothing but random label status, so it is natural the test accuracy is equal to 10%.
True T is NOT the best? One of the interesting findings we can see in Table T is that the model performance is not the best when it is trained with the true transition matrix (denoted as True T))
21

---

Published as a conference paper at ICLR 2024
To check what happens, we first report the performances of RENT with several T estimation meth- ods over the differences between the resulting estimated T and true T. Furthermore, to check any performance pattern over T estimation gap is unique for RENT or not, also report results for the performances of Forward loss as T utilization (the conventional).
w FL
EA
W/ RENT
Duff
(a) SN 20%
(b) SN 50%
(c) ASN 20%
(d) ASN 40%
Figure 12: Performances with several transition matrix estimation methods over T estimation gap (AT, calculated as 12 norm). Sub captions mean noise settings and each color represents different T estimation methods. Figures on upper lines are performances of Forward (the conventional utiliza- tion), and that on lower lines are performances of RENT as T utilization. X axes mean T estimation error (calculated as 12 norm) and y axes mean the respective performances. Dots mean different seeds. We do not report Cycle SN 50 (%) cases in the figure because its test accuracy is 10 (%).
Table 4: Pearson Correlation coefficient with regard to AT and model performances.
w/ FL
w/ RENT
SN 20%
SN 50%
ASN 20%
ASN 40%
SN 20%
SN 50%
ASN 20%
ASN 40%
Pearson Correlation
0,2234
0. 0.1448
0.2355
-0.2601
0.4648
-0.1590
-0.6444
-0.5597
Figure 12 shows the results. Interestingly, not only can we find out that the performance is not the also check that the relation between AT and the model performance is not linear. Check Pearson Correlation also in Table
best when AT is equal 0, but
At first, we conjecture that this failure of finding the correlation between AT and the model per- formance is due to two reasons: (1) considering TVV, VolMinNet and Cycle includes regularization terms for updating a classifier, those terms may have affected the classifier training process and (2) For TV, VolMinNet and Cycle, the transition matrix changes during training procedure, so the tran- sition matrix estimation gap would have reduced while training (according their original paper, the estimation error seems to decrease with training process). Therefore, the transition matrix gap of those algorithms would have been larger than the reported transition matrix estimation gap.
To analyze the impact of T estimation error to the model performance, comparing performances un- der same risk function (including no regularization terms) and other learning procedures is required. Therefore, we subtracted ér to diagonal terms and added eT (the number of classes-1) to others. Figure 13 shows the model performance over AT.
Figure 13 shows the performance over AT. For SN 20 (%), set ET =00.0, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6] and for SN 50 (%), we set éT =[0.0, 0.05, 0.1, 0.2, 0.3], since 0.7 and 0.4 would mean same as total random label flipping respectively. Lines represent performances with arbitrary corrupted transition matrix and small dots is mean performance of each cr. It again shows that the performance is not the best when the transition matrix estimation enror is 0.
Considering Forward, DualT and True, their performances are within the colored region, possibly supporting the explainability of this arbitrary transition matrix corruption experiment.
22

---

Published as a conference paper at ICLR 2024
70
Amurary (80
80-
DuiTT
TV VolkdinNet
Cyrin
02-
Trw
0.0
0.5
1.0
1.5
2.
0.0
0.2
0.4
0.5
0.B
AT betwoee: the nod inlimtedd TT and the trae (12mm)))
.77 betwaen the nnd inlimaddd and the (12mm)))
(a) SN 20 (%)
(b) SN 50 (%)
Figure 13: Performance over AT. Dots are same 12 Colored regions standard deviation over 5 seeds. For SN 20 (%), we set er = 10, 0.05, 0.1,0.2, 0.3, 0.4, 0.5, 0.6] and for SN 50 (%), we set cr [0, 0.05, 0.1, 0.2, 0.3), since 0.7 and 0.4 would mean same total random label flipping respectively.
Performance with Lin al. (2022) We compare the performance of RENT and Lin (2022) whose method name is VRNL. We also report the performances with Forward utilization (w/ FL) again as baselines, following VRNL.
Table 5: Test accuracies on CIFARIO and CIFARIO0 100 with various label noise settings including VRNL. represents the training failure case. Bold is the best accuracy for each setting.
CIFARIO
CIFAR100
Base
Risk
SN 20%
50%
ASN 20%
40%
20%
SN 50%
20%
ASN
FL
73.8:103
58.8403
79.2400
74.2=03
30.7425
15.5
34.2412
25.8:114
Forward
VRNL
76.9+a0
64.8+23
54.2+92
59.3
34.1
18.4+28
35.5+24
27.2=16
RENT
78.7#03
69.0
82.0g05
77.8=0.5
38.9
28.9
38.4+0.7
30.4=03
w/ FL
79.9.0.
71.8+03
82,9402
77.7=0.
35.2+04
23.4=10
38.3+04
28.4=26
DualT
w/ VRNL
81.2#a0
73.7+00
83.5+al
78.1=22
38.2+114
25.2+
40.0+1.
34.4=13
RENT
82.0.02
74.6204
83.3+a
80.0g09
39.8
27.1
39.8+0.7
34.0
w/ FL
74.0.005
50.4106
78. 1113
71.6103
34,5
21.0114
33.9133
28.7.108
VRNL
76.0100
51.31107
78.81003
58.5
28.5
22.9
29.9
26.4
RENT
78.8
62.5
81.0a4
74.0=00
34.0
20.0=06
34.0s0.2
25.5
FL
74.1102
46.1:22
78.8.10
69.5
29.1
25,4
22.611.
14.0.109
VolMinNet
w/ VRNL
76.3
50.3+14
72.3+99
67.4=53
28.1
26.6=22
19.5+3.
14.7=114
w/ RENT
79.4=a03
62.6#13
80.8
74.0=04
35.8+00
29.3
36.1
31.0=08
w/ FL
81.6a05
82, 82.84
54.3=03
39.9
39.410.
31.3+12
Cycle
VRNL
82.4+a6
83,0+04
54.3+04
41.9+2.1
42.1+16
32.5=12
RENT
82.54a2
70.4
81.5
70.2203
40.7
32.4
40.7+0.
32,2406
FL
76.7102
57.41:1
75.0
70.7486
34,3105
22.021.5
35.810.5
31.9210
True T
VRNL
79.3
63.8111
49.2.144
47.7162
36.611.1
25,5100
26.2133
22.5
w/ RENT
79.8
66.81.0
82.4
78.4
36.1
24.0a0.3
34.410s
27.2:06
One step further from the original paper, we also report experimental results with DualT Yao et al., 2020). TV (Zhang et al.) 20216), Cycle (Cheng et al. 2022) and True T. since can be applied to various transition matrix estimation methods orthogonally. We reported these performances follow- ing Work with VolMinNet in Practical Implementation part Lf et al. (2022). In other words, added the variance of the risk function (not including another regularization terms) for imple- menting VRNL with DualT, TV, Cycle and True T. Performances in table 5 consistently shows VRNL improves the original T estimation methods, and RENT tends to show better performances than VRNL. We demonstrate that RENT can increase the variance the risk function enough with- out the need to choose the parammer for variance regularization term (denoted in Lin et al. 2022)).
For the hyper-parametter a, followed settings from their paper.
23

---

Published as a conference paper at ICLR 2024
Table 6: Performance comparison with regard variance of the risk function over various T es- timation. Bold means the better accuracy between VRNL and RENT for each data setting and T estimation. If the performance of the integrated method is better than the original both methods, we underline the performances.
CIFARIO
CIFAR100
SN
ASN
SN
ASN
Base
Regularizer
20%
50%
20%
40%
20%
50%
20%
40%
VRNL
76.9+0.4
64.8=23
54.2=99
59.3=127
34.1+34
18.4#28
35.5+24
27.2+16
RENT
78.7
69.0.
82.0
77.8
38.9
28.9
38.4
30.4.0.3
Forward
0.0001
78.010.8
68.3_15
81.0.110
76.811.2
36.9111
25.412.1
38.5
29,911.5
0.001
77.9+1.
68.5-03
81.0=0.
72.5+s9
38.0=1.5
25.8+14
38,9+0.1
28.2+4.
0.01
77.5
68.9=07
80.9
76.8210
37.6405
27.1
37.9
30.3
0.05
78.9
71.1=04
52.7:3449
$2.0
37.7
26.9229
38.5106
30.8
VRNL
81.2.0.3
73.7
1..
78.1
38.2114
25.2
40.0
34.411.
RENT
82.0
74.6_004
83.3201
80.0209
39.8209
27.1:19
39.8
34.0.0.4
DualT
0.0001
81.6+0.4
74.2=07
82.9=a5
79.4=03
39.0=04
26.7+48
38.5+06
33.5
0.001
81.4005
74.0=112
83,5
79.3
38.5
26.7453
40.1
32.7.0.9
0.01
81.1
73.4=07
82.820.
78.7208
39.3
26.2226
38.7108
33.0214
0.05
81.8+0.5
71.8-1.5
82.6=07
80.3+04
40.0=112
27.0=28
37.1
33,4+14
VRNI.
76.0
$1,3=07
78.8
58.5
28.5=112
22,9228
29.9
26.4+23
RENT
78.8
62.5=18
81.0=04
74.0=05
34.0=00
20.0aas
34.0+0.2
25.5
TV
0.0001
77.7
61.5
78.2
74.3108
33.2
19.7.06
34.8
25,4114
0.001
78.2
60.4-15
80.0=08
13.1
31.0240
20.7:a5
33.5216
26.5
0.01
78.2+1.1
62.6-28
81.1=au
74.1+11
32.8=29
20.0=a7
33,4+10
25.6
0.05
80.3
68.8 1.4
80.7
73.7109
35.2
21.6104
34.0123
23.2144
VRNL
76.3+09
50.3=144
72.3
67.4
28. 18..6
26.6
19.5
14.7
RENT
19.4
62.6 1.3
80.8
74.0
35.8
29.3
36.1
31.0108
VolMinNet
0.0001
78.1
60.4_23
80.7
72.0211
36.3208
27.8:007
32.1
29.1
0.001
77.8+0.7
62.6
78.4
72.2+1.
31.8=40
28.2+11
35.0+25
29.5
0.01
78.3
64.2__.6
80.1
73.0112
35.9100
27.4119
34.711.1
29,411.
0.05
80.1+07
69.7=10
80.5=00
72.3+1
37.0=17
30.8+08
33.0#69
31.3+19
VRNL
82.4
83.0
54.3204
41.922.1
42.1
32.5
RENT
82.5+02
70.4=03
81.5=a
70.2=07
40.7=04
32.4=04
40.7+0.7
32.2+06
Cycle
0,0001
82.1
70.4=1.0
80.8
69.1
40.9=112
31.3=a9
40.5
31.4112
0,001
81.5
69.8
80.5
68.4109
37.9153
30.9107
35,9147
31.910.
0.01
81.5
70.5=10
80.6
69.1=04
42.0=a1
32.1
35.9+36
30.4
0.05
82.
71.6=1.
80.9
69.921.
42.2=08
31.4a03
40.8
29.1
VRNL
79.3+0.6
63.8=1.3
49.2=444
47.7+62
36.6=1.1
25,5=a7
26.2+3.
22.5
RENT
79.8
66.8
82.4
78.4:03
36.1
24.0
34.4
27.2
True T
0.0001
79.1
66.9 66.9_07
82.0
77.2105
33.9122
23.1112
33.8107
25.41112
0.001
79.5
66.6=1.5
81.0=a0
77.2+09
35.4+16
23.3+04
32.5+16
26.3202
0.01
19.1
67.4=09
81.5
77.7212
32.2a27
23.8
33.1+24
25.8119
0.05
80.5
70.2=06
9.8
35.0212
24.9
2.0200
6.6231
Then, the next question arises: would integrating RENT and VRNL enhance model performance?, since both methods can orthogonally applied to the original transition matrix estimation methods. It could show improved performance by increasing the variance of the risk function efficiently, hindering overfitting to noisy labels. It might also lead to worse performance if it prevents training process too much, e.g. when the variance increasing too much.
Table 6 shows the performance comparison with VRNL, RENT and RENT+variance increasing regularizer. Numbers in front of each represents the hyperparameter representing the variance (a in Lin et al. (2022)). We report all results with various hyperparameter values to show its sensitivity.
There cases when the resulting performance is even better than the original both methods, show- ing its possibility of future development. However, should be noted that there is no "good-for-all" hyperparameter, indicating its possible limitation of applying it robustly to diverse settings.
24

---

Published as a conference paper at ICLR 2024
Table 7: Test accuracies CIFARIO and CIFAR100 with various label noise settings for other baselines. Bold is the best accuracy for each setting.
CIFARIO
CIFAR100
Terminology
Base
SN 20%
SN 50%
ASN 20%
ASN 40%
SN 20%
SN 50%
ASN 20%
ASN 40%
Regularization
ELR
75.5ga9
47.7
79.5207
70.8a1a
34.7
18.4213
37.0199
27.7
SNL (ar 0.1)
71.7au3
46.9a03
80.5a11
72.7at2
30.2a15
17.3a83
33.4a11
26.5a00
SCE
79.5
54,8=05
79.5
69.5
34.3
18.3+a5
36,2+1.1
27.6
Robust
APL
79.3
61 .533
76.9
64.1
33.5
18.3
35.9
24.0
LRT
74.9:05
46.Sa11
77.7=19
69.2se.
33.8
20.1
35.9
25.4
Coteaching
78.7=114
76,4=3.1
817==05
73.9
37,8-40
12.5+1.3
39,7+10
26.9=32
Data cleaning
Jocor
83.4
62.9=56
80.1
65,9:34
27.5=249
7.9
34.7a12
26.9
DKNN
55.5a11
30.8
62.3-11
54.1
5.1
2.9.04
5,3
4.3=04
CORES
74.7250
26.3241
71.3225
60.7a54
37.8gn
6.5
37.8
27.2
RENT (DualT)
82.0=na
74.6a04
83.3
80.0
39.8
27.1
39.8
34.0
Performance with other baselines In the main paper, compared our method with Transition matrix based methods demonstrate the improvement effect of RENT with regard the transition matrix utilization. In this section, we compare other baselines for the learning with noisy label task itself for a wider range of comparison. Baselines included in this section are:
ELR Liu et al. (2020) suggests early learning regularization. Based on the finding that simple pat- terns are learned fast, they use the output of a learning classifier in early time iterations to regularize overfitting to noisy labels.
SCE Wang al. 2019) suggests reverse cross entropy as robust loss.
APL et al. 2020) theoretically shows any loss with normalization can be made robust to noisy labels and suggests to use two types of robust loss, active loss and passive loss.
ZRTnneeg et al. (2020) calculate the likelihood ratio between noisy label and the possible pscudo- label, which can be defined as the dimension whose output is the maximum. Based on this criterion, it arbitrary sets threshold and corrects the label into pseudo label or not.
Please refer to Section 4.4 or Appendix F.4 also for SNL Chen et al. (2020). and Section F.7 for sample selection based methods.
Again, RENT shows best or second best performance.
25

---

Published as a conference paper at ICLR 2024
More results real dataset Due to the space constraints, reported the accuracies from only some of the baselines in the main paper. this part, we present results for more baselines. As shown in table 8 RENT consistently outperforms other T utilization (FL and RW). Similar to the results for CIFARIO and CIFARTI 100 presented Table [ in the main paper, the performance gap between RENT and previous T utilization widens the noise ratio increases (please refer to the dataset description section for details on the noise ratio). When estimating I as Cycle, RENT does always exhibit the best performance. We believe this may be due the substantial gap between the estimated per-sample weights during training and the true per-sample weights, hypothesis supported by the consistently poor performance of the RW case. Please also note that although there is the case of PDN CIFAR-1ON Aggre when utilizing 7 as RW yields the best results, the gap between RW and RENT marginal.
Table 8: Whole test accuracies on CIFAR-101 and Clothing MM Bold is the best.
CIFAR-1ON
Clothing
Base
Risk
Aggre
Ranl
Ran2
Ran3
Worse
CE
X
80.8=04
75.6+0.
75.3+04
75.6#06
60.4+0.
66.9
w/ FL
79. 79.6118
76.1
76.4104
76.0102
64.5.110
67.1
Forward
w/ RW
80.7
75.810.3
76.010.5
75.8
63.91077
66.8
w/ RENT
80.8=03
77.720.4
77.5
77.2206
68.0209
68.2206
81.9=02
79.4+0.4
79.3=110
79.4#a0
72.1999
68 18.2+
DualT
RW
81.8=0
79.8+02
79.4=00
79.6+04
71.4+11.
68.5
RENT
82.0=12
80,5
80.4
80.5
73.5
69.9
w/ FL
80.5
76.4
76.2490.5
76.1
60.245.2
66.7.0.3
TV
w/ RW
80.7=04
+5.
75.2
75.4
62.3+29
67.4+05
w/ RENT
81.0=04
77.4
77.8=1.0
76.7+0.
66.9#3.1
68.1
FL
80.9
76.3
75.920.7
75.9206
61.8413
65.02n1
VolMinNet
RW
80.7
76.2.005
75.5
75.5
63.0+32
66.620.1
RENT
81.3
7..6.110
77.7.003
77.2107
66.91005
67.7.03
w/ FL
83.3=02
81.0+0.
81.6:0.7
81.2=04
51.6+10
67.1 67.1
Cycle
w/ RW
81.7=0x
79.1
78.4=04
78.2
66.0+t9
67.3+112
w/ RENT
82.0=os
80.020.
81.0=n08
80.4
70.5+0.
68.0204
W/ FL
79,8106
74.510.4
74.5
14.3
57.5
64.91004
PDN
w/ RW
80.6103
74.910.7
73.9
74.4108
58.7.105
w/ RENT
80.21066
75.2
75.0
75.7
61.61166
67.2
w/ FL
81.5
78. 1801
77.5200
77.8100
65.82110
67.22008
BLTM
w/ RW
54.0.33.9
64.412270
50.9132.9
38.01.324
43.5
67.010.4
w/ RENT
80.8
79.1.00.
78.9
79.61006
69.7.120
10.0
26

---

Published as a conference paper at ICLR 2024
Performance reproduce & comparison with our experiment setting Table [9] shows [1] the reported performance at the original paper, [2] the reproduced per- formance using our code, and [3] the model performance under our experiment settings for each T' estimation methods. Comparing three rows for each 7 estima- tion method, note that (1) we reproduced r the original performance enough and (2) the model performance drops significantly under our experiment setting. We propose again that Forward may not be enough to regularize noisy label memorization.
Table 9: Test accuracies for CIFARIO. [1] means the reported performance at the original paper. [2] means the reproduced performance using our code. [3] is the model performance under our experiment settings (same as the performance reported in the main paper as Forward (FL)). not-reported training failure.
CIFARIO
SN
ASN
T' estimation
[]] ~ [3]
20%
50%
20%
40%
[1]
83.4
87.0+
Forward
[2]
82.6=03
67.4#1.
87.9.02
82.9
[3]
73.82u3
58.8293
79.2a06
74.2aus
[1]
78.4
70,0
DualT
[2]
85.6=02
74.2:0..
87.8
81.9
Please note that there was high variance to the test accuracy with regard to the seed for TV (Zhang et al.) 2021b), so better performances could be reproduced if we
[3]
79.9
71.8+a.
82.9
77.7=06
[1]
82, 6..0
TV
[2]
87. 2005
76.6
80.6=74
75.0=13.
[3]
74.0gus
50.4
78.1
71.6203
have explored more times (Currently, we
[1]
89.6=:3
83.4+a.
VolMinNet
[2]
91.4=:0
79.2#a.
94.9 94.9
88.0s45
reported best results over 10 times for [2]
[3]
74.1=02
46.1
78.8
69.5+a3
considering TV). Also since there is no of-
[1]
90.4:02
90.62:0.0
87.3
ficial code for Cycle (Cheng al. 2022).
Cycle
[2]
90.4:04
86.5
66.4204
we reproduced it only with the paper and if some settings are not reported in the paper (e.g. augmentation), we followed settings from Li et al. (2021). Also note that the resnet structure used in TV (Zhang et al. 202 and VolMinNet (Li al.) 202T) did not include maxpooling layer unlike the standard structure (He itt al., 2016), and it made the difference in model performance. Although there are cases when ex- cluding the maxpooling layer increased the test accuracy up to 5 %. report the test accuracy with the maxpooling layer following Yao et al. (2020) and Yao et al. (2021) to show the performance utilizing the original resnet structure.
[3]
81. 31.62u5
82.8
54.3
RENT utilizes T better robustly over Experimental Settings We conducted total of 180 ex- periments under various settings and this number from the multiplication below settings.
T estimation (6): Forward, DualT, TV, VolMinNet, Cycle, True T
Optimizer (2): SGD, Adam
Network Architecture (3): ResNet 18, 34, 50
Seed (5)
Table 10: Beating number (Total 180 times) and average gap (%) of RENT over Forward loss (FL).
Perf. gap is the abbreviation of the performance gap.
CIFAR10
CIFAR100
SN
ASN
SN
ASN
Metric
20%
50%
20%
40%
20%
50%
20%
40%
Number
162 (90%)
172 (96%)
117 (65%)
148 (82%)
143 (80%)
146 (81%)
110 (61%)
101 (56%)
Perf. gap
3.5%
16.0%
1.0%
3.7%
2.2%
5.4%
1.0%
1.2%
Table 10 demonstrates the general improvement of RENT over Forward loss for T utilization. Note that the average performance gap between RENT and Forward loss is larger when the noise ratio is higher, indicating the increased difficulty of noisy label distribution matching with higher levels of noise. Another interesting point is the superiority of RENT for CIFAR 100 since RENT resamples dataset in batc, meaning it will be harder get samples from all classes when the number of class becomes more. Therefore, the gradient from the resampled samples could be biased to the classes of the selected samples. Nevertheless, it still utilizes T better than Forward (FL.).
27

---

Published as a conference paper at ICLR 2024
F.3 MORE RESULTS FOR THE IMPACT OF TO DWS
We show the additional results including other noisy label setting of CIFARIO and CIFARIOO with colored-lines in each figure reporting all 7' estimation methods experimented. Note that the interval of a value in x-axis are drawn in log-scale, meaning that we assigned more space for smaller a region (Same the figure 3). For experimental details, used the same experimental settings that we explained earlier.
00
53-
43-
DeAIT
33
20
23
10
122
12
1071
129
(a) C10 SN 20%
(b) C10 SN 50%
(c) C10 ASN 20%
(d) C10 ASN 40%
20.
32.5
31.0
25.04
25.0
16-
124
250
17.5
(0) C100 SN 20%
(f) C100 SN 50%
(g) C100 ASN 20%
(h) C100 ASN 40%
Figure 14: Test accuracy with regard to various values for CIFARIO and CIFAR100. Similar to the main paper, Star (+) and cross (x) represents the performances of RENT and RW respectively.
As shown the figure 14 performances tend to be higher with smaller & for most of the cases of CIFAR10. However, there few cases when the smaller may not be the answer considering CIFARION 100, showing higher lines than the star point (+, meaning RENT). Intuitively. it implies that larger variance may not always be the answer and there could exist optimal a for each dataset representing the difficulty and noisy property of the dataset, which could be the direction further research. Yet, please note that still the Stars are higher than the Crosses (x, meaning RW) except only one case (please refer to table [ for their exact values).
28

---

Published as a conference paper at ICLR 2024
F.4 MORE RESULTS FOR NOISE INJECTION IMPACT OF RENT
Noise injection impact comparison over various T estimation. Bold means best accuracy for each data setting and T estimation. CE. RW and RENT is the same as the one in the main paper. w.o./f and w/T' means without T and with respectively.
Table 11:
CIFARIO
CIFAR100
SN
ASN
SN
ASN
Loss
Base
20%
50%
20%
40%
20%
50%
20%
40%
CE (1..))
73.4=04
46.6207
28.4
69.7211
33.7a12
18.5207
36,9111
27.3a04
= 0.1
71.7=00
46.9=03
80.5
72.7=12
30.2
17.3=0.3
33,4
26.5=00
w.o/ T
SNL
0.2
74.2-1.1
45,9=10
79.6=13
74.8:06
35.8+24
25.5=10
36.5+10
30.5
0.3
75.2=11
46.8
78.6211
72.0a15
31.7aM
22.6272
31,7411
27.1
RW (0.0)
74.5=os
62.6210
79.6
73.1
37,2122
23,5
27,2
27.3
0.1
77.8=15
64.8=61
74.0=115
77.5+10
26.1
1.2+0.
29.9
110:86
Forward
0.2
75.9=41
65.9=33
76.7=35
76.3=36
2.0=10
1.1#02
9.1=73
3.1420
a = 0.3
77.5
45.5
76.4
74,5
1.6294
1.0g01
2.3
1.3g94
RENT
78.7.003
69.0
82.0.0
77.8204
38.9
28.9
38.4
M0.4
RW (0.0)
80.6=u6
74.1=07
82.5=202
77.9=04
38.5::1
12,0:1135
38.5
24.0:111
= 0.1
74.7=36
48.2=99
78.4=1.1
72.2:447
18.5
1.9
31.7+41
118
DualT
0.2
66.4-92
31.6250
76.6297
69.4
2.2+13
1.4
7.9-42
3,5422
0.3
59.7am1
26.21108
69.6222
64.2a22
1.5207
1.2
4.9.10
1.4196
RENT
82.0
74.6:04
83.3:0:1
80.0.09
39.8
27.1g19
39.8
34.0:04
RW (0.0)
73.7=u9
48.5=241
77.3=22
70,2:110
32.3:11
17.8:22
32.0::1.
23.2:um9
70.7210
49.0:116
76.2:41
69.8291
1,4:03
1.8110
15.0
5.6139
TV
0.2
57.62165
15.6
38.62224
43,92183
1,4guz
1.1
1.4004
1.7489
w/T
= 0.3
30.3
15.6249
30.5
19.5
1.0g01
1.1
1.2g03
1,2
RENT
78.8.00
62.5-1x
81.024
24.0
34.0
20.0.00
34.0
25.5
RW (0.0)
74.2=us
50.6=64
78.6:tus
70.4+tus
36,9-112
24.44:3
34.94-1
26.5
= 0.1
77.1=20
$6.6220
79.0=23
71.2:23
1.9.07
1.2
2.5=09
2.3+
VolMinNet
0.2
67.7=10
25.8+121
75.3=224
59.9 39.9239
1.2001
1.1sa1
1.2=02
1,2+03
0.3
21.8
15.6.42
39.9
32.8
1,1003
1.2s02
1.2=0.
1.2s02
RENT
79.4
62.6
8008
74.0.04
35.8
29.3.05
36.1
31.0208
RW (0.0)
80.2=02
$7.0=34
78. :2:0
70.6::11
37,8
30.2::n6
38. 1::1.
29.3
0.1
77.9=1a
7L1=25
75.9=22
74.4
7.9=u.
2.5
2.0=13
5.2+33
Cycle
0.2
61. 61.0a158
64.1=56
69.7=29
66.8215
1.7=03
1.3#04
1.9=05
2.7+14
a = D.3
63.0
61.22s4
58.4247
42.7am1
1.2g99
1.4107
1.2
1,4g00
RENT
82.5.02
70.4200
81.5.00
70.2.00
40.7.04
32.4
40.7
32.2
RW (0.0)
76.2=03
58.6=11
35.0
21.82us
21.3::166
21.6:1104
= 0.1
79.9=0a
68.8=06
81.3=00
78.0
30.7
24.1:05
34.5#19
27.1=24
True
0.2
76.5=21
67.3-23
81.7204
77.8att
25.7
12.4
32.5w27
27.2a24
0.3
75.7=11
59.32115
78.821
75.5
14.3
1.5g04
25.9236
23.6239
RENT
79.8=02
66.8206
82,4:04
78.4
36.1
24.0:03
34.4
27.2:06
Table [I shows model performance comparison with regard to CIFARIO and CIFARI00 under various label noise settings. In the table, we report the results under the whole T estimation baselines that we experimented in the [
Interestingly, label perturbation either (1) the naive cross entropy loss (SNL row in the table) or (2) the reweighting loss with true T (True T' row in the table) generally improves the model per- formance compared to the baselines, which refer the model performances achieved when trained solely with the basic loss itself without any random label noise. On the other hand, there are cases where the model performance become worse when the random label noise injection technique is utilized with the reweighting loss from the estimated T' under the same a. Also note that integrating the label perturbation technique directly to various T estimation methods can easily lead to training failure, especially for CIFAR100. These failure could be attributed to both the inaccurate objective function resulting from the inaccurate per-sample weights, which are calculated based on the esti- mated T, and the injected label noise. These could also imply that application of the label noise injection technique to the reweighting loss might be more sensitive to the parameter a compared to its original application, which is the application to the naive cross entropy loss. This sensitivity can be problematic in term of its applicability. In contrast, RENT tends to exhibit better performance than RW, further highlighting its robust adaptability to diverse situations.
29

---

Published as a conference paper at ICLR 2024
Figure shows additional results for CIFARIO over figure Similar to figure RENT consistently outperforms SNL and RW+< also under ASN settings. In the figure, we removed o = 0.0 case, which is RW, for ASN plots because the training failed considering both cases. Also note that RW+ is highly sensitive to the value of a and tuning the parameter is required for adequate model performances.
70
80
70
60
60
50
75
70
50
40
70
40
SNL
30
RW++ RENT
60
65
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
(a) SN 20%
(b) SN 50%
(c) ASN 20%
(d) ASN 40%
Figure 15: Test accuracies various for CIFARIO. Same with figure integration RW and the label noise perturbation technique.
RW+c denotes the
We present the model performances using different figure 4 and figure T5] We utilized the true transition matrix for getting the model performances wanted ensure that T estimation gap does not have impact the
00.000.1.0..,0333.... 0.5, 1.0) for both
in the figures, because performance gap between SNL and others. All other experimental details remain consistent with the settings described the earlier section.
F.5 MORE RESULTS FOR OUTCOME ANALYSES
20000
clsu nobey lid
10000
6000
Surple Count
Sample Conai
7500
Sumple
8000
10000
Noany
Clasia
Noiay
3000
3000
51.2%)
(94.2%)
0.000
0.005
0.3.0
0.00
0.01
0.02
0.000
0.005
0,010
Categurical pazameter
Categocion, parameter
Categorical parameter value
(a) C10 SN 20%
(b) C10 SN 50%
(c) C10 ASN 20%
8000
8000
Sample Count
150000
Sample Count
Simple
8000-
4000
Nevagy
Cleln
Nnlay
Cloun (85, (%)
Notay
Cha
2000-
(5.95()
190.550
3000
33.25))
2000
57,756)
ss
0.000
0.005
0.100
6.00
0.01
0.112
0.000
D0005
0.010
Categurical pazameler value
Calegorics, volue
Calegocioa,
value
(d) C100 SN 20%
(e) C100 SN 50%
( C100 ASN 20%
Figure 16: Histogram of we of RENT on CIFARIO 10 and CIFARI00. Same as figure we utilized Cycle (Cheng et al. 2022) as T estimation method for this plot. Blue and orange represents the number of samples with clean and noisy labels, respectively. Vertical dotted line denotes 1/B. Samples with wn < 1/B would likely be less sampled than i.i.d..
w; value Figure 16 shows the additional results the figure 5 over CIFAR1O and CIFAR100. Percentages reported in both figures calculated the ratio between the number of samples
30

---

Published as a conference paper at ICLR 2024
whose w, is within each range and the total number of samples whose labels data instances (i)) are same with (clean) or different from (noisy) the original class labels (x100%). Similar to figure 5s it consistently shows high ratio of clean samples in oversampling region and high ratio of noisy samples near zero.
With the results reported in Figure 5 and figure 16 W; value be divided easily with threshold, 1/B. However, Table 12 shows it is not true.
Table 12: Test accuracies on CIFARIO and CIFARI00 with various label noise settings comparing
RENT vs. thresholding to w,--radomm sampling. Bold the best accuracy for each setting.
CTFAR10
CIFAR100
Base
Criterion
SN 20%
SN 50%
ASN 20%
ASN 40%
SN 20%
SN 50%
ASN 20%
ASN 40%
Forward
w--andomm w/ RENT
74.0
4
80.0gas
71.3
30.8118
16.1
30.6122
24.610s
78.7=033
69.0=6.1
82.0=a5
77.8:05
38.9
28.9
38.4
30.4+03
w--raoom
74.8206
48.2
80.1
72.4
31.9109
16.5
34.7118
DualT
w/ RENT
82,0.0a
74.6.104
83.3
80.0.1.9
39.8109
27.1
39.8107
34,0104
ro-random w/ RENT
70.7
45.5211
78.7
71.9
28.2128
16.1
28.8120
21.6119
78.8
62,5
81.0
74.0
34,0
20.0
34.0102
25,5
ro-random
71.4204
45.3
76.82229
71.7=117
28. 28.7
17.2
30.4
23 23.9222
VolMinNet
w/ RENT
79.4203
62.6113
80.8
74.0
35.8109
29.3.19
36.1
31.0108
Cycle
w w--rande r--random
70.8+122
44.7=00.
78. 18.7
69.2
31.5108
16.3
33.2+16
24.5+23
RENT
82.5
70.4
81.5
70.2.107
40.7..04
32.4.0.
40.7..07
32.2100
Here, r-ndoo means when we just set a threshold for w and randomly sampling again from the selected samples. This gap happens because during training, especially in the earlier learning iterations, w of clean samples and noisy samples may be more mixed, since the model parameter yet more similar the random assignment. Therefore, sorting with arbitrary threshold may be riskier. However, Dirichlet-based resampling and RENT may be safer to this problem since the sampling probability of the samples below threshold is not This would result in the overall performance differences.
Also, choosing a good threshold may be difficult and some training iteration and noise condition adaptive strategy could be needed. Figure shows this need.
the number of (w) > 1/8)
the number of (w) > 1/8)
20000
15000
clean (High)
20000
noisy (Low)
10000
10000
clean (High)
5000
noisy (Low)
50
100
150
200
50
100 Epoch
150
200
Epoch
(a) SN 20%
(b) SN 50%
Figure 17: The number of samples whose w; value is larger than 1/B over time iterations. We visualize clean labels and noisy labels blue and orange. Dataset is CIFAR10. Note that 40,000 and 25,000 samples are total number of clean labels for 20% and 50%, respectively.
shows the number of samples whose w; value is larger than 1/B over time iterations, with clean labels and noisy labels, respectively, while training the classifier with RENT. Result shows that clean label samples whose w; larger than 1/ not as many as the final iteration (which natural), which underlines the training process adaptive strategy for threshold value.
Confidence of wrong labelled samples We show additional results over figure 6 in figure 18 for CIFAR10 and 100 The model trained with RENT consistently reports lower confidence
31

---

Published as a conference paper at ICLR 2024
20000
4000-
RW RENT
10000
2000
Uncertain
(1)
Certain (4)
Uncertain (1)
Certain (4)
Uncertain (t)
Certain (4)
(a) C10 SN 20%
(b) C10 SN 50%
(c) C10 ASN 20%
20000
5000
5000
10000
Uncertain (1) Certain (L)
Uncertain (t) Certain (4)
Uncertain (f) Cortain (1)
C100 SN 20%
(e) C100 SN 50%
(f) C100 ASN 20%
Figure 18: Training data with incorrect labels divided by the confidence (threshold=0.55) Cycle CCeng et al. 2022) for estimation, on CIFARIO and CIFARIO0.
values for noisy-labelled samples across different datasets compared to RW. This observation again supports less memorization of RENT over RW.
1.0
FINE
0.8
MCD
0.5
RENT
0.6
0.6
0.1
0.4
0.1
0.2
0.2
0.0
0.0
0.0
Precision Recall
F1
Precision Recall
F1
Precision Recall
F1
(a) C10, SN 20%
(b) C10, SN 50%
(c) C10, ASN 20%
1.0
0.8
0.8
0.6
0.6
0.6
0.1
0.4
0.2
0.2
0.0
0.00
0.0
Precision
Recall
F1
Precision Recall
F1
Precision Recall
F1
(d) C100, SN 20%
(e) C100, SN 50%
(I) C100, ASN 20%
Figure 19: Evaluation based on the selected metrics for CIFAR10 and CIFAR100. We adopted Cycle (Cheng et al. 2022) for T estimation.
Resampled dataset quality Here, we show additional results over figure in figure 19 for CI- FAR10 and CIFAR100. We show that RENT consistently surpasses the baselines in F1 score, im- plying the good quality of the resampled dataset. We provide details about the baselines that were used in experiments follows.
MCD Lee et al. (2019) assumes features of samples with noisy label will be far away from that of samples with clean label. In this sense, they assume feature distribution normal distribution and
32

---

Published as a conference paper at ICLR 2024
filter out noisy-labelled data using the Mahalanobis distance the metric to discriminate whether a sample is clean or noisy. Following the original paper, we select to use 50% samples of the data with smallest distances.
FINE Kim et al. (202I) filters out noisy-labelled samples by eigenvector. Following the original paper, we use the hyperparameter of clean probability of GMM as 0.5.
Considering FINE, there are cases when shows unimaginably bad performance. We conjecture these bad performances can be from two factors. First, please note that there differences in experimental settings. In the original paper, FINE utilizes ResNet 34 network, without max pooling layer, with SGD optimizer. We utilize ResNet 34 for CIFAR10 and ResNet 50 for CIFAR100 with Adam optimizer. Considering the dataset condition, we did not normalize input data following Zhang et al. (2021b): Li et al. (202I) but FINE did and it would make differences in performance. Second, since there is no reported result on the original paper when applying the FINE algorithm to the model trained with transition matrix based loss functions, comparing this performance to the performance from the original paper is impossible. With lower recall values, we assume eigen decomposition feature vectors would not discriminate clean samples efficiently.
F.6 ABLATION OVER THE SAMPLING STRATEGIES
Table 13: Ablation over the sampling strategies.
CTFARIO
CIFAR100
Base
Sampling
20%
SN 50%
ASN 20%
SN 20%
SN 50%
ASN 20%
Batch
78. 720
69. 69.0gn1
82.0105
38.9212
28.92111
38.4207
Forward
Global
81.8405
72.6=09
83,2+06
32.3+22
4.4=19
33.3=21
Global.C
81.8105
75.9104
83,5103
37.3108
23.3125
36.7
Batch
82.02a2
74.6204
83.3+00
39.8209
27.1
39.8:107
DualT
Global
81.7106
74.620.
83.0100
39.11114
26.9123
36.4119
Global.C
81.3+07
74.1+07
82.7+00
38.9+1.
27.3+3.
38.7=16
RENT utilizes the resampling from the mini-batch default setting. However, sampling from the mini-batch could also be changed to the dataset-level. As another ablation study on the sampling strategies of RENT, conduct sampling from the whole dataset level, which could be divided into two; I) global sampling from whole dataset, which we denote as Global, and 2) class-wise sampling from whole dataset, which we denote as Global.C. C. Table 13 shows that each strategy shows robust performances over the different experimental settings.
F.7 RENT vs.
Apart from transition matrix based methods, sample selection methods dynamically select subset of training dataset during the classifier training. by filtering out the nncorrectly-labellee instances from the noisy dataset HHn al. (2018): Yu al. (2019); Wei et al. (2020); Bahri al.) (2020); Cheng et al. (2020). Our method, RENT, shares the same property with sample selection methods by recognizing the resampling of RENT as noise-filtering procedure. Therefore, we compare the performances of RENT with Coteaching Han et al. (2018), Jocor Wei et al. (2020), DKNN Bahri ct al. (020)) and CORES Cheng et al. (2000). We report the details of the baselines as follows.
Coteaching Han et al. (2018) assumes samples with large loss assumed to be naisy-labelled. However, selecting samples based on the output of the trained classifier may cause the problem of sampling only already-traing trained samples again. Therefore, it utilizes two networks with same structures from different initialization point. It selects data with small loss from other network and train model parameter with the selected data.
Jocor Wei et al. (200)) also considers the loss value as a metric to discriminate labellla data and they utilize networks with same structures, but they optimize two network one time.
DKNN Bahri et al. 2020)) considers data whose labels are different from the K-Nearest Neighbor algorithm result as noisy-labelled data. We utilize k 500 reported in the original paper.
33

---

Published as a conference paper at ICLR 2024
CORES Cheng et al. (2020) select samples based its output confidence with regularization term making the model more confident, since a classifier can be uncertain for clean labels when training with noisy-labelled dataset. We set /3 2 following the original paper.
For training Coteaching and Jocor, noisy label ratio is required as input information to decide the followed the same
sample selection portion. However, since we do not know this information, way as DualT Yao et al. (2020) for noisy ratio estimation, i.e. get the average of the diagonal term of I-noisy ratio. For T estimation, we utilize the algorithm of Yao et al. (2020).
Table 14: Comparison with Sample selection methods
Clothing M. We did not report results
DKNN because of too much computation.
Methods
Cotcaching
Jocor
CORES
DKNN
RENT
Accuracy (%)
67.2
68.4
66.4_0.7
69.9
Table and table 14 compares test accuracies over the baselines and RENT. Our method shows competitive performances the baselines. It should be noted that Coteaching and Jocor requires the training of two different classifiers, which requires more memory space and computation than other methods.
F.8 WHEN NOISY DATASET CLASS IMBALANCED
Solving class imbalanced dataset with noisy label can be important (Koziarski et al. 2020 Chen jet al. 2021, Huang et al. 2022). Therefore, we experimented to show which utilization would work well under class im imbalance. Following studies that solves class imbalanced learning Cao ct al. (2019); Zhang et al. (003).. we first make the imbalanced dataset with (the maximum number of samples in class) (the minimum number of samples in class) is defined as imbalance ratio and between two classes the number samples in each class should increase in exponential order. Then, flipped the label to be noisy we did previously.
Figure 20 shows the performance comparison of FL., FW and RENT with several T estimation see in the figure, again, RENT shows consistently good result over FL and RW. However, since there is no treatment to solve class imbalance issue in all of FL, RW and RENT, the model performances decrease with higher class imbalance ration. This direction could be an interesting future study.
baselines. As
34

---

Published as a conference paper at ICLR 2024
Forward
11.
15.0
11.3
615
9A4
628
654
180 21
40.
min
02.5
075
DualT
750
134
11.
112
152
623
66.
V7t
TV
VolMinNet
004
17.
15.0
125
154
424
654
63.8
Cycle
(a) SN 20%
(b) SN 50%
(c) ASN 20%
(d) ASN 40%
Figure 20: Performance comparison of FL, RW and RENT over class imbalance ratio. Subcaptions represent noisy label setting and vertical captions mean the transition matrix estimation methods. RED, GREEN and BLUE represents RENT, RW and FL, respectively.
F.9 TIME COMPLEXITY
Table 15: Iterations denotes the number of iterations per epoch.
Dataset
CIFARIO
CIFARI00
Clothing
Dataset size / Iterations
50,000 / 391
50,000 / 391
1,000,000 / 10,000
froal
15.62s
20.44s
2315.10s
Sample
0.13s (0.83%)
0.12s (0.59%)
4.02s (0.17%)
To check whether there is increment in the time complexity by RENT, Table 15 presents the wall- clock time resampling procedure of RENT. footl is the total wall-clock for single epoch, and tsample is the time only for the resampling. Given a large-scale dataset, i.c. Clothing the resam- pling time becomes ignorable.
35


Reviews:
[{"review_id": "aoIrZlMeWW", "reviewer": "Reviewer_hEe7", "summary": "The paper addresses the crucial issue of learning from noisy labels, emphasizing the significance of the transition matrix in modeling the relationship between noisy and true labels. Rather than concentrating solely on learning this matrix, the authors propose a new approach that leverages it, drawing inspiration from reweighting and resampling concepts. They introduce a Dirichlet-based weighting method, which assigns individual weights to each sample drawn from a Dirichlet distribution. This distribution is parameterized using a base measure informed by the transition matrix. Empirical results indicate that this weighting method outperforms forward risk minimization and direct reweighing techniques.", "questions": "* In table 2, there are a couple of settings, where RENT does not perform as well as the other two methods, can the authors discuss the underlying reasons?\n* The author said that RENT performs better when estimated T differs from True T, the reviewer wondered if one can plot the performance gap against the differences between the estimated T and True T.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality: The concept of imposing a Dirichlet prior on individual sample weights is well-founded. This approach naturally leverages the base measure to integrate the transition matrix, creating an informative prior. Additionally, the ability to control the concentration parameter $\\alpha$ enables the fine-tuning of weight properties, including variance-based regularization and addressing noise-related issues. The practical implementation of this method is straightforward, as demonstrated in Algorithm 1.\n\nSignificance: The significance of this work is underscored by the comparison of the proposed reweighing method with forward risk minimization and an existing reweighting method based on the likelihood ratio. This comparative analysis is conducted across four datasets, and the results demonstrate the promise and potential impact of the proposed approach.\n\nClarity: Overall, the paper effectively conveys its central idea. However, there is room for improvement in terms of notation clarity.", "weaknesses": "The comparative analysis of RENT is limited to FL and RW, utilizing transition matrices obtained through various methods. A more comprehensive evaluation, including a broader range of noise-label learning techniques, such as the approach proposed by Lin et al. in 2022, would provide a more comprehensive assessment of RENT's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "htsFVl5W6A", "reviewer": "Reviewer_Epey", "summary": "The paper first extends the reweighting strategy used in noisy-label learning, with a Dirichlet-based framework. This framework encompasses both reweighting and resampling as the two extremes of the Dirichlet distribution. The paper provides analysis of the impact of the shape parameter to the empirical risk and discuss resampling is better than reweighting. The paper finally proposes a method called RENT (resampling utilizing the noise transition matrix). Various experiments show the superiority and characteristics of the proposed method.", "questions": "I already wrote some of my questions in the previous section. Some other minor comments/questions:\n\n- The 'et al.' in the references could be written out in full.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed algorithm is simple.\n- Discussions about the relationship between related work is explained in detail.\n- Experimental results are encouraging: we can see the benefit of introducing RENT.", "weaknesses": "- While reviewing the paper, I encountered some challenges in comprehending the content, primarily due to the clarity and organization of the presentation and missing notations/definitions. Some examples:\n\t- An explanation of $M$ seemed to be missing in page 4 but appears later on in page 6 (explained as a resampling budget.)\n\t- Would it be better to have Equation 3 in page 4 right after the definition of $R_{\\ell, RW}$ in page 3?\n\t- I wasn't sure how $x_1, \\ldots, x_M$ are determined in Algorithm 1 page 6. Since $\\pi_N$ is a categorical distribution (and not a joint distribution of instances and labels), it seems to me that we can only sample $\\tilde{y}_1, \\ldots, \\tilde{y}_M$ (without $x_1, \\ldots x_M$)?\n- For Proposition 3.1, it might be more clear to present the assumptions directly within the proposition rather than explaining them in the appendix.\n- Is Eq. 7 correct? Should we introduce a different notation for the quantity in Eq.7 instead of writing that this is equivalent with $R_{l, DWS}^{emp}$, since we are applying the CLT in the proof in Appendix C.1 and $N$ is finite?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mJBhTIdt93", "reviewer": "Reviewer_z7WX", "summary": "This paper studies the problem of improving the uilization of transition matrix in label noise learning, authors propose that due to the poor estimation of class posterior, existing approaches such as loss correction or re-weighting might easily fail. To counter this issue, the authors proposed REsampling method to utilize the Noise Transition matrix (RENT), which utilizes Dirichlet distribution based resampling to assign instance-dependent weights.", "questions": "1. It is well-known that instance-dependent transition matrix might exhibit high complexity when the class number increases, for instance, for cifar-100, we need to compute a tensor of size $50000 \\times 100 \\times 100$, which is prohibited in real-world applications, can DWS mitigate this issue? More discussions in this aspect might bring more contribution.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposition of improving the utilization of transition matrix is interesting, and shows good insight - due to factors such as complexity, estimation error in noise class posterior and so on, even with a perfectly estimated transition matrix, it might still exhibts subpar performances. \n\n2. The idea of re-sampling instead of re-weighting for loss correction is straight forward and intuitive.", "weaknesses": "**Major issues:**\n\n1. Some experimental results are inconsistent with prior works, why are the performances of learning with true $T$ worse than Cycle consistency and Dual-$T$?\n\n2. The discussions in section B seems problematic, in $\\sum_{j=1}^{C}\\(max_{j} T_{\\hat{y}_{i}j} \\)$, are we trying to sum over all $j$, or finding the maximum $j$? It seems that you're trying to do both, can you instead give an intuitive explaination and refine your mathmatical statements?\n\n3. The motivation of DWS is not strong enough, if we are going with the assumption that the classifier trained with noisy labels can not accurately estimate noise class posterior (poorly calibrated, high estimation error, etc.), then we can hardly assume that the transition matrix is accurate, as the transition matrix is usually estimated from the noise class posterior.\n\n**Minor issues:**\n\n1. More recent and SOTA $T$ estimation method is not included [1].\n\n[1] Yong, L. I. N., et al. \"A Holistic View of Label Noise Transition Matrix in Deep Learning and Beyond.\" The Eleventh International Conference on Learning Representations. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hdpqzLFajs", "reviewer": "Reviewer_HW3A", "summary": "The paper proposes a resampling based training method for noisy label learning. The paper builds upon the resampling framework which has been proven to be superior to the reweighting method.  The proposed method utilizes Dirichlet distribution-based per-sample weight sampling to re-formalize reweighting and resampling, aiming to improve the utilization of the transition matrix. The experiments confirm the advantage of the resampling method.", "questions": "1. What can we get from the theoretical analysis other than resampling being better than reweighting when $\\mu^*$ and $\\mu$ are different, which is a conclusion from previous work (An et al. 2020)?\n2. Following the last point, how different is the Dirichlet-based resampling from just setting a threshold for $\\mu$ and randomly sampling?\n3. Will the method work better if combined with data cleaning or abstention kind of ideas? There is one reference that I found ([1] Koziarski, Michał, Michał Woźniak, and Bartosz Krawczyk. \"Combined cleaning and resampling algorithm for multi-class imbalanced data with label noise.\" Knowledge-Based Systems 204 (2020): 106223.). Although I'm not familiar with this work, I think it might be interesting to the authors and other reviewers.\n4. Can the proposed method be compared with other noisy label learning methods (regularization, loss correction, or data cleaning)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well organized and clearly written. \nThe paper has a good angle that focuses on utilizing the transition matrix instead of estimating the transition matrix. \nThe paper formalizes the reweighting and resampling methods and provides a good analysis of both methods. \nThe paper provides comprehensive evaluation results regarding the reweighting and resampling methods.", "weaknesses": "1. The paper does not propose a \"new\" method. Reweighting and resampling are both existing methods, and may not be considered novel contributions.\n2. Although the Dirichlet based analysis provides some good angles, the theoretical results do not have a clear insight. The distance from the true weight part simply says that the distance becomes smaller as $\\alpha\\rightarrow0$, but this does not tell us much because $\\alpha\\rightarrow0$ just gives a weight assignment closer to one-hot as demonstrated. \n3. The paper lacks comparison with other means to deal with noisy label learning.\nWith these concerns, I have some doubt about the overall contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vGgpz8utjl", "reviewer": "Reviewer_KTbE", "summary": "This paper primarily explores the use of transition matrices in the context of noisy label learning. The authors conduct a meticulous analysis to expound upon the challenges that arise during the practical application of extant methods employing transition matrices. In response to these challenges, the authors introduce the Dirichlet-based per-sample Weight Sampling (DWS) framework. This framework effectively unifies two distinct methodologies, sample reweighting and resampling, facilitating their comparison within a comprehensive framework. Furthermore, the authors undertake an exhaustive analysis, establishing the resampling technique as supremely effective in addressing issues pertinent to noisy label learning.", "questions": "Why use the Dirichlet distribution to sample the weight?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The background and the motivation of the setting is well-introduced. The motivation of the work is reasonable.\n\n2. The authors integrates sample reweighting and resampling methods into a single framework for comparative analysis, highlighting the superiority of the resampling approach.\n\n3. The method is simple and the results on several datasets seem good.", "weaknesses": "1. I acknowledge the authors' theoretical contributions, but it must be said that the method proposed by the author lacks innovation. The approach in this paper bears similarity in concept to a significant category of noisy label learning methods based on sample selection, and their success strongly underscores the superiority of resampling methods.\n\n2. On CIFAR-10N dataset, RENT performs well on rand1-3 and worse scenarios but exhibits poor performance on aggre. In the context of crowdsourcing, aggre should be a more common noise setting. Therefore, it is essential to analyze the reasons for the subpar performance in this particular scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZtJe11CxhX", "reviewer": "Reviewer_4M7K", "summary": "Different from previous researches that focus more on how to estimate the transition matrix, which is significant for the risk estimator to achieve the statistical consistency, this paper proposes a new utilization of the transition matrix to deal with noisy labels.", "questions": "1.As it is mentioned above, why $\\tilde{\\mu}$ and the parameters of Categorical Distribution are calculated in the way illustrated in Algorithm 1?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.This paper proposes a new approach, named RENT, a resampling method with Noise Transition Matrix.\n\n2.The proposed approach origins from the analysis of comparing reweighting and resampling under Dirichlet distribution based per-sample Weight Sampling (DWS) framework, which is reasonable.\n\n3.Experimental results show the effectiveness of the approach.", "weaknesses": "1.Since the final algorithm chooses resampling, my main concern is the advantage of resampling comparing to reweighting, which should be given more details about and hightlighted in some sections like Introduction. \n\n2.Figure 1 should be polished up to clearly illustrate how to implement the proposed framework. Otherwise, it is difficult to understand the 4-th and 5-th paragraphs of Section Introduction.\n\n3.It is suggested to summarize the contributions at the end of Section Introduction, such as extension of DWS, the analysis of reweighting and resampling, the new resampling method.\n\n4.Some details about the implementation of the resampling should be given in the main body of the paper. For example, the reason why $\\tilde{\\mu}$ and the parameters of Categorical Distribution are calculated in the way illustrated in Algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)