Which is Better for Learning with Noisy Labels: The Semi-supervised Method or Modeling Label Noise?
Authors: Yu Yao
Mingming Gong
Yuxuan Du
Jun Yu
Bo Han
Kun Zhang
Tongliang Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: In real life, accurately annotating large-scale datasets is sometimes difficult. Datasets used for training deep learning models are likely to contain label noise. To make use of the dataset containing label noise, two typical methods have been proposed. One is to employ the semi-supervised method by exploiting labeled \textit{confident examples} and unlabeled \textit{non-confident examples}. The other one is to \textit{model label noise} and design \textit{statistically consistent} classifiers. A natural question remains unsolved: which one should be used for a specific real-world application? In this paper, we answer the question from the perspective of \textit{causal data generative process}. Specifically, the semi-supervised method depends heavily on the data generation process while the modeling label noise method is independent of the generation process. For example, for a given dataset, if it has a causal generative structure that the features cause the label, the semi-supervised method would not be helpful. When the causal structure is unknown, we provide an intuitive method to discover the causal structure for a given dataset containing label noise.

WHICH IS BETTER FOR LEARNING WITH NOISY LA- BELS: THE SEMI-SUPERVISED METHOD OR MODELING LABEL NOISE?
Anonymous authors Paper under double-blind review
ABSTRACT
In real life, accurately annotating large-scale datasets is sometimes difficult. Datasets used for training deep learning models are likely to contain label noise. To make use the dataset containing label noise, two typical methods have been proposed. One is employ the semi-supervised method by exploiting labeled confident examples and unlabeled unconfident examples. The other one is to model label noise and design statistically consistent classifiers. A natural question mains unsolved: which one should be used for a specific real-world application? this paper, we
the question from the perspective of causal data generation process. Specifically, the semi-supervised method depends heavily on the data generation process while the modeling label-noise method independent of the generation process. For example, for given dataset, if it has a causal generative structure that the features cause the label, the semi-supervised method would not be helpful. When the causal structure is unknown, we provide an intuitive method to discover the causal structure for a given dataset containing label noise.
INTRODUCTION
Deep neural networks can achieve remarkable performance when accurately annotated large-scale training datasets are available. However, annotating a large number examples accurately is often expensive and sometimes infeasible in real life. Cheap datasets which contain label errors casy to obtain (Li et al. 2019) and have been widely used to train deep neural networks. Recent results (Han) et al. 2018 Nguyen et al., 2019) show that deep neural networks easily memorize label noise during the training, which leads to poor test performance.
To reduce the side effect of label noise, there two major streams of methods. One stream of methods focus on getting rid of label errors. Specifically, they would first select confudent examples (i.e., whose labels are likely to be correct), e.g., by exploiting the memorization effect of deep networks (Jiang et al., 2018). Then, by discarding the labels of unconfident examples (i.c., whose labels are likely to be inccorect) and keeping their unlabeled instances, they (Li al. 2019 2020
Wei et al., 2020 Yao et al., 2021 Tan et al. 2021: Ciortan et al., 2021 Yao et al. (021) would employ the supervised method, e.g., mixmatch O. achieve state-of- the performance. Those methods are usually based on heuristics and lack theoretical guarantee.
Another major stream is model the label noise and then get rid of side effects. They mainly focus on estimating the label noise transition matrix T(x), i.e., T., (ax) = P(ý i|Y = = x) representing the probability that an instance with a clean label Y = i but flips to noisy label Y = j. The idea is that the clean class posterior distribution P(Y can be inferred by learning the transition matrix T(x) and noisy class posterior distribution PX) In general, when T(x) well estimated (or given), these methods are statistically consistent, i.c., they guarantce that the classifiers leamned from the noisy data converge to the optimal classifiers defined on the clean data as
the size of the noisy training data increases (Patrini et al.) 2017 Xia et al. 2019).
It naturally raises the question that which stream of methods should be used for specific real- worldworld application? Answering the question is crucial for the community learning with noisy labels. the answer is that stream of methods is dominating, future efforts should mainly focus on that

---

specific stream. However, if not, should know their differences and which method should be used for a specific real-world application.
In this paper, from a causal perspective, answer that none of the two streams of methods dominating. They have advantages and disadvantages, which are closely related to the underlying data generation process. The supeviupe methods easily incorporate heuristics (e.g., prior knowledge) to make use of the finite training sample but they do not work if the feature is the cause of the label in the data generation process. The modeling label-noise methods are not influenced by the data generation process. They can make use of all the instances and noisy labels and can be statistically consistent but they need large training sample to perform well.
Specifically, when the instance X is a cause of the clean label Y, the distributions P(X) and P(Y are disentangled (Schölkopf et al. 2012 Zhang al., 2015). which means that P(X) contains no labeling information. In other words, exploiting the unlabeled data by semi-supervised methods cannot help learn the classifier. When the clean label Y a cause of the instance X, the distributions of P(X) and P(Y|X) are entangled (Schölkopf et al. 2012 Zhang et al. (015)) then P(X) generally contains some information about P.Y[X). Then the semi- supervsse methods are helpful. In many real-world applications, we do not know the causal structure of the data generation process. To detect that on a specific noisy dataset, we proposed an intuitive method by exploiting an asymmetric property of the two different causal structures (X causes Y vs Y causes X) regarding estimating the transition matrix.
2 RELATED WORK
In this section, we first introduce the two major streams, i.e., the methods employing semi-supervised learning and the methods based on modeling label noise. Then we introduce the causal generation process of the noisy data.
Method based on semi-supervised learning. Semi-supervised learning is widely employed in learning with noisy labels. To getting rid of label errors, existing methods usually divide the dataset to confident examples and unconfident examples. Then the deep neural networks are trained on the confident examples in supervised manner (Jiang al.) 2018 (Han et al. 2018). To also make use of the unconfident examples that contain a large amount of incorrect labels, by just employing the unlabeled instances, different semi-supervised learning techniques can be employed. For example, the consistency regularization (Laine and Aila 2016 is employed by (Englesson and Azizpour 202T); FixMatch (Sohn et al.) (2020) is employed by (Li et al. 2019); the co- co-Regularization is employed by (Wei et al.] (2020); contrastive learning is employed by (Tan et al. [202]]
(022:Ciiortan Ciortan al. 202 Li
ct al. 2020 Ghosh and Lan [021 Yao et al. 2021 Zhelttoozhsski et al. 2022). Empirically, these
methods have demonstrated state-of-the-un performance.
Method based on modeling label noise. This family of methods mainly focuses on designing
statistically consistent methods by employing the noise transition matrix T(x).. Specifically, given an instance æ, its transition matrix (xx) reveals the transition relationship from clean labels to noisy labels of the instance., i.e.,
..... [Px) 1|x), P(Y L|z)] [P(Ý = ilx L[z)]
(1)
Let h: : + Ac-1 models class posterior distribution and tce be the cross-entropy loss, then arg min Ez,yleee hi ((x))) = arg min h
( T(x))(x))).
h
The above equation shows that if T(x) is given, the minimizer of the corrected loss under the noisy distribution the same as the minimizer of the original loss under the clean distribution (Liu and
Tao 2016 Patrini et al. (017). In practice, T(x:) usually is not given and needs to be estimated from noisy data (Xia et al. 2020 Li et al. 2021).
It is also worth mentioning that, methods focus design robust loss functions can closely related to modeling label-noise methods. These methods usually require the noise rate to help hyperparameter selection (Zhang and Sabuncu]
2018 Liu and Guo 2020 To calculate the noise rate, T(z) usually
have to be estimated (Yao et al. 2020).
2

---

causal generation process of noisy data. We in- troduce some background knowledge about causality and describe the data generation process by the causal graph and the structural causal model (SCM) (Spirtes) and Zhang] 2016). Specifically, in Fig. T(a), lustrate a possible data generation process when data
X
y
X
y
il-
contains instance-dependent label noise by using the causal graph which represents flow of information and reveals causal relationships among all the variables (Glymour et al., 2019). For example, Fig. [(a) shows that the latent clean label X, and both X and Y are causes of Y. The generation process can also be described by a structural causal model (SCM). Specifically,
(a) Y causes X
(b) X causes Y
Figure 1: The change of P(X) with the change P(Y) under different data gener- ation processes.
cause
instance
Y~ Py, Ux Pix, I(,UX, ~
Pug, Y
yxg(XXY,Up))
where Ux and Uy are mutually independent exogenous random variables that are also independent of Y. The occurrence the exogenous variables model the random sampling process of X and Y. and g can be linear non-linear functions. Each equation species a distribution of variable conditioned on its parents (could be an empty set). Similarly, the SCM corresponding to the causal graph in Fig. [(b) can be written as:
X Pxpx Uy ~ Pūy, Pups f(X,U),,, y YX,
causal factorization and modularity. By the conditional independence relations proposed by the Markov property (Pearl 2000), the joint distribution P(X,Y,Y, when Y causes can be factorized by following the causal direction as follows.
P(X,Y,8) PV)P
The above decomposition is called causal decomposition. According to the modularity property of causal mechanisms (Schölkopf et al., 2012] Peters et al.] (017)). the conditional distribution of each variable given its causes (which could be empty set) does not inform influence the other conditional distributions, which implies that all the distributions P(Y), P(X|Y) and P(Ý Y) disentangled. Similarly, when X causes Y, the causal decomposition of P(X, Y, Y) is as follows:
P(X,Y = -(X))XXXXX
3 LEARNING WITH NOISY LABELS FROM A CAUSAL PERSPECTIVE
In this section, we show that the modeling label-noise method is independent of different generation processes while the semi-super methods depends on different generation processes. We also proposed an intuitive method to detect the causal structure by exploiting asymmetric property regarding estimating the transition matrix.
3.1 THE INFLUENCE OF NOISY DATA GENERATION PROCESSES TO DIFFERENT METHODS
The modeling label-noise method is independent of different generation processes. The reason is that these methods mainly rely on estimating the transition matrix T(x), which can be estimated by exploiting the noisy class posterior P learned on the noisy data (Xia et al. 2020 Li et al.
2021 It is clear that the data generation process does not influence learning P(Y and T(x).
By contrast, the semi-supervised methods are influenced by data generation processes because they rely on exploiting the unlabeled data help learn the classifier. The helpfulness of unlabeled data depends on whether P(X) contains labeling information or not. According to the causal modularity property, when X causes Y. P(X) does not contain labeling information, because P(Y|X) and P(X) are disentangled with each other. However, when Y causes X, P(X) should contain labeling information, because P(X) and P(Y entangled with each other.
3

---

PXXY =
9
P(XIV - 3)
PEXIE
0
P(X = 2)
POUY A2)
WPP 0) P. D.5. PCY 2) 0,
(M -0
(PP 0. -005,
(PP -PP D-0 P
PCY =)==
PDY = 2)= 03 2( 33=
Figure 2: (a)-(d) illustrate the influence to P(X) when P(Y) changes under different data generative processes. When Y causes X, as illustrated in (a) and (b), changing P(Y) to P'(Y) influences P then P contains labeling information; when X causes Y. as illustrated in (c) and (d), changing P(Y) to PYY does not influence P(X), then P(X) does not contain labeling information.
To clearly illustrate the entanglement, we will derive that, when causes X, X) and P(X) will change simultaneously to P' PY(I)) and P((X) if we intervene on Y, i.e., change P(Y) to different distribution P'(Y).
Specifically, when P(Y) is changed to P'YY), P(X will not be influenced because of the modu- larity property (Pearl 2000). Since P(Y) is changed PY(Y), and P(X remains fixed, after the intervention, the joint distribution P(X, Y) P(Y) YY) will be changed to new joint distri- bution P"(X,Y) Y) = P(XY))(X(Y). Then P(X) will be changed to P'(X)= = P(X))
By applying Bayes' rule, P(Y = P(Y))(XX
will change to a different distribution P(Y) (XX) which is a special case.
unless P (Y)) Therefore, X(Y and P(X) generally are entangled when Y causes X.
P'(Y = PXXY)((XYY/P(CC)
To provide more intuition, we illustrate a toy example Fig. 2 For example, as illustrated in Fig. 2(a)
when P(Y = 0) -P(Y = 1) 0.5, P(Y = 2) P(Y = 3) = 0 the data is drawn from either P(X = 0) or P(XiY = 1), then P(X) = 0.5P(X = 0) + 0.5P(X = 1). However,
the class prior is changed to P' = 0) P'(Y = 1) PY = = P'(Y = 3) 0.5, as illustrated in Fig. 2(b) instcad of drawing data belonging to Y 0 and = 1, the data belonging will be drawn, and the data distribution becomes P'(X) 0.5P(X
to Y and Y
2) + O = 3). Meanwhile, the change in P(Y) also leads to change in P(Y The changes of P(X) and P(Y]X) both come from changes of P(Y), indicating that P(X) contains information about P(Y]X). Therefore the semi-supervised based methods can be useful in this case.
When feature X is cause of Y, intervention on P(Y) will change the function f' or the distribution of Uy but leave P(X) unchanged. For example, from Fig. 2(c) Fig. 2(d) the function f' will be changed output Y or Y = 1 instead of Y = 2 or Y 3 to account for the label distribution change. The change of the selected label sets will only change the classification rules (tasks). It is clear that relabeling the sampled data points with different labels according to the new rules will not influence the distribution of the sampled data points P(X), and P(X) is disentangled with the different label sets. Then P(X) generally does not contain information to learn clean label Y. Therefore the semi-supervised based methods may not work well in this case.
3.2 AN INTUITIVE METHOD FOR THE CAUSAL STRUCTURE DETECTION
In many real-world applications, the causal structure of the noisy data generation process is unknown. To discover the causal structure, we provide an intuitive casual structure detection method for learning with noisy labels (i.e., CDNL estimator). Our method relies flip rates under different generalization processes. Specifically, when X causes Y, the flip rate P(Y Y') estimated by an unsupervised classification method usually has a large estimation error, where Y' is pscudo labels estimated by the unsupervised method. However, when Y causes X, the estimation error is small.
asymmetric property of estimating
To be more specific, the flip rate P(Ý can be obtained by letting clustering method estimate pseudo labels Y on all training instances. Given pseudo labels and noise labels, P(ÝIY") then can
4

---

Algorithm 1 CDNL Estimator
Input:
noisy training sample Sw: a noisy validation sample Svat; a cluster algorithm 2; classification model h; trainable stochastic matrix A
1: Optimize and A via Eq. ( to obtain A = P(Y by employing the training set S and the validation set Svat:
2: Employ the cluster algorithm 2 to estimate the cluster IDs of all instances training set Su:
3: Obtain Y of all instances from cluster IDs;
4: Calculate PYY by Eq (6)
Output: The estimation di (( via Eq. (3)
also be estimated. On the causal dataset (X causes Y), P(X) does not contain labeling information, then y' should be very different from clean label Y. Therefore, the estimation error of P(Y|Y') is large. On the anticausal dataset (Y causes X), P(X) contains labeling information, the Y' should be "close" to clean label Y. Therefore the estimation error of P(Ý is small.
It is worth mentioning that the performance of the proposed CDLN estimator relies on the backbone unsupervised classification method. When Y causes X, the backbone method expected to have reasonable classification accuracy
training instances. Thanks to the great success of the unsuper- vised learning methods (Likas et al. 2003 Niu et al. [022]] Ghosh and Lan 2021 Zhou et al. [201]) } some of these methods can even have compatible performance with the supervised learning on some benchmark datasets such as STL10 Coates et al. (2011) and CIFARIO 10 Krizhevsky et al. (2009).
Let Y arg max; P(Y i|æ)) be the Bayes label on the clean class-posterie distribution. To
obtain the estimation error, we calculate the average difference between the noise rate estimated by the method based on modeling label noise and the noise rate estimated by clustering algorithm, i.c.
P(( j|Y P(Ý = j|Y i)
d(P())r")
ΣΣ
(3)
The intuition that given a noisy dataset, suppose that Bayes labels and pseudo labels of all instances are known and fixed, then the P(Ý and P(ÝlY')) are different in general unless Y and Y' are identical each other, which illustrated in the following theorem.
Theorem Let P(Y*|Y') be the transition relationship from the pseudo label Y to the Bayes label Y*, then d(P(())"), P() = either 1). for all E L, P(Y = ilYY = i) = or 2). for all i,j E L such that
(1 PPY" = =|lY = Pi) = j|Y =) P =)=== - =k), (4)
lepfi
where
= = =1
The above theorem shows to let d(P(Ý)Y")) )(P))) 2) has to be satisfied. To satisfy the condition 1). given all examples with Y' = i, their Bayes labels have also be i, it implies that when two variables Y and Y' are identical each other. In this case, y must cause X, because P(X) contains labeling information, i.e., by exploiting P(X), Y = Y" can be learned, The condition 2) is a special case that requires all entries in P(P and P(Y*[Y") carefully designed to make Eq. (4) holds, which can be hard satisfy in general.
either the condition 1) or the condition
Estimation of P(ÝIY"). To estimate the flip rate P(ÝIY"", clustering method is employed first to learn the clusters C. Then the clusters C' can be converted into the pseudo label Y" by exploiting the estimated Bayes label 8 and the average noise rate P P(PIY') obtained by clustering method can be directly calculated. Be more specific, let C = i denote the cluster label i, and let Sc. {
Ne:
Ng.
denote the instance with cluster label i. Similarly let Sy. = {xx) denote the instance with estimated Bayes label j by employing label-noise learning methods (Patrini al. 2017). We assign
5

---

the pseudo labels Y of all instances in set Sc; be the dominated estimated Bayes label Y i.e.
ExeEs (2LESc,)
Y
arg max JEL
(5)
Nc.
Empirically, the assignment is implemented by applying Hungarian algorithm (Jonker and Volgenant 1986). After the assignment, the pseudo labels of all training examples can be obtained. Then P(Y (") can be estimated via counting on training examples, i.c.
P( j| = = (ino..)) (8'
(6)
E(e,,,0)) 1l (8''11)
where 1. is an indicator function, (x, "''' is an training example with the estimated pseudo label, and A represents the AND operation.
Estimation of YY)) We estimate the average flip rate P( in an end-to-end manner. Specifically, let f be deep classification model that oulputs the estimated Bayes label in a one-hot fashion. Empirically, it can be achieved by employing Gumbel-Softmax (Jang et al. 2016). The distribution P(* is modeled by a trainable diagonally dominant column stochastic matrix A. Then, similar the state-of-the-an method (Li et al. 2021), A can be estimated by minimizing the empirical loss on noisy data, i.c.,
1 Σ Lee Ah(x)), s.t. max (x)x)) = 1. A.f
(A', arg min
In Section 4.1.1 we show that the estimation error of Y*) by employing our method above much smaller than employing a state-of-the oh-the-rrttttt method VolMinNet (Li al.) (222)) for both instance-dependent and Y(" mainly comes from two perspectives:
label noise. The advantage of our method to estimate
1). To estimate P(Ý by employing existing methods, the noise transition matrix P( Y. X) has to be learned in advance, but PY Y X) is hard to estimate in practice (Li et al. 2019 Yao et al. 2020). Our method avoids learning P(YJY, but directly estimates the average flip rate P(Y Y*)
Specifically, estimate P(Ý|"*) with existing methods, P(Ý X)) and P(Y|Y, X)) have to be learned first. Then both the estimated clean label Y and the Bayes label Y can be revealed by (2).
After that, P() can be estimated by using the same technique as in Eq. (6). However, because P(Ý) Y, X) usually is hard to estimate. As a result, the learned classifier (in (2) and Bayes labels will be poorly estimated, which leads to a large estimation error of P("I)=*) 2). We let h directly estimate Bayes labels but not P(Y]X). The output complexity of h reduced from continuous distribution (XY discrete distribution, the learning difficulty P(" is reduced.
EXPERIMENTS
In this section, we illustrate the performance of the proposed estimator and different methods under different data generation processes with the existence of label noise.
Baselines. We illustrate the performance of state-of-the-art modeling label-noise methods and semi-supervised methods. The modeling label-noise methods employed are: (i) Forward (Patrini et al. 2017) which estimates the transition matrix and embeds the neural network: (ii) Reweighting (Lu and Tao 2016) which gives training examples different weights according to the transition matrix by importance reweighting: (iii) T--Revision (Xia al., 2019) which refines the learned transition matrix to improve the classification accuracy. The semi-supervised methods employed are (iv) JoCoR
Wei et al. 2020) which aims to reduce the diversity of two networks during training; (v) MoPro (Li et al.) 2020) which is contrastive learning method that achieves online label noise correction
(vi) Dividemix (Li et al. 2019) which leverages the techniques FixMatch (Sohn et al.) 2020) and
Mixup (Zhang et al. 2018); (viii) Mixup (Zhang et al., 2018) which trains neural network convex combinations of pairs of examples and their labels. For all baseline methods, we follow their hyper-parameters settings mentioned in their original paper. It is worth noting that, MoPro focuses

---

(yyGuarsian. Sym)
Day Sysssaann IDN)
tynduasiionn Sym)
Symusssionn (N)
Our
Besmmror
Our Gatimror ValMinie Enthrator
Eedimano
Voll niot Latiraor
Volinnit Eotrnator
ValMinir: Eedmater
Eole -4
2.4
Reins Roce
Reies Rade
heles Rato
Neloa Rate
Figure 3: Estimation error of P(Y on synthetic datasets with instance-independen and instance- dependent label noise. Our estimator outperforms the state-of-the-an method by a large margin.
XYguassian
Sym
Instance
20% (0.196)
30% (0.142)
40% (0.081)
20% (8.)80)
30 (0. (0.)
40% (0.071)
Forward
98.9+0.21
98.35-H0.19
96.98
98.85+0.177
98,29+0024
96.72+00.3
Reweighting
98.61=0.10
99.01=0.12
96.42+11.2
09.54
99,25=+0.28
98.37=0061
T-Revision
99,4411011
98. 98.11110.12
97.08.11.48
99.54.10.23
99.26110.22
98.36.10.59
JoCoR (SSL)
98.05100.03
97.6310.16
97.11110.19
98.011.11
97.6510.21
97.26:10.09
MoPro (SSL)
96.75=0.67
95.5=1.3
79.76:14.95
95.85500.87
95.26111.78
78.2416..1
Dividemix (SSL)
97.58-10.4
13-10.9
93.31+2.17
96.61 1.05
1..9
94.14-12.28
Mixup (SSL)
96.86=0.59
96.06=+0.63
1.555+1.554
97.0=#.46
96,44+0.51
93.57+0.71
YXguassian
Sym
Instance
20% (0.021)
30% (0.008)
40% (0.005)
20% (0.023)
30 (0.013)
40% (0.005)
Forward
86.28--0.19
86.04+0.114
85.24+0.41
86.22+0.12
85.98+0.23
85.64+0.43
Reweighting
86.23=0.114
05.19:00.25
85.13:20.68
86.39+0.11
06.0440.226
85.54
T-Revision
86,43=00113
85.2=0.12
85.23+00.2
86.4=0.27
86.03++0.25
85.54+0.39
JoCoR (SSL)
86.14=0.08
0..8
85.23+00.3
86.04+00.09
85.86++0.28
85.1
MoPro (SSL)
85.17=0.711
83.73+11.2
81.11:2.35
85.17::0.49
84.410.54
82.2 06
Dividemix (SSL)
85.03++1.07
85.9 +0028
8..0
05.8+
85.74+0.54
85.8+0.33
Mixup (SSL)
85.92:10.48
2.33
82.62.12.78
86.2:10.2
85.62.10.55
82.08.14.57
Table I:
Test accuracies (%) of different methods on XYgaussian (causal) and YXgaussian (anticausal) datasets with different types of label noise. Estimations of CDNL estimator are shown in parentheses.
on image datasets, to let it work for non-image datasets, replace the strong data augmentation for images with small Gaussian Noise, which may influence its performance.
Datasets and noise types. We have employed synthetic datasets that XY gaussian and Ygguassian. We have also demonstrated the performance of our methods on 6 real-world datasets which are KrKp, Balancescale, Splice, waveform, MNIST, and CIFARIO. The causal datasets generated from X to Y are KrKp, Balancescale and Splice. The rest are anticausal datasets generated from to X. We manually inject label noise into all datasets, and 20% of data is left as the validation set. Three types of noise in our experiments are employed in our experiments. (1) symmetry flipping (Sym) (Patrini et al. (017) which randomly replaces percentage of labels in the training data with all possible labels. (2) pair flipping (Pair) (Han et al. (018)) where labels are only replaced by similar classes. (3) instance-dependent Label Noise (IDN) (Xia et al. 2020) where different instances have different transition matrices depending parts of instances.
Network structure and optimization. For a fair comparison, we implement all methods by PyTorch. All the methods are trained on Nvidia Geforee RTX 2080 GPUs For non-image datasets, a 2-hidden-layee network with batch normalization (Toffe and Szegedy 2015) and dropout (0.25) (Srivastava al., (0144) employed as the backbone method for all baselines. We employ LeNet-5 for MNIST (LeCun) 1998) dataset and ResNet-18 (He et al.) 2016) for CIFARIO (Krizhevsky et al.) 2009). To estimate P(Y])*), we use SGD to train the classification network with batch size 128, momentum 0.9, weight decay 10-4 The initial learning rate is 10-2, and decays 30th and 60th epochs with the rate 0.1, respectively. To get Y() for XYguassian, yxGuassain, KrKp, Balancescale, Splice and waveform and MNIST, K-means clustering method (Likas et al. 2003) is employed; for 1O, the SPICE" (Niu et al. 2021) clustering method is employed.
4.1 EXPERIMENTS ON SYNTHETIC DATASETS
To validate the correctness of our method, we have generated a causal dataset (from X to Y) and an anticausal dataset (from Y to X). For both datasets, P(X) is multivariate Gaussian mixture of (O I) and (11 I) with dimension 5. For causal dataset Xyguassian, the causal association f and I' between X and Y are set be linear. The parameter of the linear function is randomly drawn
7

---

KrKp
Sym
Instance
expect that the difference d(.)) obtained by employing large; when Y is a cause of X, the difference d(.) obtained by employing our estimator should be small. It is worth mentioning that, both datasets, estimations of our estimator decrease with the increase of noise rates. It is because that the labels of these two datasets are binary, and P(Ý 1) p( OlY* 1). Then, if the 20% (0.297)
30% (0.196)
40% (0.070)
20% (0.262)
30% (0.166)
40% (0.072)
Forward
93.3111.0
1993
77.7817.4
94.0=0.8
87,2513.1
80.7512.31
Reweighting
93.88-t1.43
91.16+1.099
77.31
93,5+22.3
1.52
78.22++6661
T-Revision
94.72+00.62
11881+193
77.97=5.0
94.5+1.63
90.78=2.35
79.06=4.899
JoCoR (SSL)
93.69:#0.23
89,53++0.84
67.81#2.07
93.44=0.71
87.44=2.95
67.75+6.51
MoPro (SSL)
89.47-11.13
79.47:17.00 03
65 65.9442.06
89.31-3822
79.59:6.2
62.62:14.78
Dividemix (SSL)
93.75++0.32
88.31+00.55
14.31+1444
93.47-0.15
93.34++0.22
63,94 1.45
Mixup (SSL)
93.31#1.1
18,81=+103
73.8441.18
93.19=1.31
87.25=1149
74.31+3.42
Balancescale
Sym
Pair
Instance
20% (0.099)
40% (0.071)
20% (0.113)
40% (0,109)
20% (0.110)
40% (0.090)
Forward
74.24:18774
78.8210.53
83,36.2..23
72.4819.12
75.3615..3
69.611971
Reweighting
89.76:+3.37
89 1.87
94.08 2.44
15.36+
90.72+2..8
86.24+11.88
T-Revision
92.64.10.93
89.76.1..14
92.32
81.12
89.12.1..45
85.28.12.06
JoCoR (SSL)
76.96.13.87
58,08113.43
12.32.110:43
60.16112.88
73.2814.34
51.2.16.33
MoPro (SSL.)
84.29++2.38
1.81
84.73-3.116
80.79+7.93
86.19-2.59
78.1+7.28
Dividemix (SSL)
88.16:10.32
0.993
01.12:00.99
1..96
87.52.00.4
1..04
Mixup (SSL)
86.08+-2.51
83.68+33449
16.72+1.3
67.68+17.1
84.96+2.17
75.36+5.46
Splice
Sym
Pair
Instance
20% (0.136)
40% (0.146)
20% (0.140)
40% (0.148)
20% (0.151)
40% (0.153)
Forward
71,25+3.07
66.18+3.61
73.73+1.03
65.8+3.67
65.8+4.08
61.6+5.67
Reweighting
16.96=:1.69
71.91+2.68
15.55+1888
66.68=1.54
15.64=1.95
63,5447.21
T-Revision
7..99.11.73
71.94.1..68
75.49
66.61
75.67
63,4517.17
JoCoR (SSL)
69.81114.1
63.211.89
59.3711444
51.7113.7
59.6612.44
55.315.87
MoPro (SSL.)
53.6-0.19
53.51+0.0
53.51+0.0
53.25-0043
53.79+0.38
52.17+3.27
Dividemix (SSL)
75.11=1.666
53.45=0.0
53.45=0.0
56,14=2.1
59.97=0,55
51.41=1.79
Mixup (SSL)
67.43.13.2
62.16.12.52
68.1512.63
65.52.1222
49.03.19.86
Table 2:
Test accuracies (%) different methods on causal datasets with different types of label noise. Estimations of CDNL estimator are shown in parentheses.
from the (o0, f). For YXguassian, we let the label be the mean value of the multivariate Gaussian distribution. For both datasets, we have balanced the positive and negative class priors 0.5, and the training sample size is 20000.
4.1.1 ESTIMATION ERROR OF P(*
In Fig. 3 we compare the estimation error of average flip rate (Y*) of our CDNL estimator and the ofate-of-the-ar method VolMinNet (Li et 2021), respectively. To let VolMinNet estimate P(Ý Y"), we first train VolMinNet with noisy training set and select the best model by using the validation set, then the estimated clean class- bass-posterior distribution P(Y|X) is obtained. The Bayes label y can be directly obtained via X)Y|X), and P())(Y*) can be estimated by using the same technique in Eq. (6) As illustrated in Fig. it shows that the estimation error of our method is close to 0 not only on instance-int indduceedddeeeneentt label noise but also on instance-dependee which is much smaller than the estimated error of VolMinNet. This illustrates the advantage of CDNL estimator, which does not require learning the transition matrix and clean label for each instance, but directly estimates the average level of noise rates.
label noise,
4.1.2 ESTIMATIONS CDNL ESTIMATOR AND CLASSIFICATION ACCURACY
In Tab. I we illustrate the estimations CDNL estimator and the test accuracies of modeling label- noise methods and semi- cumi-supervieed methods. The estimations of CDNL estimator are shown in the parentheses, and each estimation is averaged over 5 repeated experiments. The matrix Á P* estimated by our method is embedded into modeling label-noise methods.
It shows that the estimations the anticausal dataset YXgaussian are 10 times smaller than causal dataset XY guassisn,, which illustrates the effectiveness of our estimator. Specifically, when is cause of Y (anticausal),
estimator is
only has degree of freedom, i.e., P(ý 1 =
8

---

Waveform
Sym
Pair
Instance
20% (0.138)
40% (0.257)
20% (0.257)
40% (0.12)
20% (0.099)
40% (0.089)
Forward
74.66+7.68
74.76+3.3
70.02+10.79
66.46+3.88
39.78+12.14
36.62-12.87
Reweighting
84.58=1899
83.92=1.38
83,30+2.28
73.22#4.51
85.02=0.93
83,3=3.02
T-Revision
84.24+1.3
85.70=0.66
82.72++6.03
68.86+8.56
84.04-2.38
83.5+1.87
JoCoR (SSL)
83.44=0.83
60.28+1.46
80.64+1.299
57.14+4.17
63.84+8.8
54.56++4.44
MoPro (SSL)
76.62=7.116
76.37:7.00
79.55+22.2
58.4447.11
77.36_4.04
15.1445.66
Dividemix (SSL)
03.36-:0.63
82.06
6..74-119
58.48:10.98
73.00
66.86.1 1.26
Mixup (SSL)
81.38
79,48.1.00
80.54.12.51
72.34
78. 88 1.05
71.26.15,44
MNIST
Sym
Pair
Instance
20% (0.034)
40% (0.038)
20% (0.041)
40% (0.20)
20% (0.025)
40% (0.026)
Forward
98.75=0.08
97.86=0.22
98,8420.10
94,9240.89
96.87=0.15
90.30=0.61
Reweighting
98.71=0.11
98.13=0.19
98 53
91.50+1.27
97.99=0.13
90.30+0.61
T-Revision
98.91+0.04
98.3440.21
98.89+0.08
9183+1.08
98. 98.3940.00
96.00+0.1
JoCoR (SSL)
98.06=0.133
96.64:0.19
98.01
46,8540.43
98 62.0000006
96.0740.31
MoPro (SSL.)
90.51
1..23
96.79+11 1.04
94,96+
98.53+0.52
96.45 1,20
Dividemix (SSL.)
99.24+0.03
99.21
99.25+0.03
98.50+0.08
99.31
97.75+0.1
Mixup (SSL)
97.45=0.21
95.75=0.43
97.57+1008
92.46
96.54+1.20
10.38
CIFAR10
Sym
Pair
Instance
20% (0.010)
40% (0.009)
20% (0.010)
40% (0.026)
20% (0.037)
40% (0.042)
Forward
88.21+0.48
78.44+0.89
88.21+0.48
77.44=6.89
85 29:299.
74.72+3.24
Reweighting
86.77=0.40
83.16+0046 46
89.60+1.01
77. 77.06=6.47
88.72=0.44
84.52+2.65
T-Revision
90.33+0.52
84.9442.58
89.7540.41
80.9442.58
90.4640.11
85.3743.36
JoCoR (SSL)
85.9620.25
79.6550.43
80.3340.00
71.6241.05
89 8020.28
73.78+1.39
MoPro (SSL.)
15.15
67.70+0.56
77.92+0.81
69.89 1.02
15.75=0.15
67.61=0.24
Dividemix (SSL)
95.620.10
94.8=1.10
95.72+0.04 04
87.02 #0.41
95.5+1.17
94,5+0.23
Mixup (SSL)
93.2+0.31
86.240.3
92.23+0.71
82.43
93.3240.25
87.61+0.56
Table 3: Comparing test accuracies (%) of different methods on anticausal datasets with different levels and types of label noise. Estimations of CDNL estimator shown in parentheses.
difference between P( 1|Y =) and P( 1|Y" 1) is small, the difference between P( = 0|Y = 1) and P(L = 0|Y 1) will also be small, the estimation will be small.
The results show that the causal dataset XY XYguassian, modeling label-noise methods perform better than semi-supervised methods.
because that P(X) does not contain information of P(Y]X), then semi-supervised methods may not be helpful. On the anticausal dataset YXguassian, when the sample size large (20000), modeling label-noise methods perform slightly better than semi- supervised methods. When the complexity of anticausal datasets is high, with limited sample size, the semi-supervised method should have better performance than modeling label-noise methods (See Tab. 3). We have also conducted an additional experiment on YXgaussian by reducing the sample size to 5000, SSL-based methods clearly outperform modeling label-noise methods under the same setting. The results are illustrated in Appendix
4.2 EXPERIMENTS ON REAL-WORLD DATASETS
We illustrate estimations of CDNL estimator, test accuracies of different methods on real-world datasets Tab. ] and Tab. 2 On these datasets, when estimation CDNL estimator is lower than 0.005, semi-supervised methods demonstrate their effectiveness, otherwise, modeling label-noise methods are more helpful to improve the robustness of learning models. It is also worth mentioning that for waveform, although it is anticausal dataset, modeling label-noise methods have better performance than sami-supervised methods, and the estimation CDNL estimator is also large. The reason could be that 1). P(X) may not always contain information about P(Y X even if the data generation process is from to X, or 2). P(X) may contain information about P(Y[X), but the information can be hard be exploited by existing methods.
CONCLUSION
In this paper, we have investigated the influence of the noisy data generation process semi- supervised methods and modeling label-noise methods. We show that the semi-supervised methods can easily incorporate heuristics make of the finite training sample but their performance depends on the data generation process, while the modeling noise methods are independent of the generation process. In many real-world applications, the causal structure of the data generation

---

process is not given. Then proposed an intuitive method by exploiting the asymmetric property of estimating flip rate under different generalization processes.
REFERENCES
Madalina Ciortan, Romain Dupuis, and Thomas Peel. A framework using contrastive learning for classification with noisy labels. Data, 6(6):61, 2021.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
Erik Englesson and Hossein Azizpour. Consistency regularization can improve robustness to label noise. arXiv preprint arXiv:21 2021.
Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2703-2708, 2021.
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based graphical models. Frontiers in genetics, 10:524, 2019.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pages 8527-8537, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pages 770-778, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.00 144, 2016.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Fei-Fei. MentorNet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, pages 2309-2318, 2018.
Roy Jonker and Ton Volgenant. Improving the hungarian assignment algorithm. Operations Research Letters, 114):17 1986.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for remi-supervised learning. arXiv preprint arXiv:1610.02222,, 2016.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels semi-
supervised learning. In ICIR, 2019.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint XXX::20090079955 2020.
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. arXiv preprint arXiv:2102.0244 2021.
Aristidis Likas, Nikos Vlassis, and Jakob J Verbeek. The global k-means clustering algorithm. Pattern recognition, 36(2):451 4611, 2003.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. JEEE Transactions pattern analysis and machine intelligence, 38(3):44--46 461, 2016.
10

---

Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In ICML, pages 6226-6236. PMLR, 2020.
Due Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo. Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-ensembling. In ICLR, 2019.
Chuang Niu, Hongming Shan, and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. arXiv preprint arXiv:2103.09382, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pages 1944-1952, 2017.
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY, USA, 2000. ISBN 0-521-77362-8.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
B Schölkopf, D Janzing, J Peters, E Sgouritsa, K Zhang, and J Mooij. On causal and anticausal learning. In 29th International Conference on Machine Learning (C12), pages 1255-1262. International Machine Learning Society, 2012.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685: 2020.
Peter Spirtes and Kun Zhang. Causal discovery and inference:
concepts and recent methodological advances. In Applied informatics, volume pages 1-28. SpringerOpen, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1229-158, 2014.
Cheng Tan, Jun Xia, Lirong Wu, and Stan Z Li. Co-learning: Learning from noisy labels with self-supervision. In Proceedings of the 29th ACM International Conference on Multimedia, pages 1413, 2021.
Hongxin Wei, Lei Feng. Xiangyu Chen, and Bo An. Combating noisy labels by agreement: joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13726-13735, 2020.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable label-noise learning? NeurIPS, pages 3666, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. NerurIPS, 2020.
Yazhou Yao, Zeren Sun, Chuanyi Zhang, Fumin Shen, Qi Wu, Jian Zhang, and Zhenmin Tang. Jo-src: A contrastive approach for combating noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5192-5201, 2021.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t; Reducing estimation error for transition matrix in label-noise learning. In NeurIPS, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez- Pazz mixup: Beyond empirical risk minimization. In ICLR, 2018
Kun Zhang, Jiji Zhang, and Bernhard Schölkopf. Distinguishing cause from effect based on exogene- ity. arXiv preprint arXiv: 004400044405555 2015.
11

---

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, pages 8778-8788, 2018.
Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1657-1667, 2022.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021.
A PROOF OF THEOREM
In this section, we will prove Theorem D in our main paper.
Proof.
P =)) Σ P( = j| =---" =xYY= =)
P(Ý = =)P(Y" k|Y"= =).
The second equality holds because that Y' is learned in an unsupervised manner that does not employ Y. then Y does not contain any information about Y and independent with Y. By the above equation,
P(Ý ji =i) - P(K = -|='== =
=P(Ý= = =--r =i)- >>-_p(e P(ύ P(K = j|y = - = k|Y'=
k
-(p(Ý = =ilY"= =i) -P)-p(p = =jlY = =I)P(Y* = =ilY' = =i)) P( = =k) = k|Y =
kyi
=(1 - P(Y = i|| = P( = =|l =1) Σ P(Y = k\Y' P( =iyY = =k).
For condition 2), let the difference p(e = j|YY =1) - -P(- =|y = i) =0, -0, then according to the above equation,
(1 P(Y" = i|p i)) = 5|8 =i Σ P(Y = k/Y =i) =]lY== = k). k
For condition 1), P(Y i|Y = i) for all i, then (1 P(Y i i)) 0 and P(Y k|Y" = i) 0 for all k. It implies that both LHS and RHS will be 0 which completes the proof.
B ESTIMATION ERROR OF P(Ý
Here, we theoretically show that when X causes Y, flip rate P(Ý estimated by an unsupervised classification method usually has a large estimation error, where Y pseudo labels estimated by the unsupervised method. However, when Y causes X, the estimation error is usually small.
Theorem 2. Let P(LIY) be the transition relationship from the noisy label Y to the clean label Y; let (Y') be the transition relationship from the noisy label Ý to the pseudo label Y'. Then the estimation error is
L L
P(ÿ = =ijY = -))= PP =
= P((1)))).P')))))
ΣΣ
122
Eis P(Y = [p(x)
(P(Y'= =(X))
P(Y=j) m(V'))
P(Y =jlx)) 1 1
2
12

---

Proof. Let f(x) max; P(Y = i|X = Z) output the noisy label of every instance
P(ś = =i-Y= E ((()=-))
/ ( x|Y j)dr.
(8)
Then similarly,
p( = -) -0)=Epm EXXIN [1100=
J. () P(X = x|\² = =]d
/. (((())1))( P(x==)ppp----000> = x| = =) P(X x x\Y =
dx
P(X = x|Y =
P(X = j)
= =EMX|Y=j) [1F(X)-1 P(X = j)
(9)
The last equality is obtained by using the reweighting technique Liu and Tao (2016), which requires that P(X 5) and P(X = 3) have the same support. Then we calculate the difference p( = i|Y'= = P(Y = i|Y = j) as follows.
-) p(Ý = i|8 =) - P(Ý = =i-
P(X|Y' = -EPXII=>)
-Ei(I(= 1 [1(x)= P(XIV =}
((()=))) P(x=xx = 1) P(X x/YY = 1) =1)
P(X = =Y= dr - -111(1000))PC P(X P(x=x|V x\Y
j)dr
1(G)=D P = x|Y 1) P(X x = =))
P(X P(x==\Y- = j)dd
P(X = x = ))
U(W=D (P(X =x|Y'- = 1) P(X x/Y = =])) dz
P(YY- = -| = x)P(X = x) P(Y = = x)) =x) dx (f(x)==)
=
P(Y' =
P(Y =
P(Y' = j|x = P(V = -)) = x) P(V = = =x) = -)) =x) =))) (f
P(Y' -)) =)
P(Y = p =
P(Y' = j|x = P)Y = =j) =x) P(Y = j|- = P)Y -)) (1(x)=1) dr
P(Y' = j)) =
P(Y' = -|x = P(Y =) - P(Y - xx)x=')((''-- -) P(X= x) 2)
P(Y' [Y'==JPPM PY =
P(Y'= j(x))X =) P(Y = -(()))(X''')) = i)
=EP(X)
P(Y'=
= j)P(Y =j)
P(Y' = jIX)P(Y = - P(Y = -(x))(('= (
=P(Y- = =))
P(Y'=
( -(XX==)x)) P(Y=j) - PPY = s(x)) 1 1
=P(Y- = j)))(())
(10)
P(Y" =))
By using the above equation, the estimation error dl d(((TV'") P(Ý YY) is as follows.
1
P( i = -)-P(P = =i = j)
(((0))",,((111) = ΣΣ
12
j) Em(x)
(P(Y'=
P(Y=j) P(Y
P(Y = =(xx) 1
12
which completes the proof.
13

---

we find out that let the estimation enror be small, two class posterior P(Y of the pseudo label and clean class posterior P(Y have to be similar. For example, suppose that (X) and P(Y]X) are identical. P(Y) also identical to P(Y") because P(Y") [[ P\=-)
From Theorem 2
and P(Y) - [ Then, P(Y' = j| - x) P(Y-=) P(Y j| - x) = for all 2, and the estimation error ((P())))) P()I))) is
On the anticausal dataset, P(X) can inform P(Y[X), therefore P(Y*X)) learned by exploiting P(X) close to P(Y]X) Then P(Y) also close to P(Y") because P(Y") E[P((Y|X)) and P(Y) X[)( Therefore P(Y" = xl P(Y=j)
P(Y=))) P(Y j|X) is small. The estimation
error is small. On the causal dataset, P(X) can not inform PYY then P(Y and P(Y should have a large difference. Then P(Y" = j|X) POY=i]
P(Y=)) P(Y = jl is usually larger than
the case of Y causes X. Then the estimation error on the causal dataset is larger than the anticausal dataset.
C EXPERIMENTS ON YXGAUSSIAN WITH SAMPLE SIZE 5000
We conduct additional experiments on anticausal YXgaussian dataset by reducing the training sample size to 5000. The other settings are the same as Section in the main paper. SSL-based methods clearly outperform modeling label-noise methods. The results are illustrated in Tab.
YXguassian
Sym
Instance
20% (0.037)
30% (0.039)
40% (0.047)
20% (0.037)
30 (0.036)
40% (0.049)
Forward
84.76:0..51
82.75:11.34
79.65:12.36
84,45
82.15:1.34
72.77:12.4
Reweighting
84.31110.48
83 13.14.10.44
78.06.13.4
84,14.11.52
83.14.1.74
74.3116.75
T-Revision
84.41+0.445
83.24+0.74
78.02+3.2
84.15+1.51
83.13+0.52
74.33++6.4
JoCoR (SSL.)
85.22+0.21
84.58+0.63
79.81+3.62
85.1+0.31
84,58+0.63
72.46+6.48
MoPro (SSL)
84.17:10.67
83.2341.51
79.56:1..38
85.1340.34
83.220.51
76.15#6.12
Dividemix (SSL)
84.73-11.21
L0.34
80.15.14.34
85.6
85.13.10.46
79.82.14.51
Mixup (SSL)
84.6.10.41
83.83.12.56
79.43.1..3
85.8
85.0910.43
73.58.14.3
Table 4: Test accuracies (%) of different methods YXgaussian datasets with sample size 5000 under different types of label noise. Estimations of CDNL estimator are shown in parentheses.
14


Reviews:
[{"review_id": "eXE9vlvKee2", "reviewer": "Reviewer_TTFe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies two methods (modeling label noise and semi-supervised methods) for handling datasets with noisy labels, and the causal structure strongly influences which of the methods perform better. The paper additionally proposes a method for finding the causal structure.", "summary_of_the_review": "The paper studies an interesting problem, but there is no theoretical justification, and the experiments to not strongly support the conclusions drawn by the paper.", "strength_and_weaknesses": "The paper studies an interesting problem of comparing two major lines of work, but the conclusions are not strongly supported. There is little theoretical justification in the paper, and the experiments do not closely match the conclusions drawn in the paper.\n\nMajor Comments:\n\n1) There are many probability distributions that can be modeled under either (a) or (b) of Figure 1; it's unclear how the causal structure be directly used to determine which set of methods would perform better\n\n2) A core claim is that the estimation error of P(\\tilde Y | Y') is more accurate, but this does not seem to have any justification.\n\n3) There appears to be mismatches in the experimental results and the main conclusions of the paper. \n\nEven in the datasets where semi-supervised methods are supposed to generally perform well, it seems common that only one of the methods actually performs well, while the others still have poor performance.\n\nThe tables also have incorrect numbers bolded in a number of columns (at the very least, JoCor outperformed T-Revision in XYgaussian Sym 30%, Forward outperformed Dividemix in YXgaussian Sym 30%, and Dividemix outperformed T-revision in KrKp Instance 30%), which falsely suggests that the trends are stronger than they actually are.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Hwsv3LAKZlU", "reviewer": "Reviewer_ZHc2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors approach the challenge of huge datasets with label noise from two perspectives: a semi-supervised learning (SSL) strategy that leverages labelled and unlabeled samples, and a noise modelling (NM) approach. According to the authors, semi-supervised learning is dependent on the causal generation process, but noise modelling is not. The authors also provide a method for determining a dataset's causal structure and evaluate results between SSL and NM for different rates of causality on synthetic and real-life datasets", "summary_of_the_review": "Please see the Strengths and Weaknesses section above.", "strength_and_weaknesses": "I'd like to start by expressing that I'm not totally confident in my judgement, therefore I'm interested in the author's reponse to these statements/questions.\n\nStrengths\n\n-The work addresses a significant issue in dealing with training models on large datasets with noisy labels.\n- The authors suggest an intriguing method termed CDNL, which computes the flip rate using a clustering method to assess causality structure.\n- The proposed method appears to outperform existing methods in computing the average flip rate, which aids in detecting the dataset's causality structure.\n\nWeaknesses\n- It is not clear what the contributions and conclusions of this work are. It would help if they are explicitly and concisely listed at the end of the introduction and the abstract. Currently they seem scattered over the article and cluttered. For instance, if the proposed estimator outperforms VolMinNet in error estimation, then that should be mentioned earlier on as part of the contributions.\n\n- I am missing how the works described in the related work related to the current work. A sentence or two about the limitations of existing methods and how the current method addresses them at the end of each paragraph would make things clearer. For example, the paragraph on \"Method based on semi-supervised learning\" could have an extra such sentence(s).  Right now it is not clear how this work contributes to the literature in comparison to existing works.\n\n- I am having a hard time understanding Figure 2, could you clarify how that figure shows that semi-supervised learning does not work when P(X) does not contain information to learn clean label Y? Further, the caption in Figure 2 repeats the conditional \"When Y causes X\" twice with different conclusions, adding to the confusion further.\n\n- It is not clear why the following quoted claim is true, is there a theorem that proves it? \"when X causes Y , the flip rate P(Y˜ |Y ′ ) estimated by an unsupervised classification method usually has a large estimation error, where Y ′ is pseudo labels estimated by the unsupervised method. However, when Y causes X, the estimation error is small.\"\n\n- this work misses large-scale real-life datasets like ImageNet which is important to reliably evaluate the methods in this work.\n\n- More explanation and comparison needs to be done about why CDNL outperform VolMinNet. For example, is there inductive bias in CDNL that helps it learn better?\n\n- The table is poorly presented, it is not clear which methods are semi-supervised and which are noise modeling. It would help if the authors add a column that says something like \"SSL\" for the semi-supervised and \"NM\" for the noise modeling methods\n\n- It is not clear whether SSL is helping when there is a causal structure, so far it seems that Forward, Reweighting and T-Revision almost consistently achieve state-of-the-art except in MNIST and CIFAR10 and it is not clear why.\n\n- It is not clear what the differences are in the rows of Table 1, in both major rows, the dataset is XYguassian. Are these differentiated by casual and anticasual?\n\nOverall this work could use more clarification and more experiments and theorems before deriving any conclusion. The results seem a bit inconsistent to justify the use of SSL over noise-modeling and vice versa. It is not clear how this work can help us develop better methods for large datasets with noisy labels in practice", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ESKbu0abbG", "reviewer": "Reviewer_7ieU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of learning with noisy labels. They broadly categorize the methods into two groups 1) that model the label noise 2) semi-supervised methods. The paper focuses on the problem that in practice it is hard to figure out which group of methods should be employed on the problem at hand. They offer a solution based on \"causal data generation process\". They argue that semi-supervised methods depend heavily on the data generation process while modeling label noise is independent of the data generation process. Based on this they claim that if the given dataset has a causal generative structure i.e. features causing the labels, then the semi-supervised methods would not helpful. They also give a method to estimate this causal structure from the data, as it is not available in practice.", "summary_of_the_review": "The paper studies an interesting and relevant problem. They make strong claims about semi-supervised methods and methods that model the label noise. I am not quite convinced by the claims and the details provided to support the claims.", "strength_and_weaknesses": "Strengths:\nS1. The problem to find when a kind of methods are better than others for learning from noisy labels is interesting and of practical relevance. \nS2. The use of causal generative process to study the problem looks promising and interesting.\n\nS3. Empirical comparison of various methods in the two groups on synthetic and real datasets.\n\nWeaknesses:\nW1. The paper makes very strong claims about two groups of methods and I have two main concerns with this: a) Looking at Table 1, I don't see significant differences b/w the two groups of the methods in XY causal and YX causal settings both. b) I don't quite see how the theory is supporting their claims either.\n\nW2. I understand different data generation from Figure 1, however I could not understand how these differences affect different groups of methods.\n\nW3. The algorithm 1, is too abstract and doesn't provide more details into how $h$ and $A$ are optimized and how the clusters are obtained.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6pAltcFqdc", "reviewer": "Reviewer_pU2M", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- Assuming that there is no bi-directional causation, identify the causal direction between X and Y\n- Argue that SSL is less effective in the causal direction than in the anti-causal case\n- Argue that modeling label noise is agnostic to the causal direction\n    - suggest a method to detect causal structure based on modeling label noise\n- Introduce the CDNL estimator, which is claimed to be SOTA for $P(\\tilde{Y} \\mid Y^*)$\n    - end to end learning of both $P(Y^* \\mid X), P(\\tilde{Y} \\mid Y^*)$\n- Demonstrate that the label noise methods outperform the SSL methods on causal datasets, while mostly matching levels of SSL methods on anti-causal datasets\n    - although it is mentioned that SSL should outperform label noise methods for datasets where the ‘complexity’ is high", "summary_of_the_review": "This paper introduces a novel method to determine the causal direction of a data generating process, when label noise is present. However, there is not enough of a literature review for me to feel comfortable that this method has not been tried in some manner before. Further, the examination of how this method can be usefully applied is poorly explained and unclear to me at present, when i compare against simply running both SSL and label noise methods in my problem of choice. Finally, the argument that anticausal and causal learning leads to different outcomes for SSL methods is not new. I would like to see more examination of the previous work on this area including methods developed (or lack thereof) to identity whether SSL is appropriate.", "strength_and_weaknesses": "- There are not any experiments that demonstrate how to use the CDNL estimator in order to achieve better results than simply comparing outcomes for label noise and SSL methods\n    - Why would I use CDNL instead of doing both SSL and label noise methods?\n    - I am not convinced that the CDNL estimator is the best method for estimating causal direction when there is a vast body of literature about how to estimate causal direction which has been ignored by this work\n- The comparison with the VolMinNet is too brief: the authors would do well to outline the method of VolMinNet, what its weaknesses are, and how CDNL improves upon it\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4xT1b9uxF6g", "reviewer": "Reviewer_cjo1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper compares two families of methods for learning with noisy labels. One is to use semi-supervised learning methods, and the other is to model label noise and design statistically consistent classifiers. The paper wants to answer the question: which one is better? From the perspective of causal data generation, the authors found that the answer depends on the underlying data generating process. Specifically, when the instance X is a cause of the clean label Y, modeling noise tends to be better. When Y causes X however, semi-supervised learning methods tends to perform better. In real-world applications, such causal structures are generally unknown. The authors proposed a method to discover the causal structure from a given dataset containing label noise.", "summary_of_the_review": "The paper is marginally below the acceptance threshold. The decision was based on weighing the strong and weak points above.\n\n**Major questions and comments.**\n\n1. Figure 2 should be improved. Just to be precise, what are those contours, the shaded areas, and the straight dotted lines?\n\n2. How do you know KrKp, Balancescale and Splice are causal datasets, while waveform, MNIST and CIFAR10 are anticausal datasets?\n\n3. Just above Section 4.2, the paper says \"When the complexity of anticausal datasets is high, with limited sample size, the semi-supervised method should have better performance than modeling label-noise methods.\" In Table 3, we see that the semi-supervised methods performed better in MNIST and CIFAR10. However, these two datasets have relatively large sample sizes. I don't see how the claim can be verified.\n\n\n**Minor comments.**\n\n1. Inconsistent notation. In Page 1, $T$ is row-stochastic, while in Eq. 2, it is column-stochastic.\n\n2. Inconsistent notation. In Eq. 1, $C$ is the number of classes, while in Eq. 3, $L$ is used as the number of classes.\n\n3. Page 4, second to the last line: \"estimates\" -> \"estimate\".\n\n4. Page 6, in the paragraph just above Section 4: some citations should be in parentheses. \n\n5. Table 1: for sym 30%, the best performance should be 86.04.\n\n6. Page 9, Section 4.2, second to the last line: \"from X to Y\" should be \"from Y to X\".\n", "strength_and_weaknesses": "**Strong points**\n\nIt is of significance to study advantages and disadvantages of the two popular families of methods dealing with noisy labels, and to understand conditions under which one is better than the other. The authors made a good observation from the perspective of causal data generation (namely, when the instance X is a cause of the clean label Y, modeling noise tends to be better; when Y causes X however, semi-supervised learning methods tends to perform better), and also proposed a method to discover such causal structure from noisy data. The finding is intuitive from the theoretical point of view. The proposed method was also demonstrated its effectiveness by extensive empirical experiments.\n\n**Weak points**\n\nThe main problem is that, in the proposed method (CDNL), it is required to estimate Bayes labels $Y^*$. As Bayes labels are used to estimate $P(\\tilde{Y}|Y')$ and $P(\\tilde{Y}|Y^*)$, which is in turn used to calculate the metric (Eq. 3) that determines which family is better, the quality of $\\hat{Y}^*$ needs to be good. However, if $\\hat{Y}^*$ is estimated reasonably well from the noisy data, we will have obtained a good classifier. The why bother proceeding further? Also, it seems that some model-based or semi-supervised methods are needed to estimate Bayes labels. For example, in Page 6 Eq. 7, a diagonally dominant column stochastic matrix is used to model the label noise. The arguments seem somewhat circular.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject