Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning
Authors: Shikun Li
Xiaobo Xia
Hansong Zhang
Yibing Zhan
Shiming Ge
Tongliang Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: label-noise learning
multi-label learning
transition matrix estimation

Abstract: In label-noise learning, the noise transition matrix, bridging the class posterior for noisy and clean data, has been widely exploited to learn statistically consistent classifiers. The effectiveness of these algorithms relies heavily on estimating the transition matrix. Recently, the problem of label-noise learning in multi-label classification has received increasing attention, and these consistent algorithms can be applied in multi-label cases. However, the estimation of transition matrices in noisy multi-label learning has not been studied and remains challenging, since most of the existing estimators in noisy multi-class learning depend on the existence of anchor points and the accurate fitting of noisy class posterior. To address this problem, in this paper, we first study the identifiability problem of the class-dependent transition matrix in noisy multi-label learning, and then inspired by the identifiability results, we propose a new estimator by exploiting label correlations without neither anchor points nor accurate fitting of noisy class posterior. Specifically, we estimate the occurrence probability of two noisy labels to get noisy label correlations. Then, we perform sample selection to further extract information that implies clean label correlations, which is used to estimate the occurrence probability of one noisy label when a certain clean label appears. By utilizing the mismatch of label correlations implied in these occurrence probabilities, the transition matrix is identifiable, and can then be acquired by solving a simple bilinear decomposition problem. Empirical results demonstrate the effectiveness of our estimator to estimate the transition matrix with label correlations, leading to better classification performance. Source codes are available at https://github.com/tmllab/Multi-Label-T.

Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning
Shikun Li
Xiaobo Xia
Hansong Zhang! Yibing Zhan
Shiming 2e Tongliang Liu
Institute of Information Engineering, Chinese Academy of Sciences
School of Cyber Security, University of Chinese Academy of Sciences
3 Trustworthy Machine Learning Lab, The University of Sydney JD Explore Academy
Abstract
In label-noise learning, the noise transition matrix, bridging the class posterior for noisy and clean data, has been widely exploited to learn statistically consistent classifiers. The effectiveness of these algorithms relies heavily on estimating the transition matrix. Recently, the problem of label-noise learning in multi-label clas- sification has received increasing attention, and these consistent algorithms can be applied in multi-label cases. However, the estimation of transition matrices in noisy multi-label learning has not been studied and remains challenging, since most of the existing estimators in noisy multi-class learning depend on the existence of anchor points and the accurate fitting of noisy class posterior. To address this problem, in this paper, we first study the identifiability problem of the class dass-deeendent transition matrix in noisy multi-label learning. and then inspired by the identifiability results, we propose a new estimator by exploiting label correlations without neither anchor points nor accurate fitting of noisy class posterior. Specifically, we estimate the occurrence probability of two noisy labels to get noisy label correlations. Then, we perform sample selection to further extract information that implies clean label correlations, which is used estimate the occurrence probability of one noisy label when certain clean label appears. By utilizing the mismatch of label correlations implied in these occurrence probabilities, the transition matrix is identifiable, and can then be acquired by solving simple bilinear decomposition problem. Empiri- cal results demonstrate the effectiveness our estimator to estimate the transition matrix with label correlations, leading to better classification performance. Source codes are available at https://github.com wttpiittuu.cooo/eeoo www/mmlaa//Mulllaiic Mallla///ul----Le--T.....
Introduction
In real-world scenarios, instance is naturally associated with multiple labels, and these labels have complex entangled correlations [7]. Recently, the problem of label-noise learning in multi-label classification has received more and more attention [29, 34, 54, 50, 46, 47], since it is time-consuming and expensive to collect large-scale accurate labels and the noisy labels are much cheaper and easier acquire. In the setting of noisy multi-label learning, the multiple labels assigned be corrupted simultaneously. That is to say, any label for each class can be flipped with its respective transition matrix that denotes the transition relationship from clean labels to noisy labels.
instance may
Transition matrix has been utilized to build series of statistically consistent algorithms for noisy multi-class learning [32, 52, 48, 8). The main advantage of these consistent algorithms is that they
Shiming Ge is the corresponding author. (Email: )
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

can guarantee to vanish the differences between the classifiers learned from noisy data and the optimal ones from clean data by increasing the size of noisy examples [28, 33, 52, 39].
Fortunately, these statistically consistent algorithms for noisy multi-class learning can also be applied in such noisy multi-label learning with a little modification [54] (more details can be found in Appendix B). However, the effectiveness of these algorithms heavily relies on estimating the transition matrix. Although the estimation of the transition matrix has been investigated in noisy multi-class learning, the estimation of the transition matrix in noisy multi-label learning has not been studied and remains challenging. Specifically, a series of methods [28, 33, 57, 52, 26] has been proposed to estimate the transition matrix for noisy multi-clase learning. Most of them assume the existence of anchor points [28, 33, 57] that are defined as the training examples belonging to particular clean class surely. Nevertheless, the assumption is strong and hard to check when we only have noisy data [52]. Also, the methods need to accurately fit the noisy or intermediate class posterior of anchor points, which are rather difficult in multi-label cases, due to severe positive-negative imbalance [36].
In this paper, to address the problem of estimating the transition matrix in noisy multi-label learning, we consider utilizing label correlations among noisy multiple labels. Specifically, some label correlations that should not exist in practice are included in noisy multi-label learning. For example, "fish" and "water" always co-occur, while "bird" and "sky" always co-occur. But, due to label errors, there is slight correlation between "fish" and "sky". which is impractical. At high level, we can utilize the mismatch of label correlations to identify the transition matrix without neither anchor points nor accurate fitting of noisy class posterior.
In more detail, we first focus on the identifiability problem of the class-dependent transition matrix in noisy multi-label learning. Accordingly, new method that estimates the transition matrix by exploiting label correlations is proposed. That is, motivated by the identifiability result that the label correlation of two noisy labels can not suffice to identify the transition matrix in noisy multi-label learning, we utilize sample selection to extract useful information from noisy data, which implies clean label correlations to achieve the identifiability. Afterward, we not only estimate the occurrence probability of two noisy labels in noisy data, but also of one noisy label when a certain clean label appears in selected data. By utilizing the mismatch of label correlations implied in these occurrence probabilities, we can prove the identifiability, and transform the problem of estimating the transition matrix using label correlations into a problem of bilinear decomposition. Finally, with easy frequency counting, we can get good estimation of the noise transition matrix.
Empirical results illustrate the effectiveness of the proposed estimator for estimating the transition matrix in noisy multi-label learning, and the consistent algorithms with our estimator can achieve better classification performance.
The rest of the paper is organized as follows. In Section 2, we briefly present the problem setting of label-noise learning in multi-label classification. In Section 3, we discuss the identifiability of the transition matrix under such a noisy multi-label setting, and introduce our estimation method. Experimental results are provided in Section 4. The limitations of this work are discussed Section 5. Finally, we conclude the paper in Section 6.
2 Problem Setting
In this section, we introduce the problem setting of label-noise learning in multi-label classification. In what follows, scalars are in lowercase letters, vectors are in lowercase boldface letters, and matrices/variables are in uppercase letters. For simplicity, let [q] = (1.....9).
Preliminaries. Let D be the distribution of a pair of random variables (X, Y), where X E X C Rd
denotes the variable of instances, and Y {YY, y2 ...., y1} E {0, 1) denotes the variable of targets with possible class labels. As for Y.Y = indicates that the instance X is associated with the class j: y O. otherwise. In multi-label learning, the goal is to learn function from D which maps each unseen instance x: E X' to proper labels y. However, as discussed, Y is hard to be annotated precisely. Before being observed, their true labels
independently flipped and what we 1s where D. denotes noisy labels. Let D be
can obtain are noisy training examples D,- = {x) 7.) H the distribution of the noisy random variables (X, Y) E X x {0, 13 In noisy multi-label learning, our goal is to infer proper labels for each unseen instance by only using the noisy training examples.
Noise transition matrix. The random variables Y and Y for the class j are related through noise transition matrix T [0,1 E [4]. Generally, the transition matrix depends on instances,
2

---

i.e. (x) P(Y = k Y = i, = z). Nevertheless, given only noisy examples, the
mnstance-dependent transition matrix non -oneninntee without any additional assumption [52. 56].
For example, both P(Y) = = Ei T. P = =i|X = x) and P(Y) = -||X- =
x) =
= are valid, when T x) = Ty (X - P) = =||X =
P = i = æ). Therefore, in this paper, we assume that the transition matrix is class-
dependent and instanc--iddependet [33], i.e., P(Y k Y - X = =)) P(Y
Y
i) = T. The definition of the class-dependent label noise can be found in Appendix A, where we further discuss its differences with the class class-dependent label noise in multi-class cases.
Consistent algorithms. The transition matrix bridges the class posterior probabilities for noisy and clean data, i.e., P(Y = k X = æ) = -' n P(Y = = X æ²) Thus, it has been exploited to achieve many statistically consistent algorithms in noisy multi-class learning. Specifically, it has been utilized to build risk-consistent estimators via correcting loss functions [28, 33, 52], and to design Mlasifier-consiisern estimators via limiting hypotheses, e.g., [33, 9, 62]. Since the multi-label task can be decomposed into multiple conditionally independent binary classification problems, also can apply these consistent methods in noisy multi-label learning [54]. In this paper, without loss of generality, we adopt risk-consistent algorithm, i.c., Reweight [28, 52], to learn statistically consistent classifiers with estimated transition matrices. More details can be found in Appendix B.
Transition matrix estimation. As inaccurate transition matrices will degenerate the performances of these consistent algorithms, series of estimation methods [28, 51, 57, 63, 26] have been proposed for noisy multi-class learning to efficiently identify the transition matrix. However, most of them require the assumption of anchor points [33, 57, 26]. which is strong and hard to check in multi-label cases when only noisy data are provided [52]. Besides, severe positive-negativy imbalance in multi-label learning [36] will make it difficult to accurately approximate the noisy intermediate class posterior of anchor points, which is crucial for these methods. This motivates us to seek for a better estimator that can do without anchor points and avoid estimating noisy posterior in noisy multi-label learning.
3 Estimating Transition Matrices using Label Correlations
In this section, we first study the identifiability problem [30] of class-dependent transition matrices in multi-label cases. Furthermore, inspired by these results, we propose new estimator approximate the transition matrix by utilizing label correlations. is worth pointing out that our estimator demands neither the existence of anchor points nor accurate fitting of noisy class posterior.
3.1 Identifiability of transition matrix
Recently, Liu et al. [30] built identifiability of the noise transition matrix on the Kruskal's 's identifia- bility results. Inspired by them, with the complex correlations among class labels, we can get some identifiability results of the class-dependent transition matrix in noisy multi-label learning.
Following [30], to define identifiability, we denote an observation space by Ω. For general parametric space o denote the distribution (probability density function) induced by the parameter 0 E θ on the observation space Ω as Pg [2]. The identifiability for general parametric space is defined follows:
Definition 1 (Identifiability [30]). The parameter 0 is identifiable if Pa + Per, v0 + 0',
For the class-dependent transition matrix T for class j, the identifiability can be defined as the following, when T is a part of 0.
Definition 2 (Identifiability of T I is identifiable if Po + Po for O + 0', up to the label permutation of class j.
Here, the label permutation of class j means swapping 1 and 0 class values in class j. and the rows of T will also swap. Note that Ω does not necessarily include all observable variables. We use an example better understand it. For example, let l := (Y), x), 0 (T', [ and Po P X), the identifiability of 7 can be achieved with the anchor point assumption [33, 57, 26].
To use notations without confusion, for the identifiability of T using label correlations, we let Ω = (8 Y -J where -j means other classes having correlations with class j. 0 := P(Y)), -(Y''11>)) and Pa P(Ys, Y-). Also, Ω does not necessarily include all noisy class labels. We need to provide useful information to achieve the identifiability. The exploration of an effective Q and necessary conditions is one of the focuses of the paper.
3

---

Assumption 1.
- 0) < j E [g].
<1.j
Assumption means that the noisy label is agreed with the clean label on average, which is standard condition for analysis under the class-dependent transition matrix [32, 31].
Assumption 2. P(Y = 0 Y = 0) * P(Y" = Y = 1), i,j E [g] and i
Assumption 2 means that the multiple labels have correlations between each other, which is satisfied
by the most of (i pairs in the real-world dataset (see Appendix C). When considering multi-label
learning, the simplest case is having two class labels. In this case, the following theoretical results can be obtained.
Theorem 1. Two noisy labels (Yi,Y") will not suffice to identify T
This result tells us that the label correlations of two noisy labels can not offer enough information to achieve the identifiability of T We provide Theorem 2 based on the Kruskal's 's identifiability result [21, 40].
Theorem 2. Yf and Y are independent given Y three noisy labels {Y,, yE} are sufficient identify TV
The assumption that Y and y are independent given Y can be satisfied in certain cases, e.g., the occurrences of "blue" and "dolphin" may be independent given "sea" appearing or not. Nevertheless, due to the complex correlations among labels, this assumption is hard to hold in most cases. When the assumption can not hold, these label correlations are no more sufficient to determine T as shown in Theorem 3 in the following.
Theorem 3. Y and Y are not independent given Y2, three noisy labels {Y), Y y wwill not suffice to identify T
The inspiration from Theorem 3 is that, the increase of the number of noisy labels may make the identifiability decrease due to the entangled correlations (see Appendix G and I). To handle this problem, we prove Theorem 4 by assuming the transition relationship between noisy label for class i and clean label for class j is known.
Theorem 4. If P(Y Y*) is known, two noisy labels (Y),Y)) are sufficient to identify Té.
Theorem 4 theoretically guarantees that the identifiability of the class-dependent transition matrix can be achieved by utilizing the occurrence probabilities P(Ys, Y5) and P(Y YJ). Note that P(Y:, P) can represent noisy label correlations, and P(Y' Y*) = P(Y' P'Y's Y3), which imply clean label correlations. At high level, the mismatch of label correlations implied in the occurrence probabilities can achieve the identifiability.
The detailed proof of Theorem 1-4 is provided in Appendix E-H.
3.2 Our estimator
Our estimator is based on Theorem 4, which needs extra information to estimate p( YJ). Recently, the memorization effect [4] of deep networks has received much attention learning with noisy labels, which shows that deep networks will first memorize the training data with clean labels and then those with noisy labels. Prior works utilize this characteristic to develop sample selection methods [16. 13, 22, 17, 25], where select some examples with more likely clean labels for each class j respectively in the early learning phase. The selected examples can serve as the useful extra information that implies clean label correlations, through which we can achieve estimation via methods, a major concern is whether
counting. However, when implementing the sampling bias will lead to large estimation errors.
peppll--eleetton--ee
Generally speaking, according to the memorization effect, the sampling bias about selecting easy examples for class which usually means these examples have tasy-toddierrmminaee features. We can reasonably assume that given YJ, the distribution of the features about class j is biased, while the distribution of the features about another class i is unbiased, i.c.,
PDg (Y = - Ppp - ( P (X Y))d) / Ppy (Y x ) (X Pp: (Y')Y"),
(1)
where DÅ is the distribution of (X, Y.). Y (8), 82, YI, ,..., YrJ, Dÿ is the biased distribution of (X, Ya), and x' is the part of X. which represents all information about class appearing

---

not. When the assumption is satisfied, the sample selection will not lead to large estimation errors on P(Y' Y²), and it can converge to zero exponentially fast by counting [6]. In real-world scenarios, due to the complex label correlations, this assumption will not strictly hold. While, it may be roughly met when class labels and j do not share the major discriminative features. Intuitively speaking, the classifying of simplest examples for one label is not easily affected by the presence or absence of other significantly different labels. Also, since most label pairs from typical real-world multi-label datasets are significantly different (see Appendix D), the assumption can be roughly hold for those label pairs in typical cases. In Section 4.1, our empirical results justify this by showing a little gap between the estimation error of our estimator with biased sample selection and unbiased one.
Based on the above discussions, we can approximate P(Y Y*) and P( Y") with frequency counting, and utilize the mismatch of label correlations implied in these occurrence probabilities to estimate the transition matrix. In this work, we propose to estimate transition matrices {T' }) following two stages.
3=== by
In the first stage, we utilize sample selection to obtain useful information that implies clean label correlations. Specifically, we train classifier with the standard multi-label classification loss noisy training examples D, for a few epochs, and then perform sample selection to get a selected clean set D for each class label j. Specially, we commonly used sample selection way [3, 22] in learning with noisy labels, which extracts the subset of examples with small losses by modeling the distribution of losses for class j with a Gaussian mixture model (GMM).
In the second stage, we perform co-occurrence estimation by frequency counting, and then estimate the transition matrix by solving simple bilinear decomposition problem. For class label j. we first choose another class label and estimate P(Y Y2) and P(Y (Y)) via counting, i.e.,
1
P(Y' U. v.Y = k)
1 U, D" k] and (..D)ED
(2)
Σ E(En)ED
TT V,yy k] EiDDEDD Ty -
P(Y'= = yJ k))
(3)
where I[|] is the indicator function which takes 1 if the identity index is true and 0 otherwise. Then, these co-occurrence probabilities, which imply the mismatch of label correlations, can lead to four equations involving T
P ( 0, Y 0) P(Y O)T P(F = r 0) P(Y 1)T = 0 1),
P = 0, y 1) P(Y' = P(Y' = 1Y 0) + P(Y) = 1)T% = 1|Y 1),
P( Y 0) P(Y" 0)TЛ P(Y' / O 0) P(Y' P oly²' 1),
( 1, = 1) P(Y" O)T P(Y' = 1Y = 0) + P(Y = 1)71 1| = 1).
For simplicity, we denote
P(Y) 0, y 0) P(Y) 0, Y -
P(Y) = 1, Y = 0) P(P) = 1 1, -
e00 epi
E
E10 e11
P(Y) =0)
0 P(Y = 1) 11-P 0)
P
0 P
T
((Y' 0
YJ = =0) p(Y) = Y 0)
1 P_
p-
and
P(Y) = 0
Y =) P(Y) 1 = 1)
P+
1
pt)
P(YY'- 0 Y =0) P(Y'=1 ||| YJ = 0)
1-p' p'_
p-_
M
P(Y'= 0 Y 1) P(Y = Y = 1)
1 1-p-
Then the system of equations can be expressed as = =T') ( i.e.
(' ('-p-- p- p-
1 1- 0) (
1-pp p'
p-)
eoo
e01
1 1- -/-
Ein
e11
P+ 1-p+)
p
04
Denote the estimation of E, P. P, and M as
É
(on
é0ι
P
(1 1 ÿ
11 p
p . =
(1-1²- 8-_
84 1' P4..)
êι0
0 p
=
p+
p+
5

---

As E and M can be derived from Eq. (2) and Eq. (3), the problem is hence equivalent to a bilinear decomposition problem:
By solving the above matrix equation, we can get
p p'_) (énn + ê10)
(5)
p'_ -P's
and the estimation of the transition matrix
T' T'
(6)
Implementation of our estimator. The pseudo code of our estimator is described in Algorithm 1. A little difference from the above is that order make better use of correlations among labels, we perform R times co-occurrence estimation and bilinear decomposition for different classes i in the
second stage to get R estimations, T 1, 2, ..., R. Finally, we estimate the transition matrix T by Eq. (7):
R
T arg min
Σ I
T
where || |/2 denotes & norm.
Algorithm Estimating Label-Noise Transition Matrices using Label Correlations
Require:
Noisy training examples D., the number of classes q, the early warmup training epoch Ewarm. the threshold of sample selection т, and repeated estimation times It
Stagel: Standard Training and Sample Selection
1: Standard training with the standard multi-label loss for Ewarm epochs.
2: for 1, ...,, q do
3: Model losses with a trained classifier on Dj by a GMM.
4: Get the selected set DÍ for class label j with the threshold т.
5: end for
Stage2: Co-occurrence Estimation and Bilinear Decomposition
6: for j = 1,2, do
7:
1,2, R do
8: Choose another class label i.
9: Estimate P(Y', YJ) by P(Y', Y3) with Eq. (2) on D,, and P(Y' Y5) by P(P' | Y5)
with Eq. (3) on DJ.
10: Solve a bilinear decomposition problem (Eq. (4)) to get a estimation T by Eq. (6).
11: end for
Estimate T by T which has the minimum error with Eq. (7) from R estimations.
13:
end for
Ensure: The estimated transition matrices (+') DE
4 Experiments
Dataset We verify the effectiveness of the proposed method on three synthetic noisy multi- label datasets, i.c., Pascal-VOC2007 [11], Miscal-VOC200 [12]. and MS-COCO [27]. Pascal- VOC2007 [11] and Pascal-VOC2012 [12] datasets are two popular datasets for object recognition. They both contain images from the same 20 object classes, with an average of = 1.5 labels per image. Rascal-VOC2001 contains a training set of 5,011 images and a test set of 4,952 images. Pascal-VOC2012 consists of 11 11.540 images as the training set and 10,991 images as the test set [5]. As the labels of the test set in -OC2212 are not publicly available, we use the test set in Pascal-VOC2007 for Pascal-VOC2012 evaluation. MS-COCO [27] is widely used multi-label dataset. It contains 82,081 images as the training set and 40, images as the test set and covers 80 object classes with average of n. 2.9 labels per image. For these datasets, corrupted the training sets manually according to true transition matrices {T' ) For convenience,
the

---

(17 (1-1- p-
same true transition matrices for all classes, i.e., T TT
but do not divulge
D+ 1 1-
p+
this information for algorithms. We generate four different types of synthetic datasets by using different transition matrices: 1) p-p-= 0, p+ = P. which annotates some positive examples as negative examples, also known as multi-label learning with missing labels [45, 44]; 2) p- P. p+ 0, which annotates some negative examples as positive examples, also known as partial multi-label learn- ing [53, 55]; 3) p_ p. P+ P. where positive examples and negative examples are mislabeled with the same probability p; 4) p- n-na P: P+ = p. where positive examples and negative examples are mislabeled with the same number. In the experiments, we test the algorithms on various p. For all datasets, we leave out 10% of the noisy training examples noisy validation set. We use mAP on noisy validation set as the criterion for model selection.
na_
Implementation details For fair comparison, implement all methods with default parameters by PyTorch on NVIDIA RTX 3090. We use a ResNet-50 network [14] pre-trained on ImageNet [37] for all datasets, and the optimizer is Adam optimizer [18] with B = (0.9, 0.999). The batch size is 128, the learning rate is 5e-5. The number of training epochs is 20 for Pascal Niteal-VOO200///OC220 and 30 for MS-COCO. For the transition matrix estimation method, Ewarm is the same the normal training epoch. For our estimator, we perform sample selection based on the average losses epochs before a certain warmup epoch (10th epoch for Pascal-VOC207//0C2000
15th epoch for MS-COCO), R = - 1 and = 0.5 in all experiments. All experiments are run at least three times with different random seeds, and we report the average and standard deviation values of results. The best results are in bold, and the second- acond-best results are in blue.
4.1 Comparison for estimating transition matrices
Baselines For evaluating the effeetiveness of estimating the transition matrix under multi-label cases, compare the proposed method with the following methods: (1) T-estimator max [28, 33], which estimates the transition matrix via the noisy class posterior probabilities for anchor points that have the largest estimated noisy class posteriors. (2) T-estimator 97% [28, 33], which selects the points with 97% largest estimated noisy class posteriors to be anchor points. (3) Dual T-estimator max [57], which introduces intermediate class to avoid directly estimating the noisy class posterior, and selects the points with the largest estimated intermediate class posteriors to be the anchor points. (4) Dual Teestimator 97% [57], which selects the points with the 97% largest estimated intermediate class posteriors to be the anchor points.
Metrics We use the of estimation error for the transition matrices the estimation evaluation metric, i.e. Σÿ= I - ||//|[]' 111
Results In Tab. 1, and 3, we can that for all cases on three datasets, the proposed estimation method leads to the smallest or second-smallest estimator errors across various noise rates. Note that since the fitting of noisy or intermediate class posterior is hard be accurate in noisy multi- label learning. T-estimator and Dual T-estimator need to carefully tune a hyperparameter for better estimation under different noise rates, and it's very sensitive in some cases, e.g., MS- COCCO datasets with noise rates (0.1, 0.1). In contrast, our method uses the same hyperparameters on one dataset to get good results for all cases, which reflects robustness to various noise rates. Besides, to study the ablation of sampling bias, we also run our method with an unbiased sample selection, named "our estimator gold". We can see that sample bias is the main factor that contributes the error for our estimator, but the little error gap between our estimator and our estimator gold shows it will not lead large estimation errors.
Table 1: Comparison for estimating transition matrices on VOS220- dataset.
Noise rales (p-,
(0.0.2)
(0.0.6)
(0.2.0)
(0.6.0)
(0.10.1)
(02.022)
(0.017.0.2)
(0.034.0.4)
T-sstimator max
3.8920.03
10.52=0.58
3002
4.4
31820.22
5.28 20
30990.100
6,2840.44
T-estimator 97%
4.95
4.4220018
1.722003
2.13:0012
6.99110.10
6.94:00.17
5,38
5.1740009
Dual T-sttimaaor max
1.94.10.13
7.2910.16
1.03
2.68.1..33
2.13
4.0210.188
1.71
2.67 1.0.7
Dual T-estimator
12.59-10.06
7.4320.06
1.09=0.03
2411:0.3
14.3910.10
11.7840.06
13.71=0.16
11.15=0.09
Our cstimator
1.51=0.12
2.30-0.13
0.37=0.08
12430033
3.06::038
3211-032
2.03:0119
1.8410.32
Our estimator gold
0.44#0.05
0.51+0009
0.38=0.08
0.37=00111
0.83+0005
2.15+:0.00
0.65
1.40+:0.00
7

---

Table 2: Comparison for estimating transition matrices on Pascal-VOC2012 dataset.
Noise rales (p-, p+)
(0.0.2)
(0,0.6)
(0.2,0)
(0.6.0)
(0.1.0.1)
(0.2,022)
(0.017,0.2)
(0.034.0.4)
Testimatio max
3.9010.01
10.28_10.33
2.87
4.55.10.08
3.2910..7
5.2510.15
4.05
6.2210.20
T T-estimatoe 97%
5.42:+0.09
3.98
0.53
1.9140.07
6,43-10.16
6.20-40.11
5.76:10.27
5.16:10.114
Dual T-estimator max
1.02=0.20
5.13+0.26
1.07=0007
2.06+0.12
1,9440005
2.59+0.10
1.1740.13
193+0008
Dual T-estimator 97%
12.94=00006
7.49+00.3
1.14=0.04
2.94=:0.18
14,23+0.08
11,56+00.05
13:97=0009
11.10=0.08
Our estimator
0.8320.10
0.9420.15
0.26
0.91:20.12
1.74:0.22
1.7910.17
0.94
1007:0.1
estimator gold
0.33
0.34.10.05
0.25
0.45.10.5
0.51100.05
1.6710.29
0.42.10.06
0.9110..16
Table 3: Comparison for estimating transition matrices on MS-COCO dataset.
Noise rates [p-. p+)
(0.0.2)
(0.0.6)
(0.2.0)
(0.6.0)
(0.1.0.1)
(0.2.0.2)
(0.008.0.2)
(0.015.0.4)
T-sstimator max
06.14=0.33
39.09=0.47
10.39+0.221
11.49:0.60
13.95:0.41
20.50:10.04
16.70-10:06
28.16:20:45
F-sstiator 97%
50.49::001
25.70:10.08
4.00:10.08
3.70:10.02
51.17:10.16
39.45:1.111
49.96:10.18
37.54:10.10
Dual T-estimator max
5.04=0004
11.22:-0.0
4.65+0.07
9.55+0084
13.02:0445
15.79+0.38
7.04=0.31
6.34-0.11
Dual F-stimator 97%
61.49:0.02
30.97:00.03
1.53+00000
7.86-20.12
64.20+1000
48.67400.01
63.12-40.02
46.91+10011
Our estimator
7.42
11.23:0.111 11
0.50
0.3320006
8.8:20.10
10.27::0119
7.31=0..3
8.77=0200
estimator gold
0.82+0003
0.80=0004
0.40::0.04
0.66::0.04
1.94:0.04
8.14:10.06
0.95=0005
2.02=00.88
4.2 Comparison for classification performance
Baselines We exploit 10 baselines: (1) Standard, which trains with a standard multi-label classi- fication loss. (2) GCE [60]. which proposes a Generalized Cross-Entropy loss for robustness, (3) CDR [49], which performs different update rules for two types of parameters to achieve robust learning. (4) AGCN [58], which adopts dynamic GCN to model the relation of content-aware class representations. (5) CSRA [61], which generates class-specific features for every category by proposing spatial attention score. (6) WSIC [15] that proposes to use small set with clean labels to learn a residual net for regularization in noisy multi-label learning, and we only provide noisy datasets to it for a fair comparison. (7) Reweight-T max, which learns with a reweighting algorithm using transition matrices estimated by T-estimator max [33]. (8) Reweight- 97%, which learns with a reweighting algorithm using transition matrices estimated by T-estimator 97% [33]. (9) Reweight-Duai DualTT max, which learns with a reweighting algorithm using transition matrices estimated by Dual T-estimator max [57]. (10) Reweight-Duale DualT 97%, which learns with a reweighting algorithm using transition matrices estimated by Dual T-estimator 97% [57]. Note that Standard, AGCN, and CSRA are designed for clean multi-label data, and GCE and CDR are designed for noisy multi-class learning.
Metrics Following conventional setting [11, 7, 36, 35], we compute the mean average precision (mAP), overall F1-measure (OF1) and per-class F1-measure (CF1) as classification evaluation metrics. For each image, we assign a positive label if its prediction probability is greater than 0.5.
Results As shown in Tab. 4, 5 and 6, first, we can find those statistically consistent methods achieve the best second-best results on all three metrics in the vast majority of cases, while other methods can only achieve good results in some cases. For example, on the Pascal-VOOe VOC2012 dataset with noise rates (0.0, 0.6), CSRA achieves the best result in the mAP metric with the help of its well-designed network, but its performance is far lower than those consistent methods on the OFI and CF1 metrics, which shows the learned model can not approximate well the true class posterior P(Y X). Note that since network structure and loss correction are compatible, the risk-consistent methods can also help AGCN and CSRA perform better (shown in Appendix M). Second, theoretically, the more accurate the transition matrix is estimated, the more likely the consistent method is to achieve better results by increasing the size of noisy examples, and our experimental results verify this. Among those consistent methods, the reweighting algorithm with our estimator (named "Reweight-Oue Ours") obtains the best or second-best results on the three metrics, which is due to the smaller error of our estimation. Especially on the challenging and large-scale MS-COCO dataset, Reweight-Oun outperforms almost all state-of-the-art methods on the CF1 metric and significantly surpasses other baselines by a large margin in some cases. For example, on the MS-COCO dataset with noise rates (0.6. 0). Reweightttteeeeee Ourg achieves the best CFI result (58. 63..63+12 1.30), while the suboptimal result is 56.70 I, and the result of Standard is only 7.07 In addition, the ablation studies about loss correction ways and base learning algorithms are provided in Appendix L and M. which shows that estimator achieve much better performance with the advanced frameworks.
Besides, although our method is based on the assumption of class-dependent label noise, the experi- ments with two types of destance-dependee label noise are provided in Appendix O.
8

---

Table 4: Comparison for classification performance on RBS2a0-VOC2007 dataset.
Noise (0_ .P4
(0.0.2)
(0.0.6)
(0.2.))
(0.6.))
(0.1.0.1
(0.2.0.2)
(0.017.0.2)
(0.034.04)
Standard
84,25=:1007
77.16
52.70=0.54
1.57
83,07=0.45
78.87=0.22
85.9240059
80.9700.22
GCE
83.85:+:1099
73.324...2
83.03+00551
67.47:11.74
83.68:20.66
79.39=0.95
84.40-10.34
80.68:00.22
CDR
84.60.10.43
77.4511.23
82.7610..3
68.86....05
83.22.10.57
79.02.10.2
84.37:10.25
81.14:10.28
AGCN
83.24:1:0.77
75.50:1..56
81.09:1:51
16.47:11.29
81.09:10488
0..7
82.21:10.42
76.55:11.11
mAp
CSRA
85.11=00.51
1..2
82,93+0.65
67.36+22.25
83.69-+0.69
18.10=0.53
04.94
81:1+0001
WSIC
84.14::0.26
76.17
82.30:0.64
66.82+33.87
83.41=00.31
77.93=1000
84.17:
80.74:0:44
Reweight-T T max
84.20:10:46
76.97:11.20
83.04:0.399
71.36.12.47
83.48:10.15
79.10:0.52
84.06 0.224
81.01
Reweight- 97%
84.00=00668
78.97+0.69
83.07=0.29
73.96=1.69
02.71=0.30
78.80=0.28
84.37=00.22
81.42+00225
Reweight- DuafT max
84.46:1:0.20
77.65+:1.06
83.75:40.44
13.75+:1.61
83,94:10.31
79.48=11.24
84.60-10.30
81.77:10.26
Reweight DualT 97%
82.36.10.45
10.73
84.56
10.40
2..1
1..69.1..4
75.26
1.70
8..8
77.40:1
Keweight- Ours
3445:0046
78.7230.41
84.08:00224
14.46:10.56
84.03:10.299
80.44
0.52
84.09
809711000
Stundard
75.24.11.4D
32.02.1.5.49
78.85.10.33
15.08
79.24.10:43
75.85
1..04
59.67.111
GCF
16.17::1.57
36. 13:-4.4
79.28:10.44
04.25
79.73-10.70
76.57-00555
76.80-1-0.88
2..43
CDR
76.05=0.68
34.11+3343
09.04
14.99=0..19
79.34400.0
76.00=0.47
0..56=0.52
59.31
AGCN
74.92:1:.02
30.97:13.78
75.45:12006
16.55
78.69.10.31
7..64
75.16:10.58
56.56111.66 1.64
CSRA
7.
33.65-22.73
17
15.94
80.36:10.53
76.92=00.34
07.91+0.63
62. 19-1199199 97
OF
WSIC
75.01=1118
16,4846.78
79.02
04.8
18.55+11005
3..88+3.44
72.30-+2.882
53 53.26+99.44
Reweight- T max
76.97:10.45
41.541:2.64
79.65:0.44
47.68
80.00-1:0.27
73.58
76.94:10.37
66.77:10.99
Reweight: T 97%
77.7111..5
68.28.12.03
80.16.1024
70.67.00.70
75.28.10.97
65.011..220
78.451.0.3
1.0667
Reweight-Duin DualT max
78.38:1:.41
68.81
80.02+1
16.41
14
79.87=00.27
65.36-2..1
47.04 78.55-00.36
Reweight- Draligh--DaaTT 97%
68.1743.53
61.81+3.29
80,74+00.00
72.53+22.65
$2.99-:5500
36.75=6.71
67.55++0.64
50.41
Reweight-
78.2:00.88
65.6811
80.810025
29.04:10.29
16
09.25
74.3
Stundard
72.53:11111
30.64
76.83:00.65
14.97.0..24
15.86.1..23
70.68
1.76
03.11.10.54
52.07.2..4
GCF
73.10::1.27
33.07:14.65
77.25410.6
14.77-10.20
1.573::1.57
71.24+1.42
73.37.40.98
56.89.12.844
CDR
73.08:00.47
33.06
76.95=0079
04.88=0.11
76.09=
70.78+1.00
03.33-+0.85
54.16=3.119
AGCN
73.45=+1.04
13,41+1..5
97
16.67:10.55
76.20:100.1
69.09=0.49
1..81+1002
55.09=
CSRA
74.10:10.56
33.441:3.65
15.2811.32
15.711.0.23
77.52:10.94
03.44:0062
74.98
58.60:12.24
WSIC
70.13+2.004
13.64:6.97
75.32+2..00
14.77=00.16
14.17=1187
64.95=6.47
65.411+4.40
43.43=11.31
Reweight- I max
74.05:10.51
39.37::1.26
07.28
50.351.5.52
07.20:10.39
73.32:0.47
74.08
68.99:11.0
Reweight T 97%
76.81
1.33
77.22.10.56
1.7
14.99.10.37
68.07.00.74
17.62
74.26
Reweight-DualT max
75.27400.6
45.31
76.56:1180 80
63.99-10.42
77.27440:0
69.48-4.48
05.06
68.32.11.22
Reweight-DuafT DualT 97%
71.93+1664
63.85++..69
77.95=00.79
08.44+2.93
62,84+2.51
50.49+2.43
70.99+0.98
1.67
Reweight-Ours
76.86:10.48
61.29:11.94
77.89:0.22
66.79:12.50
78.04:10.40
0..0
07.28:10.48
72.18:20.74
Table 5: Comparison for classification performance on Pascal- VOC2012 dataset.
Noise rales (p=
(0.0.2)
(0.0.6)
(0.2,0)
(0.6.0)
(0.1.0.1)
(0.2,0.2)
(0.017.0.2)
(0.034.0.4)
Standard
33.97:0000
00.02
15.70::0.19
16.13.0.86
85.91=00110
02.4
06.05:00224
02.1=0074
GCE
86.02:1:0.21
78.71
85.96+10.19
75.02-0..00
86.29-10.15
03.18-1..3
85.84:10.37
82.96=0.83
CDR
86.09-40.14
80.52+1.61
85.61+0.188
1..53+11226
86.01=00.19
82.79=0.42
85.9240.38
83.48=0.62
AGCN
85.16::0.12
79.67-20.43
84.9110038
07.54:0.29
84.69-10.30
80.15
84.75:10.11
01.85=0.35
mAP
CSRA
86.88.10.23
81.75.19.97
85.39110.7
75.08
86.14.10.14
80.98
86.51.10.15
83.86.0.00 0.49
WSIC
86.39:10.38
80.75:10.49
85.53-10.28
77.00
85.671.0.17
8..01
86.07-10.22
03.19.0.19
Reweight- T max
85:40:::0:33
79.06=11.60
85.80:00.32
78.59
85.98:10.32
02.88-044
85.51=00.11
82.38
Reweight-11 97%
85.97:10.27
01.04
80.12+0.75
85.66:10.55
82.110052
85.99-10.51
83.28=1007
Reweight-Dun max
85.93-10.41
78.69-+2.62
86.47-10.26
00.00-10.66
86,23--0.34
84.21
86.
33.42=11.26
Reweight- Dualt 97%
83.93:40.52
19.97+1142
86.37:0.225
01.84:0.87
82.39:100.85
1..61=1001
83.33++140
10.95=11.15
Reweight- Ourrs
86.01300554
80,3311.85
86.12:0.10
$0.3411.30
06.23-10:28
83.42:0.5
85.92:10:42
83.56=1.31
Standard
71.91::0200
27.84441177
80.4720.44
14.93:0.228
81.27:10222
08.31=0.13
77.83200336
61.67=2.10
GCE
78.25::0.28
80.89:10.19
14.73-10.28
81.47-10.34
78.78:00111
78.32-10.37
63.26=11.40
CDR
78.02.10.23
29.35.16.39
80.65110.36
14,91 1.2.6
81.34.10.35
1.114 78.58
17.32
1.994
AGCN
76.11::0.66
31.06:44.73
80,4910.51
15.42-0.33
80.28:100.4
76.02=1..33
15.83
61.59::4.20
CSRA
78.61100.56
36.26
79.8810.57
15.7110.50
0..8
78.1211.11
79.05
61.77
WSIC
78.56.10078
27.9011..72
80.5611032
15.09.10.24
80.98.1049
77.36 1.51
78.52:10.65
47.56
Reweight- T max
07.76=00.72
43.96.26.52
81.2441031
48.88-2455
81.53+4.49
08.6
78.66:10.22
68 2.70
Reweight-1 T 97%
79.31::0.20
71.7942..35
81.56:00299
18.10=1181
17.81::009
71.28=1.99
79.20-10..35
73.90=1.81
Reweight DualT max
80.44-10.99
63.68115.755
74.1712.34
81.0410.336
78.71
80.47:1.335
Reweight DualT 97%
60.81111.36
58.42112.41
82.20-10.20
1.49
56.42.1.122
30.28 1.82
14.36.1:142
58.49 1.73
Reweight-C Ours
80.54:0.61
64.08
81.77
35.38
81.11:0.556
77.78-0.7
00.75
77.31= 1.49
Standard
73.15:50.22
29.76:3.17
00.3
14.87+0.29
79,05=50.26
70.51=0.22
70.07:0.44
00.32=1.99
6CE
75.25:10.84
26.25:14.44
79.68:1.32
14.69:10.27
79.13:1:0.32
7..54
76.05-1:0.84
61.99=0.71
CDR
75.32.10.66
79.20.1.20
10.25
79.14.10.45
75.56
76.1111.44
6..02
AGCN
74.16:11.06
33.33444.55
88.1100.12
15.42-0.76
78.09-10.99
73.28 1.80
13.91
58.57-4.677
CSRA
75.91=00.8
32.8344.888
45.48+0.49
79.87+0.26
1.12
76.40=+0.73
59,53-3.12
WSIC
1.39
30.19-1..09
79.49:0.57
14.98-0,22
78.50-10.91
74.10=2..39
76.05=20.90
55.11=4.30
Reweight- T max
75.54:1:0.59
13.66-45.11
79.79-10:7
49.60-13.31
79.60::0.88
76.45=0.29
76.82-20:42
63.53-2.66
Reweight- 97%
78.9410.10
1.16
79.73::029
1..38=1441
17.54+1088
1.539=1.57
78.78-10.35
7..54=1.13
Reweight Dualf max
78.37:10.54
58.96.1.3.99
80.06:10.22
1..61
79.29-10.35
76.37=144
78.69-10.25
1,37
Reweight DualT
67.96.11.13
61.89.12.99
80.11-10333
3..57
65.09
1.03
69.26
63
Reweight-Ours
788140053
66,76-43.38
79322003
72.8442.16
79.90-10.39
75:00=081
19.37
75. 1644

---

Table 6: Comparison for classification performance on MS-COCO dataset.
Noise (o- .P4)
(0.0.2)
(0.0.6)
(0.2.0)
(06.0)
(0.1.0.1)
(0.2.022)
(0:008.0.2)
(0.15.0.4)
Standard
69.9220000
63
60.77
55,45=0.48
67.77+0.28
02.50=0.23
69.76-0.09
0688==005
GCE
69.90=4:005
62.58+0.17
67.32440.11
54.01=00700
68.62:40.16
03.21=0035
69.99+0011
66.72:00.19
CDR
70.06.10.05
63.8510.28
67.32.10.8
1..20.1..2
68.01110.08
62.6510.21
69.87:100099
66.85:1.119
AGCN
71.48:10.14
68.75:10.22
69.44:10100
55.7110.61
09.42:10.23
63.96-0.111
70.90:10.13
0..86:10.26
mAp
CSRA
71.1::00100
65.2840.11
67
51,49-0.33
02.8340.122
61.80+0.98
70.76:00.16
68.02=0.15
WSIC
68.92:+0.09
63.09:0.28
66.2240.06
53.61=0..6
67.41=00155
62.33=0.18
68.95=00.15
06229=0.21
Reweight-T T max
69.99:10.18
63.9410.11
67.40:0.13
58.274:.255
67.85
0.128:0.12
69.76110.07
66.24:0.51
Reweight- 97%
67.98+40.57
62.52+0.46
68.00:0.17
59,44+081
65.69+0.48
60.03=0.11
68.13++0.04
64.40=0.18
Reweight- DuafT max
67.57:40.21
60.39:10.53
68.57:10.25
08.42-0.22
68.01
0..17
68.76:1:0.08
05.75=
Reweight DualT 97%
64.97.10.20
58.85
69.68
10.25
4..17
16.66
49.41.10.3
63.27110.36
0.886
Keweight- Ours
10.7:0
63.28
69
00.0
15
64.4610.10
70.06:0.00
67.032008
Stundard
66.48.10.50
19.18.10.97
1.00
68.64
0.54
66.07.0.115
5..7010.42
GCF
66.67:1:0.38
19.61
69.82:10.25
0.03
0.
04.98
66.58:1-0.32
0..04=0.51
CDR
66.46400554
19.60+2..8
69.72:40.31
0.00
68.75=00.09
64.68=0.54
66.03+0.399
52,49=1.18
AGCN
67.14:10.52
16.02:10..6
70.63:10.12
7.0410.02
69.66.10.25
6.07
66.61-1:0.48
52.38-1.05
CSRA
67.98:+0:40
24.08+2..00
70.14:0.117
7.06
69.70:10.31
64.89
67.37-+0.28
01.81
OF
WSIC
66.67=:0.15
23.02444.0
69.02+0.066
7,02+0000
67.78+00558
62.31=0.69
66.38:00.25
12,07=1944
Reweight- T max
67.13:100.1
39,47:1147
69.1410.111
19.26.11.76
64.03:12.00
57.68
66.45:1:0330
13.63-11.28
Reweight: T 97%
57.66.10.56 54.06.11.19
69.72.1.39
64.79.0.85
43.81110554
33.65
55.44.10.56
51.78
Reweight-Duin DualT max
65. 65.22:1:0.28
56.01
70.16+0.26
56.90
59.62 1.59
49.79-0.188
65.64440.71
61.73=2.11
Reweight- D 97%
29.39:40.36
29.30+0.74
70.24+0022
48.87=6687
25.83++0116
8.51=00.26
39,29++0445
35,78=1.10
Reweight.
70.10:00.10
61.74100.64
205
08.7810.56
64,4510477
38.60
69.40
0.49
Stundard
60.27:10.52
22.7310.115
64.66.1..22
1.00.10.02
62.38.10.27
56.7811.31
60.04110.17
45.35 0.60
GCF
60.76:1:0.08
21.06.:1.12
65.27:10.22
7.04
63.60:10.25
1.57
60.66:1-.19
44.28
CDR
60.26::0.55
12.42+
65.27=+0.10
0.07
62.63-+0.08
16.41
59,85-+0.38
05.41=0.85
AGCN
61.79:+0.98
19.30+
66.29:00111
0.05
64.09:1:039
58.97=1155
60.35++0.54
43.89=11446
CSRA
62,46:100.33
24.17:12.76
65.801:0002
7.06.00022
63.90:10.48
25.97
61.30:1::442
44.25
WSIC
61.12:+0.08
27.16:
63.71#0034
7.03++0.01
11.12=1.30
52.47=0.99
60.51+:0.25
43.52=
Reweight- I max
61.59:10.51
32.78+0..00
64.82::0224
06.68100.0
63.35:10.16
59.9310.50
60.92:10.008
08.3333.33
Reweight T 97%
55.67.10.45
52.79.11.04
63.97.10774
56.70.1.11
48.47.10.70
40.20.10.22
55.33.10.12
19
Reweight-DualT max
64.79:40.23 52.16442.06
63.5140.35
1.12
66.29-10.34
61.33++0009
68.0022.2
08
Reweight-Dial DualT 97%
32.15=00.33
30,32+0.91
65.23++0.28
83.76+8888
29.90+0.110
12.77=0.21
41.29+
37,08=1.29
Reweight-Ours
67.18:10.17
57.46:10.52
65.2210.49
58.63+1.30
06.65::0.15
61.13=1001
66.42:1:0.10
62.94:00.28
5
Limitations
This work still has certain limitations, including: 1) This work exploits the memorization effect [4] in deep learning perform sample selection, while the memorization effect has not been found in other traditional machine learning methods, and therefore, the proposed estimator can not be applied to such learning methods. 2) This work estimates occurrence probabilities using frequency counting. Although this estimation eтror will converge to zero exponentially fast [6]. when the number of one label appearing too small, c.g., less than 50, the estimation of the transition matrix for this class label is still difficult to be accurate. 3) Since our work assumes label noise is class-dependent but instance-indepente independent, when this assumption does not hold, the estimation is not guaranteed. The discussions about the relaxation of ristance-independee assumption can be found in Appendix N, which reveals its applicability in certain typical cestance-dependee cases.
6 Conclusion
In this paper, we study the estimation problem of the transition matrices in the noisy multi-labe setting. We prove some identifiability results of class-dependent transition matrices in such a setting, inspired by which we propose utilizes the information of label correlations, and demands neither anchor points nor accurate fitting of noisy class posterior. Experiments on three popular multi-label datasets illustrate the effectiveness of the proposed estimator to accurately estimate transition matrices, and the consistent algorithms with this estimator achieve better classification performance.
estimator to approximate the transition matrix. The proposed estimator
Acknowledgements
Yibing Zhan was partially supported by the Major Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Key Project (No. (021200111700) and the National Natural Science Foundation of China (No. 62002090). Shiming Ge was partially supported by grants from the Beijing Natural Science Foundation (1.192040, and National Natural Science Foundation of China (61772513). Xiaobo Xia was partially supported by Google PhD Fellowship and Australian Research Couneil Project DE- 19010147. Tongliang Liu was partially supported by Australian Research Council Projects DP180 PP180103424, DE-190101473 1C-190100031, DP-220102122 and FT-220100318.
10

---

References
[1] Hierarchy for the 600 boxable classes,
html.
[2] Elizabeth S. Allman, Catherine Matias, and John A. Rhodes. Identifiability of parameters in latent structure models with many observed variables. Annals of Statistics, 37:3099-3132, 2009.
[3] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Un- supervised label noise modeling and loss correction. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, pages 312-321, 2019.
[4] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In /CML, pages 233-242, 2017.
[5] Hong-Yu Yu Zhou Bin-Bin Gao. Learning Discover Multi-Class Attentional Regions for Multi- Label Image Recognition. TIP, 10:5920-5932, 2021.
[6] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities nonasymptotic theory of independence. In Concentration Inequalities, 2013.
[7] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In CVPR, pages 5177-5186, 2019.
[8] De Cheng, Yixiong Ning, Nannan Wang, Xinbo Gao. Heng Yang. Yuxuan Du, Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization, In NeurlPS, 2022.
[9] Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependee label noise: A sample sieve approach. In ICLR, 2021.
[10] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7:1-30, 2006.
[11] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.paccal html, 2007.
[12] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal
setwokk.ngcchalengssWooooooo2 o/o.://eeeeeeoooooooooo/o/oo/oro htmml 2012.
[13] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pages 8536-85444 8546, 2018.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[15] M. Hu, H. Han, S. Shan, and X. Chen. Weakly supervised image classification through noise regularization. In CVPR, pages 11517-11525, 2019.
[16] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks corrupted labels. In ICML, pages 2309-2318, 2018.
[17] Tachyeon Kim, Jongwoo Ko, Sangwook Cho, Jinhwan Choi, and Se- Young Yun. FINE samples for learning with noisy labels. In NeurlPS, pages 24137-24149, 2021.
[18] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun, editors, ICLR, 2015.

---

[19] Ivan Krasin, Tom Duerig, Neil Alidrin, Vittorio Ferrari, Sami Abu-EI-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont- Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng. Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https///ttrrgeegoooggappiscss www./ppenningggg/////eeeeeeeeeee
2017.
[20] Wilhelmus Petrus Krijnen. The analysis of three-way arrays by constrained PARAFAC methods. DSWO Press, Leiden University, 1993.
[21] Joseph B Kruskal. Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Lineur algebra and its applications, 18(2):95- 138. 1977.
[22] Junnan Li, Richard Socher, and Steven C. H. Hoi. DivideMix: learning with noisy labels as semi-supervised learning. In ICLR, 2020.
[23] Shikun Li, Shiming Ge, Yingying Hua, Chunhui Zhang, Hao Wen, Tengfei Liu, and Weiqiang Wang. Coupled-view deep classifier learning from multiple noisy annotators. In AAAI, pages 1667-4674, 2020.
[24] Shikun Li, Tongliang Liu, Jiyong Tan, Dan Zeng, and Shiming Ge. Trustable co-label learning from multiple noisy annotators. TMM, 2022.
[25] Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In CVPR, pages 316-325, 2022.
[26] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In ICML, pages 6403-6413 2021.
[27] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
[28] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. TPAMI, 3833)4447-461, 2016.
[29] Weiwei Liu, Xiaobo Shen, Haobo Wang, and Ivor W. Tsang. The emerging trends of multi-label learning. TPAMI, 2021.
[30] Yang Liu, Hao Cheng, and Kun Zhang. Identifiability of label noise transition matrix. CoRR, abs/2202.02016, 2022.
[31] Aditya Krishna Menon, Brendan van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class- paasabrroayyility estimation. In ICML, volume 37, pages 125-134, 2015.
[32] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurlPS, volume 26, 2013.
[33] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pages 2233-2241, 2017.
[34] Cosmin Octavian Pene, Amirmasoud Ghiassi, Taraneh Younesian, Robert Birke, and Ly- dia Yiyu Chen. Multi-label gold asymmetric loss correction with single-label regulators. ArXiv, abs/2108.02032, 2021.
[35] T. Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben-Baruch, and Asaf Noy. MI-decoder: Scalable and versatile classification head. arXiv, 2021.
[36] Tal Ridnik, Emanuel Ben Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor Asymmetric loss for multi-label classification. In ICCV, pages 82-91, 2021.
12

---

[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-255 2015.
[38] Eric Arazo Sanchez, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, 2019.
[39] Jun Shu, Qian Zhao, Zongben Xu, and Deyu Meng. Meta transition adaptation for robust deep learning with noisy labels. arXiv preprint arXiv:2006.05697, 2020.
[40] Nicholas D Sidiropoulos and Rasmus Bro. On the uniqueness of multilinear decomposition of n-way arrays. Journal of Chemometrics: A Journal of the Chemometrics Society, 14(3):229-22 2000.
[41] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: refurbishing unclean samples for robust deep learning. In ICML, pages 5907-5915, 2019.
[42] Jos MF Ten Berge and Nikolaos Sidiropoulos. On uniqueness in candecomp/parafac. Psy- chometrika, 67(3):399-44) 409, 2002.
[43] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In ICLR, 2022.
[44] Baoyuan Wu, Fan Jia, W. Liu, Bernard Ghanem, and Siwei Lyu. Multi-label learning with missing labels using mixed dependency graphs. IJC,, 126:875-899 896, 2018.
[45] Baoyuan Wu, Zhilei Liu, Shangfei Wang, Bao-Gang Hu, and Qiang Ji. Multi-label learning with missing labels. ICPR, pages 1964-1968, 2014.
[46] Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In ICML, pages 1085-112955 2021.
[47] Zhengning Wu, Xiaobo Xia, Ruxin Wang, Jiatong Li, Jun Yu, Yinian Mao, and Tongliang Liu. Lr-svm+: Learning using privileged information with noisy labels. IEEE Transactions on Multimedia, 2021.
[48] Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, Yinian Mao, and Tongliang Liu. Extended T: Learning with mixed closed-set and open-set noisy labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[49] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang. Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2021.
[50] Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In ICLR, 2022.
[51] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. NeurIPS, 2020.
[52] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurlPS, pages 6835 6846, 2019.
[53] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning. In AAAI, 2018.
[54] Ming-Kun Xie and Sheng-Jun Huang. CCMN: A general framework for learning with class- conditional multi-label noise. TPAMI, 2022.
[55] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning with noisy label identification. TPAMI, 1487776-3687, 2022.
13

---

[56] Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance- dependent label-noise learning under structural causal model. In NeurlPS, 2021.
[57] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual T: reducing estimation error for transition matrix in label-noise learning. In NeurIPS, 2020.
[58] Jin Ye, Junjun He, Xiaojiang Peng, Wenhao Wu, and Yu Qiao. Attention-driven dynamic graph convolutional network for multi-label image recognition. In ECCV, 2020.
[59] Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In ICLR, 2021.
[60] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurlPS, pages 9778-8788, 2018.
[61] Ke Zhu and Jianxin Wu. Residual attention: A simple but effective method for multi-label recognition. In CVPR, pages 184-193, 2021.
[62] Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In CVPR, pages 101 13-10123,, 2021.
[63] Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability alternative to anchor points when learning with noisy labels. Marina Meila and Tong Zhang, editors, ICML, pages 12912-12923, 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No] or [N/A] . You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section 2.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/amswers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix Q
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix E-H.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.)? [Yes] We provide the source codes at http:///gittbbboomm/mml///lluttl
14

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] We report the standard deviation value of results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 4.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you' using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "pGYGI3H1HhB", "reviewer": "Reviewer_KMFh", "summary": "\nThis paper presents a way of estimating the noise transition matrix for noisy multi-label learning, which makes use of label correlations through a bilinear decomposition based on frequency counting. It first theoretically studies the identifiable problem of estimating the noise transition matrix under the multilevel setting,  which then leads to the development of the bilinear decomposition-based method for estimating the noise transition matrix.  Experiments results derived from several image datasets show that the proposed method can better estimate the transition matrix.", "questions": "1. Label correlation plays an important role in the proposed estimation. Even though assumption 2 is validated in the appendix. Could the authors empirically show this role with some examples from the experiments? For example, labels can be strongly correlated and or weakly correlated. Is it possible to study the association between the strength of label correlation and the model performance? \n2. The server imbalance issue is indeed a challenging problem in multi-label learning, which I agree with the authors. I am wondering if the proposed method can deal with this issue? If so, how?\n3. Other common performance metrics used in multiple-label learning are the ranking-based metrics, like P@K, R@K, NDCG@k, etc. Could the authors consider one of the metrics, say NDCG@K? \n4. How will the threshold $\\tau$ affect the model performance?\n5. Given the mixed performance of the proposed method,  Is there any consistent takeaway message here? I.e., In which circumstance the proposed method can be applied and perform better?\n6. What is $n_a$ in computing $\\rho{-}$ in line 222?", "limitations": "The authors have addressed the limitations and border impact in the appendix. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nLabel-nosing learning is an important and challenging problem not just in multi-class classification and also in the multi-label setting.  This paper presents a bilinear decomposition-based approach based a simple frequency counting with a clearly described algorithm. \n\nThe proposed estimation method is well motivated with a set of theorems that study the identifiable problem in the multi-label learning setting, which seems to be thorough with proper proof in the Appendix. The proof is also easy to follow. \n\nOverall, the paper is written well, and most of the notations were clear and well-organized to guide the readers through the text. \n\n\nWeaknesses\n\nMy major concern is around the performance of the proposed estimation method.  In terms of estimation of the transition matrices, it outperforms the competitors in most of the settings, which is good. However, when coming to the classification performance, the proposed method is not a clear winner, particularly compared with those without considering noise labels across the datasets. For instance, for varying levels of noise, it is hard to conclude the behaviour of the proposed method. \n\nAblation studies showing how the proposed model benefit from label correlation and deal with imbalance issue are missing. For example, there is a running parameter $\\tau$ used in Stage 1 to select the sample set $\\mathcal{D}^j_t$, which I believe will have an impact on the performance of the proposed method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ScNtecbcONF", "reviewer": "Reviewer_9BSQ", "summary": "The paper proposes a new estimator to estimate the transition matrix in noisy multi-label learning. The main idea of the estimator is to utilize the clean label co-occurrence as a constraint to help the estimation of the transition matrix. Specifically, authors derive a two-stage method to estimate then transition probabilities. At the first stage, the paper utilizes an existing technique to select clean labels for estimating the label occurrence. Although the estimated label occurrence is biased due to the selection bias, authors claims that the selection biased cannot lead to large estimation error. At the section stage, the paper obtains the label co-occurrence with frequency counting and use the mismatch of label correlation to estimate the transition matrix.", "questions": "How to determine the threshold \\tau, especially without the clean validation set?", "limitations": "None", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. The motivation of the proposed method is reasonable. The information of label co-occurrence is useful for the transition matrix estimation.\n\n2. The paper reports extensive experiments to validate the effectiveness of the proposed method.\n\n3. The paper conducts comprehensive analyses for noisy multi-label learning.\n \n\nWeakness\n\n1. In section 3.2, authors claim that the label co-occurrence estimated by using selected clean labels is unbiased. This is based on the assumption that “given Y^j”, the features about class j is biased, while the features about another class i is unbiased. The assumption is unreasonable, since the features are biased with respect to class i and class j co-occur in an image.\n\n2. Authors select a small number of clean labels for estimating the label co-occurrence. However, the estimated label co-occurrence may be imprecise due to the insufficient labels used for frequency counting, especially for positive labels.\n\n3. There are many typos, such as page 4 line 168 Ys; Algorithm 1 line 4 Dt; page 2 line 48 “while “bird” and “sky” are always co-occurrence”.\n\n4. The references are not cited properly. In experiments, the citations for multi-label with missing labels and partial multi-label are missed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "JKzoCZfWy1Y", "reviewer": "Reviewer_n5cR", "summary": "This paper discusses the estimation problem of the transition matrices in the noisy multi-label setting. They study the identifiability problem of the class-dependent transition matrix in noisy multi-label learning and propose a new estimator by exploiting label correlations without both anchor points and accurate fitting of noisy class posterior inspired by the identifiability results. Experimentally, the effectiveness of the proposed method is also illustrated.", "questions": "1. Can the authors clarify the originality of the proof of Theorem 1-4? \n2. Some dots should be replaced by $\\cd$, e.g. the dots in line 190 and line 204.", "limitations": "They address it adequately in Appendix.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghths:** \n1. They utilize the mismatch of label correlations to identify the transition matrix without both anchor points and accurate fitting of noisy class posterior in noisy multi-label learning.\n2. The method is effective to the issue with both theoretical analyses and empirical results support this method.\n3. Clear writing and structure.\n\n**Weaknesses:** \n1. Some formulas like in line 192 are recommended to be written in more simplified way.\n2. The experimental results are recommended to be presented in different forms but not just tables.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EK_fvoWwaiH", "reviewer": "Reviewer_hyUM", "summary": "This paper deals with how to estimate the 'noise transition matrix' under a multi-label setup. Like a single-label setup, the authors attempted to make an idea without using 'anchor points' for matrix estimation. They leverage the 'sample selection' method (ie., GMM) for extra clean supervision and provide a detailed mathematical derivation for the matrix estimation. The experimental section includes VOC and MS-COCO datasets; they injected a synthetic label noise with two factors, controlling the flip ratio of '1'->'0' and '0'->'1'. Under the proposed synthetic noise, the method works well and outperforms other simple extensions of similar works (but developed under the single-label setup).\n", "questions": "Please see the weakness section above.", "limitations": "Please see the weakness section above. In addition, the paper writing should be improved, particularly the theoretical section. Too many results are omitted and borrowed from [16, 24, 25, 31].", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. This paper is the first approach to estimating the noisy transition matrix for handling noisy labels with multi-labels. \n2. Leveraging mismatched label correlation is useful.\n3. The method shows higher performance than other simple baselines.\n\n### Weaknesses\nI agree that estimating the noise transition matrix helps make a statistically consistent classifier. However, I felt some major weaknesses of this paper, as below:\n1. **Assumption of Instance-independent Noise.** As the authors said, many previous studies make the same assumption for matrix estimations. But, there have been numerous recent studies to overcome this in the same direction [1-3]. I agree that dealing with 'instance-dependent' label noise in a multi-label setup is very difficult to address. At least, the author should mention what is more challenging in this setup compared to the single-label case. And, why making this assumption is reasonable with a more specific reason. Just saying 'the vast majority of current algorithms mainly focus' is not enough as a research paper.\n2. **Noise Injection Protocol.** According to the paper, the authors assume the class-dependent label noise and aim to estimate an accurate noise transition matrix under this assumption. However, the noise injection protocol in Line 217 looks like 'class-independent label noise. Specifically, the $\\rho just means the probability of '0' (or '1') being flipped to '1' (or '0') without any consideration of class pairs like <i, j>; only class j associates with the flipping and no connection between classes. Therefore, this is not very realistic and not class-dependent label noise. Did I get it wrong?\n3. **Unrealistic Evaluation.** Related to the second weakness, the author should have included at least one realistic real noisy data. There are benchmark data with noisy multi-label instances, called OpenImages database. The author can find other better datasets if possible. Without realistic evaluation, I am not convinced of the robust results in the paper since the injection protocol is very unrealistic.\n4. **Class Imbalance Problem & Low mAP.** Unlike single-label classification, the biggest difference in multi-label setup is 'class imbalance'. The authors also mentioned that this problem is very severe in Lines 44 & 102. However, there is no detailed mention of how to resolve this issue with the proposed idea throughout the paper. Next, when I am looking at the results on MS-COCO, the mAP value is too low compared with other multi-label papers using the same ResNet-50 pre-trained models. As an example, [4] shows ResNet-50 with GAP head achieves around 80mAP on MS COCO dataset (refer to Figure 6). However, in this paper, the mAP was less than 70 with Standard at the easiest noise setup (0.0, 0.2); this is a 20% missing label scenario. There are 20% missing labels but I think 20% of missing labels do not make 10mAP drops. Could you report the performance of your method and others under (0, 0) setup? This is a very good reference to check whether your implementation is correct and whether your method is still comparable with a zero-noise setup.\n\n**[1]** Approximating instance-dependent noise via instance-confidence embedding, arXiv 2021\n\n**[2]** A Second-Order Approach to Learning With Instance-Dependent Label Noise, CVPR 2021\n\n**[3]** An Information Fusion Approach to Learning with Instance-Dependent Label Noise, ICLR 2022\n\n**[4]** ML-Decoder: Scalable and Versatile Classification Head, arXiv 2021\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept