Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations
Authors: Hao Chen
Ankit Shah
Jindong Wang
Ran Tao
Yidong Wang
Xiang Li
Xing Xie
Masashi Sugiyama
Rita Singh
Bhiksha Raj
Venue: NeurIPS 2024 poster (2024)
Keywords: Imprecise Label Learning
Primary area: learning_theory

Abstract: Learning with reduced labeling standards, such as noisy label, partial label, and supplementary unlabeled data, which we generically refer to as imprecise label, is a commonplace challenge in machine learning tasks. Previous methods tend to propose specific designs for every emerging imprecise label configuration, which is usually unsustainable when multiple configurations of imprecision coexist. 
In this paper, we introduce imprecise label learning (ILL), a framework for the unification of learning with various imprecise label configurations. ILL leverages expectation-maximization (EM) for modeling the imprecise label information, treating the precise labels as latent variables. Instead of approximating the correct labels for training, it considers the entire distribution of all possible labeling entailed by the imprecise information. We demonstrate that ILL can seamlessly adapt to partial label learning, semi-supervised learning, noisy label learning, and, more importantly, a mixture of these settings, with closed-form learning objectives derived from the unified EM modeling. Notably, ILL surpasses the existing specified techniques for handling imprecise labels, marking the first practical and unified framework with robust and effective performance across various challenging settings. We hope our work will inspire further research on this topic, unleashing the full potential of ILL in wider scenarios where precise labels are expensive and complicated to obtain.

Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations
Hao Chen Ankit Shah!, Jindong Wang? Ran Taol, Yidong Wang". Xiang Li?, Xing Xie?, Masashi Sugiyama Rita Singh', Bhiksha Raj1.
CCarnegie Mellon University. *Microsoft Research," 'Peking University. RIKEN AIP, 'The University of Tokyo. MBBZAAI
Abstract
Learning with reduced labeling standards, such noisy label, partial label, and supplementary unlabeled data, which we generically refer to as imprecise label, is a commonplace challenge in machine learning tasks. Previous methods tend to propose specific designs for every emerging imprecise label configuration, which is usually unsustainable when multiple configurations of imprecision co- exist. In this paper, we introduce imprecise label learning (ILL), a framework for the unification of learning with various imprecise label configurations. ILL maximization (EM) for modeling the imprecise label infor- mation, treating the precise labels as latent variables. Instead of approximating the correct labels for training, it considers the entire distribution all possible labeling entailed by the imprecise information. We demonstrate that ILL seamlessly adapt to partial label learning, semi-supervised learning, noisy label learning, and, more importantly, a mixture of these settings, with closed-form learning objectives derived from the unified EM modeling. Notably, ILL surpasses the existing specified techniques for handling imprecise labels, marking the first practical and unified framework with robust and effective performance across various challenging settings. We hope our work will inspire further research on this topic, unleashing the full potential of ILL in wider scenarios where precise labels are expensive and complicated obtain. Code available at: https: Pane
leverages expectation expectationmmaximizatie
//github. com/ Saahhhhaao/GGneraal
Introduction
One of the critical challenges in machine learning is the collection of annotated data for model training [T+6]. Ideally, every data instance would be fully annotated with precise labels. However, collecting such data can be expensive, time-consuming, and error- error-prone. Often, the labels can be intrinsically difficult to ascertain precisely. Factors such as lack of annotator expertise and privacy concerns can also negatively affect the quality and completeness of the annotations.
In attempt to cireumvent this limitation, several methods have been proposed permit model learning from the data annotated with reduced labeling standards, which are generally easier to obtain. We will refer to such labels imprecise. Fig. D illustraesss some typical mechanisms of label imprecision that are commonly addressed in the literature. Label imprecision requires a modification of the standard supervised training mechanism to build models for each specific case. For instance, partial label learning (PLL) (7H13) allows instances to have a set of candidate labels, instead of single definitive one. Semi- ammi-suprrvised Learning (SSL) 234 seeks to enhance the generalization ability when only a small set of labeled data is available, supplemented by larger unlabeled set.
haoc3 @ @ww@@eannndddeeeeeeee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Jabel
aadd Labe
Labd Labd Labd
Tabel 2 Tabel
Tabll Tabel
X1
21
X1
X1
X2
X2
X2
x²2
X3
X3
X4
X4
X4
X4
True Label
Candidate Label
Unlabeted
Naisy Label
(a) Full Label
(b) Partial Label
(c) SemiiSuperrvsed
(d) Noisy Label
Figure I: Illustration of the full label and imprecise label configurations. We use an example dataset of 4 training instances and 3 classes. (a) Full label, the annotation is a single true label; (b) Partial label, the annotation is a label candidate set containing true label; (c) Semi-supervised, only part of the dataset is labeled, and the others are unlabeled; (d) Noisy label, the annotation is mislabeled.
Noisy label learning (NLL) 24 37
deals with noisy scenarios where the labels are corrupted or incorrect. There is greater variety of other forms of label imprecision, including crowd-soureing 39..
38
programmable weak supervision [40 41]. and bag-level supervision [42 among others.
While prior arts have demonstrated success in handling individual configurations of label imprecision, their approaches often differ substantially. They are tailored to a specific form of imprecision, as depicted in Fig. 2 Such specificity not only imposes the necessity of devising solution for emerging types of label imprecision scenarios, but also complicates the deployment in practical settings, where the annotations can be highly complex and may involve multiple coexisting and interleaved imprecision configurations. For instance, considering a scenario where both noisy labels and partial labels appear together, it might be challenging to adapt previous methods in NLL or PLL to this scenario since they either rely on the assumption of definite labels [39] or the existence of the correct label among label candidates [88. thus requiring additional algorithmic design. In fact, few recent works have attempted to address the combinations of imprecise labels in this way, such as partial noisy label 49 50 and semi-supervised partial label learning [5T 52 However, simply utilizing more sophisticated or ad-hoc design can hardly scale to other settings. In addition, most of these approaches attempt to infer the correct labels given the imprecise information (e.g. through consistency with adjacent data [4 53 [5]. iterative refinement [55], [56. average over given labels [57 (886 etc., train the model, which inevitably accumulates error during training.
In this paper, we formulate the problem from a different perspective: rather than taking the imprecise label information provided as a potentially noisy or incomplete attempt at assigning labels instances, we treat it generically as the information that imposes a deterministic or statistical restriction of the actual applicable true labels. We then train the model over the distribution of all possible labeling entailed by the given imprecise information. More specifically, for a dataset with samples X and imprecise label information I, we treat the inaccessible full and precise labels Y latent variable. The model is then trained to maximize the likelihood of the provided information L. Since the likelihood computed over the joint probability P(X, 1) = -y P(X, I, Y: 0) must marginalize out Y. the actual information 1 provided could permit a potentially exponential number of labeling. To deal with the resulting challenge of maximizing the logarithm of an expectation, we use the common approach of expectation maaimmzaaiiinnnnn (EM) (39), where the E-step computes the expectation of P,X.I,Y; given the posterior of current belief P(Y) X, I; 8*) at time step and the M-step maximizes the tight variational lower bound over P(X, I) The overall framework is thus largely agnostic to the various nature of label imprecision, with the imprecise label only affecting the manner in which the posterior P(Y|X, 7; (0) is computed. In fact, current approaches designed for various imprecise label scenarios can be treated as specific instances of our framework. Our approach can serve as a solution towards a unified and generalized view for learning with various imprecise labels.
While there exist earlier attempts on generalized EM solutions for different (other) imprecise supervisions or fuzzy observations [60 64| 45 65 67), they usually require additional assumptions and approximations on the imprecise information for learnablility 48 68), thus presenting limited scalability on practical settings [62]. On the contrary, the unified framework we propose subsumes all of these and naturally extends to the more practical "mixed" style of data, where different types of imprecise labels coexist. Moreover, for noisy labels, our framework inherently enables the learning of a noise model, as we will show in Section 3.2 Through comprehensive experiments, we demonstrate that the proposed imprecise label learning (ILL) framework not only outperforms previous methods

---

de (X)
Lanaer
User (eodt
00
Recadal
Lixes [liakg
Heod g.
Make
0,007
Eandder
KMA
Healh
Ax(e²)
(a) Partial Label
(b) Semi-Supervised
(c) Noisy Label
(d) Imprecise Label
Figure 2: Baseline model pipelines for various imprecise label configurations. (a) PiCO [3] for
partial label learning. (b) FixMatch [8] for semi-supereised learning. (c) SOP [69 for noisy label learning. (d) The proposed unified framework. It accommodates any imprecise label configurations
and also mixed imprecise labels with an EM formulation.
for dealing with single imprecise labels of PLL. NLL, and SSL, but also presents robustness and effectiveness for mixed imprecise label learning (MILL) settings, leveraging the full potential to more challenging scenarios. Our contributions are summarized as follows:
We propose an EM framework towards the unification of learning from various imprecise labels.
We establish scalable and consistent state-of-the the-arte (SOTA) performance with the proposed method on partial label learning, supervsper learning. and noisy label learning, demon- strating method's robustness in more diverse, complex label noise scenarios.
To the best of our knowledge, our work is the first to show the robustness and effectiveness of single unified method for handling the mixture of various imprecise labels.
2
Preliminary
In this section, we illustrate the notations and baselines from different imprecise label settings that adopt various solutions. We will show later how our proposed method generalize and subsumes these prior arts. Let X denote the input space, and y = [C] = (I....,C) C} represent the label space with C' distinct labels. A fully annotated training dataset of size N is represented as D {(x,, Hie Learning with imprecise labels involves approximating the mapping function o o gog : X' + 3 from a training dataset where the true label is not fully revealed from the annotation process. Here f is the backbone for feature extraction, 9 refers to the classifier built on top of the features, and the output from f g is the predicted probability P (y//x: x: 0), where A is the learnable parameter for f g. In this study, we primarily consider three imprecise label configurations (as illustrated in Fig. [T] and their corresponding representative learning paradigms (as shown in Fig. 2), namely partial label learning, semi-supervised learning, and noisy label learning.
Partial label learning (PLL). PLL aims to learn with a candidate label set C V, where the ground truth label y E y is concealed s. The training data for partial labels thus becomes DPLL {xx, s.) kE[]]. PiCO [13] is recent contrastive method that employs class prototypes to enhance label disambiguation (as shown in Fig. 2(a)). It optimizes the cross-entropy (CE loss between the prediction of the augmented training sample A.. and the disambiguated labels &. PiCO leams set of class prototypes from the features associated with the same pseudo-targets. A contrastive loss, based on MOCO (70), is employed to better learn the feature space, drawing the projected and normalized features Zu and z, of the two augmented versions of data An (x) and A₂ 3 clooser. The objective of PiCO is formulated as:
LpICO LCE (p(y) Ap(ylAn (x); + LCont (Zw, Za.M).
(1)
Semi-supervised learning (SSL). For SSL, we can define the labeled dataset as DL
SSL
{(x], b/)), and the unlabeled dataset DU JI with NL < NU
general teriidennc-ttresooddnnn based self-training 53 18] pipeline for SSL shown in Fig. 2
Consider FixMatch [18) as an example; there are usually two loss components: the supervised
CE loss on labeled data and the unsupervised CE loss on unlabeled data. For the unsupervised
objective, the pseudo-labels ÿ" from the network itself are used to train on the unlabeled data. A "strong-weak" augmentation [53] commonly adopted. To ensure the quality of the pseudo-labels,
2 For simplicity, we use Lce for labels of the formats class indices, one-hot vectors, and class prohabilities.
We use Ay to indicate the weaker data augmentation and A, to indicate the stronger data augmentation.
3

---

only the pseudo-label, whose confidence scores p" are greater than threshold are selected to participate in training:
Frx = LeE
+ (0"h 2T CCE (p(v) 0*).
(2)
Noisy label learning (NLL). NLL aims at learning with a dataset of corrupted labels, DNLL = be)lee We illustrate the NLL pipeline (in Fig. 2(c) with the recent sparse over- parameterization (SOP) model [69]. where a sparse noise model consisting of parameters u; V; [-1,1] for each sample is adopted. The noise model transforms the network prediction from the true label distribution into the noisy label distribution. A CE loss and a wuan-squared-error (MSE) loss optimize parameter {u.} and {v.} respectively:
Lsop LCE (ó (w 0) + m) 9) + LMSE (p(y)Aw (x)): 8) + m,
(3)
where & denotes the Lo normalization and m = u, u, y v, o о (1 "oth with y referring to the one-hot version of ye. Consistency regularization with strong troog-weakk augmentation and entropy class-balance regularization are additionally utilized for better performance in SOP [69].
3 Imprecise Label Learning
Although current techniques demonstrate potential in addressing particular forms of imprecise labels, they frequently fall short in adaptability and transferability to more complicated and more realistic scenarios where multiple imprecise label types coexist and interleave. This section first defines the proposed expectation expectation-maxiizzatee maximization (EM) formulation for learning with various imprecise labels. Then, we demonstrate that our unified framework seamlessly extends to partial label learning, semi- supervised label learning. noisy label learning, and the more challenging setting of mixed imprecise label learning. Connections and generalization to previous pipelines can also be drawn clearly under the proposed EM framework.
3.1 A Unified Framework for Learning with Imprecise Labels
Exploiting information from imprecise labels. The challenge of learning with imprecise labels lies in learning effectively with inaccurate or incomplete annotation information. Per the analysis above, prior works catering to specific individual imprecise labels either explicitly or implicitly attempt to
infer the precise labels from the imprecise label information. For example, partial label learning concentrates on the disambiguation of the ground truth label from the label candidates [13 7 [5]]
or averaging equally over the label candidates 772.. In semi-supervins superiise learning, after the model initially learns from the labeled data, the -lbelss are treated as correct labels and utilized to
conduct self trff-irainnng on the unlabeled data 73 [8]. Similarly, for noisy label learning, an integral part that helps mitigate overfitting to random noise is the implementation of an accurate noise model capable of identifying and rectifying the incorrect labels [33 69), thereby ensuring the reliability of the learning process. However, inferring the correct labels from the imprecise labels or utilizing the imprecise labels directly can be very challenging and usually leads to errors accumulated during training 773 74). which is also known as the confirmation bias. In this work, take different approach: we consider all possible labeling along with their likelihood that the imprecise labels fulfill to train the model, rather than using single rectified label from the imprecise information. Such an approach also eliminates the requirements for designing different methods for various imprecise labels and provides unified formulation instead, where closed-form solutions can be derived.
unified framework for learning with imprecise labels (ILL). Let {xx JEEN represent the features as realizations from X and Jy]ee[N] represent their precise labels as realizations from Y for the training data. Ideally, Y would be fully specified for X. In the imprecise label scenario. however, Y is not provided; instead we obtain imprecise label information 1. We view 1 not as labels, but more abstractly as a variable representing the information about the labels. From this perspective, the actual labels Y would have distribution P(Y and can present in various forms. When the information I provided is the precise true label of the data, P(Y would be a delta distribution, taking a value 1 at the true label, and elsewhere. If represents partial labels, then P(Y|T) would have non-zero value over the candidate labels, and be 0 elsewhere. When / represents set of noisy labels, PTY would represent the distribution of the true labels, given the noisy labels. When does not contain any information, i.c., unlabeled data, Y can take any value.

---

By the maximum likelihood estimation (MLE) principle, we must estimate the model to maximize the likelihood of the data/information we have been provided, namely X and I. Let P(X, 1;6) represent parametric form for the joint distribution of X and 1 Explicitly considering the labels Y, we have P(X,T;0) = Ey P(x, Y, 0).. The maximum likelihood principle requires us to find:
0" arg max log P(X, 1: arg max log Σ P(X,Y,I:0).
with 0 denotes the optimal value of 0. Eq. features the log of an expectation and cannot generally be solved in closed-form, and requires iterative hill-climbing solutions. Of these, arguably the most popular is the expectation expectation-maxiiizattie maximization (EM) algorithm [59]. which iteratively maximizes a tight variational lower bound the log- leeliikl, In our case, applying it becomes:
0++1 arg max Ey 1:00 [log P(X,Y,I;0)
(5)
arg max 1 [lo P(YIX;0) 0) + log P(TXX, Y:)],
where 0' the 1hh estimate of the optimal O. Note that P(X:)) 0) is omitted from Eq. (5) since P X
does not rely on 0. The detailed derivation of the variational lower bound is shown in Appendix C.1
There are several implications from Eq. (5). (i) The expectation over the posterior P(Y|X, 1; (P)
equates to considering all labeling entailed by the imprecise label information 7, rather than any single (possibly corrected) choice of label. For independent instances setting mostly studied in this paper, we can derive closed-form training objectives from this formulation as shown in Section 3.2 (ii) The property of the second term log P(] Y; 0) is dependent on the nature imprecise label If I is derivable from true labels Y, such as the actual labels the label candidates, it can be reduced to P.) i.e., the probability of I is no longer dependent on X or 9 and thus can be ignored from Eq. (5). If I represents the noisy labels. P(] ;) instead includes a potentially learnable noise model. (iii) It is a general framework towards the unification of any label configuration, including full labels, partial labels, low-resource labels, noisy labels, etc. In this work, we specialize the proposed EM framework to PLI., SSL,, NLI,, and the mixture of them in the following.
3.2 Instantiating the Unified EM Formulation
We illustrate how to seamlessly expand the formulation from Eq. (5) to partial label learning, semi- supervised learning, noisy label learning, and mixture settings, with derived closed-form loss function for each setting here. The actual imprecise labels only affect the manner in which the posterior P(Y|X, J; 8*) is computed for each setting. We show that all learning objectives derived from Eq. (5) naturally include a consistency term with the posterior as the soft target. We also demonstrate that the proposed unified EM framework closely connects with the prior arts, which reveals the potential reason behind the success of these techniques. Note that while we only demonstrate the application of the proposed framework to four settings here, it can also be flexibly extended to other settings. More details of derivation below are shown in Appendix [
Partial label learning (PLL). The imprecise label I for partial labels is defined as the label candidate sets S containing the true labels. These partial labels indicate that the posterior P(Y|X, S; 0*) can only assign its masses on the candidate labels. Since S can be derived from true labels Y, P(S|X,Y; Y; O) reduces to Y()))), and thus can be ignored. We also demonstrate with instance dependent partial labels that maintains PSS Y; 0) in AppendixxD.2.2 Defining the label candidates as [EN) and substituting it in Eq. 5 we have the loss function of PLL derived using ILL framework:
CPLL = Σ P(Y log 0( = LCE (p(y) ((9)A)))))))))))))))))))))
(6)
YEÍC]
where p(ylAn 0') is the normalized probability that Σ k Pk = 1, and pI: = 0, vk s. Eq. (6) corresponds exactly to consistency regularization [53]. with the normalized predicted probability as the soft pseudo-targets. We use A, and A., to denote the strong and weak augmentation
"The actual parameters may apply only to some component such as P( f) of overall distribution; we will nonetheless tag the entire distribution P(X, 7;0) with @ to indicate that it is dependent on overall. TTo formulate the loss function, convert the problem to minimization of the negative likeeiiddoodd
5

---

as stated earlier. This realization on PLL shares similar insights as [12] which exploits gradually induced loss weight for PLL on multiple augmentations of the data. However, our framework much simpler and more concise as shown in Appendix D.2.2 which does not require additional techniques.
Semi-supervised learning (SSL) In SSI., the input X consists of the labeled data X and the unlabeled data X", The imprecise label for SSL is realized as the limited number of full labels y for Xt. The labels y for unlabeled X are unknown and become the latent variable. Inter- estingly, for the unlabeled data, there is no constraint on possible labels it can take. The posterior P(Y | x y=; 8), which is the actual prediction from the network, can be directly utilized as soft targets for self-training. Since Y is conditionally independent with Y given X, the second term of Eq. (5): P(Y"-IX), X Y f). is reduced to P( ;). which corresponds to the supervised objective on labeled data. The loss function for SSL thus becomes:
eSSL_
Σ P( I xL, yl; 6") log ( ",,xx - log P P(Y ;))
YEÍCI
(7)
= LoB (x") 0), p(y)Aw (x 0') + LCe (w)):
The first term corresponds to the unsupervised consistency regularization usually employed in SSL, and the second term refers to the supervised CE loss only on labeled data. Eq. (7) has several advantages over the previous methods. It adopts the prediction soft-targets of all possible labeling on unlabeled data, potentially circumventing the confirmation bias caused by pseudo-labeling and naturally utilizing all unlabeled data which resolves the quantity-qualit trade-off commonly existing in SSL 18 23 It also indicates that previous pseudo-labeling with confidence threshold implicitly conducts the EM optimization, where the maximal probable prediction approximates the expectation, and the degree of the approximation is determined by the threshold т, rationalizing the effectiveness of dynamic thresholding.
Noisy label learning (NLL). Things become more complicated here since the noisy labels Ý do not directly reveal the true information about Y, thus P(Y X; inherently involves noise model that needs to be learned. We define a simplified instance- -eneaneceiideeeeeeee noise transition model T( with parameters w, and take a slightly different way to formulate the loss function for NLL from the ILL framework:
CNLL- =
Σ P 8;
O', w') log P(Y 8;0,0 - log P(L1X:0,w)
YE(C
(8)
CCE (p(y) J: 8 (())A. g: (:0","))) w()) + LCE (p(0)An, (x): (x)
where the parameters w and 0 are learned and-to-end. The first term corresponds to the consistency regularization of prediction conditioned on noisy labels and the second term corresponds to the supervised loss on noisy predictions that are converted from the ground truth predictions. Both quantities are computed using the noise transition model given the noisy label ÿ:
p (0 x p(ylx; B)T(uly w²), and p(0]x;0,w)
Σ DE(C
(9)
Mixture imprecise label learning (MILL). We additionally consider a more practical setting, mixture of imprecise label learning, with partial labels, noisy labels, and unlabeled data interleaved together. On the unlabeled data, the unsupervised objective is the same as the unsupervised consistency regularization of SSL as shown in Eq. (7). The labeled data here present partial and noisy labels S. Thus the noisy supervised objective in Eq. (9) becomes the supervised consistency regularization as in Eq. (6) of partial label setting to train the noise transition model, and the noisy unsupervised objective becomes the consistency regularization of the prediction conditioned on noisy partial labels.
Thus we have the loss function for MILL derived as:
CMILL
LcE (p (y A,(x'), s 0, w²) (y (x),,,
+ LCE [P ( Aw (x') ( (.w.) )
(10)
+ LcE (A (x") 0), ( Art 0"))
'A more complicated instance dependeen noise model T Y, X (:)) can also be formulated under our unified framework, but not considered in this work. Also, since we use 7 both in forward fashion and backward fashion, it is unidentifiable in this work.

---

Table I: Accuracy of different partial ratio g CIFAR-10 10, CIFAR-100, and CUB-200 for partial
label learning. The best and the second best results are indicated in bold and underline respectively.
Dataset
CIFAR-10 10
CIFAR-100
CUB-200
Purtial Ratio q
0.1
0.3
0.5
0.01
0.05
0.1
0.05
Sully-Supervised
94.91a0m7
73.56xn.1
LWS II
90.30a
88.99s1.4
86.168
65.78anm2
59.56an.33
53.53a000
39.74mn.47
PRODEN 55
90. 34
89, 39.38a1:1
87.78
62. 60e00z
60.73em
26
62. 12
82.30am21
79.08amm
74.05
49.76an45
47.62an.s
35.72a0.47
55.61
MSE 79
79.97am45
75.65au2s
67 09.009
49.1 19amns
46.02e187
43
22.07
EXP 79
79.23
75.79 25
70.34.1.5 52
41 1.44
29.27
9.44
94.39 i8
94. 18-112
93.584mm
73.094m
72.74 30
69.914024
72.17
Ours
96.37
96.26inus
95.91.mu
75.31.0.19
74.58
74.00
70.77.029
We can compute both quantity through the noise transition model:
( s; w') oc p(y|x; 0) II (vvlrr"") and p(ù x x;00;;00u0 0,
Σ
p(y|x; ()T(():
(11)
JEs
sE
4 Experiments
In this section, we conduct extensive experiments to evaluate ILL. Albeit simple, the ILL framework achieves comparable sfate-of-the-art performance regarding previous methods on partial label learning, semi-supervised learning, and noisy label learning. Moreover, our experiments show that ILI. could be easily extended more practical setting with a mixture of various imprecise label configurations. For all settings, we additionally adopt an entropy loss for balancing learned cluster sizes [75] 76). Experiments are conducted with three runs using NVIDIA V100 GPUs.
similarly as 69 22]
4.1 Partial Label Learning
Setup. Following 13 we evaluate our method on partial label learning setting using CIFAR- 10
(77). CIFAR-100 777). and CUB-200 (78.. We generate partially labeled datasets by flipping negative
labels to false positive labels with a probability q, denoted as a partial ratio. The C negative labels are then uniformly aggregated into the ground truth label to form set of label candidates. We consider 9 E {0.1, 0.3, 0.5) for CIFAR-10 10. q E (0.01, 0.05, 0.1) for CIFAR-
- 100, and q = 0.05 for CUB-200. We choose six baselines for PLL using ResNet- I: LWS 11,, PRODEN (55), CC 211. MSE and EXP 779.. and PiCO [3). The detailed hyper-parameters, comparison with the more recent method R-CR 12) that utilizes a different training recipe and model [0 and comparison
with instance-dependent partial labels [8T] are shown in Appendix D.2.2
Results. The results for PLL shown in Table D Our method achieves the best performance compared to the baseline methods. Perhaps more surprisingly, on CIFAR-10 and CIFAR 100, our method even outperforms the fully-supervised reference, indicating the potential better generalization capability using the proposed framework, sharing similar insights as in Wu et al. [12]. While PiCO adopts contrastive learning objective, our method still surpasses PiCO by average of 2.1 on CIFAR-10 and 2. 2.72% on CIFAR- 100. Our approach can be further enhanced by incorporating contrastive learning objectives, potentially leading more significant performance.
4.2 Semi-Supervised Learning
Setup. For experiments of SSL, we follow the training and evaluation protocols of USB 82 on image and text classification. To construct the labeled datasct for conissupe learning, we uniformly select I/C' samples from each class and treat the remaining samples as the unlabeled dataset. We present the results on CIFAR- and STL-10 [77) for image classification, and IMDB [83] and
Amazon Review [84] for text classification. We compare with the current methods with confidence thresholding, such FixMatch (18), AdaMatch [85], FlexMatch 20), Free [22], and SoftMatch
23). We also compare with methods with the contrastive loss, CoMatch [86] and SimMatch [87]. A
full comparison of the USB datasets and hyper-parameters is shown in Appendix D.3
Results. We present the results for SSL on T Although no individual SSL algorithm dominates the USB benchmark [82), method still shows competitive performance. Notably, method
7

---

Table 2: Error rate of different number of labels 1 on CIFAR-100, STL-10,, IMDB, and Amazon Review datasets for semi-supervised learning.
Datasets
CIFAR-IN
STI.-10
IMDB
Amazon Review
# Labels /
200
400
40
100
20
100
250
1000
AdaMatch 85
22.32-1.78
16.66-a62
13.64-2.49
7.62+1.0
8.09-0.99
20
95.40+0.6
40.16.
FixMatch 18
29.60a190
19.56.052
16.15
8.11
7.72-0.3
7.33
47.61a083
43.05e0.54
FlexMatch [20
26.76a1.12
36
14,40
8.178878
7.82=0.77
45.73a1.6 3.38
45.033
CoMatch [86
35. 15.08
25.35
15.1221.88
9.56
7.44-0.3
7.72a1.14
48. 98.7669990
43.3620.21
SimMatch [7
23. 13
17.06+0.8
11.77+3.20
7.55
7.93-0.55
7.08s0.33
45, 45.9
42.2 +0,30
FrecMatch 22
21.40
15.65a026
12.73a3.2
8.52a0.53
8,94.0.11
7.95
46.4120.0
42.64
SoftMatch 22
22.67
16.84
13,55
7.84
7.76.058
7.97,072
45 95.2990
40.21
Ours
22.06s1.06
16.40.oss
11.0920.1
8. 8.1021.02
7.32
7.642067
43,9620.32
42.32a0.02
Table 3: Accuracy synthetic noise on CIFAR-10 and CIFAR- and instance noise on Clothing and Web Vision for noisy label learning. We use noise ratio of {0.2, 0.5, 0.8} } for synthetic symmetric noise and 0.4 for asymmetric label noise. The instance noise ratio is unknown.
CIPAR-I0
CIFAR-100
ClothingIM
WehVision
Noise Type
Sym.
Asym.
Sym.
Asym.
Ins.
Ins.
Noisc Ratio
0.8
0.4
0.2
CE
87.20
80.70
65.80
82.20
$8.10
47.10
23.80
43,30
69.10
Mixup
30
93.50
87.90
72,30
69.90
57.30
33.60
DivideMix
33
77.10
77.32
ELR
95.80
94.80
93.30
93.00
71.70
73.80
60.80
77.50
72,90
76.20
SOP
96.30
95.50
94.00
93.80
78.80
75,90
63.30
78.00
73,50
76.60
Ours
96.78auu11
96.60sa15
94.3aam
94.75
77,49u02s
25.51
66.4629.72
75.82x1.9
74.02aau2
79.37auu
performs best on STL-10 with 40 labels and Amazon Review with 250 labels, outperforming the
previous best by 0.68% and 1.33%. In the other settings, the performance of our method is also very close to the best-performing methods. More remarkably, our method does not employ any thresholding, re-weighting, or contrastive techniques to achieve current results, demonstrating a significant potential to be further explored.
4.3 Noisy Label Learning
Setup. We conduct the experiments of NLL following SOP 669 on both synthetic symmet- ric/asymmetric noise on CIFAR-10 10 and CIFAR-100, and more realistic and larger-scale instance noise on Clothing 1M [X), and Vision 88). To introduce the synthetic symmetric noise to CIFAR-10 10 and CIFAR-100. we uniformly flip labels for probability into other classes. For asymmetric noise, we only randomly flip the labels for particular pairs of classes. The introduced noise then treated as ground truth labels to train the model. We mainly select three previous best methods as baselines: DivideMix (33 ELR [32]; and SOP [69]. We also include the normal cross-entropy (CE) training and mixup [90] baselines. More comparisons of other methods [9] 28) and on CIFAR-100 [2]
with training details and more baselines [93] 28 are shown in Appendix D.4
Results. We present the noisy label learning results in Table 3 The proposed method is comparable to the previous best methods. On synthetic noise of CIFAR-10, our method demonstrates the best performance on both symmetric noise and asymmetric noise. On CIFAR-100, our method generally produces similar results comparable to SOP. One may notice that our method shows inferior performance on asymmetric noise of CIFAR- 100; we argue this is mainly due to the oversimplification of the noise transition model. Our method also achieves the best results on WeiVision, outperforming the previous best by 2.05%. On Clothing 1 our results are also very close to DivideMix, which trains for 80 epochs compared to 10 epochs in ours.
4.4 Mixed Imprecise Label Learning
Setup. We evaluate on CIFAR- 10 and CIFAR-100 in a more challenging and realistic setting, the mixture of various imprecise label configurations, with unlabeled, partially labeled, and noisy labeled data existing simultaneously. We first sample the labeled dataset and treat other samples as the unlabeled. On the labeled dataset, we generate partial labels and randomly corrupt the true label of the partial labels. We set / (1000, 5000, 50000) for CIFAR-10. 100 and ( (5000, 10000, 50000} } for
8

---

Table 4: Accuracy comparison of mixture of different imprecise labels. We report results of full labels, partial ratio q of 0.1 (0.01) and 0.3 (0.05) for CIFAR-I0 (CIFAR-100) and noise ratio 1j of 0.1, 0.2, and 0.3 for CIFAR- 10 and CIFAR-
CIFAR-IO, 10, (=50000
CIFAR- 100, 1=50000
Method
y=0.2
n=
g
1p00.2
7=0.3
PICO+ 94
93.64
93.13
92.18
71.42
70.22
66.14
IRNel
93.44
92.57
92.38
71.17
70.10
68.77
DALI
150
94.15
94.04
93.77
72.26
71.98
71.04
0.1
0.01
PICO+ Mixup
150
94.58
94.74
94.43
75.04
74.31
71.79
DALI Mixup
50
95.83
95.86
95.75
76.52
76.55
76.09
Ours
96.477a.11
96.09
95.83
77.53.9.34
76.96inu
76.43
PICO+
92.32
92.22
89,95
69.40
66.67
62.24
IRNet 71
92.81
92.18
91.35
70.73
69.33
68.09
DALI
[50
93.44
93,25
92.42
72.28
71.35
70.05
0.3
0.05
PICO+ Mixup 50
94.02
94.03
92.94
73.06
71.37
67.56
Mixup
94.67
76.87
75.23
Ours
96.2amu2
95.87am114
95.22s0.06
77.07=8.16
76.343nu18
75.13
Table 5: Robust test accuracy results of our method on more mixture of imprecise label configura- tions. 1, q and ly are the number of labels, partial, and noise ratio.
CIFARIO
CIFARIOD
s=00.0
2=001
y=0.2
==0.3
9
+=0.0
>=0.1
g=0.2
y=0033
95. 15
93.90-a
92.02-am
89 02
0.01
69. 93
68. 74
66.87-a31
65. 34.34-00
5,000
33.11s
92.95a0
90.14aasi
87.31
10,000
0.05
69.85
68 08.0888
66.78at43
84.83
0.5
95 95.04s010
92. 58
88.39.0.3
83.09e0.5
0.10
68, (8.92snas
67. 15s0ss
64,44s129
60 60.26.100
0.1
94.48st0
91 17
87.17
81.04a.us
0.01
65.66un2)
63.13an27
58.36u13 60.93au1
1,000
0.3
94, 35
89. 9444
82.06
69. 20
5,000
0.05
65.06.004
62 42.288a0.7
58.92m031
53. 53.24-1
93.92as.z
86.34u277
70.86.278
38.19
0.10
63.32auss
58.73x138
53,27a137
46.19
C00. For partial labels, set 9 (0.1, 0.5} for CIFAR-10, and q {0.01, 0.05, 0.1} } for CIFAR-100. For noisy labels, we set E {0, 0.1, 1..1,.0.. 0.2, 3.3) for both datasets. Since there is no prior work that handle all settings all at once, compare
partial noisy label learning with PiCO+ 194]. IRNet 771. and DALI [50). Although there are also prior efforts on partial semi-supervised learning 52). they do not scale on simple dataset even on CIFAR-IO. Thus, we did not include them in comparison. We conduct additional validation of our method on more complex settings for partial noisy labels with unlabeled data to demonstrate its robustness to various imprecise labels.
Results. We report the comparison with partial noisy label learning methods in Table 4 Compared to previous methods, the proposed method achieves the best performance. Despite the simplicity, our method outperforms PiCO+ and DALI with mixup, showing the effectiveness of dealing with mixed imprecise labels. We also report the results of our methods more mixed imprecise label configurations in T Our method demonstrates significant robustness against various settings of the size of labeled data, partial ratio, and noise ratio. Note that this is the first work that naturally deals with all three imprecise label configurations simultaneously. with superior performance than previous methods handling specific types or combinations of label configurations. This indicates the enormous potential of our work in realistic applications for handling more practical and complicated data annotations common in real world applications.
Conclusion
We present the imprecise label learning (ILL) framework, unified and consolidated solution for learning from all types of imprecise labels. II.I. effectively employs an expectation-maxiiiaaiii (EM) algorithm for maximum likelihood estimation (MLE) of the distribution over the latent ground truth labels Y, imprecise label information 1, and data X. It naturally extends and encompasses previous formulations for various imprecise label settings, achieving promising results. Notably, in scenarios where mixed configurations of imprecise labels coexist, our method exhibits substantial robustness against diverse forms of label imprecision. The potential broader impact of the ILL framework is substantial. It stands poised to transform domains where obtaining precise labels poses a challenge, offering a simple, unified, and effective approach to such contexts. Beyond the three imprecise label configurations we have demonstrated in this study, the ILI. framework shows promise for an extension to more intricate scenarios such as multi-instance learning [22 and multi-label crowd-sourcing learning (38). However, it is also crucial to acknowledge the limitations of the ILL

---

framework. Although its effectiveness has been substantiated on relatively smaller-scale datasets, additional empirical validation is necessary to assess its scalability to larger datasets. Furthermore. our study only considers balanced datasets; thus, the performance of the II.I. framework when dealing with imbalanced data and open-set data still remains an open area for future exploration. We hope that our study will constitute a significant stride towards comprehensive solution for imprecise label learning and catalyze further research in this crucial field.
Acknowledge
Masashi Sugiyama was supported by the Institute for Al and Beyond, UTokyo.
10

---

References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 10X0v:8810004805 2018.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), pages 8748 8763. PMLR, 2021.
[6] OpenAL. Gpt-4 technical report. 2023.
[7] Timothee Cour. Ben Sapp. and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research, 12:1501-1536, 2011.
[8] Jie Luo and Francesco Orabona. Learning from candidate labeling sets. Advances in Neural Information Processing Systems (NeurlPS), 2010.
[9] Lei Feng, Jiaqi Lx, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. ArXiv, 0a//2007708929, 2020.
[10] Dengbao Wang, Min-Ling Zhang, and Li Li. Adaptive graph guided disambiguation for partial label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:8796-8811, 2019.
[11] Hongwei Wen, Jingyi Cui, Hanyuan Hang. Jiabin Liu, Yisen Wang, and Zhouchen Lin. Leveraged weighted loss for partial label learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 11091-11100 PMLR, 2021.
[12] Dong-Dong Wu, Deng-Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 24212-24225. PMLR, 17-23 Jul 2022.
[13] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO: Contrastive label disambiguation for partial label learning. In International Conference on Learning Representations (ICLR), 2022.
[14] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi semi-supervised learning method for deep neural networks. In Workshop on challenges representation learning, ICML, page 896, 2013.
[15] Laine Samuli and Aila Timo. Temporal ensembling for semi- superviid learning. In Interna- tional Conference on Learning Representations (ICLR), page 6, 2017.
[16] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papemot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems (NeurlPS), 32, 2019.
11

---

[17] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-super learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations (ICIRR) 2019.
[18] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Pro- cessing Systems (NeurlPS), 33, 2020.
[19] Qizhe Xie, Minh- Kinh-TThang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 10687-10698. 2020.
[20] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting supi-supe learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[21] Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 14647-14657, 2022.
[22] Yidong Wang, Hao Chen, Qiang Heng. Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj. Bernt Schiele, and Xing Xic. Freematch: Self-adaptive thresholding for semi-supervisee learning. In International Conference on Learning Representations (ICLR), 2023.
[23] Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semi- supervised learning. In International Conference on Learning Representations (ICLR), 2023.
[24] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 2691-2699, 2015.
[25] Alan Joseph Bekker and Jacob Goldberger. Training deep neural-networks based unreliable labels. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2682-2686, 2016.
[26] Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In International Conference on Learning Representations (ICLR), 2016.
[27] Aritra Ghosh, Himanshu Kumar, and P. Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2017.
[28] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Wai-Hung Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[29] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018
[30] Junnan Li, Yongkang Wong, Qi Zhao, and M. Kankanhalli. Learning to learn from noisy labeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5046-505 000550554222 2018.
[31] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 322-330, 2019.
12

---

[32] Sheng Liu, Jonathan Niles-Weed, Weed, Nares Razavian, and Carlos Fernandez- Gemandez-Granda. Early- learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
[33] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations (ICLR), 2020.
[34] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Monazam Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[35] Qi Wei, Haoliang Sun, Xiankai Lu, and Yilong Yin. Self-filterngg A aoise- sample selection for label noise with confidence penalization. In European Conference on Computer Vision, pages 516-532. Springer, 2022.
[36] Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
[37] Qi Wei, Lei Feng, Haoliang Sun, Ren Wang, Chenhui Guo, and Yilong Yin. Fine-grained classification with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11651-11660, 2023.
[38] Shahana Ibrahim, Tri Nguyen, and Xiao Fu. Deep learning from crowdsourced labels: Coupled cross-entropy minimization. identifiability. and regularization. In International Conference on Learning Representations (ICLR), 2023.
[39] Jiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid, Abhishek Kumar, and Yang Liu. To aggregate or not? learning with separate noisy labels. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Aug 2023.
[40] Jieyu Zhang, Cheng-Yu Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision, 2022.
[41] Renzhi Wu, Shen-En Chen, Jieyu Zhang, and Xu Chu. Learning hyper label model for programmatic weak supervision. In International Conference on Learning Representations (ICLR), 2023.
[42] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International Conference Machine Learning (ICML), pages 2127-2136. PMLR, 2018.
[43] Nan Lu, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. arXiv preprint 1xXv::8808.1085.. 2018.
[44] Clayton Scott and Jianxin Zhang. Learning from label proportions: A mutual contamination framework. Advances in Neural Information Processing Systems (NeurIPS), 33:22256-22267, 2020.
[45] Y. Zhang, N. Charoenphakdee, Z. Wu, and M. Sugiyama. Learning from aggregate observa- tions. pages
2020.
[46] Saurabh Garg. Yifan Wu, Alex Smola, Sivaraman Balakrishnan, and Zachary C. Lipton. Mixture proportion estimation and pu learning: A modern approach, 2021.
[47] Lei Feng, Senlin Shu, Nan Lu, Bo Han, Miao Xu, Gang Niu, Bo An, and Masashi Sugiyama. Pointwise binary classification with pairwise confidence comparisons. In International Con- ference on Machine Learning, pages 3252-3262. PMLR, 2021.
[48] Andrea Campagner. Learnability in "learning from fuzzy labels". In 2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pages 1-6. IEEE, 2021.
13

---

[49] Zheng Lian, Mingyu Xu, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Amet: Automatic refinement network for noisy partial label learning. arXiv preprint arX14:2211.0477,, 2022.
[50] Mingyu Xu, Zheng Lian, Lei Feng, Bin Liu, and Jianhua Tao. Dali: Dynamically adjusted label importance for noisy partial label learning, 2023.
[51] Qian- Wei Wang, Yu-Feng Li, and Zhi tua-Hua Zhou, Partial label learning with unlabeled data. In IJCAI, pages 3755-3761, 2019.
[52] Wei Wang and Min-Ling Zhang. Semi-supervised partial label learning via confidence- rated margin maximization. Advances in Neural Information Processing Systems (NeurIPS), 33:6982-6992 2020.
[53] Qizhe Xie, Zihang Dai. Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems (NeurIPS). 33, 2020.
[54] Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance- dependent label-noise learning under a structural causal model. Advances in Neural Informa- tion Processing Systems, 34:4409 442000440000 2021.
[55] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 6500-6510. PMLR, 2020.
[56] Chidubem Arachie and Bert Huang. Constrained labeling for weakly supervised learning. In Uncertainty in Artificial Intelligence, pages 236-246. PMLR, 2021.
[57] Eyke Hüllermeier and Weiwei Cheng. Superset learning based on generalized loss minimiza- tion. In Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part II 15, pages 260-275. Springer, 2015.
[58] Jiaqi Lv, Biao Liu, Lei Feng, Ning Xu, Miao Xu, Bo An, Gang Niu, Xin Geng, and Masashi Sugiyama. On the robustness of average losses for partial-label learning. IEEE Transactions Pattern Analysis and Machine Intelligence, page 1-15, 2023.
[59] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1-22, 1977.
[60] Thierry Dencuxx. Maximum likelihood estimation from fuzzy data using the em algorithm. Fuzzy sets and systems, 183(1):72-91, 2011.
[61] Eyke Hüllermeier. through generalized loss minimization. International Journal of Approximate Reasoning,
Learning from imprecise and fuzzy observations: Data disambiguation
55(7):1519-1534 2014.
[62] Benjamin Quost and Thierry Denoeux. Clustering and classification of fuzzy data using the fuzzy em algorithm. Fuzzy Sets and Systems, 286:134-156, 2016.
[63] Brendan Van Rooyen and Robert C Williamson. A theory of learning with corrupted labels. J. Mach. Learn. Res., ((1):8501-85500 2017.
[64] Chen Gong, Jian Yang. Jane You, and Masashi Sugiyama. Centroid estimation with guaranteed efficiency: A general framework for weakly supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):2441-2555. 2020.
[65] Chao-Kai Chiang and Masashi Sugiyama. Unified risk analysis for weakly supervised learning. arXiv preprint arXiv:2309.082. 16, 2023.
[66] Zixi Wei, Lei Feng, Bo Han, Tongliang Liu, Gang Niu, Xiaofeng Zhu, and Heng Tao Shen. A universal unbiased method for classification from aggregate observations. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 36804 368200 PMLR, 23-29 Jul 2023,
14

---

[67] Zheng Xie, Yu Liu, Hao- Yuan He, Ming Li, and Zhi-Hua Zhou. Weakly supervised auc optimization: A unified partial auc approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[68] Andrea Campagner.
Learning from fuzzy labels: Theoretical issues and algorithmic solutions. International Journal of Approximate Reasoning, page 108969, 2023.
[69] Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by peee-parameterization In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 14153- 14172. PMLR, 17-23 Jul 2022.
[70] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020.
[71] Zheng Lian, Mingyu Xu, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Irnet: Iterative refinement network for noisy partial label learning, 2022.
[72] Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelli- gent Data Analysis, 10(5):419-444 439, 2006.
[73] Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN,, pages 1-8. IEEE, 2020.
[74] Baixu Chen, Junguang Jiang, Ximei Wang. Jianmin Wang, and Mingsheng Long. Debiased pscudo labeling in self-training. arXiv preprint arXiv:2202.07136. 2022.
[75] David MacKay John Bridle, Anthony Heading. Unsupervised classifiers, mutual information and "phantom targets. Advances in Neural Information Processing Systems (NeurIPS), 1991.
[76] Armand Joulin and Francis Bach. A convex relaxation for weakly supervised classifiers. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2012.
[77] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[78] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
[79] Lei Feng, Takuo Kaneko, Bo Han, Gang NIu, Bo An, and Masashi Sugiyama. Learning with multiple complementary labels. In Proceedings of the International Conference on Machine Learning (ICML), pages 3072-3081. PMLR, 2020.
[80] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference (BMVC). British Machine Vision Association, 2016.
[81] Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34:27119-271333 2021.
[82] Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi. Zhen Wu, Yu-Feng Li. Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj. Takahiro Shinozaki, Bernt Schicle, Jindong Wang, Xing Xie, and Yue Zhang. Usb: A unified semi-supervised learning benchmark. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[83] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.
[84] Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165-172, 2013.
15

---

[85] David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to supervipe learning and domain adaptation. In- ternational Conference on Learning Representations (ICIR), 2021.
[86] Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi- -upervvee learning with contrastive graph regularization. Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 9475-9484, 2021.
[87] Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi-supervised learning with similarity matching. arXiv preprint arXiv:2203.06915, 2022.
[88] Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2691-2699, 2015.
[89] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data, 2017.
[90] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond 1744049 2017.
empirical risk minimization. arXiv preprint
[91] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2233-2241, 2016.
[92] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110. 12088, 2021.
[93] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pages 2304-2313, PMLR, 2018.
[94] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng. Gang Niu, Gang Chen, and Junbo Zhao. Pico+: Contrastive label disambiguation for robust partial label learning, 2022.
[95] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR), pages 702-703, 2020.
[96] Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance-based approach. In IJCAI. pages 4048-4054 2015.
[97] Chen Gong, Tongliang Liu, Yuanyan Tang, Jian Yang, Jie Yang, and Dacheng Tao. A regularization approach for instance-based superset label learning. IEEE transactions on cybernetics, 48(3):967-978, 2017.
[98] Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5557-5564, 2019.
[99] Zhenguo Wu, Jiaqi Lv, and Masashi Sugiyama. Learning with proper partial labels. Neural Computation, 35(1): (3(1))58-81.. Jan 2023.
[100] Nam Nguyen and Rich Caruana. Classification with partial labels. In Proceedings of the 14th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 551-559, 2008
[101] Min-Ling Zhang, Bin-Bin Zhou, and Xu- Ying Liu. Partial label learning via feature-aware disambiguation. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1335-1344, 2016.
[102] Liping Liu and Thomas Dietterich. A conditional multinomial mixture model for superset label learning. Advances in Neural Information Processing Systems (NeurIPS), 25, 2012.
16

---

[103] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- Beighh-averaged consistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[104] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervvssd learning. IEEE transactions on Puttern Analysis and Machine Intelligence, 41(8):197--1993, 2018.
[105] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervin learning with dynamic thresholding. In Proceedings of the International Conference on Machine Learning (ICML), pages 1 11525-11536. PMLR, 2021.
[106] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR), 2019.
[107] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), pages 11557-11568, 2021.
[108] Chiyuan Zhang. Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization, 2016.
[109] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Under- standing deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
[110] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae- Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions Neural Networks and Learning Systems, page 1-19, 2022.
[111] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bai- ley. Normalized loss functions for deep learning with noisy labels. In Proceedings of the International Conference on Machine Leurning (ICML), 2020.
[112] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction, 2020.
[113] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. In IEEE Transactions on pattern analysis and machine intelligence, pages 447-461, 2016.
[114] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411 Apr 2021.
[115] Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In Proceedings of the International Conference on Machine Learning (ICML), pages 12501-12512. PMLR, 2021.
[116] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang. Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels, 2021.
[117] Junnan Li, Caiming Xiong, and Steven CH Hoi. Learning from noisy data with robust repre- sentation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 9485-9494, 2021.
[118] Shikun Li, Xiaobo Xia, Shiming Ge. and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022.
[119] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. 2018 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2018.
17

---

[120] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019.
[121] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing unclean samples for robust deep learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the International Conference Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 5907-5915. PMLR, 09-15 Jun 2019.
[122] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O' D'Connor, and Kevin McGuinness. Unsuper- vised label noise modeling and loss correction, 2019.
[123] Vinay Shukla, Zhe Zeng, Kareem Ahmed, and Guy Van den Broeck. A unified approach to count-based beakky-supprrised learning. In ICML 2023 Workshop on Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators, jul 2023.
[124] Massih-Reza Amini and Patrick Gallinari. Semi-supervised logistic regression. In ECAI, volume 2, page 11, 2002.
[125] Ning Xu, Jiaqi Lv, Biao Liu, Congyu Qiao, and Xin Geng. Progressive purification for instance-dependent partial label learning, 2022.
[126] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[127] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[128] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 113-223. JMLR Workshop and Conference Proceedings, 2011.
[129] Jiancheng Yang. Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 191-195, 2021.
[130] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Plister, and Bingbing Ni. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. arXiv preprint 1:Xii:22100114955 2021.
[131] Jong-Chyi Su and Subhransu Maji. The semi-supervised inaturalist-aves challenge at fgvc7 workshop. arXiv preprint UtXiv:2103.06933 2021.
[132] Yelp dataset: http://wwww.ylp..com/ttttc ccaallengeee..leee...e
[133] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in Neural Information Processing Systems (NeurIPS), 28:649-657, 2015.
[134] Ming- Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. Importance of semantic representation: Dataless classification. In AAAI, volume 2, pages 830-835, 2008.
[135] Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. arXiv preprint arXiv:2010.02347, 2020.
18

---

Appendix
A
Notation
We present the notation table for each symbol used in this paper in Table
Table 6: Notation Table
Notation
Definition
A training instance
A class index label
xx KEEM
A set of data instances x of size N
A set of precise label indices of size N
An imprecise label, which might contain multiple class indices
A set of imprecise labels ii] of size N Random variable of training instance Input space where x is drawn from Random variable of ground-truth labels
X
X
Y
y
Label space where y drawn from Random variable of imprecise labels Model backbone
Model classifier
Model multi-layer perceptron
fog
Model mapping X' -+
Learnable parameters of fog
p(u|x: 0)
Output probubility from model fog
f foh
Model mapping X - Z. where Z is projected feature space
D
Dataset
C
Loss function
Aw
Weak data augmentation, usually is HorizontalFlip Strong data augmentation, usually is RandAugment [95]
An
Projected features from fo on weakly-augmente
2.
Projected features from f h on grongly-augmentee
Memory queue MoCo 70
partial label, with ground south-truth label contained set partial labels, Random variable of partial lahel
[ScheeN)
ground-truth lahel contained of size
S
x
labeled training example A labeled class index
unlabeled training example unknown class index for unlabeled
g"
XL
set laheled data instances
YL
A set of labels for labeled data instances
XU
A set of unlabeled data instances
Yu
A set of unknown labels for unlabeled duta instances
ga
The maximum predicted probability unlabeled data max max(p(vix";0))
g"
The pseudo-label from the predicted probability on unlabeled data arg max(p(y) x²;")))))))))))))))
The threshold for confidence thresholding A corrupted/noisy label
geh
An one-hot version of the corrupted/noisy label Random variable of noisy labels
Y
Noise model related parameters SOP 69
T (()))
The simplified noise transition model II.I.
The parameters in the simplified noise model
Related Work
Many previous methods have been proposed for dealing with the specific types and some combinations of imprecise label configurations. We revisit the relevant work in this section, especially the state-of- the-art popular baselines for learning with individual and mixture imprecise label configurations.
Partial label learning (PLL). The prior arts can roughly divided into identification- label disambiguation 996-99 or average-based for utilizing all candidate labels 772 7 58 The
for
traditional average-based methods usually treat all candidate labels equally, which may involve the misleading false positive labels into training. To overcome these limitations, researchers have
19

---

explored edentification-basee methods, viewing the ground- label as a latent variable. They seek to maximize its estimated probability using either the maximum margin criterion [TOO] [OT] or the maximum likelihood criterion [102. Deep learning techniques have recently been incorporated into identification-basee methods, yielding promising results across multiple datasets. For example, PRODEN (55] proposed self-training strategy that disambiguates candidate labels using model outputs. CC [9] introduced classifier-consister and risk-consistent algorithms, assuming uniform candidate label generation. LWS [I] relaxed this assumption and proposed a family of loss functions for label disambiguation. More recently, Wangt et al. [3] incorporated contrastive learning into PLL, enabling the model to learn discriminative representations and show promising results under various levels of ambiguity. RCR involves consistency regularization into PLL recently [12]
Semi-supervised learning (SSL). SSL is paradigm for learning with a limited labeled dataset supplemented by a much larger unlabeled dataset. Consistency regularization and self-training, inspired by clusterness and smoothness assumptions, have been proposed to encourage the network to generate similar predictions for inputs under varying perturbations [I33] L5 [04]. Self-training
11 73 [8] is widely-used approach for leveraging unlabeled data. Pseudo Label D4 73
well-known self-training technique, iteratively creates pseudo labels that are then used within the same model. Recent studies focus largely on generating high-quality pseudo-labels. MixMatch [6]. for instance, generates pseudo labels by averaging predictions from multiple augmentations. Other methods like ReMixMatch [171, UDA [53], and Fix.Match [18] adopt confidence thresholds to generate pseudo labels for weakly augmented samples, which are then used to annotate strongly augmented samples. Methods such as Dash [05), FlexMatch 20), and FreeMatch [22 dynamically adjust these thresholds following a curriculum learning approach. SoftMatch 23 introduces a novel utilization of lseudo-labels through Gaussian re-weighting. SSL has also seen improvements through
the incorporation of label propagation, contrastive loss, and meta learning [106] 107 86 87 82).
Noisy label learning (NLL). Overfitting to the noisy labels could result in poor generalization performance, even if the training error is optimized towards zero [108] [09]. Several strategies to
address the noisy labels have been proposed [TTO]. Designing loss functions that are robust to noise is a well-explored strategy for tackling the label noise problem 229 31
[12]. Additionally,
methods that re-weight loss [13 have also been explored for learning with noisy labels. Another common strategy to handle label noise involves assuming that the noisy label originates from
probability distribution that depends on the actual label. Early works (26) incorporated these transition probabilities into a noise adaptation layer that is stacked over a classification network and trained in an end-to-end fashion. More recent work, such as Forward (9]. prefers to estimate these transition probabilities using separate procedures. However, the success of this method is contingent upon the availability of clean validation data [14 or additional assumptions about the data (115)
Noise correction has shown promising results in noisy label learning recently [16 18 69) During the early learning phase, the model can accurately predict subset of the mislabeled examples [32] This observation suggests a potential strategy of correcting the corresponding labels. This could be accomplished by generating new labels equivalent to soft or hard pseudo-labels estimated by the model [119 120]. Co-Teaching uses multiple differently trained networks for correcting noisy labels 28 SELFIE 121 corrects subset of labels by replacing them based on past model outputs. Another study in [12 uses two-component mixture model for sample selection, and then corrects labels using a convex combination. Similarly, DivideMix 33 employs two networks for sample selection using a mixture model and Mixup (90)
Mixture imprecise label settings. Various previous works have explored dealing with distinet types of imprecise labels. However, they have yet to tackle a combination of partial labels, limited labels, and noisy labels, which is highly realistic scenario. For instance, recent attention has been paid to the issue of partial noisy label learning. PiCO+ (94), an extended version of PiCO [13] is tailored specifically for partial noisy labels. IRNet 7 uses two modules: noisy sample detection and label correction, transforming the scenario of noisy PLL into a more traditional PL... DALI (50) another framework designed to reduce the negative impact of detection errors by creating a balance between the initial candidate set and model outputs, with theoretical assurances of its effectiveness.
Additionally, some work has focused on semi-supervised partial label learning (1(55) No existing research can effectively address the challenge of handling combination of partial, limited, and noisy labels simultaneously, which underscores the novelty and significance of our work.
Previous attempts towards unification of learning from imprecise labels. There are earlier attempts for the generalized solutions of different kinds of imprecise labels/observations. Denceux
20

---

660 proposed an EM algorithm for the likelihood estimation of fuzzy data and verified the algorithm on linear regression and uni-variate normal mixture estimation. Van Rooyen al. [63 developed an abstract framework that generically tackles label corruption via the Markov transition. Quost et al. [62] further extended the EM algorithm of fuzzy data on the finite mixture of Gaussians. Gong et al. (64] proposed a general framework with centroid estimation for imprecise supervision. A unified partial AUC optimization approach was also proposed earlier [67]. Zhang et al. [45] and Wei. et al. [66] proposed generalized solutions for aggregate observations. A unified solution based on dynamic programming for count-based weak supervision was also proposed 123 While relating to these works on the surface, ILL does require any assumption the imprecise information and generalizes well to more practical settings with noisy labels. Some other works for individual settings also related EM framework, but usually involved the approximation on the EM [24 25
C Methods
C.1
Derivation of Variational Lower Bound
Evidence lower bound (ELBO), equivalently variational lower bound [59 is the core quantity in EM. From Eq. (5. model log P(X, I) have:
log P(X,1:0)
Q(Y) log P(X,
/ Q(Y) log P(X,1 1;0) P(YIX, T:0)
1;0) dy
(12)
P,X,I,Y; dY / Q(Y) log P(Y (YY))(Y)
/ Q(Y) log P(X,1,Y;0) dY Q(X) log P( dY Q(Y)
Q(Y)
where the first term is the ELBO and the second term is the KL divergence Dxz (X)]QQ0)11P(XXXXX110) Replacing Q(Y) with I(Y]X,I;0') at each iteration will ob- tain Eq.
C.2 Instantiations to Partial Label Learning
The imprecise label for partial labels is defined as the label candidate sets S with (s:)ee containing the true labels. Now we can derive Eq. (6) by replacing T with S in Eq. 5):
1 [log XX 0) + log P(] [)]
= Exix,s:0 [log P(Y X: 0) + log P( Y)
Σ P(Y [log P log P(T]X,Y:0)
(13)
Σ 5( [log P(Y X: 0)] log P(I|X, 9:
Note that P(JY, X; can be moved out of the expectation because it is fixed quantity to any Y. Now we replace Y, X, and S to y, x, and for each instance, and converting the maximization problem to negative log-likelihood minimization problem drive the loss function:
CPLL
E P(y x,, s,: 8*) log (
Σ log pos.
(14)
The first term is the Cross-Entropy loss we derived in Eq. 6 S is not instance-dependent, then knowing Y also knows S, the second term thus can be ignored in Eq. (6). If S becomes instance-dependent, the second term can be maintained as supervised term in optimize 0.
C.3 Instantiations to Semi-Supervised Learning
In SSL., the input consists of the labeled data XL and the unlabeled data XU, The imprecise label for SSL is realized as the limited number of full labels L for L The labels Y for unlabeled X
21

---

are unknown and become the latent variable. Thus we can write:
1y1x,1 (log P( + log P Y] = x2 [log P(r 1 | xx + log P( ,,, [0]]
(15)
Σ P(Y )t) [log p( log p(y
The negative log-like seglliklihhod loss function for {x], y ke and (x" helwur thus becomes:
CSLL I.. = LCE (p(v)x"; 0). p(vx";; 8)) + Lee (p(u)x²;0).)
(16)
C.4 Instantiations to Noisy Label Learning
We denote the given noisy labels as Y. For noisy label learning, our method naturally supports noise transition model T(YY; ;w) with learnable parameter w, as we will show the following:
E E log P(Y]X;0) + log ((1[X,
1I log P(Y,YIXX:))
Erix.fio log X:; + log log P(T|x:0))
(17)
= Σ P(YI).X:0) log P(Y x;0) - log P(
The loss function is:
CILL Lce (p(ylx, 9: 0 p(y)l, g: 0: ,w))) + Lce (p(ū x; 8, 0.w).
(18)
Note that both term is computed from the noise transition matrix mentioned in Eq. 99
C.5 Instantiations Mixed Imprecise Label Learning
In this setting, we have both labeled data and unlabeled data, where the labels for the labeled data are both partial and noisy. On the unlabeled data, the unsupervised objective the same as the unsupervised consistency regularization of semi-supervised learning shown in Eq. 77. On the labeled data, mainly follows the Eq. (9) of noisy label learning. with the noisy single label becoming the noisy partial labels $. For noisy partial labels, the noisy supervised objective in Eq. 8 becomes the supervised consistency regularization as Eq. 6 of partial label setting to train the noise transition model, and the noisy unsupervised objective becomes the consistency regularization of the prediction conditioned on noisy partial labels:
LcE (D (y (;x (x) + LCE (p (x):)())
(19)
We can compute both quantity through the noise transition model:
S 0,w" x p(yxx 0) II
(wll w"), and p 0,w)
Σ p]T (20)
Sea
DEIC)
D Experiments
D.I. Additional Training Details
We adopt two additional training strategies for the ILL framework. The first is the "strong- weak" augmentation strategy [53]. Since there is a consistency regularization term in each imprecise label formulation ILL, we use the soft pseudo-targets of the weakly-augmented data train the strongly- augmented data. The second is the entropy loss 75 for class balancing, which is also adopted in SOP (69 and FreeMatch 222 We set the loss weight for the entropy uniformly for all experiments as 0.
22

---

D.2
Partial Label Learning
D.2.1
Setup
Following previous work [25 13 we evaluate our method on partial label learning setting using CIFAR-1O, 10, CIFAR- and CUB-200 778.. We generate partially labeled datasets by flipping negative labels to false positive labels with a probability q, which is also denoted as partial ratio. Specifically, the C 1 negative labels are uniformly aggregated into the ground truth label to form a set of label candidates. We consider & E {0.1, 0.3, 0.5) for CIFAR-I0, g E {0.01, 0.05, 0.1} for CIFAR-100, and y 0.05 for CUB-200. For CIFAR-10 10 and CIFAR-100 100, we use ResNet- 18 [ as backbone. We use SGD as an optimizer with a learning rate of 0.01, a momentum of 0.9, and weight decay of 1c-3. For CUB-200
initialize the ResNet-18 18 [ with ImageNet- 1K [26] pre- pre-trained weights. We train 800 epochs for CIFAR-10 and CIFAR-100 (77), and 300 epochs for CUB-200, with cosine learning rate scheduler. For CIFAR-I0 and CIFAR-I00, 32. For CUB-200, we use an input image size of 224. batch size of 256 is used for all datasets. The choice of these parameters mainly follows PiCO 13. We present the full syper-parameters systematically in Table 7
input image size of
Table Hyper-parameters for partial label learning used in experiments.
Plyper-paummee
CIFAR-10
CIFAR-100
CUB-200
Image Size
32
32
224
ResNet-18
Model
ResNet-18
ResNet-18
(ImageNet-18 Pretrained)
Batch Size
256
256
256
Learning Rate
0,01
0.01
0.01
Weight Decay
le-3
le-3
le-5
LR Scheduler
Cosine
Cosine
Cosine
Training Epochs
800
800
300
Classes
100
D.2.2 Discussion
We additionally compare our method with R-CR [1], which uses a different architecture as the results in Table 1 R-CR uses Wide-ResNer34x 10 backbone, and adopts multiple strong data augmentations. also adjusts the loss weight along training. For fair comparison, we use the same architecture without multiple augmentation and the curriculum adjust loss. The results are shown in Table where our method outperforms R-CR on CIFAR-10 and is comparable on CIFAR-100.
Table 8: Comparison with R-CR in partial label learning
CIFAR-10
CIFAR-100
Method
0.3
0.5
0.05
0.10
R-CR
22
97.05=oos
82.77
82.24
Ours
97.55
97.17=0.1
82.46=ams
82,22+005
also provide the comparison our method on instance-dependent partial label learning as proposed by Xu et al. [8T] [25]. Due to the nature of instance-dependdnnee we maintain the term
P(S|Y, 0) from Eq. (5) as a supervised term for optimization. We compare our method with
VALEN [8I] RCR [1], PICO [13], and POP [125) on MNIST, Kuzushiji-MNIST, Fashion-MNIST,
CIFAR- 10, and CIFAR- 100, with synthetic instance-dependent partial labels generated according to Xu et al. [25). From the results in Table 9 we proposed method demonstrate the best performance
across different datasets evaluated.
A recent work on PLL discussed and analyzed the robustness performance of different loss functions, especially the average-based methods [58]. We perform similar analysis here for the derived loss function in ILL. Following the notation in [38, let denote the candidate label set, x as the training
instance, g as the probability score from the model, and I as the classifier f(xx arg maxgi (x), iey
the average-based PLL can be formulated
Laeg-PLL((x))
1
Σ (((()),))
(21)
8s
23

---

Table 9: Comparison on instance-dependent partial label learning
MNIST
Kuzushiji-MNN
Fashion- MNNIST
CIFAR-10
CIFAR-100
VALEN 81
99.03
90.15
96.31
92.01
71.48
RCR 12
98.81
90,62
96.64
86.11
71.07
PICO 13
98.76
88.87
94.83
89.35
66.30
POP 125
99.28
91.09
96.93
93.00
71.82
Ours
99.19
91.35
97.01
93.86
72.43
Lv et al. 58 compared different loss functions on both noise-free and noisy PL. settings, where they find both theoretically and empirically that average- prerage-based PL.. with bounded loss robust under mild assumptions. Empirical study in [58] suggests that both Mean Absolute Error and Generalized Cross-Entropy loss 29 that proposed for noisy label learning achieves the best performance and robustness for average-based PLL.
Our solution for PLL can be viewed as instantiation of the average- basede PLL as in [58 with:
{(((x), = -- -5. log (x)
(22)
where 7 is normalized probability over s with detached gradient. We can further show that the above loss function is bounded for 0 l < and thus bounded for summation of all classes, which demonstrates robustness, as we show in Table
D.3
Semi-Supervised Learning
D.3.1
Setup
For experiments SSL,, we follow the training and evaluation protocols of USB [82] on image and text classification. To construct the labeled dataset for semi-supervised learning, uniformly select I/C samples from each class and treat the remaining samples as the unlabeled dataset. For image classification tasks, ImageNet-IK IK [26 Vision Transformers
4 are used, including CIFAR-100 100 (77), EuroSAT (27), STL-10 128]. TissueMNIST [129 130), Semi-Aves [33]] For text classification tasks, adopt BERT 3 backbone, including IMDB (3), Amazon Review (84), Yelp Review
[62], AG News 133 Yahoo Answer
[34 The hyper-parameters strictly follow USB, and are
shown in Table 10 and Table
Table 10: Hyper- of semi-supervised learning used in vision experiments of USB.
Hyper pppaa-prrretter
CIFAR-100
STL-10
Euro-SAT
TissuMMNIST
Semi-Aves
Image Size
32
32
224 VIT.S P16 224
Model
VTT P4:32
VIT P16 96
VIT
V4T-T-44-32
Ratch size
Unlabeled Batch size
16
Learning Rate
5c-4
10-4
5e-5
10-3
Weight Decay
Se-4
1 Layer Decay Rate
0.5
0.95
1,0
0.95
0.65
LR Schoduler
Vio coe Tak
Training epochs
20
Classes
100
10
200
Model EMA Momentum
0.0
Prediction EMA Momentum
0.999
Weak Augmentation
Random Crop, Random Horizontal Flip
Strong Augmentation
Rand Augment
95
D.3.2 Results
In the main paper, we only provide the comparison on CIFAR-100, STL-10, IMDB, and Amazon Review. Here we provide the full comparison in Table T2 and Table 13 From the full results, similar conclusion can drawn as in the main paper. Our IL.. framework demonstrates comparable performance previous methods.
24

---

Table 11: Hyper-parameters of semi-supervised learning NLP experiments in USB.
Hyper-parameter
AG News
Yahoo! Answer
IMDB
Amazom-5
Yelp-5
Max Length
512
Model
Bert-Base
Labeled Batch size
Unlabeled Batch size
Learning Rate
5e-5
lc-4
5c-5
le-5
5e-5
Weight Decay
1o-4
Layer Decuy Rate
0.65
0.65
0.75
0.75
0.75
LR Scheduler
1)
1o cool 10
16
Training epochs
Classes
4
5
Model EMA Momentum
Prediction EMA Momentum
01,999
Weak Augmentation
None Back-Trunslation
Strong Augmentation
Table 12: Error rate comparison of different number of labels CIFAR-100, STL-10, EuroSAT, TiSSU, and SemiAves for semi-supervised learning. We use USB 82 image classification task results. The best results are indicated in bold. Our results are averaged over independent runs.
Datasets
CIFAR-100
STL-10
EuroSat
TissueMNIST
SemiAves
Lahels
200
400
40
100
20
40
80
400
39,59
Pscude-Label
[4
33 99.99
25. 322.22
19.14 33
10.77a0.0
25.4621.36
15.70s2
56.92s1.54
50.86s1.79
40,35e0030
Mean-Tenee Teacher
35.47
26.03+6.0
18.67=1.
24. 19-10.1
26.83+1.16
15.85=1.6
62. 02.00:2341
55.12+2.55
38.55+0211
VAT [00
31 39.13
21.34
18.45a14
10.69sa.11
26. 160.06
10.09
57.4925.47
51.30st.72
38. 828802
MixMalch
16
38. 28.22
26.72as.1
58.77a1.8
36.74.1.24
24.85a485
17.28:267
55.5311.5
49.64x2.25
37.25
ReMixMatch
22.21
16.8610.5
13.08
4.29
5,05
5.07.0.1
58.77
49.82
30.20keas
AdaMatch
85
22.32+1.71
66.06s
13.64-2..
7.62+1.0 90
7. 7.02+0:79
4.75+1.10
58.3544877
52.40.2.0
15
FixMatch 18
29. 60e0:000
19.56e9052
16. 15
8.1
13,4423.55
5.91=2.0
55.37=4.50
51.24+1.56
31. 9.00000000
FlexMatch 20
26.76at.12
18.24
14.40
8.177s
5.17a0.57
5.58
58.3623.80
51.8923.21
22.48
Dash [105
30.61
16.22 19.38
7.85
11.19.0.00
6, 6.96=224
56.984299
51.97e155
32. 38..6
CoMatch [6
35 35.08
25.35
15.12a1.8
9.56a1.35
5.752043
4.81a105
59.0424.0
52.92s1.04
25.
SimMatch [87
23.78.108
17.06uc.s
11.77.300
7.5511.8
7.66u060
5.27,0s9
60.881431
33.85 52.9311.5
FreeMatch 22
21.40+
15.65.a.26
12.73
8.52
6. 6.50wo7s
5.78-0.1
58. 58.24.508
52.19
32.8 35
SoftMatch
22.67a1.33
16.842066
13.55e3.1
7.84
5.75
5.902142
57.982306
51.73x2.44
31 80.0.22
Ours
22.06s1.06
17.40sh.04
11.09
8.10sin2
5.86+1
5,74
57.99.216
50.95x243
33.08
Table 13: Error rate comparison of different number of labels on IMDB, AG News, Amazon Review, Yahoo Answers, and Yelp Review for semi-supervised learning. We use USB [2] text classification task results. Best results are indicated in bold. Our results are averaged over 3 independent runs.
Datasets
IMDR
AG News
Amazon Review
Yahoo Answers
Yelp Review
# Labels
20
100
40
200
250
1000
500
2000
250
1000
Pseudos-Lbbel
14
45.45
19.67-1m
19.49.xar
14,69.aa
53.45a1s
47. 47.00
37.70
32.72am:
m4.5
47.33
Mean Teacher
103
20.06a2.31
13.97a1.9
15.17a12:
13,93aae
52.14a052
47. 47.66am1
37 37.09aum
33.43am2
50.60lass
95.93
11.61-124
14.70-184
11.71-a31
49.8340.
46.54.m
14.87
31.50
52.9741
45.30-0
MixMatch
16
26. 12
15.47:005
13.50s1.1
11.75:aum
59.54sam
61 61.693333
35 75
53 12.G2a
53 98...8
$1,70anss
AdaMatch
8.09-a.
7.114000
11.73am
11.22.aot
46.72-o7
42.27
32.75wm
30.44.m
45 450m00
40. 16-019
FixMatch [8
7.72at39
7.336513
30.17
11.71=105
47.61=288
43.05
33.03sa.9
30.51
46 $2..4
40.65a045
FeexMath 20
7,82un:77
7.41as3s
16.38.334
12.08aa7s
45.73a:m
42. 42.25
35.61ai1
31.13ar1s
43.35uu4
40.51amm
Dash [105
8,34an
7.55+6.3
17.6723.9
13.76
47.10a074
43.09
35.26,0.33
31.19
45
40. 40.14,039
CoMatch 86
7,44ac03
7.72a114
11.95acee
10.75uncs
48.76ucmm
43.36umm
33.48a131
30.25
45.40
40.27amm
SimMatch [
7.93-a55
7,08-4.3
14. 51
12.45-1.7
45.914095
40
33 06
30. 16-17
46.12
40, 40.26-0ax
FreeMatch 23
8.94.mm
7.95aoas
12.98auss
11.73am4
46.41acm
42.64mmte
32.77wiz
30.32mu1
47.95a1
40.37ar8
SoftMatch
3
7.76-a5
7.97+mm
11. 10.00+m2
11.72-1.58
45.296o95
42.21
33.07w031
30.44.0
44 44.09
39,76-m
Ours
7,32-a12
7.6440m7
14.774158
12.21-aar
43.96.8.22
42.324mm
33 33.80
30.86
44.8240
39.67-mm1
D.4
Noisy Label Learning
D.4.1
Setup
We conduct experiments of noisy label learning following SOP 69). We evaluate the proposed method on both synthetic symmetric/asymmeetie noise on CIFAR- 10 and CIFAR-100, and more realistic and larger-scale instance noise on Clothing M and Web WesVision. To introduce the synthetic symmetric noise to CIFAR-10 and CIFAR-100, we uniformly flip labels for probability into other classes. For asymmetric noise, we only randomly flip the labels for particular pairs of classes. For CIFAR- 10 and CIFAR- 100, we train PreAct-ResNet-I with SGD using a learning rate of 0.02,
25

---

weight decay of 1e 3. and momentum of 0.9. We train for 300 epochs with a cosine learning rate schedule and batch size of 128. For Web Vision, we use InceptionResNet-22 as the backbone and set the batch size to 32. Other settings are similar to CIFAR-10. For Clothing M, we use ImageNet- K pre trained ResNet-50 as the hackbone. We train using SGD with an initial learning rate of 2e-3 for total of 10 epochs, where the learning rate is reduced by 10 after 5 epochs. In addition, we also conduct experiments on CIFAR-ION and CIFAR-10ON, We present the detailed hyper-parametent in Table [4
Table 14: Hyper-parameters for noisy label learning used in experiments.
popaeepurameter CIEAR-IO (CIFAR-ION)
CIFAR-100 (CIFAR-1OUN)
Clochng IM
WebWisson
Image Size
32
32
224
299
Model
RooNes-18 PreAct-RoNNet
PreAct-
18 (ResNot-34)
PreAct-ResNa-IB (RosNet-34)
(ImageNet- LK Pretrained)
ResNet:50
Inception ResNet--2
Batch Siz:
128
128
64
32
Learning
0.02
0.02
0.002
0.02
Weight Decay
LR Scheduler
Cosine
Coxine
MultiSup
MuldStep
Training Epochss
500
300
10
100
Classes Noisy Maini Scale
10
100
14
50
1.0
2.0
0,5
2.5
D.4.2 Results
In addition the results regarding noisy label learning provided in the main paper, we also present comparison results on CIFAR- 1ON and CIFAR-10ON 10ON [92] in T5] We include a full comparison on Clothing M and WeiVinion, incorporating methods like Co- Teaching, Forward, and CORES, in Table 16 As shown in Table 15 the proposed IIII. framework achieves performance comparable to the previous best method, SOP 669. On CIFAR- 1ON, our method yields results very close to SOP in the Random and Aggregate case noise scenarios and surpasses SOP in the Worst case noise scenario. However, on LIFAR-IOON, our method slightly underperforms previous methods, possibly due to the oversimplified noise model utilized in ILL. We believe that a more realistic noise transition model and further tuning of our method could lead to improved performance.
Table 15:
Test accuracy comparison of instance independent label noise on CIFAR- 1ON and CIFAR- 100N for noisy label learning. The best results indicated in bold, and the second best results are indicated underline, Our results are averaged over three independent runs with ResNet34 as the backbone.
Datasct
CIFAR-ION
CIFAR-IOG
Noisy Type
Clean
Random 1
Random 2
Random 3
Aggregate
Worst
Clean
Noisy
CE
92.92s0.1
85.02=0.65
86.46:1.79
16.161
87.77
17.69.1.5
76.70aa74
55.50s0
Forward
91
93.02a011
86.88
14,122
87.04
88.24a0.22
79.79aa.46
18.08
$7.01=1.0
Co-teaching
28
93.35a0.14
90.33.0.13
90.30ua.17
90.15
91.20s0.13
83.83an.15
73.46x009
60.37a0.7
DivideMix
95. 16
95.23-0.07
95.21-0.14
95.01
92.56=042
71.13:048
E
32
95.39x0as
94,43+0.41
94,20-0.44
94.34+0.32
94.83
91,09-1 so
78.574012
66.7240m7
CORES 135
94. 10..16el 16
94,45.0.14
94.88s0.11
94.74.0.0
95. 25.00
91. 66.60000
73.8740 16
55.72.0.42
SOP 69
96.38sa.31
95.28.0.13
15.31
95.39-0.11
95.61
93.24-0211
78.91
67. 13281
Ours
96.21.029
96.06.n.77
95.98an.12
96.10
96.40ha0.3
93,55am.14
78.53a0.2
68.07
D.5
Mixed Imprecise Label Learning
D.5.1
Setup
To create mixture of various imprecise label configurations, we select CIFAR- 10 and CIFAR-100 as base datasets. We first uniformly sample I/C labeled samples from each class to form the labeled dataset and treat the remaining samples as the unlabeled dataset. Based on the labeled dataset, we generate partially labeled datasets by flipping negative labels to false positive labels with the partial ratio q. After obtaining the partial labels, we randomly select percentage of samples from each class, and recreate the partial labels for them by flipping the ground truth label uniformly to another class. In this setting, unlabeled data, partially labeled data, and noisy labeled data exist simultaneously. which very challenging and more closely resembles realistie situations. For CIFAR-10, 10, we set I e {1000, 5000, 50000), and for 100, set I e {5000, 10000, 500003. Similarly in
26

---

Table 16: Test accuracy comparison of realistic noisy labels on Clothing IM and WebVision for noisy label learning. The best results are indicated bold and the second best results are indicated in underline. Our results are averaged over independent runs. For C we use ImageNet- IK pre trained ResNet50 as the backbone. For WebVision InceptionResNetv22 used the backbone.
Dataset
Clothing IM
WebVision
CR
69.10
Forward
91
69.80
61.10
MentorNet
66.17
63.00
Co-Teaching
69.20
63.60
DivideMix
74.76
77.32
ELR 32
72.90
76.20
CORES 135
73.20
SOP 69
73.50
76.60
Ours
74.02-0.1
79.37
the partial label setting, we set E {0.1, 0.3, for CIFAR-10, and E {0.01, 0.05, 0.13 for CIFAR-100. For noisy labels, set v E 0..1,0.2,0.3 for both datasets.
D.5.2 Results
We provide a more complete version Table Table 17 On partial noisy labels of CIFAR-10 10 with partial ratio 0.5 and of CIFAR-IOK with partial ratio 0.1, most baseline methods are more robust or even fail to perform. However, our ILL still shows very robust performance with minor performance degradation increase of noise ratios.
D.6 Ablation Strong-Augmmentttion and Entropy Loss
We provide the ablation study on the strong- arong-uggmentation and entropy loss components here, which common techniques in each setting [T8 1 69). For example, in SSL,, strong- augmentation
an important strategy for SSL algorithms widely used existing works such as FixMatch [18] and Flex (20) Thus, it is important adopt strong-weak augmentation
achieve better
performance in SSL 122 23 82 This is similar in PLL settings [13] 12). PiCO [13 [2] also used
strong augmentation). Strong-weak augmentation and entropy loss are also adopted in SOP [69] of NLL. However, we found these techniques are less important for our formulation of NLL. We provide an ablation study on the entropy loss of SSL, and both techniques for NLL and PLL here to demonstrate our discussions.
D.7 Runtime Analysis
We provide the runtime analysis CIFAR-100 of method different settings, compared with the SOTA baselines. We compute the average runtime from all training iterations on CIFAR-100. The results are shown in Table 21 Our method in general present faster runtime without complex design such as contrastive loss.
27

---

Table 17: Test accuracy comparison of mixture of different imprecise labels. We report results of full labels, partial ratio q of (0.1, 0.3, 0.53 for CIFAR-10 and 00.01,0005, 0.1} for CIFAR-100, and noise ratio of {0.1, 0.2, 0.3) for CIFAR-10 and CIFAR-100.
Dataset
Labels
Partial Ratio
Noise Ratio
0.1
0.2
0.3
PiCO+ 94
95.99
93.64
93.
92.18
IRNet 71
93.44
92.57
92.38
DALI 50
94.15
94.04
93.77
0.1
PiCO+ w/ Mixup 50
94.58
94.74
94.43
DALI w/ Mixup 50
95.83
95.86
95.75
Ours
96.55+n.0s
96.47=0.11
96.09+0.29
95.83
PiCO+ 94
95.73+0.10
92.32
92.22
89.95
IRNet
92.81
92.18
91.35
DALI 50
93.44
93.25
92.42
10
50,000
0.3
PiCO+ w/ Mixup 50
94.02
94.03
92.94
DALI w/ Mixup
50
95.52
95.41
94.67
Ours
96.52an.12
96.2sn.22
95.87an.14
95.22=0.06
PICO+
95.33+11.6
91.07
89.68
84.08
IRNet
91.51
90.76
86.19
DALI
92.67
91.83
89.8
0.5
PiCO+ w/ Mixup
93.56
92.65
88.21
DALI w/ Mixup
95.19
93.89
92.26
Ours
96.28s0.13
95.82:0.07
95.28s0.08
94.35
PICO+ 05
76.29 26.29.0.42
71.42
70.22
66.14
IRNet 7
71.17
70.10
68.77
DALI 50
72.26
71.98
71.04
0.01
PiCO+ w/ Mixup
30
75.04
74.31
71.79
DALI w/ Mixup
50
76.52
76.55
76.09
Ours
78.08a0.26
77.53.0.24
76.96
76.43=a.27
PiCO+
76. 170111
69.40
66.67
62.24
IRNet
70.73
69.33
68.09
DALI
72.28
71.35
70.05
0.05
CIFAR-100
50,000
PiCO+ w/ Mixup
73.06
71.37
67.56
DALI w/ Mixup
76.87
75.23
74.49
Ours
76.95an.46
77.07=0.16
76.3420.05
75.13
PiCO+ 94
75.55+0.21
IRNet 71
DALI 50)
0.1
PiCO+ w/ Mixup 50
DALI w/ Mixup 50
Ours
76.41=1.02
75.50=a.4
74.67=0.30
73.88
Table 20: NLL ablation
Table 19: PLL. ablation
CIFARIO
CIFARIOU
Table 21: Runtime Anal-
Table 18: SSL ablation
CITARIO
CIFARTOD
SOP
94.00
0.5
9 0.1
63.30
ysis on CIFAR-100
0.5
0.1
Ours
94.31
66.46
SOP
CIFARTOD
Ours
95.91
74.00
w/o
66.85
36.60
Seting
Algoritim
CIFAR-I00 Aug. Runtime (xiter)
t=200
1=40
PICO
Ours
22.06
11.09
wlo
91.78
66.43
Ours
SSL
PreeManch
0.2157
Ours
ent.
22.41
11.23
Ours
w/o
94.53
72.69
PLL
SSL
PLI
PSCO
Oura
Ours
0.3249
0.1186
0:2019
Ours
wlo
93.04
62.85
NI.1 NLL
Ours
SDP
0.1021
B.1176
73.75
Ours
cnl.
94.15
66.12
ent.
28

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's i contributions and scope?
Question:
Answer: [Yes]
Justification:
We discussed our contribution in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed our limitation in conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should diseuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
29

---

Justification:
All are stated in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification:
We present all details in both main paper and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily to reproduce that algorithm.
algorithm, the paper should make clear how
(b) the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reprodueibility may tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
30

---

Answer: [Yes]
Justification:
We use publically available data and code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ismiissinn ssionPolccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparame- ters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All results obtained with runs using different seeds and error bars reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the standard deviation in the results, which averaged over independent runs across different experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
31

---

It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (lype of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We showed in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://stts::ssppppsst http:///wwwpp..cctccccccaaaaaaal
Answer: [Yes]
Justification:
All behaviors follows NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special considera- tion due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed in conclusion.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address socictal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedhack over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
(uswer:[Yes)
Justification:
We discussed in experiments.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. wwtoodd..coo//ooo /amtttttsttss has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "ZNm2YLEJRH", "reviewer": "Reviewer_Z7AB", "summary": "The paper provides a unified view on various imprecise-label learning frameworks, such as semi-supervised-, partial-label-, or noise-label learning, through the lens of the expectation-maximization algorithm. In addition to unifying these existing setups, EM naturally allows treating combinations of the above setups. Experiments show that the proposed method compares favourably with existing methods specialized to just one setup.", "questions": "See Weaknesses.", "limitations": "The authors adequately concede in section 5 that the scalability of the algorithm hasn't been verified yet, sentences like\n> While there exist earlier attempts on generalized or EM solutions [...] thus presenting limited 70 scalability on practical settings.\n\n(l. 68-71) seem to suggest that the presented algorithm has this not-yet-verified scalability.", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The proposed method generalizes to a wide range of settings, and performs on par or better with more specialized algorithms in most of the evaluations.", "weaknesses": "* Neither abstract nor introduction make it clear that the paper is concerned purely with multi-class classification with deterministic labels y=f(x)\n\n* This stands in contrast to the critique in l. 70 regarding competing, `[...] they usually require additional assumptions 69 and approximations on the imprecise information for learnablility`: Assuming deterministic labels is a _very_ helpful simplification for noisy labels, as, together with some upper-bound on the noise rate, it restores identifiability of the model\n\n* A serious problem with the writing of the paper is that, for an attempt at introducing a probabilistic model that can then be used in EM, it does not actually write down the actual probabilisitic models it considers:\nIn this sentence (l. 158), the paper is extremely vague: \n> If I represents partial labels, then P(Y |I) would 158 have non-zero value over the candidate labels, and be 0 elsewhere. When I represents a set of noisy 159 labels, P(Y |I) would represent the distribution of the true labels, given the noisy labels. When I 160 does not contain any information, i.e., unlabeled data, Y can take any value.\n\nfor partial labels, is the underlying assumption $P(Y|I) = 1(Y in I) / |I|$, i.e., uniform distribution over all candidates? How about for unlabeled data?\nfor noisy labels, you either need to assume a fixed noise model, or $P(Y|I,\\theta)$ where $\\theta$ are the parameters of the noise model. \nl. 170: `Note that P (X; θ) is omitted from Eq. (5) since P (X) 170 does not rely on θ.` Why? Is this a new assumption? Earlier in the paper, $\\theta$ was introduced as the modelling parameter of the generic joint distribution `Let P (X, I; θ) represent a parametric form for the joint distribution of X and I`. The footnote claims `he actual parameters θ may apply only to some component such as P (Y |X; θ) of the overall distribution` but to me, \"may\" here means that in some situations, such a restriction is possible, whereas I guess the intended meaning is that the model is _always_ supposed to be $P(Y|X,\\theta)$?\nl. 174: `For independent instances setting` again, this reads as if the previous section had situated the paper in the independent instance setting, whereas this is the first time the topic comes up\n\n> The property of the second term log P (I|X, Y ; θ) is dependent on the nature of imprecise label I. If I contains information about the true labels Y , such as the actual labels or the label candidates, it can be reduced to P (I|Y ), i.e., the probability of I is no longer dependent on X or θ and thus can be ignored from Eq. (5).\n\nThis is not correct, just because I contains information about Y, the probability cannot be simplified.\n\nEquation (6) seems to appear out of nowhere: Even following the derivations in C.2, $\\mathcal{A}_s$ and $\\mathcal{A}_w$ just appear out of thin air in these equations. \nl. 223: `Things become more complicated here since the noisy labels $\\hat{Y}$ do not directly reveal the true information about $Y$` I'm not sure what this sentence is trying to say.  $\\hat{Y}$ should have some information about $Y$, otherwise, learning is impossible; of course, it doesn't have the full information, but neither do partial labels, so I don't see how this changes the situation compared to the preceeding paragraphs.\n\n\nThe (unreferenced in the main paper) section D.7 in the appendix claims\n> Since the settings studied in this work has loss functions derived 943 as close-form from Eq. (5), the time complexity can be viewed as O(1). Thus our method in general present faster runtime without complex design such as contrastive loss.\n\nThis argument doesn't make any sense. Solving a system of $n$ linear equations can be written in closed form, yet this is not an O(1) operation.\n\nOverall, after reading the paper, I have almost no idea what the proposed method actually does. Equations (6), (7), (8) contain data augmentations, which I suspect may be important for attaining the performance reported in the paper(?), yet they are not really discussed as part of the proposed method. There are numerous inaccuracies, gaps, and I think even some errors (not fundamental, I think, but in the way the writing describes the math). It is certainly possible that I am misunderstanding something here, but as I see it right now, the paper is not in a shape in which it should be published.\n\n\nTypos/grammar:\nl. 87: our proposed method _generalise_ and subsumes\nl. 148: we consider all possible _labeling_ along", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and informative review.\nWe recognize several of the concerns expressed by the reviewer and agree that addressing these will indeed enhance the paper.  We have attempted to do so below. We hope this is satisfactory.\n\n---\n> Neither abstract...\n\nThe setting studied in this paper is multi-class classification, as is common for almost all reported work on weak supervision, especially baselines covered in this paper. \n\n> This stands in contrast..it restores identifiability of the model.\n\nPlease note that the model is **not** dependent on the assumption of a unique label.\nIn the general case we are given a label vector $Y \\in \\mathbb{R}^C$.   For unique labels, $Y$ is one-hot. For uncertain labels, it will be dense. The loss computation remains the same in both cases, and the EM solution still holds.\n* Specifically, for noisy labels, label noise can be captured by a noise-transition matrix $\\mathcal{T}(\\hat{Y}|Y;\\omega)$ (Eq 8). The noisy label $\\hat{Y}$ is simply $\\hat{Y} = \\mathcal{T} Y$. The EM solution works with $\\hat{Y}$ to infer $\\mathcal{T}$ and the model parameters under $Y$.  The framework holds regardless of whether $Y$ is one-hot or dense. \n* **Even** with one-hot $Y$ the model is non-identifiable.  Consider any permutation matrix $P$.  We have $\\hat{Y} = \\mathcal{T}Y = (\\mathcal{T}P)(P^T Y)$, i.e. it is not permutation invariant.  Consequently, the model is not identifiable.  This is regardless of whether $Y$ is one hot or dense. \n\n> A serious...vague:\n\nAs explained above, we model $P(X, I; \\theta)$ in Eq4. The missing data here are $Y$ as evident from Eq 4 and 5. This level of description fully specifies the EM problem, although for the specific update rules the form of parameterization becomes important to state.  In our case, we use the decomposition of Eq 5, wherein we model $P(Y|X; \\theta)$ using neural nets, leading to the solutions of Eq6-10. \n\n> For partial labels...\n\nThe EM solution generalizes to any $P(Y|I)$. \n\nSpecifically, we study two types of partial labels: instance-independent and instance-dependent ones.\nThe label candidates of the first follows the uniform assumption; the second follows the instance-level strategy.  On both types of partial labels, our method presents superior performance (Tab 1 and 9). \n\n> unlabeld data…\n\nFor unlabeled data,  $P(Y|I;\\theta)$ is also learned (Eq 7) as in the case of the partial label. Our methods perform robustly with different numbers of unlabeled data (Tab 2).  Note that our solution does not preclude the use of a known or assumed prior $P(Y|I)$ (e.g. uniform), in which case this distribution would not be learned.\n\n> noisy labels…\n\nFor noisy labels, $P(Y|I,\\theta)$ is modeled through a noise transition matrix, which is learned, as explained in (Eq 8 and 9). \n\n>  l. 170: Note that P (X; θ) is omitted...assumption?\n\nThe primary component of the model learns the mapping $P(Y|X)$, since this is not a generative model for $X$.  As such, $P(X)$ need not be modeled.  This assumption is common in the training of discriminative models.\n\n$P(Y|X)$ in turn may be decomposed into $P(\\{Y_i\\} | X)$, which permits modeling of the distribution of subsets of components of $Y$. The ``may’’ merely includes this possibility.\n\n> This is not correct...simplified.\n\nAs stated in the paper, **whether $P(I|Y, X;\\theta)$ reduces to $P(I|Y)$ depends on the nature of I**.\nIf I is directly drivable from Y, i.e, $I=f(Y)$, then $P(I|Y,X;\\theta) = P(I|Y)$ e.g. $I = Y$, $I = count(Y)$, etc.\nWe have also considered other settings where $I$ is dependent on $X$, and this is not reducible, as in Tab 3, 9, and 15.\nE.g. in noisy labels, it is treated as a noise transition model.\n\n> Equation (6) seems to...equations. \n\n$\\mathcal{A}_s$ and $\\mathcal{A}_w$ represent strong and weak augmentation, and are actually defined early in the paper before Eq 1 (L101-102). \nBesides, from C.2, from Eq13 to 14, what we did is replace the random variable with actual data notation to make it a true loss function. \nIf the reviewer thinks this is still vague, we will try to make it more clear in our revised paper.\n\n> l. 174: For independent instances...up\n\nThe typical setting is where the data may be viewed as a collection of independent instances. In other settings (e.g, streaming), the labels in some instances may bear statistical dependence on those in other instances. Our framework permits both settings and experiments including both instance-independent and dependent settings.\nWe will try to make this more clear in the revised paper.\n\n> l. 223 Things become more...paragraphs.\n\nIn the noise-free case, $I$ presentes uncorrupted information about $Y$.  E.g., it might state that a collection of instances includes a particular label, without identifying the specific instances that do.  While the label information provided is incomplete, what is provided is true.\n\nIn the noisy case, an instance labeled as $y$ does not imply that its label is actually $y$.    We will clarify this in Sec 4.3.\n\n>  Equations (6), (7), (8) contain data...method. \n\nAs mentioned in the paper, these are common techniques in this field.\nBesides, we have an ablation in Tab 19 of Appendix on strong-weak augmentation.\nRemoving strong-weak augmentation leads the baselines to significant performance drop, whereas the effects on ours are much smaller. \n\n> This argument doesn't … O(1) operation.\n\nThanks to the reviewer for pointing out this error. We were trying to convey that for closed-form solutions, once sufficient statistics are gathered, the subsequent estimator has a fixed complexity, whereas for iterative solutions this is not true.  However, the actual computation for gathering statistics would be $\\mathcal{O}(N)$ as in other models. We will make this more clear in our revised paper.\n\nWe hope our above response addresses the reviewer’s concerns. If you have further concerns, please do not hesitate to let us know:)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FL5z8CXTGF", "reviewer": "Reviewer_GFuM", "summary": "The article addresses the challenge of learning with imprecise labels in machine learning tasks, such as noisy or partial labels. Traditional methods often struggle with multiple forms of label imprecision. The authors introduce a novel framework named Imprecise Label Learning (ILL) that serves as a unified approach to handle various imprecise label scenarios. ILL employs the expectation-maximization (EM) technique, viewing precise labels as latent variables and focusing on the entire potential label distribution. The framework demonstrates adaptability to different learning setups, including partial label learning and noisy label learning.", "questions": "1. Given that the method of using ground-truth labels or Bayes label distribution as latent variables coupled with variational inference in weakly supervised learning is highlighted in prior works, how does the presented framework distinguish itself or advance beyond these existing approaches in terms of innovation or application?\n1. Could you provide a detailed analysis explaining why the unified framework demonstrates superiority over specifically designed approaches?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The article is well-written, with clear and concise language that effectively conveys the main ideas and contributions of the research. Also, comprehensive derivation of loss functions for the three imprecise annotations configurations derived from equation 5 are given which ensure clarity and thorough understanding for the readers.\n2. The article offers a comprehensive solution to the prevalent challenge of imprecise annotations, enhancing the adaptability and applicability of machine learning models.\n3. The inclusion of experimental results across multiple settings provides empirical evidence of the framework's robustness and superior performance.", "weaknesses": "1. The article's innovation is limited, as the approach of considering ground-truth labels or Bayes label distribution as latent variables and using variational inference for approximation in weakly supervised learning is already a common method[1-2], which suggests that the presented techniques may not be as novel as claimed.\n\n   [1] Xu, N., Qiao, C., Geng, X., & Zhang, M. L. (2021). Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34, 27119-27130. \n\n   [2] Yao, Y., Liu, T., Gong, M., Han, B., Niu, G., & Zhang, K. (2021). Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34, 4409-4420.\n\n2. Some important baselines should be compared, such as [1,2] in SSL.\n\n   [1] Nguyen, Khanh-Binh, and Joon-Sung Yang. \"Boosting Semi-Supervised Learning by bridging high and low-confidence predictions.\" *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.\n\n   [2] Schmutz, Hugo, Olivier Humbert, and Pierre-Alexandre Mattei. \"Don’t fear the unlabelled: safe semi-supervised learning via debiasing.\" *The Eleventh International Conference on Learning Representations*. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's time reviewing this paper and suggestions on the relevant works. We address the concerns raised as follows.\n\n---\n\n> The article's innovation is limited...which suggests that the presented techniques may not be as novel as claimed.\n\nThanks for mentioning these relevant works. \nWe need to emphasize that, although there are existing methods that share the similar variational approximation for handling weak supervision (as we also discussed in our paper), there are three fundamental differences and improvement of our method:\n\n* **The algorithm/loss level**: We provide a new perspective deriving the loss objectives *directly* from the EM view of various kinds of weak supervision. Most of the existing works use the EM/ELBO to explain their proposed methods, which means their methods are *not* originated from EM and usually rely other sophisticated designs. Instead, our framework only uses the derived loss functions without other sophisticated design while achieving comparable performance. \n* **Our general nature**: Our framework is the first one that can naturally and universally handle the mixture of various weak supervision settings. \n\nIn summary, most of weak supervision methods can be explained/understood from the EM perspective, however, our method is naturally derived from and only rely on the EM, and applicable to various settings. \nWe will include the mentioned works into discussion in our revised paper.\n\n[1] Xu, N., Qiao, C., Geng, X., & Zhang, M. L. (2021). Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34, 27119-27130.\n\n[2] Yao, Y., Liu, T., Gong, M., Han, B., Niu, G., & Zhang, K. (2021). Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34, 4409-4420.\n\n\n> Some important baselines should be compared, such as [1,2] in SSL.\n\nThanks for mentioning these relevant baselines. \nWe omitted some baseline comparison on the paper mainly due the space limit. \nMore baselines of all settings can be found from our Appendix. \nFor the mentioned SSL baselines, we provide a brief comparison of error rate below: \n\n| Method \t| CIFAR-100 400 labels \t| STL-10 40 labels \t|\n|:---:|:---:|:---:|\n| [1] \t| 46.12 \t| 28.66 \t|\n|   [2] \t| 44.98 \t| 29.42 \t|\n| Ours \t| 38.24 \t| 20.19 \t|\n\n\nNote that this comparison is based on TorchSSL benchmark, instead of USB. \nWe will include these baseline comparison in our revised paper.\n\n[1] Nguyen, Khanh-Binh, and Joon-Sung Yang. \"Boosting Semi-Supervised Learning by bridging high and low-confidence predictions.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Schmutz, Hugo, Olivier Humbert, and Pierre-Alexandre Mattei. \"Don’t fear the unlabelled: safe semi-supervised learning via debiasing.\" The Eleventh International Conference on Learning Representations. 2022.\n\n> Given that the method of using...beyond these existing approaches in terms of innovation or application?\n\nAs discussed earlier, our method distinguish from existing approaches in the following aspects:\n\n* Earlier attempts usually require additional assumptions (line 68 - 71) to handle a single type of weak supervision, whereas ours does not and naturally and universally handles various type of weak supervision.\n* Unlike most of previous methods, our proposed method is *naturally derived* from EM and does not require other sophisticated design (model architecture, extra loss function, etc.). This demonstrates the scalability of our method. \n* The proposed method naturally extends to mixed weak supervision settings and also other weak supervision settings not studied in the paper. We need to highlight that this is the first method that handles mixed weak supervision settings robustly using a unified framework.\n\n\n> Could you provide a detailed analysis explaining why the unified framework demonstrates superiority over specifically designed approaches?\n\n\nThank you for this insightful question. Our unified framework demonstrates superiority over specifically designed approaches due to several key factors:\n\n* **Simplicity and generalizability**: Unlike many existing methods that require complex and specifically tailored architectures or additional loss functions for each type of weak supervision, our approach leverages a unified EM-based framework that is naturally derived without needing additional sophisticated design elements. This simplicity allows for a more straightforward implementation while still achieving competitive or superior performance across different types of weak supervision settings.\n* **Versatility in handling mixed supervision**: Many existing approaches are tailored to handle only a single type of weak supervision, often requiring additional assumptions or modifications to the model when dealing with different forms of supervision. In contrast, our framework is the first to naturally and universally handle mixed types of weak supervision without requiring any additional assumptions or modifications. This versatility makes our method more robust and adaptable to various real-world scenarios where mixed supervision is common.\n* **Scalability and robustness**: The unified nature of our framework ensures that it can be easily extended to new and unseen weak supervision settings without needing to redesign or retrain the model from scratch. This scalability is a significant advantage over specifically designed approaches, which often struggle to maintain performance when extended to different or more complex weak supervision scenarios.\n* **Theoretical foundation**: Our method is firmly grounded in the Expectation-Maximization (EM) algorithm, providing a solid theoretical foundation that ensures the method's validity and robustness. \n---\nIf you find our revise and response helpful, please consider raising the score to better support of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rajvKqtzw8", "reviewer": "Reviewer_JXvz", "summary": "This paper introduces a framework that unifies various imprecise label configurations, with an EM modeling for imprecise label information. The framework is demonstrated can be adapted to partial label learning, semi-supervised learning, and noisy label learning, and the combinations of all above. The experiments results show that the framework surpasses existing methods on various settings.", "questions": "- Can the framework be extended to handle other forms of weak supervision (such as imbalanced noisy label learning) and how?", "limitations": "-\tComputational efficiency of EM regarding the weakness above. It would be better that the author provide (theoretical or empirical) analyses regarding the computation of the algorithm.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThis paper proposed a unified framework that can unify various imprecise label learning settings, reducing the need for separate designs and solutions for each type of label imprecision.\n-\tPromising performance is achieved in individual settings and mixture settings with the unified framework.\n-\tThe proposed method is highly versatile and can be applied to the setting of a mixture of imprecise labels with robust performance.\n-\tThe framework demonstrates scalability on larger and more complex datasets.", "weaknesses": "- The implementation of EM over all possible labelings may increase the computation time.\n- More related works need to be discussed. The author considers the ground-truth or Bayes label distribution as the latent variables and leverages variational inference for estimating. I am not should this strategy is novel enough in the field of variational inference. I suggest the author add more related works to highlight the novelty of their technique.\n- The author should give an explanation of why utilizing an EM framework optimizes the variational lower bound. What is its advantage?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's efforts reviewing this paper and acknowledge of our work. We address the concerns as follows.\n\n---\n\n> The implementation of EM over all possible labelings may increase the computation time.\n\nFor all the weak supervision settings studied in this paper, our method introduce *negligible* computation time. This is because all the weak supervision considered in this paper impose conditional independent latent ground truth variable. \nA runtime analysis can be found in Table 21 of our Appendix, where the proposed method is usually *faster* than previous SOTA methods. \n\n> More related works need to be discussed. The author considers the ground-truth or Bayes label distribution as the latent variables and leverages variational inference for estimating. I am not should this strategy is novel enough in the field of variational inference. I suggest the author add more related works to highlight the novelty of their technique.\n\nThanks for pointing out this. As we discussed in our paper (line 68 - 78), EM solution for weak supervision is long desirable in the community and there are earlier attempts and relevant methods.  \nHowever, we would like to highlight that our method is the first method fully based on EM, without other model and loss design, which universally handles multiple and mixed types of weak supervision. \nThis distinguishes us from previous works.\n\nWe also added the following extra relevant works of VI and EM on weak supervision to the discussion of our paper. Per discussion above, these method do not rely on EM itself to achieve reasonable performance.\n\n[1] Xu, N., Qiao, C., Geng, X., & Zhang, M. L. (2021). Instance-dependent partial label learning. Advances in Neural Information Processing Systems.\n\n[2]  Garg, A., Nguyen, C., Felix, R., Do, T.T. and Carneiro, G., 2023. Instance-dependent noisy label learning via graphical modelling. In Proceedings of the IEEE/CVF winter conference on applications of computer vision.\n\n[3] Elhassouny, A. and Idbrahim, S., 2022, December. Deep learning with noisy labels: Learning True Labels as Discrete Latent Variable. In 2022 14th International Conference on Computational Intelligence and Communication Networks.\n\n\n\n> The author should give an explanation of why utilizing an EM framework optimizes the variational lower bound. What is its advantage?\n\n\nThe EM algorithm provides a robust method for handling latent variables by decomposing the maximization of the likelihood function into an E-step and M-step.\nEM algorithm also has provable property of convergence. \nBesides, the EM framework is inherently designed to optimize the variational lower bound, as we shown in Appendix C.1.\n\nThere is also possible improvements over EM, such as ECM [1] and PX-EM [2], which could be incorporated in our method for further improvements. Due to the scope of this study, this is left as our future work. \n\n[1] X. L. Meng and D. B. Rubin. Maximum likelihood via the ECM algorithm: a general framework. Biometrika, 1993.\n\n[2] C. Liu, D. B. Rubin, and Y. N. Wu. Parameter Expansion to Accelerate EM: The PX-EM Algorithm. Biometrika, 1998\n\n\n\n> Can the framework be extended to handle other forms of weak supervision (such as imbalanced noisy label learning) and how?\n\nYes. \nThe proposed unified framework can be easily extend to other weak supervision settings. \nIn this study, we mainly consider three types of weak supervision, where the imposed ground truth label is conditional independent given the weak supervision. \nOur framework can be extended to similar settings without any change, such as positive unlabeled learning, \n\nAs for imbalanced noisy label learning, it is a configuration regarding changes in $P(X)$ itself rather than weak supervision $P(I, Y)$. \nWhile our method is not specifically designed for such settings (change in $P(X)$), we showed in Table 3 that our method achieves comparable results on imbalanced noisy label datasets (Clothing1M). \nFor more imbalanced noisy settings, we provide an additional comparison here, following the experiment setting of [1], where we introduce 5:1 head-tail class imbalanced into noisy CIFAR-10 and CIFAR-100 with a ratio of 0.2 below.\n\n|  | CIFAR-10, $\\eta=0.2$, $1:5$ \t| CIFAR-100, $\\eta=0.2$, $1:5$ \t|\n|:---:|:---:|:---:|\n| CE \t| 87.1 \t| 64.9 \t|\n| DivideMix \t| 93.9 \t| 65.0 \t|\n| ELR+ \t| 88.2 \t| 59.6 \t|\n| ULC \t| 95.0 \t| 75.5 \t|\n| Ours \t| 94.8 \t| 73.5 \t|\n\nFrom the results, we can summarize that previous methods specifically designed for noisy label learning (DivideMix and ELR+) usually present large performance degeneration on imbalanced settings, whereas our method present comparable performance with method specifically designed for this setting (ULC), demonstrating the extensibility of our method. \nOur method can also be combined with other design or techniques for handling imbalanced data for further improvements.\n\n[1] Huang, Y., Bai, B., Zhao, S., Bai, K. and Wang, F., 2022, June. Uncertainty-aware learning against label noise on imbalanced datasets. In Proceedings of the AAAI Conference on Artificial Intelligence..\n\n> Computational efficiency of EM regarding the weakness above. It would be better that the author provide (theoretical or empirical) analyses regarding the computation of the algorithm.\n\nWe have empirical analysis about computation efficiency included in our Appendix. \nThe results demonstrate that our proposed method present better efficiency, compared to previous SOTA methods.\n\n--- \n\nIf you find our revise and response helpful, please consider raising the score to better support of our work. Also please let us know if there are further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rlw08JZI9x", "reviewer": "Reviewer_ASf6", "summary": "This paper introduces imprecise label learning (ILL), a framework for the unification of learning with various imprecise label configurations, such as partial label learning, semi-supervised learning, noisy label learning, and a mixture of these settings. They propose an EM based method with closed-form learning objectives to handle the problem.", "questions": "NA", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "● The problem of learning with imprecise labels is with great importance, and the idea of unification of different settings is interesting, impressive, and instructive.\n● The paper is well-written. \n● While some details are omitted in the main text, the comprehensive appendix furnishes ample information. Such thorough work is highly appreciated.", "weaknesses": "Given the extensive content, the main text of the paper omits certain details, which may not be immediately straightforward to follow for some parts, e.g., section 3.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's efforts reviewing this paper and acknowledge of our work. We address the concerns as follows.\n\n> Given the extensive content, the main text of the paper omits certain details, which may not be immediately straightforward to follow for some parts, e.g., section 3.2.\n\nThanks for pointing this out. Please understand that this is mainly due to the space limit that we have to include the derivation detail in Section 3.2 into Appendix.\nWe will try to make this more clear in our revised paper.\n\n\nIf you find our revise and response helpful, please consider raising the score for better support of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper tackles learning with imprecise labels and demonstrates strong empirical results, which several reviewers appreciated. The unified framework is a valuable contribution, offering an approach to handling various types of supervision.

While some concerns were raised about the novelty of applying the EM algorithm and certain theoretical aspects, the authors’ clarifications addressed many of these points. However, questions remain about how clearly the framework’s novelty is distinguished from prior work.

One of the reviewers got confused about the presentation of the mathematical material, and I agree that some details could have been clearer. As an example, the type of $f \circ g$ is inconsistent in the text. While it’s defined as a direct mapping from $\mathcal{X} \to \mathcal{Y}$, the paper explicitly states that $f \circ g$ outputs predicted probabilities $p(y \mid x; \theta)$, which would require mapping to a distribution over $C$ possible labels, not a single label. Issues of this kind and perhaps too brief explanations seem to have been the reasons for the confusion of one of the reviewers.

Considering the strengths of the framework and the potential for refinement, I am tending towards acceptance though.
Decision: Accept (poster)