Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:20:27

Abstract: Learning with reduced labeling standards, such as noisy label, partial label, and supplementary unlabeled data, which we generically refer to as imprecise label, is a commonplace challenge in machine learning tasks. Previous methods tend to propose specific designs for every emerging imprecise label configuration, which is usually unsustainable when multiple configurations of imprecision coexist. 
In this paper, we introduce imprecise label learning (ILL), a framework for the unification of learning with various imprecise label configurations. ILL leverages expectation-maximization (EM) for modeling the imprecise label information, treating the precise labels as latent variables. Instead of approximating the correct labels for training, it considers the entire distribution of all possible labeling entailed by the imprecise information. We demonstrate that ILL can seamlessly adapt to partial label learning, semi-supervised learning, noisy label learning, and, more importantly, a mixture of these settings, with closed-form learning objectives derived from the unified EM modeling. Notably, ILL surpasses the existing specified techniques for handling imprecise labels, marking the first practical and unified framework with robust and effective performance across various challenging settings. We hope our work will inspire further research on this topic, unleashing the full potential of ILL in wider scenarios where precise labels are expensive and complicated to obtain.

Main paper: 
 introduction: 
 One of the critical challenges in machine learning is the collection of annotated data for model training (He et al., 2016; Vaswani et al., 2017; Devlin et al., 2018; Dosovitskiy et al., 2020; Radford et al., 2021) . Ideally, every data instance would be fully annotated with precise labels. However, collecting such data can be expensive, laborious, time-consuming, and error-prone. Often the labels are just intrinsically difficult to ascertain precisely. Factors such as a lack of annotator expertise and concerns related to privacy can also negatively affect the quality and completeness of the annotations.In an attempt to circumvent this limitation, several methods have been proposed to permit model learning from the data annotated with reduced labeling standards, which are generally easier to obtain. We will refer to such labels as imprecise. Fig. 1 illustrates some typical mechanisms of label imprecision that are commonly addressed in the literature. Label imprecision requires a modification of the standard supervised training mechanism to build models for each specific case. For instance, partial label learning (PLL) (Cour et al., 2011; Luo & Orabona, 2010; Feng et al., 2020b; Wang et al., 2019a; Wen et al., 2021; Wu et al., 2022; Wang et al., 2022a) allows instances to have a set of candidate labels, instead of a single definitive one. Semi-supervised Learning (SSL) (Lee et al., 2013; Samuli & Timo, 2017; Berthelot et al., 2019b; a; Sohn et al., 2020; Xie et al., 2020b; Zhang et al., 2021b; Wang et al., 2022c; 2023; Chen et al., 2023) seeks to enhance the generalization ability when only a small set of labeled data is available, supplemented by a larger unlabeled set. Noisy label learning (NLL) (Xiao et al., 2015a; Bekker & Goldberger, 2016; Goldberger & Ben-Reuven, 2016; Ghosh et al., 2017; Han et al., 2018; Zhang & Sabuncu, 2018; Li et al., 2018; Wang et al., 2019c; Liu et al., 2020; Li et al., 2020; Ma et al., 2020b; Zhang et al., 2021d) deals with noisy scenarios where the labels are corrupted or incorrect. There is a greater variety of other forms of imprecision, including multiple (imprecise) annotations from crowd-sourcing (Ibrahim et al., 2023; Wei et al., 2023) , programmable weak supervision (Zhang et al., 2022; Wu et al., 2023a) , and bag-level weak supervision (Ilse et al., 2018; Scott & Zhang, 2020; Garg et al., 2021) , among others. While prior arts have demonstrated success in handling individual configurations of label imprecision, their approaches often differ substantially. They are tailored to a specific form of imprecision, as depicted in Fig. 2 . Such specificity not only imposes the necessity of devising a solution for emerging types of label imprecision scenarios, but also complicates the deployment in practical settings, where the annotations can be highly complex and may involve multiple coexisting imprecision configurations. For instance, considering a scenario where both noisy labels and partial labels appear together, it might be challenging to adapt previous methods in NLL or PLL to this scenario since they either rely on the assumption of definite labels (Wei et al., 2023) or the existence of the correct label among label candidates (Campagner, 2021) , thus requiring additional algorithmic design. In fact, quite a few recent works have attempted to address the combinations of imprecise labels in this way, such as partial noisy label (Lian et al., 2022a; Xu et al., 2023) and semi-supervised partial label learning (Wang et al., 2019b; Wang & Zhang, 2020) . However, simply introducing a more sophisticated or ad-hoc design can hardly scale to other settings. In addition, most of these approaches attempt to infer the correct labels given the imprecise information (e.g. through consistency with adjacent data (Lee et al., 2013; Xie et al., 2020a) , iterative refinement (Lv et al., 2020; Arachie & Huang, 2021) , average over the given labels (Hüllermeier & Cheng, 2015; Lv et al., 2023), etc.) , to train the model, which inevitably accumulates error during training, and reduces the generalization performance.In this paper, we formulate the problem from a different view: rather than taking the imprecise label information provided as a potentially noisy or incomplete attempt at assigning labels to instances, we treat it generically as the information that imposes a deterministic or statistical restriction of the actual applicable true labels. We then train the model over the distribution of all possible labeling entailed by the given imprecise information. More specifically, for a dataset with samples X and imprecise label information I, we treat the inaccessible full and precise labels Y as a latent variable. The model is then trained to maximize the likelihood of the provided information I. Since the likelihood computed over the joint probability P (X, I; θ) = Y P (X, I, Y ; θ) must marginalize out Y , the actual information I provided could permit a potentially exponential number of labeling. To deal with the resulting challenge of maximizing the logarithm of an expectation, we use the common approach of expectation-maximization (EM) (Dempster et al., 1977) , where the E-step computes the expectation of P (X, I, Y ; θ) given the posterior of current belief P (Y |X, I; θ t ) and the M-step maximizes the tight variational lower bound over P (X, I; θ). The overall framework is thus largely agnostic to the various nature of label imprecision, with the imprecise label only affecting the manner in which the posterior P (Y |X, I; θ t ) is computed. In fact, current approaches designed for various imprecise label scenarios can be treated as specific instances of our framework. Thus, our approach can serve as a solution towards a unified view for learning with any imprecise labels.While there exist earlier attempts on generalized or EM solutions for different (other) imprecise supervisions or fuzzy observations (Denoeux, 2011; Hüllermeier, 2014; Quost & Denoeux, 2016; Van Rooyen & Williamson, 2017; Zhang et al., 2020) , they usually require additional assumptions and approximations on the imprecise information for learnablility (Campagner, 2021; 2023) , thus presenting limited scalability on practical settings (Quost & Denoeux, 2016) . On the contrary, the unified framework we propose subsumes all of these and naturally extends to the more practical "mixed" style of data, where different types of imprecise labels coexist. Moreover, for noisy labels, our framework inherently enables the learning of a noise model, as we will show in Section 3.2.Encoder 𝒇 (Wang et al., 2022a) for partial label learning. (b) FixMatch (Sohn et al., 2020) for semi-supervised learning. (c) SOP (Liu et al., 2022) for noisy label learning. (d) The proposed unified framework. It accommodates any imprecise label configurations and also mixed imprecise labels with an EM formulation.Through comprehensive experiments, we demonstrate that the proposed imprecise label learning (ILL) framework not only outperforms previous methods for dealing with single imprecise labels, but also presents robustness and effectiveness for mixed label settings, leveraging the full potential to more realistic and challenging scenarios. Our contributions are summarized as follows:• We propose an EM framework towards the unification of learning from any imprecise labels.• We establish state-of-the-art (SOTA) performance with the proposed method on partial label learning, semi-supervised learning, and noisy label learning, demonstrating our method's robustness in more diverse, complex label noise scenarios. • To the best of our knowledge, our work is the first to show the robustness and effectiveness of a single unified method for handling the mixture of various imprecise labels. 
related work: 
 We revisit the relevant work, especially the state-of-the-art popular baselines for learning with individual imprecise label configurations. Let X denote the input space, and Y = [C] := {1, . . . , C} represent the label space with C distinct labels. A fully annotated training dataset of size N is represented asD = {(x i , y i )} i∈[N ] .Learning with imprecise labels involves approximating the mapping function f • g : X → Y from a training dataset where the true label y is not fully revealed from the annotation process. Here f is the backbone for feature extraction, g refers to the classifier built on top of the features, and the output from f • g is the predicted probability p(y|x; θ), where θ is the learnable parameter for f • g. In this study, we primarily consider three imprecise label configurations (as illustrated in Fig. 1 ) and their corresponding representative learning paradigms (as shown in Fig. 2 ), namely partial label learning, semi-supervised learning, and noisy label learning. A more comprehensive related work, including other forms of imprecision, is provided in Appendix B.Partial label learning (PLL). PLL aims to learn with a candidate label set s ⊂ Y, where the ground truth label y ∈ Y is concealed in s. The training data for partial labels thus becomesD PLL = {(x i , s i )} i∈[N ]. The prior arts in PLL can be roughly divided into identification-based for label disambiguation (Zhang & Yu, 2015; Gong et al., 2017; Xu et al., 2019; Wu et al., 2023b) or average-based for utilizing all candidate labels equally (Hüllermeier & Beringer, 2006; Cour et al., 2011; Lv et al., 2023) . PiCO (Wang et al., 2022a ) is a recent contrastive method that employs class prototypes to enhance label disambiguation (as shown in Fig. 2(a) ). It optimizes the cross-entropy (CE)1 loss between the prediction of the augmented training sample A w (x) and the disambiguated labels ŝ. The class prototypes are updated from the features associated with the same pseudo-targets.A contrastive loss, based on MOCO (He et al., 2020) , is employed to better learn the feature space, drawing the projected and normalized features z w and z s of the two augmented versions of data A w (x) and A s (x)2 closer. The overall objective of PiCO is formulated as follows:EQUATIONSemi-supervised learning (SSL). SSL is a paradigm for learning with a limited labeled dataset supplemented by a much larger unlabeled dataset. We can define the labeled dataset asD L SSL = {(x l i , y l i )} i∈[N L ], and the unlabeled dataset asD U = {x u j } j∈[N L +1,N L +N U ] ,with N L ≪ N U . SSL's main challenge lies in effectively utilizing the unlabeled data to improve the generalization performance. A general confidence-thresholding based self-training (Xie et al., 2020a; Sohn et al., 2020) pipeline for SSL is shown in Fig. 2(b ). Consider FixMatch (Sohn et al., 2020) as an example; there are usually two loss components: the supervised CE loss on labeled data and the unsupervised CE loss on unlabeled data. For the unsupervised objective, the pseudo-labels ŷu from the network itself are used to train on the unlabeled data. A "strong-weak" augmentation strategy (Xie et al., 2020a) is commonly adopted. To ensure the quality of the pseudo-labels, only the pseudo-labels whose confidence scores pu are greater than a threshold τ are selected to participate in training:L Fix = L CE p(y|A w (x l ); θ), y l + 1 (p u ≥ τ ) L CE (p(y|A s (x u ); θ), ŷu ) .(2) Noisy label learning (NLL). NLL aims at learning with a dataset of corrupted labels, D NLL = {(x i , ŷi )} i∈ [N ] . Overfitting to the noisy labels ŷ could result in poor generalization performance, even if the training error is optimized towards zero (Zhang et al., 2016a; 2021c) . Several strategies to address the noisy labels have been proposed, such as robust loss functions (Ghosh et al., 2017; Zhang & Sabuncu, 2018; Wang et al., 2019c ) and noise estimation/correction techniques (Han et al., 2018; Li et al., 2020; Bai et al., 2021; Li et al., 2021b; 2022) . We illustrate the NLL pipeline (in Fig. 2(c )) with the recent sparse over-parameterization (SOP) model (Liu et al., 2022) , where a sparse noise model consisting of parameters u i , v i ∈ [-1, 1] C for each sample is adopted. The noise model transforms the network prediction from the true label distribution into the noisy label distribution. A CE loss and a mean-squared-error (MSE) loss optimize parameter {u i } and {v i } respectively:EQUATIONwhere ϕ denotes the L ∞ normalization andm i = u i ⊙ u i ⊙ ŷoh i -v i ⊙ v i ⊙ 1 -ŷoh i, with ŷoh i referring to the one-hot version of y i . Consistency regularization and entropy class-balance regularization are additionally utilized for better performance in SOP (Liu et al., 2022) .Previous attempts to learn from imprecise labels. There are earlier attempts for the generalized solutions of different kinds of imprecise labels/observations. Denoeux (2011) proposed an EM algorithm for the likelihood estimation of fuzzy data and verified the algorithm on linear regression and uni-variate normal mixture estimation. Van Rooyen & Williamson (2017) developed an abstract framework that generically tackles label corruption via the Markov transition. Quost & Denoeux (2016) further extended the EM algorithm of fuzzy data on the finite mixture of Gaussians. Gong et al. (2020) proposed a general framework with centroid estimation for imprecise supervision. A unified partial AUC optimization approach was also proposed earlier (Xie et al., 2023) . While relating to these works on the surface, ILL does not require any assumption on the imprecise information and generalizes well to more practical settings with noisy labels. Some other works for individual settings also related EM framework to their proposed algorithms, but usually involved the approximation on the EM algorithm (Amini & Gallinari, 2002; Bekker & Goldberger, 2016; Wang et al., 2022a) . 
imprecise label learning: 
 Although current techniques demonstrate potential in addressing particular forms of imprecise labels, they frequently fall short in adaptability and transferability to more complicated and more realistic scenarios where multiple imprecise label types coexist. This section first defines the proposed expectation-maximization (EM) formulation for learning with various imprecise labels. Then, we demonstrate that our unified framework seamlessly extends to partial label learning, semi-supervised label learning, noisy label learning, and the more challenging setting of mixed imprecise label learning. Connections with previous pipelines can also be drawn clearly under the proposed framework.Exploiting information from imprecise labels. The challenge of learning with imprecise labels lies in learning effectively with inaccurate or incomplete annotation information. Per the analysis above, prior works catering to specific individual imprecise labels either explicitly or implicitly attempt to infer the precise labels from the imprecise label information. For example, partial label learning concentrates on the disambiguation of the ground truth label from the label candidates (Wang et al., 2022a; Lian et al., 2022b; Xu et al., 2023) or averaging equally over the label candidates (Hüllermeier & Beringer, 2006) . In semi-supervised learning, after the model initially learns from the labeled data, the pseudo-labels are treated as correct labels and utilized to conduct self-training on the unlabeled data (Arazo et al., 2020; Sohn et al., 2020) . Similarly, for noisy label learning, an integral part that helps mitigate overfitting to random noise is the implementation of an accurate noise model capable of identifying and rectifying the incorrect labels (Li et al., 2020; Liu et al., 2022) , thereby ensuring the reliability of the learning process. However, inferring the correct labels from the imprecise labels or utilizing the imprecise labels directly can be very challenging and usually leads to errors accumulated during training (Arazo et al., 2020; Chen et al., 2022) , which is also known as the confirmation bias. In this work, we take a different approach: we consider all possible labeling along with their likelihood that the imprecise labels fulfill to train the model, rather than using a single rectified label from the imprecise information. Such an approach also eliminates the requirements for designing different methods for various imprecise label configurations and provides a unified solution instead.A unified framework for learning with imprecise labels (ILL). Let X = {x i } i∈[N ] represent the features and Y = {y i } i∈[N ] represent their precise labels for the training data. Ideally, Y would be fully specified for X. In the imprecise label scenario, however, Y is not provided; instead we obtain imprecise label information I. We view I not as labels, but more abstractly as a variable representing the information about the labels. From this perspective, the actual labels Y would have a distribution P (Y |I). When the information I provided is the precise true label of the data, P (Y |I) would be a delta distribution, taking a value 1 at the true label, and 0 elsewhere. If I represents partial labels, then P (Y |I) would have non-zero value over the candidate labels, and be 0 elsewhere. When I represents a set of noisy labels, P (Y |I) would represent the distribution of the true labels, given the noisy labels. When I does not contain any information, i.e., unlabeled data, Y can take any value.By the maximum likelihood estimation (MLE) principle, we must estimate the model to maximize the likelihood of the data/information we have been provided, namely X and I. Let P (X, I; θ) represent a parametric form for the joint distribution of X and I3 Explicitly considering the labels Y , we have P (X, I; θ) = Y P (X, Y, I; θ). The maximum likelihood principle requires us to find:EQUATIONwith θ * denotes the optimal value of θ. Eq. ( 4) features the log of an expectation and cannot generally be solved in closed form, and requires iterative hill-climbing solutions. Of these, arguably the most popular is the expectation-maximization (EM) algorithm (Dempster et al., 1977) , which iteratively maximizes a tight variational lower bound on the log-likelihood. In our case, applying it becomes:EQUATIONwhere θ t is the t th estimate of the optimal θ. The detailed derivation of the variational lower bound is shown in Appendix C.1. There are several implications from Eq. ( 5). (i) The expectation over the posterior P (Y |X, I; θ t ) equates to considering all labeling entailed by the imprecise label information I, rather than any single (possibly corrected) choice of label. The computing of the posterior conditioned on imprecise label information I can be related to a non-deterministic finite automaton (NFA) (Hopcroft et al., 2001) ; details are in Appendix C.6. (ii) The property of the second term log P (I|X, Y ; θ) is dependent on the nature of imprecise label I. If I contains information about the true labels Y , such as the actual labels or the label candidates, it can be reduced to P (I|Y ), i.e., the probability of I is no longer dependent on X or θ and thus can be ignored from Eq. ( 5). If I represents the noisy labels, P (I|X, Y ; θ) instead includes a potentially learnable noise model. (iii) It is a general framework towards the unification of any label configuration, including full labels, partial labels, low-resource labels, noisy labels, etc. In this work, we specialize this solution to partial label learning, semi-supervised learning, noisy label learning, and the mixture of them below.We illustrate how to seamlessly expand the formulation from Eq. ( 5) to partial label learning, semisupervised learning, and noisy label learning, and derive the loss function 4 for each setting here. The actual imprecise labels only affect the manner in which the posterior P (Y |X, I; θ t ) is computed for each setting. We show that all learning objectives derived from Eq. ( 5) naturally include a consistency term with the posterior as the soft target. We also demonstrate that the proposed unified EM framework closely connects with the prior arts, which reveals the potential reason behind the success of these techniques. Note that while we only demonstrate the application of the unified framework to three learning paradigms here, it is flexible and can expand to learning with a mixture of imprecise labels with robust performance, as we will show in Appendix C.5 and Section 4.4.Partial label learning (PLL). The imprecise label I for partial labels is defined as the label candidate sets S = {s i } i∈[N ] containing the true labels. These partial labels indicate that the posterior P (Y |X, S; θ t ) can only assign its masses on the candidate labels. Since S contains the information about the true labels Y , P (S|X, Y ; θ) reduces to P (S|Y ), and thus can be ignored, as we discussed earlier. Substituting S in Eq. ( 5), we have the loss function of PLL derived using ILL:EQUATIONwhere p(y|A w (x), s; θ t ) is the normalized probability that k∈C p k = 1, and p k = 0, ∀k ∈ s.Eq. ( 6) corresponds exactly to consistency regularization (Xie et al., 2020a) , with the normalized predicted probability as the soft pseudo-targets. This realization on PLL shares similar insights as (Wu et al., 2022) which exploits a gradually induced loss weight for PLL on multiple augmentations of the data. However, our framework is much simpler and more concise as shown in Appendix D.3.2.Semi-supervised learning (SSL) In SSL, the input X consists of the labeled data X L and the unlabeled data X U . The imprecise label for SSL is realized as the limited number of full labels Y L for X L . The labels Y U for unlabeled X U are unknown and become the latent variable. Interestingly, for the unlabeled data, there is no constraint on possible labels it can take. The posteriorP (Y U |X L , X U , Y L ; θ), which is the actual prediction from the network, can be directly utilized as soft targets for self-training. Since Y L is conditionally independent with Y U given X, the second term of Eq. ( 5):P (Y L |X L , X U , Y U ; θ), is reduced to P (Y L |X L ; θ), which corresponds to the supervised objective on labeled data. The loss function for SSL using ILL thus becomes:EQUATIONThe first term corresponds to the unsupervised consistency regularization usually employed in SSL, and the second term refers to the supervised CE loss only on labeled data. Eq. ( 7) has several advantages over the previous methods. It adopts the prediction as soft-targets of all possible labeling on unlabeled data, potentially circumventing the confirmation bias caused by pseudo-labeling and naturally utilizing all unlabeled data which resolves the quantity-quality trade-off commonly existing in SSL (Sohn et al., 2020; Chen et al., 2023) . It also indicates that previous pseudo-labeling with confidence threshold implicitly conducts the EM optimization, where the maximal probable prediction approximates the expectation, and the degree of the approximation is determined by the threshold τ .Noisy label learning (NLL). Things become more complicated here since the noisy labels Ŷ do not directly reveal the true information about Y , thus P ( Ŷ |Y, X; θ) inherently involves a noise model that needs to be learned. We define a simplified instance-independent5 noise transition model T ( Ŷ |Y ; ω) with parameters ω, and take a slightly different way to formulate the loss function for NLL:EQUATIONwhere the parameters ω and θ are learned end-to-end. The first term corresponds to the consistency regularization of prediction conditioned on noisy labels and the second term corresponds to the supervised loss on noisy predictions that are converted from the ground truth predictions. Both quantities are computed using the noise transition model given the noisy label ŷ: For all settings, we additionally adopt an entropy loss for balancing learned cluster sizes (John Bridle, 1991; Joulin & Bach, 2012) , similarly as (Liu et al., 2022; Wang et al., 2023) . All experiments are conducted with three random seeds using NVIDIA V100 GPUs.p(y|x 
partial label learning: 
 Setup. Following (Wang et al., 2022a) , we evaluate our method on partial label learning setting using CIFAR-10 ( Krizhevsky et al., 2009) , CIFAR-100 (Krizhevsky et al., 2009) , and CUB-200 (Welinder et al., 2010) . We generate partially labeled datasets by flipping negative labels to false positive labels with a probability q, denoted as a partial ratio. The C -1 negative labels are uniformly aggregated into the ground truth label to form a set of label candidates. We consider q ∈ {0.1, 0.3, 0.5} for CIFAR-10, q ∈ {0.01, 0.05, 0.1} for CIFAR-100, and q = 0.05 for CUB-200. We choose six baselines for PLL using ResNet-18 (He et al., 2016) : LWS (Wen et al., 2021) , PRODEN (Lv et al., 2020) , CC (Feng et al., 2020b) , MSE and EXP (Feng et al., 2020a) , and PiCO (Wang et al., 2022a) . The detailed hyper-parameters and comparison with the more recent method R-CR (Wu et al., 2022) that utilizes a different training recipe and model (Zagoruyko & Komodakis, 2016) are shown in Appendix D.3.2.Results. The results for PLL are shown in Table 1 . Our method achieves the best performance compared to the baseline methods. Perhaps more surprisingly, on CIFAR-10 and CIFAR-100, our method even outperforms the fully-supervised reference, indicating the potential better generalization capability using the proposed framework, sharing similar insights as (Wu et al., 2022) . While PiCO adopts a contrastive learning objective, our method still surpasses PiCO by an average of 2.13% on CIFAR-10 and 2.72% on CIFAR-100. Our approach can be further enhanced by incorporating contrastive learning objectives, potentially leading to more significant performance improvements.Setup. For experiments of SSL, we follow the training and evaluation protocols of USB (Wang et al., 2022d) on image and text classification. To construct the labeled dataset for semi-supervised learning, we uniformly select l/C samples from each class and treat the remaining samples as the unlabeled dataset. We present the results on CIFAR-100 and STL-10 ( Krizhevsky et al., 2009) for image classification, and IMDB (Maas et al., 2011) and Amazon Review (McAuley & Leskovec, 2013 ) for text classification. We compare with the current methods with confidence thresholding, such as FixMatch (Sohn et al., 2020) , AdaMatch (Berthelot et al., 2021) , FlexMatch (Zhang et al., 2021b) , FreeMatch (Wang et al., 2023) , and SoftMatch (Chen et al., 2023) . We also compare with methods with the contrastive loss, CoMatch (Li et al., 2021a) and SimMatch (Zheng et al., 2022) . A full comparison of the USB datasets and hyper-parameters is shown in Appendix D.4.Results. We present the results for SSL on Table 2 . Although no individual SSL algorithm dominates the USB benchmark (Wang et al., 2022d) , our method still shows competitive performance. Notably, our method performs best on STL-10 with 40 labels and Amazon Review with 250 labels, outper- forming the previous best by 0.68% and 1.33%. In the other settings, the performance of our method is also very close to the best-performing methods. More remarkably, our method does not employ any thresholding, re-weighting, or contrastive techniques to achieve current results, demonstrating a significant potential to be further explored in SSL for even better performance.Setup. We conduct the experiments of NLL following SOP (Liu et al., 2022) on both synthetic symmetric/asymmetric noise on CIFAR-10 and CIFAR-100, and more realistic and larger-scale instance noise on Clothing1M (Xiao et al., 2015b) , and WebVision (Li et al., 2017) . To introduce the synthetic symmetric noise to CIFAR-10 and CIFAR-100, we uniformly flip labels for a probability η into other classes. For asymmetric noise, we only randomly flip the labels for particular pairs of classes. We mainly select three previous best methods as baselines: DivideMix (Li et al., 2020) ; ELR (Liu et al., 2020) ; and SOP (Liu et al., 2022) . We also include the normal cross-entropy (CE) training and mixup (Zhang et al., 2017) as baselines. More comparisons regarding other methods (Patrini et al., 2016; Han et al., 2018) and on CIFAR-10N (Wei et al., 2021) are shown in Appendix D.5.Results. We present the noisy label learning results in Table 3 . The proposed method is comparable to the previous best methods. On synthetic noise of CIFAR-10, our method demonstrates the best performance on both symmetric noise and asymmetric noise. On CIFAR-100, our method generally produces similar results comparable to SOP. One may notice that our method shows inferior performance on asymmetric noise of CIFAR-100; we argue this is mainly due to the oversimplification of the noise transition model. Our method also achieves the best results on WebVision, outperforming the previous best by 2.05%. On Clothing1M, our results are also very close to DivideMix, which trains for 80 epochs compared to 10 epochs in ours, and no doubt further training improves performance.Setup. We evaluate on CIFAR-10 and CIFAR-100 in a more challenging and realistic setting, the mixture of various imprecise label configurations, with unlabeled, partially labeled, and noisy labeled data existing simultaneously. We first sample the labeled dataset and treat other samples as the unlabeled. On the labeled dataset, we generate partial labels and randomly corrupt the true label of the partial labels. We set l ∈ {1000, 5000, 50000} for CIFAR-10, and l ∈ {5000, 10000, 50000} for CIFAR-100. For partial labels, we set q ∈ {0.1, 0.3, 0.5} for CIFAR-10, and q ∈ {0.01, 0.05, 0.1} for CIFAR-100. For noisy labels, we set η ∈ {0, 0.1, 0.2, 0.3} for both datasets. Since there is no Table 4 : Accuracy comparison of mixture of different imprecise labels. We report results of full labels, partial ratio q of 0.1 (0.01) and 0.3 (0.05) for CIFAR-10 (CIFAR-100), and noise ratio η of 0.1, 0.2, and 0.3 for CIFAR-10 and CIFAR-100. Results. We report the comparison with partial noisy label learning methods in Table 4 . Compared to previous methods, the proposed method achieves the best performance. Despite the simplicity, our method outperforms PiCO+ and DALI with mixup, showing the effectiveness of dealing with mixed imprecise labels. We also report the results of our methods on more mixed imprecise label configurations in Table 5 . Our method demonstrates significant robustness against various settings of the size of labeled data, partial ratio, and noise ratio. Note that this is the first work that naturally deals with all three imprecise label configurations simultaneously, with superior performance than previous methods handling specific types or combinations of label configurations. This indicates the enormous potential of our work in realistic applications for handling more complicated data annotations. 
conclusion: 
 We present the imprecise label learning (ILL) framework, a unified and consolidated solution for learning from all types of imprecise labels. ILL effectively employs an expectation-maximization (EM) algorithm for maximum likelihood estimation (MLE) of the distribution over the latent ground truth labels Y , imprecise label information I, and data X. It naturally extends and encompasses previous formulations for various imprecise label settings, achieving promising results. Notably, in scenarios where mixed configurations of imprecise labels coexist, our method exhibits substantial robustness against diverse forms of label imprecision. The potential broader impact of the ILL framework is substantial. It stands poised to transform domains where obtaining precise labels poses a challenge, offering a simple, unified, and effective approach to such contexts. Beyond the three imprecise label configurations we have demonstrated in this study, the ILL framework shows promise for an extension to more intricate scenarios such as multi-instance learning (Ilse et al., 2018) and multi-label crowd-sourcing learning (Ibrahim et al., 2023) . However, it is also crucial to acknowledge the limitations of the ILL framework. Although its effectiveness has been substantiated on relatively smaller-scale datasets, additional empirical validation is necessary to assess its scalability to larger datasets. Furthermore, our study only considers balanced datasets; thus, the performance of the ILL framework when dealing with imbalanced data and open-set data still remains an open area for future exploration. We anticipate that our study will constitute a significant stride towards a comprehensive solution for imprecise label learning and catalyze further research in this crucial field. An imprecise label, which might contain multiple class indicesX = {x i } i∈[N ] Data. A set of data instances x of size N XInput space where x is drawn fromY = {y i } i∈[N ]Ground-truth labels. A set of label indices y of size N Y Label space where y is drawn from A set of unknown labels for unlabeled data instances puI = {[ι] i } i∈[N ] Imprecise labels. A set of imprecise labels [ι] of size N f Model backbone g Model classifier h Model multi-layer perceptron f • g Model mapping X → Y θ Learnable parameters of f • g p(y|x; θ) Output probability from model f • g f • h Model mapping X → Z,The maximum predicted probability on unlabeled data max(p(y|x u ; θ)) ŷuThe pseudo-label from the predicted probability on unlabeled data arg max(p(y|x u ; θ)) τThe threshold for confidence thresholding ŷ A corrupted/noisy label ŷoh An one-hot version of the corrupted/noisy label Ŷ A set of noisy labels u, v, mNoise model related parameters in SOP (Liu et al., 2022)  
 
 Appendix: The forward score in forward-backward algorithm β(i, y)The backward score in forward-backward algorithm T (ŷ|y; ω)The simplified noise transition model in ILL ωThe parameters in the simplified noise modelLabel ambiguity remains a fundamental challenge in Partial Label Learning (PLL). The most straightforward approach to handling PLL is the average-based method, which assumes an equal probability for each candidate label being the ground-truth label. For instance, Hüllermeier & Beringer (2006) employed k-nearest neighbors for label disambiguation, treating all candidate labels of a sample's neighborhood equally and predicting the ground-truth label through voting strategies. However, a significant drawback of average-based methods is that false positive labels can mislead them.To overcome these limitations, researchers have explored identification-based methods for PLL. In contrast to average-based methods, which treat all candidate labels equally, identification-based methods view the ground-truth label as a latent variable. They seek to maximize its estimated probability using either the maximum margin criterion (Nguyen & Caruana, 2008; Zhang et al., 2016b) or the maximum likelihood criterion (Liu & Dietterich, 2012) . Deep learning techniques have recently been incorporated into identification-based methods, yielding promising results across multiple datasets. For example, PRODEN (Lv et al., 2020) proposed a self-training strategy that disambiguates candidate labels using model outputs. CC (Feng et al., 2020b) introduced classifierconsistent and risk-consistent algorithms, assuming uniform candidate label generation. LWS (Wen et al., 2021) relaxed this assumption and proposed a family of loss functions for label disambiguation.More recently, Wang et al. (2022a) incorporated contrastive learning into PLL, enabling the model to learn discriminative representations and show promising results under various levels of ambiguity.Nevertheless, these methods rely on the fundamental assumption that the ground-truth label is present in the candidate set. This assumption may not always hold, especially in cases of unprofessional judgments by annotators, which could limit the applicability of these methods in real-world scenarios.Consistency regularization and self-training, inspired by clusterness and smoothness assumptions, have been proposed to encourage the network to generate similar predictions for inputs under varying perturbations (Tarvainen & Valpola, 2017; Samuli & Timo, 2017; Miyato et al., 2018) . Self-training (Lee et al., 2013; Arazo et al., 2020; Sohn et al., 2020) is a widely-used approach for leveraging unlabeled data. Pseudo Label (Lee et al., 2013) , a well-known self-training technique, iteratively creates pseudo labels that are then used within the same model. However, this approach suffers from confirmation bias (Arazo et al., 2020) , where the model struggles to rectify its own errors when learning from inaccurate pseudo labels. Recent studies focus largely on generating high-quality pseudo-labels. MixMatch (Berthelot et al., 2019b) , for instance, generates pseudo labels by averaging predictions from multiple augmentations. Other methods like ReMixMatch (Berthelot et al., 2019a) , UDA (Xie et al., 2020a) , and FixMatch (Sohn et al., 2020) adopt confidence thresholds to generate pseudo labels for weakly augmented samples, which are then used to annotate strongly augmented samples. Methods such as Dash (Xu et al., 2021b) , FlexMatch (Zhang et al., 2021b) , and FreeMatch (Wang et al., 2023) dynamically adjust these thresholds following a curriculum learning approach. SoftMatch (Chen et al., 2023) introduces a novel utilization of pseudo-labels through Gaussian re-weighting. Label Propagation methods (Iscen et al., 2019) assign pseudo labels based on the local neighborhood's density. Meta Pseudo Labels (Pham et al., 2021) proposes generating pseudo labels with a meta learner. SSL learning has also seen improvements through the incorporation of contrastive loss (Li et al., 2021a; Zheng et al., 2022) . Furthermore, MixUp (Zhang et al., 2017) has shown its effectiveness in a semi-supervised learning (SSL) (Berthelot et al., 2019b; a; Cai et al., 2022) .Due to their large number of parameters, deep neural networks are prone to overfitting to noisy labels. Although certain popular regularization techniques like mixup (Zhang et al., 2017) can somewhat mitigate overfitting, they fall short of completely resolving the issue of learning with noisy labels, as they lack accurate noise modeling. Numerous techniques have been proposed to handle Noisy Label Learning (NLL) (Song et al., 2022) . They can broadly be divided into three categories: robust loss functions (Ghosh et al., 2017; Zhang & Sabuncu, 2018; Wang et al., 2019c; Ma et al., 2020b) , noise estimation (Xiao et al., 2015a; Goldberger & Ben-Reuven, 2016; Liu et al., 2020; Zhang et al., 2021d; Northcutt et al., 2021) , and noise correction (Han et al., 2018; Li et al., 2020; Liu et al., 2022) .Designing loss functions that are robust to noise is a well-explored strategy for tackling the label noise problem. The l 1 loss [59], which is a robust loss function, is a popular choice and has seen many recent extensions (Zhang & Sabuncu, 2018; Wang et al., 2019c; Ma et al., 2020a; Yu et al., 2020) . Additionally, methods that re-weight loss (Liu & Tao, 2016) have also been explored for learning with noisy labels. However, despite enabling the model to learn faster from accurate labels, these robust loss functions still lead to overfitting to corrupted labels when used with models having a large number of parameters.Another common strategy to handle label noise involves assuming that the noisy label originates from a probability distribution that depends on the actual label. The key task here is to estimate the underlying transition probabilities. Early works (Goldberger & Ben-Reuven, 2016 ) incorporated these transition probabilities into a noise adaptation layer that is stacked over a classification network and trained in an end-to-end fashion. More recent work, such as Forward (Patrini et al., 2016) , prefers to estimate these transition probabilities using separate procedures. However, the success of this method is contingent upon the availability of clean validation data (Northcutt et al., 2021) or additional assumptions about the data (Zhang et al., 2021e) .Noise correction has shown promising results in noisy label learning recently. During the early learning phase, the model can accurately predict a subset of the mislabeled examples (Liu et al., 2020) . This observation suggests a potential strategy of correcting the corresponding labels. This could be accomplished by generating new labels equivalent to soft or hard pseudo-labels estimated by the model (Tanaka et al., 2018; Yi & Wu, 2019) . Co-Teaching uses multiple differently trained networks for correcting noisy labels (Han et al., 2018) . SELFIE (Song et al., 2019) corrects a subset of labels by replacing them based on past model outputs. Another study in Arazo et al. (2019) uses a two-component mixture model for sample selection, and then corrects labels using a convex combination. They also utilize mixup (Zhang et al., 2017) to enhance performance. Similarly, DivideMix (Li et al., 2020) employs two networks for sample selection using a two-component mixture model and implements the semi-supervised learning technique MixMatch (Berthelot et al., 2019b) .Multiple Instance Learning (MIL) is a sub-field of supervised learning where each instance is associated with a label. However, MIL stands out due to its handling of incomplete label knowledge in the training set. The training data in MIL is composed of 'bags', each containing several unlabeled instances. The primary objective of MIL is to predict labels for new, unseen bags. Dietterich et al.(1997) pioneered the concept of MIL, specifically applying it to drug activity prediction. This innovative approach has since inspired the development of numerous algorithms tailored to tackle this distinctive problem. One pragmatic approach for dealing with label noise in this context involves counting the number of positive instances within a bag and setting a threshold for classifying positive bags. This technique was encapsulated as the threshold-based assumption for MIL in (Foulds & Frank, 2010) . Per this assumption, a bag is classified as positive if and only if the count of positive instances exceeds a predefined threshold. This constituted the initial integration of counting information into the MIL framework. Following this development, various efforts (Tao et al., 2004a; b) focused on carrying out bag-level predictions based on count-based assumptions.Another significant trajectory in MIL research is centered on instance-level prediction. The framework proposed by Maron & Lozano-Pérez (1997) is particularly renowned for instance-level prediction within the MIL paradigm. This framework has underpinned numerous research proposals that have demonstrated effectiveness. The foundational idea behind these frameworks is the dynamic or static labeling of instances according to the bag label. Empirical studies, as detailed in (Vanwinckelen et al., 2016) , have illustrated that superior bag-level prediction does not necessarily guarantee improved instance-level prediction. To further the field of MIL, addressing these critical issues will be essential in order to develop more robust, flexible, and scalable MIL methods that can be applied across a broad range of real-world situations (Zhang et al., 2021a) .Various previous works have explored dealing with distinct types of imprecise labels. However, they have yet to tackle a combination of partial labels, limited labels, and noisy labels, which is a highly realistic scenario. For instance, recent attention has been paid to the issue of partial noisy label learning.PiCO+ (Wang et al., 2022b) , an extended version of PiCO (Wang et al., 2022a) , is tailored specifically for partial noisy labels. It employs a distance-based method to select clean samples and uses a semi-supervised contrastive learning algorithm to train robust classifiers. This approach distinguishes between clean and noisy samples and enhances the learning of distinctive features. IRNet (Lian et al., 2022b ) is a novel framework designed for Partial Label Learning with noisy labels. It uses two modules: noisy sample detection and label correction, transforming the scenario of noisy PLL into a more traditional PLL. DALI (Xu et al., 2023) is another framework designed to reduce the negative impact of detection errors by creating a balance between the initial candidate set and model outputs, with theoretical assurances of its effectiveness.Additionally, some work has focused on semi-supervised partial label learning (Wang et al., 2019b; Wang & Zhang, 2020 ). An iterative label propagation process is proposed in these studies, operating between partially labeled examples and unlabeled instances, which helps clarify the candidate label sets of the partially labeled examples and assign valid labels to unlabeled instances. However, these methods don't scale well to larger datasets like CIFAR-10.Our study and the proposed framework hold significant importance as we have not found any existing research that can effectively address the challenge of handling a combination of partial, limited, and noisy labels simultaneously. This underscores the novelty and significance of our work, making it a valuable contribution to the field of machine learning.Evidence lower bound (ELBO), or equivalently variational lower bound (Dempster et al., 1977) , is the core quantity in EM. From Eq. ( 5), to model log P (X, I; θ), we have:EQUATIONwhere the first term is the ELBO and the second term is the KL divergenceD KL (Q(Y )||P (Y |X, I; θ)).Replacing Q(Y ) with P (Y |X, I; θ t ) at each iteration will obtain Eq. ( 5).The imprecise label I for partial labels is defined as the label candidate sets S = {s i } i∈[N ] containing the true labels. Now we can derive Eq. ( 6) by replacing I with S in Eq. ( 5):EQUATIONNow we replace Y , X, and S to y, x, and s for each instance, and converting the maximization problem to negative log-likelihood minimization problem to drive the loss function:EQUATIONThe first term is the Cross-Entropy loss we derived in Eq. ( 6). If S is not instance-dependent, then knowing Y also knows S, the second term thus can be ignored in Eq. ( 6). If S becomes instancedependent, the second term can be maintained as a supervised term as in (Wu et al., 2022) to optimize θ.In SSL, the input X consists of the labeled data X L and the unlabeled data X U . The imprecise label for SSL is realized as the limited number of full labels Y L for X L . The labels Y U for unlabeled X U are unknown and become the latent variable. Thus we can write:EQUATIONThe negative log-likelihood loss function thus becomes: L SSL ILL = L CE p(EQUATIONThe loss function is:EQUATIONNote that both term is computed from the noise transition matrix as mentioned in Eq. ( 9).In this setting, we have both labeled data and unlabeled data, where the labels for the labeled data are both partial and noisy. On the unlabeled data, the unsupervised objective is the same as the unsupervised consistency regularization of semi-supervised learning shown in Eq. ( 7). On the labeled data, it mainly follows the Eq. ( 9) of noisy label learning, with the noisy single label becoming the noisy partial labels ŝ. For noisy partial labels, the noisy supervised objective in Eq. 8 becomes the supervised consistency regularization as in Eq. 6 of partial label setting to train the noise transition model, and the noisy unsupervised objective becomes the consistency regularization of the prediction conditioned on noisy partial labels:EQUATIONWe can compute both quantity through the noise transition model: This section provides an alternative explanation for the EM framework proposed in the main paper.EQUATIONWe treat the problem of assigning labels Y to inputs X as generating a sequence of symbols Y , such that I is satisfied. Such a process can be represented as a finite-state automaton (FSA); more specifically, a non-deterministic finite automaton (NFA) (Hopcroft et al., 2001) . The NFA of imprecise labels can be formally defined with a finite set of states X, a finite set of symbols Y , a transition function f • g, a start state of x 0 , and the accepting state of I, determining the available transition paths in the NFA, as shown in Fig. 3 . We demonstrate that, Although we can directly derive the soft-targets utilized in loss functions of each setting we study from Eq. ( 5), they can also be computed by performing a forward-backward algorithm on the NFA. While there is no difference in the actual value for conducting the forward-backward algorithm and deriving the soft-targets directly, the NFA indeed becomes more necessary when only collection-level imprecise information is provided, where naively considering all possible labeling requires exponential complexity, and forward-backward on NFA reduces the complexity to linear. The reason for the equivalence in the partial, semi-sup, noisy label is because the NFA directly collapses to O(1) for computing the posterior without any statistical information imposed from I.To compute the posterior probability P (Y |X, I; θ), without loss of generality, we assume each training sample is drawn independently and identically, and training samples occur as a permutationinvariant sequence. We further assume that each y i is conditionally independent given the whole sequence of training samples X, which allows us to re-write Eq. ( 5) as:θ t+1 = arg max θ i∈[N ] y∈[C]P (y i = y|X, I; θ t ) [log P (y i = y|X; θ)EQUATION)The problem of computing the posterior then reduces to calculate the joint probability of I and the latent ground truth label of the i-th sample taking the value y given the whole input sequence X:EQUATIONSubsequently, P (y i = y, I|X; θ t ) can be computed in linear time complexity O(N C) at maximum using the forward-backward algorithm on the NFA, similar to connectionist temporal classification (CTC) (Graves et al., 2006) . More specifically, for any symbol y i in the NFA of imprecise label information, the joint is calculated with the forward score α(i, y) and the backward score β(i, y):EQUATIONwhere α(i, y) indicates the total probability of all paths of I at state x i and β(i, y) indicates the total probability of all paths of I at state x i . We can then compute the posterior probability:EQUATIONwhich can be in turn used in Eq. ( 5). Now the difference in learning with various imprecise label configurations lies only in how to construct the NFA graph for different configurations.An illustration for the NFA of full labels, partial labels, limited labels with unlabeled data, and noisy labels is shown in Fig. 3 . For full precise labels, there is only one determined transition of Y . For partial labels, the transition paths can only follow the label candidates, resulting in the normalized soft-targets with masses on the label candidates as we shown in Section 3.2. For semi-supervised learning, the transition paths for the labeled data strictly follow Y L . For unlabeled data, since there is no constraint on the transition paths, all transitions are possible, corresponding to the soft pseudo-targets. For noisy labels, all transition paths are possible, but the joint probability needs to be computed with the noise transition model. Since the settings studied in this paper has relatively simple NFA representations, we can directly derive the closed-form formulation from Eq. ( 5 Figure 4 : of results comparison to recent SOTA baselines on benchmarks on PLL, SSL, NLL, and Mixed imprecise labels. We report the average accuracy over all settings for each dataset.extended to other settings such as multiple instance learning, learning with count/proportion, etc, by constructing more complex NFA of each case, and actually use forward-backward algorithm on it to compute EM. We will show, in future work, the more extensions of the ILL framework on collection-level and statistical imprecise labels, with the help of NFA.We adopt two additional training strategies for the ILL framework. The first is the "strong-weak" augmentation strategy (Xie et al., 2020a) . Since there is a consistency regularization term in each imprecise label formulation of ILL, we use the soft pseudo-targets of the weakly-augmented data to train the strongly-augmented data. The second is the entropy loss (John Bridle, 1991) for class balancing, which is also adopted in SOP (Liu et al., 2022) and FreeMatch (Wang et al., 2023) . We set the loss weight for the entropy loss uniformly for all experiments as 0.1.Here we provide an overview of the performance of our method, compared to recent state-of-the-art (SOTA) baselines, as shown in Fig. 4 . For each evaluated datasets in each imprecise label setting, we average the results and report the average accuracy. It is demonstrated that the proposed method in general achieves better performance compared to recent SOTA baselines or achieves performance comparable to them.Following previous work (Xu et al., 2022; Wen et al., 2021; Wang et al., 2022a) , we evaluate our method on partial label learning setting using CIFAR-10, CIFAR-100, and CUB-200 (Welinder et al., 2010) . We generate partially labeled datasets by flipping negative labels to false positive labels with a probability q, which is also denoted as a partial ratio. Specifically, the C -1 negative labels are uniformly aggregated into the ground truth label to form a set of label candidates. We consider q ∈ {0.1, 0.3, 0.5} for CIFAR-10, q ∈ {0.01, 0.05, 0.1} for CIFAR-100, and q = 0.05 for CUB-200. For CIFAR-10 and CIFAR-100, we use ResNet-18 (He et al., 2016) as backbone. We use SGD as an optimizer with a learning rate of 0.01, a momentum of 0.9, and a weight decay of 1e-3.For CUB-200, we initialize the ResNet-18 (He et al., 2016) with ImageNet-1K (Deng et al., 2009) pre-trained weights. We train 800 epochs for CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) , and 300 epochs for CUB-200, with a cosine learning rate scheduler. For CIFAR-10 and CIFAR-100, we use an input image size of 32. For CUB-200, we use an input image size of 224. A batch size of 256 is used for all datasets. The choice of these parameters mainly follows PiCO (Wang et al., 2022a) . We present the full hyper-parameters systematically in Table 7 . We additionally compare our method with R-CR (Wu et al., 2022) , which uses a different architecture as the results in Table 1 . R-CR uses Wide-ResNet34x10 as backbone, and adopts multiple strong data augmentations. It also adjusts the loss weight along training. For fair comparison, we use the same architecture without multiple augmentation and the curriculum adjust on loss. The results are shown in Table 8 , where our method outperforms R-CR on CIFAR-10 and is comparable on CIFAR-100. We also provide the comparison of our method on instance-dependent partial label learning as proposed in Xu et al. (2021a; 2022) . Due to the nature of instance-dependence, we maintain the term P (S|Y, X; θ) from Eq. ( 5) as a supervised term for optimization. We compare our method with VALEN (Xu et al., 2021a) , RCR (Wu et al., 2022) , PiCO Wang et al. (2022a) , and POP (Xu et al., 2022) on MNIST, Kuzushiji-MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, with synthetic instance-dependent partial labels generated according to Xu et al. (2022) . From the results in Table 9 , we proposed method demonstrate the best performance across different datasets evaluated. A recent work on PLL discussed and analyzed the robustness performance of different loss functions, especially the average-based methods (Lv et al., 2023) . We perform a similar analysis here for the derived loss function in ILL. Following the notation in Lv et al. (2023) , let s denote the candidate label set, x as the training instance, g as the probability score from the model, and f as the classifier f (x) = arg max i∈Y g i (x), the average-based PLL can be formulated as:EQUATIONLv et al. ( 2023) compared different loss functions ℓ on both noise-free and noisy PLL settings, where they find both theoretically and empirically that average-based PLL with bounded loss are robust under mild assumptions. Empirical study in Lv et al. (2023) suggests that both Mean Absolute Error and Generalized Cross-Entropy loss (Zhang & Sabuncu, 2018) that proposed for noisy label learning achieves the best performance and robustness for average-based PLL. Worst case noise scenario. However, on CIFAR-100N, our method slightly underperforms previous methods, possibly due to the oversimplified noise model utilized in ILL. We believe that a more realistic noise transition model and further tuning of our method could lead to improved performance. (Patrini et al., 2016) 69.80 61.10 MentorNet (Jiang et al., 2018) 66.17 63.00 Co-Teaching (Han et al., 2018) 69.20 63.60 DivideMix (Li et al., 2020) 74.76 77.32 ELR (Liu et al., 2020) 72.90 76.20 CORES (Cheng et al., 2020) 73.20 -SOP (Liu et al., 2022) 73 To create a mixture of various imprecise label configurations, we select CIFAR-10 and CIFAR-100 as base datasets. We first uniformly sample l/C labeled samples from each class to form the labeled dataset and treat the remaining samples as the unlabeled dataset. Based on the labeled dataset, we generate partially labeled datasets by flipping negative labels to false positive labels with the partial ratio q. After obtaining the partial labels, we randomly select η percentage of samples from each class, and recreate the partial labels for them by flipping the ground truth label uniformly to another class.In this setting, unlabeled data, partially labeled data, and noisy labeled data exist simultaneously, which is very challenging and more closely resembles realistic situations. For CIFAR-10, we set l ∈ {1000, 5000, 50000}, and for CIFAR-100, we set l ∈ {5000, 10000, 50000}. Similarly in the partial label setting, we set q ∈ {0.1, 0.3, 0.5} for CIFAR-10, and q ∈ {0.01, 0.05, 0.1} for CIFAR-100. For noisy labels, we set η ∈ {0.1, 0.2, 0.3} for both datasets. We provide the runtime analysis on CIFAR-100 of our method on different settings, compared with the SOTA baselines. We compute the average runtime from all training iterations on CIFAR-100.The results are shown in Table 21 . Since the settings studied in this work has loss functions derived as close-form from Eq. ( 5), the time complexity can be viewed as O(1). Thus our method in general present faster runtime without complex design such as contrastive loss. For simplicity, we use L CE for labels of the formats of class indices, one-hot vectors, and class probabilities.We use Aw to indicate the weaker data augmentation and As to indicate the stronger data augmentation.The actual parameters θ may apply only to some component such as P (Y |X; θ) of the overall distribution; we will nonetheless tag the entire distribution P (X, I; θ) with θ to indicate that it is dependent on θ overall.To formulate the loss function, we convert the problem to minimization of the negative log-likelihood.A more complicated instance-dependent noise model T ( Ŷ |Y, X; ω) can also be formulated under our unified framework, but not considered in this work.Although there are also prior efforts on partial semi-supervised learning(Wang et al., 2019b;Wang & Zhang, 2020), they do not scale on simple dataset even on CIFAR-10. Thus, they are ignored for comparison. 


Decision: The paper tackles learning with imprecise labels and demonstrates strong empirical results, which several reviewers appreciated. The unified framework is a valuable contribution, offering an approach to handling various types of supervision.

While some concerns were raised about the novelty of applying the EM algorithm and certain theoretical aspects, the authors’ clarifications addressed many of these points. However, questions remain about how clearly the framework’s novelty is distinguished from prior work.

One of the reviewers got confused about the presentation of the mathematical material, and I agree that some details could have been clearer. As an example, the type of $f \circ g$ is inconsistent in the text. While it’s defined as a direct mapping from $\mathcal{X} \to \mathcal{Y}$, the paper explicitly states that $f \circ g$ outputs predicted probabilities $p(y \mid x; \theta)$, which would require mapping to a distribution over $C$ possible labels, not a single label. Issues of this kind and perhaps too brief explanations seem to have been the reasons for the confusion of one of the reviewers.

Considering the strengths of the framework and the potential for refinement, I am tending towards acceptance though.

Reviews: review: paper_summary: paper_summary: The paper provides a unified view on various imprecise-label learning frameworks, such as semi-supervised-, partial-label-, or noise-label learning, through the lens of the expectation-maximization algorithm. In addition to unifying these existing setups, EM naturally allows treating combinations of the above setups. Experiments show that the proposed method compares favourably with existing methods specialized to just one setup.
strength_weakness: strength_weakness: * Neither abstract nor introduction make it clear that the paper is concerned purely with multi-class classification with deterministic labels y=f(x)

* This stands in contrast to the critique in l. 70 regarding competing, `[...] they usually require additional assumptions 69 and approximations on the imprecise information for learnablility`: Assuming deterministic labels is a _very_ helpful simplification for noisy labels, as, together with some upper-bound on the noise rate, it restores identifiability of the model

* A serious problem with the writing of the paper is that, for an attempt at introducing a probabilistic model that can then be used in EM, it does not actually write down the actual probabilisitic models it considers:
In this sentence (l. 158), the paper is extremely vague: 
> If I represents partial labels, then P(Y |I) would 158 have non-zero value over the candidate labels, and be 0 elsewhere. When I represents a set of noisy 159 labels, P(Y |I) would represent the distribution of the true labels, given the noisy labels. When I 160 does not contain any information, i.e., unlabeled data, Y can take any value.

for partial labels, is the underlying assumption $P(Y|I) = 1(Y in I) / |I|$, i.e., uniform distribution over all candidates? How about for unlabeled data?
for noisy labels, you either need to assume a fixed noise model, or $P(Y|I,\theta)$ where $\theta$ are the parameters of the noise model. 
l. 170: `Note that P (X; θ) is omitted from Eq. (5) since P (X) 170 does not rely on θ.` Why? Is this a new assumption? Earlier in the paper, $\theta$ was introduced as the modelling parameter of the generic joint distribution `Let P (X, I; θ) represent a parametric form for the joint distribution of X and I`. The footnote claims `he actual parameters θ may apply only to some component such as P (Y |X; θ) of the overall distribution` but to me, "may" here means that in some situations, such a restriction is possible, whereas I guess the intended meaning is that the model is _always_ supposed to be $P(Y|X,\theta)$?
l. 174: `For independent instances setting` again, this reads as if the previous section had situated the paper in the independent instance setting, whereas this is the first time the topic comes up

> The property of the second term log P (I|X, Y ; θ) is dependent on the nature of imprecise label I. If I contains information about the true labels Y , such as the actual labels or the label candidates, it can be reduced to P (I|Y ), i.e., the probability of I is no longer dependent on X or θ and thus can be ignored from Eq. (5).

This is not correct, just because I contains information about Y, the probability cannot be simplified.

Equation (6) seems to appear out of nowhere: Even following the derivations in C.2, $\mathcal{A}_s$ and $\mathcal{A}_w$ just appear out of thin air in these equations. 
l. 223: `Things become more complicated here since the noisy labels $\hat{Y}$ do not directly reveal the true information about $Y$` I'm not sure what this sentence is trying to say.  $\hat{Y}$ should have some information about $Y$, otherwise, learning is impossible; of course, it doesn't have the full information, but neither do partial labels, so I don't see how this changes the situation compared to the preceeding paragraphs.


The (unreferenced in the main paper) section D.7 in the appendix claims
> Since the settings studied in this work has loss functions derived 943 as close-form from Eq. (5), the time complexity can be viewed as O(1). Thus our method in general present faster runtime without complex design such as contrastive loss.

This argument doesn't make any sense. Solving a system of $n$ linear equations can be written in closed form, yet this is not an O(1) operation.

Overall, after reading the paper, I have almost no idea what the proposed method actually does. Equations (6), (7), (8) contain data augmentations, which I suspect may be important for attaining the performance reported in the paper(?), yet they are not really discussed as part of the proposed method. There are numerous inaccuracies, gaps, and I think even some errors (not fundamental, I think, but in the way the writing describes the math). It is certainly possible that I am misunderstanding something here, but as I see it right now, the paper is not in a shape in which it should be published.


Typos/grammar:
l. 87: our proposed method _generalise_ and subsumes
l. 148: we consider all possible _labeling_ along
questions: questions: See Weaknesses.
limitations: limitations: The authors adequately concede in section 5 that the scalability of the algorithm hasn't been verified yet, sentences like
> While there exist earlier attempts on generalized or EM solutions [...] thus presenting limited 70 scalability on practical settings.

(l. 68-71) seem to suggest that the presented algorithm has this not-yet-verified scalability.
paper_summary: paper_summary: The article addresses the challenge of learning with imprecise labels in machine learning tasks, such as noisy or partial labels. Traditional methods often struggle with multiple forms of label imprecision. The authors introduce a novel framework named Imprecise Label Learning (ILL) that serves as a unified approach to handle various imprecise label scenarios. ILL employs the expectation-maximization (EM) technique, viewing precise labels as latent variables and focusing on the entire potential label distribution. The framework demonstrates adaptability to different learning setups, including partial label learning and noisy label learning.
strength_weakness: strength_weakness: 1. The article's innovation is limited, as the approach of considering ground-truth labels or Bayes label distribution as latent variables and using variational inference for approximation in weakly supervised learning is already a common method[1-2], which suggests that the presented techniques may not be as novel as claimed.

   [1] Xu, N., Qiao, C., Geng, X., & Zhang, M. L. (2021). Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34, 27119-27130. 

   [2] Yao, Y., Liu, T., Gong, M., Han, B., Niu, G., & Zhang, K. (2021). Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34, 4409-4420.

2. Some important baselines should be compared, such as [1,2] in SSL.

   [1] Nguyen, Khanh-Binh, and Joon-Sung Yang. "Boosting Semi-Supervised Learning by bridging high and low-confidence predictions." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.

   [2] Schmutz, Hugo, Olivier Humbert, and Pierre-Alexandre Mattei. "Don’t fear the unlabelled: safe semi-supervised learning via debiasing." *The Eleventh International Conference on Learning Representations*. 2022.
questions: questions: 1. Given that the method of using ground-truth labels or Bayes label distribution as latent variables coupled with variational inference in weakly supervised learning is highlighted in prior works, how does the presented framework distinguish itself or advance beyond these existing approaches in terms of innovation or application?
1. Could you provide a detailed analysis explaining why the unified framework demonstrates superiority over specifically designed approaches?
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces a framework that unifies various imprecise label configurations, with an EM modeling for imprecise label information. The framework is demonstrated can be adapted to partial label learning, semi-supervised learning, and noisy label learning, and the combinations of all above. The experiments results show that the framework surpasses existing methods on various settings.
strength_weakness: strength_weakness: - The implementation of EM over all possible labelings may increase the computation time.
- More related works need to be discussed. The author considers the ground-truth or Bayes label distribution as the latent variables and leverages variational inference for estimating. I am not should this strategy is novel enough in the field of variational inference. I suggest the author add more related works to highlight the novelty of their technique.
- The author should give an explanation of why utilizing an EM framework optimizes the variational lower bound. What is its advantage?
questions: questions: - Can the framework be extended to handle other forms of weak supervision (such as imbalanced noisy label learning) and how?
limitations: limitations: -	Computational efficiency of EM regarding the weakness above. It would be better that the author provide (theoretical or empirical) analyses regarding the computation of the algorithm.
paper_summary: paper_summary: This paper introduces imprecise label learning (ILL), a framework for the unification of learning with various imprecise label configurations, such as partial label learning, semi-supervised learning, noisy label learning, and a mixture of these settings. They propose an EM based method with closed-form learning objectives to handle the problem.
strength_weakness: strength_weakness: Given the extensive content, the main text of the paper omits certain details, which may not be immediately straightforward to follow for some parts, e.g., section 3.2.
questions: questions: NA
limitations: limitations: NA

Comments: title: Rebuttal by Authors (Continued)
comment: Thank you for your response.  We will address the issues below:

a. Please note that  mixture distributions are *in general* not identifiable.  This is not unique to our model.  Widely-used and very popular models such as mixture multinomials, Gaussian mixtures,  HMMs, and even unconstrained factor analysis (which permits us to permute bases) are all provably non-identifiable. Nonetheless, they are perfectly good for modeling data distributions, computing likelihoods or making predictions. This remains true of our model as well. Regardless of the nature of the label $Y$ identifiability is not a requirement for our model, nor indeed for EM estimation of any latent-variable model.  

b. Point taken about the presentation -- we will rephrase "if $I$ contains information about $Y$".   In the rest of the paper we have actually clearly separated cases where $I$ depends only on $Y$, and cases where it depends on both $Y$ and $X$.  If $I = f(Y)$, the statement still holds regardless of whether $Y$ is one-hot.   For $P(I|Y, X) = P(I | Y)$ all that is required is that $I$ is not a function of $X$.  For instance, $Y$ could be a probability vector, and $I$ could just be $Y$ itself (i.e. the information given is the actual uncertain vector $Y$), in which case $P(I|Y)$ is just a unit volume delta at $Y$.

c. $P(X, Y; \theta)$ is being parameterized as $P(Y|X; \theta) P(X)$, where $P(X)$ is the *true* probability distribution of $X$.  This is a perfectly valid decomposition and maximum-likelihood estimation still holds.  One does not need to model *every* component of a distribution, e.g. if a particular component of the distribution is known (or, as in our case, not relevant to the model), it would be redundant to model it.
As for whether Eq. 4 holds by itself,  we refer the reviewer to "Maximum Likelihood from Incomplete Data via the EM Algorithm" by Dempster, Rubin and Laird where the EM algorithm was originally laid out.  Please observe that the algorithm is described entirely in terms of generic PDFs, similar to Eq. 4.,  and specific formulae are only used when specific examples of EM are elucidated.   The EM algorithm is always just specified in terms of PDFs observed and unobserved variables, and model estimation with unobserved variables.  In our case this variable is $Y$. 

d.  In lines 101 and 104 $\mathcal{A}_w(x)$  and $\mathcal{A}_s(x)$ are defined as augmented data samples.  We also point to citation [13] as where this form of Augmentation is introduced.   Thus, we have briefly described the variable, assigned it a symbol, and referred the reader to the specific citation from which this notation and description has been drawn.

e.  Imprecise labels can be of many kinds.  Eq. 14 specifically refers to the partial-label case where the labels belong to a set, which indeed is as the reviewer interprets it.  So, for instance, given a picture, we could be informed ``this is either a dog or a cat''.  But imprecise labels can be of other kinds.  E.g.  given  a collection of pictures,  one may be informed that ``this collection includes 3 pictures of cats''.  This information is true, but it is imprecise because it does not mention *which* three of the pictures are cats.   This does not fall under the partial label condition.   The information is true -- there are indeed three pictures of cats, we just dont know which ones.  So the label is not noisy.
In the noisy label case, we may be informed ``this is a picture of a cat'', but sometimes this information is wrong, it is actually a table.

The challenge in these settings is that each of these settings is different and traditionally have been handled with different algorithms.  The algorithms are specialized -- so, for instance, if we had a collection of data, each with partial labels, we would employ a partial-label algorithm from the literature.  However, if you were to add to this same data set a small bag of instances with the information -- this bag has 3 cats --,  that algorithm would not apply to this bag.
We provide a generic formulation that allows us to consider *all* kinds of imprecise labels under a single umbrella.  So, for instance, the above scenario could be handled.

We understand that the reviewer may be unfamiliar with this problem framework, but we assure them that it is indeed a genuine challenge in the literature, and as the other reviewers recognize, our unifying approach is not only novel, but also a significant contribution.

We hope we have answered the reviewers doubts, and are happy to provide more explanations.
Identifyability:
If the model is non-identifiable, in particular, if you could put an arbitray permutation matrix on the labels, then no model could hope to learn something useful, no? So you generally _need_ to put some assumptions/inducitve priors on the noise transition matrix; e.g., _if_ labels are deterministic, then I think having the largest entry in each row if $\\mathcal{T}$ be the diagonal restores identifiability, but that condition wouldn't be sufficient with randomized  labels.

---

> The property of the second term $\\log P(I|X, Y ; θ)$ is dependent on the nature of imprecise label I. If I contains information about the true labels Y, such as the actual labels or the label candidates, it can be reduced to P (I|Y ), i.e., the probability of I is no longer dependent on X or θ and thus can be ignored from Eq. (5).

To clairfy, what I was objecting to here was the writing of the paper, not the statement that $P(I|Y,X;\theta)$ can be reduced to $P(I|Y)$ in some settings:
The paper writes **If I contains information about the true labels Y**, which is a much weaker condition than what you stated in your rebuttal, **if $I$ is directly drivable from $Y$**.
(this phrase is even repeated later in the paper, **Since S contains the information about the true label Y...**)

Note that this step also implicitly assumes that $Y$  is a one-hot vector; otherwise, if it has non-zero weight for labels from two different bags, $I$ is no longer a measurable function of $Y$.

---

I don't think eq. (4) models $P(X, I;\theta)$; this equation is just the maximum-likelyhood principle and the law of total probability; it doesn't actually tell me anything new about $P(X, I;\theta)$
In fact, I think this goes together with my later point. One of the modeling assumptions is $P(X, I;\\theta) = P(I | X; \\theta) P(X)$, i.e., the parameters $\\theta$ are not used to model the distribution over $X$.

---

You are right, $\mathcal{A}_s$ and $\mathcal{A}_w$ are _mentioned_ earlier in the paper; however, I don't think they are actually defined; they appear in a sentence discussing the Pico method
> It optimizes the cross-entropy (CE) loss between the prediction of the augmented training sample Aw(x) and the disambiguated labels s. 
PiCO learns a set of class prototypes from the features associated with the same pseudo-targets. A contrastive loss, based on MOCO [67 ], is employed to better learn the feature space, drawing the projected and normalized features zw and zs of the two augmented versions of data Aw(x) and As(x)
closer.

This does not actually give any information about what  $\mathcal{A}_w$ and $\mathcal{A}_s$ are.
More importantly, though, the way I read the paper, this is a description of a trick used to improve performance.

But then in 3.1, you try to derive a general framework from first principles, which then is applied in 3.2 to these various settings. This is what happens in C.2.
In the main paper, however, it is not equation (14) that is presented, but (6) which contains these additional augmentations. There is no mention of these in the preceding section 3 or the derivation in C.2;
no reason given why two different agumentations should be used in this cross-entropy formulation. I'm still not quite sure what $p(y|A_w(x), s; θ_t)$ actually is.
If I have four classes, and $s\_1=\\{1, 2\\}, s\_2=\\{3, 4\\}$, and $s=1$, according to the text, $p$ would be something like $\\{0, 0, 0.5, 0.5\\}$; i.e., zero on the classes that are consistent with $s$ and uniform (in the instance-independent case) on the others???


---

> In the noise-free case, presentes uncorrupted information about. E.g., it might state that a collection of instances includes a particular label, without identifying the specific instances that do. While the label information provided is incomplete, what is provided is true. In the noisy case, an instance labeled as
does not imply that its label is actually . We will clarify this in Sec 4.3.

I still don't understand this. In the noise-free, partial label case, the observation tells you some distribution over possible labels. In the noisy-label case, the observation also gives you a distribution over labels. In both cases, you do get some probabilistic information about the true label (assuming an underlying deterministic labeling). 

---

I don't really see the novelty in the described approach. If I come upon an incomplete labelling scenario for the first time, what would be the advantage of me reading this paper, compared to just reading about EM in general? Maybe I'm just missing something fundamental that the other reviewers seemed to get (note my low confidence score).
Dear Reviewer Z7AB,

As the discussion period coming towards the end, we would appreciate if you review our response and we hope it will help resolve your concerns. 
Also please do not hesitate to let us know if there is any more question.
Your insights have been invaluable, and we thank you sincerely for your time and effort.

Best,
Authors.
Thanks for the constructional suggestions!
Thanks for your efforts and suggestion on this paper again!
Thanks for the response, and I maintain my original score.
Thanks for your reply, which addressed my concerns and I decided to raise my score.
The author has addressed my concerns, and I maintain my original decision to _accept_.
Here we address other concerns from the reviewer due to limited space in rebuttal.


> The footnote...P(Y|X;\theta)?

Thanks for pointing this out. 
In the noisy label setting, this is not the case, where the noise transition model also depends on $\theta$ and $X$, as we discussed above.


> Typos/grammar: l. 87...

Thanks for pointing out these typos. 
We have fixed them in our revised paper.

> L68 - 71 seem...scalability.

The "scalability" in the limitation section is meant for larger datasets such as ImageNet-1K that we have not investigated due to the heavy computation cost.
However, our method indeed demonstrates scalability on various settings where the runtime advantage compared to previous SOTA methods is shown in Table 21.