Partial-Label Learning with a Reject Option
Authors: Tobias Fuchs
Florian Kalinke
Klemens Böhm
Venue: Accepted by TMLR ()

Abstract: In real-world applications, one often encounters ambiguously labeled data, where different annotators assign conflicting class labels. Partial-label learning allows training classifiers in this weakly supervised setting, where state-of-the-art methods already show good predictive performance. However, even the best algorithms give incorrect predictions, which can have severe consequences when they impact actions or decisions. We propose a novel risk-consistent nearest-neighbor-based partial-label learning algorithm with a reject option, that is, the algorithm can reject unsure predictions. Extensive experiments on artificial and real-world datasets show that our method provides the best trade-off between the number and accuracy of non-rejected predictions when compared to our competitors, which use confidence thresholds for rejecting unsure predictions. When evaluated without the reject option, our nearest-neighbor-based approach also achieves competitive prediction performance.

Florian Kalinke Published in Transactions on Machine Learning Research (01/2025)
Partial-Label Learning with a Reject Option
Tobias Fuchs
tobias.fuchs Okit. edu
Karisruhe Institute of Technology, Germany
forion. &kit. edu
Karlsruhe Institute of Technology, Germany
Klemens Böhm
klemens. bochm Skit edu
Karlsruhe Instätute of Technology, Germany
Reviewed on OpenReview: https: // ottp://veeprreeeeee.mmmo net/ formm? id= uS1f
Abstract
In real-world applications, one often encounters ambiguously labeled data, where different annotators assign conflicting class labels. Partial-label learning allows training classifiers in this weakly supervised setting, where state-of-the the-art methods already show good predictive performance. However, even the best algorithms give incorrect predictions, which can have severe consequences when they impact actions or decisions. We propose novel risk-consistent neaarest-neighoorbbeee based partial-label learning algorithm with reject option, that is, the algorithm can reject unsure predictions. Extensive experiments on artificial and real-world datasets show that our method provides the best trade-off between the number and accuracy of non-rejected predictions when compared to our competitors, which use confidence thresholds for rejecting unsure predictions. When evaluated without the reject option, our nearest- peeeest-niighho--baee approach also achicves competitive prediction performance.
1 Introduction
Real-world data is often noisy, for example, human annotators might assign different class labels to the same instance. In partial-label lcarning (PLL; Hüllermeier & Beringer 2005 Liu & Dictterich 2012: Zhang & Yu 2015: Xu et al. 2023), training instances can have multiple labels, known as candidates, which only one is correct. While some cases possible to sanitize such data, cleaning is costly, especially for large-scale datasets. Instead, one wants to predict the class labels of unseen instances having sets of candidates only, that is, without knowing the exact cound-truth labels of the training data. PLL algorithms allow for handling such ambiguously labeled data.
However, even the best algorithms give incorrect predictions. These errors can have severe consequences when they impact actions or decisions. Consider, for example, safety-critical domains such as the classification of medical images (Yang et al.) 2009 Lambrou et al. 2011 Senge et al., 2014 Kendall & Gal 2017 Reamaroon
ct al. 2019) or the control of self-driving cars (Xu ct al., 2014 Varshney Alemzadeh] 2017 Hubmann et al,
2017 Shafaei al. 2018 Michelmore et al. 20200). One option to limit fallacies is to employ so-called reject
options, which allow one to abstain from certain predictions if unsure and, instead, let humans decide the label of an instance or the actions to take (Mozannar et al. 2023) Naturally, there is a trade-off arising between the number and accuracy of non-rejected predictions. In the supervised setting, reject options have
already been studied, both, for multi-class classification (Charoenphaklle et. al. 2021 Cao et al. 2022 Mao
et al. 2024 Narasimhan et al. 2024) and regression tasks (Zaou et al. 2020 Cheng et al. 2023).
In the weakly supervisod PLL setting, obtaining sensible reject options is more challenging than in the supervised case ground truth is not. available. Still, a reject option allows for mitigating misclassifications

---

Published in Transactions on Machine Learning Research (01/2025)
and improving prediction quality. Determining whether to reject prediction of PLL algorithm is difficult as the uncertainly involved in prediction-making originates from multiple sources (Hora 1996 Sale et al. 2023: Wimmer et al.) 2023). On the one hand, there is inherent uncertainty due to ambiguously labeled data aleatoric uncertainty). This ambiguity is inffuenced by both instances and classes: In some regions of the instance space, annotators are more likely to mislabel instances than others. Additionally, some class labels are inherently more similar than others. On the other hand, there is uncertainty due to the lack of knowledge regarding the relevance of each candidate label (epistemic uncertainly). To the best of our knowledge, we are the first to study reject options in the context of PLL.
We propose a novel partial-label learning approach based on Dempstcr-Shafer theory (DST: Dempster 1967
Shafer 1986). In contrast to existing PLL approaches, which use point estimates of the candidate label weights Hiillermeier & Beringer 2005 Liu & Dietterich, 2012 Zhang & Yu 2015 Ni et al. 2021 Xu et al.)
2023), our method maintains feasible region, also known as credal set. Maintaining such a credal set is
beneficial in assessing the certainty of predictions as our experiments show.
Contributions. We summarize contributions as follows.
Algorithm with reject option. We introduce DST-PLL, a novel haarest-neighhor-beee based partial-labe learning algorithm with a reject option that Icarns from ambignously labeled data. The algorithm elfectively determines whether to reject a prediction and provides the best. trade-off between the number and accuracy of non-rejected predictions when compared to the state-of-the-art PLI mothods.
Experiments. Extensive experiments on artificial and real-world data support our claims. We make our code and data openly available
Theorelical analysis. We analyze DST-PLL, give a closed-form expression of its expected decision boundary under mild assumptions, and prove its risk consistency. The runtime analysis shows that the proposed method's runtime is dominated by k-nearest-nniihhoo search, which has average time complexity of O(dk n)), with d features and n training instances.
Structure of the paper. We discuss related work Section P define the problem setting in Section 3 and propose our method in Section | Soction 5 features experiments and Section concludes. All proofs and additional experiments are in the appendices.
2 Related Work
Imperfect data often renders the application of supervised methods challenging. Weakly supervised learning tackles this sctting and encompasses a varicty of problem formulations (Bylander 1994 Hady & Schwenker 2013
Ishida et al. 2019) including PLL. We discuss related work regarding PLL in Section 2.1 Section 2.2
elaborates on related work dealing with uncertainty in ML. through the lens of reject options.
2.1
Partial-Label Learning
The early PLL approaches transfer common supervised learning frameworks to the PLL context: Grand
valet (2002) proposes a logistic regression formulation, Jin & Ghahramani (2002) propose an expectation- method, Nguyen & Caruana 2008) propose an extension of support-vector machines, and Cour et al. (2011) introduce an average loss formulation allowing for the use of any supervised method. However, these approaches (i) cannot model the relevance of candidate labels in the labeling process or (ii) are not robust to non-uniform noise in the (i) arises from the lack of knowledge about the candidate label relevancies of the PLL model (epistemic uncertainty) and (ii) arises from the random label noise in the candidate sets and is inherent to the PLL problem (aleatoric uncertainty).
maximization strategy, Hüilermeier & Beringer (2005) propose a x-nearest--eighiee
candidate sets. We emphasize that (i) and (fi) stem from two different sources of uncertainty:
2

---

Published in Transactions on Machine Learning Research (01/2025)
More recent approaches address (i) and (ti) in the following ways: Zhang & Yu 2015 Zhang al. (0016)
Xu et al. (201 Wang et al. (2019); | Feng & An (2019) Ni et al. 20021) leverage ideas from representalin aa
learning Bengio et al. 2013 Yu & Zhang (2017) Wang et al. (2019); Feng & An (2019); Ni et al. (2021)
extend the maximum- margin idea, Liu & Dictterich (2012): Lv et al. (2020) propose extensions of the
expectalioon-maxiiiiaaiioo strategy, Zhang et al. (2017) : Tang & Zhang (2017); Wu & Zhang (2018) propose
stacking and boosting ensembles for more robust prediction-making, [v et al. (2020); Cabannes
(2020)
introduce minimum loss formulation to iteratively remove noise, and Feng et al. (2020); Lv al. (2020); Xu
et al. 2021];; Wang et al. y He et al. (2022) Xu et al. (2023); Fan et al. (024) employ deep-learning
techniques such as auto-encoders, data augmentation, and transformers in the context of PLL. These methods
address (i) and (ii), featuring already good performance. However, considering a reject option of those
methods based on a confidence threshold, as proposed in the supervised setting (Ni et al. 2019)) yields
sub-optimal results as all of the listed methods mix uncertainty stemming from (i) and (ii). We claim that keeping both separate provides a better trade-off for the number and accuracy non-rejocted predictions
(Section 5).
In the PLL context, consider (Hüllermeier & Beringer 2005 Zhang & Yu 2015 Zhang al. 2016
Xu et al. 2019 Wang et al.) 2019) as most closcly related to our proposed method as these approaches consider an instance's neighborhood to infer its class label. To the best of our knowledge, we are the first to propose an extension of the k-NN classifier that keeps uncertainty from (i) and (ii) separate by leveraging Dempster-Shafer theory.
2.2
Reject Options
Recently, much attention has been given to the study of reject options (Mozannar al. 2023 Mao et al., 2024
Narasimhan et al. 2024). A reject option allows one abstain from predictions and defer them humans rather than making uncertain and possibly harmful decisions.
There are two common strategies in rejecting predictions in the supervised setting: The confidence-based and the classifier- approach (Ni et al. 2019 Cao et al.) (022). The confidence-bast based strategy uses a threshold on the models' confidence in order to accept or reject predictions. Common model choices for quantifying the confidences are Bayesian methods (Kingma & Welling 2014 Kendall & Gal 2017) and ensembles (Lakshminarayanan et al., 2017 Wimmer et al. 2023). In contrast, the classifeer-egjccoor approach jointly learns the classifier and rejector (Ni al.) 2019 Mao et al. (2024), which can have beneficial theoretical properties. However, the rlassifter-rjeectorr approach is less flexible than the confidence based strategy as it is coupled with the concrete loss formulation of the classifier. We propose an extension of the confidence-based strategy for partial-label learning in Section
4.2
Calibration methods (Nacini et al. 2015 Guo et al. 2017 Ao et al. 2023 are also related to the confidence- based rejection strategy as both are used to make statements about the certainty of predictions. While reject options provide a binary decision, calibration methods modify the predicted confidences such that they align with the observed accuracies. In this sense, both approaches are orthogonal and cannot. directly be compared. In our work, we focus on reject options.
3 Problem Statement and Notations
This section formally defines the partial-label learning problem with reject option, establishes notations used throughout this work, and summarizes Dempster-Shafer theory.
3.1 Partial-Label Learning (PLL) with Reject Option
Let. X' = Rd denote a d-dimensional real-valued feature space and y = [1] := (1,..., . the finite set of < N classes. A partial-label learning training dataset = {(,,, ) E [n]] of n instances contains feature vectors n, E X' and a set of candidate labels 8; y for cach E [n]. All instances have an unknown ground-truth label y E y, and yi E s;. Further, the candidate labels s; can be partitioned into s; = {1r} U2 with y/ $ 21, that. is, z C )" {we} are the false-positive labels.
3

---

Published in Transactions on Machine Learning Research (01/2025)
The sample space is st = X x x2>>2> It is equipped with the Borel o-algebra B(2), yielding the measurable
space ((2) with respect to which the set of all probability measures (1, (())) is defined. Tet (X) ~ IP (2, B()),, where the random variables X : S2 52 + X, Y Y - and S 52 + 2 govern an instance's ' features, true label, and candidate labels, respectively. We denote the corresponding marginal measures by associating the correspouding index to P, for example, Px is the distribution of X. Pxy denoles the joint distribution of (X
PLL aims to train classifier g X' + Y that, given measurable loss function L: x2 - R20 minimizes the risk R(g) E ((x,))~ (()(())))) Common choices for L include the average loss (Nguyen & Caruana 2008 Cour et al. 2011) and the minimum loss (Lv ct al., 2020 Xu et al. 2021). The empirical version of the risk substitutes the expectation with a sample mean, that is, R(g) 2 L(g(zi), st).
2i
As misclassifications can be quite harmful, we look at the possibility of rejecting predictions, that is, abstaining from making these prodictions and, instcad, deferring the decisions to humans. In the PLL sctting, a reject option rg : X' + 2 associated with a trained classifier 9 either returns g's prediction (accepl) or abstains from making any prediction at all (reject), that is, [x(x) {0, {9(x)}) for z E X, with [x) 0 denoting a reject. We then define the rejection probability r(I'g) (x
0) and the expected error err(F,) = (xx.s)-Pxs(C(ol () (T,())00)) of accepted predictions. Naturally, a trade-off arises between the number and accuracy of accepted predictions, which leads us to the risk
Rx (xx ([xxs)))xss((((X)), () + (r,()))0)) err(T,) + Ar(Tg))
(1)
for A 2 0; L characterizes the cost of misclassification and parameter À the cost of rejecting a prediction. In order to make well-informed decisions about when to reject, we are using Dempster-Shatee theory, which we elaborate in the following.
3.2 Dempster-Shafer Theory (DST)
Dempster-Shafer theory (DST; Dempster 1967: Shafer 1986) allows for dealing with uncertainty by assigning probability mass to sets of events without specifying the probabilities of individual labels; incorrect labels do not oblain any probability mass. DST builds upon two core quantities, so-called belief and plausibililyy Informally, belief collects all evidence that supports a hypothesis and plausibility collects all evidence that does not contradict a hypothesis. We argue that DST is a perfect fit for partial-label learning as one may interpret the ambiguous candidate sets as evidence regarding a label hypothesis. We further exploit belief and plausibility to inform our reject option, taking into account the difference between the supporting and non-conflicting evidence. In contrast, existing PLL approaches (Hüllermeier & Beringer 2005 Cour et al,,
2011 Liu & Dietterich 2012 Zhang & Yul 2015: Ni et al. 2021 Xu et al. 2023) initially assign some probability mass to each label candidate and subsequently refine them. By doing so, most probability mass is first allocated to labels that are certainly incorrect, as only one candidate is the true label. This known method renders handling the noise coming from incorrect candidate labels challenging.
With these intuitions, we now recall DST formally. In DST, a basic probability assignment (bpa) m 2 + [00, 1] assigns probability mass to subsets of y. m satisfics m()) = and Eagy m(A) =. This differs from standard probability as P())) = 1 for any IP E .M (1',2) 2') but m()) S 1. Also, the mass allocated to non-intersecting sets does not necessarily add up to the mass allocated to the union, that is, one can have m ((((122) # m({1]) + m({))). In this sense, DST allows for more flexibility as one can allocate mass on the set (1,2) without needing to specify any mass for {1} and (2) if uncertain. The sets C y with m(A) > are called focal sets of m. The mass allocated to the set of all possible alternatives m m()) can be interpreted as the degree of ignorance; is the mass not supporting a specific alternative within )
The basic probability assignment m does not induce single probability measure P on (,,2²) but rather set of probability measures (,, which is called credal set (Abellán et al. 2006 Cuzzolin 2021). The probability measures IP E Cm (0,2"") 2²) are restricted by imposing lower and upper bounds, which are called belief and plausibility, respectively. They are defined as follows:
belm (A) == Σ mB) and (A) == belm (D))A)=
Σ
m(B) for 2". (2)
BCA
BCILANB4)
4

---

Published in Transactions on Machine Learning Research (01/2025)
Recall that belief collects all evidence that supports hypothesis A E 2 (or more specific one B S A), and plausivility collects all evidence that does not contradict a hypothesis A E 2 (or more specific one B S A). With belief and plausibility as above, the set of all probability measures supporting m is
{ MT (>),2)) belm (A) S (P(A) S (A) for all A A C CM
Further, DST provides rules to combine m-s from multiple sources (Dempster 1967 Yager 1987a b). This is beneficial in the PLJ setting as there is several conflicting evidence about the class labels within a neighborhood of instances. Estimating credal set (),2²) from such a neighborhood allows us to construct
an effective reject option, which we detail in Section 4.2
Several methods already leverage DST in supervised learning [aadler Schümann 1988 Denoeux 1995
Tabassian et al., 2012; Sensoy et al. 2018 Denocux 2019; Tong et al.] (021). We consider the ncarest neighbor approach by Denoeux (1995) to be most closely related to our approach. Here, basic probability assignments are constructed from the nearest neighbors of an instance. Then, Dempster's rule is used to combine them into single bpa. Their analysis is, however, not transferable to our case because they only have singletons or the full label space as focal sets, making set intersections in the combination rule easy to handle. In this sense, we examine a more general set setting since we allocate probability mass to arbitrary subsets.
4 Our Method: DST-PLL
This section introduces our novel partial-label learning method DST-PII Based on the labeling information of an instance's nearest neighbors, we construct basic probability assignments within Dempster-Sie -haffer theory. These bpas inform the prediction and rejection decisions as discussed in Section 4.1 and Section 4.2 respectively. Regarding the reject option, we propose novel variation of the confidence- based rejection strategy: The confidence threshold is adaptively selected per-instance basis dependent the amount of incorrect label noise. The more noise from incorrect labels there is, the more confident the model needs to be to accept prediction.
Algorithm 1 outlines DST-PLL, which we summarize in the following. We denote by (N S X' x2 the set of the k-nearest neighbors instance z with their associated candidate labels. To predict the class label of an instance (Tine 11), the algorithm first transforms information from ä's neighbors NN&(#) into bpas m; (Lines 3 7), collects these into evidence set & (Line 8), and combines the bpas into using Yager's rule (Line 10; Yager 1987a bj. Section 4.1 elaborates on these steps. Section 4.2 elaborates on how we extract our reject option from (Line 12). Section 4.3 shows that the resulting classilication rule is risk-consistent. We
analyze our algorithm's 's runtime in Section
4.4
4.1 Making Predictions
Basic probability assignments. Following the standard assumption that neighboring instances in feature space are also close in label space, we combine the evidence from the k-nearest neighbors (2;, E NN+(#) of a given instance ã E X' with its candidate labels 5 C (3 = ) if & is a test instance).
When looking at a neighboring instance (I4,, 8,) E NN, there are generally two cases: either (i) (a2, 8)) provides information about the correct label of Z or (ii) (Iz s;) is irrelevant for finding the correct label of æ. To address (i), we allocate probability mass on the candidates 8; of neighbor x;. To address (ii), we allocate probability mass on the full label space y indicating uncertainty about the correct label of T.
More formally, for fixed E [ke], the candidate labels s; do not provide any valuable information if they support all (3 C s;) or none (8ns 8; 0) of the labels in 8; we use bpa of (x) = 1 (Line 4). We set m((A) = 1/2 if A or A 3 $4, else m, = 0 (Line 6), where 1/2 equally weights evidence. We later elaborate further on this choice and demonstrate the application of the proposod classification rule Example 4.1 Note that we make the common assumption that the true label of instance is always in á (Cour et al. 2011 Liu &
Dietterich 2012 Lvv et al. 2020 Ni et al.] 2021). While our definition of the m is similar to Denoeux
5

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm DST-PLL (Our proposed method)
Input: Partially-labeled dataset D = {( s:) & x2 E [n]], number of nearest neighbors k, instance
I E for inference with candidate labels 3 ) ($ y if ă is an unseen test instance);
Output: Prediction g(z) and reject option y (2) for instance ā;;
1: E-0
2: for (e),s)) E NNN (z) do
if S or 8 @ then
if 8, clsc
4:
m. 2 10 A
else
1/2 A
A == =ns
m. 2 [0, AH
0
else
7: end if
E E {m,)
end for
10: in 4 yager combination(,, E)
arg maxyes m((v))
if maxyes m((v}) > 0,
11: g(ā)
Randomly pick from arg max m((A) else;
[a(())} A 0 as defined
12: (T) +
clse;
13: return (g(#), I (x))
1995 we target more general setting our focal sets be arbitrary subsots instead of only singletons the full label set.
The bpa m, has the following four effects on belief and plausibility as defined in [2] (i) A set of candidates A has maximal belief, that is, belm. (Al = 1, if covers 8, that is, 3 S A. (ii) A set of candidates A is plausible, that is, plm, (A) > 0, it supports at least one of the candidate labels in 8, that is, An ø * 0. (iii) There is a 8 but does not cover all candidates
gap, that is, belm, (A) < plm, (4), if A supports some candidate s or supports some candidate in but does not cover all candidates of (iv) Class labels y E 8 8 8; are maximally plausible, that is, plm, ((v)) 1.
Evidence weighting. Our definition of the m.-s (Algorithm) Line 6) also permits a more general view, that is, (A) = a if A = 8, mx(A) = 1 - a if A = 8 8i, and mA) = 0 otherwise, for some a E 1). However, without further assumptions, one cannot know how relevant the information from a particular neighbor is. The setting of a = 1/2, which we use, weights supporting and conflicting evidence all neighbors oqually. In other words, if neighbor's evidence excludes some candidate labels from consideration, it is of equal importance compared to supporting some candidate labels. Therefore, set 1/2.
Evidence combination. Given the set = {m, [k]], we combine all m using Yager's rule Yager 1987a Dempster's original rule (Dempster) 1967) enforces mQ() by normalization, which is criticized for its unintuitive results when facing high conflict. (Zadeh) 1984). Instead, Yager's rule first collects overlapping
evidence in q 2 + [o, 1] and creates a valid bpa in 2 [0, 1] by
A = 0,
q(A) = Σ
Il (A,)
and
m(A) = 9())) q())
if A = ), else.
(4)
A Ag....AAY j=1 nt Ai=A
((AA)
We implement this efficiently with hash maps storing only the focal sets!!

---

Published in Transactions on Machine Learning Research (01/2025)
Classification rule. After the combination into in by 4), we extract prediction g(ē) for instance ø (Line 11). We predict the class label with the highest probability mass arg maxyes mv(y)) if any has non-zero mass, or else randomly pick from the subset with the most mass arg maxAçs m(A). When tied, we use the subset with the smallest cardinality. In the following, we present an example of our classification rule.
Example 4.1 (Classification rule). Let & = 3, F an unseen test instance, (zi, 8;) E NNA (T), ) (1,2, 3), 81 = =11], $2 = [1,2}, 2), and $3 = {1,3}. Then, mi{{1)
m)) = 1/2, m2 2}) = m²()/) = 1/2, and mm((113]) = ()) = 1/2. All other subsets receive a mass of zero. Using Yager's 's combination rule, we obtain m{[1)) = 5/8, m((1, 2]) = m({1,3)) = m()) = 1/8, and m(A) = for the remaining A C y.
Therefore, we predict label 1.
4.2 Proposed Reject Option
To limit the impact of misclassification, our method provides a reject option F'g, that is, the algorithm can abstain from individual predictions if unsure (Algorithm [ Line 12). Our formulation builds on the confidence-bret based rejection strategy (Section 2.2), that is, A == conf(g) - 9 with conf(g) e [0, 1] being the model's confidence and 0 E [0, 1] the confidence threshold. We adapt this sctting to the PLL context by changing the confidence threshold On based on the amount of noise present.
Recall from (3) that the belief and plausibility regarding in act as a lower and upper bound of the probability mass, respectively. The intuition of our reject option is as follows. If the lower bound (belicf) on the probability mass of our predicted label exceeds the maximal upper bound (plausibility) on the probability mass regarding any other label, we can safely make the prediction.
In other words, if there is a class label different from the prodicted one that is quite plausible (high maxyes (+) Pla Pla(()))), we require a high belief mass in order to be certain about the prediction, that is, the belief must satisfy belm(())) > max (0) pl (v)). If, instead, there is no other plausible candidate label, we can be sure of our prediction with less belief mass.
We formalize this intuition in the following. Let i be the resulting bpa as determined by Algorithm ! and
Aa belm (101)
((v)) with f:= arg max m ((v)).. (EA)(0)
max
(5)
yEa
(=))nf(y))
(=0m)
In other words, we instantiate the model's confidence with the model's belief mass conf(g) bela. (10)) of the predicted instance ú and the confidence threshold 0n based on the amount of noise regarding other labels y * 8, that is, 6m = max)es)(() pl. (fv)). Note that the dependence on m allows for setting the threshold adaptively.
satisfies several desirable properties, which we collect in Theorem 4.2 and elaborate on in the following. Theorem 4.2. Let y be the label space, 3 X the instance of interest, $ C y its candidate labels (s=Y² if T is test instance), g(ō) our algorithmm's 's prediction, and in the resulting probability mass as determined by A Algorithm| Then, the following hold:
(i) If (ā) has been picked randomly (Algorithm 1 Line 11, second case), then Am S 0,
(it if Am 0, then P((g()))) P((v)) for all P (,)())2))) and y E 3 (9()), and (i if (((((())) > 1/2, then Au > 0. The converse of (tii does not hold.
Based on these considerations, we define the accept and reject regions of our method as follows.
1. Accept. When Am > 0, we are in the accept region and I (z) = { The lower-bound probability of the class label ÿ is greater than the upper-bound probability of any other class label y + g. that is, P((0)) > P((v)) for all P E (),, 2²)) and y + ÿ-
Reject. When Am I O, we are in the reject region and (, = 0: The lower-bound probability of the class label ÿ is less than or equal to the upper-bound probability of another class label y + g. There exists IP C. (Y, 2²) and y # 9 with P((O)) P(()))
7

---

Published in Transactions on Machine Learning Research (01/2025)
We remark that Theorem 4.2 (itz) implics that DST-PLL rejects fewer predictions than the k-nearest neighbor reject option by |Hellma (9700), which requires more than 1/2 of all votes to be certain. Our method can accept predictions with less than 1/2 of all probability mass on a single class label, while still satisfying Theorem 4.2 (it), that is, a prediction is accepted, the decision remains unchanged independent of PP E Cm, It. follows that having fewer rejections does not come al the expense of an increase in the expocted error. In the following, we provide an example of the proposed reject option.
Ezample 4.3 (Reject option). Assume the setting and result of Example 4.1 m({1}) = 5/8, m = mi(1, 3)) = m)) = 1/8, and (mA) = 0 for the remaining A S y. Our prediction is 9 = arg maxyes m((y)) = with = y. Hence, Aa = vocl((()) maxyea)(0) pla((v)) = bela ((1}) - -pla (12)) = 5/8 2/8 = 3/8.
Since Am = 3/8 > 0, we accept the prediction y =
4.3 Consistency
Our classification rule yields a consisenstt classifier, which we demonstrate in the following. As is common
in the literature (Cour et al., 2011 Liu & Dietterich] 2012 Feng et al.] 2020 Lv et al., 2020) and required to
obtain statistical guarantocs in the PLL sctting, we fix label (noise) distribution (Assumption 4.4] that
permits further analysis of the proposed algorithm. Appendix D experimentally verifies that Assumption 4.4
is satisfied on worl-world datasets.
Assumption 4.4. Let ă X' be the instance of interest with hidden true label ÿ and 1= 1 3 classes. Its k partially-labeled neighbors are (z), E NN+(ā). Label Ne E < (ū) denotes the class label that co-occurs most frequently with label ÿ in i's neighborhood. We assume that the true label dominates the neighborhood, that is,
if y, s 0, E 2, D E 2, if
(i)
y= = 8, Be 2, and 8 Y
(i1)
P(S - x (v) Partitioning valid (c))-(c))
Y-
21-211
2 P2
if y = D and Ne $2
(iii)
2-1-1 P3
y + ū. $ 2 2, and 8 +0,
(iv)
with P1.P2.P8 E (0, P1 +p 1, and pı 2 P2 p3
Bocause P1 + P2 > Ps, Assumption 4.4 implics Pxy (Y = - X = x;) > PX x0 x;),, that is, points that are close in feature space are likely to have similar class labels. We note that this assumption is related to the ambiguity degree condition by Cour et al. (011)) as both make sure that the noise labels do not overwhelm the PLL algorithm. Assumption
4.4 enforces this as Pl p2 2 pa > 0. Having all cases spelled
out as in Assumption 4.4 benefits the proof of Lemma 4.5
In the following, we elaborate on the four cases (ir))
If a candidate set contains all labels (s = )), no labels (s= 0), or does not allow to be partitioned into true label and false-positive labels 2, it is not a valid candidate set (Hüllermeier & Beringer 2005 Cour et al. 2011 Liu & Dictterich 2012 Zhang & Yu 2015) (t) handles this pathological
setting by assigning zero probability.
In (it), H's label ÿ is also the label of neighbor x, and Ve, with which ÿ is most commonly confused, is also part of the candidate set 84- There are 2/-2 sets 8 that contain ğ and Te and are not y
In (iit, I's label is also the label of neighbor zy. Label ye is not part of the candidate set s; of neighbor 24. There are 2 sets that contain but not Øe-
In (iv), neighbor x, has different labels altogether and is irrelevant for explaining the class label of instance ă. There are 2/-1 sets that do not contain the label ÿ (excluding the empty set). Each of the |s| candidates can be the neighbor's correct label.
Theorem 4.6 which establishes risk-consistency, hinges on the following intermediate results in Lemma 4.5
The fact that the my- can have arbitrary focal sets renders the direct application of Yager's rule (4) in our
8

---

Published in Transactions on Machine Learning Research (01/2025)
analysis challenging. The proof of Lemma 4.5 mediates this issue by providing closed-form solution for the resulting bpa m.
Lemma 4.5. Assume the setting of Assumption 4-4 let ã E X be the instance of interest, ÿ E y its hidden true label, We (' the incorrect label with which ÿ is most often confused, and in the resulting probability mass as determined by Algorithm
Then, the following hold:
(1) Ep(m((()) X - > 0, and
(1) Bp|m (ū)) X x] E- M ((e))
X 2].
As the probability mass of the hidden true label ÿ positive in expectation by (i), we can reduce our analysis the first case of our decision rule (Algorithm [ Line 11). (ti) shows how Assumption 4.4 propagates when applying Yager's rule all k neighbors' m,-s. The label that dominates the neighborhood obtains the highest probability mass. To our knowledge, it is the first time that such result has been shown for arbilrary focal sets in m. Denoeux 1995 analyze the special case when all focal sets are singletons or the full label space.
To put Theorem 4.6 into context, we recall the concept of risk consistency (Devroye et al. 1996.. The Bayes classifier is defined by g² = arg ming:x->> Rgy); it has the least overall risk. Let gn be the classifier trained by Alg. | with n instances. Its empirical risk R(gn) can be computed by substituting the expectation with a sample mean. It is makk-consistent R(gn) - R(g") for n + oo almost surely. Theorem 4.6 establishes the risk consistency of the proposed classifier.
Theorem 4.6. Assume the setting of Assumption 4-4 let 9n : X - y be the classifier trained by Algorithm with n training instances, 9" : X + y the Bayes classifier, and it E a fixed instance with unknown true label ÿ. Then, the following hold:
(i) E
(x) Ø. and
(1180) ced Pxs Pxs
(ii) lim (R(g)) - R(g")) = 0 almost surely.
4.4 Runtime Complexity
We decompose the overall runtime of our approach in Algorithm Q into k-times querying one nearest neighbor and creating its bpa (Lines 2-9), the cost of Yager's 's rule (Line 10), as well as extracting predictions in Lines 11-12. Using the ball-tree data structure (Omohundro 1989), querying one neighbor takes (dd n) time on average. In the worst case, query time is Odd). One builds a ball-tree in O(dnlog n) time. We construct bpa m; by storing its focal sets within a hash map and combine all m,-8 as defined in ( There are at most min (2*,2) 2 focal sets of m: Each m, has at most two focal sets producing 2 combinations and there are at most 2' possible subsets of y. We take the minimum as both are upper bounds. Looking up focal set in the hash-map requires (l)) time as the key length is variable. Extracting prediction and the reject option then requires (12) time. Combining the above yields worst-case complexity of Oddn I (min (2*, 2') + max(k: ())- Since I and are constant, the nearest-neighbor search dominates. The average total runtime of the search is logn..
5 Experiments
Section 5.1 lists the tested methods, Section 5.2 shows our experimental setup, and Section 5.3 collects our main findings. Additional results and description of all hyperparameters can be found in Appendix D
5.1
Algorithms for Comparison
While many PLL algorithms exist (soc Scction 2p. we focus on statc-of-thh-art methods commonly used in the literature. We consider ten methods: PL-KNN (Hüllermeier Beringer 2005). PL-SVM (Nguyen Caruana
2008), IPAL. (Zhang & Yu 2015), PI-Fooc (Zhang et al. 2017), PRODEN [Ly al. 2020), Cc (Feng et al.
9

---

Published in Transactions on Machine Learning Research (01/2025)
CROSEL TTan et al. 2024 and DST-PLL (our proposed method). Note that we decided to not consider PICO (Wang et al. 2022) as it is only applicable to image data. In our experiments, however, we consider various image and non-image datasets as discussed in Scction 5.2
20200, VALEN (Xu et al. 2021). PoP (Xu et al. 2023
We choose the parameters of all methods as recommended by their respective authors and use the same base models for all of the neural network approaches. Appendix D.2 discusses the choices of all hyperparameters in more detail. As base models, we pick the LeNet architecture (LeCun et al., 1998) for the MNIST datasets and 1-300-300-3004 MLP (Werbos 1974] for all other datasets. For PL-KNN and our method, use variational auto-encoders (Kingma & Welling, 2014) to reduce the feature space dimensionality of the MNIST datasets and compute the nearest neighbors on the hidden representations. As our competitors do not provide a reject option, we use threshold on their model's confidences, that is, the maximum probability outputs, to evaluate the trade-off between the fraction and accuracy of non-rejected predictions.
5.2 Experimental Setup
Data. Following the default protocol [Cour et al. 2011 Zhang & Yu 2015
Lv et al., 2020 Xu et al., 2023)
we conduct several experiments using datasets for supervised learning with added artificial noise as well as experiments on real-world partially-labcld data. We repeat all experiments five times to report averages and standard deviations. For the supervised datasets, we use the ecoli (Horton & Nakai 1996). medllpple-
Duin 2002), pen--iggtss (Alpaydin & Alimoogu
1998). semeion (Buscemna & Terzi 2008), solar-flare [Dosson]
& Hedeman 1989). statlog-landsat landsat (Srinivasan 1993) and theorem datascts (Bridge et al. 2013) from the
UCI repository (Bache & Lichman 2013p. These datasets contain between 336 and 10 992
instances each.
Also, we use the popular MNIST [eCun et al. 1999). KMNIST (Clamuwat et al. 2018 and FMNIST
datasets (Xiao et al. 2018), which contain 60 000 images each similar to other datasets like CIFAR-10 and
CIFAR-100 (Krizhevsky 2009). )009). For the partially labeled data, we use the bind-song (Briggs et al. 2012) )
flickr Huiskes & Lew 2008), yahoo-news (Guillaumin et al., 2010). and msrc-v2 datasets (Liu & Dietterich 2012
They contain between 1755 and 22 762 instances.
Noise Generation. We use three noise generation strategies to partially label the supervised datasets: uniform, class-dependent, and instance-dependen noise. Uniform noise (Liu & Dietterich 2012) adds three uniform random noise labels to a fraction of 70 70% of all instances. Class-dependent noise (Cour et al., 2011) randomly partitions all class labels into pairs and adds the partner label as noise to 70 % of all instances having the other label. Instance-dependent noise (Zhang et al.] (021) first trains a supervised classifier g : X + MI (V,27 2²"). Given an instance x with true label y, a flipping probability of (x( == 9/(z)/ / 9y' (I) for 7 + y determines which noise labels to randomly pick.
5.3 Results
Prediction Performance. Table [abllssswwss the average test-set accuracies and standard deviations over all UCI datasets with class- and instance-dependent noise, MNIST datasets with class- and instance-dependent noise, and the real-world datasets. The algorithms with the highest accuracies as well as the algorithms with non-significant differences using paired t-test with level a = 0.05 are emphasized. Our approach (DST-PLL) performs comparably to the other methods. We note that none of the methods is best across all settings. For example, Cc performs best in four out of five settings but significantly outperformed by our approach on the real-world experiments.
Reject Option. To compare our reject option with the other methods, we use a threshold of Am 0 for our proposed approach (Algorithm [1 Line 12), confidence threshold of 90 % for classifiers outputting probability distribution over the class labels, and a threshold of 50 % of all votes for PI.-KNN to not reject a prediction, which is in line with the reject option by Hellman (1770).
Table shows the average empirical reject-option risk and standard deviation across all experiments for varying X. We compute (T,)) = eir(T,) + AF(T,) by using ground-truth information to calculate the non-reject error (T,(T,) and counting the number of rejects to calculate the reject rate f(Tg). The algorithms with the lowest risks well as the algorithms with non-significant differences using paired t-test with level
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Average test-set accuracies (# std.) the UCI, MNIST, and real-world datasets. The UCI and MNIST datasets are artificially augmented with class-dependent and instance-dependent noise. The best algorithms (highest accuracy) as well as algorithms with non-significant differences are emphasized. The significance analysis uses a paired student t-test with level a = 0.05.
UCI datasets
MNIST-like datasets
Algorithms
Real-world
Class-dep.
Inst.-dep.
Class-dep.
Inst.-dep.
PL-KNN (2005)
81.2 (# 13.9)
75.8 (# 11.1)
92.2 (+ 4.1)
84.8 (+ 7.2)
53.4 (# 10.9)
PL-SvM (2008)
62.3 (= 16.3)
43.8 (# 16.4)
67.5 (+ 9.8)
47.8 (+ 8.2)
39.1 [= 9.6)
IPAL (2015)
79.3 (1: 17.1)
75.3 (1 18.3)
92.9 (1 4.3)
88.5 (1 6.6)
58.7 (+ 9.8)
PL-Ecoc (2017)
63.7 (#: 13.1)
66.5 (# 12.5)
64.3 (1 14.3)
51.7 (1 10.7)
46.2 (# 10.2)
PRODEN (2020)
81.6 (+ 14.0)
78.1 (+ 13.2)
93.9 (+ 4.4)
88.2 (+ 6.0)
64.2 (+ 8.2)
Cc (2020)
81.3 (+ 14.2)
78.8 (+ 13.6)
93.9 (+ 4.5)
89.6 (+ 5.7)
49.2 29.7)
VALEN (2021)
79.7 (# 15.3)
75.6 (# 12.7)
91.8 (# 4.2)
83.4 (# 8.4)
63.6 (= 9.7)
POP (2023)
81.5 (# 14.0)
78.1 (# 13.1)
93.9 = 4.5)
88.1 (# 6.0)
63.6 (= 8.4)
CROSEL (2024)
79.9 (1: 17.2)
78.4 (4: 14.5)
94.2
(+ 4.5)
88.9 (4 6.7)
46.3 (# 28.9)
DST-PLL (ours)
80.8 (#: 14.1)
75.4 (# 10.7)
92.0 (# 4.2)
84.5 ( 7.2) 52.0 (= 11.6)
Table 2: Average empirical reject option risk (Tg) defined in 1 for various values of A. The best algorithms (smallest risk) well as algorithms with non-significant differences are emphasized. The significance analysis uses paired student t-test with level a 0.05.
Average RA(T,) (+ std.) across all experimental settings
Algorithms
À = 0.00
X = 0.05
À 0.10
À 0.15
À = 0.20
Pt.KNN (2005)
0.11 (+ 0.17)
0.15 (+: 0.18)
0.18 (+ 0.19)
0.21 (+: 0.20)
0.25 (+ 0.21)
PL-SVM (2008)
0.19 (# 0.27)
0.23 (# 0.28)
0.27 (# 0.29)
0.31 (# 0.30)
0.35 (# 0.31)
[PAL (2015)
0.19 (+ 0.17)
0.20 (+ 0.18)
0.21 (# 0.19)
0.22 (# 0.20)
0.23 (# 0.21)
PL-Ecoc (2017)
0.25 (1 0.27)
0.29 (1 0.27)
0.34 (1 0.28)
0.38 (1 0.28)
0.43 (1 0.28)
PRODEN (2020)
0.12 (1 0.11)
0.13 (1 0.11)
0.14 0.14 0.12)
0.15 (1 0.12)
0.16 (1 0.13)
Co (2020)
0.16 (1: 0.18)
0.16 (1 0.19)
0.17 (1 0.20)
0.18 (1 0.20)
0.19 (1 0.21)
VALENN (2021)
0.20 0.14)
0.20 (+ 0.14)
0.20 (+ 0.14)
0.20 (+ 0.14)
0.21 (+ 0.14)
Pop (2023)
0.12 (+ 0.11)
0.13 (+ 0.11)
0.14 (+ 0.12)
0.15 (+ 0.12)
0.16 (+ 0.13)
CROSEL (2024)
0.15 (# 0.20)
0.16 (# 0.21)
0.17 (# 0.21)
0.18 (# 0.21)
0.19 (# 0.22)
DST-PLL (ours)
0.05 (+: 0.07)
0.07 (+ 0.08)
0.10 (#: 0.10)
0.12 (+ 0.11)
0.15 (#: 0.12)
Table 3: Fraction of rejects and non-rejected test-set accuracy of all methods with standard deviations across all experimental settings and five repetitions with different random seeds. To obtain the results, tune the thresholds 0 such that each competitor has number of rejects that is comparable to that of our method.
Algorithms
Fraction rejects (# std.)
Non-rejected test accuracy (# std.)
PI-Kmn (2005)
50.19 % (+ 20.98 %)
91.23 % (+ 10.12 %)
PI-Svm (2008)
50.19 % (+ 20.98 %)
74.40% (+ 19.77 %)
Ipal (2015)
50.19% %
(1: 20.98 %)
83.52% (1: 16.08 %)
PI-Ecoc (2017)
50.19 (1: 20.98 %)
73.92% (1: 17.11 %)
Proden (2020)
50.19% % (#: 20.98 %)
94.03% % (#: 8.23 %)
Cc (2020)
50.19% (+: 20.98 %)
90.13% (+ 17.89 %)
Valen (2021)
50.19 % (+ 20.98 %)
86.81 % (+ 13.01% %)
Pop (2023)
50.19% % (+ 20.98 %)
94.02% % (+ 8.20%)
CroSel (2024)
50.19 % (+ 20.98 %)
89.71 % (+ 18.67
Dst-Pll (ours)
50.15 % (+ 21.16 %)
95.49 % (+ 7.01 %)
11

---

Published in Transactions on Machine Learning Research (01/2025)
Our method
PL-SVM
PL-Ecoc PRODEN
Cc
POP
PL-KNN
IPAL
VALEN
CROSEL
Non-Rejected Test Acc.
1.00
1.00
1.0
0.95
0.98
0.9
0.90
0.96
0.8
0.85
0.94
0.7
0.80
0.92
0.6
0.75
0.90
0.5
0.00
0.25 0.50 0.75
1.00
0.00
0.25
0.50 0.75 1.00
0.00
0.25
0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
(a) Ecoli (inst.-dep. noise)
(b) KMNIST (inst.-dep. noise)
(c) msre-v2
Figure 1: Trade-off between the fraction of rejected predictions and the accuracy of non-rejected predictions for three experiments: Ecoli with instanco-dependent noise, KMNIST with instance-dependen noise, and the real-world dataset msrc-v2. We show the trade-off curves for varying confidence (0 to 1) and An (-1 to 1) thresholds. We highlight, the points corresponding to a threshold of Am = 0 for our method, a confidence threshold of 90 % for methods with a probability output, and a threshold of 50 % all votes for PL-KNN. We refer to Appendix D.4 for all reject tradc-off curves across all experimental settings.
a 0.05 are emphasized. When misclassification is costly (A S 0.2), our method provides the significantly best trade-off compared to our competitors. In contrast, when rejecting predictions is costly (A > 0.2), the methods in Table 1 are to be preferred.
Table shows the non-rejected test accuracy of all methods across all experimental settings for a fixed fraction of rejects. To obtain the results, we tune the confidence thresholds 0 E [0 such that each competitor rejects a similar number of instances as our proposed approach. Our approach uses the threshold A > O, for which we have proved several desirable guarantees in Theorem 4.2 Our method achieves superior test-set accuracy on the non-rejected predictions.
Figure I shows the reject trade-off for varying confidence (0 to 1) and An (-1 to 1) thresholds the ccoli and KMNIST datasets with instance-dopendent noise as well as on the msrc-v2 real-world dataset. The x-axes show the fractions of predictions that are rejected. The y-axes show the accuracies of predictions that are not rejected. The plots show (fraction of rejects, non-rejected test-set sccuracy)-pains corresponding to different settings of the thresholds. Most methods have monotonic growth: The more predictions are rejected, the more accurate are non-rejectod predictions. Also, it desirable be close the top-left corner of the plots as one wants achieve high accuracy while rejecting as few predictions possible. Note that the point (1, 1) cannot be observed as the test-set accuracy is undefined if all predictions rejected.
Given a desired rejection rate i(Tg), Figure also allows for numerically finding the appropriate value of À that minimizes (Ty). Varying À in ①1 while having efr(T,) (Tg) and f(T,) fixed, yields a straight line in Figure I showing all possible trade-offs.
Our method provides the significantly best trade-offs compared to competitors (Table 2 These results empirically demonstrate our method's superiority in reliably rejecting unsure predictions.
6 Conclusions
When misclassification is costly, reject options provide principled way of alleviating the consoquences of incorrect predictions. In this work, we have presented a novel nearest-neighbor-beee partial-label learning algorithm that can reject predictions if uncertain. We have demonstrated the desirable properties of the proposed reject option both from thcoretical (Theorem 4.2) and practical perspective. Our wide range of experiments showed the effectiveness of our classification rule and reject option on supervised datasets with added artificial noise and partially labeled real-world datasets.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Acknowledgments
We thank the anonymous reviewers for their constructive comments. This work was supported by the German Research Foundation (DFG) Research Training Group GRK 2153: Energy Status Dala Informaties Methods for its Collection, Analysis and Exploitation and by the KiKIT (The Pilot Program for Core-Informaties the KIT) of the Helmholtz Association.
References
Joaquín Abellán, George J. Klir, and Serafin Moral. Disaggregated total uncertainty mcasure for credal sets. International Journal of General Systems, 35(1):29 44, 2006.
Ethem Alpaydin and Fevzi Alimoglu. Pen-based recognition of handwritten digits. UCI Machine- Repository, 1998.
Shuang Ao, Stefan Rueger, and Advaith Siddharthan. Two sides of miscalibration: Identifying over and under-confidence prediction for network calibration. In Uncertainty in Artificial Intelligence, volume 216, pp. 77-87, 2023.
Kevin Bache and Moshe Lichman. UCI Machine-Learning Repository, 2013. URL htttss//arccivve uci.edu/
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. Transactions Pattern Analysis and Machine Intelligence, 35(8):1798-15) 2013.
James Bridge, Sean Holden, and Lawrence Paulson. First-order theorem proving. UCT Machine-Learning Repository, 2013.
Forrest Briggs, Xiaoli Z. Fern, and Raviv Raich. Rank-loss support instance machines for MIML instance annotation. In Conference on Knowledge Discovery and Data Mining, pp. 534-542, 2012.
Massimo Buscema and Stefano Terzi. Semeion handwritten digit. UCI Mlachine-eaaringg Repository, 2008 https://doi.orr/I1 240.0224322/S5SC5
Tom Bylander. Learning linear threshold functions in the presence of classification noise. In Conference on Computational Learning Theory, pp. 340 347, 1994.
Vivien Cabannes, Alessandro Rudi, and Francis R. Bach. Structured prediction with partial labelling through the infimum loss. In International Conference on Machine Learning, volume 119, pp. 1239, 2020.
Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, and Masashi Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In Advances in Newral Information Processing Systems, 2022.
Nontawat Charocnphaklce, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning, volume 139, pp. 1507-1517, 2021.
Xin Cheng, Yuzhou Cao, Haobo Wang, Hongxin Wei, Bo An, and Lei Feng. Regression with cost-based rejection. In Advances in Neural Information Processing Systems, 2023.
Tarin Clanuwal, Mikel Bober-Trizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamolo, and David Ha. Deep learning for classical japanese literature. Technical report, 2018.
Timothée Cour, Benjamin Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12:1501-1536, 2011.
Fabio Cuzzolin. The Geometry of Uncertainty The Geometry of Imprecise Probabilities. Springer, 2021.
Arthur P. Dempster. Upper and lower probabilities induced by multivalued mapping. The Annals of Statistic, 1967.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Thierry Denoeux. A k-nearest neighbor classification rule based on Dempster-Shafer theory. Transactions Systems, Man, and Cybernelics, 25(5):804-8)) 813, 1995.
Thierry Denoeux. Decision-making with belief functions: A review. International Journal for Approzimate Reasoning, 109:87 109;87-110, 2019.
Lue Devroye, László Györfi, and Gábor Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1996.
Helen W. Dodson and Emma R. Hedeman. Solar flare. /ttp///o.rrg11.24333///////
UCI Machine-Loarning Repository, 1989.
Robert Duin. Multiple features. UCT Machine-Learning Repository, 2002. https///b...og///..443////////// CCCC00
Jinfu Fan, Linqing Huang, Chaoyu Gong, Yang You, Min Gan, and Zhongjie Wang. KMT-PLI: K-means cross-attention transformer for partial label learning. Transactions Neural Networks and Learning Systems, 2024.
Lei Feng and Bo An. Partial label learning with self-guided retraining. In AAAI Conference on Artificial Intelligence, pp. 1542-3549) 2019.
Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Advances in Newral Information Processing Systems, 2020.
Ronald L. Graham, Donald E. Kmuth, and Oren Patashnik. Concrete Mathematics: A Foundation for Computer Science. Addison Addison-Wesley, 1994.
Yves Grandvalet. Logistic regression for partial labels. In Information Processing and Management of Uncertainty in Knouledge-bused Systems, 2002.
Matthieu Guillaumin, Jakob Verbeek, and Cordelia Schmid. Multiple instance metric learning from auto- matically labeled bags of faces. In European Conference on Computer Vision, volume 6311, pp. 634-647, 2010.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. International Conference on Machine Learning, volume 70, pp. 1321-1330, 2017.
Mohamed Farouk Abdel Hady and Friedhelm Schwenker. Semi-supervised learning. In Handbook on Neural Information Processing, volume 49, pp. 215-239. Springer, 2013.
Shuo Hc, Lei Feng, Fongmao Lv, Wen Li, and Guown Yang. Partial label lcarning with semantic label representations. In Conference on Knowledge Discovery and Data Mining, pp. 545 553, 2022.
Martin E. Hellman. The nearest neighbor classification rule with reject option. Transactions on Systems Science and Cybernetics, 6(3):179-18) 115799911111111 1970.
Stephen C. Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering and System Safety, 54(2-3))217-223, 1996.
Paul Horton and Kenta Nakai. probabilistic classification system for predicting the cellular localization sites of proteins. In International Conference on Intelligent Systems for Molecular Biology, pp. 109 115, 1996.
Constantin Hubmann, Marvin Becker, Daniel Althoff, David Lenz, and Christoph Stiller. Decision making for autonomous driving considering interaction and uncertain prediction of surrounding vehicles. In Intelligent Vehicles Symposium, pp. 1671-1678, 2017.
Mark J. Huiskes and Michael S. Lew. The MIR flickr retrieval evaluation. In International Conference on Multimedia Information Retrieval, pp. 39 43, 2008.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Eyke Hilllemmeer and Jürgen Beringer. Learning from ambiguously labeled examples. Intelligent Data Analysis, 10(5):168-1)) 179, 2005.
Takashi Ishida, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. Complementary-label learning for arbitrary losses and models. In International Conference on Machine Learning, volume 97, pp. 2971-2980, 2019.
Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In Advances in Neural Information Processing Systems, pp- 897-904, 2002.
Alex Kendall and Yarin Gal. What uncertainties do need in Bayesian deep learning for computer vision? In Aduances in Newrll Information Processing Systems, pp. 5574-5584, 2017.
Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL http://www. cs. www.:utoottoccaaoooo wttppptt..////tttt///eonneeeeeee
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Antonis Lambrou, Harris Papadoopulos, and Alex Gammerman. Reliable confidence measures for medical diagnosis with evolutionary algorithms. Transactions on Information Technology in Biomedicine, 15(1): 93 99, 2011.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-233) 2324, 1998.
Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. The MNIST database of handwritten digits. http:// httt://yyan..eee http://yyam.lecum.com www.ll.cc./.ee.e........e. hte////ddd/iissse /mnist, 1999.
Li-Ping Liu and Thomas G. Dietterich. A conditional multinomial mixture model for superset label learning.
In Advances in Neural Information Processing Systems, pp. 557-565, 2012.
Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In International Conference on Machine Learning, volume 119, pp. 6500 6510, 2020.
Eberhard Mandler and Jürgen Schümann. Combining the classification results of independent classifiers based on the Dempster-Shafer theory of evidence. Machine Intelligence and Pattern Recognition, 7:381-393, 1988.
Anqi Mao, Mchryar Mohri, and Yutao Zhong. Predictor-rejoctor multi-class abstention: Thcoretical analysis and algorithms. In International Conference on Algorithmic Learning Theory, volume 237, pp. 822 867, 2024.
Rhiannon Michelmore, Matthew Wicker, Luca Laurenti, Luca Cardelli, Yarin Gal, and Marta Kwiatkowska. Uncertainty quantification with statistical guarantees in end-to-end autonomous driving control. In International Conference on Roboties and Aulomation, pp. 7344-73500 2020.
Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David A. Sontag. Who should predict? Exact algorithms for learning to defer to humans. International Conference on Artificial Intelligence and Statistics, volume 206, pp. 10520 10545, 2023.
Mahdi P. Nacini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In AAAT Conference Artificial Intelligence, pp- 2901-2907, 2015.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, Neha Gupta, and Sanjiv Kumar. Learning to reject meets long-tail learning. In International Conference on Learning Representations, 2024.
Nam Nguyen and Rich Caruana. Classification with partial labels. In Conference on Knowledge Discovery and Data Mining, pp. 551 559, 2008.
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. In Advances in Neural Information Processing Systems, pp. 2582-25992 2019.
Peng Ni, Suyun Zhao, Zhi-Gang Dai, Hong Chen, and Cui-Ping Li. Partial label learning via conditional- label-aware disambiguation. Journal of Computer Science and Technology, 36(3):590-60) 2021.
Stephen M. Omohundro. Five balliree construction algorithms. Berkeley, 1989.
Narathip Reamaroon, Michael W. Sjoding, Kaiwen Lin, Theodore J. Iwashyna, and Kayvan Najarian. Accounting for label uncertainty in machine learning for detection of acute respiratory distress syndrome. Journal of Biomedical and Health Informatics, (3(1):477441,, 2019.
Yusuf Sale, Michele Caprio, and Eyke Hüllermeier. Is the volume of a credal set good mcasure for epistemic uncertainly? In Uncerlainly in Artificial Intelligence, volume 216, pp. 1795 1795-1804, 2023.
Robin Senge, Stefan Bösner, Krzysztof Dembezynski, Jörg Haasenritter, Oliver Hirsch, Norbert Donner- Banzhoff, and Eyke Hillermeier. Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255:16-29, 29, 2014.
Murat Sensoy, Lance M. Kaplan, and Melih Kandemir. Evidential deep learning quantify classification uncertainty. In Advances in Neural Information Processing Systems, pp. 3193, 2018.
Sina. Shafaei, Stefan Kugele, Mohd Hafeez Osman, and Alois C. Knoll. Uncertainty in machine learning: A safety perspective on autonomous driving. In Computer Safety, Reliability, and Security, volume 11094, pp. 458-464, 2018.
Glenn Shafer. The combination of evidence. Intelligent Systems, 1(3):155-179) 1986.
Shai Shalev-Shwartz, Yoram Singer, and Nathan Srebro. Pegasos: Primal Estimated sub- GrAdient SOlver for SVM. In International Conference on Machine Learning, volume 227, pp. 807-814, 2007.
Ashwin Srinivasan. Statlog (landsat satellite). hupps//doi.oor//1..22 11111/10224442/11 /C55887.
UCI Machine-Lcarning Repository, 1993.
Mahdi Tabassian, Reza Ghaderi, and Reza Ebrahimpour. Combining complementary information sources the Compsteer-hafer framework for solving classification problems with imperfect labels. Knowledge Based Systems, 27:92-102, 2012.
Cai-Zhi Tang and Min-Ling Zhang. Confidence-rated discriminative partial label learning. In AAAI Conference on Artificial Intelligence, pp.
2017.
Shiyu Tian, Hongxin Wei, Yiqun Wang, and Lei Feng. CroSel: Cross selection of confident pscudo labels for partial-label learning. In Conference on Compuler Vision and Pattern Recognition, 2024.
Zheng Tong, Philippe Xu, and Thierry Denoeux. An evidential classifier based on Dempster-Shafer theory and deep learning. Neurocomputing. 450:275-221 293, 2021.
Kush R. Varshney and Homa Alemzadeh. On the safety of machine learning: sciences, and data products. Big Data, 5(3):246-255, 2017.
systems, decision
Deng- Bao Wang, Li Li, and Min-Ling Zhang. Adaptive graph guided disambiguation for partial label learning. In International Conference on Knowledge Discovery and Data Mining, pp. 83 91, 2019.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO: Contrastive label disambiguation for partial label learning. In International Conference on Learning Representations, 2022.
Paul Werbos. Beyond regression: New tools for prediction and analysis in the behanioral sciences. PhD thesis, Harvard University, 1974.
Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainly in Arlificial Intelligence, volume 216, pp. 2282 22922 2023.
Xuan Wu and Min-Ling Zhang. Towards enabling binary decomposition for partial label learning. In International Joint Conference on Artificial Intelligence, pp. 2868-2874, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. Technical report, 2018.
077747
Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In AAAI Conference on Artificial Intelligence, pp. 5557-5564, 2019.
Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In Advances in Newral Information Processing Systems, pp. 27119 27130, 2021.
Ning Xu, Biao Liu, Jiaqi Lv, Congyu Qiao, and Xin Geng. Progressive purification for instance-dependent partial label learning. In International Conference on Machine Learning, volume 202, pp. 38551 -38665, 2023.
Wenda Xu, Jia Pan, Junqing Wei, and John M. Dolan. Motion planning under uncertainty for on-road autonomous driving. In International Conference Robotics and Automation, pp. 2507-2512, 2014.
Ronald R. Yager. On the Dempster-Shaiee Shafer franework and new combination rules. Information Sciences, 41 (2):93-133) 137, 1987a.
Ronald R. Yager. Quasi-associative operations in the combination of evidence. Kybernetes, 16(1):37-4) 1987b.
Fan Yang, Hua-zhen Wang, Hong Mi, Cheng-de Lin, and Wei-wen Cai. Using random forest for reliable classification and cost-sensitive learning for medical diagnosis. BMC Bioinformatics, 10(1), 2009.
Fei Yu and Min-Ling Zhang. Maximun margin partial label learning. In Asian Conference on Machine Learning, volume 106, pp. 573 193-593, 2017.
Lotfi A. Zadeh. Book review: mathematical theory of evidence. Al Magazine, 5(3):81 83, 1984.
Ahmed Zaoui, Christophe Denis, and Mohamed Hebiri. Regression with reject option and application to KNN. In Advances in Neural Information Processing Systems, 2020.
Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance- based approach. In International Joint Conference Artificial Intelligence, pp. 4048-4055 2015.
Min-Ling Zhang, Bin- Min-Bin Zhou, and Xu- Ying Liu. Partial label learning via feature-aware disambiguation. In International Conference on Knowledge Discovery and Data Mining, pp. 1335-1344, 2016.
Min- Linn Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-fre partial label learning. Transactions on Knowledge and Data Engineering, 29(1)):2155 2167, 2017.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature- dependent label noise: A progressive approach. In International Conference Learning Representations, 2021.
17

---

Published in Transactions on Machine Learning Research (01/2025)
A Proofs
This section collects our proofs. Appendix A.1 contains the proof of Theorem
4.2
The proof of Lemma 4.5
in Appendix A.2 We prove our main statement, Theorem 4.6 in Appendix A.3
A.1 Proof of Theorem 4.2
Part (i). Given an instance & E A' with its candidate set 3 S y and its associated prediction g(t) as described in Algorithm| [
we assume that g(ø) has been picked randomly from arg max miA) (Algorithm | Line 11, second casc). so maxyes in m(vl) must be zero because we are not in the first case in Linc 11. Therefore,
@
(tv))
m(lv)) = 0 for all y E V. Substitution in Am yields
Aa = ((v)) max pln ((v/)) = =0 - max Plm((t)) S, y'ERL())
independent of the choice of E V. Therefore, An S.
Part (ii). Given an instance E X' with its candidate set S y and its associated prediction g as described in Algorithm 1 we assumee that Am > 0. By the contraposition of part (i), g(ā) y
with ý waggex mm From (5) and An 0, it follows that beln, ((7)) > max pla ((y)), Cm (V, 2"), it holds that which is equivalent to belm (173) > plg ((y) for all y 3 (0). For all P E Cm
bela( A) < P(A) S (4) for all A S by 33. Therefore, P((v)) S (l,(())) < bela ((0)) S P(()) for all \ (0)-
E &
Part (iii). (+): Given an instance 5 E X' with its candidate set C y and its associated prediction gi) as described in Algorithm 1 we assume that m({g(())) > 1/2. Because maxges m(())) O, we are in the first case in Line 11 of Algorithm D Therefore, g()) 9 with 1 arg maxyes m((y))- As Eacy m(4) and m((o)) > 1/2, it holds that ΣAcx,4*0)) m(A) 1/2. Then, for all y E 8 with y & 0, #0 m(A) 1/2. Hence, plm((1)) belm ((9)) for all y E 3 with y + ÿ. Therefore, An = bbela (101) - max (fv}) > O.
Pla (lv))
(4)): In the following, we provide a counter-example. Let 8= (1,2,3) and in be defined by ((A) = 0.4 if A [1}, mAA = 0.3 if A {1. or A = {1,3), else m(A) = 0. Then, y our prediction since it has the highest probability mass. The prodiction is not rejected because Am 0.4 - 0.3 0.1 > 0. However, m((v)) < 1/2.
A.2 Proof of Lemma 4.5
Part (i). The expected probability mass of ÿ is given by 7 The non-negativity of the binomial coefficients implies that it is sufficient to show that
h t: -1- (I
l:
h
N
Σ
(-1)bt]
k (k--)) (
(k -i-
1-2
(
k kk
(6)
ja
i b
Jr -
where & E N, h, i,ja E No with 0 S < k h and 0 S ja < k- h with a E [l 2], and that the inequality is strict in at least one case.
The proof of 6 proceeds by induction over k. Note that the choices of and ja are interchangeable (see Appendix B Hence, w.l.o.g, we assume for the remainder of this proof that < ja for all a E [L 2]. Further, we set K' h N.
Proposition. For k' E N, it holds that
1''
(K' - b (K'
K 1o
D)
b41
ja
6)
i
b
-o-
with 0 ja < for all [1 1-2] 2].
18

---

Published in Transactions on Machine Learning Research (01/2025)
Base case. For k' = 1,,1 it holds that h = ja =0 (a [1 [[-)). 2]). Then, all binomial coefficients the Lh..s. are 1. The sum on the r.h.s. is empty and thus O. Hence, l.h.s. > r.h.s. (1 >> 0), which also renders 7 strictly positive.
Induction step. Let k' = W/ + 1. The l.h.s. yields
k' + i
++1
1)) G.
Ja
#+1
W +1-1aa Ja
a=1
W
L-2
K' +1
(6)
6) K (1-
II
1-2
K'' 6) Ge b)
W
+
1 +1 Ja
6=1
b=1
c=1
6+1
1+1
(K)
W+1
W - b)
6)
M+1-1
++1- 3c
Je b)
+1
R
W ++1-- (K' 1-2
W +1
K + 1 -
h 6)
641
#+1+ - b
b
W +1 -
*+
- b 1/' + 1 - jc Je b)
(()+)
(1
(1) (t)
K' 1'+1
k 11
W +1-b
1-2
K +1+101 b)
1
W w+11-b
i-b
jc b
+1+11
K +1+11b
1-2
(W' +1- b) je - b
(-1) b+1
(bb
b
b c=1
where (i) holds as
k' (+1))
(K' + 1)!
K' +1
k!!
k ++ 1 k' -+1--i i
K"
al(k' +1-))
-+1
the proposition yields (it), in (iit) we rearrange the sum and collect both products, and (iv) expands the fractions. In (v), we rearrange the factor within the product and apply the (i')-s, which hold by similar calculations as (i), and Lemma B.1 with k = k + 1 implies (vi). These derivations and the observation that the base case guarantees the inequality to be strict conclude the proof.
Part (ii). By comparing the closed-form expressions (with m i 1) of and Te given in and respectively, we obtain
Ep(m((i))
x= > Ep femm@tt (1ūe)) X = 3]
k-h-m
k-h-m
21-2P2 P2
21-1 P3
2/-1
21 21 P2
2
P3
2
>1
= 2 >1
where (i) holds as Assumption 4.4 guarantees that 2 P3 > 0. The last statement is satisfied for / 2 3, which shows that the dominance assumption w.r.t. the label distribution propagates through the computation of belief when using Yager's rule, We note that the result is independent of pt as a candidate set can not distinguish the belief in ğ and de when these co-occur.
A.3 Proof of Theorem 4.6
With Assumption 4.4 Theorem 4.6 is a consequence of Lemma 4.5 We detail this in the following.
Part (i). Let i X be fixed instance with unknown true label g. Our algorithm is trained with n samples
((x;, s L.d. Ixss that satisfy the label distribution in Assumption 4.4 Especially, the dominance of the
19

---

Published in Transactions on Machine Learning Research (01/2025)
true label within ë's neighborhood of instances 2; holds. Since the expected probability mass of ý is positive (Lemma 4.5; part (i)), that is, Ep [p (ū)) x -i > O, are, in expectation, in the first case of our two-case decision rule (Algorithm ! Line 11). Furthermore, label receives, in expectation, the maximum probability mass among all labels because the dominance label in the neighborhood propagates to the label masses by Lemma 4.5 (part (ii)), that is, [p(m((()) = ã > Ep(m(([ee)
ã). Therefore, in expectation, must hold that, given an instance ž, we consistently predict the true label y, that is,
E
((a))))) - Pxs (n y.
Part (it). Given n samples ((2,, s)))==1 ltd. Pxs, the classifier In trained by Algorithm from the n samples, and the minimum loss L(g(r), 8) = minyes (o(x))t) (L.v et al. 2020 Feng et al. 2020 the expected empirical risk is
Ep Rp(R(gn)) = Ep
((gn(z)),ss)
Bp (xi),, _
=
which is zero by part (1) of the statement as, in expectation, we consistently predict the true label yi for any fixed label Tz.
By the law of large numbers, we have
1 Σ (z4),
lim R(gn)= = lim
almost surely,
n n1000
n n
because [p]RR(g]] 0 stated above. As R(g") 2 0, we conclude
0 < lim (R(gx) - R(g")) < lim R(gn) =0 almost surely,
n-+20
n-220
which establishes (Mi).
B Auxiliary Results
This section lists results underpinning the findings in Appendix A Scction
B.1 introduces a closcd-form expression of the expected probability mass. Section B.2 presents Lemma which needed the proof of Lemma 4.5
B.1 Expected Probability Mass
This section, first, presents a closed-form expression for the expected probability mass of ÿ and Øe- Second, we compare the computation with a simulation to numerically validate our closed-form expression and the results Appendix A
Closed-form expression. Let & E X be the instance of interest with hidden true label ÿ y and k associated neighbors (wi, s:) E (T). We recall that our label space y has / 2 classes. Then, the expected
20

---

Published in Transactions on Machine Learning Research (01/2025)
probability mass of ū given instance 2 is
B(XY m((()) X
(xx.s)><(((()))
x
= a]
(1))
B(XY,S)
Σ
II
mms(A
==1
(1)
N
1- K-
(186)
Σ
II (4,)) j=1
(II A x =xz))
h=0
0=
11=0
12=0
$1-2=0
A Al cy:
BI...,B.CII
with |Bi|=h,] Ba
(v)
(vi)
Ba+2|= for 1-1-2]
0 Ar for rE lkj
A, => for re Bii
E.C
Bzi
2e+2E Arg for rg E Bgt2
and q [1-2]
(in)
k-1
1 1--h-1k h-
h-h-1
カールー
E
2
2
P1
21-2P2
h=0
1=0
=2=0
j1-2=0
A A cy:
BI..., B.C (es)
(18)
with [B, [
1 al for 1-(-2]]
ØE A, re k];
A, =y=y y for re Bi
3.E
Bai
E Ar, for ry E Rg+s
and q [1-2]
(vit)
ΣΣ
Σ
Σ
h-
k
1-2
k h
=
h h
i
Ja
h=0
01
Ji 2
a=l
(wis)
k
min(i.ji 52 St 2)
(-1)b+1
k h)
1k (h:
h b)
1-2
h: h
t: - h
b
i i-
je b
b=1
c=1
(oit)
x 2k 21-2 2 p1) ) )
(2222222))
h-h-s
(7)
In the following, we show that transformations (i) (vvitt hold.
In (i), we apply (4. For any y E V/ it holds that m((v)) = a((v)) as (n) ل 0 and {4} * V.
In (ii), we insert that is, how all m as defined by Algorithm [ are combined into q using
Yager's rule
In (iii), we then compute the expected value by taking into account all possible combinations of sets Ai,..,, A C y producing the intersection {ū)
Term (ir) enumerates all cases where n Ar = (ū). To produce this intersection, label ÿ needs to be contained in all sets Ap. At most k -- sets can be the full label space, that is, sets satisfy A, y. The label De with which ÿ is most often confused has to be missing in least one set A,, that is, sets A, satisfy de A,. All remaining labels .s..... 0 y also have be missing in least. set. A, each, that is, ja sets A, satisfy Dat2 Ar, respectively.
21

---

Published in Transactions on Machine Learning Research (01/2025)
Term (t) calculates the value that given instantiation of all Ai...., Ag has. Because we only sum over focal elements of the my-8 (j E [k]) by the construction of the sum in (v), term (vi) simplifies to 1/2*. Recall that all focal sets have a value of 1/2 (Algorithm Line 6).
Expression (vi) computes the probability that a given instantiation of all A1,..., As has. The variables h, i, ..,, ft-2 directly allow to quantify the probabilities in a way to apply Assumption 4.4 which allows to simplify the expression.
As the summands within (vit) do not depend on the AI.... A anymore, we can simplify by counting their possible instantiations. We first count the possible combinations of sets where we pick to intersect h-times with the full label space and (k - h-)times with the neighbors' candidate sets of which i contain label De and ja contain the remaining labels for all a E [1 - 2]. Second, we need to subtract all combinations that produce the whole label space within at least one candidate set as the probability of s' 3 is zero per Assumption 4.4 We do this using an inclusion-exclluionn strategy. We have k - h sets with which we intersect and that are not the full label space. Out of those l: - h scts, at least one set ought to contain all possible labels leaving ("g-)²) and (5/--)) possibilities for the other sets. This overcounts combinations though as multiple sets can contain all labels. We employ the inclusion-exclusie strategy to remove combinations that produce the same allocations of labels to sets.
Similar to we also express the expected mass of the most frequently co-occurring label Te as
E(x. ((((e)) X = =] = E(x,rs)) (g((et)) x = #]
k
= Bxx
Σ
II
mm) IX =]
Ac
j=1
(1=(1e)
k- 1-1k-
1k
k-h-1
k 1)
(k=
h - 1
k h ja
Σ
Σ
k k- h
t a=1
0=0
-=
k
min(t. Ja dt 2)
(-1)ut2 (k-h)
hi) (*
kk- h b)
1-2
k h - b)
Σ
h
1-b b
Je b
b=1
h-
1 x 2 21-2
1
1
1 P1
21-1- P3
(8)
To produce the interscction fūe} in Yager's rule, all ks sets with which we intersect need to contain ğe- The variable i denotes how many out of k k- h sets contain the true label g. Therefore, ÿ and Te co-occur in a set in cases, with which the probability (= is associated. In k - h t sets, the true label is not contained, with which the probability (2 (g Ps)* h-t is associated according to Assumption 4.4
Simulation. To simulate the expected probability masses, we randomly draw l candidate sets 8; according to the label distribution in Assumption 1.4 apply Algorithm [ and report the belief of ğ and Øe. We repeat this 100 000 times and average the results. For 1= 3, ğ 1, and ÿe 2, we obtain the label distribution
P(S- {1. 2) Y = 1|x = zi) = P1. = (1,3), Y 1 X
(1, Y0,, =1 1/
X- x = X|
||x = xx) 2 P2,
P(S (2) Y = 2 X x,) = P(S (3) Y = 3 x 22) P3 P(S = [2,3). Y = 2 X = x;) = P(S = (2.3). Y = 3 x = x2 P3:
regarding neighboring instance 2; with E [k]. All other probabilities that are not listed are zero Naturally,
Evey Evev P(S= s = s X .x) = p1 2 + 2 2 1.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 2 compares the computation and simulation of the expected probability masses. The values of the computation and simulation are indiscernible. Note that the expected probability mass of any single-item set converges towards zero as k increases. The more sets one intersects with in ( the more likely it is to produce the empty set. As shown in Lemma 4.5 (part (it)), the expected probability mass of ÿ is greater than the mass of De, independently from the number of neighbors lt E N.
B.2 Auxiliary Lemma to the Proof of Lemma 4.5
This section presents an auxiliary inequality, which is used in the proof of Lemma 1.5
Lemma B.1. Let I E N with I 1 3 denote the number of class labels and k E N the number of neighbors. Let further i, E No with 0 S S Ja < l for all a E [1-22) -2]. Then, it holds that
(-1))+1
k
1-2
k
k b
2
Jo b) 2 Σ
(-1)at
6+1
k
k b
k b) Jo b)
k-b)
b bb
6
b
6
- b
Proof. We first transform the statement as
(-1) b41
k
2-2
k
b)
b)
Σ
6+1
(k)
k b
II
k k)bb)
b b-b
b
b
b
jo
b=1
1-2
k
k 2 o.
(k-6)
1
k
b
je b)
b=1
We then write the latter
6+1
h
1-2
k
k
b
b) je b)
1
h b
(1)
6+1
1-3
k
b
k
b)
1-2
h b
b))
b
je
(8)
(-1)641
k
b
k
k b)
b= b
k
b
j. b)
(65E)
(-1)641
k
k b)) ja b)
b
(00)
6+1
Σ
kh
(
k b je b)
k 51
i
(k-b-2) 2
k b 1 je b
k
11 1
(11)
Σ (-1)) '
Σ
4 b )-1
(-1) -b (
(k (k-i
J1 -1 (k-i-1) 1
II
1-2
k b
k - j1
je b
0
1-2 k 6--b-1)
(vé))
i
k
h
k
2 0,
カー σイ i
51
-1
k- b
Ja - b-
00=0
23

---

Published in Transactions on Machine Learning Research (01/2025)
which shows the statement to be demonstrated as the last term is sum of products whose factors are all non-negative. Recall that 0 ك i S ja < k for all a E [ - 2]. In the following, we show that all transformations
are sound.
Using Theorem
C.2
(1) holds as
1-2
k
1-2
1-3)
1-3
k
1-1
k-
1-
k
bb
ht k- b
6)
k
1)
+
k --b b
k
k
1-3
k
1-1
b 1)
+
+ +1
b
k ---b. b)
k - -1
1-3
k
b
1-3
k
k
= k
k b)
In (ii), we extract the first factor of the product such that it starts with c 2. Subsequently, we transform the binomial coefficients (tii) as
b
k
k
b k - b
k 5
b
i
b
J1 - b)
C.4
b
k b)
kc b
51 b)
C 1
b
i!
(I: b)!
k b
bl
(i
6)1 (f1 6)1(( (k-
51 (i)
(v86))
i
(1-1)!
(k: b - 1)!
k
ke ji
(b 1)! - 6)1 (h
b)! (I: 51 11)!
C.1
i
(k)
( (b=1) (*: (k-b- -b-1) 1)
k J1 b
f1 - b
where (vilt) rearranges terms.
Then, in (iv), we move terms that are independent of outside of the summation.
In (v), we shift the summation to start at b b= 0.
Finally, by defining ji - b 1 E No, we express (vi)
i b (
شز (
-
( - 2 = (i-1)
+ ---jj in 11-1) 1
b
r
1)
(i h- k
(-1)"
b
(z2) (-1)- Σ (i-1)
1 (I1 -k) k)
b'
8 B
b+r'=h 1
C.5
(-1ح
1 + f1 - k - 1)
51 - 1)
C.3
(-1) ا-ط (-1)---
(k- - - j1 j1-1
= (-1)-b (k カーー i i-1 1)
j1 -
24

---

Published in Transactions on Machine Learning Research (01/2025)
0.4
Bp bel (m) ((1)) X
Calculation
0.3
Bp ((()) X =
Simulation
Belief
Bp ((Be X T Calculation
0.2
Bp bel ((bE X
Simulation
0.1
0.0
20
40
60
100
Number of neighbors &
Figure 2: Comparison of simulating and calculating the expected belief of the true label ÿ and of the most
frequently co-oceuriig label de for k E [100], I = 3, 0.4, P2 = 0.35, and = 0.25.
Thereby, (iz) holds all summands that are introduced by E 20+r'=j-11 ((x)) (5) are zero. The only non-zero summand for R' = b and r' r, that is, + r = J1 1. In general, (*) = 0 if S m < n for m E R and n No.
C
External Theorems
This section collects identities and theorems that use in the proofs of our statements.
Definition C.1. (Binomial coefficients; Graham et al. 1994 p. 154). When both arguments of a binomial coefficient are natural, one defines
n! k! (n k)!
for n, & E No with n 2 0. One extends this definition using falling factorials ym = 11 (r - i) for any and m E No. Binomial coefficients are then expressed as
m
m!!'
which allows for real arguments r.
Theorem C.2. (Binomial theorem; Graham et al. 1994 p. 162). Given z, y E R. and n No, the binomial theorem states
+ v)"
Theorem C.3. Graham et al. 1994 P. 164). As special case of the binomial theorem, one obtains
(-1)* (-1)
-(r+b=1
1 --1
k
for R and l E No.
25

---

Published in Transactions on Machine Learning Research (01/2025)
Theorem C.4.
Graham et al. 1994 p. 168). Given E R and m, E No with m 2 k, it holds that
r
m
r
k
=
k
m
k
Theorem C.5. (Chu- Vandermonde identity; Graham al. 1994 p. 169). Given s, R and n E No, the
Chu-Vander Vandermonde identily states
8
+
11
n
k
n k:
k=
Additional Experiments
This section contains additional experiments and details their hyperparameters. Section D.1 provides experiments justifying Assumption 4.4, Section D.2 lists the values of all relevant hyperparameters, the parameter sensitivity of our approach is discussed in Section D.3 and Section D.4 contains reject trade-off plots for all experimental settings considered.
D.1 Applicability of Assumption 4.4
Given a fixed instance ã, Assumption 4.4 describes the label distribution of the neighbors' candidate sets. Thereby, assume that the true label y dominates the neighborhood and that label je co-occurs most often with the true label J. Otherwise, we assume uniform noise among the remaining noise labels.
Figure 3 demonstrates that those assumptions are largely satisfied on the four real-world datasets (see Section 5.3 In most cases, the true label ÿ occurs most often in the neighborhood instance ž. There are one or two other labels with which ÿ is commonly confused, which we model by de. Apart from that, all remaining noise labels are distributed uniformly.
D.2 Hyperparameters
As mentioned Section 5.1 we consider ten commonly used PLL approaches. We choose their parameters as recommended by the respective authors.
PL-KNN (Hlllemeiier & Beringer 2005): For all MNIST datasets, we use k 10 neighbors recommended by the authors. For the MNIST datasets, variational auto-encoder
the hidden representation of a instance features and use h = 20. The variational auto-encoder has 768-dimensional inpul layer (flat MNIST input), 512-dimensional second layer, and 48-dimensional bottleneck layers for the mean and variance representations. The decoder uses a 48-dimensional first layer, a 512-dimensional second layer, and a 768-dimensional output layer with sigmoid activation. Otherwise, we use ReLU activations between all layers. Binary cross-entropy is used as a reconstruction loss. We choose the Adam W optimizer for training.
PI-SVM Nguyen & Caruana, 2008 : We use the PEGASOS optimizer SShleeyShwaard et al. 2007
and I.
IPAL Zhhng & Yu
2015;:
We use A = 10 neighbors, 0.95, and 100 iterations.
PL-Ecoc /Zhang et al. 2017 We use L [10 (l)] and 0.1 recommended.
2020 For a fair comparison, we use the same base models for all neural-network- based approaches. We a standard d-300-300-000-1 MLP (Werbos) 1974) for the non- MNN-MNIST datasets with ReLU activations, batch normalizations, and softmax output. For the MNIST datasets,
PRODEN (Lv al.
we use the LeNet-5 architecture (LeCun et al. 1998) We choose the Adam optimizer for training.
Cc (Feng et al. 2020 We use the same base models mentioned above for PRODEN.
26

---

Published in Transactions on Machine Learning Research (01/2025)
Trme label
label
10
10
Labels neighbors 6,
Labels in neighbors
Dataset bird-song.
(b) Dataset mår-ftickr.
0.1
40
The lnbel y
True 0
0.2
12
14
200
200
8
12
16
40
80
120
160
200
Labels in neighhors 81
Labels in neighbors
(c) Dataset msrc-u2.
(d) Dataset yahoo-news.
Figure 3: Counts of the class labels in the candidate sets 8; of instance i's 10-ncarest neighbors z,-
VALEN Xu et al.
2021 :
We use the same base models as mentioned above for PRODEN.
Pop (Xu et al. 2023)
We use the same base models as mentioned above for PRODEN. Also, we set ep = 0.001, cend = 0.04, and e. = 0.001. We abstain from using the data augmentations discussed in the paper for a fair comparison.
CROSEL Tian et al. 2024]: We use the same base models as mentioned above for PRODEN. We use 10 warm-up epochs using Cc and A = 2. We abstain from using the data augmentations discussed in the paper for a fair comparison.
DST-PLL (our proposed approach): Similar to PL-KNN and IPAL, we use k = 10 neighbors for the non-MNIST datasets. For the MNIST datasets, we use the hidden representation of a variational auto-encoder as instance features and use = 20. The architecture of the variational auto-encoder is the same as described above for PI,KNN.
We have implemented approaches PYTHON using the PYTORCH library. All experiments need two three days on machine with 48 cores and one NVIDIA GeForce RTX 3090.
D.3 Parameter Sensitivity
Figure shows the sensitivity of the number of neighbors regarding the test-set performance, the fraction of confident / non-rejected predictions, and the non-rejected prediction performance. The shaded areas indicate the standard deviation regarding the 5-fold cross-validation. As for default k-nearest neighbor classification, changes of k have relatively large impact. We show parameter sensitivity for each of the real-world datasets scparately. Naturally, different datasets have different optimal parameter settings. The configuration k 10, which is also recommended within PL.-KNN (Hüllermeier & Beringer 2005) and IPAL (Zhang & Yu 2015 provides a good trade-off between the number confident predictions and how accurate confident predictions are. Indood, this setting produces good number of confident predictions on most datasots (Figure 4 center plot). At the same time, produces good MCC performance of confident predictions on most datasets (Figure right plot).
27

---

Published in Transactions on Machine Learning Research (01/2025)
1.0
Frac. of confident pred.
1.0
MCC of confident pred.
1.0
Test-Set MCC
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Number of Neighbors
Number of Neighbors
Number of Neighbors
bird
flickr
lost
msre
yahoo
Figure 4: Sensitivity of regarding the test-set MCC score, the fraction, and the MCC score of confident non-rejected predictions.
When increasing k, our method's behavior on different datasets can be assembled into two groups. On the bird-song, mir-flickr, and yahoo-news datascts, incrcasing k past ten ncighbors also increases the amount of irrelevant labeling information from those neighbors. Therefore, our approach produces less confident. predictions. At the same time, the MCC score of confident predictions remains at roughly the same level. This because irrelevant labeling information from neighbors increases most at the same rate as k. In contrast, on the lost and msre-v2 datasets, the MCC score of confident predictions drops sharply at a certain point while the number of confident predictions decreases similarly. This is because irrelevant labeling information increases more rapidly than k: The decrease of confidence in predictions is slower than the increase of irrelevant candidate labels.
D.4 Reject Trade-off Curves
Figure (xxxiv) shows the reject trade-off for varying confidence (0 to 1) and An (-1 to 1) thresholds and augments Figure I by considering all datasets and noise generation strategies. The x-axes show the fractions of predictions that are rejected. The y-axes show the accuracies of predictions that are not rejected. The plots show (fraction of rejects, non-rejected test-set accuracy) -pairs corresponding to different settings of the thresholds. In most cases, our method provides a better trade-off between the number of rejected predictions and the accuracy of the non-rejected predictions. Table P summarizes all plots by showing the average empirical risks across all experimental settings and for different trade-off parameters A. We recall that our method provides the significantly best trade-offs for [0.0.2]
28

---

Published in Transactions on Machine Learning Research (01/2025)
Our method
PL-SVM
PL-Ecoc
- - CC
POP
PL-KKN
IPAL
PRODEN
VALEN
CROSEL
Non Rei-RRejeted Test Acc.
1.00
1.00
1.00
0.95
0.95
0.95
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.80
0.80
0.75
0.75
0.75
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
(i) ccoli (uniform noise)
iii cooli (class-dep. noise)
(ii) ccoli (inst.-dep. noise)
Our method
PL-SVM
PL-ECoc - Cc
POP
PNN-KNN
TPAL
PRODEN
VALEN
CROSEI
Von-Rejected Test Acc.
1.00
1.00
1.00
0.95
0.95
0.95
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.80
0.80
0.75
0.75
0.75
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
(iv) m.-feats (uniform noise)
(v) m m.-feats (class-dep. noise)
(vi) m.-feats (inst.-dep noise)
Our method
PL-SVM
PL-Ecoc
Cc
POP CROSEL
PL-KNN
IPAL
PRODEN
VALEN
Non- Rejectejcc Test Acc.
1.00
1.00
1.00
0.99
0.99
0.99
0.98
0.98
0.98
0.97
0.97
0.97
0.96
0.96
0.96
0.95
0.95
0.95
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
(vii) pen-digits (uniform noise)
(viii) digidsgits (class-dep. noise)
(ix) pen-digits (inst.-dep noise)
Our method
PL-SVM
PL-ECoc
Cc
POP CROSEL
PL-KNN
IPAL
PRODEN
VALEN
Non- Rejectejccddd Test Acc.
1.00
1.00
1.00
0.98
0.98
0.98
0.96
0.96
0.96
0.94
0.94
0.94
0.92
0.92
0.92
0.90
0.90
0.90
0.00
0.25
0.50 0.75 1.00 0.00
0.25
0.50 0.75 1.00
0.00
0.25 0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
xx) semeion (uniform noise)
(xi) semeion (class-dep. noise)
(xii) semeion (inst.-dep noise)
29

---

Published in Transactions on Machine Learning Research (01/2025)
Our method
PL-SVM
PL-Ecoc
- - CC
POP
PL-KKN
IPAL
PRODEN
VALEN
CROSEL
Non Rei-RRejeted Test Acc.
1.00
1.00
1.00
0.95
0.95
0.95
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.80
0.80
0.75
0.75
0.75
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
(xiii) solar-flare (uniform noise)
(xiv) soler-flare (class-dep. noise)
(xv) solar-flare (inst.-dep. noise)
Our method
PL-SVM
PL-ECoc - Cc
POP CROSEI
PN.-KNN
IPAL
PRODEN
VALEN
Non- Non-Rejected Test Acc.
1.00
1.00
1.00
0.95
0.95
0.95
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.80
0.80
0.75
0.75
0.75
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
(xvi) -landst (uniform noise)
(xvii) slad.-landsat (class-dep. noise)
(xvii) stat. aaa.-landsat (inst.-dep. noise)
Our method
PL-SvM
PL-Ecoc
CC
POP
PL-KNN
IPAL
PRODEN
VALEN
CROSEL
Non-Rejected Test Acc.
1.0
1.0
1.0
0.9
0.9
0.9
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50 0.75 1.00
Fraction of Rejects
Fraction Rejects
Fraction of Rejects
(xix) theorem (uniform noise)
(xx) theorem (class-dep. noise)
(xxi) theorem (inst.-dep. noise)
Our method
PL-SVM
PL-ECoc
- Cc
POP CROSEL
PL-KNN
IPAL
PRODEN
VALEN
Non-Rejee Rej-Reejcted Test Acc.
1.00
1.00
1.00
0.99
0.99
0.99
0.98
0.98
0.98
0.97
0.97
0.97
0.96
0.96
0.96
0.95
0.95
0.95
0.00
0.25
0.50 0.75 1.00
0.00
0.25
0.50 0.75 1.00
0.00
0.25
0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
(xxii) MNIST (uniform noise)
(xxili) MNIST (class-dep. noise)
(xxiv) MNIST (inst.-dep. noise)
30

---

Published in Transactions on Machine Learning Research (01/2025)
Our method
PL-SVM
PL-Ecoc PRODEN
Cc
POP
NN.-KNN
IPAL.
VALEN
CROSEI.
Non- Ra--RReected Test Ace.
1.00
1.00
1.00
0.98
0.98
0.98
0.96
0.96
0.96
0.94
0.94
0.94
0.92
0.92
0.92
0.90
0.90
0.90
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.50 0.75 1.00 Fraction of Rejects
0.25
Fraction of Rejects
Fraction of Rejects
(xxV) FMNIST (uniform noise)
(xxvi) FMNIST (class-dep. noise)
(xxvii) FMNIST (inst.-dep. noise)
Our method
PL-SVM
PL-ECoc
Cc
POP
PN.-KNN
IPAL
PRODEN
VALEN
CROSEI
Non-Rejected Test Ace.
1.00
1.00
1.00
0.98
0.98
0.98
0.96
0.96
0.96
0.94
0.94
0.94
0.92
0.92
0.92
0.90
0.90
0.90
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75 1.00
Fraction of Rejects
Fraction of Rejects
Fraction of Rejects
(xxviii) KMNIST (uniform noise)
(xxix) KMNIST (class-dep. noise)
(xxx) KMNIST (inst.-dep. noise)
Our method
PL-SVM
Pt-Ecoc
Cc
POP
PL-KNN
IPAL
PRODEN
VALEN
CROSEL
Non-Rejectee Test Ace.
1.00
1.0
0.95
0.90
0.8
0.85
0.7
0.80
0.6
0.75
0.5
0.00
0.25 0.50 0.75 1.00
0.00 0.25 0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
(xxxi) bird-song
(xxxii) mir-ftickr
Our method
Pt-SVMM
Pt.-Ecoc
- Cc
POP
PL-KNN
IPAL
PRODEN
VALEN
CROSEL
Non-Rejected Test Acc.
1.0
1.0
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.00
0.25 0.50 0.75 1.00
0.00
0.25 0.50 0.75 1.00
Fraction of Rejects
Fraction of Rejects
(xxxiii) yahoo-news
(xxxiv) msrc-v2
Figure 5: Trado-off between the fraction of rejected predictions and the accuracy of non-rejected predictions.
31


Reviews:
[{"review_id": "CBXffEsNcz", "reviewer": "Reviewer_xnXJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper addresses an important problem and appears to be the first to introduce a \"reject option\" under the PLL setting. \n- The use of DFT to aggregate candidate label sets into a predicted label and confidence score is novel and very interesting.\n- Experiments suggest that the proposed approach provide better trade-offs between accuracy and rejection rates compared to the baselines.\n- The authors provide theoretical consistency results for their method under some specific distributional assumptions.\n\nWeaknesses:\n- Some of the design choices in applying DFT to PLL have not been aptly justified. As a reader new to DFT, many elements of the proposed algorithm seemed initially to be arbitrarily chosen. Only upon reading background material on DFT, was I able to convince myself of the rationale behind the design choices made (see changes requested).\n\n- The paper fails to provide sufficient intuition for why the confidence measure or the label prediction rule proposed is suitable for PLL. Yes, the proposed algorithm is derived using principles from DFT, but the writing could provide some concrete intuition for why one would expect the method to work in practice (see changes requested).\n\n- Finally, the paper could do a better job of providing background on DFT. Currently, Section 3.2 is short and packed with notations and jargons, making it hard for an unfamiliar reader to grasp the basics of DFT.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the problem of Partial Label Learning (PLL), where the training samples contain instances with multiple candidate labels, of which one is the true label. They propose a new nearest neighbor (kNN) style learning algorithm that allows the classifier to optionally *abstain* from making a prediction. \n\nHaving found the $k$-nearest neighboring instances to the current test instance and their associated candidate label sets, the algorithm uses Dempster-Shafer theory (DFT) to predict a label and its associated confidence score. By thresholding the confidence score, they then decide whether to return the predicted label or abstain from making a prediction.\n\nDFT is an alternative to classical probability theory for reasoning about uncertain outcomes. In the context of this paper, each of the $k$ candidate label sets is seen as a source of evidence, and are associated with a mass function that assigns probabilities to subset of labels. Applying tools from DFT, the authors aggregate the $k$ mass functions to derive both a predicted label and a confidence score.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would be more inclined to accepting the paper if the authors address the following questions/comments:\n\n- **Justification for probability assignments in Algorithm 1:**\n\nThe choice of basic probability assignments $m_i$s needs further justification. Taking $\\tilde{s} = \\mathcal{Y}$ for an unseen test instance, your choosen $m_i$ assigns a probability of 1/2 to both the set $s_i$ and the set of all labels $ \\mathcal{Y}$. The paper mentions why you choose equal probabilities for the two sets, but whats unclear to me is why $m_i$ assigns non-zero probabilities to only two sets. I could imagine a probability assignment where  $s_i$ receives a probability of 1/2 and all other possible subsets other than $s_i$ receive equal probabilities, such that $m_i$ totals to 1.\n\nCould the authors elaborate on their rationale behind $m_i$ having a support of size 2?\n\nFurthermore, wouldn't it be more accessible to present Algorithm 1 for an unseen test instances $\\tilde{x}$, instead of a generic $\\tilde{s}$, thereby simplifying lines 3-7 to only the Else part. \n\n- **Illustrations to convey intuition:** \n\nI would urge the authors to include a simple toy example to go over the workings of their algorithm. For example, a particular corner case of interest is when the candidate label sets have exactly one label in common - it would be useful to show that your method would indeed predict this label with high confidence. \n\nThe following is an example that helped me understand and work through your method. Suppose the candidate label sets from k=3 neighbors are {1, 2}, {1, 3}, {1, 4}, then the belief probability assignment $\\tilde{m}$ from (4) evaluates to:\n\n{1}:  1/2\n\n{1, 2}:  1/8\n\n{1, 3}:  1/8\n\n{1, 4}:  1/8\n\n{1, 2, 3, 4}:  1/8\n\nIts intuitive that because 1 is the lone common label in across candidate sets, it receives a higher probability mass. The classification rule predicts label 1. The rejection confidence score evaluates to 1/2 - 1/4 = 1/4 > 0; therefore, we accept the prediction.\n\n- **Intuitive background description for DFT:**\n\nPlease elaborate on Sec 3.2 to provide a more intuitive and self-contained discussion on DFT.\n\n- **Experimental metrics:**\n\nThe experimental results in Table 1 and Table 2 use particular thresholds for different methods (50% for some baselines, 90% for proposed method). Instead, for a more fair comparison, I would strongly recommend tuning the thresholds to reach a fixed rejection rate and reporting test accuracies for all methods at that fixed rejection rate. In other words, reporting accuracies for different fixed rejection rates may be a more useful comparison than fixing the thresholds and varying the rejection cost $\\lambda$.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4C6s4c86Fx", "reviewer": "Reviewer_z1xF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. This paper is easy to follow.\n2. Authors provide theoretical justification and analysis for the proposed method.\n\nCons:\n1. The authors assumpt that the near samples in feature space attribute to the reliability of the target sample, and thus a reject option. But it is more from an intuition and lacks of theoretical justification or experimental results.\n2. The novelty. The reject option machinical has been studied in recent years. The authors seems to simply apply it on the PLL tasks.\n3. The experimental results in Table 1 could not validate the effectiveness of proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focus on the limitation of potentially incorrect prediction of current partial-label learning methods. The authors utilize a nearest-neighbor algorithm to avoid the risk of incorrect prediction and propose a confidence-based strategy. They also conduct a series of experiment to validate the effectiveness of proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please find in weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E1ce1mFFGQ", "reviewer": "Reviewer_xj75", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* The tackled problem is important fits very well to the venue.\n* The paper presentation is sound - it is sufficiently format, includes analyses such as for the runtime and consistency.\n* The evaluation setup is sufficiently broad in terms of used comeptitors and datasets\n\n# Weaknesses\n* Unclear if the proposed method is adding value based on the empirical results. The base approach is only on par with other approaches without reject option\n* Unclear how the the proposed reject option relates to calibration methods and how it would perform against calibrated classifiers", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with learning a classifier with a reject option under partial labels. The authors propose to use a KNN classifier, where the class probabilities are adjusted based on Dempster-Shafer Theory (DST) and combined using Yager's rule. The proposed confidence-based rejection strategy encodes that high probability mass is required for a class prediction if other viable options exist, but lower probability mass is sufficient if there are no other significantly competitive options.\n\nThe approach is evaluated against 10 SOTA partial label learning approaches on supervised datasets augmented with noise (including datasets from UCI and MNIST variants) as well as partial label datasets (including bird-song, flickr. yahoo-news). The authors evaluate approach without reject option as well with it. The results show that the approach is on par with SOTA approaches if no reject option is used, but improves the results when reject options are used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please clarify the following points:\n* How would the reject option for other the other approaches perform if confidences where calibrated or if the best threshold would be searched for? It is partially visible in Figure 1, but not for all results in the table.\n* How can I interpret the results on the far right in Figure 1? Does this end with > 99% but < 100% of labels rejected? \n* Couldn't one use the reject option approach also for the other approaches and if yes, would it increase their performance?\n* How does the approach, in general, relate to calibration of classifiers? It would refer to the related work section on the confidence-based approach, but only bayesian methods and ensembles are mentioned.\n\nLastly, it would be beneficial to have a designated \"Conclusions\" section to summarize the results instead of ending the paper with the section on results.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We uphold the positive reviewer consensus and recommend the paper for publication. The revised version has improved the manuscript in readability and scope of results, and is expected to be of interest to the community.
Decision: Accept (poster)