Estimating Expected Calibration Error for Positive-Unlabeled Learning
Authors: Ryuichi Kiryo
Futoshi Futami
Masashi Sugiyama
Venue: Accepted by TMLR ()

Abstract: The reliability of probabilistic classifiers hinges on their calibration---the property that their confidence accurately reflect the true class probabilities.
  The expected calibration error (ECE) is a standard metric for quantifying the calibration of classifiers.
  However, its estimation presumes access to ground-truth labels.
  In positive-unlabeled (PU) learning, only positive and unlabeled data are available, which makes the standard ECE estimator inapplicable.
  Although PU learning has been extensively studied for risk estimation and classifier training, calibration in this setting has received little attention.
  In this paper, we present PU-ECE, the first ECE estimator for PU data.
  We provide non-asymptotic bias bounds and prove convergence rates that match those of the fully supervised ECE with an optimal bin size.
  Furthermore, we develop an information-theoretic generalization error analysis of PU-ECE by formalizing the conditional mutual information (CMI) for a PU setting.
  Experiments on synthetic and real-world benchmark datasets validate our theoretical analysis and demonstrate that our PU-based ECE estimator achieves performance comparable to that of the fully-labeled ECE estimator.

Published in Transactions on Machine Learning Research (12/2025)
Estimating Expected Calibration Error for Positive-Unlabele Learning
Ryuichi Kiryo
The University of Tokyo, Japan
Futoshi Futami
uutami..utosshi es hww.:ossii..sssssouuu.......
The University of Osaka, Japan
The University of Tokyo, Japan RIKEN AIP, Jopan
Masashi Sugiyama
sugi u-tokyo. ac.jp
RIKEN AIP, Japan
The University of Tokyo, Japan
Reviewed on OpenReview: /ttps://openreee openreveee.....e....mmmmm net/ forum9 id SvoBtLIrP2
Abstract
The reliability of probabilistic classifiers hinges on their calibration the property that their confidence accurately reflects the true class probabilities. The expected calibration error (ECE) is a standard metric for quantifying the calibration of classifiers. However, its esti- mation presumes access to ground-trute truth labels. In positive-umlaeled (PU) learning, only positive and unlabeled data are available, which makes the standard ECE estimator inappli- cable. Although PU learning has been extensively studied for risk estimation and classifier training, calibration in this setting has received little attention. In this paper, we present PU-FCE, the first ECF estimator for PU data. We provide non-asymptotic bias bounds and prove convergence rates that match those of the fully supervised ECE with optimal generalization error analysis of
bin size. Furthermore, we develop an information-theores PU-ECE by formalizing the conditional mutual information (CMI) for a PU setting. Ex- periments on synthetic and real-world benchmark datasets validate our theoretical analysis and demonstrate that our PU-based ECE estimator achieves performance comparable to lhat of the fully-labeled ECE estimator.
1 Introduction
Deep learning models achieve state-of-thee-at accuracy in various classification tasks (Devlin et al. 2019 Dosovitskiy el al., 2021 Radford et al. 2021). However, their deployment in safely-critical applications requires more than just high prediction accuracy. For instance, in medical diagnosis (Jiang et al. 2012 autonomous systems (Muthali et al. 2023), and financial bocision-mmkinn (Bahnson al.] 2014). the con- fidence of the predictions provided by machine learning models must accurately reflect. the true probability of the predicted events. This property, known colibration (Dawid 1982 Guo et al.) 2017]. is essential for reliable decision-making The calibration error, true calibration error (TCE) (Naeini et al. 2015 Guo
ct al. 2017 Roclofs et al. 2002)), is a widely used metric for evaluating the calibration of classifiers. The TCE is typically est by partitioning the confidence score space into bins, followed by the calculation of the average difference between the confidence score and the empirical probability of the true label given the confidence score each bin (Zadrozny & Elkan 2001 Nacini et al. 2015). The resulting estimator is called
the expected calibration error (ECE) (Roelofs et al. 2022). Theoretical analysis of the ECE has also been
As some researchers have pointed out, both TCE and ECE are sometimes called "ECE". For clarity, we follow the convention of Futami Fujisawa 2024

---

Published in Transactions on Machine Learning Research (12/2025)
conducted, establishing bias bounds and the consistency of the ECE to the TCE as the number of samples increases, with an appropriate binning strategy (Gupta & Ramdas 2021 Futami & Fujisawa 2024). The results show an (nn1//3 convergence rate of the ECE, where n is the mumber of test samples. The naive computation of the ECE requires a fully labeled dataset, where each sample has both confidence scores and true labels.
However, in practice, fully labeled data may not be available or may be costly to obtain (Jaskic & Spanias, 2019 Li et al. 2022 Bepler et al. 2019). For instance, when identifying protein particles in microscopy
images, annotating negative data is costly due to diverse backgrounds (Bepler et al.) 2019). In such cases, positive-unlaeeled (Pt) learning, which aims to train a classifier from only positive and unlabeled data, is
a promising approach (Liu et al. 2002 Elkan & Noto 2008 Natarajan et al. 2013 du Plessis et al. 2014
Despite extensive research on PU learning methods, the majority of these methods have focused on maximiz-
ing classification accuracy (Elkan & Noto 2008 du Plessis et al.) 2015 Kiryo al. 2017 Chen et al.) 2020;
Xinrui et al. 2023). On the other hand, calibration in the PU setting has been largely overlooked. This gap is partly because evaluating calibration itself is challenging in the PU setting- -he absence of negative labels makes the standard ECE estimator inapplicable, which prevents practitioners from assessing the reliability of their PU classifiers.
In this paper, we address the problem of estimating the calibration error in PU learning settings. We introduce PU-ECE, a novel method for estimating the ECE using only PU data, without requiring negative labels. The core idea is reframe the ECE computation by leveraging the statistical properties of PU datasets. By applying Bayes' rule within each bin, we can estimate from PU data the conditional expectation of the true label given the confidence score, which is necessary for the ECE computation. Our theorctical analysis establishes non-asymptolic bias bounds of PU-ECE, which guarantee the precision of our estimator from the TCF in a finite sample. With these bounds, we prove that with an optimal binning strategy, our estimator achieves a convergence rate that matches the convergence rate of the ECE in fully-supervised settings of O where is the number of test samples (Futami & Fujisawa, 2024). An information- theoretic generalization error analysis of PU-FCC is also provided by formalizing the functional conditional mutual information (fCMI) (Steinke & Zakynthinou 2020 Hellström & Durisi 2022 for PU setting, In addition, we demonstrate through extensive experiments on synthetic and benchmark datasets that our approach achieves estimation biases comparable to those of the fully-supervised method.
The rest of this paper is organized as follows. In Section 2 we formally describe the problem setting and define notations. In Section 3 we present the PU-ECE method and its theoretical properties and Section we present its information-theoreti analysis. Section 5 provides experimental results to validate our approach. Finally, conclude the paper in Section
2 Problem Formulation
We denote random variables by capital letters and their realizations by lowercase letters. Let X' be the input space and y = {0, 13 be the label space. Let P( be the probability distribution over X' x }, P(X) be the marginal distribution of X, P(X Y = y) be the conditional distribution of X given Y = y E y, and тp- P(Y = 1) be the class prior. The hypothesis space is denoted by H S R and classifier h E outputs real-valued score for given input x X. The model's confidence score is then obtained by applying an output activation function R + [0, 1], which we denote by f(x) == ahh(r)). We assume a is measurable and monotone increasing. Examples include the sigmoid function and the probit function. If the sigmoid function (2) := (1+ is used as o, we refer to h(x) as the logit score. We define F == (o((((()) h E H} for a fixed a. We denote the parameterized classifier and its confidence score function by hnw and frr with w E W, where W is the parameter space. If the parameter w is fixed, we simply write h and f. Let A be the indicator function, which returns 1 if the condition A is true and 0 otherwise. We define [n] (1. for E N. Let I (X;Y) Y Z) denote the conditional mutual information between random variables X and Y given Z.
2

---

Published in Transactions on Machine Learning Research (12/2025)
2.1 Supervised Learning
In supervised learning, the dataset is set of independent and identically distributed (ii.d. samples drawn
from P(X,Y) Let I x y + Rg be the loss function, e.g., the (-1 loss (Wald 1945) for the mis- classification rate. Let SPN == {(xm ) nPN be a dataset of npx samples drawn from P(X, Y). The expected risk of on P) is given by Exy ((((X)) Y)). The empirical risk on Spn is given by 1. 2m=1 I(N(XP) ).N). A standard approach for training classifiers to minimize this quantity, npw
}
principle known Empirical Risk Minimization (ERM) (Vapnik & Chervonenkis, 1971
2.2 Calibration Error Estimation
A classifier is said to be perfectly calibrated if its confidence score matches the true class probability (Murphy
& Epstein 1967 Gupta & Ramdas 2021). Mathematically, this means that for all confidence scores p [0. 1]. we have P(Y
f(X) = pp) p almost surely (Gupta & Ramdas 2021). However, this ideal state is
rarely achieved in practice (Guo et al. 2017). To quantify the deviation from this perfect calibration, the
true calibration error (TCE) (Nacini et al. 2015
Gupta & Ramdas 2021 Roelofs et al. 2022 Futami &
Fujisawa. 2024 used as a calibration metric.
2.2.1
True Calibration Error (TCE)
The TCE of f on P(X,Y) is defined as
TCE(F) Ex [ [ f(x)]
This measures the average absolute difference between the true conditional expectation of Y given f(X) and the confidence score S(X). It quantifies how well the confidence scores align with the true probabilities of the positive class. A lower TCE indicates better calibration, meaning that the confidence scores are closer the true probabilities. However, calculation of TCE(f) requires access to the conditional distribution P(Y f(X)), which is usually unvaiilable
2.2.2 Expected Calibration Error (ECE)
Instead, to estimate TCE(f), we often use a binning method Zadrozny Elkan 2001 Guo et al, 2017
Futami & Fujisawa 2024). This method involves set of B E N bins (intervals), denoted by I = {I b=
that covers the entire range of f. We define the true binned function as
fz(Ø) Σ B(((X) ((X) 1.) (())E
Then its TCE is calculated as follows
Futami & Fujisawa 2024):
B
TCE(fr) = Σ ((((())))LL f(X) e 4 - [[(() f(X) 40ll
TCE(fz) can be estimated from supervised evaluation dataset S. {Xx ) of size ne N, which can be validation dataset a test dataset, by the plug-in method. The resulting empirical estimate is called the expected calibration error (ECE):
B
ECE(F, Sc) =
Σ Pb To.s. fo.s.l
(1)
D0 Lexgge is the fraction of samples in bin b, D6.s. 2m L The
1oxg Y
where pb
is the average
D true label in bin b, and J.S. = mm 1 tex is the average confidence score in bin b. We assume
m 1ossee
ne 2 2B, where ne is the supervised evaluation dataset size and B is the number of bins.
3

---

Published in Transactions on Machine Learning Research (12/2025)
For I, there are two common choices: uniform width binning (UWB) (Guo et al. 2017 and uniform mass binning (UMB) Zadrozny & Elkan 2001). In UWB, the bins are defined as 1. = 5_ UMB, the bins are defined as Ia = (ub-1, u6] for b E [B], where 44 = 0, up = f(s)) for b
h for b E [B]. B
B
E [B - 1], and
= 1. Here, ly = " J, where [-] denotes the floor function that returns the greatest integer less than equal to the input value, and f(e) denotes the k-th order statistic of the set (f(Xm))
2.3 Unlabeled Learning
Positive-umlaeledd (PU) learning is classification problem where only positive and unlabeled data are observed, but no negalive data. Among various settings of PU learning (Elkan & Noto 2008 Natarajan et al. 2013: du Plessis et al. 20144 Kato et al. 2019 Bekker et al. 2020), we focus on the relected-completety at-random (SCAR) and casc-control scenario. Concretely, we observe two independent sample sets: a set of
unlabeled samples St.u := { [x m=1
" iidd. P(X) and set of positive samples Sr. = {X " i.dd.
P(X ===11 1), with the full training data denoted by S (Su,u. Su.P) (du Plessis al. 2014 m m=1
This
SCAR and case-control scenario is commonly used in the PU learning literature (Natarajan et. al.
2013 3
du Plessis et al.
2014 Kiryo et al. 2017 Yao et al. 2021p. Throughout the paper, assumne that the class
prior Tp = P(Y 1 is known-lh in practice, it can be estimated from data Blaachard et al. 2010 Garg
et al.
2021 Zhu et al. 2023
In PU
learning, an unbiased risk estimator of on Sty is given as follows
(Ntarajan et al.
2013 du Plessis
et al.
2014
2015):
Der.U Σ
(h Deer
TP
Σ
(((h(xm))) -1(()) 0)}
1(h(x ).
(2)
Utr,U
The estimator suitable for estimating the expected risk E [I(((X),,)) Y) or training simple classifiers (u Plessis al. 2014 2015), but it does not work well for training more complex classifiers such as
deep neural networks due the negative risk issue (Kiryo al. 2017). To overcome this issue, non-negative risk estimators were proposed (Kiryo et al. 2017 Lu et al. 2020 The non-negative risk estimator is given by
PAP.P
D ((h(x ).0)
(h) = TP
Σ m=1
((h(XttP)) ), + lec
TP
Σ (h(xx-P))0) m=1
3.0)
Drr.P
Btr m=1
Ttt,.
where loc R - Rzo is called a consistenl-corretioon function. Several functions have been proposed for use as lees e.g., max 0} (Kiryo et al.) 2017 Lu et al. 2020).
3 PU-ECE: An Estimator for Calibration Error in PU Learning
Conventional ECE estimation, as shown in Eq. ( requires labeled data (both positive and negative), but in the PU environment, negative data are unavailable, as introduced in Section 2.3 The main challenges are (i) estimating the true positive rate within bins, and (ii) handling the fact that the data do not come from a single distribution P(X,Y) but from P(X) and P(X 1). In this section, we propose method to estimate the ECE in the PU environment. The proposed method is based on the idea of using positive and unlabeled data to estimate E Y f(X) Io]. We denote the PU evaluation data as Sy := {X U y Sp == {XP yop and Spu == (Su, Sp). The evaluation data are independent of the training data and f. and can be a validation dataset or a test dataset in practice.
3.1 Proposed Estimator
Since Y E 1), the conditional expectation can be expressed as
B|y f(X) 41] = P(Y 11()) 1)) P(Y = 11,((X E E P(((X) T,)
TpP(f(X) E Lo Y =1 P(f(X) (b)
(3)

---

Published in Transactions on Machine Learning Research (12/2025)
In the PU learning setting, we have access to positive samples Sp {X [XP]1g I drawn iid. from P(X = 1)
and unlabeled samples Su = [x drawn i.idd from P(X). The term P(f()) E Io | Y = 1) in Eq. 3 can be estimated by the empirical frequency of positive samples falling into bin Ig: Ps,P = op 2m=1 [(xgDe Similarly, P(((X) E It) can be estimated by the empirical frequency of unla- beled samples falling into bin Tg: Pb,u = no
Em S(M))4 These estimators are unbiased, that is, P(f(X) E It). Let [4, Em= ((gg)),, and
E [P..p] P(f(X) E Is | Y = 1) and E [p..u] Nu. -Em=1 s(x)))) be the number of positive and unlabeled samples in bin Ib, respectively. Then Po.P Ho.p//pp and Po,u = Lt.ul/nu. Using these estimates, an estimator for the conditional probability P(Y =1 =1] E T)) is naively obtained Ss. pPp6.P The ECE for PU learning (PU-FCE) can then
PA.U
estimated as
B
Σ
Pa.U S0 S0.seel,
where Sb.Spu = Σ D xxx f(x
is the average confidence score for unlabeled samples in bin Io. If Lo,ul 0, then Pa,U = O. In this case, Sp. would involve division by zero. However, this is a removable singularity. For all cases where Ps,U > O, we can algebraically reformulate the estimator as follows:
1..0
Ef
Σ
TPP R0p
nP
1 SX
1 BU
Σ
RU
fx
This new form is mathematically equivalent for all Pa.U > O, and remains well-defined at Рb, 0, and its value at Pa, 0 coincides with the limit of the estimator as Pb.t 0. The computational complexity of PU-ECE is Onnu + np) given the bins. Including bin construction (sorting), the total complexity is O(nu log mu + np log mp), which is the same as that of the standard ECE.
We define UWB and UMB for the PU setting as follows: in UWB, the bins are defined in the same way as in supervised learning, while in UMB, the bins are defined by the unlabeled data; 4. = (us-1, un] for b E [B], where uy = 0, uh f(k)) for [B 1], and ug = 1. Here, Ko = 1 nub and f(x) denotes the k-th order statistic of the set (f(XW) mar ie., the k-th smallest value.
3.2 Bias Analysis of PU-ECE
We analyze the bias of the proposed estimator given the evaluation data. In the analysis, we assume the following:
Assumption 1. The number of posilive samples np salisfies np 1. The number of unlabeled samples ny
satisfies nu 2 2B for UMB and nu 2 for UWB.
Assumption 2. f(X) is absolutely continuous with respect the Lebesgue measure, i.e., f(X) has probability densily function.
A Assumption 3. E E S(X)) sulisfies L--Lipschita continuity with respect to I(X), iee, |E [(x) = v] -
E [Y f(X) w'll s L/u-00 o'l for all v, v' [0, 1].
The (total) bias of the PU-ECE estimator defined as
Biastor (f. Spv) == (CEpu(f..Sp)) - TCE(f))-
Then, we have the following theorem, which provides non-asymptotic bias bounds for the PU-ECE estimator. Theorem 1. Under Assumptions
and have
1+L + 2 B
Blog
(for UWB),
Espu [Biastot (J, Spu)] <
mp
Du
1+L B
+
(3 B
+ 2BB V2Blog2
1+L
+
1
(for UMB).
ne-B
Vau-B
Rp
An
B
5

---

Published in Transactions on Machine Learning Research (12/2025)
Furthermore, for any 8 E (0, 1), with probability at least 1 - over SpU, we have
1+4 + 2
(B 1og 2 log(1/6))
(for UWB),
Biascot Spu) <
pp
1t B
+ Av- B
(3+21) 22 - log(1/66)
VaD-B
1+I
nD-B B
(for UMB).
The proof of Theorem [ is provided in Appendix A.11 Briefly, the proof is based on decomposing the total bias into two parts: the bias due to binning and the bias due to the estimation by the empirical frequency, and then bounding each part separately by concentration inequalities.
Minimizing the upper bound of the total bias with respect to B, we have the following upper bound orders: Corollary 1. Under Assumption | the scale of the optimal B that minimizes the upper bound in Theorem for both UWB and UMB is given by
(
aP
B B
and the corresponding expected total bias convergence rate given by
((
Espu (f (pu)
"T
np
nu
The proof can be found in Appendix A.2 In practice, this means that to minimize the bias of the PU-ECE estimator, the number of bins B should be chosen according to the formula above, balancing the sample sizes of the positive and unlabeled data. The result corresponds to the convergence rate of the supervised setting; the optimal B is (n²) and the expected total bias is O(n-*), where n is the size of the fully-labeled evaluation data Futami & Fujisawa, 2024). The term mi in these expressions is a constant factor that does not depend on the sample size, but indicates that smaller class prior тт leads
smaller expected bias.
In practical PU learning scenarios, the true class prior Tp is often unknown and must be estimated from data (Bepler et al. 2019 Ito & Sugiyama, 2023), which introduces additional estimation error into the calibration process. It is therefore important to understand how inaccuracies in class prior estimation affect PU-ECE. We also analyze how the class prior estimation error affects the PU-ECE estimation.
Theorem (Error bound due to class prior estimation error). Let कp be an estimate of the class prior Tp, and define the PU-ECE estimator using ीp as
13
ECEpu Spv) =
Σ Σ
Rp
1 (XEEEL
Σ f(x ((X))e
nu m=l
b=1
m=l
Then, we have the following bound on the difference between the PU-ECE with the true prior and that with the estimated prior:
EC SpU) EC SPV) < |m #p/
The proof is provided in Appendix A.3 This inequality does not require expectation and holds for any f and SPU. This theorem shows that when the class prior unknown and must be estimated, the calibration error estimation incurs additional error of at most the class prior estimation error.
4 Information-Theoredl Generalization Analysis of PU-ECE
The previous analysis focused on the bias of the ECE estimation using evaluation data, which is independent of the training data used for learning fw- However, in practice, may recalibrate classifiers using recalibration dataset and then evaluate the PU-ECE on the same recalibration dataset (Kuaar et al. 2019

---

Published in Transactions on Machine Learning Research (12/2025)
or we may calibrate classifiers using the training dataset (Futamii & Fujisawa 2024). In these cases, the data used for PU-ECE evaluation and the classifier canalirratio/trainiii are no longer independent. Therefore, analyzing the generalization error of the PU-ECE is crucial to ensure reliable model selection and evaluation in practice. In this section, we exlend the conditional mutual information (CMI) based analysis framework (Steinke & Zakynthinou 2020 Haghifam et al. 2021) used in supervised learning to the PU setting and analyze the generalization error of PU-ECE using it.
4.1 Information-Theoreti Generalization Error Analysis for Supervised Learning
A rigorous understanding of the generalization error is crucial for evaluating the reliability of machine learn- ing algorithms. Classical paradigms for assessing generalization include the concepts of Vapnik-Chervonenle
(VC) dimension Vaapnik & Chervonenkis 1968 Blumer et al. 1989) and metric entropy (Kolmogorov & Tikhomirov 1959; Dudley 1974). The VC dimension quantifies how many points a hypothesis class can shatter, while metric entropy measures how fincly a function class can be approximated using a finite set of representative functions at a given precision. Despite the established significance of these tools, recent advances information-theoreiic perspectives, particularly those centered on CMT (Steinke & Zakynthinou 2020 Haghifam et al.) 2021), have provided new insights into generalization. For example, functional con- ditional mutual information (fCMI) (Steinke & Zakynthinou 2020) measures the dependency between the predictions of learned model on supersample (the collection of both training and held-out data) and the random split that determines which part of the supersample is used for training, conditioned on the super- sample itself. By bounding this data-model dependency, CMI-based analysis methods provide new insights into why certain learning algorithms exhibit strong generalization performance. For instance, Futami & Fu- jisawa (0244) applied this approach to derive non-asymptotic generalization error bounds for the supervised ECE in the supervised setting.
4.2 Functional Conditional Mutual Information (fCMI) for PU Learning
To analyze the generalization error, we extend the CMI-based analysis framework to the PU setting. We consider supersample X [X";XP] t which is (Ner,u Mer,P) x 2 array of samples. Let X E X be the unlabeled supersample and X E X be the positive supersample. For convenience, we use the O-start-index for the columns of the arrays and the I-start-index for the rows of the arrays. We denote the i-th row of X by X, Analogous to X, we have a supersample membership vector M [A A E {0, 1,",, where M E {0, 1, and M e {0, 11 are the membership vectors for the unlabeled and positive supersamples, respoctively. The entry M, specifies which column of the i-th row X, is retained for the i-th data point for training: M, selects the left column (index 0) and M; 1 selects the right column (index 1). All entries are drawn independently as M; ~ Bern(1/2). We denote the PU training vector by XM. The remaining entries of X that are not selected by M can be regarded as the PU evaluation data. Let A: : X x R + W be randomized algorithm that maps a training dataset and a source of randomness to the parameter space W. The source of randomness, denoted by R E R, is independent of any other random variables. Given a training dataset Sur and randomness R, the algorithm outputs the parameters W = A(SS,, R)) E W, and we denote the correspouding predictor by sw.
We define the functional conditional mutual information (FCMI) for PU data as
[CMI I(sw (X)((X); M(X),
where fw (X) is the matrix calculated by the elementwise application of fw to X. Similarly, we also define fCMI on XU as
fCMIu == (:
By the data processing incquality, we have ICMI, < fCML. The PU fCMI metrics can be bounded by model complexity measures such the VC dimension (Steinke & Zakynthinou 2020 Harutyunyan et al. 2021) or the metric entropy (Futami & Fujisawa 2024h. Detailed discussions and definitions of these quantities are provided in Appendix B
We use the term 'array" for notational convenience, although each entry is an element of X, which can be high-dimensional vector rather than scalar.

---

Published in Transactions on Machine Learning Research (12/2025)
4.3 Generalization Bound for PU-ECE
This subsection analyzes the generalization error of PU-ECE using the fCMTs defined above. We follow Futami & Fujisawa (2024) and use Biastot (fw, St) [TCE(fw) ECEpu(Jw, Su)l as the generalization error of ECEpu (fw. (fw. Stz).
Theorem (Expected generalization error bound of PU-ECE)). Under Assumptions and we have
BR [Biastot (fw. Sw)l
1+L
+
8
x ner,P
{fCMI + B 2}
(for UWB),
Mer.U
1+L B
+
8
x Atr.P
[fCMI + B 2o + (1 L) {fCMly + B Bogg2) (for UMB).
Par.U
The proof of Theorem is provided in Appendix C.1 Briefly, the proof is similar to the proof of Theorem except the dependency the training data is handled by the rCMIs.
Similar to Corollary [ the optimal bin size to minimize the upper bound in Theorem 3 and the order of the upper bound can also be derived a learning algorithm has a sufficiently small fCML. For instance, such an assumption holds when H finite VC class since is also finite VC class, and hence fCMT is bounded by Ollog(Trr.F ( mer,u)) (Steinke & Zakynthinou, 2020). For such a case, we can derive the following result.
Corollary 2 (Total bias upper bound order). Under Assumptions and 3 and assuming fCMI
1/33 ) the scale of the optimal B that minimizes the upper bound in Theorem is given
0
+ 1
((1)))
1/3
B B
Phr.P
and the convergence rate of the corresponding expected generalization error is given by
1/3
ER,s, [Biastot (Sw, Str)] =0
Mtr.P (
Tee,
The proof is provided in Appendix A.2
By Corollaries 1 and ER,St [Biastor (fw, Su)) < Essy [Biastor (fasrorfww,ao (fw,Spu)) and the classifier generalizes well. This would actually happen in practice since the training dataset is often much larger than the evaluation dataset. Such a property is also observed in the supervised setting (Futami & Fujisawal 2024). This represents an important future direction for PU learning to develop a mothod that jointly optimizes the TCE and accuracy, although this beyond the scope of this paper.
if RtrPP >> np and Ru.U > ny
5 Experiments
In this section, empirically evaluate the theoretical properties of the proposed PU-ECE estimator. We empirically validate the theoretical bias and convergence properties of our proposed PU-ECE estimator on both synthetic and real-workd benchmark datasets.
5.1 Experimental Setup
The sigmoid function was used the output activation function. Results averaged over 100 trials, each with different random seed. The code for all experiments was implemented in Python and is available
8

---

Published in Transactions on Machine Learning Research (12/2025)
Dataset
Model
Learning rate
MNIST (LeCun et al.
1998)
60,000
0.49
2-layer MLP 1784-100-100-
1.0 x 10-3
CLFAR-10 Krizhevsky 2009
50,000
0.40
ResNet-18 (He et al.
2016
1.0 x 10-5
DDI (Herrero-Zazo al. 2013
27,792
0.14
R-BERT (Wu & He
2019)
2.0 x 10-5
Table 1: Specification of benchmark datasets, models, and learning rates. Mtr,U denoles the number of
unlabeled samples for training PU classifiers.
Synthotic data Following previous work (Vaiccnavicius et al. 2019 Zhang ct al. 2020 Futami & Fujisawa, 2024), we generated data from a logistic model where P(Y-
X - x) - sig 2r), with the conditional
probability densities p(X) 0.5 (2; 1) + 0.5 (x;; 1).. We evaluated a classifier f(z) sig (Bo + 31z) under two settings: less-calibrated case (Bo = 0.5, f1 1.5) and a better-calibrated case (Bo 0.2, 81
1.9). The ground-truth TCE(f) computed by the trapezoidal rule. Further details are provided in
Appendix D
Benchmark data We used the MNIST, CIFAR-10 and DDI datasets, with specifications detailed in Table Since these datasets are multiclass, we converted them into binary classification tasks by selecting positive classes and treating all other classes as negative. For MNIST, we used the digits 5-9 as the positive class, while for CIFAR-10, we selected the vehicle class (airplane, automobile, ship, and truck) as the positive class. For DDI, we treated drug pairs with annotated drug-drug interaction (DDI) relation the positive class, and all other pairs (without a DDI relation) as the negative class. Classifiers were trained using the nnPU learning method [Kryo et al.) 2017) for MNIST and CIFAR-10 and the imbalanced nnPU learning method (Su et al. 2021) for DDI, and the outputs of the classifiers were used as logit scores, The logistic loss (a.k.a. binary cross- eooss-y loss) log(Esig ((2 - 1)h(z))) or sigmoid loss ( - 1)h(z)) was used as the loss function, depending on the experiment. The sigmoid loss was used as a noiso-robust (Ghosh) eet al. 2017) and dlassffiaaioon-calibaaeeld (Charoenphakdee et al.] 2019) loss function, and hence are widely weakly-supervised learning including PU learning (Kiryo et al. 2017 Chen et al. 2020 Zhao et al.
used
2022
Ye et al. 2023). Although the sigmoid loss is not strictly proper Chhroenphakkee al. 2019), its
confidence scores are conventionally interpreted as class-posterior probabilities (Platt 1999 Chen et al.
2020
Ye et. al.
2023). The sigmoid loss was used for MNIST and CIFAR-10, while the logistic loss was used
for DDI. Classifiess were trained on PU training data with 10,000 positive for MNIST and CIFAR-10 and 2,000 positive for DDI, and U,r,U unlabeled samples specified in Table [ The other details are provided in Appendix
Ground-truth TСE computation To compute TCE(f) for these datasets, we first need to assume parametric models for the true conditional probability densities. Fortunately, the TCE can be computed using the distribution of the logit score h(X) or the confidence score f(X), without modeling the distribution of X, which would otherwise suffer from the curse of dimensionality (see Appendix for details). We observed that the confidence scores were heavily saturated near 0 or 1, while the logit scores were spread out. Therefore, we modeled the logit score h(X) instead of the confidence score f(X), specifically modeling the distributions of the logit score, P(h(X) Y = 1) and P(h(X) were multi-modal, a single-mode parametric family, such the logistic- beta distribution used by Roclofs et al. 2022), is inadequate. For instance, the MNIST logits exhibited two clear peaks. Instead, we fitted Gaussian Mixture Models (GMMs) to P(h(X) Y == 1) and P(A(X) Y = 0), selecting the number of components for cach GMM from 1 to 10 via the Bayesian Information Critcrion (BIC) (Schwarz) 1978). In MNIST, the GMMs with 4 positive class components and 3 negative class components were selected, while in CI 10, the GMMs with 2 positive class components and negative class components were selected, and in DDI, the GMMs with positive class components and 3 negative class components were selected. With these estimated densities, we sampled the logit scores from the GMMs. TCE(f) was calculated using the trapezoidal rule at 10, 000 points.
= 0). The empirical logit histograms
9

---

Published in Transactions on Machine Learning Research (12/2025)
(a) Synthetic
(b) Synthetic
(c) MNIST
(d) CIFAR-10
(e) DDI
less-calibrated better-calibrated
Figure Total bias ECE and PU-ECE, as a function of sample size N. Both axes are log-scale. The lines represent the total bias, and the shaded arcas represent the 90% percentile interval. The dashed lines represent the theoretical convergence rate of O(N-1/3).
(a) Synthetic loss-calibrated
(b) Synthetic better-calibrated
(c) MNIST
(d) CIFAR-10
(c) DDI
Figure 2: Convergence of total bias of PU-ECE varying np. Both axes are log-scale. The lines represent the total bias, and the shaded areas represent the 90% percentile interval. The dashed lines represent the theoretical convergence rate of O(N-1/3).
We use UMB by default, but also evaluate UWB side-by-side to study stability in Appendix I The number of bins B was set to (mp/np+1/nc)-bn for PU-ECE and [NpN]] or [(NNN/ / 1/8 for ECE, following the optimal order derived in our analysis and in Futami & Fujisawa (2024).
Binning strategy
5.2 Empirical Bias Convergence Rate
Let N denote an integer ranging from 10 to 10, 000. We evaluated the convergence behavior by varying N, comparing PU-ECE in settings with (ni = standard ECE with n = N. For settings with either nu = 0o or np = oo, the corresponding components (e.g., (X) E I)) when ny oo) were computed via numerical integration using the trapezoidal rule with up to 10,000 points.
=N, ny = 20), (np = o0 = N), (np = NN, my = 10NN, and
Figure 1 shows the total bias of PU-ECE and standard ECE (absolute bias compared to the TCE) on both synthetic and benchmark datasets. This experiment is designed to validate the theoretical convergence rate of our estimator. According to our analysis, the total bias of PU-ECE is expected to decrease at a rate of mp 1lmp+///n--1/// When both np and ny grow proportionally with N (or are set to oo), this simplifies to (NN333))
O(
The empirical results clearly exhibit this O/N3 convergence, providing strong evidence in support of our theoretical claims. Notably, this rate matches the known optimal convergence rate for standard ECE (Futami & Fujisawa 2024), demonstrating that PU-ECE can achieve competitive calibration performance even without access to fully labeled data.
5.2.1 Finite Positive Sample Size Effect
We also investigated the effect of fixing np by comparing PU-ECE, with (mp = 100, ny = N), (mp 1, 00000 ny N), (rp 10, 000, nu = N), and (np N, = oo).
Figure 2 investigates the effect of fixing ne while increasing nu- We observe that PU-ECE converges to the limiting case of PU-ECE with np N. ny = oo but not to the true TCE. This behavior is consistent with
10

---

Published in Transactions on Machine Learning Research (12/2025)
410
(a) Synthetic less-calibrated
(b) Synthetic
(c) MNIST
(d) CIFAR-10
e DDI
better-calibrated
Figure 3: Effect of prior estimation error on PU-ECE. The lines represent the total bias, and the shaded areas represent the standard error. We perturb the class prior up by +5% and +10% in PU-ECE computation and observe shifts proportional to the prior estimation error, consistent with Theorem 2
our theoretical prediction: even nu oo, the bias remains bounded below by the term depending npp This residual error creates error floor that prevents convergence to the TCE when np is finite.
These results highlight a key practical insight: sufficiently large sets of both positive and unlabeled samples are essential for effectively estimating the TCE.
5.3 Robustness to Class Prior Estimation Error
In many PU applications, the class prior тт is estimated from data. Section 3.2 Theorem 2 shows that using an estimated prior dp perturbs PU-ECE by at most - Ap/ Here, +5% and 100% prior estimation errors mean μ - 7p| = 0.05 Tp and 0.10 Tp, respectively. Since тp known our experiments, we can compute the corresponding error bound from Theorem 2 and compare with the observed total bias. Note that this bound controls only the perturbation due to prior misspecification and does not bound the overall total bias. Consequently, at small sample sizes the observed total bias exceed the prior-error bound. Figure shows that as the data size grows and the total bias itself becomes small, it no longer exceeds the bound and lypically converges to a value below it. The bound becomes tight only in the synthetic datasets when the prior is overestimated. In this case, the total bias approaches the error bound as the sample size grows. In all other settings, including benchmark datasets, the total bias converges to smaller values than the bound, indicating that the theoretical bound is conservative in practice.
6 Conclusions
We proposed PU-ECE, the first ECE estimator that can be computed from PU data. Our theoretical analysis provides non-asymptotic bias bounds for PU-ECE and shows that, with an optimal binning strategy, the bias of our estimator converges to the TCE at rate of O((xp/pp convergence rate of the standard ECE in the supervised setting. We further extended the CMI framework to PU learning, deriving generalization error bounds with the same convergence rate as the bias. Experiments on synthetic and benchmark datasets corroborated the theory: PU-ECE performs comparably to the standard ECE, despite lacking negative labels. These results also reveal two practical caveats: (i) with the number of positive samples fixed, increasing unlabeled data alone does not yield convergence to the true TCF, underscoring the importance of having sufficient positive data for accurate estimation; and (ii) any nonzero prior error makes PU-ECE converge to a biased limit, even though the observed error often below the theoretical bound.
1/mx) This rate matches the
Acknowledgements
FF was supported by JST PRESTO Grant Number JPMJPR22C8. MS was supported by JST ASPIRE Grant Number JPMJAP2405.
11

---

Published in Transactions on Machine Learning Research (12/2025)
References
Kendall Atkinson. An introduction to numerical analysis. John Wiley & Sons, New York, 2 edition, January 1989.
Alejandro Correa Bahnsen, Aleksandar Stojanovic, Djamila Aouada, and Björn Ottersten. Improving credit card fraud detection with calibrated probabilities. In Proceedings of the 2014 SIAM International Con- ference on Data Mining, pp. 677-685, Philadelphia, PA, April 2014. Society for Industrial and Applied Mathematics.
Jessa Bekker, Pieter Robberechts, and Jesse Davis. Beyond the selected completely at random assumption for learning from positive and unlabeled data. In Machine Learning and Knowledge Discovery in Databases, pp. 71 11- Springer International Publishing, 2020.
Tristan Bepler, Andrew Morin, Micah Rapp, Julia Brasch, Lawrence Shapiro, Alex J Noble, and Bonnie Berger. Positive-umlaeeldd convolutional neural networks for particle picking in cryo-electron micrographs. Nature Methods, 16(11):1153-1160 November 2019.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. Journal of Machine Learning Research, 11(99):2973 3009, 2010.
Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth. Learnability and the chervnk-hervvnenkiis dimension. Journal of the ACM, 36(4):929-966) October 1989.
Nontawat Charoenphaklee, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learning from corrupted labels. Tn Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, volume 97, pp. 961 970. PMLR, Jamuary 2019.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self- PU: Self boosted and calibratod positive-unlabcled training. In Hal Daumé Iî and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1510-1519. PMIR, July 2020.
Alexander P. Dawid. The well-calibrated bayesian. Journal of the Americon Statisticnl Association, (3797) 605 610, 1982.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapler of the Association for Compu- tational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. Minncapolis, Minncsota, June 2019. Association for Computational Linguistics.
Monroc D. Donsker and Srinivasa R. S. Varadhan. On a variational formula for the principal eigenvalue for operators with maximum principle. Proceedings of the National Academy of Sciences of the Unittdd States of America, (2(3):780-788 783, 1975.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, May 2021.
Marthinus C. du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Newral Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Marthinus C. du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Confer- ence on Machine Learning, volume 37 of Proceedings of Machine Learning Reseorch, pp. 1386-1394 Lille, France, 2015. PMLR
12

---

Published in Transactions on Machine Learning Research (12/2025)
Richard M. Dudley, Metric entropy of some classes of sets with differentiable boundaries. Journal of Approzimation Theory, 10(3):227-22 236, 1974.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 213 220, New York, NY, USA, August 2008. ACM.
Futoshi Futami and Masahiro Fujisawa. Information generalization analysis for expected calibration error. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.); Advances in Neural Information Processing Systems, volume 37, pp. 2024.
Curran Associates, Inc.,
Saurabh Garg, Yifan Wi,, Alex Smola, Sivaraman Balakrishnan, and Zachary Chase Lipton. Mixture pro- portion estimation and PU learning;a modern approach. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 8532 8544. Curran Associates, Inc.; 2021.
Aritra Ghosh, Himanshu Kumar, and S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, February 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, August 2017.
Chirag Gupta and Aaditya Ramdas. Distribution-free calibration guarantecs for histogram binning without sample splitting. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3942-3952. PMLR, July 2021.
Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, and Dan Roy. Towards a unified information- theoretic framework for generalization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 26370- 26381. Curran Associates, Inc., 2021.
Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Information- gener- alization bounds for black-box learning algorithms. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Pro- cessing Systems, pp. 24670 24682. Curran Associates, Inc., 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Fredrik Hellström and Giuseppe Durisi. A new family of generalization bounds using samplewise evaluated cmi. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (ods.), Advances in Neural Information Processing Systems, volume 35, pp. 10108 10121. Curran Associates, Inc., 2022.
María Herrero-Zavo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. The ddi corpus: An annotated corpus with pharmacological substances and drug-drug interactions. Journal of Biomedical Informatics, 46(5):914-92)) 9200, 2013. ISSN 1532-0464.
Nobutaka Ito and Masashi Sugiyama. Audio signal enhancement with learning from positive and unlabeled data. In ICASSP 2023 2023 TEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5, 2023.
Kristen Jaskie and Andreas Spanias. Positive and unlabeled learning algorithms and applications: A survey, In 2019 1 10th International Conference on Information, Intelligence, Systems and Applications, pp. 1-8, 2019.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno- Machado. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Associalion, 19(2):263- 274, March 2012.
Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with a selection bias. In International Conference on Learning Represenlations, May 2019.
Ryuichi Kiryo, Gang Niu, Marthinus C. du Plessis, and Masashi Sugiyama. Positive- learning with non-negative risk estimator. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Andrey N. Kolmogorov and Vladimir M. Tikhomirov. &-entropy and e-capacity of sets in function spaces. Uspekhi Matematicheskiki Nauk, 14:3 86, 1959.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Meelis Kull, Miquel Perello Nicto, Markus Kängsepp. Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ananya Kumar, Percy S. Liang, and Tengyu Ma. Verified uncertainty calibration. In H. Wallach, H. Larochelle, A. Beygelzimer, F. AAlch--Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems, volume 32. Curran Associates, Inc., 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang, BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinfor- malics, 36(4):1234-122
2019.
Fuyi Li, Shuangyu Dong, André Leier, Meiya Han, Xudong Guo, Jing Xu, Xiaoyu Wang, Shirui Pan, Cangzhi Jia, Yang Zhang, Geoffrey I Webb, Lachlan J M Coin, Chen Li, and Jiangning Song. Positive-umlabeeed lcarning in bioinformatics and computational biology: brief review. Briefings in Bioinformatics, 23(1): bbab461, Jamuary 2022.
Bing Liu, Wee Sun Lee, Philip Yu, and Xiaoli Li. Partially supervised classification of text documents. In Proceedings of the 19th International Conference on Machine Learning, pp. 387 394. Morgan Kaufmann Publishers Inc., July 2002.
Nan Lu, Tianyi Zhang, Gang Niu, and Masashi Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the 23rd International Conference on Artificial Intelligence and Statástics, volume 108 of Proceedings of Machine Learning Research, pp. 1115 1125. PMLR, August 2020.
Allan H. Murphy and Edward S. Epstein. Verification of probabilistic predictions: A brief review. Journal of Applied Meteorology, 6(5):748- October 1967.
Anish Muthali, Haotian Shen, Sampada Deglurkar, Michael H. Lim, Rebecca Roelofs, Aleksandra Faust, and Claire Tomlin. Multi-agent reachability calibration with conformal prediction. Tn 2023 62nd IEEE Conference on Decision and Control, pp. 6596 6603, 2023.
Mahdi P. Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 29, February 2015.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep K. Ravikumar, and Ambuj Tewari. Learning with noisy labels. In C.J. Burges, T. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Newral Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in large margin classifiers. MIT Press, 1999.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.); Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, July 2021.
Christian Robert and George Casella. Monte Carlo Statistical Methods. Springer texts in statistics. Springer, New York, NY, November 2010.
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating bias in calibration error estimation. In Gustau Camps- Valls, Francisco J R Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Slabistics, volume 151 of Proceedings of Machine Learning Research, pp. 1136-40544 PMLR, March 2022.
Gideon Schwarz. Estimating the dimension of a model. Annals of Statistics, 6(2):461-464) March 1978.
Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. Conference on Learning Theory, pp. 3437 3452. PMLR, July 2020.
Guangxin Su, Weitong Chen, and Miao Xu. Positive-umlabeled learning from imbalanced data. In Zhi- Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 2995 3001. International Joint Conferences on Artificial Intelligence Organization, 2021.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 3459-3467. PMLR, April 2019.
Vladimir N. Vapnik and Aleksei Y. Chervonenkis. The uniform convergence of frequencies of the appearance of events to their probabilities. Doklady Akademii Nauk, 181(4):781 783, 1968.
Vladimir N. Vapnik and Alexcy Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability & Its Appliculions, 16(2):264 280, 1971.
Martin J. Wainwright. High Dimensional Statistics Non Asymptotic Vieupoint. Cambridge University Press, 2019.
Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Mathematics, (6(2):265-280) 1945.
D S Wilks. Statistical methods in the atmospheric sciences. Elsevier, 2019.
Shanchan Wu and Yifan He. Enriching pre-trained language model with entity information for relation classification. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19, pp. 2361 2364, New York, NY, USA, 2019. Association for Computing Machinery.
Wang Xinrui, Wenhai Wan, Chuanxing Geng, Yuan Li, and Songcan Chen. Beyond myopia: Learning from positive and unlabeled data through holistic predictive trends. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 67589-67602 67602. Curran Associates, Inc., 2023.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, and Dacheng Tao. Re- thinking class-prior estimation for positive-unlabeled learning. In International Conference on Learning Representations, May 2021.
Xichen Ye, Xiaoqiang Li, songmin dai, Tong Liu, Yan Sun, and Weiqin Tong, Active negative loss functions for learning with noisy labels. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Advances in Newrul Information Processing Systems, volume 36, pp. 6917-997-66 6940. Curran Associales, Inc., 2023.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the 18th International Conference on Machine Learning, pp. 609 616. Morgan Kaufmann Publishers Inc., June 2001.
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-match Ensemble and compositional methods for uncertainty calibration in deep learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11117 11128. PMLR, July 2020.
Yunrui Zhao, Qianqian Xu, Yangbangyan Jiang, Peisong Wen, and Qingming Huang. Dist-PU: Positive- unlabeled learning from label distribution perspective. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognition, pp. 14461-14470, June 2022.
Yilun Zhu, Aaron Fjeldsted, Darren Holland, George Landon, Azaree Lintereur, and Clayton Scott. Mix- ture proportion estimation beyond irreducibility. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.), Proceedings of the 40th International Conference on Machine Learning, volune 202 of Proceedings of Machine Learning Research, pp. 42962 42982. PMLR, 2023.
16


Reviews:
[{"review_id": "VgxYUOxHHi", "reviewer": "Reviewer_k7h8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces PU-ECE, the first estimator of expected calibration error (ECE) in positive–unlabeled (PU) learning. Since standard ECE requires full labels (positive + negative), it cannot be applied in PU settings. The authors propose to reframe the ECE computation using Bayes’ rule with empirical frequencies from positive and unlabeled data. They provide non-asymptotic bias bounds, show convergence rates that match the supervised case, and extend information-theoretic generalization error analysis (via conditional mutual information) to PU learning. Experiments on synthetic, MNIST, and CIFAR-10 datasets validate the theoretical claims.\n\nStrengths\n\nAddresses an important problem namelycalibration in PU learning.\n\nProvides rigorous bias analysis and convergence guarantees.\n\nExtends fCMI-based generalization analysis to a new setting.\n\n\nWeaknesses\n\nNovelty is somewhat incremental; PU-ECE is a natural adaptation of supervised ECE.\n\nHeavy reliance on a known class prior, which is unrealistic in practice; robustness to misestimation is not empirically studied.\n\nExperimental evaluation is narrow (only MNIST/CIFAR binary subsets); lacks validation on real PU application domains.\n\nLimited ablations (no empirical comparison of UMB vs UWB binning).\n\nPractical interpretability is under-discussed (e.g., calibration diagrams, downstream decision costs).", "claims_and_evidence_yn": "No", "claims_explanation": "The theoretical claims are well supported with rigorous derivations and proofs. The convergence rates and bias bounds are clearly demonstrated.\nHowever, the empirical support is thin where results are confined to small-scale datasets and controlled settings. The claim of “comparable performance to supervised ECE” is not convincingly validated beyond synthetic and toy cases. The assumption of a known prior is a significant limitation that weakens the practical strength of the claims.", "audience_yn": "Yes", "audience_explanation": "he work will interest researchers studying calibration, weak supervision, and uncertainty estimation. Extending calibration theory to PU settings fills a genuine gap..", "requested_changes": "Add experiments testing PU-ECE under estimated priors (e.g., +-10% error) to evaluate robustness.\n\nInclude at least one real-world PU dataset (e.g., bioinformatics, medical, or text classification) to demonstrate practical relevance.\n\nProvide empirical comparison of UMB vs UWB binning strategies.\n\nDiscuss or simulate extensions to multiclass PU calibration.\nAdd reliability diagrams or calibration plots comparing PU-ECE with supervised ECE.\n Clarify computational complexity and scalability relative to standard ECE.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u69mMGugLs", "reviewer": "Reviewer_wNcX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors seem to have taken the methodology of Futami &Fujisawa 2024 (cited in the paper) and extended it from the standard supervised learning regime into the PU learning regime.\n\nThe difference in problem setup means that the empirical distribution of positive examples in each histogram bin is not available, and this affects what theoretical results can be derived. A natural alternative estimator is proposed which is appropriate for the problem setup. Theorem 1 analyzes the statistical estimation error and the binning discretization errors of this estimator. Compared to Futami &Fujisawa 2024 Corollary 1 they provide corrections deriving from the different problem setup. The authors also provide a concentration result bounding the deviation of the bias term from the expected bias. (n.b. the 'bias' term is a random varable that depends on the evalation set so knowing how large deviations are plausible for a single evaluation dataset is valuable). Theorem 2 introduces an error analysis based on the assumed known class marginal distribution $\\pi_P$ which is important for this problem setup. Theorem 3 presents analysis of the generalization error, parallell to Futami &Fujisawa 2024 Theorem 5. The proof in appendix points out several of the reasons for discrepancies in the formulas.\n\nThe authors also provide a numerical section showing that the theoretical results are observed in practice.", "claims_and_evidence_yn": "Yes", "claims_explanation": "As far as I can determine, the theoretical claims are supported, and the proofs are clear and possible to follow.\nThe numerical claims are based on reasonable scenarios and evaluation strategies.\n\nWhile I find some problems in the clarity of the presentation (see requested changes), I find the article clear enough to understand.", "audience_yn": "Yes", "audience_explanation": "PU learning is an important scenario in applied machine learning, and finite sample bounds on calibration errors can be used both for inference and for guiding study design (inform sample sizes etc).\n\nOne statement in the conclusions \"if the number of positive samples is fixed and the number of unlabeled samples increases, PU-ECE does not converge to the true TCE\" is a good example of what TMLR readers should be interested in learning from the paper.", "requested_changes": "Would strengthen your work:\n- Section 2.1 states \"Classifiers are trained by the empirical risk minimization\" but many other training methods are used in supervised learning. Regularization often falls outside ERM. Distributional robust optimization is used. Modern ML use in-context learnin. The list goes on. your quote is this misleading.\n- First equation in section 2.2.2 has a sum and an indicator function, effectively removing the sum. It corresponds to eq (3) in Futami &Fujisawa 2024, but in that case x is different in the conditional expectation and the indicator. This looks like a typesetting mistake. Please clear it up\n- Last paragraph page 3 discuss extensions to TCE/ECE, but this has no bearing on the article. Should be put in background (related work) or preferrably just removed.\n- Section 4.1 should define what CMI is in this context. For a reader familiar with CMI as defined in information theory, the explanation that \"CMI quantifies the amount of information that a model retains about the training data after observing a test sample.\" is confusing. I think clarity would be improved by moving the definition of fCMI from appendix B to the main text.\n- 5.1 Common Setup / Synthetic data. Defining a distribution in terms of P(Y|X) and P(X|Y) instead of P(Y|X) and P(X) is unnecessarily confusing. Even though your presentation is *correct* it is harder to read than it should be.\n- 5.2 Motivational Study should be part of the background. All problems should be motivated there, not in the Experiments section. Paragraph 3, starting with \"This collapse is likely due to...\" does not belong in a main text section. Such speculation should either be addressed with experiments proving it is the case, or it should be placed in a discussion/conclusion/outlook section. By mixing speculation with the experiment setup, it is harder to separate what is well supported science and what is not. The main point you present in section 5.2 also seems to be \"PU learning is considerably more prone to miscalibration than standard supervised learning\" which seems besides the point if the article given that I understand the research question as \"What are the statistical properties of our propsed estimator?\"\n- I suggest you remove Figure 2 and 3. The text in 5.3 explains very well that you made sensible choices for estimating the TCE, so the figures don't really add to that. Possibly, keep Figure 2 and remove figure 3, since they are equivalent up to a transformation of the x-axis.", "broader_impact_concerns": "I do not see any problematic ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wNyoLtS9MB", "reviewer": "Reviewer_z4Bc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel approach to measure the expected calibration error (ECE) for positive-unlabeled data (PU) to gain more insides into the reliability of a model trained with PU data only.", "claims_and_evidence_yn": "No", "claims_explanation": "The authors provide a theoretical framework for estimating ECE in a setting where only positive labeled and unlabeled data is given. \n\nHowever, the proposed framework is limited to binary classification tasks, where the actual usability of ECE is questionable. \nFor a two-class problem, the desired confidence is typically 100% for a correct prediction and 0% for an incorrect one. Consequently, the binning strategy inherent to ECE may not be well-suited to this problem and could provide a misleading measure of calibration. This choice of metric requires a more thorough justification.", "audience_yn": "Yes", "audience_explanation": "The theoretical framework provided is intersting.", "requested_changes": "Requested Changes:\n\nPlease provide formal definitions for the variables n and B immediately following Equation (1).\n\nOpen Questions:\n1. The authors should provide a justification for the choice of using Softmax as the output activation function, especially considering its potential limitations for the specific task at hand.\n2. The applicability of the proposed approach appears to be very limited, as all experiments are only conducted on binary classification tasks. The authors should discuss the challenges of extending their method to a multi-class setting.\n3. The use of Expected Calibration Error (ECE) for a binary classification problem is questionable. For a two-class problem, the desired confidence is typically 100% for a correct prediction and 0% for an incorrect one. Consequently, the binning strategy inherent to ECE may not be well-suited to this problem and could provide a misleading measure of calibration. This choice of metric requires a more thorough justification.", "broader_impact_concerns": "no broader impact statement required or given", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)