Rethinking Class-Prior Estimation for Positive-Unlabeled Learning
Authors: Yu Yao
Tongliang Liu
Bo Han
Mingming Gong
Gang Niu
Masashi Sugiyama
Dacheng Tao
Venue: ICLR 2022 Poster (2022)
Keywords: Positive-Unlabeled Learning
Class-Prior Estimation

Abstract: Given only positive (P) and unlabeled (U) data, PU learning can train a binary classifier without any negative data. It has two building blocks: PU class-prior estimation (CPE) and PU classification; the latter has been well studied while the former has received less attention. Hitherto, the distributional-assumption-free CPE methods rely on a critical assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution. If this is violated, those CPE methods will systematically overestimate the class prior; it is even worse that we cannot verify the assumption based on the data. In this paper, we rethink CPE for PU learning—can we remove the assumption to make CPE always valid? We show an affirmative answer by proposing Regrouping CPE (ReCPE) that builds an auxiliary probability distribution such that the support of the positive data distribution is never contained in the support of the negative data distribution. ReCPE can work with any CPE method by treating it as the base method. Theoretically, ReCPE does not affect its base if the assumption already holds for the original probability distribution; otherwise, it reduces the positive bias of its base. Empirically, ReCPE improves all state-of-the-art CPE methods on various datasets, implying that the assumption has indeed been violated here.

Published as a conference paper at ICLR 2022
RETHINKING CLASS-PRIOR ESTIMATION DOSITIVE-UNLABER LEARNING
FOR
Yu Yao
Tongliang Liu't Bo Han? Mingming Gong"
Gang -
Masashi Sugiyama*> 4.5
Dacheng Tao 8.1
The University of Sydney
Hong Kong Baptist University Thhe University of Melbourne
'RIKEN AIP
"The University of Tokyo
"JD Explore Academy, China
ABSTRACT
Given only positive (P) and unlabeled (U) data, PU learning can train a binary classifier without any negative data. It has two building blocks: PU class class-prior estimation (CPE) and PU classification; the latter has been well studied while the former has received less attention. Hitherto, the CPE methods rely on a critical assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution. If this is violated, those CPE methods will systematically overestimate the class prior; it is even worse that we cannot verify the assumption based on the data. In this paper, we rethink CPE for PU learning can we remove the assumption to make CPE always valid? We show an affirmative answer by proposing Regrouping CPE (ReCPE) that builds an auxiliary probability distribution such that the support of the positive data distribution is never contained in the support of the negative data distribution. ReCPE can work with any CPE method by treating it the base method. Theoretically, ReCPE does not affect its base if the assumption already holds for the original probability distribution; otherwise, it reduces the positive bias of its base. Empirically, ReCPE improves all state-of-the -f-the--rt CPE methods on various datasets, implying that the assumption has indeed been violated here.
INTRODUCTION
Pesilive-unlabeled (PU) learning can date back to 1990s (Denis, 1998; De Comité et al., 1999; Letouzey et al., 2000), and there has been surge of interest in this learning scenario in recent years because of the difficulty to annotate large-scale datasets (Ren et al., 2014; du Plessis et al., 2014; 2015; Christoffel et al., 2016; Jain et al., 2016; Ramaswamy et al., 2016; Sakai et al., 2018; Kato et al., 2018; Bekker & Davis, 2018; Gong et al., 2019; Bai et al., 2021; Xia et al., 2021; Yao et al., 2021). It is also fallen into different applications, such knowledge-
completion (Galárraga et al., 2015; Neelakantan et al., 2015), text classification (Lee & Liu, 2003; Li & Liu, 2003), and medical diagnosis (Claesen al., 2015; Zuluaga et al., 2011).
PU learning can be divided into two different settings based on different data generation processes. The first setting called censoring PU learning (Elkan & Noto, 2008), which follows a one-sample configuration. Specifically, a sample S is randomly drawn from the unlabeled data distribution Pu- and a positive sample Sp is then distilled from it, i.e., randomly selecting some positive instances contained in the unlabeled data to be the positive sample. The second setting is called case-control PU learning (Kiryo et al., 2017). In this setting, a positive sample Sp {x;
randomly
drawn from the positive class-conditional distribution P, = P(X = 1), and unlabeled sample S, = {x; J is randomly drawn from the unlabeled data distribution P,. Because case-control PU learning is more general than censoring PU learning (Niu et al., 2016), therefore, we will focus on the setting of case-control PU learning.
Under the setting of case-control PU learning, lot of classification methods have been proposed (Ren et al., 2014; du Plessis et al., 2014; 2015; Christoffel et al., 2016; Sakai et al., 2018; Kato et al., 2018; Bekker & Davis, 2018; Kwon al., 2019; Tanielian & Vasile, 2019; Gong et al., 2019).
"Correspondence Tongliang Liu

---

Published as a conference paper at ICLR 2022
ASP.
Figure 1: (a) The unlabeled data distribution P. and the positive class-conditional distribution Pp are given. (b) Assume that the latent negative class-cou canditoondii distribution P. is fixed, i.e., 0.5Pa is shown by the green curve, and that the class-prior is 0.5, i.e., P, 0.5P, 0.5Pp. (c) The existing
CPE methods will output 0.7 instead of 0.5 because they of Pu in Pp. (d) Applying the proposed ReCPE method, auxiliary distribution Pp will be created and the existing CPE methods will output x = 0.49 instead of 0.7 with input Per and Pa instead of Pp and Pa.
always output the maximum proportion
However, the class-prior estimation (CPE) (Elkan Noto, 2008; Jain et al., 2016; Ramaswamy et al., 2016; Christoffel et al., 2016; Kato et al., 2018) has received less attention. Formally, CPE is defined as a problem of estimating π = Ply = 1) E (0, 1) given a sample from the marginal distribution Pa and sample from positive dlass-conditional distribution Pp. The marginal distribution P. mixed with both positive and negative class-conditional distributions, i.e., P. TP, + (1 π T). CPE acts crucial building block for state-of-the-an PU classification methods, and it essential to build aatistically-consistt PU classifiers (du Plessis et al., 2014; Scott, 2015; Jain et al., 2016; Kiryo et al., 2017). The formulation of these classification methods involves the class-prior и,, but ı is usually unknown in practice. If is poorly estimated, the classification accuracy of the tfate-of-the-ar PU classification methods (du Plessis al., 2014; 2015; Kiryo et al., 2017) could be degraded.
The mixture proportion estimation (MPE) is closely related to CPE (Blanchard et al., 2010; Scott, 2015). In the setting of MPE, there a mixture distribution
F = (1 x² H
where H and G are called component distributions. Given the samples randomly drawn from F and H, respectively, MPE aims to estimate the maximum proportion x* (0, 1) of H in F. Thereby, if the maximum proportion '² is identical to the class-prior т, the MPF methods can be employed to obtain x by letting P. and P, be the mixture distribution F and the component distribution H. respectively; otherwise, the MPE methods cannot be employed. To the best of our knowledge, most of state-of-the-an CPE methods (Blanchard et al., 2010; Liu Tao, 2015; Scott, 2015; Ramaswamy et al., 2016; Jain et al., 2016) are based on MPE, which do not rely assumptions that the data are drawn from a given parametric family of probability distributions (i.e., they are iistributional-assmnpe -ass-aattooo----faaa free methods).
To let these distributional- cutriiionn-ssuumtttoooniii free methods can be used to identify class-prior n, '² must be identical to the class-prior The irreducibility assumption (Blanchard et al., 2010) has been proposed to make them identical, which is employed by all these CPE methods implicitly or explicitly, the best of our knowledge. It assumes that the support of the positive class-conditional distribution P. is not contained in the support of the negative class-conditional distribution P,. However, it is strong and hard to be verified in PU learning, since Pa is latent distribution, such that do not have any prior knowledge about it. Additionally, since the applications of PU learning are diverse (Hsieh et al., 2019; Bekker & Davis, 2020), is hard to guarantee that the support of Pp not in the support of P..
If the irreducibility assumption cannot be satisfied, the existing distributional--n wwwpriononaassutiiioonnire CPE methods will suffer from an overestimation of я. For example, in Figure (a)), we show both the unlabeled data distribution P. and the component distribution Pp. In Figure 1(b), we assume the latent negative class-conditional distribution P. is fixed as shown in the green color, and the positive class-pric prior ग = 0.5. In Figure ((c), we show the existing distributional distributionallaasspttioo
free CPE methods will output the biased class-prion 0.7. It is different from the ground truth 0.5, since the support of P, contained in the support of P,, When the irreducibility assumption is not held, how to improve the estimations of distributionallasuuptioooi
PU learning methods is challenging but useful.
2

---

Published as a conference paper at ICLR 2022
Because the irreducibility assumption is impossible to check without making any assumption Pu. Thereby. in this paper, we rethink those CPE methods and propose a novel method called Regrouping CPE (ReCPE) which improves the estimations of the current PU learning methods without irreducibility assumption. The main idea of our method is that, instead of estimating the maximum proportion of Pp in Pu. we build distribution Po always guaranteeing the irreducibility assumption. Then we use the existing CPE method to obtain the maximum proportion of Pe in Pu. which denoted by æ'. We show that, with both theoretical analyses and experimental validations, when the irreducibility assumption holds, our ReCPE method does not affect the prediction of the existing estimators; when the irreducibility assumption does not hold, our method will help the current estimators have less estimation bias, which could improve the performances create new class-conditional (auxiliary) distribution Pp. By solving it, n' 0.51. The estimation bias of the existing estimators will reduce to - π = 0.01 instead of x* πT = 0.2.
CPE problem by creating new auxiliary
PU classification tasks. For example, in Figure (d)) we
The rest of the paper is organized as follows. In Section 2, we review the irreducibility assumption and its variants. We discuss the difficulty of checking the assumptions. In Section 3, we provide inriiuutiooal-asuupttiii
the estimation biases of the existing consistent we propose our method ReCPE, followed by theoretically analysis of its estimation bias and the implementation details. All the proofs listed in Appendix A. The experimental validations given in Section 4. Section 5 concludes the paper.
CPE methods. Then
2 IRREDUCIBILITY OF CPE
In this section, we briefly review the assumptions used for existing distributionallasuuppiiiooo CPE estimators. Then we provide the estimation bias introduced by consistent cuaributionallassuuptii free CPE methods when the assumptions do not hold.
The irreducibility assumption. Let Pp and P. be probability measures (distributions) on measur- able space (X', G), where X' is the sample space, and 6 is the o-algebra. Let x² be the maximum proportion of Pp Pu. To let x* be identical to æ, the irreducibility assumption was proposed by Blanchard et al. (2010).
Definition (Ireeduciiility). Pn and Pp are said to satisfy the irreducibility assumption if Pn is not mixture containing Pp. That is, there does not exist a decomposition Pa (1 B)Q + 3Pp where Q is probability distribution on the measurable space (X, G), and 0 < B < 1.
Equivalently, the assumption assumes the support of P. is hardly contained in the support of Pn- It means that with the selection of different sets S. the probability (.(S) can be arbitrarily close O, and (S) > O. Suppose we can access the distributions P.. Pp and the set C containing all possible latent distributions, then the class-prior can be found as follows:
inf P.(S) SE6,P,(S)>0 P,(S)
x² 2 sup(a P. = - a)) aap K EC
(2)
To the best of our knowledge, all existing
CPE methods (Blanchard et al., 2010; Scott et al., 2013; Liu & Tao, 2015; Scott, 2015; Ramaswamy et al., 2016; Ivanov, 2019) are variants of estimating the maximum proportion x of P. in P.. Many of them are statistically consistent estimators (Blanchard et al., 2010; Scott et al., 2013; Liu & Tao, 2015; Scott, 2015).
The variants of the irreducibility. Based on the irreducibility assumption, estimators can be designed with theoretical guarantees that they will converge to the class-prior x (Blanchard et al., 2010). However, the convergence rate be arbitrarily slow (Scott, 2015). The reason is that the irreducibility assumption implies the following fact (Blanchard et al., 2010; Scott et al., 2013)
inf Pa(S) = 0, (E) PA(S)
(3)
ie., the maximum proportion of Pp in P. approaches to 0. To obtain the plass-prior म. it requires finding sequence of the sets S converging to the infimum, which empirically can be hard to find. Therefore, the convergence rate of the designed estimators based on Eq. (3) will be arbitrarily slow. To ensure fixed rate of convergence, the anchor sel assumption, stronger variant of the irreducibility
3

---

Published as a conference paper at ICLR 2022
assumption, has been proposed (Scott, 2015; Liu & Tao, 2015; Xia al., 2019; 2020; Yao et al., 2020). It assumes that
(S)
P, Pp(S) 0,
min
(4)
i.c., there exists set can achieve the minimum O, which is called an anchor set. Another stronger variant is the separability assumption (Ramaswamy al., 2016) which extends the anchor set assumption to function space. It is proposed to bound the convergence rate of the method based on kernel-mean-matching (KMM) technique (Gretton al., 2012).
REGROUPING FOR CPE (RECPE)
In this section, we propose general method named regrouping for CPE (ReCPE). We discuss how to theoretically and empirically mitigate the overestimation problem of the class-prior π.
3.1 MOTIVATION
In general, it impossible to verify the irreducibility assumption for CPE. To check the assumption, we need to make Pa itself to be observable and verify that whether the distribution Pa is mixture containing the distribution Pp. which obviously contradicts the setting of PU learning. However, in practice, the irreducibility assumption may not hold for many real- world problems, because the negative class is diverse (Hsieh et al., 2019; Bekker & Davis, 2020) in PU learning. the assumption does not hold, P, is said to be reducible Pp. and introduce an estimation bias.
attpaattiionaaasuuptiiioo free CPE methods will
(.(S) PS) be the maximum proportion of Pp in Pn, given P. = (1-1 Pn Pn + TP p. for 0 1 we have
Proposition Let 8r inf infses.
inf Ps(S) (5))2, P,(S) T (1-))
x = π + (1 -
(5)
According Proposition if the irreducibility assumption does not hold, then there exists 0. In this case, maximum proportion x² can still be obtained, but it is different from but equal to (1 π) T" In this case, we directly employ existing methods, they could introduce an arbitrary estimation bias (1 - 3 which depends on P..
CPE
To reduce the estimation bias, propose ReCPE. The process of regrouping change the original class-cont conditional distributions Pn and Pp into new class dlass-conditions distributions Pn' and Pp by transporting the probability mass of the regrouping, new class-conditional distributions are guaranteed to satisfy the irreducibility assumption, and therefore, the new positive class-prior x' can identified by current CPE methods. To get the intuition, provide concrete example as follows.
from the negative class the positive class. After
Suppose that Pp is the uniform on [t. 1], Pn uniform on [0, 1J. and म = 1 Then we have P. such that it is uniform on (0, }) and 13, 11. respectivlly.. Specifically, the probabilities are
P. 00.
1 4' (1.)))
P. (1
3 4"
In this case, by Eq. the maximum proportion of P, in P. i* Lete > 0 small constant,
and let A = (1 1]. In this case, the mass of A in P. from P., is * f. After transporting
the mass & from P, to Pp. have new positive class-prior n' and new class-conditional Pp
which is uniform on [ p) and p. 1), respectively. Specifically,
+ Pp 1) I+p"
1-2p 2p Per (1 - 1) 3p
2
1+p
The left equation above shows that the new class class-prior n' is dependent P or the size of A. By controlling set or p to be small, x' can be as close to म as possible. This is the intuition of how regrouping works.
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 ReCPE
Input: An unlabeled sample S, i.i.d. drawn from Pu. positive sample Sp i.i.d. drawn from Pp. and the percentage of the sample needed to copy from S. to Sp.
1: Train binary classifier h with the unlabeled sample S, and positive sample Sp by treating S. as a negative sample;
2: Assign each example
Su with the negative class-posterios probability P(Y - x) predicted by the trained classifier h;
3: Obtain Sp by copying p |Sul examples with the smallest negative class-posterior probability
P(Y = = = z) from S. to Sp:
4: Estimate the class-prion n' by employing an algorithm based on Eq. (5) with inputs S, and Spr.
Output: The estimated new class-prior #'.
3.2 PRACTICAL IMPLEMENTATION
In practice, we have to implement the aforementioned idea of regrouping based on positive sample Sp and unlabeled sample S,, Since the negative sample unavailable, we cannot "cut and paste" any example from negative class to positive sample Spi instead, we can "copy and past" some unlabeled examples to Sp. When doing so, we should select small set of samples Á which look the most similar to the positive class and dissimilar to the negative class, which could encourage the difference between the original Pa. Pp. and and ग, Pp'. and I to be small. This is why A = (1 - p. 1] was selected in the above intuitive example, i.e., A belongs geometrically and visually to the positive class with the highest confidence among all subsets of (0, 1] of size
A
E (0, 1) is introduced to control the size of set A", theoretically, we prefer the set A" to have small size. Empirically, cannot be so small: the existing estimators are insensitive to tiny modifications (they are designed to be robust in such a way, in order to be good estimators). For example, the difference between the estimated class-priors by employing samples Su and Sp and the one by employing samples Sy and Sp can be hardly observed if Sp and Sp only differ from in one or two points. Specifically, p 10% is selected for the experiments leads to a significant improvement of the estimation accuracy. The details on the selection of the syper-parameter value will be explained in Section 4.1. The algorithm is summarized in Algorithm 1.
all datasets, which
There are two fundamental concerns for copying Á Sp- 1). When we have irreducibility, might regrouping make #' be a worse approximation? 2). When we lack irreducibility, must regrouping make 7' be better approximation? While these concerns will be formally clarified later, we give here intuitive implications of regrouping.
1). If we have irreducibility, the (AA)) should be rather small (if not zero), and Á should be drawn from the positive component P. of the mixture P.. In this case, regrouping will generally have small influence to Pp. Hence, it will not make n' worse.
2). If we lack irreducibility, A may be drawn from either P, P.. By regrouping. A becomes present Sp., which encourages the probability of the set Å' in Pp to large. This will modify Pp as we expected towards irreducibility. As a consequence, regrouping will make #' better.
3.3 THEORETICAL JUSTIFICATION
In the regrouping approach described above, the auxiliary class-conditional distribution Pp and P. are created by regrouping small set A from Pp and P... Here, we analyze the properties of regrouping and theoretically justify it.
formal definition of regrouping In order to analyze the properties, we need formally define how to split, transport, and regroup a set A (or the mass of A).
Definition 2. Let M be probability measure on measurable space (X, 6). Given a set A 6, we define a measure M on the a-algebra I as follows:
AA G, M M(S
(6)
5

---

Published as a conference paper at ICLR 2022
It is easy to that given two measures M and M obtained according to Definition 2, where A = X \ A, then M A and M A" have the following property.
Lemma 1. Let M be a probability measure over a measurable space (X, 6). For any set A E 6, we have M = A = M.
Now, we introduce the theory of regrouping. Fixing set A E G, we split Pn as PA and PA transport PA to Pp to regroup them together, i.e.,
Pa - (1 T Pn + TPp - (1 (P + P. + TPP -1- P) + ((- + PPp)
split inio two Finally, we can rewrite the unlabeled data distribution Pu as a mixture of two new class-conditional distributions Pa and Per defined in Theorem 1 by normalization.
regroup une
Theorem 1. Let Pa (1 Pa + Tpp. Let A C support(Pu).. By regrouping P. to Pp. Pa can be written as a mixture, i.e. Pa = (1 -x)),, + I T'Pp, where
n' = π + (1 (A),,
PAA
(1-x) PA toP (1 -1--7) (n +
Pe PA(AA)' Pp
and Par and Pp satisfy the anchor set axsumption.
When class-conditional distributions Pn and Pp do not satisfy the irreuucibility assumption, π cannot be obtained by using CPE methods based on MPE, which will lead an estimation bias discussed before. However, Theorem shows that the new proportion x' is always identifiable as Pa and Pp always satisfy the anchor set assumption. Thus, after regrouping, x' is identifiable and can be estimated by the existing CPE methods.
Bias reduction According Theorem I, make T' closer æ,, we expect to find the set A looks most dissimilar to the negative class, i.e., (A) is small.
P.(A): 1 from Pp Theorem 2. Let Pe and Ps: be obtained by regrouping set A := arg min Age P:
and Pn. 1). If Pu and Pp satisfy the irreducibility assumption, then n' = n; 2). if Pa and Pp dissatisfy the irreducibility assumption, then < n <T < =
Theorem 2 shows how properly select used for regrouping make n' good approximation of т. Specifically, once A is selected for regrouping, if Pn and P. satisfy the irreducibility assumption, the new estimation s' will be identical π; if Pa and Pp dissatisfy the irreducibility assumption, n' obtained by employing the distributions Pa and Pe will contain a smaller estimation bias compared to xe obtained by employing the distributions P. and Pp.
Convergence analysis For completeness, we illustrate the convergence property of ReCPE, which presented by employing the estimator proposed by Blanchard al. (2010). Let S,, Sp and Sp be the samples i.i.d. drawn from P., Pp and Ppr, respectively. Let A be the set used for regrouping. Let X - R, h E H. be a function that predicts 1 for all elements the set A and otherwise, where H denotes hypothesis space. Let |S| denote the cardinality of set S. Let 1 (h(=)==) be an indicator function which returns 1 if (xx) predicts 1 and O otherwise. Then (A) can be expressed where Pp is the density function of the distribution P.. Let P. be the empirical version of (A). i.e., (A) = S.l E [(())==1)- Similarly, let (A) be the empirical version of Pp. (A). Let the error es, n (Sa) denote the difference between (A(A) and Pa Pa(A) obtained by exploiting the empirical Rademacher complexity (Mohri et al., 2018). Similarly, let (Sp' denote the difference between Ppr (A) and Ppr (A). We have the following theorem.
as Seex Pp
Theorem 3. Let Pu (1- P, + TPp. By selecting set A and regrouping PA Pp. Then, with P((S) satisfies
probability 1 - 26, the estimated class-prion x' based on solving inf infses. ($))0 PS)
(S,)(Sw) + (A) (S))(S)))) + (A.(A) (Sp) (1 P.
(Sp)
| - m|
We have defined that the fraction tends to infinite if its numerator is larger than 0 and its denominator is 0. Additionally, the infimum may not always exist, if it does not exist, we could use sequence of sets that converges to the infimum value, but the convergence rate can he arbitrarily slow (Scott, 2015).

---

Published as a conference paper at ICLR 2022
where (5) _ 29: (H) + +3 log 2
and RH) is the empirical Rademacher complexity of H.
To make H (S) converge to 0 with the increasing of the sample size of S, a universal approximation assumption has been proposed by Scott (2015) to ensure that the hypothesis space is large enough to represent a wide variety of interesting functions. Under the assumption, Scott (2015) proved that, with increasing of the size of samples S,, and Spr, the eтror between (A( and (A( will converge
to 0 at a rate O (v log [Sa]
similarly to (A) and (A). Since the empirical Rademacher complexity (H) of a hypothesis space H can be upper-bounded by its VC-dimension (Mohri
[...
al., 201 the both errors based on the empirical Rademacher complexity will also converge to zero with increasing of the sample size. Consequently, the estimation R' Pu(4) will converge to (r)(4)
n' P.(A) Parl = ग + (1 (1- (A) at a rate O (V log(min([S.... min([Sal (1Sa1.15pr))
Computationally efficient identification of A" The following theorem presents how to identify A' with P. and Pp. Let us define another auxiliary distribution (XX, C). where C' E {0, 1 is the positive- vs-unlabeled label i.e., class label distinguishing between the positive component and the whole mixture. Specifically, priors are q(C = 1) == 1-"
x and q(C = 0) == 1 1- conditional densities
are q(X(C'== = 1) := Pp and ( = 0) == Pu; class- class-posterior probabilities are g(C = x and qC = 1|x). We have the following theorem.
Theorem 4. Let Pu and Pp be density functions of Pa and P... respectively. Let q P(C O)ps + P(C = 1)pp Let 1 : X > {0, } be the identity function which outputs 1 ifx E X is in the set A, and O otherwise. Then the set A arg arg ACe Ex =[)
For the above optimization, its objective function has two expectations over q, which can have the "exact" empirical solution obtained by replacing expectations with empirical averages: A arg migmin ACS ErEA (C=](Xmm))
EeeA (C=0 xa))
Approximation of Pp with surrogate As we do not have examples drawn from Pn, it is hard to Pd create Ppr, let alone sample from it. We approximate Pp by using Py = P.(A)+I The following proposition shows that when (A) is small, Pp is almost identical to Ppr-
PA+Pa P((AFII and (A(A) < e. Ve > 0 and G, (Pr((S) - P3.(S) S (ee).
Proposition 2. Let Ps.-
Since the gap has the same order as (A) uniformly over S, it is guaranteed that whenever Pu(A) is small, the gap is also small. Practically, we can control the parameter e in the above proposition to be small. Specifically, using a small value of the hyper-parameter p in Algorithm will lead to the set A in Theorem to be small, as well as (A).. As a consequence, the practical implementation of regrouping is good approximation of the theory of regrouping as we expected. By now, we have analyzed all of the properties of regrouping and theoretically justified all of the points in its design.
EXPERIMENTS
We run experiments on 2 synthetic datasets and 9 real word d The objectives of employing synthetic datasets are to validate whether the proposed regrouping CPE method reduces the estimation eтror of the consistent distributional distributioallassuupiooo aatnaallssumption----o free CPE method on the dataset satisfying the irreducibility assumption and does not influence the prediction of the CPE method on the dataset dissatisfying the irreducibility assumption. The datasets. The real-world datasets are used to illustrate the effectiveness of our methods. Although we have introduced a p and used approximations in the implementation, empirical results on all synthetic and weal-world datasets consistently show the superiority of ReCPE.
is also selected from the synthetic
To have a rigorous performance evaluation, for each dataset, 6 3 x 10 experiments conducted via random sampling. Specifically, we select {0.25, 0.5, 0.75) fraction of positive examples to be the
The real word datasets are downloaded from the UCL. machine learning database. Multi-class datasets are used as binary datasets by either grouping or ignoring classes.
7

---

Published as a conference paper at ICLR 2022
sample of the positive distribution Pp. We let the rest of the examples be the sample of the unlabeled distribution Pu. In such a way, 3 pairs of empirical positive and unlabeled distributions are generated. Then, we create other 3 pairs of distributions by flipping the labels of all instances in the original datasets. For each pair of distributions, we randomly draw positive and unlabeled samples with sizes of 800, 1600, and 3200, respectively, which are used input data. Note that, the positive and unlabeled samples have the same size as did in Ramaswamy et al. (2016). For each sample size, 10 repeated experiments are carried out with random sampling. For all experiments, we employ a neural network? 3 with 2 hidden layers. Each hidden layer contains 50 hidden units. The batch normalization (loffe Szegedy, 2015) is also employed. The stochastic gradient descent optimizer is used with the batch size 50. The network is trained for 350 epochs with a learning rate 0.01 and momentum O. The weight decay is set to le - 5. The model with the best validation accuracy is used to estimate the positive class-posted posterior probability P(Y = 1|x = x)). We sample the validation set with 20% of the training data size.
4.1 EXPERIMENTS ON SYNTHETIC DATASETS
We create two datasets with one satisfying the irreducibil- ity assumption while the other not. The dataset satisfying the inreuciiiility assumption is created by sampling from 2 different 10-dimensional Gaussian distributions as the com- ponent distributions. One of the distributions has means and a unit covariance matrix. Another one has unit means and unit covariance matrix. The dataset dissatisfying the ir- reducibility assumption is also created by drawing examples from 2 different 10-dimensional Gaussian distributions. One of the distributions has zero means and unit covariance ma- trix. Another one has unit means and covariance matrix. Then we remove all the data points with P(Y = 1[x) 2 0.98 or P(Y= = 1[X) S 0.02. For simplicity, in Figure 2, we name two datasets irreducible data and reducible data, respectively.
0.025
KM42 FReX322
0.020
0.015
0.005
0.000
0.10
0.15
Copy Fraction p
(a)
0.025
KM2 AAE RRKM2 AE
To validate the correctness of our method and to select a suit- able value of the hyper-parameter p, we carry out two ex- periments. The consistent CPE method KM2 is used as the baseline, which is compared to method ReKM2, i..., re- grouping version of the KM2. Firstly, we compare the mag-
0.020
0.015
0.010
0.005
nitude differences between के and #' (i.e., â #') with the
different fractions of points to be copied from the mixture sample to the component sample, which is illustrated in Fig- ure 2(a). Then we compare differences of the absolute error (i.e.. ক| | = between the baseline and our method with the increasing of the copy fractions. Note that each point in Figure 2 is obtained by averaging over 6 x 3 10 experi- ments.
0.000
0.10
Copy Praction p
(b)
Figure 2: Experiments of the hyper- parameter selection on synthetic datasets. With increasing of the copy fraction p, (a) average estimation differences between KM2 and Re- grouping KM2 (ReKM2) and (b) av- erage differences of the absolute er- ror between KM2 and Regrouping KM2 (RcKM2).
Figure 2(a) validates the correctness of our Theorem 2 and Eq. (7). Theorem 2 states that, by properly selecting the set A, on the dataset dissatisfying the irreducibility assumption (re- ducible data), TT should be smaller than the maximum propor- tion R²; on the dataset satisfying the irreducibility assumption (irreducible data). x' should be close to æ. Figure 2(a) perfectly matches this statement. It shows that, on the reducible data, the values of A' are continuously smaller than á with the copy fraction < 22.5%; on the irreducible data, A' and स have the similar values until the copy fraction 2 17.5%. According to Eq. (7), the positive bias of our estimator should become larger with the increase of P, (A). This fact reflected by the differences of म #' become smaller on both datasets when the copy fraction > 15%
Figure 2(b) illustrates the average differences of absolute error between the baseline and the proposed method. On the reducible data, our method continuously outperforms the baseline with the copy
We employ the neural network hecause has high approximation capability (Csáji al., 2001).
8

---

Published as a conference paper at ICLR 2022
AM
RaAM
1SPT.
RamP.
FN
KaFN
KM
ReKMI
KMC
REKMI
ROC
KaROX
RRI
KoRRO
ed0le (1800)
wdalt (800)
0.127
0.222
0120
00080
00890
0.108-
01089
- 0.31
0.29
182055
0.131
0.091
0.1
0.72
011
0.126
01121
0.104
0.000
0.123
0.135
0.134
0.137
ed33l t (3200)
10.105
0.088
030054
00057
0.200
0.279
0.054
0.04.
0'082
10080
13099
01000
0.114
0.128
1 (8020)
0.108
0.1:2
0 1.129
0.147
0.447
03.422
00.105
0.075
0:104
0.081
002015
0.228
0.111
111
0.104
0.084
0.430
0.418
010186
0.076-
0.108
0.092
0.191
0.36
0.121
avila (3200)
0,156
0.133
0.05+
0.001
0.436
0.42
0.092
0.078
0.112
0.093
0.131
0.095
0.121
0.122
bank (800)
0.136
0.116
0.116+
0.132
0.282
0.2664
0.395
0.216
0.206
0.238
0.163
0.15
0.163
0.185
bamx (10000)
0.117
0.107
0.087:
0.305
0.202
0.244
11.178
03:128
0:200
0.198
0.129
0.118
0.137
0.161
(3200))
0.073-
0.248
0.297
01124
0.00
0.35
0.093
0.106
0.159
0.18
caEd (500)
0.127-
0.174
0,161
0.465
0.
01.295
0.176
0:208
0.156
0.217
0.233
0.117
caled (1600)
0.175
0.14
0.14
0.4
0.450
0.437
0.10
0.135
0:150
0.129
0.191
0.163
0.126
0.155
maam (1200)
0.104
0.134
0.127
0.12
0:455
0-435
0.101
0:115
09142
0:122
0.130
0.152
0:11
D: T08
CovEype (800)
0.16
0.123
0.155
0.151
0.367
0.345
0:157
0.142
0.129
0:13
0.291
0.256
0.116
0 1010
covt.ype (1600)
0.552
0,709
0.364
0.116
0.115
0.121
0.123
0,150
0.101
covttyp (3200)
0.125
6.09
0.005
0.083-
0.354
0.53 3.34
0'087
0.100
0.124
01128
0:157
0.113
0.100
107
*gg (M00)
0.165
0.106-
6:218
0.225
0.303
0.305
0.175
012204
0.119
0:131
0.476
0.3336
0.171
0. 0.124
#ge (1600)
0.137
0.12
0.121
0.142
0.486
0.388
0.2234
01:214
0:116
0.108
0.3115
0.2338
0.151
01:14
aqe (3200)
0.125
0.057+
0.485
0.059
0.22
0.193
0.134
0.113
0.165
0.139
0.142
0.102
neg negiie04 (800)
0.077
0.072
0.071
0.312
0.111
0.1
0.071
0.064
0.341
0.124
0.065
0.064.
magg200 (1000)
0.071
0.056
0.044
0.0435
0.292
0.274
0.088
0.072
00079
0.066
0.1
0.073
DO0GS
0.062
magico0 0 (3200)
0,009
0.054
0.035+
0.006
0,274
0.258
0.07
0.047
0.085
0.063
0.085
0.047
0.004
0.052
zebot (800)
0.053
0.082
0.039
0.047
0.19
0.187
0,252
0.215
0.111
0.114
0.119
0.344
0.077
0.088
raboot (1600)
0.058-
0.087
0.034
0.130
0.33
0.15
0.141
0.098
0.29
0.075
0.076
0.070
rolott (3200)
0.052
0.033
0.150
0.119
00081
0.085
0.079
0.077
0108
0184
0.003
0.043
0.06
0.000
chutcle (800)
0.083
0.031
0.016
0.02
0.041
0.033
0.058
0.083
0.035
0.03
0.042
0.017
0.035
0.051
Le (1600)
0.008
0.045
0
0008
000
0.034
0.048
0.079
0.023
0.05
0.029
0.03
0.028
0.050
le
0.07
averaae
0.116
0.
0009
0.092-
0.311
0.148
0.121
DITT
0.111
0.157
0.136
0.105
Table 1: Absolute estimation errors on real-world datasets. The first column provides the names of the datasets and sample size. We bold the smaller average estimation errors by comparing each baseline method with its regrouped version. The smallest average estimation error among all methods for each row is highlighted with s. The last row obtained by averaging the results of all experiments. Variances and the results of Wilcoxon signed-rank test are reported in Appendix B. The proposed Regrouping methods are significantly better than most of the baselines.
fraction < 22.5%. However, the differences of average absolute error start to decrease with the copy fraction > 15%. On the irreducible data, the differences of average absolute error are close to zero until the copy fraction > 15%.
By observing Figure 2, we found the prediction of the KM2 estimator will not change much if the copy fraction p is too small. For example, the difference between the estimated mixture proportion by employing samples Sn and Xp and the ones by employing samples Sn and X can be hardly observed if Xp and X only differ from in one or two points. For simplicity and consistency, we select hyper-paramene parameter to be 10% for all the following experiments.
4.2 EXPERIMENTS ON REAL-WORLD DATASETS
We illustrate the absolute estimation errors of different estimators on the real-world datasets. Totally, 7 baseline methods are used in the experiments, which are AlphaMax (AM) (Jain et al., 2016). DEDPUL (DPL) (Ivanov, 2019), Elkan-Noto (EN) (Elkan & Noto, 2008), KMI, KM2 (Ramaswamy et al., 20, ROC (Scott, 2015), and Rankpruning (RPG) (Northcutt et al., 2017). By using our method, the regrouped version of them are implemented, which are called ReAM, ReDPL,, ReEN, ReKMI, ReKM2, ReROC, and ReRPG. In Table 1, we compare the absolute estimation errors of each baseline with those of its regrouped version on different datasets with different sample lengths. Each number in Table is the average over 6 x 10 experiments.
Table 1 reflects the effectiveness of our regrouping CPE method. Overall, by using our method, the estimation accuracy is increased for most of the popular CPE methods among most of the datasets with different sample lengths. By observing the last row, the regrouped version of the estimators has much smaller average estimation errors except DPL, KM2, and RPG. On the real- world datasets, Regrouping AlphaMax (ReAM) has the smallest average estimation error among all methods.
5 CONCLUSION
In this paper, we investigate how to reduce the estimation bias of the distributional awsumpttosss -ree CPE method without irreducibility assumption for PU learning. We have proposed regrouping CPE which can be employed on top of most existing CPE methods. We have also theoretically analyzed the estimation bias of ReCPE. Empirically, it improves all popular CPE methods various datasets. One future work will focus on how to generate a sample from Pp instead of using an approximation.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
For theoretical results, we have clearly explained any assumptions. A complete proof of the claims can be founded in the appendix. We have also included an anonymous source code in our supplementary material. For any datasets used in the experiments, complete description of the data processing steps is provided In Section 4.
ACKNOWLEDGMENTS
TL. was partially supported by Australian Research Council Projects DP180103424, DE-190101473. IC-190100031, and 1P-220102121. BH was supported by the RGC Early Career Scheme No. 22200720 and NSFC Young Scientists Fund No. 62006202. MG is supported by ARC DE210101624. GN and MS were supported by JST AIP Acceleration Research Grant Number JPMJCR20U3, Japan. MS was also supported by the Institute for Al and Beyond, UTokyoo
REFERENCES
Yingbin Bai, Brkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.
Jessa Bekker and Jesse Davis. Estimating the class prior in positive and unlabeled data through decision tree induction. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: survey. Mach. Learn., 109(4):719-760 2020.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. Journal of Machine Learning Research, 1 (NNov):2973-3009 2010.
Marthimus Christoffel, Gang Niu, and Masashi Sugiyama. Class Class-prior estimation for learning from positive and unlabeled data. In Asian Conference on Machine Learning, pp. 221-236, 2016.
Mare Claesen, Frank De Smet, Pieter Gillard, Chantal Mathieu, and Bart De Moor, Building classifiers to predict the start of glucose-lowering pharmacotherapy using belgian health expenditure data. arXiv preprint arXiv: /5Xv005504 2015.
Balázs Csanád Csáji et al. Approximation with artificial neural networks. Faculty of Sciences, Ervs Lornd University, Hungary, 24(48):7, 2001.
Francesco De Comité, François Denis, Rémi Gilleron, and Fabien Letouzey. Positive and unlabeled examples help learning. In International Conference on Algorithmic Learning Theory, pp. 219-230. Springer, 1999.
François Denis. Pac learning from positive statistical queries. In International Conference on Algorithmic Learning Theory, pp. 112-126. Springer, 1998.
Marthinus du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In International conference on machine learning, pp. 1386-1394, 2015.
Marthinus C du Plessis. Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Advances in neural information processing systems, pp. 703-711, 2014.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 213-220. ACM, 2008
Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. Fast rule mining in ontological knowledge bases with amie. The VLDB Journal, 24(6)7707-73, 2015.
10

---

Published as a conference paper at ICLR 2022
Chen Gong, Hong Shi, Tongliang Liu, Chuang Zhang, Jian Yang, and Dacheng Tao. Loss decompo- sition and centroid estimation for positive and unlabeled learning. IEEE transactions on pattern analysis and machine intelligence, 2019.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773, 2012.
Yu-Guan Hsieh, Gang Niu, and Masashi Sugiyama. Classification from positive, unlabeled and biased negative data. In International Conference on Machine Learning, pp. 2820-2829, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Dmitry Ivanov. Dedpul: Method for mixture proportion estimation and positive-unlabeled classifica- tion based on density estimation. arXiv preprint arXiv: 1902.0696,, 2019.
Shantanu Jain, Martha White, Michael Trosset, and Predrag Radivojac, Nonparametric semi- supervised learning of class proportions. arXiv preprint arXiv: 1601.0994,, 2016.
Masahiro Kato, Liyuan Xu, Gang Niu, and Masashi Sugiyama. Alternate estimation of classifier and the class-prior from positive and unlabeled data. arXiv preprint
2018.
Ryuichi Kiryo, Gang Niu, Marthinus du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non non-negative risk estimator. In Advances in neural information processing systems, pp. 1675-1685, 2017.
Yongchan Kwon, Wonyoung Kim, Masashi Sugiyama, and Myunghee Cho Paik. Principled ana- lytic classifier for positive-unlabeled learning via weighted integral probability metric. Machine Learning, pp. 1-20, 2019.
Wee Sun Lee and Bing Liu. Learning with positive and unlabeled examples using weighted logistic regression. In ICML, volume 3, pp. 448-455, 2003.
Fabien Letouzey, François Denis, and Rémi Gilleron. Learning from positive and unlabeled examples. In International Conference on Algorithmic Learning Theory, pp. 71-85. Springer, 2000,
Xiaoli Li and Bing Liu. Learning to classify texts using positive and unlabeled data. In IJCAI, volume 3, pp. 587-592, 2003.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting, IEEE
Transactions on pattern analysis and machine intelligence, 38(3)447-461,, 2015.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
Arvind Neelakantan, Benjamin Roth, and Andrew McCallum. Compositional vector space models for knowledge base completion. ACL, 2015.
Gang Niu, Marthinus Christoffel du Plessis, Tomoya Sakai, Yao Ma, and Masashi Sugiyama. Theo- retical comparisons of positive-unlabeled learning against positive-negative learning. In Advances in neural information processing systems, pp. 1199-1207, 2016.
Curtis G Northcutt, Tailin Wu, and Isaac L Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. stat, 1050:9, 2017.
Harish Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In International Conference Machine Learning, pp. 2052-2060, 2016.
Yafeng Ren, Donghong Ji, and Hongbin Zhang. Positive unlabeled learning for deceptive reviews detection. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 488 488-498, 2014.
11

---

Published as a conference paper at ICLR 2022
Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Semi-supervised optimization based on positive-unlabele learning. Machine Learning, 107(4):777-79, 2018.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In Artificial Intelligence and Statistics, pp. 838-846, 2015.
Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. In Conference On Learning Theory, pp. 489-511, 2013.
Ugo Taniclian and Flavian Vasile. Relaxed softmax for pu learning. In Proceedings of the ACM Conference on Recommender Systems, pp. 119-127, 2019.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7610, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint 10Xiv:2006.00445. 2021.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. Advances in neural information processing systems, 33:7260-787 7271, 2020.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under structural causal model. Advances in Neural Information Processing Systems, 34, 2021.
Maria A Zuluaga, Don Hush, Edgar JF Delgado Leyton, Marcela Heráández Hoyos, and Maciej Orkisz. Learning from only positive and unlabeled data to detect lesions in vascular et images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 9-16. Springer, 2011.
12


Reviews:
[{"review_id": "w202zp0LeX5", "reviewer": "Reviewer_EaHM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper is well-written and easy to follow. The mathematical contents are not hard to parse and examples are given. \n- This work is well-motivated and novel. The paper addresses the overestimation problem of $\\pi$ in PU learning by proposing a regrouping technique.\n- Both the theoretical and experimental proofs are given. They help support the authors' claims.\n- Although the gain is small in absolute value, the large number of repeat experiments makes it statistically significant (and statistical tests are included). \n\n**Weaknesses and Clarifications**\n- The authors exclusively compare the estimation of $\\pi$. While I understand that the method is aimed at estimating the prior, I think that it would be beneficial if the authors can provide the PU learning results with different priors (fixed, CPE, ReCPE, etc. ). A simple nnPU would suffice. \n- Many \"irreducibility\" has a typo (\"irreduciblility\").\n", "summary_of_the_paper": "This paper studies the prior $\\pi$ in PU learning. More specifically, it studies the estimation method of the prior $\\pi$ in a more general case, i.e., without the irreducibility assumption. When unsatisfied, such an assumption will often lead to the overestimation of $\\pi$. However, current class-prior estimation methods are usually based on the irreducibility assumption. The authors propose a new CPE problem based on a new auxiliary distribution that always satisfies the irreducibility requirement. The technique is called regrouping, which transforms the true distributions to  Both theoretical and experimental results are included in the paper, showing the validness of the proposed ReCPE method. ", "summary_of_the_review": "The authors give an insightful demonstration of limitations of existing CPE methods, and also give feasible solutions to the limitations. Theoretical proofs are given, and experiment results show significant superiority of the proposed method compared to existing CPE methods. So I believe this work would be beneficial to the community and therefore give a score of 8. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "rwn3M0pjTS", "reviewer": "Reviewer_8SWo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+++Strength:\n1. PU learning is apparently an important machine learning problem, which has the potential to be used in many real-world applications.\n2. The analyses on the prior studies appear reasonable.\n3. Experiments have been conducted to support the proposed method and the results appear good.\n\n\n---Weakness:\n1. While the proposed probability distribution transformation makes sense to avoid relationship between the prior of positive data and that of negative data, the proposed method appear pretty simple. This might not be a weakness, but methodology is not a strength of this paper.\n2. A big gripe of this paper is its writing. There exist too many grammatical/syntactical mistakes. Here I list few in the below.\n\nPage 2: \"To let these distributional-assumption-free methods can be used to identify class-prior $\\pi$, $\\kappa^*$ must ...\". This sentence can be definitely written in a better way.\n\nPage 2: \"Because the irreduciblility assumption is impossible to check without making any assumption on $P_n$. Thereby, in ...\". This is an incomplete sentence, which is a mistake.\n\nPage 3: \"... by creating a new auxiliary distribution $P_{p'}$ always guaranteeing the irreduciblility assumption...\" Here using an attributive clause is much better than using a present participle.\n\nPage 4: \"... i.e., there exists a set can achieve the minimum 0, ...\" This sentence is simply wrong.\n\n3. As we all know, deep learning has become the main stream of machine learning in the past decade, which is particularly true in ICLR. Many research on PU learning using deep neural networks have been proposed and demonstrated good performance. As a result, without comparing to such methods, it is harder for reader to appreciate the effectiveness of the paper.", "summary_of_the_paper": "This paper studies positive and unlabeled (PU) learning, which apparently is an important problem in machine learning. In many existing researches, the PU classifiers assumes that the irreducibility assumption holds on input data, which, though, may not always be true in many real-world applications. As a result, the learn classifier might not work well due to estimation bias on data prior. To address this problem, this paper proposes to construct a transformed the probability distributions of input data by a re-grouping operation, such as the positive prior cannot be a component (or called as a support) of the negative prior. Good experiment results have been obtained to support the proposed method.", "summary_of_the_review": "This paper studies positive and unlabeled (PU) learning by constructing a transformed the probability distributions of input data by a re-grouping operation, such as the irreduciblility assumption holds on the data to the subsequent classifier. Overall, this paper is clearly written. But because of the following concerns, also mentioned above, I suggest the authors to further improve this paper before publication:\n1. Please proofread this manuscript carefully to improve the writing.\n2. More comprehensive experiments to compare deep neural network based PU learning should be added to convince the readers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WeYTEzG93xE", "reviewer": "Reviewer_6Jj8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This paper introduces a particular situation in which the irreducibility assumption cannot be satisfied, which has not been studied (even considered) in previous PU learning literature.\n\n2. The idea of \"regrouping\" is novel. The authors give a very simple and intuitive example in Subsection 3.1 as the motivation of the proposed method, which is very interesting and helps the readability of this paper. In Subsection 3.3, some theoretical results are introduced and these results can help the readers to understand the proposed \"regrouping' idea.\n\n3. The experimental results on synthetic data validate the correctness of the theoretical part and show the effectiveness of the proposed method in \"reducibility\" setting.\n\nWeaknesses:\n\n1. The experiments on real-world data use same  hyper-parameter $p$ across all datasets. Considering that this paper actually aims to address the hyper-parameter (class-prior $\\pi$) selection problem and the proposed method introduces another hyper-parameter $p$ (which seems to have significant impact to the performance), the parameter sensitivity analysis is needed in experiment. \n\n2. More related work about \"irreducibility\" can be introduced before Section 3. I don't know is there any solution for the irreducibility problem in mixture proportion estimation after reading the whole paper.\n\nOther comments:\n\n1. Is the indicator function really needed in \"Convergence analysis\" part? It seems that $\\mathbb{1}_{\\{h(x)=1\\}}$ is equivalent to $h(x)$.\n\n2. Can you explain that why \"In this case, regrouping will generally not change $P_p$ but only increase number of examples in $P_p$\".\n\n3. Typos: \"diffferent\" --> \"defferent\", \"the trained classifier a h\" --> \"the trained classifier h\".", "summary_of_the_paper": "This paper addresses the problem of class-prior estimation (CPE) in positive-unlabeled (PU) learning. The authors consider that the existing methods usually fail if the data distribution dissatisfies the irreducibility assumption. To address this problem, the authors introduce a method named Regroping CPE (ReCPE) which tries to transform the original data distribution to an auxiliary data distribution, such that the produced distribution always guaranteeing the irreducibility assumption. \n\nA practice implementation is proposed by firstly training a binary classifier and then picking some samples whose outputs are mostly dissimilar to the negative examples as the pseudo positive samples. The proposed methods can be considered as a pre-processing method and used together with any CPE method. The experiments on synthetic data show the intuition of the proposed ReCPE and the results on real-world data show the effectiveness of the proposed method when it is combined with other CPE methods.", "summary_of_the_review": "This paper considers a particular PU learning setting where the irreducibility assumption is not satisfied, and gives a new method to address this problem. The proposed method can be combined with any class-prior estimation method and the experimental results show positive effect of the proposed practical implementation. This paper is well-motivated and novel, and it will be better with the weaknesses addressed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5xqz8xhvli", "reviewer": "Reviewer_JBHN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n1.Compared with the previous class-prior estimation methods, the authors consider the practical problems in the actual scene, that is, the distribution of the data set generally does not meet the ideal conditions. This is a relatively practical research topic.\n2.In general, this article is well written, which is reflected in the clear organizational structure and concise and easy-to-understand presentation. The summary of related work is very informative.\n3.The experimental design is generally reasonable, and the experiments with real -world data sets give more substantial results.\nweaknesses:\n1.Why is case-control PU learning more general than censoring PU learning? There is no clear explanation about this issue, please give a more detailed explanation in the part Introduction.\n2.Regarding the main research scenarios in this article, that is, \"assuming that the support of the positive class-conditional distribution is contained in the support of the negative class-conditional distribution\", the article gives a theoretical definition. Can the author give a more intuitive explanation? Please give some more examples of this in actual problems. In what scenarios will such distributed data appear?\n3.Why choose the most likely positive sample among the unlabeled samples? If the support of the positive class-conditional distribution is contained in the support of the negative class-conditional distribution, then intuitively these contained positive samples should look more like a negative sample.\n4.In the ReCPE algorithm, the author proposes to find the most unlikely negative data among the unlabeled samples. The way to achieve this step is to train a binary classifier with the unlabeled sample and positive sample by treating unlabeled sample as a negative sample. When there are many positive samples in unlabeled samples, the \"least unlikely negative sample\" selected by the binary classifier obtained by this method is unreliable. Thereby the reliability of the next steps cannot be guaranteed.\n5.The author mentioned that 10% was chosen as the value of the hyperparameter p in all experiments of the data set, but this approach seems to be contrary to Theory 1. This approach is too casual, I am worried that the new class-conditional distribution, which is obtained by transporting the probability mass of the set A from the negative class to the positive class, are really guaranteed to satisfy the irreduciblility assumption?\n6.Why assume that the positive class-priorπis close to the new positive class-priorπ’? If the support of the positive class-conditional distribution is strongly contained in the support of the negative class-conditional distribution, The two class-priors should be quite different.\n7.In general, although this research scenario is very meaningful, the ReCPE method proposed in this article is too strategic. It only proposes solutions intuitively, and there seems to be no strong guarantee in theory. The authors may be able to strengthen this article from this perspective.\nMinor comments: The spelling of \"real word\" in Note 2 on page 7 is incorrect.\n\n", "summary_of_the_paper": "This paper studies a relatively little-concerned problem, that is, the problem of class-prior estimation(CPE) in PU scenes. The authors concluded that the existing CPE methods are based on a critical assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution. However, in actual scenarios, this assumption may not be satisfied. It is also difficult to prove that a certain data set satisfies this assumption. Existing CPE methods will systematically overestimate the class prior when the data does not meet the critical assumption. To remove the assumption to make CPE always valid, a strategy called Regrouping CPE (ReCPE), which builds an auxiliary probability distribution, are proposed so that the support of the positive data distribution is never contained in the support of the negative data distribution. Theoretically, this method can give a more accurate estimate regardless of whether the dataset meets the assumption. The authors also proved the effectiveness of the method through a series of experiments.", "summary_of_the_review": "This article focuses on practical issues, and its writing style is clear and easy to read. The structure of the article is reasonable, and the experiments are full and practical.\nHowever, there are many parts that need to be improved in the method part, which are listed in Main review. In general, although this research scenario is very meaningful, the ReCPE method proposed in this article is too strategic. It only propose solutions intuitively, and there seems to be no strong guarantee in theory.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received a majority vote for acceptance from reviewers and me. I have read all the materials of this paper including manuscript, appendix, comments and response. Based on collected information from all reviewers and my personal judgement, I can make the recommendation on this paper, *acceptance*. Here are the comments that I summarized, which include my opinion and evidence.

**Research Motivation and Problem**

This paper is well motivated by the agnostic of CPE assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution. To tackle this problem, the authors built an auxiliary probability distribution such that the support of the positive data distribution is never contained in the support of the negative data distribution.

**Technical Contribution**

The technical part is simple and clear. The regrouping idea is also easy to implement. The theoretical justification is a good complement of the proposed ReCPE algorithm.

**ReCPE does not affect its base if the assumption already holds**

The authors employed the synthetic datasets to verify this point. This is a plus.

**Experimental Results**

The authors demonstrated their ReCPE algorithm can be used as a booster on seven base PU classifiers.

**Presentation**

The presentation has been much improved with the guidance of one reviewer. But I found two extra minor ones. (1) "PU Learning" -> "PU learning" at the beginning of the second paragraph on Page 1. (2) Two typos in "9 real word datasets." on Page 7 -> "9 real-world dataset.", where the footnote should be placed after the period.

**Layout**

(1) Too many lines in Table 1. It is suggested to remove the horizontal lines among the same dataset, (2) Appendix should go after the main manuscript, rather than a separate file.

No objection from reviewers was raised to again this recommendation.
Decision: Accept (Poster)