PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference
Authors: Xutao Wang
Hanting Chen
Tianyu Guo
Yunhe Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: PU learning
causal inference
semi-supervised learning

Abstract: Positive-Unlabeled (PU) learning aims to achieve high-accuracy binary classification with 
limited labeled positive examples and numerous unlabeled ones. Existing cost-sensitive-based 
methods often rely on strong assumptions that examples with an observed positive label were 
selected entirely at random. In fact, the uneven distribution of labels is prevalent in 
real-world PU problems, indicating that most actual positive and unlabeled data are subject 
to selection bias. In this paper, we propose a PU learning enhancement (PUe) algorithm 
based on causal inference theory, which employs normalized propensity scores and normalized 
inverse probability weighting (NIPW) techniques to reconstruct the loss function, thus 
obtaining a consistent, unbiased estimate of the classifier and enhancing the model's 
performance. Moreover, we investigate and propose a method for estimating propensity scores 
in deep learning using regularization techniques when the labeling mechanism is unknown. 
Our experiments on three benchmark datasets demonstrate the proposed PUe algorithm significantly 
improves the accuracy of classifiers on non-uniform label distribution datasets compared to 
advanced cost-sensitive PU methods. Codes are available at https://github.com/huawei-noah/Noah-research/tree/master/PUe and https://gitee.com/mindspore/models/tree/master/research/cv/PUe.

PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference
Xutao Wang, Hanting Chen, Tianyu Guo, Yunhe Wang"
Huawei Noah's Ark Lab.
(xutao. wang, chenharting, tesiiinniiiinnnnn tianyu. guo, yunhe. wang @huawei com,
Abstract
Positive-Unlabeled (PU) learning aims to achieve high-accuracy binary classi- fication with limited labeled positive examples and numerous unlabeled ones. Existing cost-sensitive-base amples with an observed positive label were selected entirely at random. fact, the uneven distribution of labels is prevalent in real-world PU problems, indicating that most actual positive and unlabeled data are subject to selection bias. In this paper, propose a PU learning enhancement (PUe) algorithm based on causal inference theory, which employs normalized propensity scores and normalized inverse probability weighting (NIPW) techniques to reconstruct the loss function, thus obtaining consistent, unbiased estimate of the classi- fier and enhancing the model's performance. Moreover, we investigate and propose a method for estimating propensity scores in deep learning using reg- ularization techniques when the labeling mechanism is unknown. Our exper- iments on three benchmark datasets demonstrate the proposed PUe algorithm significantly improves the accuracy of classifiers on non-uniform label distri- bution datasets compared to advanced cost-sensitive PU methods. Codes are available at htttts:///// http://///...aooooooo/aaaaaa//aa researchh tree/ cree/master/PJe and http:///ttt.teeceooooom////tt//p
methods often rely strong assumptions that ex-
PV/PU
Introduction
In the era of big data, deep neural networks have achieved outstanding performance across various tasks, even surpassing human performance in many instances, particularly in traditional binary classification problems. The success of these deep neural networks often hinges on supervised learning using large quantities of labeled data. However, in reality, acquiring even binary labels can be challenging. For instance, in recommendation systems, users' multiple clicks on films may be considered as positive samples. Nonetheless, all other films cannot be assumed uninteresting and thus should not treated as negative examples; instead, they should be regarded as unlabeled ones.
The same issue emerges in text classification, where it is typically more straightforward to define partial of positive samples. However, due the vast diversity of negative samples, it becomes difficult or even impossible to describe a comprehensive set of negative samples that represent all content included in the positive samples. Similar situations occur medical diagnosties, malicious URL detection, and spam detection, where only a few labeled positives are available amidst plethora of unlabeled data. This scenario is variant of the classical binary classification setup, known as PU. In recent years, there has been growing interest in this setting. Positive- Unlabeled (PU) learning primarily addresses the challenge of learning binary classifiers solely from positive samples and unlabeled data.
*Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In previous research, numerous PU algorithms have been developed, with cost-sensitive PU learning emerging as a popular research direction. Methods such [99 18 reweight positive and negative
risks by syper-parameters and minimize it. In addition, Self-PU introduces self self-supervision to
nnPU through auxiliary tasks, including model calibration and distillation; ImbPU 22 oversamples
and modifies sample weights to address unbalanced data. Dist-PUfE PUSSPP224] corrects the negative preference
of the classification model through prior information. PUSB [1] maintains the order-preserving assumption. However, these methods necessitate the assumption that their checked set is uniformly sampled from the population, or else the PU learning risk estimator ceases to be an unbiased or consistent estimator, otherwise resulting in reduced model accuracy.
In reality, the labeled is often biased and does not conform to the Selected Completely At Random (SCAR) assumption [I which posits that the observed labeled examples are a random subset of the complete set of positive examples. Consequently, the labeled set and replace it with more general assumption about the labeling mechanism: the probability of selecting positive examples be labeled depends on their attribute values, known as the Selected At Random (SAR) assumption (10)
essential to relax the assumption of
The work most closely related us is the paper 7 by W Gerych et al. The focus of this article is to de- velop a method for generating identifiable propensity scores. Moreover, this paper proves that the general propensity score is unidentifiable in any standard PU data assumption except for the probability gap scenario. In this paper, two sets of assumptions are proposed to estimate propensity scores: one is Local Certainty and the other is Probabilistic Gap. Under the local certainty hypothesis, there is no overlap between the positive and negative hypotheses, p(x*ly = 0) = O. Under the Probabilistic Gap assumption, it is assumed that e = k p(y = 1|z) is linear and there is an anchor point.
Labeled Positive Samples
Unlabeled Negutive Samples
Unlabeled Positive Samples
Clastiication Boundary
Tradition PU
Progosed
Figure 1: Our method and traditional PU method classification diagram, our solution uses reweighting to make the classification plane more accurate.
However, the above assumptions are too strict. We show in the appendix that the estimated propensity scores can- not be completely unbiased. To address the aforemen- tioned problem, we propose a causal-inferencc--nspiree PU learning framework, termed PUe. Our approach aims to weight the original PU learning risk estimator under biased conditions, thereby obtaining new unbiased consistent PUe learning risk estimator, which achieved through the inverse probability weighting technique of known normalized propensity scores. Given that propensity scores for samples are typically unknown, apply regularization techniques to the deep learning classifier to estimate the propensity score of each sample in the marker set. An illustration of the proposed method is shown in Figure | We also provide an estimate of the generalization error bound for empirical risk minimization. Experimental results demonstrate that the proposed PUe framework be applied to advanced PU algorithms, such as PUbN and Dist-PU, to enhance their performance in the SAR scenario. Our main contributions are summarized follows:
We relax our assumption of local determinism.
We use deep learning model to estimate propensity scores instead of a linear model.
Our propensity score estimation method can be extended to the case of negative classes (e.g. PUbNe).
Our new algorithm can simply be coupled with most cost-sensitive algorithms to improve perfor- mance. The PUe algorithm achieved superior results over existing solutions on standard benchmarks (MNIST, CIFAR10) and the Alzheimer's Disease Neuroimaging Initiative (ADNI) database.
2 Methodology
In this section, we conduct a thorough review of existing PU algorithms, examining their limitations and challenges, particularly the context of biased labeling scenarios that result diminished accuracy. To address these issues, we introduce a novel PUc algorithm, specifically designed to

---

overcome the aforementioned limitations, thus enhancing the performance of PU algorithms under biased labeling conditions.
2.1 Review of PU classification
In standard PN classification, let z E IRd and E {+1, be the input samples and its corre- sponding labels. We are given positive data and negative data that are sampled independently from pr(æ)) p(ærly = +1) and pv (x) = p(aly = -1) as XP {=ff J: and XN = {a: y We denote the class prior probability as π = p 1). where we follow the convention assume л as known throughout the paper [66.. In fact, in the paper [14] by S Jain et al., it is pointed out that class prior can be estimated in the case of the proportion of classes that are biased in the labelling data. Let g R2 IR be the binary classifier and 0 be its parameter and the L R (+1,-15 - R+ be loss function.
Let us denote by Rp(g. +1) = Eg-pp (a)) [L(g(z, +1))I, Rx(g.-1) = Ex-pw(=) (a)) [gg - 1))] Re(g, +1) ) is the expected risk loss of a positive sample and Rx (g, -1) is the expected risk loss of a negative sample.
The risk of classifier 9. RyN (g) can be approximated by:
( = n (r +1) + (1 - -T) RN -1);
(1)
where Rr(g,+1) +1) E L(g(=))) +1) and (g minimizing (g). We obtain the ordinary empirical risk minimizer 9PN-
E.h L(gao") -1). By
ur
Bx
In standard PU classification,, instead of N data XN. we only have set of U samples drawn from the { ) ~ x(æ).. Because of the fact p(æ:) <pp (x) + (1-111)))) ())))))))
marginal density p(æ), Xv
we have the unbiased risk estimator, which key idea is to use the following equality:
-1 1) 11) Rv(g -1) Rr -1).
(2)
where Rg, 11 Rus) [g((())) -1] and Rp(g. -1) = B (eppp(a) ((g(())) -1)]. As result, we can approximate the unbiased classification risk (1) by:
(epgg,+11 +1) + Ry (g. -1) 10---110 RRpp. -1), ( = n RR
(3)
where Rp(g, -1) up E. L(g(=)). 1) and Ru -1. nu En (g(x=)) -1)). By mini- mizing (9) obtain the ordinary empirical risk minimizer grU.
In theory, the risk (1 Rg 1) Ry (g, 1) x RRrgg is also positive. However, the model of is too flexible, RPo (ÔPU) indeed goes negative and the model severely overfits the
training data. We use the non-negative risk estimator for PU learning to alleviate overfitting/I6:
2.2 PU Classification in Biased Scenarios
PU learning mostly assumes that it runs under an ideal situation, that is, all labeled samples are selected completely at random from all positive samples, which is also called The Selected Completely At Random (SCAR).
Labeled examples are selected com- pletely at random, independent from their attributes, from the positive distribution. The propensity score e(x), which is the probability for selecting a positive example, is constant and equal the label frequency:
Defintion 1. SCAR(Selected Completely At Random 66:
<(x: = p() p(== |2 = p(s 1y
However, many PU learning applications suffer from labeling bias. The Selected Completely At Random (SCAR) assumption does not conform to realtity. For example, whether someone clicks
3

---

on sponsored search ad is influenced by the position in which it is placed. Similarly, whether patient with a disease will see doctor depends on his vocio-economic status and the severity of his symptoms. The Selected At Random (SAR) assumption, is the most general assumption about the labeling mechanism: the probability for selecting positive examples to be labeled depends on its attribute values [1.
The labeling mechanism depends on the values of the attributes of the example. However given the attribute values, it does not depend on the probability of the example being positive. Instead of assuming a constant probability for all positive examples to be labeled, it assumes that the probability is function of subset of an example's attributes: e(x) = p(s s= = 1|x, = 1).
Defintion 2. SAR(Selected At Random [3):
In biased labeling scenarios, the accuracy of common PU algorithms decreases significantly. So we need to develop the PUe algorithm. The mathematical representation of equation is exactly the same as that of the propensity score in causal inference. Causal inference is the process of determining the actual and independent effects (effects) of given phenomenon (cause) within a larger system. Causality inference can give evidence of causality model established by causal reasoning. Causal inference is therefore fundamentally a missingdata problem [13, which is very similar to the PU problem. The PU problem is essentially classification problem of missing negative labels.
What we're re're actually using here reverse causal reasoning. In general. we study the average causal effect of treatment. However, we first complete the causal analysis of the processing (i.e. selection) and the label itself, and then use the known causal analysis, combined with the selected samples and the unselected samples to determine the distribution of positive samples. In this paper, accept processing means that the sample is selected for labeled. And the result is refers whether the sample itself is positive sample. Without noise in the detection, whether the sample itself is positive sample is determined at the time when the sample is generated, and the behavior of selecting is performed after the sample is generated. In causal inference, what happens after the result is produced has no effect on the result, i.e. whether the sample is selected has a causal effect on the label value is O. In essence, individual causality is also O. Based on such simple causality, we use the method of reweighting the sample to estimate the positive probability of the sample.
Since the causal effect of the labeling mechanism on the sample category is zero, known causality can be utilized to improve the performance the case of biased PU problems. Rubin Causal Model of (RCM) [12] causal inference are extensively used within Statistics, Political Science, Economics, and Epidemiology for reasoning about causation. Due to the presence of confounders, estimating the average treatment effect directly using the average of observed treatment/control outcomes would include spurious effect, which is called selection bias, brought by the confounders. A key concept in RCM is balancing score [1I. reweighting samples by which is one of the most effective methods to overcome selection bias. Besides, propensity score special case of balancing score. The propensity score is defined as the conditional probability of treatment given background variables 222. Propensity score is one of the core concepts of assignment mechanism in RCM. Propensity scores play a central role in observational studies of causality. Propensity scores can be used to reduce selection bias by equating groups based on these covariates, which is called inverse propensity weighting (IPW) 211.
In causal inference, inverse probability weighting (IPW) 21 is standard statistical technique about propensity score for calculating statistics standardized to a ppeudo-population different from that in which the data was collected. Study designs with a disparate sampling population and population of target inference are common in application. Weighting, when correctly applied, can potentially improve the efficiency and reduce the bias of unweighted estimators. In the context of causal inference and survey methodology, propensity scores are estimated (via methods such as logistic regression, random forests, or others), using some set of covariates. These propensity scores are then used as estimators for weights to be used with Inverse probability weighting methods. We incorporate the propensity score when learning in a PU setting by using the propensity scores to reweight the data. The examples are weighted with the inverse of their propensity score.
crucial difference with the propensity score from causal inference is that our score is conditioned on the class being positive. IPW cannot be applied when working with positive and unlabeled data, because we have zero probability for labeling negative examples. But we can only do a weighting on positive. For each labeled example (x,, s = 1), which has a propensity score es, there are expected

---

to be 1 positive examples. So we can use normalized inverse probability weighting on positive to e: emtimate the unbiased expectations on positive.
Rp(g, +1) = Eer-p(tr +1)) (e d]. (5)
up
Rgp +1) _ ē(xx²)
((g(x)))+1)
(6)
Similarly, Rr(g, -1) = Eæ (e) [1g(2, -1)) = (0 ( (a) -1))ee,))))((()) c]}-
(g -1) Σ I(g(x.)) -1), 2=1 e(xx))
where e(xx²) is the true propensity scores of positive samples and é(x!!) is the estimate of propensity scores, while ((x=²) and ē(xx{" the normalized inverse probability weighting on positive samples. We can use the real normalized propensity score of positive samples ē(x!") construct the PUe's loss function, or use the estimated normalized propensity score of positive samples ((xf") to construct the loss function. Since then, we use ē(x=") to represent them.
From the normalization, we can know the following formula: EnS E(x!))
1, Note, ΣØ= claf) = Np. where Np means the
1. Diml a(co
and (((x))
a(m
eto
number of all positive data.
np nr, is the number of labeled samples, Np is the number of positive samples, n is the number ap of samples. is clear that to be unbiased, we have to make Ep(z|g=1) = Np which indicates that e(x) npp. And Ep(r(==1)
c(x=") Np. Since e(x,²) 0, existing propensity score estimation methods such SAR-EM in [ often underestimate the propensity score of positive samples, which will cause co-directional bias or produce an unidentifiable result (overfitting) that all labeled samples have propensity score close to and all unlabeled samples have a propensity score close to O.
The normalization technique is used to make the deviations no longer in the same direction (co- directional error will produce worse results). Moreover, the maximum estimation error ratio of sample weights is reduced. The regularization technique is used to alleviate the overfitting problem, and the work does not assume known propensity score or positive sample distribution. After normalization
used, 0.A < Σ, [(a)T
A/O does not have such co-direction error, where
and
Σ, claf)
0 < Ply = 1 1 11
To calculate the unbiased PU loss in the biased scenario, we must know the propensity score of each labeled sample. But this propensity score we usually don't 't know. So we consider two cases: (I) the true propensity scores know and (2) must estimate the propensity scores from data.
Case 1: Know the True Propensity Scores. In standard PN Classification, the true PN risk with real class y of size can be formulated follows:
n
Rrw(gly) : Prr(g +1) R -1)
Σ (gg(())) +1) + [g)())) -1)-
(8)
Defintion 3 (IPW). Given the propensity scores and PU labels, the inverse probability weighting estimator of (g) is:
Rrve (g) x xRp(
+1) + Ru, -1) Rpp( -1).
(9)
5

---

0,30
Step 2
1
inverse probability
Estimated network
Xx
E R,(x-1)
RoCS 1)
3tep.3
9
9
y
r Xy
x Xa
R.ft 1-1-1 4
Classification network
- ==1
y=0
y
Figure 2: Overview the PUe framework. Our goal is to estimate the propensity score of the labeled samples and modify the sample weights using the normalized inverse probability weighting technique to obtain the loss function uniformly unbiased estimator.
This estimator is unbiased.
1 n Σ L.L((())))+t) + (1- ()) -1).
E E[Rpue = RPN(glv)
(10)
When the propensity score is known, not only the loss function reconstructed by the PUe algorithm is unbiased, the expectation of the PUe loss function is equal to the real PN loss function. but also the difference between the PUe loss function value and the real PN loss function value probability bounded. However, in other PU algorithms where the label distribution is biased, the expectation of the loss function biased
Theorem 1: Inverse Probability Weighting Error Bound. For any predicted classes ğ and the real class y, with probability 1 1-1, 17. the inverse probability weighting estimator RP (g) does not differ from the true PN loss function ReN (gly) more than: [Rpx Rt / 1 2
2n
with Lmax is the maximum absolute value of cost function L(g. y).
Theorem Error Bounds of Common PU Algorithm in Biased Scenarios. For any pre- dicted classes and the real class y, with probability 1 - n,, the inverse probability weight- ing estimator R.P (g) does not differ from the true PN loss function RPN more than: [RPN(gl)) - (g)(g)) < 2T
L2 In
Rey
2n
In the biased scenario, the error bound of PU algorithm larger than that of PUe algorithm.
Case 2: Estimate the Propensity Scores from Data. But in practice, the probability of the sample being labeled usually unknown. For example, ad recommendation. we not actually know the probability of everyone clicking on the ad, that is, the propensity score is unknown. So in order to get the loss function of PUe algorithm for the unknown label distribution, we must have an estimate of the propensity score.

---

R = (9 x +1) (9. -R -
(11)
In this case, the bias of the inverse probability weighting estimator is:
72 ya
E(x,) (T(g(ass) +1) [(g(a.)) -1]]- Np Np²(x;;)
bias hias(Rrua(g))
(12)
From the bias, find that the propensity scores only need to be used accurately for positive examples. Apparently, (zg( -1g))(((m;), -1) < 0 for positive examples. And when incorrect propensity score is close 0 or I (especially tend towards 0), there will be large bias. Overestimate normalized inverse propensity scores.
Underestimated propensity scores expected to result in model with higher bias. Lower propensity scores result in learning models that estimate the positive class to be more prevalent than it is, which results in larger (L)g(z;), +1) L(g(x,), -]] for positive samples.
up
ny
((x) arg min
Σ
(c(
M(e( -1) + ael
Σ x,EXPUXU
e(x;) npl.
7P
nU
(13)
where म +p+to
In reality, it is almost impossible know the propensity of the sample, which requires estimate the propensity score of the sample. However, the traditional statistical estimation method not be used directly in deep neural network. This requires to design new method to estimate sample propensity scores.
According to the paper 77: In the Local Certainty scenario, assume the relationship between the feature
observed features and the true class is deterministic function f + Y, where space and Y = {0, 1), while allowing the propensity score to be arbitrary probability function. but this assumption is too strong. In fact, we only need to have hypotheses [P(y = 1|z) - P(y = 0xx)| > 9 and P(y 1|x²) Pvy 0jx²) to estimate the normalized propensity score well for sample classification. (where I. represents the labeled sample and used to distinguish from the positive category.)
Otherwise, a linear function usually used to estimate the propensity score in scenarios where the local certainty scenario is not met. Our supplementary experiments in the supplemental materials show that the probability function fitted using deep learning is better than the linear estimation the MNIST experiment.
We use labeled data as positive samples and unlabeled data as negative samples. We use binary network and regularization technology to construct a scoring network e(z). Finally, the output of the network passes through the sigmoid function to ensure that the output value is between 0 and 1. The score it outputs for each sample is the estimated probability of the sample being labeled (propensity score). The purpose of introducing the regularization technique Otherwise, the output value of the labeled sample is very close to and the output value of the unlabeled sample is very close to O. This not what we expect. The ideal propensity score value should be far away from 0 and 1. Therefore, we introduce such a constraint in the classification loss function by combining the nature of propensity scores, where the sum propensity scores of all samples is equal to the number of labeled samples: ae Ex,ExpUxo e(x,) npl-
prevent overfitting.
where Og ajusts the importance of regular term.
Theoretically, for each labeled example (æ;, +1), which has porpensity score e,, there expected to be _ positive examples, and of which ب - 1 did not get selected to be labeled. Each labeled example gets weight 1 and the negative example is added to the dataset that gets negative weight
e,
According 37. - Sores For Classification. However, as discussed above, existing methods always underestimate
c, -.) + s1)6g. using Propensity
7

---

Table 1: Comparative results MNIST, CIFAR-I0 and Alzheimer.
Dataset
Method
ACC (%)
Prec. (%)
Rec. (%)
FI (%)
AUC (%)
AP (%)
uPU
85.17 (1.90) 78.38 (2.36)
96.69 (1.33)
86.55 (1.50)
$5.34(1887
77.41 (2.26)
nnPU
87.81 (1.65) 83.78 (2.14)
93.40 (1.68)
88,31 (1.51)
87.89 (1.63)
81.62 (2.28)
PUbN
87.07 (1.73) 83.43 (2.74)
92.21 (2.55)
87,55 (1.55)
(2.14)
80.70 (2.59)
Dist-PU
89.79 (0.59) 92.22 (1.67)
88.86 (1.48)
90.49
06.60
96.92 (0.21)
uPUe
92.28 (1.32) 89.41 (1.87)
35.71
92.33 (1.25)
87.69 (1.92)
MNIST
nnPUe
92.45 (1.61) 90.45 (2.26)
94.73 (1.24)
02.53
92.48 (1.60)
88.29 (2.43)
PUbNe
94.70 (0.63) 97.16 (1.59)
91.33 (1.58)
94.46 (0.71)
94.66 (0.50)
93.18 (0.55)
Dist-PUe
92.57 (0.78) 93.02 (0.86)
90.58 (1.47)
92.77 (0.84)
97.13 (0.57)
97.43 (0.44)
uPUe
92.39 (1.36) 39.33
96.07 (1.70)
92.56
92.44 (1.87)
87.76 (1.93)
nnPUe'
92.94 (1.27) 91.11 (1.84)
94.98 (2.38)
92.98 (1.29)
92.96 (1.27)
89.00 (1.77)
PUbNe'
94,48 (0.93) 97.74 (0.46)
90.90 (2.10)
94.18 (1.06)
94.43 (0.95)
93.45 (1.00)
Dist-PUe'
91.73 (0.95) 93.34 (0.42)
89.60 (2.38)
91.41 (1.11)
96.48 (1.54)
96.60 (1.37)
uPU
(1.76) 83.59 (0.65)
50.81 (5.57)
63.01 (4.35)
(4.40)
78.68 (2.22)
nnPU
82.47 (0.69) 75.30 (1.35)
83.68 (1.64)
79.25 (0.75)
90.01 (0.73)
85.22 (0.73)
PUbN
84.54 (0.54) 82.18 (1.38)
78.40 (2.70)
80.20 (0.99)
91.62 (0.27)
87.31 (0.40)
Dist-PU
84.45 (1.00) 80.90 (1.51)
80.04 (1.62)
80.46 (1.26)
90.89 (1.20)
85.67 (1.54)
uPUe
79.31 (0.90) 81.37 (2.16)
(5.02)
70.73 (2.58)
86,76 (0.82)
80.07 (1.56)
CIFAR-10
nnPUe
83.33 (1.04) 76.83 (2.75)
83.88
80.11 (0.85)
90.53 (0.79)
85.33 (1.36)
PUbNe
85.66 (0.58) 83.18 (1.63)
(2.29)
81.79 (0.66)
92.69 (0.74)
89.41 (1.24)
Dist-PUe
86.97 (0.62) 82.11 (1.69)
86.46 (2.00)
84.19 (0.76)
93.50 (0.66)
89.38 (1.78)
uPUe'
85.11 (1.75
(1.58)
$7.99644966
68.60
86.57 (1.56)
81.56 (2.18)
nnPUe'
84.24 (0.67) 78.40 (0.86)
83.66 (2.35)
80.92 (1.02)
91.35 (0.99)
87.15 (1.83)
PUbNe*
87.02 (0.47) 83,84 (1.44)
83.78 (2.68)
83.76 (0.82)
93,80 (0.42)
90.69 (0.65)
Dist-PUe'
86.17 (2.52) 80.97 (3.20)
85.57 (3.32)
83.19 (3.06)
92.65 (2.20)
88.22 (3.19)
uPU
63.46 (1.72) 66.80 (3.45)
55.06 (9.83)
59.65 (4.70)
68.08 (2.16)
67.78 (2.22)
unPU
69.19 (0.85) 65,46 (1.06)
(5.22)
72.48 (1.62)
73.50 (0.84)
68.74 (1.83)
PUbN
68.75 (0.96) 66.65 (2.04)
75.64 (5.58)
70.68 (1.52)
72.57 (0.63) 66.99 (0.80)
Dist-PU
69.74 (0.50) 67.74 (1.57) 75.69 (4.72)
71.37 (1.33)
(1.10)
69.53 (1.39)
Alzheimer
uPUe
65.30 (3.36) (1.99)
(11.33)
(3.99) (7.58)
70.28 (3.09)
nnPUc
69.38 (0.56) 65.97 (0.68)
80.06 (2.77)
72.30 (0.91)
74.49 (0.34)
69.98 (0.60)
PUbNe
69.52 (0.85) 68. (1.28)
72.95 (1.99)
70.51 (0.82)
74.68 (1.08)
70.33 (0.98)
PUs
70.51 (0.62) 68.91 (1.00) 75.02 (2.26)
71.81 (0.83)
75.19 (0.69) 71.23 (2.08)
propensity scores. As result, PU learning classification is performed by using the foregoing formula, which reduces model classification performance.
We found that in cost-sensitive PU learning technique, the training instances are properly reweighted. and a common way to score propensity is inverse probability weighting (a reweighting technique). Conditionally, the two methods can be combined to solve the defects of the two methods themselves. The disadvantage of the cest-sensi PU learning technique is that the classification effect of the application deteriorates when the labeled samples are biased. The propensity score is estimated by the fact that the propensity score of positive samples is always underestimated, which leads to the decline of the classification effect of the model. This is why we chose to couple with cost-sensitive PU learning.
However, in practice, the disturbance of propensity score has a great influence on the results due to the reciprocal, which makes the variance of our model large. We suggest that the existing prior class in PU algorithm is used, and the propensity score of samples is weighted by normalized inverse probability, which is used as the weight of samples. This advantage is that there are many estimation methods for the prior class [2]. and that the error estimation of the propensity score of individual samples is prevented from greatly affecting the model (the propensity score is close to 0), thereby enhancing the robustness of the model and improving the accuracy. The loss function of the basic PUe algorithm can be obtained as follows:
R x RR +1) + (g. 11) RR -1).
(14)
The main algorithm process of PUe is shown in Figure2 The specific algorithm process of PUe shown in supplementary material. In summary, the first step of PUe algorithm to estimate the propensity score of positive samples using regular binary classification. If the propensity score is
8

---

known, the first step can be skipped. The second step is to modify positive sample weights with normalized propensity scores. The third step use the PU algorithm (that is, PUe) with modified weights to perform PU binary classification. We also consider the case where negative classes are also selectively labeled and conduct related experiments (see PUBN), which are not mentioned in other papers estimating propensity scores and applying them to PU learning.
3 Experiment
In this section, we experimented our idea on several popular PU methods and compare its performance against the initial baseline methods.
Table 2:
Ablation results on CIFAR-10 10 with V'ndicating the enabling of the corresponding loss term and different labeled distripution.
method
aa labeled distripution
ACC (%) Prec. (%) Rec. (%) Fl (%) AUC (%) AP (%)
Dist-PUe
255,225,22,2251
91.79 (0.86) 89.02 (2.63) 90.78 (188)) 89.84 (0.92) 97.11 (0.52) 95.38 (1.18)
Dist-PUe
225.
91.30 (0.80) 89.22 (1.56) 89.05 (1.93) 89.11 (1.03) 96.64 (0.53) 94.85 (0.77)
Dist-PU
(225.25.225.255
91.88 (0.52) 89.87 (1.09) 89.84 (0.81) 89.85 (0,62) 96.92 (0.45) 95.49 (0.72)
Dist-PUe
10.0.10
91.50 (0.32) 89.07 (1.15) 89.78 (0.97) 89.41 (0.34) 96.67 (0.14) 95.64 (0.26)
Dist-PUe
[10..10.330.550
90.40 (0.66) 88.86 (3.15) 87.14 (0.74) 87.89 (0.74) 95.71 (0.40) 94.19 (0.65)
Dist-PU
110.110,30.550
90.79 (0.72) 91.31 (1.54) 85.12 (2.44) 88.07 (1.07) 95.81 (0.83) 91.28 (2.35)
Dist-PUe v
100
86.97 (0.62) 82.11 (1.69) 86.46 (2.00) 84.19 (0. 76) 93.50 (0.66) 89.38 (1.78)
Dist-PUe
1.7...5..100.033
86.44 (0.67) 82.19 (1.54) 84.45 (1.78) 83.28 (0.82) 92.90 (0.63) 88.39 (1.32)
Dist-PU
172..15.10..03)
84.45 (1.00) 80.90 (1.51) 80.04 (1.62) 80.46 (1 (1.26) 90.89 (1.20) 85.67 (1.54)
Dist-PUe V
86.8.66.02.044
88.15 (1.22) 88.05 (2.69) 31.70 84.56 (2.23) 94,46 (1.O1) 92.79 (1.09)
Dist-PUe
008.866002.04)
86.39 (2.11) 84.47 (3.51) 81.05 (4.32) 82.62 (2.77) 92.42 (2.25) 89.84 (2.40)
Dist-PU
(088.86.02.004)
86.74 (1.58) 86.56 (2.51) 79.30 (4.60) 82.65 (2.48) 93.30 (1.32 91.08 (1.69)
3.1 Experimental settings
Datasets. We conducted experiments on two benchmarks commonly used in PU learning: MNIST for parity classification and CIFAR-10 [17 for vehicle class recognition. And on the simulated datasets of MNIST and CIFAR-IO, we know the propensity score of the sample a priori, and we compare proposed method with the ideal propensity to know the propensity Morevvr, tested our method on the Alzheimer's ' dataset 2 [usdd to identify Alzheimer's ' disease in order to test the performance of our proposed method in real-world scenarios. More information is displayed in supplementary material.
Baselines. We mainly consider four common PU learning methods, including uPU, nnPU, PUbN, and Dist-PU.
Evaluation metries. For each model, we counted six commonly used indicators of the test set results, including accuracy (ACC), Precision (Prec.), Recall (Rec.), F1, Area Under ROC Curve (AUC) and Average Precision (AP). Accuracy (ACC) is the main index for more comprehensive comparison. The experiments are repeated with six random seeds, and the mean and standard deviation of each index were recorded.
Implementation details. All the experiments are run by PyTorch. Backbones of each dataset summarized in supplementary material. The training batch size is set as 256 for MNIST and CIFAR- 10, while 128 for Alzheimer. We use Adam as the optimizer with cosine annealing scheduler, where the initial learning rate is set as 5 x 10 while weight decay is set as 5 x 10-3, PU learning methods first experiences warm-up phase of 60 epochs, and then trains another 60 epochs with depth, where the value of a is searched in the range of [0, 20).
'Dubey, S. Alzheimer's Dataset. Available online: http://wwww..kalllcooomoooossose alzheimers-dataan daw.aatt---alaaaeeeeeeeee------- of

---

3.2 Comparision with state-of-the-art methods
Competitors. We compare our method with 4 competitive baseline PU algorithms including uPU [5]. nnPU [6]., PUbN [9], and Dist-PU 24]. Due to the space limitation, the detailed descriptions are provided in supplementary material.
Results. The results on all the datasets are recorded in Tab It shows that in most metrics, our proposed PU+e method significantly outperforms our competitors on all biased datasets, improving the performance of the original PU method by about 1% to 5%. This proves the effectiveness of our proposed method. addition, some observations can be made: (1) model with known propensity score is not necessarily the best (who has e' in Ta but model with an estimated propensity score can perform better in many cases. This is a conclusion that is consistent with causal inference (2) In the ablation study, the performance of the PU+e algorithm is comparable to that of the most advanced PU algorithm, even when the labels are evenly distributed.
3.3 Ablation studies
Effectiveness of hyper- apperrparameters. Ablation experiments were conducted to verify the validity of pyprpaarameters which is shown F It can be seen that the PUe algorithm sensitive to hyperparameters and does not change monotonically. When &ee 15, the PUe algorithm has the best performance. Therefore, in order to obtain the best performance of the PUe algorithm, fine adjustment required. When the best performance is achieved remains to be further studied. However, the overall performance Cifar- 10 is better than that of the original Dist-PU algorithm.
Effectiveness of labeled samples distribution. The deviation of la- bel data distribution will signifi- cantly affect the promotion effect of our proposed model in T In general, when the distribution of labeled samples is more biased, the improvement effect of our pro- posed method is more obvious. However, when the distribution deviation is very large, the lifting effect of our model is weakened. This is because the number of la- beled samples for some classes is small not labeled. As sult, the propensity score is close O, which affects the model per- formance. On the Cifar 10 dataset, our algorithm improves the accu- racy of the Dist-PU algorithm by up to 2.5%.
Accuracy (%)
87
Preciston (%)
R2
86-
10
15
10
15
20
Recall (%)
86
FL. (96)
8
18
15
10
15
20
de
dee
AUC (26)
AP (%)
0
10
15
20
10
15
20
Ce
Figure 3: Influences of different ag on CIFAR-10 by Dist-PUe.
Conclusion
In this paper, new PU learning method, namely PUc, is proposed from the perspective of propensity score for the PU learning with biased labels in deep learning. The PUe algorithm can enhance the original cost-sensitive PU algorithm, improve the prediction precision in the case of biased sample labeling, and has the degenerate ability, and the prediction precision in the case of unbiased labeling is not lower than that of the original PU algorithm. PUe consistently outperforms state-of-te of-the-art methods on most metrics on real-world biased labeled datasets, including MNIST, CIFAR-10, 10, and Alzheimer's. We hope that the proposed propensity score estimation scheme for deep learning can also provide some inspiration other weakly supervised scenarios, especially those where the label distribution is unknown.
10

---

Acknowledgement
We gratefully acknowledge the support of MindSpore 20), CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] Jessa Bekker and Jesse Davis. Positive and unlabeled relational classification through label frequency estimation. In Inductive Logic Progranming: 27th International Conference. ILP 2017, Orléans, France, September 4-6, 2017, Revised Selected Papers 27, pages 16-30. Springer, 2018.
[2] Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning. 109:719-700 760, 2020.
[3] Jessa Bekker, Pieter Robberechts, and Jesse Davis. Beyond the selected completely at random assumption for learning from positive and unlabeled data. In Machine Learning and Knowledge Discovery in Databoses: European Conference, ECML PKDD 2019, Würzhurg, Germany, September 16-20, 2019, Proceedings, Part II, pages 71-85. Springer, 2020.
[4] Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang- Self-pu: Self boosted and calibrated positive-unlabeled training. In International Conference on Machine Learning, pages 1519, PMLR, 2020.
Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data, Advances neural information processing systems, 27, 2014.
[6] Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 213-220, 2008.
[7] Walter Gerych, Thomas Hartvigsen, Luke Buquicchio, Emmanuel Agu, and Elke Rundensteiner. Recover- ing the propensity score from biased positive unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1194-6702, 2022.
[8] Keisuke Hirano, Guido W Imbens, and Geert Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica. 71(4): 161-1189, 2003.
[9] Yu-Guan Hsieh, Gang Niu, and Masashi Sugiyama. Classification from positive, unlabeled and biased negative data. In International Conference on Machine Learning, pages 2820-2829. PMLR, 2019.
[10] Antti Hyuiinen Frederick Eberhardt, and Patrik O Hoyer. Experiment selection for causal discovery. Journal of Machine Learning Research, 14:3041-3071, 2013.
[11] Kosuke Imai and Mare Ratkovic. Covariate balancing propensity score. Journal of the Royul Statisticul Society: Series B: Statistical Methodology, pages 243-263, 2014.
[12] Guido W Imbens and Donald B Rubin. Rubin causal model. In Micrneconometrics, pages 229-241. Springer, 2010.
[13] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[14] Shantanu Jain, Justin Delano, Himanshu Sharma, and Predrag Radivojac. Class prior estimation with biased positives and unlabeled examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4255-4263, 2020.
[15] Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with a selection bias. In International conference on learning representations, 2019.
[16] Ryuichi Kiryo, Gang Niu, Marthinus C. Du Plessis, and Masashi Sugiyama. Positive-umlahhleed learning with neganiee risk estimator. Advances in neural information processing systems, 30, 2017.
[17] Alex Krizhevsky, Geoffrey Hinton,
Learning multiple layers features from tiny images. 2009,
[18] Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S Yu. Building text classifiers using positive and unlabeled examples. In Third IEEE international conference data mining, pages 179-186. IEEE, 2003.
[19] Bing Liu, Wee Sun Lee, Philip S Yu, and Xiaoli Li. Partially supervised classification of text documents. In ICML, volume 2, pages 387-394, Sydney, NSW, 2002.

---

[20] MindSpore. Mindspore Mindspor/httts:: httllpppwhtttpss//www..oo http:///swmmmmmmoopp.......mm en/
[21] Paul R Rosenbaum. Model-based direct adjustment. Journal of the American statistical Association, 32(398):387-394, 1987.
[22] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
[23] Guangxin Su, Weitong Chen, and Miao Xu. Positive-unlaheled learning from imbalanced data. In IJCAI, pages 2995-3001, 2021.
[24] Yunrui Zhao, Qianqian Xu, Yangbangyan Jiang, Peisong Wen, and Qingming Huang. Dist-pu: Positive- unlabeled learning from a label distribution perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14461-14470, 2022.
12

---

A
Algorithm
Algorithm PUe algorithm
Require: data XP-XU, size n,p,,uy hyperparameter ac,
1: Step 1:
2: Compute ē(x)) by minimizing
x up Ei L(g(ao 1,+1) +
1-x no Ex= L(g(a)) -1)
ael Ex.Exuuu e(Ts) -pl:;
3: Step 2:
4: Compute the weight of labeled samples: w! a(x?)
5: Step 3:
6: for lori=1...
7: Shuffle (XP, xv) into M mini-batches
8:
for each mini-batch (xex x do
9:
Compute the corresponding (g)e(g)
10: Use A to update with the gradient information Va (g)
11: end for
12:
end for
B Experiment Details
Table 3: Summary of used datasets and their corresponding models.
Datasct
Input Size
n1p
ng
Testing
Positive Class
true e(X)
Model
MNIST
28
2500 60,000
10,000
Even (0, 2, 4, and 8) 165.115.1.007.003
6-layer MLP
CIFAR-10 3 x 32 x 32
1,00050,000
10,000
0.4 Vehicles (0, I, 8 and 9) [.72,.15.11.00 13-layer CNN
Alzheimer3 x 224 x 224
769 5.121 1,279
0.5 Alzheimer's Disease
unknow
ResNet-50
C
Complementary Experiment
Table 4: Supplemental Experiments on MINST
method
labeled distripution ACC.(%) Prec. (%) Rec,(%) F1.(%)
AUC
AP(%)
LRe 165..15.100.0..0000 86. 19(0.55) 92.94(0.64) 77. 77.89(1.38) 84.75(0.993 88 88.06(0.93) 88.72(1.09)
nnPUe
11..15..00.00.0333999. (1.61)90.45 (2.26)94.73 (1.24)92.53 (1.55)92.48 (1.60 (60088.299 (2.43)
nnPU without Self-PU
normalize|.66,.15..1100 07.003))..... (0(666()))) 88. 18(2.40) 94.38(2.74) 91.13(1.56) 91.00(1.61) 85 85,98(2.25)
anchor 166..15.10.00.0033 01..003999000000 90.08(0.47) 90.08 (0.47)89.35 (1.17)90.70 (1.73)90.00 (0.53)85.61 (0.69)
88. 94.66(1.41) 80.70(3.15) 87.06(1.37) 92.36(1.92 93.37(2.33)
LRe: Logistic regression estimation of propensity scores for PU learning.
According to paper 77]. it cannot estimate identifiable PS without making certain assumptions about the data. But according to the formula we gave in the first question, it's approximate. This is not explained by the self-monitoring method. Results in the above table show that our scheme is better than PU in the case of biased label datasets.
D Proofs
D.1
error bound of bias
We may assume that the error of propensity scores estimated by the NN method is the same as that estimated by the linear method. (In fact, the NN methods are usually more general, which may produce results with less error.) That is, the estimate of the propensity score has maximum error ratio of B, with ) S ((x?²) S e(x()) of the following shows that our regularization technique can yield smaller error ratio with respect to sample weights. Obviously, the sample If has a sample weight of ne(z5)- in (Formula 1) with an error bound of bias( ne(ut)) " ne(af] (} 1).
13

---

D.2 error ratio
1
In our approach, Sample of has a weight of a "E, {(v)))
a(a
p(re(at) < E(xf) S e(xf)) = a where
the of samples is S1 .P(Se(at) (((x) S ?e(xx)) = 1 where the set of samples is
3 < y < 1 and Eies: c[at] Diess c(xt5) B. So that Ej c(5 -B Np. For of St,
we have clzt) S a(x+) < na(a))) For ot E S2, we have )r(x+) S B(x7) < Be(af)) so we can
get B(1 + 4) S Ej ह
< B + }) and we have bias(T
<max ne(af) (1+7)
Σ) B(
1), ne(x?] コ )
1), ne(n]] (1
3+1 )1 < no 2 B 1) and obviously we
11+
have (147] (1+
<
3.00
0
<
3 <
2 _+
< 1 3 < 1, which shows that our
regularization technique has a smaller error ratio with respect to sample weights.
D.3 expectation
One understanding is that, according to the PS definition, each labeled sample xz.' corresponds to e positive samples. So Sp e(af)) = Np it's true. Because P(z|s = 1) = P(z, = 1|s = 1),
we have
P(r,y 118 1) P(==1(
Σ P(a ((2.y)
P(==l)
P(s=1lyy=]
Σ P(r.y=1)
P(==1) uD Σ P(r,y= y= 1)
2 N.Ne Ne= Ne,
up mp
1 = Np- It indicates that E}=1 c(2,)
D.4 PUbN
The PUbN formula is as follows:
Let (xx) p(s +1 +1]z), however, the a (xx) is actually unknown,we should replace (1:) by its estimate (ar..We can get the classification risk of PUbN (RpusN(g)) as the following expression:
RPUSN (g) (g +1) + pRaN(g, -1) Ro=-imo(9) where
+ (o 1,n (g)
= Benptell
a(x))]
1-6(1))
(n (--(())) [000))
)(x)
Then Ran (g, 1) and Rs - n (g) can also be approximated from data by RoN -1)
RAN - L(g(x.)) ( 1) R = - in (g)
- ny
La -1))) o(x(\)1
1>1 L(
+ e
E L(g - )(agT\)
o(x;))
a
RPU R +1) pRaN (g, -1) R
D.5 PUbNe
Our PUbNe formula is as follows:
R (g) πT RR +1) + p (g. -1) + R.
im (g)
where R (g. -1) MAN
1 2
L(g(atN). - -1) and
14

---

Σ - [ Tacs))) ( -1)(1
o(x5))]
+ Minlaad )يح) (n - -
+p Em
naw
((ug.))) 1 ((a7))>01((((0 -1) 1-0((tt))
o(o;))
a(o75))
D.6 unbiased
B[Rpue())]
=
=E +1) Rg -1) x xhplg, -1]
=
Σ c(x+))
np
+1) - L(g(ao)) -1)) + 1 Σ
L(g(x;), -1)]
n ==1
L(g(2 )2 +1)+ + (1-
E
c(x?f) -(oo(p))) L(g(z))) -1)
Σ (1 - s,)L(g(x;;) -1)]
c(ar.))
im1
11 (1--1 e(x!!)
E
8 c(xx))
L(g(2 ), +1) + 8,
[(g(x)), + s-))(((((x;))) - -1)]
π1
1
1 L(g(a;;); +1) + yie; 1 (1-((ar)) c(,,,)
1
L(g(o)), -1) + (1 - (egx)) -1)
Mie c(ar,)
"
y:Llg (x,) +11 + 3 (e: 1) L(grx)) -1 + (1 ( -1)
1
+1) + (1- -1 ((g(e)) -1)
(gl()
(15)
The change of (g)e(g) will be no more than Lmmx//n if some 2, E Xr U Xu replaced, and McDiarmid's inequality gives us:
Pr[ Pr((RPue())- (eN( 2 2 = PP-[(pppu() - E(Rrue(o)l 2 S 2cxp (-
262 in(Lmax/m))
And make the right side of the previous formula equal to ry:
2c²
2 exp
[Tn(Lmx//m))
r7
2ca
2
exp
ni [mmax n2)
2xa
In
L 2 nn
I 2 In
2
2e
L In
2
2n
Equivalently, with probability at least 7
L2 In
(g) RPN (glv) = (g) - [[] <
(16)
2n
15

---

And because we know the expressions of Rove(g) and Ro)
R nlt + Ru(g (9,(,1111 -1) + T1pp(0.-111 -1),
(17)
(g)(0) = T Rr(g +1) + Rg -1) (g. -1),
(18)
Since we know that 2ك c(ae))
Np, we can get the following formula:
RRpue (g) Ro)(
т M2 +1)
RCply- -1) - ((Rp(s,+)) (1]p(g,+) +1) - Rp(s -1))
NP L +,1e))+++1]++)
1
2p
Np
+111
Σ c(xx))
[g -1)]
c(xx))
np
np
v Np Linax 2n Lmmax
(19)
Then, we can prove:
RRN(sll)) Rpv(w))
I (PN(( RPUe(g)) + (Rpue(g) Ro((g))
(20)
L2 max In
< 2x Immax +
2n
16


Reviews:
[{"review_id": "HPNwFoLUX8", "reviewer": "Reviewer_U7Dp", "summary": "This paper noted that existing Positive-Unlabeled (PU) learning methods assumed that positive samples are selected entirely at random, ignoring the prevalent selection bias in real-world PU problems. To overcome this limitation, the authors proposed a PU learning enhancement (PUe) algorithm based on causal inference theory. They adapted sample re-weighting methods, to be specific, inverse propensity weighting (IPW) to PU learning, which assigns a weight to each positive sample. Based on the weights, they obtained a new unbiased PU risk estimator. They also proposed a method that can compatible to deep networks to estimate the weights when the labeling mechanism is unknown. ", "questions": "Please see the main review.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The problem is interesting and valuable. The adaptation of causal inference to PU learning is sound and original to the best of my knowledge. \n* The resulting algorithm is unbiased, which is a very nice result (but I quickly checked the proofs and have some concerns).\n* The algorithm is simple and easy to follow.\n* The empirical studies well support the method's superiority with a clear margin.", "weaknesses": "* I'm a bit puzzled by the fact that the risk estimator is unbiased. \n\n$R_{PN}$ in current version should be $\\hat{R_{PN}}$ for notation consistency, while $R_{PN}$ denotes the expected risk. Then if $\\mathbb{E}[\\hat{R_{PUe}}(g)] = R_{PN}(g)$, we say $\\hat{R_{PUe}}$ is unbiased. \nWhile in this paper, the authors proved $\\mathbb{E}[\\hat{R_{PUe}}(g)] = \\hat{R_{PN}}$ (Eq.11), which looks a little weird.\n\n* The examples given by the authors give readers a very clear idea that biased selection is likely to be present in the PU problem, but it seems to me that Figure 1 doesn't present a very clear picture of the setting discussed in the paper. Wouldn't it be more telling if all the blue dots were centered, like, in the upper left corner? But if so could ePU also have difficulty learning good classification boundaries, since this bias could also seriously mislead the estimation of $e$ (step 1 in Figure 2)?\n* $s$ should be introduced in Eq.4.\n* In line 113, \"it does not depend on...\", \"it\" here seems very vague.\n* Tables 1 and 2 are too dense a presentation of the results, and I would suggest labeling the best results more obviously.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1:* **$R_{PN}$ should be $\\hat{R_{PN}}$. Eq.11 looks a little weird.**\n\n*A1:* Thank you for pointing out the issue. It is true that the expression forms here are not uniform. The empirical risk should be represented by $\\hat{R_{PN}}(g|y)=\\frac{1}{n}\\sum_{i=1}^{n}[y_iL(g(x_i),+1)+(1-y_i)L(g(x_i),-1)]$ and the expected risk should be represented by $R_{PN}(g|y)=E_{P(x,y)}(L(g(x),y))=E_i(\\hat{R_{PN}}(g|y))$. Obviously, the expectation of empirical risk equals the expectation of expected risk. Using the two-stage expectation $\\mathbb{E}_i[\\mathbb{E}_s]$, we can obtain that the risk estimation of our algorithm is unbiased.\n\n$\\mathbb{E}[\\mathbb{E}(\\hat{R_{PU_e}}(g))]=R_{PN}(g|y)$. It does not affect the conclusion. We'll add this to the final version of the paper.\n\n*Q2:* **If all the blue dots were centered, could PUe learn good classification boundaries? Or is it possible that the bias can mislead the estimation of $e$ (Step 1 in Figure 2)?**\n\n*A2:* Thank you for your question. The scenario you said is different from the condition setting of our paper and contradicts the Probabilistic Assignment assumption in causal inference. If all the blue dots were centered, like, in the upper left corner. Without additional assumptions, machine learning cannot learn good classification boundaries, and unlabeled positive classes cannot be correctly classified. Our scenario actually requires that all orthodox classes be labeled with probability, even if they are small, which means that Probabilistic Assignment is $0<P(s,x|y_i=1)<1$ for all samples.\n\nThe scenario you're hypothetical, which is denoted as partial positive, has a tendency score of 0. For example, in the odd and even category, a scenario that doesn't select 8 at all for labeling is similar to your assumption. According to the no free lunch principle, the same algorithm does not exist for the same data, and the classification of 8 must be accurate. If algorithm A correctly classifies eight of [0, 2, 4, 6, 8] and [1, 3, 5, 7, 9], algorithm A can not correctly classifies eight of [0, 2, 4, 6] and [1, 3, 5, 7, 8, 9]. 8 is not labeled, so it is impossible to determine whether 8 is positive or negative without additional assumptions. In our scenario, even if the probability of 8 being labeled is very small, the accuracy of the final algorithm is improved as long as there is labeling. The results are as shown in the paper.\n\n*Q3:* **$s$ should be introduced in Eq.4.**\n\n*A3:* Sorry for the unclear statement. $s=1$ here means that the sample was selected into the set of annotations. $s=0$ indicates that the sample is not selected into the annotation set. We'll clearly state this symbol in the final version of the paper.\n\n*Q4:* **In line 113, \"it does not depend on...\", \"it\" here seems very vague.**\n\n*A4:* Thank you for your correction. Here \"It\" means the labeling mechanism.It's not clear here. We will refine it in the final version of the paper.\n\n*Q5:* **Tables 1 and 2 are too dense a presentation of the results, and I would suggest labeling the best results more obviously.**\n\n*A5:* Thanks for your advice. We will mark the best results in the final version of the paper."}, {"review_id": "Kp31ZmsDar", "reviewer": "Reviewer_nitz", "summary": "The paper introduces an algorithm for Positive-Unlabeled (PU) learning that tackles the problem of biased labeled data. By utilizing causal inference theory, the algorithm utilizes normalized propensity scores and inverse probability weighting to reconstruct the loss function and achieve an unbiased estimate of the classifier. Compared to existing cost-sensitive PU methods, the proposed PUe algorithm demonstrates significant improvements in classifier accuracy, particularly on datasets with non-uniform label distributions. Additionally, the paper proposes a method for estimating propensity scores in deep learning models when the labeling mechanism is unknown. Overall, the PUe framework effectively addresses selection bias, enhances PU learning, and improves model performance.\n\n-------------------------------\nAfter rebuttal: I would like to increase my rate a little bit. However, I think the novelty part still needs more clarification. \n\nAfter the author-review discussion: I could increase my rate a little bit for the thorough rebuttal provided by the authors. However, I think the rebuttal provides a lot of information that may not be easily included in the paper. I would like to see a plan on how to incorporate the reviews/rebuttals into the paper. \n\nDuring the final discussion: The plan has given me more confidence in how the paper can be if we accept it.", "questions": "Comparison with \"Recovering the Propensity Score from Biased Positive Unlabeled Data\" presented at AAAI 2022: A direct comparison between the paper and the mentioned work would require a thorough analysis of both papers. \n\nSignificance of causal inference in the context of the proposed work: Although the paper mentions causal inference as a motivation, it is valid to question the significance of causal inference in this work. If the process of generating biased data is explicitly defined and the proposed loss function can be straightforwardly derived from it, the direct application of causal inference may not be evident. It is important to consider whether the paper effectively incorporates causal inference methods or frameworks that address causal assumptions and mechanisms beyond straightforward reweighting.\n\nComparison with self-supervised methods like selfPU: Evaluating the performance of the proposed approach compared to self-supervised methods like selfPU would require a detailed analysis and comparison of the methodologies and experimental results from both approaches.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Importance of the studied problem (biased positive data): The problem of studying biased positive data is highly significant and practical. In many real-world scenarios, the selection of positive samples is often biased, while existing methods typically assume that the selection of positive samples is completely random. The proposed PUe algorithm in this paper addresses this issue and effectively handles biased labeled data, filling a research gap in the field of PU learning.\n\nWell-written: The paper is well-written, with clear descriptions of the problem background, method principles, and experimental design and results. The authors introduce causal inference theory and utilize techniques such as normalized propensity scores and inverse probability weighting to provide a viable solution, which is presented in a clear and concise manner.\n\nExperimental results demonstrate the effectiveness of the method: The paper provides experimental evidence of the proposed PUe algorithm's effectiveness.", "weaknesses": "Lack of novelty: One of the limitations of the paper is that it may lack novelty. Similar work, such as the paper \"Recovering the Propensity Score from Biased Positive Unlabeled Data\" presented at AAAI 2022, may have explored similar ideas and methods earlier. It is important for a research contribution to demonstrate novelty and differentiate itself from existing approaches in the field.\n\nLimited causal inference aspect: While the paper is motivated by causal inference, it falls short in incorporating substantial causal elements. The approach of reweighting samples based on sampling probabilities is straightforward and may not fully capture the complexity of causal relationships. To strengthen the causal inference aspect, the paper could have explored additional methods or frameworks that explicitly address causal assumptions and mechanisms.\n\nLack of significant contribution: The primary contribution of the paper should lie in the proposed method for estimating propensity scores. However, the approach of using model predictions as weights for propensity scores is not fundamentally different from classical self-supervised methods. Although a regularizer is introduced, it may not provide a meaningful improvement. Additionally, the process of normalizing the estimated propensity scores could have been done separately after learning the scores, making the proposed regularizer less essential or impactful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1:* **Comparison with paper in AAAI 2022.**\n\n*A1:* Thanks for the nice concern. The main content of aaai's article is introduced and explained respectively in the case of Local Certainty and Probabilistic Gap Scenario PS estimation which is s identifiable. There are four points about the novelty and contribution of this paper. First, the assumption of Local Certainty is relaxed, and the algorithm effect is improved. Second, use deep learning to estimate propensity scores(PS), rather than linearity, better than logistic regression. Third, our PS estimation method can be extended to the case with negative classes (e.g., PUbNe). Fourthly, our new algorithm can be simply coupled with most cost-sensitive algorithms to improve performance, as shown in paper. We’ll include more analysis and results in final version.\n\n1. The paper relaxes the assumption of the Local Certainty and does not require the relationship between the observed features and the true class is a deterministic function . When x has a large Probabilistic Gap to y and positive instances that resemble negative instances are less likely to be labeled, this algorithm can also be true, i.e. $|P (y=1|x) -P(y=0|x)|>0.8$ and  $P(y=1|x^L) > P(y=0|x^L)$,  L means the labeled samples.\n\nAnd the Local Certainty method in the aaai paper is used to estimate the PS, the $\\sum e = n_p$ restriction and $\\hat{e}(x_i^N)>0$ on the negative sample will result in underestimated PS on labeled samples. Normalization can improve the estimation effect.\n\nOnly labeled samples are considered. We have $0.9<P(y=1|x_i^L) \\leq 1$. Obviously, $0.9e(x_i^L) < \\hat{e}(x_i^L)\\leq e(x_i^L)$. This shows that our estimated PS for the labeled sample is smaller than the true PS. Moreover, the PS of the estimated negative sample is greater than 0, i.e. $\\hat{e}(x_i^N)>0$. According to the PS definition, we have the following equation $\\sum \\hat{e_i} = {n_p}$. In aaai Local Certainty Scenario, PS of negative samples also share part of the probability, the PS of the positive sample is underestimated. Therefore, we need to use the normalization method to improve the accuracy of our classification algorithm. \n$\\frac{\\frac{1}{e(x_i^L)}}{\\sum_{j} \\frac{1}{e(x_j^L)}} = A$.\n\n$0.9A < \\frac{\\frac{1}{\\hat{e}(x_i^L)}}{\\sum_j \\frac{1}{\\hat{e}(x_j^L)}} < 10A/9$. This shows that our estimate is relatively stable.\n\n2. In the Probabilistic Gap, We don't limit the PS to be linear. Deep learning is used for estimation. The regularization method is used to prevent overfitting, and the normalization method is used to alleviate the problem of underestimating PS. We added experiments on MINST using LR (Logistic regression) to estimate PS, showing that LR is inferior to deep learning.\n\n|Method|Distribution|ACC|Prec|Rec|F1|AUC|AP|\n|-|-|-|-|-|-|-|-| \n|LR | [.65,0,.15,0,.1,0,.07,0,.03,0] | 86.19(0.75) | 92.94(0.64) | 77.89(1.38) | 84.75(0.93) | 88.06(0.93) | 88.72(1.09)|\n| nnPUe | .. | 92.45 (1.61) | 90.45 (2.26) | 94.73 (1.24) | 92.53 (1.55) | 92.48 (1.60) | 88.29 (2.43)|\n|nnPUe w/o normalize | .. | 90.95(1.61) | 88.18(2.40) | 94.38(2.74) | 91.13(1.56) | 91.00(1.61) |  85.98(2.25) |\n|Self-PU | .. | 90.08(0.47)  | 90.08 (0.47) | 89.35 (1.17) | 90.70 (1.73) | 90.00 (0.53) | 85.61 (0.69)|\n\n*Q2:* **Significance of causal inference in the context of the proposed work.**\n\n*A2:* Thanks for the nice question. The simple method is applied because we analyze the causality here is simple. Our main contributions are indeed not here. See the first question for our main contributions. We'll explore more complex ways to study this in the future.\n\nWe're using causal reasoning in reverse here. In general, we study the average causal effect of a treatment. However, we first complete the causal analysis of the processing (i.e. selection) and the label itself, and then use the known causal analysis, combined with the selected samples and the unselected samples to determine the distribution of positive samples.\n\nIn this paper, accept processing means that the sample is selected for labeled. And the result is refers to whether the sample itself is a positive sample. Without noise in the detection, whether the sample itself is a positive sample is determined at the time when the sample is generated, and the behavior of selecting is performed after the sample is generated. In causal inference, what happens after the result is produced has no effect on the result, i.e. whether the sample is selected has a causal effect on the label value is 0. In essence, individual causality is also 0.\n\nBased on such a simple causality, we use the method of reweighting the sample to estimate the positive probability of the sample.\n\n*Q3:* **The primary contribution of the paper should lie in the proposed method for estimating PS.**\n\n*A3:* It makes sense for us to introduce a regularizer here. The regularization method is used to solve the overfitting problem, and the normalization method is used to solve the problem of small estimated PS, which is a method to deal with different problems.\n\nAblation experiment (in the above table) showed that when the regularizer was removed, the performance was reduced. This will make it very easy to overfit, when we estimate the PS. The estimated PS in the label set is close to 1, with unlabeled is close to 0. The regularization method smooths the estimated PS. The regularization method is to solve the problem that the model underestimates PS. We also do additional experiments here to demonstrate that the performance of the model deteriorates when the regularization effect is removed under the nnPUe algorithm.\n\n*Q4:* **Comparison with selfPU.**\n\n*A4:* According to the aaai paper, it cannot estimate identifiable PS without making certain assumptions about the data. But according to the formula we gave in the first question, it's approximate. This is not explained by the self-monitoring method. Results in the above table show that our scheme is better than self-PU in the case of biased label datasets.\n"}, {"review_id": "Kxo3lr3VlZ", "reviewer": "Reviewer_7yJd", "summary": "This paper considers the problem that existing cost-sensitive-based methods often rely on strong assumptions that examples with an observed positive label were selected entirely at random in Positive-Unlabeled (PU) learning. The authors propose a PU learning enhancement (PUe) algorithm based on causal inference theory, which employs normalized propensity scores and normalized inverse probability weighting (NIPW) techniques to reconstruct the loss function, thus obtaining a consistent, unbiased estimate of the classifier and enhancing the model’s performance. Moreover, they investigate and propose a method for estimating propensity scores in deep learning using regularization techniques when the labeling mechanism is unknown. Experiments on three benchmark datasets demonstrate the proposed PUe algorithm significantly improves the accuracy of classifiers on non-uniform label distribution datasets compared to advanced cost-sensitive PU methods.", "questions": "Could the authors provide a more detailed explanation of how the causal inference method addresses the issue of bias in the labeled set not conforming to the SCAR assumption?", "limitations": " A significant limitation of this paper is its overall clarity and readability. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. One strength of this paper is its focus on addressing the prevalent issue of bias in the labeled set, which often does not conform to the Selected Completely At Random (SCAR) assumption. This assumption posits that the observed labeled examples are a random subset of the complete set of positive examples. In many real-world scenarios, this assumption does not hold true, leading to biased learning outcomes.", "weaknesses": "1.  This paper contains several spelling errors, which can potentially hinder the reader's understanding and overall perception of the work. For instance, on line 131, the term \"oftreament\" appears to be a typographical error where two words are mistakenly joined. Similarly, on line 149, \"porpensity\" is likely a misspelling of \"propensity,\" and on line 189, \"Apparemtly\" should be corrected to \"Apparently.\" Lastly, on line 220, \"pi\" might be a typo or a symbol that is not clearly defined in the context. These errors suggest a lack of thorough proofreading and can detract from the overall quality and credibility of the paper. It is highly recommended that the authors conduct a careful review of the manuscript for spelling, grammar, and typographical errors before final submission. \n2. A significant weakness of this paper lies in its lack of clear explanation and connection between the problem it aims to solve - the bias in the labeled set not conforming to the Selected Completely At Random (SCAR) assumption - and the causal inference method it employs to address this issue. Specifically, lines 123 to 145 appear to attempt to elucidate this connection, but the explanation falls short due to the lack of clear definitions and elaboration on key concepts. This lack of clarity can make it difficult for readers to understand the rationale behind the proposed method and its relevance to the problem.\n3. A notable weakness in the experimental section of the paper is the presentation of results in Tables 1 and 2. The current format does not clearly highlight the best-performing algorithm, making it difficult for readers to quickly and intuitively discern which method is superior and by what margin.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1:* **This paper contains several spelling errors.**\n\n*A1:* Thank you for your advice. We will carefully check spelling, grammar and typographical errors, thoroughly proofread the paper, improve the overall quality and credibility of the paper, and carefully revise it in the final version. Lines 131, 149, 189 do have errors. In line 220, \"pi\" should actually be \"$\\pi$\".\n\n*Q2:* **Could the authors provide a more detailed explanation of how the causal inference method addresses the issue of bias in the labeled set not conforming to the SCAR assumption?**\n\n*A2:* Thanks. There are four basic assumptions in causal inference. First, assignment mechanism must be probabilistic, $0<P(s=1|x_i, y_i=1) \\leq1$. for positive sample, (It can be extended to negative classes, see PUbNe) If the selection probability is 0, the sample is difficult to accurately determine without additional assumptions. Second, SUTVA means that the potential outcome (label value) of any individual does not change depending on whether other individuals are treated or not. Third. Consistent, there is no noise. Fourth, the core assumption on which our work relies, is that the assignment mechanism must be unconfounded, requiring that all the assignment probabilities $P(s |x_i, t_i=1)$ are free from dependence on the potential outcomes. allocation mechanism is independent of potential outcomes.\n\nAnd we think that this allocation mechanism should be completely dependent on the covariate that can be observed in the sample, i.e. $e(x_i)$. Usually, PU learning studies the case of $e(x_i)=c$. Here we look at the case where $e(x_i)$ is a variable function, that is, the SCR hypothesis (which does not conform to the SCAR hypothesis). And under the unconfounded hypothesis, according to the theory of causal inference, we can know\n$s_i \\perp x_i |e(x_i)$, or, equivalently, $e(x_i)=P(s_i=1|x_i, e(x_i),y_i=1)= P(s_i=1|e(x_i),y_i=1)$. and $s_i \\perp y_i(0),y_i(1) |e(x_i)$. where $y_i(0), y_i(1)$ is the potential result, and in PU learning there is $y_i(0)=y_i(1)=1$, i.e., the final potential result is positive regardless of whether the positive class is selected into the annotation set.\n\nWe are actually using causal reasoning backwards here, first completing the causal analysis of the processing (i.e. selection) and the label itself, and then using known causal analysis, combining selected and unselected samples to determine the distribution of positive samples.\n\nDetermine the causality: In PU case, accepting processing means selecting samples for labeling. The result refers to whether the sample itself is a positive sample. Whether a sample is a positive sample is determined when the sample is generated, and the behavior of selecting a sample for marking is executed only after the sample is generated.\nIn causal inference, what happens after the result is produced has no effect on the result, i.e. whether the sample is selected or not has a causal effect on the label value is 0. In essence, individual causality is also 0, which ensures that we can estimate the positive probability of samples based on selected samples and unselected samples.\n\nIn causal inference, we can think that the samples in each block are generated by completely randomized experiments, and estimate the causal effects in each block separately. Given knowledge of the propensity score, the sample weights can also be adjusted using propensity score adjustment weights to estimate causal effects. The individual causal effect is 0, and we know the distribution of covariates of the labeled and unlabeled samples, so we can estimate the positive probability with the estimated propensity score. Simply put, we want to increase the weight of samples with low labeling probability and reduce the weight of samples with high labeling probability. What kind of weight adjustment is used is based on the inverse probability weighting in causal inference.\n\nAccording to Rubin's formulas for Weighting Estimators and Blocking Estimators:\n\n$\\hat{\\tau}^{ht}= \\frac{1}{N}\\sum_{i=1}^{N} \\frac{s_{i}{Y_{i}}^{P}}{e({x}^{P}_{i})} $\n\n$-\\frac{1}{N} {\\sum_{i=1}}^N \\frac{(1-s_i){Y_i}^{P}}{1-e({x_i}^{P})}$\n\n$\\hat{\\tau}^{dif}(j)=\\frac{\\sum_{i:B_{i}(j)=1}s_i {Y_{i}}^{P}}{\\sum_{i:B_{i}(j)=1}s_i}- \\frac{\\sum_{i:B_{i}(j)=1}(1-s_i) {Y_{i}}^{P}}{\\sum_{i:B_{i}(j)=1}(1-s_i)}$\n\nOnly positive are labeled, so $Y^{P}_{i}=1$. Because the causal effect is 0. We have the following equation:\n\n$\\frac{1}{N_{p_j}}\\sum_{i=1}^{N_{p_j}} \\frac{s_{i}Y^{P}_{i}}{e({x_i}^{P})} $\n\n$= \\frac{1}{N_{p_j}}\\sum_{i=1}^{N_{p_j}} \\frac{(1-s_i){Y_i}^{P}}{1-e({x_i}^{P})}$,\n\n$N_{p_j}$ is the number of positive samples in the jth block. \n\nThe jth block represents a positive sample with a propensity score between $b_j$ and $b_j+\\epsilon$,\n\n $b_j \\in [0,1)$, $\\epsilon > 0$ and $b_j+\\epsilon \\leq 1$. $s_i=1$ means sample i is selected into the labeled set. \n\nSo we can get $T_j= \\sum_{i=1}^{n_{p_j}} \\frac{1}{e({x_i}^{L})}=\\sum_{i=1}^{N_{p_j}-n_{p_j}} \\frac{1}{1-e({x}^{'}_{i})}$, \n\n${n_{p_j}}={n_{L_j}}$ is the number of labeled samples in the jth block, ${x}^{'}_{i}$ are unlabeled positive samples.\n\n$(\\frac{b_j}{b_j+\\epsilon})N_{p_j} \\leq \\frac{n_{p_j}}{b_j+\\epsilon} < T_j \\leq \\frac{n_{p_j}}{b_j} < (\\frac{b_j+\\epsilon}{b_j})N_{p_j}$.\n\nMake $\\epsilon \\to 0$, $T_j \\to N_{p_j}$. So $\\sum T_j = \\sum_{i=1}^{n_{L}} \\frac{1}{e(x^{L}_{i})} = N_p$, which is used in the unbiased proof in the paper.\n\n*Q3:* **A notable weakness in the experimental section of the paper is the presentation of results in Tables 1 and 2. The current format does not clearly highlight the best-performing algorithm.**\n\n*A3:* Thanks for your advice, we will mark the best results, clearly highlighting the best performing algorithms so that readers can quickly identify them. In the final version of the paper, we will refine the presentation of the results."}, {"review_id": "aUveg61hDb", "reviewer": "Reviewer_fhZG", "summary": "This paper considers PU learning issue. Existing cost-sensitive-based methods often rely on strong assumptions that examples with\nan observed positive label were selected entirely at random. This work relaxes the assumption and proposes a unbiased novel method PUe based on the causal theory.", "questions": "See weaknesses", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation to relax the Selected Completely At Random assumption is practical and interesting.\n\n2. Generalization theory to guarantee the effectiveness.\n\n3. Excellent experiment performance to guarantee the effectiveness of the algorithm. ", "weaknesses": "I am very confused on how to estimate the prior pi.\n\nPlease conduct experiments with wild OOD detection method Training OOD Detectors in their Natural Habitats.\n\nI have not got where you have used the causal theory. Could you point out the causual theory you have used and clarify what causal assumptions you have used in you theorems?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1:* **I am very confused on how to estimate the prior $\\pi$.**\n\n*A1:* Thanks. There is a lot of work on estimating $\\pi$, such as the paper \"Learning from corrupted binary labels via class-probability estimation\" presented at ICML 2015, \"Mixture proportion estimation via kernel embedding of distributions\" presented at ICML 2016, \"Estimating the class prior and posterior from noisy positives and unlabeled data\" presented at NIPS 2016, \" Class-prior estimation for learning from positive and unlabeled data\" presented at Machine Learning 2017.\n\nCommon methods for estimating $\\pi$ include kernel method, kernel logistic regression, mixture proportion estimation and univariate transforms.\n\nEstimates on $\\pi$ are not our main contribution. In papers on PU learning loss functions, $\\pi$ is usually assumed to be known. Our main contribution is four points. Firstly, we relax the assumption of local certainty and propose a simple method to estimate the propensity score under the assumption of SAR, and apply it to PU learning to improve the performance of the algorithm. Second, using deep learning to estimate propensity score (PS) rather than linearity is better than logistic regression. Third, our PS estimation method can be extended to the case of negative classes (e.g. PUbNe). Fourth, our new algorithm can simply be coupled with most cost-sensitive algorithms to improve performance, as shown in the paper and supplementary experiments. We will include more analysis and results in the final version.\n\n*Q2:* **Please conduct experiments with wild OOD detection method \"Training OOD Detectors in their Natural Habitats\".**\n\n*A2:* Thank you very much for your question. Wild OOD detection can also be regarded as a PU problem. $P_{wild}=\\pi P_{out}+(1-\\pi)P_{in}$. \nWe separate in-distribution (ID) data and OOD data from input data by using a small number of labeled samples and a large number of wild data samples. We can regard ID data as P class, OOD data as N class, wild data as unlabeled data, and labeled ID data as label data.\nWild OOD detection can be tested and compared with our algorithm. However, the common dataset of PU work in the previous literature is not this OOD task, so I didn't consider doing this experiment at first. And due to time constraints and additional experiments, we did not complete code refactoring and experimental simulations. We will supplement this experimental comparison in the final paper to further explore the performance of our algorithm.\n\n*Q3:* **Could you point out the causal theory you have used and clarify what causal assumptions you have used in you theorems?**\n\n*A3:* Thanks for your question. There are four basic assumptions in causal inference. 1st, assignment mechanism must be probabilistic, $0<P(s=1|x_i, y_i=1) \\leq1$. for positive sample, (It can be extended to negative classes, see PUbNe) If the selection probability is 0, the sample is difficult to accurately determine without additional assumptions. 2nd, SUTVA means that the potential outcome (label value) of any individual does not change depending on whether other individuals are treated or not. 3rd. Consistent, there is no noise. 4th, the core assumption on which our work relies, is that the assignment mechanism must be unconfounded, requiring that all the assignment probabilities $P(s |x_i, t_i=1)$ are free from dependence on the potential outcomes. allocation mechanism is independent of potential outcomes.\n\nAnd we think that this allocation mechanism should be completely dependent on the covariate that can be observed in the sample, i.e. $e(x_i)$. Usually, PU learning studies the case of $e(x_i)=c$. Here we look at the case where $e(x_i)$ is a variable function, that is, the SCR hypothesis (which does not conform to the SCAR hypothesis). And under the unconfounded hypothesis, according to the theory of causal inference, we can know\n$s_i \\perp x_i |e(x_i)$, or, equivalently, $e(x_i)=P(s_i=1|x_i, e(x_i),y_i=1)= P(s_i=1|e(x_i),y_i=1)$. and $s_i \\perp y_i(0),y_i(1) |e(x_i)$. where $y_i(0), y_i(1)$ is the potential result, and in PU learning there is $y_i(0)=y_i(1)=1$, i.e., the final potential result is positive regardless of whether the positive class is selected into the annotation set.\n\nWe are actually using causal reasoning backwards here, first completing the causal analysis of the processing (i.e. selection) and the label itself, and then using known causal analysis, combining selected and unselected samples to determine the distribution of positive samples.\n\nDetermine the causality: In PU case, accepting processing means selecting samples for labeling. The result refers to whether the sample itself is a positive sample. Whether a sample is a positive sample is determined when the sample is generated, and the behavior of selecting a sample for marking is executed only after the sample is generated.\nIn causal inference, what happens after the result is produced has no effect on the result, i.e. whether the sample is selected or not has a causal effect on the label value is 0. In essence, individual causality is also 0, which ensures that we can estimate the positive probability of samples based on selected samples and unselected samples.\n\nIn causal inference, we can think that the samples in each block are generated by completely randomized experiments, and estimate the causal effects in each block separately. Given knowledge of the propensity score, the sample weights can also be adjusted using propensity score adjustment weights to estimate causal effects. The individual causal effect is 0, and we know the distribution of covariates of the labeled and unlabeled samples, so we can estimate the positive probability with the estimated propensity score. Simply put, we want to increase the weight of samples with low labeling probability and reduce the weight of samples with high labeling probability. What kind of weight adjustment is used is based on the inverse probability weighting in causal inference.\n"}, {"review_id": "nSnoWONn7p", "reviewer": "Reviewer_w1uD", "summary": "This paper works on PU learning with a selection bias. They propose a weighted risk estimator to solve this problem, and estimate the weight via neural networks. Extensive experimental results validate the effectiveness of the proposed approach.", "questions": "Please see \"Weaknesses\".", "limitations": "None.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "- The studied problem is very important for PU learning.\n- The experimental results is good, since the proposed mechanism improves previous PU learning methods under the biased setting.", "weaknesses": "- First, I found potential mistakes in the derivations of  $\\mathbb{E}[\\hat{R}\\_{PUe}(g)]=R_{PN}(g|y)$ in the supplemental materials. In Line 10 of Section 3, the authors use $n_{P}/n$ to replace $\\pi$, but I do not think so. Since there are positive examples in the unlabeled data ($s=0, y=1$), the class prior is under estimated. I will reconsider it if I need to be corrected.\n\n- Second, the major problem of this paper is that it is very similar to [3]. The proposed approach and theoretical results are very similar. For the theoretical results, Theorem 1 and 2 are very similar to Proposition 3 and 4 in [3]. For the methodology, it only changes the MLE method in [3] to a neural network. Therefore, the contribution is limited.\n\n- Third, this work actually works on “single-training-set PU learning”, while uPU and nnPU work on \"case-control PU learning\". Please see Gong et al. (2021) for more explanations. The authors should discuss the problem here.\n\n- Besides, I also have questions on the weight estimation via a neural network. The paper says that it regards positive examples as \"positive\" and unlabeled examples as \"negative\". My concern is that the model output is $p(s=1|x)$, instead of $p(s=1|y=1,x)$. Can the authors explain it?\n\n- It seems that the paper was written in a hurry since it is very difficult to read. For example, what's after Line 96? It has not reached the bar of a scientific paper up to now. I suggest revising this paper in detail.\n\n---\nReference\n- Gong et al, Instance-Dependent Positive and Unlabeled Learning with Labeling Bias Estimation, TPAMI 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1:* **The class prior is under estimated by using $n_P/n$ to replace $\\pi$.**\n\n*A1:* Thanks for the nice conern. We actually use $N_P/n$ to replace $\\pi$, that is, the proportion of the total number of positive samples to the total number of samples. $n_P=n_L$ is the number of labeled samples, $N_P$ is the number of positive samples, $n$ is the total number of samples, And $\\sum_{j=1}^{n_P} \\frac{1}{e(x_j^P)}=N_P$. Obviously, $N_P>n_P$. Therefore, the result is reasonable and does not underestimate the class prior.\n\n*Q2:* **The major problem of this paper is that it is very similar to [3]. It only changes the MLE method in [3] to a neural network.**\n\n*A2:* Thanks. Our work differs greatly from [3] with four aspects. First, the propensity score estimated in [3] is actually unidentifiable, and we stabilize our estimate of the propensity score by relaxing the Local Certainty assumption. Second, the deep learning model is used to estimate the propensity score instead of the linear model, which is better than logistic regression. Third, we improve the accuracy of cost-sensitive PU learning under biased labeling sets through normalized coupling commonly used PU learning, which is not involved in [3]. Finally, our method can be generalized to negative classes, and experiments are done (see PUbNe), which is not involved in [3]. The impact is great and the contribution is not small.\n\nSince our initial conditional assumptions are similar (based on the four basic assumptions of causal inference: Probabilistic, SUTVA, Consistent, unconfounded), the theoretical results look similar. In fact, our deep learning method coupling the cost-sensitive PU learning, which is different from [3].\n\n*Q3:* **This work actually works on “single-training-set PU learning”, while uPU and nnPU work on \"case-control PU learning\".**\n\n*A3:* Thanks for the nice conern. Our method can be implemented on both scenarios.\n\nThe observed positive examples are generated from the same distribution in both scenarios. Hence, the learner has access to a set of examples drawn i.i.d. from the true distribution and a set of examples that are drawn from the positive distribution according to the labeling mechanism that is defined by the propensity score e(x). As a result, most methods（includes uPU and nnPU can handle both scenarios, but the derivation differs. Our method is coupled to cost-sensitive methods such as uPU, nnPU, and PUbN. Therefore, it can be implemented in two scenarios.\n\nIn our paper, we mainly conduct experiments on the single-training-set, which is following the previous works. \n\n*Q4:* **My concern is that the model output is $p(s=1|x)$, instead of $p(s=1|x,y=1)$.**\n\n*A4:* Thanks for the nice concern. It's true that the estimate here is $p(s=1|x)$, but I'll show later that our estimate is reasonable, and that's what contributes to our relaxation of the Local Certainty assumption. In fact, we would point out that method in [3] of calculating the propensity score (PS) achieves this goal by maximizing the probability of observing the data, treating both the class posterior and the PS as latent variables. Unfortunately, this did not produce a identifiable PS; There are multiple false assumptions about the PS, which are as likely to be returned by this method as the true PS. Further, the estimated PS in the Local Certainty Scenario is something else, but in the Probabilistic Gap Scenario, the PS is unidentifable without additional assumptions. But in Local Certainty setting, we assume the relationship between the observed features and the true class is a deterministic function, this assumption is too strong to be realistic. We consider a large probabilistic gap. and positive instances that resemble negative instances are less likely to be labeled, i.e. $|P (y=1|x) -P(y=0|x)|>0.8$ and $P(y=1|x^L) > P(y=0|x^L)$, L means the labeled samples. According to the definition of probabilistic score, we have the $\\sum e = n_p$ restriction, but $\\hat{e}(x_i^N)>0$ on the negative will result in underestimated PS on labeled samples. Normalization can improve the estimation effect.\n\nOnly labeled samples are considered. We have $0.9<P(y=1|x_i^L) \\leq 1$. Obviously, $0.9e(x_i^L) < \\hat{e}(x_i^L)\\leq e(x_i^L)$. This shows that our estimated PS for the labeled sample is smaller than the true PS. Moreover, the PS of the estimated negative sample is greater than 0, i.e. $\\hat{e}(x_i^N)>0$. According to the PS definition, we have the following equation $\\sum \\hat{e_i} = {n_p}$. In Local Certainty Scenario and [3], PS of negative samples also share part of the probability, the PS of the positive sample is underestimated. Therefore, we need to use the normalization method to improve the accuracy of our classification algorithm. \n$\\frac{\\frac{1}{e(x_i^L)}}{\\sum_{j} \\frac{1}{e(x_j^L)}} = A$.\n\n$0.9A < \\frac{\\frac{1}{\\hat{e}(x_i^L)}}{\\sum_j \\frac{1}{\\hat{e}(x_j^L)}} < 10A/9$. This shows that our estimate is relatively stable.\n\n*Q5:* **It seems that the paper was written in a hurry since it is very difficult to read. For example, what's after Line 96?**\n\n*A5:* Thanks for the advice. We will carefully check for spelling, grammar and typographical errors, thoroughly proofread the paper, improve the overall quality, credibility and readability of the paper, and carefully revise it in the final version.\nLine 96 should end with a period instead of a colon \":\".\n"}]

Author rebuttal:

Meta-review: This paper notes that existing Positive-Unlabeled (PU) learning methods assume that positive samples are selected entirely at random, ignoring the prevalent selection bias in real-world PU problems. To overcome this limitation, the authors propose a PU learning enhancement (PUe) algorithm based on causal inference theory. They adapt sample re-weighting methods, to be specific, inverse propensity weighting (IPW) to PU learning, which assigns a weight to each positive sample. Based on the weights, they obtain a new unbiased PU risk estimator. They also propose a method that can compatible to deep networks to estimate the weights when the labeling mechanism is unknown.

The strength of this paper includes the importance of the studied problem (biased positive data), writing and experiments. For example, the problem of studying biased positive data is highly significant and practical. In many real-world scenarios, the selection of positive samples is often biased, while existing methods typically assume that the selection of positive samples is completely random. The proposed PUe algorithm in this paper addresses this issue and effectively handles biased labeled data, filling a research gap in the field of PU learning. Moreover, this paper is well-written, with clear descriptions of the problem background, method principles, and experimental design and results. The authors introduce causal inference theory and utilize techniques such as normalized propensity scores and inverse probability weighting to provide a viable solution, which is presented in a clear and concise manner. Last but not least, experimental results demonstrate the effectiveness of the method: The paper provides experimental evidence of the proposed PUe algorithm's effectiveness.

Overall, the clarity and novelty are above the bar of NeurIPS. While the reviewers had some concerns, the authors did a particularly good job in their rebuttal. Thus, most of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (poster)