Title: PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference
Venue: NeurIPS 2023
Accepted: yes
Keywords: PU learning

causal inference

semi-supervised learning
Abstract: Positive-Unlabeled (PU) learning aims to achieve high-accuracy binary classification with 
limited labeled positive examples and numerous unlabeled ones. Existing cost-sensitive-based 
methods often rely on strong assumptions that examples with an observed positive label were 
selected entirely at random. In fact, the uneven distribution of labels is prevalent in 
real-world PU problems, indicating that most actual positive and unlabeled data are subject 
to selection bias. In this paper, we propose a PU learning enhancement (PUe) algorithm 
based on causal inference theory, which employs normalized propensity scores and normalized 
inverse probability weighting (NIPW) techniques to reconstruct the loss function, thus 
obtaining a consistent, unbiased estimate of the classifier and enhancing the model's 
performance. Moreover, we investigate and propose a method for estimating propensity scores 
in deep learning using regularization techniques when the labeling mechanism is unknown. 
Our experiments on three benchmark datasets demonstrate the proposed PUe algorithm significantly 
improves the accuracy of classifiers on non-uniform label distribution datasets compared to 
advanced cost-sensitive PU methods. Codes are available at https://github.com/huawei-noah/Noah-research/tree/master/PUe and https://gitee.com/mindspore/models/tree/master/research/cv/PUe.

PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference

Abstract

Positive-Unlabeled (PU) learning aims to achieve high-accuracy binary classification with limited labeled positive examples and numerous unlabeled ones. Existing cost-sensitive-based methods often rely on strong assumptions that examples with an observed positive label were selected entirely at random. In fact, the uneven distribution of labels is prevalent in real-world PU problems, indicating that most actual positive and unlabeled data are subject to selection bias. In this paper, we propose a PU learning enhancement (PUe) algorithm based on causal inference theory, which employs normalized propensity scores and normalized inverse probability weighting (NIPW) techniques to reconstruct the loss function, thus obtaining a consistent, unbiased estimate of the classifier and enhancing the model's performance. Moreover, we investigate and propose a method for estimating propensity scores in deep learning using regularization techniques when the labeling mechanism is unknown. Our experiments on three benchmark datasets demonstrate the proposed PUe algorithm significantly improves the accuracy of classifiers on non-uniform label distribution datasets compared to advanced cost-sensitive PU methods. Codes are available at https://github.com/huawei-noah/Noah-research/ tree/master/PUe and https://gitee.com/mindspore/models/ tree/master/research/cv/PUe .

1 Introduction

In the era of big data, deep neural networks have achieved outstanding performance across various tasks, even surpassing human performance in many instances, particularly in traditional binary classification problems. The success of these deep neural networks often hinges on supervised learning using large quantities of labeled data. However, in reality, acquiring even binary labels can be challenging. For instance, in recommendation systems, users' multiple clicks on films may be considered as positive samples. Nonetheless, all other films cannot be assumed uninteresting and thus should not be treated as negative examples; instead, they should be regarded as unlabeled ones.

The same issue emerges in text classification, where it is typically more straightforward to define a partial set of positive samples. However, due to the vast diversity of negative samples, it becomes difficult or even impossible to describe a comprehensive set of negative samples that represent all content not included in the positive samples. Similar situations occur in medical diagnostics, malicious URL detection, and spam detection, where only a few labeled positives are available amidst a plethora of unlabeled data. This scenario is a variant of the classical binary classification setup, known as PU. In recent years, there has been a growing interest in this setting. Positive-Unlabeled (PU) learning primarily addresses the challenge of learning binary classifiers solely from positive samples and unlabeled data.

In previous research, numerous PU algorithms have been developed, with cost-sensitive PU learning emerging as a popular research direction. Methods such as [19, 18, 6] reweight positive and negative risks by hyper-parameters and minimize it. In addition, Self-PU [4] introduces self-supervision to nnPU through auxiliary tasks, including model calibration and distillation; ImbPU [23] oversamples and modifies sample weights to address unbalanced data. Dist-PU[24] corrects the negative preference of the classification model through prior information. PUSB [15] maintains the order-preserving assumption. However, these methods necessitate the assumption that their checked set is uniformly sampled from the population, or else the PU learning risk estimator ceases to be an unbiased or consistent estimator, otherwise resulting in reduced model accuracy.

In reality, the labeled set is often biased and does not conform to the Selected Completely At Random (SCAR) assumption [10, 3], which posits that the observed labeled examples are a random subset of the complete set of positive examples. Consequently, it is essential to relax the assumption of the labeled set and replace it with a more general assumption about the labeling mechanism: the probability of selecting positive examples to be labeled depends on their attribute values, known as the Selected At Random (SAR) assumption [10].

The work most closely related to us is the paper [7] by W Gerych et al. The focus of this article is to develop a method for generating identifiable propensity scores. Moreover, this paper proves that the general propensity score is unidentifiable in any standard PU data assumption except for the probability gap scenario. In this paper, two sets of assumptions are proposed to estimate propensity scores: one is Local Certainty and the other is Probabilistic Gap. Under the local certainty hypothesis, there is no overlap between the positive and negative hypotheses, p ( x P | y = 0) = 0 . Under the Probabilistic Gap assumption, it is assumed that e = k ∗ p ( y = 1 | x ) is linear and there is an anchor point.

However, the above assumptions are too strict. We show in the appendix that the estimated propensity scores cannot be completely unbiased. To address the aforementioned problem, we propose a causal-inference-inspired

Proposed PUe

Tradition PU

PU learning framework, termed PUe. Our approach aims to weight the original PU learning risk estimator under biased conditions, thereby obtaining a new unbiased or consistent PUe learning risk estimator, which is achieved through the inverse probability weighting technique of known normalized propensity scores. Given that propensity scores for samples are typically unknown, we apply regularization techniques to the deep learning classifier to estimate the propensity score of each sample in the marker set. An illustration of the proposed method is shown in Figure 1. We also provide an estimate of the generalization error bound for empirical risk minimization. Experimental results demonstrate that the proposed PUe framework can be applied to advanced PU algorithms, such as PUbN and Dist-PU, to enhance their performance in the SAR scenario. Our main contributions are summarized as follows:

•We relax our assumption of local determinism.

•We use a deep learning model to estimate propensity scores instead of a linear model.

•Our propensity score estimation method can be extended to the case of negative classes (e.g. PUbNe).

·Our new algorithm can simply be coupled with most cost-sensitive algorithms to improve performance. The PUe algorithm achieved superior results over existing solutions on standard benchmarks (MNIST, CIFAR10) and the Alzheimer's Disease Neuroimaging Initiative (ADNI) database.

2 Methodology

In this section, we conduct a thorough review of existing PU algorithms, examining their limitations and challenges, particularly in the context of biased labeling scenarios that result in diminished accuracy. To address these issues, we introduce a novel PUe algorithm, specifically designed to

overcome the aforementioned limitations, thus enhancing the performance of PU algorithms under biased labeling conditions.

2.1 Review of PU classification

In standard PN classification, let x ∈ R d and y ∈ { +1 , -1 } be the input samples and its corresponding labels. We are given positive data and negative data that are sampled independently from p P ( x ) = p ( x | y = +1) and p N ( x ) = p ( x | y = -1) as χ P = { x P i } n P i =1 and χ N = { x N i } n N i =1 . We denote the class prior probability as π = p ( y = 1) , where we follow the convention to assume π as known throughout the paper [16]. In fact, in the paper [14] by S Jain et al., it is pointed out that class prior can be estimated in the case of the proportion of classes that are biased in the labelling data. Let g : R d → R be the binary classifier and θ be its parameter and the L : R ×{ +1 , -1 } → R + be a loss function.

Let us denote by R P ( g, +1) = E x ∼ p P ( x ) [ L ( g ( x , +1))] , R N ( g, -1) = E x ∼ p N ( x ) [ L ( g ( x , -1))] . R P ( g, +1) is the expected risk loss of a positive sample and R N ( g, -1) is the expected risk loss of a negative sample.

The risk of classifier g , ˆ R PN ( g ) can be approximated by:

where ˆ R P ( g, +1) = 1 n P ∑ n P i =1 L ( g ( x P i ) , +1) and ˆ R N ( g, -1) = 1 n N ∑ n N i =1 L ( g ( x N i ) , -1) . By minimizing ˆ R PN ( g ) . We obtain the ordinary empirical risk minimizer ˆ g PN .

In standard PU classification, instead of N data χ N , we only have a set of U samples drawn from the marginal density p ( x ) , χ U = { x U i } n U i ∼ p ( x ) . Because of the fact p ( x ) = πp P ( x )+(1 -π ) p N ( x ) , we have the unbiased risk estimator, which key idea is to use the following equality:

where R U ( g, -1) = E x ∼ p ( x ) [ L ( g ( x ) , -1)] and R P ( g, -1) = E x ∼ p P ( x ) [ L ( g ( x ) , -1)] . As a result, we can approximate the unbiased classification risk (1) by:

where ˆ R P ( g, -1) = 1 n P ∑ n P i =1 L ( g ( x P i ) , -1) and ˆ R U ( g, -1) = 1 n U ∑ n U i =1 L ( g ( x U i ) , -1) . By minimizing ˆ R PU ( g ) we obtain the ordinary empirical risk minimizer ˆ g PU .

In theory, the risk (1 -π ) R N ( g, -1) = R U ( g, -1) -πR P ( g, -1) is also positive. However, if the model of g is too flexible, ˆ R PU (ˆ g PU ) indeed goes negative and the model severely overfits the training data. We can use the non-negative risk estimator for PU learning to alleviate overfitting[16]:

2.2 PU Classification in Biased Scenarios

PU learning mostly assumes that it runs under an ideal situation, that is, all labeled samples are selected completely at random from all positive samples, which is also called The Selected Completely At Random (SCAR).

Defintion 1. SCAR (Selected Completely At Random [6]): Labeled examples are selected completely at random, independent from their attributes, from the positive distribution. The propensity score e ( x ) , which is the probability for selecting a positive example, is constant and equal to the label frequency:

However, many PU learning applications suffer from labeling bias. The Selected Completely At Random (SCAR) assumption does not conform to realtity. For example, whether someone clicks

on a sponsored search ad is influenced by the position in which it is placed. Similarly, whether a patient with a disease will see a doctor depends on his socio-economic status and the severity of his symptoms. The Selected At Random (SAR) assumption, is the most general assumption about the labeling mechanism: the probability for selecting positive examples to be labeled depends on its attribute values [1].

Defintion 2. SAR (Selected At Random [3]): The labeling mechanism depends on the values of the attributes of the example. However given the attribute values, it does not depend on the probability of the example being positive. Instead of assuming a constant probability for all positive examples to be labeled, it assumes that the probability is a function of a subset of an example's attributes: e ( x ) = p ( s = 1 | x , y = 1) .

In biased labeling scenarios, the accuracy of common PU algorithms decreases significantly. So we need to develop the PUe algorithm. The mathematical representation of equation is exactly the same as that of the propensity score in causal inference. Causal inference is the process of determining the actual and independent effects (effects) of a given phenomenon (cause) within a larger system. Causality inference can give evidence of causality model established by causal reasoning. Causal inference is therefore fundamentally a missingdata problem [13], which is very similar to the PU problem. The PU problem is essentially a classification problem of missing negative labels.

What we're actually using here is reverse causal reasoning. In general, we study the average causal effect of a treatment. However, we first complete the causal analysis of the processing (i.e. selection) and the label itself, and then use the known causal analysis, combined with the selected samples and the unselected samples to determine the distribution of positive samples. In this paper, accept processing means that the sample is selected for labeled. And the result is refers to whether the sample itself is a positive sample. Without noise in the detection, whether the sample itself is a positive sample is determined at the time when the sample is generated, and the behavior of selecting is performed after the sample is generated. In causal inference, what happens after the result is produced has no effect on the result, i.e. whether the sample is selected has a causal effect on the label value is 0. In essence, individual causality is also 0. Based on such a simple causality, we use the method of reweighting the sample to estimate the positive probability of the sample.

Since the causal effect of the labeling mechanism on the sample category is zero, known causality can be utilized to improve the performance in the case of biased PU problems. Rubin Causal Model of (RCM) [12] causal inference are extensively used within Statistics, Political Science, Economics, and Epidemiology for reasoning about causation. Due to the presence of confounders, estimating the average treatment effect directly using the average of observed treatment/control outcomes would include a spurious effect, which is called selection bias, brought by the confounders. A key concept in RCM is balancing score [11], reweighting samples by which is one of the most effective methods to overcome selection bias. Besides, propensity score is a special case of balancing score. The propensity score is defined as the conditional probability of treatment given background variables [22]. Propensity score is one of the core concepts of assignment mechanism in RCM. Propensity scores play a central role in observational studies of causality. Propensity scores can be used to reduce selection bias by equating groups based on these covariates, which is called inverse propensity weighting (IPW) [21].

In causal inference, inverse probability weighting (IPW) [21] is a standard statistical technique about propensity score for calculating statistics standardized to a pseudo-population different from that in which the data was collected. Study designs with a disparate sampling population and population of target inference are common in application. Weighting, when correctly applied, can potentially improve the efficiency and reduce the bias of unweighted estimators. In the context of causal inference and survey methodology, propensity scores are estimated (via methods such as logistic regression, random forests, or others), using some set of covariates. These propensity scores are then used as estimators for weights to be used with Inverse probability weighting methods. We incorporate the propensity score when learning in a PU setting by using the propensity scores to reweight the data. The examples are weighted with the inverse of their propensity score.

A crucial difference with the propensity score from causal inference is that our score is conditioned on the class being positive. IPW cannot be applied when working with positive and unlabeled data, because we have zero probability for labeling negative examples. But we can only do a weighting on positive. For each labeled example ( x i , s = 1) , which has a propensity score e i , there are expected

to be 1 e i positive examples. So we can use normalized inverse probability weighting on positive to emtimate the unbiased expectations on positive.

Similarly, R P ( g, -1) = E x ∼ p P ( x ) [ L ( g ( x , -1))] = E c ∈ (0 , 1] { E x ∼ p P ( x ) [ L ( g ( x , -1)) | e ( x ) = c ] } .

where e ( x P i ) is the true propensity scores of positive samples and ˆ e ( x P i ) is the estimate of propensity scores, while ˜ e ( x P i ) and ˜ ˆ e ( x P i ) is the normalized inverse probability weighting on positive samples. We can use the real normalized propensity score of positive samples ˜ e ( x P i ) to construct the PUe's loss function, or use the estimated normalized propensity score of positive samples ˜ ˆ e ( x P i ) to construct the loss function. Since then, we use ˜ e ( x P i ) to represent them.

From the normalization, we can know the following formula: ∑ n P i =1 1 ˜ e ( x P i ) = 1 , ∑ n P i =1 1 ˜ ˆ e ( x P i ) = 1 ,

number of all positive data.

n P = n L is the number of labeled samples, N P is the number of positive samples, n is the number of samples. It is clear that to be unbiased, we have to make E p ( x | y =1) = n P N P , which indicates that ∑ n j =1 e ( x ) = n P . And E p ( x | s =1) 1 e ( x P j ) = N P . Since e ( x N i ) > 0 , existing propensity score estimation methods such as SAR-EM in [3] often underestimate the propensity score of positive samples, which will cause co-directional bias or produce an unidentifiable result (overfitting) that all labeled samples have a propensity score close to 1 and all unlabeled samples have a propensity score close to 0.

The normalization technique is used to make the deviations no longer in the same direction (codirectional error will produce worse results). Moreover, the maximum estimation error ratio of sample weights is reduced. The regularization technique is used to alleviate the overfitting problem, and the work does not assume a known propensity score or a positive sample distribution. After normalization

To calculate the unbiased PU loss in the biased scenario, we must know the propensity score of each labeled sample. But this propensity score we usually don't know. So we consider two cases: (1) the true propensity scores we know and (2) we must estimate the propensity scores from data.

Case 1: Know the True Propensity Scores. In standard PN Classification, the true PN risk with real class y of size n can be formulated as follows:

Defintion 3 (IPW). Given the propensity scores e and PU labels, the inverse probability weighting estimator of ˆ R PUe ( g ) is:

1

0

ො

𝑦 = 0

ො

𝑦 = 1

E𝑥[1 - 𝑠]

E𝑥[𝑠]

𝑥 ∈ X𝑃

1

0

ො

𝑦 = 0

ො

𝑦 = 1

E𝑥[1 - 𝑠]

E𝑥[𝑠]

𝑥 ∈ X𝑈

1

0

𝑦 = 0

R𝑥(𝑠, 1)

𝑥 ∈ X𝑃

R𝑥(𝑠, -1)

1

0

𝜋𝑃

𝑥 ∈ X𝑈

𝜋𝑁

Step 1

Step 3

1

0

e(x)

inverse probability

Step 2

𝑦 = 1

𝑦 = 0

𝑦 = 1

Estimated network

Classification network

This estimator is unbiased.

When the propensity score is known, not only the loss function reconstructed by the PUe algorithm is unbiased, the expectation of the PUe loss function is equal to the real PN loss function, but also the difference between the PUe loss function value and the real PN loss function value is probability bounded. However, in other PU algorithms where the label distribution is biased, the expectation of the loss function is biased [3].

Theorem 1: Inverse Probability Weighting Error Bound. For any predicted classes ˆ y and the real class y , with probability 1 -η , the inverse probability weighting estimator ˆ R PUe ( g ) does not differ from the true PN loss function R PN ( g | y ) more than: | R PN ( g | y ) -ˆ R PUe ( g ) | ≤ √ L 2 max ln 2 η 2 n . with L max is the maximum absolute value of cost function L ( g, y ) .

Theorem 2: Error Bounds of Common PU Algorithm in Biased Scenarios. For any predicted classes ˆ y and the real class y , with probability 1 -η , the inverse probability weighting estimator ˆ R PU ( g ) does not differ from the true PN loss function R PN ( g | y ) more than: | R PN ( g | y ) -ˆ R PU ( g ) | ≤ 2 πL max ∗ n u n u + n p + √ L 2 max ln 2 η 2 n .

In the biased scenario, the error bound of PU algorithm is larger than that of PUe algorithm.

Case 2: Estimate the Propensity Scores from Data. But in practice, the probability of the sample being labeled is usually unknown. For example, in ad recommendation, we do not actually know the probability of everyone clicking on the ad, that is, the propensity score is unknown. So in order to get the loss function of PUe algorithm for the unknown label distribution, we must have an estimate ˆ e of the propensity score.

In this case, the bias of the inverse probability weighting estimator is:

From the bias, we can find that the propensity scores only need to be used accurately for positive examples. Apparently, L ( g ( x i ) , +1) -L ( g ( x i ) , -1) < 0 for positive examples. And when incorrect propensity score is close to 0 or 1 (especially tend towards 0), there will be a large bias. Overestimate normalized inverse propensity scores.

Underestimated propensity scores are expected to result in a model with a higher bias. Lower propensity scores result in learning models that estimate the positive class to be more prevalent than it is, which results in a larger [ L ( g ( x i ) , +1) -L ( g ( x i ) , -1)] for positive samples.

where π 1 = n P n P + n U .

In reality, it is almost impossible to know the propensity score of the sample, which requires us to estimate the propensity score of the sample. However, the traditional statistical estimation method can not be used directly in deep neural network. This requires us to design a new method to estimate sample propensity scores.

According to the paper [7]: In the Local Certainty scenario, we assume the relationship between the observed features and the true class is a deterministic function f : X → Y , where X is the feature space and Y = { 0 , 1 } , while allowing the propensity score to be an arbitrary probability function. but this assumption is too strong. In fact, we only need to have hypotheses | P ( y = 1 | x ) -P ( y = 0 | x ) | > θ and P ( y = 1 | x L ) > P ( y = 0 | x L ) to estimate the normalized propensity score well for sample classification. (where L represents the labeled sample and is used to distinguish it from the positive category.)

Otherwise, a linear function is usually used to estimate the propensity score in scenarios where the local certainty scenario is not met. Our supplementary experiments in the supplemental materials show that the probability function fitted using deep learning is better than the linear estimation in the MNIST experiment.

We use labeled data as positive samples and unlabeled data as negative samples. We use binary network and regularization technology to construct a scoring network e ( x ) . Finally, the output of the network passes through the sigmoid function to ensure that the output value is between 0 and 1. The score it outputs for each sample is the estimated probability of the sample being labeled (propensity score). The purpose of introducing the regularization technique is to prevent overfitting. Otherwise, the output value of the labeled sample is very close to 1 and the output value of the unlabeled sample is very close to 0. This is not what we expect. The ideal propensity score value should be far away from 0 and 1. Therefore, we introduce such a constraint in the classification loss function by combining the nature of propensity scores, where the sum of propensity scores of all samples is equal to the number of labeled samples: α e | ∑ x i ∈ χ P ∪ χ U e ( x i ) -n P | .

where α e ajusts the importance of regular term.

Theoretically, for each labeled example ( x i , s = +1) , which has a porpensity score e i , there are expected to be 1 e i positive examples, and of which 1 e i -1 did not get selected to be labeled. Each labeled example gets a weight 1 e i and the negative example is added to the dataset that gets a negative weight 1 -1 e i .

According to [7, 3], P ( ˆ Y | E,L ) = 1 n ∑ n i =1 s i ( 1 e i δ 1 +(1 -1 e i ) δ 0 ) + (1 -s i ) δ 0 , using Propensity Sores For Classification. However, as discussed above, existing methods always underestimate

propensity scores. As a result, PU learning classification is performed by using the foregoing formula, which reduces model classification performance.

We found that in cost-sensitive PU learning technique, the training instances are properly reweighted. and a common way to score propensity is inverse probability weighting (a reweighting technique). Conditionally, the two methods can be combined to solve the defects of the two methods themselves. The disadvantage of the cost-sensitive PU learning technique is that the classification effect of the application deteriorates when the labeled samples are biased. The propensity score is estimated by the fact that the propensity score of positive samples is always underestimated, which leads to the decline of the classification effect of the model. This is why we chose to couple with cost-sensitive PU learning.

However, in practice, the disturbance of propensity score has a great influence on the results due to the reciprocal, which makes the variance of our model large. We suggest that the existing prior class π in PU algorithm is used, and the propensity score of samples is weighted by normalized inverse probability, which is used as the weight of samples. This advantage is that there are many estimation methods for the prior class [2], and that the error estimation of the propensity score of individual samples is prevented from greatly affecting the model (the propensity score is close to 0), thereby enhancing the robustness of the model and improving the accuracy.The loss function of the basic PUe algorithm can be obtained as follows:

The main algorithm process of PUe is shown in Figure2. The specific algorithm process of PUe is shown in supplementary material. In summary, the first step of PUe algorithm is to estimate the propensity score of positive samples using regular binary classification. If the propensity score is

known, the first step can be skipped. The second step is to modify positive sample weights with normalized propensity scores. The third step is to use the PU algorithm (that is, PUe) with modified weights to perform PU binary classification. We also consider the case where negative classes are also selectively labeled and conduct related experiments (see PUbN), which are not mentioned in other papers estimating propensity scores and applying them to PU learning.

3 Experiment

In this section, we experimented our idea on several popular PU methods and compare its performance against the initial baseline methods.

3.1 Experimental settings

Datasets. We conducted experiments on two benchmarks commonly used in PU learning: MNIST for parity classification and CIFAR-10 [17] for vehicle class recognition. And on the simulated datasets of MNIST and CIFAR-10, we know the propensity score of the sample a priori, and we compare our proposed method with the ideal propensity to know the propensity score.Moreover, we tested our method on the Alzheimer's dataset 2 used to identify Alzheimer's disease in order to test the performance of our proposed method in real-world scenarios. More information is displayed in supplementary material.

Baselines. We mainly consider four common PU learning methods, including uPU, nnPU, PUbN, and Dist-PU.

Evaluation metrics. For each model, we counted six commonly used indicators of the test set results, including accuracy (ACC), Precision (Prec.), Recall (Rec.), F1, Area Under ROC Curve (AUC) and Average Precision (AP). Accuracy (ACC) is the main index for a more comprehensive comparison. The experiments are repeated with six random seeds, and the mean and standard deviation of each index were recorded.

Implementation details. All the experiments are run by PyTorch. Backbones of each dataset are summarized in supplementary material. The training batch size is set as 256 for MNIST and CIFAR10, while 128 for Alzheimer. We use Adam as the optimizer with a cosine annealing scheduler, where the initial learning rate is set as 5 × 10 -3 ; while weight decay is set as 5 × 10 -3 . PU learning methods first experiences a warm-up phase of 60 epochs, and then trains another 60 epochs with depth, where the value of α is searched in the range of [0 , 20] .

3.2 Comparision with state-of-the-art methods

Competitors. We compare our method with 4 competitive baseline PU algorithms including uPU [5], nnPU [16], PUbN [9], and Dist-PU [24]. Due to the space limitation, the detailed descriptions are provided in supplementary material.

Results. The results on all the datasets are recorded in Tab.1. It shows that in most metrics, our proposed PU+e method significantly outperforms our competitors on all biased datasets, improving the performance of the original PU method by about 1% to 5%. This proves the effectiveness of our proposed method. In addition, some observations can be made: (1) A model with a known propensity score is not necessarily the best (who has e' in Tab1), but a model with an estimated propensity score can perform better in many cases. This is a conclusion that is consistent with causal inference[8]. (2) In the ablation study, the performance of the PU+e algorithm is comparable to that of the most advanced PU algorithm, even when the labels are evenly distributed.

3.3 Ablation studies

Effectiveness of hyper-parameters. Ablation experiments were conducted to verify the validity of hyperparameters, which is shown in Figure3. It can be seen that the PUe algorithm is sensitive to hyperparameters and does not change monotonically. When α e = 15 , the PUe algorithm has the best performance. Therefore, in order to obtain the best performance of the PUe algorithm, fine adjustment is required. When the best performance is achieved remains to be further studied. However, the overall performance of Cifar-10 is better than that of the original Dist-PU algorithm.

Effectiveness of labeled samples distribution. The deviation of label data distribution will significantly affect the promotion effect of our proposed model in Tab.2. In general, when the distribution of labeled samples is more biased, the improvement effect of our proposed method is more obvious. However, when the distribution deviation is very large, the lifting effect of our model is weakened. This is because the number of labeled samples for some classes is too small or not labeled. As a result, the propensity score is close to 0, which affects the model performance. On the Cifar10 dataset, our algorithm improves the accuracy of the Dist-PU algorithm by up to 2.5%.

4 Conclusion

In this paper, a new PU learning method, namely PUe, is proposed from the perspective of propensity score for the PU learning with biased labels in deep learning. The PUe algorithm can enhance the original cost-sensitive PU algorithm, improve the prediction precision in the case of biased sample labeling, and has the degenerate ability, and the prediction precision in the case of unbiased labeling is not lower than that of the original PU algorithm. PUe consistently outperforms state-of-the-art methods on most metrics on real-world biased labeled datasets, including MNIST, CIFAR-10, and Alzheimer's. We hope that the proposed propensity score estimation scheme for deep learning can also provide some inspiration for other weakly supervised scenarios, especially those where the label distribution is unknown.











e

















e



















e

















e



















e

















e









References

Jessa Bekker and Jesse Davis. Positive and unlabeled relational classification through label frequency estimation. In Inductive Logic Programming: 27th International Conference, ILP 2017, Orléans, France, September 4-6, 2017, Revised Selected Papers 27 , pages 16-30. Springer, 2018.

Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning , 109:719-760, 2020.

Jessa Bekker, Pieter Robberechts, and Jesse Davis. Beyond the selected completely at random assumption for learning from positive and unlabeled data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II , pages 71-85. Springer, 2020.

Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Conference on Machine Learning , pages 1510-1519. PMLR, 2020.

Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. Advances in neural information processing systems , 27, 2014.

Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 213-220, 2008.

Walter Gerych, Thomas Hartvigsen, Luke Buquicchio, Emmanuel Agu, and Elke Rundensteiner. Recovering the propensity score from biased positive unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6694-6702, 2022.

Keisuke Hirano, Guido W Imbens, and Geert Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica , 71(4):1161-1189, 2003.

Yu-Guan Hsieh, Gang Niu, and Masashi Sugiyama. Classification from positive, unlabeled and biased negative data. In International Conference on Machine Learning , pages 2820-2829. PMLR, 2019.

Antti Hyttinen, Frederick Eberhardt, and Patrik O Hoyer. Experiment selection for causal discovery. Journal of Machine Learning Research , 14:3041-3071, 2013.

Kosuke Imai and Marc Ratkovic. Covariate balancing propensity score. Journal of the Royal Statistical Society: Series B: Statistical Methodology , pages 243-263, 2014.

Guido W Imbens and Donald B Rubin. Rubin causal model. In Microeconometrics , pages 229-241. Springer, 2010.

Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015.

Shantanu Jain, Justin Delano, Himanshu Sharma, and Predrag Radivojac. Class prior estimation with biased positives and unlabeled examples. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4255-4263, 2020.

Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with a selection bias. In International conference on learning representations , 2019.

Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. Advances in neural information processing systems , 30, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S Yu. Building text classifiers using positive and unlabeled examples. In Third IEEE international conference on data mining , pages 179-186. IEEE, 2003.

Bing Liu, Wee Sun Lee, Philip S Yu, and Xiaoli Li. Partially supervised classification of text documents. In ICML , volume 2, pages 387-394. Sydney, NSW, 2002.

MindSpore. Mindspore https://www.mindspore.cn/ .

Paul R Rosenbaum. Model-based direct adjustment. Journal of the American statistical Association , 82(398):387-394, 1987.

Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika , 70(1):41-55, 1983.

Guangxin Su, Weitong Chen, and Miao Xu. Positive-unlabeled learning from imbalanced data. In IJCAI , pages 2995-3001, 2021.

Yunrui Zhao, Qianqian Xu, Yangbangyan Jiang, Peisong Wen, and Qingming Huang. Dist-pu: Positiveunlabeled learning from a label distribution perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14461-14470, 2022.

A Algorithm

2: Compute ˆ e ( x ) by minimizing π n P ∑ P i =1 L ( g ( x P i ) , +1) + 1 -π n U ∑ U i =1 L ( g ( x U i ) , -1) + α e | ∑ x i ∈ χ P ∪ χ U e ( x i ) -n P | ;

Algorithm 1 PUe algorithm Require: data χ P , χ U , size n , n P , n U , hyperparameter α e , π . 1: Step 1: n n 3: Step 2: 4: Compute the weight of labeled samples: w P i = π ˜ ˆ e ( x P i ) 5: Step 3: 6: for i = 1 . . . do 7: Shuffle ( χ P , χ U ) into M mini-batches 8: for each mini-batch ( χ j P , χ j U ) do 9: Compute the corresponding ˆ R PUe ( g ) 10: Use A to update θ with the gradient information ∇ θ ˆ R PUe ( g ) 11: end for 12: end for

B Experiment Details

C Complementary Experiment

LRe: Logistic regression estimation of propensity scores for PU learning.

According to paper [7], it cannot estimate identifiable PS without making certain assumptions about the data. But according to the formula we gave in the first question, it's approximate. This is not explained by the self-monitoring method. Results in the above table show that our scheme is better than self-PU in the case of biased label datasets.

D Proofs

D.1 error bound of bias

We may assume that the error of propensity scores estimated by the NN method is the same as that estimated by the linear method. (In fact, the NN methods are usually more general, which may produce results with less error.) That is, the estimate of the propensity score has a maximum error ratio of β , with βe ( x L i ) ≤ ˆ e ( x L i ) ≤ e ( x L i ) . of the following shows that our regularization technique can yield a smaller error ratio with respect to sample weights. Obviously, the sample x L i has a sample weight of 1 n ˆ e ( x L i ) . in (Formula 1) with an error bound of bias ( 1 n ˆ e ( x L i ) ) ≤ 1 ne ( x L i ) ( 1 β -1) .

D.2 error ratio

In our approach, Sample x L i has a weight of π 1 ˆ e ( x L i ) ∑ j 1 ˆ e ( x L ) . P ( γe ( x L i ) < ˆ e ( x L i ) ≤ e ( x L i )) = α

1) , 1 ne ( x L i ) (1 -2 1 β + 1 γ ) , 1 ne ( x L i ) ( 2 (1+ 1 γ ) β -1) , 1 ne ( x L i ) (1 -2 γ β +1 )] < 1 ne ( x L i ) ( 1 β -1) and obviously we have 2 (1+ γ ) < 2 (1+ 1 γ ) β < 1 β , 0 < 1 -2 γ β +1 < 1 -2 1 β + 1 γ < 1 -β < 1 β -1 , which shows that our regularization technique has a smaller error ratio with respect to sample weights.

D.3 expectation

One understanding is that, according to the PS definition, each labeled sample x P j corresponds to 1 e ( x P j ) positive samples. So ∑ n p j =1 1 e ( x P j ) = N p it's true. Because P ( x | s = 1) = P ( x, y = 1 | s = 1) , we have

D.4 PUbN

The PUbN formula is as follows:

Let σ ( x ) = p ( s = +1 | x ) , however, the σ ( x ) is actually unknown,we should replace σ ( x ) by its estimate ˆ σ ( x ) .We can get the classification risk of PUbN ( R PUbN ( g ) ), as the following expression:

D.5 PUbNe

Our PUbNe formula is as follows:

E [ ˆ R PUe ( g )]

(15)

The change of ˆ R PUe ( g ) will be no more than L max /n if some x i ∈ χ P ∪ χ U is replaced, and McDiarmid's inequality gives us:

And make the right side of the previous formula equal to η :

Equivalently, with probability at least 1 -η ,

And because we know the expressions of ˆ R PUe ( g ) and ˆ R uPU ( g ) :

Since we know that ∑ n P j =1 1 e ( x P j ) = N P , we can get the following formula:

Then, we can prove:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on PU learning with a selection bias. They propose a weighted risk estimator to solve this problem, and estimate the weight via neural networks. Extensive experimental results validate the effectiveness of the proposed approach.\n\nSTRENGTHS:\n- The studied problem is very important for PU learning.\n- The experimental results is good, since the proposed mechanism improves previous PU learning methods under the biased setting.\n\nWEAKNESSES:\n- First, I found potential mistakes in the derivations of  $\\mathbb{E}[\\hat{R}\\_{PUe}(g)]=R_{PN}(g|y)$ in the supplemental materials. In Line 10 of Section 3, the authors use $n_{P}/n$ to replace $\\pi$, but I do not think so. Since there are positive examples in the unlabeled data ($s=0, y=1$), the class prior is under estimated. I will reconsider it if I need to be corrected.\n\n- Second, the major problem of this paper is that it is very similar to [3]. The proposed approach and theoretical results are very similar. For the theoretical results, Theorem 1 and 2 are very similar to Proposition 3 and 4 in [3]. For the methodology, it only changes the MLE method in [3] to a neural network. Therefore, the contribution is limited.\n\n- Third, this work actually works on “single-training-set PU learning”, while uPU and nnPU work on \"case-control PU learning\". Please see Gong et al. (2021) for more explanations. The authors should discuss the problem here.\n\n- Besides, I also have questions on the weight estimation via a neural network. The paper says that it regards positive examples as \"positive\" and unlabeled examples as \"negative\". My concern is that the model output is $p(s=1|x)$, instead of $p(s=1|y=1,x)$. Can the authors explain it?\n\n- It seems that the paper was written in a hurry since it is very difficult to read. For example, what's after Line 96? It has not reached the bar of a scientific paper up to now. I suggest revising this paper in detail.\n\n---\nReference\n- Gong et al, Instance-Dependent Positive and Unlabeled Learning with Labeling Bias Estimation, TPAMI 2021.\n\nQUESTIONS:\nPlease see \"Weaknesses\".\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-10"}

comments: SUMMARY:
This paper considers PU learning issue. Existing cost-sensitive-based methods often rely on strong assumptions that examples with
an observed positive label were selected entirely at random. This work relaxes the assumption and proposes a unbiased novel method PUe based on the causal theory.

STRENGTHS:
1. The motivation to relax the Selected Completely At Random assumption is practical and interesting.

2. Generalization theory to guarantee the effectiveness.

3. Excellent experiment performance to guarantee the effectiveness of the algorithm.

WEAKNESSES:
I am very confused on how to estimate the prior pi.

Please conduct experiments with wild OOD detection method Training OOD Detectors in their Natural Habitats.

I have not got where you have used the causal theory. Could you point out the causual theory you have used and clarify what causal assumptions you have used in you theorems?

QUESTIONS:
See weaknesses

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem that existing cost-sensitive-based methods often rely on strong assumptions that examples with an observed positive label were selected entirely at random in Positive-Unlabeled (PU) learning. The authors propose a PU learning enhancement (PUe) algorithm based on causal inference theory, which employs normalized propensity scores and normalized inverse probability weighting (NIPW) techniques to reconstruct the loss function, thus obtaining a consistent, unbiased estimate of the classifier and enhancing the model’s performance. Moreover, they investigate and propose a method for estimating propensity scores in deep learning using regularization techniques when the labeling mechanism is unknown. Experiments on three benchmark datasets demonstrate the proposed PUe algorithm significantly improves the accuracy of classifiers on non-uniform label distribution datasets compared to advanced cost-sensitive PU methods.\n\nSTRENGTHS:\n1. One strength of this paper is its focus on addressing the prevalent issue of bias in the labeled set, which often does not conform to the Selected Completely At Random (SCAR) assumption. This assumption posits that the observed labeled examples are a random subset of the complete set of positive examples. In many real-world scenarios, this assumption does not hold true, leading to biased learning outcomes.\n\nWEAKNESSES:\n1.  This paper contains several spelling errors, which can potentially hinder the reader's understanding and overall perception of the work. For instance, on line 131, the term \"oftreament\" appears to be a typographical error where two words are mistakenly joined. Similarly, on line 149, \"porpensity\" is likely a misspelling of \"propensity,\" and on line 189, \"Apparemtly\" should be corrected to \"Apparently.\" Lastly, on line 220, \"pi\" might be a typo or a symbol that is not clearly defined in the context. These errors suggest a lack of thorough proofreading and can detract from the overall quality and credibility of the paper. It is highly recommended that the authors conduct a careful review of the manuscript for spelling, grammar, and typographical errors before final submission. \n2. A significant weakness of this paper lies in its lack of clear explanation and connection between the problem it aims to solve - the bias in the labeled set not conforming to the Selected Completely At Random (SCAR) assumption - and the causal inference method it employs to address this issue. Specifically, lines 123 to 145 appear to attempt to elucidate this connection, but the explanation falls short due to the lack of clear definitions and elaboration on key concepts. This lack of clarity can make it difficult for readers to understand the rationale behind the proposed method and its relevance to the problem.\n3. A notable weakness in the experimental section of the paper is the presentation of results in Tables 1 and 2. The current format does not clearly highlight the best-performing algorithm, making it difficult for readers to quickly and intuitively discern which method is superior and by what margin.\n\nQUESTIONS:\nCould the authors provide a more detailed explanation of how the causal inference method addresses the issue of bias in the labeled set not conforming to the SCAR assumption?\n\nLIMITATIONS:\nA significant limitation of this paper is its overall clarity and readability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an algorithm for Positive-Unlabeled (PU) learning that tackles the problem of biased labeled data. By utilizing causal inference theory, the algorithm utilizes normalized propensity scores and inverse probability weighting to reconstruct the loss function and achieve an unbiased estimate of the classifier. Compared to existing cost-sensitive PU methods, the proposed PUe algorithm demonstrates significant improvements in classifier accuracy, particularly on datasets with non-uniform label distributions. Additionally, the paper proposes a method for estimating propensity scores in deep learning models when the labeling mechanism is unknown. Overall, the PUe framework effectively addresses selection bias, enhances PU learning, and improves model performance.\n\n-------------------------------\nAfter rebuttal: I would like to increase my rate a little bit. However, I think the novelty part still needs more clarification. \n\nAfter the author-review discussion: I could increase my rate a little bit for the thorough rebuttal provided by the authors. However, I think the rebuttal provides a lot of information that may not be easily included in the paper. I would like to see a plan on how to incorporate the reviews/rebuttals into the paper. \n\nDuring the final discussion: The plan has given me more confidence in how the paper can be if we accept it.\n\nSTRENGTHS:\nImportance of the studied problem (biased positive data): The problem of studying biased positive data is highly significant and practical. In many real-world scenarios, the selection of positive samples is often biased, while existing methods typically assume that the selection of positive samples is completely random. The proposed PUe algorithm in this paper addresses this issue and effectively handles biased labeled data, filling a research gap in the field of PU learning.\n\nWell-written: The paper is well-written, with clear descriptions of the problem background, method principles, and experimental design and results. The authors introduce causal inference theory and utilize techniques such as normalized propensity scores and inverse probability weighting to provide a viable solution, which is presented in a clear and concise manner.\n\nExperimental results demonstrate the effectiveness of the method: The paper provides experimental evidence of the proposed PUe algorithm's effectiveness.\n\nWEAKNESSES:\nLack of novelty: One of the limitations of the paper is that it may lack novelty. Similar work, such as the paper \"Recovering the Propensity Score from Biased Positive Unlabeled Data\" presented at AAAI 2022, may have explored similar ideas and methods earlier. It is important for a research contribution to demonstrate novelty and differentiate itself from existing approaches in the field.\n\nLimited causal inference aspect: While the paper is motivated by causal inference, it falls short in incorporating substantial causal elements. The approach of reweighting samples based on sampling probabilities is straightforward and may not fully capture the complexity of causal relationships. To strengthen the causal inference aspect, the paper could have explored additional methods or frameworks that explicitly address causal assumptions and mechanisms.\n\nLack of significant contribution: The primary contribution of the paper should lie in the proposed method for estimating propensity scores. However, the approach of using model predictions as weights for propensity scores is not fundamentally different from classical self-supervised methods. Although a regularizer is introduced, it may not provide a meaningful improvement. Additionally, the process of normalizing the estimated propensity scores could have been done separately after learning the scores, making the proposed regularizer less essential or impactful.\n\nQUESTIONS:\nComparison with \"Recovering the Propensity Score from Biased Positive Unlabeled Data\" presented at AAAI 2022: A direct comparison between the paper and the mentioned work would require a thorough analysis of both papers. \n\nSignificance of causal inference in the context of the proposed work: Although the paper mentions causal inference as a motivation, it is valid to question the significance of causal inference in this work. If the process of generating biased data is explicitly defined and the proposed loss function can be straightforwardly derived from it, the direct application of causal inference may not be evident. It is important to consider whether the paper effectively incorporates causal inference methods or frameworks that address causal assumptions and mechanisms beyond straightforward reweighting.\n\nComparison with self-supervised methods like selfPU: Evaluating the performance of the proposed approach compared to self-supervised methods like selfPU would require a detailed analysis and comparison of the methodologies and experimental results from both approaches.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper noted that existing Positive-Unlabeled (PU) learning methods assumed that positive samples are selected entirely at random, ignoring the prevalent selection bias in real-world PU problems. To overcome this limitation, the authors proposed a PU learning enhancement (PUe) algorithm based on causal inference theory. They adapted sample re-weighting methods, to be specific, inverse propensity weighting (IPW) to PU learning, which assigns a weight to each positive sample. Based on the weights, they obtained a new unbiased PU risk estimator. They also proposed a method that can compatible to deep networks to estimate the weights when the labeling mechanism is unknown.\n\nSTRENGTHS:\n* The problem is interesting and valuable. The adaptation of causal inference to PU learning is sound and original to the best of my knowledge. \n* The resulting algorithm is unbiased, which is a very nice result (but I quickly checked the proofs and have some concerns).\n* The algorithm is simple and easy to follow.\n* The empirical studies well support the method's superiority with a clear margin.\n\nWEAKNESSES:\n* I'm a bit puzzled by the fact that the risk estimator is unbiased. \n\n$R_{PN}$ in current version should be $\\hat{R_{PN}}$ for notation consistency, while $R_{PN}$ denotes the expected risk. Then if $\\mathbb{E}[\\hat{R_{PUe}}(g)] = R_{PN}(g)$, we say $\\hat{R_{PUe}}$ is unbiased. \nWhile in this paper, the authors proved $\\mathbb{E}[\\hat{R_{PUe}}(g)] = \\hat{R_{PN}}$ (Eq.11), which looks a little weird.\n\n* The examples given by the authors give readers a very clear idea that biased selection is likely to be present in the PU problem, but it seems to me that Figure 1 doesn't present a very clear picture of the setting discussed in the paper. Wouldn't it be more telling if all the blue dots were centered, like, in the upper left corner? But if so could ePU also have difficulty learning good classification boundaries, since this bias could also seriously mislead the estimation of $e$ (step 1 in Figure 2)?\n* $s$ should be introduced in Eq.4.\n* In line 113, \"it does not depend on...\", \"it\" here seems very vague.\n* Tables 1 and 2 are too dense a presentation of the results, and I would suggest labeling the best results more obviously.\n\nQUESTIONS:\nPlease see the main review.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}