SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From the bias, we can find that the propensity scores only need to be used accurately for positive examples. Apparently,  $L(g(\pmb{x}_i), +1) - L(g(\pmb{x}_i), -1) < 0$  for positive examples. And when incorrect propensity score is close to 0 or 1 (especially tend towards 0), there will be a large bias. Overestimate normalized inverse propensity scores.
Underestimated propensity scores are expected to result in a model with a higher bias. Lower propensity scores result in learning models that estimate the positive class to be more prevalent than it is, which results in a larger  $[L(g(\pmb{x}_i), + 1) - L(g(\pmb{x}_i), - 1)]$  for positive samples.
$$
\hat {e} (x) = \arg \min  _ {e} \frac {\pi_ {1}}{n _ {P}} \sum_ {i = 1} ^ {n _ {P}} L \left(e \left(\boldsymbol {x} _ {i} ^ {P}\right), + 1\right) + \frac {1 - \pi_ {1}}{n _ {U}} \sum_ {i = 1} ^ {n _ {U}} L \left(e \left(\boldsymbol {x} _ {i} ^ {U}\right), - 1\right) + \alpha_ {e} \left| \sum_ {x _ {i} \in \chi_ {P} \cup \chi_ {U}} e \left(x _ {i}\right) - n _ {P} \right|, \tag {13}
$$
where  $\pi_1 = \frac{n_P}{n_P + n_U}$ .
In reality, it is almost impossible to know the propensity score of the sample, which requires us to estimate the propensity score of the sample. However, the traditional statistical estimation method can not be used directly in deep neural network. This requires us to design a new method to estimate sample propensity scores.
According to the paper [7]: In the Local Certainty scenario, we assume the relationship between the observed features and the true class is a deterministic function  $f: X \to Y$ , where  $X$  is the feature space and  $Y = \{0, 1\}$ , while allowing the propensity score to be an arbitrary probability function. But this assumption is too strong. In fact, we only need to have hypotheses  $|P(y = 1|x) - P(y = 0|x)| > \theta$  and  $P(y = 1|x^L) > P(y = 0|x^L)$  to estimate the normalized propensity score well for sample classification. (where  $L$  represents the labeled sample and is used to distinguish it from the positive category.)
Otherwise, a linear function is usually used to estimate the propensity score in scenarios where the local certainty scenario is not met. Our supplementary experiments in the supplemental materials show that the probability function fitted using deep learning is better than the linear estimation in the MNIST experiment.
We use labeled data as positive samples and unlabeled data as negative samples. We use binary network and regularization technology to construct a scoring network  $e(x)$ . Finally, the output of the network passes through the sigmoid function to ensure that the output value is between 0 and 1. The score it outputs for each sample is the estimated probability of the sample being labeled (propensity score). The purpose of introducing the regularization technique is to prevent overfitting. Otherwise, the output value of the labeled sample is very close to 1 and the output value of the unlabeled sample is very close to 0. This is not what we expect. The ideal propensity score value should be far away from 0 and 1. Therefore, we introduce such a constraint in the classification loss function by combining the nature of propensity scores, where the sum of propensity scores of all samples is equal to the number of labeled samples:  $\alpha_{e} \left| \sum_{x_{i} \in \chi_{P} \cup \chi_{U}} e(x_{i}) - n_{P} \right|$ .
where  $\alpha_{e}$  adjusts the importance of regular term.
Theoretically, for each labeled example  $(\pmb{x}_i,s = +1)$ , which has a porosity score  $e_i$ , there are expected to be  $\frac{1}{e_i}$  positive examples, and of which  $\frac{1}{e_i} - 1$  did not get selected to be labeled. Each labeled example gets a weight  $\frac{1}{e_i}$  and the negative example is added to the dataset that gets a negative weight  $1 - \frac{1}{e_i}$ .
According to [7, 3],  $P(\hat{Y} | E, L) = \frac{1}{n} \sum_{i=1}^{n} s_i \left( \frac{1}{e_i} \delta_1 + \left( 1 - \frac{1}{e_i} \right) \delta_0 \right) + (1 - s_i) \delta_0$ , using Propensity Sores For Classification. However, as discussed above, existing methods always underestimate
Table 1: Comparative results on MNIST, CIFAR-10 and Alzheimer.
Dataset	Method	ACC (%)	Prec. (%)	Rec. (%)	F1 (%)	AUC (%)	AP (%)
MNIST	uPU	85.17 (1.90)	78.38 (2.36)	96.69 (1.33)	86.55 (1.50)	85.34 (1.87)	77.41 (2.26)
nnPU	87.81 (1.65)	83.78 (2.14)	93.40 (1.68)	88.31 (1.51)	87.89 (1.63)	81.62 (2.28)
PUbN	87.07 (1.73)	83.43 (2.74)	92.21 (2.55)	87.55 (1.55)	87.14 (2.74)	80.70 (2.59)
Dist-PU	89.79 (0.59)	92.22 (1.67)	88.86 (1.48)	90.49 (0.47)	96.60 (0.30)	96.92 (0.21)
uPUe	92.28 (1.32)	89.41 (1.87)	95.71 (1.43)	92.44 (1.25)	92.33 (1.31)	87.69 (1.92)
nnPUe	92.45 (1.61)	90.45 (2.26)	94.73 (1.24)	92.53 (1.55)	92.48 (1.60)	88.29 (2.43)
PUbNe	94.70 (0.63)	97.16 (1.59)	91.33 (1.58)	94.46 (0.71)	94.66 (0.50)	93.18 (0.55)
Dist-PUe	92.57 (0.78)	93.02 (0.86)	90.58 (1.47)	92.77 (0.84)	97.13 (0.57)	97.43 (0.44)
uPUe'	92.39 (1.36)	89.33 (1.87)	96.07 (1.70)	92.56 (1.30)	92.44 (1.87)	87.76 (1.93)
nnPUe'	92.94 (1.27)	91.11 (1.84)	94.98 (2.38)	92.98 (1.29)	92.96 (1.27)	89.00 (1.77)
PUbNe'	94.48 (0.93)	97.74 (0.46)	90.90 (2.10)	94.18 (1.06)	94.43 (0.95)	93.45 (1.00)
Dist-PUe'	91.73 (0.95)	93.34 (0.42)	89.60 (2.38)	91.41 (1.11)	96.48 (1.54)	96.60 (1.37)
CIFAR-10	uPU	76.33 (1.76)	83.59 (0.65)	50.81 (5.57)	63.01 (4.35)	84.40 (3.12)	78.68 (2.22)
nnPU	82.47 (0.69)	75.30 (1.35)	83.68 (1.64)	79.25 (0.75)	90.01 (0.73)	85.22 (0.73)
PUbN	84.54 (0.54)	82.18 (1.38)	78.40 (2.70)	80.20 (0.99)	91.62 (0.27)	87.31 (0.40)
Dist-PU	84.45 (1.00)	80.90 (1.51)	80.04 (1.62)	80.46 (1.26)	90.89 (1.20)	85.67 (1.54)
uPUe	79.31 (0.90)	81.37 (2.16)	62.86 (5.02)	70.73 (2.58)	86.76 (0.82)	80.07 (1.56)
nnPUe	83.33 (1.04)	76.83 (2.75)	83.88 (2.86)	80.11 (0.85)	90.53 (0.79)	85.33 (1.36)
PUbNe	85.66 (0.58)	83.18 (1.63)	80.52 (2.29)	81.79 (0.66)	92.69 (0.74)	89.41 (1.24)
Dist-PUe	86.97 (0.62)	82.11 (1.69)	86.46 (2.00)	84.19 (0.76)	93.50 (0.66)	89.38 (1.78)
uPUe'	79.11 (1.75)	85.11 (1.58)	57.92 (4.96)	68.80 (3.61)	86.57 (1.56)	81.56 (2.18)
nnPUe'	84.24 (0.67)	78.40 (0.86)	83.66 (2.35)	80.92 (1.02)	91.35 (0.99)	87.15 (1.83)
PUbNe'	87.02 (0.47)	83.84 (1.44)	83.78 (2.68)	83.76 (0.82)	93.80 (0.42)	90.69 (0.65)
Dist-PUe'	86.17 (2.52)	80.97 (3.20)	85.57 (3.32)	83.19 (3.06)	92.65 (2.20)	88.22 (3.19)
Alzheimer	uPU	63.46 (1.72)	66.80 (3.45)	55.06 (9.83)	59.65 (4.70)	68.08 (2.16)	67.78 (2.22)
nnPU	69.19 (0.85)	65.46 (1.06)	81.46 (5.22)	72.48 (1.62)	73.50 (0.84)	68.74 (1.83)
PUbN	68.75 (0.96)	66.65 (2.04)	75.64 (5.58)	70.68 (1.52)	72.57 (0.63)	66.99 (0.80)
Dist-PU	69.74 (0.50)	67.74 (1.57)	75.69 (4.72)	71.37 (1.33)	74.34 (1.10)	69.53 (1.39)
uPUe	65.30 (3.36)	68.53 (1.99)	56.73 (11.93)	61.32 (7.58)	70.49 (3.99)	70.28 (3.09)
nnPUe	69.38 (0.56)	65.97 (0.68)	80.06 (2.77)	72.30 (0.91)	74.49 (0.34)	69.98 (0.60)
PUbNe	69.52 (0.85)	68.28 (1.28)	72.95 (1.99)	70.51 (0.82)	74.68 (1.08)	70.33 (0.98)
Dist-PUe	70.51 (0.62)	68.91 (1.00)	75.02 (2.26)	71.81 (0.83)	75.19 (0.69)	71.23 (2.08)
propensity scores. As a result, PU learning classification is performed by using the foregoing formula, which reduces model classification performance.
We found that in cost-sensitive PU learning technique, the training instances are properly reweighted and a common way to score propensity is inverse probability weighting (a reweighting technique). Conditionally, the two methods can be combined to solve the defects of the two methods themselves. The disadvantage of the cost-sensitive PU learning technique is that the classification effect of the application deteriorates when the labeled samples are biased. The propensity score is estimated by the fact that the propensity score of positive samples is always underestimated, which leads to the decline of the classification effect of the model. This is why we chose to couple with cost-sensitive PU learning.
However, in practice, the disturbance of propensity score has a great influence on the results due to the reciprocal, which makes the variance of our model large. We suggest that the existing prior class  $\pi$  in PU algorithm is used, and the propensity score of samples is weighted by normalized inverse probability, which is used as the weight of samples. This advantage is that there are many estimation methods for the prior class [2], and that the error estimation of the propensity score of individual samples is prevented from greatly affecting the model (the propensity score is close to 0), thereby enhancing the robustness of the model and improving the accuracy. The loss function of the basic PUs algorithm can be obtained as follows:
$$
\hat {R} _ {P U \hat {e}} (g) = \pi \hat {R} _ {P} ^ {\hat {e}} (g, + 1) + \hat {R} _ {U} (g, - 1) - \pi \hat {R} _ {P} ^ {\hat {e}} (g, - 1). \tag {14}
$$
The main algorithm process of PUe is shown in Figure2. The specific algorithm process of PUe is shown in supplementary material. In summary, the first step of PUe algorithm is to estimate the propensity score of positive samples using regular binary classification. If the propensity score is
known, the first step can be skipped. The second step is to modify positive sample weights with normalized propensity scores. The third step is to use the PU algorithm (that is, PUe) with modified weights to perform PU binary classification. We also consider the case where negative classes are also selectively labeled and conduct related experiments (see PUbN), which are not mentioned in other papers estimating propensity scores and applying them to PU learning.
In this section, we experimented our idea on several popular PU methods and compare its performance against the initial baseline methods.
Table 2: Ablation results on CIFAR-10 with  $\checkmark$  indicating the enabling of the corresponding loss term and different labeled distribution.
method	αe labeled distribution	ACC (%)	Prec. (%)	Rec. (%)	F1 (%)	AUC (%)	AP (%)
Dist-PUe	[.25,.25,.25]	91.79 (0.86)	89.02 (2.63)	90.78 (1.81)	89.84 (0.92)	97.11 (0.52)	95.38 (1.18)
Dist-PUe	[.25,.25,.25]	91.30 (0.80)	89.22 (1.56)	89.05 (1.93)	89.11 (1.03)	96.64 (0.53)	94.85 (0.77)
Dist-PU	[.25,.25,.25]	91.88 (0.52)	89.87 (1.09)	89.84 (0.81)	89.85 (0.62)	96.92 (0.45)	95.49 (0.72)
Dist-PUe	[.10,.10,.30,..50]	91.50 (0.32)	89.07 (1.15)	89.78 (0.97)	89.41 (0.34)	96.67 (0.14)	95.64 (0.26)
Dist-PUe	[.10,.10,.30,..50]	90.40 (0.66)	88.86 (3.15)	87.14 (0.74)	87.89 (0.74)	95.71 (0.40)	94.19 (0.65)
Dist-PU	[.10,.10,.30,..50]	90.79 (0.72)	91.31 (1.54)	85.12 (2.44)	88.07 (1.07)	95.81 (0.83)	91.28 (2.35)
Dist-PUe	[.72,.15,.10,.03]	86.97 (0.62)	82.11 (1.69)	86.46 (2.00)	84.19 (0.76)	93.50 (0.66)	89.38 (1.78)
Dist-PUe	[.72,.15,.10,.03]	86.44 (0.67)	82.19 (1.54)	84.45 (1.78)	83.28 (0.82)	92.90 (0.63)	88.39 (1.32)
Dist-PU	[.72,.15,.10,.03]	84.45 (1.00)	80.90 (1.51)	80.04 (1.62)	80.46 (1.26)	90.89 (1.20)	85.67 (1.54)
Dist-PUe	[.08,.86,.02,.04]	88.15 (1.22)	88.05 (2.69)	81.70 (5.41)	84.56 (2.23)	94.46 (1.01)	92.79 (1.09)
Dist-PUe	[.08,.86,.02,.04]	86.39 (2.11)	84.47 (3.51)	81.05 (4.32)	82.62 (2.77)	92.42 (2.25)	89.84 (2.40)
Dist-PU	[.08,.86,.02,.04]	86.74 (1.58)	86.56 (2.51)	79.30 (4.60)	82.65 (2.48)	93.30 (1.32)	91.08 (1.69)
Datasets. We conducted experiments on two benchmarks commonly used in PU learning: MNIST for parity classification and CIFAR-10 [17] for vehicle class recognition. And on the simulated datasets of MNIST and CIFAR-10, we know the propensity score of the sample a priori, and we compare our proposed method with the ideal propensity to know the propensity score. Moreover, we tested our method on the Alzheimer's dataset  ${}^{2}$  used to identify Alzheimer's disease in order to test the performance of our proposed method in real-world scenarios. More information is displayed in supplementary material.
Baselines. We mainly consider four common PU learning methods, including uPU, nnPU, PUbN, and Dist-PU.
Evaluation metrics. For each model, we counted six commonly used indicators of the test set results, including accuracy (ACC), Precision (Prec.), Recall (Rec.), F1, Area Under ROC Curve (AUC) and Average Precision (AP). Accuracy (ACC) is the main index for a more comprehensive comparison. The experiments are repeated with six random seeds, and the mean and standard deviation of each index were recorded.
Implementation details. All the experiments are run by PyTorch. Backbones of each dataset are summarized in supplementary material. The training batch size is set as 256 for MNIST and CIFAR-10, while 128 for Alzheimer. We use Adam as the optimizer with a cosine annealing scheduler, where the initial learning rate is set as  $5 \times 10^{-3}$ ; while weight decay is set as  $5 \times 10^{-3}$ . PU learning methods first experiences a warm-up phase of 60 epochs, and then trains another 60 epochs with depth, where the value of  $\alpha$  is searched in the range of [0, 20].
$^{2}$ Dubey, S. Alzheimer's Dataset. Available online: https://www.kaggle.com/tourist55/ alzheimers-dataset-4-class-of-images
3.2 Comparison with state-of-the-art methods
Competitors. We compare our method with 4 competitive baseline PU algorithms including uPU [5], nnPU [16], PUbN [9], and Dist-PU [24]. Due to the space limitation, the detailed descriptions are provided in supplementary material.
Results. The results on all the datasets are recorded in Tab.1. It shows that in most metrics, our proposed  $\mathrm{PU + e}$  method significantly outperforms our competitors on all biased datasets, improving the performance of the original PU method by about  $1\%$  to  $5\%$ . This proves the effectiveness of our proposed method. In addition, some observations can be made: (1) A model with a known propensity score is not necessarily the best (who has  $\mathbf{e}^{\prime}$  in Tab1), but a model with an estimated propensity score can perform better in many cases. This is a conclusion that is consistent with causal inference[8]. (2) In the ablation study, the performance of the  $\mathrm{PU + e}$  algorithm is comparable to that of the most advanced PU algorithm, even when the labels are evenly distributed.
Effectiveness of hyper-parameters. Ablation experiments were conducted to verify the validity of hyperparameters, which is shown in Figure3. It can be seen that the PUe algorithm is sensitive to hyperparameters and does not change monotonically. When  $\alpha_{e} = 15$ , the PUe algorithm has the best performance. Therefore, in order to obtain the best performance of the PUe algorithm, fine adjustment is required. When the best performance is achieved remains to be further studied. However, the overall performance of Cifar-10 is better than that of the original Dist-PU algorithm.
Effectiveness of labeled samples distribution. The deviation of label data distribution will significantly affect the promotion effect of our proposed model in Tab.2. In general, when the distribution of labeled samples is more biased, the improvement effect of our proposed method is more obvious. However, when the distribution deviation is very large, the lifting effect of our model is weakened. This is because the number of labeled samples for some classes is too small or not labeled. As a result, the propensity score is close to 0, which affects the model performance. On the Cifar10 dataset, our algorithm improves the accuracy of the Dist-PU algorithm by up to  $2.5\%$ .
Figure 3: Influences of different  $\alpha_{e}$  on CIFAR-10 by Dist-PUe.
In this paper, a new PU learning method, namely PUs, is proposed from the perspective of propensity score for the PU learning with biased labels in deep learning. The PUs algorithm can enhance the original cost-sensitive PU algorithm, improve the prediction precision in the case of biased sample labeling, and has the degenerate ability, and the prediction precision in the case of unbiased labeling is not lower than that of the original PU algorithm. PUs consistently outperforms state-of-the-art methods on most metrics on real-world biased labeled datasets, including MNIST, CIFAR-10, and Alzheimer's. We hope that the proposed propensity score estimation scheme for deep learning can also provide some inspiration for other weakly supervised scenarios, especially those where the label distribution is unknown.
We gratefully acknowledge the support of MindSpore [20], CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
[20] MindSpore. Mindspore https://www.mindspore.cn/.
[21] Paul R Rosenbaum. Model-based direct adjustment. Journal of the American statistical Association, 82(398):387-394, 1987.
[22] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
[23] Guangxin Su, Weitong Chen, and Miao Xu. Positive-unlabeled learning from imbalanced data. In IJCAI, pages 2995-3001, 2021.
[24] Yunrui Zhao, Qianqian Xu, Yangbangyan Jiang, Peisong Wen, and Qingming Huang. Dist-pu: Positive unlabeled learning from a label distribution perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14461-14470, 2022.
Algorithm 1 PUs algorithm
Require: data  $\chi_P,\chi_U$  size  $n,n_P,n_U$  , hyperparameter  $\alpha_{e},\pi$    
1: Step 1:   
2: Compute  $\hat{e} (x)$  by minimizing  $\frac{\pi}{n_P}\sum_{i = 1}^{n_P}L(g(\pmb {x}_i^P), + 1) + \frac{1 - \pi}{n_U}\sum_{i = 1}^{n_U}L(g(\pmb {x}_i^U), - 1) +$ $\alpha_{e}|\sum_{x_{i}\in \chi_{P}\cup \chi_{U}}e(x_{i}) - n_{P}|;$    
3: Step 2:   
4: Compute the weight of labeled samples:  $w_{i}^{P} = \frac{\pi}{\hat{e}(x_{i}^{P})}$    
5: Step 3:   
6: for  $i = 1\dots$  do   
7: Shuffle  $(\chi_P,\chi_U)$  into M mini-batches   
8: for each mini-batch  $(\chi_P^j,\chi_U^j)$  do   
9: Compute the corresponding  $\hat{R}_{\mathrm{PUe}}(g)$    
10: Use A to update  $\theta$  with the gradient information  $\nabla_{\theta}\hat{R}_{\mathrm{PUe}}(g)$    
11: end for   
12: end for
Table 3: Summary of used datasets and their corresponding models.
Dataset	Input Size	nP	nU	#Testing πP	Positive Class	true e(x)	Model
MNIST	28 × 28	2500	60,000	10,000	0.5	Even (0, 2, 4, 6 and 8)	[.65,.15,.1,.07,.03]
CIFAR-10	3 × 32 × 32	1,000	50,000	10,000	0.4	Vehicles (0, 1, 8 and 9)	[.72,.15,.1,.03]
Alzheimer3	3 × 224 × 224	769	5,121	1,279	0.5	Alzheimer's Disease	unknown	ResNet-50
C Complementary Experiment
Table 4: Supplemental Experiments on MINST
method	labeled distribution	ACC.(%)	Prec.(%)	Rec.(%)	F1.(%)	AUC.(%)	AP.(%)
LRe	[.65,.15,.10,.07,.03]	86.19(0.75)	92.94(0.64)	77.89(1.38)	84.75(0.93)	88.06(0.93)	88.72(1.09)
nnPUe	[.65,.15,.10,.07,.03]	92.45 (1.61)	90.45 (2.26)	94.73 (1.24)	92.53 (1.55)	92.48 (1.60)	88.29 (2.43)
nnPU without normalize	[.65,.15,.10,.07,.03]	90.95(1.61)	88.18(2.40)	94.38(2.74)	91.13(1.56)	91.00(1.61)	85.98(2.25)
Self-PU	[.65,.15,.10,.07,.03]	90.08(0.47)	90.08 (0.47)	89.35 (1.17)	90.70 (1.73)	90.00 (0.53)	85.61 (0.69)
anchor	[.65,.15,.10,.07,.03]	88.22(0.95)	94.66(1.41)	80.70(3.15)	87.06(1.37)	92.36(1.92)	93.37(2.33)
LRe: Logistic regression estimation of propensity scores for PU learning.
According to paper [7], it cannot estimate identifiable PS without making certain assumptions about the data. But according to the formula we gave in the first question, it's approximate. This is not explained by the self-monitoring method. Results in the above table show that our scheme is better than self-PU in the case of biased label datasets.
D.1 error bound of bias
We may assume that the error of propensity scores estimated by the NN method is the same as that estimated by the linear method. (In fact, the NN methods are usually more general, which may produce results with less error.) That is, the estimate of the propensity score has a maximum error ratio of  $\beta$ , with  $\beta e(x_i^L) \leq \hat{e}(x_i^L) \leq e(x_i^L)$ . Of the following shows that our regularization technique can yield a smaller error ratio with respect to sample weights. Obviously, the sample  $x_i^L$  has a sample weight of  $\frac{1}{n\hat{e}(x_i^L)}$ . In (Formula 1) with an error bound of bias  $\left(\frac{1}{n\hat{e}(x_i^L)}\right) \leq \frac{1}{n e(x_i^L)}\left(\frac{1}{\beta} - 1\right)$ .
D.2 error ratio
In our approach, Sample  $x_{i}^{L}$  has a weight of  $\pi \frac{\frac{1}{\hat{e}(x_{i}^{L})}}{\sum_{j} \frac{1}{\hat{e}(x_{j}^{L})}}$ .  $P(\gamma e(x_{i}^{L}) < \hat{e}(x_{i}^{L}) \leq e(x_{i}^{L})) = \alpha$  where the set of samples is  $S_{1}$ .  $P(\beta e(x_{i}^{L}) < \hat{e}(x_{i}^{L}) \leq \gamma e(x_{i}^{L})) = 1 - \alpha$  where the set of samples is  $S_{2}$ .  $\beta < \gamma < 1$  and  $\sum_{i \in S_{1}} \frac{1}{e(x_{i}^{L})} = \sum_{i \in S_{2}} \frac{1}{e(x_{i}^{L})} = B$ . So that  $\sum_{j} \frac{1}{e(x_{j}^{L})} = 2B = N_{p}$ . For  $x_{i}^{L} \in S_{1}$ , we have  $\frac{1}{e(x_{i}^{L})} \leq \frac{1}{\hat{e}(x_{i}^{L})} < \frac{1}{\gamma e(x_{i}^{L})}$ . For  $x_{i}^{L} \in S_{2}$ , we have  $\frac{1}{\gamma e(x_{i}^{L})} \leq \frac{1}{\hat{e}(x_{i}^{L})} < \frac{1}{\beta e(x_{i}^{L})}$ , so we can get  $B(1 + \frac{1}{\gamma}) \leq \sum_{j} \frac{1}{\hat{e}(x_{j}^{L})} < B(\frac{1}{\gamma} + \frac{1}{\beta})$  and we have  $bias(\pi \frac{\frac{1}{\hat{e}(x_{i}^{L})}}{\sum_{j} \frac{1}{\hat{e}(x_{j}^{L})}}) \leq \max[\frac{1}{ne(x_{i}^{L})}(\frac{2}{(1 + \gamma)} - 1), \frac{1}{ne(x_{i}^{L})}(1 - \frac{2}{\beta + 1}), \frac{1}{ne(x_{i}^{L})}(1 - \frac{2}{\beta + 1})] < \frac{1}{ne(x_{i}^{L})}(\frac{1}{\beta} - 1)$  and obviously we have  $\frac{2}{(1 + \gamma)} < \frac{2}{(1 + \frac{1}{\gamma})\beta} < \frac{1}{\beta}, 0 < 1 - \frac{2}{\beta + 1} < 1 - \frac{2}{\beta + 1} < 1 - \beta < \frac{1}{\beta} - 1$ , which shows that our regularization technique has a smaller error ratio with respect to sample weights.
D.3 expectation
One understanding is that, according to the PS definition, each labeled sample  $x_{j}^{P}$  corresponds to  $\frac{1}{e(x_{j}^{P})}$  positive samples. So  $\sum_{j=1}^{n_{p}} \frac{1}{e(x_{j}^{P})} = N_{p}$  it's true. Because  $P(x|s = 1) = P(x, y = 1|s = 1)$ , we have
$$
\begin{array}{l} E _ {P (x \mid s = 1)} \frac {1}{P (s = 1 \mid x , y = 1)} \\ = \sum P (x, y = 1 | s = 1) \frac {1}{P (s = 1 | x , y = 1)} \\ = \sum \frac {P (s = 1 | x , y = 1) P (x , y = 1)}{P (s = 1)} \frac {1}{P (s = 1 | x , y = 1)} \\ = \sum \frac {P (x , y = 1)}{P (s = 1)} = \frac {n}{n _ {P}} \sum P (x, y = 1) \\ = \frac {n}{n _ {P}} \frac {N _ {P}}{n} = \frac {N _ {P}}{n _ {P}}. \\ \end{array}
$$
It indicates that  $\sum_{j=1}^{n_p} \frac{1}{e(x_j^P)} = N_p$ .
D.4 PUbN
The PUbN formula is as follows:
Let  $\sigma(x) = p(s = +1|x)$ , however, the  $\sigma(x)$  is actually unknown, we should replace  $\sigma(x)$  by its estimate  $\hat{\sigma}(x)$ . We can get the classification risk of PUbN  $(R_{PUbN}(g))$ , as the following expression:
$$
R _ {P U b N} (g) = \pi R _ {P} (g, + 1) + \rho R _ {b N} (g, - 1) + \bar {R} _ {s = - 1, \eta , \dot {\sigma}} (g)
$$
where  $\bar{R}_{s = -1,\eta ,\hat{\sigma}}(g)$ $\mathbb{E}_{x\sim p(x)}[\mathbb{1}_{\hat{\sigma} (x)\leq \eta}L(-g(x))(1 - \hat{\sigma} (x))]$
$$
+ \pi \mathbb {E} _ {x \sim p _ {P} (x)} [ \mathbb {1} _ {\hat {\sigma} (x) > \eta} L (- g (x)) \frac {1 - \hat {\sigma} (x)}{\hat {\sigma} (x)} ] + \rho \mathbb {E} _ {x \sim p _ {b N} (x)} [ \mathbb {1} _ {\hat {\sigma} (x) > \eta} L (- g (x)) \frac {1 - \hat {\sigma} (x)}{\hat {\sigma} (x)} ]
$$
Then  $R_{bN}(g, - 1)$  and  $\bar{R}_{s = -1,\eta ,\hat{\sigma}}(g)$  can also be approximated from data by  $\hat{R}_{bN}(g, - 1) =$ $\frac{1}{n_{bN}}\sum_{i = 1}^{n_{bN}}L(g(x_i^{bN}), - 1)\hat{\bar{R}}_{s = -1,\eta ,\hat{\sigma}}(g) = \frac{1}{n_U}\sum_{i = 1}^{n_U}[\mathbb{1}_{\hat{\sigma} (x_i^U)}\leq \eta L(g(x_i^U), - 1)(1 - \hat{\sigma} (x_i^U))]$
$$
\begin{array}{l} + \frac {\pi}{n _ {P}} \sum_ {i = 1} ^ {n _ {P}} [ \mathbb {1} _ {\hat {\sigma} (x _ {i} ^ {P}) > \eta} L (g (x _ {i} ^ {P}), - 1) \frac {1 - \hat {\sigma} (x _ {i} ^ {P})}{\hat {\sigma} (x _ {i} ^ {P})} ] + \frac {\rho}{n _ {b} N} \sum_ {i = 1} ^ {n _ {b N}} [ \mathbb {1} _ {\hat {\sigma} (x _ {i} ^ {b N}) > \eta} L (g (x _ {i} ^ {b N}), - 1) \frac {1 - \hat {\sigma} (x _ {i} ^ {b N})}{\hat {\sigma} (x _ {i} ^ {b N})} ] \\ \hat {R} _ {P U b N, \eta , \hat {\sigma}} (g) = \pi \hat {R} _ {P} (g, + 1) + \rho \hat {R} _ {b N} (g, - 1) + \hat {\bar {R}} _ {s = - 1, \eta , \hat {\sigma}} (g) \\ \end{array}
$$
D.5 PUbNe
Our PUbNe formula is as follows: