Beyond Myopia: Learning from Positive and Unlabeled Data through Holistic Predictive Trends
Authors: Wang Xinrui
Wenhai Wan
Chuanxing Geng
Shao-Yuan Li
Songcan Chen
Venue: NeurIPS 2023 spotlight (2023)
Keywords: positive and unlabeled learning
machine learning
deep learning
temporal point process
data imbalance
TL;DR: Learning from Positive and Unlabeled Data through Holistic Predictive Trends

Abstract: Learning binary classifiers from positive and unlabeled data (PUL) is vital in many real-world applications, especially when verifying negative examples is difficult. Despite the impressive empirical performance of recent PUL methods, challenges like accumulated errors and increased estimation bias persist due to the absence of negative labels. In this paper, we unveil an intriguing yet long-overlooked observation in PUL: \textit{resampling the positive data in each training iteration to ensure a balanced distribution between positive and unlabeled examples results in strong early-stage performance. Furthermore, predictive trends for positive and negative classes display distinctly different patterns.} Specifically, the scores (output probability) of unlabeled negative examples consistently decrease, while those of unlabeled positive examples show largely chaotic trends. Instead of focusing on classification within individual time frames, we innovatively adopt a holistic approach, interpreting the scores of each example as a temporal point process (TPP). This reformulates the core problem of PUL as recognizing trends in these scores. We then propose a novel TPP-inspired measure for trend detection and prove its asymptotic unbiasedness in predicting changes. Notably, our method accomplishes PUL without requiring additional parameter tuning or prior assumptions, offering an alternative perspective for tackling this problem. Extensive experiments verify the superiority of our method, particularly in a highly imbalanced real-world setting, where it achieves improvements of up to $11.3\%$ in key metrics.

Beyond Myopia: Learning from Positive and Unlabeled Data through Holistic Predictive Trends
Xinrui Wang" Wenhai Wan" Chuanxing Geng Shaoyuan Lit Songean Chen
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory Pattern Analysis and Machine Intelligence
Abstract
Learning binary classifiers from positive and unlabeled data (PUL) is vital in many real-world applications, especially when verifying negative examples is difficult. Despite the impressive empirical performance of recent PUL methods, challenges like accumulated errors and increased estimation bias persist due to the absence of negative labels. In this paper, unveil an intriguing yet long-overlooked observation in PUL: resampling the positive data in each training iteration to ensure a balanced distribution between positive and unlabeled examples results strong early-stage performance. Furthermore, predictive Irends for positive and negative classes display distinctly different patterns. Specifically, the scores (output probability) of unlabeled negative examples consistently decrease, while those of unlabeled positive examples show largely chaotic trends. Instead of focusing on classification within individual time frames, we innovatively adopt holistic approach, interpreting the scores of each example as temporal point process (TPP). This reformulates the core problem of PUL as recognizing trends in these scores. We then propose novel inspirre measure for trend detection and prove its asymptotic unbiasedness in predicting changes. Notably, our method accomplishes PUL without requiring additional parameter tuning or prior assumptions, offering alternative perspective for tackling this problem. Extensive experiments verify the superiority of our method, particularly in a highly imbalanced real-world setting, where it achieves improvements of up 11.3% in key metrics. The code is available at https://gituub.com/wee wtt.b//oooooooo//oi/ioott
1 Introduction
Positive and Unlabeled Learning (PUL) is binary classification task that involves limited positive labeled data and large amount of unlabeled data [36]. This learning scenario naturally arises in many real-world applications like matrix completion[25] deceptive reviews detection[45], fraud detection[33. [35] and medical diagnosis(56). It also serves key component of more complex machine learning problems, such as out-of-distribution detection[63] and adversarial (raining[18], Two main categories of PUL methods are cost-sensitive methods and sample-selection methods. However, both approaches face their challenges. The cost-sensitive methods rely on the negativity assumption, which may introduce estimation bias due to the mislabeling of positive examples as negative[49]. This bias can be accumulated and even worsen during later training stages, making its elimination challenging. The sample-selection methods struggle with distinguishing reliable negative examples, particularly during the initial stage, which also results in error accumulation during the training process[23; 57].
As a basic component for various PUL methods, resampling the positive labeled data shows its potential in alleviating the bias brought by negative assumption [49; 52; 30; 33; 61]. For example,
"Equal contribution: Xinrui Wang <wangxinrui @ www..cdu.c.. wdw.endudd...eeeeeeeee...cm and Wenhai Wan <wwh35 @ (anan55@uaa..oo www.eaaaaaucc.m...mmmmcm edu.en> Corresponding authors: Songcan Chen <s.chen @ wwwaaaa.oee wwwa..cu.com wdw.en> and Shaoyuan <lisy wowaaacod.co
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

06
CIPARIO-1 CIPARIO-2
FMNIST-
0.75
Anwraged Predbiiing Store
PMNISTR
04
0.65
Averaged
CIRURT
0.45
CIBARID22
0.1
FMNISE4
FMNIST2
2500
5120
7680 Steratinn
10240
12800
15360
2500
5120
7680
10240
12800
15360
Deraioon
Figure 1: Averaged predicting scores (output probability) of positive (left) and negative (right) examples in unlabeled dataset during the first 15,360 iterations of training (30 epochs).
[30] resamples positive examples according to the given class prior and assumed label mechanism to achieve decent performance. In this paper, we dive deeper into this class strategies. Instead of relying one single-step prediction which is prone to model uncertainty, and examine the predictive trend of unlabeled data during the training process. Specifically, we treat the unlabeled data as negative. In each training epoch, resample over the labeled positive data to ensure a balanced class distribution. We evaluate the model's performance on CIFAR1O and FMNIST [3tasets[32; 55] with 4 experimental settings. Our pilot experiments show that this resampling method achieves comparable or even state-of-the-an test performance at the outset, but underperforms soon after. Furthermore, the averaged predicting scores (output probability) of unlabeled negative examples exhibit a consistent decrease, whereas those of unlabeled positive examples display an initial increase before subsequent decreasing oscillating. Conclusively, the averaged predictive trends for different classes exhibit significant differences, as depicted in Figure 1. One possible explanation for these observations the model's early focus on learning simpler patterns, which aligns with the early learning theory of noisy labels [37]. Although the resampling strategy enjoys these advantages, selecting an appropriate model can be more challenging than the classification task itself due to the lack of precise validation set.
take a holistic view
To break the above limitation, we propose novel approach that treats the predicting scores of each unlabeled training example as a temporal point process (TPP). It takes a holistic view and surpasses existing methods that focus on examining loss values tuning confidence thresholds based on a limited history of predictions. By centering on the difference in trends of predicting scores, our approach provides more comprehensive understanding deep neural network training in PUL. To further investigate whether this difference in trends is prevalent in individual unlabeled examples, we apply the Kenn-l Test, pana-arrettric statistical test used detect trends in the temporal point process [20], to the continuously predicting scores of each example. These scores are classified into three types: Decreasing, Increasing, and No Trend. The statistical test reveals clear distinction in the trends of predicted scores for each positive and negative example, supporting our observation. Our findings suggest that utilizing the model's classification ability in the early stages may be sufficient for successfully classifying unlabeled examples. This discovery offers new perspective on reformulating the problem of distinguishing positive and negative examples the unlabeled set identification of their corresponding predictive trends.
We then propose a novel TPP-inspired measure, called trend score to quantify the distinctions in predictive trends. It is obtained by applying a robust mean estimator [3] to the expected value of the ordered difference in TPP (sequence predicting scores for each example)[19]] Subsequently, we introduce modified version of Fisher's Natural Break to distinguish these predictive trends, identifying a natural break point in the distribution of trend score. This approach divides examples into two groups: the group with high trend score represents positive examples, while the group with low trend seore corresponds to negative examples. Our approach simplifies by circumventing threshold selection when assigning pseudo-labels. Once the unlabeled data is classified, the remaining problem becomes a binary supervised learning task, and issues such as estimating class priors can be casily addressed. In summary, our main contributions are:
training process

---

We demonstrate the effectiveness of the proposed resampling strategy. It is also observed that predictive trends for each example can serve as an important metric for discriminating the categories of unlabeled data, providing novel perspective for PUL.
We propose a new measure, trend score, which is proved to be asymptotically unbiased in the change of predicting scores. We then introduce modified version of Fisher's Natural Break with lower time complexity to identify statistically significant partitions. This process does not require additional tuning efforts and prior assumptions.
We evaluate our proposed method with various state-of-ther tftte-of-thharttt aut approaches to confirm its superiority. Our method also achieves a significant performance improvement in a highly imbalanced real-world setting.
2 Our Intuition and Method
2.1 Preliminary
We first revisit some important notations in PUL. Formally, let x E Rd be the input data with d dimensions and y {0.1} be the corresponding label. Different from the traditional binary classification, PUL dataset is composed of a positive set P = {æi, yi 0),²1 and an unlabeled set U { Vi where the unlabeled set U/ contains both positive and negative data. Throughout the paper, we denote the positive class prior as т Fly 0).
2.2 Resampling Strategies for Positive and Unlabeled Learning
CESMRLD-SCCN
CIPARID-2CCN7
2MN
100+
Resempling
Desamppligg
Ressmpling DisEUUCCVRR2022)
Has ampling DistFU Early Stopping
P'MDXCCE 220223
P*MIXICLLa2222
Barly Stapping
Stppping
Early Stappiig
Lause Oat
Zaro
Laseve Zero
Leave Oat
1022
1830
2M0
kicz
Telo
1360
15300
013
12h
1odez 13489 19560
Figure 2: The accuracy of our resampling method (first 30 epochs). The horizontal line represents the accuracy of the state-of-the-art methods. Farly stopping and Leave Zero Out represent different model selection strategies.
Resampling strategies have long been a baseline for dealing with imbalanced data or limited labels, which naturally fits PUL since its key challenge lies in limited labels and potentially imbalanced data distribution(5]. Different from popular resampling strategies applied in PUL[30], we follow the training scheme [47; 58] independently sample positive and unlabeled data different data batches and the loss function is defined accordingly.
C
Σ (E..
E(vi, (i) +
Σ ((ji,,) 1), i = [(x)). Z.EB.
(1)
Bpp
[Bu|
Here, we denote of E F as a binary classifier, ((..)) as the loss function, B, and B, as the positive and unlabeled training batches respectively. We ensure that |B,] [B,] achieve a balanced class prior during the training process. This approach emphasizes the labeled data and mitigates the imbalance of positive and pseudo-negative labels, which also provides a good theoretical explanation when dealing with high-dimensional data conforming to different Gaussian distributions. As shown in AppendixA.I, an optimal decision hyperplane can be attained when /P|/(< equals 1. Figure2 details the performance of our resampling baseline on two datasets under four different settings. It can be observed that the proposed method performs comparably or even better than state-of -the-art methods (P' MIX(333 and DistPU[61]) in the early stages of training, as demonstrated by its test performance at certain epochs. However, the method's performance quickly degrades in all 4 settings as the estimation bias worsens during training due to the false negatives introduced by the negativity assumption. We also explore alternative model selection strategies, such as holding out a validation set from given labeled examples or using different versions of augmented data for model selection, as inspired by prior studies [34; 39]. addition to the common practice of selecting the model from an additional positive validation set, we also implement LZO[34], which selects the model based on the
3

---

mixup-induced validation set. As shown in Table the performance gap persists, especially when most of the unlabeled data belongs to the positive class.
Table 1: Classification accuracy (Recall rate is reported on Credit Card) on unlabeled training data. Resampling-P represents the model selected on an extra positive validation set. Resampling- represents the model selected through LZO. Resampling* represents the best model selected on the test set which is an ideal case.
Dataset
F-MNIST-I
F-MNIST-2
CIFARIO-1
CIFARIO-2
STLI0-1
STL10-2
Credit Card
Alzheimer
Resampling-P
89.93
84.29
81.06
72.93
60.75
70.09
Resampling-L20
93.37
92.04
84.87
82.98
67. 24
74.11
Resampling* 94,92
94.57
76.30
P'MIX-C 91.59
87.65
86.05
88.14
76.21
68.01
To tackle the above issues, some denoising- based semi-supervised PUL methods, such as [8; 52; 49), have leveraged some threshold tuning sample selection techniques to achieve acceptable empirical performance. These techniques have been criticized in [54] for relying solely on prediction scores or loss values, as they do not account for uncertainty in the selection process. This becomes even more problematic in PUL, where the noise ratio is typically higher when making negativity assumption[2].
To break the above limitations, we record the whole predicting process of each unlabeled training example to take a holistic view of the training. It is evident that averaged model preedciin scores for positive and negative data display two distinct trends when implementing the above resampling strategy in the early training stages. Meanwhile, the standard deviation of predictions for positive examples increases rapidly during training, making it increasingly difficult select appropriate threshold for distinguishing between positive and negative examples. The appropriate threshold interval for discriminating positive and negative examples quickly shrinks training progresses, indicating that existing denoising techniques cannot fundamentally alleviate the issues of accumulated errors and increased estimation bias. Therefore, more robust evaluation measure necessary beyond relying on raw model-predicte scores or loss values. Implementation details in model selection and visualizations of threshold tuning are provided in AppendixA.
2.3
Identifying Predictive Trends: A Key to Successful Classification
CLPA
100%
100%
87%
Nogottiee
80%
Nogettee
80%
Negottee
80%
Negathre
60%
41%
407%
40%
40%
20%
20%
Becrweiing No Townd Incowoiing
Decreeinng
Trwad
Incrweeing
Decreedin
No Thend Incrweinn
Decrweking Trend Incresdin
Figure 3: The Mann-Kendall Test is performed on 4 settings of CIFARIO and FashionMnist datasets. The figure reports the fractions of positive and negative examples in an unlabeled dataset exhibiting different predictive trends during the early training stage (first 30 epochs).
While deep neural networks have strong learning capabilities, they are at risk of overfitting all provided labels, regardless of their correctness. This can result in all unlabeled examples being predicted as negative [1; 59]. We expect the predictive scores of negative examples in the unlabeled set consistently decrease because all negative examples are given true negative labels by the negativity assumption. On the other hand, the predictive scores of positive examples in the unlabeled training set may not decrease initially because the resampled labeled examples are consistently emphasized from the start of training. To provide more evidence, we use the Mennall test to analyze the model--peeicten scores of each example [20]. This test categorizes the prediction sequence into three situations: Decreasing, Increasing, and No Trend. The calculation process of the Menn-ll Test is detailed in AppendixB. Figure shows a contrast between the trends of predicted scores for positive and negative examples. Even when certain positive and negative examples exhibit similar trend of decreasing prediction scores during training, we observed significant differences in the significance index y across different classes.

---

Our next objective is to measure the differences between positive and negative examples. To accomplish this, we require an evaluation measure that captures the significance of the observed trends in model-predicted scores. Before developing our own measure, an important notation in the TPP is first introduced, ElAp], which represents the expected value of the ordered difference in series of predicting scores.
2 t(t
E App
lim
1) Pij. Apis Pj Ps-
700
where p; is the predicting score (output probability) at i-th epoch, is the number of training epochs.
2 $ t(t 1) ;=1
Pig. Apis Pj Pi-
While S is the empirical mean and unbiased estimation of E[Ap]. it can be unreliable for non- Gaussian examples and may not handle outliers or heavy-tailed data distributions well as illustrated in[3]. To address these issues, we propose robust mean estimator inspired by[54; 20], called the trend score S, which measures the difference between each ordered pair of prediction scores:
$= 2
Maadis)) Apij = Pp Pi.
(4)
1(1
(Apr) = sign( sign(A)))))) log(1 + /Apsl + A22,
(5)
in which a > 0 is scaling parameter, and sign() is the sign function that returns -1 if its argument is negative, 0 if its argument is zero, and 1 if its argument is positive. The function (/(+) can result more robust estimation by flattening the values of Apij and reducing the influence of minority outlier points on the overall estimation. Besides, we also provide simplified version as:
$ $=
Σ (aDDii)) ApG Pj p:-
(6)
Notably, S, Š are all caleulated each example. Experiments show that both S, exhibit better empirical results than $ in Section3. For choosing the stopping epoch t, we implement the LZO[34] algorithm as described in Section2.2. We also derive a concentration incquality between our trend score and the expected value of the ordered difference E[Ap].
Theorem 2.1. Let P {pall
1,2 tti < j} be an observation set of changes in predictions in which E, A is the expected values of the ordered difference in temporal point process and o² is the variance of P. By exploiting the nec-eescgg influence function 16(+), for any 0, have the following bound with probability at least 1 2c:
2logle-1)
2000
$ - o <
((l1)
0 0 (1))g(())))(--)))
2log(c-2) a[I-1)a=0d
It illustrates that the measure we propose is an asymptotically unbiased estimation with linear weighting of E[Ap]. The proof is provided in AppendixC. It is also proved in [3] that the deviations of this robust mean estimator can be of the same order as the deviations of the empirical mean computed from a Gaussian statistical sample, which further verifies the advantage of this estimator.
2.4 Clustering Unlabeled Data by the Fisher Criterion
The topic of accurately labeling unlabeled data is widely discussed in various fields, including PUL. In the existing literature, threshold- based criteria and small loss criteria are the two primary approaches used for selecting reliable or clean examples, as seen in studies such as [47; 58; 29; 49]. However, previous works generally select examples based solely on current predictions, ignoring the inherent uncertainty in training examples, leading to longer training times and poor generalization ability[54; 41]. Besides, they often require extensive hyperparameter tuning efforts to choose
5

---

appropriate thresholds ratios for data selection. In this section, we introduce new labeling approach based on our proposed trend score tackling the above issues.
Our proposed trend score the naturally comparable one-dimensional data and allows the Fisher Criterion to be viable choice. It identifies natural break point in the trend score distribution, which could be used to divide the data into two groups: one with high trend scores and one with low trend scores representing positive and negative examples respectively. Specifically, the objective function of finding this Fisher's natural break point can be formed as follows:
Exec, (S, - 11)² ExEc, (S. k22
min C1
1c1
CC2
8.1. C1 C2n 0, C 22.
where Sz our derived trend score for example x, C1 and C2 are the two clusters, J4 the mean of cluster C,, and N is the total number of data points. We utilize the Fisher natural break point method to automatically determine a threshold value that divided the trend score distribution into two distinct groups. Our implementation introduces an improved algorithm, which reduces the time complexity from (NN2) to (ooggg( (N)). as explained in AppendixD. This method eliminates the need for manual threshold selection hyperparameter tuning, both of which can be time-consuming and error-prone prone. Furthermore, the data-driven approach we used optimizes the threshold value for the specific dataset under analysis, rather than relying on arbitrary or pre-defined values.
Once the unlabeled data is classified, the remaining task becomes a straightforward supervised learning problem. We directly train by cross-entropy loss the estimated labels given by Eq.8 on the backbone network given in Table4. Besides, issues such as estimating class priors can be addressed easily when unlabeled data classified.
3
Experiments
3.1 Classification on Unlabeled Training Set
In this subsection, we first evaluate the performance of our method on the unlabeled training set compared with some state-of-the-art methods. As shown in Table2, our method demonstrates excellent classification performance on the unlabeled training data (the true labels of unlabeled data are not available STL10). Moreover, comparison with state-of-the-art prior estimation methods in PUL is conducted to further verify the effectiveness of our approach, and the results are presented in Table3.
Table 2: Classification accuracy (Recall rate is reported on Credit Card) on unlabeled training data.
Dataset
F-NNIST-2 F-NNIST-I
CIFARI0-1 CIFARIO-2 STLI0-1 STLI0-2
Credit Card
Alzheimer
unPU
85.31
82.46
83.11
83,23
62.53
64.01
PGPU
92.02
90.17
85.67
88.38
42.12
75.09
Self-PU
94.04
91.59
84.06
83.77
71.00
70.05
P M
91.59
87.65
86.05
88,
76.21
68.01
Ours 95.41
96.00
91.42
91.17
98.90 75.13
Table 3: Absolute estimation error with the true positive prior in the first row. We implement an oracle early stopping for the extant methods as defined in [15]. Our method significantly reduces estimation error when compared with existing methods.
Algorithm F-MNIST-I F-NNIST-2
CIFARI0-1
CIFARIO-2
STI.10-1
STL.1022
Credit Card
Alzheimer
π
0.40
0.60
0.40
0.60
0.50
0.50
0.50
KM2
0.146
0.106
0.115
0.164
0.096
0.101
0.094
BBE"
0.082
0.073
0.034
0.059
0.046
0.064
0.112
0.026
(TED)"
0.026
0.020
0.042
0.044
0.024
0.021
0.014
Ours
0.014
0.021
0.016
0.031
0.018
0.009
0.011
3.2
Test Performance
We use three synthetic prevalent benchmark datasets including FashionMnist (F-MNIST) [55]. CIFARIO [32] and STL10 [10] and two real-world datasets on fraud detection' and Alzheimer

---

Table 4: Dataset description and corresponding backbones.
Dataset
#Trainset
#Testset
Input size
Backbone
F-MNIST
60,000
10,000
28
LeNet-5
CIFAR-10
50,000
10,000
3x32x32
7-Layer CNN
S11-10
8,000
7-1ayer CNN
Alzheimer
5.890
1,279
3x224x224
ResNet-50
Credit Fraud
8,392
2098
30
6-Layer MLP
diagnosis" as our test set. We provide the dataset description and corresponding backbones in Table4, and the positive priors each setting are given in Table3. More detailed description of benchmark datasets, dataset split and implementation details are given in AppendixF. For each dataset, we run our method for times with different random seeds and report the averaged classification accuracy. We follow the settings in [52; 61] when making the comparison: randomly select 769 positive examples in Alzheimer dataset, 100 positive examples in Credit Fraud dataset and 1000 positive examples in others as the labeled set in training. Classification accuracy on test sets is reported as the main criterion. For highly imbalanced distributed (Credit Fraud) and biasedly selected (Alzheimer) datasets, we provide additional metrics such as Recall, F1 score and AUC on test sets for a more comprehensive comparison.
Table 5: Results of classification accuracy (%) on 3 generic datasets with 6 settings (mean=std),
Algorithm
F-MNIST-I
F-MNIST-2
CIFARIO-1
CIFARIO-2
STLI0-1
STL.0--2
uPU
81.6+
85.7+22.6
76.5++2.5
71.6+ 1.4
76.7+3388
78.2+44.1
unPU
91.4+0.6
90.2=0..7
84.7#2.4
83.7
77.1#4.5
80.4+2.7
Self-PU
90.8.10.4
89.110.77
85.1
83,912.6
78.511.1
80.812.1
87.742.44
89,913.2
87.020.3
82.81100
71.742255
79,8+1.
YPU
92.6.t1.2
90.5+
8..
82.5+1.1
78.411.11
82.9
MIXPUL
90.4+1.2
89,6=1.2
87.0=199
87.0=1.1
77.8+0.7
1..9+119
PULNS
91.0110.5
89.110.8
87.210.6
83.7.2.9
80.210.8
83.610.7
Dist-PU
94.7++0.4
92.4+00.4
86.8+00.7
87.2+0.9
79.8++0.6
82.9++0.4
P'
92.6:10.4
91.8+0022
88.2+10.4
84.7.0..5
80.2:10.9
83.7:100.7
MIX-X
92.8+0066
90.4=0.1
88.7=0.4
87.9
80.7+0.7
84.1=20.3
Ours
95,8+0033
96.0=20.3
91.1-#0.2
90.3+0.11
83.7++0.3
85,3++0.6
3.2.1 Sythetic datasets
Our proposed method consistently outperforms all PUL baselines by 1% 4% on all generic benchmark datasets and settings, as shown in Table 5, demonstrating its superior performance. Furthermore, many existing PUL methods rely on a given positive prior or make various assumptions that are not available in real-world settings, whereas our method does not require any of them. To avoid inherent challenges such as accumulated errors and estimation bias, we transform above challenges into much simpler task of discerning the trend of the model-predicting scores. Considering we can achieve outstanding classification accuracy in unlabeled data, it is natural to expect our method to outperform existing PUL methods. While using some tricks for label noise learning like Co-teaching[22] and large loss criterion could possibly further improve the performance of our method, believe that in most scenarios, method effectively solve existing PUL problems with simplicity.
3.2.2 Real-world datasets
This subsection presents experimental results imbalanced Credit Fraud dataset. In fraud detection, recall is typically more important than precision accuracy, as the consequences of missing a fraudulent transaction can be much more severe than flagging a legitimate transaction as fraudulent. As shown in Table 6, our proposed method achieves significantly higher recall rates and Fl scores, as well comparable accuracy and precision. indicating its ability to better handle highly imbalanced scenarios. Our approach offers novel perspective compared to traditional prediction-basse methods, as the model's predictive trends are affected by the positive prior, as long as the observation outlined in Section 2.3 holds. Furthermore,
real-world datasets, including highly

---

Table 6: Comparative results(%) Credit Card Fraud dataset (meantstd).
Algorithm
Fl score
Recall
Accuracy
Precision
AUC
uPU
89.513.1
13
97.0:1..2
96.5=1.66
93.413.1
nnPU
89.9+1.0
83,4+1.3
98.4+0.1
97,4+1.1
94.2+0.9
mPUU+mixu
89.0+2.8
82.9+1.6
98.1+0.1
96.0+3.2
93.8+2.9
Self-PU
89.0+22.4
85,842..0
99.2+10.1
92.4=3.4
95.642.8
PAN
91.5++0.9
85,4+1.3
99.1
98.5=1.0
96.611.1
VPU
91.7+3.9
84.9+5.7
98.6:0..5
99.7=0.66
96.9=3.1
MIXPUL
82.9+2.8
86.611.3
98.410.3
79.2=3.5
91.3200.7
PULNS
89.0+2.0
83.2+2.1
99.0+0.1
95.6+1.9
94.5+00.7
Dist-PU
87.9+3.4
80.2+4.1
98.8+0.4
97,2+1.66
96.5+2.7
P'MIX-E
91.9+2.1
87.7+2.0
99.0+-.11
96.5++188
97.5+009
M
90.2+t1.4
86.5+1.8
98.8+0.11
94.1+112
97,3+1.2
Method
99.1-10.2
99.0+10.2
99.1110.1
99.3
99.7:1.11
Table 7: Comparative results(%) on Alzheimer dataset (mean=std).
Algorithm
score
Recall
Accuracy
Precision
AUC
uPU
67.6+-2.8
66.1+6.11
68.5+22.2
69.7+:3.5
73.8+:2.9
nnPU
68.6++3.2
69.5+7.2
68.3+22.1
68.0+2.3
72.9+
RP
62.1+:5.6
64.6+15.9
61.6+t:3.2
61.9+:4.5
16.1+
PUSB
69.2+12.4
69.3:12244
69.2:1:244
69.2422244
2..4
PUbN
70,4113.2
72.0:18.4
70.0:11.3
69.41:225
1.3
Self-PU
72.1
75.41:5.1
70.9110.7
69.312.55
1.
aPU
70.5+3.4
75.7++8.2
68.5+118
66.2+0.99
70.7+33.7
VPU
70.2+1.1
76.7+33.6
67.4++0.7
64.7+1.11
73.1
ImbPU
68.8+t1.
70.6:16.5
68.2+008
67.5
73.8:10.7
Dist-PU
73.7:11.6
80.1115.1
71.6:10.66
68.5111.2
77.110.77
Our Method
74.5:12
79.5:1:588
72.8:10.9
70.2:11.6
77.1
our method also demonstrates comparable performance on the Alzheimer dataset to the state-of- the-art method DistPU, which employs various regularization techniques and data augmentation strategies. In both two real-worke settings, our method achieves balanced good performance on all evaluation metries which further illustrates its effectiveness.
90
90-
85
Kay mwdi
85
dlarl
dlar2
[mnik]
cilrrt
1nnid2
faniet
65
alchsiner
coeda
ccedit
60
60
5120
20430
Figure 4: Sensitivity analysis was performed on two parameters: a (left) and stopping iteration (right).
The stopping iteration of LZO (also the use) is denoted by '*' the right.
3.2.3 Ablation Study
To investigate the specific effects of different components (Resampling, trend score, and Fisher Natural Break Partition) in our method, we conducted a series of ablation studies and compared them with some popular alternatives. From Table 8, we can draw several observations: (1) The resampling strategy plays crucial role in our method as it maximizes the discrepancy of the trends in different classes of examples, particularly in the Credit Fraud dataset. serves as an important factor in amplifying the model's early success, which is the foundation of our further approach

---

Table 8: Ablation results (%) on CIFAR-I0 (acc), Credit Fraud (recall) and Alzheimer (fl score).
indicates the enabling of the corresponding components.
Trend Measure
Clustering
Dataset
Resampling
Simplified TS
MK
Natural break
k-means
CIFARI0-1
Credit Fraud
Alzheimer
84.1
88.6
69.2
89.4
99.3
70.5
90.2
99.0
69.7
90.7
99.2
73.9
91.1
99.1
74.5
towards achieving better performance. (2) Our proposed trend score provides a better evaluation metric than the statistic S used in the standardized Mann- Kendall test, and the simplified trend score also shows competitive performance. (3) Fisher Natural Break Partition derives deterministic optimal partitions with better statistical properties and empirical performance compared to heuristic k-means. Moreover, is unrelated to initialization and less time-consuming than the original version, as detailed in AppendixD.
3.2.4 Sensitivity Analysis
In this subsection, we investigate the impact of two hyperparameters, namely the scaling parameter and the stopping iteration (we do not need to manually tune it), on the evaluation of predictive trends for each example. To facilitate comparisons, we set discussed in Section 2.2 for selecting the stopping epoch in our experiments involving mixed labeled data. As depicted in Figure 4, our approach consistently delivers robust outcomes across diverse hyperparameter values. Moreover, the model tends to perform better when 1 and demonstrates basically consistent performance. Figure 4 confirms the effectiveness the LZO strategy which is free of manual intervention in the stopping epoch.
and employ the 1.70 algorithm [34]
Related Works
For long time, learning with limited supervision has been a striking task in the machine learning community and PUL is an emerging paradigm of weakly supervised learning [64; 17]. Despite its close relations with some similar concepts, the term PUL is generally accepted from [36; 12; 14]. Currently, the mainstream PUL methods this problem as cost-sensitive classification task through importance reweighting, among which uPU [13] is the widely known one. Later, the authors of nnPU [31] suggest that uPU gets overfitting when using flexible and complex models such as Deep Neural Networks and thus propose a non non-negativv risk estimator. Some recent studies attempt to combine the cost-sensitive method with model's capability calibrate and distill the labeled set with various techniques like denoise [49], self-paced curriculum [8] and heuristic mix up [33; 52].
Parallel with the cost-sensitive methods, another branch of PUL methods adopts heuristic two-step method. The early trials of two-step methods mainly focus on the sample-selection task to form reliable negative set and further yield the semi-supervised learning framework [57; 35; 23; 6; 27J. Other two-step methods are mainly derived from the large margin principle to correct the bias caused by unreliable negative data such Loss Decomposition [46], Large margin based calibration and label disambiguation [16; 60]. Plus. different techniques have been employed to assign labels for unlabeled data in PUL like Graph- models [4; 62], GAN [24; 27] and Reinforcement learning [38] in recent years. Plus, decision tree based PU methods are also investigated in [53].
Most PUL methods oriented from SCAR (selected completely at random) assumption established on given class prior. In this respect, there emerges some class prior estimation algorithms specially designed for PUL. PE attempts minimize the Pearson divergence between the labeled and unlabeled distribution, PEN-LI [9] and MPE [15] are then proposed to modify PE by using simple Best Bin Estimation (BBE) technique. Unfortunately, most class prior estimation algorithms still rely on specific assumptions and the estimates will be unreliable otherwise[40J. Regarding the possibility of selection bias in the labeling process. the SCAR assumption relaxed in [30]. VAE-PU is the first generative PUL model without a supposed labeling mechanism like SCAR assumption [42] and further investigated in [51]. For more details about PUL, readers are referred recent survey for comprehensive understanding of this subject [2].

---

Conclusion
This study introduces a novel method for Positie-Unnlablled Learning (PUL) that takes a fresh perspective by identifying the unique characteristics of each example's 's predictive trend. Our approach is based on two key observations: Firstly, resampling positive examples to create a balanced training distribution can achieve comparable or even superior performance existing state-of-the-ari methods in the early stages of training. Secondly, the predicting scores of negative examples tend to exhibit consistent decrease, while those of positive examples may initially increase before ultimately decreasing or oscillating. These insights lead us to reframe the central challenge of PUL as task of discerning the trend of the model predicting scores. We also propose novel labeling approach that uses statistical methods to identify significant partitions, circumventing the need for manual intervention determining confidence thresholds or selecting ratios. Extensive empirical studies demonstrate the effectiveness of our method and its potential to contribute to related fields, such as learning from noisy labels and semi-supervised learning.
mcksowledgments and Disclosure of Funding
This work supported by the Natural Science Foundation of China (NSFC) (Grant No.62376122 the National Key R&D Program of China (2022/D01 14801), National Natural Science Foundation of China (61906089), Natural Science Foundation of China (NSFC) (Grant No.62106100 Natural Science Foundation of Jiangsu Province (BK20210292). Graduate Research and Practical Innovation Program Nanjing University of Aeronautics and Astronauties (xcxjh20221601).
10

---

References
[1] D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMLR, 2017.
[2] J. Bekker and J. Davis. Learning from positive and unlabeled data: A survey, Machine Learning, 109(4):71--76, 2020.
[3] O. Catoni. Challenging the empirical mean and empirical variance: a deviation study. In Annales de I'IHP Probabilités et statistiques, volume 48, pages 1148-1188 2012.
[4] S. Chaudhari and S. Shevade. Learning from positive and unlabelled examples using maximum margin clustering. In International Conference on Neural Information Processing, pages 465 445-473.600 Springer, 2012.
[5] N. V. Chawla, K. W. Bowyer, L.. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[6] H. Chen, F. Liu, Y. Wang, L. Zhao, and H. Wu. A variational approach for learning from positive and unlabeled data. Advances in Neural Information Processing Systems, 33: 14844-14884 2020.
[7] Chen, X. Jin, X. Li, and I.. Xu. A generalized catoni's m-estimator under finite a-th moment assumption with e(1, 2). Electronic Journal of Statistics, 15(2):5523-5544, 2021.
[8] X. Chen, W. Chen, T. Chen, Y. Yuan, C. Gong, K. Chen, and Z. Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Conference on Machine Learning. pages 1510-1519. PMLR, 2020.
[9] M. Christoffel. G. Niu, and M. Sugiyama. Class-prior estimation for learning from positive and unlabeled data. In Asian Conference on Machine Learning, pages 221-236. PMLR, 2016.
[10] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings. 2011.
[11] O. Coudray, C. Keribin, P. Massart, and P. Pamphile. Risk bounds for positive- uanabveed learning under the selected at random assumption. Journal of Machine Learning Research, 24(107):131, 2023.
[12] Denis, Gilleron, and Letouzey. Learning from positive and unlabeled examples. 2005.
Theoretical Computer Science,
[13] M. Du Plessis, G. Niu, and M. Sugiyama. Convex formulation for learning from positive and unlabeled data. In International conference on machine learning, pages 1386-1394 PMLR, 2015.
[14] C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 213-220, 2008
[15] S. Garg. Y. Wu, A. J. Smola, S. Balakrishnan, and Z. Lipton. Mixture proportion estimation and pu learning: A modern approach. Advances in Neural Information Processing Systems, 34:8532-8544, 2021.
[16] C. Gong, T. Liu, J. Yang, and D. Tao. Large-margin label-calibrated support vector machines for positive and unlabeled learning. IEEE transactions neural networks and learning systems, 30(11):3471-3483 2019.
[17] C. Gong, J. Yang, J. You, and M. Sugiyama. Centroid estimation with guaranteed efficiency: A general framework for weakly supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 446612841128555 2020.

---

[18] T. Guo, C. Xu, J. Huang, Y. Wang, B. Shi, C. Xu, and D. Tao. On uusitie--ullbeeed classifi- cation gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8385-8393, 8393, 2020.
[19] K. H. Hamed. Trend detection in hydrologic data: the mann kendall trend test under the scaling hypothesis. Journal of hydrology, 34:(344)3350-300 1635000-3---00 2008.
[20] K. H. Hamed and A. R. Rao. A modified mann- kendall trend test for autocorrelated data. Journal of hydrology, 204(1-4):182 1968882996666199 1998.
[21] Z. Hammoudeh and D. Lowd. Learning from positive and unlabeled data with arbitrary positive shift. Advances in Neural Information Processing Systems, 01113388889999 2020.
[22] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[23] Hc, Liu, G. I. Webb, and D. Tao. Instance-dependent pu learning by bayesian optimal relabeling, 2018.
[24] M. Hou, B. Chaib-Draa C. Li, and Q Zhao. Generative adversarial positive-unlabelled learning. 2017.
[25] C.-J. Hsieh, N. Natarajan, and I. Dhillon. Pu learning for matrix completion. In International conference on machine learning, pages 4453-443. PMLR, 2015.
[26] Y.-.. Hsieh, G. Niu, and M. Sugiyama. Classification from positive, unlabeled and biased negative data, In International Conference on Machine Learning, pages 2820-2829, PMLR, 2019.
[27] W. Hu, R. Le, B. Liu, F. Ji, J. Ma, D. Zhao, and R. Yan. Predictive adversarial learning from positive and unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7806-781 14, 2021.
[28] Huang, L. Qu, R. Jia, and Zhao. O2u-net: A simple noisy label detection approach for deep neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3326-3334, 2019.
[29] L. Jumnan and S. C. Hoi. Dividemix: Learning with noisy labelsas semi-supervised learning. In ICLR. International Conference on Learning Representations (ICLR), 2020.
[30] M. Kato, T. Teshima, and Honda. Learning from positive and unlabeled data with selection bias. In International conference on learning representations, 2019.
[31] R. Kiryo, G. Niu, M. C. Du Plessis, and M. Sugiyama. Pusitive-unlabeled learning with non-negative risk estimator. Advances in neural information processing systems, 30, 2017.
[32] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009,
[33] C. Li, X. Li, L. Feng, and J. Ouyang, Who is your right mixup partner in positive and unlabeled learning. In International Conference on Learning Representations, 2022.
[34] W. Li, C. Geng, and S. Chen. Leave zero out: Towards a no-cross-validation approach for model selection, 2020.
[35] X. Li, B. Liu, and S.-K. Ng. Learning to identify unexpected instances the test set. In IJCAI, volume 7, pages 2802-2807, 2007.
[36] X.-L. Li and B. Liu. Learning from positive and unlabeled examples with different data distributions. In European conference on machine learning, pages 218-229. Springer, 2005.
[37] S. Liu, J. Viles-Weed N. Razavian, and C. Seemandez-Granda. Farly-learning regularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:2033 1-20342, 2020.
12

---

[38] C. Luo, P. Zhao, C. Chen, B. Qiao, C. Du, H. Zhang, W. Wu, S. Cai, B. He, S. Rajmohan, et al. Pulns: Positive-unlabeles learning with effective negative sample selector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8784-8792, 2021.
[39] M. Mahsereci, L.. Balles, C. Lassner, and Hennig. Early stopping without a validation set. arXiv preprini arXiv: / 703.09580, 2017.
[40] A. Menon, B. Van Rooyen, C. S. Ong, and B. Williamson. Learning from corrupted binary labels via class-probability estimation. In International conference machine learning, pages 125-134, PMLR, 2015.
[41] D. Moore. Uncertainty. the shoulders of giants: new approaches numeracy. la steen, 1990.
[42] B. Na, H. Kim, K. Song, W. Joo, Y.-Y. Kim, and I.-C. Moon. Deep generative positive-unlabeled learning under selection bias. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1155-1164, 2020.
[43] C. G. Northcutt, T. Wu, and L. Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. In Conference on Uncertainty in Artificial Intelligence, 2017.
[44] H. Ramaswamy, C. Scott, and A. Tewari. Mixture proportion estimation via kerne embeddings of distributions. International conference on machine learning, pages 2052-2060 PMLR, 2016.
[45] Y. Ren, D. Ji, and H. Zhang. Positive unlabeled learning for deceptive reviews detection. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 488 098,4998222 2014.
[46] H. Shi, S. Pan, J. Yang, and C. Gong. Positive and unlabeled learning via loss decomposition and centroid estimation. In IJCAI, pages (689,2 2018.
[47] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang. H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-608, 2020.
[48] G. Su, W. Chen, and M. Xu. Positive-umlabeled learning from imbalanced data. In IJCAI, pages 1095-3001, 2021.
[49] D. Tanaka, D. Ikami, and K. Aizawa. A novel perspective for unlablee learning via noisy labels, 2021.
[50] V. Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[51] C. Wang. J. Pu, Z. Xu, and J. Zhang. Asymmetric loss for positive-umlabeled learning. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2021.
[52] T. Wei, F. Shi, H. Wang, W.-. T. Li, et al. Mixpul: Consistency based augmentation for positive and unlabeled learning, 2020.
[53] J. Wilton, A. Koay, R. Ko, M. Xu, and N. Ye. Positive-unlabeled learning using random forests via recursive greedy risk minimization. In Advances in Neural Information Processing Systems, 2022.
[54] X. Xia, Liu, B. Han, M. Gong. J. Yu, G. Niu, and M. Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106. 00445, 2021.
[55] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist. novel image dataset for benchmarking machine learning algorithms, 2017.
[56] Yang, X.-I.. Li, J.P. Mei, C.-K. Kwoh, and S.-K. Ng. Positive-unlaheled learning for disease gene identification. Bioinformatics, 28(20))2640-26477 2012.
13

---

[57] H. Yu, J. Han, and K. C.-C. Chang, Pebl: positive example based learning for web page classification using svm. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 239-248, 2002.
[58] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34, 2021.
[59] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
[60] C. Zhang, D. Ren, T. Liu, J. Yang, and C. Gong, Positive and unlabeled learning with label disambiguation. IJCAI, pages 4250-4256, 2019.
[61] Y. Zhao, Q. Xu, Jiang, P. Wen, and Q. Huang. Dist-pu: Positive-unlabbledd learning from label distribution perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14461-14470, 2022.
[62] D. Zhou, O. Bousquet, Lal, J. Weston, and B. Scholkopf. Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
[63] Z. Zhou, L.-Z. Guo, Z. Cheng, Y.-F. Li, and S. Pu. Step: Out-of-distribation detection in the presence of limited in-distribution labeled data. Advances in Neural Information Processing Systems, 34:29168-29188 2021.
[64] Z. Zhou. brief introduction weakly supervised learning. National science review, 5(1):44-5) 03,4453,320 2018.
14


Reviews:
[{"review_id": "FmzNFqIkM7", "reviewer": "Reviewer_cdNW", "summary": "The paper proposed an interesting idea of using the trend detection technique for positive and unlabeled learning (PUL). The core problem of PUL is thus reformulated as recognizing trends in these scores. A new TPP-inspired measure is then utilized to solve the task. The idea of the paper is interesting; it is well-motivated and well-supported with a range of experiments from fraud detection and computer vision tasks.\n\n-----------------------\nAfter rebuttal: I am satisfied with the rebuttal and would like to increase my score.", "questions": "1. The comparison with ImbalancedPU in Table 10 in Appendix is interesting to me. Can authors provide a more detailed experimental setting? As the proposed technique is clearly distinguished from prior works. Can we expect the method also achieves decent performance under a class-conditional label noise or imbalance? Some reference:\n   1. \"Centroid Estimation with Guaranteed Efficiency: A General Framework for Weakly Supervised Learning\" \n   2. \"Learning from Corrupted Binary Labels via Class-Probability Estimation\"\n2. It's mentioned that the proposed approach does not rely on the \"selected completely at random\" (SCAR) assumption? Some further discussions are suggested. \n3. What does stopping iteration mean in Sec3?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed technique is well motivated and clearly distinguished from prior works.\n2. The PUL model achieves good performance in the early training stage is an interesting phenomenon and worth studying.\n3. The proposed method is technically sound and it demonstrates a strong performance against both two-step methods and popular reweighting strategies. Plus, comprehensive experimental results on benchmark datasets clearly demonstrate the phenonmenon and the effectiveness of the proposed method.", "weaknesses": "1. The paper attributes the performance downgrade after the early stage success to the label noise problem and the data imbalance problem. However, it is unclear whether all existing reweighting or resampling methods can achieve the same state-of-the-art (SOTA) test set performance in the early training stage and witness a quick performance degradation.\n2. std should be included in the experimental results such as those demonstrated in Figure 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer cdNW:\n\nThanks for your insightful comments. We address your concerns as follows.\n\n### Q1: It is unclear whether existing methods witness the same phenomenon.\n\nA： Most PUL methods currently use a combination of regularization techniques and refinement on mislabeled examples brought by the negativity assumption. To investigate the early learning phenomenon of some recent methods during the training process, we compared their performance in Tables 1 and 2. Our observations suggest that some methods, such as Self-PU and PGPU, perform better than others but are similar to vanilla resampling strategies. Therefore, we opted for the simpler and more commonly used technique of resampling strategies to effectively showcase our discoveries.\n### Q2: std should be included in Figure1\n\nA: Thanks for your advice, it will be added in the revised version.\n\n### Q3: What does stopping iteration mean in Sec3?\n\nA: It represents when should we stop the phase 1 training. Concretely, it equals the *MaxEpoch* in **Algorithm 2**.\n\n### Q4: Detailed experimental setting & extention on class-conditional label noise\n\nA: For a detailed description of our experimental settings, we kindly request the reviewer to refer to Section F in the Appendix. As for the extension on class-conditional label noise, we have included the relevant settings in Table 10. In this case, positive labels are randomly assigned while the class prior is highly imbalanced. Although learning from corrupted binary labels is an interesting problem and highly related to our topic, it is beyond the scope of the PUL and we will talk about it in related works and leave it for our future work.\n\n### Q5: Further discussions on method's reliance on the SCAR assumption\nUnlike traditional reweighting methods that rely on assumptions to derive an unbiased risk estimator, our method takes a new perspective to tackle PUL. By considering the existence of the early learning phenomenon, the proposed trend score can be perceived as a criterion to differentiate positive and negative data even without the SCAR assumption."}, {"review_id": "ApPZ4tQnnh", "reviewer": "Reviewer_vUs8", "summary": "The paper addresses the positive and unlabeled learning problem and presents interesting and meaningful observations. Inspired by the observations, the authors propose an innovative approach that effectively discriminates between positive and negative examples within unlabeled data. The performance of this approach is validated on both synthetic and real-world datasets, further confirming its superiority.\n", "questions": "1. Is it feasible to apply trend prediction methods in other fields, such as semi-supervised learning?\n2. Does recording the predicted values at every iteration for trend analysis yield the best performance? What if we record the predictions every 3 iterations instead? how about every 5 iterations?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- (Clarity) The paper is well organized and clearly written,  the figures are also informative and well-designed.\n- (Novelty) The paper is thought-provoking! Handling the PUL problem from the perspective of predicting trends is highly innovative. Furthermore, the proposed techniques are also novel as far as I can tell. \n- (Quality) The paper is of good quality in my opinion. The algorithm is well designed for identifying trend prediction (via Trend Score) and clustering unlabeled data (via the improved Fisher Criterion). The technical details are all correct as far as I can tell. The empirical evaluation is also comprehensive, covering 6 datasets, including synthetic and real-world datasets, and comparing against popular baselines.\n- (Significance) The proposed techniques are simple, easy to implement and experimentally highly effective, making the algorithm a strong, potentially impactful baseline for future researchers & practitioners to use and/or improve upon.", "weaknesses": "1. It would be appropriate to discuss why there can be significantly different prediction trends for positive and negative examples in unlabeled data.\n2. Both Table 1 and Table 2 lack the experiments on STL10, although you mentioned in Table 2 that the reason is \"the true labels of unlabeled data are not available in STL10.\" However, a better choice would have been to provide an explanation for the absence of this experiment from the first occurrence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We address your concerns as follows.\n\n### Q1: Why prediction trends for positive and negative examples are significantly different?\n\nThe divergence in trends between positive and negative examples can primarily be attributed to the strong fitting capabilities of deep neural networks (DNNs). This often leads to the prediction of all unlabeled examples as negative. However, due to the consistent emphasis placed on resampled labeled examples, positive examples exhibit discernible trends in their predictive scores that differ from those of negative examples.\n\n### Q2: Table 1 and Table 2 lack the experiments on STL10\n\nWe sincerely apologize for any confusion caused and appreciate your valuable suggestions. In order to address the absence of this experiment from the initial entry in Table 1, we will provide a detailed explanation.\n\n### Q3: Is it feasible to apply trend prediction methods in other fields?\n\nIn relation to this question, we are inclined to provide a positive response. We can offer intuitive and straightforward approaches for leveraging out-of-distribution (OOD) data detection or semi-supervised learning, such as:\n- In the context of semi-supervised learning, we can consider unlabeled data as the \"U\" in PUL (Positive-Unlabeled Learning). By utilizing our proposed trend score, we can identify unlabeled examples that belong to a specific class. This enables the formation of multiple binary classifiers for a multi-class classification task.\n- Similarly, we can utilize the predictive trend of a deep neural network (DNN) to detect OOD data. By analyzing the trend exhibited by the DNN's predictions, we can effectively identify instances that lie outside the distribution of the training data.\n\nOverall, we firmly believe that our proposed method can serve as a valuable component within more intricate machine learning techniques.\n\n### Q4: Does the recording frequency affect the model's performance?\n\nHere, we provide some further experiments on the influence of recording frequency on the model's performance.\n\n| Frequency   | CIFAR10-1   | CIFAR10-2 | FMNIST10-1 | FMNIST10-2 |\n| :---:       |    :----:   |      :---:| :---:      | :---:      |\n| 1           | 91.1 $\\pm$ 0.2     | 90.3  $\\pm$ 0.1 | 95.8 $\\pm$  0.3   | 96.0 $\\pm$  0.3   |\n| 3           | 90.9 $\\pm$  0.2    | 90.4 $\\pm$ 0.3  | 94.8 $\\pm$  0.2   | 95.5 $\\pm$  0.2   |\n| 5           | 90.6 $\\pm$  0.5    | 90.0 $\\pm$ 0.3  | 95.1 $\\pm$  0.4   | 95.7 $\\pm$  0.4   |\n\nOur observations indicate that the recording frequency (capturing predictions at intervals of 1/3/5 iterations) does not appear to have a substantial influence on the performance of the model."}, {"review_id": "eLJtneOCEm", "reviewer": "Reviewer_3dQ9", "summary": "In this paper, the authors investigate the topic of positive and unlabeled learning, and solve it from an interesting perspective, namely trend score. The story begins with the empirical observations of different temporal output score trends of positive instances, unlabeled negative instances, and unlabeled positive instances. Based on the observations, the authors propose to compute the trend score under the temporal point process, and use the trend scores to distinguish positive and negative from unlabeled instances. The authors conduct numbers of experiments, and the results show the effectiveness of the proposed method.", "questions": " Overall, I argue the idea of this paper is interesting, but I still have several questions:\n1.\tIf the trend score is the major contribution, whether the resampling is a must in the framework? What are the relationships between them?\n2.\tThe authors compute the trend scores for all unlabeled instances and then use them to identify the positive instances from them. Is that right?\n3.\tIn the problem formulation, the positive label is denoted as y=0. So why the predictive trend of positive instances is “increasing”?\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": " The idea is novel for PU learning.\n- The paper is easy-to-follow.\n- Extensive empirical results.\n", "weaknesses": "Please refer to the following Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We address your concerns as follows.\n\n### Q1: Whether resampling is a must in the framework? What's the relationship between trend score and resampling?\n\nWe argue that resampling is essential in this framework. As we emphasized in Sections 2.2 and 2.3, a benign early learning performance serves as the foundation for our later observations on trend score. Additionally, resampling also amplifies the differences in trend score predictions between examples from different classes.\n\n### Q2: Is it right to compute the trend scores for all unlabeled instances?\nIndeed, we calculate trend scores for all unlabeled instances to enhance the differentiation between positive and negative classes. This process is similar to a typical threshold selection procedure, where we compare the trend score of each instance with the derived threshold to make the classification. \n\n### Q3: Why the predictive trend of positive instances is “increasing”?\n\nWe apologize for any confusion caused. The predicting scores represent the probability of an example belonging to the positive class: $P(y=0)$. Therefore, the predictive trend of positive examples does increase in the early stages."}, {"review_id": "RdNpZjf1vY", "reviewer": "Reviewer_EzoK", "summary": "The paper focuses on addressing the positive and unlabeled learning problem. The authors make an observation that a simple resampling method can yield strong early-stage performance, which has been overlooked in previous literature. Building on this insight, the paper proposes a trend detection measure to tackle the PU learning problem. The proposed method is supported by both theoretical and empirical results.", "questions": "1. In addition to the positive and unlabeled learning problem, can the proposed approach be applied to other machine learning topics such as semi-supervised learning and learning with noisy labels\n2.Why does the performance of the proposed approach vary with different choices of positive priors For example, in Tab10, when highly imbalanced positive priors are used, the proposed algorithm displays a notable gap in performance.\n1. Can the proposed Fisher natural break point method be considered a universal approach for multi-class classification, or are there limitations to its applicability", "limitations": "This is an algorithmic work, and the authors have addressed some limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work studies an important problem, and it is well-written and easy to follow.\n2. The motivation and proposed method are clearly described. It's definitely a new way to approach the positive and unlabeled learning problem from the perspective of trend detection.\n3. The proposed PU learning approach requires no explicit computation of the class prior and avoids the need for additional tuning efforts and assumptions.\n4. The experiments are well-conducted, and comprehensively compared to recent SOTA methods. Ablations and sensitivity analysis are also abundant.\n\nOverall, the paper presents a novel insight for the positive and unlabeled learning problem , and the contribution is definitely relevant and significant.", "weaknesses": "1. Although it is mentioned that the proposal can be applied to more machine learning topics, further explanation should be included.\n2. Can other clustering methods could yield better empirical performance compared with Algorithm 1 ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We address your concerns as follows.\n\n### Q1: Further discussions on how this method applies to other areas\nPlease kindly refer to our responses to Question 3 from Reviewer vUs8.\n### Q2: Whether the method can perform well with different positive priors?\nAs shown in Tab10, our method achieves very competitive performance under various positive priors compared with methods specialized for imbalanced cases. We admit that there still exists a performance gap between different positive priors. This gap may be explained by both the upper bounds and the lower bounds mentioned in Lemma G.1 and G.2. However, we maintain our belief that the predictive trends derived from our proposed resampling method can be a viable choice for the general imbalanced scenarios.\n\n### Q3: Is Fisher natural break point a universal approach?\n Yes, the Fisher natural break point is a universal approach that can be used to find any number of break points and can be applied to multi-class classification tasks. For a detailed exploration, please refer to: \n \n Jenks, George F. \"The data model concept in statistical mapping.\" International yearbook of cartography 7 (1967): 186-190.\n\n Jiang, Bin. \"Head/tail breaks: A new classification scheme for data with a heavy-tailed distribution.\" The Professional Geographer 65.3 (2013): 482-494.\n\n### Q4: Can other clustering methods achieve better performance?\nAs demonstrated in Table 8, the Fisher break point consistently outperforms the traditional k-means algorithm across the majority of datasets. Moreover, the Fisher Natural Break Partition generates deterministic optimal partitions with superior statistical properties compared to many existing clustering methods that heavily rely on heuristics. Additionally, this method is independent of initialization and requires less computational time, thanks to its time complexity of $O(nlogn)$."}, {"review_id": "sz4ZpDilWF", "reviewer": "Reviewer_3ytg", "summary": "In this paper, the authors investigate a phenomenon appearing in positive and unlabeled data learning. i.e. predictive trends is different for positive and negative classes in binary classification task. Based on this observation, the authors propose a holistic approach to capture the predictive trends of different instances. Extensive experiments on real-world and synthetic data sets validate the effectiveness of proposed method.", "questions": "By using trend score, the problem of unlabeled data classification is transformed as threshold selection, which requires that the trend score exhibits obvious difference for these two classes.  Although the authors leverage Fisher Criterion to induce the positive and negative classes, as shown in Figure 1, the predictive trend on CIFAR10 is not completely consistent with the assumption. How does the author view this issue?\n\n=====after rebuttal=====\nThe authors' responses have well addressed most of my concerns of this paper. I would like to raise my overall score from 6 to 7 and recommend acceptance on this paper.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The phenomenon appearing in positive and unlabeled data learning is interesting, i.e. predictive trends is different for positive and negative classes in binary classification task.\n(2) Extensive experiments on real-world and synthetic data sets validate the effectiveness of proposed method.", "weaknesses": "However, there still exists several shortcomings to be overcome as follows:\n(1) In experiments, the comparing algorithms are not presented.\n(2) In Figure 3, what does it mean \"No trend\"? The trend score is a constant or not?\n(3) By using trend score, the problem of unlabeled data classification is transformed as threshold selection, which requires that the trend score exhibits obvious difference for these two classes.  Although the authors leverage Fisher Criterion to induce the positive and negative classes, as shown in Figure 1, the predictive trend on CIFAR10 is not completely consistent with the assumption. How does the author view this issue?\n\n=====after rebuttal=====\nThe authors' responses have well addressed most of my concerns of this paper. I would like to raise my overall score from 6 to 7 and recommend acceptance on this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We address your concerns as follows.\n\n### Q1: Missing comparing algorithms in experiments\nThank you for your valuable suggestions. Actually, due to space limitations, we have included the comparing algorithms in Section F of the Appendix. We will also add a brief introduction of the comparing methods in the experiments section.\n### Q2: Explanation of 'No trend'\nWe apologize for any confusion that may have occurred. It is important to clarify that the Mann-Kendall Trend Test is a statistical test used to analyze the presence of a monotonic trend in a chronological series of a variable. The test can yield three possible results: 'Increasing' if there is a significant increasing trend, 'Decreasing' if there is a significant decreasing trend, or 'No trend' if there is no significant monotonic tendency observed.\n\nIn some cases, there may be chronological series that do not exhibit a clear increasing or decreasing trend. These series are categorized as having 'No trend'. It's worth noting that the Mann-Kendall Trend Test is primarily focused on detecting monotonic trends rather than more complex patterns.\n\nAdditionally, the trend score is calculated to represent the predictive trend of each example. Once the phase 1 training is completed, the trend score can be considered a constant value for that particular example.\n### Q3: Predictive trend on CIFAR10 is not completely consistent with the assumption.\nWe acknowledge that the assumptions regarding the predictive trend of unlabeled positive examples may not hold true for all cases. However, based on the statistical results shown in Figure 3 for CIFAR, it appears that the majority of examples are consistent with our assumption.\n\nEven in cases where unlabeled positive examples exhibit a decreasing trend, the significance index $\\gamma$ demonstrates a substantial difference between the positive and negative classes. This suggests that the proposed trend score is still able to distinguish between the two classes effectively, despite the specific trend direction.\n\nIt's important to consider when the assumption may not hold universally, so the proposed trend score adopts a more robust mean estimator such that the overall statistical results and differences in significance indices support the reliability of our method. If you have any more questions or need further clarification, please let us know."}]

Author rebuttal:

Meta-review: This paper studies positive and unlabeled learning (PU learning) by observing that predictive trends for positive and negative classes are distinct. Motivated by this, the authors propose a holistic method based on TPP to detect the trend and classify the unlabeled data. The proposed method is novel and verified by empirical studies.
All reviewers agree with the acceptance of this paper. During the discussion stage, the authors provided responses to the reviewers and the reviewers maintained their initial score. Therefore, I recommend the acceptance.
Decision: Accept (spotlight)