SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinrui Wang* Wenhai Wan* Chuanxing Geng Shaoyuan Li† Songcan Chen†  
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics  
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence
Learning binary classifiers from positive and unlabeled data (PUL) is vital in many real-world applications, especially when verifying negative examples is difficult. Despite the impressive empirical performance of recent PUL methods, challenges like accumulated errors and increased estimation bias persist due to the absence of negative labels. In this paper, we unveil an intriguing yet long-overlooked observation in PUL: resampling the positive data in each training iteration to ensure a balanced distribution between positive and unlabeled examples results in strong early-stage performance. Furthermore, predictive trends for positive and negative classes display distinctly different patterns. Specifically, the scores (output probability) of unlabeled negative examples consistently decrease, while those of unlabeled positive examples show largely chaotic trends. Instead of focusing on classification within individual time frames, we innovatively adopt a holistic approach, interpreting the scores of each example as a temporal point process (TPP). This reformulates the core problem of PUL as recognizing trends in these scores. We then propose a novel TPP-inspired measure for trend detection and prove its asymptotic unbiasedness in predicting changes. Notably, our method accomplishes PUL without requiring additional parameter tuning or prior assumptions, offering an alternative perspective for tackling this problem. Extensive experiments verify the superiority of our method, particularly in a highly imbalanced real-world setting, where it achieves improvements of up to  $11.3\%$  in key metrics. The code is available at https://github.com/wxr99/HolisticPU.
Positive and Unlabeled Learning (PUL) is a binary classification task that involves limited positive labeled data and a large amount of unlabeled data [36]. This learning scenario naturally arises in many real-world applications like matrix completion[25], deceptive reviews detection[45], fraud detection[35] and medical diagnosis[56]. It also serves as a key component of more complex machine learning problems, such as out-of-distribution detection[63] and adversarial training[18]. Two main categories of PUL methods are cost-sensitive methods and sample-selection methods. However, both approaches face their challenges. The cost-sensitive methods rely on the negativity assumption, which may introduce estimation bias due to the mislabeling of positive examples as negative[49]. This bias can be accumulated and even worsen during later training stages, making its elimination challenging. The sample-selection methods struggle with distinguishing reliable negative examples, particularly during the initial stage, which also results in error accumulation during the training process[23; 57].
As a basic component for various PUL methods, resampling the positive labeled data shows its potential in alleviating the bias brought by negative assumption [49; 52; 30; 33; 61]. For example,
*Equal contribution: Xinrui Wang <wangxinrui@nuaa.edu.cn> and Wenhai Wan <www35@nuaa.edu.cn>
† Corresponding authors: Songcan Chen <s.chen@nuaa.edu.cn> and Shaoyuan Li <lisy@nuaa.edu.cn>.
Figure 1: Averaged predicting scores (output probability) of positive (left) and negative (right) examples in an unlabeled dataset during the first 15,360 iterations of training (30 epochs).
[30] resamples positive examples according to the given class prior and assumed label mechanism to achieve decent performance. In this paper, we dive deeper into this class of strategies. Instead of relying on one single-step prediction which is prone to model uncertainty, we take a holistic view and examine the predictive trend of unlabeled data during the training process. Specifically, we treat the unlabeled data as negative. In each training epoch, we resample over the labeled positive data to ensure a balanced class distribution. We evaluate the model's performance on CIFAR10 and FMNIST datasets[32; 55] with 4 experimental settings. Our pilot experiments show that this resampling method achieves comparable or even state-of-the-art test performance at the outset, but underperforms soon after. Furthermore, the averaged predicting scores (output probability) of unlabeled negative examples exhibit a consistent decrease, whereas those of unlabeled positive examples display an initial increase before subsequent decreasing or oscillating. Conclusively, the averaged predictive trends for different classes exhibit significant differences, as depicted in Figure 1. One possible explanation for these observations is the model's early focus on learning simpler patterns, which aligns with the early learning theory of noisy labels [37]. Although the resampling strategy enjoys these advantages, selecting an appropriate model can be more challenging than the classification task itself due to the lack of a precise validation set.
To break the above limitation, we propose a novel approach that treats the predicting scores of each unlabeled training example as a temporal point process (TPP). It takes a holistic view and surpasses existing methods that focus on examining loss values or tuning confidence thresholds based on a limited history of predictions. By centering on the difference in trends of predicting scores, our approach provides a more comprehensive understanding of deep neural network training in PUL. To further investigate whether this difference in trends is prevalent in individual unlabeled examples, we apply the Mann-Kendall Test, a non-parametric statistical test used to detect trends in the temporal point process [20], to the continuously predicting scores of each example. These scores are classified into three types: Decreasing, Increasing, and No Trend. The statistical test reveals a clear distinction in the trends of predicted scores for each positive and negative example, supporting our observation. Our findings suggest that utilizing the model's classification ability in the early stages may be sufficient for successfully classifying unlabeled examples. This discovery offers us a new perspective on reformulating the problem of distinguishing positive and negative examples in the unlabeled set as identification of their corresponding predictive trends.
We then propose a novel TPP-inspired measure, called trend score to quantify the distinctions in predictive trends. It is obtained by applying a robust mean estimator [3] to the expected value of the ordered difference in a TPP (sequence of predicting scores for each example)[19]. Subsequently, we introduce a modified version of Fisher's Natural Break to distinguish these predictive trends, identifying a natural break point in the distribution of trend score. This approach divides examples into two groups: the group with high trend score represents positive examples, while the group with low trend score corresponds to negative examples. Our approach simplifies the training process by circumventing threshold selection when assigning pseudo-labels. Once the unlabeled data is classified, the remaining problem becomes a binary supervised learning task, and issues such as estimating class priors can be easily addressed. In summary, our main contributions are:
- We demonstrate the effectiveness of the proposed resampling strategy. It is also observed that predictive trends for each example can serve as an important metric for discriminating the categories of unlabeled data, providing a novel perspective for PUL.
- We propose a new measure, trend score, which is proved to be asymptotically unbiased in the change of predicting scores. We then introduce a modified version of Fisher's Natural Break with lower time complexity to identify statistically significant partitions. This process does not require additional tuning efforts and prior assumptions.
- We evaluate our proposed method with various state-of-the-art approaches to confirm its superiority. Our method also achieves a significant performance improvement in a highly imbalanced real-world setting.
2 Our Intuition and Method
We first revisit some important notations in PUL. Formally, let  $x \in \mathbb{R}^d$  be the input data with  $d$  dimensions and  $y \in \{0,1\}$  be the corresponding label. Different from the traditional binary classification, PUL dataset is composed of a positive set  $\mathcal{P} = \{x_i, y_i = 0\}_{i=1}^{n_p}$  and an unlabeled set  $\mathcal{U} = \{x_i\}_{i=1}^{n_u}$ , where the unlabeled set  $\mathcal{U}$  contains both positive and negative data. Throughout the paper, we denote the positive class prior as  $\pi = \mathbb{P}(y = 0)$ .
2.2 Resampling Strategies for Positive and Unlabeled Learning
Figure 2: The accuracy of our resampling method (first 30 epochs). The horizontal line represents the accuracy of the state-of-the-art methods. Early stopping and Leave Zero Out represent different model selection strategies.
Resampling strategies have long been a baseline for dealing with imbalanced data or limited labels, which naturally fits PUL since its key challenge lies in limited labels and potentially imbalanced data distribution[5]. Different from popular resampling strategies applied in PUL[30], we follow the training scheme as [47; 58] to independently sample positive and unlabeled data as different data batches and the loss function is defined accordingly.
$$
\mathcal {L} = \frac {1}{| \mathcal {B} p |} \sum_ {\left(x _ {i}, y _ {i}\right) \in \mathcal {B} _ {p}} \ell \left(\hat {y} _ {i}, y _ {i}\right) + \frac {1}{| \mathcal {B} u |} \sum_ {x _ {i} \in \mathcal {B} _ {u}} \ell \left(\hat {y} _ {i}, 1\right), \quad \hat {y} _ {i} = f \left(x _ {i}\right). \tag {1}
$$
Here, we denote  $f \in \mathcal{F}$  as a binary classifier,  $\ell(\cdot, \cdot)$  as the loss function,  $\mathcal{B}_p$  and  $\mathcal{B}_u$  as the positive and unlabeled training batches respectively. We ensure that  $|\mathcal{B}_p| = |\mathcal{B}_u|$  to achieve a balanced class prior during the training process. This approach emphasizes the labeled data and mitigates the imbalance of positive and pseudo-negative labels, which also provides a good theoretical explanation when dealing with high-dimensional data conforming to different Gaussian distributions. As shown in AppendixA.1, an optimal decision hyperplane can be attained when  $|\mathcal{P}| / |\mathcal{U}|$  equals 1. Figure2 details the performance of our resampling baseline on two datasets under four different settings. It can be observed that the proposed method performs comparably or even better than state-of-the-art methods (P<sup>3</sup>MIX[33] and DistPU[61]) in the early stages of training, as demonstrated by its test performance at certain epochs. However, the method's performance quickly degrades in all 4 settings as the estimation bias worsens during training due to the false negatives introduced by the negativity assumption. We also explore alternative model selection strategies, such as holding out a validation set from given labeled examples or using different versions of augmented data for model selection, as inspired by prior studies [34; 39]. In addition to the common practice of selecting the model from an additional positive validation set, we also implement LZO[34], which selects the model based on the
mixup-induced validation set. As shown in Table1, the performance gap persists, especially when most of the unlabeled data belongs to the positive class.
Table 1: Classification accuracy (Recall rate is reported on Credit Card) on unlabeled training data. Resampling-P represents the model selected on an extra positive validation set. Resampling-LZO represents the model selected through LZO. Resampling* represents the best model selected on the test set which is an ideal case.
Dataset	F-MNIST-1	F-MNIST-2	CIFAR10-1	CIFAR10-2	STL10-1	STL10-2	Credit Card	Alzheimer
Resampling-P	89.93	84.29	81.06	72.93	-	-	60.75	70.09
Resampling-LZO	93.37	92.04	84.87	82.98	-	-	67.24	74.11
Resampling*	94.92	94.57	89.56	85.46	-	-	87.54	76.30
P3MIX-C	91.59	87.65	86.05	88.14	-	-	76.21	68.01
To tackle the above issues, some denoising-based semi-supervised PUL methods, such as [8; 52; 49], have leveraged some threshold tuning or sample selection techniques to achieve acceptable empirical performance. These techniques have been criticized in [54] for relying solely on prediction scores or loss values, as they do not account for uncertainty in the selection process. This becomes even more problematic in PUL, where the noise ratio is typically higher when making a negativity assumption[2].
To break the above limitations, we record the whole predicting process of each unlabeled training example to take a holistic view of the training. It is evident that averaged model-predicting scores for positive and negative data display two distinct trends when implementing the above resampling strategy in the early training stages. Meanwhile, the standard deviation of predictions for positive examples increases rapidly during training, making it increasingly difficult to select an appropriate threshold for distinguishing between positive and negative examples. The appropriate threshold interval for discriminating positive and negative examples quickly shrinks as training progresses, indicating that existing denoising techniques cannot fundamentally alleviate the issues of accumulated errors and increased estimation bias. Therefore, a more robust evaluation measure is necessary beyond relying on raw model-predicted scores or loss values. Implementation details in model selection and visualizations of threshold tuning are provided in AppendixA.
2.3 Identifying Predictive Trends: A Key to Successful Classification
Figure 3: The Mann-Kendall Test is performed on 4 settings of CIFAR10 and FashionMnist datasets. The figure reports the fractions of positive and negative examples in an unlabeled dataset exhibiting different predictive trends during the early training stage (first 30 epochs).
While deep neural networks have strong learning capabilities, they are at risk of overfitting all provided labels, regardless of their correctness. This can result in all unlabeled examples being predicted as negative [1; 59]. We expect the predictive scores of negative examples in the unlabeled set to consistently decrease because all negative examples are given true negative labels by the negativity assumption. On the other hand, the predictive scores of positive examples in the unlabeled training set may not decrease initially because the resampled labeled examples are consistently emphasized from the start of training. To provide more evidence, we use the Mann-Kendall test to analyze the model-predicted scores of each example [20]. This test categorizes the prediction sequence into three situations: Decreasing, Increasing, and No Trend. The calculation process of the Mann-Kendall Test is detailed in Appendix B. Figure 3 shows a contrast between the trends of predicted scores for positive and negative examples. Even when certain positive and negative examples exhibit a similar trend of decreasing prediction scores during training, we observed significant differences in the significance index  $\gamma$  across different classes.
Our next objective is to measure the differences between positive and negative examples. To accomplish this, we require an evaluation measure that captures the significance of the observed trends in model-predicted scores. Before developing our own measure, an important notation in the TPP is first introduced,  $\mathbb{E}[\Delta p]$ , which represents the expected value of the ordered difference in a series of predicting scores.
$$
\mathbb {E} [ \Delta p ] = \lim  _ {t \rightarrow \infty} \frac {2}{t (t - 1)} \sum_ {i <   j} ^ {t} \Delta p _ {i j}, \Delta p _ {i j} = p _ {j} - p _ {i}. \tag {2}
$$
where  $p_i$  is the predicting score (output probability) at  $i$ -th epoch,  $t$  is the number of training epochs.
$$
\tilde {S} = \frac {2}{t (t - 1)} \sum_ {i = 1} ^ {t - 1} \sum_ {j = i + 1} ^ {t} \Delta p _ {i j}, \Delta p _ {i j} = p _ {j} - p _ {i}. \tag {3}
$$
While  $\tilde{S}$  is the empirical mean and unbiased estimation of  $\mathbb{E}[\Delta p]$ , it can be unreliable for non-Gaussian examples and may not handle outliers or heavy-tailed data distributions well as illustrated in [3]. To address these issues, we propose a robust mean estimator inspired by [54; 20], called the trend score  $S$ , which measures the difference between each ordered pair of prediction scores:
$$
\hat {S} = \frac {2}{t (t - 1)} \sum_ {i = 1} ^ {t - 1} \sum_ {j = i + 1} ^ {t} \psi \left(\alpha \Delta p _ {i j}\right), \Delta p _ {i j} = p _ {j} - p _ {i}. \tag {4}
$$
$$
\psi \left(\Delta p _ {i j}\right) = \operatorname {s i g n} \left(\Delta p _ {i j}\right) \cdot \log \left(1 + \left| \Delta p _ {i j} \right| + \Delta p _ {i j} ^ {2} / 2\right). \tag {5}
$$
in which  $\alpha > 0$  is a scaling parameter, and  $sign()$  is the sign function that returns  $-1$  if its argument is negative,  $0$  if its argument is zero, and  $1$  if its argument is positive. The function  $\psi(\cdot)$  can result in a more robust estimation by flattening the values of  $\Delta p_{ij}$  and reducing the influence of minority outlier points on the overall estimation. Besides, we also provide a simplified version as:
$$
\dot {S} = \frac {1}{t - 1} \sum_ {i = 1} ^ {t - 1} \psi \left(\alpha \Delta p _ {i j}\right), \Delta p _ {i j} = p _ {j} - p _ {i}. \tag {6}
$$
Notably,  $\tilde{S},\hat{S},\dot{S}$  are all calculated on each example. Experiments show that both  $\hat{S},\dot{S}$  exhibit better empirical results than  $\tilde{S}$  in Section3. For choosing the stopping epoch  $t$ , we implement the LZO[34] algorithm as described in Section2.2. We also derive a concentration inequality between our trend score  $\hat{S}$  and the expected value of the ordered difference  $\mathbb{E}[\Delta p]$ .
Theorem 2.1. Let  $P = \{p_{ij} | 1 \leq i \leq t - 1, 2 \leq j \leq t, i < j\}$  be an observation set of changes in predictions in which  $\mathbb{E}[\Delta p]$  is the expected values of the ordered difference in a temporal point process and  $\sigma^2$  is the variance of  $P$ . By exploiting the non-decreasing influence function  $\psi(\cdot)$ , for any  $\epsilon > 0$ , we have the following bound with probability at least  $1 - 2\epsilon$ :
$$
\left| \hat {S} - \alpha \mathbb {E} [ \Delta p ] \right| <   \frac {2 \alpha \sigma \sqrt {\frac {2 \log (\epsilon^ {- 1})}{t (t - 1)}}}{1 - \sqrt {\frac {2 \log (\epsilon^ {- 1})}{t (t - 1) \alpha^ {2} \sigma^ {2}}}} = O \left(\left(l o g (\epsilon^ {- 1})\right) ^ {\frac {1}{2}} t ^ {- 1}\right). \tag {7}
$$
It illustrates that the measure we propose is an asymptotically unbiased estimation with a linear weighting of  $\mathbb{E}[\Delta p]$ . The proof is provided in AppendixC. It is also proved in [3] that the deviations of this robust mean estimator can be of the same order as the deviations of the empirical mean computed from a Gaussian statistical sample, which further verifies the advantage of this estimator.
2.4 Clustering Unlabeled Data by the Fisher Criterion
The topic of accurately labeling unlabeled data is widely discussed in various fields, including PUL. In the existing literature, threshold-based criteria and small loss criteria are the two primary approaches used for selecting reliable or clean examples, as seen in studies such as [47; 58; 29; 49]. However, previous works generally select examples based solely on current predictions, ignoring the inherent uncertainty in training examples, leading to longer training times and poor generalization ability[54; 41]. Besides, they often require extensive hyperparameter tuning efforts to choose
appropriate thresholds or ratios for data selection. In this section, we introduce a new labeling approach based on our proposed trend score tackling the above issues.
Our proposed trend score is the naturally comparable one-dimensional data and allows the Fisher Criterion to be a viable choice. It identifies a natural break point in the trend score distribution, which could be used to divide the data into two groups: one with high trend scores and one with low trend scores representing positive and negative examples respectively. Specifically, the objective function of finding this Fisher's natural break point can be formed as follows:
$$
\min  _ {C _ {1}, C _ {2}} \frac {\sum_ {x \in C _ {1}} \left(\hat {S} _ {x} - \mu_ {1}\right) ^ {2}}{\left| C _ {1} \right|} + \frac {\sum_ {x \in C _ {2}} \left(\hat {S} _ {x} - \mu_ {2}\right) ^ {2}}{\left| C _ {2} \right|} \tag {8}
$$
$$
s. t. C _ {1} \cap C _ {2} = \emptyset , C _ {1} \cup C _ {2} = x _ {1}, x _ {2}, \dots , x _ {N}.
$$
where  $\hat{S}_x$  is our derived trend score for example  $x$ ,  $C_1$  and  $C_2$  are the two clusters,  $\mu_i$  is the mean of cluster  $C_i$ , and  $N$  is the total number of data points. We utilize the Fisher natural break point method to automatically determine a threshold value that divided the trend score distribution into two distinct groups. Our implementation introduces an improved algorithm, which reduces the time complexity from  $O(N^2)$  to  $O(N\log (N))$ , as explained in AppendixD. This method eliminates the need for manual threshold selection or hyperparameter tuning, both of which can be time-consuming and error-prone. Furthermore, the data-driven approach we used optimizes the threshold value for the specific dataset under analysis, rather than relying on arbitrary or pre-defined values.
Once the unlabeled data is classified, the remaining task becomes a straightforward supervised learning problem. We directly train by a cross-entropy loss on the estimated labels given by Eq.8 on the backbone network given in Table4. Besides, issues such as estimating class priors can be addressed easily when unlabeled data are classified.
3.1 Classification on Unlabeled Training Set
In this subsection, we first evaluate the performance of our method on the unlabeled training set compared with some state-of-the-art methods. As shown in Table2, our method demonstrates excellent classification performance on the unlabeled training data (the true labels of unlabeled data are not available in STL10). Moreover, a comparison with state-of-the-art prior estimation methods in PUL is conducted to further verify the effectiveness of our approach, and the results are presented in Table3.
Table 2: Classification accuracy (Recall rate is reported on Credit Card) on unlabeled training data.
Dataset	F-MNIST-1	F-MNIST-2	CIFAR10-1	CIFAR10-2	STL10-1	STL10-2	Credit Card	Alzheimer
nnPU	85.31	82.46	83.11	83.23	-	-	62.53	64.01
PGPU	92.02	90.17	85.67	88.38	-	-	42.12	75.09
Self-PU	94.04	91.59	84.06	83.77	-	-	71.00	70.05
P³MIX-C	91.59	87.65	86.05	88.14	-	-	76.21	68.01
Ours	95.41	96.00	91.42	91.17	-	-	98.90	75.13
Table 3: Absolute estimation error with the true positive prior in the first row. We implement an oracle early stopping for the extant methods as defined in [15]. Our method significantly reduces estimation error when compared with existing methods.
Algorithm	F-MNIST-1	F-MNIST-2	CIFAR10-1	CIFAR10-2	STL10-1	STL10-2	Credit Card	Alzheimer
π	0.40	0.60	0.40	0.60	0.50	0.50	0.05	0.50
KM2	0.146	0.106	0.115	0.164	0.096	0.101	0.236	0.094
BBE*	0.082	0.073	0.034	0.059	0.046	0.064	0.112	0.026
(TED)n	0.026	0.020	0.042	0.044	0.024	0.021	0.018	0.014
Ours	0.014	0.021	0.016	0.031	0.018	0.009	0.004	0.011
3.2 Test Performance
We use three synthetic prevalent benchmark datasets including FashionMnist (F-MNIST) [55], CIFAR10 [32] and STL10 [10] and two real-world datasets on fraud detection<sup>1</sup> and Alzheimer
<sup>1</sup>https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud
Table 4: Dataset description and corresponding backbones.
Dataset	#Trainset	#Testset	Input size	Backbone
F-MNIST	60,000	10,000	28×28	LeNet-5
CIFAR-10	50,000	10,000	3×32×32	7-Layer CNN
STL-10	105,000	8,000	3×96×96	7-Layer CNN
Alzheimer	5,890	1,279	3×224×224	ResNet-50
Credit Fraud	8,392	2098	30	6-Layer MLP