Who Is Your Right Mixup Partner in Positive and Unlabeled Learning
Authors: Changchun Li
Ximing Li
Lei Feng
Jihong Ouyang
Venue: ICLR 2022 Poster (2022)
Keywords: Positive and Unlabeled Learning
Mixup
Heuristic

Abstract: Positive and Unlabeled (PU) learning targets inducing a binary classifier from weak training datasets of positive and unlabeled instances, which arise in many real-world applications. In this paper, we propose a novel PU learning method, namely Positive and unlabeled learning with Partially Positive Mixup (P3Mix), which simultaneously benefits from data augmentation and supervision correction with a heuristic mixup technique. To be specific, we take inspiration from the directional boundary deviation phenomenon observed in our preliminary experiments, where the learned PU boundary tends to deviate from the fully supervised boundary towards the positive side. For the unlabeled instances with ambiguous predictive results, we select their mixup partners from the positive instances around the learned PU boundary, so as to transform them into augmented instances near to the boundary yet with more precise supervision. Accordingly, those augmented instances may push the learned PU boundary towards the fully supervised boundary, thereby improving the classification performance. Comprehensive experimental results demonstrate the effectiveness of the heuristic mixup technique in PU learning and show that P3Mix can consistently outperform the state-of-the-art PU learning methods.

Published as a conference paper at ICLR 2022
WHO Is YOUR RIGHT MIXUP PARTNER IN POSITIVE AND UNLABELED LEARNING
Changchun Li Ximing Lit-*,, Lei Feng",",Jhonn Jihong Ouyang* CCollege of Computer Science and Technology. Jilin University, China
"College of Computer Science, Chongqing University, China IImperfect Information Learning Team, RIKEN Center for Advanced Intelligence Project, Japan { [changchun1i9 93, iximingg 6 fmaal. com, 1 f cqu. edu. en, ouy je jlu. edu. en
ABSTRACT
Positive and Unlabeled (PU) learning targets inducing binary classifier from weak training datasets of positive and unlabeled instances, which arise in many real-world applications. In this paper, we propose a novel PU learning method, namely Positive and unlabeled learning with Partially Positive Mixup (P' Mix), which simultaneously benefits from data augmentation and supervision correc- tion with a heuristic mixup technique. To be specific, we take inspiration from the decision boundary deviation phenomenon observed in our preliminary experi- ments, where the learned PU boundary tends to deviate from the fully supervised boundary towards the positive side. For the unlabeled instances with ambigu- ous predictive results, we select their mixup partners from the positive instances around the learned PU boundary, so transform them into augmented instances near to the boundary yet with more precise supervision. Accordingly, those aug- mented instances may push the learned PU boundary towards the fully supervised boundary, thereby improving the classification performance. Comprehensive ex- perimental results demonstrate the effectiveness of the heuristic mixup technique in PU learning and show that Pd Mix can consistently outperform the state-of of-the- art PU learning methods.
INTRODUCTION
Positive and Unlabeled (PU) learning refers specific binary classification problem, where only a small number of positive training instances are manually annotated but all other instances are unlabeled (Liu et al.) 2002). Such kind of datasets naturally arise in many significant real-world scenarios such as product recommendation (Hsieh et al., 2015). deceptive reviews detection (Ren let al.) 2014), and medical diagnosis (Yang al. 2012). For specific example, many diseases, e.g., Alzheimer's disease, Amyotrophic Lateral Sclerosis, and Parkinson's disease, are very infrequent and with long latency, hence only few diagnosed patients are known but much larger population of undiagnosed individuals may be either diseased or healthy, Treating the diagnosed ones as positive instances and the undiagnosed ones unlabeled instances results in such PU datasets of medical diagnosis. To meet those practical demands, PU learning has drawn increasing interest from the machine learning community (Bekker& Davis 2020).
Formally, let x E Rd and E {0, } be the feature representation and category label, respectively, where the positive instance indicated by y and the negative one by y 0. In the context of PU learning, the training dataset is composed of the sets of positive instances P {(x; y/ = 1) 72
and unlabeled instances 44 N" where U contains both positive and negative instances. The target learn binary classifier based on such weak training dataset U4.
During the past decades, many PU learning methods have been proposed, where, naturally, the es- sential idea estimate the negative instances from the set of unlabeled instances U. Generally, most of existing PU learning methods can be divided into two categories, termed as sample-selection
Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, China Corresponding author.

---

Published as a conference paper at ICLR 2022
0.36
Real positive prior Disambiguation-Fi Supervised
0.16
Disumbiguation free test
Disambiguation-tre training
0.33
0.12
Supervised test
Error rate
Supervised training
(a)
0.30
0.05
PP
0.27
0.04
0.24
0.00
50
100
150
200
50
100 Epech
150
200
Epoch
0.56
0.25
Real positive prior
Disambiguation free test
Disambiquation-fee
Disambigaation free truining
0.48
Supervised
Supervised test
Error rate
Supervised training
(b)
0.40
0.15
PP
0.32
0.24
0.65
Epoch
Epech
Figure 1: Rates of training instances Predicted Positive (PP rate) and rates of disambiguation free and fully supervised objectives on (a) FashionMNIST and (b) CIFAR-10 (Xiao et al. 2017 Krizhessky 2016 [hen et al., (2220a). The disambiguation free objective suffers from much lower PP rates than the real positive prior as well as the fully supervised objective. This implies the decision boundary deviation phenomenon, which results in higher error rates.
methods and cost-sensitive methods. The sample-selection methods, the name suggests, mainly select reliable negative instances from 4 using various heuristic strategies, e.g. Naive Bayes (Liu et al. 2002), kNN (Zhang & Zuo| 2009), k-means (Chaudhari & Shevade 2012), and reinforce- ment learning (Luo et al. 2002);; and then apply supervised methods over positive and those reliable negative instances. In contrast, the cost-sensitive methods treat all unlabeled instances corrupted negative ones, and correct the estimation bias of the objective by employing well-designed misclas- sification risks such as unbiased risk estimator (du Plessis al. 2014 2015 Kiryo et al., (017)) and maximum margin loss (Shi et al. 2018 Gong et al. 2019b Zhang et al. 2019: Gong et al. (019a).
Orthogonal the aforementioned techniques, we note that some PU learning methods such as (Chen et al. 2020a; Wei et al. 2020) have made preliminary attempts to integrate with the art of mixup, L.e., an cconomic-yet-effectt
data augmentation method (Zhang al.) [2088). Formally, mixup generates an augmented instance (x, y) with the convex combination of any pair of instances {(x;, W:); (})) drawn from the training dataset:
X Ax; + N)x,, 1 Ay, (1 A)y
À Beta(a, a), a (0)
Previous studies have indicated that mixup approximately equivalent applying adversarial train- ing (Zhang et al. 2021), enabling to improve robustness with even scarce and noisy supervision Thulasidasan al. 2019: Carratino al., 2020 Zhang et al. 202I]. Accordingly, it has been successfully used to solve various learning problems with weak supervision, e,g., semi-supervised learning (Berthelot et al. 2019), noisy label learning (Li et al. 2020b), and partial label learning
Yan &
Our story and contribution. Inspired by the recent success of mixup in learning with weak su- pervision, original goal is to thoroughly investigate the impact of mixup in PU learning. To this end, we begin with a naive disambiguation-free objective of PU learning, where all unlabeled instances are treated as pseaiienegative instances, denoted by i {(x, = 0) tp... and the
binary classifier is trained based on P U U. In preliminary experiments, we found an interesting phenomenon, where the number of training instances predicted as positive by the disambiguation free classifier tends to be smaller than usual as illustrated in Fig.1. This phenomenon implies that
SSpecially, apply the early learning regularization (Lu
2020 into the objective to keep stable.
2

---

Published as a conference paper at ICLR 2022
houndary
Supervised boundary
Our l'xtix boundary
Mixup with (c((L))
Labeled positive instance
Untabeled positive instunce
A Unlabeled negative insfance
Augmentte instance
A--o
Am-
Av-
A
A--n
A--
A
Cy-t
A--o
Asn
0-11
A--
Cy-nn
A--1
O
Areu
AS-
Al-0
(a)
(b)
Figure 2: Toy examples of (a) the decision boundary deviation phenomenon and (b) the proposed heuristic mixup for marginal pseudo-negative instances. Best viewed in color.
the disambiguation-free boundary tends deviate from the fully supervised boundary towards the positive side, expressed by a toy example shown in Figgla). We consider that the decision bound- ary deviation mainly caused by the marginal pseudo-negative instances, which lie between the two boundaries. Such instances are more likely to be positive but actually annotated by negative. Motivated by this observation, we extend mixup to a specific heuristic version for PU learning, en- abling to achieve data augmentation and supervision correction simultaneously. Its basic idea is to transform the marginal pseudo-negative instances into augmented instances which are partially positive and yet also lie between the two boundaries, so to push the learned boundary towards the fully supervised one. This can be achieved by selecting the mixup partners for marginal pseudo- negative instances from the positive instances that are around the learned boundary, expressed in 2).. With this insight, we propose a novel PU method, namely Positive and unlabeled learning with Partially Positive Mixup Mix) Generally, Mix is masy-to-implement, where, specifically, we can define the marginal pseudo-negative instances using the predictive results and the positive instances around the boundary using the entropy values of predictive results. To evaluate the ef- fectiveness of Mix,x, we conduct a number of experiments on benchmark datasets. Experimental results demonstrate that MiM can consistently outperform the state-of-the-an PU methods.
THE PROPOSED POMIX METHOD
In this section, introduce the proposed Mix method for PU learning. We first revisit and clarify some important notations: the set of positive instances P = {(x; y/s = 1)] and the set of unlabeled instances Ut {xx))=,, By treating all unlabeled instances as negative, translate
u into the set of pseudo-negative negatiie instances ü {(x,, ys 0)) / Given batches X, P and
X. U. the disambiguation-free objective of PU learning can be formulated as follows:
L( 0)
1
Σ
((((x:))
+
Σ
((f(x;))
X
X
(x.))
(x.
where f(: ②) is trainable neural network, i.e., the binary classifier, parameterized by θ; E(. is the loss function; and B is the coefficient parameter.
To achieve data augmentation and supervision correction simultaneously, M transforms X, and Xr into the batches of augmented instances &, and Lt using the proposed heuristic mixup tech- nique. Accordingly, the objective of pa Mix then expressed as follows:
L(R Ry θ)
1
Σ
((((x; 9). +
Σ ()(())),)),
(2)
(
&.
RS &. = HeuristicMixup(x,.KK a))
(3)
where E (0, 0o) is hyperparameter of mixup. Next, describe the details of heuristic mixup.
3

---

Published as a conference paper at ICLR 2022
Algorithm Training procedure of Mix,, P'Mix-E and P'Mix-C
Input:
P U training instances; 3: coefficient parameter; ".: thresholding parameter: k; size of the candidate mixup pool; o: hyperparameter of mixup; yf coefficient parameter of carly-learning > fj 0 for på Mix and Mixi--C
regularization
Output:
O: binary classifier parameters
1: Initialize θ the mean-teacher parameters O and the candidate mixup pool Mend randomly,
translate U into U:
2: for t = 2, ... .Mar do
3: Shuffle P uu into I mini-batches and denote the i-th mini- by (A (K, N;):;
4: for 1,2, .... do
5:
Estimate marginal pseudo-negative instances Xmpn using Eq. 6
6:
Select the mixup partners for each instance within X3 U X using Eq.5
7:
Set labels of { {xx, y 0)}(x, y E X f(x; ①) > ? to & Optional for P*Mix-C
8:
Construct R U X by applying Eq. 4 to X U X and their mixup partners;
1 |+ [
9:
Estimate
j=1
for instances in X U X by f(:
Optionall for P'Mix-E
10: Update + by Ve (C( Ru θ) + Ratr ( (1 y, },) +
with Adam;
end for
12:
Update Xend using Eq.
13:
Update @ by ④ with the move-average;
> Optional for M'xix-E
14:
end for
2.1 TRAINING WITH HEURISTIC MIXUP
Basically, for each instance (xi, Wi) E Xp U Xr we select a mixup partner (x,3;) to generate an augmented instance (x:, Di) using the modified mixup operator Berthelot et al. 2019):
x, X'x; + (1-X)xx, D = +1 X)15,
X' max(A, 1 A), A Beta(a, a (0, (4)
accordingly forming the augmented instance sets R, and X,,
Our heuristic mixup refers to a guidance of mixup partner selection to refine the imprecise su- pervision within Xy. We take inspiration from the phenomenon, where the boundary learned by Eq.(T) tends to deviate from the fully supervised boundary towards the positive side as illustrated in aa.. The marginal pseudo-negative instances Xmpm C X, lie between the two boundaries, and they are more likely to be positive but actually annotated by negative. To resolve this problem, for each of them we uniformly select a mixup partner from the candidate mixup pool Xond C P of pos- itive instances that are around the current learned boundary, which partially positive and yet also lies between the two boundaries as expressed Fig (b). Besides, for positive instances Xx and other seeudo-negative instances Xr &mpps uniformly choose their mixup partners from X, U Xz The overall mixup partner selection is formulated as follows:
generate augmented instance
Uniform (Xan))
if (xx, Us) E Xmpm:
(x,, Us)
(5)
Uniform(24 (X, U X.) (x,, yi) E X, Xr, \ Xmpn.
In what follows, we introduce how to estimate the marginal construct the candidate mixup pool X
instances Xmpo and
BBecause compute individual loss terms for positive instances and pseudo-negative in Eq ap- propriately, we define X' maax(X, 1 - X) to guarantee the feature each augmented instance & is closer x; than the mixup partner xy. Consequently, (R,, T.) assigned into X, if (xs, v.) E Xx P, otherwise.
4

---

Published as a conference paper at ICLR 2022
Marginal pseudo-negative instance estimation. Because the fully supervised boundary is actly unknown, we have to estimate the set of marginal negudd-eegative instances Xmpn from X,- In this work, define them the "unreliable" pseudo-negativg instances measured by the predic- tive scores with thresholding parameter 7 E [0.5, 1]:
Xmpn {xx, y 0) (x, y/ - 0) E Xu, 1 - f(x; 5
where 7 = 0.5 implies Xmpn 0. and y = 1 means Xmpn Xu.
Candidate mixup pool. We maintain candidate mixup pool Xend containing the positive in- stances around the current learned boundary from P. To be specific, for each positive instance we compute its entropy value of the predictive score, and update the candidate mixup pool with the top-k positive instances as follows:
Xend { (x, y (x) y
P, (x((x; θ)) Rank (a)((()( O))
where H())) is the entropy, and Rank(-) outputs a set positive instances with the top-k maximum entropy values. For efficiency, we update Xend per-epoch. The full training procedure is shown in Algorithm
2.2 ROBUSTNESS
The augmented instances within & also suffer from imprecise supervision using the heuris- tie mixup. To make pś more robust, we employ two tricks, ie., early-learning regularization (Liu et al. 2020) and pseudo-negative instance correction. We call the versions with early-learning regularization and pseudo-negative instance correction as M' and M'Mix-C, respectively.
Early-learning regularization. We employ the early learning regularization prevent the mem- orization of imprecise supervision (Liu et al. 2020). For each mixup instance within X, U &g, we estimate auxiliary target vector y, and formulate the early-learning regularization below:
Rey (f (1(3, v.) 1 Ral, θ) 1
1
1=1 log (1 (f(2,
Here, we estimate the target vector y of each instance by using the mean teacher technique (Tar- vainen & Valpola 2017), and incorporate Eq. to Eq, (
Pseudo-negative instance correction. We concentrate on the pseudo-negative instances with high confidence to be positive {xx, (x, y 0) E X.. f(x Q) > }. We directly revise their labels to positive before their corresponding mixup operators.
EXPERIMENT
3.1
EXPERIMENTAL SETTINGS
Datasets. In the experiments, employ three prevalent benchmark datasets, including Fashion- MNIST (F-MNIST) (Xiao et al. 2017) CIFAR- Krizhevsky 2016 and STL-10 (Coates et al, 2011 The dataset statistics are described in Table Note that all benchmark datasets have TO category labels, and we denote them with integers ranging from 0 to 9 following the default settings in torchvision 0.10.0. For each dataset, group those category labels into two disjoint sets as positive or negative, and generate two synthetic PÚ datasets by reversing the definitions of positive and negative labels. Following the protocol of (Chen al. 2020a), the specific definitions of la- bels ("positive" vs "negative") are as follows: F-MNIST-I: "14,7" vs "0,2.3,5.6,8,9" F-MNIST-2: "0,2.3.556,8.9" vs "1,4,7"; CIFAR-I0-1: "0,1,8,9" "2,3,4.5,6,7". CIFAR-10-2: "2.3,4.5,6,7" vs "0,1,8,9"; STL-10-1: "0,2,3,8.9* vs "1,4,5,6,7", STL-10-2: "1,4,5,6,7" vs "0,2,3,8,9*. For each dataset, we randomly select 1. 1,000 positive instances from the training set, and 500 instances as the validation set.
http://ww..e..oorrooeeced//t///t htm

---

Published as a conference paper at ICLR 2022
Table Specification of datasets and corresponding backbones.
Dataset
#Train
#Test
Input size
Backbone
F-MNIST
60,000
10,000
28x28
LeNet-5
CIFAR-10
50,000
10,000
3x32x32
7-layer CNN
STL-10
105,000
8,000
3x96x96
7-layer CNN
Table 2: Results of classification accuracy (mean=std). The highest scores among PU learning methods are indicated in bold.
Dataset
F-MNIST-I
F-MNIST-2
CIFAR-10-1
CIFAR-I0-2
STL-10-1
STL-10-2
uPU 1..
84.0%:000
76.5+2.5
71.611.4
76.7+3.8
78.2=:4.1
anPU
89.7.10.8
88,8.0.9
84.7112.4
83.7.10.6
77.114.55
80.4112.7
unPU+mixup
91.4.10.3
88.2.10.7
87.2.10.6
85.811.22
79.8:10.8
82.2.10.9
Self-PU
90.8.10.4
89.110.77
85.1-0..8
83.9112.6
78.511.1
80.8-2.11
PAN
88.7=1.2
83.6#2.5
87.0=0.3
82.8+10
77.7#2.5
19.8=1.4
VPU 90.6+1.2
86.820.8
86.8+1.2
82.5+1.1
78.41
82.9=0.7
MIXPUL
87.5+1.5
89.0+00.5
87.0#1.9
87.0#1.1
77.8+0.77
18.9=119
PULNS
90.7++0.5
87.9+0.5
87.2++0.6
83.7+-2.9
80.2++0.8
83.6++0.7
P P'Mix
91.9:1..3
89,5
88.2+:0.4
84.7+00.5
80.24:0.9
83.7-00.7
Pd M'MiX-C
92.0-10.4
89.4
88.7:10.4
87.9-0.55
80.7+10.7
84.1-10.3
Supervised
95.2.10.2
95.2.10.2
91.3110.3
91.3.10.3
85.6.10.6
85.6
Baseline methods. To verify the effectiveness pixix, we utilize eight PU learning baselines, including uPU (du Plessis et al.) 20144) nnPU (Kiryo et al.] (017). nnPU+mixup, Self-PU (Chen ett al. 2020b), PAN (Hu et al.J 2021)) VPU (Chen al., (022aa). MIXPUL (Wei et al.) 2020) and PULNS (uo et al. 2021), well the supervised method for comparison. The corresponding implementation details of baselines are present in Appendix[A] For all comparing methods, we adopt the classifiers (including the discriminator of PAN) by LeNet-5 for F-MNIST, and 7-layer CNN for CIFAR-10 10 and STL-10. Specially, the baseline methods of uPU, nnPU and Self-PU require the prior knowledge of class proportion, however, the prior actually unknown for STL-10 since it contains many "real" unlabeled instances. Accordingly, we estimate the class proportion of STL-10 by using the SOTA KM2 method (Ramaswamy al. 2016) before evaluating uPU, nnPU, and Self-PU.
Implementation details of Mix. We implement pd Mix, Mixi and p-Mix-C by using Py- torch (Paszke et al. 2019) with the Adam algorithm (Kingma & Ba 2014). We employ the cross to and the size k of the candidate mixup pool Xend to 100, and choose the coefficient parameter 3 from {0.8, 0.9, 1.0), the thresholding parameter y from {0.85, 0.9, 0.95). We will make the sensitiv- ity analysis on {B,r} later. Specially, the early-learning regularization parameter of PaMix-E is chosen from (1.0, 2.0, 4.0, 5.0).
entropy function
the loss function l of (2. fix the mixup hyperparameter
3.2 CLASSIFICATION PERFORMANCE
For each dataset, independently run each comparing method 5 times and report the average clas- sification accuracy in Table Generally, our MiMix-E and P'Mix-C consistently outperform all PU learning baselines on all benchmark datasets, indicating their superior performance. Compared with unPU+mixup, VPU and MIxPUL, which utilize the typical mixup technique, both MiMix-E and Mix-- achieve significant performance gain in most cases, iee. about 1% 5% improvements. These results imply that our proposed heuristic mixup benefits to the supervision correction within marginal pseudo-negative instances. Compared with the three cost-sensitive PU learning methods uPU, nnPU and Self-PU, P*Mix-E and P'Mix-C also outperform them by about 1% 4% in all cases. Besides, we observe that all discriminative PU learning methods except uPU perform bet- ter than the GAN-based PU learning method PAN in most cases. The possible reason is that the imprecise supervision within unlabeled instances makes the identification fake instances for the discriminator more difficult, resulting in worse classifier.

---

Published as a conference paper at ICLR 2022
Table 3: Results of ablative study (mean:tstd). The highest scores are indicated in bold.
Dataset
F.MNIST-I
F-MNIST-2
CIFAR-10-1
CIFAR-10-2
STL-10-1
STL-10-2
DF
75.24112
62.74:2.8
72.0=3.2
57.413.7
78.1
80.6:E2.4
DF+mixup
1.,4+1.7
1,
79.2+:3.0
67.4+2255
78.9+00.3
1.
M'Mix
87.011.1
79.0111.
87.011.1
84.310066
79.8
83,4:10.7
DF-E
90.1+0.7
74.24:5.5
82.4+1.6
69.4+3300
67.3+22.0
75.0+:3.7
90.6+0.77
86.1+2.5
85.7+0.7
76.4+0.9
78.3+1.1
79.3++2.3
P'MiX-E
91.9110.3
89,510.55
88.2110.4
84,7.10.5
80.2.10.9
83.7.10.7
DF-C
89.6+1.8
87,42244
87.2+0.8
84.7+1.11
80.2+3.0
82.7+2.6
Dm-c+m
91.6+0.33
88,3+122
87.711
81,3+3,6
79.9+3.11
81.6+288
Pd MiMx-C
92.0.10.4
89.4
88.7
87.9
80.7-10.7
84.1-10.3
F.MNIST-I
CIFAR-10-1
STI-110-1
0.96
Accuracy
0.92
Accuracy
0.84
Accuracy
0.80
0.99
0.73
0.72
0.88
0.56
0.48
0.56
0.884
0.36
0.45
011
0.2
01.3
0.6
11. 0.9 1.0
0.1
01.
0.5
0,4
11.5
R.7
0.8
3.0
0.2
11.1
11.0
U.5.
R. 1.0
B
F-MNIST-2
CIFAR-10-2
STI-1102
0.99
0.90
0.88
Accaracy
0.90
Accuracy
0.85
Accurucy
0.80
0.66
0.73
0,64
0.80
0.46
0.56
B
Figure 3: Sensitivity analysis of the coefficient parameter 3.
3.3
ABLATION STUDY
To evaluate the effectiveness of our proposed heuristic mixup, we conduct the ablation experi- ments on all benchmark datasets. Specifically, we compare M'xix, pê and PêMix-C with the Disambiguation-Free (DF) objective of [q. DF+mixup, augmented by the typical mixup technique, and their versions with early-learning regularization ("-E") and pseudo-negative instance correction ("-C"). The experimental results are reported in Table 3 It clearly demonstrates that the proposed heuristic mixup can significantly improve the classification performance. This result is expected because the heuristic mixup simultancously achieve data augmentation and super- vision correction by refining the imprecise supervision within marginal pseudo-negative instances. Besides, also observe that both early-learning regularization and pseudo-negativy instance correction contribute to the improvement of the classification performance in all cases, proving the effectiveness of those tricks in improving the robustness of models.
3.4
SENSITIVITY ANALYSIS
In this section, examine the sensitivities of the coefficient parameter B and the thresholding parameter 7.
Sensitivity of B. We examine the impact of different /3 values over the set (0.1,0.2. 1.00) by P-Mix-C and plot the experimental results in Fig We omit the results of 2aMix and MiMix-E due to their similar performance curves and also page-limitation. Obviously, the performance achieves the highest and is relatively stable when 3 > 0.8, and it sharply drops as the values become smaller especially on CIFAR- 10-1R- 10-2 STL-10-1 and ST-- Notice that the coefficient pa- rameter / is used to balance the importance of the positive and pseudo-negatiog parts in Eq. (2). The larger or smaller value of 3 will result the indecent importance of the aseudo-negation part, lead- ing unstable classifier. Therefore, suggest tuning 3 over the set (0.8, 0.9, 1.0) in practice.
7

---

Published as a conference paper at ICLR 2022
F-MNIST-II
CIFAR-10-1
STL-10-1
Accuracy
0.93
Accuracy
0.48
Accuracy
0.85
0.84
0,80
0.84
0.40
0.75
0.80
0.70
0.76
0.5
0.5
0.7
0.8
0.9
1.0
0.5
0.5
0.8
0.9
1.0
F-MNIST-2
CIFAR-10-2
STI.-1022
0.54
0.35
Accuracy
01:00
Accuracy
0.88
Accuraty
0,88
0.76
0.30
0.84
0.72
0.70
0.76
0.72
0.5
11.0
11.7
11.8
11.0
1.31
0.5
0.6
0,7
0.8
0.9
1.0
01.5
11.
0..
11.1
1.11
Figure 4: Sensitivity analysis of the thresholding parameter y.
Sensitivity 7. Generally, the thresholding parameter y utilized to estimate the set of marginal pseudo-negative instances Xmpn from pscudo-negative instances Xz However, both the early- learning regularization and pseudo-negative instance correction may be affected by y since they mainly used to control the corrupted negative instances within Xu Xmpn- Accordingly, perform the sensitivity analysis of y by Pd Mix. Specifically, the set {0.5, 0.55. ... .L00. As shown in 4ig PoMix achieves the best performance when {0.85, 0.9, 0.95], and performs relatively worse when is too small too large. This result is expected because the smaller value of y implies that Xmpm contains fewer marginal pseudo-negatee instances, e.g., Xmpn = when y 0.5, and it will hurt the supervision correction of heuristic mixup. Besides, when becomes too large, too many pseudo-negative instances will be treated as marginal pseudo-negative ones, e.g., Xmpm Xr when y 1.0, resulting in the reduction of the variety of augmented instances. In summary, its suggested setting is given by {0.85, 0.9, 0.95}.
vary the value of
4 RELATED WORK
In this section, we review the representative studies on PU learning, especially the ones most related Pd Mix. Besides, we briefly introduce the recent studies of mixup for data augmentation.
4.1 PU LEARNING
Weakly supervised learning (Zhou 2018) mainly tackles datasets with weak supervision, such as
incomplete labels (aan Engelen Hoos [2020] Li al. (2021a), inexact labels (Feng et al.) [2020a b
Li & Wang 2020, Li al. 2020a| 2021b), and inaccurate labels (Li et al. 2020b Nguyen al., 2020). PU learning is an emerging paradigm of weakly supervised learning- The early PÚ learning works focus on the sample-selection paradigm. As the name suggests, the basic idea of sample- selection methods is to select reliable negative instances from unlabeled instances to form pseudo- binary dataset before applying supervised methods. Existing methods have proposed various heuris-
tic strategies for negative sample selection, e.g., 1-DNF (Yu et al., 2002) 2004 Peng et al. 2008
Naïve Bayes (Liu al. 2002). Rocchio extraction (Li & Liu 2003), ANN (Zhang & Zuo
2009
k-means (Chaudhari & Shevade 2012), large margin method (Gong et al.) (218) and reinforcement
learning (Luo et al. 2021). Early works based on sample-selection spirit mainly focus on exploiting various traditional classification and clustering approaches to construct the heuristic strategy. Re- cently, PULNS (Luo et al.) 2021) employs a negative selector trained by a reinforcement learning framework, where the selector, the selection of negative instances, and the performance of the clas- sifier are treated as the agent, action and reward, respectively. Then the classifier is induced from the mixture of positive instances and negative ones selected by the selector.
In contrast, the community of PU learning has recently paid more attention to the cost-sensitive methods, which directly treat all unlabeled instances corrupted negative instances and correct
8

---

Published as a conference paper at ICLR 2022
the estimation bias of the objective by employing well-designed misclassification risks. The uPU (du Plessis et al. 2014, 2015) made an early attempt of unbiased risk estimation,, which reformu- lates the misclassification risk as an equivalent and ubiased form depending only PU datasets. However, as reported in (Kiryo et al. 2017), the risk of uPU would become negative due to over- fitting when using the flexible and complex models such as deep networks. To remedy this issue, the authors of (Kiryo et 2017)) suggest the nnPU method, i.e., the non-negative version of uPU. Besides, the Self-PU (Chen al. 2020b) further considers the learning capability of the model itself, and jointly employs three self superviieis techniques, i.e., a self-paced strategy to discover confident positive and negative instances, self-calibrated instance-aware loss explore meaning- ful supervision over unconfident instances, and a self-supervision consistency by teacher-students learning. Other cost-sensitive methods are based on the maximum margin objective, and refine the bias of corrupted negative instances by various tricks, such as unbiased centroid estimation of unla- beled instances (Shi et al. 2018 Gong et al., 2019b), label calibration with a hat loss (Gong et al., 2019a), and margin- based label disambiguation (Zhang et al. 2019))
In parallel with the aforementioned methods, several GANebased PU learning methods (Hou et al, 2018 Chiaroni et al. 2018, Guo et al., 20200] Na et al. 2020 Hu et al. 2021) have been proposed. The GenPU (Hou et al., 2018) generates positive and negative instances, and induces a classifier from those generated instances. The PAN (Hu et al. 2021) is based on adversarial learning on the
probability distributions of discriminator and a classifier.
Orthogonal to those PU learning methods, our pa Mix concentrates on data augmentation and extends the well-established mixup technique (Zhang et al., (018) to a specific heuristic version for PU learning, enabling achieve data augmentation and supervision correction simultaneously. The recent PU learning works most related to pd Mix are VPU (Chen et al., 2020a) and MixPUL (Wei al. 2020), in which the mixup is used as a regularization to improve the robustness of classifiess and performed among labeled and unlabeled instances randomly, In contrast, our M constructs a heuristic mixup partner selection to refine the imprecise supervision within unlabeled instances.
4.2 MIXUP AUGMENTATION
The mixup technique Zhang et al. 2018) generates augmented instances with convex combinations of training instances. Despite its simplicity. it effectively improve the robustness with even scarce and noisy supervision Thuulasidasan et al.] 2019; Carratino et al., [,20000] Zhang et al. 2021) Further, a number of modified mixup versions (Verma et al.] [019]] Yun al. 2019 Guo al. 2019 Kim et af. 2020 Hendrycks et al. 2020) have been proposed, e.g., manifold mixup that generates convex combinations on the latent feature space (Verma et al. (201)) and puzzle mixup that further explores the saliency information and underlying statistics of instances (Kim et al. 2020). In this work, we extend the typical mixup to a specific heuristic version for PU learning.
CONCLUSION
In this paper, we propose novel PU learning method named P'Mix, which extends the typical mixup technique to heuristic version. The story begins with the observation of the decision bound- ary deviation phenomenon, which inspires to propose a guidance of mixup partmers selection, especially for the marginal pseudo-negative instances. Fortunately, this heuristic mixup technique can simultancously achieve data augmentation and supervision correction for PU learning. Gen- erally, our Mi is casy-to-implement, and we also employ two tricks to improve the robustness of Mix. We compare Mix against a number of existing PU learning methods benchmark datasets. Experimental results show the superior performance of paMix and the effectiveness of the heuristic mixup technique.
ACKNOWLEDGMENTS
We would like to acknowledge support for this project from the National Key R&D Program of China 2No.2021ZD01 12501, No.2021ZD01125000 the National Natural Science Foundation of China (NSFC) (N0.61876071, No.62006094), the Key R&D Projects of Science and Technology Department of Jilin Province of China (N0.201802010035)
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109(4):11--70, 2020.
David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Paperno, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. Neural Information Pro- cessing Systems, pp. 5050-5060. 2019.
Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regular- ization. arXiv preprint (eXXi:20066000499 2020.
Sneha Chaudhari and Shirish K. Shevade. Learning from positive and unlabelled examples using maximum margin clustering. In International Conference Neural Information Processing, pp. 465-473, 2012.
Hui Chen, Fangqing Liu, Yin Wang. Liyue Zhao, and Hao Wu. A variational approach for learning from positive and unlabeled data. In Neural Information Processing Systems. 2020a.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Confer- ence on Machine Learning, pp. 1510-1519. 2020b.
Florent Chiaroni, Mohamed-Cheri Rahal, Nicolas Hueber, and Frédéric Dufaux. Learning with A generative adversarial network from a positive unlabeled dataset for image classification. In IEEE International Conference Image Processing, pp. 1368 1878-133..020 2018.
Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In International Conference Artificial Intelligence and Statistics, pp- 215-223. 2011.
Marthinus Christoffel du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Neural Information Processing Systems, pp. 703-711. 2014.
Marthinus Christoffel du Plessis. Gang Niu, and Masashi Sugiyama. Convex formulation for learn- ing from positive and unlabeled data. In International Conference on Machine Learning, pp- 1386 1394. 2015.
Lei Feng, Takuo Kancko, Bo Han, Gang Niu, Bo An, and Masashi Sugiyama. Learning with multiple complementary labels. In International Conference on Machine Learning, pp. 3072-3081. 2020a.
Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Prov- ably consistent partial-label learning. Neural Information Processing Systems, pp. 10948- 10960. 2020b.
Chen Gong, Tongliang Liu, Jian Yang, and Dacheng Tao. Large-margin label-calibrated support vector machines for positive and unlabeled learning. IEEE Transactions Neural Networks and Learning Systems, 30(11):3471-348 2019a.
Chen Gong, Hong Shi, Tongliang Liu, Chuang Zhang, Jian Yang, and Dacheng Tao. Loss decompo- sition and centroid estimation for positive and unlabeled learning.. IEEE Transactions Pattern Analysis and Machine Intelligence, 43(3):918 9912.9992222 2019b.
Tieliang Gong, Guangtao "ang, Ye, Zongben Xu, and Ming Lin. Margin based PU learning. In AAAI Conference on Artificial Intelligence, pp. 3037-3044. 2018
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup locally linear out-of-manifold regular- ization. In AAAI Conference on Artificial Intelligence, pp. 3714-3722 2019.
Tianyu Guo, Chang Xu, Jiajun Huang, Yunhe Wang, Boxin Shi, Chao Xu. and Dacheng Tao. On positive-unlabele classification in GAN. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 8382-8390. 2020.
10

---

Published as a conference paper at ICLR 2022
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: simple data processing method to improve robustness and uncertainty. In International Conference Learning Representations. 2020.
Ming Hou, Brahim Chaib-Draa, Chao Li, and Qibin Zhao. Generative adversarial positive-unlabeled learning. International Joint Conference on Artificial Intelligence, pp. 2255-2261. 2018.
Cho-Jui Hsieh, Nagarajan Natarajan, and Inderjit Dhillon. learning for matrix completion. In International Conference on Machine Learning, pp. 4445-2453. 2015.
Wenpeng Hu, Ran Le, Bing Liu, Feng Ji, Jinwen Ma. Dongyan Zhao, and Rui Yan. Predictive ad- versarial learning from positive and unlabeled data. In AAAI Conference on Artificial Intelligence, pp. 7806-7814. 2021.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pp. 5275-5285. 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv::412.1 6980, 2014.
Ryuichi Kiryo, Gang Niu, Marthinus Christoffel du Plessis, and Masashi Sugiyama. Positive- unlabeled learning with non-negative risk estimator. In Neural Information Processing Systems, pp. 1675-1685. 2017.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, Uni- versity of Toronto, 2016.
Changchun Li, Ximing Li, and Jihong Ouyang. Learning with noisy partial labels by simultancously leveraging global and local consistencies. In ACM International Conference Information and Knowledge Management, pp. 725-734. 2020a.
Changchun Li. Ximing Li, and Jihong Ouyang. Semi-supervised text classification with balanced deep representation distributions. In Annual Meeting of the Association for Computational Lin- guistics, pp. 5044-5053, 2021a.
Changchun Li, Ximing Li, Jihong Ouyang, and Yiming Wang. Learning with noisy partial labels by simultaneously leveraging global and local consistencies. In ACM International Conference Information and Knowledge Management, pp. 902. 2021b.
Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi-
supervised learning. In International Conference on Learning Representationss 2020b.
Xiaoli Li and Bing Liu. Learning to classify texts using positive and unlabeled data. In International Joint Conference on Artificial Intelligence, pp- 587-594. 2003.
Ximing Li and Yang Wang. Recovering accurate labeling information from partially valid data for effective multi-label learning. In International Joint Conference on Artificial Intelligence, pp- 1373-1380. 2020.
Bing Liu, Wee Sun Lee, Philip S. Yu, and Xiaoli Li. Partially supervised classification of text documents. In International Conference Machine Learning, pp. 387-394. 2002.
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. In Neural Information Processing Systems. 2020.
Chuan Luo, Pu Zhao, Chen Chen, Bo Qiao, Chao Du, Hongyu Zhang, Wei Wu, Shaowei Cai, Bing He, Saravanakumar Rajmohan, and Qingwei Lin. PULNS: positive-unlabeled learning with effective negative sample selector. In AAAI Conference on Artificial Intelligence, pp. 8784-8792. 2021.
Byeonghu Na, Hyemi Kim, Kyungwoo Song, Weonyoung Joo, Yoon- Yeong Kim, and II-Chul Moon. Deep generative positive-unlabeles learning under selection bias. ACM International Conference Information and Knowledge Management, pp. 1155-1164. 2020.
11

---

Published as a conference paper at ICLR 2022
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi-Phuuog--Nung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. SELF: learning to filter noisy labels with self- ensembling. In International Conference on Learning Representations. 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems, pp. 8024-8035. 2019.
Tao Peng, Wanli Zuo, and Fengling He. SVM based adaptive learning method for text classification from positive and unlabeled documents. Knowledge and Information Systems, 16(3):281-300 2008.
Harish G. Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In International Conference on Machine Learning, pp. 2052-2060. 2016.
Yafeng Ren, Donghong Ji, and Hongbin Zhang. Positive unlabeled learning for deceptive reviews detection. In Conference on Empirical Methods in Natural Language Processing, pp- 488-498. 2014.
Hong Shi, Shaojun Pan, Jian Yang, and Chen Gong. Positive and unlabeled learning via loss de- composition and centroid estimation. In International Joint Conference on Artificial Intelligence, pp. 2689-2695. 2018.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-average consis- tency targets improve semi- sunersuper deep learning results. In Neural Information Processing Systems, pp. 1994-1204. 2017.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A. Bilmes, Tanmoy Bhattacharya, and Sarah Micha- lak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks In Neural Information Processing Systems, pp. 13888-13888 13899. 2019.
Jesper E. van Engelen and Holger H. Hoos. A survey on semi-supervised learning. Machine Learn- ing, 109(2):333-44,, 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438-6447. 2019.
Tong Wei, Feng Shi, Hai Wang, Wei-Wei Tu, and Yu-Feng Li. Mixpul: Consistency-basee augmen- tation for positive and unlabeled learning. arXiv preprint (0V:22004009388, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017.
2017.
Yan Yan and Yuhong Guo. Partial label learning with batch label correction. AAAI Conference on Artificial Intelligence, pp. 6575-6582. 2020.
Peng Yang, Xiaoli Li, Jian-Ping Mei, Chee Keong Kwoh, and See-Kiong Ng. Positive-unlabeled learning for disease gene identification. Bioinformatics, 28(20):260022477,211 2012.
Hwanjo Yu, Jiawei Han, and Kevin Chen-Chuan Chang. PEBL: positive example based learning for web page classification using SVM. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 239-248. 2002.
Hwanjo Yu, Jiawei Han, and Kevin Chen-Chuan Chang. PEBL:
web page classification without negative examples. IEEE Transactions on Knowledge and Data Engineering, 16(1):70-81, 2004.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In JEEE/CVF International Conference Computer Vision, pp. 6022-6031. 2019.
12

---

Published as a conference paper at ICLR 2022
Bangzuo Zhang and Wanli Zuo. Reliable negative extracting based on knn for learning from positive and unlabeled examples. Journal of Computers, 4(1):94-101. 2009.
Chuang Zhang, Dexin Ren, Tongliang Liu, Jian Yang, and Chen Gong, Positive and unlabeled learning with label disambiguation. International Joint Conference on Artificial Intelligence, pp. 4250 4256. 2019.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference Learning Representations. 2018.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference Learning Representa- tions. 2021.
Zui-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1): 44-53, 2018.
A DETAILS OF BASELINE METHODS
Eight existing PU learning baselines and the supervised method are employed for comparison in this paper. The details of baseline methods are presented below.
unbiased PU learning (uPU) (du Plessis et al.
2014): ) A cost-sensitive method based on unbiased risk estimation. We use the public code from the net.
non-negative PU learning (nnPU) (Kiryo et al. 2017): A cost-sensitive method based on non-negative risk estimation. We the public code from the net [suggested settings: 3 = -0 and y 1.0]
nnPU+mixup: cost-sensitive method incorporating the typical mixup technique into the nnPU method by mixing positive instances and unlabeled ones separately.
Selr-PU (Chen al. 2020b): A cost-sensitive method with self-supervision scheme. We
use the public code from the net [suggest settings: a = 10.00 B 0.3, = 1/ /6, Pacel 0.2 and Pace2 0.3]
Predictive Adversarial Networks (PAN) (Hu et al.) 2021): A GAN-based PU learning method with a discrininator and a classifier. We use the public code from the net [sug- gested settings: X = 1e 4]
Variational PU learning (VPU) (Chen et al.) 2020a): A PU learning method based on the variational principle. We use the public code from the net [suggested settings: 0.3,
3 E {le- 3. 4, le 3, .. ,1,3]]
MIxPUL (Wei et al. 2020): A PU learning method based on the consistency regularization with the mixup technique. We use the public code from the net [suggested settings: 1.0, 8 1.0, 17 = 1.0]
Dositive-Unlabelee Learning with effective Negative sample Selector (PULNS) (Luo et al. 2021): A sample-selection method with reinforcement learning. We implement an in-house python code with a 3-layer MLP selector suggested by the paper. [suggested settings: 1.0 and 3 (0.4. 0.6, 0.8, 1.0}]
Supervised: The classifiers trained on the fully supervised datasets.
13

---

Published as a conference paper at ICLR 2022
Table 4: Results of classification accuracy (mean=ssd)) on the credit card fraud detection dataset. The highest scores among PU learning methods are indicated in bold.
Metric
Accuracy
Precision
Recall
uPU
97.0+0.2
96.5+3.6
83.4+:.3
nnPU
98.410.1
97.411.1
8..4
nnPU+mixup
98.1=0.1
96.0#3.2
8.
Self-PU
99.2.t0.1
92,4+3.4
85.8+2.0
PAN
99.1110.1
98.5.1.00
1.
VPU
98.6.10.5
99.7110.6
84.915.77
MIXPUL
98.4=0.3
79.2+3.5
86.6+1.3
PULNS
99.0+t0.1
95.6+119
83.2+2.1
P MiX-E
99.010.11
96.511.8
87.71:2.0
P M'Mi-CC
98.8++0.1
94.1+112
86.5+118
B ADDITIONAL EXPERIMENTAL RESULTS ON REALWORLD DATASET
To examine the performance of our proposed pd Mix-E and P P'MiX in practice, we perform the ex- periments the Credit Card Fraud Detection task of Kaggle We utilize subset of the original Credit Card Fraud Detection dataset, which contains all (492) fraudulent instances and 10000 gen- uine ones selected randomly from the original dataset. In this subset, the proportion of the positive instances (frauds) is about 0.0469. We use 20% of the constructed subset as the test dataset, and all others the training one stances. The accuracy, precision and recall are utilized as metrics. Table the average results of independently running 5 times of all comparison methods. Overall, our proposed P'Mix-E and P'Mix-C gain the best recall score, and also achieve the competitive performance the accuracy and precision scores, even the dataset highly unbalanced.
learning, in which 100 frauds are selected randomly as positive
htt.:/aggg....ommoogoob/////ree www.eeeeediaaaaaaaeaaeeaaa tcardfraud
14


Reviews:
[{"review_id": "Mq8MYj0HUsT", "reviewer": "Reviewer_XfpN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ideas of marginal pseudo negative instances and candidate mixup pool are interesting. Since PU learning is different from ordinary supervised learning, it would be reasonable to develop a specific mixup approach for PU learning. \n\n---\n\nIt is not really clear to me that when to use the proposed early-learning regularization. If possible, it is nice to write when to use early-learning regularization (and also, pseudo-negative instance correction) in Algorithm 1 or create Algorithm 2 including the techniques in Section 2.2 (Robustness).\n\nIt is expected that the proposed method would be compared with one of the PU learning methods, e.g., nnPU, with the ordinary mixup technique. Probably, we can respectively augment P and U data by a mixup technique and then use the existing PU learning methods. This approach can be regarded as a simple baseline against the proposed method. It is expected that such a simple baseline would be included in the experiments.\n\nIn Section 3.4, it is reported that \\beta \\in {0.8, 0.85, 0.9, 0.95} is better on the basis of the experiments. But, it is not clear the relation between the class-prior probability and \\beta. When we set \\beta=1/(2\\pi) and the sigmoid loss is used, minimization of Eq. (1) is equivalent to minimization of the objective function of uPU. It is also known that a rough class-prior estimation is sufficient when the true-class prior is known to be large (du Plessis et al., NeurIPS2014). The suggested \\beta might not be generalized to the settings other than the settings of this paper. That is, the suggest \\beta might correspond to the rough estimation of the class-prior, by chance. To support the effectiveness of the suggested \\beta, it would be necessary to show the comprehensive experiments to illustrate the relation between \\beta and the class-prior.\n\nIn Section 2.1, it is not obvious the rule of assigning the generated samples into \\hat{X}\\_p or \\hat{X}\\_u. Since this part is very important in the proposed method, without explicitly-written rules, it would be difficult to reproduce the results from this paper.\n\nIn Figure 1, it seems that \"disambiguation-free\" is labeled as \"discrimination-free.\"", "summary_of_the_paper": "This paper proposes a variant of the mixup technique for positive-unlabeled learning. Based on the observation that the learned PU boundary tends to deviate towards the positive side, the authors suggest selecting samples between the learned PU and supervised boundaries. The proposed P3MIX method and its variant improve the classification performance of PU learning.", "summary_of_the_review": "This paper proposes a mixup method specialized for PU learning. The idea of marginal pseudo-negative instance estimation is interesting. In the current manuscript, there are however several unclear points and it lacks a simple baseline in the experiments. Having such a simple baseline, the advantage of the proposed method will become more clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BkR4TVBXYhF", "reviewer": "Reviewer_fDKn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The motivation of this paper is strong, and the research problem is interesting. The authors found a phenomenon that the decision boundary learned with PU data tends to shift to the positive side compared to the boundary learned with PN data. Because shifting of the decision boundary leads to the bias of learned classifiers, the authors try to reduce the bias by exploiting a heuristic mixup technique.\n+ The proposed method has strong empirical performance. The experimental results on different datasets show that the proposed method can consistently outperform the state-of-the-art PU learning methods.\n+ This paper is well structured and easy to follow. \n\nWeaknesses\n+ It seems that “directional boundary” is not commonly used in existing papers. To avoid confusion, it is better to add a specific definition or change to other words. I assume that it is the same as the “decision boundary”.   \n+ To select the marginal pseudo-negative instance, predictive scores can be different by employing different learning models. It is better to add some discussion that what should pay attention to when choosing the learning model for estimating predictive scores.\n+ In the abstract, “For the unlabelled instances with ambiguous predictive results…”. The word “ambiguous” is not clear. I think that the authors should high-levelly explain the word “ambiguous”.", "summary_of_the_paper": "The authors proposed to reduce the bias of classifiers learned on PU data by a heuristic mixup technique that partially selects the unlabelled instances and mixes them up with the positive instances around the decision boundary learned with PU data. The experimental results demonstrate the effectiveness of the heuristic mixup technique.", "summary_of_the_review": "This paper has a strong motivation as I mentioned above. I like the idea of using a heuristic mixup technique for reducing bias in PU learning. I think the idea that adds heuristic to mixup could be extended to other weakly supervised machine learning tasks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2I_ZzhvLDU", "reviewer": "Reviewer_eqge", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well written and easy-to-follow.\n2. The motivation and the proposed method are clearly described. Especially, the observed phenomenon and the key idea of correcting marginal pseudo-negative examples with a heuristic mixup technique is interesting, and may potentially be useful for semi-supervised learning.\n3. The proposed PU learning approach does not require explicit computation of a class prior.\n4. The experiments are well-conducted, and comprehensively compared to recent SOTA methods. Ablations and sensitivity analysis are also shown. \n\nWeakness:\n1. Does the proposed approach rely on the \"selected completely at random\" (SCAR) assumption? \n2. Showing sensitivity analysis on \\alpha would be better.\n3. Why is the size of the candidate mixup pool fixed as 100? Bigger candidate mixup pool, better performance?\n4. Could the authors release the code?\n", "summary_of_the_paper": "In this paper, the authors focus on the problem of positive and unlabeled learning. They show an interesting phenomenon, where the learned PU boundary tends to deviate the supervised boundary towards the positive side when treating unlabeled examples as pseudo-negative examples. The phenomenon may imply there are a number of marginal pseudo-negative examples that are more likely to be positive but labeled as negative. Based on this, the paper proposes a PU learning approach building on a novel heuristic mixup technique, which can achieve both data augmentation and supervision correction. They also present many empirical results to show the superior performance comparing with SOTA PU learning methods.", "summary_of_the_review": "The problem is interesting and the proposed method is promising. I vote for accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0fB2QHb00o7", "reviewer": "Reviewer_x3p8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea that achieves data augmentation and supervision correction by refining mixup is interesting and easy to implement. This paper is well-organized and well-motivated. Extensive experiments on several datasets and ablation studies prove the effectiveness of the proposed method and its components. Exhaustive discussion of related work is presented. \n\nConcerns:\n1.\tThe method is evaluated on several image classification datasets. It will be better to do experiments on practical PU learning problems, such as product recommendation and medical diagnosis, as mentioned in the paper. The data distributions of these practical problems may be different from image dataset. Thus, the conclusions may be different. \n2.\tMixPUL is a recently proposed PU learning method, which also applies mixup. It may be better to compare with MixPUL and discuss the difference.\n3.\tWhat is the main difference between positive and unlabeled (PU) learning and anomaly detection?\n4.\tDoes the proposed method also work well with Cutmix[1] augmentation?\n\n[1] Cutmix: Regularization strategy to train strong classifiers with localizable features, Yun et al 2019\n", "summary_of_the_paper": "This paper studies an interesting weakly supervised binary classification problem called positive and unlabeled (PU) learning. The authors propose a novel PU learning method inspired by the boundary deviation phenomenon observed in experiments. Specifically, a new mixup method is proposed, which selects the mixup partners for unlabeled examples heuristically to obtain more correct supervised signals. Extensive empirical results, including ablation study and sensitiveness analysis, are provided to evaluate the proposal.", "summary_of_the_review": "The paper proposes a new positive-unlabeled learning method which achieves data augmentation and supervision correction simultaneously by refining mixup strategy. The method is well-motivated by empirical findings. Extensive experiments and ablation study demonstrate the effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Mixup is very helpful when the training sample is scarce or has weak supervision. The paper studies how to adapt mixup to positive and unlabeled (PU) learning, a representative weakly supervised learning problem. By studying the specific properties of PU learning, the authors propose the concept of marginal pseudo-negative instances, which are more likely to be positive but actually annotated by negative. A novel mixup variant has been proposed for PU learning by mixuping the marginal pseudo-negative instances with the positive instance around the classification boundary. The effectiveness has been empirically shown.
Decision: Accept (Poster)