AUC Maximization under Positive Distribution Shift
Authors: Atsutoshi Kumagai
Tomoharu Iwata
Hiroshi Takahashi
Taishi Nishiyama
Yasuhiro Fujiwara
Venue: NeurIPS 2024 poster (2024)
Keywords: AUC maximization
distribution shift
domain adaptation
PU learning
imbalanced data
Primary area: other
TL;DR: In this paper, we propose a method for maximizing the AUC under the positive distribution shift by using labeled positive and unlabeled data in the training distribution and unlabeled data in the test distribution.

Abstract: Maximizing the area under the receiver operating characteristic curve (AUC) is a popular approach to imbalanced binary classification problems. Existing AUC maximization methods usually assume that training and test distributions are identical. However, this assumption is often violated in practice due to {\it a positive distribution shift}, where the negative-conditional density does not change but the positive-conditional density can vary. This shift often occurs in imbalanced classification since positive data are often more diverse and time-varying than negative data. To deal with this shift, we theoretically show that the AUC on the test distribution can be expressed by using the positive and marginal training densities and the marginal test density. Based on this result, we can maximize the AUC on the test distribution by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution. The proposed method requires only positive labels in the training distribution as supervision. Moreover, the derived AUC has a simple form and thus is easy to implement. The effectiveness of the proposed method is shown with four real-world datasets.

AUC Maximization under Positive Distribution Shift
Atsutoshi Kumagai
Tomoharu Iwata
NTT
NTT tomoharu. iwata@ntt.com
atsutoshi. kumagai@ntt.comm @ntt.
Hiroshi Takahashi
Taishi Nishiyama NTT Security Holdings, NTT
NTT
www.hhibm.taaaaahiiiiee... ontt. com
taishi hww..ssssiiiammsscrrtttyyy... ntt
Yasuhiro Fujiwara
NTT
yasuhiro. auguhiro..fiivaaaattt.coo
Abstract
Maximizing the area under the receiver operating characteristic curve (AUC) common approach to imbalanced binary classification problems. Existing AUC maximization methods usually assume that training and test distributions are iden- tical. However, this assumption is often violated in practice due to positive distribution shift, where the negative-conditions density does not change but the positive-conditional density can vary. This shift often occurs in imbalanced classifi- cation since positive data are often more diverse or time-varying than negative data. To deal with this shift, we theoretically show that the AUC on the test distribution can be expressed by using the positive and marginal training densities and the marginal test density. Based on this result, distribution by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution. The proposed method requires only positive labels in the training distribution as supervision. Moreover, the derived AUC has simple form and thus is easy to implement. The effectiveness of the proposed method experimentally shown with six real-world datasets.
maximize the AUC on the test
Introduction
In many real-world binary classification problems such as intrusion detection 37 medical diag- nosis [60]. and visual inspection [42] class- iaaasiimaleacee frequently occurs where positive data is much smaller than negative data [22]. In this case, classification accuracy, which is the standard
performance measure for ordinary binary classification, is not suitable measure [54] 61. Instead,
the area under the receiver operating characteristic curve (AUC) is commonly used 4 19 The AUC the probability that classifier will rank randomly drawn positive instance higher than randomly drawn negative one [61]. Due to the nature of the ranking, the AUC can adequately measure the with imbalanced data. By maximizing the AUC,
performance of the classifier accurate classifiers even from imbalanced data [5]
obtain
32 65
Existing AUC maximization methods usually assume that training and test distributions are identical to ensure the generalization performance. However, real-world applications, this assumption often violated due to distribution shifts. This paper considers positive distribution shift (16), ie., the negative-conditions density does not change but the positive-conditional density can vary, because this shift often occurs in imbalanced problems. For example, in intrusion detection, malicious
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

adversaries rapidly change their attacks (positive data) to bypass detection systems while the benign class's data (negative data) do not change much ( 16 67]. In medical diagnosis, the distribution of disease data (positive data) can change due to the disease progression or emergence of new pathogens, but the distribution of data of healthy people is generally stable. In visual inspection, the types of anomalous products (positive data) are diverse but normal ones (negative data) have some degree of stationarity. When such a distribution shift occurs, the performance of AUC maximization methods drastically deteriorates.
Although labeled data drawn from the test distribution can alleviate this problem, such data are often time- consuming and expensive to collect whenever the dis- tribution shift occurs [4]). In addition, labeled negative data the training distribution are also difficult to collect in some applications. For example, in intrusion detec- tion, although some malicious data can be collected from public sources such as blacklists, benign data are often unavailable due to privacy reasons, and identifying clean benign data from given unlabeled data requires high level of expertise 37
the training distribution
Data the distribution
In this paper, we propose a method for maximizing the AUC under the positive distribution shift by using labeled positive and unlabeled data in the training distribution and unlabeled data the test distribution. Figure | lustrates examples of given data in our problem setting. Since no labeled data are available in the test distribu- tion, a challenge is how to maximize the AUC the test distribution. To address this challenge, we theoret- ically show that the AUC on the test distribution can be expressed by using the positive and marginal training densities and the marginal test density when assuming the positive distribution shift. This result enables us to maximize the AUC the test distribution with positive and unlabeled data in the training distribution and unla- beled data in the test distribution. The derived AUC has a simple form and is easy to implement. In addition, it does not depend on the class-prior the test distribution, which is usually difficult to obtain with unlabeled data in the test distribution. Also, it can be applied to any differentiable classifiers such as linear classifiers, kernel- based classifiers, and neural networks. Hence, our method is easy for practitioners to use.
Labeled positive training distribution
Unisbeled positive data in training distribution
Uniebaled posttive cata in the test distt bution the trainingrtest distribution
Uniabeled
Figure I: Illustration of given data in our problem setting. The triangle, circle, and cross represents negative data, positive data in the training distribution, and pos- itive data in the test distribution, respec- tively. Orange and gray represents labeled and unlabeled data, respectively. Our set- ting assumes that the distribution of neg- ative data does not change (triangles) but that of positive data can vary (circles and crosses).
The main contributions of our work are summarized as follows:
We propose a novel and practical problem setting, where the aim is to maximize the AUC under the positive distribution shift on imbalanced data.
We theoretically show that the AUC the test distribution can be maximized with positive and unlabeled data in the training distribution and unlabeled data in the test distribution when assuming the positive distribution shift.
We empirically demonstrate that the proposed method outperformed various existing meth- ods with six real-world datasets.
2 Related Work
Many AUC maximization methods have been proposed [5] 61 64 32 65 In previous studies [65]
66 12 55]. they have been reported to often perform better than other methods for imbalanced classification such as class balanced loss (), focal loss 31,, or sampling- methods {8] 36]. However, these AUC maximization methods require labeled positive and negative data. In addition, they assume that the training and test distributions are the same. Therefore, they are inappropriate for our problem setting where there no labeled negative data and the distribution shift occurs.

---

Unsupervised domain adaptation aims to adapt to the distribution (domain) shift by using unlabeled
data in the test distribution and labeled data in the training distribution 44 [6]. One representative approach is to learn domain-invariinn
feature representations by minimizing the discrepancy of the
Although this approach is promising, it often deteriorates the performance since it minimizes only the feature discrepancy without considering the relationship between features and labels [70]. Another representative approach is to explicitly minimize the loss on the test distribution by assuming the types of the distribution change, such as 69 35 and class-prior shift [68] [. The proposed method belongs to this covariate shift [50 23
features in both domains 33 5
52 13 27
48
approach. By assuming the shift type, this approach can adapt to the shift in theoretically guaranteed manner. The existing methods aim to minimize the classification risk (or negative classification accuracy), which is an inappropriate metric for imbalanced data (e.g., if the imbalanced data ratio is : 99, a naive classifier that classifies 'every' instance as negative has 99% accuracy, but it is definitely not a good classifier). One domain adaptation method tries to maximize the AUC by learning domain-invariant feature representations [62]. However, this method is not designed for the positive distribution shift. Moreover, all these methods require labeled positive and negative data in the training distribution, which are unavailable in our problem setting.
Positive-unlabeled (PU) learning methods aim to learn classifiers by using only positive and unlabeled data 22 [0] The proposed method is related to the PU learning since it assumes positive and unlabeled data in the training distribution. A representative PU learning method is the empirical minimization- based approach, which rewrites the classification risk by using positive and unlabeled 49 [21]. Although they are effective, they cannot maximize the AUC. Recent studies have shown that the AUC can be rewritten from positive and marginal densities by using the technique
densities 9 25
of the PU learning 45 58 6 (9]. However, these all methods assume that the training and test.
distributions are identical.
Several PU learning methods consider the distribution shift such as covariate shift [46., class-prior shift 7 39,, or positive distribution shift [16]. They require positive and unlabeled data in the training distribution and unlabeled data in the test distribution. However, they consider the classification risk and cannot maximize the AUC. Due to the pairwise formulation of the AUC, their methods that use ordinary instance-wist loss functions such the cross-entropy loss cannot be applied to the AUC. In addition, the method for the positive distribution shift [16] requires the class-prior on the test distribution, which is generally difficult to know with unlabeled data in the test distribution. In contrast, the proposed method does not need to know the class-prior and thus is more practical.
3 Preliminary
We briefly explain the AUC maximization. Let instance E RD and its corresponding label y E {-1,+1} } be equipped with probability density p(x, y). where +1 and -1 means a positive and negative class, respectively. Here, px(x) p(xly +1) and (x(x) p(xly = 11) is the conditional probability density of positive and negative class, respectively. Furthermore, let 8 RR R be score function that outputs the positivity of an input instance. The classifier is defined by the score function with threshold L: y = sign(s(x) - 1), where sign is sign function.
The AUC is the probability of a randomly drawn positive instance being ranked before randomly drawn negative instance [61). Specifically, the AUC with score function s can be formulated as
AUC(s) = (x)) Exe (x) [T(((xP) > s(x")))
(2) is the indicator function that outputs 1 if is true and 0 otherwise, and E is the expectation. Since the gradient of indicator function / is zero everywhere except for the origin, the AUC cannot be optimized via gradient descent methods. To avoid this, the following smoothed AUC often used by replacing the indicator function with a sigmoid function (22 1/ (1 + exp(-2))
20 28 29
(s) Exp=p"(x (xx (xxxx)) s(xx)))
(2)
Given NP positive instances (xf,.... } drawn from px()) and N negative instances {x]..... .... drawn from px(),, the empirical estimate of the smoothed AUC is calculated as
NP N
1
(s) =
(x(((x))) - -s(x",))
(3)
NP
3

---

By maximizing this empirical smoothed AUC with respect to the parameters of s, we can obtain good score functions to maximize the AUC when the training and test distributions are identical [6]]
4 Proposed Method
In this section, we first describe our problem setting (subsection 4.1 that the AUC on the test distribution can be maximized with positive and unlabeled data in the training distribution and unlabeled data in the test distribution under the positive distribution shift (subsection 4.2)
Then, we theoretically show
Then, we discuss class-priors used in the proposed method (subsection 4.3). Lastly, we explain an extension of the proposed method (subsection 4.4).
4.1 Problem Setting
Suppose that we are given set of positive instances X and a set of unlabeled instances Xtr drawn from the training distribution:
X
{x In (= +1),
NP
(4)
Er
X [ M (x) = (x) (1 Tez )pt (x),
(5)
where Ptr (x) is the marginal density of the training distribution, pf. (x) and p (x) := Per (xly - 1) densities of the training distribution, respectively, and Ter == (v +1) is the positive class-prior. Although
are positive and negative-conditin
assume that class-prior Wir is known in this
paper, it can be estimated from positive and unlabeled data
63 15]. In addition, we suppose that a of unlabeled instances Xte drawn from the test distribution is also given:
X = [xta,
Nic | ~ Pto(x) = TepPl + (1 - (x))
(6)
where (x. := (xly = +1), (x.) := (x 1), and Tte == Peely = +1). We assume the class- maassimmallance in both unlabeled data, i.e., Tir. Tee < As shown later, the proposed method does not need to know The to maximize the AUd which is beneficial in practice.
We consider a situation of the positive distribution shift between the training and test distributions, where the negative-condition density does not change but the positive-conditional one can vary,
p (x) = (x), pf. (x) + (x).
This situation often occurs imbalanced classification problems as described in Section 1 Our aim is to learn score function 8 R - IR that can maximize the AUC on the test distribution by using xP U. U.ee For score function s, we can use any differentiable function such as linear classifiers, kerne--bace banel- classifiers, or neural networks. Note that we also discuss the negative distribution shift, i.., phr (x) a phc (x) and (x) = pic (x), in Section D
4.2 Positive Distribution Shift Adaptation
In this subsection, we explain how to maximize the AUC under the positive distribution shift. The objective function to be maximized is the following smoothed AUC on the test distribution.
AUC, (s) (x) Bxr (x)
[f x")],
(8)
where we set s(xP, x²) := (s(PP) ))x")). Since this AUC depends on Plo and phe (x). it seems be impossible calculate in our setting where neither positive negative data in the test distribution are given. However, we show that calculation is possible. First, from the definition of marginal density Pro in Eq. (6). the positive-conditions test density p (x) can be expressed
Phe (x)
[x)e (1 (tie)
(9)
Tto
where we assume that Tee > 0. By substituting Eq. 9 into Eq. we can obtain
1
AsC+
[Exxpla Exe [f(xx (1 Tee E Ex (x) "pp (x) [xx", xx"]]
Tee
(10)
To be precise, the specific value of Twe is not required, but we require that Tie o O, which will be discussed later.

---

Algorithm Training procedure of the proposed method
Require:
Positive and unlabeled data in the training distribution X U Xu.. unlabeled data in the test distribution Xte. positive class-prior in the training distribution Tix mini- size M, and positive mini-batch size P
Ensure: Model parameters of score function 8
1: repeat
2: Sample positive data with size P form tr
3: Sample unlabeled data with size M P from X X
4: Calculate the loss in Eq. (14) on the sampled positive and unlabeled data with Ttr
5: Update model parameters of score function with the gradient of the loss
6: until End condition is satisfied;
Here, the second term in Eq. (10) becomes constant (1 Tto (x)) "ep ((xx"x"")
(1- / because (z))+( (-z) for all z E R [58 59) Therefore, we can ignore the second
term in Eq. (T)) to learn score function 8. Next, by using the assumption of the positive distribution
shift, (x) Pee (x), and the definition of the marginal density Ptr (x) in Eq. (5). we can express the negative-conditiona test density Pie (x) as
pio(x) = pir (x) I Thr
[ptr Teppe (x)]
(11)
where we assume that oer 1. As before, by substituting Eq. (11) into the first term in Eq. 10 we can obtain
A+),(()=
E
(xx, x)] Ter Ex) Exo (x) [0(x, xP]] +C,
teo (1 Ter)
(12)
where C' represents constant terms that do not depend on score function Since coefficient 1/ T()) does not affect the optimization for s, our loss function to be minimized is as follows,
(T
((s) -E Exx-pi Ex.pp. (xx) [x, x)] + Ex Exe (x) [f(xxx))
(13)
This loss depends on the positive and marginal training densities, marginal test densities, and class- prior. Therefore, we can approximate this loss with given data X.. U X,r U X,x and Ter- Specifically,
the empirical estimate of the loss function is given as follows,
Neo.Ner Σ
Ne..ND
((s) No..NT
1
[Xtee, Xtr,m)
Ttr
Σ
(Xxn, X
(14)
N NP
Algorithm shows the pseudocode of training procedure with stochastic gradient methods. Note that although we have used the sigmoid function to represent the AUC in Eq.
(2). as long as we symmetric functions (i.e., function a satisfying a(=) (-=) K for any IR and K is
constant [6]), we can derive the loss function of the same form in Eq. 13 The symmetric functions include wide range of functions such as sigmoid, ramp, and unhinged functions (6]. addition, the
loss corrections for the proposed method such as the non-negative correction [25), which is commonly
used in ordinary PU learning avoid overfitting, are discussed in detail in Section
B
4.3 Discussion about Class-priors
In the process of deriving our loss function, we assume mer < and Teo > We discuss why this assumption is necessary in our setting. When Tue = 0, all unlabeled data X. become negative data. In this case, our given datasets XP U Xtx U do not have any information about positive data in the test distribution since (x) * pl. (x). Therefore, distribution. However, in this case, we can easily create trivial optimal classifiers that classify all data as negative. Thus, this is not an issue.
cannot calculate the AUC the test
When गe = 1, all unlabeled data X become positive data on the training distribution. X. U X (positive data in the training distribution) have no information to extract positive and negative data from X.. since (x) A (x). Thus, in this case, we also cannot calculate the AUC on the test distribution. However, it is extremely rare for all unlabeled data to be positive in imbalanced data problems where the ratio of positive data in unlabeled data is low. Thus, this also not a problem in practice.
5

---

4.4 Extension
In some applications, small number of labeled negative data as well as labeled positive data might be available from the training distribution. The proposed method can be easily extended to such cases. Specifically, by Eq. (TO), the AUC on the test distribution without constant terms is represented as
(s) oc Ex-pii (x) Bx (x) [fx, x")] = Exppii (x) (x) [x(xxx)) =: R(s), (15)
where we used assumption (x) = pie (x). Loss R(s) can be approximated with test unlabeled data and labeled negative data. Thus, the following modified loss function be minimized can be used:
Lmodifted (s) == af(s) - a)R(s),
(16)
where a E [0, 1] is a weighting hyperparameter.
5
Experiments
In this section, we empirically demonstrate the effectiveness of the proposed method under the positive distribution shift with real-world datasets.
5.1
Data
We utilized four widely used real- world datasets in the main paper: MNIST 30]. FashionMNIST [57]. SVHN [40], and CIFARIO 26). MNIST consists of hand-writter images of 10 digits. Each image is represented by gray-scale with 28 x 28 pixels. FashionMNIST consists of images of 10 fashion categories where each image is represented by gray scale with 28 x 28 pixels. SVHN consists of 32 x 32 RGB images of printed 10 digits clipped from photographs of house number plates. We converted SVHN into gray-scale for simplicity. CIFARIO consists of 32 32 RGB images of 10
animal and vehicle categories. In Section E.3 we also evaluated the proposed method with two tabular datasets with distribution shifts (HReadmission and Hypertension) 14]
For MNIST and SVHN, we used even digits as the negative class and odd digits as the positive class. Following the previous studies [16] 18], data with digits '7' and "9" were used as positive data appearing in the training distribution and data with digits '1', '3' '5', '7', and '9' were used as positive data appearing in the test distribution. This simulates situation where new types of positive data, which did not appear during the training, appear in the test environment. For MNSSS, following the study [59], we used upper garments ("T-shift', 'Pullover', "Dress", 'Coat', and 'Shift') as the negative class and the others as the positive class. Positive data with the 'Trouser' and 'Bag' appeared in the training distribution and those with the 'Trouser', 'Bag', , 'Sandal', 'Sneaker', and 'Ankle boot' appeared in the test distribution. For CIFARIO, 10, we used the animal categories as the negative class and the vehicles as the positive class. Positive data with the 'Airplane' appeared in the training distribution and those with the 'Airplane", 'Automobile', 'Ship', and 'Truck' appeared in the test distribution as in the previous study [16].
For each dataset, we used 10 positive and 5, 000 unlabeled data in the training distribution and 5, 000 unlabeled data in the test distribution for training. In addition, unlabeled data in the training distribution and 500 unlabeled data in the test distribution for validation. We used 1, 500 positive and 1, 500 negative data in the test distribution as test data for evaluation. There is no overlap between training, validation, and test datasets. The positive class-prior on the training distribution Ter was set to 0.1 and that the training distribution Tie was changed within {0.1. 0.2, 0.3]. For each case of the positive class-pric pairs, we conducted eight experiments while changing the random seeds and evaluated mean test AUC.
used 5 positive and 500
5.2 Comparison Methods
We compared the proposed method (Ours) with nine neural network-basee comparison methods: CE, nnPU (25) PuAUC [58), PULA (1I. AnnPU, ApuAUC, APULA, CpuAUC, and PURR [16].
CE, NnPU, PuAUC, and PULA use positive and unlabeled data in the training distribution for learning classifiers. These methods do not adapt to the test distribution. CE learns neural network parameters by minimizing the cross-entropy loss. This method naively treats unlabeled data as negative data. nnPU is widely used PU learning method with non-negative risk estimator. This method rewrites

---

the classification risk with positive and unlabeled data and minimizes the rewritten non- -egnattte classification risk learn neural network parameters. PULA is recent PU learning method that learns a neural network classifier with the label distribution alignment. nnPU and PUL.A used the class-prior in the training distribution as in the proposed method. puAUC learns a neural network classifier by maximizing the AUC that is calculated from positive and unlabeled data.
AnnPU, ApuAUC, APULA, CpuAUC, and PURR use positive and unlabeled data in the training distribution and unlabeled data in the test distribution. AnnPU, ApuAUC, and APULA use unlabeled data in both the training and test distributions instead of those in the training distribution of nnPU, puAUC, and PULA respectively. AnnPU and APULA used the composite class-prior on the combined unlabeled data (i.e., π = Nu. tu,.Mtttm.NN. Me).. In our preliminary experiments, the methods that use Nu. unlabeled data from the test distribution instead of data from both distributions were also evaluated. However, they performed worse than AnnPU, ApuAUC, and APULA. Thus, we omitted them. CpuAUC learns the invariant feature representations to mitigate the discrepancy of training and test distributions. Specifically, this method minimizes the negative AUC loss used in puAUC and the CORAL loss, which is a widely used in domain adaptation studies to match the second-order statistics of both distributions [53 52]. PURR is a PU learning method for the positive distribution shift. This method rewrites the classification risk assuming the wegative-condition density does not change. To rewrite the risk, PURR requires the class-prior in the test distribution be available, whereas the proposed method does not.
5.3 Settings
For MNIST, FaShionMNIST, and SVHN, all methods used feed-forward neural network with three hidden layers. The number of hidden nodes was 128 and the ReLU activation function was used for each hidden layer. For CIFAR1O, all methods used a convolutional neural network, which consisted of two convolutional blocks followed by feed-for beed-forward neural network with two hidden layers. The first (second) convolutional block comprised a (16) filter 5 x convolution, the ReLU activation function, and a 2 x 2 max-pooling layer (38] The numbers of nodes in the two hidden layers were 120 and 84, and the ReLUU activation function was used. For PULA and APULA, margin P was selected from 10.1, 1, 10). For CpuAUC, the CORAL loss was applied to the last hidden layer and its regularization weight was selected within {100, 10, 1, 0.1, 0.01). For PULA, APULA, and CpuAUC, the best test results were reported. For all methods, we used the Adam optimizer (24) with a learning rate of 10-4. We set a mini-batch size M to 512, a positive mini-batch size P to 10, and the maximum number of epochs to 200. The loss validation data used for early stopping to avoid overfitting. All methods were implemented using Pytorch [43] and all experiments were conducted on Linux server with an Intel Xeon CPU and A100 GPU.
5.4 Results
We evaluated the performance of the proposed method under the positive distribution shift with imbalanced datasets. Table T shows the average test AUCs of cach method on the four datasets (their standard deviations are reported in Section E.4) The proposed method performed the best or comparably to it in almost all cases (11 out of 12 cases). The non-adaptation methods (CE, nnPU, puAÙC, and PULA) that do not use data in the test distribution performed worse than the proposed method in many cases. This result indicates that adaptation using data in the test distribution is essential when there are distribution shifts. Although AnnPU, ApuAUC, and APULA used unlabeled data in the test distribution, they also did not work well. This is because their loss functions that naively use unlabeled data in the test distribution are not designed to maximize the performance on the test distribution. CpuAUC that learns invariant feature representations to mitigate the distribution gap tended to perform worse than the proposed method since the invariant features are not theoretically guaranteed maximize the performance under the positive distribution shift. PURR performed well with MNIST and FashionMNIST since it is designed to adapt to the positive distribution shift and both datasets are relatively simple data where training is easy even imbalanced data settings. We note that PURR used information on true class-priors on the test distribution, which is not used in the proposed method. By maximizing the AUC, the proposed method outperformed PURR with SVHN and CIFAR 10 by large margin, which are more complex than MNIST and Fashion As for the results of each class-prior pair, the proposed method performed better as the positive class- pris- on the test distribution eee increased. Since unlabeled data X has many positive data
7

---

Table 1: Average test AUCs with different class-prior pairs. Values in bold are not statistically different at the 5% level from the best performing method in each row according to paired t-test. "# best' row represents the number of results of each method that are the best comparable to it.
Data
Tte
Ours
CE
nnPU
PuAUC
PULA
AnnPU
ApuAUC
APULA
CpuAUC
PURR
MNIST
0.1
0.723
0.796
0.782
0,801
0,802
0.782
0.798
0.793
0,804
0,815
0.2
0.854
0.796
0.784
0.801
0.802
0.785
0.793
0.769
0,803
0,871
0.3
0.902
0.796
0.783
0.800
0.802
0.780
0.786
0.745
0.802
0,914
Fashion
0.787
0.932
0.825
0.937
0.954
0.772
0,920
0,939
0,943
0,920
MNIST
0.2
0.891
0.930
0.825
0.937
0.955
0.771
0.890
0.911
0.943
0,929
0.3
0.960
0.930
0.825
0.938
0.955
0.866
0.870
0.863
0.944
0,961
SVHN
0.554
0.504
0.501
0.518
0.494
0.501
0.512
0.492
0.524
0.511
0.2
0.660
0.503
0.501
0.518
0.494
0,501
0.507
0.490
0.523
0.522
0.3
0.736
0.503
0,501
0.518
0.494
0.501
0.507
0,490
0.523
0.519
CIFARIO
0.1
0.727
0.682
0,455
0.749
0.751
0.489
0.739
0.750
0.750
0.434
0.2
0.825
0.679
0,467
0.741
0.739
0.491
0.732
0.742
0.744
0,536
0.3
0.874
0.682
0,451
0.750
0.738
0.545
0.722
0.750
0.749
0.710
# best
11
2
0.9
0.70
8.88
Thes AUC
Teas AUC
3.65
6.7
0.7
0.7
0.60
the aecond
0.02
0.30
1V 0.20
0.35
0.50
0.10
Fl. 0.30
0.15
0.50
7 0.20
0.35
0.03
0.50
0.35
030
(a) MNIST
(b) MaSS
(c) SVHN
(d) CIFARIO
Figure 2: Ablation study of our loss function: average test AUCs and standard emror over different class-priors on the test distribution Tie within {0.1, 10.1,0.2,0 0.3) when changing the class-pria on the training distribution Try Ours w/o the second is Ours without the second term in Eq. (13)
when tee is large, the proposed method more easily extracted information on positive data from X and consequently performed well.
We conducted an ablation study of the proposed method. When positive class-prior on the training distribution Tir is small, all unlabeled data Xtr might be treated as negative data. In this case, existing methods can be directly applied to maximize the AUC the test distribution by using unlabeled data in the test distribution X1 and (pseudo) negative data X 45 approach is equivalent the proposed method without the second term (or mer 0) Eq. (13). denoted as Ours w/o the second. Figure shows the average test AUCs and the standard errors of the proposed method (Ours) and Ours w/o the second when changing the value of class-prior Ti. within (0.01, 0.05, 0. 0.15, 0.2}. Ours consistently outperformed Ours w/o the second across all class- priors Ter except for SVHN. As class-prior कez is increased, the difference between the two methods tended to become larger. This is because the assumption of Ours w/o the second (i.e., unlabeled data X are all negative data) is significantly broken when Ttr is large. With SVHN, the performances of both methods did not differ. This result suggests that it is acceptable to consider all unlabeled data as negative data in some datasets. However, overall, these results show that the naive approximation is generally ineffective and our theoretically grounded loss function is important even with small Wir-
58
6 59 This
We evaluated the proposed method with estimated class- priors on the training distribution since class-prior infor- mation might be unavailable in practice. To estimate the Mlas--prior from positive and unlabeled training data, we used the kernel embedding- based class- class-prior esti- mation method 44). Table 2 shows the results. Here, Ours w/ true and Ours w/ est are the proposed method using true class-prior Tir and estimated class-prior est respectively. We found that true class-prior Ttr = 0.1 was accurately estimated from positive and unlabeled data in the training distribution. As a result, Ours w/
Table 2: Results with estimated class-priors on the training distribution क average test AUCs [%] over different class- class-priors on the test distribution when training class- prior Tt is 0.1. FMNIST is an acronym for MNIST.
Data
Ours w/ true
Ours w/ est
MNIST
0.117
0.826
0.845
FMNIST
0.077
0.879
0.857
SVHN
0.078
0.650
0.652
CIFARIO
0.146
0.809
0.811
8

---

0.003
0.50
0.22
0323
0.823
0.000
"eet AMC.
2.64
Teal AUC.
0.20
0.500
0375
0.78-
0850
0.25
0.00
0.76
05
0.3
0.3
(a) MNIST
(b) MashionMNIST
(c) SVHN
(d) CIFARIO
Figure 3: Results in the case where true and input class-prion on the training distribution मer can be different: average test AUCs and standard errors over different class-priors on the test distribution Tte within 1, 0.2, 0.3} with true training class-prior 0.1 when changing the input class-prior on training distribution.
0.38
0.923
0.725
0.100
0.83
0.82
3 0.90
0.881
0.80
0.79
0.1
0.2
0.4
05
0.1
0.2
0.4
0.
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.4
12-4
(a) MNIST
(b) FashionMNIST
(c) SVHN
(d) CIFARIO
Figure 4: Results in the case where true and input class-priors on the training distribution Ttr can be different when the positive distribution drastically changed: average test AUCs and standard errors with true training class-prior = 0.1 when changing the input class-prior on the training distribution.
est and Ours w/ true mostly performed similarly. This result indicates that the proposed method works well even when class-prior information is unavailable in the training distribution, which is preferable in practice. Additionally, investigated how the performance of the proposed method changes when there is a difference between true and input dlass-priors on the training distribution
Tur- Here, the input class-prior is used in the loss function of Eq. (13) and the true class- priorr used for data generation. Figure S shows the average test AUCs and the standard errors of the proposed method (Ours) when changing the input class-priors. Interestingly, the performance tended to improve when values larger than the true class-prior (0.1) were inputted in most datasets. One reason for this result is that since there some overlap between the types of positive data in the test and training distributions, strengthening the influence on labeled positive data by increasing the input class-pricn in Eq. (13) improved the performance. To confirm this, conducted additional experiments by excluding the types of positive data from the test distribution that were used in the training distribution (e.g., digits '7" and "9" were excluded from the test distribution MNIST). Figure 4 shows the results. As expected, the performance did not improve even with large input class-priors.
We investigated how the performance of the proposed method changes when the number of labeled positive data NP is increased. 3 shows the average test AUCs and the standard errors with different numbers of labeled positive data. As expected, the performance of the proposed method tended improve as NP increased. By using much information labeled positive data in the training distribution, the proposed method can accurately estimate the AUC on the test distribution.
We investigated the performance of the proposed method when changing the number of unlabeled data in the test distribution Table 3 As expected, the performance of the proposed method tended to increase as the number of unlabeled data in the test distribution Nt. increased. The proposed method tended to outperform PuAUC, which does not use unlabeled data in the test distribution, even when No. 500. Since many unlabeled data often easy collect, the proposed method is useful in practice.
We evaluated the modified loss function in Eq. 16 for the cases where positive, negative, and unlabeled data in the training distribution and unlabeled data in the test distribution are available. Table shows the results. Here, the number of labeled negative data was set to 20 and other settings

---

0.88
0.92
0.08
d.t-
oas.
0.50
0.90
d 0.22
0.54
Thet
0.66
0.53
0.82
MNIST
(b) MaSSionMNIST
(c) SVHN
(d) CIFARIO
Figure 5: Average test AUCs and standard errors of the proposed method over different class-priors on the test distribution when changing the number of labeled positive data the training distribution Np.
Table 3: Results of the proposed method with different numbers unlabeled data in the test distribution average test AUCs over different class-prior pairs the in the test distribution within {0.1, 0.2, 0.3} when training class-prior Trr is 0.1. 'Base" represents the result of puAUC that does unlabeled data in the test distribution.
Nic
100
500
1000
2000
5000
Base
MNIST
0.773
0.813
0.823
0.826
0.827
0.801
Fashion MNIST
0.863
0.913
0.918
0.911
0.879
0,938
SVHN
0.516
0.579
0.616
0.626
0.650
0.518
CIFARIO
0.683
0.757
0.770
0.778
0.809
0.746
Table 4: Results when few labeled negative data are available on the training distribution: average
test AUCs for each test class class-prior. FMNIST is an acronym for FashionMNIST.
Data
MNIST
FMNIST
SVHN
CIFAR10
0.1
0.2
0.3
0.1
0.2
0.3
0.1
0.2
0.3
0.2
0.3
a=
1.0
0.723
0.854
0.902
0.787
0.891
0.960
0.334
0.660
0.736
0.727
0.825
0.873
0.909
0.727
0.875
0.904
0.816
0.939
0.947
0.542
0.662
0.743
0.731
0.838
0.870
0.0
0.517
0.577
0.661
0.732
0.794
0.838
0.503
0.506
0,510
0.602
0.594
0.697
are the same as those in the previous experiments. First, the proposed method with = 0, which maximizes the AUC on the test distribution in Eq. (15) with labeled negative and test unlabeled data, did not work well. This would because the number positive data test unlabeled data was small, and thus it was difficult to extract information such positive data from unlabeled data by using a few labeled negative data only. The proposed method with a 0.999, which uses positive, negative, and unlabeled data in the training distribution and unlabeled data in the test distribution, slightly tended to perform better than that with a = 1.0, which equivalent to the proposed method described in Section 4.2 This result suggests that using a few labeled negative data is useful in our framework.
Conclusion
In this paper, we proposed a AUC maximization method under the positive distribution shift. We theoretically showed the AUC on the test distribution can be maximized by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution when assuming the positive distribution shift. The derived AUC has the advantage of being simple and easy to implement. The experiments with six real-world datasets demonstrated that the proposed method outperformed existing methods under the positive distribution shift with imbalanced data.
7 Limitations
The proposed method performs well in the positive distribution shift. As stated in Section I we can expect that the positive distribution shift must occur many real-world applications. However, the proposed method is not guaranteed to keep working well if different distribution shifts occur unexpectedly (i.e., the negative-conditional density changes). deal with this problem, methods should be developed that predict the types of the shift from given data.
10

---

References
[1] Y. Bai, Y.-J. Zhang, P. Zhao, M. Sugiyama, and Z.-H. Zhou. Adapting to online label shift with provable guarantees. NeurlPS, 2022.
[2] J. Bekker and Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109(4):719-760 2020.
[3] S. Bickel, M. Brückner, and T. Scheffer. Discriminative learning under covariate shift. Journal of Machine Learning Rexearch, 10(9), 2009.
[4] A. P. Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1 1159, 1997.
[5] U. Brefeld, T. Scheffer, al. Auc maximizing support vector learning. In Proceedings of the ICML 2005 workshop on ROC Analysis in Machine Learning, 2005.
[6] N. Charoenphakdee, J. Lee, and M. Sugiyama. On symmetric losses for learning from corrupted labels. In ICML., 2019.
[7] N. Charoenphakdee and M. Sugiyama. Positive-umlabeled classification under class prior shift and asymmetric error. In SDM, 2019.
[8] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[9] M. Du Plessis, G. Niu, and M. Sugiyama. Convex formulation for learning from positive and unlabeled data. ICML, 2015.
[10] C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In SIGKDD, 2008.
[11] B. Fernando, A. Habrard, M. Sebban, and T. Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In ICCV, 2013.
[12] A. Fujino and N. Ueda. A semi-supervised auc optimization method with generative models. In ICDM, 2016.
[13] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In JCML, 2015.
[14] Gardner, Z. Popovic, and L. Schmidt. Benchmarking distribution shift in tabular data with tableshift. NeurlPS, 2023.
[15] S. Garg, Y. Wu, A. J. Smola, S. Balakrishnan, and Z. Lipton. Mixture proportion estimation and pu learning: A modern approach. NeurlPS, 2021.
[16] Z. Hammoudeh and D. Lowd. Learning from positive and unlabeled data with arbitrary positive shift. NeurlPS, 2020.
[17] H. He and A. Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 1269):123-11284, 2009.
[18] Y.G. Hsieh, G. Niu, and M. Sugiyama. Classification from positive, unlabeled and biased negative data. ICML, 2019.
[19] J. Huang and C. X. Ling. Using auc and accuracy in evaluating learning algorithms. JEEE Transactions knowledge and Data Engineering, 17(3):299-311 2005.
[20] Iwata and Y. Yamanaka. Supervised anomaly detection based on deep autoregressive density estimators. arXiv preprint arXiv:1904.06034 2019.
[21] Y. Jiang, Q. Xu, Y. Zhao, Z. Yang, P. Wen, X. Cao, and Q. Huang. Positive-unlabeled learn- ing with label distribution alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

---

[22] J. M. Johnson and T. M. Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data. 6(1):1-54, 2019.
[23] T. Kanamori, S. Hido, and M. Sugiyama. least-squares approach direct importance estimation. The Journal of Machine Learning Research, 10:1391-1445 2009.
[24] D. P. Kingma and J. Ba. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] R. Kiryo, G. Niu, M. C. Du Plessis, and M. Sugiyama. Positive-unlabeled learning with non-negative risk estimator. NeurIPS, 2017.
[26] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] A. Kumagai and T. Iwata. Unsupervised domain adaptation by matching distributions based on the maximum mean discrepancy via unilateral transformations. In AAAI, 2019.
[28] A. Kumagai, T. Iwata, and Y. Fujiwara. Transfer anomaly detection by inferring latent domain representations. NeurlPS, 2019.
[29] A. Kumagai, T. Iwata, H. Takahashi, and Y. Fujiwara. Meta-learning for robust anomaly detection. In AISTATS, 2023.
[30] Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://ann..ccu..om//ee/////a///
2, 2010.
[31] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In ICCV, 2017.
[32] M. Liu, Z. Yuan, Y. Ying, and T. Yang. Stochastic auc maximization with deep neural networks. In ICLR, 2020.
[33] M. Long, Y. C. J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. ICML, 2015.
[34] N. Lu, T. Zhang, G. Niu, and M. Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: consistent risk correction approach. In AISTATS, 2020.
[35] J. I. S. Martin, S. Mazuelas, and A. Liu. Double-weighting for covariate shift adaptation. In ICMI., 2023.
[36] G. Menardi and N. Torelli. Training and assessing classification rules with imbalanced data. Data mining and knowledge discovery, 18:92-122, 2014.
[37] Y. Mirsky, T. Doitshman, Y. Elovici, and A. Shabtai. Kitsune: an ensemble of autoencoders for online network intrusion detection. In NDSS, 2018.
[38] Nagi, F. Ducatelle, G. A. Di Caro, D. Cireşan, U. Meier, A. Giusti, F. Nagi, J. Schmidhuber, and I.. M. Gambardella, Max-pooling convolutional neural networks for vision-based hand gesture recognition. In 201/ / IEFE international conference signal and image processing applications (ICSIPA), pages 342-347. IEEE, 2011.
[39] S. Nakajima and M. Sugiyama. Positive-umlabele classification under class-prior shift: a prior-invariant approach based density ratio estimation. Machine Learning, 112(3):889 9999 2023.
[40] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 7. Granada, Spain, 2011.
[41] S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 12450):33551135, 2009.
[42] J.-K. Park, B.-K. Kwon, J.-H. Park, and D.-J. Kang. Machine learning-based imaging system for surface defect inspection. International Journal of Precision Engineering and Manufacturing- Green Technology, 3:303-310, 2016.
12

---

[43] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison,
L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
[44] H. Ramaswamy, C. Scott, and A. Tewari. Mixture proportion estimation via kernel embeddings of distributions. In ICML, 2016.
[45] T. Sakai, G. Niu, and M. Sugiyama. Semi-supervised auc optimization based on positive- unlabeled learning. Machine Learning, 107:767-794, 2018.
[46] T. Sakai and N. Shimizu. Covariate shift adaptation on learning from positive and unlabeled data. In AAAI, 2019.
[47] I. Sharafaldin, A. Gharib, A. H. Lashkari, and A. A. Ghorbani. Towards reliable intrusion detection benchmark dataset. Software Networking, 2018()):177-000, 2018.
[48] J. Shen, Y. Qu, W. Zhang. and Y. Yu. Wasserstein distance guided representation learning for domain adaptation. In AAAI, 2018.
[49] M. Sugiyama, H. Bao, T. Ishida, N. Lu, and T. Sakai. Machine learning from weak supervision: an empirical risk minimization approach. MIT Press, 2022.
[50] M. Sugiyama and M. Kawanabe. Machine learning in non- stationary environments: Introduc- tion to covariate shift adaptation. MIT press, 2012.
[51] M. Sugiyama, M. Yamada, and M. C. du Plessis. Learning under nonstationarity: covariate shift and Llassbblaace change. Wiley Interdisciplinary Reviews: Computational Statistics, 5(6):465 477, 2013.
[52] B. Sun, Feng, and K. Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016.
[53] B. Sun, J. Feng, and K. Saenko. Correlation alignment for unsupervised domain adaptation. Domain adaptation in computer vision applications, pages 153-171, 2017.
[54] N. Ueda and Fujino. Partial auc maximization via nonlinear scoring functions. arXiv preprint arXiv: 1806.04838, 2018.
[55] G. Wang, S. W. H. Kwok, M. Yousufuddin, and F. Sohel. A novel auc maximization imbalanced learning approach for predicting composite outcomes in covid-19 19 hospitalized patients. IEEE journal of biomedical and health informatics, 2023.
[56] G. Wilson and D. Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-6,, 2020.
[57] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv:1708.0774) 2017.
[58] Z. Xie and M. Li. Semi-supervised auc optimization without guessing labels of unlabeled data. In AAAI, 2018
[59] Z. Xie, Y. Liu, H.-Y. He, M. Li, and Z. Zhou. Weakly supervised auc optimization: a unified partial auc approach. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[60] C. Yang, M. Zhao, C. Zhu, S. Xie, and Y. Chen, Automated detection of breast cancer metastases. In DShSS, 2021.
[61] Yang and Y. Ying. Aue maximization in the era of big data and ai: A survey. ACM computing surveys, 55(8):1-37, 2022.
[62] 7. Yang, Q. Xu, S. Bao, P. Wen, Y. He, X. Cao, and Q. Huang. Auc-oriented domain adaptation: From theory to algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[63] Y. Yao, T. Liu, B. Han, M. Gong, G. Niu, M. Sugiyama, and D. Tao. Rethinking class-prior estimation for positive-unlabeled learning. In ICLR, 2021.
13

---

[64] Y. Ying, L. Wen, and S. Lyu. Stochastic online auc maximization. NeurIPS, 2016.
[65] Z. Yuan, Z. Guo, N. Chawla, and T. Yang. Compositional training for end-to-end deep auc maximization. ICLR, 2021.
[66] Z. Yuan, Y. Yan, M. Sonka, and Yang. Large-scale robust deep auc maximization: new surrogate loss and empirical studies medical image classification. In ICCV, 2021.
[67] J. Zhang, M. F. Khan, X. Lin, and Z. Qin. An optimized positive-unlabeled learning method for detecting a large scale of malware variants. In DSC, 2019.
[68] K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang. Domain adaptation under target and conditional shift. ICML, 2013.
[69] Zhang, I. Yamane, N. Lu, and M. Sugiyama. A one-step approach to covariate shift adaptation. In ACML, 2020.
[70] H. Zhao, R. T. Des Combes, K. Zhang, and G. Gordon. On learning invariant representations for domain adaptation. In ICML, 2019.
14

---

A Impact Statements
Although the proposed method performed well, there a possibility of misclassification in practice. In particular, the misclassification can lead to serious incidents in cases such as cyber/physical security and medical care, which are typical examples of imbalanced data. Therefore, this method should be used as support tool for humans to make final decision.
B Proposed Method with Loss Corrections
In this section, we discuss the loss (risk) corrections commonly used in ordinary PU learning to mitigate overfitting (25 [6]. Specifically, the loss correction is used to prevent empirical estimates of the risk from taking negative values even when the risk never takes negative, which often leads to overfitting.
First, we provide summary of our results: (1) The non negative loss correction, which is widely used for ordinary PU learning [25 16]. is not effective in our AUC maximization framework. This is because zero is not tight lower bound of expected loss. (2) When class-prior in the test distribution tte is available, we can derive a tighter lower bound. By using this for the loss correction, we can often enhance the performance of the proposed method. Below, we describe the details.
Non-negative loss correction. In this section, for clarity, we consider minimizing the AUC risk,
(s) := Exp-pp. (xx Exp (fx"," xP)].
(17)
which is equivalent to maximize the AUC in Eq. (8) since AUC, (s) = - (s) The minimum
value of this risk is zero. Then, the corresponding loss for Eq. (13) becomes
(s) == Exxpia (x) Exrxp. [/ x)] Ter Ter (x)
x)].
(18)
This loss is derived from the AUC risk with unlabeled and negative test densities, i.e.,
(s) (1 Tu (x) pptt (f [f("x, x)]]
(19)
Since the AUC risk does not take negative values, the loss should not also take negative values. Therefore, to prevent negative values of its empirical estimate (s), we can use the empirical loss with the absolute value function [s)] for the optimization. This type of correction is successfully used in PU learning [16] other weakly supervised learning [34]. Table 5 shows the results of the proposed method with Lrisk [s)l (Ours w/ nn). The results of Ours w/ nn and the proposed method without the loss correction (Ours) were almost identical and thus the non-negative loss correction was not effective in our framework unlike ordinary PU learning studies 25
The reason of the ineffectiveness of Ours nn is that the non-negative constraint is insufficient/weak in our loss (Eq. (8)). In fact, since pie (x) is contained in Pre (x) Teelle (x) + (1 - *te )pe (x) ), the minimum value the expected loss (riek(s in Eq. (99 actually be greater than zero. Thus, the empirical loss Lrisk(() (s) could not be sufficiently constrained with the non-negativity; the performance of Ours w/ nn did not improve.
Note that the non-negative correction effective in ordinary PU learning studies (25 16] This because the minimum value of the risk R, (g) := Exxp"" "x- (x) [[(g(x), - -1]] used in the study 25 is zero, where f is a posnt-wise loss and is a decision function. This creates a gap in the effectiveness of the non-negative correction in framework and the previous study 25
Loss correction with class-prior in the test distribution Tie- class-prior Teo is known, we can derive tighter lower bound of our loss Crisk (s). Specifically, we can obtain
(x", x)] (1-Ter) Ter (1 mee)
(risk(() = (1 Mer Fx Ex E
b 0.
(20)
Here, we used Pto (xx (x) TeePl (x) + (1 Tea Tea) Dt (x) and the fact that the AUC risk between the same densities is 1/2 as described in Section 4.2 As result, we can use ( b) + for the optimization. Our method with this correction (Our w/ b) tended to enhance the performance of it without the correction (Ours) in T However, note that Ours has the strong advantage of not requiring the class-prior and performed better than existing methods.
15

---

Table 5: Comparison with the proposed methods that use the loss corrections. Ours w/ nn used the non non-negative loss correction and Ours w/ b uses b = (1 - Tic)(1 T2r for the correction: average test AUCs over different class-prior pairs The the test distribution within (.21,0.200.3 when training class-prior Ter is 0.1.
Data
Ours
Ours w/ nn
Ours w/ b
MNIST
0.827
0.816
0.885
Fashion MNIST
0,879
0,879
0,954
SVHN
0.650
0.650
0.572
CIFARIO
0.809
0.792
0.829
Dynamics of training loss, validation loss, and test AUC. Lastly, we compared the dynamics of training loss, validation loss, and test AUC of these methods (Ours, Ours w/ nn, and Ours w/ b) in Figure 6 Here, the validation loss was calculated with validation PU data (i.e., PU data in the training distribution and U data in the test distribution). As learning progressed, the training losses of Ours and Ours w/ nn became smaller than the lower bound AUCs tended to stop improving or to become worse after the training losses were below b = 0.315. However, since the validation loss and test AUC were well correlated, Ours and Ours w/nn could select good models by using early-stopping with the validation loss. Ours w/ b tended to maintain good test AUCs without overfitting even when the learning is processed by evading the training loss to take smaller values than b = 0.315.
0.315; the validation losses and test
C How to Determine the Classification Threshold
The proposed method allows us to sort data in score order. In practical use, this is beneficial many situations. For example, in anomaly detection, experts operators can check data with high scores within the cost they can spend or until anomalous data do not appear. In disease diagnosis, patients with higher scores can be prioritized for detailed examination. In recommendation systems, products can be presented to users in order of score. However, a classification threshold may be required to classify positive and negative data in some situations. Below, explain how to determine the classification threshold.
First, we have Nue unlabeled data in the test distribution. When the true positive class-prior in the test distribution is Tie we can regard that Noe positive data are included in the No unlabeled data. Thus, when the Nte unlabelled data are sorted by score, the top Noe instances can be considered positive (assuming this scoring is accurate), and the score of the To instance can be the boundary separating positive and negative. Thus, we can use the score of the toe instance as the threshold. Since true prior Tte unknown, we want to estimate it and use the estimated prior The instead.
Next, we explain the procedure obtain the estimated class-prior in the test distribution mic Specifically, we first extract negative data in unlabeled data from the training distribution by applying (off-theeshell PU learning to PU data in the training distribution. Since PU data and the flass-prior in the training distribution are available in our setting, we can perform it. Then, since the negative distribution does not change our setting, the extracted negative data can regarded as negative data in the test distribution. We can estimate the class-prior in the test distribution by applying to the extracted negative and unlabeled data in
existing class-prior estimation methods the test distribution.
63 15
D Discussion about Negative Distribution Shift
In the main paper, we considered the positive distribution shift. Here, we consider negative distribution shift, i.e., (x) p (x) but (x)() * (x). In this case, when positive data in the training distribution and unlabeled data in the test distribution are available, we show that the
negative distribution shift can be addressed by using existing AUC maximization methods [45 58 59
16

---

(a) MNIST
(b) Fashion MNIST
105
175
200
25
(c) SVHN
(d) CIFARIO
Figure 6: Dynamics of training loss, validation loss, and test AUC when (Tr.. Tte) = (0.1,0.3). Each column represents the results of Ours, Ours w/ nn, and Ours w/ b, respectively from left to right. The value = (1=)( xte)/2 / 0.315 for all datasets.
Specifically, in this case, the AUC in the test distribution is
AUC, (s) = Bx) [f(xxxx")
Tee (x) [f(xP, xx)] + C,
(21)
where C' is constant, and in the second equal sign, used that (x) -- |pto TeoPP (x)).
pP (x) o (x), and fact that AUC between the same densities a constant as described in
Section 2 This derived AUC is equivalent to that in the existing studies [45] 58 59 and can be
maximized with test unlabeled and training positive data. Note that we do not require to know Tte since it does not affect the optimization.
17

---

Table 6: Comparison with BPURR: average test AUCs for each test class-prion Values in bold not statistically different at the 5% level from the best performing method in each row according to a paired t-test. FMNIST is acronym for Fashi
Data
MNIST
FMNIST
SVHN
CIFARIO
The
0.1
0.3
0.1
0.2
0.3
0.1
0.3
0.1
0.2
0.3
Ours
0.723
0.854
0.902
0.787
0.891
0.960
0.554
0.660
0.736
0.727
0.825
0.873
BPURR
0.863
0.904
0.907
0.939
0.929
0.917
0.495
0.516
0.532
0.745
0.786
0.790
E
Additional Experimental Results
E.1
Comparison with PURR using Class Balanced Classification Loss
In the comparison methods, PURR is the PU learning method under the positive distribution shift to minimize the test classification risk [16]. Since the classification risk is not suitable for imbalanced classification, created (the balanced class rate) 6 instead of the classification risk, called BPURR. Even in this case, BPURR requires the class-prior the test distribution, which is usually difficult to know and is not required in the proposed method. Table(ō]shows the mean test AUC with each class-prios on the test distribution. BPURR used the class-prior on the training distribution as that on the test distribution. When The = 0.1, BPURR worked well since the used class-prior was equal to the true value. However, in other cases, BPURR performed worse than the proposed method in many cases.
method of PURR that uses the class balanced classification loss
E.2 Results with Small Class-priors on the Training Distribution
In the main paper, the positive class-prior the training distribution Ter was set to 0.1. Here, investigated how the performance of the proposed method changes when Ttr is smaller. Table shows the average test AUCs of each method on the four datasets. The proposed method performed best or comparably to almost all cases (32 out 36). As the positive class-prior on the training distribution Ttz decreased, the proposed method tended to perform well. Since the amount of noise (positive data in the training distribution) in Xt decreases as Tu decreases, negative data (which is essential information maximize the AUC on the test distribution) might be casy to extract from Xtr when Tir is small.
E.3 Results with Tabular Datasets
We evaluate the proposed method with two tabular datasets with distribution shifts (HReadmission and Hypertension) (14] In HReadmission, the task is to predict the 30-day readmission of diabetic hospital patients. Each patient is represented by -33-dimensional feature vector. In Hypertension. the task is a hypertension diagnosis for high-risk age. Each survey subject is represented by 100-dimensional feature vector. We constructed the positive distribution shift using positive-shifted data (positive ood data) and non-shifted data (positive and negative training data). The experimental setting, such as the number data, is the same as that the main paper. Table&sshows the result. The proposed method outperformed the others. This result shows that the proposed method is effective for tabular datasets.
E.4 Results with Standard Deviations
In the main paper, omitted the standard deviations of results due to the limited space. Table 9 shows the mean test AUCs with the standard deviations.
E.5 F1 Score and G-mean of TPR and TNR
Although this paper focuses on maximizing the AUC since it a representative evaluation metric for imbalanced data, other metrics for imbalanced data exist, such as the Fl score and the G-mean of TPR and TNR [7]. Therefore, we investigated whether the proposed method also effective for the F1 score and the G-mean of TPR and TNR. Unlike the AUC, the F1 score and the G-mean of TPR and TNR require determine the classification threshold. In this experiment, when N test data, in which the ratio of positive data is Tte, sorted by score of each method, regard the top N Note
18

---

Table 7: Average test AUCs with different class-prior pairs. Values in bold are statistically different at the 5% level from the best performing method in each row according to paired t-test. "# best' row represents the number of results of each method that are the best comparable to it.
Data MNIST
T1e
Ours
CE
nnPL
PoAUC
PULA
AnnPU
ApuAUC
APULA
CpuAUC
PURR
0.01
0.871
0.813
0.814
0.811
0.786
0.805
0.803
0.818
0.895
0.01
0.2
0.923
0.813
0.724
0.814
0.811
0.786
0.798
0.783
0.817
0.948
0.01
0.3
0.953
0.813
0.724
0.814
0.811
0.788
0.792
0.757
0.817
0.957
0.05
0.741
0.806
0.768
0.807
0,808
0.782
0.802
0.800
0.811
0.871
0.05
0.2
0,869
0.806
0.772
0,807
0,808
0.794
0.796
0.775
0.811
0.921
0.05
0.3
0.936
0.806
0.767
0.807
0.808
0.789
0.790
0.755
0.811
0.943
0.1
0.723
0.796
0.782
0.801
0.802
0.782
0.798
0.793
0.804
0.815
0.1
0.2
0.854
0.796
0.784
0.801
0,802
0.785
0.793
0.769
0,803
0.871
0.902
0.796
0.783
0,800
0,802
0.780
0.786
0.745
0,802
0,914
Fushion
0.01
0.969
0,955
0,776
0,949
0.917
0.723
0.501
0,947
0,951
0.934
MNIST
0.01
0.2
0.978
0.955
0.777
0.949
0.917
0.745
0.872
0.926
0.954
0.936
0.01
0.985
0,955
0.777
0.949
0,916
0.785
0.845
0,903
0,955
0,955
0.05
0.925
0.935
0.796
0,950
0.973
0.740
0.902
0.947
0,950
0.928
0.05
0.2
0.968
0.942
0.797
0.948
0.973
0.715
0.872
0.923
0.948
0.944
0.05
0.982
0.933
0.796
0,949
0.973
0.792
0.843
0.904
0,949
0.962
0.1
0.787
0.932
0.825
0.937
0.954
0.772
0.920
0.939
0.943
0.920
0.1
0.2
0.891
0.930
0.825
0.937
0.955
0.771
0.890
0,911
0.943
0.929
0.955
0.863
0.944
SVHN
0.01
0.586
0.503
0.503
0,522
0,499
0.503
0,521
0.500
0.538
0.501
0.01
0.2
0.752
0.503
0.503
0.521
0.499
0.503
0.520
0.500
0,533
0.501
0.503
11499
0,05
0.1
0.587
0.503
0.501
0.513
0.498
0.501
0.514
0.498
0.510
0.501
0.05
0.2
0.715
0.505
0.501
0.513
0.498
0.501
0.513
0.495
0,517
0,501
0.05
0.779
0.505
0.513
0.498
0.501
0.512
0.493
0,517
0.501
0.1
0.1
0.554
0.504
0,501
0,518
0.494
0.501
0.512
0.492
0.524
0.511
0.1
0.2
0.660
0.503
0.501
0.518
11494
0.501
0.507
0.490
0.523
0.522
0.736
0.503
0.518
0.494
0.507
0,523
CIFARIO
0.01
0.1
0,805
0.738
0.414
0.748
0.765
0.454
0.746
0.724
0.738
0.468
0.01
0.2
0.883
0.743
0.414
0.749
0.765
0.534
0.719
0.748
0.757
0.593
0.01
0.741
0.765
0.721
0.733
0.914
0.05
0.1
0.740
0.736
0.419
0.745
0.724
0.478
0.737
0.729
0.751
0.489
0.05
0.2
0.861
0.737
0.419
0.745
0.724
0.520
0.723
0,731
0,750
0.518
0.05
0.894
0.740
0.745
0.724
0.536
0.712
0.736
0.750
0.845
0.1
0.1
0.727
0.682
0.455
0.749
0.751
0.489
0.739
0.950
0.750
0.434
0.825
0.679
0.467
0.741
0.739
0.491
0.732
0.744
0,744
0.536
0.3
0.874
0.682
0.451
0.750
0.738
0.545
0.722
0.749
0.749
0.710
best
32
4
10
5
14
Table 8: Results with tabular data: average test AUCs over different class-prior pairs ate in the test. distribution within (..1,0.2,0 0.3) when training class-prior Ter is 0.1. Boldface denotes the best and
comparable methods according the paired t-test and the significance level of 5%.
Data
Ours
CE
anPU
puAUC
PULA
AnnPU
ApuAUC
APULA
CpuACC
PURR
HReadmission
0.747
0.529
0.562
0.502
0,519
0.531
0.492
0.495
0,511
0.695
Hypertension
0.630
0.596
0.607
0.571
0.571
0.565
0.608
0.539
0.571
0.643
data as positive and the remaining as negative. Tables and and G-means of TPR and TNR with different test and training class-prior pairs, respectively. The proposed method tended to outperform the other methods although it maximizes the AUC. This result may imply that AUC maximization can also help improve other evaluation metries for imbalanced data.
and 11 show the average test F1 scores
19

---

Table 9: Average test AUCs and standard deviations with different class prior pairs.
Data
TTe
Ours
CE
nnPU
puAUC
PULA
MNIST
0.1
0.723(0.094)
0.782(0.034) 0.796)
0.801(0.011)
0.802(0.020)
0.2
0.854(0.062)
0.796(0.014)
0.784(0.035)
0.801000010)
0.802(0.020)
0.3
0.902(0.034)
0.796(0.013)
0.783(0.035)
0.800(0.011)
0.802(0.020)
Fashion
0.1
0.787(0.091)
(.932(0.020)
0.825(0.075)
0.937(0.033))
0.954(0.030)
MNIST
0.2
0.891(0.071)
0.930(0.2.1)
0.825(0.075)
0.937(0.038)
0.955(0.030)
0.3
0.960(0.037)
0.930(0.020)
0.938(0.038) 0.825(0.075)
0.955(0.030)
SVHN
0.1
0.554(0.045)
0.501(0.009) 0.504(0.015)
0.518(0.027)
0.494(0.013)
0.2
0.660(0.047)
0.503(0.016)
0.501().009)
0.518(0.027)
0.494(0.013)
0.3
0.736(0.013)
(.503().015)
(.501(0.009)
0.518(0.027)
0.494(0.013)
CIFARTO
0.1
0.727(0.070)
0.682(0.090)
0.455(0.098)
(.749(..050)
0.751(0.022)
0.2
0.825(0.048)
0.679(0.10)) 100)
0.741(0.055) 0.467(0.121)
0.739(0.047)
0.3
0.874(0.022)
0.682(0.090)
0.750(0.050) 0.451(0.082)
0.738(0.048)
Data
Tee
APULA
AnnPU
ApuAUC
CpuAUC
PURR
MNIST
0.798(0.009) 0.782(0.035) 0.793(0.021)
0.804(0.011)
0.815(0.037)
0.2
0. 0.769(0.017)
0.785(0.017)
0.793(0.009)
0.803(0.011)
0.871(0.014)
0.3
0.745(0.017)
0.780(0.017)
0.786(0.010)
0.802(0.010)
0.914(0.014)
Fashion
0.1
0.939(0.066)
0.772(0.073)
0.920(0.046)
0.943(0.025)
0.920(0.010)
MNIST
0.2
0.911(0.082)
0.771(0.104
0.899().059)
0.943(0.026)
0.929(0.01))
0.3
0.863(0.098)
0.866(0. 105)
0.870(0.049)
0.944(0.026)
0.961(0.225)
SVHN
0.1
0.492(0.013)
0.501(0.009)
0.512(0.022)
0.524(0.02)
0.511(0.009)
0.2
0,490(0.012)
0.501(0.009)
0.507(0.015)
0.523(0.026)
0.522(0.009)
0.3
0.490(0.008)
(.501(0.009)
0.507(0.017)
0.523(0.025)
0.519(0.008)
CIFARTO
0.1
0.489(0. 0.750(0.029)
0.750(0.05T) 0.735(
(.434))
0.2
0. 0.742(0.038)
0.491(0.139)
0.732(0.056)
0.744(0.051)
0.536(0.162)
0.3
0.750(0.044)
0.545(0.144)
0.722(0.060)
0.749(0.049)
0.710(0.197)
Table 10: Average test Fl scores with different class-prior pairs. We set training class-prior Tee to 0.1. Values in bold are not statistically different at the 5% level from the best performing method in each row according the best or comparable to it.
paired I-test. '# best' row represents the number of results of each method that are
Data
Ours
CE
nnPU
PuAUC
PULA
AnnPU
ApuAUC
APULA
CpuAUC
PURR
MNIST
0.1
0.395
0.445
0.432
0.443
0.425
0.418
0.437
0,383
0.445
0.469
0.2
0.637
0.548
0.534
0.551
0.541
0.504
0.538
0.454
0.553
0.645
0.3
0.750
0.618
0.603
0.621
0.621
0.581
0.608
0.525
0.629
0.782
Fashion
0.1
0.619
0.702
0.445
0.727
0.778
0.342
0.683
0.738
0.746
0,746
MNIST
0.2
0.763
0.755
0.545
0.786
0.844
0.466
0.667
0.722
0.794
0.792
0.3
0.883
0.793
0.631
0.821
0.874
0.718
0.687
0.697
0.830
0.883
SVIIN
0.1
0.163
0.110
0.111
0.112
0.101
0.111
0.121
0.097
0.129
0.116
0.2
0.368
0.207
0.201
0.217
0.195
0.206
0.211
0.193
0.227
0.208
0.3
0.542
0,310
0.310
0.323
0.293
0.309
0.314
0,287
0.324
0.321
CIFARIO
0.1
0.337
0.299
0.090
0.381
0.371
0.120
0.367
0.370
0.381
0.070
0.2
0.574
0.401
0.181
0.493
0.491
0,201
0.482
0.484
0.498
0,260
0.3
0.716
0.489
0.235
0.583
0.570
0.348
0.551
0.566
0.583
0.531
best
3
2
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claim described in the abstract and introduction accurately reflects our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
20

---

Table 11: Average test G-mean of TPR and TNR with different class- masss-proor pairs. We set training class-prion trr to 0.1. Values in bold are not statistically different at the 5% level from the best
performing method in each row according to paired t-test. *# best' row represents the number of
results of each method that are the best or comparable to it.
Data
R1e
Ours
CE
nnPU
puAUC
PULA
AnnPU
ApuACC
APULA
CpuAUC
PURR
MNIST
0.1
0.604
0,646
0.634
0.644
0.629
0.624
0.640
0.596
0.646
0.663
0.2
0.760
0.697
0.683
0.699
0.692
0.664
0.690
0,626
0.700
0.767
0.3
0.818
0.719
0.707
0.721
0.721
0.690
0.711
0.647
0.721
0.842
FMNIST
0.1
0.767
0.823
0.636
0.837
0.868
0.555
0.809
0.832
0.850
0.746
0.2
0.846
0.842
0.690
0.861
0.899
0.628
0,780
0.812
0.866
0.866
0.3
0.915
0.850
0.727
0.870
0.909
0.792
0.771
0.776
0.877
0.916
SVHN
0.1
0.382
0.313
0.315
0.328
0.300
0.315
0.330
0.294
0.340
0.322
0.2
0.555
0,407
0,406
0.417
0.394
0,406
0,411
0.392
0.427
0,408
0.3
0,660
0,468
0.467
0.479
0.451
0.466
0.470
0,446
0.480
0.477
CIFARIO
0.1
0.556
0.319
0.258
0.395
0.587
0.292
0.582
0,586
0.593
0.229
0.2
0.716
0.578
0.363
0.655
0.654
0,388
0.646
0.648
0.659
0.169
0.3
0.793
0.616
0.394
0.691
0.682
0,493
0.667
0.678
0.691
0.643
best
3
4
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work are discussed in Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
21

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The full sets of assumptions and proof are described in Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The details of the experimental settings (data, comparison methods, network architecture, and byperparameters) are described in Section []
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification:
The code proprietary. The datasets and experimental procedures are de-
scribed in Section 5 We described the pseudocode of our method in Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttpcccpccccccccccccceoooo httpi//ggguieee/eeeeuuubiiiiii cetisubimsssiinii. ssissssiooooooo ssioiioiccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details of the experimental settings are described in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted a statistical significance test (paired t-test) in tables 1 and The standard deviations of Table are reported in Table The standard errors are reported in all graphs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We described the computer resource in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We confirmed the NeurIPS Code of Ethnies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Ye]]
Justification:
We discussed the potential social impacts in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release such high-risk models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Datasets used in this paper are cited in Section 5
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not provide new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "WBpncE1xiG", "reviewer": "Reviewer_QCqD", "summary": "This paper addresses the challenge of maximizing the Area Under the Receiver Operating Characteristic Curve (AUC) in imbalanced binary classification problems where there is a positive distribution shift--this shift is where negative data remains constant, but positive data varies. A new method is proposed that utilizes labeled positive and unlabeled data from the training distribution, along with unlabeled data from the test distribution, to maximize the AUC effectively in the presence of such shifts.", "questions": "Q1: Please specify the scenarios where both class imbalance and positive distribution shift occur. Providing detailed examples will help readers grasp the practical significance of this research problem.\n\nQ2: Why did the authors choose to conduct their experiments exclusively with image datasets? Are there any other real-world problems?\n\nQ3: AUC maximization can be implemented not just as a loss function for neural networks, but across various machine learning methods. Why did you choose to focus on proposing a loss function?\n\nQ4: The reviewer is not convinced that Lines 4-5 in Algorithm 1 sufficiently demonstrate the training process. There needs to be a more detailed and mathematical explanation of how model parameters are updated using the proposed loss function.", "limitations": "L1: An expansion to include various metrics, such as F-1 and G-mean of TPR and TNR, which are also relevant for imbalanced data classification, could enrich this paper.\n\nL2: For extremely imbalanced cases, training difficulties are likely to arise. It is necessary to address how the proposed loss function can be effectively minimized in these scenarios.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper introduces a new loss function designed for AUC maximization under positive distribution shifts. Previous research has focused separately on AUC maximization and positive distribution shifts, but this study found the intersection of these two areas. The authors have successfully identified and explored this new research niche. The proposed loss function, derived from mathematical foundations, can be readily integrated into neural network training, offering a practical application for enhancing model performance. This paper is well-structured and clearly written, making it easy to follow.", "weaknesses": "Despite its strengths, this research primarily offers a simple proposal of a loss function, suggesting its contributions to the field might be limited. An expansion to include various metrics, such as F-1 and G-mean of TPR and TNR, which are also relevant for imbalanced data classification, could enrich this paper. Additionally, the experimental validation is somewhat restricted, utilizing only four datasets, all of which are image datasets. A more comprehensive evaluation using a broader range of datasets is necessary to fully assess the proposed loss function's effectiveness. Therefore, the reviewer believes that the contribution of this research may not be substantial enough for acceptance at a top-tier conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and constructive feedback.\n\n> An expansion to include various metrics, such as F-1 and G-mean of TPR and TNR, which are also relevant for imbalanced data classification, could enrich this paper.\n\nThank you for the suggestion. We agree that extensions to maximize other metrics such as F-1 could enrich our paper. However, AUC is the most representative metric for imbalanced data, and many papers (including top conference and journal papers such as ICLR, AAAI, ICCV, ICDM, NeurIPS, and TPAMI) focus solely on AUC maximization [12, 29, 41, 54, 55, 57, 58, 60, 62]. We, therefore, believe that the current proposal is still worthy of publication. Extending our problem setting to maximize other metrics is an interesting future work.\n\nAs a supplement, we evaluated F1 scores of each method.\nThe results are described in Table 1 of the PDF file in the global response.\nOur method also outperformed the others even when it maximizes the AUC.\nThis result may imply that AUC maximization can also help improve other evaluation metrics for imbalanced data.\nWe will include this result and the above discussion (future work) in the final version.\n\n> the experimental validation is somewhat restricted, utilizing only four datasets, all of which are image datasets. A more comprehensive evaluation using a broader range of datasets is necessary to fully assess the proposed loss function's effectiveness.\n\nThank you for the constructive comment.\nWe additionally evaluate our method with two tabular datasets (Hospital Readmission and Hypertension [a]).\nIn Hospital Readmission, the task is to predict the 30-day readmission of diabetic hospital patients.\nIn Hypertension, the task is a hypertension diagnosis for high-risk age.\nBoth datasets are widely used for distribution shift adaptation studies [a].\nWe used positive-shifted data to construct the positive distribution shift.\nThe experimental settings, such as the number of data, are the same as the submitted paper.\nThe results are described in Table 2 of the PDF file.\nOur method outperformed the others. These results show that our method works well in tabular datasets. We will include these results in the final version.\n\n[a] Gardner et al. \"Benchmarking distribution shift in tabular data with tableshift.\" NeurIPS2023.\n\n> Q1: Please specify the scenarios where both class imbalance and positive distribution shift occur. Providing detailed examples will help readers grasp the practical significance of this research problem.\n\nWe have described the motivating examples (cyber security, medical diagnosis, and visual inspection) where both class imbalance and positive distribution shifts occur in the first and second paragraphs of Section 1. For example, in cyber security, malicious data (positive data) are much smaller than benign data (negative data). In addition, malicious adversaries often change their attacks (malicious data) to bypass detection systems, while benign data do not change much [9, 15, 63]. Thus, our problem setting is suitable for this application.\n\n> Q3: AUC maximization can be implemented not just as a loss function for neural networks, but across various machine learning methods. Why did you choose to focus on proposing a loss function?\n\nWe focus on the loss function because it has a high degree of generality: it can be combined with any (differentiable) models such as linear models, kernel models, and neural networks, as described in Lines 66--67. This characteristic is beneficial in practice because the available computing resources vary depending on the application site.\nFor example, when resources such as GPUs are scarce, which is often the case, our loss function can be used with lightweight models such as liner models. When ample resources are available, huge models such as large neural networks can also be used.\n\n> Q4: The reviewer is not convinced that Lines 4-5 in Algorithm 1 sufficiently demonstrate the training process. There needs to be a more detailed and mathematical explanation of how model parameters are updated using the proposed loss function.\n\nSorry for the lack of explanation.\nWe describe a more detailed and mathematical explanation of Lines 4 and 5 in Algorithm 1 below:\n\n- Let $\\\\{ {\\bar x} _{{\\rm tr},m} ^{\\rm p} \\\\} _{m=1}^{P}$ be sampled positive data from $X _{{\\rm tr}} ^{\\rm p}$ and $\\\\{ {\\bar x} _{{\\rm tr},m} \\\\} _{m=1} ^{M _{{\\rm tr}}} \\cup \\\\{ {\\bar x} _{{\\rm te},m} \\\\} _{m=1} ^{M _{{\\rm te}}}$ be sample unlabeled data from $X _{{\\rm tr}} \\cup X _{{\\rm te}}$. Then, in Line 4, we calculate the loss in Eq. 14 on the sampled data. That is, \n${\\hat {\\mathcal L}} _{{\\rm sampled}} (\\theta) := - \\frac{1}{M _{{\\rm te}} M _{{\\rm tr}}} \\sum _{n,m=1} ^{M _{{\\rm te}}, M _{{\\rm tr}}} f({\\bar {\\bf x}} _{{\\rm te}, n}, {\\bar {\\bf x}} _{{\\rm tr}, m}) + \\frac{\\pi _{{\\rm tr}}}{M _{{\\rm te}} P} \\sum _{n,m=1} ^{M _{{\\rm te}}, P } f({\\bar {\\bf x}} _{{\\rm te}, n}, {\\bar {\\bf x}} _{{\\rm tr}, m} ^{{\\rm p}})$,\nwhere $\\theta$ is parameters of score function $s$.\n\n- In Line 5, we update $\\theta$ by using a stochastic gradient descent. That is, \n$\\theta \\leftarrow \\theta - \\mu \\frac{\\partial {\\hat {\\mathcal L}} _{{\\rm sampled}}}{\\partial \\theta} (\\theta)$, \nwhere $\\mu \\in \\mathbb{R} _{>0}$ is a learning rate. \n\nWe will add this explanation in the final version.\n\n> L2: For extremely imbalanced cases, training difficulties are likely to arise. It is necessary to address how the proposed loss function can be effectively minimized in these scenarios.\n\nAs you mentioned, highly imbalanced data will be difficult to learn for all methods, including ours. This is due to the difficulty in extracting positive data information from unlabeled data. However, in Table 4 of Section C.2, we confirmed that our method outperformed the others when small class-priors on the training distribution (e.g., $\\pi _{{\\rm tr}}=0.01$) are used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GWG6fAkiwb", "reviewer": "Reviewer_4KuN", "summary": "This paper considers AUC maximization when the conditional probability distribution of the positive class changes in the test phase. To this end, the unbiased loss function is derived. The loss is approximated by positive and unlabeled data from training distribution, unlabeled data from test distribution, and class-prior of training distribution. In experiments, the proposed method outperformed the existing methods over the four benchmark datasets.", "questions": "Is the proposed loss function unbiased to its supervised counterpart? \n\nIt would be valuable if there were discussion or experimental results showing the effect of the number of unlabeled data from the test distribution. In some applications, collecting a lot of unlabeled data from the test distribution might be difficult. In such a situation, the experimental results would help practitioners understand how many samples are necessary to collect.\n\nAccording to the literature, the non-negative risk estimator plays a crucial role in training deep neural networks. However, the proposed method does not mention the non-negativity of the risk estimator. Did the authors encounter that the risk estimator went to a large negative value in experiments? If not, what points in the proposed method avoid the issue?\n\nRegarding the Extension in Section 4.4, it would be nice to cite the existing work.", "limitations": "The limitations are discussed in Appendix A.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This is the first study on AUC maximization for positive distribution shift.\n- The proposed method outperformed the existing methods.\n- The proposed method does not require the class-prior of the test distribution.", "weaknesses": "- Unlike the existing study [15, 42], the negative distribution is not considered.\n- It lacks theoretical analyses of the proposed method.\n- The extension of the proposed method is discussed but not evaluated in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and constructive feedback.\n\n> Unlike the existing study [15, 42], the negative distribution is not considered.\n\nThe method in [15] assumes the positive distribution shift as in our method. Thus, it does not consider the negative distribution change. \nThe method in [42] assumes the covariate shift, i.e., $p _{{\\rm te}} (x) \\neq p _{{\\rm tr}} (x)$ but $p _{{\\rm te}}(y|x) = p _{{\\rm tr}} (y|x)$. Although the covariate shift has been traditionally studied, the assumption of $p _{{\\rm te}}(y|x) = p _{{\\rm tr}} (y|x)$ is often restrictive in practice.\nOne of the main contributions of our work is to highlight an important problem (both class imbalance and positive distribution shift occur) that has been overlooked despite its many applications, as described in Section 1.\n\n> The extension of the proposed method is discussed but not evaluated in the experiments.\n\nWe have already evaluated the extension of our method (Eq. 16) in Table 6 of Section C.4.\nThe extension of our method ($\\alpha=0.999$) slightly tended to perform better than our original method ($\\alpha=1.0$) by using additional labeled negative data. \n\n> Is the proposed loss function unbiased to its supervised counterpart?\n\nYes. As long as the assumption of positive distribution shift (Eq. 7) is satisfied, the derived AUC in Eq. 12 is equivalent to the original AUC in Eq. 8. \n\n> It would be valuable if there were discussion or experimental results showing the effect of the number of unlabeled data from the test distribution. \n\nThank you for the constructive comment. We additionally evaluate our method with a small number of \nunlabeled data from the test distribution.\nThe results are described in Table 3 of the PDF file in the global response.\nAs expected, the performance of our method tended to increase as the number of unlabeled data $N_{{\\rm te}}$ increased. Nevertheless, our method tends to outperform puAUC, which does not use unlabeled test data, even when $N _{{\\rm te}}=500$. Since many unlabeled data are often easy to collect, we believe that our method is useful in practice. We will include these results in the final version.\n\n> According to the literature, the non-negative risk estimator plays a crucial role in training deep neural networks. However, the proposed method does not mention the non-negativity of the risk estimator.\n\nThank you for the insightful comment. In our preliminary experiment, we evaluated our method with the non-negative loss correction (Ours w/ nn), which prevents the empirical loss from being smaller than zero. However, the results with and without the loss correction (Ours w/ nn and Ours) were almost identical, as described in Table 4 of the PDF file. Thus, we proposed the current simpler loss function.\n\nWe describe the non-negative correction used in this experiment.\nFor clarity, we consider to minimize the AUC risk, $R _{\\rho} (s) := \\mathbb{E} _{{\\bf x} ^{{\\rm p}} \\sim p ^{{\\rm p}} _{{\\rm te}} ({\\bf x} )} \\mathbb{E} _{{\\bf x} ^{{\\rm n}} \\sim p ^{{\\rm n}} _{{\\rm te}} ({\\bf x} )} [ f({\\bf x} ^{{\\rm n}}, {\\bf x} ^{{\\rm p}}) ] $, which is equivalent to maximize the AUC in Eq. 8 since  $ {\\rm AUC} _{\\rho} (s) = 1 - R _{\\rho} (s) $.  The minimum value of this risk is zero. Then, the corresponding loss for Eq. 13 becomes ${\\cal L} _{{\\rm risk}}(s) :=  \\mathbb{E} _{{\\bf x} \\sim p _{{\\rm te}} ({\\bf x}) } \\mathbb{E} _{{\\bar {\\bf x} } \\sim p _{{\\rm tr} } ({\\bf x}) } [ f({\\bar {\\bf x}} ,{\\bf x}) ] - \\pi _{{\\rm tr} } \\mathbb{E} _{{\\bf x} \\sim p _{{\\rm te}} ({\\bf x}) } \\mathbb{E} _{{\\bf x} ^{{\\rm p}} \\sim p ^{{\\rm p}} _{{\\rm tr}} ({\\bf x}) } [ f({\\bf x} ^{{\\rm p}} ,{\\bf x}) ] $. \nSince this loss is derived from the AUC risk, $\\mathbb{E} _{{\\bf x} \\sim p _{{\\rm te}} ({\\bf x} )} \\mathbb{E} _{{\\bf x} ^{{\\rm n}} \\sim p ^{{\\rm n}} _{{\\rm te}} ({\\bf x} )} [ f({\\bf x} ^{{\\rm n}}, {\\bf x}) ]$, it should not take negative values. Thus, to prevent negative values of its empirical estimate ${\\hat {\\cal L} _{{\\rm risk}}} (s) $, we used the loss with the absolute value function $|{\\hat {\\cal L} _{{\\rm risk}}} (s)| $ for the optimization. This correction is successfully used in PU learning [15] or other weakly supervised learning [a].\n\nBy the way, if class-prior $\\pi _{{\\rm te}}$ is known, we can derive a tighter lower bound on the AUC risk. Specifically, we can obtain $\\mathbb{E} _{{\\bf x} \\sim p _{{\\rm te}} ({\\bf x} )} \\mathbb{E} _{{\\bf x} ^{{\\rm n}} \\sim p ^{{\\rm n}} _{{\\rm te}} ({\\bf x} )} [ f({\\bf x} ^{{\\rm n}}, {\\bf x}) ] \\geq (1-\\pi _{{\\rm te}})/2$. Here, we used the definition of $p _{{\\rm te}} ({\\bf x} )$ and the fact that the AUC risk between the same densities is $1/2$ as described in Lines 163--164. By substituting Eq. 11 for this, we can obtain ${\\cal L} _{{\\rm risk}}(s) \\geq (1-\\pi _{{\\rm te}})(1-\\pi _{{\\rm tr}})/2 =: b > 0$. As a result, we can use $| {\\hat {\\cal L} _{{\\rm risk}}} (s) - b| + b$ for the optimization. Our method with this correction (Our w/ b) tended to enhance the performance of it without the correction (Ours) in Table 3. However, note that Ours has the strong advantage of not requiring the class-prior and performed better than existing methods. We will include this result in the final version.\n\n[a] Lu, Nan, et al. \"Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach.\" AISTATS2020.\n\n> Regarding the Extension in Section 4.4, it would be nice to cite the existing work.\n\nThank you for the suggestion. Since the extension in Section 4.4 uses positive, negative, and unlabeled data in the training distribution, it is related to semi-supervised learning.\nA few studies [41,b] especially use the PU learning for semi-supervised learning.\nHowever, semi-supervised learning does not consider the distribution shift. The final version will include a more detailed discussion and citations.\n\n[b] Sakai et al. Semi-supervised classification based on classification from positive and unlabeled data. In ICML2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kHrM20p0pj", "reviewer": "Reviewer_dvdg", "summary": "The paper proposes a method for AUC maximization in binary classification problems under positive distribution shift. They introduce their method, which is simple and easy to implement/understand, and then show it works well in some experiments.", "questions": "How should the practitioner choose classification threshold after training their classifiers using your method?", "limitations": "The authors discuss limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and easy to understand;\n- The paper proposes a well-motivated method and show how it can be easily implemented in practice;\n- The experiments are convincing.", "weaknesses": "- The authors do not discuss how the classification threshold can be chosen in a practical situation under positive distribution shift.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and constructive feedback.\n\n> How should the practitioner choose classification threshold after training their classifiers using your method?\n\nThank you for the insightful question. \nIn practical use, there are many situations where it is beneficial just to be able to sort data in score order. For example, in anomaly detection, experts or operators can check data with high scores within the cost they can spend or until anomalous data do not appear. In disease diagnosis, patients with higher scores can be given priority for detailed examination.\nIn recommendation systems, products can be presented to users in order of score.\n\nHowever, when a classification threshold is required, we can use the estimated class-prior in the test distribution to determine the threshold. \nSpecifically, we first extract negative data in unlabeled data from the training distribution by applying (off-the-shelf) PU learning to PU data in the training distribution. Since PU data and the class-prior in the training distribution are available in our setting, we can perform it. Next, since the negative distribution does not change in our setting, the extracted negative data can be regarded as negative data in the test distribution. We can estimate the class-prior in the test distribution by applying existing class-prior estimation methods [40, 59, 14] to the extracted negative and unlabeled data in the test distribution. The threshold can be set to the top $N_{{\\rm te}} \\pi_{{\\rm te}}^{{\\rm est}}$-th score of unlabeled data, where $\\pi_{{\\rm te}}^{{\\rm est}}$ is the estimated class-prior. We will include this discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sb9O6MTkWa", "reviewer": "Reviewer_CKbq", "summary": "Due to a positive distribution shift, training and test distributions are not identical. However, existing AUC maximization methods don’t take it into account. To address this shift, this paper theoretically shows a new way to maximize the AUC on the test distribution by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution. Finally, four real-world datasets validate the effectiveness of the proposed method.", "questions": "please refer to Weakness", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "-\tThe proposed setting is novel and practical in AUC optimization. The distribution of negative data is generally stable but the distribution of positive data is more diverse or time-varying in medical diagnosis, intrusion detection, and visual inspection. \n-\tThe method presentation is easy to understand. This paper first introduces basic AUC fundamental knowledge. Then, it gives the problem setting of the proposed positive distribution shift. Based on this setting, the final expression is obtained through some intuitive and simple derivation. To be specific, the AUC maximization on the test distribution can be accomplished by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution.", "weaknesses": "- The effect of the proposed methods on MINST and Fashion MINST datasets is not significant, which is inconsistent with those on the other datasets. The authors don’t give any explanation.\n- The authors do not fully compare their method with the latest ones. For example, \n  - Positive-Unlabeled Learning with Label Distribution Alignment. (TPAMI 2023)\n  - Dist-PU: Positive-Unlabeled Learning from a Label Distribution Perspective. (CVPR 2022)\n  - Positive-unlabeled learning using random forests via recursive greedy risk minimization. (NeurIPS 2022)\n- All theoretical derivations are only based on the sigmoid surrogate loss. As far as I know, square loss is also popular. Can the theoretical results extend to the other losses?\n- There are some typos. For example,\n  - In line 105, “However, these all methods assume that” should be “However, all these methods assume that”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and constructive feedback.\n\n> The effect of the proposed methods on MINST and Fashion MINST datasets is not significant, which is inconsistent with those on the other datasets. The authors don’t give any explanation.\n\nAs described in Line 279, since MNIST and Fashion MNIST are relatively simple data, the performances of our method and PURR that does not consider class-imbalance might be similar. More specifically, if the supports of positive and negative conditional densities are separated (i.e., data are simple), both losses of positive and negative data can be minimized without conflict between them, even when there is a class imbalance. Thus, class imbalance might not be a severe problem in this case. In contrast, when positive and negative-conditional densities overlap (i.e., complex data), minimizing losses for positive data located in the overlapped region causes to increase losses for negative data in the same region. Thus, when there is a class imbalance, positive data that are few can often be ignored. Therefore, in this case, class imbalance becomes a more severe problem. Since SVHN and CIFAR10 are more complex than MNIST and Fashion MNIST, our method worked well. We will clarify this.\n\n> The authors do not fully compare their method with the latest ones.\n\nThank you for sharing relevant works.\nWe additionally compared our method with the method [a] in your comment (PURDA).\nThe results are described in Table 5 of the PDF file in the global response.\nPURDA used positive and unlabeled data in the training distribution in this experiment since it is designed for ordinary PU learning.\nMargin $\\rho$ is selected from $\\\\{0.1,1,10\\\\}$ by validation data.\nOur method performed better than PURDA.\nThis is because PURDA does not consider the distribution shift.\nAll the methods in your comment do not consider the distribution shift and, thus, are unsuitable for our setting.\nIn the final version, we will include this result and add all related works in your comments in Section 2.\n\n[a] Positive-Unlabeled Learning with Label Distribution Alignment. (TPAMI 2023)\n\n> All theoretical derivations are only based on the sigmoid surrogate loss. As far as I know, square loss is also popular. Can the theoretical results extend to the other losses?\n\nThis is an insightful question. As long as we use symmetric losses (i.e., loss $l$ satisfying $l(z)+l(-z)=K$ for any $z \\in \\mathbb{R}$ and $K$ is a constant) [6], we can derive the same final loss function in Eq. 13. This is because the second term in Eq. 10 becomes constant, and thus, we can ignore it.\nWe note that symmetric losses include a wide range of losses such as sigmoid, ramp, unhinged, and zero-one losses, although the squared loss is not included [6].\nWe will include this discussion in the final version.\n\n> There are some typos.\n\nThank you for pointing out the typos. We will again carefully proofread and prepare the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all reviewers,

Thank you very much for the detailed and constructive feedback on our paper. We would like to revise the paper based on the comments. A pdf file with additional experiments is attached to this global response.

Best regards,

Authors

Meta-review: The addressed task is very relevant in practice. One of the reviewers was a bit unhappy that it only applies for AUC. It would indeed be useful to know about whether similar approaches could be developed for other objectives, but it is perfectly fine that the paper addresses only AUC. The appendix includes information about F1 as well, but this is just informative, not stating that the method should be applied to improve F1 when there is positive distribution shift.

The proposed method is quite simple and uses a trick that is somewhat similar to many other methods in transfer learning, but this is rather a strength not a weakness.

One of the reviewers (4KuN) had first lower grades for presentation, soundness, and contribution, but raised the overall rating from 4 to 5 after a lengthy discussion with the authors, stating eventually that "The textual explanation is fine".

One of the most positive reviewers (CKbq) claimed that "The authors do not fully compare their method with the latest ones" and gave 3 references. However, these papers did not address the shift on the positives as the current paper does. Regardless, in the rebuttal phase, the authors included one of those methods into the comparison. As expected, that method was outperformed by the method proposed in the current paper.

The reviewers had no major criticisms about the quality of presentation.

Therefore, I recommend the paper for acceptance.
Decision: Accept (poster)