AUC Maximization under Positive Distribution Shift

Authors: atsutoshi  kumagai
tomoharu  iwata
hiroshi  takahashi
taishi  nishiyama
yasuhiro  fujiwara
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 05:26:34

Abstract: Maximizing the area under the receiver operating characteristic curve (AUC) is a popular approach to imbalanced binary classification problems. Existing AUC maximization methods usually assume that training and test distributions are identical. However, this assumption is often violated in practice due to {\it a positive distribution shift}, where the negative-conditional density does not change but the positive-conditional density can vary. This shift often occurs in imbalanced classification since positive data are often more diverse and time-varying than negative data. To deal with this shift, we theoretically show that the AUC on the test distribution can be expressed by using the positive and marginal training densities and the marginal test density. Based on this result, we can maximize the AUC on the test distribution by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution. The proposed method requires only positive labels in the training distribution as supervision. Moreover, the derived AUC has a simple form and thus is easy to implement. The effectiveness of the proposed method is shown with four real-world datasets.

Main paper: 
 introduction: 
 In many real-world binary classification problems such as intrusion detection [37] , medical diagnosis [60] , and visual inspection [42] , class-imbalance frequently occurs where positive data is much smaller than negative data [22] . In this case, classification accuracy, which is the standard performance measure for ordinary binary classification, is not a suitable measure [54, 61] . Instead, the area under the receiver operating characteristic curve (AUC) is commonly used [4, 19] . The AUC is the probability that a classifier will rank a randomly drawn positive instance higher than a randomly drawn negative one [61] . Due to the nature of the ranking, the AUC can adequately measure the performance of the classifier even with imbalanced data. By maximizing the AUC, we can obtain accurate classifiers even from imbalanced data [5, 61, 64, 32, 65] .Existing AUC maximization methods usually assume that training and test distributions are identical to ensure the generalization performance. However, in real-world applications, this assumption is often violated due to distribution shifts. This paper considers a positive distribution shift [16] , i.e., the negative-conditional density does not change but the positive-conditional density can vary, because this shift often occurs in imbalanced problems. For example, in intrusion detection, malicious adversaries rapidly change their attacks (positive data) to bypass detection systems while the benign class's data (negative data) do not change much [9, 16, 67] . In medical diagnosis, the distribution of disease data (positive data) can change due to the disease progression or emergence of new pathogens, but the distribution of data of healthy people is generally stable. In visual inspection, the types of anomalous products (positive data) are diverse but normal ones (negative data) have some degree of stationarity. When such a distribution shift occurs, the performance of AUC maximization methods drastically deteriorates. Although labeled data drawn from the test distribution can alleviate this problem, such data are often timeconsuming and expensive to collect whenever the distribution shift occurs [41] . In addition, labeled negative data in the training distribution are also difficult to collect in some applications. For example, in intrusion detection, although some malicious data can be collected from public sources such as blacklists, benign data are often unavailable due to privacy reasons, and identifying clean benign data from given unlabeled data requires a high level of expertise [37, 47] .In this paper, we propose a method for maximizing the AUC under the positive distribution shift by using labeled positive and unlabeled data in the training distribution and unlabeled data in the test distribution. Figure 1 illustrates examples of given data in our problem setting.Since no labeled data are available in the test distribution, a challenge is how to maximize the AUC on the test distribution. To address this challenge, we theoretically show that the AUC on the test distribution can be expressed by using the positive and marginal training densities and the marginal test density when assuming the positive distribution shift. This result enables us to maximize the AUC on the test distribution with positive and unlabeled data in the training distribution and unlabeled data in the test distribution. The derived AUC has a simple form and is easy to implement. In addition, it does not depend on the class-prior in the test distribution, which is usually difficult to obtain with unlabeled data in the test distribution. Also, it can be applied to any differentiable classifiers such as linear classifiers, kernel-based classifiers, and neural networks. Hence, our method is easy for practitioners to use.The main contributions of our work are summarized as follows:• We propose a novel and practical problem setting, where the aim is to maximize the AUC under the positive distribution shift on imbalanced data.• We theoretically show that the AUC on the test distribution can be maximized with positive and unlabeled data in the training distribution and unlabeled data in the test distribution when assuming the positive distribution shift.• We empirically demonstrate that the proposed method outperformed various existing methods with six real-world datasets.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our claim described in the abstract and introduction accurately reflects our contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Many AUC maximization methods have been proposed [5, 61, 64, 32, 65] . In previous studies [65, 66, 12, 55] , they have been reported to often perform better than other methods for imbalanced classification such as class balanced loss [6] , focal loss [31] , or sampling-based methods [8, 36] . However, these AUC maximization methods require labeled positive and negative data. In addition, they assume that the training and test distributions are the same. Therefore, they are inappropriate for our problem setting where there are no labeled negative data and the distribution shift occurs.Unsupervised domain adaptation aims to adapt to the distribution (domain) shift by using unlabeled data in the test distribution and labeled data in the training distribution [41, 56] . One representative approach is to learn domain-invariant feature representations by minimizing the discrepancy of the features in both domains [33, 53, 52, 13, 27, 11, 48] . Although this approach is promising, it often deteriorates the performance since it minimizes only the feature discrepancy without considering the relationship between features and labels [70] . Another representative approach is to explicitly minimize the loss on the test distribution by assuming the types of the distribution change, such as covariate shift [50, 3, 23, 69, 35] and class-prior shift [68, 51, 1] . The proposed method belongs to this approach. By assuming the shift type, this approach can adapt to the shift in a theoretically guaranteed manner. The existing methods aim to minimize the classification risk (or negative classification accuracy), which is an inappropriate metric for imbalanced data (e.g., if the imbalanced data ratio is 1 : 99, a naive classifier that classifies 'every' instance as negative has 99% accuracy, but it is definitely not a good classifier). One domain adaptation method tries to maximize the AUC by learning domain-invariant feature representations [62] . However, this method is not designed for the positive distribution shift. Moreover, all these methods require labeled positive and negative data in the training distribution, which are unavailable in our problem setting.Positive-unlabeled (PU) learning methods aim to learn classifiers by using only positive and unlabeled data [2, 10] . The proposed method is related to the PU learning since it assumes positive and unlabeled data in the training distribution. A representative PU learning method is the empirical minimization-based approach, which rewrites the classification risk by using positive and unlabeled densities [9, 25, 49, 21] . Although they are effective, they cannot maximize the AUC. Recent studies have shown that the AUC can be rewritten from positive and marginal densities by using the technique of the PU learning [45, 58, 6, 59 ]. However, these all methods assume that the training and test distributions are identical.Several PU learning methods consider the distribution shift such as covariate shift [46] , class-prior shift [7, 39] , or positive distribution shift [16] . They require positive and unlabeled data in the training distribution and unlabeled data in the test distribution. However, they consider the classification risk and cannot maximize the AUC. Due to the pairwise formulation of the AUC, their methods that use ordinary instance-wise loss functions such as the cross-entropy loss cannot be applied to the AUC. In addition, the method for the positive distribution shift [16] requires the class-prior on the test distribution, which is generally difficult to know with unlabeled data in the test distribution. In contrast, the proposed method does not need to know the class-prior and thus is more practical.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of our work are discussed in Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 We briefly explain the AUC maximization. Let instance x ∈ R D and its corresponding label y ∈ {-1, +1} be equipped with probability density p(x, y), where +1 and -1 means a positive and negative class, respectively. Here, p p (x) := p(x|y = +1) and p n (x) := p(x|y = -1) is the conditional probability density of positive and negative class, respectively. Furthermore, let s : R D → R be a score function that outputs the positivity of an input instance. The classifier is defined by the score function with threshold t: y = sign(s(x) -t), where sign is a sign function.The AUC is the probability of a randomly drawn positive instance being ranked before a randomly drawn negative instance [61] . Specifically, the AUC with score function s can be formulated asEQUATIONI(z) is the indicator function that outputs 1 if z is true and 0 otherwise, and E is the expectation. Since the gradient of indicator function I is zero everywhere except for the origin, the AUC cannot be optimized via gradient descent methods. To avoid this, the following smoothed AUC is often used by replacing the indicator function with a sigmoid function σ(z) = 1/(1 + exp(-z)) [20, 28, 29] :EQUATIONGiven N p positive instances {x p 1 , . . . , x p N p } drawn from p p (x) and N n negative instances {x n 1 , . . . , x n N n } drawn from p n (x), the empirical estimate of the smoothed AUC is calculated asEQUATIONBy maximizing this empirical smoothed AUC with respect to the parameters of s, we can obtain good score functions to maximize the AUC when the training and test distributions are identical [61] . 
proposed method: 
 In this section, we first describe our problem setting (subsection 4.1). Then, we theoretically show that the AUC on the test distribution can be maximized with positive and unlabeled data in the training distribution and unlabeled data in the test distribution under the positive distribution shift (subsection 4.2). Then, we discuss class-priors used in the proposed method (subsection 4.3). Lastly, we explain an extension of the proposed method (subsection 4.4).Suppose that we are given a set of positive instances X p tr and a set of unlabeled instances X tr drawn from the training distribution:EQUATIONEQUATIONwhere p tr (x) is the marginal density of the training distribution, p p tr (x) and p n tr (x) := p tr (x|y = -1) are positive and negative-conditional densities of the training distribution, respectively, and π tr := p tr (y = +1) is the positive class-prior. Although we assume that class-prior π tr is known in this paper, it can be estimated from positive and unlabeled data [44, 63, 15] . In addition, we suppose that a set of unlabeled instances X te drawn from the test distribution is also given:EQUATIONwhere p p te (x) := p te (x|y = +1), p n te (x) := p te (x|y = -1), and π te := p te (y = +1). We assume the class-imbalance in both unlabeled data, i.e., π tr , π te 1. As shown later, the proposed method does not need to know π te to maximize the AUC 1 , which is beneficial in practice.We consider a situation of the positive distribution shift between the training and test distributions, where the negative-conditional density does not change but the positive-conditional one can vary, p n tr (x) = p n te (x), p p tr (x) = p p te (x).This situation often occurs in imbalanced classification problems as described in Section 1. Our aim is to learn score function s : R D → R that can maximize the AUC on the test distribution by using X p tr ∪ X tr ∪ X te . For score function s, we can use any differentiable function such as linear classifiers, kernel-based classifiers, or neural networks. Note that we also discuss the negative distribution shift, i.e., p n tr (x) = p n te (x) and p p tr (x) = p p te (x), in Section D.In this subsection, we explain how to maximize the AUC under the positive distribution shift. The objective function to be maximized is the following smoothed AUC on the test distribution,EQUATIONwhere we set f (x p , x n ) := σ(s(x p ) -s(x n )). Since this AUC depends on p p te (x) and p n te (x), it seems be impossible to calculate in our setting where neither positive nor negative data in the test distribution are given. However, we show that calculation is possible. First, from the definition of marginal density p te (x) in Eq. ( 6), the positive-conditional test density p p te (x) can be expressed asEQUATIONwhere we assume that π te > 0. By substituting Eq. ( 9) into Eq. ( 8), we can obtain Sample positive data with size P form X p tr 3:EQUATIONSample unlabeled data with size M -P from X tr ∪ X te 4:Calculate the loss in Eq. ( 14) on the sampled positive and unlabeled data with π tr 5:Update model parameters of score function s with the gradient of the loss 6: until End condition is satisfied;Here, the second term in Eq. (10) [58, 6, 59] . Therefore, we can ignore the second term in Eq. (10) to learn score function s. Next, by using the assumption of the positive distribution shift, p n tr (x) = p n te (x), and the definition of the marginal density p tr (x) in Eq. ( 5), we can express the negative-conditional test density p n te (x) asbecomes constant (1 -π te )E x n ∼p n te (x) E xn ∼p n te (x) [f (x n , xn )] = (1 -π te )/2 because σ(z) + σ(-z) = 1 for all z ∈ REQUATIONwhere we assume that π tr < 1. As before, by substituting Eq. ( 11) into the first term in Eq. ( 10), we can obtainEQUATIONwhere C represents constant terms that do not depend on score function s. Since coefficient 1/(π te (1π tr )) does not affect the optimization for s, our loss function to be minimized is as follows, L(sEQUATIONThis loss depends on the positive and marginal training densities, marginal test densities, and classprior. Therefore, we can approximate this loss with given data X p tr ∪ X tr ∪ X te and π tr . Specifically, the empirical estimate of the loss function is given as follows, L(s) = - 1 N te N trAlgorithm 1 shows the pseudocode of our training procedure with stochastic gradient methods. Note that although we have used the sigmoid function to represent the AUC in Eq. ( 2), as long as we use symmetric functions (i.e., function σ satisfying σ(z) + σ(-z) = K for any z ∈ R and K is a constant [6] ), we can derive the loss function of the same form in Eq. (13) . The symmetric functions include a wide range of functions such as sigmoid, ramp, and unhinged functions [6] . In addition, the loss corrections for the proposed method such as the non-negative correction [25] , which is commonly used in ordinary PU learning to avoid overfitting, are discussed in detail in Section B.In the process of deriving our loss function, we assume π tr < 1 and π te > 0. We discuss why this assumption is necessary in our setting. When π te = 0, all unlabeled data X te become negative data. In this case, our given datasets X p tr ∪ X tr ∪ X te do not have any information about positive data in the test distribution since p p tr (x) = p p te (x). Therefore, we cannot calculate the AUC on the test distribution. However, in this case, we can easily create trivial optimal classifiers that classify all data as negative. Thus, this is not an issue.When π tr = 1, all unlabeled data X tr become positive data on the training distribution. X p tr ∪ X tr (positive data in the training distribution) have no information to extract positive and negative data from X te since p p tr (x) = p p te (x). Thus, in this case, we also cannot calculate the AUC on the test distribution. However, it is extremely rare for all unlabeled data to be positive in imbalanced data problems where the ratio of positive data in unlabeled data is low. Thus, this is also not a problem in practice.In some applications, a small number of labeled negative data as well as labeled positive data might be available from the training distribution. The proposed method can be easily extended to such cases. Specifically, by Eq. ( 10), the AUC on the test distribution without constant terms is represented asEQUATIONwhere we used assumption p n tr (x) = p n te (x). Loss R(s) can be approximated with test unlabeled data and labeled negative data. Thus, the following modified loss function to be minimized can be used:EQUATIONwhere α ∈ [0, 1] is a weighting hyperparameter. 
experiments: 
 In this section, we empirically demonstrate the effectiveness of the proposed method under the positive distribution shift with real-world datasets.We utilized four widely used real-world datasets in the main paper: MNIST [30] , FashionMNIST [57] , SVHN [40] , and CIFAR10 [26] . MNIST consists of hand-written images of 10 digits. Each image is represented by gray-scale with 28 × 28 pixels. FashionMNIST consists of images of 10 fashion categories where each image is represented by gray scale with 28 × 28 pixels. SVHN consists of 32 × 32 RGB images of printed 10 digits clipped from photographs of house number plates. We converted SVHN into gray-scale for simplicity. CIFAR10 consists of 32 × 32 RGB images of 10 animal and vehicle categories. In Section E.3, we also evaluated the proposed method with two tabular datasets with distribution shifts (HReadmission and Hypertension) [14] .For MNIST and SVHN, we used even digits as the negative class and odd digits as the positive class. Following the previous studies [16, 18] , data with digits '7' and '9' were used as positive data appearing in the training distribution and data with digits '1', '3' , '5', '7', and '9' were used as positive data appearing in the test distribution. This simulates a situation where new types of positive data, which did not appear during the training, appear in the test environment. For FashionMNIST, following the study [59] , we used upper garments ('T-shift', 'Pullover', 'Dress', 'Coat', and 'Shift') as the negative class and the others as the positive class. Positive data with the 'Trouser' and 'Bag' appeared in the training distribution and those with the 'Trouser', 'Bag', 'Sandal', 'Sneaker', and 'Ankle boot' appeared in the test distribution. For CIFAR10, we used the animal categories as the negative class and the vehicles as the positive class. Positive data with the 'Airplane' appeared in the training distribution and those with the 'Airplane', 'Automobile', 'Ship', and 'Truck' appeared in the test distribution as in the previous study [16] .For each dataset, we used 10 positive and 5, 000 unlabeled data in the training distribution and 5, 000 unlabeled data in the test distribution for training. In addition, we used 5 positive and 500 unlabeled data in the training distribution and 500 unlabeled data in the test distribution for validation. We used 1, 500 positive and 1, 500 negative data in the test distribution as test data for evaluation.There is no overlap between training, validation, and test datasets. The positive class-prior on the training distribution π tr was set to 0.1 and that on the training distribution π te was changed within {0.1, 0.2, 0.3}. For each case of the positive class-prior pairs, we conducted eight experiments while changing the random seeds and evaluated mean test AUC.We compared the proposed method (Ours) with nine neural network-based comparison methods: CE, nnPU [25] , puAUC [58] , PULA [21] , AnnPU, ApuAUC, APULA, CpuAUC, and PURR [16] .CE, nnPU, puAUC, and PULA use positive and unlabeled data in the training distribution for learning classifiers. These methods do not adapt to the test distribution. CE learns neural network parameters by minimizing the cross-entropy loss. This method naively treats unlabeled data as negative data. nnPU is a widely used PU learning method with a non-negative risk estimator. This method rewrites the classification risk with positive and unlabeled data and minimizes the rewritten non-negative classification risk to learn neural network parameters. PULA is a recent PU learning method that learns a neural network classifier with the label distribution alignment. nnPU and PULA used the class-prior in the training distribution as in the proposed method. puAUC learns a neural network classifier by maximizing the AUC that is calculated from positive and unlabeled data.AnnPU, ApuAUC, APULA, CpuAUC, and PURR use positive and unlabeled data in the training distribution and unlabeled data in the test distribution. AnnPU, ApuAUC, and APULA use unlabeled data in both the training and test distributions instead of those in the training distribution of nnPU, puAUC, and PULA respectively. AnnPU and APULA used the composite class-prior on the combined unlabeled data (i.e., π = πtrNtr+πteNte). In our preliminary experiments, the methods that use unlabeled data from the test distribution instead of data from both distributions were also evaluated. However, they performed worse than AnnPU, ApuAUC, and APULA. Thus, we omitted them. CpuAUC learns the invariant feature representations to mitigate the discrepancy of training and test distributions. Specifically, this method minimizes the negative AUC loss used in puAUC and the CORAL loss, which is a widely used in domain adaptation studies to match the second-order statistics of both distributions [53, 52] . PURR is a PU learning method for the positive distribution shift. This method rewrites the classification risk assuming the negative-conditional density does not change. To rewrite the risk, PURR requires the class-prior in the test distribution to be available, whereas the proposed method does not.For MNIST, FashionMNIST, and SVHN, all methods used a feed-forward neural network with three hidden layers. The number of hidden nodes was 128 and the ReLU activation function was used for each hidden layer. For CIFAR10, all methods used a convolutional neural network, which consisted of two convolutional blocks followed by a feed-forward neural network with two hidden layers. The first (second) convolutional block comprised a 6 (16) filter 5 × 5 convolution, the ReLU activation function, and a 2 × 2 max-pooling layer [38] . The numbers of nodes in the two hidden layers were 120 and 84, and the ReLU activation function was used. For PULA and APULA, margin ρ was selected from {0.1, 1, 10}. For CpuAUC, the CORAL loss was applied to the last hidden layer and its regularization weight was selected within {100, 10, 1, 0.1, 0.01}. For PULA, APULA, and CpuAUC, the best test results were reported. For all methods, we used the Adam optimizer [24] with a learning rate of 10 -4 . We set a mini-batch size M to 512, a positive mini-batch size P to 10, and the maximum number of epochs to 200. The loss on validation data was used for early stopping to avoid overfitting. All methods were implemented using Pytorch [43] and all experiments were conducted on a Linux server with an Intel Xeon CPU and A100 GPU.We evaluated the performance of the proposed method under the positive distribution shift with imbalanced datasets. Table 1 shows the average test AUCs of each method on the four datasets (their standard deviations are reported in Section E.4). The proposed method performed the best or comparably to it in almost all cases (11 out of 12 cases). The non-adaptation methods (CE, nnPU, puAUC, and PULA) that do not use data in the test distribution performed worse than the proposed method in many cases. This result indicates that adaptation using data in the test distribution is essential when there are distribution shifts. Although AnnPU, ApuAUC, and APULA used unlabeled data in the test distribution, they also did not work well. This is because their loss functions that naively use unlabeled data in the test distribution are not designed to maximize the performance on the test distribution. CpuAUC that learns invariant feature representations to mitigate the distribution gap tended to perform worse than the proposed method since the invariant features are not theoretically guaranteed to maximize the performance under the positive distribution shift. PURR performed well with MNIST and FashionMNIST since it is designed to adapt to the positive distribution shift and both datasets are relatively simple data where training is easy even in imbalanced data settings. We note that PURR used information on true class-priors on the test distribution, which is not used in the proposed method. By maximizing the AUC, the proposed method outperformed PURR with SVHN and CIFAR10 by a large margin, which are more complex than MNIST and FashionMNIST. As for the results of each class-prior pair, the proposed method performed better as the positive class-prior on the test distribution π te increased. Since unlabeled data X te has many positive data when π te is large, the proposed method more easily extracted information on positive data from X te and consequently performed well.We conducted an ablation study of the proposed method. When positive class-prior on the training distribution π tr is small, all unlabeled data X tr might be treated as negative data. In this case, existing methods can be directly applied to maximize the AUC on the test distribution by using unlabeled data in the test distribution X te and (pseudo) negative data X tr [45, 58, 6, 59 ]. This approach is equivalent to the proposed method without the second term (or π tr = 0) in Eq. ( 13), denoted as Ours w/o the second. Figure 2 shows the average test AUCs and the standard errors of the proposed method (Ours) and Ours w/o the second when changing the value of class-prior π tr within {0.01, 0.05, 0.1, 0.15, 0.2}. Ours consistently outperformed Ours w/o the second across all classpriors π tr except for SVHN. As class-prior π tr is increased, the difference between the two methods tended to become larger. This is because the assumption of Ours w/o the second (i.e., unlabeled data X tr are all negative data) is significantly broken when π tr is large. With SVHN, the performances of both methods did not differ. This result suggests that it is acceptable to consider all unlabeled data as negative data in some datasets. However, overall, these results show that the naive approximation is generally ineffective and our theoretically grounded loss function is important even with small π tr . We evaluated the proposed method with estimated classpriors on the training distribution since class-prior information might be unavailable in practice. To estimate the class-prior from positive and unlabeled training data, we used the kernel embedding-based class-prior estimation method [44] . Table 2 shows the results. Here, Ours w/ true and Ours w/ est are the proposed method using true class-prior π tr and estimated class-prior π est tr , respectively. We found that true class-prior π tr = 0.1 was accurately estimated from positive and unlabeled data in the training distribution. As a result, Ours w/ est and Ours w/ true mostly performed similarly. This result indicates that the proposed method works well even when class-prior information is unavailable in the training distribution, which is preferable in practice. Additionally, we investigated how the performance of the proposed method changes when there is a difference between true and input class-priors on the training distribution π tr . Here, the input class-prior is used in the loss function of Eq. ( 13) and the true class-prior is used for data generation. Figure 3 shows the average test AUCs and the standard errors of the proposed method (Ours) when changing the input class-priors. Interestingly, the performance tended to improve when values larger than the true class-prior (0.1) were inputted in most datasets. One reason for this result is that since there was some overlap between the types of positive data in the test and training distributions, strengthening the influence on labeled positive data by increasing the input class-prior in Eq. ( 13) improved the performance. To confirm this, we conducted additional experiments by excluding the types of positive data from the test distribution that were used in the training distribution (e.g., digits '7' and '9' were excluded from the test distribution in MNIST). Figure 4 shows the results. As expected, the performance did not improve even with large input class-priors.We investigated how the performance of the proposed method changes when the number of labeled positive data N p tr is increased. Figure 5 shows the average test AUCs and the standard errors with different numbers of labeled positive data. As expected, the performance of the proposed method tended to improve as N p tr increased. By using much information on labeled positive data in the training distribution, the proposed method can accurately estimate the AUC on the test distribution.We investigated the performance of the proposed method when changing the number of unlabeled data in the test distribution in Table 3 . As expected, the performance of the proposed method tended to increase as the number of unlabeled data in the test distribution N te increased. The proposed method tended to outperform puAUC, which does not use unlabeled data in the test distribution, even when N te = 500. Since many unlabeled data are often easy to collect, the proposed method is useful in practice.We evaluated the modified loss function in Eq. ( 16) for the cases where positive, negative, and unlabeled data in the training distribution and unlabeled data in the test distribution are available. Table 4 shows the results. Here, the number of labeled negative data was set to 20 and other settings are the same as those in the previous experiments. First, the proposed method with α = 0, which maximizes the AUC on the test distribution in Eq. ( 15) with labeled negative and test unlabeled data, did not work well. This would be because the number of positive data in test unlabeled data was small, and thus it was difficult to extract information on such positive data from unlabeled data by using a few labeled negative data only. The proposed method with α = 0.999, which uses positive, negative, and unlabeled data in the training distribution and unlabeled data in the test distribution, slightly tended to perform better than that with α = 1.0, which is equivalent to the proposed method described in Section 4.2. This result suggests that using a few labeled negative data is useful in our framework.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The code is proprietary. The datasets and experimental procedures are described in Section 5. We described the pseudocode of our method in Section 4. 
conclusion: 
 In this paper, we proposed a AUC maximization method under the positive distribution shift. We theoretically showed the AUC on the test distribution can be maximized by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution when assuming the positive distribution shift. The derived AUC has the advantage of being simple and easy to implement. The experiments with six real-world datasets demonstrated that the proposed method outperformed existing methods under the positive distribution shift with imbalanced data.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The details of the experimental settings are described in Section 5. 
limitations: 
 The proposed method performs well in the positive distribution shift. As stated in Section 1, we can expect that the positive distribution shift must occur in many real-world applications. However, the proposed method is not guaranteed to keep working well if different distribution shifts occur unexpectedly (i.e., the negative-conditional density changes). To deal with this problem, methods should be developed that can predict the types of the shift from given data.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We conducted a statistical significance test (paired t-test) in tables 1, 6, and 7.The standard deviations of Table 1 are reported in Table 9 . The standard errors are reported in all graphs.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discussed the potential social impacts in Section A. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does not release such high-risk models or datasets. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Datasets used in this paper are cited in Section 5. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Although the proposed method performed well, there is a possibility of misclassification in practice.In particular, the misclassification can lead to serious incidents in cases such as cyber/physical security and medical care, are typical examples of imbalanced data. Therefore, this method should be used as a support tool for humans to make a final decision.In this section, we discuss the loss (risk) corrections commonly used in ordinary PU learning to mitigate overfitting [25, 16] . Specifically, the loss correction is used to prevent empirical estimates of the risk from taking negative values even when the risk never takes negative, which often leads to overfitting.First, we provide a summary of our results: (1) The non-negative loss correction, which is widely used for ordinary PU learning [25, 16] , is not effective in our AUC maximization framework. This is because zero is not a tight lower bound of our expected loss. ( 2) When class-prior in the test distribution π te is available, we can derive a tighter lower bound. By using this for the loss correction, we can often enhance the performance of the proposed method. Below, we describe the details.Non-negative loss correction. In this section, for clarity, we consider minimizing the AUC risk,EQUATIONwhich is equivalent to maximize the AUC in Eq. ( 8) since AUC σ (s) = 1 -R σ (s). The minimum value of this risk is zero. Then, the corresponding loss for Eq. ( 13) becomesEQUATIONThis loss is derived from the AUC risk with unlabeled and negative test densities, i.e.,EQUATIONSince the AUC risk does not take negative values, the loss should not also take negative values. Therefore, to prevent negative values of its empirical estimate Lrisk (s), we can use the empirical loss with the absolute value function | Lrisk (s)| for the optimization. This type of correction is successfully used in PU learning [16] or other weakly supervised learning [34] . Table 5 shows the results of the proposed method with | Lrisk (s)| (Ours w/ nn). The results of Ours w/ nn and the proposed method without the loss correction (Ours) were almost identical and thus the non-negative loss correction was not effective in our framework unlike ordinary PU learning studies [25] .The reason of the ineffectiveness of Ours w/ nn is that the non-negative constraint is insufficient/weak in our loss (Eq. ( 18)). In fact, since p n te (x) is contained in p te (x)(= π te p p te (x) + (1 -π te )p n te (x)), the minimum value of the expected loss L risk (s) in Eq. ( 19) actually be greater than zero. Thus, the empirical loss Lrisk (s) could not be sufficiently constrained with the non-negativity; the performance of Ours w/ nn did not improve.Note that the non-negative correction is effective in ordinary PU learning studies [25, 16] . This is because the minimum value of the risk R - n (g) := E x∼p n (x) [ (g(x), -1)] used in the study [25] is zero, where is a point-wise loss and g is a decision function. This creates a gap in the effectiveness of the non-negative correction in our framework and the previous study [25] .Loss correction with class-prior in the test distribution π te . If class-prior π te is known, we can derive a tighter lower bound of our loss L risk (s). Specifically, we can obtainEQUATIONHere, we used p te (x) = π te p p te (x) + (1 -π te )p n te (x) and the fact that the AUC risk between the same densities is 1/2 as described in Section 4.2. As a result, we can use | Lrisk (s) -b| + b for the optimization. Our method with this correction (Our w/ b) tended to enhance the performance of it without the correction (Ours) in Table 5 . However, note that Ours has the strong advantage of not requiring the class-prior and performed better than existing methods. The proposed method allows us to sort data in score order. In practical use, this is beneficial in many situations. For example, in anomaly detection, experts or operators can check data with high scores within the cost they can spend or until anomalous data do not appear. In disease diagnosis, patients with higher scores can be prioritized for detailed examination. In recommendation systems, products can be presented to users in order of score. However, a classification threshold may be required to classify positive and negative data in some situations. Below, we explain how to determine the classification threshold.First, we have N te unlabeled data in the test distribution. When the true positive class-prior in the test distribution is π te , we can regard that N te π te positive data are included in the N te unlabeled data. Thus, when the N te unlabelled data are sorted by score, the top N te π te instances can be considered positive (assuming this scoring is accurate), and the score of the N te π te -th instance can be the boundary separating positive and negative. Thus, we can use the score of the N te π te -th instance as the threshold. Since true prior π te is unknown, we want to estimate it and use the estimated prior π est te instead. Next, we explain the procedure to obtain the estimated class-prior in the test distribution π est te . Specifically, we first extract negative data in unlabeled data from the training distribution by applying (off-the-shelf) PU learning to PU data in the training distribution. Since PU data and the class-prior in the training distribution are available in our setting, we can perform it. Then, since the negative distribution does not change in our setting, the extracted negative data can be regarded as negative data in the test distribution. We can estimate the class-prior in the test distribution by applying existing class-prior estimation methods [44, 63, 15] to the extracted negative and unlabeled data in the test distribution.In the main paper, we considered the positive distribution shift. Here, we consider a negative distribution shift, i.e., p p te (x) = p p tr (x) but p n te (x) = p n tr (x). In this case, when positive data in the training distribution and unlabeled data in the test distribution are available, we show that the negative distribution shift can be addressed by using existing AUC maximization methods [45, 58, 59 ]. Specifically, in this case, the AUC in the test distribution isEQUATION)where C is a constant, and in the second equal sign, we used that p n te (x) = 1 1-πte [p te (x)-π te p p te (x)], p p te (x) = p p tr (x), and the fact that the AUC between the same densities is a constant as described in Section 4.2. This derived AUC is equivalent to that in the existing studies [45, 58, 59] and can be maximized with test unlabeled and training positive data. Note that we do not require to know π te since it does not affect the optimization. In the comparison methods, PURR is the PU learning method under the positive distribution shift to minimize the test classification risk [16] . Since the classification risk is not suitable for imbalanced classification, we created a new method of PURR that uses the class balanced classification loss (the balanced class rate) [6] instead of the classification risk, called BPURR. Even in this case, BPURR requires the class-prior on the test distribution, which is usually difficult to know and is not required in the proposed method. Table 6 shows the mean test AUC with each class-prior on the test distribution. BPURR used the class-prior on the training distribution as that on the test distribution. When π te = 0.1, BPURR worked well since the used class-prior was equal to the true value. However, in other cases, BPURR performed worse than the proposed method in many cases.In the main paper, the positive class-prior on the training distribution π tr was set to 0.1. Here, we investigated how the performance of the proposed method changes when π tr is smaller. Table 7 shows the average test AUCs of each method on the four datasets. The proposed method performed the best or comparably to it in almost all cases (32 out of 36). As the positive class-prior on the training distribution π tr decreased, the proposed method tended to perform well. Since the amount of noise (positive data in the training distribution) in X tr decreases as π tr decreases, negative data (which is essential information to maximize the AUC on the test distribution) might be easy to extract from X tr when π tr is small.We evaluate the proposed method with two tabular datasets with distribution shifts (HReadmission and Hypertension) [14] . In HReadmission, the task is to predict the 30-day readmission of diabetic hospital patients. Each patient is represented by a 183-dimensional feature vector. In Hypertension, the task is a hypertension diagnosis for high-risk age. Each survey subject is represented by a 100-dimensional feature vector. We constructed the positive distribution shift using positive-shifted data (positive ood data) and non-shifted data (positive and negative training data). The experimental setting, such as the number of data, is the same as that in the main paper. Table 8 shows the result. The proposed method outperformed the others. This result shows that the proposed method is effective for tabular datasets.In the main paper, we omitted the standard deviations of results due to the limited space. Table 9 shows the mean test AUCs with the standard deviations.Although this paper focuses on maximizing the AUC since it is a representative evaluation metric for imbalanced data, other metrics for imbalanced data exist, such as the F1 score and the G-mean of TPR and TNR [17] . Therefore, we investigated whether the proposed method is also effective for the F1 score and the G-mean of TPR and TNR. Unlike the AUC, the F1 score and the G-mean of TPR and TNR require to determine the classification threshold. In this experiment, when N test data, in which the ratio of positive data is π te , are sorted by score of each method, we regard the top N π te data as positive and the remaining as negative. Tables 10 and 11 show the average test F1 scores and G-means of TPR and TNR with different test and training class-prior pairs, respectively. The proposed method tended to outperform the other methods although it maximizes the AUC. This result may imply that AUC maximization can also help to improve other evaluation metrics for imbalanced data. NeurIPS Paper Checklist• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.38th Conference on Neural Information Processing Systems (NeurIPS 2024).To be precise, the specific value of πte is not required, but we require that πte = 0, which will be discussed later. 


Decision: The addressed task is very relevant in practice. One of the reviewers was a bit unhappy that it only applies for AUC. It would indeed be useful to know about whether similar approaches could be developed for other objectives, but it is perfectly fine that the paper addresses only AUC. The appendix includes information about F1 as well, but this is just informative, not stating that the method should be applied to improve F1 when there is positive distribution shift.

The proposed method is quite simple and uses a trick that is somewhat similar to many other methods in transfer learning, but this is rather a strength not a weakness.

One of the reviewers (4KuN) had first lower grades for presentation, soundness, and contribution, but raised the overall rating from 4 to 5 after a lengthy discussion with the authors, stating eventually that "The textual explanation is fine".

One of the most positive reviewers (CKbq) claimed that "The authors do not fully compare their method with the latest ones" and gave 3 references. However, these papers did not address the shift on the positives as the current paper does. Regardless, in the rebuttal phase, the authors included one of those methods into the comparison. As expected, that method was outperformed by the method proposed in the current paper.

The reviewers had no major criticisms about the quality of presentation.

Therefore, I recommend the paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenge of maximizing the Area Under the Receiver Operating Characteristic Curve (AUC) in imbalanced binary classification problems where there is a positive distribution shift--this shift is where negative data remains constant, but positive data varies. A new method is proposed that utilizes labeled positive and unlabeled data from the training distribution, along with unlabeled data from the test distribution, to maximize the AUC effectively in the presence of such shifts.
strength_weakness: strength_weakness: Despite its strengths, this research primarily offers a simple proposal of a loss function, suggesting its contributions to the field might be limited. An expansion to include various metrics, such as F-1 and G-mean of TPR and TNR, which are also relevant for imbalanced data classification, could enrich this paper. Additionally, the experimental validation is somewhat restricted, utilizing only four datasets, all of which are image datasets. A more comprehensive evaluation using a broader range of datasets is necessary to fully assess the proposed loss function's effectiveness. Therefore, the reviewer believes that the contribution of this research may not be substantial enough for acceptance at a top-tier conference.
questions: questions: Q1: Please specify the scenarios where both class imbalance and positive distribution shift occur. Providing detailed examples will help readers grasp the practical significance of this research problem.

Q2: Why did the authors choose to conduct their experiments exclusively with image datasets? Are there any other real-world problems?

Q3: AUC maximization can be implemented not just as a loss function for neural networks, but across various machine learning methods. Why did you choose to focus on proposing a loss function?

Q4: The reviewer is not convinced that Lines 4-5 in Algorithm 1 sufficiently demonstrate the training process. There needs to be a more detailed and mathematical explanation of how model parameters are updated using the proposed loss function.
limitations: limitations: L1: An expansion to include various metrics, such as F-1 and G-mean of TPR and TNR, which are also relevant for imbalanced data classification, could enrich this paper.

L2: For extremely imbalanced cases, training difficulties are likely to arise. It is necessary to address how the proposed loss function can be effectively minimized in these scenarios.
paper_summary: paper_summary: This paper considers AUC maximization when the conditional probability distribution of the positive class changes in the test phase. To this end, the unbiased loss function is derived. The loss is approximated by positive and unlabeled data from training distribution, unlabeled data from test distribution, and class-prior of training distribution. In experiments, the proposed method outperformed the existing methods over the four benchmark datasets.
strength_weakness: strength_weakness: - Unlike the existing study [15, 42], the negative distribution is not considered.
- It lacks theoretical analyses of the proposed method.
- The extension of the proposed method is discussed but not evaluated in the experiments.
questions: questions: Is the proposed loss function unbiased to its supervised counterpart? 

It would be valuable if there were discussion or experimental results showing the effect of the number of unlabeled data from the test distribution. In some applications, collecting a lot of unlabeled data from the test distribution might be difficult. In such a situation, the experimental results would help practitioners understand how many samples are necessary to collect.

According to the literature, the non-negative risk estimator plays a crucial role in training deep neural networks. However, the proposed method does not mention the non-negativity of the risk estimator. Did the authors encounter that the risk estimator went to a large negative value in experiments? If not, what points in the proposed method avoid the issue?

Regarding the Extension in Section 4.4, it would be nice to cite the existing work.
limitations: limitations: The limitations are discussed in Appendix A.
paper_summary: paper_summary: The paper proposes a method for AUC maximization in binary classification problems under positive distribution shift. They introduce their method, which is simple and easy to implement/understand, and then show it works well in some experiments.
strength_weakness: strength_weakness: - The authors do not discuss how the classification threshold can be chosen in a practical situation under positive distribution shift.
questions: questions: How should the practitioner choose classification threshold after training their classifiers using your method?
limitations: limitations: The authors discuss limitations.
paper_summary: paper_summary: Due to a positive distribution shift, training and test distributions are not identical. However, existing AUC maximization methods don’t take it into account. To address this shift, this paper theoretically shows a new way to maximize the AUC on the test distribution by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution. Finally, four real-world datasets validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: - The effect of the proposed methods on MINST and Fashion MINST datasets is not significant, which is inconsistent with those on the other datasets. The authors don’t give any explanation.
- The authors do not fully compare their method with the latest ones. For example, 
  - Positive-Unlabeled Learning with Label Distribution Alignment. (TPAMI 2023)
  - Dist-PU: Positive-Unlabeled Learning from a Label Distribution Perspective. (CVPR 2022)
  - Positive-unlabeled learning using random forests via recursive greedy risk minimization. (NeurIPS 2022)
- All theoretical derivations are only based on the sigmoid surrogate loss. As far as I know, square loss is also popular. Can the theoretical results extend to the other losses?
- There are some typos. For example,
  - In line 105, “However, these all methods assume that” should be “However, all these methods assume that”.
questions: questions: please refer to Weakness
limitations: limitations: Yes

Comments: title: Thank you for your prompt response.
comment: Thank you very much for your response. We are pleased to learn that you have increased the rating to 5. We will ensure that the content, results, and figures from our discussions are incorporated into the final version. Your feedback has undoubtedly contributed to the improvement of our work.
We sincerely appreciate the time and effort you have devoted to reviewing and discussing our paper.
Thank you very much for your response. We are pleased to hear that you have increased the confidence level to 5. We sincerely appreciate the time and effort you have dedicated to reviewing and providing feedback on our paper.
Thank you for your answers. The textual explanation is fine.

I sincerely thank the authors for their humble and respectful responses during discussions.
If the answers, results, and figures are really included in the final version, my final score will be five.
Thanks for the rebuttal! The authors have clarified all of my concerns. Hence, I decide to keep my rating and increase my confidence.
Thank you for your response.

> If there are the results of the learning curve, like [22], the discussion in this thread will be more convincing.

We agree on this. However, according to the NeurIPS rebuttal instructional email, links to external sites are not allowed, so we cannot present figures like [22]. Thus, please let me continue with the textual explanation.

> The results in Fig. 2 in [22] show that even if we use early stopping, the test error of uPU at the best point sometimes has a gap from that of nnPU.

This is correct. The same thing was observed in our experiments: our method with the appropriate loss correction (Ours w/ b) tended to have a higher test AUC than it without the loss correction (Ours) at the best point.

As previously mentioned in this thread, the test AUC of our method with and without the non-negative correction did not differ significantly. This is because the minimum value of our expected loss ${\cal L} _{{\rm risk}} (s)$ is actually greater than zero, which is theoretically derived in the previous response. Thus, the non-negative correction is insufficient for our loss.

On the other hand, the non-negative correction was effective in ordinary PU learning [22] because the minimum value of the risk $R _{{\rm n}} ^{-} (g) := \mathbb{E} _{{\bf x} \sim p ^{\rm n} ({\bf x}) } [ \ell (g({\bf x}), -1)]$ used in [22] is zero, where $\ell$ is a point-wise loss and $g$ is a decision function.
This creates a gap in the effectiveness of the non-negative correction in our paper and in [22].

We would like to include the figures and this discussion in the final version.
Thank you for your answers.

The results in Fig. 2 in [22] show that even if we use early stopping, the test error of uPU at the best point sometimes has a gap from that of nnPU. If there are the results of the learning curve, like [22], the discussion in this thread will be more convincing.
We would like to kindly inform you that we have sent a response to your question. We may have posted it just as the OpenReview email notifications temporarily ceased, so we wanted to ensure you received it by notifying you again. We apologize for the inconvenience and appreciate your understanding.
Thank you for your reply.

We are pleased that you recognize the quality of the paper. We agree that the practical approach to determining the thresholds, as discussed, is an important point.

If you have any additional questions, please feel free to contact us.
Thank you again for taking the time to peer review and discuss the paper.
Thank you for your response.

We investigated the learning process for the training loss, validation loss, and test AUC. Here, we used ${\hat {\cal L} _{{\rm risk}}} (s)$ for the loss. Note that the validation loss is the value of ${\hat {\cal L} _{{\rm risk}}} (s)$ calculated with validation data (PU data in the training distribution and U data in the test distribution).

The training loss of our method became smaller than the lower bound $b$ as learning progressed and took negative on simple datasets (MNIST and Fashion MNIST). The validation loss (test AUC) tended to decrease (increase) initially but gradually increased (decreased) or stopped improving as learning progressed. These trends are consistent with ordinary PU learning, such as the study [22]. 

However, the validation loss and test AUC were well correlated, and our method was able to select good models by using early-stopping with the validation loss. This is one of the reasons for the good performance of our method without the loss correction. Early-stopping was effective in preventing overfitting in the PU learning context.
Note that all methods in our experiments used early-stopping.

We will include this result and discussion in the final version.
Thank you for your response. I think your paper is good, but discussing this issue we have been talking about will be an important part of your work.

I will keep my score as "weak accept".
Thak you for your answer. 

I still have questions about the discussion `The reason of the ineffectiveness of ...`. 
According to [22], even though the expected risk is non-negative, the empirical risk becomes negative.
The PU learning method without risk correction performed poorly [22]. I was wondering why this does not happen with the proposed method.
We appreciate your reply.

> The paper [42] discussed the negative distribution shift and the risk to adapt the shift.

Thank you for pointing this out. As you mentioned, the paper [42] discussed the negative distribution shift as well as the covariate shift.
Specifically, this paper considers the case, $p _{{\rm te}} ^{{\rm p}} ({\bf x}) = p _{{\rm tr}} ^{{\rm p}} ({\bf x}) $ but $p _{{\rm te}} ^{{\rm n}} ({\bf x}) \neq p _{{\rm tr}} ^{{\rm n}} ({\bf x}) $.
When positive and unlabeled data in the training distribution and unlabeled data in the test distribution are available, this paper showed that the original PU learning without re-weighting can deal with the negative distribution shift. This method requires class-prior $\pi _{{\rm te}}$.

We illustrate that the same can be said for AUCs easily. That is, negative distribution shifts can be addressed by the existing AUC maximization studies [41,54,55]. Specifically, the AUC in the test distribution is 

${\rm AUC}_{\sigma} (s) = \mathbb{E} _{{\bf x} ^{{\rm p}} \sim p ^{{\rm p}} _{{\rm te}} ({\bf x} )} \mathbb{E} _{{\bf x} ^{{\rm n}} \sim p ^{{\rm n}} _{{\rm te}} ({\bf x} )} \left[ f({\bf x} ^{{\rm p}}, {\bf x} ^{{\rm n}}) \right] = \frac{1}{1-\pi _{{\rm te}}} \mathbb{E} _{{\bf x} ^{{\rm p}} \sim p ^{{\rm p}} _{{\rm tr}} ({\bf x} )} \mathbb{E} _{{\bf x} \sim p _{{\rm te}} ({\bf x} )} \left[ f({\bf x} ^{{\rm p}}, {\bf x}) \right] + C$,

where $C$ is a constant, and in the second equal sign, we used $p^{{\rm n}} _{{\rm te}} ({\bf x} ) = \frac{1}{1-\pi _{{\rm te}}} [ p _{{\rm te}}({\bf x}) - \pi _{{\rm te}} p _{{\rm te}} ^{{\rm p}}({\bf x}) ]$ and $p _{{\rm te}} ^{{\rm p}} ({\bf x}) = p _{{\rm tr}} ^{{\rm p}} ({\bf x})$, and the fact that the AUC between the same densities is a constant as described in Lines 163--164. The form of the derived AUC is equivalent to that in existing studies [41,54,55] and can be maximized with test unlabeled and positive training data. Unlike ordinary PU learning, test class-prior $\pi _{{\rm te}}$ is not required since it does not affect the optimization.

We will include this discussion in the final version.

> It would be better to put Table 6 in the main body.

Thank you for the suggestion. We will put Table 6 in the main body of the final version.

> It is nice to mention this for clarification.

We agree on you. We will mention this in the final version for clarity.

> If there is an interpretation of why this happens, the contribution of this paper will become stronger.

The reason of the ineffectiveness of Ours w/ nn is that the non-negative constraint is insufficient/weak in our setting. Specifically, in Ours w/ nn, the non-negativity of the empirical loss, ${\hat {\cal L} _{{\rm risk}}} (s)$, is derived from the non-negativity of the corresponding expected loss, ${\cal L} _{{\rm risk}} (s) = (1-\pi _{{\rm tr}}) \mathbb{E} _{{\bf x} \sim p _{{\rm te}} ({\bf x} )} \mathbb{E} _{{\bf x} ^{{\rm n}} \sim p ^{{\rm n}} _{{\rm te}} ({\bf x} )} [ f({\bf x} ^{{\rm n}}, {\bf x}) ]$.

Here, $f({\bf x} ^{{\rm n}}, {\bf x}) := \sigma (s({\bf x} ^{{\rm n}}) - s({\bf x})) \geq 0$, and when $s({\bf x}) \gg s({\bf x} ^{{\rm n}})$, $f({\bf x} ^{{\rm n}}, {\bf x})$ becomes zero.
However, since $p _{{\rm te}} ^{{\rm n}} ({\bf x})$ is contained in $p _{{\rm te}} ({\bf x}) (= \pi _{{\rm te}} p _{{\rm te}} ^{{\rm p}} ({\bf x}) +  (1-\pi _{{\rm te}}) p _{{\rm te}} ^{{\rm n}} ({\bf x})$), the minimum value of the expected loss ${\cal L} _{{\rm risk}} (s)$ actually be greater than zero. Thus, the empirical loss could not be sufficiently constrained with the non-negativity, and thus, the performance did not improve.

Additional information is required to know better/tighter constraints of the expected loss ${\cal L} _{{\rm risk}} (s)$.
When class-prior in the test distribution $\pi _{{\rm te}}$ is known, the tighter constraint can be derived, ${\cal L} _{{\rm risk}}(s) \geq (1-\pi _{{\rm te}})(1-\pi _{{\rm tr}})/2 =: b > 0$, and using this (Ours w/ b) tends to enhance the performance of Ours in Table 3 in the PDF file. (Note that Ours has the advantage of not requiring $\pi _{{\rm te}}$ and performed better than the existing methods even without the loss correction).

In the final version, we will include this discussion of why non-negativity is ineffective in our method.
Thank you for your answer. 
Some of my questions and concerns are resolved.

> The method in [42] assumes the covariate shift

The paper [42] discussed the negative distribution shift and the risk to adapt the shift.

> We have already evaluated the extension of our method (Eq. 16) in Table 6 of Section C.4.

It would be better to put Table 6 in the main body.

> Eq. 12 is equivalent to the original AUC in Eq. 8.

It is nice to mention this for clarification.

> the results with and without the loss correction (Ours w/ nn and Ours) were almost identical

If there is an interpretation of why this happens, the contribution of this paper will become stronger.
Thank you for your valuable comment. We understood the intent of your question.

If the estimated score function perfectly ranks the positives and negatives in the unlabeled data and the estimated number of positive data in the unlabeled data with the estimated class-prior is correct, then using the threshold determined by our method leads to recall=100% and precision=100%. Thus, this threshold would also be best for other metrics defined by the combination of recall and precision, such as the F1 score. Our threshold determination method is based on these principles.

Of course, this is an ideal situation. In reality, an estimation error is involved. 
A small error may approximate recall=100% and precision=100%, but whether it works well in practice depends on the data and the application.
Thus, if we want to more rigorously determine the threshold for each application (recall-oriented, precision-oriented, etc.), we would need extra knowledge, such as some labeled data in the test distribution. This could be a limitation of our method, but as there are no labels in the test distribution, it is not easy to make a strict adjustment. 
(Note that this is not specific to our method but is expected in methods where only unlabelled data is available in the test distribution (e.g., unsupervised domain adaptation and unsupervised anomaly detection)).

Of course, this limitation is a case where a strict threshold is required. As you have acknowledged, there are many cases where just sorting data is sufficient.

In the final version, we will incorporate your suggestion by emphasizing that our method can sort data and has many real-world applications. Additionally, we will mention that there is a way to determine the threshold by using estimating class prior. Then, we will describe that if a strict threshold is required for the application, additional information, such as some labels, is desirable in the limitation section. Finally, we will describe that research on methods that maximize desired metrics, such as the combination of recall and precision, other than AUC under our problem setting is a future challenge.
I think the main issue here is the following. When deciding on a classification threshold, we try to balance some metrics (e.g., recall and precision), by optimizing some combination of them. It is not clear why setting the threshold in the way you propose will solve the problem for all kinds of scenarios (sometimes you value more precision and sometimes you value more recall, for example). The problem is that I am not sure you can do that under the things you know or can estimate. If this is possible, that'd be a good direction to explore in a future version, if not, I would point it as a limitation. Does that make sense?

It would be good to put more emphasis on the ordering aspect as well. In some circumstances, ordering is enough.
Thank you for your prompt response!
We are pleased to hear that you agree with sorting the data. We will address this point in the final version.

Here, we will explain the threshold in more detail.

Now, we have $N_{{\rm te}}$ unlabeled data in the test distribution. When the true positive class prior (the ratio of positive data in unlabeled data) in the test distribution is $\pi_{{\rm te}}$, we can regard that $N_{{\rm te}} \pi_{{\rm te}}$ positive data are included in the $N_{{\rm te}}$ unlabeled data. 
Thus, when the $N_{{\rm te}}$ unlabelled data are sorted by score, the top $N_{{\rm te}} \pi_{{\rm te}}$ instances can be considered positive (assuming this scoring is accurate), and the score of the $N_{{\rm te}} \pi_{{\rm te}}$-th instance can be the boundary separating positive and negative. Thus, we can use the score of the $N_{{\rm te}} \pi_{{\rm te}}$-th instance as the threshold. Since true prior $\pi_{{\rm te}}$ is unknown, we used the estimated prior $\pi _{{\rm te}}^{{\rm est}}$ instead.

Does this response meet your satisfaction?
If you have any further questions or suggestions, please do not hesitate to contact us.
Thank you for your reply!

1. I agree with you sorting the data; that's a good point;
2. I am a bit confused with your suggestion here. What is the purpose/rationale of setting the threshold to the $N_{te}\pi_{te}$-th score?