Positive-unlabeled AUC Maximization under Covariate Shift
Authors: Atsutoshi Kumagai
Tomoharu Iwata
Hiroshi Takahashi
Taishi Nishiyama
Kazuki Adachi
Yasuhiro Fujiwara
Venue: ICML 2025 poster (2025)
Keywords: Distribution shift
Covariate shift
AUC
PU learning
Imbalanced data
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We propose a method for maximizing the AUC under covariate shift by using positive and unlabeled data in the training distribution and unlabeled data in the test distribution.

Abstract: Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced binary classification tasks. Existing AUC maximization methods typically assume that training and test distributions are identical. However, this assumption is often violated due to {\it a covariate shift}, where the input distribution can vary but the conditional distribution of the class label given the input remains unchanged. The importance weighting is a common approach to the covariate shift, which minimizes the test risk with importance-weighted training data. However, it cannot maximize the AUC. In this paper, to achieve this, we theoretically derive two estimators of the test AUC risk under the covariate shift by using positive and unlabeled (PU) data in the training distribution and unlabeled data in the test distribution. Our first estimator is calculated from importance-weighted PU data in the training distribution, and the second one is calculated from importance-weighted positive data in the training distribution and unlabeled data in the test distribution. We train classifiers by minimizing a weighted sum of the two AUC risk estimators that approximates the test AUC risk. Unlike the existing importance weighting, our method does not require negative labels and class-priors. We show the effectiveness of our method with six real-world datasets.

Positive-unlabeled AUC Maximization under Covariate Shift
Atsutoshi Kumagai
Tomoharu Iwata
Hiroshi Takahashi
Taishi Nishiyama
Kazuki Adachi" 12 Yasuhiro Fujiwara'
Abstract
much smaller than negative data (Johnson & Khoshgoftaar, 2019). In this situation. classification accuracy, which is the standard performance measure for ordinary classification, is not a suitable measure (Ueda & Fujino, 2018; Yang & Ying, 2022). Instead, the area under the receiver operating characteristic curve (AUC) is widely used (Bradley, 1997; Huang Ling, 2005; McDermott et al., 2024). The AUC the probability that classifier will rank a randomly chosen positive instance higher than randomly chosen negative one (Yang & Ying, 2022). Thanks to the nature of the ranking, the AUC can adequately measure the classifier's performance even with imbalanced data. By maximizing the AUC, we can learn accurate classifiers from imbalanced data (Brefeld et al., 2005; Yang & Ying. 2022; Ying et al., 2016; Liu al., 2020; Yuan et al., 2021a).
Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced binary classification tasks. Exist- ing AUC maximization methods typically assume that training and test distributions are identical. However, this assumption is often violated due to a covariate shift, where the input distribution can vary but the conditional distribution of the class label given the input remains unchanged. The importance weighting common approach the covariate shift, which minimizes the test risk with seprrtance-weigheed training data. How- ever, it cannot maximize the AUC. In this paper, to achieve this, we theoretically derive two esti- mators of the test AUC risk under the covariate shift by using positive and unlabeled (PU) data in the training distribution and unlabeled data in the test distribution. Our first estimator is calculated from amportance-weighter PU data in the train- ing distribution, and the second one is calculated from apportance-weiighed positive data in the training distribution and unlabeled data in the test distribution. We train classifiers by minimizing a weighted sum of the two AUC risk estimators that is approximately equivalent to the test AUC risk. Unlike the existing importance weighting, our method does not require negative labels and class- priors. We experimentally show the effectiveness of our method with six real-world datasets.
Existing AUC maximization methods usually assume that training and test distributions are identical. However, this assumption is often violated in practice due to distribution shifts. This paper considers a covariate shift, where the in- put distribution can vary but the conditional distribution of the class label given the input remains unchanged between training and test stages (Shimodaira, 2000). The covariate shift is the most common distribution shift (He al., 2023) and often occurs in imbalanced classification tasks. For example, in medical care, the distribution of patients (input instances) can vary due to the differences in hospitals and measurement instruments, even when the conditional dis- tribution of the class label given the input does not change (Matsui et al., 2019). In cyber security, attackers rapidly generate new attacks (input instances), and thus the input distribution can vary over time (Kumagai & Iwata, 2016).
1. Introduction
When labeled positive and negative data in the training distribution and unlabeled data in the test distribution are available, the covariate shift on ordinary classification can be alleviated using the widely used importance weighting framework (Sugiyama et al., 2012; Lu al., 2022). It first estimates importance weights that are the ratio between training and test input densities and then learns classifiers by minimizing the importance weighted empirical training risk that is approximately equivalent to the test risk. How- ever, it is not designed for AUC maximization. In addition, labeled negative data in the training distribution are often difficult collect in some applications. For example, in
In many real-world binary classification tasks such as cyber security (Mirsky al., 2018; Bagui & Li, 2021), medical care (Yang et al., 2021), and product inspection (Park et al., 2016), class-imbalance often occurs where positive data is
'NTT Corporation, Japan 'Yokohama National Univer- sity, Japan. Correspondence to: Atsutoshi Kumagai <atsu- toshi Kislgiiumagai@ee eatiii@tttt......
-eeightee
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Positive-unlabeled AUC Maximization under Covariate Shift
cyber security, although some malicious data (positive data)
and unlabeled data in the test distribution.
can be collected from public sources such blocklists, be- nign data (negative data) are often unavailable due to privacy reasons, and identifying benign data within given unlabeled data requires a high level of expertise (Mirsky et al., 2018; 8 Sharafaldin et al., 2018). Although an importance weight- ing method (Sakai& & Shimizu, 2019) does not use labeled negative data exceptionally, it cannot maximize the AUC.
We theoretically derive two estimators of the test AUC risk under our problem setting. which can be used for classifier learning without class-priors.
We develop a dynamic approach for the importance weighting with the derived AUC risk estimators.
We experimentally show that the proposed method outperforms various existing methods with real-world datasets.
In this paper, we propose method for maximizing the AUC under the covariate shift by using labeled positive and un- labeled (PU) data in the training distribution and unlabeled data in the test distribution. On the basis of the importance weighting, we theoretically derive two estimators of the AUC risk on the test distribution. The first one calcu- lated from wepigte PU data in the training distribution. While this estimator is effective, unlabeled data in the test distribution are only used for estimating importance weights and are not directly used in classifier learning, even if they have useful information. This draw- back is common in ordinary importance weighting methods (Sugiyama & Kawanabe, 2012; Fang et al., 2020; Sakai & Shimizu, 2019). The second estimator is calculated from importance-Vee weighted positive data in the training distribu- tion and unlabeled data in the test distribution. This estima- tor directly uses unlabeled data in the test distribution for classifier learning. Our loss function for classifier learning is a weighted sum of these two AUC risk estimators that is also approximately equivalent to the test AUC risk. By using this loss, we can directly use all available data in our setting for classifier learning. Moreover, unlike the existing method (Sakai & Shimizu, 2019), our method does not re- quire class-priors for training, which is beneficial since they are generally difficult or impossible to estimate (Zhao et al., 2023; Yao et al., 2021).
2. Related Work
Many AUC maximization methods have been proposed (Brefeld et al., 2005; Yang & Ying, 2022; Ying et al., 6;;6; Liu et al., 2020; Yuan et al., 2021a). As reported in pre- vious studies (Yuan et al., 202 Fujino & Ueda, 2016; Wang et al., 2023), they often outperform other methods for imbalanced classification such as class balanced loss (Charoenphakdee et al., 2019), focal loss (Lin et al., 2017), or sampling- bassd methods (Chawla et al., 2002; Menardi & Torelli, 2014). However, these AUC maximization methods require labeled positive and negative data and assume that the training and test distributions are identical. Thus, they are inappropriate for our problem setting where there are no labeled negative data and the covariate shift occurs.
weeigta
Covariate shift adaptation methods attempt to learn accurate models under the covariate shift by using unlabeled data in the test distribution and labeled data in the training distri- bution (Pan & Yang, 2009; Sugiyama & Kawanabe, 2012). The importance weighting is a representative approach for the covariate shift (Sugiyama & Kawanabe, 2012). This approach first estimates importance weights and then mini- mizes the importance- -eeighteeee empirical training risk. Al- though this two-step approach is theoretically sound, it often does not work well for complex models or data (Fang et al., 2020; 2023). To overcome this difficulty, Fang et al. (2020; 2023) have recently proposed a dynamic approach that iter- ates importance weight estimation and classifier learning. which enables the importance weighting to work well in such difficult cases. Our method also uses the dynamic approach. Another approach is to learn invariant feature representations by minimizing the discrepancy of the fea- tures between the training and test distributions (Long et al., 2015; Sun et al., 2017: 2016; Ganin & Lempitsky, 2015; Kumagai & Iwata, 2019; Shen al., 2018). Although this approach is promising, often deteriorates the performance since it does not explicitly minimize the test risk (Zhao et al., 2019; Kumagai et al., 2024). These existing methods usually minimize the classification risk (or negative clas- sification accuracy), which is an inappropriate metric for imbalanced data. One distribution adaptation method has been proposed to maximize the AUC by learning invariant
Although we can learn classifiers by minimizing the loss, the importance weights are difficult to estimate, especially when using complex models such as neural networks or complex data such as image data (Fang et al., 2020; Kato & Teshima, 2021; Rhodes et al., 2020). Thus, the above two-step importance weighting often does not work well in such cases. To deal with this problem, following recent works (Fang et al., 2020; 2023), we use dynamic approach that iterates the importance weight estimation and classifier learning while sharing a neural network for feature extrac- tion. By training the shared feature extractor with simpler classifier learning, the importance weights can be estimated more easily; the classifier learning can be performed without biases by using the estimated importance weights.
Our main contributions are as follows:
We propose a novel and practical problem setting, where the aim is to maximize the AUC under the co- variate shift with PU data in the training distribution
2

---

Positive-unlabeled AUC Maximization under Covariate Shift
feature representations (Yang et al.. 2023). However, all these methods including the method (Yang et al., 2023) re- quire positive and negative data in the training distribution, which are unavailable in our problem setting.
instance being ranked before a randomly drawn negative instance (Yang & Ying, 2022). Specifically, the AUC with score function can be formulated as
AUC(s) = upp (xx)) Ex -p" [1x((x)) 2 s(x))] = 1 - Ex (xx Ex [1(((x)) < s(x")] (1)
PU learning methods aim to learn binary classifiers by us- ing only PU data (Bekker & Davis, 2020). Our work is closely related PU learning since it assumes PU data in the training distribution. A representative approach for PU learning is the empirical risk minimization approach, which rewrites the empirical classification risk by using only PU data (Du Plessis et al., 2015; Kiryo et al., 2017; Sugiyama et al., 2022; Jiang et al., 2023). Although they are effec- tive, they cannot maximize the AUC. Recent studies have shown that the AUC can be maximized from only PU data by using the techniques of PU learning (Sakai et al., 2018; Xie & Li, 2018; Charoenphakdee et al., 2019; Xie et al., 2024). However, these methods assume that training and test distributions are the same.
where [(2) is the indicator function that outputs 1 if is true and 0 otherwise, and E is the expectation. Maximizing the AUC is equivalent to minimizing the AUC risk,
R(s)= == Exx (x) [T(((P)) < s(x")) (2)
Since the gradient of indicator function is zero everywhere except for the origin, the AUC risk cannot be minimized via gradient descent methods. To avoid this, the following smoothed AUC risk is often used by replacing the indicator function with sigmoid function (z) = 1/(1 + exp(-=)) (Iwata & Yamanaka, 2019; Kumagai et al., 2019; 2024):
Several PU learning methods for distribution shift have been proposed that use unlabeled data in the test distribution and PU data in the training distribution (Sakai & Shimizu, 2019; Kumagai al., 2024; Nakajima & Sugiyama, 2023). Nakajima and Sugiyama (2023) considers class-prior shit, where the class-prior can vary, but their method cannot treat the covariate shift and maximize the AUC. Although Sakai and Shimizu (2019) consider the covariate shift, they treat the classification risk and cannot maximize the AUC. Due to the pairwise formulation of the AUC, this method using ordinary instance-wise loss functions cannot be straightfor- wardly applied to the AUC. Kumagai et al. (2024) maximize the AUC under positive distribution shift, where negative- conditional density does not change but positive-conditiona density can vary. This method, however, cannot deal with the covariate shift that often occurs in practice. Additionally, these methods assume that the class-prior on the training distribution is available, which is generally difficult to esti- mate (Yao et al., 2021). In contrast, the proposed method does not require it.
R. (s)==Ex Exp Exs (o(=<(x")+<<("))
(3)
Given NP positive instances {x ...,..., x drawn from pd p*(x) and N negative instances {x",,.. .... from p" (x), the empirical estimator of the smoothed AUC risk is calculated as
} drawn
NP
N"M
1
(s(
[a(-s(x))) + + (xx))) (4)
NPN
m=11
By minimizing this empirical smoothed AUC risk w.r.t. the parameters of s, we obtain good score functions to maximize the AUC when the training and test distributions are identical (Yang & Ying, 2022).
4. Proposed Method
In this section, we first explain our problem setting (subsec- tion 4.1.. Then, we theoretically derive two estimators of the AUC risk on the test distribution under the covariate shift and define our loss function for classifier learning on the ba- sis of the derived two AUC risk estimators (subsection 4.2). After describing the importance weight estimation method (subsection 4.3), we finally explain our training procedure (subsection 4.4).
3. Preliminary
We briefly explain AUC maximization. Let input instance x E X and its class label y E +1) be equipped with probability density p(x, y). where +1 and -1 mean posi- tive and negative class, respectively. px(x) : p(xly = +1) and px)) == p(x|y= 1) are the conditional probability densities of positive and negative classes, respectively. Let 8 X' + RR be a score function that outputs the positivity of an input instance. The classifier is defined by the score function with threshold t: y = sign(s(x) - t), where sign a sign function.
4.1. Problem Setting
Suppose that we are given a set of positive instances XP and a set of unlabeled instances X drawn from the training distribution:
XP = (xx (xpmnnn
NE ~ pp((x) ;= (xe(x = +1),
(5)
The AUC is the probability of randomly drawn positive
X1r = {xuen) ~ px(()) = (x) + (1 - न.. )r (x),
Ner
(6)
3

---

Positive-unlabeldd AUC Maximization under Covariate Shift
where Pt (x) is the marginal density of the training distribu- tion, (x) and Pte (x) = (xly 1 are positive- and negative-conditiona densities of the training distribution, respectively, and Ter == (1 = +1) is the positive class- prior. We also suppose that a set of unlabeled instances Xte drawn from the test distribution is given:
where we used the Bayes' theorem and the assumption of the covariate shift (Eq. (8) in the first and third equali-
ties, and w(x) == P=x(x] is referred as the importance weight. Similarly, negative-conditional density Pic (x) can be represented as
(x)
1 1 Tir
Xie = [Xiemm Nie ~ Pte(x)= = Te + -1-
(1-TTe ) (x), (7)
Px)()) Pir (x))((x)
(11)
Aec
By substituting Eqs. (10) and (11) into Eq. (9), we can obtain
where (x) == (xy(x +1), (x) == Pee (xly
-1), and The := Py = +1). As shown later, we do not need to know specific values of Tur and Tie to maximize the
(g) = Tex ( -tc))
Tee nee)
AUC! !, which is beneficial in practice.
x E Exp (xx) Bxm (xx) ( x")] (xxx)x))) (12)
We consider the covariate shift between the training and test distributions, where the marginal density can vary but the conditional density of the class label given the input instance remains unchanged,
From the definition of marginal density Ptr in Eq. (6), regative-conditioaa density pt, (x) can be expressed as
1 1 1 -tr
Plr (x)
[p - TTPP,
(13)
Pur (x) # (x), Ptr (y|x) = Pte (y)x)) =: p())x)) (8)
By substituting Eq. (13) into Eq. (12), we can obtain
This situation often occurs in imbalanced classification tasks as described in Section 1. Our goal is to learn the score function X IR that can maximize the AUC on the test distribution by using X U Xxr U Ute- In the following, C' represents any constant that does not depend on models to be learned (e.g., score function s).
(s)
Atr
Exp Sar (xc) Bx
x)]
Tue Tte)
2
Air
E Rxp (xx) ppp (x((x))((*)) [xx", xP)
Tte (1 Tee)
(14)
4.2. Importance-weighter AUC Risks
Here, since (2) + o(-2) = for all 2 E R, the second term of Eq. (14) becomes
In this subsection, we theoretically derive two estimators of the AUC risk on the test distribution under the covariate shift. The objective function to be minimized is the following smoothed AUC risk on the test distribution,
Exe (x) Exp (o(x))=<(*))))))))) xP)]
Exrwpt (x) Exr [w(xP) yw (xP) C, (15)
(s) == Exprpli (x) E Ex" [f(x",x")
(9)
where C' represents constant that does not depend on Therefore, the AUC risk on the test distribution in Eq. (14)
where we set f(XP, x") (-x((x)) + s)x")). Since this AUC risk depends on pie (x) and ple (x), it seems to be impossible to calculate directly in our setting. However, by using the importance weighting framework, this AUC risk can be calculated as described below.
can be represented as
Re = Tea
Tir
- (1-
x Exe (xx) (xx (x) (u(x))u(x))((x) x)] + C. (16)
First AUC Risk Estimator We derive the AUC risk es- timator that is calculated from PU data in the training dis- tribution. First, positive-conditiona density (x) can be rewritten as
Since this AUC risk depends on positive-conddtional training density (x) and marginal training density Px)(x), we can
(x)
obtain the following empirical AUC risk estimator,
Ple (x) p(y + 1/x))
Rdc((s) (s) = Teo ((m No.No
Mtr
Tto
NP Nor
p(y = + 1 1|xx) ( (x) (x) Ter
(w f Xnrmm]]
x + C.
(x) Tte
= (x) Pie (x) Tue
= pP, (x))x()))))
Aur
(10)
(17)
Per (x) Tte
Tte
In this estimator, training instances with higher importance weights have a greater influence. The estimation method for importance weight wxx) described in subsection 4.3.
We only require Wer, Tse E (0, 1) evade zero-division when deriving our AUC risk estimators in subsection 4.2.
4

---

Positive-unlabeled AUC Maximization under Covariate Shift
Second AUC Risk Estimator In Eq. (17), unlabeled data in the test distribution Xte are not directly used even if they have useful information for learning s, although they are used for estimating importance weights as described later. This drawback is common in existing importance weighting methods (Sugiyama & Kawanabe, 2012; Fang et al., 2020; Sakai & Shimizu, 2019). To deal with this problem, we de- rive another AUC risk estimator that is calculated from posi- tive data in the training distribution and unlabeled data in the test distribution. Specifically, since wx) phe (x) (x), the equation (x) (wx())g(e [x)])]]] = (x)) [x(x)] is satisfied for any function g. Therefore, Eq. (16) can be rewritten as
and l is a constant), we can derive the loss function of the same form in Eq. (20). This is because the second term in Eq. (14) also becomes constant. The symmetric functions include many popular functions such as sigmoid, ramp, and unhinged functions (Charoenphakdee et al., 2019).
4.3. Importance Weight Estimation
In Eq. (20), importance weight w(x) pte(X) is unknown. A common way for estimating importance weights is to use density-ratin estimation methods that directly estimate the ratio between training and test densities from data without density estimation (Sugiyama et al., 2012; Kanamori et al., 2009; Huang et al., 2006). However, they are known to be unstable since w(x) is unbounded, i.e., it takes extremely larger values around low-density region of the denominator (Yamada et al., 2013). To alleviate this problem, we use the following relative density-ratio as the importance weight,
(x(
(2((s = Tio(l Tie)
Ttr
x Exp (x) (x)) [xx(x)) x)] + C.
(18)
This AUC risk depends on positive-conditional training den- sity per (x) and marginal test density (x). Thus, we can obtain the following empirical AUC risk estimator,
(x) Wa wx(x) == Oxto(
(21)
a (x)
(s)
Uer
where a E [0. 1] is hyperparameter (Yamada et al., 2013). (x)) is bounded above by 1/a, and it is equivalent to
Tee - T(e) No..No
NP
Nen
w(x) when a O. Thus, Wa (x) is a bounded extension of
[f (x Xte,m) +C
(19)
w(x). The importance weighting with the relative density- ratio has been reported to perform excellently in various data (Yamada et al., 2013; Sakai & Shimizu, 2019).
1==1
Since Eqs. (17) and (19) do not explicitly depend on Xte and X1, except for importance weights, respectively, we consider using both simultancously for learning s.
Let rxx) E [o, 1/a] be a model such as a neural network for estimating w.. (x). To learn parameters of r, we minimize the expected squared error between true importance weight wa (x) and r(x) as in the previous studies (Yamada et al., 2013; Kanamori et al., 2009):
Our Loss Function for Classifier Learning Our loss function for learning s is a weighted sum of two estimators of the AUC risk in Eqs. (17) and (19),
((r = -Epo(x) [wa(x)) - r(x))
Ru (s)= = BR;; (s) + (1- B)R (1-B1 Tea [1 Tto)
Tir
= =Epc(x) [ar(x)²- - 2r(x)]
+ (1 - a: (x)()) [r(xx)]] + +C,
(22)
NP
Nur
B
[uxxfr,nn Yw (Xer,m) (xfr, X]rmm]]
NP N
where (x) = (1 (x) and C' is constant term that does not depend on r. The empirical estimate of J can be obtained as
tr
NP
Nic
1- 3
[ (xxxeen) m x) +C, (20)
NP 1r
Nic
1
J(r)
[ar(Xte,n) 2r 2r(xte,))
Nie
where 3 E [0, 1] is a weighting hyperparameter. This loss directly uses all data X.. UXr U X for learning s. Since coefficient [I- and constant C' do not affect the opti- mization for s, we can safely ignore them for training. This loss without the coefficient does not depend on class-priors Ter and Tie, which is beneficial in practice since they are generally difficult or impossible to estimate (Zhao et al., 2023). Note that although we use the sigmoid function in the AUC risk in Eq. (3), when we use symmetric functions (i.e., function a satisfying (2) (-2) = & for any z E TR.
n=1
1 No.
(xxer,)) ) CC. C.
(23)
N₂
4.4. Training Procedure
To calculate R5c in Eq. (20), the standard approach is to first estimate the importance weights and then use them to calculate the importance- Sugiyama et al., 2012; Kanamori et al., 2009). However, in
risk (Yamada et al., 2013;
5

---

Positive-unlabeldd AUC Maximization under Covariate Shift
this two-step approach, errors in the importance weight es- timation propagate to the subsequent importance-weighern risk calculation, which degrades the performance of the learned classifiers (Fang et al., 2020; Zhang et al., 2020). To alleviate this, Fang et al. (2020; 2023) recently proposed a dynamic approach that iterates between the importance weight estimation and classifier learning for ordinary super- vised learning.
Algorithm 1 Training procedure of the proposed method
Require: PU data in the training distribution and unlabeled
data in the test distribution X U Xue U
size M, positive mini-batch size P, relative parameter
ar and weighting parameter B.
Ensure: Parameters of neural networks h, u, and v.
1: repeat
Sample unlabeled data with size M form Xtr U Xte 3: Sample positive data with size P form X [Importance weight estimation]
The proposed method follows this dynamic approach. Specifically, we use the following neural networks for mod- cling the score function and importance weight,
4: Calculate the loss in Eq. (23) on the sampled data
5: Update parameters of ' with the gradient of the loss fixing feature extractor h [Classifier learning)
s(x) == (h(x))), r(xx == v(h(x)),
(24)
where h : X' + RK u R + R. and v : RR + are neural networks for feature extraction, score function, and importance weights, respectively. By sharing feature extrac- tor h, we can effectively perform the importance weighting.
Calculate the loss in Eq. (20) on the sampled data with current importance weights
Update parameters of u and h with the gradient of the loss fixing the importance weights
Algorithm shows the training procedure of the proposed method with stochastic gradient descent methods. We first randomly sample PU data from the training distribution and unlabeled data in the test distribution (Lines 2-3). Then, we calculate the loss in Eq. (23) for the importance weight estimation (Line 4) and update parameters of with the gradient of the loss fixing feature extractor h (Line 5). We fixed h to avoid overfitting as in (Fang et al., 2020). By using the estimated importance weights, we calculate the loss in Eq. (20) for classifier learning (Line 6). We update parameters of both u and h by using the gradient of the loss (Line 7). In this step, we fixed the importance weights to avoid learning meaningless model, r r(x) 0 for all x.
8: until End condition is satisfied;
selection bias in the training and testing distribution follow- ing a previous study (Aminian et al., 2022). Specifically, for MNIST and SVHN, we used even digits as negative and odd digits as positive. For the training distribution, 90% of data were selected from the digits 0, 1, 2, and 3, and 10% of data were selected from the remaining digits. We reversed this ratio for the test distribution: 10% of data were selected from the digits 0, 1, 2, and 3, and 90% of data were selected from the remaining digits. For FashionMNIST, following the study (Xie et al., 2024; Kumagai et al., 2024), we used upper garments (0, 2, 3, 4, and 6) as negative and the others as positive, where numbers in parentheses represent class labels. Similar to MNIST and SVHN for the training distri- bution, 90% of data had the class labels 0, 2, and 5, and 10% of data had the remaining class labels. We reversed this ratio for the test distribution. For CIFAR10. we used the animal categories (2, 3, 4, 5, 6, and 7) as negative and the vehicles as positive. For the training distribution, 90% of data had the class labels 0, 1, 3, and 4, and 10% of data had the remaining class labels. We reversed this ratio for the test distribution.
5. Experiments
5.1. Data
We used four real-world datasets in the main paper: MNIST (LeCun et al., 1998), MaShT (Xiao et al., 2017), SVHN (Netzer et al., 2011), and CIFARIO (Krizhevsky et al., 2009). These datasets have been commonly used in PU learning or distribution adaptation studies (Kumagai et al., 2024; Fang et al., 2020; Xie et al., 2024; Kiryo al., 2017; Jiang et al., 2023). MNIST consists of hand-written images of 10 digits. Each image is represented by gray-scale with 28x x 28 pixels. FashionMNIST consists of images of 10 fashion categories where each image is represented by gray scale with 28 x 28 pixels. SVHN consists of 32 x 32 RGB images of 10 printed digits clipped from photographs of house number plates. We converted SVHN into gray- scale for simplicity. CIFAR10 consists of 32 x 32 RGB images of 10 animal and vehicle categories. In Appendix D.3, we also used real-world tabular datasets: epsilon and Hreadmission (Gardner et al., 2023).
For training in each dataset, we used 50 positive and 3,000 unlabeled data in the training distribution and 3,000 unla- beled data in the test distribution. Additionally, we used 20 positive and 250 unlabeled data in the training distribution and 250 unlabeled data in the test distribution for validation. For unlabeled data in training and validation, we changed class-prior := Ttr The within (0.01, 0.05, 0.1). We used 3,000 data in the test distribution as test data for evalu- ation. Training, validation, and test datasets did not overlap. We conducted 10 experiments for each positive class-pric prior changing the random seeds and evaluated mean test AUC.
To create the situation of the covariate shift, we imposed a
6

---

Positive-unlabeled AUC Maximization under Covariate Shift
5.2. Comparison Methods
performance as in previous studies (Lu et al., 2020; Ham- moudeh & Lowd, 2020). trPU, tePU, PAUC, and CPU used information on class-prior of the training distribution Tir for training although the proposed method does not require it.
We compared the proposed method with eight methods: non-negative PU learning method with PU data in the train- ing distribution (trPU) (Kiryo et al., 2017), non-negative PU learning method with positive data in the training and unlabeled data in the test distributions (tePU), AUC maxi- mization method with PU data in the training distribution (trAUC) (Xie & Li, 2018; Xie et al., 2024), AUC maxi- mization method with positive data in the training and unla- beled data in the test distributions (teAUC), AUC maximiza- tion method with PU data in the training distribution and unlabeled data in the test distribution (trteAUC), unsuper- vised domain adaptation method with AUC maximization (UDAUC), AUC maximization method for positive distribu- tion shift (PAUC) (Kumagai et al., 2024), and covariate shift adaptation method for PU learning (CPU) (Sakai & Shimizu, 2019). All methods including the proposed method used neural networks for modeling classifiers.
5.3. Settings
For all methods, the sigmoid loss was used instead of indi- cator functions as in the previous studies (Kiryo et al., 2017; Kumagai et al., 2024). The empirical risk with validation data was used to select hyperparameters and carly-stopping to mitigate overfitting. All methods were implemented us- ing Pytorch (Paszke al., 2017), and all experiments were conducted on a Linux server with Intel Xeon CPU and A100 GPU. The details of neural network architectures and hyperparameters are described in Appendixes B and C.
5.4. Results
Table 1 shows the average test AUCs of each method on the four class-imbalanced datasets. Full results including the standard deviations are described in Appendix D.6. The proposed method performed the best or comparably to it in all cases. trPU and tePU did not work well since they cannot deal with the dlass-iballane.. trAUC and teAUC performed better than trPU and tePU, which indicates that the AUC maximization is appropriate for the PAUC performed poorly because the shift assumption used in PAUC (i.e., positive distribution shift) is different to the covariate shift and it struggles with small Tte due to its re- liance on extracting positive data from unlabeled test data as described in the paper (Kumagai et al., 2024). Although CPU is a method for covariate shift, it did not work well because it does not consider the dlass-mballanee. Although UDAUC that learns invariant feature representations to miti- gate the distribution gap performed better than other distri- bution shift methods (PAUC and CPU), it performed worse than the proposed method. This result suggests that the ap- proach of learning invariant features was often sub-optimal for the covariate shift problem. The proposed method per- formed better than tUt, trAUC, and teAUC, which can be regarded as the special cases of the proposed method (i.e., non-importance weighting versions of our method) as described in Section 5.2. This result indicates that the im- portance weighting with AUC maximization is useful for the dlass-imbllanced data under the covariate shift. As for the results of each class-prior π. the performance of AUC- based methods (the proposed method, trAUC, teAUC, and trteAUC) tended to improve as the value of π decreased. This is because they essentially use loss functions of the form, (xxx Exxp(x) (xP,, x)], where pP(x)) and p(x) are positive and marginal densities. When и is small, p(x) can be regarded as negative density (x).. In this case, the above loss function becomes the original AUC risk. Thus,
tPPU learns the neural network by minimizing the non- negative PU risk with PU data in the training distribution. This method does not consider the class- -tas--ibalaane. trAUC learns the neural network by maximizing the AUC with PU data in the training distribution. trAUC is equivalent to the proposed method that minimizes Rie, (s) in Eq. (17) without importance weights (i.e., (xx = 1 for all x). tPPU and trAUC do not adapt to the test distribution. tePU and teAUC use unlabeled data in the test distribution instead of unlabeled data in the training distribution within tpPU and trAUC, respectively. Especially, teAUC is equivalent to the proposed method that minimizes R (s) in Eq. (19) for all x).
data.
without importance weights (i.e., wxx)
AUC, UDAUC, PAUC, and CPU use PU data in the train- ing distribution and unlabeled data in the test distribution as in the proposed method. trteAUC uses a weighted sum of the losses of trAUC and teAUCt for training. Thus, trteAUC is equivalent to the proposed method that minimizes R (s) in Eq. (20) without importance weights (i.e., w(x)= = for all x). UDAUC learns the invariant feature representation to mitigate the discrepancy of the training and test distri- bution by using CORAL loss, which is a widely used in domain adaptation studies (Sun et al., 2016; 2017), as in the previous method (Yang et al., 2023). Specifically, this method minimizes the weighted sum of the loss of trAUC and the CORAL loss for training. PAUC is a recently pro- posed AUC maximization method for positive distribution shift, where regative-condition but the positive-conditional density can vary between the training and test phases. CPU is a PU learning method for covariate shift adaptation. We used the dynamic approach for the importance weighting in CPU. This method does not consider the class-imbalance. For trPU, tePU, and CPU, the absolute-value risk correction was used improve the
density does not change
these methods work well with small æ. Since PAUC has
7

---

Positive-unlabeled AUC Maximization under Covariate Shift
Table 1. Average test AUCs with different positive class-prior и := Air = Tee. Values in bold are not statistically different at the 5% level from the best performing method in each row according a paired t-test. '# best" represents the number of results of each method that are the best or comparable to it.
Data
π
Ours
trPU
tePU
trAUC
teAUC
trteAUC
PAUC
UDAUC
CPU
MNIST
0.01
0.817
0.677
0.654
0.795
0.813
0.795
0.402
0.794
0.667
0.05
0.812
0.750
0.695
0.793
0.806
0.806
0.473
0.795
0.734
0.1
0.783
0.764
0.691
0.774
0.779
0.779
0.504
0.774
0.748
Fashion
0.01
0.925
0.912
0.906
0.920
0.912
0.920
0.634
0.926
0.920
MNIST
0.05
0.907
0.902
0.890
0.906
0.871
0.871
0.730
0.906
0.893
0.1
0.890
0.834
0.790
0.899
0,839
0.839
0.718
0.899
0.829
SVHN
0.01
0.715
0.504
0.504
0.548
0.724
0.725
0.236
0.549
0.504
0.05
0.686
0.504
0.504
0.559
0.698
0.698
0.246
0.559
0.504
0.1
0.666
0.504
0.504
0.533
0.677
0.677
0.266
0,540
0,504
CIFARTO
0.01
0.896
0.456
0.475
0.874
0.874
0.874
0.364
0.873
0.508
0.05
0.893
0,456
0.482
0.878
0.884
0.884
0.475
0.871
0.534
0.1
0.870
0.469
0.587
0.871
0.874
0.874
0.560
0.869
0.542
# best
12
2
6
7
6
2
0.81
0.615
035-MM6
0.80
A00
0.50
AM0
0.035
0.600
0.880
0.88
0.575
0.78
0.87
0.350
0.875
0.4
1.0
0.0
0.5
066
1.0
Bota
Bita
Bota
(a) MNIST
(b) FashionMNIST
(c) SVHN
(d) CIFAR10
Figure 1. The average test AUCs with the standard errors of the proposed method when changing weighting hyperparameter B.
Table 2. Comparison with the two-step importance weighting of the proposed method (Ours w/ 2step). Average test AUCs over different positive class-priors on cach dataset. FMNIST is an
abbreviation of FaShionMNIST.
MNIST
FMNIST
SVHN
CIFARIO
Ours
0.817
0.907
0.690
0.886
Ours w/ 2step
0.820
0.911
0.608
0.876
CIFARIO, smaller values of 3 tended to yield better results, while for FashionMNIST, larger values of / led to better results. The proposed method was able to select good / by using validation data. These results show the effectiveness of using the weighted sum of two AUC risk estimators.
Figure 2. Importance weight distribution of our method with Fash- ionMNIST when = 0.1. 'Training' (blue) and 'Test' (orange) represent data the training and test distributions, respectively.
different form of loss, its trend was different.
visualizes the distribution of the importance weights estimated by the proposed method with FashionM- NIST. The data the test distribution tended to have larger importance weights than the data in the training distribution. This result shows that the proposed method can estimate importance weights as expected.
Figure 2
Figure I shows the average test AUCs with the standard errors of the proposed method when changing 3. which con- trols the effect of two estimators of the AUC risk on the test distribution. When 8 = 1, our loss function becomes the AUC risk estimator with PU data in the training distribution (Eq. (17)). When 9 = 0, our loss function becomes the AUC risk estimator with positive data in the training and unlabeled data in the test distribution (Eq. (19)). The trends of the results varied across datasets: for MNIST, SVHN, and
Table 2 compares the proposed method and Ours w/ 2step, which is the proposed method using conventional two-step importance weighting. As expected, the proposed method, which uses the dynamic approach, tended to perform better
8

---

Positive-unlabeled AUC Maximization under Covariate Shift
Table Results under class-prio shift; average AUCs with different positive test class-prior The with postive training class-prior Tur 0.01. Values in bold are not statistically different at the 5% level from the best performing method in each row according to a paired t-test. '# best' row represents the number of results each method that are the best or comparable to it.
Data
Tte
Ours
trPU
tePU
trAUC
teAUC
trteAUC
PAUC
UDAUC
CPU
MNIST
0.01
0.938
0.729
0.729
0.938
0.938
0.938
0.562
0.938
0.729
0.05
0.929
0.734
0.717
0.930
0.924
0.930
0.762
0.930
0.733
0.1
0.930
0.736
0.709
0.932
0.921
0.932
0.866
0.932
0.731
Fashion
0.01
0.983
0.899
0.895
0.982
0.984
0.984
0.818
0.982
0.906
MNIST
0.05
0.983
0.900
0.770
0.983
0.980
0.983
0.929
0.983
0.897
0.1
0.988
0.901
0.743
0.988
0.983
0.988
0.970
0.988
0.895
SVHN
0.01
0.644
0.504
0.504
0.642
0.632
0.655
0.499
0.648
0.504
0.05
0.648
0.504
0.504
0.625
0.600
0.635
0.505
0.624
0.504
0.1
0.639
0.504
0.504
0.641
0.619
0.648
0.507
0.645
0.504
CIFARTO
0.01
0.884
0.432
0.427
0.867
0.872
0.872
0.519
0.866
0.481
0.05
0.890
0.433
0.424
0.878
0.872
0.878
0.680
0.878
0.480
0.1
0.891
0.432
0.424
0.885
0.867
0.886
0.785
0.885
0.482
# best
12
0
0
10
3
10
0
10
0
than Ours w/ 2step. Note that Ours w/ 2step is also our proposal since there are no existing importance weighting methods for AUC maximization.
Table 4. Results without distribution shift: average test AUCs
over different positive class-prior π := Wer The within {0.01, 0.05, 0.1
Although the proposed method assumes the covariate shift, other forms of distribution shifts may arise in real- real-world see- narios. Therefore, we additionally evaluated the proposed method under class-prior shift, in which the Mlass-prior changes, but the class-con density remains the same (Lu et al., 2022). Table 3 shows the results. The proposed method empirically worked well. This would be because the proposed method does not depend on class-priors and thus is relatively robust against the class-prior shift.
Data
Ours
trPU
trAUC
MNIST FashionMNIST
0.927
0.814
0.927
0.982
0.916
0.982
SVHN
0.625
0.504
0.630
CIFARIO
0.885
0.436
0.875
7. Limitations
The proposed method assumes a specific type of distribu- tion shift: the covariate shift. Although we found that the proposed method works well under the covariate and class- prior shifts in our experiments and we can often expect the validity of the covariate shift assumption to certain extent in each real-world application, the proposed method is not guaranteed to work well if other distribution shifts occur. Therefore, a method should preferably be developed that can estimate the type of distribution shifts from data.
Additionally, scenarios without any distribution shift may also arise in practice. Table 4 shows the results without distribution shifts (i.c., Ptr (x, b)= = Pte y)). Since there were no shifts, we compared trPU and trAUC, which do not consider shifts. The proposed method and trAUC showed comparable results, indicating that the proposed method robustly works well when no shift exists.
6. Conclusion
Impact Statement
In this paper, we proposed a method for maximizing the AUC under a covariate shift. To construct the method, we theoretically derived two estimators of the AUC risk on the test distribution: the first calculated from importance- weighted PU data in the training distribution, and the second is calculated from inportance-weighted positive data in the training distribution and unlabeled data in the test distribu- tion. Our loss function for classifier learning is a weighted sum of these two AUC risk estimators. The experiments show that our method outperformed various existing PU learning and distribution adaptation methods.
Although our method performed well, misclassification is possible in practice. In particular, misclassification can lead to serious incidents in cases such as cyber security and medical care, which are typical examples of imbalanced data. Thus, our method should be used as a support tool for humans to make final decision.
References
Aminian, G., Abroshan, M., Khalili, M. M., Toni, L., and Rodrigues, M. An information-theoretiee approach to

---

Positive-unlabeled AUC Maximization under Covariate Shift
semi-supervised learning under covariate-shift. In A/S- TAV', 2022.
He, Y., Shen, X., Xu, R., Zhang, T., Jiang, Y., Zou, W., and Cui, P. Covariate-shift generalization via random sample weighting. In AAAI, 2023.
Bagui, S. and Li, K. Resampling imbalanced data for net- work intrusion detection datasets. Journal of Big Data, (1):6, 2021.
Huang, J. and Ling, C. X. Using auc and accuracy in evaluat- ing learning algorithms. IEEE Transactions on knowledge and Data Engineering. 17(3):299-33 2005.
Bekker, and Davis, Learning from positive and unla- beled data: A survey, Machine Learning, 109(4):719-760, 2020.
Huang, J., Gretton, A., Borgwardt, K., Schölkopf, B., and
Smola, A. Correcting sample selection bias by unlabeled data. NeurIPS, 2006.
Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101- mining discriminative components with random forests. In ECCV, 2014.
Iwata, T. and Yamanaka, Y. Supervised anomaly detection based on deep autoregressive density estimators. arXiv preprint arXiv: vrXivv/904.000 06044, 2019.
Bradley, A.
The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7): 145 1159, 1997.
Jiang, Y., Xu, Q. Zhao, Y., Yang, Z., Wen, P., Cao, X., and Huang, Q. Positive-unlabeled learning with label distribution alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Brefeld, U., Scheffer, T., Auc maximizing support vec- learning. In Proceedings of the ICML 2005 workshop ROC Analysis in Machine Learning, 2005.
Johnson, M. and Khoshgoftaar, T. M. Survey on deep learning with class imbalance. Journal of Big Data, 6(1): 1-54, 2019.
Charoenphakdee, N., Lee, J., and Sugiyama, M. On sym- metric losses for learning from corrupted labels. In ICML, 2019.
Kanamori, T., Hido, S., and Sugiyama, M. A least least-qquares approach to direct importance estimation. The Journal of Machine Learning Research, 10:1391-1445, 2009.
Chawla, N. V., Bowyer, K. W., Hall, I.. O., and Kegelmeyer, W. P. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Kato, M. and Teshima, T. Non-negative bregman divergence minimization for deep direct density ratio estimation. In ICML, 2021.
Du Plessis, M., Niu, G., and Sugiyama, M. Convex formu- lation for learning from positive and unlabeled data. In ICML, 2015.
Kingma, D. and Ba, J. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Fang, T., Lu, N., Niu, G., and Sugiyama, M. Rethinking importance weighting for deep learning under distribution shift. NeurlPS, 2020.
Kiryo, R., Niu, G., Du Plessis, M. C., and Sugiyama, M. Positive-unlabeled learning with non-negative risk esti- mator. NeurIPS, 2017.
Fang, T., Lu, N., Niu, G., and Sugiyama, M. Generalizing importance weighting to universal solver for distribution shift problems. NeurIPS, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers features from tiny images. 2009.
Fujino, A. and Ueda, N. A cemi-supervisedd auc optimization method with generative models. In /CDM, 2016.
Kumagai, and Iwata, Learning future classifiers with- out additional data. In AAAI, 2016.
Ganin, Y. and Lempitsky, Unsupervised domain adapta- tion by backpropagation. In ICML, 2015.
Kumagai, A. and Twata, T. Unsupervised domain adap- tation by matching distributions based on the maximum mean discrepancy via unilateral transformations. In AAAI, 2019.
Gardner, J., Popovic, Z., and Schmidt, L. Benchmarking distribution shift in tabular data with tableshift. NeurlPS, 2023.
Kumagai, A., Iwata, T., and Fujiwara, Y. Transfer anomaly detection by inferring latent domain representations. NeurIPS, 2019.
Hammoudeh, Z. and Lowd, D. Learning from positive and unlabeled data with arbitrary positive shift. NeurIPS, 2020.
Kumagai, A., Iwata, T., Takahashi, H., Nishiyama, T., and Fujiwara, Y. Auc maximization under positive distribu- tion shift. In NeurlPS, 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
10

---

Positive-unlabeled AUC Maximization under Covariate Shift
LeCun, Y., Bottou, L.,, Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(11):228 86(11):2278-2324, 1998.
inspection. International Journal of Precision Engineer- ing and Manufacturing-GGri GGren Technology, 3:303-310, 2016.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, Focal loss for dense object detection. In ICCV, 2017.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L.,, and Lerer, A. Automatic differentiation in pytorch. 2017.
Liu, M., Yuan, Z., Ying, Y., and Yang, T. Stochastic auc maximization with deep neural networks. In ICLR, 2020.
Rhodes, B., Xu, K., and Gutmann, M. U. Telescoping density-ratio estimation. NeurlPS, 2020.
Long, M., Cao, Y., Wang, J., and Jordan, M. Learning transferable features with deep adaptation networks. In ICML, 2015.
Sakai, T. and Shimizu, N. Covariate shift adaptation on learning from positive and unlabeled data. In AAAI, 2019.
Lu, N., Zhang, T., Niu, G., and Sugiyama, M. Mitigating overfitting in supervised classification from two unla- beled datasets: consistent risk correction approach. In AISTATS, 2020.
Sakai, T. Niu, G., and Sugiyama, M. Semi-supervised optimization based positive-unlabeled learning. Machine Learning, 107:767-794, 2018.
Sharafaldin, I,, Gharib, A., Lashkari, A. H., and Ghorbani, A. A. Towards reliable intrusion detection benchmark dataset. Software Networking, 2018(1):177-20) 2018.
Lu, N., Zhang, T., Fang, T., Teshima, T., and Sugiyama, M. Rethinking importance weighting for transfer learning, In Federated and Transfer Learning, pp. 185-231. Springer, 2022.
Shen, J., Qu, Y., Zhang, W., and Yu, Y. Wasserstein distance guided representation learning for domain adaptation. In AAAI, 2018.
Matsui, K., Kumagai, W., Kanamori, K., Nishikimi, M., and Kanamori, Variable selection for nonparametric learning with power series kernels. Neural Computation, 31(8):1718-177) 1750, 2019.
Shimodaira, H. Improving predictive inference under covari- ate shift by weighting the bog-liied function. Jour- nal of statistical planning and inference, 90(2):227-244, 2000.
McDermott, M., Hansen, L. H., Zhang, H., Angelotti, G., and Gallifant, J. A closer look at auroc and aupre under class imbalance. In NeurIPS, 2024.
Sugiyama, M. and Kawanabe, M. Machine learning in non- stationary envirouments: Introduction to covariate shift adaptation. MIT press, 2012.
Menardi, G. and Torelli, N. Training and assessing clas- sification rules with imbalanced data. Data mining and knowledge discovery, 28:92-122. 122, 2014.
Sugiyama, M., Suzuki, T., and Kanamori, T. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Mirsky, Y., Doitshman, T., Elovici, Y., and Shabtai, A. Kit- sune: an ensemble of autoencoders for online network intrusion detection. In NDSS, 2018.
Sugiyama, M., Bao, H., Ishida, T., Lu, N., and Sakai, T. Machine learning from weak supervision: empirical risk minimization approach. MIT Press, 2022.
Nakajima, S. and Sugiyama. M. Positive-unlabeled classifi- cation under class-prior shift: a prior-invariant approach based on density ratio estimation. Machine Learning, !12 (3):889-9)) 919, 2023.
Sun, B., Feng, J., and Saenko, K. Return of frustratingly easy domain adaptation. In AAAI, 2016.
Sun, B., Feng. J., and Saenko, K. Correlation alignment for unsupervised domain adaptation. Domain adaptation computer vision applications, pp. 153 173-171,22 2017.
Netzer, Y.. Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. Granada, Spain, 2011.
Ueda, N. and Fujino, A. Partial auc maximization via nonlin- ear scoring functions. arXiv preprint Xiiv:1800.00433. 2018.
Pan, S. J. and Yang, Q. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10): 1359, 2009.
Wang, G., Kwok, S. W. H., Yousufuddin, M., and Sohel, F. A novel auc maximization imbalanced learning ap- proach for predicting composite outcomes covid-19 hospitalized patients. IEEE journal of biomedical and health informatics, 2023.
Park, J.-K., Kwon, B.-K., Park, J.H and Kang, D.-J. Ma- chine learning-bar based imaging system for surface defect

---

Positive-unlabeled AUC Maximization under Covariate Shift
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint
2017.
Xie, Z. and Li, M. soni-supervised auc optimization with- out guessing labels of unlabeled data. In AAAI, 2018.
Xie, Z., Liu, Y., He, H.-Y., Y. Li, M., and Zhou, Z.-H. Weakly supervised auc optimization: a unified partial auc ap- proach. IEEE Transactions on Puttern Analysis and Ma- chine Intelligence, 2024.
Yamada, M., Suzuki, T., Kanamori, T., Hachiya, H., and Sugiyama, M. Relative density-ratio estimation for ro- bust distribution comparison. Neurul computation, 25(5):
1324-1370. 2013.
Yang. C.. Zhao, M.. Zhu, C., Xie, S., and Chen, Y. Auto- mated detection of breast cancer metastases. In DSINS, 2021.
Yang, T. and Ying, Y. Auc maximization in the era of big data and ai: A survey. ACM computing surveys, 55(8): 1-37, 2022.
Yang. Z., Xu, Q.. Bao, S., Wen, P. He. Y., Cao, X., and Huang, Q. Auc-oriented domain adaptation: From theory to algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Yao, Y., Liu, T., Han, B., Gong, M., Niu, G., Sugiyama, M., and Tao, D. Rethinking class-prior estimation for positive-unlabeled learning. In ICLR, 2021.
Ying, Y., Wen, L., and Lyu, S. Stochastic online auc maxi- mization. NeurlPS, 2016.
Yuan, Z., Guo, Z., Chawla, N., and Yang, T. Compositional training for end-to-end deep auc maximization. ICLR, 2021a.
Yuan, Z., Yan, Y., Sonka, M., and Yang. T. Large-scale robust deep auc maximization: a new surrogate loss and empirical studies on medical image classification. ICCV, 2021b.
Zhang, T., Yamane, I., Lu, N., and Sugiyama, M. A one-step approach to covariate shift adaptation. In ACMI, 2020.
Zhao, H., Des Combes, R. T., Zhang, K., and Gordon, G. On learning invariant representations for domain adaptation. In ICML, 2019.
Zhao, H. Wang, X., Li, J., and Zhong, Y. Class frior-free positile-unlabeled learning with taylor variational loss for hyperspectral remote sensing imagery. In ICCV, 2023.
12

---

Positive-unlabeled AUC Maximization under Covariate Shift
A. Derivation of Eq. (15)
Lemma A.1. Exx-ppe (x) Exp (x) (x(x))))(x>)(((xx,>>>>)))))))))) xP)] = (x0 [xx(P)w((XP) = C, where C is a constant that does not depend on s.
Proof. Since f(x", o( ((xs(x))))))) s(x²)) and (()) + o(-2) for all E R,
Ex (xx (e (xx(x" xP]
E pp f (x) Bxx pph (xxx)))(("" x(xx,x²)]]
Ex (xx 1B pp (x) (xx()))((())))) (xx))(
E Exp (xx Exe app
xP]
(25)
is clear that the lemma follows from this equation.
B. Neural Network Architectures
For MNIST, FaShi and SVHN, three-layered feed-forward neural network with ReLU activation was used for the feature extractor h. The number of hidden and output nodes was 32. For CIFARIO, a convolutional neural network, which consisted of two convolutional blocks followed by a two-layered feed-forward neural network, was used for the feature extractor The first (second) convolutional block comprised a 6 (16) filter x 5 convolution, the ReLU activation, and 2 2 max-pooling layer. The numbers of the hidden and output nodes were 120 and 84, respectively. One-layered and two-layered feed-forward neural networks were used for u and v, respectively. For the output activation of e, we used (-) where is sigmoid function, to match the value range of the relative density-ratio (importance weights) when a >> 0. When 0 (i.e., using ordinary density-ratio), we used the softplus function for output activation. For all comparison methods, the same neural network architecture (i.e., u(h(x))) was used for the classifier (score function). For CPU, the architecture of e was also the same as that of the proposed method.
C. Hyperparameters
For all methods, the empirical risk with validation data was used to select hyperparameters and early-stopping to mitigate overfitting. Specifically, for the proposed method, the weighted risk in Eq. (20) was used. For the proposed method and CPU, relative parameter a was set to 0.5 for all datasets. For the proposed method and tUC, weighting parameter / was selected from (0.0, 0.01, 0.1, 0.3. 0.5, 0.7, 0.9, 0.99, 1.0). For UDAUC, the CORAL loss was applied to h(x). The weighting parameter of the CORAL loss was chosen from {1, 10-1, 10-2 10-3). The mini-batch size M was set to 512 size was set to 50. For all methods, we used the Adam optimizer (Kingma & Ba, 2014). We
and the positive set the learning rate 10 The maximum number of epochs was 200. All methods were implemented using Pytorch (Paszke al., 2017), and all experiments were conducted on Linux server with Intel Xeon CPU and A100 GPU.
D. Additional Experimental Results
D.1. Results with Different Numbers of Labeled Positive Data in the Training Distribution
Figure 3 shows the average test AUCs with the standard error of the proposed method with different numbers of labeled positive data in the training distribution NP. As expected, the performance of the proposed method improved as NP increased.
D.2. Results with Different Relative Parameters
Figure 4 shows the average test AUCs with the standard error of the proposed method with different values relative parameter a. Although the tendency of the results varied across datasets, the proposed method with a = 0.5 worked well for all datasets.
13

---

Positive-unlabeled AUC Maximization under Covariate Shift
0.01
0.75
0.07
0.70
0.80
6.575
0.45
0.055
6.850
0.40
D.84
6.23
0.3
6.800
20
160
20
160
60
69
160
Labered data
# Labslle P
cuta
# Labulee cata
Labe od data
(a) MNIST
(b) FashionMNIST
(c) SVHN
(d) CIFARIO
Figure 3. The average test AUCs with the standard emrors of the proposed method when changing the number of labeled positive data in the training distribution N
0.315
0.700
0.09
0.000-
0.675
0.550
0.01
105
0.005
0.623
0.800
0.900
085
0.0
0.6
08
0.7
(a) MNIST
(b) FashionMNIST
(c) SVHN
(d) CIFARIO
Figure 4. The average test AUCs with the standard errors of the proposed method when changing relative hyperparameter a.
Table 5. Average test AUCs over different positive class class-prior и := Tte within (0.01, 0.05, 0.1] on tabular datasets. Values in bold are not statistically different at the 5% level from the best performing method in each row according paired t-test.
Data
Ours
trPU
tePU
trAUC
teAUC
trteAUC
PAUC
UDAUC
CPU
epsilon 0.530
0,479
0.480
0.529
0.530
0.530
0.514
0.528
0.490
Hreadmission
0.694
0.541
0.561
0.700
0.553
0.553
0.477
0.699
0.551
D.3. Results with Tabular Data with Distribution Shifts
We evaluated the proposed method with two tabular datasets: epsilon and Hreadmission (Gardner al., 2023). In epsilon, the feature vector size is 2,000. To create the covariate shift, we followed the procedure used in a previous study (Sakai & Shimizu, 2019). Specifically, we first calculated the Euclidean distance between instance X,, and the mean vector of all data x, = ||x ||. Then, we found the median Cmed from all {c..). We split all (c.,) } into the first set whose elements were smaller than Cmed and the second set whose elements were larger than Cmed- With probability 0.9 and 0. instances whose indices were in the first set were selected as data in the training and test distributions, respectively. In contrast, instances whose indices were the second set were selected as data in the training and test distributions with probability respectively. In Hreadmission, the task is predict the 30-day readmission diabetic hospital patients. Each patient is represented by 183-dimensional feature vector. This dataset has distribution shift where the training and test distributions were created by "admission source". The experimental settings such as the number of data the same those in the main paper. Table 5 shows the average test AUCs these two tabular datasets. The proposed method performed the best or comparably to it in all cases.
and 0.9,
D.4. Computation Cost
We investigated the training time of the proposed method MNIST with æ 0.1. We used Linux server with 2.20Hz Central Processing Unit. For comparison, we also evaluated the methods that use PU data in the training distribution and unlabeled data in the test distribution as in the proposed method. Table shows the results. Since the proposed method and CPU learned both importance weights and classifiers, they had slightly longer training times than the other methods. However, the differences were not significant. This result indicates that the proposed method is practical in terms of computation costs.
"hups//www.csie..m ntu www.tw/
14

---

Positive-unlabeldd AUC Maximization under Covariate Shift
Table 6. Training time [s] of the proposed method on MNIST
Ours
IrteAUC
PAUC
UDAUC
CPU
90.2
86.83
87.37
87.93
89.49
Table 7. Performance comparison across different learning rates.
Method
Data
10-6 6
10
10
10- 3
10
10
Ours
MNIST
0.709
0.760
0.804
0.806
0.796
0.738
Ours
FahionFMNIST
0.854
0.914
0.907
0.873
0.874
0.789
Ours
SVHN
0.507
0.532
0.689
0.682
0.669
0.564
Ours
CIFARIO
0.810
0.897
0.886
0.884
0.789
0.644
trAUC
MNIST
0.693
0.756
0.787
0.785
0.779
0.784
trAUC
FaShionMNIST
0.846
0.916
0.909
0.895
0.891
0.876
trAUC
SVHN
0,500
0.503
0.547
0.553
0.537
0.533
trAUC CIFARIO
0.761
0.879
0.874
0.870
0.865
0.738
Table 8. Average test AUCs and their standard deviations with different positive class-prior == Ter Tse-
Values in bold are not
statistically different at the 5% level from the best performing method each row according to a paired t-test.
Data
Ours
uPU
tePU
tAUC
teAUC
uteAUC
PAUC
UDAUC
CPU
MNIST
0.01
0.817)0.021)
0.677(0.020)
0.654(0.009)
0.795(0.226)
0.813(0.022)
0.795(0.023)
0,402(0.313)
0.794(0.023)
0.667(0.008)
0.05
0.812(0.024)
0.750(0.047)
0.793(0.028) (.695(0.022)
0.806(0.027)
0.806(0.027)
(.473(0.043)
0.795(0.029)
0.734(0.038)
0.1
0.764(0.026) 0.783(0.027)
0.774(0.022) (0.018)
0.779(0.026)
0.779(0.026)
0.504(0.043)
0.774(0.022)
0.748(0.037)
Fashion
0.01
0.925(0.028)
0.912(0.032)
0,906(0.029)
0,912(0.021) ..920(0027))
0.920(0.027)
0.926(0.026) 0.634(0.038)
0.920/0.029)
MNIST
0.05
0.907(0.040)
0.962(0.066)
0.890(0.068)
0.906(0.036)
0.871(0.033)
0.87100.033)
0.730(0.031)
0.906(0.036)
0.893(0.663)
0.1
0.890(0.048)
0.834(0.053)
0.790(0.046)
0.899(0.04))
0.839(0.045)
0.839(0.045)
0.899(0.047) 0.718(0.035)
0.829)0.058)
SVHN
0.01
0.715(0.016)
0.504(0.008)
0.504(0.008)
0.725(0.027) 0.724(0.024) 0.548(0.022)
0.549(0.020) 0.236(0.006)
0.504(0.008)
0.05
0.686(0.024)
0.504(0.009)
0.504(0.008)
0.559(0.016)
0.698(0.014)
0.698(0.014)
0.559(00133) 0.246(0.007)
0.504(0.008)
0.1
0.666(0.037)
0.504(0.008)
0,504 (008)
0.533(0.024)
0.677(0.040)
0. 0.266(0.008) 0.677(0.940)
0.540(00024)
0.504(0.008)
CIFARIO
0.01
0.896(0.016)
0.456(0.070)
0.874(0.013) 0.475(0.095)
0.874(0.025)
0.874(0.025)
0.873(0.013) 0.364(0.33)
0.508(0.075)
0.05
0.893).0025)
0.456(0.064)
0.482/0.1144 114)
0.878(0.020)
0.884(0.026)
0,475(0.046) 0.884(0.026)
0.871(0.021)
0.534(0.115)
0.1
0.870(0.038)
0.469(0.071)
0.587(0..744
0.871(0.024)
0.874(0.025)
0.874(0.025)
0.56000.033)
0.869(0.027)
0.542(0.120)
D.5. Results with Different Learning Rates
We investigated the performance obtained by varying the learning rates of the Adam optimizer. Table shows the average test AUCs over different class-prior within (0.01, 0.05, 0.1) of the proposed method and trAUC, which is the most basic baseline. As observed, the value 10-1 used in our experiments consistently shows good performance across all datasets.
D.6. Full Results with Standard Deviations
Table 8 shows the average test AUCs and their standard deviations for each method. As class-prior decreased, the standard deviations of the proposed method also decreased. This is because when π is small, unlabeled data can be approximately regarded negative data; ranking function ff in Eqs. (17) and (19) can stably rank data well such that the scores of positive data are higher than those of negative data. Thus, its training becomes easier and more stable.
D.7. Results with Larger Data
We performed the additional experiments on larger datasets. In this experiment, for each dataset, we used 100 positive and 9,000 unlabeled data in the training distribution and 9,000 unlabeled data in the test distribution for training. In addition, we included the recent PU learning method (PURA) (Jiang et al., 2023) for comparison. Since it is designed for ordinary PU learning, it used PU data in the training distribution. Margin P was selected from f0.1, 1, 10) by validation data. The results are described in Table 9. The proposed method outperformed the other methods. Since PURA does not consider distribution shift, it did not work well.
15

---

Positive-unlabeled AUC Maximization under Covariate Shift
Table Results in the large-data regime: average AUCs with different positive class-prior и fir Mie- Values in bold
statistically different at the 5% level from the best performing method in each row according to paired t-test. '#f best' row represents the
number results of each method that are the best or comparable
Data
π
Ours
trPU
tePU
trAUC
teAUC
trteAUC
PAUC
UDAUC
CPU
PURA
MNIST
0.01
0.849
0.719
0.690
0.823
0,842
0.823
0.417
0.823
0.704
0.808
0.05
0.829
0.766
0.738
0.813
0,814
0.813
0.507
0.813
0.774
0.775
0.1
0.810
0.779
0.736
0.806
0.805
0.805
0.524
0.806
0.757
0.776
Fashion
0.01
0.942
0.928
0.922
0.942
0.925
0.942
0.668
0.945
0.920
0.924
MNIST
0.05
0.935
0.864
0.750
0.945
0.912
0.912
0.728
0.946
0.854
0.952
0.1
0.913
0.897
0.820
0.925
0.854
0.854
0.701
0.926
0.898
0.946
SVHN
0.01
0.763
0.504
0.504
0.628
0.780
0.780
0.252
0.628
0,504
0,503
0.05
0.757
0.504
0.504
0.599
0.760
0.765
0.272
0.605
0.504
0.506
0.1
0.750
0.504
0.504
0.587
0.749
0.751
0.304
0.586
0.504
0.503
CIFARIO
0.01
0.920
0.450
0.476
0.915
0.922
0.922
0.343
0.914
0.509
0.854
0.05
0.910
0.464
0.520
0.904
0,910
0.910
0.487
0.906
0.521
0.849
0.1
0.900
0.552
0.654
0.885
0.877
0.879
0.547
0.900
0.579
0.864
# best
11
8
0
2
Table 10. Results with the Food101 dataset: average test AUCs with different positive class-prior π
π
Ours
trPU
tePU
trAUC
teAUC
trteAUC
PAUC
UDAUC
CPU
0.01
0.590
0.470
0.563
0.585
0.601
0.585
0.385
0.577
0.509
0.1
0.585
0.488
0.551
0.578
0.554
0.578
0.441
0.590
0.506
avg
0.588
0.479
0.557
0.582
0.578
0.582
0.413
0.584
0.508
D.8. Results with the Food101 dataset
We additionally evaluated the proposed method with the Food101 dataset (Bossard et al., 2014) and the ResNet-18 model (He et al., 2016), which allows us to evaluate our method with a larger dataset and model. The Food101 dataset consists of image data from 101 food categories and is widely used for image classification tasks. The maximum length each image is 512 pixels. We resized each image to 224 224 pixels. To create a binary classification problem, we divided the original 101 categories into sweets-related (positive) and main dish-related (negative) classes. Then, following the procedure described in Section 5.1, we split the original categories within each positive/negative into two groups, assigning the first half with smaller class indices to the first group and the remaining the second group. We then created the covariate shift by using the group ratio of 9:1 for the training and 1:9 for the testing. We used 2,500 (200) positive and 25,000 (2,000) unlabeled training data and 25,000 (2,000) unlabeled test data for training (validation). We used 3,000 test data for evaluation. As for R8sNet-18, we did not use pre-trained weights to purely investigate the performance with the given data. Table 10 shows the results. The proposed method performed slightly better than these methods on average.
16


Reviews:
[{"review_id": "gSoQvWZBuc", "reviewer": "Reviewer_Su7H", "summary": "The paper aims to optimize the AUC under a covariate shift, i.e., when the test distribution of inputs differs from the training distribution. Considering the difficulty of collecting negative examples, this work focuses on the positive-unlabelled (PU) setting. To solve this problem, the paper first proposes two importance-weight-based risk estimators that depend only on PU training and unlabelled test data. A similar technique estimates the importance weights with a learnable model. Empirical studies on MNIST, FashionMNIST, SVHN, and CIFAR10 are conducted to validate the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive comments.\nWe will revise the paper according to your suggestion regarding the structure and notation.\nWe added experimental results in the anonymous URL: https://anonymous.4open.science/r/icml25s-6C74/results.pdf\n\n> the benchmark datasets could be further improved: only small-scale datasets are used to validate the effectiveness.\n\nWe performed the additional experiments on larger datasets.\nOur method outperformed the others in Table A of the anonymous URL.\nPlease see the answer to the first comment of Reviewer Gocn for details.\n\n> This work focuses on the covariate shift, i.e., the input distributions are different. However, as described in lines 315~328, the training data and test data are sampled under different class priors (categories of the images). A sampling strategy based on the inputs (e.g., style, noise, color) instead of the classes would be more reasonable.\n\nThank you for the constructive comment.\nAs described in Line 317, we created the covariate shift by using the sampling strategy of the AISTATS paper of covariate shift adaptation (Aminian et al., 2022).\nSpecifically, we first constructed positive and negative classes by partitioning the dataset's original classes into two groups.\nThen, we created $p_{{\\rm tr}} ({\\bf x}) \\neq p_{{\\rm te}} ({\\bf x})$ by reversing the high- and low-density input regions based on the original classes between the training and testing.\nIn our setting, training and test class-priors were the same ($\\pi_{{\\rm tr}} = \\pi_{{\\rm te}}$).\nAlso, we used the epsilon dataset in Appendix E.3, where the covariate shift was created based on the inputs (the distance between the feature vectors) as in (Sakai & Shimizu, 2019).\nWe would like to use the suggested strategy in future.\n\n> Details of the competitors (teAUC, trteAUC, UDAUC) are missing.\n\nAs described in the second paragraph of Section 5.2, the loss functions of teAUC and trteAUC are equivalent to Eq. (19) and Eq. (20) with $w({\\bf x})=1$ for all ${\\bf x}$ (i.e., they do not use importance weights), respectively.\nThe loss function of UDAUC is a weighted sum of the loss of trAUC and the coral loss that is calculated from unlabeled training and test data.\nThe coral loss is used to minimize the feature discrepancy.\nteAUC used positive training and unlabeled test data.\ntrteAUC and UDAUC used positive training, unlabeled training data, and unlabeled test data as in our method.\nWe will describe their specific loss functions in the revised paper.\n\n> Some results seem counter-intuitive. PAUC only achieves about 0.23~0.5 under most settings. \n\nPlease see our response to Reviewer 9PVZ's comment '(2) Why do the PAUC results in Table 1 perform particularly poorly.'\n\n> As $\\pi$ varies from 0.01 to 0.1, the tasks become more manageable with more positive examples, but the test AUC drops for most methods.\n\nAUC-based methods, except for PAUC, prefer small $\\pi$.\nThis is because they essentially use loss functions of the form, $\\mathbb{E} _{{\\bf x} ^{{\\rm p}} \\sim p ^{{\\rm p}}} \\mathbb{E} _{{\\bf x} \\sim p} \\left[ f({\\bf x} ^{{\\rm p}}, {\\bf x} ) \\right]$, where $p ^{{\\rm p}}$ and $p$ are positive and marginal densities.\nWhen $\\pi$ is small, $p$ can be regarded as negative density $p ^{{\\rm n}}$. In this case, the above loss function becomes the original AUC risk.\nThus, these methods work well with small $\\pi$.\nSince PAUC has a different form of loss, its trend was different. \nWe will clarify this.\n\n> sensitive analysis on $\\alpha$ should be provided.\n\nWe have performed the sensitive analysis on $\\alpha$ in Appendix E.2.\nAlthough the tendency of the results varied across datasets, our method with $\\alpha = 0.5$ worked well.\n\n> the learning rate is set to $10^{-4}$ for all methods, which might be suboptimal for some competitors.\n\n In our preliminary experiments, we tested learning rates of $10^{-3}$ and $10^{-4}$ and confirmed no significant difference in performance. Based on this, we used $10^{-4}$ in this paper. We will mention this.\n\n> A key to removing the positive-positive loss in Eq. (15) is using a symmetric function... Is it possible to practically use other surrogate losses, such as hinge or square loss?\n\nYes, we can use non-symmetric losses, although the second term in Eq. (14) is not a constant as you mentioned.\nIn this case, $\\pi_{{\\rm tr}}$ is necessary for training, but it can be estimated from PU data (Kumagai et al., 2024).\nSince the sigmoid function worked well in many PU learning and AUC maximization works, we also used it. We will discuss this in the revised paper.\n\n> How does the proposed method perform without a covariate shift?\n\nWe additionally evaluated our method when there were no shifts.\nTable B of the anonymous URL shows the results.\nSince there were no shifts, we compared trPU and trAUC, which do not consider shifts.\nOur method and trAUC showed comparable results, indicating that our method robustly works well when no shift exists.\nWe will include this result.", "claims_and_evidence": "The claims are overall clear and well-supported:\n\n**Claim 1.** New setting: The paper proposes to optimize AUC under the covariate shift with PU data.\n\n**Evidence:** Related work in Section 2 indicates that such a setting is different from previous work and reasonable.\n\n**Claim 2.** New methods: Two estimators of the AUC risk are proposed, which can be used to learn a score function without class priors; a dynamic method for predicting importance weights is provided.\n\n**Evidence:** According to Eq. (17) and Eq. (19), the proposed estimators are unbiased and do not require the class priors. Besides, Eq. (23) provides a loss to fit a bounded extension of the importance weights.\n\n**Claim 3.** Superior experimental performance.\n\n**Evidence:** Table 1 shows that the proposed method outperforms most of the competitors in various datasets.", "methods_and_evaluation_criteria": "The proposed method makes sense since it can optimize AUC using only PU data without class priors. However, the benchmark datasets could be further improved: only small-scale datasets (resolution $< 32 \\times 32$ and about tens of thousands of images) are used to validate the effectiveness. The results would be more convincing if larger benchmarks were used, e.g., Tiny-ImageNet-LT or iNaturalist.", "theoretical_claims": "I have checked and confirmed the correctness of the proofs.", "experimental_designs_or_analyses": "The experiments could be further improved in the following aspects:\n1. Both datasets and models (a few layers) are small-scale. It should be validated whether the proposed method is scalable to larger datasets or backbones (e.g., ResNet or DenseNet).\n2. This work focuses on the covariate shift, i.e., **the input distributions are different** ($p\\_{tr}(x) \\neq p\\_{te}(x)$). However, as described in lines 315~328, the training data and test data are sampled under different **class priors** (categories of the images). A sampling strategy based on the inputs (e.g., style, noise, color) instead of the classes would be more reasonable.\n3. Details of the competitors (teAUC, trteAUC, UDAUC) are missing. To ensure fairness, how these methods utilize different data could be clarified.\n4. Some results seem counter-intuitive. PAUC only achieves about 0.23~0.5 under most settings, but even a random guessing achieves 0.5 of AUC. Please provide more analyses and results, such as the AUC in the training distributions. Another issue is the effect of class prior $\\pi$. As $\\pi$ varies from 0.01 to 0.1, the tasks become more manageable with more positive examples, but the test AUC drops for most methods.\n5. The hyperparameter $\\alpha$ controls the trade-off between the upper bound and the approximation error in the importance weight estimation. Therefore, sensitive analysis on $\\alpha$ should be provided.\n6. According to Appendix D, the learning rate is set to $10^{-4}$ for all methods, which might be suboptimal for some competitors. A common strategy is searching for the best learning rate for each method.", "relation_to_broader_scientific_literature": "The paper extends AUC optimization, PU learning, and covariate shift. These areas are well studied, but the paper is the first work to jointly consider these problems. The main idea follows the mainstream covariate shift methods and develops some novel techniques. It could be applied to cyber security and medical care.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### **Other Strengths**\n1. Some real-world applications, such as cyber security and medical care, support the idea of jointly considering AUC optimization and covariate shift.\n2. The presentation is clear and easy to understand overall.\n\n### **Other Weaknesses**\n1. The main concern is the soundness of the experiments (see Experimental Designs Or Analyses). This work would be more attractive if the experiment issues were addressed.\n2.  The organization could be further improved. For example, the main paper presents too many detailed proofs. Although it is clear for readers who spend more time reading this paper, it is hard to find the main conclusions at first glance.", "other_comments_or_suggestions": "1. Some equations are overwidth (e.g., Eq. (6), Eq. (7), Eq. (14)).\n2. An abused notation $m$ is the index of unlabelled data before Eq. (20) and indicates the weights estimator in Eq. (22).", "questions_for_authors": "1. A key to removing the positive-positive loss in Eq. (15) is using a symmetric function ($\\sigma(z)+\\sigma(-z)=1$). However, such a surrogate loss might suffer from a gradient vanishing problem if the predicted scores are near $0$ or $1$. Is it possible to practically use other surrogate losses, such as hinge or square loss?\n2. How does the proposed method perform without a covariate shift? In this case, is the proposed method still comparable to previous PU & AUC optimization methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJCZYsvLif", "reviewer": "Reviewer_7MQt", "summary": "This paper proposes a new method for AUC maximization under covariate shift using positive-unlabeled (PU) data from the training distribution and unlabeled data from the test distribution. Given the challenges of estimating class priors of the training and test distribution, this paper theoretically derives two estimators for maximizing AUC such that they do not require the estimation of class priors. The paper also proposes an approach for importance weighting estimation for the AUC risk estimators. Empirical experiments on image and tabular datasets show overall performance improvement over existing approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments on our paper.", "claims_and_evidence": "The main claim of the paper – that learning from PU training data and unlabeled test while adapting to covariate shift, using their AUC risk estimators improves on the state-of-the-art – is supported by theoretical proofs and experimental results.", "methods_and_evaluation_criteria": "Proposed method, the methods which are included for comparison, and evaluation on image and tabular benchmark datasets makes sense for this problem.", "theoretical_claims": "Theoretical derivation of the two AUC risk estimators, approach to combine them, and approach to estimate importance weights seem to be correct. The steps described in the paper are detailed and make good use of the page limits.", "experimental_designs_or_analyses": "Covariate shift is simulated using standardized approaches from existing literature. Paper takes appropriate steps to for fair comparison among different approaches. Statistical significance tests confirm performance improvement across different datasets and class priors.", "relation_to_broader_scientific_literature": "The key contributions of the paper build on existing approaches in this domain but highlight the importance of specific parts of their proposed approach (such as recommending joint estimation of importance weighting during classifier learning over two-step importance weighting) which allow this approach to outperform existing state-of-the-art models.", "essential_references_not_discussed": "The related works sufficiently describes relevant literature on this topic. The proposed method is compared against a large set of existing approaches published in recent years.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g4ARGkBUDW", "reviewer": "Reviewer_Gocn", "summary": "The paper addresses the problem of maximizing the AUC in binary classification tasks under covariate shift, where the input distribution changes between training and test phases, but the conditional distribution of the class label given the input remains the same. The authors propose a novel method that leverages positive and unlabeled (PU) data from the training distribution and unlabeled data from the test distribution. They derive two estimators for the test AUC risk: one based on importance-weighted PU data from the training distribution, and another based on importance-weighted positive data from the training distribution and unlabeled data from the test distribution. The final loss function is a weighted sum of these two estimators. The authors also introduce a dynamic approach for importance weight estimation and classifier learning, which iteratively updates the importance weights and the classifier.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and constructive feedback.\nWe added experimental results in the anonymous URL: https://anonymous.4open.science/r/icml25s-6C74/results.pdf\n\n> The scale of the adopted datasets is too small, e.g., only \"50 positive and 3,000 unlabeled data in the training distribution and 3,000 unlabeled data in the test distribution\". In fact, for benchmarks like MNIST/CIFAR10, the authors could use much more examples (e.g., 10000+ samples) for training and test. Besides, more recent AUC optimization methods and PU learning methods should be compared.\n\nThank you for the constructive suggestion. We performed the additional experiments on larger datasets.\nFor each dataset, we used 100 positive and 9,000 unlabeled data in the training distribution and 9,000 unlabeled data in the test distribution for training.\nIn addition, we included the recent PU learning method (PURA) [a] for comparison. Since it is designed for ordinary PU learning, it used PU data in the training distribution.\nMargin $\\rho$ was selected from $\\\\{0.1,1,10\\\\}$ by validation data.\nThe results are described in Table A of the anonymous URL.\nOur method outperformed the other methods.\nSince PURA does not consider distribution shift, it did not work well.\nWe will include these results in the revised paper. \n\n[a] Positive-Unlabeled Learning with Label Distribution Alignment, TPAMI'23\n\n> While the paper covers a broad range of related work, there are a few essential references that could provide additional context for the key contributions:\n\nThank you for sharing the papers. \nWe briefly explain the difference between our work and these papers.\nThe paper [1] proposed a PU learning method under class-prior shift. Unlike our method, it cannot deal with the covariate shift.\nThe paper [2] proposed a PU learning method with label distribution consistency, but it cannot treat distribution shifts.\nThe paper [3] proposed a PU learning method from noisy positive and unlabeled data. It cannot treat distribution shifts.\nIn addition, all these methods are not designed for AUC maximization.\nWe will include a more detailed discussion of these papers in Section 2.\n \n> The paper draws on the importance weighting framework from covariate shift literature (e.g., Sakai, T. and Shimizu, N. Covariate shift adaptation on learning from positive and unlabeled data. In AAAI, 2019) but adapts it for AUC maximization (e.g., Kumagai, A., Iwata, T., Takahashi, H., Nishiyama, T., and Fujiwara, Y. AUC maximization under positive distribution shift. In NeurIPS, 2024). In general, the novelty might be somewhat incremental.\n\nThe novelty of our paper is mainly twofold.\nThe first is to propose a novel and significant problem setting of AUC maximization under covariate shift with PU data.\nThe second is to theoretically derive two novel AUC risk estimators to address this problem based on importance weighting, effectively utilizing the properties of AUC maximization and PU learning.\nMoreover, we also proposed to use the recently proposed dynamic importance weighting framework (Fang et al., 2020; 2023) for AUC maximization with PU data on complex models and data. \nAs other reviewers acknowledged, we believe our method has sufficient technical novelty.\n\n> The method assumes a specific type of distribution shift (covariate shift), which may limit its applicability in scenarios with other types of shifts.\n\nAs you mentioned, our paper assumes the covariate shift.\nThis is because the covariate shift is the most common and important shift in practice (He et al., 2023). \nNote that unsupervised distribution shift adaptation methods require some assumption about the shift type since no supervision is available in the test distribution.\nSince many papers at top conferences focus on the covariate shift only [b, c, d, e, f], we believe that our work has sufficient contributions. \n\n[b] Test-time Adaptation for Regression by Subspace Alignment, ICLR'25\n\n[c] Robust importance weighting for covariate shift, AISTATS'20\n\n[d] Double-weighting for covariate shift adaptation, ICML'23\n\n[e] Adapting to continuous covariate shift via online density ratio estimation, NeurIPS'23\n\n[f] An information-theoretical approach to semi-supervised learning under covariate-shift, AISTATS'22\n\nMeanwhile, we additionally evaluated our method under a class-prior shift, in which the class-prior changes, but the class-conditional density remains the same.\nTable C in the anonymous URL shows the results.\nOur method empirically worked well.\nThis would be because our method does not depend on class-priors and thus is relatively robust against the class-prior shift.\nWe will include these results.", "claims_and_evidence": "The claims made in the paper are generally supported by experimental results.", "methods_and_evaluation_criteria": "The proposed methods are well-suited for the problem of AUC maximization under covariate shift.", "theoretical_claims": "I checked the correctness of the proofs for the theoretical claims in the paper. Specifically, I reviewed the derivation of Lemma B.1 in Appendix B, which is used to prove Eq. (15). This proof is clear and correct.", "experimental_designs_or_analyses": "The scale of the adopted datasets is too small, e.g., only \"50 positive and 3,000 unlabeled data in the training distribution and 3,000 unlabeled data in the test distribution\". In fact, for benchmarks like MNIST/CIFAR10, the authors could use much more examples (e.g., 10000+ samples) for training and test. Besides, more recent AUC optimization methods and PU learning methods should be compared.", "relation_to_broader_scientific_literature": "The key contributions of this paper are closely related to several areas of the broader scientific literature, including AUC maximization, covariate shift adaptation, and positive-unlabeled (PU) learning.", "essential_references_not_discussed": "While the paper covers a broad range of related work, there are a few essential references that could provide additional context for the key contributions:\n1. Nakajima S, Sugiyama M. Positive-unlabeled classification under class-prior shift: a prior-invariant approach based on density ratio estimation\n2. Zhao Y, Xu Q, Jiang Y, et al. Dist-pu: Positive-unlabeled learning from a label distribution perspective\n3. Jain S, White M, Radivojac P. Estimating the class prior and posterior from noisy positives and unlabeled data", "other_strengths_and_weaknesses": "Strengths:\n1. The paper addresses a significant problem in PU learning under covariate shift, which is common in real-world applications.\n2. The paper is generally well-written, explaining the proposed method, with detailed descriptions of each module.\n\nWeaknesses:\n1. The paper draws on the importance weighting framework from covariate shift literature (e.g., Sakai, T. and Shimizu, N. Covariate shift adaptation on learning from positive and unlabeled data. In AAAI, 2019) but adapts it for AUC maximization (e.g., Kumagai, A., Iwata, T., Takahashi, H., Nishiyama, T., and Fujiwara, Y. AUC maximization under positive distribution shift. In NeurIPS, 2024). In general, the novelty might be somewhat incremental.\n2. The method assumes a specific type of distribution shift (covariate shift), which may limit its applicability in scenarios with other types of shifts.\n3. The experimental design is insufficient to convincingly support the paper’s claims.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "42m2adq4rL", "reviewer": "Reviewer_9PVZ", "summary": "In this paper, the authors focus on addressing the problem of AUC maximization under distribution shifts between training and testing data, specifically under the scenario of covariate shift, where the input distributions of the training and test data differ, but the conditional distribution of the class labels remains unchanged. To tackle this issue, the authors theoretically derive two estimators of the AUC risk, and combine positive instances with unlabeled data for training, without the need for negative examples or class-prior information. Extensive experiments on benchmark datasets validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and constructive feedback.\nWe added experimental results in the anonymous URL: https://anonymous.4open.science/r/icml25s-6C74/results.pdf\n\n> Weaknesses: (1) The paper lacks a clear explanation of the motivation for deriving the two estimators of the AUC risk on the test distribution under covariate shift. \n\nWe have described the motivation for deriving the two AUC risk estimators in Section 4.2 and the fourth paragraph of Section 1.\nSpecifically, under the covariate shift, existing AUC maximization methods cannot minimize the test AUC risk.\nTo deal with this, we derived two AUC risk estimators that can approximate the test AUC risk.\nSince these estimators are calculated with different data for classifier learning (i.e., Eqs. (17) and (19)), we can use all available data $X ^{{\\rm p}} _{{\\rm tr}} \\cup X _{{\\rm tr}} \\cup X _{{\\rm te}}$ for classifier learning by using both estimators (i.e., Eq. (20)). We will clarify this.\n\n> there is no analysis of their error bounds.\n\nThank you for the important feedback. We would like to work on analyzing the error bounds in the future.\n\n> (1) Has the issue of noise been considered? When the unlabeled data contains significant noise or redundant information, what impact would this have on the performance of the proposed AUC maximization method? \n\nYes, we have considered the issue of noise.\nThis is because unlabeled data can be regarded as noisy negative data since it contains a small number of positive data (noise).\nAs class-prior $\\pi$ becomes large (the noise in unlabeled data becomes large), the performance of our method tends to decrease, as described in Table 1. Nevertheless, our method outperformed the other methods.\n \n> (2) ... in real-world applications, such inconsistencies might also manifest as changes in class distributions (positive distribution shift) or shifts in class priors. In these inconsistent scenarios, can the proposed method still effectively perform AUC maximization?\n\nThank you for the insightful question.\nPlease see the answer to the last comment of Reviewer Gocn.\n\n> (1) The design in the paper relies on the sigmoid loss function. Would other loss functions be compatible with this solution approach?\n\nWe can derive the same form of the loss function of Eqs. (17) and (19) when using symmetric loss functions as described in Lines 259--267.\nThe symmetric functions include many popular losses, such as sigmoid, ramp, and unhinged losses.\n\n> (2) Why do the PAUC results in Table 1 perform particularly poorly, even being the worst in most cases? They are troublingly bad. \n\nPAUC strongly depends on the assumption that the negative-conditional density does not change between the training and testing to derive the loss function.\nHowever, the negative density changed significantly in our datasets.\nIn addition, as noted in the PAUC paper, PAUC performs poorly when $\\pi_{{\\rm te}}$ is small due to its reliance on extracting positive data from unlabeled test data.\nFor these reasons, PAUC did not work well in our setting.\nNote that we have confirmed that PAUC performs well in the setting used in the PAUC paper.\n\n> (3) The results on the SVHN dataset are consistently anomalous.\n\nSince each dataset has different properties, it is unsurprising that the trends of the results differ across datasets. \nMeanwhile, the SVHN is not the only data that shows different patterns. \nFor example, although trPU, tePU, and CPU worked relatively well in MNIST and FashionMNIST, they performed poorly (AUC is about 0.5) in both SVHN and CIFAR10 in Table 1.\n\n> (4) Fig.2 shows the importance weight distribution on the FashionMNIST dataset when the positive class prior is 0.1. Does this result suggest that the method might overly rely on the test data during training? Furthermore, how do the distributions behave when the priors are set to 0.01/0.05?\n\nThis result does not mean our method overly relies on the test data for classifier learning.\nThis is because the (estimated) importance weights are used for only PU data in the training distribution in Eqs. (17) and (19).\n\nAs described in the second paragraph of Section 5.1, the covariate shift was created so that the high- and low-density regions of the inputs were reversed between the training and testing.\nSince importance weights are defined as density-ratio $w({\\bf x}) = p_{{\\rm te}}({\\bf x})/p_{{\\rm tr}}({\\bf x})$, test data would tend to have larger weights than training data.\n(This is also valid for the relative density-ratio in Eq. (21)).\nFigure 2 examined whether the estimated weights correctly reflect this characteristic.\nWhen the class-priors are $0.01$ or $0.05$, similar results are obtained in Figure A in the anonymous URL.\nWe will clarify this.", "claims_and_evidence": "The claims presented in the paper are supported by clear theoretical derivations and experimental validation.", "methods_and_evaluation_criteria": "The proposed method fills the gap in current AUC maximization techniques under covariate shift.", "theoretical_claims": "The theoretical claims in the paper are both correct and rigorous.", "experimental_designs_or_analyses": "Overall, the experiments are comprehensive, but I still have a few concerns: these include the absence of experiments under noisy conditions and concerns regarding some outliers in the experimental results. The specific details will be provided in the Questions.", "relation_to_broader_scientific_literature": "The contribution of this paper lies in proposing a new AUC optimization method that addresses covariate shift in weakly supervised learning, building on Kumagai et al.'s (2024) work on positive class distribution shift and Lu et al.'s (2022) advancements in importance weighting methods for domain adaptation and transfer learning.\n\nKumagai, A., Iwata, T., Takahashi, H., Nishiyama, T., and Fujiwara, Y. Auc maximization under positive distribution shift. In NeurIPS, 2024.\nLu, N., Zhang, T., Fang, T., Teshima, T., and Sugiyama, M. Rethinking importance weighting for transfer learning. In Federated and Transfer Learning, pp. 185–231. Springer, 2022.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n(1)\tThe method proposed in the paper is innovative and addresses a key issue regarding covariate shift in AUC maximization, an area that has not yet been explored in the literature.\n(2)\tThe two estimators of the test AUC risk under covariate shift proposed in the paper are effective. This is demonstrated both theoretically and experimentally.\n(3)\tThe paper is well-written and easy to understand.\nWeaknesses:\n(1)\tThe paper lacks a clear explanation of the motivation for deriving the two estimators of the AUC risk on the test distribution under covariate shift.\n(2)\tThe proposed estimators are not unbiased. But there is no analysis of their error bounds.\n(3)\tThe structure of the paper is essentially identical to that of Kumagai et al. (2024).", "other_comments_or_suggestions": "(1)\tHas the issue of noise been considered? When the unlabeled data contains significant noise or redundant information, what impact would this have on the performance of the proposed AUC maximization method?\n(2)\tThe paper explores a new scenario of \"covariate shift\" and argues that this scenario more closely approximates real-world situations. However, in real-world applications, such inconsistencies might also manifest as changes in class distributions (positive distribution shift) or shifts in class priors. In these inconsistent scenarios, can the proposed method still effectively perform AUC maximization? Does the paper address these issues and provide corresponding designs? Is the proposed method solely applicable to covariate shift?", "questions_for_authors": "(1)\tThe design in the paper relies on the sigmoid loss function. Would other loss functions be compatible with this solution approach?\n(2)\tWhy do the PAUC results in Table 1 perform particularly poorly, even being the worst in most cases? They are troublingly bad.\n(3)\tThe results on the SVHN dataset are consistently anomalous. For example, in Table 1, the UDAUC method exhibits unexpectedly poor results on the SVHN dataset compared to the other three datasets. In Table 2, the experimental results on SVHN show an anomalous reverse pattern compared to the other datasets.\n(4)\tFig.2 shows the importance weight distribution on the FashionMNIST dataset when the positive class prior is 0.1. Does this result suggest that the method might overly rely on the test data during training? Could this lead to overfitting the test data? Furthermore, how do the distributions behave when the priors are set to 0.01/0.05? Would they show more pronounced features?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an innovative approach aimed at maximizing the AUC in binary PU classification tasks, particularly in the context of covariate shift. The proposed methods and estimators are underpinned by a robust theoretical framework and exhibit good performance. Following the rebuttal, all reviewers expressed a consistently positive stance toward the work. Therefore, I recommend accepting this paper. However, it is strongly advised that the authors incorporate all relevant details in their rebuttal in the final version
Decision: Accept (poster)