Quantitative Analysis of the Effect of Density Ratio Estimation in Covariate Shift Adaptation
Authors: Chenglin Yu
Zhengyu Zhou
Weiwei Liu
Venue: Accepted by TMLR ()

Abstract: In supervised learning, it is essential to assume that the test sample and the training sample come from the same distribution. But in reality, this assumption is frequently broken, which can lead to subpar performance from the learned model. We examine the learning problem under \emph{covariate shift}, in which the conditional distribution of labels given covariates does not change despite the covariate distribution shifting. Two-step procedures, which first compute the density ratio and then carry out importance-weighted empirical risk minimization, are a popular family of methods for addressing covariate shift. However, the two-step techniques' performance could degrade due to estimation error of the density ratio. 
Unfortunately, the extent of the density ratio estimation error that affects the accuracy of learning algorithms is rarely analyzed. This paper accordingly provides a quantitative answer to this question. Specifically, we formulate the two-step covariate adaptation methods as a meta-algorithm. We show that the effect of the density ratio estimation error on the excess risk bound of the meta algorithm is of the fourth order, i.e., $\mathcal{O}\left(\epsilon_{1}\left(\mathcal{G}, S_{s1}, S_t, \delta/2\right)^4\right)$, if the true risk satisfies a requirement known as the \emph{derivative vanishing} property, where $\epsilon_{1}\left(\mathcal{G}, S_{s1}, S_t, \delta/2\right)$ is the convergence rate of the density ratio estimation algorithm, $\mathcal{G}$ is the density ratio function class, $S_{s1}$ and $S_t$ are the samples generated by training distribution and test distribution respectively, and $\delta/2$ is the confidence parameter. Moreover, we analyze the impact of two specific density ratio estimation algorithms, Kullback-Leibler Importance Estimation Procedure and Kernel unconstrained Least-Squares Importance Fitting, on the final classifier's generalization error. We also report the experimental results of two-step covariate shift adaptation with a toy classification dataset using KLIEP.

Published in Transactions on Machine Learning Research (03/2026)
Quantitative Analysis of the Effect of Density Ratio Estima- tion in Covariate Shift Adaptation
Chenglin Yu School of Computer Science Wuhon University
clyycl @y com
Zhengyu Zhou
zaysince: 1999 (ww.9999mmmmmmm.m com
School of Computer Science Wuhon University
Weiwei Liu'
luweeiwei8600@mmac
School of Computer Science Wuhan University
Reviewed on OpenReview: https: /ttp:/://operevvteeee net/ forum? TtWsRXTYUV
Abstract
In supervised learning, it is essential to assume that the test sample and the training sample come from the same distribution. But in reality, this assumption is frequently broken, which can lead to subpar performance from the learned model. We examine the learning problem under covariale shift, in which the conditional distribution of labels given covariates does not change despite the covariate distribution shifting. Two-step procedures, which first compute the density ratio and then carry out importanco-wcighteed empirical risk minimization, popular family of methods for addressing covariate shift. However, the two-step techniques' performance could degrade due to estimation error of the density ratio. Unfortunately, the extent of the density ratio estimation error that affects the accuracy of learning algorithms rarely analyzed. This paper accordingly provides a quantitative answer to this question. Specifically, we formulate the two-step covariate adaptation methods as a meta-algorithm. We show that the effect of the density ratio estimation error on the excess risk bound of the meta algorithm is of the fourth order, i.c., O 0(,,, (G. SaL, S,, 5/2)*), if the true risk satisfies a requirement known as the derivative vanishing property, where (G, S.1. S, 8/2) is the convergence rate of the density ratio estimation algorithm, g is the density ratio function class, S,1 and S, are the samples generated by training distribution and test distribution respectively, and 5/2 the confidence parameter. Moreover, we analyze the impact of two specific density ratio estimation algorithms, Kullback-Leibler Importance Estimation Procedure and Kernel unconstrained Least-Squares Importance Fitting, on the final classifier's generalization error. We also report the experimental results of two-step covariate shift adaptation with toy classification dataset using KLIEP.
1 Introduction
A fundamental assumption in machine learning is that training and test sample are independent and identical distributed (t.i.d.). Because it makes it simple to extrapolate conclusions from training data to test data, this assumption is essential to learning. is possible to provide statistical guarantees on the performance of traditional machine learning algorithms, so long as the data, can be modeled under this assumption (Christopher M & Nasser M. 2006 Shalev-Shwartz & Bav-David 2014). Real-world data,
* *Corresponding author

---

Published in Transactions on Machine Learning Research (03/2026)
Patterns for engineering controls fluctuate due to the non-stationarity of environments (Raza et al. 2015 Shi & Liu 2024 2025;; incoming signals used for natural language and image processing, bioinformaties, or econometric analyses change in distribution over time and scasonality (Liu et al. 2018]; patients used for prognostic factor ident idcatioiiation in clinical trials may not come from the target population due to sample selection bias (He
however, rarely meet this assumption (Zhou & Liu 2023
Wang et al. 2024 Zhou & Liu
2026
ct al. 2021).
Covariate shift is more common and realistic for real data when compared to the i..d. assumption. It states that the conditional distribution of the labels given features is the same for both the training and test data (Maldonado et al.j (023). However, the marginal distributions of the training and test data are different. In the case of cancer diagnosis, for instance, the training set might. contain an excessive number of sick patients, frequently of a particular subtype that is endemic in the area where the data was collected. Similarly, the distribution of covariates in advertising systems is non-stationary because of temporal variations in user interest. Additionally, the covariate shift is feasible in numerous real-world applications, including speaker identification (Yamada et. al. 2010), brain-compuler interfacing (Li et al. 2010 emotion recognition (Jiraycharoennsak et al. 2014), human activity recognition (Hachiya et al. 2012). and spam filtering [Bickel& & Scheffer 2007).
The model that is trained using standard machine learning techniques, like empirical risk minimization (ERM) (Vapnik] 1998 Schölkopf & Smola 2002), is biased because of the difference between the training and test distributions under covariate shift. Many scholars suggest two-step methods to overcome this drawback: first, they estimate the ratio of test and training input (instance) densities, and then they minimize a weighted empirical loss (Sugiyama et al., 2011]. In each case; the density ratio for training example indicates the training example's importance to the test distribution. With such mechanism, the examples which are important to the test distribution will be given large weights, and minimizing the weighted loss will inevitably result in a hypothesis that fits these examples well. Density ratio estimation is challenging, though, and as a result, estimation error may worsen learning outcomes in real-world scenarios (Yamada et al. 2011 Rugretfully, it is still unclear how the estimation affects the final hypothesis's accuracy.
In order to comprehend the impact of density ratio estimation error on the ultimate hypothesis's performance, pose the following query: how much will the density ratio estimation error impact the accuracy of the hypothesis relurned by the learning algorillm? Inspired by the orthogonal stalistical learning framework (Dylan J & Vasilis 2023), we formulate the two-step approaches as a meta algorithm to address this question. We demonstrate that the impact of the density ratio estimation error on the excess risk bound of the meta algorithm is of the fourth order, i.c., O (a S,, Se. 5,,6 provided that the true risk satisfics condition known as derivative vanishing, where 61 (9, 5/2) is the convergence rate of the density ratio estimation algorithm:; & denotes the density ratio function class; S51 and S, represents samples coming from training and test distributions respectively; finally, 5/2 is the confidence parameter. Moreover, based on our theoretical results, we investigate two commonly used density ratio estimation techniques: Lulbback-Lebller Importance Estimation Procedure (KLIEP) (Sugiyama et al.) 2008) and Kernel unconstrained Least-Squares Importance Filling (KuLSIF) (Kanamori el al. 2012). Using Imporlance Weighted Empirical Risk Minimization (IWERM) for a linear hypothesis class, we derive excess risk bounds of the hypothesis for each of these estimation techniques. We also report the experimental results of two-step covariate shift adaptation with a toy classification dataset using KLIEP.
Our contributions are as folows: (1) unified meta-algorithmic framework that formulates diverse two-step covariate shift adaptation methods as a single meta-algorithm (Algorithm D cleanly decomposing the excess risk into components arising from density ratio estimation and from learning on the weighted sample; (2) a precise quantification of error propagation under the noted assumptions (satisfied by square loss); (3) explicit. excess risk bounds for classical estimators, where, while convergence rates for KLIEP and KuLSIF are known individually, integrating them into our framework yields novel, comparative perspective by showing that their rates, when used in this two-step pipeline, fall short of the oracle O(n rate.
2

---

Published in Transactions on Machine Learning Research (03/2026)
2
Preliminaries
This section is organized as follows. First, in Section 2.1 we establish our notation, which is mostly standard. Then we recall covariate shift assumption and covariate shift adaptation in Section 2.2 Lastly, we present the two-step covariate shift adaptation approaches and formulate them as a meta algorithm in Section 2.3
2.1 Notation
Let Z = X' x y denote the sample space with X C Rd (d E M) being the instance space and y = 10, being the label space. For a function f X + R and a distribution D over X, the functional (EE= (a) PP). We define the stor hull of E X as star (X,x)
Lp-norm defined as: [I)l,,, (t- + (1 (1-t) t) x' x' t 1]}. In addition, denote the convex hull of X conv (X). We denote the transpose of the vector/ veclor/matrix malrix by the superscript. T.
2.2 Covariate Shift Adaptation
Let S. =(at, =/ be the i.i.d. training sample drawn from training distribution D, with density (s, y), which be decomposed into the marginal distribution and the conditional probability distribution, i.c., (x, = P. (x),,, Let {(r], 1/) be test sample drawn from test distribution D, with density (z(z, v) (z) covariate shift assumption (Shimodaira) 2000), the input marginal distributions vary between training and test, Lee, Ps (x) # Pt (z), but the conditional distribution
The corresponding marginal distributions are denoted as D and Dj. Under the
the labels (given features) exhibits invariance across training and test stages, i.e., Ps (y/z) = Pe (ylz). The objective of covariate shift adaptation to select classifier e: X {0,1} out of hypothesis class with small risk with respect to the test distribution D,, _ (h) Eg,D, [ujl where Boollea expression) is the indicator function (oqual to 1 if the expression is true and 0 otherwise).
2.3 Two-Step Covariate Shift Adaptation Approaches
Let H be class of functions from X' to [0, 1] and let the risk LD, (h) == Eg-D, (x), .y)], where é [0, 1] x R+ denotes the loss function that measures the discrepancy between the predicted value h(x) and the true output value In covariate shift adaptation, the only labeled sample we observe is S,=((af, of )i which is drawn i.i.d. from D.. However, we need to find a minimizer h* of risk with respect to D., for which we only have í. draws from Dj. Reweighing data to ensure that the reweighted distribution matches the p((x)
is known, test distribution is a popular method for dealing with covariate shift. In particular, if go (x) (.) importance weighting be used for the problem:
LD, (h) -h)=Ez-pp EZLD, (1 y) - / (n (x).
f (((()) pe (x))))
(1)
1 pa(s) (h (z) (.) EZND, 1e (2)) .)) g0 (x)
Px(z) (z)
Let ((h (2) (x) = (x)) (x), where g X R+ a density ratio function. Denote E,-D, (1 (I), ,g ( (z) by LD. (h, We then have LD. (h) Lo, (h, The importance weighted empirical risk with respect the training distribution is therefore an unbiased estimator of the risk with respect to the test distribution for any fixed h E H. The ability to precisely estimate the density ratio turns into a crucial component in the covariate shift adaptation process. Consequently, the majority of the rescarch is focused on estimating the density ratio (Sugiyama et al., 2007b Kanamori et al. 2009 Huang et al., 2006).. Next, a predictive model in the training phrase is trained using the estimated density ratio, frequently with the help of the IWERM technique (Shimodaira) 2000; SSugiyama & Kawanabe 2012). In order to analyze the effect of the density ratio estimator error on the final hypothesis returned by IWERM at a high level, we formulate the two-step covariate shift adaptation approaches a meta algorithm presented Algorithm T
3

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 1 Two-step Covariate Shift Adaptation Meta Algorithm
Input: Sample set S, = f(0,, v1) ~ D,, St = f Di
Output: h, the prediction hypothesis,
1: Split S, into two subsets Sa1 = (at, yf) and S22 = S.1
2: Let & be the output of DRE(G, Ss1, St).
3:
return h, the output of IWERM S22;9).
In the two-step covariate shift adaplation meta algorithm, we first use a Densily Ralão Eslimation (DRE) algorithm to estimate the density ratio function 9o, and then plug the estimated density ratio into IWERM. For subsequent analysis, give accurate definitions of DRE and IWERM with their convergence properties being assumed.
Definition 1 (DRE algorithm and its convergence rate). Let G be a densily ralio function class from X to R². Define Iglls = I| (Dg). Given S51 ~ D., St ~ Dj, let the algorithm DRE(G,Sa.. St) can output a density rutio function 9, for which
ll gelle S c1 (9, St,
with probability at least 1 1
Definition (TWERM algorithm and its convergence rate). Suppose h is a minimizer of Ī.D, (h. Given can output predictor h E H. for which
Sa ~ DD and g G, let the algorithm IWERM(H, S S2i;g
(h, - LD. (h",g) .9) S (H, S22,559)
(3)
with probability at least 1-6. 8.
3 Main Results
We aim to quantitatively analyze the effect of density ratio estimation error (el S,,, Q)) on the final hypothesis's excess risk, i.e., L.D, (figo go) - T.D, (h*,9o) For convenience of illustration, we first give the definition of directional derivative on the vector space of functions.
Definition 3 (Directional Derivative). Let T be vector space of functions. For a functional T: T -> R, the directional derivative of T at function t E T in the direction 1' T is defined as
d T (t ut')
T(t + 8t') T(t) 8-0
0.T (1) [1']
lim
dj
w==
The k-th order directional derivative of a function in the directions th....,tth E T defined as
gk
OTTOM, (1] 3n .pp
T(I mily ... + mall)) w==0....
For functional T(h, g) TTx7 T R, the partial derivative of T with respect h the direction tt
d 4 (h (t == 8-:
T(h + 0t T(h,
0, g)[(] du
= lim
8
The partial derivative of T with respect to g in the direction i/2 is
d -T (h.g + pt2))
T(h 812) T(h,
(hT(h,g)(tt
lim 8-20
dp
#=0
The second partial derivative of with respect to in the direction ti then h in the direction to is Og (h, (h.))T(() denoted as TR ,)(,,,t)). The second partial derivative of T with respect h in the direction then g in the direction t22 is 0 (0, g)) denoted as T(,, (),g)tt Eg). Similar definitions hold for Th, 9) 14 12) and 0,0, (h, g)44 12]
4

---

Published in Transactions on Machine Learning Research (03/2026)
The directional derivative defined in Deftnition (also called the Gateaux derivative is generalization of the directional derivalive in mallivuriable calculus (Daryoush, & Encych Dehgham 2008).
Remark 1.
The first assumption make is that the risk function has the so-called derivative vanishing property, which implies that some partial derivatives vanish.
Assumption 1 (Derivative Vanishing Loss). LD, (h. g) is derivative vanishing with respect to H. and g, that is,
I, (h",90 [11 h",g .1 gol = = wh E H. Vg Eg.
(4)
Our main theorem requires a first-order condition for the hypothesis class in the sense that h is local minimizer, in addition to the derivative vanishing property.
Assumption 2 (First Order Condition). The minimizer for the risk satisfies the first-order condition:
OnL,, (h*,90) lh - hh] 2 0 Whu E star (H,
(5)
Further, need few technical assumptions.
Assumption 3. Define the norm over H as: ||lll (EawDy ((2g (2) (2) There erist constants k1, K2 such that the following holds:
The risk LD, is strongly conver with respect to the hypothesis: 4h1 H. and gi g,
LgD (h, 9) h², h1 h"1=>|lh--"" 1 x1|91 gollg 2hz wh E star h)).
(6)
b There erists constant B1, such that the following bound holds: Vhn E H and E star (H, h)):
0g, go) - h", hh,,,, 1²] B1 |1/1 h1* 11
There erists a constant 82, such that the following bound holds: wh, E star (H,h"), 91 E g, and g E star (0)
1 (l', [h1 h", 91 h",g1
90. -90] S B2 |161 12" 11 11
goll
(8)
An observation that for hn 1 and ką 2, Items and Assumption represent socond-order and higher-order smoothness of the true risk, respectively. Moreover, Assumptions to 3 are easily met whenever the square loss is applied to the hypothesis class prediction in order to determine the true risk. We shall discuss this in more depth in Section 3.1
The main theorem is presented below.
Theorem 1. Suppose EZ-D, le (h (z) (x): L.D, (h, salisfies Assumplions to 3 Let & be the oulput. of DRE(S, S.1, St). We then have:
If 32 in Ie 2 of Assumplion is 0, then the Algorithom will produce predictor h such that, with probability at least 6,
BIK e1 LD. (h. go) - LD, (h*,go)= =
5) ( S..
2k2
B1
5.0 H. S22. 5:
+
(9)
2x
Tf B2 in Item - of Assumption probability at least 1 6, 6,
is then the Algorithm will produce predictor h such that, with
31K
3153 2xa
8) 2bs 23
1 So. - i.p (h", 900s
9, S,,Sa,, St.
H, S. ::
(10)
5

---

Published in Transactions on Machine Learning Research (03/2026)
Proof of Theorem 1 The proof is provided in Appendix B.
Remark 2. Theorem 7 shows that, for Algorithm 7 the impact of the unknown density ratio function on the prediction has 2kg-order growth: O (e1 (9, St, -)) This suggests that if the desired oracle rate of Algorithm with the true density rutio function being known is of order O(n it suffices to take e1 (9, St, 2) On
(())=>=))) ) addition, we can also analyze the excess risk for specific two-slep domain adaptation algorithms using this theorem.
3.1 Square Loss Satisfies Assumptions to
We now show that the sufficient conditions introduced in Theorem 1 (Assumptions to 3 are not merely abstract technicalities: they are naturally and exactly satisfied by the square loss under mild boundedness conditions. This concrete instantiation makes the error propagation result in Theorem [ directly applicable to core class of regression problems.
In particular, several basic properties of the weighted square loss, that is, assuming 3h" E H such that Λ* [[y] Then, we show that the true risk satisfies Assumptions [ to under a little more conditions posed on the hypothesis class and density ratio function class. Finally, a corollary with concrcte constants is presented.
y)² , - are obtained
Lemma Suppose Elylæ] h² (x)). Then (5,)) = (5- (5)) -y has the following properties:
2. E (z): - he- (z))] = 0
vh E star (H, h*)
3. B Vel (z). (z): Vg E star (G, go)
Proof of Lemma i The proof is provided in Appendix C.
Properties in Lemma and some additional conditions lead to Theorem 2
Theorem 2. Suppose Ely|æ] ht (I), suPaex, H h (x) < Mr, Vg E G (z) 0, 90 (z) n.
Then LD. gh, (h. satisfies Assumptions B2 0.
with 1. kı 1, ky 2, k=
2 B 1 and
Proof of Theorem The proof is provided in Appendix D.
Combining Theorem and Theorem 2 we obtain the following corollary:
Corollary 1. Suppose Elyla| h+ (x), sup th (x)) s Mn, Vg G,x E (g) 2 n> 0, 90 (x) n.
Algorithm will then produce predictor h such that, with probability at least 1 - 6,
Mh np
8 St. )) + 4e2 H. S22 7
(o. LD,(h',
(11)
Corollary shows that, for Algorithm I when using square loss, the impact of the unknown density ratio function on the prediction has fourth-order growth: O(er SG, 1) )) This suggests that if the desired oracle rate with the true density ratio function being known is of order O(n-1), it suffices to take én Sa1, , St, 4) = -O(n -t). In addition, Corollary instant iates the coefficients before the two convergence rates in Theorem I We will instantiate the bound further in Section
Remark 3. The theoretical analysis of Corollary translates into clear, quantitative guidance for practitioners: the precision required in the density ratão estimation step is less stringent than the final target accuracy for the predictor. DRE method converging rate of roughly n con sufficient to support a final predictor converging at the desirable -1 rale.
6

---

Published in Transactions on Machine Learning Research (03/2026)
4 Effect of Density Ratio Estimation Error for Linear Hypothesis Class
Our main theorem in Section 3 is stated at high level of generality. In this section, we will examine two concrete estimation techniques: KLIEP and KuLSIF. Particularly, we first analyze the estimation error of IWERM on a linear hypothesis class. Then, we obtain the convergence properties of KLIEP and KuL.IF. It's worth mentioning that our bounds have concrete probabilities which are not explicitly stated the original papers of those two algorithuss. Finally, we plug the convergence rales into the general bound in Corollary respectively.
4.1 Excess Risk Bound of IWERM
This section analyzes the excess risk bound when the hypothesis class is the composition of the sigmoid
function sig (f) I+exp[-r] over the class of linear functions, i.e. H = {o + sig (wwx)) ||wl|22 s 1J. particular, we derive the relationship between the risks of the true minimizer and estimated risk minimizer with the density ratio function being g. For convenience, we denote J= {f (I, y) + (x) - yyhh E H}. Nx FA,F) denotes the A-covering number of with respect to the metric px (f. f') I fo We have the following theorem:
Theorem 3. Let H far + sig ((w,,) ||l|l2 - 1} and E(h(x:), v) (h[x)) 10--yy) y)². Suppose Ya E X, | S B and Ag E G, suppzex 9 (z) S M. When given S,2 D, and any g E 9, IWERM(H, S g) outputs a predictor E H for which, with probubility at least 1 - 5,
LD. (h.g) - LD, (h'
2B d log (1 + 22
log (
(12)
S 8M
inf (4
2A
+ 2M
where D = =0= MMPmmar'EEan (..0. suPrex (ii (ww,))) - sig ((w', (ww,,))) and ne is the size of Saa.
Proof of Theorem
The proof is provided in Appendix E.
4.2 KLIEP
KLIEP estimates the density ratio go (x) p=(x) (x(
and models go (x) using the following linear model:
9 (z)
Σ aepp (x)
(13)
where {ar} are the parameters to be learned from S, and S,, and (yr (z) are basis functions The target input density pt (x) can be estimated by
such that ye (x) 2 O, for X and e = 1,2....,b.. b. Pe (x) = 9 (x) p. (z). Parameters {ael can be optimized by minimizing the Kullback-Leibler divergence between Pr (x)) and D (x):
KL Pa, (I) pl (+)] = / Pt (I) log
Pt (x)
/ Pt (x) log
Pe (x) dx - / Pt (z) (x) dx
Po dx =
(14)
Ps (x)
Based on the empirical approximations, the optimization problem can be formulated as follows.
EnH log (Ein arpi (z:))]
maximize {a,
(15)
subject to En Eb arpe (x;)=n,1 and an, o2....,, 04.2
There global solution available for this convex optimization problem. We shall analyze how fast the resulting estimated density ratio function converge to the real density ratio function. For simplicity, assume
7

---

Published in Transactions on Machine Learning Research (03/2026)
R₂1 n1. Additionally, the Hellinger distance between g and g' with respect to Pe (x) is defined follows:
hp (g,9') (Volo)) 2 Pa (x) (x)dr)
(16)
Lemma 2. Let do E 90 (+]), Yos maxt- 11 Σド= log (i(z))) + m Σド= log (aj go 0}.
The KLIEP algorithm then returns 9 such that hy, (go, 9) O nn log (3) + vms
with probability at
least 1 - 5, where 0 <w<< < is constent.
The proof of Lemma can be easily adapted from Theorem 1 of (Sugiyama et al. 2008
Note that the bound in Lemmna 2 is with respect to Hellinger distance. We shall bridge the gap between Hellinger distance and I4-functional norm.
Lemma 3. Suppose Vy 9,8up <
M,
(x)
S m
Then 1 golle,(D))) S
24 (n1 + M) hp, 3
Proof of Lemma The proof is provided in Appendix F.
Combining Lemma 2 and Lemma 3 we obtain the bound of the distance between the empirical density ratio function returned by KLIEP and the real density ratio function, presented as Theorem
Theorem 4. Suppose Vg E 9,
E X, g (x) M, 90 (x) m. Then || go gollL,(D;)
((1=
-
) with probability least - -d, where 0 < w < 2 is a con-
2 (m + M) o
- log (3) + vmo))
stant.
The generalization error difference between the hypothesis returned by IWERM using the real density ratio function and the one using the estimated density ratio function is bound by the following corollary.
Corollary 2. Let H {ar + sig ((w,x) || S 1}. Suppose Elyla] ht (x). Vy E G E X,0<n < 9 (z) M, n go (x) < m and ||0|| B, vo X. Algorithm will then produce predictor h such that, with probability at least 1 - 6,
1 2m + +M2 ((1
log( 8, 5 + Van
2)
Lp, go) go) - LD,
(h", (h" s 1/3
(17)
2B dlog (1 + 2A
2 log (2)
32M inf A (A.F)
2
+MM
/12
112
210
where D wuPp, (B((1) SuPaex sig (ww, sig ((w xx))1, 0 w < constant.
Proof of Corollary The result follows from Corollary Thcorems 3 and 4
Corollary 2 shows that, the impact of KLIEP Algorithm 1
Since 0 < 2, can tell that -1 < 2 -.. It implies that there is still a distance between the prediction error rate that can be achieved by KLIEP and fast rate, i.c., O(n )).
is
O
n1
Tiw
Remark 4. The speed with which a learning algorillum converges as il is presented with more dala is a central problem in machine learning. Alezey Chervoonenkis and Vladimir Voprák present quantitiave bounds on the deviation between the empérical and expected risk as function of the sample size n (Vapnik) 1998). According to Vapnil (1998), in Vapnik's book co- authoreee by Chervonenkis VN & A Ya [1979), they present 'slow' and 'fast' bounds for Empirical Risk Minimization (FRM) when used with 0-1 loss. They show that in
8

---

Published in Transactions on Machine Learning Research (03/2026)
the realizable or 'optimistic" case (where there is predictor in the hypothesis class that almost surely predicts correctly, so that the mönimum achievable risk is zero) one cun achieve fast ((/nn) convergence, as opposed to the "pessimistic" case where one does not have such a predictor in the hypothesis class and the best uniform bound is ((1/v/) (typically so-called slow rate, page 127 in Vapnák (1998)). This difference is important becouse if one is in such a Jast vate' vegime, one can uchieve good performance with less data.
4.3 KuLSIF
Analogous to KLIEP, KuLSIF also models go(x) using a linear model:
((z) (epe(x)
(18)
where the parameters to be learned from S51 and St are fac)e-r The non-negativity condition spr(x) 2 0 is satisfied by the basis functions ye, 1,....b.
KuL.IF assumes that the model for the density ratio, i.e., g, Reproducing Kernel Hilbert Space (RKHS) endowed with a kernel function & X' x X, the inner product and the norm being denoted as (x1) and Ilg, respectively! The optimal solution to
A E9llا s.t. gEg
min
(2f)
(zf)
The
yields the estimator ÿ. In order to prevent overfitting, the regularization parameter A(2 0) is introduced
along with the regularization term A 111ll:11
According to Theorem of (Kanamori et al. 2012) the KuLSIF estimator given as:
nx1
((x)
a,k xt) +
n&d Σ Mx,
(19)
The coefficients = (ōi,... an.1) yT are given by the solution of the linear equation:
(_kn Kn + Una a 11174A
K12ln,
(20)
n11
where Ta the na1 by n11 identity matrix and I, is the column vector defined as 1,, (1...., 1)T E R", K11 and K12 are the sub-matrices of the Gram matrix: (Kn)ær ={xf,=)), (K12)i i = ....,......1............
k(af, 2/), where = 1,...,, nt. We shall analyze how fast the resulting estimated densily ratio function converges to the real density ratio function.
Lemma 4. Assume G is a RKHS endowed with the kernel function k defined on X x X. The norm and inner product on G are denoted by ||-l|cı and (.-))1. Suppose that supzex k(a, x) < cc. Let GM = [g E Glllgllg < M), and that the bracketing entropy Ht(t, GMs, D2) bounded above by O M yr, where ? is conslant salisfying 0 <T 2, and I. is an arbilrary wumber such that 2 < 1. Sel the regularization parameter X Aaaio such that
2+7
An.1,, O, A=1 .o. -0 (ma)) ( (nal, n 700)
lim Boi +00
where na1 A n min(nes, ne). Then, for go E G, we have ||@ - (00ooolx((D)) = O (2log(()) with probability al least 1-8. 8.
The proof of Lemma can be easily adapted from Theorem 2 of Kanamori et al. 2012
Note that the bound in Lemma 4 is not with respect to L4-functional norm either, but with respect to L2- 22-fuctional norm. We bridge the gap between them.
Too avoid confusion with the previous notation ||l defined Definition

---

Published in Transactions on Machine Learning Research (03/2026)
Lemma 5. Suppose vgg E G, Sup x g(z) M. Let Md c² log(2), where is the same constant in Lemma 4 Then, |10 goll La(py) & (M + Ma) Do (D:))
Proof of Lemma 5
The proof is provided in Appendix G.
Combining Lemma 4 and Lemma 5 obtain the bound of the distance between the estimated density ratio function returned by KuLSIF and the real density ratio function, presented Theorem
Theorem 5. Suppose Vgg E supeexx g(x) & M. Then, llğ (00-oolla(Dp5) I I0 log( ( (M))))))mm))) with probability at least d, where с is a constont.
Proof of Theorem The proof is provided in Appendix H.
The following corollary provides a bound the difference between the generalization error of the hypothesis returned by IWERM using estimated density ratio function by KuLSIF and true risk minimizer in H. Corollary 3. Let H = far + sig x)) l|voll2 S 1}. Suppose Elyla] = ht (x). Vg E 9 E 2, < S 9 (x) S M, S 9o (x) S M and ||ll S B, vx E X. Algorithm with probability at least 1 - 6,
will then produce predictor such that,
((2100 3)²² Aanimme
LD (h. - (h" (h"" 1 173
B 2B, dloo (1+
2 log (3)
(21)
22
32M inf AC0,D).
2
8M
V/12
12/2
(x(A:))
210
where D w' su |ii ((w,x)) sig ((w', x))], and Ansime is the regularization parameter which satisfies the following condition:
lim Re +oo
Am. 0,
A Aol =0 ((na)/r))z (1s1, n4 700)
(22)
where is an arbitrary number such that
2 2+y
<t <t<1 (y is a constant satisfying 0 0<<<2)
Proof of Corollary The result follows from Corollary Theorems and
Corollary 3 shows that, the impact of KuLSIF on Algorithm is ((m.1 A m))) Since 247 < <i<], have -1 2 <t < 0. Therefore, two-step covariate shift adaptation for KuLSIF's convergence rate does not reach the fast rate, i.., O(n
5 Experiments
In this section, we compare the classification performance of exponentially-Baateeed importance weighted ERM(E/WERM [Shimodaira, 2000)) using density ratio estimates returned by KLIEP with different kernel widths and true density ratios.
Let us consider a two-dimensional binary classification problem. by
The class posterior probabilities are defined
p(u = +1|z) = (a ( +1
(23)
2
where ply -12) ppy = +1/x) and x = (1)) x (2))T. Let the training and test instance densitics be
1
(x(r)=
(x:; m1 Σn) + (1;; m2 E2).
2
(24)
((( = ((x; my, Ex) +
((x; m4 Ex). 2
2
10

---

Published in Transactions on Machine Learning Research (03/2026)
0g 1000
n, 2000
0.3
0.24
0.28
0.22
0.24
188t efror
test error
0.18
0.18
0.2
c-0.T, NMSE-111322 1.1324x 10
n=0.1, NMSE-9.7192x 10
0.16
c-0.2, NMSE-1 00015xxx
NMSE-1 1.404x 10
0.12
o-0.2, NMSE-7.7170x
10
denstty mato
density ratlo
0.14
0.1
0.2
0.3
0.4
0.5
0.7
0.9
0.1
0.3
0.5
0.6
0.7
0.8
0.9
A
Figure 1: Test error (misclassification rate) functions of the tuning parameter À on different density ratio estimates and real density ratio.
where ((x; m, Σ) is the multivariate normal density with mean and covariance matrix E, and m, (-2,3)
m2 = (2,3) = (0, (0) .ma = (4) Σ = Σ2 ( 2 Es En
(1 - We create training
0
n. test instances { following Pz(r) and test labels 1m following p(y|z))). This a typical example of covariate shift.
instances {xi) following (r) and training labels {uf J==1 following p(y)=;)) (xf)). Similarly, we create n, = 1000
Setup. For learning, employ the following linear model:
f(x; 0) 8, + Σ 0zz
(25)
where d the input dimension and (0 the i-th element of a. The parameter vector learned by EIWERM:
n Eolt) (167;:) );)
DRIWERM aarg aargmin min nx
(26)
where S < hyperparameter controlling the trade-off between consistency and stability of EIWERM and g is the density ratio function. The above minimizer ÓETWERO ERM analytically given by
θEIWERM (XT DAX) x DAy.
(27)
where
1
1
22
X
(28)
D is the diagonal matrix with the i-th diagonal element D2 g(xf) and = (gi.v..... .., The classification result of a test instance x² is obtained by:
i = s
(29)
where sgn(+) denotes the sign of a scalar. We use values Xe (0.00...2.. .,103 to vary the trade-offs between consistency and stability of EIWERM.
11

---

Published in Transactions on Machine Learning Research (03/2026)
The performance of KLIEP depends on the choice of the basis functions {ipe (z)] Following Sugiyama et al. 2008 we use Gaussian kernel model centered at subset of { for computational efficiency, i.e.
b Σ ae
0 (x)
(30)
where (x(x,z') is the Gaussian kernel with kornel width d:
1ix-x'1 2x
(o x²) ) =
Ce template instance randomily chosen from { and U(S n4) prefixed number, as referenced on Eq. (7) of Sugiyama et al. (2008). We use values E (0.1,0.2, 0.8} to vary the quality of the density ratio estimates produced by KLIEP, which evaluated by the normalized mean squared ervor(NMSE):
((x))
[o())) Ex P== (o( (.)))
NMSE =
(31)
Thss p= ((7))
Results. Fig. depicts the test errors for 1000 and 2000. It is illustrated that the test error of the predictor returned by EIWERM for most of the A values decreases when NMSE decreases. The predictor obtained using real density ratio achieves lower test error compared with that obtained using density ratio estimates when À is greater than 0.5. The results show that reducing the density ratio estimation crror is helpful for reducing the lest error.
6 Related Work
6.1 Density Ratio Estimation
When there a covariate shift, empirical risk minimization may not work as planned because the training and test distributions differ. In order to reduce the impact of covariate shift, importance weighting is utilized (Sugiyama et al., (207a), which results in IWERM. The importance frequently related the ratio of the test and training instance densities. Thus, the accuracy of the density ratio estimation becomes critical to the covariate shift adaptation process.
Density ratio istimmation algorithms are divided into two categories. The first involves estimating the two probability densities independently first, and then calculating the ratio of the estimated densities. However, it is well-known that density ostimation is a challenging problem (Kanamori et al. 2010). As a result, this naive strategy might not work. Estimating the density ratio directly, without estimating the densities, the second method. Various direct density-ratio estimation methods can be divided into four types: Moment Matching, Probabilistic Classification, Density Fitting and Density-Ratio Fitting (Masashi et al. (0022). Of all the direct. methods, KLIEP (Sugiyama et al.) 2008) and KuLSIF (Kanamori al. 2012) are two commonly used ones which are computationally efficient and comparable to others in terms of performance (Kato & Teshima
2021
Tiao et al. 2021). We include them as our analysis.
6.2
The Analysis of Effect of Density Ratio Estimation Error
The closest previous work ours is Cortes et al. (2008). They consider the problem of figuring out how much the weight estimation error impact the precision of the hypothesis that weight-sensitive learning problem returns in the missing at random. (MAR) bias problem. Their theoretical results differ from our work in three aspects.
First, MAR bias and covariate shift differ significantly from one another. Compared to covariate shift, the MAR bias places more stringent requirements on the relationship between the training and test distributions. In other words, all cases of MAR bias can be modeled as covariate shift, but some cases of covariate shift cannot result. from MAR bias Consider the data analysis problem with brain-computer interface, where it
12

---

Published in Transactions on Machine Learning Research (03/2026)
is known that experiments progress - the subjects get tired, the sensor configuration varies, etc. the distribution over incoming signals will alter. In this example, a covariate shift that cannot be represented by sample selection bias occurs. The relationship between covariate shift and MAR bias can be further referred to Appendix I.1 and I.3.
Second, the risks analyzed in Cortes et al. (2008) and this paper are different. Tet Ago denote the predictor returned by IWERM(H, go). Theorems and in Cortes al. (2008) provide bounds on the difference between the generalization error of hg and h (i.e., L.D (b,,(,gg (h. ) - LD, (h 9o)). In contrast, we consider the difference between the generalization error of the hypothesis returned by IWERM based on the estimated density ratio and the best one in the hypothesis class (i.e. LD, (h, (0) (1k.9o)-Lp, LD, (h", 90)).. For the weight estimation error, they only consider the weights for the training sample, while our analysis consider the difference between the real density ratio function and the estimated density ratio function. Therefore, the analysis in Cortes et al. 2008 is orthogonal to our work.
Third, the techniques of deriving bounds are different. Cortes et al. 2008 derives bounds via the distributional stability of kernel-based regularization algorithms. In contrast, we derive excess risk bounds via the derivative vanishing property of the true risk. Details can be found in Appendix I.3.
Reddi et al. (0155 is another work that is relevant to ours. Using the unweighted solution variance- reducing proxy for the correct weighed solution, this paper proposes doubly robust estimator to address the problem that many covariate shift correction algorithms face: the variance increases significantly whenever samples are reweighted. Moreover, they analyze the generalization bounds for the Penalized Risk Minimization (PRM) estimation algorithm based on a meta theorem. However, they only give the generalization bound of the final hypothosis. The effect of the density ratio estimation error on the learning algorithm is not explicitly illustrated by Reddi et al. (2015)
7 Conclusion
KLIEP and KuLSIF are two classical and fundamental density ratio estimation approaches in two-step covariate shift adaptation and still the effect of the estimation error on the final classifier's accuracy is comparatively less explored. To study this problem, we first formulate the two-step covariate shift adaptation approaches as a meta algorithm. We then show that if the true risk satisfies a condition called derivalive vanishing property, the impact of the density ratio estimation error on the excess risk bound of the meta algorithm is of the fourth order. Furthermore, we analyze the convergence rates of KLIEP and KuLSIF with specific probability and give an upper bound of estimation error of IWERM for linear hypothesis class. Finally, the excess risk bounds for the two density ratio estimation algorithms are obtained and the quantitative effects of the density ratio estimation error are illustrated. Nunerical experiments collaborate the theoretical findings.
A natural question is whether the derivative-vanishingg property holds for other common losses (e.g., logistic, hinge) or more complex model classes. While beyond the scope of this work which focuses on establishing a clear foundation with a canonical loss-we we view this as a valuable direction for future investigation.
Acknowledgments
This work supported by the Key R&D Program of Hubei Province under Grant 2024BAB038, the National Key R&D Program of China under Grant 2033Y7C360400..
References
Storkey Amos. When training and test sets are different: characterizing learning transfer. Dalaset shift in machine learning, 30:3 28, 2009.
Shai Ben-David, Tyler Tu,, Teresa Lau, and Dávid Pål. Impossibility theorems for domain adaptation. Tn AISTATS, 2010.
Steffen Bickel and Tobias Scheffer. Dirichlet-enhanced spam filtering based biased samples. NewrIPS, 2007.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Stéphane Boucheron, Olivier Bousquet, and Gábor Lugosi. Theory of classification: A survey of some recent advances. ESATM: Probabilily and slatistics, 9:323-377 2005.
Bishop Christopher M and Nasrabadi Nasser M. Pattern recognition and machine learning, volume 4. 2006.
Corinna Cortes, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh. Sample selection bias correction theory. In ALT, 2008.
Behmardi Daryoush and Nayeri Encyeh Dehghan. Introduction of fréchet and gâteaux derivative, Applied Mathematical Sciences, (17-20):975-980, 2008.
Foster Dylan J and Syrgkanis Vasilis. Orthogonal statistical learning. The Annals of Statistics, 51(3):879-900 908, 2023.
Hirotaka Hachiya, Masashi Sugiyama, and Naonori Ueda. Importance- least-squares probabilistic classifier for covariate shift adaptation with application to human activity recognition. Neurocomputing, 80: 93-101, 2012.
Yue He, Zheyan Shen, and Peng Cui. Towards non-i..d. image classification: A dataset and baselines. Pattern Recognition, 110:107383, 2021.
Jiayuan Huang, Alexander J. Smola, Arthur Gretton, Karsten M. Borgwardt, and Bernhard Schölkopf. Correcting sample selection bias by unlabeled data. NewrIPS, 2006.
Sooyong Jang, Sangdon Park, Insup Lee, and Osbert Bastani. Sequential covariate shift detection using classifier two-sample tests. In ICML, 2022.
Suwicha Jirayucharoensak, Setha Pan-Ngum, and Pasin Israsena, Eeg-based based emotion recognition using deep learning network with principal component based covariate shift adaptation. The Scientific World Journal, 2014, 2014.
Paulo Mauricio Gonçalves Jr., Silas Garrido Teixcira de Carvalho Santos, Roberto Souto Maior de Barros, and Davi Carnauba de Lima Vieira. A comparative study on concept drift detectors. Expert Syst. Appl., 2014.
Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct importance ostimation. Journal of Machine Learning Research, 10:1391-1445, 2009.
Takafumi Kanamori, Taiji Suzuki, and Masashi Sugiyama. Theorctical analysis of density ratio estimation. IEICE Transactions on Fundamentals of Electronics, Communications and Compuler Sciences, 93-A(4): 787-798, 2010.
Takafumi Kanamori, Taiji Suzuki, and Masashi Sugiyama. Statistical analysis of kernel-basee least-squares density-ratio estimation. Machine Learning, 86(3):335-331) 2012.
Masahiro Kato and Takeshi Teshima. Non-ncgative bregman divergence minimization for deep direct density ratio estimation. In ICML, 2021.
Atsutoshi Kumagai, Tomoharu Iwata, and Yasuhiro Fujiwara. Meta-learning for relative density- estimation. In NeurIPS, 2021.
Michel Ledoux and Michel Talagrand. Probability in banach spaces: Tsoperimetry and processes. 1991.
Yan Li, Hiroyuki Kambara, Yasuharu Koike, and Masashi Sugiyama. Application of covariate shift adaptation techniques in brain computer interfaces. IEEE Transactions on Biomedical Engineering, 57(6))1318 1324, 2010.
Anjin Liu, Jie Lu, Feng Liu, and Guangquan Zhang. Accumulating regional density dissimilarity for concept drift detection in data streams. Pattern Recognition, 76:256-277 272, 2018.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Song Liu, Akiko Takeda, Taiji Suzuki, and Kenji Fukumizu. Trimmed density ratio estimation. In NeurIPS, 2017.
Sebastián Maldonado, Ramiro Saltos, Carla Vairetti, and José Delpiano. Mitigating the effect of dataset shift in clustering. Pattern Recognition, 134:109058, 2023.
Wainwright Martin J. High- High-ddmensional statistics: non-asymptotic viewpoint, volume 48. 2019.
Sugiyama Masashi, Suzuki Taiji, and Kanamori Takafumi. Density ratio estimation in machine learning. 2012.
Aditya Krishna Menon and Cheng Soon Ong. Linking losses for density ratio and class-probability estimation. In ICML, 2016.
Jose G. Moreno-Torres, Troy Raeder, Rocío Alaix-Rodriguez, Nitesh V. Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45(1):521 530, 2012.
Aleksandr Podkopaev and Aaditya Ramdas. Tracking the risk of a deployed model and detecting harmful distribution shifts. In ICLR, 2022.
Haider Raza, Girijesh Prasad, and Yuhua Li. EWMA model based shift-detcctin methods for detecting covariate shifts in non-stationary enviromments. Pattern Recognition, 48(3):659 669, 2015.
Sashank Jakkam Reddi, Barnabás Póczos, and Alexander J. Smola. Doubly robust covariate shift correction. In AAAI, 2015.
Bernhard Schölkopf and Alexander Johannes Smola. Learning with Kernels: support vector machines, regularization, oplimization, and beyond. 2002.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms. 2014.
Lianghe Shi and Weiwei Liu. A closer look at curriculum adversarial training: From an online perspective. In AAAT, 2024.
Lianghe Shi and Weiwei Liu. Adversarially robust unsupervised domain adaptation. Artificial Intelligence, 347:104383, 2025.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihoodd function. Journal of Statistical Planning and Inference, 90(2):227-244 2000.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stationary environments: Introduction covariate shift adaptation. 2012.
Masashi Sugiyama and Robert Müller. Input-dependent estimation of generalization error under covariate shift. Statistics Risk Modeling, 23(4):249-25 279, 2005.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robers Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8:985-1005, 2007a.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In NewrIPS, 2007b.
Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statisticnl Mathematics, 60(4):699-746, 2008.
Masashi Sugiyama, Makoto Yamada, Paul von Bünau, Taiji Suzuki, Takafumi Kanamori, and Motoaki Kawan- abe. Direct density-ratio estimation with dimensionality reduction via least-squares hetero-distributional subspace search. Newral Nelworks, 24(2):183 198, 2011.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Louis C. Tiao, Aaron Klein, Matthias W. Seeger, Edwin V. Bonilla, Cédric Archambeau, and Fabio Ramos. BORE: bayesian optimization by rensity-ratio estimation. In ICML, 2021.
Vladimir Vapnik. Statistical learning theory. 1998.
Vapnik VN and Chervonenkis A Ya. Theory of pattern recognition (in russian) nauka, moscow, 1974. German translation: WN Wapnik, A Ja. Tscherwonenkis Teorie der Zeichenerkensrang, Akademia, Berlin, 1979.
Robin Vogel, Mastane Achab, Stéphan Clémengon, and Charles Tillier. Weighted empirical risk minimization: Sample selection bias correction based on importance sampling. CoRR, abs/2002.05145,, 2020.
Vladimir Vovk. Testing for concept shift online. CoRR, abs/2012.142465 2020.
Zekai Wang, Zhengyu Zhou, and Weiwei Liu. DRF: improving certified robustness via distributional robustness framework. In AAAI, 2024.
Härdle Wolfgang, Müller Marlene, Sperlich Stefan, and Werwatz Axel. Nonparametric and semiparametric models, volume 2004.
Makoto Yamada, Masashi Sugiyama, and Tomoko Matsui. Semi-supervised speaker identification under covariate shift. Signal Processing, 90(8):2353 2361, 2010.
Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Relative density-ratio estimation for robust distribution comparison. In NewrIPS, 2011.
Bianca Zadrozny. Learning and evaluating classifiers under sample selection bias. In ICML, 2004.
Yu-Jie Zhang, Zhen-Yu Zhang, Peng Zhao, and Masashi Sugiyama. Adapting continuous covariate shift. via online density ratio estimation. In NewrIPS, 2023.
Zhengyu Zhou and Weiwei Liu. Sample complexity for distributionally robust learning under chi-square divergence. Journal of Machine Learning Research, 24:230:1-230:27, 2023.
Zhengyu Zhou and Weiwei Liu. Rademacher complexity for distributionally robust learning. AAAI, 2026.
Appendix
In appendices, we provide proofs of Theorem 1, Lemma 1, Theorem 2, Theorem 3, Lemma 3, Lemma 5 and Theorem 5 in the main paper. For completeness, present the necessary assumptions, lemmas and theorems in the main paper again. In addition, we provide related work in detail.
A Assumptions
Assumption 1 (Derivative Vanishing Loss). LD, (h, g) is derivative vanishing with respect to H. and 9, that 0g0, L.D (h", ,9) [h1 h² 9h - gol = 0 Wh, E H. Vg EG.
is,
(32)
In addition to derivative vanishing property, our main theorem requires first-order condition for the hypothesis class in the sense that h" is local minimizer.
Assumption 2 (First Order Condition). The minimizer for the risk satisfies the first-order condition:
TrL,, [ h-] 2 0 Yhi E star (H,h").
(33)
Further, we need a few technical assumptions.
Assumption 3. Define the norm over H. as: ||l||x = (EwDy [(2g0 (z) (2) (x))))) ki, ko such that the following holds:
There exist constants
16

---

Given Published in Transactions on Machine Learning Research (03/2026)
The risk Lo, is strongly conver with respect to the hypothesis: 4h1 H. and I1 9,
LLD, (h, ,) h², h1 h-12>llhp-h' -xl911 900lg
2kz wh E star h).
(34)
b There crists constant B1, such that the following bound holds: 4h, H and h star h))
L.D (b) h", ,,, hi) B1 ||/1 11" 1x
(35)
There erists a constant 82, such that the following bound holds: wh, E star (H,h"), 91 E g, and 9 E star go):
LD, (h",g) [h1 - h1,,., -
90- goll S B2 1 11 1191 - -9ollg
(36)
Lemma 4. Assume 9 is RKHS endowed with the kernel function k deftned on X X. The norm and Let GM {g E
inner product on G are denoted by Illllaa and (:-))1. Suppose that supzex k(x,z)
Gillells < M), and that the brackeling entropy (n(,, 9M, D2) is bounded above by O O(4),, where is constant satisfying 0 < <<y < 2, and t is an arbitrary number such that 1 parameter X Anamn. such that
2 <t< 1. Set the regularization
247
lim Annim. -, =0 ((na) (n +200
Dhat Ne +00
where n11 A min ne). Then, for = go E G, we have 11 gollen(D)) = ) probability at least 8.
with
Definition 1 (DRE algorithm and its convergence ratc). Let g a density ratio function class from X to Rt. Define و = l|gاl (D(g). Given S,1 D., St Di, let the algorithm DRE(G, S,1, St) can output
density ratio function 9, for which
1 - و e1 S.,,
(37)
with probability least 1-
Definition (IWERM algorithm and convergence rate). Suppose h is a minimizer of LD, (h, S.2 ~ D, and E 9, let the algorithm IWERMíH, can output a predictor h E H, for which
(h,9h - LD, (h',g) S E2 (H, S22, 6;g)
(38)
with probability at least 1 - 5.
B Proof of Theorem 1 in the Main Manuscript
Theorem 1. Suppose EZ-D, 10,, (h (x)) ,g (x) =)] = LD, (h, satisfies Assumptions to Let & be the output
of DRE(G, Sa,, S.). We then have:
If 32 in Item g of Assumption 3 is 0, then the Algorithmm i will produce predictor h such that, with probability at least 6,
LD, go) LD, (h"))))=
BIx
Sc
8) 2b2
B1 62
H. S22 5i0)
(39)
2X
If B2 in Item C of Assumption probability at least 1 -6, 6,
3 is
then the Algorithm
1 will produce predictor h such that, with
(h. Lo, (h", 90)5
8183
a a
5)
2hz
2
1. H. 9:
(40)
X
2xa
62
Since our proof repeatedly invokes Taylor's theorem, we first review it and provide its directional derivative generalization version.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Proposition 1 (Taylor expansion). Let S b be fized and let f - R, where S R. is open interval containing b. If I is (k + 1)-limes differentiable, then there exists E {a, bj such that
b 1 (b) b)' * 1)!
(a)
1
((k (c) (a- b)
(41)
Let F: R, where vector space of functions. For any 9.9 E F, if (t g 1) -).99 g') is
(k + 1)-times differentiable over an open interval containing [0, 11, then there exists g conv ((,g'') such that
F(g)= + Σ 1! (9) g.....g +
1 g F
1
(0) Ig' -'.....'-- gl.
(42)
times
Remark 5. The Gateaux derivative a one-dimensional calculation along a specified direction. Because it's one-dimensional, we can use ordinary one-dimensional calculus compute il. Thus, il is usually easy to compute Gateaur differential even when the space F is infinite dimensional. The Taylor expansion for the Gateaur derivative can be obtained by applying standard Taylor exponsion to the function of the single real variable T(t) = F(t g (1 (1-t) ... g'), where t E [0, 1].
Proof of Theorem 7 We start with second-order Taylor expansion on the risk at go to decompose the excess risk into two components. There exists h E star (H, such that
LD. go)
go) - (h", = GnLo (h" go) 1+]+ 2 LRLD, (h, go) ht, 1*]
(43)
By Assumption 2 we know the first part in Eq. 43 is non-negative. So we defer the focus on this item. Additionally, by Item in Assumption 3 we conclude
L,,,, (h, 90) 11
h- h*] < B1 1 11 2k1
(44)
Combining Eq.
43 and Eq.
44 leads to the following inequality:
LD, (h, go) LD,(h' (h",90) S InL (h", (h",9))(h - h-] + 31 11 h- 112
(45)
2
We shall bound l h 11 first. Using a second-order Taylor expansion on the risk at I, there exists h star (H, h²) such that
LD (h,0)) Ln, (h" On Lo, (h 1h- h*] 2
L.. (h, ( h² -h-,h-1h h-1]]
(46)
Applying second-order Taylor expansion to the first item of Eq. 46) again yields that there exists star (G.go) such that
On LD, (h', [1 h1] O LD. (h", go) 1h [1-11*] + 0,,0, LD (h*,, ..) 1² - gol
1 LgoLL.. (h* ))) [h h-t 9o. gol 2
(47)
Combining these two Taylor expansions (replacing the first item of Eq. 46 with RHS of Eq.
47
implics:
Lp, (h.0) (h" 9) = L,,L, (h" go) bi ] + 0,0n Lo. (h*, .0) h hh- - 1
(48)
+ 2 22 Lo. (h", ()" [h h², h go. g - gol + LRLL. (h.g) Ih h" h - h
Eq. (48 is the main incquality get the upper bound of I h" 11k For the sccond item of Eq. (48
Assumption yields:
0,0n I.D, .0) h- gol
(49)
18

---

Published in Transactions on Machine Learning Research (03/2026)
For the fourth item of Eq. 48 Item of Assumption 3 leads to:
LR (h. 9) Ih h" h*) A 2k xl6ø -
10oolll Whe star (H, h*).
(50)
For the third item of Eq. 48 Item of Assumption 3 concludes:
1/ 2
1 02 (h', (h",g) 1h h", -90.9. 9ol
-2 110
goll 11 h- 11
(51)
Let d' 0 be any constant, solle and b= 11
ht
Vd. By the Algorithmic Geom-Geomcrricc Mean
va
(AM-GM) inequality, namely, Va, b R, ab v a²4b²
2
a
+
b
gollg 2kk
l| - 19-1oll
Ro I1 11 = ab <
+ 11A - 111 d
(52)
2
2
Based on Eqs.
48
to 52 we get:
h ht 11 S ف|ا - g0llg
2hz
-
2
[h..(h.)
(53)
B2
322
On OnLL, (h" g0) [h- ho
|10
go g0
2
11
11"
1122/
2dA
2x
To combine like items, we need to discuss this inequality in two cases.
82 =0 Replacing B2 with O in Eq. 53
|/ h 11
7 11 gol
g
2kk
(h, Lo, (hi,, (h",0)
2 x LnIlo (h*,,00) Sh nt)
(54)
Combining Eq.
45 and Eq.
54h:
LD, (h, L.D, (h',,'') 31 l1 golle 2kk B1 (Lo, th,
(55)
B1 LDD (h* gu) jh n-1
(1 X
By Assumption 2 and using the fact that 4 2 1 without loss of generality, we get:
$1 101
31 (iro (h, - LD, (h* (h+ .))
(o, (,"",,, (h" -
(56)
2x
Lastly, we use the convergence rates assumed in Definitions 1 land 2
PS. [ll gollg > e1 (9, S.1. Se 8/2) 8
(57)
8
8
PS (Tp, (h. Lp, (h* e2 S.2, 5: 00]
Since Sa1, Sr, S.2 are mutually independent, we have
PSA.DD.S. [l] gulle > c S.1. S₁ 8/2)U
8
8
(58)
Ln, (h, LD, (h* 9) ea (H, S.2 5i 9)]
+
d
2
2
3 6 S,, S,, 8/2) 2kz
31
8
L.,, (h. Ti" (li,,0)) s
E2 (H, S.2, 5:
(59)
2x
with probability least 1 - 8.
19

---

Published in Transactions on Machine Learning Research (03/2026)
2. [2 +
Let d Eq.
Then
0 2x
and
Rad 2X
Moving items and combining same kind of items in
32
2dx
53
[1 h" 20g
2k
82
110
90
gullg*
2
4 x
2 0n AnIn,(h", - h-1
x2
x
(60)
Analogous the above case, we conclude the following inequality by combining Eq.
45
and Eq.
(60)
T.D, (h, 90) ID,,(h' (h,, g0) S (xx
111
31
|1 go 0ollo 2k=
212
(61)
+ 2 (h. - LD, (1" . Lo, (1*,
(1
31
On LnLD, (h", (h)0l h²]
X
Likewise, by Assumption and using the fact that
$1 2 1 without loss of generality, we have:
Bix
3133
28 (Lo,(h,3) - (h",g))
(h. LD. (h', (h',g)) go) S
11
go 2 +
(62)
2x
g
A
Using the assumed rates again, we get the desired result:
81K A
3133
2 (7H, S
8
(h, (h, g0) T.D,(h', (1" go) s
61 (G. S,,
z:
(63)
212
with probability at least 1 1-6. 6.
C Proof of Lemma 1 in the Main Manuscript
Lemma 1. Suppose Elylæ] h² (x). Then ē (G, = (S - y) -y has the following properties:
1. V (2)) [z]] =0
2. E| E (h" (z) (z)::)(((())) - ht (z))] = 0 vh E star (H. h*)
3. V (ht (z)) = Ag E star (G,
Proof of Lemma
1 Property 1.
Vge(h" (z) (x) 90 (x) 2- (h" (x) y)
V VeL (x) (z) (h' (x) y)
(64)
E V (h* (x) .9 (z) [x] 2E( (x) [x] = 2E(h (x) - -Eyy = 0
where the final formula follows from h" (z) E[y[z].
Property 2.
V (z)
(x))
Elgo (z) - 2 (x) (z) (x)]
(65)
2IE[E[go (I) (h" (x) (x h1" (x)) |<]
= 2E[go (x) (z))) h² (z)) (1 (x) - -) |x] =
20

---

Published in Transactions on Machine Learning Research (03/2026)
Property 3.
(=) (z) 2(" (z) - y)
V, (.E (h* (x) v)
(66)
D Proof of Theorem 2 in the Main Manuscript
Theorem 2. Suppose Elyla:] = ht (a), Me h (x) S Mr, Vg E G E (x 21 0, 90 (x) 2 4/
Then LD, g) Elê (h, (h,g: g;=]] satisfies Assumptions to S
with À 1.4 1 1, A/ =,
2/ B1 1 and
B2 0.
Proof of Theorem
1. Assumption 1
On 900 1², 9nl
V. V,, (1* (g(x)))))))))))--))))))))))))>))
(s(x) 90(x))]
(67)
B[(h(x)- h1 (x)) V
(g(x) go(2))]
From Lemma
we know that E V (h* (x) (0)))()))))))) (x); Then, the above formula is zero.
Assumption is proved.
2. Assumption 2
[x(x)) we have B[[ (z)) ( (x); (M(x)) - 11" (x)) 2 0, vh E star (H, ht,
O D. (h",,))]l - h* (M
(68)
By Lemma
Therefore,
Ön LDD (h*, 9o h*] 2 0. Assumption is satisfied.
3. Assumption 3
(a) Item a
Vh' E H, [ho,,
(69)
haga://oohjaaaa4H444-0001H-AAAA4 -(()))
Therefore,
(1(c)) ((()()(((((())))))))((((((((((() (x)))
(70)
Let a = (2)(z)(((x)- - h*(x)), b (1)20(rr) (2)(z))(((z)) h" (*)))) Then + b h²())) By AM-GM inequality, we have
2h(x)(h(x) ²²)))) = d² d²> 2 (a (a+ 1²
(71)
1 230
Combining Eq. 70 and Eq.
lcads to:
8. (h, g) ht 1:1
(72)
2 h h E[ - (MG)))
Our following task is to lower bound the second part of Eq.
72
2
(2 - 2g(xx)
1
(v2g0(z) - 22g(x))²
(go (2) ()))))
(73)
(2)) + 22((z)
2n
21

---

Published in Transactions on Machine Learning Research (03/2026)
Multiplying (h(z) h" (z))) by both sides of Eq.
73
implies:
(2t) 27y
(74)
Let c >> 0 be any number.
2)V=(00(z) g(x)) b (((h()))><<))33))) AM-GM inequality
yields:
1
(g0 ( ( (h(z) (x)) 2// (00))))))))))
211
a² b2
Ty.C (20 g(0)) + - -
= ab -
(75)
2
2
(go(x)) - g(x))*
(h(x) - h1" (x))²
811²c
Combining Eq.
74 and Eq.
75
leads to the following inequality:
(g(())
C (b()))+)==))11
8712c
(76)
20 Il c (1(
81/2c
Therefore, suffices to upper bound [((((x) h² (x))* using | 11 1
[^(x) - 11" (x)) < M(x) + 11 (x) < 2Mr
IM(x)) - h²(x))² < 4MA
(77)
[1(x;) - -'(x)]1 < 4M)()(s) - (x))²
[[(h(z) - --(x))) < 4M) (x))
Since Ag E G, x E X. 2 O,
E[200 (w) (x) h (x))²
z 2m
(78)
E E(h (I) h" (x))²
Combining Eq.
72 Eq. 76 Eq.
and Eq. 78 yields:
cM2 }| h" 1+
LiD. (h, g) h h* h h+1
l1g
gu l1
(79)
87/²c
Let 7 aMg.
Then,
cMR 1
M
11 Si1=
2T
1
M2 2/y²
L.D. (h () -h,, 127 2 1 111 - 1--112
|1g goll
(80)
M
Item a is proved with corresponding constants X = 2, 27
(b) Item
By Eq.
69
we get:
0 LD. (h,9) [h - h² h - - El2g (x) - - h²- (x))²
(81)
h/ h" hl < 1 ||/ 11
Ttem b is satisfied.
22

---

Published in Transactions on Machine Learning Research (03/2026)
(c) Item
For all h E star (H, h"), 9, and star (9,90):
0y Lp. (h',g) [1- [-) (g(x) 90
(82)
E B 77 Vel (h' (s). -
Let [(X) V Vel (h* (z),
- -,
=
(83)
By Lemma 1 we know: Vg star (9, E())
82 Ld. (h", wow.::.......0000.... | 90()))
(84)
Using Cauchy-Schward [ngg
implies:
(h" ( xg
(85)
00v (=) BlliVlll()xxx00)) (0
By Eq. 78
we tell | - ht (1,,)p - 1 h h 1|
22n
0 (k" [h- h², .9 9u. 9 goll s 0- 1²1l 119
9ul1
(86)
Ttem c is proved.
In summary, there exist constants k1 1, 1: 2 such that the conditions in Assumption 3 holds.
E Proof of Theorem 3 in the Main Manuscript
Theorem 3. Let H = fo sig ((w,x)) x)) l|lle2 - 1} and ((h(z). y) (h(z) -y) Suppose Ve X, |2||| S B and Ag E G, supapex (z) & M. When given Sa2 ~ D, and any g, IWERM(H. S.z; g) outputs a predictor h H for which
İ.D, (h. - L.D (h",9)
2B d dlog (1+ +
2 2 (+)
(87)
26
S 8M
inf
24
2M
Mx
/n2
112
where D = sup an EB(0)1) SuP=Ex iig ((w, 23) sig (ww', ())) and no is the size of S.2-
In this section, we provide the proof of Theorem 3 in detail. First, we state several key definitions and lemmas.
E.1 Definitions
Definition 4 (Rademacher average (Boucheron et al., 2005 ). Let A C R be a bounded set of vectors
a = (ai,.... on); the Rademacher average associated with A is denoted as:
Rr (A) = E, (sup
Σ a,aill
a n
1=1
where (aiy..., .n) is the Rademacher vector, namely, 01,-.. Tn are independent wniform (+1).
23

---

Published in Transactions on Machine Learning Research (03/2026)
Definition 5 is Gaul-Gnssnnn if there is posilive number of such that.
random variable Martin J. 2019 ). A random variable V with mean μ B[]]
RlEAV-
222 S
AAER
(88)
Definition 6 (Sub- Gaussian process (Martin J.
2019 A collection of zero-mean random variables {X,,f f E
F) is sub-Gaussian process with respect to metric px F is
x x
E(e^(xx-X:))
<
for all f.f' E and À E R
Definition 7 (Covering number (Martin J. 2019 A 8-cover of a set T with respect to metric p is
set ..'...., 0 C. T such that for each E T, there crists some i {1....,N} such that p (0,0*) S 8. The 6-covering number (8;; T, p) is the cardinality of the smallest 8-cover.
E.2 Lemmas
Lemma 6. A Rademacher random variable is sub- Gaussian with parameter 1 1.
Proof of Lemma
0 By laking expectations and using the power-series for the exponential, we obtain:
Ele 1 (e- 1
}
2k
2h
E 4f
(89)
k!
k!
(2
2
k=
which shows that o; is sub-Gaussian with parameter 1 as claimed.
Let { ff E F} be zero-mean sub- nab-Gaussian process with respect the metric px. Then, for any A [0, DJ such that Nx (A;F) 2 10, we have
Lemma (Proposition 5.17 of (Martin J. 2019
E
sup (IEF
(X, - Xp) 2E
sup SoTEF Px(In')s
(X, - (X, Xgr) + 44/D2 De Nx (A;F)
(90)
where D = sup Px (F.8) denotes the diameter of F.
Lemma 8. Let B (0, 1) be the unit ball with respect to 11-112, and
N
(0, 10,1,,11-l
the -covering vamber of (0, 1) with respect to the l |1-llammetrie. Then, Nx (A;F) s
N
B (0. 10.,),.1. 11-1/-
Proof of Lemma Assume {Wi, (wi...., wn) is 44 -ooeerin of B 10. We then have: Va B (0,1). 3w 4A
B Then, Af' sig ((w', - y, 3ʃt sig ((w,, - y such that IIfe f |x S
such that l|w; w'll2
B
1 Hll:; - w ||0||| <
44
= A. Therefore, Nx FA;F) < N (4: B (0, 1), 11-1122)
B
Lemma 9. log N A B (0, 1), 11-112) < d dog (1+
B 23
Proof of Lemma
Example 5.8 of Martin J.
2019h concludes log N (A; 1),111)))))) dlog (1 2)
Therefore, log
(0, 1), 11-112 S dlog (1 x
Lemma 10. (t) 2 is p p-iipschitz over the set C (t: ||| s ()
Proof of Lemma 10 For any 4, E C, 10f 12 = 1/1 + 12/1/1
1a 2 g/4 122 = p/h 12
24

---

Published in Transactions on Machine Learning Research (03/2026)
Lemma 11 (Lemma 1 of (Vogel et al, 2020)): ) When given sample 21,..., from distribution D, and any g 9, IWE 9) outputs predictor h H. for which
1 ofe (h, x)ll
LD, (h. LD. gh) S 4|lg| Be
sup Eo
Σ
hEH T
(91)
2 2oo (1)
2|1g|o0 sup (M.))Hx2
(h, V
n
with probability at least 1 8.
Proof of Theorem Lemma 11 has given an upper bound of the estimation error except that the Rademacher complexity item is not well-bounded. Therefore, it suffices to bound that item.
- 1/2 U2....,, M(T,) vn)) : h E HJ. f (=) - h (x) y, and a; = h (2;) - - f (z)) Let. (1) = 12 Then, Φ o contraction principle (Ledoux & Talagrand 1991)) states that o R - R. is a function with ( = 0 and Lipschitz constant L& and Φ
Let A = (=(((()) 1/,
h(22) -1,
d : E H). E, (SuPREn 1 - age [h)] = R, ( A). The
the set vectors of form (a1) ...... (an))
E R" with (aI....,an) E A,
then Ri (ø A) RN (A). This principle plus the fact that ((() (2 is 2-Lipschitz (Lemina 10) yields: Rn (db a A) 29,, (A) 2 [suprer 2 #E ozf (z:]],, where F {s f(z) Mxx) y. H). So
bounding R. (A) is our following goal.
1 Σ o. ( (2)) and F = FU-F We then have Let X v
n o.f (zx)
Rs (A) = E,
sup-
Σ i=1
Eg
sup
Σ
ord (zi)
fEF 71
J 11
1=1
(92)
= E,
sup
X
=
Eg
sup X
5
Es
sup X, - X's -Xy
Vn
LEFF
v
Lire
It is easy to prove that f E F) is a sub-Gaussian process with respect to the metric px (f.f) =
|| - -flloo whose proof is as follows:
Eg (c (X1
B Ao Ein ((()))))) II: E, le do. ((()))))((
(93)
[(((()))((x))
< II 1,, 1211-1112
1911-1112
II Bo,
(Lemma
Definition
25

---

Published in Transactions on Machine Learning Research (03/2026)
Let D supj.reF (f. By Lemma for any A [0, DI, we have:
Eg
sup S.
X, - X3
2
sup
(X, - + +4, +4 log Nx (A;F)
To'er (x())
2
sup EY
os (2)) - -
+4 D2 log Nx (A:F)
vn 1=1
lb SA
ooll (z)) - (201)
$ 2E,
sup T. E 1ow
+4V D2 Nx (A:F)
(94)
i=1
A
7
$ 2
+4V D2 log Nx (gg Nx (A;
4A B
< 22/nA + 4D +4D
log N Voog
(0, (0,11),1-lb))
(Lemmna
B
$ <2vnA + 4D ddlog
1+
(Lemma g
2A
Moreover,
D sup px (f.f'= Li'
sup sup {ssg ((w, - y sig ((w', + y1} S.E'F (m.))xxy))
(sig((w, x)) sig((w' 0s fi (x.)) 1 (1, (1) ,
sup f.f'FF (xx))xxx
sup
sup
sup
(95)
1
B
sup sup f.f'FF EX
4 -w0-
S
sup f. (x))
sup
Ilw cou--wll221 12122|10112 S
Combining Eq. (94) and Eq. (95) yiclds: E. [sup 5 X,
2/T4 2B dlog (1 2. Since Eq. (94
holds for any A E 00 D), we take the inf of the upper bound, namely
B
E
sup ,
<
inf (x
2/1 + 2B dlog 1+
(96)
24
SEF
Combining Eq.
992 and Eq. 96
obtain:
1 Vn AE
B
Rn (A)=
E sup Xy L
inf
2/n dlog
1 2A
(97)
Vn
Lastly, by Lemma with probability at least - 8,
(h.gh - (D, (h),
2 2log(
< 4|gg| 20 R m ~ RRn (cb +)]+2|lglle
sup E (hh (h,))HHZ
2B, V
dlog B
23
2 lo
(98)
<81gg l00 inf
2A
21|gll (N)))HXZ
sup
l (h, vn
AE
vn
n
(x
2B dlog (1
2 log (+)
<SM
inf AC00.0).
2A
24
12M
(by sup g (x) wex
Vn
n
(x(4-))210
where D supp..wwwww w. supxex sig ((w,x)) sig ((w',
26

---

Published in Transactions on Machine Learning Research (03/2026)
F Proof of Lemma 3 in the Main Manuscript
Lemma
Suppose Vg
S,supzex (x) S M, 90 (x)
m
Then 11 gollaa(oy)
2 (n + M) hp. (9. 90)
Proof of Lemma
hp, (0. (0.90))=1
//6
(V0 (o) (x)
Po (I) dr-
(0(x) 90 (x)))
P (2))
x
V (x) V (2)
(99)
(g(x) 9u (x))
1
2 x (VM = VTII
2 Po (x) dx 2
11 (D))
(M m))
1 (x) 90 (x:))* Ps (2)) dr
ll 9ollin(02)
x
0 (xx) 90 (x) (0(x))
111 + M Po (x) do dos(m //2
-9o (x)
)os (x)dx
(100)
m + M
(n + M)² 2 1 go golli. S 2 (m + My hp, 90)
Both sides take the 1/4 power yields the statement of the lemma.
G Proof of Lemma 5 in the Main Manuscript
Lemma 5. Suppose Vg E Supuree g(x) M. Let M& c² log(f), where e is the same constant in Lemma 4
Then, 119 go Il (P;;) (M + 2 9u (P::)
Proof of Lemma
l1 (ooolli ( p.())do / ( (0(x)-90) pa(x)dd
M+M
(101)
((x) g0(z)) ) (2) 1 -oo (0!1)
S(M + Ms)
M + Ms
H Proof of Theorem 5 in the Main Manuscript
Theorem 5. Suppose Yg E G, suparex g(x) & M. Then, - 1oloooll.. S sO (c² (cllo(())A with probability
al least 8, where is a constant.
Proof of Theorem By Lemma
we have
|1 (ol S M + e² (lo8(
110 lio-solli sullin
(102)
-0 + 0005+t 10gt ())²
$ (2110(())
7 *xtt [Lomma O (e&log log( ) st.m.)
27

---

Published in Transactions on Machine Learning Research (03/2026)
Related Work
1.1 Covariate Shift and Sample Selection Bias
The situation where training and test distributions are different is called dataset shift -ooreso-Toorrss et al.) 2012), in which generalization guaranlees on the test distribution are impossible without further assumptions OSen-Davite et al. 2010). Well-chosen assumptions, which can be categorized into three classes: covariate, prior, and concept shift (Moreno- Torres et al., 2012 Amos 2009), can make possible algorithms
with non-vacuous performance guaranlees.
There are several possible causes for covariate shift, out of which this section mentions the two deem most important: the mássing at random (MAR) case in the sample selection bias and non-stationary environments. Sample selection bias (Cortes et al. 2008) relies on a model of the data generation process. Test instances are drawn from (x)) Training instances are drawn by first sampling æ from the test distribution (x()). A selector variable s then decides whether x is moved into the training set (s = 1) or moved into the rejected set (s 0). For instances in the training set, a label is drawn from p(y|z). For the instances in the rejected set, the labels are unknown. A typical scenario for sample selection bias credit scoring. The labeled training sample consists of customers who were given loan in the past and the rejected sample are customers that asked for but were not given loan. New customers asking for loan reflect the test distributiin. The true but unobserved test distribution Dj can be expressed by
pc(x,) v) p(u, y 1)) p(s= p(s = 111,1 (m(
(103)
p(s = 1 x.
In the MAR case, the selector variable is only dependent on v, but not on yi that is, p(s 1,y, p(s 1 lz)) Therefore, the conditional distribution of the labels given features is the same for both the training and the test data:
(ulx)) = p(ylæ,, = 1)= ((p)p(yl=)pp( = 12,,8) = (x(2))
p(z)p(s 1|z)
The MAR bias leads to covariate shift. The second cause for covariate shift appears when the training environment is different from the test one, whether it is due to temporal or spatial change (Jang et al.
2022 r. et al.
2014
Vovk 2020:
Podkopaev & Ramdas 2022).
1.2 Density Ratio Estimation
Empirical risk minimization may fail under covariate shift due to the difference between the training and test distributions. Importance (often referred to the ratio of test and training instance densities) weighting is used to mitigate the influence of covariate shift (Shimodaira) 2000 Sugiyama & Müller 2005 Sugiyama et al.
2007a Zadrozny| 2004), which leads to IWERM. Therefore, how to estimate the density ratio accurately becomes the key to success of covariate shift adaptation.
There are two types of density ratio estimation algorithms. The first one is to first separately estimate the two probability densities, and then take the ratio of the estimated densities. However, density estimation is known to hard problem [Vapnik 1998 Wolfgang al. 2004 Kanamori et al. 2010). Therefore, this naive approach may not be effective. The second one is to directly estimating the density ratio without estimating the densities. Various direct density-ratio estimation methods can be divided into four types: Moment Matching, Probabilistic Classification, Density Fitting and Density-Ratio Fitting (assashi et al. 2012). Of all
the direct methods, KLIEP (Sugiyama et al. 2008) and KuLSIF (Kanamori et 2012) are two commonly used ones which are computationally efficient and comparable to others in terms of performance YYamada)
et al. 2011 Menon & Ong. 20166: Liu et al.) [0117] Kumagai et al. 2021 Kato & Teshima
2021 Tiao et al,
2021
Zhang ct al.
2023 We include them our analysis.
1.3
The Analysis of Effect of Density Ratio Estimation Error
The closest previous work to ours is Cortes et al. (2008). They consider the problem of determining
what extent. the error in the weight estimation can affect the accuracy of the hypothesis returned by
28

---

Published in Transactions on Machine Learning Research (03/2026)
weight-sensitive learning algorithm in MAR bias problem. Their theoretical results differ from our work in three aspects.
First, there is a key difference between MAR bias and covariate shift. The MAR bias imposes a stricter criterion on the relationship between the training distribution and the test distribution than covariate shift. That to say, there are some instances of covariate shift that cannot arise from MAR bias, but every instance of MAR bias can be modeled as covariate shift. Consider the problem of data analysis using a brain-computer interface, where the distribution over incoming signals is known to change experiments go (subjects tire, the sensor setup changes, etc.). In this example, covariate shift that cannot be represented by sample selection bias occurs.
We further illustrate the difference between MAR bias and covariate shift in terms of the correction met hod. The MAR bias correction technique commonly used in machine learning consists of reweighting the loss on training examples to closcly reflect the test distribution. Proposition / illustrates that minimizing the loss on examples weighted by p(s 1|x)- in fact minimizes the expected loss with respect to Dr.
Proposition 2 (Coortes et al., 2008 Zadrozny 2004)). The expected loss with respect to D, is proportional to the expected loss with respect to D. with weights p(s 1[x)) for the loss incurred by each x, provided that the support of D2 is contained in the support of D
1 EL-D,I((h(a), v\l x BawD, pls 1|x) (()()).))
(104)
When the model is implemented, correcting for MAR bias amounts to estimating p(s = =/2). This cstimation can be accomplished by building classifier that discriminates the training against the rejected examples. The target model is then learned by following Proposition 2 and weighting training examples by p(s 1 [x)) No test examples drawn directly from DE are needed to train the model; only labeled selected and unlabeled rejected examples are required. This is in contrast to the covariate shift model that requires examples drawn from the test. distribution, but no selection process is assumed and no rejected examples are needed. Covariate shift models can be applicd to learning under MAR bias by treating the selected examples as a labeled sample and the union of selected (ignoring the labels) and rejected examples as an unlabeled sample. In contrast, MAR bias models cannot be applied to learning under covariate shift problems that are not caused by MAR bias. Compared with Cortes et al. 20088), we consider a more general case in which covariate shift is analyzed matter what the cause is.
Second, the risks analyzed in Cortes et al. (2008) and this paper are different. Let hgo denote the predictor returned by IWERM(H, S.2; Theorems 2 and 4 in Cortes et al. (2008) provide bounds on the difference between the generalization error of h. and h (i.e., LDD (h. LD, gh 9o)). In contrast, we consider the difference between the generalization error of the hypothesis returned by IWERM based on the estimated density ratio and the best one in the hypothesis class (i.c. Lo, (h, 90) LDD
For the weight estimation error, they only consider the weights for the training sample, while our analysis consider the difference between the real density ratio function and the estimated density ratio function. Therefore, the analysis in Cortes et al. (2008) is orthogonal to our work.
Third, the techniques of deriving bounds are different. A weighted sumple Sw is a training sample S, that is augmented with a non-negative weight. W; for each example 2j. The sample weight W of Sw defines a distribution over S,. For a fixed learning algorithm L, denote by hw the predictor returned by L for the weighted sample Sw. Denote by d(W, W") a divergence measure for two distribuutions W and W The key concept for deriving bounds Cortes et al. (2088) distributional B--tability.
Definition 8 (Distributional 3-Stability (Cortes et al. (2008)) A learning algoritlom L. is said to be distributional B-stable for the divergence measure d if for any two weighted samples Sw and Sw
Vz E (,1((h(()) - E(hwr (x), s Sd(W,w")
(105)
Thus, an algorithm is distributionally stable when small changes to a weighted sample's distribution, as measured by a divergence d, result in a small change in the loss of the predictor at any example. Proposition 2 in Cortes et al. (2008) shows that kernel-based regularization algorithms are distributionally f-stable.
29

---

Published in Transactions on Machine Learning Research (03/2026)
Proposition (Proposition 2 in Cortes al. (2008)). ). Let L be distributionally B-stable algorithm and let hw(ww) denole the hypotthesi relurned by I. when trained on the weighted sumple. Sw (respectivelyy Sw Then the following holds
LD, (hw) - LD, (hw 3d(,.w).
(106)
In contrast, we demonstrate that the true risk satisfies a. condition called derivative vamishing property, the impact of the density ratio estimation error on the excess risk bound of the two-step covariate shift adaptation algorithm is of the fourth order.
Another work relevant to ours is Reddi et al. 2015p. To address the issuc that plagues many covariate shift correction algorithms, namely that the variance increases considerably whenever samples are reweighted, this paper proposes a doubly robust estimator using the unweighted solution as a variance-reducing proxy for the correct weighed solution. Moreover, they analyze the generalization bounds for the Penalized Risk Minimization (PRM) estimation algorithm based on meta theorem. However, they only give the generalization bound of the final hypothesis. The effect of the density ratio estimation error the learning algorithm is not explicitly illustrated by Reddi et al. (2015).
30


Reviews:
[{"review_id": "l1Xl1hlIDD", "reviewer": "Reviewer_6oqd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a theoretical analysis of two-step covariate-shift adaptation methods that combine density-ratio estimation with importance-weighted ERM. Under a derivative-vanishing assumption and technical smoothness/convexity conditions, it shows that the excess risk contribution from density-ratio estimation is fourth order for square loss. The framework is instantiated for KLIEP and KuLSIF using known convergence rates, and a small toy experiment illustrates qualitative trends.\n\nStrengths: clean error decomposition; clear fourth-order result for square loss.\nWeaknesses: strong assumptions; limited novelty beyond recombining known rates; very weak empirical validation.", "claims_and_evidence_yn": "No", "claims_explanation": "The theoretical claims are supported under the stated assumptions, but these assumptions are restrictive and only clearly justified for square loss. Empirical evidence is minimal (single toy example) and does not validate the claimed rates or practical relevance.", "audience_yn": "Yes", "audience_explanation": "The results may interest researchers studying covariate shift and importance weighting, but are unlikely to appeal broadly due to limited practical validation.", "requested_changes": "Clarify or narrow the scope of the derivative-vanishing assumption.\n\nStrengthen experiments (non-toy data, include KuLSIF, scaling studies).\n\nClearly articulate novelty relative to prior work.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aEMcx5lwXq", "reviewer": "Reviewer_q3yo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies two-step covariate shift adaptation: (i) estimate the density ratio using a density ratio estimation; (ii) run importance-weighted ERM (IWERM) using the estimated ratio. This paper provides a quantitative answer to how does density ratio estimation error propagate to the final learner’s excess risk? The authors formalize the two-step pipeline as a meta-algorithm (Algorithm 1) and show that under a derivative vanishing property of the true risk, the impact of density-ratio estimation error on the excess risk bound is higher order. They further instantiate the framework for KLIEP and KuLSIF, deriving density-ratio error rates and plugging them into the meta bound, plus a toy experiment demonstrating that improved ratio estimation correlates with lower test error.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The core assumption (“derivative vanishing”) is strong. The main result and derivation of the analysis hinge on Assumption 1. The paper argues square loss satisfies the assumptions under additional conditions, but it is not clear how broadly this property holds for common classification losses and modern models.\n\n2. Scope of the empirical validation is very limited. The experiments section is essentially a toy 2D synthetic classification with KLIEP, varying kernel width and showing test error decreases as NMSE decreases.\n\n3. Practical implications are not fully developed. If the main message is “DRE error enters at higher order,” then the paper should translate this into actionable guidance.", "audience_yn": "Yes", "audience_explanation": "1. A clear, well-motivated question with a clean abstraction.\nThe “meta-algorithm” formalization of two-step covariate shift adaptation makes the analysis modular.\n\n2. The claim that density ratio estimation error can enter the final excess-risk bound, which can be insightful.\n\n3. Concrete instantiations for KLIEP and KuLSIF.\nThe paper attempts to connect high-level theory to classical DRE methods", "requested_changes": "1. How often does derivative vanishing hold beyond square loss?\n\n2. Can you empirically test the predicted scaling?\n\n3. Add experiments at least one realistic benchmark.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6WT3I2kteL", "reviewer": "Reviewer_2KMo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to clarify the effect of the error in the density ratio estimation on the training of a model for covariate shifts.  \nIn machine learning problems, the distributions between training and test samples may be different, and the covariate shift is an assumption in the difference; the covariate shift assumes that the distribution of the input variables between the training and test samples are different ($p\\_s(x) \\\\neq p\\_t(x)$), but the distributions of the output variables conditioned by the input variables are unchanged ($p\\_s(y|x) = p\\_t(y|x)$). In such a case, it is known that we have only to train the model by, for each training sample, we set the weight on the loss for the sample by the density ratio of the test distribution to the training distribution at the sample.  \nHowever, since the true density ratio is usually unknown and we therefore need to estimate it, the error of the estimation affects the performance for the test dataset.  \nThe paper showed the following facts. Firstly, they showed that an upper bound of the difference in the loss (defined by the loss function) between case using the true density ratio and the estimated density ratio is quartic of the possible difference caused by the density ratio estimation algorithm (not the difference of the result of the estimation algorithm; see Definition 1 and Corollary 1). Then, for two density ratio estimation algorithms KLIEP and KuLSIF, they showed that the upper bound is reduced in $\\\\mathcal{O}(n^{a})$ ($-1 < a < 0$), which was worse than a desirable order $\\\\mathcal{O}(n^{-1})$.", "claims_and_evidence_yn": "No", "claims_explanation": "Analyses made in the current paper themselves look sound, but there were several points that are not sufficiently convincing to the reviewer due to the lack of desired analyses. Also, the experiment was not sufficient to convince the reviewer. Please see \"Requested changes\" part for details.", "audience_yn": "Yes", "audience_explanation": "Covariate shift is extensively studied as an assumption of the distribution changes between training and test distributions, and novel analyses have been made. Also, they analyzed two existing density ratio estimation methods to derive the rate of the error reduction with respect to training sample size. They are fundamental things to be analyzed.", "requested_changes": "## Points critical to securing my recommendation for acceptance\n\n- Section 2.3, Definitions 1: It defines $\\\\|g\\\\|\\_\\\\mathcal{G} = \\\\|g\\\\|\\_{L\\_4(\\\\mathcal{D}\\_s^x)}$, but in the preceding sentence $\\\\mathcal{G}$ is defined as a density ratio function class. What does the definition intend? Does it just intend that the norm of $g\\\\in\\\\mathcal{G}$ is calculated by $\\\\|g\\\\|\\_{L\\_4(\\\\mathcal{D}\\_s^x)}$?\n- Section 3, Definition 3: Since the definition is given for $\\\\partial\\_t T(t)[t^\\prime]$, the definition of the expression $\\\\partial\\_a T(b)[c]$ when $a\\neq b$, or the expression $\\\\partial\\_a T(b)[c]$ when $c$ includes the variable $a$ are not clear (but used in many places like Assumption 1). Please provide clear definitions even for such cases.\n- Section 4.1, Theorem 3 (including conclusions of Sections 4.2 and 4.3): can the $\\\\Delta$ that attains \"inf\" be analytically or computationally identified? If difficult, can its lower (upper?) bound be computed?\n- Section 4.2, end of the section (same for Section 4.3): It states that \"It implies that there is still a distance between the prediction error rate that can be achieved by KLIEP and fast rate, i.e., $\\\\mathcal{O}(n^{-1})$\", but is the rate $\\\\mathcal{O}(n^{-1})$ is considered as sufficiently fast in this field? If so, please add brief explanation about this fact and/or add references about existing discussions on the rate.\n  - Related to the above, are KLIEP and KuLSIF almost the best existing methods of density ratio estimations under covariate shift in terms of the convergence rate?\n- Section 5: Experiment with only one model and one dataset is too limited to support the proposed method. Consider adding for both, and (if the page is limited) show one result in the main paper and others in the appendix.\n- Section 5: It states that \"It is illustrated that the test error of the predictor returned by EIWERM for most of the $\\\\lambda$ values decreases when NMSE decreases\", but the reviewer is not convinced with this consideration.\n  - The relationship that smaller NMSE of density ratios yields smaller test error can be seen when $\\\\lambda$ is relatively small (near to 0). However, to make the sample weights in the training being similar to true density ratio, larger $\\\\lambda$ (near to 1) is needed. This discrepancy made the reviewer unconvinced. Are there any plausible reasons why such results are produced?\n\n## Points that simply strengthen the work in my view\n\n- Overall: Please add references for functional derivatives (e.g., in Definition 3) and their theorems (e.g., Proposition 1 in Appendix B). It seems that they are not so obvious for researchers in this field.\n- Section 2.1: It looks that the term \"starhull\" is not so standard. (The reviewer heard the term for the first time, and there was no result in Google search for the term.) It seems better to \"define\" the term, for example, \"We define the *starhull* of $x \\\\in \\\\mathcal{X}$ as $\\\\mathrm{star}(\\\\mathcal{X}, x) = \\\\{ t\\\\cdot x + (1 - t)\\\\cdot x^\\\\prime \\\\mid x^\\\\prime \\\\in \\\\mathcal{X}, t \\\\in [0, 1]\\\\}$.\"\n- Section 2.3, Definitions 1 and 2: These definitions are the *assumption of the existence* of a DRE or an IWERM algorithm satisfying the inequality. So, it is better to write like \"Given $S\\_{s\\_1}\\\\sim\\\\mathcal{D}\\_s, S\\_t\\\\sim\\\\mathcal{D}\\_x$, let the algorithm $DRE(\\\\mathcal{G}, S\\_{s\\_1}, S\\_t)$ can output a density ratio function $\\\\hat{g}$,\".\n- Section 3, Assumption 3: In the first expression $(\\\\mathbb{E}[(2 g\\_0(x) h(x))^2])^{\\\\frac{1}{2}}$, for what distribution the expectation $\\\\mathbb{E}$ is taken? Please specify. (Perhaps $\\\\mathbb{E}_{z\\\\sim\\\\mathcal{D}_s}$?)\n- Section 3, Remark: It states that \"This suggests that if the desired oracle rate\", but what rate does it intend? (Perhaps the rate of hypothesis's excess risk?)\n- Section 3.1, Lemma 1: When we use the notation $\\\\nabla\\_\\\\zeta$ or the like, if the variable name is replaced, it becomes difficult to identify the variable with which the gradient is taken. For example of Point 1 of Lemma 1, instead of writing $\\\\nabla\\_\\\\zeta \\\\nabla\\_\\\\gamma \\\\tilde{\\\\ell}(h^\\*(x), g\\_0(x); z)$, it is clearer to write as $\\\\nabla\\_\\\\zeta \\\\nabla\\_\\\\gamma \\\\tilde{\\\\ell}(\\\\zeta, \\\\gamma; z)|\\_{\\\\zeta = h^\\*(x), \\\\gamma = g\\_0(x)}$.\n- Section 4.1, Theorem 3 (including conclusions of Sections 4.2 and 4.3): As far as the reviewer's understanding, $N_X(\\\\Delta, \\\\mathcal{F})$ is a decreasing function with respect to $\\\\Delta$. So, instead of defining \"$\\\\inf\\_{\\\\Delta\\\\in[0, D],\\~N_X(\\\\Delta,\\\\mathcal{F})\\\\geq 10}(\\mathrm{omitted})$ where $D = \\\\sup\\_{w, w^\\\\prime \\\\in \\\\mathbb{B}(0, 1)} \\\\sup\\_{x\\\\in\\\\mathcal{X}} |\\\\mathrm{sig}(\\\\langle w, x \\\\rangle) - \\\\mathrm{sig}(\\\\langle w^\\\\prime, x \\\\rangle)|$\", defining \"$\\\\inf\\_{\\\\Delta\\\\in[0, D]}(\\mathrm{omitted})$ where $D = \\\\min\\\\{ \\\\mathrm{arg}\\\\max_{\\\\Delta > 0} N_X(\\\\Delta,\\\\mathcal{F})\\\\geq 10,\\~ \\\\sup\\_{w, w^\\\\prime \\\\in \\\\mathbb{B}(0, 1)} \\\\sup\\_{x\\\\in\\\\mathcal{X}} |\\\\mathrm{sig}(\\\\langle w, x \\\\rangle) - \\\\mathrm{sig}(\\\\langle w^\\\\prime, x \\\\rangle)| \\\\}$\" looks more straightforward.\n- Section 5: It states that \"We use values $\\\\sigma\\\\in\\\\{0.1, 0.2, 0.8\\\\}$ to vary the quality of the density ratio estimates produced by KLIEP\", but what is $\\\\sigma$? It looks that it is not stated anywhere in the paper; please specify the definition.\n- Appendix B, just before expression (54): It states that \"$\\\\beta\\_1/\\\\lambda\\\\geq 1$ without loss of generality\", but it implies that we need to choose $\\\\beta\\_1$ and $\\\\lambda$ satisfying $\\\\beta\\_1/\\\\lambda\\\\geq 1$ *before applying Theorem 1*. A possible workaround is, in Theorem 1, to replace all factors $\\\\frac{\\\\beta\\_1}{\\\\lambda}$ in expression (9) (and also (10)) with $\\\\max\\\\{ \\\\frac{\\\\beta\\_1}{\\\\lambda}, 1 \\\\}$.", "broader_impact_concerns": "The reviewer did not find any specific concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)