From Weighting to Modeling: A Nonparametric Estimator for Off-Policy Evaluation
Authors: Rong J.B. Zhu
Venue: Accepted by TMLR ()

Abstract: We study off-policy evaluation in the setting of contextual bandits, where we aim to evaluate a new policy using historical data that consists of contexts, actions and received rewards. This historical data typically does not faithfully represent action distribution of the new policy accurately.  A common approach, inverse probability weighting (IPW), adjusts for these discrepancies in action distributions.  However, this method often suffers from high variance due to the probability being in the denominator. The doubly robust (DR) estimator reduces variance through modeling reward but does not directly address variance from IPW.  In this work, we address the limitation of IPW by proposing a Nonparametric Weighting (NW) approach that constructs weights using a nonparametric model.  Our NW approach achieves low bias like IPW but typically exhibits significantly lower variance. To further reduce variance, we incorporate reward predictions -- similar to the DR technique -- resulting in the Model-assisted Nonparametric Weighting (MNW) approach. The MNW approach yields accurate value estimates by explicitly modeling and mitigating bias from reward modeling, without aiming to guarantee the standard doubly robust property.  Extensive empirical comparisons show that our approaches consistently outperform existing techniques, achieving lower variance in value estimation while maintaining low bias.

Published in Transactions on Machine Learning Research (3/2026)
From Weighting to Modeling: A Nonparametric Estimator for Off-Policy Evaluation
Rong J.B. Zhu
mmgghuu@fudan.edu.ccm
Institute of Science and Technology for Broin- iniin-isspired Intelligence, Fudon University
Reviewed on OpenReview: https: // openreview. net/ net//orum? id- EW6PYOAU3
Abstract
We study off-policy evaluation the setting of contextual bandits, where we aim to evaluate a new policy using historical data that consists of contexts, actions and received rewards. This historical data typically does not faithfully represent action distribution of the new policy accurately. A common approach, inverse probability weighting (IPW), adjusts for these discrepancies in action distributions. However, this method often suffers from high variance due to the probability being in the demominator. The doubly robust (DR) estimator reduces variance through modeling reward but does not directly address variance from TPW. In this work, we address the limitation of IPW by proposing a Nonparametric Weighting (NW) approach that constructs weights using a nonparametric model. Our NW approach achieves low bias like IPW but typically exhibits significantly lower variance. To further reduce variance, incorporate reward predictions similar to the DR technique resulting in the Mosee-assisted Nonparametric Weighting (MNW) approach. The MNW approach yields accurate value estimates by explicitly modeling and mitigating bias from reward modeling, without aiming to guarantee the standard doubly robust property. Extensive empirical comparisons show that our approaches consistently outperform existing techniques, achieving lower variance in value estimation while maintaining low bias.
1 Introduction
We study the off-policy value evaluation problem for decision making in environments where feedback is observed only for the actions that are taken. In this sctting, the goal is to estimate the value of a target policy using data collected under an actual ootiinngeneraling (behavior) policy (Langford & Zhang 20088] Streh] et al.) 2010). This problem central many real-world applications of reinforcement learning, especially in situations where directly deploying and evaluating new policy is impractical due to high costs, potential risks, ethical and legal concerns (Li et al. 2011h. For example, in health care, data are typically collected under a treatment assignment policy that records outcomes only for the treatments actually administered, with no counterfactual information for treatments not received. The objective is then to evaluate the value of a new treatment policy using such observational data.
The first approach address off-policy learning contextual bandits is inverse probability weighting (IPW) Horvitz & Thompson 1952), uses importance weights to correct for action imbalances in the logged data, but often incurs high variance, especially when the data logging policy has low probability of choosing some actions. The second approach, the Direct Method (DM), which estimates the reward function from data and substitutes it as substitute for the true reward to evaluate policy value across contexts. DM heavily relics on correct model specification and can suffer high bias when the reward model is misspecified an issue common in practice. The third approach, doubly robust (DR) estimator (Cassel et al.) 1976 Robins & Rotuitzky 1995 Lunceford & Davidian 2004 Kang & Schafer 2007), combines DM and IPW and retains unbiasedness if either component correctly specified. Although DR reduces variance through reward modeling, does not directly address the variance introduced by the weighting mechanism itself.

---

Published in Transactions on Machine Learning Research (3/2026)
In this paper, address the high variance of the IPW approach by introducing the nonparametric weighting (NW) method, which uses nonparametric model to link target policy-weighted rewards to behavior policy probabilities, thereby reducing the instability inherent in IPW. By employing P-splines to estimate this ftexible function, the NW approach constructs weights that substantially reduce variance while maintaining low bias, similar to IPW. We establish convergence rates for both the bias and mean squared error of the NW ostimator.
To further reduce variance, extend our framework by incorporating reward predictions, akin to the DR technique, resulting in the Model-assisted Nonparametrie Weighting (MNW) approach. We show that the MNW estimator corrects bias from the reward model and provide convergence rates for its bias and mcan squared error. When applied to policy evaluation, both the NW and MNW estimators consistently outperform existing methods, achieving lower variance without sacrificing bias.
2 Off-policy Evaluation
Let X' be input space, A finite action space with K [Al, and R reward space. Consider policy evaluation problem in contextual bandits that is specified by distribution D over pairs (I,r), where E X is the context and E Rk is vector of rewards for all actions. A dataset of size is generated as follows: the environment draws a sample (xi, r;), and the policy chooses an action a E A from an unknown distribution The reward Tía corresponding to the context-action pair (2,, a) then revealed. We assume
Tia Pia + Cia:
where Mio [sra denotes the expected reward under the context-action pair (I),a), a), and Eia zero-mean noise term with finite variance o
Two tasks are typically considered: policy evaluation and policy optimization. In policy evaluation, our goal is to evaluate the policy т. Given Fi, the policy π chooses an action: Tia = <aax;;), a E A. We aim to estimate the value of a stationary policy n.
v = (a,r)D [ra(w)]
In policy optimization, the aim is to find an optimal policy with maximum value म" arg max, V*, In this paper, focus the problem of policy evaluation. It is expected that more accurate evaluation generally leads to better policy optimization Strehl et al. (2010). Further investigation is needed into variants of this approach.
The key challenge in estimating policy value, given the data as described in the previous section, is the fact that we only have partial information about the reward, hence we cannot directly simulate our proposed policy on the data set S. Given data set {2, a, Fia. P collected as above. Define the probability associated with nn as
in
Pio = P(a|x;).
In practice, this probability needs to be estimated, and we denote its estimator as pra- Generally, there are three main approaches for overcoming this limitation.
DM. The direct method forms estimate Piia (x:) of the expected reward on the context and action a2),a). The policy value is then estimated by
- Edes Tiafria
dm
IPW. The inverse probability weighting (IPW) uses the inverse probability weighting to correct the gap between data-collection policy and the target policy:
n 1pw
=n -1 "n "
1=1 Pie, Tia, Tia.
2

---

Published in Transactions on Machine Learning Research (3/2026)
DR. The doubly robust (DR) approach takes advantage of both the DM and IPW approaches and constructs the following DR estimator:
=n E
Pir i=1 PPass Tia, (ria; [iaa + EnEA Tiaftra
In practice, the estimated Pro is used in place of the true Pia in the IPW and DR approaches. The DM approach requires an accurate reward model of rewards; however, if the model is poorly specified, the DM method can incur high bias. Generally, accurately modeling rewards can be challenging, making this specification potentially restrictive. The IPW approach often suffers from large variance, particularly when the past policy differs substantially from the policy being evaluated. The DR approach improves the inference reliability by integrating the DM and IPW approaches. However, it primarily reduces variance through reward estimation, rather than specifically addressing the variance introduced by the IPW technique itself.
3
Nonparametric Framework of Policy Evaluation
3.1
Nonparametric Model Framework
In this section, we present a nonparametric framework for policy evaluation. The framework is built upon the novel representation results introduced below.
3.1.1
Equivalent representations for off-policy evaluation
We now present representation results for off-policy evaluation. Define
(Praa)) ElMraPialPial-
(1)
Based on this definition, we obtain the following representation results. Proposition 3.1. Under the definition of f (Pta) in Eqn. 11) the value v of the target policy admits the following equivalent represenlations. (a) Design-based represenlation:
= Reg o [Pio.
S (pia:)]
(2)
(b) Model-based representation:
T Biz; Enes (T(aa)
(3)
Proof. Using the unbiasedness of the IPW estimator and the definition of f (pia), we obtain
v En, Eu,rp(alas) [Pra, Tia, rinss = Ex, Barpaaxp() [E [ptn, | Tin ria; [po:]]
E,, En [pm) (Pea.)]
Thus, Eqn.
2 is proved. We also have
4 Ex, EnEA TiaFia = Ez,
Σ aEa (Taaaaaa
Ex;
EneA f (=((pa))
where the last. step is from the definition of f (psa). Thus, Eqn.
3 is proved.
In Proposition 3.1 Eqn. shows that (p((a)) possesses a design-based representation that takes expectation
with respect to the chosen action, analogous to the IPW estimator, and Equ. (3) shows that also admit a model-based representation that take expectation over all actions, analogous to the DM estimator. is worth noting that if we instead define f (pia) Eirlo the resulting quantity aia (Pta) does not inherit these representation properties.
3

---

Published in Transactions on Machine Learning Research (3/2026)
3.1.2 Nonparametric model framework
In the off-policy evaluation problem, the data are assumed to be collected under a mechanism in which the action assignment a, for unit relies only on Piz. That is, conditional on Pla, the action assignment as is independent of Ta- Consequently,
Tia [Paa] =E (Taaii PPaa 0; a].
Therefore, by the definition of (.) in Eqn. 1 we obtain the following model, which links Tiaiia to Pia based the data set {I,, di Tin, T Hi denoted by S":
Tio Pie. f (pia.) + Ca, i= i.
(4)
where euy is an error term with mean zero and variance o² a.ia"" From the proposed modeling framework the function f²(( (-) can be estimated from the data set S The function f*(() captures the systematic part of Tia, that can be explained by Pia,, while the remainder part absorbed by the model error term e,,
In the model framework 4 we allow for flexible structure of (-)-) without imposing a specific functional form, maintaining an agnostic stance toward model specification. By employing the nonparametrie estimation procedure outlined in the next section, facilitate elficient estimation of (*)() within this framework.
3.1.3 Illustration of the two cases
We now connect two weighting methods to two distinct models and demonstrate that the resulting estimators achieve optimal efficiency under their respective models.
Case 1: IPW. The IPW estimator can be viewed as model assisted estimator Little 2004). We construct
the following working linear model to relate ria to the probability Pia- For each (3i, a), model
TsaTia = Pia + PiaSia:
(5)
where {Eia) are assumed be independently distributed with mean O and variance o². We estimate 3 from S*. Given S", the model follows the generalized least squares estimator 3 is given as
=== E
1= Pia, Tia:Pia.-
It follows that the estimator of El is given by Piia Pia.3TT The corresponding estimate of V* is v" n- Ex=1 Eae Pia 3r where the last equality holds due to the fact EaEA Pia 1 for each This demonstrates 77 Ve indicating that the IPW off-policy evaluation can be considered as a prediction under the model 5
Case 2: Simple Weighting. Simple weighting could perform best under an alternative model. Consider the which actions taken uniformly random regardless of the state, i.e., Pia is the same for every (i, a). It then follows that all (pi.) are equal; denoted this common value by This leads the following model
TiaTsa 4t + Cias
where Ei are assumed to be independently and identically distributed with mean zero and finite variance. This model means that there is no relation between ria and Pia- Given the dataset S", the model G leads to the following estimator for μ::
Σ
1=1 Tio, Tia,
which minimizes the variance. The corresponding value estimator is Va = AT. This demonstrates that the simple weighting estimator is the most efficient estimator, achieving the minimum variance.
4

---

Published in Transactions on Machine Learning Research (3/2026)
The results from the two cases above indicate that the efficiency of the IPW SW estimators depends which model is valid. Accordingly, modeling ViaTia through flexible function f (pta) allows the estimator to capture the underlying relationship and preserve efficiency.
We illustrate how the relationship between Tia with the exploration policy Pia can conform to the function form of f(.).
A Toy Example. Suppose ria + dea, where dia (0, o²), and consider a target policy of the form Tio uPia + (1- w)/K with 0 S uuL 1. Then Tia wpPi (1 w)//KK + Tiadia- Hence, it corresponds - / and 3 wy. When
to linear function f(Pta) a + Spic: where
0, Pio has no explanatory power for TiaTia, in which case simple weighting is efficient; when w = 1,the IPW estimator fully captures their relationship. This toy example illustrates that the functional form f() captures the relationship between Taaiaa and Pio and can be estimated using nonparametric methods.
3.2 Nonparametric Estimation
We provide a brief introduction to the P-spline approach (Ellrr & Marx 1996), which we adopt in this paper. It is worth noting that, under the general framework various nonparametric methods can be employed. Here, we use the P-spline as a representative example due to its wide adoption in the nonparametric modeling community (Eilers& & Marx 1996). Other alternatives may also be promising, though their exploration is beyond the scope of this paper.
Consider a univariate regression model
V/ f(x) + So i= 1=,...,," n,
where, conditionally given Tú, & has mean zero and variance o² (a4)). We assume f() E ( bj, where WT[a b) denotes the Sobolev space of order q; that is, f(-) has 1 absolutely continuous derivatives and satisfies (x)) oc. For simplicity, we assume x E (0, 1). We adopt the P-spline approach (Eilers & Marx 1996), which represents the function
f(x) = Σien
J(ni+d BxBx(r),
where (Bx(2) k= = I, (n)) + d) are d-th degree B-spline basis functions with knots no < K1 < KJ(a) The number of knots (n) is chosen such that J(n) = o(n).
The coefficient vector 8 = (B,, -a(m))a) estimated using penalized least-squares approach. Specifi- cally, the estimator 3 is obtained by minimizing the following objective function:
Σn y1 - Siitt Be An Σ Din)t (A
J(n)+d
1
where A the first-oder difference operator, i.e., A8r- BR 3h-1- Solving this optimization problem yields
the estimator 3 = My, where M is the smoothing (hat) matrix resulting from the penalized least-squares procedure and y = (31, Bnn) Denoting B(z)= (z) ... B)())((z)) T, the resulting nonparametric
regression estimator of f(x) is given by
BBx)=B(xTTB =(xx) My.
Under regularity conditions, the penalized spline estimator is consistent; see Eubank 1999 Claeskens et al. (2009), and Xiao (211)). Specifically, for any E (a, b), as long as 2, + oc, the mean squared error satisfics (x) f(=)) = O(A,/n) + ((20/(2-1)) 1/(2))) In particular, when X, O(n O(n112(+2)) we obtain the optimal convergence rate:
E((a(x)) - [(x)) -0(mm220(113)) 21/(1))4)
(8)
Throughout the theoretical analysis in the paper, we adopt this optimal rate.
5

---

Published in Transactions on Machine Learning Research (3/2026)
3.3 Nonparametric Weighting for Policy Evaluation
Under the model [4) we have V E [E Ix (pia) ]. To estimate (pia) from the sample S", we apply the
nonparametric estimation described in Scction 3.2 Let y = (T Plays' Ana, Tna.)". Using the P-spline approach in 7 we obtain the estimator f (pia) = "nw (Pia). which is linear in y. Here, the weight
vector Wow (Pia) = B(psa) M derived from the penalized least-squares estimator presented in 7 We then estimate V by
DTT
Σ. EneA (pia).
This reveals that V.T is weighted average of the elements in {Te, Tio, Pi-1: with the overall weight vector n EaeA Wnw(Pia) We refer to this approach as Nonparametric Weighling (NW).
The procedure summarized in Algorithm below.
Algorithm Policy Evaluation using the Nonparametric Weighting approach
Policy x to be evaluated and the values of Pie, (or their estimates thereof obtained from the collected data).
Input:
Step 1: Apply the P-spline approach in regress (Ta, Pio, V on {Pta, J and obtain the fitted function f*(();
Step 2: Compute the estimate V according (
3.4 Robustness to Behavior Policy Estimation
practice, the true values Pia are typically unknown and must estimated from the data. To address this, the function f(-) is subsequently fitted using the estimates Pia. We now examine the implications of using Pia, accounting for both the estimation error and bias arising from potential misspecification in the estimation of Pia- Denote pha E[pra] and write Pia = Pi, + (1)) The difference p., Pio captures the bias due to potential model misspecification for estimating Pia: while the term (1) represents estimation error.
First, we investigate the impact of estimation error. Consider no-bias case, that is, P.a = Pia- Assuming that f()) is Lipschitz continuous, we have
(fr.) ((pia) + o,))
Hence, evaluating the function at the estimated probabilities is asymptotically equivalent to evaluating it at the true probabilities, up to negligible error term. This indicates that our approach remains valid and is robust to estimation error in the logging policy.
Second, we investigate the impact of bias arising from model specification. Suppose that Pia is a transformation of Pia through a mapping h(.), i..,, Pi, = h(pia). Although h(pia) is a biased proxy for Pias using a flexible regression model such as a nonparametric method allows us to approximate E[ h(ppa)) If h(pia) is sufficient for Pia with respect Tiaia namely, B[TtaTta h((pa)] = E[Tuaia Pia), then modeling based Pia or h(paa) yiclds equivalent results. A simple example of this cquivalence is that h(p) is one-to-one function. More generally, when the biased proxy h(pia) is not exactly sufficient but is highly correlated with Pia, the resulting fit can still provide reasonable approximation to the true conditional mean. In this cases, the flexibility of the regression function mitigates the impact of the bias, rendering its effect on the final estimator limited.
3.5 Error Analysis
We now analyze the error bounds of the NW estimator Vx relative to Va address the question: How well does this estimator perform? Let (7()) f (p) (p), and define v = = EaeA f (Pia) as the

---

Published in Transactions on Machine Learning Research (3/2026)
sample average of f (pia). Then, the error of the NW estimator can be decomposed as
V=
v nw
1 + V
V*
nw
Ap(pia) + n
Euea (pioo) V".
(10)
GEA
- we see from Eqn. 10 that the uncertainty in VT relative to V mainly stems from (piaa) which captures the estimation error in the nouparametric procedure. The following proposition establishes the convergence rates for the bias and mcan squared error of Vin.
Noting (P(A)()))))) O(n ((122)1(+2))) as established in Eqn. and [[((T Vー) = O(n
Proposition 3.2. (a) The bias of the NW eslimator is given by
B(Vaa) V =O(Kn -
(b) The MSE of the NW estimator is given by
BI(V
Proposition 3.2 shows that the convergence rates depend on both K and n, under the condition that Kn (1((+2g)) n - 00. As result, convergence guaranteed even in the presence of large action space, provided that K = (129)())2))
3.6 An Illustrative Example
We present Example 1 to demonstrate how our proposed NW method can improve efficiency. We simulate a K-armed bandit. setting with 300 contexts. For each context and arm k == 1, K, yik is independently generated from (O0, 1), and we consider the squared rewards yog- We construct three distinct reward- gencrating sccnarios:
Sorting {y2, . in increasing order; Sorting {vā,-.. .. vix) } in decreasing order;
Leaving (vā.... .vik unsorted.
For each context i, we draw values from a uniform distribution over (0, 1) and sort them in increasing order. An arm then sampled with probability proportional to these sorted values.
Sorting the rewards in increasing order induces a strong positive correlation between the rewards and the sampling probabilities, while sorting them in decreasing order induces strong negative correlation. In such cases, approximate working model can exploit this explanatory power through nonparametric modeling of the probabilities. In contrast, when the rewards remain unsorted, there is no correlation bet wetween the rewards and the sampling probabilities. In this case, the probabilities offer no additional explanatory power, and the SW estimator yields the most efficient evaluation.
We K and obtain a sample of size n 300. Our goal is to evaluate the uniform target policy Th 1/K, leading to the objective of estimating V (nK)-1> "" > and IPW estimators, well as the proposed NW estimator. With B 2000 iterations, caleulate the bias, standard deviation (s.d.), and root mean square error (RMSE) for each estimator. The results are presented in Table D From the table, the variance of the NW estimator is significantly smaller than that of the IPW estimator across all three cases, resulting in much higher efficiency in terms of RMSE, while the NW estimator exhibits slightly larger absolute bias compared to the IPW estimator. Compared to the SW estimator, which has the lowest variance in the unsorted and docrcasing cases but the largest absolute bias, the NW estimator performs much better in the decreasing and increasing cases while being slightly worse in the unsorted case, expected.
y We estimate V using the SW

---

Published in Transactions on Machine Learning Research (3/2026)
Table Performance of Example
decreasing
increasing
unsorted
method
Bias
8.d.
RMSE
Bias
8.d.
RMSE
Bias
s.d.
RMSE
SW
-0.586
0.040
0.587
0.577
0.090
0.584
0.018
0.081
0.083
IPW
0.017
0.491
0.492
-0.002
0.045
0.045
0.004
0.258
0.258
NW
-0.070
0.148
0.164
0.025
0.031
0.040
0.007
0.110
0.110
4 Model-assisted Nonparametric Weighting
In this section, we extend the NW approach by integrating the DM estimator and propose a model-assisted nonparametric weighting (MNW) estimator. Given an estimate Ωia of the expected reward for the context- action pair (I; a), compute the residual (ra Pia). We model the relationship between Tua (Taa Pio) and the probability Pio using the following nonparametrie model for each (Iz, a):
(ra - (iia) - 9 (Dia) Cia
(11)
where (5% are independently distributed with mean 0 and variance v2
Similar to Section 3.3 we estimate the function g²(()) from the sample S and obtain the nonparametric estimate (.) under the model Given g²(()), (.), we construct the MNW estimator as
>" E dEd (0% + Ttafita).
(12)
Like the DR estimator, the MNW estimator incorporates Aia as baseline when reward model available, reducing variance by nonparametrically modcling the rosiduals Tia -ia- At the same time, this incorporation does not introduce bias. In other words, the MNW estimator remains robust to misspecification of puu: when Pia is biased, V compensates through the nonparametric adjustment. provided by (-), which captures and corrects the residual bias. As result, the MNW estimator effectively adjusts for misspecification in the reward model (a(+), achieving higher efficiency when (a(()) is accurate and maintaining robustness when it is not. The procedure summarized Algorithm below.
Algorithm Policy Evaluation using the Model-assisted Nonparametric Weighting approach
Input: Policy to be evaluated and the values of Pio, (or their estimates obtained from the collected data).
Step 1: Estimate Uio and obtain the fitted values Piai
Step 2: Apply the P-spline approach to regress (Tia, (Pia, - Pi on {pia, 1i yiclding the fitted function (%)();;
Step 3: Compute the estimate v mnw according to 12
4.1 Error Analysis
We now analyze the bias and MSE of the MNW estimator. Denoting pta TE(f))) the bias introduced by the reward model is given by Pia pia. Under the nonparametric model we have
E (riaaiii - Piia) = Tia (Hia -Pia) 9" (pia).
(13)
Define y Van n- EE EnEd Tialfa. Using 13 we obtain
Tat mnw
Σ EOEA
(Teattin g" (pie)).
(14)
8

---

Published in Transactions on Machine Learning Research (3/2026)
Let (7(p) (p) (p). From 14
have
V mnw
V mmw
Ap(ree + (Dio Mto)
This leads to the
following decomposition:
V
mow
V
mow
V
mnw +
V
mnw
V
Σ
Σ aEA
(piia)) + (Dia [eal +
Σ
V
muw (1) V*].
(15)
where VT Vinw (i) = EnEA (Tsalia + + (pia))
Based on this decompasition, we derive the convergence rates for the bias and mean squared error of V as stated in the following proposition.
mnws
Proposition 4.1. Assume Ei - -Maa) 0 Then, we have the following results: (a) The bias of the MNW eslimalor is given by
B(V V O(Kn (((et))())t)),
(b) The MSE of the MNW estimator is given by
E[(V m =
Proposition 4.1 demonstrates that the MNW estimator remains consistent despite discrepancies in reward modeling (i.e., Mia (ia), as these discrepancies are corrected by gd (-).
Comparison with the NW estimator. Eqn. 15 shows that fz mmw muw can be decomposed into two components: Ag (Pta), which reflects the error in estimating g" (-), and Aii Mla the deviation of the estimated reward from its expectation. When the variability of Pio Pia is smaller than that of A (pia); the dominant source of error arises from A7 (Pte). Comparing (Pta) in the MNW estimator with A (Pia) in the NW estimator, it becomes evident that the MNW estimator can achieve higher efficiency when TiaPia effectively captures context-dependent reward information.
4.2 An Illustrative Example
We present Example to illustrate how the MNW approach can improve efficiency over the NW approach. The setting is similar to that in Section 3.6 but the reward-generating process differs. Specifically, generate 2s 2 + v where Jik (O0, and Yak ~ (0,, 1) are independently generated. Here, the term a represents the baseline reward. We adopt the baseline reward model μ(x) = ,z2², with 8 = corresponding correct specification and 3 = 0.5 corresponding to misspecification. As in Example 1, we construct three newarddgenerating scenarios by sorting {= for each according to the values of 20-armed bandit setting with 300 contexts, where rewards are independently generated across contexts and arms. The corresponding results are presented in Table
We simulate
Table 2: Performance of Example
decreasing
increasing s.d.
unsorted
method
Bias
s.d.
RMSE
Bias
RMSE
Bias
s.d.
RMSE
SW
-0.604
0.157
0.624
0.553
0.177
0.581
-0.036
0.174
0.178
IPW
0.005
0.693
0.693
-0.002
0.345
0.345
0.007
0.444
0.444
NW
-0.148
0.259
0.298
-0.003
0.243
0.243
0.000
0.265
0.265
MBW(6 0.5)
-0.118
0.220
0.250
0.003
0.184
0.184
-0.013
0.213
0.213
MNW(6 1)
-0.080
0.221
0.235
0.000
0.157
0.157
-0.025
0.194
0.195
From the table, the MNW estimator exhibits substantially lower variance than the NW estimator across all three cases, resulting in higher efficiency terms of RMSE, while maintaining negligible bias comparable to that of the NW estimator. The MNW estimator works better when I = 0.5 corresponds to misspecified baseline model, in which only part of the term x3 is explained by the model. The results show that the MNW estimator still performs better, expected, since the nonparametric
than when B 0.5. Note that
9

---

Published in Transactions on Machine Learning Research (3/2026)
weighting approach efficiently mitigates bias arising from model misspecification. This example illustrates that the MNW estimator can improve performance when the baseline model captures some of the outcome variation, in a manner analogous to doubly robust estimation.
5 Experiments
This section provides empirical evidence supporting the effectiveness of NW-type estimators over IPW-type estimators. Although more recent methods for off-policy evaluation have been developed (see Section 6 below), we restrict our comparison to the IPW and DR estimators. These estimators are conceptually representative and sufficient to highlight the essential differences between weighting beisht and nonparametric approaches. Including a larger set of methods would not necessarily provide additional insights, our primary aim is to examine the relative performance and robust of the NW-type estimators in comparison with the standard IPW framework.
Following Dudik et al. (0011); ; we consider a multi-class classification problem with bandit feedback, using public benchmark datasets. We use the same datasets as in Dudík et al. (201T). p. See Table G
the appendix for summary of using the datasots usod. All experiments were conducted on a MacBook Air equipped with M3.
In classification task, we assume that the data are drawn independently and identically distributed from fixed distribution: (z ~ P, where x E X' is the feature vector and c E (1..... .K} is the class label. The goal is to find a classifier π : X + {1,-... K} that minimizes the classification error ex = (p(((T()) * c)]- We turn the data point (z, into a cost-sensitive classification example (x. ,1..,.,,.x'' where la denotes the loss for predicting a. We consider a noisy loss where l, is drawn from a Gaussian distribution with mcan I(a & c) and 0.2. Under this formulation, the classifier can be viewed as an action-selection policy, and its classification error exactly the policy's expected loss. The target policy is defined as the deterministic decision of logistic regression classifier learned on the multi-class data.
The logging policy b is generated by sampling probability proportional to values drawn from a uniform distribution; specifically, we draw Pa ~ unif(0, 1) and assign action a according to ha x Pa- To predict the value for action a for unit in the testing dataset, we first fit an lg-regularized least-squares estimates using the training data, and then generate predictions for cach data point in the test set.
We randomly split data. into training and test sets of (roughly) the same size. On the training set, we train multinomial classifier to define the policy म, and compute the classification error the test data, treating it as the ground truth for comparing various estimales. We perform Monte Carlo simulation with 500 ilerations to generate a partially labeled set from the test data using the logging policy b. We compute policy evaluation estimates using various mothods and calculate the resulting bias and root mcan squared error (RMSE). We repeat this entire process 20 limes, and report the average bias and RMSE values across these 20 runs in Table The data sizes used for policy evaluation vary across datasets (see Table X in the appendix for a summary). We further examine the performance under varying sample sizes, and report results for a representative dataset, Page, in Figure I in the appendix. The results demonstrate that our approaches are robust to the sample size used for policy evaluation.
From the table, the RMSE of the NW estimator is consistently lower than that of the IPW estimator across all datasets, while its bias is remains negligible and comparable to that of the IPW estimator. The DM estimator performs the worst, reflecting the poor fit of the linear model used to predict rewards. In comparison, the MNW estimator achioves substantially lower RMSE than the DR estimator, while maintaining a similar level of bias.
How much does estimating the logging policy affect the results? To assess the impact of estimating the logging policy b, we conduct another experiment using perturbed logging policy defined as b x Pa 6, where 8 ~ (1, 0.09). Here, 8 introduces Gaussian noise into the probability estimates, simulating errors in estimating the logging policy that generated the data. We calculate the bias and RMSE values of each estimator using i and report the results in Table 4 As shown in the table, the IPW and DR estimators exhibit significantly larger RMSE, highlighting their sensitivity to errors in probability estimation. Notably, the IPW estimator shows marked increase in bias, suggesting that it may become biased under noisy probability
10

---

Published in Transactions on Machine Learning Research (3/2026)
Table 3: Bias and RMSE of various estimators for classification error with the true logging policy,
Bias
RMSE
Data
DM
IPW
DR
NW
MNW
DM
IPW
DR
NW
MNW
letter
0.507
0.001
0.009
0.000
0.009
0.507
0.070
0.075
0.036
0.045
glass
0.218
-0.003
0.034
0.000
0.036
0.218
0.233
0.238
0.238
0.193
ecoli
0.215
0.002
0.037
0.000
0.034
0.215
0.270
0.229
0.243
0.208
opt
0.292
0.001
-0.001
0.000
-0.001
0.292
0.047
0.060
0.027
0.037
page 0.086
0.000
0.012
-0.001
0.012
0.086
0.021
0.028
0.014
0.020
pen 0.400
0.000
0.004
0.000
0.004
0.400
0.033
0.051
0.022
0.032
sal 0.192
0.000
0.020
0.000
0.021
0.192
0.041
0.043
0.024
0.031
vehicle 0.239
0.001
0.017
0.000
0.016
0.239
0.079
0.078
0.058
0.057
yeast
0.215
0.000
0.070
-0.002
0.070
0.215
0.139
0.140
0.098
0.106
Table 4: Bias and RMSE of various estimators for classification error under perturbed logging policy.
Bias
RMSE
Data
DM
IPW
DR
NW
MNW
DM
IPW
DR
NW
MNW
letter
0.505
0.066
-0.085
0.000
0.009
0.505
0.319
0.337
0.034
0.041
glass
0.215
0.044
0.015
-0.002
0.039
0.215
0.302
0.282
0.226
0.188
ecoli
0.206
0.060
0.012
0.002
0.039
0.206
0.334
0.272
0.221
0.190
opt
0.291
0.006
-0.052
0.000
-0.001
0.291
0.057
0.115
0.025
0.035
page 0.088
0.004
-0.001
0.000
0.012
0.088
0.043
0.048
0.013
0.019
pen 0.398
0.020
-0.095
0.000
0.004
0.398
0.081
0.216
0.020
0.030
sat 0.192
0.028
-0.007
-0.002
0.021
0.192
0.074
0.085
0.023
0.030
vehicle
0.236
0.021
-0.027
-0.001
0.015
0.236
0.126
0.164
0.054
0.053
yeast
0.218
0.044
-0.043
-0.003
0.069
0.218
0.230
0.470
0.089
0.101
estimates. contrast, the NW and MNW estimators yield RMSE values similar to those obtained when the logging policy is known, demonstrating that our proposed methods are robust to inaccuracies in the estimated probabilities. We further conduct an experiment in which the logging policy is estimated; the corresponding performance results reported in Table 5 of the appendíx, The results show that our approaches perform consistently across settings.
6 Related Work
Off-policy evaluation for bandits has been extensively studied. Below, we provide concise review of the most relevant work to the best of our knowledge.
To address the high variance of Iaaeb methods, several simple techniques have been proposed. One common approach is weight clipping, which truncates large inverse probability weights (Ionides) 2008 Swaminathan & Joachims 2015a. [Su et al., 2019). Another normalization, which rescales the weights stabilize estimates (Swaminathan & Joachims) 2015b). In contrast, our paper takes a fundamentally diffcrent. approach by modeling the probabilities directly, thereby avoiding the issue of large weights in a data-driven way.
Doubly robust estimation Robiis et al. 1994 Lunceford & Davidian, 2004 Kang & Schafer 2007 is widely used for parameter estimation in statistical inference. Dudík et. al. (2011 first applied this framework to policy evaluation and optimization, with later extensions to the reinforcement learning setting by Jiiag & Li (2016) and Thomas Brunskill (2016). More recent developments include the work of Wang al. (017)) Farajtabar et al. (2018), and (Su et al. (2020). While the goal of our MNW approach is guarantee the standard doubly robust property, it explicitly models and mitigates the bias introduced by reward modeling.
11

---

Published in Transactions on Machine Learning Research (3/2026)
Doubly robust estimation has also been explored in observational settings for estimating average treatment effects using asymptotically optimal estimators (Hirano et. al. 2003 Tmbens et al. 2007
Several data-driven approaches have been proposed for off-policy evaluation. Saito et al. 2021 selected one of multiple logging policies as a pseudo-target policy, directly estimated its value from the dataset, and used it to identify the off-policy estimator with the best performance. [Cdagawa et al. (2023) introduced two surrogate policies constructed from the logged data. Cief et al. (2024) proposed cross-validated off-policy evaluation framework. In contrast to these methods, our approach leverages flexible modeling, offering both ease of implementation and reliable performance.
Finally, it is worth noting that nonparametric modeling of sampling probabilities is not new. In survey sampling, such techniques have been employed to correct for sample selection bias and enable finite population inference; see the review article (Little 2004). Motivated by this line of work, we propose a nonparametric wighting approach for policy evaluation, and further enhance it by incorporating the regression-basee adjustment to improve accuracy.
7 Conclusion
In this paper, we have proposed nonparametric modeling for off off-policy evaluation, leading to nonparametric weighting estimators for off-policy evaluation. The weights are constructed using nonparametric framework rather than relying explicit selection bias adjustment. Instead, the approach is grounded in the general model specification 4 and leverages data-driven relationships. The method exhibits strong robustness duc to its flexible modeling structure. When the rewards are only weakly correlated with the selection rrobabilities, there is little relationship for the nonparametric model to capture; in such cases, the NW estimator essentially reduces to simple averaging. As the relationship becomes stronger, the nouparametric model able to capture more of the underlying structure, thereby improving its explanatory and predictive power. Furthermore, we incorporate the DM estimator to further reduce the variance in the rewards and develop model-assisted nonparametric weighting estimator. Extensive experiments on multi-class classification with bandit feedback, using public benchmark dalasets, demonstrate the elficacy of these estimators and emphasize the role of nonparametric weighting in achieving superior performance. As a result, we anticipate that the NW approach will become standard alternatives to the IPW approach.
This study has some limitations. First, we adopt the P-spline approach to const the NW-lype estimators. Although the P-spline method demonstrates robust performance in our experiments, implementing the framework with alternative nonparametric methods, particularly neural networks modcls, could further enhance its flexibility and predictive accuracy. This direction warrants further investigation. Second, in many reinforcement learning problems, the rewards are specified as binary variables. However, the general model framework
( does not explicitly account for the discrete nature of such reward distributions. Incorporating the discrete characteristics of the outcome may further improve the performance and applicability of the proposed approach.
Several promising directions for further research arise from this work. Given the widespread use of IPW in policy evaluation across various domains, we anticipate that our NW approach will have competitive applications in these areas. For instance, extending it to combinatorial contextual bandits (Swaminathan ct al.) 2017) could offer valuable insights, particularly in decision-making scenarios with complex action spaces. Furthermore, integrating this methodology within the broader reinforcement. learning framework presents a promising opportunity to improve policy evaluation and learning in sequential decision decision-making problems. Finally, our paper primarily focuses on settings with small action spaces. Extending our approach to large action spaces is a promising direction, as importance weighting can break down in such cases due extreme variance from large importance weights (Saito & Joachims 2022
12

---

Published in Transactions on Machine Learning Research (3/2026)
Acknowledgements
sincerely thank the three anoпуmoоs reviewers for their insightful and constructive review comments and discussions, which substantially improved the manuscript. I also thank the Assigned Action Editor, Dr. Inigo Urteaga, and for the valuable discussions and guidance on the manuscript.
References
C. M. Cassel, C. E. Särndal, and J. Wretman. Some results on gencralized difference estimation and generalized regression estimation for finite populations. Biomelrika, 63:615 620, 1976.
M. Cief, B. Kveton, and M. Kompan. Cross-validated off-policy evaluation. In AAAI, 2024.
G. Claeskens, T. Krivobokova, and J. D. Opsomer. Asymptotic properties of penalized spline estimators. Biometrika, 96(3):529-544 2009.
M. Dudik, Langford, and L. Li. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on Machine Learning, 2011.
P.H.C. Eilers and B.D. Marx. Flexible smoothing with b-splines and penalties (with discussion). Statistical Science, 11:89-12 121, 1996.
R.L. Eubank. Nonparametric Regression and Spline Smoothing. New York: Marcel Dekker., 1999.
M. Farajtabar, Y. Chow, and M. Ghavamzadeh. More robust doubly robust off- evaluation. Proceedings of the 95th International Conference on Machine Learning, 2018.
K. Hirano, G. W Imbens, and G. Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71:1161 1189, 2003.
D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from finite universe. Journal of the American Statistical Association, 47:663-688 1952.
G. Imbens, W. Newey, and G. Ridder. Mean-squared-error calculations for average treatment effects. Available SSRN: http://dx.doi.orgg 11..019//ssrn..... 9547488 2007.
E. L lonides. Truncated importance sampling. Journal of Computational and Graphical Statistics, 17(2): 295-311, 2008.
N. Jiang and Li. Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of the 36th International Conference on Muchine Learning, 2016.
J. D. Y. Kang and J. L. Schafer. Demystifying double robustness: comparison of alternative strategies for estimating population mean from incomplete data. Statistical Science, 22:523-533 2007.
J. Langford and T Zhang. The opoch-greedy algorithm for multi-armed bandits with side information. In Advances in Neural Information Processing Systems, 2008.
L. Li, W. Chu, J. Langford, and Wang. Unbiased offline evaluation contextuul-banddi-beeell news article recommendation algorithms. In International Conference on Web Search and Data Mining, 2011.
Roderick J Little. To model or not to model? competing modes of inference for finite population sampling. Journal of the American Statistical Association,
2004.
J. K. Lunceford and M. Davidian. Stratification and weighting via the propensity score estimation of causal treatment effects: A comparative study. Statistics in Medicine, 23:2937-2900 -960, 2004.
J. Robins and A Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data.
Journal of the American Statislical Associalion, 00:122-1200 1995.
13

---

Published in Transactions on Machine Learning Research (3/2026)
J. M. Robins, A. Rotnitzky, and L. P. Zhao. Estimation of regression coefficients when regressors are not always observed. Journal of the American Stalistical Associalion, 89:846- 19:846-866, 1994.
Y. Saito and T. Joachims. Off- Off-policy evaluation for large action spaces via embeddings. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Y. Sailo, T. Udagawa, H. Kiyohara, K. Mogi, Y. Narita, and K. Tateno. Evaluating the robust.ness of off-policy evaluation. In Proceedings of the 15th ACM Conference on Recommender Systems, 2021.
A. Strehl, J. Langford, L. Li, and S. M Kakade. Learning from logged implicit exploration data. In Advances in Neural Information Processing Systems, 2010.
Y. Su, L. Wang, M. Santacatterina, and T. Joachims. CAB: Continuous adaptive blending estimator for policy evaluation and learning. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Y. Su, M. Dimakopoulou, A. Krishnamurthy, and M. Dudík. Doubly robust. off-policy evaluation with shrinkage. In Proceedings of the 37th International Conference on Machine Learning, 2020.
A. Swaminathan and Joachims. The self-normalized estimator for counterfactual learning. In Advances in Newral Information Processing Systems, 2015a.
A. Swaminathan and Joachims. The self-normalized estimator for counterfactual learning. In Advances in Neural Information Processing Systems, 2015b.
A. Swaminathan, A. Krishnamurthy, A. Agarwal, M. Dudik, Langford, D. Jose, and I. Zitouni. Off-policy evaluation for slate recommendation. In Advances in Neural Information Processing Systems, 2017.
P. Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. Proceedings of the 36th International Conference on Machine Learning, 2016.
Udagawa, Kiyohara, Y. Narita, Y. Saito, and K. Tateno. Policy-adaptive estimator selection for off-policy evaluation. In Proceedings of the AAAT Conference on Artificial Intelligence, 2023.
Y.X. Wang, A. Agarwal, and M. Dudik. Optimal and adaptive off-policy evaluation in contextual bandits. Proceedings of the 94th International Conference on Machine Learning, 2017.
L. Xiao. Asymptotic theory of penalized splines. Electvonic Journal of Statistics, 13:747 794, 2019.
Appendix
The Proof of Proposition 3.2
Equ. 10
directly follows
B (pia)) Tiattia =R(A7(pia)) O(n -/(11+2)))
Thus, we have the bias
n nw
T =n
Σ a
E [AA = O(Kn -9((+4)))
Noting
E((V nw
(V.
V*] = from
10 we have the mean squared error of ya mw:
nw
E (VV A
v*
= [V nw
V.H + v
(16)
14

---

Published in Transactions on Machine Learning Research (3/2026)
For the first term of the righ-hand side of 16
we have
[[(V nw
V
191
IE'
E aen Eoee
n
I E aEd
E [[]
(PA)())))))
2== i=1
=kK2 20/(1+20])
(17)
For the second term of the righ-hand side of 16p. V is sample simple average of set of sample with expectation V*, following we have
B[(V nw [=)3] -0(n
(18)
Inserting
and
18
16 the bound on the MSE in Proposition 3.2 is proved.
The Proof of Proposition
Similar the proof of Proposition 3.2 we have
B mnw
V O(Kn (OKKnn/(ett)) 0/2211)) O(Kn
For the MSE term, similar the proof of Proposition
3.2 we have
( maw V= =R(( mnw
7 ( + E((V V=
[[(V mnw
VT -0
Now we investigale the term Bi(V nnw
V kunw) )2], and have
Bi(
VT (anw)
)
=E
E aEA
(p)()))) + Tealfia - mmta)
In |A4) L i=1 Σ aEA
E
(A4 + TialPr - Mia)
22n |A Σ i=1 En aEA
(R[A) + E [Tii (Dia - 1(te))
=0 n -4/(2+24) + O(K²n-1))
Thus, the MSF expression in the theorem follows.
More results on the experiments
Performance under the estimated logging policy. We conduct an experiment where the logging policy is estimated. Unlike the previous experiment, the logging policy is estimated via logistic regression using four-fifths of the test data (Table 6). and policy value estimates are computed using various methods. We calculate the resulting bias and RMSF. We repeat this entire process 20 times, and report the average bias and RMSE values across these 20 runs in Table b
Performance across evaluation sample sizes. We examine performance across a range of evaluation sample sizes under the true logging policy. Figure Sogureereppnr reports results for the Page data as a representative example. We report the RMSE along with its standard error, computed over 100 repetitions and displayed error bars, well the bias and its standard error. The results show that approach consistently outperforms competing methods across sample sizes.
15

---

Published in Transactions on Machine Learning Research (3/2026)
Table 5: Bias and RMSE of various estimators for classification error under estimated logging policy.
Bias
RMSE
Data
DM
IPW
DR
NW
MNW
DM
IPW
DR
NW
MNW
letter
0.503
0.006
0.000
0.001
0.009
0.503
0.030
0.035
0.029
0.034
glass
0.221
0.290
-0.317
-0.038
0.022
0.221
0.566
0.638
0.268
0.292
ecoli
0.197
0.115
-0.208
-0.009
0.003
0.197
0.305
0.417
0.112
0.128
opt
0.290
0.009
-0.042
-0.001
-0.004
0.290
0.036
0.062
0.023
0.030
page 0.090
0.009
0.001
0.000
0.010
0.090
0.020
0.033
0.011
0.017
pen 0.396
0.002
-0.004
-0.001
0.003
0.396
0.018
0.025
0.017
0.023
0.193
0.005
0.019
-0.001
0.022
0.193
0.021
0.027
0.019
0.029
vehicle
0,239
0.009
-0.002
-0.005
0.012
0.239
0.054
0.051
0.045
0.044
yeast
0.215
0.015
0.065
0.000
0.069
0.215
0.084
0.097
0.074
0.095
0.15
0.10
mmse
bias
0.00
-0.05
171
342
684
1368
2737
684
1368
2737
size
size
Figure 1: Performance across various sample size for the Page dataset. Left: RMSF as function of sample size; Right: Bias function of sample size.
Table Summary of the datascts, where n denotes the total sample size, is the dimension of covariates, n. is the sample size of the evaluation, and np is the sample size used to estimate the training behavior policy.
Data
d
fly
np
letter
20000
16
10000
7500
glass
214
10
107
80
ccoli
336
8
168
126
opt.
3823
64
1912
1434
page
5473
10
2737
2052
pen
7494
16
3747
2810
sat
6435
36
3218
2413
vehicle
846
18
423
317
yeast
1484
9
742
556
16


Reviews:
[{"review_id": "RatNx2kS8X", "reviewer": "Reviewer_dMcV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses off-policy evaluation in contextual bandits by proposing two novel estimation approaches that aim to reduce the high variance associated with inverse probability weighting (IPW) method.\n\nThe paper proposes two novel estimation approaches for off-policy evaluation in contextual bandits. The proposed approaches aim to reduce the high variance associated with the existing methods like inverse probability weighting (IPW). The authors propose nonparametric weighting (NW), where they model the relationship between policy-weighted rewards and behavior policy probabilities using a flexible nonparametric function (P-splines). In addition, building on nonparametric weighting, they propose model-assisted nonparametric weighting (MNW) method that incorporates reward predictions to further reduce variance, similar to the existing method known as doubly robust (DR) estimation.\n\nThe authors provide theoretical analysis of the NW and MNW estimators, establishing convergence rates for bias and mean squared error relative to the true policy value. The authors empirically evaluate NW and MNW on 9 multi-class classification datasets with bandit feedback and show robust results under two different conditions - with true logging policy (Table 2) and when it is estimated (Table 3). They compare the proposed methods against existing methods like direct method (DM), IPW and DR.  Overall, NW and MNW achieve lower variance in value estimation and maintain low bias while demonstrating robustness to probability estimation errors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In the abstract, the authors state - \"To further reduce variance, we incorporate reward predictions—similar to the DR technique— resulting in the Model-assisted Nonparametric Weighting (MNW) approach. We show that MNW yields accurate value estimates when either the reward model or the behavior policy model is well specified.\" and in the related work section, the authors state \"While the goal of our MNW approach is not to guarantee the standard doubly robust property, it explicitly models and mitigates the bias introduced by reward modeling\". I find the claim made in the abstract to be a bit ambiguous, and I would like to request the authors to clarify the ambiguity.\n\nUpdate - The authors have clarified the ambiguity and updated the paper.", "audience_yn": "Yes", "audience_explanation": "Yes, the findings of the paper would be of interest to researchers working on contextual bandits and off-policy evaluation. While the current work focuses on the bandit setting, the core methodology may have implications for the broader reinforcement learning community.", "requested_changes": "1. Typo in page 5, 'demonstrates' should be 'demonstrate'.\n2. Possible typo in page 7, the equation for the residual in the first paragraph might be wrong.\n3. In paragraph 2 of introduction, 'DM and IPS' instead of 'DM and IPW'\n4. In paragraph 1 of introduction, 'collect a set of data according a policy' -> 'according to a policy'\n5. In paragraph 1 of introduction, 'were not receive' -> 'did not receive'\n6. In section 3.1, 'there is no any relation' -> 'there is no relation'\n7. While the authors discuss relevant related work in Section 6, the empirical comparisons are limited. I would like to see comparisons with more recent methods mentioned in the related work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "93nQ8o8CYC", "reviewer": "Reviewer_mo5Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of offline policy evaluation for contextual bandit. A method is proposed that models the expected reward of a hypothetical policy pi using a univariate non-parametric function with input being the probability of action of a policy used to generate the data. Extension to doubly robust regimes is also considered", "claims_and_evidence_yn": "Yes", "claims_explanation": "Mostly the math is solid. I only have one question regarding the motivation of the model, which I will detail later in my comment.", "audience_yn": "Yes", "audience_explanation": "Offline policy evaluation is very important in sequential decision making problems such as RL which is at the heart of modern AI research. While this paper only studies contextual bandit, it is reasonable to believe that people who work on RL or related fields will find the method relevant.", "requested_changes": "I only have one major question that I'm requesting the authors to elaborate on.\n\nIn Eq. (5), the expected reward of a policy pi on a context t, is modeled as a nonparametric function with input being the probability of seeing an action in the data generating process. I struggle to understand the motivation of this:\n\n1. Why would pi_{ia} be on the left-hand side of this model? We know pi_{ia} (since this is the policy we want to evaluate). Can we simply fit a model of r_{ia} using p_{ia}, and then recover r_{ia}pi_{ia} by simply multiplying the estimated reward with pi? This way, we don't have to fit a different model for different policies pi.\n\n2. More importantly, why would r_{ia} (or pi_{ia}r_{ia}) have anything to do with p_{ia}? p_{ia} is the probability of taking action a on context x_i in the policy that generates the data set. I don't see why this has anything to do with the actual reward of this particular action. For example, if I use a randomized policy to generate the data, then r_{ia} has abosolutely nothing to do with p_{ia}.\n\nThis shows that certain assumptions, either implicit or explicit, have to be placed to motivate this model. I would suggest making such assumptions explicit. This means that Eq. (5) needs to be discussed in much more details, preferably standalone sections, to convince readers that this is a suitable model.\n\n2.1. Perhaps, there is some requirement that the exploration policy is good. In this way, if an action is taken with higher probability, then the expected reward of that action is higher, justfying some connection between r_{ia} and p_{ia}. It would be greatly appreciated if this can be made rigorous. Perhaps with a toy example showing / deriving such connections explicitly.\n\n2.2. Perhaps, the alignment between the exploration policy and the evaluate policy is good, so that p_{ia} closely aligns with pi_{ia} and therefore pi_{ia}r_{ia} is closely related to p_{ia}. Again, formalizing this and some toy examples will be helpful.\n\nOnce the key model in Eq.(5) is well motivated, I think the idea of using univariate non-parametric estimates is reasonable. This is actually quite intereting that I believe will inspire ideas in other forms of offline evaluation problems.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m2ZevRDgH8", "reviewer": "Reviewer_m39c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new approach for off-policy value estimation, called non-parametric weighting, using a P-spline approach. The paper compares this to two common weighting methods (inverse probability weighting and simple weighting) and shows the connection between them. The paper establishes some theoretical properties of NW, performing an error analysis to determine the rates of convergence for various quantities under usual assumptions. The paper also produces a model-assisted variant of NW and compares it to the DR estimator. Through experiments, the paper establishes the improved performance of the NW(/MNW) estimator compared to the other approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The introduction does not over-claim about the scope of the paper.\n- The theoretical claims seem to be correct and well-supported.\n    - I did not find the section \"Robustness to the estimation of $p_{ia}$.\" to be very convincing.\n- The empirical set-up seems reasonable, though has some weaknesses in my opinion:\n    - The results presented are mostly snapshots with the dataset size chosen by the authors. It would be valuable to see comparisons across dataset sizes.\n    - The experiment to determine \"How much does estimating the logging policy affect the results?\" uses a proxy estimate for $b$ rather than actually estimating $b$. It would be interesting to know what happens when $b$ is actually estimated in practice, because this may affect the relative performance of the approaches. Or, perhaps seeing how the metrics change when the noise scale $\\mathbb{E}\\left\\[\\delta\\right\\]$ changes.\n    - \"Including a larger set of methods would not necessarily provide additional insights, as our primary aim is to examine the relative performance and robustness of the NW-type estimators in comparison with the standard IPW framework.\" This is ok, but I still think it would be nice to make some comparisons to other existing methods, as this would help to determine whether NW estimators are actually worth using in practice (compared to existing techniques).\n    - Some details are missing from experiments to be able to replicate them based on the paper.", "audience_yn": "Yes", "audience_explanation": "The problem of off-policy evaluation is a key challenge in the study of bandits and reinforcement learning. Improved off-policy evaluation for contextual bandits, which is directly addressed in this paper, could lead to improvements in various contextual bandit applications. Though not covered directly in this work, future works may build upon this work to apply the approach in reinforcement learning, when off-policy value estimates are required.", "requested_changes": "**The following changes would aid in securing my recommendation, but are not necessarily critical.**\n\n- I did not follow the argument in \"Robustness to the estimation of $p_{ia}$.\" Perhaps it can be clarified/strengthened.\n- The empirical results presented are mostly static snapshots computed with a fixed $n$. I would find the results more helpful if the work provided plots comparing the approaches at different values of $n$ (on the $x$-axis) for a few different values of $K$ (on different plots). Most of these plots could go into an appendix.\n- It's not clear to me how many knots were used, or what degree spline was used in the empirical results.\n- The results for \"How much does estimating the logging policy affect the results?\" should either be based on using an actually-estimated logging policy, or provide some more elaboration for why the Gaussian noise approach used is appropriate.\n\n\n\n**The below changes are not critical for securing my recommendation, though I believe they would strengthen the work:**\n\n- Most of the text is well-written and free of errors, but the first paragraph of the introduction requires some editing.\n- There is a typo in the second line of pg 7\n- Sec 3.5 should be titled \"An illustrative example'\n- I found it difficult to follow the procedure in Sec 3.5. I'm still not 100% sure I could replicate the procedure, though I think I understand the idea. In particular:\n    - does $y^2_{iK} = y^2_{jK}$? I think the answer is no, and that you sample $\\vec{y}_i$ from an $K$-dimensional normal distribution $\\mathcal{N}(\\vec{0}, I)$ for each $i$\n    - What is meant by \"We perform sampling without replacement\"? I thought that we would sample one $y_{ia}$ from $\\vec{y}_i$ at each $i$ so sampling with/without replacement would be irrelevant\n- I would find it useful to have an illustrative example of MNW outperforming NW.\n- A table comparing convergence rates of various quantities between the different methods would be useful.", "broader_impact_concerns": "n/a", "additional_comments": "In general, I am uncertain about the positioning of this work within the broader context of related work, and will defer to other reviewers to make judgements on this aspect (including whether the experiments are missing important baselines).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)