Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies
Authors: Koichi Tanaka
Kazuki Kawamura
Takanori Muroi
Yusuke Narita
Yuki Sasamoto
Kei Tateno
Takuma Udagawa
Wei-Wei Du
Yuta Saito
Venue: ICLR 2026 Poster (2026)
Keywords: off-policy evaluation; ranking; common support; deterministic logging
Primary area: reinforcement learning
TL;DR: We propose novel OPE estimators that work even when your logs come from a fully deterministic ranker by using user click randomness instead of policy randomness.

Abstract: Off-Policy Evaluation (OPE) is an important practical problem in algorithmic ranking systems, where the goal is to estimate the expected performance of a new ranking policy using only offline logged data collected under a different, logging policy. Existing estimators, such as the ranking-wise and position-wise inverse propensity score (IPS) estimators, require the data collection policy to be sufficiently stochastic and suffer from severe bias when the logging policy is deterministic. In this paper, we propose novel estimators, Click-based Inverse Propensity Score (CIPS) and Click-based Doubly Robust (CDR), which exploit the intrinsic stochasticity of user click behavior to address this challenge. Unlike existing methods that rely on the stochasticity of the logging policy, our approach uses click probability as a new form of importance weighting, enabling low-bias OPE even under deterministic logging policies where existing methods incur substantial bias. We provide theoretical analyses of the bias and variance properties of the proposed estimators and show, through synthetic and real-world experiments, that our estimators achieve significantly lower bias compared to strong baselines, particularly in settings with completely deterministic logging policies.

Published as a conference paper at ICLR 2026
OFF-POLICY EVALUATION FOR RANKING POLICIES UNDER DETERMINISTIC LOGGING POLICIES
Koichi Tanaka
Kazuki Kawamura
Keio University
Sony Group Corporation
kouichi 1207@keio. Jp
Kazuki Kawamura@sony com
Takanori Muroi
Yusuke Narita
Sony Group Corporation
Yale University
Takanori. Muroi (esiiisony... com
yusuke www.taeaale.eceecmmmoom edu
Yuki Sasamoto
Kei Tateno
Sony Group Corporation
Sony Group Corporation Kei. Tateno@sony com
Yuki. Booki.Sssmmttosoonyyco com
Takuma Udagawa
Wei-Wei Du
Sony Group Corporation Takuma Udagawalsony, com
Sony Group Corporation
weiwei du@sony, com
Yuta Saito
Hanuku- kaso, Co., Ltd. sa vaiton kwskkaaak..oooo..oo.kkaakokkkkoa com
ABSTRACT
Off-Policy Evaluation (OPE) is an important practical problem in algorithmic ranking systems, where the goal is to estimate the expected performance of a new ranking policy using only offline logged data collected under a different, logging policy. Existing estimators, such as the ranking-wise and position wise inverse propensity score (IPS) estimators, require the data collection policy be sufficiently stochastic and suffer from severe bias when the logging policy is fully deterministic. In this paper, propose novel estimators, Click-based Inverse Propensity Score (CIPS), exploiting the intrinsic stochasticity of user click behavior address this challenge. Unlike existing methods that rely on the stochasticity of the logging policy, our approach uses click probability as a new form of importance weighting, enabling low-bias OPE even under deterministic logging policies where existing methods incur substantial bias. We provide theo- retical analyses of the bias and variance properties of the proposed estimators and show, through synthetic and real-world experiments, that our estimators achieve significantly lower bias compared strong baselines, for a range of experimental settings with completely deterministic logging policies.
INTRODUCTION
Many intelligent systems, such as recommender systems, search engines, and e-commerce plat- forms, present items in the form of rankings. These systems interact with users through the con- textual bandit process, where ranking policy repeatedly observes context, produces a ranking, and observes rewards (Kiyohara et al. 2022). As ranking policies interact with the environment, they collect logged data that valuable for Off-Policy Evaluation (OPE). OPE allows researchers to estimate the performance of new policies using only logged data, reducing potential risks and ethical concerns associated with repeated A/B testing (Saito & Joachims 2021 Dudik et al. 2014 Su et al. 2020 Shimizu al. 2024).

---

Published as a conference paper at ICLR 2026
Table 1: Examples of clicks, potential rewards, and observable rewards.
position
action
observable reward CR
click C
potential reward R
k
d
3000
k 2
42
1000 0
1000
k = 3
43
2000
k= = 4
44
0
0
500
A typical OPE study often considers the Direct Method (DM), Inverse Propensity Score (IPS), and Doubly Robust (DR) the baseline estimators (Saito & Joachims 2021p. DM predicts rewards using a learned model, but it can suffer from substantial bias when the model is misspecified. IPS reweights observed rewards by the ratio of the probabilities of taking the observed action, and DR combines both to reduce variance while being unbiased (Dudik et al.) 2014). However, these typical estimators become impractical in ranking problems with extremely large action spaces, because the probability of observing the exact same ranking is often negligible (Mclnerney et al. 20200)
Although many estimators for the ranking setup already exist to address the challenges of large rank- ing action spaces (Li et al.) 2018 Kiyohara et al., 2022, McInerney et al.) 2020), they require the logging policy to be sufficiently stochastic to produce low-bias estimates, For example, the ranking- wise IPS estimator, which reweights observed rewards by the ratio of ranking- probabilities, requires the logging policy to explore all the rankings that a new policy may select. The indepen- dent IPS estimator, which uses the ratio of position-wisd -wise probabilities, requires the logging policy to select all actions at each position (Li et al. 2018). Because of this strong dependence on the stochas- ticity of the logging policy, these existing estimators can only evaluate the value of actions that the logging policy explored sufficiently (Sachdeva et al. (2220). However, large-scale industrial systems often deploy deterministic policies, as the large action spaces make stochastic policies inefficient and risky to use. In such cases, existing methods suffer from severe bias, making them impractical.
To address the substantial bias caused by deterministic logging policies, we propose novel class of estimators, namely Click-based Inverse Propensity Score (CIPS), which newly exploits the intrinsic stochasticity of user click behavior. the ranking setting, policy recommends a ranked list of multiple actions (e.g., videos, products, or job listings) to an incoming user. Therefore, even if the logging policy is fully deterministic, the user may still view and click each action in the presented ranking with some non-zero probability. Building on this property of ranking settings and user behavior, our approach uses marginalized click probabilities under the new and logging policies to define a new form of importance weights, unlike existing methods that depend solely on the stochasticity of the logging policy. This design enables unbiased or low-bias OPE even in situations where deterministic logging policies cause existing methods to produce substantial bias.
In our thcoretical analysis, we show that CIPS achieves unbiasedness under the stochasticity of user click behavior and a relatively mild independence condition the reward structure. This is, the best of our knowledge, the first method to provide low-bias value estimation of new ranking policies under completely deterministic logging. Extensive experiments show that our estimator achieves lower mean squared error than existing methods in fully deterministic logging scenarios.
2 OFF-POLICY EVALUATION FOR RANKING
This section formulates OPE for ranking policies in the contextual bandit process following existing relevant studies (Li et al. 2018 Kiyohara al.) 2022 2023 Mclnerney al., 2020). Let x e X' C RR be context vector, such as user demographics, sampled from an underlyin ring distribution p(x). We use A to denote a discrete set of unique actions, with a E A corresponding to a specific action such as movie, news article, vector, where K denotes the ranking length. We use A(k) to denote the action at the k-th position. The function π X A[TIx (A)) is called ranking policy, where II (A) represents the set of K K-permutations of A, i.e., the ranking space, and A(-) is the probability simplex.
product. Let A = (a1,ag, ... aa-- ax) be ranking
To closely reflect realistic ranking scenarios, introduce two types of rewards: clicks and poten- tial rewards. The latter refers to downstream outcomes (c.g., product purchase, movic play) that may occur after click, thus modeling the two-stage decision process of users. Let first C denote
2

---

Published as a conference paper at ICLR 2026
Table 2: A toy example of importance weight under completely deterministic logging policy.
(a) Logging and new policy
(b) Importance weights of IPS
A1
Aa
Ag
A4
Ag
Ag
A1
A2
As
A4
As
As
01
01
02
02
03
03
w((1, A)
0.1
NA
NA
NA
NA
NA
k 2
a3
02
03
01
02
01
(c) Importance weights of UIP
(AAar)
0.1
0.3
0.3
0.1
0.0
0.2
w(x1, A(k))
a1
a2
03
1.0
0.0
0.0
0.0
0.0
0.0
k 1
0.4
NA
NA
k
NA
0.3
NA
k 3
NA
NA
0.4
click vector sampled from an unknown distribution conditional on the presented ranking, (CC2x,A) where C(k) is a binary signal indicating whether a click occurs at position k. Let R then denote potential reward vector drawn from p 22 A). We refer to R "potential" because it is only observable through CR,, meaning that we observe R((k) only if C(k) = 1, i.e. after click. This structure based on real-world ranking systems, such product searches, video recommendations, and many other large-scale e-commerce or content platforms, where user interactions naturally fol- low or more sequential phases, i.e., exposure and click, followed by downstream engagement (e.g., purchase, watch time, or dwell time). Table [ provides an example of clicks and potential rewards. For instance, we observe R(2) because action d2 clicked at the 2nd position, while we cannot observe R(1), R(3), or R(4) because 01, a3 and are not clicked. Thus, the potential reward vector never fully observable on its own.
As logging policy To interacts with the environment, we observe C and CR as the rewards corre- sponding to the selected rankings A. Therefore, we obtain the logged dataset
D = [(x,, Ai, C₁ C.Ri)) ~ P(:()))0
P(p()) 2 (C) [., Ad),
==1
which contains independent observations drawn under To.
The ultimate goal of OPE is to accurately estimate the effectiveness of a new ranking policy using only the logged data collected under a logging policy that differs from the new policy. We typically
define the effectiveness of ranking policy π as (Melnerney et al. 2020 Kiyohara al. 2022)
K Σ
K
V(=) = (Fp(t))((Al))eCC.R.....
C(K)R())
= (p(e))(AAa))
Σ
yk (x, A)
(I)
where qo (x, A) = ((C(K)R(K) x. is the position-wise expected reward function regarding position k. This quantity (IT), referred the policy value, represents the expected sum of rewards (e.g., sales or play duration) obtained under the deployment of policy ग, and serves as a natural measure of policy performance.
In OPE, aim construct estimator V (T:; D) that accurately estimate the policy value using only the logged data D. The accuracy of an estimator is typically measured by the mean squared error (MSE) (Uchara al. 2022p:
MSE(() == Ey(D) (VV(:: ((m))) BBas [ D)] + Var [((:
where Ep(D) denotes the expectation over the distribution of the log D and
Bias [(T:D)] Ep(D) [((:;) ((T),, Var [T:: = En(D) (V Ep(D) ((::))
As thoroughly discussed in Appendix A most existing studies in ranking OPE focus mitigating high variance. In contrast, under deterministic logging policies, the bias term in the MSE becomes the dominant issue, making bias reduction the central focus of this study.
3

---

Published as a conference paper at ICLR 2026
Table 3: A toy example click importance weight under completely deterministic logging policy. The new and logging policy are the same the previous example given in Table 2 (a) click probability
(b) importance weights of CIPS
Pe(211
A)
A1
A2
A3
A4
A5
A6
a1
d2
(1
k
0.8
0.5
0.7
0.2
0.4
0.4
Po a,
0.55
0.39
0.48
k
2
0.5
0.6
0.6
0.5
0.3
0.4
Pe(21, a,
0.8
0.5
0.2
k
0.2
0.1
0.5
0.4
0.2
0.1
W( a, a, mo)
0.6875
0.78
2.4
Existing Estimators. Here, we review notable existing estimators in the literature of ranking OPE and discuss their limitations under deterministic logging. A more comprehensive overview of related work is available in Appendix A
First, the ranking-wist wise IPS estimator is considered the most naive baseline (Kiyohara et al. 2023 IPS reweights observed rewards by the ratio of ranking-wise probabilities under two policies as
x(A TO Ag (T.)
(T: D) Σ
Σ C.())R(K)
where w(xx A) = T(Al))/mo(A is called the ranking-wise importance weight. This estimator is unbiased under the canking-wise common support condition in the following.
Condition 2.1 (Ranking-wise Common Support). The logging policy Tu has common support if (AAx)) > (A()))) > 0 for all A E A[T (A)) and x X.
Although ranking-wise IPS only requires Condition 2.1 for for unbiased evaluation, this condition is never satisfied under a deterministic logging policy. To provide intuition for the common support violation, Table presentss a toy example with X {=1}- A {a1, 22 0z), and K 3. Table 2a shows the ranking probabilities of the new and logging policies. The logging policy selects ranking A with probability I and never selects any other ranking, meaning
completely deterministic. Table 2b shows the corresponding ranking- wise importance weights. For example, w((21, A1) (A1 21 no (A1 = 0.1/1.0 0.1, whereas for the other rankings, the weight undeñned due to condition violations. This example illustrates that the common support condition is heavily violated under deterministic logging, leading IPS to incur substantial bias (Sachdeva et al. 2020
Theorem 2.1 (Bias of IPS). IPS produces the following bias under violated Condition 2.
Bias (PP)((TD)) - -Ep(w)
Σ
*(Ax²)
Σ 2N (x, A) k=1
AELoo(a,mo)
where Uo(I, mp) fA A(Tx TT(A2z) = 0} the set of unsupported rankings for x.
Theorem 2.1 shows that IPS suffers from downward bias, or underestimation, proportional to number of unsupported rankings, which becomes large under deterministic logging. This occurs be- cause IPS assigns zero reward estimates to unsupported actions. In Table 2b the set of unsupported rankings is Ud [Ag, Ag, A4, As, As), mcaning almost all available rankings unsupported. As a result, the bias becomes substantial, posing a critical challenge for accurate OPE.
In the typical literature ranking OPE with sufficiently stochastic logging policies, the most promi- nent challenge is that sanking-wise IPS suffers from severe variance due to large action spaces. To mitigate this, other estimators incorporate user behavior conditions (Li al. [018 Kiyohara al. 2022 McInerney et al., 2020). For example, under the independence condition, which posits that interact with items without being influenced by other items in ranking, the independent IPS (IIPS) estimator can be defined as follows (Li et al. 2018).
K
क (A,(k) T (A(K) (A((((k)
(TT D)
(3)

---

Published as a conference paper at ICLR 2026
(AA(4) is the position-wise importance weight, and T (A((k) (x) = Ex ((A()) A(k)] To ensure unbiased evaluation, IIPS requires the logging policy to satisfy the following exploration condition.
where ((())(())
Condition 2.2 (Position-Wins Common Support). The logging policy To has position-wiss wise common <z) 0 for all E A. and x X.
support if π (A(())
Although Condition 2.2 imposes weaker stochasticity requirements than Condition 2.1 (needed for unbiasedness of ran Bunking-woll wise IPS), still severely violated under a fully deterministic logging policy. Tables 2b and 2c provide intuition for the distinction between the mon support. In this example, the ranking-win common support fails in of 6 cases, whereas the position-wee wwis counterpart fails in 6 of indicating that IIPS satisfies its support condition slightly more often than IPS. Yet under deterministic logging, the position-wise common support condition never satisfied either, which leads IIPS incur substantial bias.
versions of com-
Other than ranking-wiss IPS and IPS,, McInerney et al. (2020) propose RIPS under the cascade condition, which posits that users examine items sequentially from the top position downward.
(A, k) x4) (.()))(()),
Viurs (T; D)
To (A,(11 B) (x))
where TAA(1 I x)= T(A' <'(( k) A(1 k)). RIPS requires the logging policy to be stochastic, like IPS and IIPS, and the user behavior to follow the cascade condition for unbiasedness. However, shown in Appendix B these requirements are also violated under deter- ministic logging. To enable effective OPE even in the practical scenario of deterministic logging, we introduce novel idea designed circumvent the severe bias caused by the lack of exploration.
THE PROPOSED ESTIMATOR
This section proposes estimator that mitigates the excessive bias caused by fully deterministic logging policies. The key idea to exploit the intrinsic stochasticity of user click behavior rather than relying on the stochasticity of the logging policy.
To introduce our method, we first focus on user click behavior for each action and reformulate the policy value from Eq. as
((T) = (lppe))((AA)))(C........
Σ C(a)R(a)
(5)
LaE.1
where C(a) and R(a) denote the click indicator and the potential reward for action respectively. This reformulation replaces the summation over positions in Eq. with a summation over actions and serves merely as a different representation of the same quantity to aid our analysis. It is worth noting that this notation does not impose any independence assumption, i.c., can generally depend on the actions presented at different positions. For example, the independence assumption used in the IIPS estimator can be written as E(C(a))(a)))) = (E(((a) R(a) which shows that our notation itself does not imply independence.
We then propose our new estimator, Click-based Inverse Propensity Score (CIPS), which lever- ages the ratio of click probabilities as a new form of importance weighting.
Pe(a,, a, Pe a, T))
VCIPS( (T; D) =
(a(a)
(6)
YL
i=1
aEA
where pe(x, a, (x,
(z, a, mo) is the click importance weight, and
(a a) ( (z, A)] = (BC(())
is the marginalized probability that a user with context x clicks action a under a ranking policy n.
The key idea behind method is that, even if the logging policy is fully deterministic, the user may still view each action with a positive probability. We leverage this property by weighting with click-based probabilities.
5

---

Published as a conference paper at ICLR 2026
Theoretical Analysis. We first analyze the bias of CIPS under click-based common support, which is much less restrictive than the previous support conditions required by existing estimators. Condition 3.1 (Click-wise Common Support). The logging policy To has click-wise common sup- port if Pe(a, a, x) > 0 = Pe(z, a, To) > 0 for all E and x E X.
This condition is more likely to hold because click probabilities are inherently stochastic. Table illustrates the advantage of using click importance weights under deterministic logging policy. As discussed earlier, existing estimators fail to meet their respective support conditions in this setting by definition. In contrast, Condition 3.1 is satisfied even when the logging policy is deterministic, intuitively showing that CIPS can mitigate bias by leveraging the stochasticity of click behavior.
Additionally, we formally introduce the independence of potential rewards condition, which is needed for formally showing when CIPS becomes unbiased.
Condition 3.2 (Independence of Potential Rewards). The expected potential rewards satisfy inde- pendence if E 2r, A] = ([((a) x] = q, (r, a) for all E and x E X.
Condition 3.2 states that the expected potential reward for an action depends only on that action a, on other actions in the ranking. This is generally more realistic than the independence condition required by UPS (Li et al. 2018). For example, in e-commerce it means that, once a user clicks on product in search, their decision purchase after a click depends solely on that product's 's attributes.
Under Conditions 3.1 and 3.2 CIPS is unbiased.
Theorem 3.1. Under Conditions See Appendix for the proof.
3.1 and 3.2 CIPS is unbiased, i.e., Ep(D) (eurs) D)] V(T)..
Theorem 3.1 shows that CIPS stays unbiased even with deterministic logging, provided these two conditions hold. This is unlike IPS, IIPS, and RIPS, as their stricter support conditions are never satisfied with deterministic logging, leading to their unavoidable and often substantial bias.
It is worth noting that the above bias analysis assumes access the true click probabilities, which we rarely know in reality. In practice. these probabilities must be estimated from logged data. We can still characterize the bias of CIPS in this case.
Theorem 3.2. Under Conditions 3.7 and 3.2
CIPS has the following bias when using estimated
click probabilities pe( 0, T):
Bias (VeIPs) = (Ep(()
Σ polz,, a, mo)
(a (,,
[ola, a)
To) pa(2,, a, m0)) gr (
(7)
LaEd
where Pelx,, a) = (E pela,, a, A)), wox, a, a, = pela, a, (p/.// a, mo). and (x) E[R(a) x].
Theorem 3.2 shows that the bias depends on the discrepancy between the ratios of estimated and true click probabilities. Thus, even if the click probabilities themselves are perfectly accurate, the bias remains small as long as these ratios are estimated well. Empirically, we will show that CIPS performs much better than existing methods by substantially reducing the bias even when the click probabilities are estimated based only on observable data.
Next, we analyze the variance of CIPS.
Theorem 3.3 (Variance of CIPS). Under Conditions 3.1 and 3.2
CIPS has the following variance:
nVD
Vairs]
mo) Σ (m(())o(AAl)) [u² no))o,,,To00" (2,,
(8)
aeAA
+ Ex) (x, a, n, mo) (r, a) 2V (a(A))) (a, A)]] + Vp(e) a,
where 0² -
A]
Theorem 3.3 shows that the variance of CIPS depends on the magnitude of the click importance weights, which remain stable in deterministic logging. For further variance reduction, it is straight- forward to extend CIPS by including weight clipping, noffaloomalizaiinn, or other recent variance reduction techniques. In the following section, seeking better bias varsinainnce trade-off, we develop the Click-based Doubly Robust (CDR) estimator, an extension of CIPS, that lowers variance with- out introducing additional bias.

---

Published as a conference paper at ICLR 2026
4 EXTENSION TO CDR
We now extend CIPS to a more sophisticated estimator, Click-based Doubly Robust (CDR), by incorporating regression model for the expected potential reward.
VEDR (T; D)
a, Pe(2,, a.)
(C)(a)R,(a) Pe(xs; a, A); Ba(Alo:) (pcl,,, a, A)gr( a)l
n 1= REA
where (e(x, a, A) E[C(a) x; A].
We first analyze the bias of CDR under Conditions 3.1 and 3.2 both with and without access to the true click probability.
Theorem 4.1. Under Condition See Appendix for the proof.
3.1 and 3.2
CDR is unbiased, i.e. EpCD) D)] = ((T)..
Theorem 4.2. Under Condition probability.
3.1 and 3.2
CDR has the following bias for a given estimated click
Bias (VEDR) = Bias (VCIPs)
= Ep(x)
Σ pol mo)
(2, To)
Pola, (. a,
0,(x,,,7) (a(z, a, To)
(z, a))
Theorems 4.1 and 4.2 show that CDR produces exactly the same bias CIPS, regardless of the accuracy of the regression model.
Next, we analyze the variance of CDR and show that it is often smaller than that of CIPS.
Theorem 4.3 (Variance of CDR). Under Conditions 3.7 and 3.2 CDR has the following variance.
nVD
[CDR]
Σ
Bp(3)to(A1=>)
(x To o (x,
aEA
+ Ep(x))
w (I, a, a,) (z (Al A)] + Vo(n) pola, a, a,)
where A, (z, (x, a) (r,
The key difference between the variance of CIPS (Theorem 3.3 and CDR lies in the term (x, a), which captures the error of the potential reward model. When gr (x, a) is more accurate than simple zero-filling, CDR is expected to reduce variance relative to CIPS.
EMPIRICAL EVALUATION
this section, we first evaluate CIPS under deterministic logging policies using synthetic data, aiming evaluate and compare the proposed methods against the baselines under fully controlled environments. We then assess the real-world applicability of the proposed method on a public rec- ommendation dataset. The implementation code is shared supplementary material for repro- ducibility, and detailed experimental setups can be found in Appendix D
5.1 SYNTHETIC DATA
To generate synthetic data for our experiment, we first randomly sample 10-dimensional context vectors (z) from the standard normal distribution. We then synthesize the expected click probability and the expected potential reward functions follows.
(x, A(k)) sigmoid( A(k)) +
Wc(A()),AH))
(10)
k
(x, = A(k)) +
W, (A())))
(1I)
k
1zk
7

---

Published as a conference paper at ICLR 2026
IIPS
RIPS
CIPS
CIPS (true CTR)
Normalized MSE
Squared Blas
Varfance
0.5
0.125
0.000
0.200
0.325
300
3890
2000
4000
500
2006
2003
4030
540
2000
2005
Iooqed
Fi Figure 1: MSE, Squared Bias, and Variance with varying logged data sizes (7)).
Nonralized MSE
Squared Bias
Variance
0.75
0.00
11:11
0.
targt oarkking
kengh rranking
of nanking
Figure 2: MSE, Squared Bias, and Variance with varying ranking lengths (K).
Squared
Veriance
0.
8.0
0:25
violatinn indupendinne
Vilution indppendnnce
atlation
Figure 3: MSE, Squared Bias, and Variance with varying violations of independence (A).
No Normalized MSE
Squared Bles
Variance
0.3
03
02
decemb wotti user thveshold
ristic user threshold
determinitti user threshold
Figure 4: MSE, Squared Bias, and Variance with varying deterministic user thresholds (a:).
((A(), A(k)) denotes the interaction term that measures the effect of action A(I) (presented at po- sition 1) on the reward of action A(k) (presented at position k). We can see that the click probability of A(k) in Eq. 10 depends on both its position k and the entire ranking A. We can control the extent of violation of the independence condition through the experimental parameter À in Eq. II where the independence condition holds when À = 0.0. Based on the above reward functions, we sample the click signal C(k) from binomial distribution with mean (z, A(k)) and the potential reward R(kk) from a normal distribution with mean (I, A(k)) and standard deviation a 1.0.
We then synthetically define logging policy mo following the Plackett-Luce model (Plackett 1975. > a.
K A=
if (1)
De'edrana exp(Jo(zça")
To (o(A
(12)
K
[(k)) = argmax fo(x,a") if (1)) <a d'EAQaL.
where fo(ar,a= 0, + aạa + Oz.a- We sample 0z Oa. and Oz.a from the standard uniform distri- bution. Here, (1) denotes the first dimension of the context x. The experiment parameter a thus controls the stochasticity of the logging policy; when (1(1) S a, the logging policy for a user with context becomes completely deterministic. We set = co as the default to evaluate our proposed estimators under a fully deterministic logging policy. In one of the experiments, however, we vary this experimental parameter to compare different methods across a range of determinism levels in the logging policy. We finally define the new policy π as
K (1 . 1 a(k)
(A (A|2))
II
argmax M(( + A
(13)
d'EA
8

---

Published as a conference paper at ICLR 2026
where [0, 1] controls the quality and stochasticity of œ. When 0.0, the new policy becomes deterministic and always selects the action that maximizes fi (z, a) at each position. When = 1.0, the policy becomes completely uniform. We use = 0.3 as the default setting.
Compared methods. We compare CIPS with IPS, IPPS, and RIPS, whose definitions appear in Section 2 We estimate click probabilities to implement CIPS using 3-layer neural network and the logged data D. Regarding CIPS, we also report the results of CIPS (true CTR), which uses the true click probability from Eq. just just as a reference (CTR refers to Click Through Rate).
5.1.1 RESULTS
Figures present the MSE, squared bias, and variance, each normalized by the true policy value V(x). We compute 100 simulations with different random seeds to generate synthetic data and report the averaged results. The shaded regions in the plots represent the 95% confidence intervals estimated with bootstrap. Unless otherwise specified, the logged data size is set to 1000, the number of unique actions is 6, the ranking length is 6, and parameter À in Eq. 11 is 0.5. Note that, beyond the MSE comparison between CIPS and the baselines, we empirically show that CIPS improves policy selection and that applying CDR further reduces estimation variance in Appendix D.2.
How does CIPS perform when varying the logged data size? Figure D varies the logged data size (n) from 500 4000. We can see that CIPS consistently outperforms the baselines across all logged data sizes by substantially reducing bias. While the baselines suffer from severe bias due to violations of their support conditions, CIPS achieves low-bias estimates by leveraging the still remaining stochasticity in the click data through its click importance weights. As expected, the variance the baselines remains small because variance not a major concern under completely deterministic logging policy. We also observe that CIPS achieves performance close to that of CIPS (true CTR), although a small additional bias arises from estimating click probabilities Pe(x, a, IT, showing its real-world applicability even without access to the true CTR.
How does CIPS perform when varying the ranking length? Figure reports the performance of the estimators as vary the ranking length K. The results show that CIPS provides robust estimations regardless of this experiment parameter. Specifically, CIPS maintains low bias across different lengths, while the bias of the baselines increases with ranking length. Although CIPS exhibits a moderate increase in variance, this effect does not dominate the MSE. Furthermore, its performance remains close to CIPS (true CTR), consistent with the trend we saw in Figure
How does CIPS perform when violating the independence of potential rewards condition? In
Figure 3 we examine the effect the independence of potential rewards condition (Condition) 3.2
on CIPS, which is one of the necessary conditions for its unbiasedness, by varying À in Eq. | Larger values of À correspond to stronger violations of this condition. The results show that the MSE of CIPS increases only marginally À grows. CIPS continues to provide accurate estimates even when the independence condition is not strictly satisfied and remains the best-performing approach among the compared methods, thanks its strong bias reduction effect. From these results, can empirically confirm that the relative advantage of CIPS over existing baselines remains unchanged in more practical scenario where the independence of potential rewards condition is violated.
How does CIPS perform with different levels of logging policy stochasticity? Figure reports the performance of the estimators for varying a in Eq. 12 Larger a values correspond to lower stochasticity in the logging policy. Since the context is sampled from the standard normal distribu- tion, varying a over 1-5,, policies over [0.07, ... 0.84, 0.93). We find that CIPS maintains low bias even at large a, in con- trast to the baselines, which exhibit substantial bias under near deterministic logging. For smaller a values, where the logging policy is more stochastic and the key bias issues are absent, IIPS performs on par with CIPS. This outcome is expected, as our primary focus is on substantially improving OPE under deterministic environments rather than the typical stochastic setup.
1.0, 1.5] changes the proportion of users with deterministic logging

---

Published as a conference paper at ICLR 2026
IPS
IIPS
RIPS
---
CIPS
-
CIPS (true CTR)
MSE
Squarad
0.000
1.4
1.5
1.
01810
0.3005
10C5
4090
0.5
2000
2000
500
4000
ligged
lepped dana
loggud size
Figure 5: MSE, Squared Bias, and Variance with varying logged data sizes (71).
Normalized MSE
Squared Bles
Variance
2.5
2.0
2.0
13
03
00
3.5
Jatamm threshold
detarmmati thrsshald
datermnaticc user theeshold
Figure 6: MSE, Squared Bias, and Variance with varying deterministic user thresholds (a:).
5.2
DATA
This section conducts ranking OPE experiments real-world dataset, KuaiRec (Gao al. 2022 which contains recommendation logs from the video-sharing application Kuaishou. KuaiRec pro- vides fully observed user-item interactions with nearly 100% density for subset of users and items. By leveraging this property, we construct OPE experiments with minimal reliance on synthetic com- ponents. Specifically, we use the user-item interaction matrix recorded in the original data as the potential reward function gr (x, A(k)) and define the expected click probability as
(c(=,(()) = A(k)) if gr (1, A(k)) > 2.0
(14)
A2,A(R)) otherwise
where 17 noise parameter sampled independently from the uniform distribution over [0, 0.5).
We define the logging and new policies following Eq. 12 Eq. 13 from the synthetic experiments.
We conduct real-world experiments with 0.0 to simulate the most challenging problem with both deterministic new and logging policies. Note that the number of unique actions is set to 10, the ranking length is 6, and = co.
Results. Figure 5 compares the estimators under deterministic new policy while varying the logged data size (n). We observe that CIPS consistently achieves lower MSE than the haselines, de- spite exhibiting some non-negligible bias. This bias stems from violations of the click-wise common support, which can occur when both the new and logging policies are deterministic, an extremely challenging setting for any estimator. Nevertheless, even under partial violations of its common support, CIPS remains more accurate than the baselines, which completely fail under deterministic logging due to full violations of their support conditions.
the estimators as the deterministic user threshold (a) varies. As the logging policy becomes more deterministic, the relative advantage of CIPS over the baselines increases. Although all estimators experience bias growth with increasing a, CIPS limits this growth most effectively.
CONCLUSION
We studied Off-Policy Evaluation (OPE) for ranking under deterministic logging, where existing methods relying on logging-policy stochasticity incur severe bias. To address this previously un- tackled problem, we proposed the Click-based Inverse Propensity Score estimator, which exploits the intrinsic stochasticity of user click behavior via click importance weights, substantially relax- ing the support condition and enabling unbiased or low-bias estimation under deterministic logging. Overall, this work challenges the long-held belief that accurate OPE is infeasible under deterministic logging. Our method makes reliable evaluation possible in this previously intractable setting via the new weighting design, paving the way for broader applicability of OPE in real-world systems.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485-511, 2014.
Nicolo Felicioni, Maurizio Ferrari Dacrema, Marcello Restelli, and Paolo Cremonesi. Off-policy evaluation with deficient support using side information. Advances in Neural Information Pro- cessing Systems, 35. 2022.
Nicolò Felicioni, Michael Benigni, and Maurizio Ferrari Dacrema. Automated off-policy estimator selection via supervised learning, arXiv preprint arXiv:2406. /8022, 2024.
Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang. Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recom- mender systems. In Proceedings of the s1 ACM International Conference on Information & Knowledge Management, pp. 140-550, 2022.
Daniel Suuzman-Olivares, Philipp Schmidt, Jacek Golcbiowski, and Artur Bekasov. Clustering con- text in off-policy evaluation. arXiv preprint arXiv:2502.21304, 2025.
Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. Doubly robust off-policy evaluation for ranking policies under the cascade behavior model. In Proceedings of the 15th International Conference on Web Search and Data Mining, 2022.
Haruka Kiyohara, Masatoshi Uehara, Yusuke Narita, Nobuyuki Shimizu, Yasuo Yamamoto, and Yuta Saito. Off-policy evaluation of ranking policies under diverse user behavior. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1154 1163, 2023.
Haruka Kiyohara, Masahiro Nomura, and Yuta Saito. Off-policy evaluation of slate bandit policies via optimizing abstraction. Proceedings of the ACM on Web Conference 2024, pp. 3150-3161, 2024.
Shuai Li, Yasin Abbasi Yadkor,, Branislav Kveton, S Muthukrishnan, Vishwa Vinay, and Zheng
Wen. Offline evaluation of ranking policies with click models. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 1685-1694, 2018.
James McInerney, Brian Brost, Praveen Chandar, Rishabh Mchrotra, and Benjamin Carterette. Counterfactual evaluation of slate recommendations with sequential reward interactions. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1779-1788 2020.
Yuta Natsubori, Masataka Ushiku, and Yuta Saito. Cross-domain off-policy evaluation and learning for contextual bandits. In The Thirteenth International Conference on Learning Representations, 2025.
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg. Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duch- esnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12: 2825 2835, 2011.
Robin L Plackett.
The analysis permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193-25 1975.
Noveen Sachdeva. Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 965-975, 2020.
Yuta Saito and Thorsten Joachims. Counterfactual learning and evaluation for recommender sys- tems: Foundations, implementations, and recent advances. In Proceedings of the 15th ACM Conference Recommender Systems, pp. 828-88888 228-830,20 2021.

---

Published as a conference paper at ICLR 2026
Yuta Saito and Thorsten Joachims. Off- Off-policy evaluation for large action spaces via embeddings. In International Conference on Machine Learning, pp. 10089-99999991 19122. PMLR, 2022.
Yuta Saito, Qingyang Ren, and Thorsten Joachims. Off-policy evaluation for large action spaces via conjunct effect modeling. arXiv preprint (xXx:2335.00066 2023.
Tatsuhiro Shimizu, Koichi Tanaka, Ren Kishimoto, Haruka Kiyohara, Masahiro Nomura, and Yuta Saito. Effective off-policy evaluation and learning in contextual combinatorial bandits. In Pro- ceedings of the 18th ACM Conference on Recommender Systems, pp. 733-741, 2024.
Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudik. Doubly robust off-policy evaluation with shrinkage. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 9167-9176. PMLR, 2020.
Muhammad Faaiz Taufig, Arnadd Doucet, Rob Cornish, and Jean-Francois Ton. Marginal density ratio for off-policy evaluation in contextual bandits. Advances in Neural Information Processing Systems, 36:52648-5269) 2023.
Takuma Udagawa, Haruka Kiyohara, Yusuke Narita, Yuta Saito, and Kei Tateno. Policy-adaptive estimator selection for off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2023.
Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off off-policy evaluation in rein- forcement learning. arXiv preprint (rXiv:2212.063355 2022.
12

---

Published as a conference paper at ICLR 2026
A RELATED WORK
In this section, we distinguish our work from related studies and clarify the contributions of our methods and analysis.
First, we review OPE for ranking policies Li ct al. (2018); Kiyohara et al. (2022) ; McInerney et al. 2020):Kiyohanl et al. (2023) under the standard setting with sufficiently stochastic logging policies. Prior work mainly addresses the severe variance caused by large ranking spaces. To reduce the high variance of the "naïve" canking-wise IPS estimator, recent methods introduce user behavior assumptions such as independence Li et al.] (2018) or cascade models McInerney et al.] (2020); Kiyohara al. 22022) to improve the bias-variance trade-off. In contrast, we target the severe bias introduced by deterministic logging policies rather than the variance problem. In this setting, all existing estimators fail drastically because they rely on the logging policy's stochasticity and suffer from substantial bias due to the lack of exploration in the logged data.
Second, we compare our contributions with studies addressing the deficient support problem in the typical (non-ranking) OPE formulation Sachdeva SSaculationnaccdeva et al. (2020): Felicioni et al. (2022. Deficient sup- port refers to a milder situation where the logging policy assigns zero probability to some actions that the new policy may select. This situation is problematic for OPE because importance weighting cannot evaluate the reward of actions that have no probability of being selected under the logging policy. Compared to the deficient support problem, our work addresses an even more challenging setting, a fully deterministic logging policy. Deterministic logging is a strict special case of defi- cient support, climinating all stochasticity, which makes existing methods severely ineffective and motivates our development of a new approach tailored to this scenario.
Third, distinguish our contributions from studies OPE for large action spaces Saito et al. (2023) ; Saito & Joachims (2022); Taufiq et al. (2023); Bluzman-Olivaree et al.] (2025) Kiyohara al. (2024). These studies aim deal with the severe variance problem caused by large action spaces, similar to the typical OPE studies for ranking Li et al. (2018): ; McInerney et al. (2020);
Kiyohara al. (2022 2023). Their key technique is marginalize importance weights leveraging observed action embeddings to prevent the weights from taking excessively large values. Our pro- posed estimators also marginalize importance weights, but we do so using click probabilities rather than embeddings. While this shares the general idea, our goal is to address the lack of stochasticity in deterministic logging policies rather than variance in large action spaces. Furthermore, meth- ods for large action spaces require additional information, such as pre-observed action embeddings, which our method does not require.
Finally, we mention a recent work addressing OPE in multiple domains Natsubori et al. (0255) That study uses logged data collected from multiple domains (c.g., multiple countries or hospitals) to compensate for the lack of stochasticity in the logging policy of the target domain. Although it tackles a similar problem to ours, it depends on multi-domain logged data, which is rarely avail- able. Our method addresses deterministic logging without requiring data from multiple domains by exploiting the intrinsic stochasticity of user click behavior.
B EXAMPLES OF IMPORTANCE WEIGHTS UNDER COMPLETELY DETERMINISTIC LOGGING POLICY
Here, we provide useful intuition for the common support violation. Table presents a toy example with X' = {x1} A = {a1,ay a2 (z). and K = 3. This example is the same as Table 2 in the main text. In Table we also consider importance weights of RIPS as well as those of ranking-wing -wie IPS and position-wist IPS. As we discussed in the main text, the ranking-wise common support fails in of cases, whereas the position-wise counterpart fails in 6 of 9, indicating that IIPS satisfies its support condition slightly more often than IPS. Table 4d shows importance weights of RIPS. We can see that the common support of RIPS fails of 15 cases, suggesting that RIPS has severe bias under deterministic logging policies. This also indicates that RIPS satisfies its support condition slightly more often than IPS, but less often than IIPS. We can consider RIPS as an estimator that lies between IPS and IIPS. Thus, all baseline estimators in ranking OPE suffer from violation of support conditions, resulting in introducing substantial bias. However, CIPS achieves low-bias OPE under deterministic logging policies by leveraging click importance weights.
13

---

Published as a conference paper at ICLR 2026
Table 4: A toy example of importance weight under completely deterministic logging policy.
(a) Logging and new policy
(b) Importance weights of IPS
A1
Ag
Ag
A4
Ag
A6
A1
A2
A3
A4
As
As
01
01
02
02
03
03
w(x1 A)
0.1
NA
NA
NA
NA
NA
2
43
02
03
01
02
01
(c) Importance weights of LIPS
77 (Alar))
0.1
0.3
0.3
0.1
0.0
0.2
w(x1, A(k))
a1
a2
03
To (A)
1.0
0.0
0.0
0.0
0.0
0.0
k
0.4
NA
NA
k
NA
0.3
NA
k 3
NA
NA
0.4
Importance weights of RIPS
A(1 : 1) Q1 02
w A(1 1)) 0.4
NA
NA
A(1: :2)
(a1.0) 02)
(a1. 03)
(a2, a1)
(a2, .g)
(ag, a1)
(a3, a2)
w1 A(: 2))
0.1
NA
NA
NA
NA
NA
A(1: :3)
(01, a2 oz)
(01, 03, a2)
(a2, 01 0z)
(ag, a3, 01)
(az, 01, a2)
(a3, 02, a1)
w(r1,A(1 A(1: 3))
0.1
NA
NA
NA
NA
NA
C OMITTED PROOFS
Here, we provide the derivations and proofs that are omitted in the main text.
C.1 PROOF OF THEOREM
We show that CIPS is unbiased under Conditions 3.1
and 3.2
De Po ai
E Bede))a(Ala)p(C.RRe..
Pel ai mo) C.(a)
1= aEA
(A
Σ
(e a, T
E (((a))((a)zz
(e a, T0
LAEA
Pe(x, a,
Pe(x, a, To)
LEE
Pe(x; a, T.
Epp(a)
I. [R(aa) x
To A]
L (2r, a, To)
Po(#, U, Pa( a, To)
EE((w)
IF (2
L
Ep(x))
Σ
Pelu, a T E [
LaEd
Ep(aa)
Σ Σ
LaEA
14

---

Published as a conference paper at ICLR 2026
Σ ((())((a)
LaEA
(TT
C.2
PROOF OF THEOREM 3.2
Bias (VCIPs)
n
Pe as T a.
(((a)ta(a))) V(x)
= [ede))o
Deloi, ai, T()
i=1 GEA
Pclas a. a) ((a)R(a)
Σ
c(o)R(a)
= (Aleema(Ala)ppCCCnnn...)
Polz, a, mo)
LEAA
Σ a
(p(())m((Aaa)
Ei A]
po a.
LoEA
(A(())((A=))
Σ
AI[(C)), A]
(a a, σ)
po(x))o(AAa)
([
Pe( a, mD)
LaEA
Σ
LaEAA
(15)
Palx, a)
= By(w) LaEd
Σ
dr (2,
Σ To (C)(),)( Al]
PolE, a, To)
Ep(())
Σ (g( a) Σ [(AA)E) A)
LaEd
Po (e, a)
Ep(x))
dr (x, a)poot,
-Ep(z)
Σ (1
Palx, a, mo)
Laed
LaEd
(c(x, a,(x,
= Epp(s)
Pe a, mo) LaEA
Σ
a, To)
az, To)
2r
(2, a)
where we use Condition 3.2 Eq.
C.3 PROOF THEOREM
We can derive the variance of CIPS by setting (x, a) 0 in the variance of CDR.
nVD
[cIPs]
Σ { (p(c))o(AA)))
[w2 (z, a, mo)o" (x, (z, a, A)]
aeA
Ep(z) ( ( a, mo)g a) To(a l,clo, A)]] V Pos a, ()
where 03(,, a, A) = V(C(a)R(a) ar, A).
C.4 PROOF OF THEOREM
We show that CDR is unbiased under Conditions 3.1 and
Po a. T) Pa(x; a, T6)
(C,(a)R,(a) pola, a, ()G, a)) Ex((())) (pc a, (x,, a)]
n
aEA
15

---

Published as a conference paper at ICLR 2026
(C(a))(() pa(a, ( + (x(2)) (ec a, () a)l
Pe(a, a, To)
LEA
Pe(a, a,
(p(s)mo(A(e)
([()( A] pa(c, () (2; + (AIo) pa(a, a, A) (2, a)]
uEA a, no)
Po(
(Bp())o(AAl))
a a,,,,, 100 Bu(als) (p(a,, A) (x,
LEe a,
Pe(
pp(s)
Σ (0(A Bx(A/=) [po a, (z, a)]
Pa(a, a, To)
Pe(I, a.
= =Bp(s)
Σ LaEA
(0.(2, q. pe + polx, a
Po(x, a, To)
= =Ep(s)
pala, LaEA
Σ
(1r( a) (z, a)) + a a (1,
Bp(x)
Σ LaEd
pola, a, a,
pp(r)
Σ
T (Alr))po a, A)q-(zz
Σ Po(x, a, (z)
(p(x))(A(a))
LEEA
((a)
LAEA
-i)
C.5 PROOF OF THEOREM 4.2
Bias (VODR)
1
ae (C.(a))a(a)) pe(x;, A1) Ea(A mn) Pe a, A)
Ep(D)
V((x)
n
i=1 aEA
a,
Ep()
(C(a)
Bx(A))) a, V(TT)
LaEA Po(ox, a, mo)
a,
= (Apl))
(pe(x, a, A)q, f, A1q + fc (2,
-V(T)
Po(I, mo)
LEA
a,
Ep(a)
a +
- VV(T)
De(x,a,m))
LEA
Pe(I, aT)
=pp(.)
Σ
Pe a (0)0
Ep(s) (,
L po(x, a, no)
Epp(a)
Σ Pe(ar, a, no)
a,)
Pe(x, a, TT Pe(x, a, x0)
gr (x)
(2 a, np)
Ladd
= -Bias (Virps)
(16)
16

---

Published as a conference paper at ICLR 2026
C.6
PROOF OF THEOREM 4.3
nVD VCDR
ae(
=dVD
(Ci(a)RR(a)) Pe(a,, a, A)g, (((n) lolon, a, (2)) A)
n Po( a,
Pa(a, a, x)
V
- + Pe(a, a, (z,
Po(I, a, To)
mEA
-Bp(a)to(Ala))
V VI(C.R(2A))
Σ w(2,a,T,m)) (C(a)) (((la)))))))(((())))))())))))))) Pa(t),, A) a)) + E (app) Po(x, (z,
aEd
+V
[P(C,
Σ w(2,a,,,m)) (C(a) pe(x, a, (x,- + Ba(Al)) (pa A) a)]
Σ wi)
-Bp(s)m(AAl))
aEA
+ +V((s)ooA(la)
Σ
(2c a, (),r(r, () + Ba(A(a) [pel (x, a)]
oEA
=Bp(s)moAAl))
w (2, (4,
+ Ep(() V (T(())=)
Σ w(z,a,,,m)) (pol0, a, (z), po(a, a, A)G, (2, + Em(((a) lola, a, A) a)] aed
+ px(x) BTo()|
Σ w(z,a,7,m)) (pol0, a, A)g, pol2, a, (z, a)) + (a(( po(x, A) (z. a)]
aEA
(ppa))o(A(l))
u ((
+ Ep(x)
aEA
+ +V px))
a (g) (
(z Σ (z A) + En(apa) [po a, (x,
aEA
(All))o(AAln)
w (1
aEA
+ Ex(e))
Σ
(x, ( A)]
aEA
10(,,,,T,m (0) (() a (2, ((t + Br(A)E) poca, a, () a)]
+ p((x)
aEA
(pp())oo(All))
+ Ep(e)
Σ aEd
w (, q (x) me (aa Al]
+ Vp(x)
a,( (g, (x, + (
aeA
(
(AA
aEAA
+ Ep(())
Σ
w (2, (x, 0) g (x, (oal A1]
GEA
17

---

Published as a conference paper at ICLR 2026
+ p(z))
Σ
aed
{ Ep(o)m((Ala)) aEA
-E
[ (I. 4))
Byco) (1) a a, (3, a)V no (A + +
where (I, = 4r (z, dr (x)
D ADDITIONAL EXPERIMENTAL SETUPS AND RESULTS
This section describes the detailed experimental settings and reports additional results.
D.1 SYNTHETIC EXPERIMENTS
Detailed Setup. We first describe the synthetic experiment settings in detail, followed by how define the expected reward function. In the synthetic experiments, the expected reward function is defined as follows:
(e A(k)) 1 k
Σ 14k
Wc(A(t)
A(k)) = (0, +
W.(A()),
k -
tych
where W. and W, are sampled from a uniform distribution with range [-3.0,3.0] and [-1.0, 1.0). respectively. Additionally, (a( A(k)) and (r, A(k)) are defined as follows.
(x, A(k)) xx M My Ta + (02) (09) Taa
(2r A(:)) My MM a. Fa + (0²)²* x + (0") Nas
where Ta denotes action context for A(k) = a represented by one-hot vector. Ma Og and 0g are sampled from uniform distribution with range [-
We then synthetically define a logging policy as in Eq. 12 We adopted a modified version of the Plackett-Luce based formulation in Eq. [2] it allows smooth transition between stochastic and deterministic regimes through the parameter. This design enables controlled experiments in which we can vary the determinism of the logging policy while holding other factors constant, making it ideal for studying bias behavior across regimes in our experiments. Similarly, the new policy Eq. [3 follows a standard e-greedy form that allows adjustable stochasticity e, a setup widely used in prior OPE studies Saito & Joachims 2002 Kiyohara al. (2023).
we report additional results from the synthetic experi- ments demonstrate that CIPS outperforms the baselines. We vary the new ranking policies, the definition of logging and new policies, and the noise on the click probability of CIPS. Note that the logged data size set to 1000, the number of unique actions is the ranking length is oo, é = 0.3, and X 0.5.
Additional Results. From Figure 7
How does CIPS perform with varying new ranking policies? Figure 7 shows comparisons of estimators under varying new ranking policies (e {0.0, 0.2, 0.4, 0.6, 0.8, T.03 in Eq. 13). A larger value makes the new ranking policy closer to a random uniform distribution. policy is completely deterministic. We observe that CIPS outperforms the baselines across all values of e. As increases, the probability of the supported action increases. Consequently, the basclines reduce their bias. However, the baselines still exhibit severe bias, particularly when the new ranking policy is completely deterministic (c 0.0).
0.0 means the
How does CIPS perform with different definition of logging and new policies? Figure reports comparisons of estimators with new different definition of logging and new policies from Eq. 12
18

---

leverage data-driven Published as a conference paper at ICLR 2026
For the logging policy, define fo in Eq. 12 fu(z, d.(æ, A(k)) + a, where a is sampled from the standard normal distribution. We then construct the new policies by mixing the softmax and epsilon greedy policies follows.
and 13
K
(A
II0.5. 0.5
(1-c) aaak) argmax A((z,a)) d'EA/SLLA-I
TA a1
exp(fi (z,), & an
+ 0.5
En' En'E/ exp(fi(z,a')
we can see that CIPS outperforms the baselines under more complex definition of logging and new policies. This results suggest that the advantage of CIPS is not limited by the structures of logging and new polices.
In Figure 8
How does CIPS perform with varying noise on the click probability CIPS? In Figure 9 we compares CIPS against the baselines we vary the accuracy of click probability used in CIPS. The x-axis represents levels of noise, which is sampled from a uniform distribution with range [- 8]. Figure 9 shows that CIPS is superior to the baselines with noisy click probability, while its bias gradually increases. This increase demonstrate the robustness of CIPS even with larger noise its click probability model and its consistent relative advantage over the baselines (note that the baselines are independent of the noise parameter).
bias is consistent with Theorem 3.2 These results further
From these results, we can see that it is important to use accurate click probabilities for low MSE es- timations. Regarding model selection for estimating click probabilities, estimator selection techniques, such as the ones proposed by [Uaaawwa et al. (2023) and Felicioni al. (202)) By treating CIPS with different click-probabiltty estimators as separate estimators, these techniques allow us to automatically and data-drivenly identify the click click-probability model that minimizes the resulting MSE.
D.2 ABLATION STUDY SYNTHETIC DATA
In addition to the main performance comparisons, we conduct two ablation studies.
Figure 10 compares CDR against CIPS we vary the accuracy of the regression model d. used in CDR. Here, the x-axis represents the estimation error of the regression model; larger values corre- spond to less accurate estimates of gr. The results show that, when Gr is estimated with reasonable accuracy, CDR outperforms CIPS by leveraging its variance reduction effect, consistent with our theoretical analysis. We also observe that as the estimation noise in the regression model increases, the variance of CDR grows steadily, eventually eliminating its advantage over CIPS.
Next, Figure IT evaluates the policy selection accuracy of the estimators under varying logged data sizes (n) and deterministic user thresholds (a:). We measure how accurately each estimator identifies the better policy between the new policy я and the logging policy To across 100 independent trials. Note that, our setup, the new policy consistently outperforms the logging policy in expectation. The left plot demonstrates that CIPS achieves the highest selection accuracy across all data sizes, while the baselines fail entirely with zero accuracy. This failure is expected because, under severe common support violations, as shown in Theorem 2 baseline methods substantially underestimate the value of the new policy æ, leading systematic errors policy selection. The right plot shows that CIPS maintains high selection accuracy even at larger a values, where the logging policy be- comes more deterministic and the baselines deteriorate substantially. These findings highlight the superiority of CIPS over the baselines in both policy selection and policy evaluation.
D.3 REAL-WORLD EXPERIMENTS
Detailed Setup. The detailed real-world experimental settings are described the main text. We conduct OPE experiments consists of recommendation logs from the video-sharing app Kuaishou. KuaiRec contains fully observed user-item interactions with nearly 100% density for subset of its users and items. By leveraging these user user-item item interactions, we can construct OPE experiments with minimal synthetic components Gao et al. 2022
real-world dataset called KuaiRec Gao et al. (2022). which
19

---

Published as a conference paper at ICLR 2026
KuaiRec includes user-feature csv, which ture dimension reduction using PCA implemented scikit-learn (Pedregosa et al. 2011). We then the user-item interaction matrix recorded in the original data as the potential reward function 4r (x, A(k)). We construct yr (x, A(k)) by randomly sampling rows and columns from the user-item matrix. We also define the expected click probability
context vectors (a). We apply fea-
if ga, A(k)) > 2.0
4. (c(x,
(17)
AR))(R)
otherwise
where is noise parameter sampled independently from the uniform distribution over [0, 0.5]. We set the threshold for binarization by following the example provided on the KuaiRee webpage. This definition reflects essential user behaviors, namely that more relevant items to be clicked, and that user responses remain probabilistic even for highly relevant items. Thus, this setting has some logic and is consistent with formulations, and have designed it with available modeling rationale in mind.
likely
Additional discussion about further bias mitigation. In 5 we observe that CIPS exhibits some non-negligible bias. This is because both the logging and new policies are deterministic, violating even the click-wise common support (Condition 3.1 of our methods. this extreme case, no OPE method, including ours and existings, can remain strictly unbiased due to the absence of overlapping actions. Nevertheless, our results show that CIPS maintains significantly smaller bias and MSE than existing methods because it can still utilize some stochasticity embedded in user clicks. To further mitigate bias in the case with both deterministic logging and new policies, we can additionally leverage action embeddings when available as proposed in Saito & Joachims 2022 which allows us to use similaritics across different actions in OPE.
IPS
IIPS
RIPS
- CIPS
CIPS (true CTR)
Normalized
Squared Bias
Variance
1.2
1.
1.0
1.0
0.225
0.8
0.200
0.6
0.675
0.4
0.050
0.2
0.09%
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.000
0.0
0.4
0.6
0.8
3.0
epsilon
epsilon
epsilon
Figure 7: Comparison of MSE, Squared Bias, and Variance with varying new policies (e).
IPS
IIPS
CIPS
CIPS (true CTR)
Normalized MSE
Squared Bias
Variance
0.6
0.6
0.4
0.00
8.2
0.04
0.0
0.0
0.00
1000
logged
logged
logged data
Figure 8: Comparison of MSE, Squared Bias, and Variance with different policy definition.
20

---

Published as a conference paper at ICLR 2026
IPS
---
IIPS
RIPS
CIPS
1.2
Normalized MSE
1.2
Squared Blas
Variance
0.25
0.9
0.9
0.20
0.0
0.35
0.3
0.3
0.06
0.0
3.0
0.5
3.0
3.5
2,0
2,5
3.0
3.5
0.0
1.0
1.5
2.0
2.5
3.0
3.5
0.00
0.0
0.5
1.0
3.5
2.0
2.5
3.0
3.5
estimation noise on
estimation nois
estimation noise on
Figure 9: Comparison of MSE, Squared Bias, and Variance with varying estimation noise on Pe.
CIPS
CDR
+ CDR (true CTR)
Squared
0.12
0.22
0.00
8.0
0.06
0.000
0.02
0,0000
0.0
2.0
4.0
6.0
8.0
0.9
2.0
4.0
6.0
8.0
0.0
2,0
4.0
6.0
8.0
astimation noise on
estimation noise on Pr
estimation noise on
Figure 10: Comparison of CIPS and CDR' MSE, Squared Bias, and Variance with varying estima- tion noises on 9r-
IIPS
RIPS
CIPS
CIPS (true CTR)
policy selection accuracy
1.0
policy selection accuracy
1.0
0.4
0.4
0.0
2000
-1.0
00.5
1.0
logged data size
deterministic user threshold
Figure 11: Comparison of the policy selection accuracy with varying logged data sizes (n) and deterministic user thresholds (a).
21


Reviews:
[{"review_id": "h2mwZwoWrG", "reviewer": "Reviewer_WNN2", "summary": "This paper studies the off-policy evaluation for ranking policies. Existing approaches require the data collection (logging) policy to be sufficiently stochastic and do not perform well when it is deterministic. This paper has overcome this challenge by proposing novel estimators, referred to as Click-based Inverse Propensity Score (CIPS), based on the intrinsic stochasticity of user click behavior. This paper proceeds as follows: the considered off-policy evaluation problem is formulated in Section 2, and the CIPS estimator is proposed and analyzed in Section 3. Preliminary experimental results are demonstrated in Section 4.", "questions": "- Please try to address the weaknesses listed above. I will consider increasing my score if some of them are addressed.\n\n- If possible, I recommend moving the Click-based Doubly Robust estimator to the main body of this paper. I think it is important.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper has considered a very practical problem: how to do off-policy evaluation for ranking policies when the data collection (logging) policy is deterministic (or not sufficiently stochastic). This is a practical and important problem in algorithmic ranking systems, such as recommendation systems.\n\n- The proposed algorithm, CIPS, is natural and simple.\n\nOverall, I think this is an interesting paper, and recommend accepting it.", "weaknesses": "- I have concerns about some key math notations in this paper. In particular, notations like $C(a)$, $R(a)$, $C(k)$, and $R(k)$ can be misleading. Specifically, $C(k)$ hints that the click event **only** depends on the position $k$, while $C(a)$ hints that the click event **only** depends on the action $a$. I do not think this is what this paper has assumed. I recommend that the authors clean up such notations.\n\n- The idea of this paper is very interesting. However, given the idea and the CIPS algorithm, the analyses for the CIPS algorithm (Theorems 3.1-3.3) are relatively straightforward based on existing literature and techniques. If the authors disagree, please explain the key challenges and novelties in the proofs for Theorems 3.1-3.3, compared to existing literature.\n\n- In the experiments, this paper has only considered a fixed logging policy defined in equation 11 and a fixed \"new\" policy defined in equation 12. First, please better motivate and explain why choosing such a logging policy and such a new policy. Second, I strongly recommend that the authors also include experimental results with a different logging policy and a different new policy to justify that the advantages of CIPS are not limited to such policies.\n\n- In Section 4.2, the click probability model for real-world data defined in equation 13 seems pretty arbitrary. Please better motivate and justify it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UP2lFyGgpP", "reviewer": "Reviewer_m2rz", "summary": "This paper addresses the challenge of Off-Policy Evaluation (OPE) for ranking policies, which aims to estimate a new policy's performance using data collected by a different, existing policy. The authors highlight that standard estimators, like Inverse Propensity Score (IPS), fail and suffer from severe bias when the logging policy is deterministic.\nThe paper's contribution is an estimator called the Click-based Inverse Propensity Score (CIPS). The key finding is that reliable OPE is possible even with deterministic logging by exploiting a different source of randomness: the intrinsic stochasticity of user click behavior. Instead of weighting by the policy's action probabilities, CIPS uses the ratio of marginalized click probabilities as a new form of importance weighting. Theoretical analysis shows CIPS is unbiased under specific conditions. Experiments on synthetic and real-world data demonstrate that CIPS achieves lower bias and more accurate estimations than existing methods in deterministic settings.", "questions": "1- Your theoretical analysis for CIPS's unbiasedness relies on the \"Independence of Potential Rewards\" (Condition 3.2). How robust is the CIPS estimator in practical scenarios where this assumption is strongly violated, such as systems with significant item complementarity or competition, and what is the expected impact on its bias?\n\n2- Theorem 3.2 indicates that the bias of CIPS depends on the accuracy of the ratio of estimated click probabilities. How sensitive is the estimator's performance to errors in this ratio, particularly in low-data regimes, and what are the practical implications for model selection when estimating these click probabilities from logged data?\n\n3- Your real-world experiment noted non negligible bias when both the logging and new policies were deterministic. This suggests violations of the clickwise common support (Condition 3.1). Could you elaborate on the performance degradation in this specific scenario and discuss potential extensions or clipping methods to mitigate this bias when even clickbased support is deficient?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper's primary strength is proposing an approach to address the Off-Policy Evaluation (OPE) issue when the data-logging policy is fully deterministic, a setting where existing methods fail.\n\n* The CIPS estimator method interestingly shifts the source of randomness away from the deterministic policy and instead exploits the intrinsic stochasticity of user click behavior for importance weighting. This approach is shown to \"significantly\" reduce the severe bias of standard estimators in this challenging but practical setting.", "weaknesses": "The following are some of the items that it make the paper better to see them developed/explained/addressed further:\n\n* The theoretical unbiasedness of CIPS depends on Condition 3.2, which assumes that the expected potential reward for an item (e.g., a purchase after a click) depends only on that item, not on the other items shown in the ranking. The paper's own experiments show that as this condition is more strongly violated, the Mean Squared Error (MSE) of CIPS, while still the best, does increase. The experiment section didn’t develop much on this aspect and how to potentially address it.\n\n* The CIPS estimator relies on knowing the true click probabilities for both the logging and new policies. In practice, these are unknown and must be estimated from data. As Theorem 3.2 shows, any inaccuracy in the ratio of these estimated click probabilities will introduce bias into the final policy evaluation.\n\n* While CIPS is designed for deterministic logging policies, it can still suffer from \"non-negligible bias\" in the most challenging scenarios. Specifically, the real-world experiment (which used a deterministic new policy in addition to a deterministic logging policy) showed this bias. The authors state this is due to violations of the click-wise common support (Condition 3.1), which can happen when both policies are deterministic. It would be great to develop further how to address this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IwchOQWfYJ", "reviewer": "Reviewer_RmhY", "summary": "This paper investigates the severe bias introduced by existing Off-Policy Evaluation (OPE) methods when applied under deterministic logging, and illustrates this problem through concrete examples and theoretical analysis. To address this issue, the authors propose a Click-based Inverse Propensity Score (CIPS) estimator, which exploits the intrinsic stochasticity of user click behavior through click importance weights, substantially relaxing the support condition and enabling unbiased or low-bias estimation under deterministic logging. The effectiveness of the proposed method is thoroughly validated through rigorous theoretical analysis that establishes its unbiasedness and consistency properties, and extensive experimental results that empirically confirm its superior performance over existing OPE approaches under deterministic logging.", "questions": "1. In real-world scenarios, although the IPS exhibits relatively large bias compared to other approaches, why does this bias remain unchanged when the logging policy shifts from stochastic to more deterministic?\n2. In real-world scenarios, why does the CIPS method appear less robust to bias across varying degrees of deterministic policies compared to synthetic data? What might be the underlying causes of this discrepancy?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper is motivated by a well-founded research question — existing OPE methods are primarily developed for stochastic logging policies, which inherently limit their reliability under deterministic settings. This motivation is both sound and clearly articulated.\n2. To alleviate the bias commonly induced by the OPE methods under deterministic logging, the paper relaxes the stochasticity assumption of existing OPE methods and takes a novel perspective by leveraging the inherent randomness in user click behaviors. By leveraging click probabilities for reweighting, the proposed CIPS method represents a clever and conceptually elegant reformulation of the problem.\n3. The paper clearly illustrates the bias issue of IPS-based estimators under deterministic logging through intuitive examples, making the problem easy to understand. Moreover, the theoretical analysis is thorough and well-justified, and the experimental results further validate that CIPS effectively mitigates bias across varying degrees of determinism in logging policies.", "weaknesses": "1. The CIPS method implicitly assumes consistency in the action space between the old and new policies. It remains unclear whether CIPS can maintain its robustness when new actions appear under the new policy.\n2. In the synthetic data section, the exact forms of functions appearing in Equations (9) and (10) are not clearly specified.\n3. The paper lacks a dedicated related work section. The relation between this work and related work should be discussed. \n4. Baselines are limited. More recent baselines should be used. The strongest baseline, RIPS, was proposed in 2020; it is unclear whether more recent OPE methods should be considered for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y8suul0h3P", "reviewer": "Reviewer_wdph", "summary": "This paper studies off-policy evaluation (OPE) for ranking policies under deterministic logging, where existing estimators such as IPS, IIPS, and RIPS suffer from severe bias due to the lack of exploration. The authors propose a new estimator, Click-based Inverse Propensity Score (CIPS), which leverages the stochasticity of user click behavior instead of the logging policy. The paper provides theoretical analyses proving that CIPS is unbiased under mild assumptions and demonstrates through synthetic and real-world experiments that CIPS substantially reduces bias compared to existing methods.", "questions": "1. Could the authors clarify the rationale behind the reward design, particularly the two-stage structure of “clicks” and “potential rewards”? How is this modeling choice connected to real-world ranking scenarios, and to what extent does it affect the theoretical guarantees or the empirical behavior of CIPS? For example, would CIPS still perform similarly if the downstream reward R were not conditioned on the click C?\n2. In the real-world experiments (KuaiRec), the click probabilities were generated using a simulated noise model (Eq. 13).\nCould the authors provide more discussion or evidence about how well this setting reflects real deterministic logging systems?\n3. While CIPS focuses on bias reduction, could the authors elaborate on its variance behavior in practical settings where click probabilities are estimated with noise? How sensitive is CIPS to misspecification of the click model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important and realistic problem in OPE that has been largely overlooked — deterministic logging policies.\n2. The idea of exploiting user click stochasticity as a new source of importance weighting is conceptually novel.\n3. Theoretical results are clearly stated and well justified.\n4. The paper is clearly written and easy to follow.", "weaknesses": "1. The effectiveness of CIPS depends on the accuracy of estimated click probabilities, which may be difficult to obtain in real systems.\n2. The independence of potential rewards assumption may not hold in many practical ranking scenarios.\n3. Real-world validation relies on simulated click probabilities (Eq. 13), limiting the strength of empirical evidence.\n4. No statistical significance or confidence intervals are reported for the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)