Uncertainty-aware off policy learning
Authors: Xiaoying Zhang
Junpu Chen
Hongning Wang
Hong Xie
Hang Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: off-policy learning
uncertainty

Abstract: Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applications, such as search engines, recommender systems, etc.  While the ground-truth logging policy, which generates the logged data, is usually unknown, previous work directly takes its estimated value in off-policy learning, resulting in a biased estimator. This estimator has both high bias and variance on samples with small and inaccurate estimated logging probabilities. 
In this work, we explicitly model the uncertainty in the estimated logging policy and propose a novel  \underline{U}ncertainty-aware \underline{I}nverse  \underline{P}ropensity \underline{S}core estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world  recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.

Under review as a conference paper at ICLR 2023
UNCERTAINTY-AWAN OFF POLICY LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Off- Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applica- tions, such as search engines, recommender systems, etc. While the ground-truth logging policy, which generates the logged data, is usually unknown, previous work directly takes its estimated value in off-policy learning, resulting in a biased estimator. This estimator has both high bias and variance on samples with small and inaccurate estimated logging prohabilities. In this work, we explicitly model the uncertainty in the estimated logging policy and propose a novel Uncertainty- aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
INTRODUCTION
In many real-world applications, including search engines (Agarwal et al. (2019)), ), online advertise- ments (Strehl et al. (2010)), recommender systems (Chen et al. (2019): Liu et al. (2022) ), only logged feedback data is available for subsequent policy optimization. For example, in recommender sys- tems, various complex recommendation models (i.e., policies) (Zhou et al. ( (2018); B Guo et a (2017)) were optimized with logged user interactions (e.g.. clicks or staytime) to items recommended by pre- vious recommendation policies. However, such logged data is known to be biased, since one does not know the feedback on items that previous policy (which is generally referred as the logging pol- icy) did not take. This inevitably distorts the evaluation and optimization of new policy when it tends to select items that are not in the logged data.
Off-policy learning (Thrun & Littman (2000); |rrccup (200)) emerges favorable way to learn an improved policy only from the logged data by addressing the mismatch between the learning policy and the logging policy. One of the most commonly used off-policy learning methods is Inverse Propensity Scoring (IPS) (Chen et al. (2019) Munos et al. (2016)), which assigns per-sample importance weight to the training objective on the logged data, as get unbiased optimization objective in expectation. The importance weight in IPS is the probability ratio between the learning policy and the logging policy.
However, the ground- truth logging policy is unavailable to the learner, c.g., it is not recorded in the data. One common treatment taken by previous work (Strehl et al. (2010): Liu et al. (2022) Chen eet al. (2019): Ma et al. (2020) is to first employ a supervised learning method (e.g., logistic regres- sion, neural networks, etc.) to estimate the logging policy, and then take the estimated logging policy for off-policy learning. We theoretically show that such an approximation results in biased estima- which is sensitive to those inaccurate and small estimated logging probabilities. Worse still, the small values of the estimated logging probabilities usually mean that there are fewer related samples in the logged data, so its estimation usually has high uncertainties, i.e., inaccurate estimation with high probability. Figure| [ shows piece of empirical evidence from large-scale recommendation benchmark KuaiRec dataset (Gao et al.] (2022)) ) where items with lower frequencies the logged dataset have lower estimated logging probabilities and higher uncertainties concurrently. The high bias and variance caused by these samples greatly hinder the performance of off-policy learning.
In this work, we explicitly take the uncertainty of the estimated logging policy into consideration and design a novel Ucertainty-awee aware Inverse Propensity Score estimator (UIPS) as the optimization objective for policy learning. UIPS introduces propensity from the estimated one, and learns an improved policy by alternating: (1) Find the opti- mal weight that makes the estimator accurate as possible, taking into consideration the uncertainty
additional weight to approach the ground-truth

---

Under review as a conference paper at ICLR 2023
Patssund Lnagreg Protahlts
Uscottaas
0.0
Loa(item (requency)
(a) Estimated Logging Probability
(b) Uncertainty Estimation
Figure Estimated logging policy and its uncertainty under different item frequency on KuaiRec.
of the estimated logging policy; (2) Improve the policy by optimizing the resulting estimator. We further find a closed-form solution for the optimal weight by deriving an upper bound on the mean squared error (MSE) the ground-truth policy value. The optimal weight adjusts sample weights considering both the uncertainty of estimated logging probabilities and the propensity scores, rather than simply boosting or penalizing samples with high uncertain logging probabilities. Experiment results on the synthetic and three real-world recommendation datasets demonstrate the efficiency of UIPS. All data and code can be found in supplementary materials for reproducibility.
To summarize, our contribution in this work is as follows:
We point that directly using the estimated logging policy leads sub-optimal off-policy learn- ing, since the resulting biased estimator is greatly distorted by samples with inaccurate and small estimated logging probabilities.
We take the uncertainty of the estimated logging policy into consideration and propose UIPS for more accurate off- off-policy learning.
Experiments synthetic and three real-world recommendation datasets demonstrate UIPS's strong advantage over tfte-of-the--ee methods.
PRELIMINARY: OFF-POLICY LEARNING
We focus on the standard contextual bandit setup to explain the kev concepts. Following convention (Joachims et al. (2018); Saito & Joachims (2022); Su et al. (2020)). let E X C Rtd be d- dimensional context vector drawn from an unknown probability distribution p(x). Each context is associated with finite set of actions denoted by A. where |A| < co. Let A X - [o, 1] denote a stochastic policy, such that (al(²) is the probability of selecting action a under context æ and Under given context, reward Tz.a is observed when action is chosen. Take
EaeA <aa(==) news recommendation for example, æ represents the state of a user, summarizing his/her interaction history with the recommender system, each action is a candidate recommendation algorithm, and the reward Te.a denotes the user feedback on article a, e.g., whether the user clicks the article. Let ((T) denote the expected reward value of the policy w.
article, the policy
V(T) =
[r
(1)
We look for a policy T(a(æ)) to maximize (TT). the rest of the paper, we denote Exxxia))) as [.] for simplicity.
In contrast to performing online updates by following the learning policy ((alx), in off off-policy learning we can only access set of logged feedback data denoted by D == {(x,, any T Ta,, a.. )n E [N]), where [N] == [1....,N). Given E.., the action On was generated by a stochastic logging policy 3*, i... the probability action a, was selected is (a (a,) The actions (aI.... ax and their corresponding rewards (fz].ar (Tziaar..... : Fax.av } are generated independently given 3* Due the nature of policy optimization, the learning policy π (aaxz) x) is expected to different from 8 falx;,, unless 3* (a|z) is already optimal. Moreover, in practice the situation could be further complicated. Again, consider the news recommendation scenario. Due to the scalability requirement, industrial recommender systems usually adopt two-stage framework (Ma et al. (2020)). where one several
2

---

Under review as a conference paper at ICLR 2023
candidate generation models first produce a candidate set and separate ranking model reranks candidate items to present top- item to users. While 8 (a)az) depicts the whole two-stage process, the learning policy x(a ()) is usually employed in particular stage (e.g., the reranking stage), implying drastic differences between the logging and learning policies. The main challenge off- policy learning is then address the distribution discrepancy between / (alar) and (alaa)) and learn& a policy π T [c) to maximize V (T) with access only to the logged dataset D.
One of most widely used methods address the distribution shift between I (aa x)) and 3 ()) is the Inverse Propensity Score (IPS) (Chen et al. (2019) MMnoss et al. (0166). One can easily get that:
((T) = E3-
s(alm) (-(aa))) T2.a
yielding the following empirical estimator of ((=):
Vips(*) Σ man mn))
(" [r.) O
(2)
where (a, ((n)//))/// (an is referred to as the propensity score. In the rest of paper, without further specification, we use the empirical estimation of expectation in our practical calculation. Various algorithms can be readily used for policy optimization under (Trs(T), including value- based methods (Silver et al. (2016)), policy-based methods (Levine & Koltun (2013); [Schulman et al.) (2013); Williams] (1992)). In this work, we adopt a well-known policy gradient algorithm,
REINFORCE (Williams (1992). ). Assume the policy π (alæ)) is parameterized by 9, via the "log-
trick", the gradient of (ir)(TT)) with respect to & can be readily derived as follows:
V (TV) N
maan (a,, (en) Tan.an V log (o (a,,(x)))
(3)
Approximation with unknown logging policy. In many real-world applications, the ground-truth logging policy, i.e., the 3 (a(1z) of each observation (æ,, a), is unknown. One reason is the legacy issue, i.e., the probabilities were not logged when collecting data. Another reason is that the exact value of (a(ar) is intrinsically unavailable such as in the two-stage recommender systems. As the solution, previous work employs various supervised learning methods (c.g.. logistic regression (Schnabel al. (2016), nerural networks (Chen et al. (2019), etc.) to estimate the logging pol- icy, and replaces 3 (alar) with its estimated value ((a)z) to get the following estimator for policy learning:
(T = Σ En=1 B(a., (2.) T.
Talanla.. (
However, as shown in the following proposition, inaccurate ((a(æ) leads to high bias and variance of VIIPS(TT (TT). Worse still, smaller inacourate ((a(z) further enlarges this bias and variance.
Proposition 1. The bias and variance of (surs(TT)) can be derived as follows:
Bias (To) = ED (UBIPS(TT) - V(TT)) = ETo |an (a(()))
- -1]]
A(aaar)
N Varn (ViIPs(TT)) = Var (B'(alm)) B(aa=)
(al() (-(a((]
-1) 1)
3 oz) v m.a
Ta,aa
+ Ero
Smaller 3(a]z) usually implies fewer related training samples in the logged data, and thus 3(a a
will be inaccurate with a higher probability. To make it more explicit, we take KuaiRec dataset Gaa al. 2022)) as an example and estimate the logging policy following (Chen et al. (2019)). Figure shows the estimated ((aaz) and its corresponding uncertainties in items of different observation frequencies in the logged dataset. As uncertainty measures how large the confidence interval is about the current estimation, higher uncertainty implies that the true value may be away from the empirical mean estimate with high probability. We defer the discussion about our detailed uncer- tainty calculation in Section 3] We can observe from Figure T that as item frequency decreases, the estimated logging probability also decreases, but the estimation uncertainty increases. This implies that smaller 3(a xx) usually 1) more inaccurate and 2) associated with high uncertainty.
As a result, with high bias and variance caused by inaccurate 3(a[z). it is erroneous to improve To (a/z) by simply optimizing (n We propose uncertainty-aware off-policy learning to address this challenge.
3

---

Under review as a conference paper at ICLR 2023
3 UNCERTAINTY-AWARE OFF-POLICY LEARNING
Our idea is incorporating the uncertainty of the logging policy estimation into policy learning. Ob- serving that
V(TT) = Es-- (e(ala)) A(a m) ["(a]]]
(())))
we propose to learn the optimal policy by optimizing the following empirical estimator:
VUIPS(TO) = # E
nolanla. Blanken
Oze On
Tn On
(5)
where a,, is a weight, which reflects 3(am, (ann/ (an)) be selected to make Vurps (ma) as close to (TT) as possible, Intuitively, one should give small weights to samples whose ((a (x) is far below the ground-truth 3* (a|x). Thus, we divide offline policy improvement into two steps, and repeat them until certain convergence condition is met:
Uncertainty aware policy evaluation: Derive the optimal uncertainty aware óz to make
(TI accurate as possible.
Policy Improvement: Learn an improved policy To x>) by optimizing VUIPS(TO).
3.1 Uncertainty Aware Policy Evaluation
Optimal uncertainty aware weight Φa.a. We measure the accuracy of VUIPs(Te) by its mean squared error (MSE) to (m following previous work (Su et al. (2020); Saito & Joachims 20222) )- MSE captures both the bias and variance of an estimator, since it is the summation of squared bias and variance. We then locate the Φx that can minimize the MSE. particular, we demonstrate the optimal Φz.a has a closed-fone formm formula which relates to both the value of (a)a))) (((((z)) (x) and the estimation uncertainty of }(a(æ:).
More specifically, instead of directly minimizing the MSE, which is intractable, we find the desirable Φn by minimizing the upper bound of MSE in the following theorem.
Theorem 1. Assume Ta..a E [0, 1), the mean squared error (MSE) between Vuurs(=o) and ground- truth estimator V(Ta) is upper bounded as follows:
MSE (VUIsS(TO)) = ED (VUIPS(TO) - V(TO)) = Bias (VUIPS(TO)) + Var (VUIPS(TO)
S Ero
[ a.a (a(o))
xo(al Eg.
(
(a/2z) A(ale)
Da.a - 1) 2
+ Eg.
[xa(alx)² M(alaaa Pa.a
The upper bound in Theorem
I strictll increases with the two expectations related to dr.a. which implies that for some choice X E [0, 2o]. the MSE-optimizing Φara can be derived by minimizing:
AIE3- (
(a(a)) B(a|=))
Pz.a - 1)}]
Ta(al=) B(a x)a=)² 2.0
Eg-
We cannot directly minimize [q. since the unknown (a(=x) is involved. However, various ways (Gal & Ghahramani (2016); Xu et al. (202I)) can employed to get the confidence interval which will contain 8 (a x with high probability. More specifically, following previous work (Joachims) jtt al. (018)). ) we assume 3 (a|x) can be modelled by a softmax function on top of an unknown function fe- (x, a), i.e., the realizable assumption. Then can get:
(ala: Σ., expifo
exp(ff= (m,a)) (z,a"))² (x..)))))
B(alx)) =
cxp(fo(mmaa)
E., >.. exp(fo(a.a"))
where fo(æ, a) is an estimate of fo- (z,a). Following the conventional definition of confidence in- terval (Abbasi- Yadkori al. (201T), we define ? and Un,a such that Ife- (xx, fo a)l S yUa,a holds with probability at least 1-8, where 7 is function of ó (typically the smaller å is, the larger y is). Then yUn. measures the width of confidence interval of fo(æ, a) against its ground- truth So- (2, This implies that 8* (alæx E B with probability at least 1-ō, where:
B
22spp- Hompl-nUles a) 3(a)2)) 20
2mmpfyUz.al 2.
7" n== Ea [d))))),) 2 E.r exp(fo(a')a))
4

---

Under review as a conference paper at ICLR 2023
Since 5 (a) can be any value in Bæ.a. we adopt the idea of robust optimization Chen et al. 2020 ) and find the optimal Φz by solving the following optimization problem:
minea.a max Ba.a
(
(
Ox Oz - 1) + +1 ((olz)
[a(al=)
3(oo2))
Pz.a
(8)
The following theorem derives closed-form formula for the optimal solution of (8)
Theorem 2. Let "h-?h 122 E (exp(-UUmat exp(TUmax) where U = maxa Un The optimiza- tion problem in Eq.18) has a closed-form solution as follows:
22 min
( exp(-uUm.a) + (x//))=xxp(-<(())
(
2/2/ (exp (7Um,,) + exp (-7 Un,a))
Insights on 22.0 The second term of &z (i.e., 2/2/ (exp (?Um..)) ) + exp (->Us,a)) acts like with small To To / BB xz) shown in Lemma in Appendix| [A.4 The key component the first term, and Lemma implies that:
capping threshold ensure 2r.a 2n/ holds
the propensity score Th (a|=)/ (((l))) u) is above the threshold vλ/n, CTPS will assign smaller weight to a sample with more inaccurate $(a)zz) æ>) to prevent its distortion from a large propensity score but an inaccurate logging probability.
the propensity score Th (a (a)a)) / ((((zz))) is below the threshold but not small enough to ac- tivate the second term, then the propensity score at the worse case (i.e., taking B B M(a (x)) 2 exp (~TU2,a) /7" as denominator) matters. If the propensity score at the worse case is under control, i.e., (ala)) B Vλ, larger Un,a implies small propensity score T To(a|z)/ /()l=)))))))),.).))))))))))))))) and UIPS tends to boost this safe sample with a higher 27 Otherwise 6".0 still decreases as Uz. becomes higher.
Uncertainty estimation. Now we describe how to calculate Uz i... the uncertainty of the es- timated ((a)x). In this work, we choose to estimate 3 (a|az) using neural network, due to its encouraging representation learning capacity. And various ways (Gal Ghahramani (2016); Xu et al. (202T) can be leveraged to perform the uncertainty estimation in neural network. For ex- ample, (Gal & Ghahramani 2016) proposed to estimate uncertainty using dropout; and (Xu et al.] 2021 provided a theoretical bound. Here adopt the result in (Xu et al. (202I) due to its com- putational efficiency and theoretical soundness. Following the proof of Theorem 4.4 in (Xu et al. 2021 given the logged dataset D, we can get with high probability Ey:
( an) for (2 a,,)) S S>y VV g(x,, aT, M, M a.)
where g(æn; an) is the gradient of fo(#n: an) regarding to its last layer, i.., g( an) Ve., fol Fz an), where 0n c O is the parameter of the last layer of fe(an, an). And MD g( an an)g(a, am) implying Uwn,
N
gg an) MD g(x,
3.2 Policy Improvement
After getting the optimal oz.a as in Theorem 2 the policy To (ala)) can be updated by the following REINFORCE gradient:
VOVUIPS Ez-
(ala))
3())=) oi Tz.a V log(=
(9)
UIPS then iterates policy evaluation and policy improvement for policy learning until converge. The
whole algorithm framework and important notations are summarized in Algorithm T and Table
Appendix A. respectively.
EMPIRICAL E EVALUATION
In this section, we evaluate UIPS on both synthetic datasets and three real-world datasets with unbi- ased data. We compare UIPS with the following baselines, which can be grouped into five categories:
Cross-Entropy (CE): A supervised learning method with the cross-entropy loss its objective, which is the commonly used learning approach for a model with softmax output. No off off-policy correction is performed in this method.
5

---

Under review as a conference paper at ICLR 2023
IPS-Cap (Chen et al. )2019)): The standard IPS based off-policy learning, which prunes propen-
[a(a| as the propensity score. Setting c to sity scores to control variance, i... taking min(c, 8(a//))
small value can reduce variance, but introduces bias.
MinVar & stableVar (Zhan et al. (202T), Shrinkage (Su et al. (2020)) This line of work im- proves off-policy evaluation estimators by reweighing each sample. For example, Min and sta-
Var reweigh each sample by En Rata' with hg. =a(a/=)² and ha v (a(aa)) respectively,
A(a|m)
(al=)
since they find that To / is directly related to variance. Su et al. (2020) proposes to
shrink the propensity score by multiplying a weight A/(A +
=o(al=) which is a special case of
p(a|z))
the proposed UIPS with U 0 and 7/1 1. All these work simply treats M(a(zz) as 8 (alar) and none of them consider the accuracy or uncertainty of (((aa)))
SNIPS (S (Swaminathan & Joachims 2015c)), BanditNet (Joachims et al. 2. POEM ( Swami-- nathan & Joachims (2015b)). POXM (Lopez et al. (202T), Adaptive (Liu et al. (2022)) This line of work aims for more stable and accurate policy learning. For example, SNIPS normalizes the
estimator by the sum of propensity scores in each batch. BanditNet extends SNIPS and leverages an additional Lagrangian term normalize the estimator by an approximated sum of propensity scores of all samples. POEM jointly optimizes the estimator and its variance. POXM controls estimation variance by pruning samples with small logging probabilities. Adaptive proposes new formulation to utilize negative samples.
UIPS-P and UIPS-O Two variants of our proposed UIPS with different ways of leveraging uncertainties. UIPS-P directly penalizes samples whose estimated logging probabilities are of high uncertainties, i.e., taking Φz.a 1.0/ exp(1U which follows previous work on offline reinforcement learning (Wu et al. [2021; An et al. (202I)). UIPS-O adversarially uses the worst propensity scores (T (a(z)) B. B for policy learning, i.e., Oz,a = 1.0/ exp(-rUp,.).
4.1 Synthetic Data
Data generation. Following previous work (Ma et al. (2020); Lopez et al. (2021)). we generate a synthetic dataset by a supervision-to-bandeee conversion on Wiki ViKi10-31K dataset (Bhatia et al. (2016). which is an extreme multi-label classification dataset. The Wiki10-31K K dataset contains approxi- mately 20K samples. Each sample is associated with a feature vector of 101,938 dimensions and a label vector ya of 31K classes with more than one positive class. Let ya.o denote the label of class a under and we take each class as an action. We adopt the Wikil0-31K dataset rather than ones in the UCI machine learning repository (Swaminathan & Joachims 2015a)), since it will be much harder with such a large action space.
We then split the dataset into train, validation, test sets with size I1K:3K:6K. The test set is from the official split. Since the original feature vector E is too sparse, for ease learning, we first embed to dimension d by W æt, and synthesize the ground-truth logging policy 3 (al (z) by:
exp( 03/T)
3 (alx) = E.. expl x /7/7))
(10)
where W and {0:} are pre-learned parameters by applying a logistic regression model on the train set, T is hyper-parameter that controls the skewness of logging distribution. A small value of T leads to a near-deterministic policy, while a larger 7 makes logging policy smoother. Due to space limit, more details on data generation and implementation can be found in Appendix A.2
Evaluation metrics. To evaluate the learned policy (a)æ)) we calculate Precision@ K (P@ K), Recall @K (R @ K) and NDCG@ K as in previous work (Lopez et al. (202T): Ma et al. (2020)
Higher P@ K, R R@K K and NDCG@N K imply a better policy.
Table shows the mean performance and standard deviations of all algorithms under 10 random seeds on three synthetic datasets generated under different t. Since the ground-truth logging policy is accessible on the synthetic datasets, we include a new baseline IPS-GT, which depicts the per- formance the IPS estimator can achieve, assuming the ground-trutt logging probabilities are known and sample size is sufficiently large. We calculate p-value under t-test between UIPS and the best baseline on each dataset to investigate the significance of improvement. First, we can observe that UIPS achieves similar and even better performance than IPS-GT when 7 0.5 and = 1, but

---

Under review as a conference paper at ICLR 2023
0.5
T=1
==2
Algorithm
PG15
RG5
N
5665
Rof5
N
P@5
Ror5
N
IPS-GT
0.559
0.352612
0,1565
100007 Eie-
0.353T 13K
CE
U,555366 -1
0.157322 -1
0.603755
0.55 1.5510
4,1561
0.599544
0.5586
0,1524 476
0.5874-3"
IPS-Cap
0.581515-77
0,1553
0,6031
0.5522 2m
1561
06016111
0.5409
1.1529
0.59m1
MinVar
0.5340
0,1509 1d:
0,5857
0.5282 22
0.1491
0,5791
0.5036.44
0.14
0.5543 L3
StableVar
0,4577m5-**
0.1310
0.5111
0.5373
0.1523
0.5866
0.5279
0.1492ea
0.578L
Strinkage
0.5526
0.1562 a7e
0.6024
0.5499
0. 15455
0.6040
0.5347:22
0,1513 2ne
0.5824 23e
SNIPS
0,2616
0.0749
0.3150
0.3538at<
0.0987:16
0,41
0.4379:5
0.1226
0.5177
BandiiNet
0.401 ==
0.1131
0.4830
0.3894444
D. 101055511
0.4122=
0.1153
0.4934
PORM
0154800
0.1539.a--4
0.550248e-4 22
D,1581
160000
0.5399
181
0.5893
POXM
0.4006
0.1130
0.3616444 0
D. 0.
0.522
0,3816
1069
0.4680
Adaptive
0.150-a--
04382
044734:1e
D. 1325-4tm
0.3326-3
0.3936-1
109744m
D.43689
UIPS-P
0.40195%-
0.113
0
0.3904
D. 10966eee
0.4749
0.4109=2
0. 0.419911
0.492269-
UIPS-O
04135
6. 1.4
D. 1096
0.7799.377
0.4519
126688877
0.5296635-2
UIPS
0.5608 L20-3
0.1589 Lia
1.5572..x"
0.1571 Li
0.6074..-7
0.5432 150
0.5946 1.1534
136-
P-value
Ze
20
Table I: Experimental results on synthetic datascts. The best and second best results are highlighted with bold and underline respectively. The p-value under the t-test between UIPS and the best bascline on cach datasct is also provided.
Low Frequent Action Related Samples High Uncertaiuty)
High Frequent Action Related Samples (Low Uncertainty)
Algorithm CE
5QR5(RI)
SDCG
R@5(RT)
NDCG
0.5186
0,1228
0,5521
0.5931
0.1921
0.6575
IPS-Cap
0.5170(-0.319)
0.1218(-0811))
0.553X+0.3220
0.5996(+11188)
O. 0.1935(-0.73%)
0.6647(+1.. 10%)
Shrinkage
0.5145(-0.79%)
0.1212(-1.30%)
0.551%(00.04%)
0.5982(-0 86%)
0.6628(+0.819 (.1931()
%c
UIPS
0.5276(+1.749))
0.125)(+1.79%)
0..82)(+1.855)
0.6055(+2.099)
(.1961(-2.08%)
0.6715(+2.13%)
Table 2: Performance under different uncertainties.
NDCG@5
0.6033
0.603
0:6033
0.60:38
0.6039
0.610
Algorithm
MSE
10
0.0024
0.0048
0.6058
0. 0.5065
0.6070
0.008
IPS-CaP
0.4953
min Var
0.8928
20
0.6002
0.6057
0.6083
0.6098
0.6112
stable Var
0.8112
0.604
25
0.6004
0.606.
0.6099
0.6106
0.6113
0.602
Shrinage 0.5125
UIPS
0.4516
0.5891
0.6054
0.5087
D. 9.8094
0.6107
0.600
0.5960
0.6012
0.6043
0.6008
0.6068
0.598
Table 4: MSE of different off-policy evaluation estimators.
i
10
15
20
-0.99
A
Table 3: Effect of and y on NDCG@5.
performs worse than IPS-GT on the dataset with = 2. Although IPS-GT can access the ground- truth logging probabilities, it still suffers from high variance caused by samples with small logging probabilities, which is the main cause of its worse performance when = 0.5 and T 1. When the ground-truth logging policy is smoother (e.g.. = 2), the variance of the IPS estimator becomes much smaller, and off-policy correction with the ground-truth logging probabilities, rather than the estimated ones, leads to better model performance. We can then observe that as T increases, i.e., the probability of selecting positive actions decreases, the performance of most algorithms drop, including CF. IPS-Cap, UIPS, Shrinkage, POEM, Adaptive, etc. However, UIPS still achieves the best performance on all three datasets under all three metrics. And as T decreases, the improvement of UIPS becomes larger and more significant. SNIPS, BanditNet, POXM are more robust to small logging probabilities of positive actions. UIPS consistently outperforms Shrinkage (a special case of UIPS with uncertainties always being zero) on all three datasets, demonstrating the benefits of considering the estimation uncertainty. Finally, regardless of the scale of propensity scores, blindly reweighing through uncertainties also leads to poor performance, as shown by UIPS-P and UIPS-O.
Performance under different uncertainty levels. As shown in Figure I low-frequency actions in the logged dataset suffer higher uncertainties in their propensity estimation. Thus, we divide the test set into two subsets according to the average frequency of associated actions, where the uncertainty in the subset associated with low-frequency actions is average 9% higher than that in the subset associated with nigh-frequency actions. Table 2 shows the results on these two subsets when 7 = 0.5. We only report the results of the best three baselines due to space limit. One can clearly observe that only UIPS performed better than CE on the test set associated with low-frequency actions, implying the advantage of UIPS in dealing with the inaccurately estimated logging probabilities.
7

---

Under review as a conference paper at ICLR 2023
Yahoo
Cont
KualRee
Algorithm
PGj5
R65
NDCGBS
Por5
R65
NDCG@5
PGP30
R4550
NDCG 50
CE
0.281958-*
0.75
0.607377
0.27995te
0.461821e*
0.4529-76"*
0.802=20
0.0240=56
0.8e
IPS-Cap
0.2751=8
0.7419-5-
0.592847e
0.2758
0.4582-70*
0.4
0.8750
0.028-767
0.8788
MinVar
0.284326
0.7685=2-
0616841
0.2815
0.4668-
0.4414
0.8827=1"
0.0240-5~
0
StableVar
0.2787+a
0.7499-7
0,5919476
0.2840=3:
0,4662-1
0,4393=77
0.8524
0.0231
0.8570=
Shrinkage
0.2843+3
0.7654
0.624477-7
0.2790=te
0.4636=4
0.4
0.8744
0.0238
0.8771 =te
SNIPS
0.2222
0.5828
0.4357
a.2643
0.4287
0,4009
0,8411
0.0228
0.843] a
BunditNet
0.2413a5
0.642==a-
0.4988
0.2781=5
0.4527
0.8758=5e
0.0239
0.88 1
POEM
0.2732
0.7357mie
0.5880
0.2791
0.4566
0.4375
0,7785
0.0310
0.7779
POXM
0.2250
0
0.4512+2-7
0.2663=56
0.430
0.4006=16-3
0.8962=1=
0.0245-4-7
0.9041=1e"
Adaptive
D.2762
0,5919
0.2830=5
0.4634
0.4217
0.8375
0.0227=4
0.8460
U
0.182 %
0.4500 3e
0.33000 te
D.2685 To
044164 SM
0.4087 1.7x
0.8638 M
0.0235 36
0.8685 Te
UPPS-O
1947
0.4959-177
3600
0,2657
0,300-
0.4146=8-7
0.8681
0.0235-44
UIP
0.2868
0.7742
0.6274.5.
0.2877 1
0.4757 1e
0.4576 s..""
0.9120 16
0.0250 54
0.9174 Tx
P-value
4-
3e
22-
6e
Table 5: Experimental results on real-world datasets. The hest and second best results are highlighted with bold and underline respectively. The p-value under the t-test between UIPS and the best baseline on each dataset is also provided.
Ablation Study. In this experiment, we aim to answer two questions: (1) Can (T)))( in Eq. (D lead to more accurate off-policy evaluation? (2) How will UIPS perform with different hyper- parameters. Due to space limit, we report results on synthetic dataset with T = 0.5.
To answer the first question, we evaluate the following e-greddy policy: <<da=) I-c Ilaa E
[M]]
M2) + e/I-A,, where M2 contains all positive actions associated with feature vector æ. Then for each æ in the test set, we sample IK data points in a similar way discussed previously to calculate the value of estimators. T
the MSE of the estimators to ground-truth policy value under off-policy evaluation estimator, i.e.,
20 different random seeds. We only compared with baselines IPS-Cap, Min Stable Var and Shrinkage. One can observe from Table 4 that UIPS does lead to the smallest MSE, implying the most accurate off-policy evaluation.
For the second question, 7 and vf/AA /A are the two most important hyperparameters as discussed in Appendix[A.21 Thus we fix 1/, 1/2, and vary À and y to track the performance of UIPS Recall that a larger 7 implies higher chance the derived interval contains 3 (alæz), while v\/m) is closely related to how UIPS works as discussed in "Insights on & " in Section 3.1. Figure 3 reports NDCG@5 under diferent 7 and X. Results P@5 and R@5 can be found in Appendix A.2. We can observe that to make UIPS perform, Bæ, needs to be of high confidence, c.g., 7 25
performed the best when T = 0.5. Moreover, the threshold vX/mn cannot be too small or too large.
4.2 Real-Worle Data
Off-policy learning has its utility in recommendation scenarios (Chen al. (2019) Ma al. (020 where context vector æ denotes the state of a user and each candidate item is taken as an action. To further demonstrate the efficiency of UIPS in real-world scenarios. we evaluate it on three recom- mendation datasets with unbiased testing data: (1) Yahoo!R3 (2)Coa Kuaiuae (Gao et al. (2022), ) from music, fashion and micro-video recommendation scenario respectively. All these datasets contain an unbiased test set collected from a randomized controlled trial where items are randomly selected. The statistics of the three datasets and implementation details, e.g., model archi- tectures and dataset splits, can be found in Appendix A.2.2
We still adopt P@KK R@K and NDCG@ K as our evaluation metrics. Following (Ding et al. (0022) ). take K = 5 on Yahoo!R3 and Coat datasets, and K 50 on KuaiRec dataset. The p-value un- der the t-test between UIPS and the best baseline on each dataset is also reported to investigate the significance of the improvements. We can first observe that on all three datasets, the proposed UIPS achieves the highest precision, recall and NDCG. IPS-Cap cannot outperform CE due to the inac- euracy of the estimated logging probabilities. BanditNet, POEM and POXM tend to perform better with a larger action space, while Min Stable and Shrinkage as well as Adaptive are more suitable for scenarios with small action size. UIPS still outperforms Shrinkage. highlighting the im- portance of modeling uncertainty in the estimated logging policy. However, reweighing based solely on uncertainties, ignoring the corresponding propensity scores, will also lead to poor performance, as shown by UIPS-P and UIPS-O
https:// sandbox. wwwcc.acc....oo com
https:: www. corne11 schh///chaattt//marr
8

---

Under review as a conference paper at ICLR 2023
5 RELATED WORK
This work is the first of its kind to take into consideration the uncertainty of the estimated logging policy for improved policy learning. The following two lines of work are related to this paper.
Off-policy learning. In many real-world applications, such as search engines, recommender sys- tems, etc., interactive online model update is expensive and risky QJiang & Li (2016)) Off-policcy learning has therefore attracted increasing interest, since it can leverage the already logged feedback data (Agarwal et al. (2019); Chen et al. 2019); Liu et al. (2022)) The main challenge off-policy learning is how to address the mismatch between the logging policy and the learning policy. One line of work (Achiam et al. 2017); |Schulman et al. (2015) circumvents this by constraining the learn- ing policy not too far from the logging policy. However, such constraint is too restrictive thus not applicable in some scenarios such as recommender systems where user behaviors and items change rapidly. Another more common and widely-applic approach is to leverage Inverse Propensity Score (IPS) method to correct the discrepancy between two policies. And various methods are proposed for stabilized learning (Swaminathan & Joachims (2015c b)) and variance control (Lopez et al. (20TT] Liu et al. (2022)) on top of TPS. However, all these work directly use the estimated logging policy for off-policy correction, leading sub-optimal performance
L
shown in our experiments. Some other work further extend IPS-based off-policy learning for more complex problems. such as slate recommendation (Swaminathan al. (2017). two-stage recommender systems (Ma et al. )2020), etc. But they still fail to realize the effect of accuracy of the estimated logging policy. A recent work (Ding et al. (2022)) on causal recommendation also argues that propensity scores may not be correct due to unobserved confounders. However, they assume the effect of unobserved con- founder for any sample can be bounded by a pre-defined hyper-parameter, and adversarially search for the worst-case propensity to update model parameters. Adapting off- off-policy learning, it is special case of our UIPS-O O variant with uncertainty as a pre-defined constant.
Off-policy learning can be directly built off-policy evaluation. In this line of research, several work (Su et al.] (2020): Zhan et al. (202T) also propose to control the high variance of learning caused by small logging probabilities by instance reweighing. However, they directly take the timated logging policy as true logging policy for correction, thus worse than UIPS as shown in experiments. A recent work (Saito & Joachims (2022)) assumes additional structure in action space and proposes the marginalized IPS. Instead, our work considers the uncertainty when estimating the logging policy and thus does not add new assumptions about the problem space.
Uncertainty- aware Learning. Estimation uncertainty has been extensively used for making trade- off's between exploration and exploitation in online learning (Xu et al. (202T): Zhou al. (2020); Abbasi Yadkori et al. (201T)) Recently, several work on offline reinforcement learning Ww et al. (2021); et al. (2021)) D; Bai et al. (2022)) penalize the value function of out-o--distribution states and actions by directly subtracting uncertainty to tackle the extrapolating error. However, blindly penalizing samples of high uncertainty (i.e., UTPS-P) is problematic, as shown our experiments. Proper correction depends on both uncertainty in logging policy estimation and the actual value of estimated logging probabilitics.
6 CONCLUSION
In this paper, we propose a novel Uncertainty-awarc Inverse Propensity Score estimator (UIPS) to explicitly model the uncertainty about the estimated logging policy for improved off-policy learning. UIPS weighs each logged instance to approach the ground-truth estimator and closed-form solution of the optimal weight is derived by minimizing the upper bound of the mean squared error (MSE). An improved policy can be obtained by optimizing the resulting estimator. Extensive experiments on synthetic datasets and three real-world datasets demonstrate the efficiency of UIPS
As demonstrated in this work, explicitly modeling the uncertainty of the estimated logging policy is crucial for effective off-policy learning; but the best use of this uncertainty is not to simply down- weigh or drop instances with uncertain estimations, but to balance it with the actually estimated logging probabilities in a per-instance basis. As our future work, it is promising to investigate how UIPS can be extended to value- based learning methods, e.g., critircrities. And on the other hand, it is also important to analyze how tight our upper bound analysis of MSE is; and possible, find new ways to tighten it for improvements.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems, pp. 2312-2320, 2011.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Aman Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Li, Mare Najork, and Thorsten Joachims. Estimating position bias without intrusive interventions. In Proceedings of the Twelfth ACM International Conference Web Search and Data Mining, pp.
2019.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline re- inforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7444 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhihong Deng, Animesh Garg. Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.11566 2022.
K. Bhatia, K. Dahiya, H. Jain, P. Kar, A. Mittal, Y. Prabhu, and M. Varma. The extreme classi- fication repository: Multi-label datasets and code, 2016. URL
org/
dbwnlooads/KC///MLeepsssss
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. Top-k off-policy correction for a reinforce recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 456 2019.
Ruidi Chen, Ioannis Ch Paschalidis, et al. Distributionally robust learning, Foundations and Trends@ in Optimization, 4(1-2):1-243, 1-243, 2020.
Sihao Ding, Peng Wu, Fuli Feng, Yitong Wang, Xiangnan He, Yong Liao, and Yongdong Zhang. Addressing unmeasured confounder for recommendation with sensitivity analysis. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 305-315, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050~1059.
PMLR, 2016.
Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Ji- axin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recommender systems. In Proceedings of the 31st ACM International Conference on Informa- fion and Knowledge Management, CIKM '22, 2022. doi: 10.1145/3511808 188355511100..355220 URL https://dol: wttpg//...1///311800333 311./551111888.8.5222221 3557220
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfim factorization- machine based neural network for etr prediction. arXiv preprint arXiv: 0703.0 2017.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference Machine Learning, pp. 652 661.661.66 PMLR, 2016.
Thorsten Joachims, Adith Swaminathan, and Maarten De Rijke. Deep learning with logged bandit feedback. In International Conference on Learning Representations, 2018.
Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning, pp. 1-9. PMLR, 2013.
Yaxu Liu, Jui-Nan Yen, Bowen Yuan, Rundong Shi, Peng Yan, and Chih-Jen Lin. Practical coun- terfactual policy learning for top-k recommendations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1141-1151, 2022.
Romain Lopez, Inderjit S Dhillon, and Michael I Jordan. Learning from extreme bandit feedback. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 8732-8740, 2021.
10

---

Under review as a conference paper at ICLR 2023
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang. Minmin Chen, Jiaxi Tang, Lichan Hong, and Ed H Chi. Off-policy learning in two-stage recommender systems. In Proceedings of The Web Conference 2020, pp. 463-473, 2020.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Yuta Saito and Thorsten Joachims. Off-policy evaluation for large action spaces via embeddings. arXiv preprint arXiv:2202.06317, 2022.
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning, pp. 1670-1679. PMLR, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiesser Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(75877:4844449 2016.
Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit explo- ration data. Advances in neural information processing systems, 23, 2010.
Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning, pp. 9167-9176.
PMLR, 2020.
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research, 16(1):1731-1755, 2015a.
Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference Machine Learning, pp. 814-823, PMLR,, 2015b.
Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learn- ing. advances in neural information processing systems. 28. 2015c.
Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. Advances in Neural Information Processing Systems, 30, 2017.
Sebastian Thrun and Michael L Littman. Reinforcement learning: an introduction. Al Magazine, 21 (1):103-103, 2000.
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-22 2266, 1992.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint 21550:2005.08140, 2021.
Pan Xu, Zheng Wen, Handong Zhao, and Quanquan Gu. Neural contextual bandits with deep representation and shallow exploration. In International Conference on Learning Representations, 2021.
Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey. Off evaluation via adap- tive weighting with data from contextual bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pp. 2125-2135, 2021.

---

Under review as a conference paper at ICLR 2023
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-baseed exploration. In International Conference on Machine Learning, pp. 14992-1150,, PMLR, 2020.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1059-1068, 2018
A APPENDIX
A.1 NOTATIONS AND ALGORITHM FRAMEWORK.
For ease of reading, we list important notations in Table and summarize the main framework of the proposed UIPS in Algorithm D
Notation
Description
context space
A
action set
x E Rd
context vector
a
action
Tæ.o
reward
7(a (
targeted policy to evaluate the unknown ground-truth logging policy
3"(( (D)
B(a a))
the estimated logging policy
V(T)
value function
c n, On Tw,,
12 N
logged dataset containing N samples the optimal uncertainty-awe aware weight the unknown ground- ground-truth -ruth function
$2.0
fe (x)
that generates /3* (a (z)) Σ., exp (a.af)
exp( (a,a))
fo(ar,a)
the estimate of fo- (ax, a) that generates (aaxr) confidence interval of (()
Br.a
Unaa
uncertainty defined as fo- a) folæ, a) S0mma
g(In, an.
gradient of fe(æ, a regarding to the last layer.
Table Notations
Computation Cost. The additional computation cost of UIPS over IPS comes from two parts:
Pre-calculating uncertainties (line 1-5 in Algorithm 1): This part calculates uncertainty of the logging probability for each (s, a) pair, and "it only needs to be executed once". The computational cost of this step O(Nd² + dd), where O(Nd)) is for calculating uncertainties in each (s, a) pair and (dd) is for matrix inverse.
Calculating o during training (line 8 in Algorithm 1): It only takes O(1) time, the same computational cost calculating IPS score.
Note that calculating logging probability for each sample, which is essential for both UIPS and IPS, takes ((N])A)) time. Since the dimension d is usually much less than action size [A] and samples size N, UIPS does not introduce significant computational overhead compared the original IPS solution.
A.2 EXPERIMENTS DETAILS
A.2.1 SYNTHETIC DATA
Data generation. Given the ground-truth logging policy 3 (a(x), we generate the logged dataset as follows. For each sample in train set, we first get the embedded context vector form its original
12

---

Under review as a conference paper at ICLR 2023
Algorithm 1: UIPS
Input: The logged dataset == (Inn On, Tan.on ) E [N]], the estimated logging policy
model ((((x) Ed exp(_o((m,aa))'
expl fol (x(a)))) latent dimension d.
Init: MD Iaxd
// calculate MD for uncertainty calculation.
for n 1,2, N do
MD MD anNoooonn
Mnn (m)
for n = 1,2, N do L UR U,,.a, = Vofo(Tn: 0n) ) M Vofo(In,an))
// Main part of UIPS while not converge do for n = 1,2,..., do
Calculating aa..a. in Theorem 2 Calculating gradients as in Equation (9) and updating To <o(a)z).
Output: The learnt policy N (a/2z)
feature vector T. We then sample action a according 8 [z), and obtain the reward Tw.a Yæ.a, resulting a bandit feedback (æ, Tw a). where Va.a is the label of class under the original feature vector Ø. We repeat above process N times collect the logged dataset. In our experiments, we take d = 64, N = 100.
Implementation Details. We model the logging policy as in Equation ( with fo(æ, a) x O where (6.) } are parameters to learn. To train the logging policy, we take all samples in the logged dataset D as positive instances, and randomly sample non-selected actions as negative instances as in (Chen al. (2019)). We use grid search to select the hyperparameters based on the model's performance on validation dataset: the learning rate was searched {le-5, le 1e-3, le le=2]; Y, were searched in (0.5, 0.1, 1, 2.5, 10, 15, 20, 25. 30, 40, 50}. And 172 searched in {1, 10, 100, 1000}. For baseline algorithms, we perform a similar grid search as mentioned above, and the search range follows the original papers.
Hyperparameter tuning. Although UIPS has four hyperparameters (A, 7, 7h and 77),, only needs to carefully finetune two of them, i.e., 7 and v/ />, to obtain good performance of UIPS. This is because:
Ablation Study:
122 acts like a capping threshold to ensure Φz.a S 272 holds even with small propensity scores. Hence, it should be set to a large value (e.g., 100).
The key component (i.e., the first term) of Φ can rewritten in the following way. While all (a, a) pairs will be multiplied by oi.a. 7h in the numerator will not affect final performance too much, and the key is to find good value of ni / to balance the two terms in the denominator:
/// T (alar)²
m/ exp (- +
3(a[z)² exp (~>Ux,
Thus with and 77 fixed, effect of hyperparameter and À on precision and recall can be found in Figure 2a and Figure Figure22jrespectinn respecctiivl..
A.2.2 REAL-WORLD DATA
Statistics of data. The statistics of three real-world recommendation datasets with unbiased data can be found in Table
All these datasets contain a set of biased data collected from users' interactions the platform, and a set unbiased data collected from a randomized controlled trial where items are randomly selected. As in (Ding al. (2022)), each dataset, the biased data is used for training, and the
13

---

Under review as a conference paper at ICLR 2023
P@5
R@5
0.1561
0.5525
0.6947
0.5546
6.5546
0.5053
10
0.1561
0.1563
0.1563
0.1963
0.1964
0.5
20
0.5505
0.5352
0.5575
0.5387
0.539
20
0.1556
0.1569
0.1576
0.1579
0.3583
0.157
0.554
V
25
0.5495
0.5558
0.5589
0.5500
0.5600
0.552
25
0.1554
0.1573
0.1583
0.1586
0.1509
0.156
30
0.5481
0.5544
0.5590
0.5591
0.5602
0.550
0.1550
0.1569
0.1580
0.1584
0.1580
-..448
-.155
40
0.5541
0.5554
0.5364
40
0.1547
0.1558
0.1567
0.15
0.1574
0.546
10
15
20
1
10
15
20
Precision85
(b)
Figure 2: Effect of and 7 on Precision @5 and Recall 5.
Dataset
#User
#Item
#Biased Data
#Unbiased Data
Yahoo
15,400
1,000
311,704
54,000
Coat
290
300
6,960
4,640
KuaiRec
7,176
10,729
12,530,806
4,676,570
Table 7: The statistics of three real-world datasets.
unbiased data is for testing, with small part of unbiased data split for validation purpose (5% on Yahoo and Coat, and 15% on KuaiRec). We take the reward as if (1) the rating is larger than 3 in Yahoo!R3 and Coat datasets; (2) the user watched more than 70% of the video in KuaiRec. Otherwise, the reward is labeled as
Implementation Details. We adopt two-tower neural network architecture to implement both the logging and learning policy, as shown in Figure 3 For the learning policy, the user representation and the item representation are first modelled through separate neural networks (i.e., the user tower and the item tower), and then their element-by -yynentt---elementt product vector is projected to predict the user's preference for the item. We then re-use the user state generated from the user tower of the learning policy, and model the logging policy with another separate item tower, following (Chen) al.) (2019). We also block gradients prevent the logging policy interfering the user state of the learning policy. In each learning epoch, we will first estimate the logging policy, and then take estimated logging probabilities as well as their uncertainties optimize learning policy. All hyperparameters are searched in similar way described in Section 4.1
((ulx)
a xx)
MLP
MLP
goalert
item cmb
neural net
item tower
neural net
item
item festure
user feature
Figure 3: Model architecture of the logging and the learning policy in real-world datasets
14

---

Under review as a conference paper at ICLR 2023
A.3 EXPERIMENTS ON THE DOUBLY ROBUST ESTIMATORS.
The doubly robust (DR) estimator (Jiang & Li (016)), which is a hybrid of direct method (DM) estimator and inverse propensity score (IPS) estimator, also widely used for off- aff-policy evaluation. More specifically, let vj X' x A -> R be the imputation model in DM that estimates the reward of action under context vector and ((aaa) be the estimated logging policy in the IPS estimator. The DR estimator evaluates the policy π based on the logged dataset D == {(2 Ore I Tan.an) E [M]] by:
(DR(TT) = (TM + 1 E
x(On (Fænian i) En an)) B(anlan))
(11)
where (T) is the DM estimator:
(TM(()= En EAEA x (allaa) a).
(12)
Again assuming the policy Taax)) is parameterized by 0, the REINFORCE gradient of (TT) with respect to & can be readily derived as follows:
V VoVDn(FO)
N E 1 N E
Vo (EnEA To(aa a) V log( log(TT((a
mtann (mn)) (Ten.an Mxn an V log (an (Zn))
(13)
((anllen)
The imputation model f0 a) is pre-trained following previous work (Liu et al. (022)) with the same model architecture the logging policy model. Besides the standard DR estimator, also adapt UIPS and the best two baselines on off-policy evalaution estimator (i.e., Min Var and Shrink- age) to doubly robust setting using the same imputation model.
Table 8 and Table 9 show the results on the synthetic datasets and three real-world woold datasets respec- tively. For ease of comparison, we also include the experimental results of IPS-Cap and UIPS on each dataset in two tables. Two p-values are also provided: (1) P-value der the t-test between UIPSDR and the best DR baseline on each dataset; (2) P-value(UIPPS) The p-value under the t-test between UIPS and the best DR baseline each dataset. From Table and T we can first observe that DR cannot consistently outperform IPS-Cap: outperforms IPS-Cap on the Coat and KuaiRec dataset, while achieving much worse performance on the syn- thetic datasets and Yahoo dataset. This is because the imputation model also plays an important role in gradient calculation as shown in Equation [3]. its accuracy greatly affects policy learn- ing. When the imputation model is sufficiently accurate, for example, on the Coat dataset with only 300 actions, incorporating the DM estimator not only leads to better performance of DR over IPS, but also improved performance of UIPSDR over UIPS. And in particular, in this situation UIPSDR performs better than DR with the gain being statistically significant. When the imputation model is not accurate enough, for example, on the KuaiRec dataset with large action space but sparse reward feedback, DR is still worse than UIPS, and UIPSDR also performs worse than UIPS due to the distortion of the imputation model.
The p-value un-
A.4 THEORETICAL PROOF.
Proof of Proposition] 1
Proof. With the linearity of expectation, we have ED (VEIP((TO)]
Eg-
[x(aa)) ((aa=) rw.a
thus:
Bias (VBIPS(TO)) = Ep
VIIPS(TT)
- V (TT)
= E
[a(ak))
((a m) Taaa
Egg
(a pelaaee [e)
(a(p))
Tana
Es--
To(alz) [-]aa Ta
(a(xx
1) [(a(o)]
Era
Tw.a
(a(m)
[(a [x)
1)]
(14)
15

---

Under review as a conference paper at ICLR 2023
0.5
Algorithm
Pars
NDCGGS
Pord
RG5
NDCCG
Pars
R.05
NDCGG0
1P5--Cap
0.551522
0,1553 28c
0.003T24
0.5526522-1
0,150T=9:"
0.60
0,500
0.1529214
0.5901=2
UIPS
0.899633
0,15K3-1,"
0.6095-1-3
0.557249-
6.1571
0.607-6
0.5452e9-
0,153464
0.5916-2
DR
0 3846663
0. 1082
0,684414
0.36575007
0.1017
D. 0.3500
0.0955777
0.4470-5
MinVaDDR
0.322244
0.1062a%
0
0.3234
0.0910
0.4059-4-
StrinkageDR
04159-2-75
0.1161-7e-1
0.4969-14-4
D.3944aa-
0.1101
0.4777-0744
D 4080----
0.1135-77
0.4901 -bind
UIPSDR
0.4278:22
0.1200
0.306922
0.4008:2-
0.1126
0.4847=2
0.4144
0.1162:
0.4972:2
P-aaluet UIPSD)
26
30
U.VSllee(UPPS)
6r
4x
4c-12
86"
8
2n"
Table 8: Experiment results on synthetic datasets. The best and second best results are highlighted with bold and underline respectively. Two p- p-values are calculated: (1) P-Value(UPPSDR): The p-value under the t-test
between UTPSDR and the best DR baseline on each dataset; (2) P-value(UIPSS: The p-value under the t-test hetween UTPS and the best DR baseline on each dataset.
Yahoo
Coal
KuaiRee
Alganilius
P85
R65
NDCG65
F85
RG5
NDCO6S
P8:50
NDCG
IPS-Cap
0.2751
0.7419
0.3928477
0.275Made
0.552-7- 7+
0.4399M
0.8700
0.028
0.8788-5e
UIPS
08688122
0.7742
0.6274.5
0,2877
0.4757
0.487615
0.120.1
0.0250
0.9174.7>
DR
0:267
0.7174
0.5636a1
02284421
0,4760
0.4541ate
0.8794
0.0240:11
0.8824.2>
MinVarDR
0:222255-2
0,5989 1
0.4625 (1
0,27041 4m
0,4334
1.4137
0.8600 70
0.235 A0
0.8657 1T
ShrinkageDR
0.5713ate
0.2895
0,4749
0.4526at
0.877822
0.0239
0.8800
UIPSDR
0.2721 1
0.7294 4-
0.5750
0.29461-4
0.4854
0.4647
0.8899
0.0242
0.8996 11x
Povalue (IPSDR)
Ie
be
2r
P-yzloe(UIPS)
1e
G--13
3e"
Be-1
1e-1
2--4
2e-1
Table 9: Experimental results on real-world unbiased datasets. The best and second best results are highlighted with bold and underline respectively. Two p-values are calculated: (1) P- P-value(UIPSDDR The p-value under
the t-test between UIPSDR and the best DR baseline on each dataset; (2) E-value(UIPS) The p-value under the 1-test between UIPS and the best DR baseline on each dataset.
For variance, since samples are independently sampled from logging policy, thus
VarD (VeIPS(TO)) = N
To(alz>
Varg.
Tw,a
B(al20)
By re-scaling, we get:
N. VarD (VeIPs(To)) = Varg-
[a(a|m) B(a|=) Tata
Eg.
solalan?
a.a
Egg-
[a(aaz)
A(ape)
Tanaa
)²
E
[a(a)))) (-(a(
8" (ala)) 2
Malaz) 2..
8" (alxx)
p(a
ra.a
2
= Varza
(a)(a))))
rw.a) + Er
[a(a
- 11
(a|m))
22.0
(15)
B(a a²)
(-(aa=)
B[aaa)}
Then we complete the proof.
Proof of Theorem 1
Proof. We can get:
MSE (Vonps(TO)) EE (Vores(To)-V(oo) (ooo))-((TO)) 27
Ep (VIPS(TT) ((TT)) )
+ Varn (VoIPS(TT) - V(TT))
Ep
(UIPS(TT) V(TT)) 2
+ Varp (VoIPS(TO))
-Bias(Vurs(TTo)) + Var Var(Vuurs(To)).
16

---

Under review as a conference paper at ICLR 2023
We first bound the bias term:
Bias Bias(VUiPs (VUIPS(TT)) En
(To V(TT)
Eg.
[a(a))) A(alæ) oz T220
V (TT))
Eg.
(T(aa)) [(alz)
Pa,aTa
na (al x+)
3- (-(a(zr) Faza
Eg-
Tana (a)))
mo(ala)
(alm)) B(alæ)
Ozea - 11]
E
(a(al) (
V Eg
(
((((al)) 2ze - K(aa))
1)²
27
(2)
Equality (1) follows the linearity of expectation. Inequality (2) due the Cauchy-Schwarz in- equality. We then bound the variance term:
Var ((VVIIS((TT = N
1 Varg.
(a(all) ox. A(a m))
(1)
N Eg
Ka(a w² B(alm)) aa 2a
Eggt
xa(ale) B(alar)
Du aa manaa
)>
(16)
<
N E,5-
na(alw)²
s Es-
ma (a(a) Bla x²)2
12
(17)
Bla æ)2
2.0 a,a
Combining the bound of bias and variance, we can complete the proof.
Proof of Theorem 2
Proof. We first define several notations:
T( Pr,a) ABB
(
Aes Aee ((aale) Pn - -1) + Eg
27
Te(al=)²
B(aaa)² 0
((n,a) = max EB T(Da,a Sa,a) denotes the maximum value of inner problem.
T mins... ((ox,a) = min CB T(ox Px denote the optimal min- value. And &z,a arg min T(mma).
B Zexpl- (alxx) and B Zexpl7Ue.l B(alx))
We first find the maximum value of inner problem, i.e., T(ba,) for any fixed dz And there are three cases shown in Figure
8ia
B1a
Bode
82.
8..
F2
9xa
Case
Case
Case
Figure Three cases for maximizing inner problem.
Case 1: When B(aaa)) B 2.0' T()),) achieves the maximum value at Bz B In other Omn,
words, T(Obaa) = T(do,a Brad) when Du. S B(alz)
B
17

---

Under review as a conference paper at ICLR 2023
B[amm]
B a,a ie., Z' "oxplsttee S 22 S
Z' ('exp(tUoo) then (
Case II: When B
$2.0
2
2
will be the maximum between Tonn, Bz, and T(mma Bzaa).
More specifically, when A(alm))
B Baa th Baa i.e., Da,a 2
28(a(m)
(()m,)) = T(Da,a a
txa 2B(a)w))
2
B
Otherwise when @ar.
(dza)) = (oz Bz,a).
Bž. Ba.
Case III: When Px.a
M(lmm) implying ((al))
S B B T(ow,o) T(oa, B²aa).
Ba.a
$0,0
Overall, we get that:
T(o Ba
02,0 E (-00,
28(a)a)
T()2,))
B Ba
(18)
T( B Uxa E
28(a(z)) 00)
B
Next try to find the minimum value of T( We first observe that without considering constraint on Pz,as when
A
0,0
Re (a)a))
Z.
B(alar)
T(dm,a Bi,a) achieves the global minimum value. However, 2
va. E 22(aaa)) B B 00), the minimum value of T T( a achieves 28(a(a) which implies when
B(alx) B
B B..
On the other hand, without considering any constraint Φz,a global minimum value of T( B achieves at:
A
(19)
2 expl Up.a.)) Rala/mn)2 (aaw))2 exp mlmma
2.
28(((())
28(a)m) B Bz.
Thus if 6= S B Ba,
Dz.a a
Otherwise Pn,a
Overall,
6 m.a
min
A
2 2 2xxp(=Un.a) expl Unaa)
(20)
2expl ) Usan))
A(a)m)a
expp o
We use 7/1 and 1/2 to represent S in the two terms respectively. We can get
Se. min exp(-NUeaatt ((ote) (Up.))
2na ( exp(>Ux.. texpl -000a)
(21)
where 1/2
(exp(->Umax) maxx) since 2 exp(->UUms
Z
En exp( (a'[x)) S 2 exxp(U)max). Usually we set ?/ S 12- We introduce two parameters since the scale of 171 is closely related to the scale of X, while the scale of m2 is independent.
Then we complete the proof.
Lemma 1. Assume ?11 S 72. then with fixed (a(a)) and B(aaæ), and ax.a
X11-m) nf
exp(- (o0(-UUUn..l we have the following observations:
2mm
If (a(alz) 3(a(x)
S Og V 2/// [exp(>Um,a) + exp( exp((>Un.a)} Otherwise dta
A/ exp (~>Ua,) + 32(a))) wxpppxpppU..... 1] In other words, &'.a S 2np always holds.
18

---

Under review as a conference paper at ICLR 2023
2 LA then &2,0 will always decrease as Un.a increases. B(alz) nn [o(a)))
If Oxra
I nelalæ) p(a
vA exp(~nWa.a) larger Uwaa brings larger 8z Otherwise o still m
decreases as Uz, increases.
Proof. For the first observation, deriving
A
21/2
exp (- (xpll))
m))
exp (yU2,a) + exp (~VU2,a)
m
we can get the result.
For the second and third observation, since 7 72. then aza can have:
vA m
Let L(u)
22m
(a)=) À exp(-Nu) + M(aaz)) exp((>u) -yu)"
m
exp( (u)) + 7T x:)²
Vul(u) = - m
exp(yu)
8(a|x))
By letting V.L(u) 2 we can get a log v
This implies when Una 2 Pnna will decrease as Un. increases. Otherwise as Un. increases, On also log ( increases.
a ))
Specially, when [a(am)) M(aa=) m always decrease as Un. increases in this case.
v log (il(al)) (ala))
< 0. Given Una 2 0, this implies that o will
Otherwise, when Un
Tala/æ] B(a)æ)
vI
exp(=>Um.a) with
To (a
V Un,a S log
1
VA (a(a))
((aaa)) exp(->Um.o)
larger Un.a implies larger dz.a. Otherwise &z still decreases as Un increases. This completes the proof.
19


Reviews:
[{"review_id": "dN9ZDELHG_", "reviewer": "Reviewer_DgbC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an extension to the classical inverse propensity score estimator for off-policy learning.\nThe authors propose considering the uncertainty in estimation to the logging policy when this policy is unknown, which is typically the case in many practical scenarios. They do so by introducing a new learnable parameter ( phi ), that increases for samples with low uncertainty, and decreases for samples with high uncertainty.\nThrough various datasets, the authors show that their proposed technique is able to outperform several standard methods for off-policy evaluation.\n", "summary_of_the_review": "A well-written paper that highlights a practical issue of estimating logging policies when these are not available.\nThis paper tackles the problem by taking into account uncertainty in estimating the logging policy, and making corrections to the classical propensity scores based on uncertainty.\nThroughout several experiments on a number of datasets, the authors show that the propose technique improves results over other commonly used techniques.", "strength_and_weaknesses": "Strengths:\n- The paper is clearly written and well organized. It was pleasant to read, and the reasoning flowed nicely.\n- The paper tackles a common problem that's encountered across several industries, how to apply off-policy evaluation when the logging policy is unknown. The typical approach of estimating a logging policy from data suffers from high bias, and this paper addresses that issue by including uncertainty when weighting samples with probability ratios.\n- The experimental results show that UIPS consistently outperforms the chose benchmark techniques, when the logging policy is estimated.\n\n\nWeakness:\n- I would have liked to see a comparison in the synthetic dataset of how close the proposed technique gets to the ground-truth logging policy, when it is estimated. That oracle baseline would be very helpful to understand how significant the improvements of UIPS are in contrast to the performance of the other methods. \nFor ex. if UIPS achieves P@5 of 0.5666 (table 1), the difference to CE 0.5559 is a lot more meaningful if the performance of using the true logging policy at P@5 is 0.5700, than if it is 0.988.\n\n- In my experience, doubly-robust (DR) estimators are often some of the best performing benchmarks for off-policy evaluation. I would have liked to see the use of a DR estimator as part of the evaluations.\n\n- For experimental results, please include standard deviation numbers as well as mean performance as it helps to identify how significant those improvements are.\n\n\nNitpick:\n- Page 8, section 4.1, The second sentence makes reference to Table 3 showing the average MSE, but table 3 is actually showing P@5, R@5. Maybe a wrong table reference, or something got lost in editing?\n\n- Table 2: Any comment on why would CE outperform other off-policy eval methods on low frequency actions? I'm very intrigued by that, as I would expect CE to be heavily biased specially in low frequency samples. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "c3PuvgU_IuX", "reviewer": "Reviewer_4K6H", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies off-policy learning when there is distribution shift between the data distribution and target distribution. The paper points out that the vanilla inverse propensity score (IPS) estimator suffers from large variance and bias when the estimated data distribution is inaccurate. To this end, this paper proposes to take into consideration the uncertainty in the logging probability estimation, and constructs an uncertainty-aware IPS estimator. A mathematical framework about how to solve for the \nUIPS estimator is described. Experimental results on both synthetic data and real-world data show that the proposed estimator performs better than a few benchmark algorithms on recommendation tasks.\n", "summary_of_the_review": "My current recommendation is reject. \n\nThe main reason is that the technical novelty is unclear to me. I did not see what the technical contribution of this paper is. \n\nHowever, this paper does have good and quite extensive experimental results. \n", "strength_and_weaknesses": "***Strength***:\n\n1. The paper is written very clearly. The problem is well-formulated. The theorems are stated clearly and there is no ambiguity. The experimental settings and results are presented with details.\n\n2. The experimental results are extensive. Various datasets and cases are studied. The results corroborate the proposed estimator.\n---\n***Weakness***: \n\n1. My major concern is that the technical novelty is unclear. \n\n(i). The existence of bias for the IPS estimator when the data distribution estimation is not accurate is a well-known result in the area of offline policy evaluation. \n\n(ii). IPS-based estimator is quite thoroughly studied. This paper studies the single-step setting, while IPS-type estimator is already extensively studied in the case of dynamic decision making/reinforcement learning. For example, in the DICE line of work ([1-3]), a series of distribution correction estimators are proposed (Also, these works are not discussed). Therefore, the correction term in this paper does not seem novel. \n\n(iii). The most significant technical contribution seems to be the objective function equation 8. However, this follows directly from the robust optimization as mentioned by this paper. \n\nOverall, it is not clear what is the technical novelty and contribution of this paper.\n\n2. There is no theoretical guarantee for the sample complexity of the proposed estimator. (minor issue)\n---\n[1] O. Nachum, Y. Chow, B. Dai, and L. Li: DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems 32 (NeurIPS), spotlight, 2019\n\n[2] R. Zhang, B. Dai, L. Li, and D. Schuurmans: GenDICE: Generalized offline estimation of stationary values. In the 8th International Conference on Learning Representations (ICLR), 2020\n\n[3] B. Dai, O. Nachum, Y. Chow, L. Li, Cs. Szepesvari, D. Schuurmans: CoinDICE: Off-policy confidence interval estimation. In Advances in Neural Information Processing Systems 33 (NeurIPS), spotlight, 2020\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S6CPQISEcU0", "reviewer": "Reviewer_FTLw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is concerned with accurately estimating the logging policy in off-policy learning when it is unknown or unavailable. Specifically, the paper derives an uncertainty-aware inverse propensity score algorithm called UIPS that is more accurate for rarely-taken actions, and provides theoretical and empirical arguments in favour of the algorithm.", "summary_of_the_review": "I am conflicted. The proposed algorithm is interesting and performs well in experiments, but my concerns with clarity and readability prevent me from wholeheartedly recommending acceptance. If the authors address my concerns, improve the clarity of the paper, and drastically reduce the grammatical errors, I would recommend acceptance.\n\n**Update:** The authors have addressed enough of my concerns that I now recommend accepting the paper for publication.", "strength_and_weaknesses": "**Strengths:**\n+ The paper draws on work from several areas to build the proposed algorithm.\n+ It seems like future algorithms for uncertainty estimation in neural networks could be dropped into the proposed method with only minor modifications, making the algorithm somewhat future-proof in this respect.\n+ Strong performance in experiments.\n\n**Potential Weaknesses/Questions:**\n- Some of the citations do not seem to cite the correct papers. For example, the earliest use of probability ratios in off-policy RL that I'm aware of is Sutton and Barto (1998), and the earliest peer reviewed paper using probability ratios in off-policy RL that I'm aware of is Precup et al. (2000).\n- The title of the paper seems overly broad, as the uncertainty is only in the estimated logging policy, not in the resulting policy or value estimates, and the proposed algorithm is only for the contextual bandit setting.\n- ~~The example of news recommendation at the beginning of Section 2 seems a little weird. If the context summarizes the user's interaction with the recommender system, then their choice of action would affect future contexts, which would be reinforcement learning and not a contextual bandit problem. If that's true, why use the contextual bandit setting?~~\n- It seems like the proposed algorithm is limited to the finite action case, and does not support continuous actions spaces, or generalization between actions.\n- ~~Can the authors please comment on the computational complexity of the proposed method in comparison to the competitor methods used in the experiments? Does UIPS use more computation than the other methods?~~\n- It would be good to explicitly specify the algorithm somewhere self-contained (even in an appendix would be fine if space is a concern), to make it easier to reproduce and easier for practitioners to implement.\n- There's a lot of notation to remember which makes the paper harder to understand, and some of it is not necessary. For example, on page 5, why define the function $g(x_n,a_n)$ as the gradient of $f_\\theta(x_n,a_n)$ instead of just writing $\\nabla_\\theta f_\\theta(x_n,a_n)$?\n\n**Minor comments/questions:**\n- Page 3: \"enlarges the this bias\" has a typo.\n\n**References:**\n* Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An Introduction.\n* Precup, D., Sutton, R. S., & Singh, S. P. (2000). Eligibility Traces for Off-Policy Policy Evaluation. ICML.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0uxdOH-v-d", "reviewer": "Reviewer_NQXb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies off-policy evaluation and learning, under unknown propensities, which is pretty common for most complicated systems. The paper proposes a novel Uncertainty-aware Inverse Propensity Score estimator, which builds upon the original IPS estimator by adding an additional weight to reflect the uncertainty of the estimated propensity score. This specific weight is derived by optimizing an upper bound of the MSE of the estimator. Empirical results on synthetic and real-world dataset shows better average performance compared to other baselines, in terms of MSE (for evaluation), Precision@K, Recall@K, etc (for learning). ", "summary_of_the_review": "I like the topic this paper studies, which seems to be relevant to lots of practitioners who work on OPE/OPL problems. This paper proposes a novel uncertainty-based IPS estimator, which is clearly motivated. Some minor theoretical guarantee of the proposed estimator and learning performance is missing. The empirical results are comprehensive, however it is unclear the significance of the gain, as well as the applicability of the method, due to hyper-parameter tuning as well as the training cost. Based on the current version of the paper, it is around the borderline from my perspective, but I would consider changing the score based on the authors' reply. ", "strength_and_weaknesses": "Strength:\n \n1). The unknown propensities are very common for real-world problems, especially when the underlying system is complex and the propensities are hard to calculate. This paper proposes efficient off-policy evaluation/learning method under this realistic setting.\n\n2). The paper is well-written and easy to follow.\n\n3). The method is well-motivated, solid, and the idea about reweighing based on uncertainty is neat. \n\n3). It has comprehensive ablation studies and empirical results, with both synthetic and real-world datasets.\n\nWeakness:\n\n1). The proposed method seems have several important hyper-parameters, that control the balance of bias and variance term, as well as the confidence bound. The methodology part does not give a clear guideline about how to choosing them. From empirical studies, it seems to be selected from the unbiased validation set? If I understand it correctly, it might be hard to get this validation set in real-world scenarios. Could the authors comment more on how these are selected for different experiments? \n\n2). The uncertainty estimate is crucial for the uncertainty weight in Theorem 2. The NTK theory provides a nice quantification about the uncertainty for deep neural networks. However, I am concerned about the dimensionality of the gradient vector $g$, as well as the inversion stability of the matrix M. What are their corresponding dimensionality for the experiments? [1] uses the inverse of the diagonal of the matrix M as approximations. Could the authors comment more on how the quality of the uncertainty estimate affects both the evaluation and learning? It should be reflected in the MSE of the estimator as well as the policy optimization regret bound.\n\n3). All the results are average results, without standard deviation. It would be great if the std is provided, and this could provide us more information about whether the gain over previous method is significant. \n\n4). Could the authors comment more on the convergence criteria? For the policy optimization step, is it performed until full convergence? This part seems missing from the paper, and it is worth commenting more about the computational cost as well. \n\n\n\nReference:\n[1]. Neural Contextual Bandits with UCB-based Exploration\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject