A General Framework for Off-Policy Learning with Partially-Observed Reward
Authors: Rikiya Takehi
Masahiro Asami
Kosuke Kawakami
Yuta Saito
Venue: ICLR 2025 Poster (2025)
Keywords: off-policy learning
partially-observed rewards
contextual bandits
Primary area: reinforcement learning
TL;DR: We introduce HyPeR, a method that enhances off-policy learning in contextual bandits with partially-observed rewards by leveraging secondary rewards to optimize policies effectively.

Abstract: Off-policy learning (OPL) in contextual bandits aims to learn a decision-making policy that maximizes the target rewards by using only historical interaction data collected under previously developed policies. Unfortunately, when rewards are only partially observed, the effectiveness of OPL degrades severely. Well-known examples of such partial rewards include explicit ratings in content recommendations, conversion signals on e-commerce platforms that are partial due to delay, and the issue of censoring in medical problems. One possible solution to deal with such partial rewards is to use secondary rewards, such as dwelling time, clicks, and medical indicators, which are more densely observed. However, relying solely on such secondary rewards can also lead to poor policy learning since they may not align with the target reward. Thus, this work studies a new and general problem of OPL where the goal is to learn a policy that maximizes the expected target reward by leveraging densely observed secondary rewards as supplemental data. We then propose a new method called Hybrid Policy Optimization for Partially-Observed Reward (HyPeR), which effectively uses the secondary rewards in addition to the partially observed target reward to achieve effective OPL despite the challenging scenario. We also discuss a case where we aim to optimize not only the expected target reward but also the expected secondary rewards to some extent; counter-intuitively, we will show that leveraging the two objectives is in fact advantageous also for the optimization of only the target reward. Along with statistical analysis of our proposed methods, empirical evaluations on both synthetic and real-world data show that HyPeR outperforms existing methods in various scenarios.

Published as a conference paper at ICLR 2025
A GENERAL FRAMEWORK FOR OFF-POLICY LEARN- ING WITH PARTIALLY-OBSERVED REWARD
Rikiya Takehi
Masahiro Asami
Waseda University
HAKUHODO Technologies Inc.
Sikiya.takehi@ee waseda jp
ma masahi ro. asami @hakkuhooo-teechnoo.oie.. co
Kosuke Kawakami
Yuta Saito
HAKUHODO Technologies Inc.
Comell University
kosuke kawakami www.uhhccctttechhollleeeeee
ys5228cornel1.
ABSTRACT
Off-policy learning (OPL) in contextual bandits aims to learn a decision-making policy that maximizes the target rewards by using only historical interaction data collected under previously developed policies. Unfortunately, when rewards are only partially observed, the effectiveness of OPL degrades severely. Well-known examples of such partial rewards include explicit ratings in content recommen- dations, conversion signals on e-commerce platforms that are partial due to delay, and the issue of censoring in medical problems, One possible solution to deal with such partial rewards is to use secondary rewards, such as dwelling time, clicks, and medical indicators, which are more densely observed. However, relying solely on such secondary rewards can also lead to poor policy learning since they may not align with the target reward. Thus, this work studies new and general problem of OPL where goal to learn policy that maximizes the expected target reward by leveraging densely observed secondary rewards as supplemental data. We then propose a new method called Hybrid Policy Optimization for Partially-Observe Reward (HyPeR), which effectively uses the secondary rewards in addition to the partially-observed target reward to achieve effective OPL despite the challenging scenario. We also discuss case where aim to optimize not only the expected target reward but also the expected secondary rewards to some extent; counter- intuitively, will show that leveraging the two objectives is in fact advantageous also for the optimization of only the target reward. Along with statistical analysis of our proposed methods, empirical evaluations on both synthetic and real-world data show that HyPeR outperforms existing methods in various scenarios.
INTRODUCTION
In applications that involve dynamic decision-making. such as ad placement and recommendation systems, exploration can be costly and risky. These constraints limit the use online exploration of actions, thereby motivating the study of offline policy learning methods. Off-ppolcy learning (OPL)
addresses this need by enabling the policy optimization using only logged bandit data generated
under old (or logging) policies (Joachims et al. 2018 Su et al. 2019 2020a Uehara et al. 2022
typical approach OPL is using policy gradient (PG), which we can estimate unbiasedly based on the logged data via techniques like Inverse Propensity Score (IPS) and Doubly Robust (DR) (Dudik al. 2014). OPL methods based on PG iterations perform effectively ideal scenarios with large sample sizes and relatively small action spaces (Saito & Joachims) 2021 [Sachdeva et al., 2023) Taufiq et al.) 2024). However, in many application scenarios, rewards are only partially observed due to missing data (Jakobsen et al. 2017: Wang [2il22199 Jadidinejad et al. 2019 Christakopoulou
et al. 2022), censoring (Ren et al. 2019; Wang et al. (019a)) delayed observation (Wang et al.
2022b Imbens al. 2022 Wang et al. [2023] Saito et al. (024a). data fusion (Imbens al. 2022]
and multi-stage rewards (Wan & McAuley Hadasha et al. 2018 Ma et al.) 2018] Saato 2020
For instance, e-commerce platforms, binary conversions
target reward maximize.

---

Published as a conference paper at ICLR 2025
Table I: Examples of Partially Rewards in Various Real-Life Scenarios
Scenario
Reward
Reason for Partial Observation
Streaming Platform
Explicit Rating
Mulli-stage & Missing (adaah al.
2018
E-commerce
Conversion
Multi-stage Delayed Klena
Ad-placement
Product Purchase
Missing Delayed (Ashkan Clarke 2009
Clinical Trials
Long-term Health
Censoring (Pratap et al. 2020
Financial Investment
Long-term Return
Delayed Arroyo ct al. 2019
Subscription Platform
User Retention
Delayed Wang cl al.
2022b
Remediation
Academic Performance
Delayed (Barnell 2011
Unfortunately, conversions are only observed for products that were clicked and seen by the user. To make matters worse, conversion signals are obtained only after weeks-long delays (Ktena et al.) 2021), and most of them are yet to be observed when policy learning is conducted. Table I provides a list of real-life examples and causes of partially-observed rewards. When rewards are only partially observed due to such causes, the estimation of PG would result in high variance and inefficient OPL.
Our work thus proposes new formulation and method to address this challenging but highly general problem: OPL with partial observations of the reward. In scenarios with only partial observations of the target reward, one possible idea to perform OPL more effectively is to more frequently observed secondary rewards instead. For instance, in e-commerce recommender systems, we often observe not only the conversion signals but also other implicit feedback such clicks and dwell time, which are much more densely observed with no missingness (Liu et al., 2010 Jadidinejad al., 2019). Thus, in our problem formulation, we consider both the targer reward that only partially observed (e.g., user ratings, conversions, retention, future earnings, and survival days) and the secondary rewards that are fully observed (e.g., clicks, dwell time, and short-term medical indi- cators). Considering the case where we also aim to optimize secondary rewards, we explore learning a policy that maximizes a weighted sum of the target reward and secondary rewards. This represents more general objective rather than focusing only on the optimization of the target reward.
With the availability of the secondary rewards, one feasible approach to perform OPL is to maximize some aggregation of the secondary rewards significant drawback of this approach is the potential for high PG estimation bias, as secondary rewards often do not align perfectly with the target reward (Liu et al.) 2010: Jadidinejad et al., 2019). We also do not know how to construct an appropriate aggregation function of the secondary rewards. Therefore, there is dilemma between the target and secondary rewards: the former more accurately aligns with our ultimate objective, while the latter provides more observations. Using only the former leads to high variance in PG estimation due to missing observations, while relying solely on the latter results in high bias due to potential misalignment with the ultimate objective.
surrogate for the target reward. Nonetheless,
To solve this new OPL problem more effectively, we focus on developing a method that can leverage both target and secondary rewards in a principled way. Specifically, we propose Hybrid Policy Optimization for Partially-Observer Reward (HyPeR), method which estimates the PG leveraging both types of rewards. We show that our PG estimator can substantially reduce the estimation variance compared to typical estimators such as IPS and DR while maintaining unbiasedness under reasonable conditions. In addition, while our approach generally performs effectively using the predefined weight between the target and secondary rewards within the objective, we demonstrate that further enhancement can be achieved by strategically tuning the weight to improve the bias- variance trade-off of the PG estimation, which we can perform based only on observable logged data. Finally, conduct comprehensive experiments where the HyPeR algorithm outperforms a range of existing methods in terms of optimizing both the target reward objective and the combined objective of the target and secondary rewards.
both synthetic and real-world datasets,
The key contributions of our work can be summarized as follows.
We formulate the general problem of Off-Policy Learning (OPL) for contextual bandits with partially-observed rewards, encompassing many prevalent scenarios such as missing data, delayed rewards, and censoring, all of which are instances of this general problem.
We propose new method to address OPL with partial rewards by leveraging more densely observed secondary rewards to estimate the policy gradient with reduced variance.
We consider combined objective defined by weighted sum of the target and secondary rewards and introduce the novel concept of a strategic use of an incorrect weight in our method to maximize the advantage of combining the two types of rewards.
2

---

Published as a conference paper at ICLR 2025
2 OFF-POLICY LEARNING
2.1 THE CONVENTIONAL FORMULATION
We first formulate the conventional OPL. problem regarding the typical contextual bandit set- ting (Dudik et al.] 2014 Swaminathan & Joachims 2015a; Farajtabar et al. 2018). In the typical formulation, context E X' S _ is a dy-dimensional vector that is drawn iid. from an unknown distribution p(z). Given context æ, a possibly stochastic policy π ((a [z) chooses an action a within finite action space denoted here by A. The (target) reward r E [0, rmaar (e.g., ratings, conversions, retention, survivals) is then sampled from an unknown conditional distribution p(r|a,,))
The existing literature defines the objective function in OPL. by the expected target reward (often referred as the policy value) as below (Swaminathan & Joachims 2015b).
V (TT) = peteeea [r] = (p(e))(aaa))gg
(I)
where q(z, a) := E[|| x, al is the expected reward given z and which we call the q-function. In OPL, the goal is learn a policy To. parameterized by that would maximize the policy value: 0" E arg max (mr)). In particular, OPI. aims to learn such policy using only logged data consisting of tuples (ar, a, r) generated under the logging policy denoted by To. More specifically, the logged data we can
perform OPL can be written as D {(I,, a. r:) p(D) where the data distribution is induced by the logging policy, i.c., p(D) = II. p(a,) ( (r))p(r a:).
2.2 OFF-POLICY LEARNING VIA POLICY GRADIENT
Most existing approaches in OPL are based on PG iterations (Ma al., 2020 Chen et 2021
This method updates the policy parameter 0 via iterative gradient ascent: 0.+1 + 0, + V mp where the policy gradient is represented as VoV (m²)
(a)=)) [g(x; a) Vo log Ty ( [x]] Saito
et al.] (2024b). This form of PG can be derived by the reg-deeivative trick and suggests to update the policy parameter
that the resulting policy Ty can choose actions that have high expected reward.
To implement the PG iteration, we first need to estimate the PG, since it is an unknown vector. To achieve this using only the logged data collected under the logging policy wo, the relevant liter- ature relies on estimators such as IPS and DR, which enable unbiased estimation of the PG (Dudík) ct al. 2014, Metelli et al. 202I). These PG estimators are specifically defined as follows.
1 n = w(2, (.) n
V. (PPT:
1
= V VORDR(TTO
[w(x, (r: ((0, a:)))((7;; + (rotala)))(( a)go ax]}
where w(z, a)
<e (r)/ x)) is the importance weight and ge(zz a) Ve log (e xx) is the policy score function. q(I, a) is an estimator of the q-function, which we can obtain by performing reward regression the logged data D.
These estimators are unbiased (i.e., Ep(P) VooinS D)] E. ED( PV (wo: D)] VoV(T)))
under full support, which requires that the logging policy sufficiently explore the action space.
Condition (Full Support). The logging policy mo is said to have full support if (a(a))) > 0 for all x E X and a E A.
The importance weighting technique used by IPS and DR enables an unbiased estimation of the PG,
thereby resulting in effective OPL even without additional exploration. However, if the problem is far from ideal with smaller data sizes, large action spaces, and noisy or partial rewards, existing OPL methods easily collapse due substantial variance in PG estimation (Saito & Joachims 2022
3

---

Published as a conference paper at ICLR 2025
2.3 -BBERVVE REWARDS
As discussed in the introduction and in Table[] there are many real-life cases where we can observe the target reward only partially. To precisely formulate such a scenario, we introduce an additional random variable, E (0, 1). to represent whether the reward is observed for each data point. If
of L, the reward r; is observed; otherwise, it is unavailable, as indicated by setting T3 N/A. Considering a scenario where the observation indicator comes from an unknown distribution 0 <
p(olz) < 1, we can generalize the data generating process as D ((ri, O,, 0z, r.); r=) p(D)
Ih- (a; ) a;), which encompasses many realistic situations like missing data, delayed rewards, data fusion, multi-stage rewards, and censoring.
To implement OPL with such partially-observ rewards, we can simply apply the PG approach using only the data with observed rewards, which we call r-IPS and r-DR:
N VoRAIPS(TT: (To; D) : 11
Σ i=1
0:
w(xi, a.) (s)rigo(ss 0z),
(4)
p(o)(a)))
V VOV.DR(TT: D):
Σ
Oi p(o,(ar.)
{w(s, ar)(r ((24 (4)))) + Raooa (,)( (r) a]]}
where we can see that these estimators use only the data with reward observations (o; 1) to esti- mate the PG. We can readily show that these estimators are unbiased, but they are often substantially inefficient and produce high variance in PG estimation. This is because they use only a part of the data in D and naively discard all the information when o, 0.
3 OFF-POLICY LEARNING WITH SECONDARY REWARDS
To deal with the issue of inefficiency in OPL when the rewards are partial, we propose a new for- mulation of OPL with secondary rewards. In many real-life scenarios. we not only observe the target reward such as the conversion signals that we are optimizing, but also secondary rewards such as clicks and dwell time (Wan & McAuley| [2018] Jadidinejad et al. 2019 Christakopoulou et al.] 2022). By leveraging these secondary rewards, aim to reduce variance in PG estimation and to achieve a more efficient OPL even under the challenging scenarios with partially-observed rewards.
To implement this idea, we extend the typical formulation of OPL by introducing the secondary rewards denoted by E Rd,, which can be to be sampled from unknown conditional distribution of the form: p(s|z, a) after taking action for context æ. The logged dataset that we can use in the setting can be written as follows.
We consider the secondary rewards
D == {(T ai 0g, 8,,r; ~ p(D) = II (a (.tp st),
(6)
In addition to the introduction of secondary rewards, we consider the generalized objective called the combined policy value, which is defined weighted sum of the expected target and secondary rewards, considering a situation where we aim to optimize the secondary rewards as well.
Vo(m;;)) = (1 (1-3))=(=+))) (V. = (1- (3) + (Bp(s)
Salr, a)
where fa(æ, a) := E[sa|2z a] is the d-th dimension of the expected secondary reward and 3 [0, 1) is a parameter to control the prioritization between the optimization of the target reward and that of the secondary rewards. When 3 0, this generalized objective reduces to the typical policy value defined in Eq. (T). With some positive value of 3, we can address practical situations where we aim to optimize not only the target reward but also the secondary rewards to some extent.
Given this extended problem of OPL with secondary rewards and with the combined policy value, can consider baseline method of using some aggregation of the secondary rewards surrogate
4

---

Published as a conference paper at ICLR 2025
for the target reward. This baseline approach of using only the secondary rewards surrogates, which we call s-IPS and s-DR, estimates the PG as follows:
Vol. DosarsstTr (To: D) = waz, F(s)))))))))
(8)
VoV.DD(TTO:7 D)
(9)
1
w(2, ((2)) - F(f(x), (.)))))()s) a:) + (F(()) 1=1
where f(x, a) is an estimator of f(z, a) and F(s) some aggregation of the secondary rewards, such as their weighted average, designed to imitate the target reward. By replacing the target reward with F(s), s-IPS and s-DR can use all the data points irrespective of the observation indicator og, thus reducing the variance. However, unless the function F(s) accurately describes the target reward, which is untestable, the estimators often produce substantial bias against the true PG regarding the target reward, which leads to ineffective OPL as we will show in our experiments.
4 HYBRID POLICY OPTIMIZATION FOR PARTIALLY LABTERLLY-OBSERVED REWARD
This section proposes a new OPL algorithm maximize the combined policy value in Eq. 7 with only partially-observe target rewards and secondary rewards that do not necessarily align accurately with the target reward. We also newly introduce the concept of strategically using a different value of the balancing factor compared to the true value of 33 in the objective in Eq. (7) to improve the snite-sample effectiveness of the algorithm.
The key idea behind our algorithm is the introduction of a new estimator of the PG using secondary rewards reduce variance while also optimizing the secondary rewards depending on the value of 3 in the objective function.
To derive our method, we first focus on the PG estimation regarding the target policy value V, (mg). Particularly to address the variance issue in existing methods, we propose leveraging secondary rewards through the following estimator for (TV)(TT).
VaVi(TO; D) = Bmo(alz.) (0(2,, (go a)) + w(a;, az) (f(#), a;, ((x;, az)) go (o(1;,a))
w w(z, a.) (r, ((x, a, 9:)) go (zi (z) (10)
p(o.(x:))
where q(z, a, 8) is an estimator of the q-function conditional on the secondary rewards (i.e., ((x, a, s)), which we can derive, e.g., by solving using D. Note also that we can estimate the conditional reward-observation probability p(o, (x,) when it is unknown by regressing We need to perform this estimation task before performing OPL not only for our method, but also for baseline methods such as r-IPS and r-DR.
arg ming E(r...rre (x, a, s))²
observed in the logged data D supervised classifier.
We first show that our PG estimator is unbiased under the same conditions as r-DR.
Theorem 1. (Unbiasedness) Under Condition Eg. 70 is unbiased against the true PG regarding the target reward, i.e.,
E VoV D)] VoVi(T)) V VoVa(mo3 B =
See Appendix B.7 for the proof.
In addition to its unbiasedness, our PG estimator fully utilizes the information from the secondary rewards, thus reducing the variance compared r-DR in most cases. The following demonstrates that the variance of Eq. (10) can be much lower than r-DR.
5

---

Published as a conference paper at ICLR 2025
Theorem 2. (Variance Reduction) Under Condition we have
D)] VD(VoV,(T;
p²
= =Epte)moldlsppssla..
w1, gz a)² p(o|z)2
- 12 (2 (A
a,)))
where p² = V[olz] is the variance of the observation indicator. A a, == q(x, a, 8) - ((x, a) and Agg(s, a, == q a, 8) ((x, a, 8) are the estimation error of (( a)) and ((x, a)s), respectively. See Appendix B.2 for the proof.
Theorem 2 indicates that there is reduction in variance if ( a, 8) is better than ((2;, a) in esti- mating { s), which is often the case since secondary rewards are typically somewhat correlated with the larget reward. Thus, Eq. (10) is expected to perform better than r-DR while being unbi- ased under the same condition. To demonstrate this, in the experimental sections, we investigate the difference in performance between Eq. (10), which we name HyPeR(y = 0), against r-DR.
Based on the new estimator for the PG regarding the target reward defined in Eq. 100) we finally introduce our HyPeR estimator.
Ve WHyPeRR(TT: D,y) (1- x) VoV.(TO: + (=Vo
(11)
where the first term Vo VoVi(TT) D) estimates the PG regarding the target policy value V-(mT) in Eq. (10). The second term Vo (To; D) estimates the PG regarding the secondary policy value (TT)), which we can define by simply applying DR to the sum of secondary rewards as
(oVo(To;
d.
E (al
Σ fa(me, ()) ai) w(x; a:)
Σ
( -a(ss, go go (.))aa))))) d=1
n
In Eq. [I], y E [0, 1) is tunable parameter that defines the mixture ratio between the estimators of the PG regarding the target reward and secondary rewards. When using the predefined weight (i.e., (), Eq. 11 unbiased against the combined policy gradient (i.e., E Viypen((To; D)] (TV()) but the following discusses a strategic tuning of y to further improve the finite-sample effectiveness of HyPeR algorithm.
4.1 STRATEGIC TUNING THE WEIGHT ?
Although the natural choice of y in our PG estimator is B, which defines the true objective in Eq. 7 we argue that intentionally using 7 * 3 can further improve the combined policy value in the finite sample setting. This is due to the potential differences in variance between the different types of rewards. Specifically,, strategically shifting weights from the predefined balance, B,, can lead to better estimation due to variance reduction, even though it introduces some bias. For example, consider objective function that variance than estimator Y, it is likely better give more weight to X to achieve less variance, even at the cost of introducing some bias. When dealing with multiple reward types, this can occur in various situations, such when one reward is less noisy than the other, or as our study, when one reward type (secondary reward) is more frequently observed than the other (target reward).
of two estimands, X Y. If estimator X carries less
In HyPeR, the PG regarding the target reward V VoVi(TO: D) often carries much higher variance compared to that regarding the secondary rewards V, V. (my; D) due to the pastial-observation na- ture. Thus, although setting y = 3 will make the HyPeR estimation unbiased and at least reduce the variance from existing methods. it can still have high variance, particularly when 3 takes a small value. On the other hand, increasing the weight of y will likely lead to less variance since this prior- itizes the second term more, while introducing some bias. This creates interesting bias-variance trade-off, and we aim to tune 7 to achieve the optimal combined policy value by the resulting policy:
arg max V (TO(1:7 D);
(12)
>E(0.1)

---

Published as a conference paper at ICLR 2025
where ( D) is policy optimized under the weight used in our policy gradient estimator in Eq. I and its value is evaluated under the originally defined weight B.
Since the true policy value is unknown, we need to perform Eq. (12) from only the logged dataset D. The most straightforward method is via splitting the dataset D> into training (Dtr) and validation (Deat) sets. Then, we train policy using Der (i.e., To (13 7, D),)) and estimate the value using Daal (i.e.. (TT: B, Deat)). However, an issue with this naive procedure is that the policy :e(: Dir) is trained on a smaller dataset Dr, instead of the full dataset D that is used in real training. In the PG estimation, variance inversely proportional to the data size (D), while bias is unaffected. Therefore, the naive tuning procedure may end up selecting a higher weight than the optimal weight y" by overly prioritizing variance reduction. To address this issue, we ensure that [Drrl = D through bootstrapping:
D'r = [ d' 8/, of 1i=1,..,,,))
(13)
where (10), di, (, Der is sampled independently with replacement. By doing so, alleviate the issue regarding the variance estimation, since Dir now has the same size as the full dataset D. Using Dyr,. propose solve the bi-level optimization data-drivenly tune y:
arg max V (1 PPo)) 3, Deat). ee/0,1]
(14)
SYNTHETIC EXPERIMENT
Synthetic Data Generation. To create synthetic data, we sample 10-dimensional context vectors r from a standard normal distribution, and the sample size is fixed at n = 2000 by default. We then synthesize the logging policy = = softmax( (0xx" Mx, 0z + 0o)). where M, On,, 0g are parameter matrices randomly sampled from a uniform distribution with range [-1, 1]. and actions A are sampled following this logging policy (IAA) = 10). o is a parameter that controls how deterministic the logging policy would become, and we set this at & = - 2.0 by default. We then synthesize each dimension of the 5-dimensional expected secondary rewards given x and as
(15) and secondary rewards are sampled from a normal distribution sa N(aa(z, (U)( o)). In the main text, we set the default to og = 0.5, and the results for other values of as can be found in Appendix C.2 We finally symthesize the expected target reward function as
fa(x, a) = x²' Mxx + 0=0' + +a a'
+ x² 0% + a aTo" + x" Mx.Ff + a MAPf)+ q(z, a, a(),))) = (1 - (x M
(16)
E [0, 1] is an experimental parameter to control how much secondary rewards are correlated with the target reward. When À = 1, secondary rewards are completely correlated with the tar- get reward, while a smaller A will make it less correlated. We use X = 0.7 as a default setting throughout the synthetic experiment. The target reward is sampled from a normal distribution as r ~ ((g(x, x. a, fax ()), of) with default o, = 0.5 (results with other or values are provided in Appendix C.2 The target reward observation probability is an experimental parameter and is set to p(olz)) = 0.2 for all x by default The true weight 8, which is used to define the combined policy value Ve(=; B). is set to 0.3 and it is also one of the experimental parameters.
In the synthetic experiments. we generally use the predetined weight /3 for HyPeR, and represent as HyPeR(y (9.. We compare HyPeR(y = 9) against HyPeR(? = 0), r-IPS, r-DR, s-IPS and s- S-DR. For s-IPS and s-DR, we use F(s) = s" (0, + Err, where 0, is identical to that of Eq. (16). and E ~ ./0 o is the noise to simulate the inaccuracy of describing the target reward.
5.1 RESULTS AND DISCUSSION
We run OPL simulations 100 times with different train-test splits. We report the relative policy values calculated by (V(T*) - V (TTe)))(V(( - V (TTuii))), where x² is the optimal policy and Tunif is a uniform random policy. This way, the performance of a random policy will have a value of O. and an optimal policy would have value of 1, thus easier to interpret. Note that the shaded regions in the plots represent 95% confidence intervals estimated via bootstrapping.
'Appendix C.2 presents additional experimental results with estimated observation probabilities, which support the same conclusion the main text.
7

---

Published as a conference paper at ICLR 2025
r-IPS
a-IPS
rD
SDDR
HyPeR(y= 0)
Hy B)
Combined Policy Value
Target Policy Value
Secondary Palicy Value
06
0.75
0.6
2:00
0:25
0.0
(Hojol)
(plose))
Figure 1: Comparing the combined, target, and secondary policy values of OPL methods with vary- ing target reward observation probabilities (p(olz)) on Synthetic Data.
Combined Policy Value
Target Policy Value
Secondary Policy Value
ca
0.
0.
06-
04
04
02
022
0.2
450
1800
3000
5400
12830
450
800
3200
6400
12800
420
.900
1000
6400
129000
Training Data Size
Training Data Size (p)
Training Data Sibe (n)
Figure 2: Comparing the combined, target, and secondary policy values of OPL methods with vary- ing training data sizes (n) on Synthetic Data.
How does HyPeR perform with varying target reward observation probabilities? Figure evaluates the relative policy value when we vary the observation probability of the target reward p(o[z). We observe that HyPeR(y = (3) provides the highest combined policy value in all cases. also able to optimize both the target and secondary policy values in a balanced way. In ad- dition, HyPeR(y 3) outperforms all the baselines regarding the target policy value, and it even outperforms HyPeR(? 0) when the observation probability low. This because the secondary reward maximization component of HyPeR reduces the variance of the PG estimation, as secondary rewards are denser. We also observe that HyPeR(y = 0) (target reward maximization component of HyPeR) is always performing better than r-DR, which suggests the effective use of secondary rewards enhances the estimation of the target gradient VoV)(TO) alone, as in Theorem 2
How does HyPeR perform with varying training data sizes? Figure 2 evaluates the methods' policy values with different training data sizes. Larger data size generally makes gradient estimations more accurate as decreases the variance. Therefore, as the data size increases, the methods that use the target reward in their estimation (i.e., HyPeR(y = 3). HyPeR(y = O), r-IPS, and r-DR) perform increasingly better compared to the ones that do not (i.e., s-IPS and s-DR). The left plot in Figure 2 shows that HyPeR(y 33) performs the best in most cases regarding the combined policy value (T). It even performs the best terms of the target policy value (T),,, particularly when the data size is small due to the use of secondary rewards and resulting variance reduction.
How does HyPeR perform with varying correlation between target and secondary rewards? Ssows the results with different degrees of correlation between the secondary rewards and the target reward, controlled by X in Eq. (16). A larger À indicates that the target reward can be more predictable by the secondary reward. We can also see that HyPeR(y in almost all cases for both the combined policy value and target policy value. We also observe that HyPeR(y B), s-IPS, and s-DR perform comparatively well when secondary rewards are more correlated (larger X), as they find more advantage in using the secondary rewards as surrogates.
:) is the best option
How do HyPeR and data-driven weight selection perform with varying true weight !? Sec- 4.1lwwe that data-driven tuning of the weight y will potentially improve the performance of HyPeR. To empirically evaluate whether our beight-tuning method can make further improvement, in addition HyPeR(y 3),, add HyPeR(Tuned 4*) and OpPenaO y²) for comparison.
8

---

Published as a conference paper at ICLR 2025
Combined Policy Value
Target Policy Value
Secondary Policy Value
c
c.
0.2
0.0
0.0 02
0.4
(A]
Figure 3: Comparing the combined, target, and secondary policy values of OPL methods with vary- ing degrees of target-secondary reward correlation (A) on Synthetic Data. Secondary rewards can completely explain the target reward X 1, and they are less correlated with smaller A.
HR
DR
H) 0)
HyPoRly = P)
HyPoR/Tuned 8*)
HyPe[[pplimal y*)
Combined Policy Value
Target Policy Value
Secondary Policy Value
06
08
0.
0.
0.30
0.2
Beta Value
(8)
Figure 4: Comparing the combined, target, and secondary policy values of OPL methods with vary- ing B, which is weight that balances target and secondary policy values in the combined policy value. Higher 3 means the secondary policy value becomes more dominant.
HyPeR(Tuned ?*) estimates the optimal weight through estimation using the method described in Section 4.1 HyPeR y²) is a skyline that performs HyPeR with truly optimal weight y", which is not feasible in practice but it is a useful reference. Due to the increase in the number of comparisons, here we reduce r-IPS and s-IPS from the baselines, which always perform worse than DR-based methods (Appendix C. shows the complete results including r-IPS and s-IPS). Figure provides the results, including HyPeR(Tuned 4*) and HyPeR(Optimal 7²), with varying weight 3. From the results, we can see that HyPeR(Tuned 4*) always outperforms HyPeR(y 3) and all other feasible methods, suggesting the effectiveness of our weight tuning procedure. This obser- vation also interestingly implies that incorrect weight (i.e., " * 3) can lead to better policy performance compared to & = 3. due to the variance reduction at the cost of some bias in the PG estimation as discussed in Section 4.1 Appendix Ca.l
also empirically demonstrates the advantage of leveraging the bootstrapping procedure in our tuning process.
6
EXPERIMENT
To assess the et al. 2022) This is a publicly available fully-observed user-item matrix data collected on a short video platform, where 1,411 I users have viewed all 3,317 videos and left watch duration as feedback. This unique feature of KuaiRec enables to perform an OPL experiment without synthesizing the reward function (few other public datasets retain this desirable feature).
applicability of HyPeR, we now evaluate it on the KuaiRec dataset GGao
Setup. We use watch ratio (= watch duration/video length) as target reward r. We use four- dimensional secondary rewards; each dimension has a realistic reason why the platform would want to maximize it. The first dimension is binary, where $1 = 1 if r 2 2.0, and $1 if < 2.0; this reward maximization lets the platform prioritize sessions with an exceptionally long watch ratio. The second dimension is also binary, where 82 = -1 if r < 0.5, and 82 0 if r 2 0.5. This dimension is built to strictly punish and refrain from sessions with an exceptionally low watch ratio (raising the engagement floor). The third dimension of the secondary rewards is time since video upload (multiplied by -1), which is implemented to prioritize newer videos over older ones. The last
hhups//kuairec.com/

---

Published as a conference paper at ICLR 2025
+DR
HyPaRly 0)
HyPoRly B)
HyPoR(Tuned V
HyPoR|(pttmal
Combined Policy Value
Combined Policy Value
Combined Policy Value
0.7
0.76
0.00
0:92
0.1
0.
0.25
0.0
010
103
0.00
500
1030
2000
40033
8000
19300
0.00
0.0
0.2
0.4
0.5
0.5
1.5
Target Reward witseration Probobilty (plo
Traniin Date Size (n)
Uler Varue (2)
Figure 5: Comparing the combined policy values of OPL methods under (Ieft) varying target reward observation probabilities (p(olz)), (center) varying data sizes (n), and (right) varying true weights in the combined policy value on the KuaiRee dataset.
dimension is video length, which is designed to prioritize longer videos when watch ratios are sim- ilar. Note that all continuous rewards normalized to the range [0, 1]. We use the first dimension of the secondary rewards to express F(s) for s-IPS and s-DR.
To perform OPL experiments on the dataset, we randomly choose 988 users (70%) for training and 423 users (30%) for evaluation. We set the target reward observation probability to p(o|z) 0.2 for all æ, training data size to n 1000, and weight 3 0.3, as default experimental parameters. The actions are chosen randomly with size |A| = 100, and Appendix C.3 shows results with varying numbers of actions. We define the logging policy as To (a/zz) = sollmax( Mx, + x x + a (a),, with @ = 2.0, and run 100 simulations with different train-tess splits
Results. Figure 5 provides real-world experiment results with varying target reward observation probabilities p(o x). varying training data sizes n, and varying weights 3 in the combined policy value. In this section, we provide comparisons of the combined policy values for r-DR, s-DR, HyPeR(y = 0), HyPeR(y 3), HyPeR[Tuned 5*) and HyPeR(Optimal 7²). In Figure 5 we observe that HyPeR(y 3) and HyPeR(Tuned RT)) both outperform the baselines by far. Moreover, by comparing ElyPeR(Tuned 5*)) against HyPeR(y = 3). we can see that intentionally using the tuned weight leads better performance, particularly under more challenging scenarios (i.e., sparse target reward observation and small training data size). We also observe, in Appendix C.3 that HyPeR outperforms the baseline methods in terms of not only the combined policy value, but also the target and secondary policy values. It is also interesting to see that, in terms of the secondary policy value, the policy performance becomes much worse when we use only the target reward (i.e., performance of r-DR at 5 = 1), compared to the case where we use only the secondary rewards (i.e., performance of HyPeR(? we used in our real experiment is not highly correlated with the target reward, ensuring the problem non-trivial and making crucial to leverage both types of rewards via our hybrid approach.
(3) at 3 1). This demonstrates that the secondary rewards that
CONCLUSION AND FUTURE WORK
This paper studies the general problem of off-policy learning (OPL) in contextual bandits with partially-observed target rewards. Relying only on the target reward learn a policy often leads to high variance and inefficient learning in this setup. We propose Hybrid Policy Optimization for Partially-Observed Reward (HyPeR), novel approach that integrates secondary rewards to improve policy gradient estimation. HyPeR effectively reduces variance while maintaining unbi- asedness and can easily be extended to deal with the combined objective of maximizing both the target and secondary rewards from available data. Experiments synthetic and real-world datasets demonstrate that HyPeR outperforms existing methods in optimizing both the target and combined policy values. In future work, it would be valuable to extend our formulation and method to the problem of offline reinforcement learning. Conducting an experiment in a real environment would also further demonstrate the effectiveness, applicabliiyy and real-world impact of our method.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Shima Alizadeh, Aniruddha Bhargava, Karthick Gopalswamy, Lalit Jain, Branislav Kveton, and Ge Liu. Pessimistic off-policy multi-objective optimization. October 2023. URL http://
wwx.v.org/abs//200 2//.//21111111111111..11 18617 arXiv preprint U1Xiv:2310.1861
Gholamali Aminian, Armin Behnamnia, Roberto Vega, Laura Toni, Chengchun Shi, Hamid R. Ra- biee, Omar Rivasplata, and Miguel D. Rodrigues. Semi-supervised batch learning from logged data, 2024. URL http://aaxvvv..o http:://wwt.....oo/aa/////////0/
Javier Arroyo, Francesco Corea, Guillermo Jimenez-Diaz and Juan A. Sacio-Garcia. Assessment of machine learning performance for decision support in venture capital investments. IEEE Access, 7:1 7:124233-124243 2019. doi: 10.1 A0.100/AACCSSSS22999933666
Azin Ashkan and Charles L.A. Clarke. Characterizing commercial intent. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM '09, pp. 67-76, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 978 605585113. doi: 10.
1145/1645953. 1645965. /ttpo/////////////.oo/////001110
W. S. Barnett. Effectiveness of early educational int Science, 33346045) 2011. doi: 10.1126/science.12244
science. 1204534
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. Top-k off-
policy correction for reinforce recommender system, 2021.
Konstantina Christakopoulou, Can Xu, Sai Zhang, Sriraj Badam, Trevor Potter, Daniel Li, Hao Wan, Xinyang Yi, Ya Lc, Chris Berg, Eric Bencomo Dixon, Ed H. Chi, and Minmin Chen. Reward shaping for user satisfaction in reinforce recommender, 2022.
Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. Statistical Science, 29(4). November 2014. ISSN 0883-4237. doi: 10.1214/ 14-sts500. htILhttp://
org 00 1211/14-
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation, 2018.
wwwp:/wwwwwwwwpbx/xxxoooo/ 03493
Germano Gabbianelli, Gergely Neu, and Matteo Papini. Importance-weighhed offline learning done right. In International Conference on Algorithmic Learning Theory, pp. 614-634 PMLR, 2024.
Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recom- mender systems. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, pp. 540-550, 2022. doi: 10.1145335118083355222 URL
3557220
Guy Hadash, Oren Sar Shalom, and Rita Osadchy. Rank and rate: multi-task learning for recom- mender systems. In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys '18. ACM, September 2018. doi: 10. 10.1145/3240333.224444 3244400 URL
dof o
T0.11145/240323.. 3240406
Guido Imbens, Nathan Kallus, Xiaojie Mao, and Yuhao Wang. Long-term causal inference under persistent confounding via data combination. arXiv preprint arXiv:2202.07344 2022.
Amir H. Jadidinejad, Craig Macdonald, and Iadh Ounis. Unifying explicit and implicit feedback for rating prediction and ranking recommendation tasks. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of on Information Retrieval, ICTIR '19, pp. 149-156, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 1500368888. 3341981.334422 URL http::ttt:///t /ttp://ooi.coo http:///ooo..011.1111131333 1ew.e11143333333 11t.://01/1.1.13311111111111 334423
doi: 10.1145/
Janus Christian Jakobsen, Christian Gluud, Jøm Wetterslev, and Per Winkel. When and how should multiple imputation be used for handling missing data in randomised clinical trials practical guide with flowcharts. BMC Medical Research Methodology, 17(1):162, December 2017. ISSN
1471-2288. doi: 10.1 1866s 12874-01--04441111

---

Published as a conference paper at ICLR 2025
Olivier Jeunen and Bart Goethals. Pessimistic reward models for off-policy learning in recom- mendation. In Proceedings of the 15th ACM Conference on Recommender Systems, RecSys '21, pp. 63-74, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384582. doi: 10.1145/3460231 3460231 3474247
URL
Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. Deep learning with logged bandit feedback. In International Conference Learning Representations, 2018. URL https://
openreview net/ forum?id SJaP -XAb
Sofia Ira Ktena, Alykhan Tejani, Lucas Theis, Pranay Kumar Myana, Deepak Dilipkumar, Ferenc Huszar, Steven Yoo, and Wenzhe Shi. Addressing delayed feedback for continuous training with neural networks in ctr prediction, 2021. URL petttts:://axxie wtaaxiivvvo. orgg abs///b/// / 1907. 06558
Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
Nathan N. Liu, Evan W. Xiang, Min Zhao, and Qiang Yang. Unifying explicit and implicit feedback for collaborative filtering. In Proceedings of the 19th ACM International Conference Infor- mation and Knowledge Management, CIKM '10, pp. 1445-1448, New York, NY, USA, 2010. doi: 14144///871437.18116633
Association for Computing Machinery. ISBN 9781450300995. 1871643
URL httpp::: //c:: wwt.////......111111111111111111 /Attpp://od..oog/10011//////
Ben London and Ted Sandler. Bayesian counterfactual risk minimization. In International Confer-
ence on Machine Learning. pp. 4125 4133. PMLR, 2019.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang, Minmin Chen, Jiaxi Tang, Lichan Hong, and Ed H. Chi. Off-policy learning in two-stage recommender systems. In Proceedings The Web Con- ference 2020, WWW '20, pp. 463-473, New York, NY, USA, 2020. Association for Com- puting Machinery. ISBN 9781450370233.
doi: 10.1145/336642333388333 URL https:
//doi.org/10. 111.11111115333362210 3380110
Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang. Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In The 41st International ACM SIGIR Conference Research Development in Information Retrieval, pp- 1137-1140, 2018
Alberto Maria Metelli, Alessio Russo, and Marcello Restelli. Subgaussian and differ- entiable importance sampling for off-policy evaluation and learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 8119-8132. Curran Associates, Inc.,
2021. URL https::/proceedde
neurips cc wwppaper..pp.ep.pppppeeeeeee...e wwwes//pper.///// pdf
F1le/44763 4 F11e44478222200000ee8ebccccc2222
Abhishek Pratap, Elias Chaibub Neto, Phil Snyder, Carl Stepnowsky, Noémie Elhadad, Daniel Grant, Matthew H. Mohebbi, Sean Mooney, Christine Suver, John Wilbanks, Lara Mangravite, Patrick J. Heagerty, Pat Areán, and Larsson Omberg. Indicators of retention in remote digital cross-study evaluation of 100,000 participants. npj Digital Medicine, 3(1):21, doi: 10. 10.1038//4174--10 077744660200-2---8 1038/ 4417/411464-644 020 0224-8
health studies:
2020. ISSN 2398-6352.
TRLhttps: //ooi///oo.. 10.
Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang, Weinan Zhang, Lin Qiu, and Yong Yu. Deep recurrent survival analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 33 (01):4798-448) 4805, Jul. 2019. doi: 10. 1609//aaiiaiaaa v33i01 33014798. URL wtw.nttpss////// httttttss//osss.o. org/index php/ AAAT /tA.///ttccle//ee////aaaa.
Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 965-975, 2020.
Noveen Sachdeva. Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley. Off-policy evaluation for large action spaces via policy convolution, 2023. URL https: : http///xxvvvo////// abs/ bbss/3100 0. 5433
12

---

Published as a conference paper at ICLR 2025
Yuta Saito. Doubly robust estimator for ranking metries with post-click conversions. In Proceedings of the 14th ACM Conference on Recommender Systems, pp. 92-100, 2020.
Yuta Saito and Thorsten Joachims. Counterfactual learning and evaluation for recommender sys- tems: Foundations, implementations, and recent advances. In Pruceedings of the 15th ACM Conference on Recommender Systems, pp. 828-830, 2021.
Yuta Saito and Thorsten Joachims. Off-policy evaluation for large action spaces via embeddings. In International Conference on Machine Learning, pp. 19089-19124 PMLR, 2022.
Yuta Saito and Masahiro Nomura. Hyperparameter optimization learning and how to deal with it. arXiv preprint 2rXiv:2404.15084 2024.
be harmful in off-policy
Yuta Saito, Himan Abdollahpouri, Jesse Anderton, Ben Carterette, and Mounia Lalmas. Long-term off-policy evaluation and learning. In Proceedings of the ACM on Web Conference 2024, pp. 3432-3443, 2024a.
Yuta Saito, Jihan Yao, and Thorsten Joachims. Potec: Off-policy learning for large action spaces via two-stage policy decomposition. arXiv preprint arXiv:2402.06151, 2024b.
Otmane Sakhi, Imad Aouali, Pieree Alquier, and Nicolas Chopin. Logarithmic smoothing for pes- simistic off-policy evaluation, selection and learning. arXiv preprint arXivv2405.1433. 2024.
Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive blending estimator for policy evaluation and learning, 2019. URL https://arxiv.org
abs /ess/111.. 02672
Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage, 2020a.
Yi Su, Pavithra Srinath, and Akshay Krishnamurthy. Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pp. 9196-9205. PMLR, 2020b.
Adith Swaminathan and Thorsten Joachims. The self-normalize estimator for counterfactual learn- ing. In Proceedings of the 28th International Conference on Neural Information Processing Sys- tems Volume 2, NIPS' 15, pp. 3231-3239 Cambridge, MA, USA, 2015a. MIT Press.
Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback, 2015b. URL httpp:::/aax...oor///bb//////// 502. 02362
Muhammad Faaiz Taufiq, Amaud Doucet, Rob Cornish, and Jean-Francois Ton. Marginal density ratio for off-policy evaluation in contextual bandits. Advances in Neural Information Processing Systems, 36, 2024.
Takuma Udagawa, Haruka Kiyohara, Yusuke Narita, Yuta Saito, and Kei Tateno. Policy-adaptive estimator selection for off-policy evaluation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 10025-10033, 2023.
Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in rein- forcement learning. arXiv preprint arXiv:2212.06355, 2022.
Mengting Wan and Julian McAuley. Item recommendation monotonic behavior chains. In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys '18, pp. 86-94, New York. NY, USA, 2018. Association for Computing Machinery, ISBN 155355999 doi:
10. 00.114553240333.3240000
URL. htttp:://d/l wrg./0 /111/3244//333333 32115/332223233323333333 3240369
Nan Wang, Hongning Wang, Maryam Karimzadehgan, Branislav Kveton, and Craig Boutilier. Imo3.
Interactive multi-objectivv off-policy optimization. January 2022a. URL http://arxiv:
org/abs/ 2201. 09798 arXiv preprint arXiv:2201.09799.
Ping Wang, Yan Li, and Chandan K. Reddy. Machine learning for survival analysis: A survey. ACM Compur. Surv., 51))), feb 2019a. ISSN 0360-0300. doi: 10. 145/321430. URL hetps:
7/doi. 1145//1..11//31143300
13

---

Published as a conference paper at ICLR 2025
Siyu Wang, Xiaocong Chen, Dietmar Jannach, and Lina Yao. Causal decision transformer for rec- ommender systems via offline reinforcement learning, 2023. URL bttps://axxiv.orr
115/2304.07923)
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recom- mendation on data missing not at random. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, www...pressvv9// https////////////////////aa 9n. html
PMLR, 09-15 Jun 2019b. URL
httpps:/prroeedding ngs. htwpp//........./...ppp........e
Yuyan Wang, Mohit Sharma, Can Xu, Sriraj Badam, Qian Sun, Lee Richardson, Lisa Chung, Ed H. Chi, and Minmin Chen. Surrogate for long-term user experience in recommender sys- tems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDDD '22, pp. 4100-4109, New York, NY, USA, 2022b. Association for Com- puting Machinery. ISBN 9781450393850. doi: 10. 10.1145/55446883333003.
URL https:
//doi
33339073
14

---

Published as a conference paper at ICLR 2025
A
RELATED WORK
A.1 OFF-POLICY LEARNING
The contextual bandit framework has been used as one of the favored methods for online decision- making under uncertainty (Lattimore & Szepesvári 2020). There is also growing demand for refining decision-making using only historical datasets without requiring active exploration. Con- sequently, ooolyy learning (OPL) methods, which learn decision-making historical datasets collected under different policy in the contextual bandit framework. have at- tracted attention (Joachims et al. 2018). This area of research could be applied to many real-world interactive systems like recommender systems and ad placement (Saito & Joachims 2021 Aminian et al. 2024).
policies solely from
There are mainly two families of approaches in OPL: regression- approach and policy- based approach. The regression-based method uses a regression estimate that was trained on the data to predict the rewards from the logged data (Sachdeva et al., 2020 Jeunen & Goethals 202T). The policy is then made by simply choosing the action with the highest predicted reward deterministi- cally or from a distribution based on the estimates. A drawback of this approach is the bias formed by the inaccurate regression estimation. In contrast, the policy-based approach aims to update the policy by gradient ascent iterations. The policy gradient needs to be estimated from data using tech- niques like IPS and DR (Dudik et 2014, [Su al., 2020a). These estimators are unbiased under certain assumptions. However, when rewards are only partially observed, they often suffer from extremely high variance (Saito & Joachims) 202T] Sachdevva et al. 2023; Taufiq et al.) (024). One possible approach to address this variance issue is through the use of secondary rewards that are more frequently observed instead, which lowers variance substantially. However, wards often misalign with the target rewards, this suffers greatly from bias. Targeting this problem, HyPeR effectively uses both types of rewards to enhance the estimation of the gradient.
secondary
We did not compare pessimistic OPL methods in our experiments because they are not relevant to our context (Swaminathan & Joachims 2015a London & Sandler 2019] Gabbianelli et al.) 2024 Sakhi et al.) 2024). Our main motivation is to address the prevalent problem of partial target rewards, an issue that pessimistic techniques are not designed solve. However, our proposed method could easily be combined with a pessimistic approach if desired. It is also important to note that when tuning hyperparameters in OPL, there is the issue of overestimation in the validation estimate of the policy value. In such cases, it may be beneficial to use correction methods as proposed in (Saito) & Noouura 2024). Additionally, the line of work on adaptive estimator selection (Su et al. 2002bb] Udagawa al. 2023) is relevant in this context. However, their primary focus is on data-adaptive estimator selection for OPE, rather than on effectively tuning hyperparameters for OPL.
Our work is also related to the multi-objective optimization problems (Wang et al. 2022a Alizadeh cet al. 2023). However, these works solely focus on achieving weights that meet the designer's desired balance of multiple rewards, where the rewards generally have relation to each other. In contrast, our work is crucially different from the fact that we use secondary rewards to enhance the estimation of the target rewards and the combined reward function, particularly in practical situation where target rewards are only partially observed.
A.2 PARTIAL TARGET REWARDS AND SECONDARY REWARDS
Our work is related to various fields that deal with partial observations of rewards due reasons like missing data (Jakobsen et al.] 2017
Wang et al. 2019b Jadidinejad et al., 2019) Christakopoulou 2019; Wang et al. 2019a). delayed observation (Wang et al., 20226 2023 Saito et al. 2024a). data fusion (Imbens et al. 2022). and multi-staee rewards (Wan & McAuley, 2018) Hadash al. 2018). Problems involving partially-observer data have been ex- tensively studied. For example, Liu et al.; (2010) propose to unify implicit and explicit feedback in a recommendation setting, to enhance collaborative filtering. Wan McAuley (2018) have ad- dressed the implicit-explicit feedback problem by modeling them into a monotonic behavior chain and considering them as multi-stage rewards. Christakopoulou et al. (2022) have proposed to de- sign reinforcement learning model that uses user trajectories (secondary reward) to maximize user satisfaction (target reward). Wang et al. (2022b) investigate the relationship between short-term surrogates and long-term user satisfaction.
al. 2022). censoring (Ren al.
15

---

Published as a conference paper at ICLR 2025
In the field of off- off-policy evaluation and learning, perhaps the closest setting to work is LOPE (Saito et al., 2024a), which aims to estimate the long-term performance of the evaluation policy. To achieve this, in addition to the long-term rewards from historical datasets, [Sato et al. (022a) leverage short-term rewards collected in an online experiment. However, while Saito et al. (2244)) specifically focus such setting where short-term online experiments
be done, do not assume access to such online experiment data. In addition, under their formulation, the two types of rewards have no difference in observation density, i.c., the long-term reward is not partial.
B PROOFS AND DERIVATIONS
B.1 PROOF THEOREM I
Proof. To demonstrate the unbiasedness the estimator in Eq. (10) we show that its expectation is equal the true policy gradient given in Eq. (
Ep VoV, D)] ED 1 Σ
E (alm)) [0(zis ()ge((; a)]
+ no (as (2A))
0z To
(r: [( a, (.)ss)) go (z as)
(g(x;, 04 84) a(I,, a:3) go az)
no (a, (r;)
p(olar) no(us
= -Ex())
E (al(a [0(x, (gg,a)))(rc a)]
Ta(a To(a|z)
+ (mo(alx)p((alaaa))
(g(2, s) - [(1 a)) a)) ge(2,
Te(a
(r - ((æ, a 8)) 90 a))
+ Bmm.aaabppdueeeapaaaaaaaahe.eeee
p(olx) To To(a|zz)
Ta(a/>)
= -px()) Enolala) (ā(x, ()) a)) + E Enolal)
(g(x, a, f(x, - ((v, a)) go (0( a)
To(a())
<a(a)))
+ (Boo(a=)m((oa)) p(o|z) mu(a|z) (g(2, a, f(x, - ((x, a, ()), (()
= -Ep(z) Eny(als) [0(x, (((x,))o0(z00 a)] + E Exo(als) (a(z, a, f(x, - g(0, a)) go a)
To(a)z) (q(2, a f(x, a)) 6( (()((=)))))((aaaaaa)))))))ll)l)) (o(x))))))
+ +Bno(alz)
g(x, s(E, [pxx))
VoV((T)
16

---

Published as a conference paper at ICLR 2025
B.2 PROOF OF THEOREM!
2
Proof. Using the law of total variance, the difference in variance of the j-th element of the gradient estimator be decomposed into
(TVD/VoV.DR(Te D)] V VolVoV.(To: D)II)
o p(olz) (
wla, a)g (0) (0, (x, .)) dz a))
(17)
Wowhmmaaaaaeeecaaaaaoa wa,s) w...)) Brocalssl(l (aaeeeeaaaaaaaaaaaaaaaaaaae a)gb (x, a)
+ p(olz) w(x w( (r -(x))(rrrxaaaa)))))) ())9 (c.
2 (mro wlr, a)go w(r,
o
((x,a,))
Perr m.0,)) E Bno(al=)l((1 a) (x. (2.
+ +w(x, ()(z,, 8) ((x, a) a))g (xr, a) p(olz) w(s a) ((x, a, s)) (z, (c,,)]
plojz)
where oa, a, Wfrlz, a, s). and 9. ( (z, a) is the policy score function of the j-th element. We now focus on each the terms. After canceling out the first and the third term, the second term further simplifies using the law of total variance:
E (ppeemotalleoolelm...) p(o|z)2
0²
w²(x, a)ge
( (x, (3,))2a,, s( al,, 0,))
(18)
E Bno a) (1) (z, a)) + w(x, ()(x a, 8) - ((x, a)). (z. a)
+ + (ete)rooall)p)ell...
where p² = Vlolz). and A (x, (( ((x, a). The fourth term can be simplified into
0² p(ooz)2 2² (x, a)9. 6 (z. a)² 2 (0, a, s)²
Bro(ol=) a)g. (z, a)] + w(r, a)(((z) a, -((,)))( (9 (x, (1)
+ [Vle)mm
where ( a. a q(z,, a, s). Thus, we finally have
n(Vo VoV. D)] (TV D)])
(19)
p²
w(1, a)² go a)² (Ag,0-((z, a) Ang(2, p(ooz)z Mo
C ADDITIONAL EXPERIMENTS
C. DETAILED SYNTHETIC EXPERIMENT RESULTS
In this section, we provide more detailed experimental results for the comparisons conducted in Section 55] Specifically, we present results for additional baseline methods, including what we refer to as r-DM and s-DM. These are regression-baats use regression estimates to predict expected target rewards based solely on logging data. The policy then derived by choosing the action with the highest predicted reward in deterministic manner. Note that r- r-DM trains its regression model on a partial dataset due to the use of target rewards, whereas D-DM trains the full dataset but uses only secondary rewards for regression.
methods (also known Direct Methods), which
For the experiment involving weight-tunel methods (experiments with varying 3 values), we also compare the values of parameters 4" and 7" used for HyPeR (Tuned 4*) and HyPeR (Optimal ?*).
17

---

Published as a conference paper at ICLR 2025
Table Comparison of the tuned y values for varying values
Mean 7 w/o Replacement
Mcan 7 w/ Replacement
Mean 7
0.0
0.5595
0.3971
0.4788
0.2
0.6075
0,5762
0.7241 0.5067
0.4
0.7390
0.5542
0.6
0.8681
0.8198
0.6371
0.8
0,9019
0.8771
0.8014
1.0
1.0000
1.0000
1.0000
r-DM
S-DM
r-IPS
s-IPS
r-DR
S-DR
HyPeR(y= 0)
HyPeR(y = B)
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.75
0,50
Target Reward Observatio Probobilyy (ploje)) Target Observation Proboblity (p(otk)
Target Osservatio Probobillty (plo)k))
Comparing the combined, target, and secondary policy values of OPL methods with vary- ing target reward observation probabilities (p(olz)).
Figure 6:
Results. Figures 6 and [ present additional experiments related to the research questions ad- dressed in Section including r-DM and s-DM. Specifically, Figure 6 evvaluaee the policy values under varying observation probabilities for the target rewards, Figure evallatee the methods with different training data sizes, and Figure presents results for different degrees of target-secondary reward correlation. Across all experimental scenarios, we observe that r-DM and s-DM perform significantly worse than the corresponding IPS and DR methods due to high bias in the regression estimates.
Figure shows results with varying true weights 3 in the combined objective. In this experiment, we additionally compare two more baseline methods: HyPeR(Tuned 9 w/o replacement) and DR with F(s,r). HyPeR(Tuned 5* w/o replacement), unlike our tuning method, performs tuning of y using sampling without replacement, i.e, without bootstrapping. We compare this baseline method to empirically show the effectiveness of our tuning method of leveraging the bootstrapping procedure. DR with F(s,r) is another additional baseline that naively combines the target and secondary rewards in its policy gradient estimation; it uses Doubly Robust where the reward is defined as F(s,r)
when 1, and otherwise F(s,r) F(s) in the method s-DR.
In Figure 9 we also observe that the additionally compared baseline methods all performed worse than the corresponding DR methods. The result also demonstrates that parameter tuning with placement empirically outperforms tuning without replacement for a range the true weight / values in the combined policy value objective. Moreover, by comparing HyPeR against the method of DR with F(srr) r), the results empirically demonstrate how combining the target and secondary rewards is indeed crucial and that HyPeR does it effectively.
We also share the tuned 6" values and compare them with the ground truths in Table below.
C.2 ADDITIONAL. SYNTHETIC EXPERIMENTS
In this section, additionally address three research questions: two questions regarding the optimization performance under varying noise levels in each reward type and one concerning the effectiveness of HyPeR with estimated observation probabilities p(o[z))
18

---

Published as a conference paper at ICLR 2025
Combined Policy Value
Target Policy Value
Secondary Policy Value
08
0.75
0,6
06
0.50
04
0,2
400
000
1000
2200
6400
12000
800
1800
3200
6400
12000
100
1000
3200
6400
12000
Training Data Size (n)
Trainigg Data Size (n)
Training Data Size (n)
Figure 7: Comparing the combined, target, and secondary policy values of OPL methods with vary- ing training data sizes (n)
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.6
0.0
0.0
0.2
0.5 Correaatio (A)
0.4
0.5
1.0
0.0
02
06 Coreeation (A)
0.1
08
1.0
0.0
02
0.4
0.6
0.0
1.0
Correlation (A)
Figure 8: Comparing the combined, target, and secondary policy values of OPL methods with vary- ing degrees of target-secondary reward correlation (A). Secondary rewards completely plain the target reward at À = 1, and they are less correlated with smaller
r-DM
s-DM
r-IPS
s-IPS
r-DR
s-DR
HyueRd y w/o replacement
DR with Fls, r)
HyPeR(y 0)
HyPeRly
HyPeR y
HyPeR(Optimal) y')
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.48
0.40
0,22
0.
0.
0.0
0.2
0.6 Seta Vallue (F)
0-4
0.8
1.0
0.0
0.2
0.4
0.6
0.0
1.0
0.0
02
0.4
0.8
0.8
1.0
Deta Value (P)
Bela Valum (B)
Figure 9: Comparing the combined, target, and secondary policy values of OPL methods with vary- ing B, which is weight that balances target and secondary policy values in the combined policy value. Higher 3 means the secondary policy value becomes more dominant.
19

---

Published as a conference paper at ICLR 2025
r-DM
S-DM
r-IPS
s-IPS
r-DR
S-DR
Hy = 0)
HyPeR(y= = B)
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.40
0.45
0.30
0.24
0.0
0.5
1.5 Nolie of (du)
1.0
2,0
2.5
0.0
0.5
1.0
1.5
2.0
25
0.0
0.5
1,3
1.5
2.0
25
NoBia of (O6)
Nass of (0t)
Figure 10: Comparing the combined, target, and secondary policy values of OPL methods with varying noise of secondary reward (o.).
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.5
0.45
0.32
0.24
02
0.0
0.5
1.5 Naise of (c%)
1.0
255
0.0
1.5 Noice of ((0))
1.0
2.0
0.0
0.5
1.0
1.5
2.0
25
Notie of (0%)
Figure 11: Comparing the combined, target, and secondary policy values of OPL methods with varying noise of target reward (o,).
How does HyPeR perform under varying noise levels in secondary rewards? Figure 10 eval- uates the relative policy value when the noise level in the secondary rewards is varied. As the secondary rewards become noisier, their variance increases. Consequently, methods that incorpo- rate secondary rewards in their estimation (i.e., HyPeR( s-DM) perform worse as noise increases. Most notably, the performance of s-IPS and r-DR degrades the most. contrast, while HyPeR- based methods are also affected by the noise, their performance does not degrade as severely as that of the baseline methods that rely solely on secondary rewards, demonstrating its robustness to the noise due to the effective combination of the two types of re- wards.
3), HyPeR(? 0), s-DR, s-IPS, and
How does HyPeR perform under varying noise levels in target rewards? Figure II presents the results for different noise levels in the target rewards. As the noise increases, the variance in the target rewards becomes higher. This affects methods that utilize target rewards in their estimation (i.e., HyPeR(> 39), HyPeR(y 0), r-DR, r-IPS, and r-DM). While HyPeR methods negatively impacted, they consistently outperform the baseline methods that rely exclusively on target rewards, showing robustness as well.
How does HyPeR perform with estimated observation probabilities? To investigate the robust- ness of HyPeR to estimated probabilities p(ooar), we conducted experiments under unknown p (al(z) with varying levels of noise on The results are shown in Figure 12] where the methods that rely on p(olz) are affected by the estimation error and the presence of noise. We observed, however, that HyPeR consistently achieves better combined policy values compared to the baseline methods even under the realistic case with estimated probabilities p(o[z). In particular, HyPeR (Tuned 4T) demonstrates better robustness to the noise in p(o[z), as it can adaptively assign greater weight (via the tuning procedure) to the secondary policy gradient when effective.
20

---

Published as a conference paper at ICLR 2025
r-DM
S-DM
r-Ips
s-IPS
rDR
SDDR
HyPeR(y 0)
HyPeR(y B)
HyPeRiTuned
HyPeR[Optimal y*)
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.60
0.4
0.45
0.45
0.2
0.15
0.0
1.0
3.0 Noise of (ou)
2.0
4.0
5.0
0.0
1.0
3.0 Noise of (oa)
2.0
4.0
5.0
0.0
1.0
2.0
3.0
4.0
5.0
Noise of (7g)
Figure 12: Comparing the combined, target, and secondary policy values of OPL methods with varying noise in the observation probability (0%). Note that the conditional probabilities (p(olz)) are fully unknown and they are estimated based on the logged data.
Table 3: Comparison of of and 5th values for different scenarios.
(a) Varying Target Reward Observation Proba- (b) Varying Training Data Sizes (ni (c.f. Fig-
bilities (p(olz)) (c.f. Figure 13
ure [4]
x
Mean y
Mean 7*
Data Size
Mean y
Mean 8
0.0
1.0000
1.0000
500
0.7205
0.5458
0.2
0.6262
0.5926
1000
0,6612
0.5890
0.4
0.5811
0.5542
2000
0.5678
0,6392
0.6
0,5124
0,5002
4000
0,4462
0.5309
0.8
0.4771
0.4701
8000
0.3356
0.4794
1.0
0.4299
0,4512
16000
0.2878
0.4306
(c) Different Beta Values (3) (c.f. Figure 15
(d) Different Action Sizes ((A4) (c.f. Figure 16
8
Mean 7
Mean 7"
Action Size
Mean 7
Mean 7"
0.0
0.5743
0.5287
25
0,5998
0.5312
0.2
0.6603
0.5833
50
0.6057
0.5159
0.4
0.7193
0.6386
100
0.5686
0.5008
0.6
0.7577
0.6549
200
0.6241
0.4925
0.8
0.8592
0.8021
400
0.5972
0.5328
1.0
1.0000
1.0000
800
0.6124
0.5086
C.3 DETAILED AND ADDITIONAL REAL-WORLD EXPERIMENT RESULTS
In this section, we share more detailed experiment results of the real-world data experiments in Section We add r-DM, r-IPS, s-DM, and s-IPS baselines compared to the main text.
and 15 present more detailed results from the real-data experiments in S including additional methods for comparison. Figure evaluatees the policy values under varying observation probabilities of the target rewards, while Figure [4 evaluates the methods with different training data sizes. Figure 15 evaluates the methods with different true weights B in the objective. In this experiment, we also compare DR with F(s, r) (the yellow line), which serves as a baseline that naively utilizes both target and secondary rewards when available.
Results. Figures
In this section, we also evaluate the relative policy values when varying the size of the action set |A], shown in Figure 16 As the action size increases, the performance of all methods degrades due to an increase in variance in gradient estimation. However, we observe that HyPeR methods consistently outperform all other methods across all evaluated policy values.
B and 3 present the tuned 6* values for the four real-world experiments, com- pared to the trie optimal value 7", averaged over the number of simulations.
Below, Tables 3
21

---

Published as a conference paper at ICLR 2025
r-DM
S-DM
r-IPS
s-IPS
rDD
SDDR
DR with F(s, r)
HyPeR(y
HyPeR(y
(TTned 9*)
HyPeR(Optimall y")
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.6
0.4
0.4
0.0
0.0
0.0
0.7
0.4
0.5
0.8
1.0
0.0
0.7
0.4
0.5
0.5
1.0
0.0
0.7
0.4
0.5
0.5
1.0
Phimary Reward Obsenaation Probabillty golojk))
Phimary Rewardd Otservation Probabelly (olof)3)
Primary Rewardd Obssevattoon Probabilly (elofk))
Figure 13: Comparing the combined, target, and secondary policy values of OPL. methods with varying target reward observation probabilities (p(olz)) on KuaiRee Dataset
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.75
0.7
0,50
0.50
0.25
0:25
0.00
0.00
0.00
500
1000
2000
4000
5000
199000
500
1000
2000
4000
8800
16000
500
1020
2000
4200
5020
18000
Traniin Data Size
Training Data Size (n)
Thiining Data Size
Figure 14: Comparing the combined, target, and secondary policy values of OPL methods with
varying training data sizes (n) on KuaiRec Dataset
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.75
0.75
0.45
0.50
0.50
0.30
0.25
0.25
0:15
0.00
0.00
0.0
0.5
0.0
0.2
0.4
1.0
0.4 0.6
0.8
Beta Vatumm
Bats Value
Valuan (F)
Figure 15: Comparing the combined, target, and secondary policy values of OPL. methods with varying B, which weight that balances target and secondary policy values in the combined policy value. Higher 3 means the secondary policy value becomes more dominant on KuaiRec Dataset.
Combined Policy Value
Target Policy Value
Secondary Policy Value
0.46
0.2
0.00
25
30
100
200
400
500
25
30
100
200
400
600
25
100
200
400
000
Number of Actions ((AI)
Number of Actions (JAI)
Number of Actions (IAI)
Figure 16: Comparing the combined, target, and secondary policy values of OPL methods with varying action size |A| KuaiRee Dataset.
22


Reviews:
[{"review_id": "AogQrQ7xFZ", "reviewer": "Reviewer_nmRS", "summary": "This paper studies off-policy learning with partial rewards. In the considered setting, the rewards are partially observed and the authors assume a secondary reward information.  The authors provide a weighted objective for the original and secondary reward functions based on the off-policy policy gradient algorithms. The authors then give an unbiased estimator of the gradient of the policy value and prove that the variance is reduced if the policy learned by the secondary information is better than the policy only based on the partially observed reward.  A strategy to tune the balancing weight is also given. Both synthetic and real-world datasets are provided to validate the performance.", "questions": "See the comments in weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper has the following strengths:\n+ The paper provides an unbiased estimator for the value based on secondary information and prove that the expected gradient is the same as the one without considering secondary information ($\\beta=0$).\n+ The authors formally prove the variance reduction due to the introduction of the secondary information and show the reduction exists if the policy learned by the secondary information is better than the policy only based on the partially observed reward.", "weaknesses": "The paper can be improved in the following aspects. \n- It is common to consider a weighted combination of the original and secondary information in machine learning algorithms.  Similar strategies are used in few-shot learning [2], informed learning [1], etc. The authors need to compare the proposed algorithm with them and emphasize their uniqueness.\n- The paper considers the setting with some types of reinforcement learning settings and secondary information, and the authors only consider policy gradient, so I would not regard the framework as \"general\". \n-  The strategy to tune the weight $\\gamma$ is straightforward. Can the authors show the improvement in theory?\n\n\n\n\n\n\n\n[1] Von Rueden, Laura, et al. \"Informed machine learning–a taxonomy and survey of integrating prior knowledge into learning systems.\" IEEE Transactions on Knowledge and Data Engineering 35.1 (2021): 614-633.\n\n[2] Wang, Yaqing, et al. \"Generalizing from a few examples: A survey on few-shot learning.\" ACM computing surveys (csur) 53.3 (2020): 1-34.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eazDqRmgz1", "reviewer": "Reviewer_G83n", "summary": "This work presents an off-policy learning approach to efficiently optimise partially-observed reward. The authors first model the data generation process of this problem, assuming access to dense, secondary rewards that are correlated to the target reward. They introduce HyPer, an unbiased policy gradient method that leverage secondary rewards to better optimise the true target. Experiments validate the approach on synthetic and semi-synthetic problems.", "questions": "In addition to the weaknesses, I have the following questions:\n- How can we test if the DGP hold in real world problems? The real world experiment section in your paper is still a synthetic experiment simulated (with the DGP you suggest) on real world data.\n- In real world scenarios, we never have access to the observation probability $p(o|x)$. How do you estimate it in the real world? And what is the impact of its estimation in the efficiency of your algorithm?\n- Why did you need to introduce the combined policy value? And why did you define it with the sum of secondary rewards? In addition, It is better to change the name to \"policy combined value\", \"policy secondary value\" as it reflects more the quantities defined.\n- The baselines s-IPS and s-DR use aggregation function $F(s)$ over the secondary rewards, why they do not use also the primary reward when available, giving $F(s, r)$?\n- If it is difficult to derive the theoretical value of $\\gamma^*$ as defined in your paper, one can derive the theoretical value of $\\gamma^{**}$ to minimise the MSE of the gradient, did you explore this path? \n- There is a new line of work [2] that optimises for long term reward, using dense, secondary/surrogate reward, is there a reason why you did not compare to it?\n- Is there a reason why you did not include the large spectrum of pessimistic OPL? \n\n[2] Long-term Off-Policy Evaluation and Learning. Yuta Saito, Himan Abdollahpouri, Jesse Anderton, Ben Carterette, Mounia Lalmas", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and is easy to follow.\n- The problem of partially observed/delayed reward is of high importance.\n- The formulation of the data-generation process is simple and can model diverse problems.", "weaknesses": "- The contribution of the paper is twofold, first, the introduction of the framework of partially-observed rewards with presence of secondary rewards, that comes with a new data-generation-process (DGP), and secondly, the derivation of an unbiased policy gradient that can efficiently leverage all information to optimise the target. These two main contributions present some weaknesses:\n   + The framework's data generation process (DGP) is **restrictive and was not properly tested**: If the DGP models missing, non delayed rewards, the claim that it can model delayed rewards is not supported, as these rewards can be observed after many interactions with the system and attribution of the reward to a certain action becomes the main difficulty [1]. \n   + The policy gradient approach relies on $p(o|x)$ **which is unknown**, thus the PG is never unbiased as we need to model $p(o|x)$. In addition, this quantity can be really hard to model in real life scenarios (For example, we do not know what explains why a user did not purchase an item after he clicked on it).\n- The introduction of the combined policy value overloads the paper, and the story would have been simpler by defining a clear goal, which is to optimise the target reward, considered the north start in the majority of applications. \n- There is a problem in Equation (10), which is the core result of the paper. The gradient boils down to a doubly robust gradient that does not use any secondary reward. This is a major typo that needs fixing.\n- The paper does not discuss important, theoretical contributions to OPL, even in the extended related work. These contributions are based on the pessimism principle and were proven to be optimal, contrary to using naive policy gradients. The following lines of work should be incorporated to the extended related work and discussed:\n   + Counterfactual Risk Minimization: Learning from Logged Bandit Feedback. Adith Swaminathan, Thorsten Joachims\n   + Bayesian Counterfactual Risk Minimization. Ben London, Ted Sandler Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4125-4133, 2019.\n   + PAC-Bayesian Offline Contextual Bandits With Guarantees. Otmane Sakhi, Pierre Alquier, Nicolas Chopin Proceedings of the 40th International Conference on Machine Learning, PMLR 202:29777-29799, 2023.\n   + Importance-Weighted Offline Learning Done Right. Germano Gabbianelli, Gergely Neu, Matteo Papini Proceedings of The 35th International Conference on Algorithmic Learning Theory, PMLR 237:614-634, 2024.\n   + Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning. Otmane Sakhi, Imad Aouali, Pierre Alquier, Nicolas Chopin.\n\n[1] Fixed point label attribution for real-time bidding. Bompaire et. al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5FCsb1WAs8", "reviewer": "Reviewer_8JAf", "summary": "The paper addresses the problem of off-policy learning in contextual bandits where rewards are only partially observable. In this setting, the authors propose using an auxiliary variable to reduce the variance of the double-robust inverse propensity weighted (IPW) estimator for the reward. They introduce a Hybrid Policy Optimization for Partially-Observed Reward (HyPeR) estimator, which incorporates an auxiliary-conditioned reward estimator to refine average reward predictions. Theoretical and empirical results demonstrate that when the auxiliary variable is highly correlated with the target reward, the HyPeR method surpasses baseline approaches in performance.", "questions": "No further questions. My score reflects my current concerns about the technical soundness of this paper. I'm open to further discussions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Incorporating auxiliary variables for variance reduction is a well-established idea in the ML community. However, the authors innovatively integrate this concept with the doubly robust estimator to tackle the challenge of limited data coverage under partially observable rewards—a contribution that appears novel. The experiments further support the theoretical insights, and the paper is generally clear and easy to follow, aside from a few errors and typos.", "weaknesses": "I noticed several potential technical issues (or possibly typos) that may impact the rigor of the results:\n\n1. In eqn (6), you write down the distribution in a way that allow $r_i$ to also depend on $o_i$. This will cause the problem of confounding bias, which does not give rise to an unbiased estimator of $q(x, a)$. See Kato et al. 2021 or Pearl, 2009 for details. Is this a typo or is there something missing? I found in the proof of Theorem 1 that you indeed assume $r$ to be independent of $o$ when conditioned on other observable variables $(x, a, s)$. If this is an assumption, you should clearly state it.\n\n2. In Eqn (10), you perhaps miss out a $o_i / p(o_i\\,|\\, x_i)$ term in the second line, as you cannot evaluate this term when $o_i = 0$ and $r_i$ is missing from the observational data. \n\n3. In the proof of Theorem 2, what are $s_\\theta^{(j)}(x, a)$, $\\sigma^2(x, a, s, o)$? Are they also typos? You also miss out a variance symbol in the first line of the proof. \n\n\nDespite these concerns, I believe the variance reduction claim remains valid, as the paper’s approach can be interpreted as an application of the well-known Rao-Blackwell theorem.\n\n***References***\n1. Kato, M., Imaizumi, M., McAlinn, K., Kakehi, H., & Yasui, S. (2021). Learning causal models from conditional moment restrictions by importance weighting. arXiv preprint arXiv:2108.01312.\n2. Pearl, J. (2009). Causality. Cambridge university press.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IYAKfxwZDx", "reviewer": "Reviewer_tksu", "summary": "The paper deals with the problem of Off-Policy Learning (OPL) in the contextual bandit domain. They focus, in particular, on the case where a secondary reward is fully observed while the target reward is, in general, partially observed. This is the case in many real-world applications, such as recommendation systems. The paper introduces Hybrid Policy Optimization for partially observed Reward (HyPeR), a method for improving off-policy learning (OPL) when target rewards are partially observed. By utilizing secondary rewards, the method aims to reduce variance in policy gradient estimation, thus improving learning efficiency. An empirical analysis (on synthetic and real data) is presented to support theoretical findings.", "questions": "- Do you agree that the cross-validation procedure that you proposed is biased by design? In my opinion (if I am correct), you can still leave the section as is, but just acknowledge this caveat. Indeed, empirically, this procedure seems to work despite being biased, so it is not a big deal.\n\n- Can you elaborate a bit on the design choices for the secondary rewards in your experimental settings?\n\n- Can you provide ablation studies on the secondary reward dimensions? I would like to see which dimensions are responsible for the HyPeR good performance. My intuition would say that, in the real data case for example, it is because of s_1 and s_2, since you can almost construct the target reward r based on those. \n\n- Can you provide an ablation study on what happens when you perform cross-validation with sampling without replacement? You claim that it is important to have sampling with replacement. Do you have empirical data to support this claim? It can be useful for researchers in the future\n\n\nFinally, it is not a question but more of a comment:\nYour estimator of the policy gradient also seems robust to missing secondary rewards: if you set \\hat{q}(x,a,s) to resort to \\hat{q}(x,a) when there are missing s (for any reason), your estimator reduces to Doubly Robust, which is still an unbiased estimator.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Addresses a relevant issue in OPL with partially observed rewards: the issue is very relevant, since there are many real-world application domains where the target reward may be partially observed, but there are a lot of secondary reward signals available. This paper makes good use of those signals and practitioners may be interested in this\n\n- New estimator with theoretical properties: the HyPeR estimator for the policy gradient is still unbiased and provably reduces (as long as \\hat{q}(x,a,s) is more accurate than \\hat{q}(x,a)) the variance of the estimation by introducing a sort of \"control variate\" using the secondary rewards. This is an important contribution.\n\n- Empirical analysis on two datasets: I like the fact that the empirical analysis is done on both synthetic and real data, and also with a lot of different settings (e.g., varying data size, secondary-target reward correlation, etc.).", "weaknesses": "- Section on the tuning procedure: in my opinion, this section may be a bit misleading (not intentionally, of course). This is because the authors claim that we can simply apply a cross-validation procedure to tune a hyperparameter as we do in the supervised setting. However, this is not so trivial as they claim: the fact that we are dealing with an Off-Policy problem means that the data is not iid with the target data. Hence, performing a cross-validation on such data will not give an unbiased estimation of the performance of the target policy. Nevertheless, it may be a heuristic way of performing hyperparameter tuning (I've seen papers in the OPE/OPL domain doing it), but it must be acknowledged that it is a biased way of performing hyperparameter tuning.\n\n- I am not convinced of the design choices for the secondary targets in the experimental setting: let us consider the real data experiment. The authors say that the target reward is the watch ratio r. We would like however to provide a good PG estimate even when we do not observe this reward. They decide that they will do it with secondary rewards that they design to be *very* correlated with r. In particular, with s_1 and s_2, you can almost reconstruct r. Hence I am wondering if this experiment is really meaningful, since we would like to see what happens when we do not observe r, but we almost exactly observe it (albeit quantized: if s_2 = -1, then r < 0.5, if s_2 = 0 and s_1 = 0, then r \\in [0.5, 2), if s_1=1, then r>= 2).\n\nA very minor issue that did not impact the score:\n- If I recall correctly, you never mention the fact that you focus only on contextual bandits in the introduction. Perhaps it is better to mention it somewhere in the intro, otherwise readers may be confused and think you are focusing on full RL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)