MultiScale Contextual Bandits for Long Term Objectives
Authors: Richa Rastogi
Yuta Saito
Thorsten Joachims
Venue: NeurIPS 2025 poster ()
Keywords: policy learning at multiple timescales
timescale of interventions vs feedback
off policy contextual bandits
Primary area: applications
TL;DR: This paper studies the disconnect in timescales of short term interventions and the long term feedback. As a solution it introduces a framework to learn interventions from the corresponding timescale feedback at multiple interdependent levels.

Abstract: The feedback that AI systems (e.g., recommender systems, chatbots) collect from user interactions is a crucial source of training data. While short-term feedback (e.g., clicks, engagement) is widely used for training, there is ample evidence that optimizing short-term feedback does not necessarily achieve the desired long-term objectives. Unfortunately, directly optimizing for long-term objectives is challenging, and we identify the disconnect in the timescales of short-term interventions (e.g., rankings) and the long-term feedback (e.g., user retention) as one of the key obstacles. To overcome this disconnect, we introduce the framework of MultiScale Policy Learning to contextually reconcile that AI systems need to act and optimize feedback at multiple interdependent timescales. 
Following a PAC-Bayes motivation, we show how the lower timescales with more plentiful data can provide a data-dependent hierarchical prior for faster learning at higher scales, where data is more scarce. 
As a result, the policies at all levels effectively optimize for the long-term. We instantiate the framework with MultiScale Off-Policy Bandit Learning (MSBL) and demonstrate its effectiveness on three tasks relating to recommender and conversational systems.

MultiScale Contextual Bandits for Long Term Objectives
Richa Rastogi
Yuta Saito Department of Computer Science
Thorsten Joachims
Coreell University
Abstract
The feedback that Al systems (e.g., recommender systems, chatbots) collect from user interactions is crucial source of training data. While short-term feedback (e.g., clicks, engagement) is widely used for training, there is ample evidence that optimizing short-term feedback does not necessarily achieve the desired long-term objectives. Unfortunately, directly optimizing for long-term objectives challeng- ing, and identify the disconnect in the timescales of short-term interventions (e.g., rankings) and the long-term feedback (e.g., user retention) as one the key obstacles. To overcome this disconnect, we introduce the framework of MultiScale Policy Learning to contextually reconcile that Al systems need to act and optimize feedback at multiple interdependent timescales. Following a PAC-Bayes motiva- tion, we show how the lower timescales with more plentiful data can provide a data-dependent hierarchical prior for faster learning at higher scales, where data is more scarce. As result, the policies at all levels effectively optimize for the long- term. We instantiate the framework with MultiScale Off- Off-Policy Bandit Learning (MSBL) and demonstrate its effectiveness on three tasks relating to recommender and conversational systems.
Introduction
Many interactive Al systems (e.g., recommender systems, conversational systems) use abundantly collected short-term feedback for learning. However, it is well known that over-optimization of short-term feedback can adversely affect the long-term goals [4 21 26). Similarly, challenges with competing time horizon objectives 440 [5), and issues like reward hacking [35] [2 30 and user
manipulation [8 have been previously reported. For example, optimizing for engagement on social media platforms can lead to clickbait-y or toxic feeds. This neither reflects the user preferences nor the platform's i goals of retaining users. We wish to design systems that optimize for long-term objectives, that are beneficial for various stakeholders in the system
key problem in achieving this goal is that the long-term feedback (e.g., user retention) is different timescale than the short-term interventions (e.g., rankings of recommended products) that are used to optimize them. For instance, optimizing rankings for clicks recommender system relatively straightforward since clicks can be attributed to an individual ranking. Indeed, learning methods such bandits that optimize for users' immediate response recommendations such as clicks, likes, views, etc are widely used to learn ranking policies [9 46). Now, consider optimizing rankings for the feedback on users' subscription renewal, which is observed monthly. We note disconnect between the timescale of rankings presented and the long-term objective of subscription renewal, making this a much harder problem than the previous one. While a Markov Decision Process in principle model sequential dependencies to achieve long-term goals, in practice the large resulting state spaces, moddit-assigmment problems, and the sparsity of long-term feedback prohibit straightforward applications of reinforcement learning 23
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To overcome this problem, our approach is to contextually reconcile the disconnect between short-tern interventions and long-term objectives by learning interventions and policies at multiple timescales. Consider a recommender system for a video streaming platform as depicted in Figure | At the lowest level, the system acts by providing rank- ings of recommended products. These rankings (short-term interventions) can be optimized for clicks (short-term reward). While clicks are an important signal for learning that is abundantly available at the lowest level of Figure I an unmit- igated maximization of clicks is not necessarily aligned with higher-level goals. In particular, the platform may sacrifice some clicks if that leads to a higher weekly return rate. This metric is a more reliable indicator of user satisfaction, but it is available at a slower timescale. Finally, at the highest level, the platform ultimately cares about user retention and subscription renewal. This feed- back lives at an even slower timescale, and it is thus more scarce but even more valuable. Similar multi-scale levels of feedback metries also exist for other AI systems (e.g., tutoring chatbots that aim to achieve long-term learning outcomes), and additional settings are discussed in Appendix B
Interventoos
Feedback
Level (Long Termi
Subsecripioon Remerall
0"
Phransta lo ramard whotier Mod and furn
Lovel (Modiuu Term]
Weekly Return Rate pf2
Dacoding Stnttay Deccsinng
(Short Tarmy)
Engagement
Resporue ganaration
a" p y
Figure 1: MultiScale feedback r with cor- responding interventions a at each level (L1, L2, L3). At the short-term level, engage- ment feedback (e.g., responses, clicks) is ob- served at the fastest timescale. At the next higher level, we observe feedback like the weekly return rate. And at an even higher level, subscription renewal is observed at the slowest timescale.
The main contributions of this work follows:
1. We introduce a new framework - MultiScale Policy Learning that formalizes a multi- level approach for optimizing long-term objectives. The framework introduces a recursive construction of priors that are informed by data at the lower levels to speed up learning at the higher levels.
2. We demonstrate the practicality and generality of this framework by developing MultiScale Off-Policy Bandit Learning (MSBL) as a simple recursive algorithm for training multi- scale contextual bandit policies. It includes two widely applicable constructions for nested training of policies that enable the use of both abundant short-term and sparse long-term data for optimizing long-term outcomes.
3. We demonstrate the effectiveness of our approach empirically on three tasks ranging from recommender to conversational systems. Ablations show robustness of our method for optimizing the long term reward.
2 Related Works
Hierarchical Reinforcement Learning. Hierarchical RL (HRL) approaches, such as the options framework [37 and feudal learning [7], learn to operate on different levels of temporal abstraction. HRL aims to accomplish complex tasks by dividing them into sub-goals with the higher level assigning sub-tasks to the next lower level, such as in robotic applications [31]. While our framework shares the idea of creating hierarchy, the type of hierarchy fundamentally different, In particular, our tasks do not have subgoal decomposition as in HRL, where the discovery of the subgoals and their execution critical and guides the micro policy. Instead, we exploit the hierarchy of feedback timescales to construct hierarchical prior over the policy space to speed up learning for sparse long-term feedback. Our goal is to steer towards improving long-term outcomes, even at the expense of shorter-term rewards. This is fundamentally different from the macro actions (options) in HRL, which are abstractions over micro actions, and the goal is to combine these options as subroutines.
Long Term Optimization. There has been growing interest in the study of recommender systems that go beyond optimizing engagement and clicks 25 5 23 26 2 Maystre et al. (23 provide an RL perspective to long- tergtermmiee learning in recommender systems and discuss challenges with credit attribution. Other approaches, such as 5 do not learn at the macro level and instead formulate long-term macro interventions that they aim to fulfill with minimum impact on short-term engagement metries. These works operate on a single timescale and assume that the macro intervention given and fixed. Our work can be viewed as learning the macro interventions themselves from long- term

---

feedback. In fact, these methods can be used for single level-specific learning within our MultiScale framework, making them special case our approach.
Multi-Objective Optimization, Recent work has highlighted the importance of selecting weights for linear scalarization for multi-objective learning in recommender systems [27 16 42) and in text generation tasks 43 A different line of work explores learning conditional policies as a single Our work leverages the idea of conditional policies and multi-objective learning from the perspective of optimizing for long-term outcomes. In doing so, we elevate single- stage learning to multiple levels with the macro level contextually selecting the objective to optimize at the micro level.
family of policies 9 42 43
Additional related works are deferred to Appendix C
3 Multi-Scale Policy Framework
We begin by providing a PAC-Bayesian motivation for why a hierarchical approach that exploits feedback across multiple scales can be substantially more data efficient. For simplicity of notation, we restrict to two levels - the micro level operating at the faster timescale /1 and the macro level operating the slower timescale 12. As we will later, the framework naturally extends to multiple scales of policy learning.
Our goal is to learn policy =<a|1; (z) that selects action for given context We assume that contexts are drawn i.i.d. from an unknown distribution x ~ p(z), but we conjecture that our approach can be extended to stateful models well. Our approach rests the realization that observe rewards at different timescales in many Al systems. For two levels, we observe a reward
L p(r p(r" Iz a)
at the micro level for each action a and context corresponding to the short-term engagement (e.g.. clicks). We also record rewards
p22 12pp(r (x,, at). (II+T, ((+T)
after we have taken a sequence of actions. This corresponds to the long-term feedback (c.g., weekly returns, subscription renewal) which we would like to optimize, since it typically better reflects stakeholder objectives. This leads us to the following policy learning objective,
π arg max (TT(, where V (T) B aeaar,, (2)). Tell
Unfortunately, for a large and complex policy space II, finding the optimal policy x by simply replacing the expected reward (1T with its empirical estimate v-I.2(T)) on some training data D is typically intractable. The long-term reward is too infrequent for learning policies from scratch. As result, existing approaches predominantly 44 optimize the more frequent reward signal ,.41 the micro level.
L1 arg max (Ll(m,, TIll
(T)) where VL Ez,a, [n-1]
Note that even the policy π Li* that perfectly optimizes the micro level reward can have substantially worse reward the macro level, V (z= <,,2(, However, 1,41* is typically much better than random policy from II. This raises the following question.
How do exploit feedback at the micro level to learn faster at the macro level? To provide theoretical motivation, we make the following PAC-Bayesian argument. PAC Bayes generalization bounds 20 24 provide uniform convergence over all posterior distributions Q(T)) for any given prior distribution (TT). Specifically, with probability - 5,
S O [I(QII))++(110))))) ln(1/8)
E Ex L2 E,,
n
Since this bound holds for all Q, it also holds for any (approximately) optimal posterior Q12+ that maximizes the macro level reward. For discrete policy space, Q22- is the Dirac delta distribution centered on L22++ The bound states that this learning problem is "easy' (i.e., requires small number of training examples) if the KL-divergence between QL2+ and the prior P is small.
3

---

Can we improve the prior for the macro level with data from the micro level? While the optimal policy x flle the micro level may be suboptimal at the macro level, policy that is learned based on finite data the micro level via i <- arg = LLL (x) can provide useful prior information for learning at the macro level.
Consider the following illustrative example, where each policy T(-1,,0) 0) II defined via parameter vector 8. We denote 6t1 as the parameters of policy f'1, and g as those of the optimal macro policy x For simplicity of demonstration, we define the target policy distribution as Q'22+ N( ) an uninformed prior distribution Po N(. Σo) for some arbitrary 9g and an informed prior pil N(
Σ centered at the learned micro policy # The difference in training samples noo - 22 to get the same confidence interval for Q42- in Equation (3) is proportional
p We show Section t.a for an appropriately chosen E the improvement in required training samples by moving to the informed prior PL1 is at least
K KL[Q Pa) K [(Q (
no - n2 KL(Q421 Pa) KL(Q4> (PL)) E O(1022> BolMr - |0L2 (4
(4)
where 10L2* Oolar is the squared Mahalanobis distance in the parameter space This shows that the policy #21 ~ pL1 learned at the micro level can provide training sample savings at the macro level, if L and 222 are close compared to the uninformed prior Po. In particular, if we can learn almost all parameters of π 1.2+ at the micro level, the distance ( gh | will be small, resulting in higher macro level sample savings. The figure on the right illustrates how informed prior PL1 pulls the center of the prior Po closer to the parameters of π 222 ~ Q12+.
pl
Q
Po
To quantify the reduction in macro level training data, we construct numerical example (detailed in Section E with I E R such that 49 parameters are learned well at the micro level and only parameter needs to be adjusted at the macro level. This results in saving = 98% training samples which means gathering enough macro level data may only take weeks instead of years.
4 Multi-Scale Policy Learning
In order to put the theoretical insights from the previous section into a practical policy learning algorithm, we introduce factorization of contexts and policies at each level. We propose the policy factorization in a way that the micro level learns a large part of the parameter space, even if it is not aligned with the long term expected reward. This simplifies learning macro level compared to learning from scratch.
Multi-Scale Contexts. The context x ~ p(x)) can be factorized as p(x) é p(x p(x''112> At the upper level, contexts x2 2(x²2) arrive at timescale t2. An example of an upper-level context could be a user as described by demographic features or some long-term profile. For each such upper-level context, a sequence of lower-level contexts = ~ p(x"'1 (22 is drawn conditionally on ,.!2 Such lower-level contexts could be search queries, or chat requests.
Multi-Scale Policies. We consider the following factorization of policy space IT,
(5) where II consists of miero policies, that as we will see later, can provide strong inductive bias for the long term optimal policy. In contrast, the macro level is only concerned with learning within the space of reasonable policies obtained after micro learning and has much smaller policy space I
II 2 11 12
Specifically, for each upper-level context x222 the upper-level policy T22 selects an action a² from action space AL2, Exxmmles of a2 are diversity boosts, aggressiveness of spam filtering, the decoding strategy of an L.M policy, etc.
a L2 (a (12)
LL2 42
Importantly, as shown in Figure (a), the action مع corresponds to a lower-level policy 411 This means that the action space 12 the upper level isomorphic to family of policies f { L2 E AT.2), learned empirically the lower level.
A 1.2
'This calculation does not consider the investment of training samples n for constructing pL), Generally, these n 41 samples are significantly cheaper than the macro samples n. L2 since they are I' times more frequent.

---

Micro Level Data
Macro Level Data S D
D
Macro action indexes to select the micro policy
Policy
Feadback
Modification
L #L1
Macro Level Palicy Space II 11
III41
sf
1741
4f = fi
L2 = f
Micro Level Policy Space
(x) MultiScale policies (Inference)
Learning family promising policies ft using micro level data
Learning policy g
Figure 2: (a) At inference, macro action indexes to select the particular micro policy from family of micro policies. The macro action space A is isomorphic to the family of policies f' (b) Learning micro policies: Abundant micro-level data is used learn promising policies Ti using policy or feedback modification (c) Macro-level data is used to learn a macro policy. For more than two levels, (b) and (c) recursively called, narrowing down micro policy space/ macro action space.
The lower-level policy T indexed by the chosen upper-level action a.2 will be executed for the subsequent lower-level contexts.
a ~ 1. (att/zz))
The actions a E A41 at the lower level are rankings, chat responses, or push notifications, just like in the conventional contextual bandit framework.
Multi-Scale Data. We collect contextual bandit feedback at both the upper and the lower level. At the lower level, we get the conventional data
D41 = [(x a.. .r 11))1* )
where we use the logging policy To" and record the propensity pf Fh (a. (x.-1) to enable off- policy learning. However, unlike for conventional bandit policies, we also get data for the upper-level, which includes the long-term rewards
D = {x
L1 L2 Mapz, P
12)) with propensity p! = m (a (2.2)
We thus need to devise an algorithm for learning both #22 as well each of the policies in A11 We approach this problem by using the upper-level data D22 for learning a²2 and we use the lower-level data D41 to learn a comparably small set of policies 11 that serve as actions for the upper-level. This provides the affordance to use the abundant feedback in DL1 for learning in the large action space A thus narrowing down the set of potential actions for learning N from the comparably scarce data in 1 as shown in Figure 2 (b) and (c).
4.1 Learning Family of Mícro Policies
The following proposes two options, which both define a comparably small 4 based the more abundant data D4i available at the lower level.
Policy Modification. In this procedure, modify to define the family ii In particular, we first train 411 to optimize the expected reward at the lower level according to Eq. (2). This single policy then modified by each action a 42 where each action takes the form of a function from [ - A141
first learn single policy x41 from DL1 that then
1.1 = a (*
In the example of text generation, a decoding strategy L2 might modify the learned LLM policy for more varied response generations. Similarly, applying boost 12 items of a particular type in recommender system updates the ranking policy x I.1. by ranking boosted items higher up, or
items suspected to be click-bait lower. As result, Eq. O defines policy update *L1 for given
5

---

intervention a2 as form of perturbing the short-term optimized policy # While 724 could provide lower expected short-term reward compared to that from a it can be more effective at optimizing the reward at the upper-level (c.g., fewer clicks by more aggressively pruning suspected click bait can lead to better weekly returns).
Feedback Modification. In this procedure, each upper-level action '2 takes form of a loss function that acts upon the observed feedback at the lower level when learning the lower-level policy. In this case we assume that the feedback 1. is vectorial (e.g., clicks, likes, purchases, add-to-carts, and each 22 is different function (e.g.. convex combination) for combining the feedback vector into scalar loss. Then, for any given L2, we optimize
T = arg max (alll (1221)
x ena
to get a family of policies fi
For more efficient implementation that does not require us to explicitly enumerate the policies in f1-1, we include 22 in the context and parameterize the reward (12(1 during training n for every a_22 E A442, In this way, we only learn single policy that parameterized by a to represent all policies in fi At inference, a2 chosen from the learned macro policy is included the context of the micro policy, selecting the particular 4242-
L1 As a result, while Eq. (7) in principle refers to
discrete set of policies, practically we only train one micro policy. For LLM policies, this can be implemented as described in 43
Note again that the transformation of the rewards can lead to lower short-term reward some primary metric (e.g.. clicks), but that the upper-level policy now has space of actions that can optimize the longer-term metric (e.g., user refention).
4.2 Learning the Macro Policy
Once the family of policies fi that correspond the upper-level actions a E A is fixed, can chose from a wide range of policy-leariing methods that use the data D to optimize the expected upper-level reward V V12 (172)). In particular, that optimize the inverse propensity weighted empirical average
use off off-policy policy-gradient methods
L2 10.2110
10 0) r
L2 arg max
L2
8
L2
7 (11-00))
as an estimate of the expected upper-level reward ( (2) the policy 422 (1).,, 0) is differentiable in its parameters 0, we can use stochastic-gradies descent for training [17
4.3 MultiScale Bandit Learning Algorithm
We can now summarize our approach to learning nested set of policies across multiple levels in Algorithm I The algorithm uses off-policy contextual bandits [10] 38 33 at each level. Algorithm [ limited to two levels for conciseness of notation, but the full recursive procedure for an arbitrary number of levels is given in Appendix D.2 In the experiments, we will explore policy spaces with two and three levels.
Algorithm 1 MultiScale Training: Off-Policy Contextual Bandits
Procedure PolicyL .mbb)
Collect Micro Logged dataset D41 = {x att ; rft .rl M ~ mb1
Learn Micro policies fFg. (6) or 7 using DLI)
Collect Maero Logged dataset D (xf22 a , po
- ) mp
Leamn Macro Policy a +- arg m
122 12, D 122) (Eq. 8
return learned policies 1.2 f
The procedure requires the logging policies m L2 I1 as input. We first collect logged bandit data DLI and learn the micro policies either policy feedback modification to get family of policies

---

it' := a E J With the learned policies ff we now collect logged bandit data D for the macro level. Note that the logged datasets D and D can be collected asynchronously, because during micro policy learning, we learn 11 for all a E AA?
Overall, at each lower level T(k 1), we use data DL(k-1) to learn the promising policies (k=((-1) that provide prior information and serve as the action space for the next upper level L(k). While the training involves bottom-up learning of the policies, deployment involves top-down inference from learned policies at each of the levels. Figure (a) shows the inference with actions chosen from the upper-most level policy, indexing the next lower level policy. For e.g., a² is first selected from the learned 7 (.(xx22 and the lower level action at1 is selected according to the indexed policy * (.1...). In this way, learning family of policies enables adaptation (according to the intervention a 2 for the miero level at inference time. We provide formal inference algorithm in Section D
Since we use off-policy learning independently at each level, convergence for the contextual bandit learning at each level depends on the number of samples and the action space [1O]. As result, with smaller macro action space A compared to A we can utilize the scarcer data at the upper level and get similar convergence as the micro policy that utilizes the more abundant data.
5 Experiments
We conduct experiments on three scenarios related to conversational systems (Anthropic Helpful Assistant 4 and new simulator), and a conventional recommendation system (KuaiRand video streaming benchmark) for two and three timescale levels. We provide additional experiment on a toy domain that is simple enough to make RL tractable Section F. and as expected, we find that our approach is competitive. In the following experiments, we compare our approach against single-stage policies that cannot personalize at the macro level, a random baseline policy (denoted by a²2 TT²2) that selects interventions uniformly, and oracle skyline policy. We subscript index to denote the static policies, c.g., af² is fixed action policy that always applics first macro intervention.
Complete experiment setup, hyperparameters, and training details are provided in Section F
5.1 Multi-turn Conversation
Figure 3 (a) shows two level setup for multi- turn conversations. In this task, we learn the preference weight vector a 12 for harm- lessness and helpfulness with a bandit policy " 222). The macro intervention L42 is applied as a feedback modification to the lower level LLM policy. The upper-level con- text مب- represents a user persona, such as "Child" and "Expert". At the micro level, the context =11 starts from a question in the An- thropic dataset. For each subsequent turn, the trained LLM policy - responds, and the user persona LLM asks follow-up question. The short term observed reward is the user LLM's evaluation of a single-turn. We simu- late five turns of the prompt-response cycle. At the end of the five turns, an LLM evaluator at the macro level scores user satisfaction for the given user persona and full conversation in {0, 1}.
Long Term (Multi- turnl (Muli-tuurll Rewardd
0.00
MSBL
User
Short tarm (Singie-tumi Reward
(b) Figure 3: Multi-turn conversation: (a) Setup for learning preference weights a²2 using feedback modification (b) Comparison of long-term (user sat- isfaction of multi-turn) vs short-term (single-turn) rewards for all users across 5 random seeds.
(a)
Results. Figure 3 (b) shows that our approach of learning a macro level policy that selects the helpful/harmless tradeoff 122 for each user persona based on macro level feedback achieves the best long term reward for the overall conversation as compared the con-adaptive baselines. Optimizing only the per-turn response can adversely affect the overall conversation. This is due to harm hanu-inducing responses, which provide acceptable individual answers at the lower level but adversely affect the conversation over five turns. This experiment demonstrates that macro level learning is crucial and that MSBL effectively learns to optimize for long-term reward even when the reward is highly non-linear (i.c.. Llama-3-70b [3] evaluations). Further results are given in Section F4
7

---

MSBL
mi
Return Rate (Longer Term))
Oracle
(Subscription Renewal)
Long Term
Level 3 opt. (MSBL)
af
Level Reward
a5a
Level 2 opt. (MSBL)
at²
opt.
0.6
0.5
Short Term (Egagemeent)
Medium Term (Weekly Return Rate)
0.45
0.50
0.55
0.60
9.70
Engagement (Short term) Level 1 Reward
(a)
(b)
Figure 4: Conversational recommender system: (a) Tradeoff between longer-term Level and short-term Level rewards using decoding temperature 222 E {0.0, 0.2, 0.4, 0.6, 0.8, 1.0) as policy modification. (b) Tradeoff between expected rewards at all three levels. Expected rewards are reported across random seeds for all users.
5.2 Conversational recommender system
To increase the complexity of the experiments and evaluate more than two levels of feedback, we built simulated conversational recommender system based the motivating example in Figure [ We simulate 1500 users for training and test 300 users. At the lowest level, we use a pretrained LLM policy N LLLM that acts personalized agent, generating cuisine suggestions y users at L1 timescale ě = (1,...10) Each query consists of a system prompt specifying the agent's 's expertise and query q. We learn bandit policy T (a="1>211 that selects the particular LLM agent d1 according to user context x"1 Next, we generate response ~ π (.
3/- from the LLM policy, given the agent selection a²1 and append the previous timestep response yr-1 the current query q. The L1 reward is the inverse perplexity conditional on the optimal action, which simulates the engagement (relevance) metric. At the second level, we simulate two user groups with unknown preferences for relevance and diversity. L2 reward is a non-linear function of relevance and diversity, representing weekly return rate at every 10 timesteps of L1. The L2 policy n selects the decoding temperature a² E {0.0, 0.2, 0.4, 0.6, 0.8, 1.0] as policy modification to the LI policy. At the third level, we simulate two user groups, with different long term preferences and use feedback modification for L2 training to get a family of policies fi = (# LL E A
Results with two levels. Figure[[ illustrates the tradeoff between Level and Level 2 expected rewards. While greedy decoding with af provides the best overall short term reward, leads to low longer term reward at Level 2. Stochastic decoding temperatures a! through ag applied to the lower level policy improve the longer term reward depending on the user group but fall short. MSBL learns nested contextual policies A and a 222 to achieve nearly optimal longer term reward for all users in the system with little sacrifice to the short term relevance of responses. In Appendix F.3 we analyze the performance for each of the user groups well as robustness of MSBL for varying feature noise in user contexts both levels. We find that MSBL learns the weighted preference of relevance and diversity for each user group and is robust to noisy features.
Scaling three levels. Next, we analyze all three levels and compare short, medium, and long-term expected rewards in (b). First, note that the corners of the simplex are red, blue, and brown corresponding the Level 3 opt. (MSBL), Level 2 opt. (MSBL), and Level opt. policy respectively. Level opt. (MSBL) refers to three nested bandit policies with Level 1 policy obtained via policy modification with 422 and Level 2 policy obtained via feedback modification with macro intervention L3 The short term optimizing policy performs poorly for both the medium and long term, validating that optimizing engagement leads to sub-optimal performance in the longer term. Level 2 opt. policy (MSBL with 2 levels) performs optimally in the medium term with little sacrifice to the short term but does not perform well for the long term. Level 3 opt. policy (MSBL) achieves the best expected long term reward with little sacrifice to the medium and short term rewards. Taking a random intervention at the third level lies strictly inside the Level opt. policy. This experiment validates the scalability of MSBL for more than two levels. In Appendix E3
analyze the tradeoff in feedback across all
8

---

user groups from all three timescales. We find that MSBL optimizes the long term rewards for each of the user groups.
5.3 Recommender System
Groups
3 Groups
Groups
5 Groups
MCBL
Longstem (Retur Rate!
of
Stort
25 Short-tem (Clicks)
Short-term (Clicks)
Short-term (Clicks)
341
Short-term (Clicks)
Figure 5: Recommender system: Tradeoff between long term return rate and clicks by varying groups using the boost a.2 to the relevance scores as policy modification. Expected short and long term rewards are reported across 5 random seeds.
In the final setting, we use the KuaiRand dataset [2] to simulate two levels of short and long term feedback. This allows us evaluate based real-world user features, and their historical interactions, thereby evaluating for increasing action spaces. We use the simulator developed in [45] and modify for the contextual bandit setting. We use training dataset of size 14,265 and test randomly selected 1,771 users. At the lower level, each user logs T = 5 interactions, and a transformer model uses item and user embeddings predict relevance scores Action aLd represents top-k selection of items for context x from the policy T + arg max; 8, with the micro reward as average clicks per user.
At the upper level, we simulate a macro intervention 1.2 as the boost to the scores s. This is an instance of policy modification since the intervention a is applied post optimization to the micro policy. We learn a policy 4-2 to select the boost 22 for a given user I.12 We simulate user and item groups such that each user group has unknown preference for particular item group that is not evident in the micro-level feedback, but only in the macro-level feedback (e.g., less click bait). We simulate the macro reward of return rate non-linear function of the long-term preference-weighted fraction of selected items.
Scaling Level 2 action space. The purpose of applying macro intervention a is to boost certain item groups for certain user groups. We increase the macro action space by increasing the granularity of user and item groups. As the groups increase, the top-k selected items may not belong to the preferred item groups, making the problem setting harder. Figure 5 shows the tradeoff in the long term user return rate and short term reward of clicks (under the intervention of boost from the macro policy) for top-10 selection. As the number of groups varies E (2, 3, 4, 5). MSBL maintains a high return rate compared to all the baseline policies, with some sacrifice the short-term clicks. Policies indicated by af² use the same micro policy 7 4L1 as MSBL, but apply the boost only to item group for all users in the system. The short-term optimization policy A has no macro intervention applied to the scores and maximizes the clicks, but results in low user return rates across all group sizes.
We study the changes in outcomes when the multiscale policies are updated asynchronously after deployment in Appendix F.4 We find that as new data becomes available, updating only the micro policy improves short term outcomes but may deteriorate long term outcomes requiring an update to the macro policy. We also evaluate the robustness of macro learning with varying noise in the micro policy and analyze the tradeoff with increasing Level 1 actions (selection set size). We find that MSBL robust to perturbations in micro policy, and outperforms baselines consistently across varying selection sizes.
6 Conclusion, Limitations and Future Work
We study the problem of how to train AI systems so that they achieve long-term desirable outcomes. Focusing on the contextual bandit setting, we introduce MultiScale Policy Learning framework that can use plenty of data at the lower levels as prior information for enabling learning from scarce data at the higher levels. We show how this bridges the disconnect in timescales between short-term

---

actions and long-term feedback when optimizing for long-term objectives. Furthermore, we show how this framework can be implemented in a practical algorithm, which we found to be effective in optimizing long-term outcomes across range of domains. However, there are many other ways of instantiating our MultiScale framework with other algorithms, which provide many directions for future work.
Limitations. One limitation of this work is the availability of real-world multi-scale datasets in the public domain. We hope that our work accelerates research and sparks interest in the community, particularly within the industry to open source real-world data. Another limitation lies in the focus on contextual bandits, and the principled extension to stateful policies is an interesting future work. Finally, we assume access to the structure of levels and contexts at each level. Discovering the timescales and inferring the contexts at each level of the multiscale framework in a data driven way is another important direction for future work.
7 Acknowledgements
This research supported in part by NSF Awards IIS-2312865 and OAC-2311521. Yuta Saito was supported by the Funai Overseas Scholarship. All content represents the opinion of the authors, which not necessarily shared or endorsed by their respective employers and/or sponsors. We thank Ben London for helpful feedback on the PAC Bayesian theory, Zhaolin Gao for help with the LLM experiment setup questions, and Taran Singh, Woojeong Kim, and Kiante Brantley for helpful discussions in brainstorming.
References
[1] Himan Abdollabpouri, Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar Jannach, Toshihiro Kamishima, Jan Krasnodebski, and Luiz Augusto Pizzato. Beyond personalization: Research directions in multistakeholder recommendation. CoRR, abs/1905.01986, 2019. URL
http://arxiv. wwwp:/..g//b/a/aet. 1905. 01986
[2] Arpit Agarwal, Nicolas Usunier, Alessandro Lazaric, and Maximilian Nickel. System-2 rec- ommenders: Disentangling utility and engagement in recommendation systems via temporal point-processes. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, page 1763-1773, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400704505. htttps://do.. http:::///oo..rr/100. 1144/113011000 365900 3659004
doi: 10. 10.114523630106.365900 URL
[3] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli. Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Satficld-Dodde Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. BRLphttps: : //ta://ariiii arxiv. org/ abs/ 2204. 05862
[5] Kianté Brantley, Zhichong Fang. Sarah Dean, and Thorsten Joachims. Ranking with long- term constraints. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM '24, page 47-56, New York, NY, USA. 2024. Association for Computing Machinery. ISBN 9798400703713, doi: 10.114553616855. 3635819. URL https //doi. 10. 114111111111111 /31333366855 3635819
[6] Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, and Kun Gai. Reinforcing user retention in billion scale short video recommender system. In Companion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, page 421-44 126,, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394192. doi: 10. 45555/33553773333444000 URL, https:://toi...ooo org/ 1145/ 3543873 3584640
10

---

[7] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In S. Hanson, J. Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems, volume 5. Morgan- Kaufmann, 1992. UtI.httpss//pproceedinng neurips.... cc/paper fwwes/paper///eeeeeeaeaaaaaaeeat Filcd4222e66eer33400888666662222 Pappeeeeeeeeeeeeeaee......
[8] Shehzaad Dhuliawala, Ilia Kulikov, Ping Yu, Asli Celikyilmaz, Jason Weston, Sainbayar Sukhbaatar, and Jack Lanchantin. Adaptive decoding via latent preference optimization, 2024.
URL. httpttt/tt/:axii...... wwwp:/iww..ooooaooooaoo attp///s./211....//// 09661
[9] Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. In International Conference Learning Representations, 2020. URL
https::/openrevies: net/ f Formm worooiiiiiiiaiieii Saaimiidd--aa H
[10] Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learn- ing. In Proceedings of the 28th International Conference International Conference Machine Learning, ICML' page 1097-1104, Madison, WI, USA, 2011. Omnipress. ISBN 978 1450306195.
[11] Tom Everitt, Mareus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective. Synthese, 198:pp. S6435-S6467, 2021. ISSN 00397857, 15730964. URL https://www jsttr.org/sttateeee.. wtrgsstable/48822222
[12] Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wengiang Lei, Peng Jiang, and Xiangnan He. Kuairand: An unbiased sequential recommendation dataset with randomly exposed videos. In Proceedings of the s1 ACM International Conference on Information & Knowledge Management, CIKM '22, page 9953-3957, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781450392365. http::://ao..rrg/// 10. 11151111111118 31511111188883666244
doi: 10. 180.73577844. URL
[13] Groq. Api reference- chat create, 2024. URL https://console.gron htt.://wwooo..g...oooooooooo/o/e
api-rroferenceHcat---rratt
[14] Henning Hohnhold, Deirdre O'Brien, and Diane Tang. Focusing on the long-term: It's good for users and business. In Proceedings of the 2/th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, page 1849-1858, New York, NY. USA, 2015. Association for Computing Machinery. ISBN 9781450336642. doi: 10.11452783258.2785538 URL https://doi. org 10. 1145/2783258 2788583
[15] Dietmar Jannach and Himan Abdollahpouri. A survey multi-objective recommender sys- tems. Frontiers in Big Data, Volume 6 2023, 2023. ISSN 2624-909X. doi: 10.3389/fdata. 2023.1 115789. URL httph::tt/////t
http:://wwwwwwwo.rsesssee frontiersin wttpiiin..g////ooe hwtp:/ooonnall/iiddaaaaaaaaaaa 1157899
10. wtw.999fffaaa...mm.
[16] Olivier Jeunen, Jatin Mandav, Ivan Potapov, Nakul Agarwal, Sourabh Vaid, Wenzhe Shi, and Aleksei Ustimenko. Multi-objective recommendation via multivariate policy learning. In Proceedings of the 18th ACM Conference on Recommender Systems, RecSys '24, page 712-721, New York, NY, USA, 2024. Association for Computing Machinery, ISBN 9798400705052. doi:
10. 10.1145/364447,...o 11117/644457..8881111 132. HRL/https://do1 org 10. 1145/164444555 2444//3364445733881222 3688132
[17] Thorsten Joachims, Adith Swaminathan. and Maarten de Rijke. Deep learning with logged bandit feedback. In International Conference on Learning Representations, 2018. URL
https://openreviee
-XAb
[18] Atoosa Kasirzadeh and Charles Evans. User tampering in reinforcement learning recommender systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES '23, page 58-69, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702210. doi: 10. 10.1145/3600211 1.3604669. URL, https::tt::///... org/10.011111111 1145/ 3600211.360469
[19] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextul-banddi-bass news article recommendation algorithms. Proceedings of the Fourth ACM International Conference Web Search and Data Mining, WSDM 11, page 297-306.
11

---

New York, NY. USA. 2011. Association for Computing Machinery. ISBN 9781450304931. doi: 10.1145/1935826. 1935878. URL htttp:://doi.org/111 htt.////......1111...111111 1145/1935826 1935878
[20] Ben London and Ted Sandler. Bayesian counterfactual risk minimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4125- URL http::::/ptoceeeeeeeeeeeee amliiigggmm.......m mml. www..press. (eress/v97// 11ndon191
4133, PMLR, 09-15 Jun 2019.
html
[21] Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. Feedback loop and bias amplification in recommender systems. P Proceedings of the 29th ACM International Conference Information & Knowledge Management, CIKM '20. page 2145-2144 2148, New York, NY, USA, 2020. Association for Computing Machinery. doi: 10.1 14145/3340551.4412222 URL WtRpsttps://doo oww.llorg1110 1trg/10.15 1145/
ISBN 9781450368599.
3340531 .410531.4122152 3412152
[22] Matthicu Martin, Panayotis Mertikopoulos, Thibaud Rahier, and Houssam Zenati. Nested bandits. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15122, PMLR, 17-23 Jul 2022. URL /ttpttps::/prrtttt https:/pproceednngss mlr wwwps/eesvvvee.. wttss//v12mmaaarnn/nnn httmtiiinn2a.mmm11111m html
[23] Lucas Maystre, Daniel Russo, and Yu Zhao. Optimizing audio recommendations for the long-term: A reinforcement learning perspective, 2023.
[24] David McAllester. Some pac-bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory, pages 230-234, 1998.
[25] Thomas M. McDonald, Lucas Maystre, Mounia Lalmas, Daniel Russo, and Kamil Ciosek. Impatient bandits: Optimizing recommendations for the long-term without delay. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '23. page 1687-1697, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/35803055333 35599386....
URL https://doi.ii.... orw.oi...rg11..0.. 1145/
3580305 3599386
[26] Smitha Milli, Luca Belli, and Moritz Hardt. From optimizing engagement to measuring value. In Proceedings of the 2021 ACM Conference on Fairness, Accountability; and Transparency, FAceT 21, page 714-722, New York, NY, USA, 2021. Association for Computing Machinery. doi:
ISBN 978 9781450383997.
014553442188.3445000 URL htttt:://///oo org 10. 1145/
3442188 3445933
[27] Smitha Milli, Emma Pierson, and Nikhil Garg. Choosing the right weights: Balancing value, strategy, and noise in recommender systems, 2024. URL https:://axii..oort www.:/ogg/////o//eooooo..o /305 17428
[28] Marco Morik, Ashudeep Singh, Jessica Hong, and Thorsten Joachims. Controlling faimess and bias in dynamic learning-to-rant Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 429 438. ACM, July 2020. doi: 10. 10.1145/3377271.34011
http:tt:: //dx doi. org/10 100 1145/
3397271 3401100
[29] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. International Conference on Learning Repre- sentations, 2022. URL /oLLitttts///operrrrii http:/ppnrrrrieeeeeeeeffrrrrraa/ www./eeeun/iiinie Jdddddd=iwLyyyyeeee
[30] Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. Feedback loops with language models drive in-context reward hacking. InMC 2024.
[31] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical rein- forcement learning: comprehensive survey. ACM Comput. Surv., 54(5), June 2021. ISSN 0360-0300. doi: 10. 14/3445/3453.60, URL https: /tttps://doi.oo org/10 1145/ 1eeel.114555550 /4453660
12

---

[32] Matt Post and Marcin Junczys-Dowmunt. Escaping the sentence- level paradigm in machine translation, 2024. URL https://arxiv org/ abs//3/4//////.////// 12959
[33] Noveen Sachdeva, Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley. Off- policy evaluation for large action spaces via policy convolution. In Proceedings of the ACM Web Conference 2024, WWW '24, page 3576-3585, New York, NY. USA, 2024, Association for Computing Machinery. ISBN 9798400701719, doi: 10.1 10.1145738993443644551. URL
[34] Rajat Sen, Alexander Rakhlin, Lexing Ying, Rahul Kidambi, Dean Foster, Daniel N Hill, and Inderjit S. Dhillon. Top-k extreme contextual bandits with arm hierarchy. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9422-94333 9433, PMLR, 18-24 Jul 2021. URL http::://proceeeinngss mlr, wwwpprresssee.ee se///a.nn.a..ee///e html
[35] Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreviie http:forumm/dd/yoooo00..33
[36] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16857-16867. Curran Associates, Inc., 2020. URL httpps neurips cc/ paper httpa//pparrroooo/aaaaaaaaaaaaaa
pdf
[37] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[38] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning, pages 814-823. PMLR, 2015.
[39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Seialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL (ttps::ts::// //tp:ii::///xxv....// ha/p/axxvvvvoro org/ abs/ 2wt.//9077000228888
[40] Shoujin Wang, Longbing Cao, Yan Wang. Quan Z. Sheng, Mehmet A. Orgun, and Defu Lian. A survey on session-based recommender systems. ACM Comput. Sure., 54(7), July 2021. ISSN 0360-0300. doi: 10. 10.1144/3465401.
URL httpt:://ot....... org/ 1145/4665401
[41] Yequan Wang, Jiawen Deng, Aixin Sun, and Xuying Meng. Perplexity from plm unreliable for evaluating text quality, 2023. htttps:///// www.:/orggooooo wttgaaa///1....000 2210. 25892
[42] Timo Wilm, Philipp Normann, and Felix Stepprath. Pareto front approximation for multi- objective bassi recommender systems. Proceedings of the 18th ACM Conference on Recommender Systems, RecSys '24, page 809-812, New York, NY, USA, 2024. Association doi: 10. 10.1145/3640457.388 3688048. URI.
for Computing Machinery. ISBN 9798400705052.
attps://dol: org/10. 111.1/1..1111111 1145/ 36880488
13

---

[43] Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-contxt: Multi-objective alignment of foundation models with dynamic preference adjustment. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 56276-56297. PMLR, 21-27 Jul 2024. URL https: ptop:e///proceeiiggs mlr. press/ w23g/yang24q html
[44] Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. ACM Comput. Surv., 52(1), February 2019. ISSN 0360-0300. doi: 10.1145/3285029, URL http::ttsss::iitrreeeeeseses org 10. 1 1145/ 3285029
[45] Kesen Zhao, Shuchang Liu, Qingpeng Cai, Xiangyu Zhao, Ziru Liu, Dong Zheng. Peng Jiang, and Kun Gai. Kuaisim: A comprehensive simulator for recommender systems. In Thirty-eeventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
URL https:///pperreeieeet wtt.:///ew.......ooooooooooaaaa.
[46] Shi Zong, Hao Ni, Kenny Sung. Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cas- cading bandits for large-scale recommendation problems. Proceedings of the Shiony-Second Conference on Uncertainty in Artificial Intelligence, UAI'16, page 835-844- Arlington, Virginia, USA, 2016. AUAI Press. ISBN 9780996643115.
14

---

A Broader Impacts
Optimizing for long term outcomes is a desired goal for many interactive Al systems. This paper makes progress in the area by highlighting the issue of disconnect in timescales of feedback and interventions. As a solution, we propose a general framework and a learning algorithm that aims to reconcile this disconnect.
While the proposed framework provides affordances to design and optimize for beneficial outcomes at multiple scales, those design choices must be carried out responsibly. For instance, the design of interventions and how they interface between longer and shorter-term levels are of crucial importance. These interventions could enable user agency in steering the system towards their personalized long term outcomes, but could also be used adversely. Transparency in the design choices and optimization objectives is especially important for that reason.
B
Additional MSBL Example
Consider a conversational system, consisting of multiple agents, where each agent specializes in specific type of food cuisine. The platform assists users with various cuisine preferences over session consisting of multiple queries. Within these preferences, some users prefer more diverse recommendations in session than others. According to the users' interaction history, selecting the most relevant agent for a query is optimized at the lowest level. The feedback at this level the relevance of the generated response from a L.M policy, with relevant responses leading to better engagement. At the next higher level, a decoding strategy acts as an upper-level intervention to the LLM policy. The intervention of the decoding strategy can steer the responses towards more less diverse responses depending on user preferences, leading to better user return rate on the platform. While optimizing for this return rate seems important, at even higher level, some users may not prefer to use the platform as heavily as compared to others. For such users, leveraging an intervention that ultimately provides more system value than optimizing their return rate is ideal.
C Extended Related Works
Prior works have studied the misalignment between micro and maero objectives to be substantial 5.228. Specifically, S compares the ranking deg metric for the micro/short term utility of micro actions (ranking) and the macro violation for the long term constraint violation. Similarly, (28] demonstrate the tradeoff between utility and amortized exposure (collected over time horizon T) as the macro goal. These works consider optimizing for LI actions only (rankings). They do not learn L2 interventions such as reward weights (26). ads, message notifications, decoding strategies etc that are available the system and can directly steer the system towards long term objectives. In contrast, MSPL framework learns these interventions via the macro policy and extends more than two timescales of competing objectives. In conversational systems, misalignment between token-level metrics (e.g.. perplexity) and multiple responses/document- in paragraph) has been studied 41 32.
metrics (c.g.. diversity
Recent works have explored hierarchy in the action space and arms in bandits 22 34 These works do not learn for long-term outcomes but for efficient multi-task learning. [2 proposes tree-based
hierarchical structure, where each tree node represents action abstraction, grouping similar actions its child nodes. contrast, our focus study hierarchy the timescale of feedback, and clear separation of action space at different levels. For long term optimization, [25] proposes bandit algorithm in the progressive feedback setting, where the long term outcome increasingly predictable as more short term information is revealed. It models the long term outcome as the delayed outcome
of micro level actions. This setting is different than ours as it does not take into account the tradeoff in short and long-term outcomes.
For multi-objective optimization, 66) proposes learning an RL-based policy for the weights of clicks, shares, likes etc. in recommender systems. Similarly, [16] proposes learning weights with off- policy bandits for a north star goal. Different than these works, our framework learns a single policy as family of policies at the micro level and separate policy at the macro level. This allows learning with more abundant feedback at the micro level narrow down the potential actions for the upper level and learn macro policy from relatively sparse data. Our framework also learns contextually and
15

---

handles a richer class of reward functions than the linear scalarization approach in [6 16 9 first introduced learning single model trained with distribution of losses instead of learning multiple
models, each trained with a single loss function. 44 learns family of models for recommender
systems given distribution of preference vectors. L.MM Alignment literature, [43] introduces
learning family of policies by including preference weighting in the prompt context of L.M policies.
These works do not learn the preference weights, operate on a single level, and do not consider learning long-term outcomes.
Related to the adverse effects of over optimization on short term feedback, recently showed that feedback loops within in context learning can cause reward hacking via output and policy refinement. In this paper, we propose policy and feedback modification as two ways construct priors and mitigate the over-optimization of short term feedback.
D MSBL Algorithms
D.I MSBL Inference Algorithm
Below, we provide the inference procedure for two levels. The process follows top-down approach, where macro action is selected from upper level policy. This macro action then selects the lower level policy action a² a a
Algorithm 2 MultiScale Inference
foreach upper level timescale t2
do
Observe upper level context x
Get upper level action a_2 ~ L2 12 1يعي
foreach lower level timescale /1
E {1, ... T) do
Observe lower level context x p( x x22)
Get lower level action a=1 7 atta (a
(1l)
D.2 Extending MSBL to l levels
Algorithm presents MSBL extended to multiple levels by calling the PolicyLearning procedure recursively for any two levels. We start with the highest level k, and recursively call the next lower level until the base of k=1 is reached. The Algorithm would then return the PolicyL_earning procedure the next two upper levels and so on.
Algorithm 3 MultiScale Off-Policy Contextual Bandits (for multiple levels)
Require: l 2 2
Procedure PolicyLeamiin(Tng To if k: then
L(K), I(k-1))
return;
// Base case: when at lowest level
PolicyLeamnngg= L(k-1) 2)k zu
Collect Micro Logged dataset
D=(k-1) [( ((k-1) aj I(k-1) L(k-1) Pi 1) in
114-13 L(k
no
Learn Micro policies () (Eq. (6) using D41)
Collect Macro Logged dataset
DT( := {(x [(2, E(k) L(N).
E(K)
L())
T I(K)
.r. .p.
Learn Macro Policy
L(M)) < arg max LA)) (nr (()) D(()) (Eq. (
return learnt policies A (RR) 1)
16

---

E
Multi-Scale Policy Framework
The PAC Bayes generalization bounds we consider in Eq. are derived for counterfactual risk minimization (e.g., clipped inverse propensity estimator) in 20
As part of the motivation, we illustrate the following example.
E.1 Gaussian Model
We consider Gaussian parameterizations, where each policy T(.I., defined via a parameter vector 0 E TR.. We define the target policy distribution as Q N(022* 5222), an uninformed prior distribution Po N (0o. Eg) for some arbitrary f,, and an informed prior pL1 N( E centered at the learned micro policy n/11
The KL divergence between any two multivariate gaussian KL(Q42~1| (PP) ) is given by,
KL[Q
log
- tn tn + (0) B (00 042)
where (9,
(Lo)) (60 - ( 100 -/ is the squared Mahalanobis distance in the
parameter space 0.
Using the above, the gain in the number of samples from using an informed prior PL1 instead of an uninformed prior Po is,
ng n2 x KL(Q - (1(t))
tr(_; ( tr((=(1)) y (0 - 00) En (0 - 0o) - -0112 (11) p(i1)
+ log
[Eal
(9)
Setting Σ'1 En Ep and since the squared Mahalanobis distance symmetric, we have
KL(Q K2 = 2e - ' (@ 6o) (0 - (0 gh1)
As result, can have the sample gain from using the informed prior PL1 instead of Po as
no - n 12 x KL
PP) K (L (( E 0(10 - 0o0 M - 1022 - -0
(10)
For the special case of isotropic Gaussian distributions, where y = = == ool = apl. Eq. 10 can also be written in terms of I.2 distance as,
((((Q(121P)))) E 0(100'2. 0ol1 -1 0z 113)
However, the above a conservative estimate, since it can be beneficial to pick variances LL < do that increasingly reduce the variance going from uninformed prior Po to p!.1.
Next, we simulate a toy example with isotropic Gaussian distributions for 0 E R.30 To calculate no 41 we use the exact KL divergence Eq. g We start with uninformed prior variance Gg = 200 and learn all 50 parameters with target variance L2 = 1.0. For n2 we start with an informed prior pLl that has 49 parameters learned with LL 1.0, and we only need to adjust more parameter. The coefficient constant for KL[Q]PP is used as 5.0c for all cases. This provides
98% as the reduction in samples needed at the macro level L2. Note that the number of no
samples used to form the prior 41 is significantly cheaper than n since they occur I times as frequently. Empirically, even when taking the additional LI samples into account, this still provides
88.2% 2% reduction with a T = 10 horizon.
F Experiment Details
For training the bandit policy at given level, Importance Sampling estimator (IPS) [38
17

---

Level Manager
Algorithm
Estimators
Reward Simulator
egg. MSBL algorithm, single stage policy, oracle
e-g., IPS, direct method, doubly robust
Manage levels and their estup
Feward functions
Figure 6: MultiScale simulator modules
For any given level, we use a uniform random policy as the logging policy no, and a softmax policy π Taa lr, 0) parameterized by weights 0,
exp(30(x, a, 0))
π (alar, 0) EA exp(Só(z,a', a()
(11)
where 3 >> 0 the inverse temperature parameter, (b.) a function mapping given context, action to real value with dimension d. parameterized by 0, defined as ((. : x A + Rd
IPS is an unbiased estimator and only requires the full support condition, that is mo(alz) > O v(I,a) a) E X' A.
For feedback modification, we additionally pass the macro action with the features, so that,
exp(Bé(x*) _ _, .0)
x (a "
, L2 Eatl'et EAt exp(36(x*1) a.2 LL ())
(12)
Ea
We sample 42 from a uniform distribution during training. At inference 22 is selected from the learned macro policy.
Simulator The simulator code can be found at httttt:://gitii http://ww...ommiiiicatattssseses shows the structure of the simulator with four key modules. A level manager module is responsible for the overall orchestration of level initialization and invokes the algorithm module. The algorithm module implements recursive policy learning of the MSBL algorithm and other non-adaptive baselines. This module may invoke one of the off-policy estimators such as IPS, the direct method, or the doubly robust estimator. Finally, the reward simulator implements reward functions that provide the observed feedback at different timescales. Policy learning options for the algorithm, estimators, and the type of reward function configurable.
Figure
Compute Resources We use NVIDIA RTX A6000-48GB GPUs for experiments conversational systems and 24GB NVIDIA GeForce RTX 3090 for experiments in conventional recommender system.
F1 A toy example illustrate comparison with flat RL
We consider a finite horizon setting MDP with the goal of learning π (alx)) that optimizes the long term reward y222 (I), (I44T, (I14r,a(4T) observed after T timesteps.
80
MSBL Eplsodes (ni2))
QLearning Episodes
We use (tabular) Q learning to learn policy that optimizes for the sparse reward ,.42 for the following setup. Users arrive with
40-
context 24 p(æ.) and select & out of n items based on relevance vector (x,)) E R". The action a₂ can be one out of & choose n combinations. We simulate long term reward for each user is observed based on their preference for the item. We
10 items, and 2 contexts 2,. The
10
time horizon of T 5.
Selection size (k)
While Q learning is provably optimal, it computationally expensive to form the Q table by learning from episodes {(x,, a.), ... (I++T; d:4T), ,22). especially when 72 is sparse. In contrast, MSBL leverages the problem structure as follows. Its mi- cro policy is simply argmax on item relevances. Even though this myopic policy doesn't t lead optimal long-term reward, it provides prior for the macro level, which only needs to learn the macro intervention a 122 as the amount of boost to the relevance vector r.. We construct [ 8, that 42 only needs select the best L2 for each user context.
Figure 7: Ratio of Q learning vs MSBL episodic samples for toy setup as the action space increases with kt
As a side note, while function approximation and other advancements (actor-critic, etc.) would provide benefit to the RL baseline, they orthogonal to the key idea of enabling faster learning
18

---

via the use of hierarchical priors for sparse long-tern feedback. Figure shows the ratio of episodic samples no to the MSBL macro policy samples n n2 to achieve the same in expectation. We can see that 10 < T I 62 as k varies in {2, 4, 6}. Scaling this baseline for continuous context features, time horizon, selection size k, and number of items n is challenging and motivates our approach.
F.2 Multi-turn conversation
Experiment Setup. We use Llama-2-77 2-1---chatht 39 as the base model at the lower level. We use huggingface reward models gpt2-large- wawtaang-halleseeeeeee harmless-reward model and sanp22largehelpful-rewaa model for harmless and helpful reward models R1, Rg. Following rewards in context learning [43], we train a micro policy as a family of LLM policy fi by including a2 in the context prompt and optimizing Σ af The conversation starts from one of the query prompts x11 1 (a question in the Anthropic dataset), and for each subsequent turn, the trained LLM policy fi responds. This completes one turn. Then the user LLM asks a follow-up question. As a result, each generated response y π (. U<t)) consists of the conversation upto that turn. For the user LLM, another Llama-2-7b-che model. The user LLM scores a single-turn conversation, which we use as the short-term reward. This is instance of feedback modification that steers the LLM policy using modeled feedback (where feedback is defined by a model). This differs from the observed feedback (i.e, user LLM evaluations, which are not modeled and only observed for an action that taken). At the upper level, we learn interventions a E [0.8 (0armm-,,--,, 0.2((elpff)) (0.2 (harmll)) ), 0.8(helpf-)]. 22 refers to the same level policy as other baselines but using the fixed level 2 intervention of preference weights in (0.8(harll-,, 0.2(helpf-)), while 1.2 refers to using the fixed level 2 intervention of preference weights in (0.2(harll-), O. (.8(helpf-)).
We use the upper level reward simulator as Llama-3-706 API [13), which scores user satisfaction for the given user persona after a full 5-turn conversation as a binary score in {0, 13. We transform the user persona text with MPNet model (36) and project down with PCA to 10- dimensional embedding. This forms the upper-level con- text 4.42 for bandit policy π nl.2
Long (Multi-turn) Reward
D
al2
.f²²
0.80
a|2
a/?
8.60
0.40
Group-wise Results Figure [8 shows the long term multi turn reward vs per-conversation short term reward for each of the user personas Child and Expert. We note that MSBL achieves the best multi-turn reward for each user persona while the other policies optimal only for type of user group.
Overall
Group A Group B
0.45
0.0
655
5.85
Short term (Single-tum) Reward
Figure 8: Long-term (multi-turn) vs short-term (single-turn) rewards for all users (overall) and for each user group.
Training Details. For π = we 5 layer neural network with hidden dimension 256. We train using AdamW optimizer, with a batch size of 256, learning rate le-3, weight decay le-1 for 1000 epochs. We use the inverse temperature parameter 3 0.8 in Eq. (11).
Prompt Templates Below, we include the prompt templates for the LLM evaluators and prompt generator.
We use the template below and the corresponding user persona for obtaining the upper level reward.
#
User Persona Prompts
child: You are a child user who wants to avoid harmful, illegal, or unethical content.,
expert: You are an expert user who values accurate and relevant content.
### Llama 3-b0b Evaluation Template prstem_prompp You are an expert conversation evaluator. Given a user personality and conversation, output only a score o or 1 indicating user satisfaction. Repetition of responses in the conversation should not have any impact on your evaluation.
User Personality: {user persona]
19

---

output format: score: [0.0 or 1.0]
Here are some examples of how you might respond: (Examples)
template {role: system,
content: {system prompt),
role: user,
content:
Conversation to evaluate: (covvrratoooooaaaiiooo
Evaluate user satisfaction. Please provide a score and explanation in the required format
User Llama-2-7b Evaluator and Prompt Generator
For the next prompt generation, we use the below template with the history of previous turn conversa- tion.
### User Llama-2-7b Evaluator and Prompt Generator system prompt You are a user having conversation with an AI assistant.
{user persona]
Given the conversation history, generate only the next user message or question (one or two sentences) to continue the conversation.
output format:
User: [next user message]
template { role: system,
content: {system prompt),
role: user,
content: Conversation history: ccoverratio)
Respond with just your next message the exact format.
For evaluating single turn, the below template. This serves the simulated lower level reward.
system prompt = You are evaluating AI assistant 's response as a user
{user persona).
You are provided with a user's prompt and an assistant 's response. Rate the usefulness and quality of the response from 0.0 to 1.0.
You must use this exact format for your reaponse: score: [between 0.0 and 1.0]
template = { role: system,
content: {system prompt),
role: user,
content: User: (prompt) Assistant: (response}
Provide score in the required format.
F33 Conversational recommender system
Experiment Setup At each of the three levels, simulate 5-dimensional contexts sampled from a normal distribution N (H σy) belonging to two user groups. The context 441 represents users'
demographics and cuisine preferences. At Level 1, there are 10 bandit actions a 441 corresponding
20

---

to cuisine:| "aassin:"EEthiopiaa"" "Mexican", "French", "Japanese". "Spicy Indian", "Thai", "Carribean', "Peruvian", "Russian", "Italian", J. We use Llama-2-7b-chat (39 the LLM policy x 7 that acts as personalized agent, generating cuisine suggestions y to users at L11 timescale (1 = (1.... 10). Each query consists of system prompt specifying the agent's expertise and user query 4- We learn bandit policy (Ll [x (x)) that selects the particular LLM agent a=1, Next, we generate response Yt ~ TTLL (-Ia"1 9, 1t-1) from the LLM policy, given the agent selection a and append the previous timestep response 3/-1 the current query. The LI reward is simulated as inverse perplexity conditional on the optimal prompt, representing the engagement (relevance) metric, as follows
# tokens in Ne
r perplexity(vsla'" (1n) exp
Σ
log pro.(yy..s....ii--1... attlt)
# tokens in response y/
where a = [ccusiine is selected according to rr/ and the optimal action a LI+ corresponds to the optimal prompt according to user preference.
Intuitively, the inverse perplexity metric means that the response Ye generated using the bandit action aLd should give the highest perplexity the optimal action a_1* is chosen. For example. if the user's preference was french, but the bandit policy n-1 selected an action corresponding to mexican cuisine, then the response generated y consisting of mexican cuisine would have low perplexity given the prompt of french cuisine (i.e, the optimal action a al))))
The query q at every timestep t is follows,
You my personal chef experienced in {cuisine} Cuisine. Your responses should be professional and concise (100 words or less). Previously, you suggested: {previous generated response}.
What should I eat today?
At the second level, we simulate two user groups with context L.2 representing preferences for engagement (relevance) and diversity. We the notion of n-gram repeats multiple responses (t1 E (1,... 10}} for computing diversity, where we use n = 3. In particular, we define diversity as normalized 3-gram repeats, where the 3-gram repeats computed over T 10 responses generated at Level for each user. We simulate the reward
L2 -0 10
L1
(1 (w) (diversity)
10
where (x) = a(x b) is scaled sigmoid function with scaling factors a = 60, b 0.6. 8n is the unknown user's relevance-diversity trade-off parameter, and that prefer relevance and diversity respectively. The second level policy 222 selects an intervention of decoding temperature _² {0.0, 0.2, 0.4, 0.6, 0.8, 1.0) as a policy modification the Level policy, with a122 ag² referring to each of the L2 interventions.
3, 3 (0.9, 0.1) for groups
Some sample responses (decoding temperatures are logged after generation) are,
"Hola! How about our delicious carne asada burritos with sauted onions and bell peppers, topped with melted cheese and served with side of sliced avocado and a drizzle of homemade salsa? decoding_temp.1..5 "For lunch today, I recommend
Carne Burrito. It 's a satisfying blend of slow-cooked beef, beans, and vegetables wrapped creamy sour cream and fresh cilantro. It's a hearty and flavorful dish that will warm you up on a chilly day! decoding. temp_0. 0.87",
'Chili
large flour tortilla, topped with
"Great! Based on your previous preferences, our 'Shiro Not'
recommend trying creamy stew made with chickpeas, onions garlic, and a blend dish that sure to satisfy your taste buds. Pair it with side of 'Kik Alicha' sauted vegetables in a mild sauce, and enjoy! decoding Berrodiing_tmp_.00 21", 21",,
spices. It's popular vegetarian
21

---

Return Rato (Larger Torm) Rowwrdd
Group
Orack
2.4
034
#2
6483 5:30
346
333
0.40
0.00
0.56
6.5
660
0.00
Engagement (Short term) Level Reward
Enga gementt (Short term) Level Reward
(a)
(b)
Figure 9: Conversational recommender system: Tradeoff between longer-term Level 2 and short- term Level I rewards using decoding temperature a E {0.0, 0.2, 0.4, 0.6, 0.8, 1.0) policy modification across 5 random seeds. (a) for all users (b) for users belonging to group A that prefer relevance (c) for users belonging to group B that prefer diversity among multiple responses
Long
Long
Medium
Short
Medium
Short
Level opt. (MSBL)
Level opt. (MSBL)
Level opt.
a² mg²
Figure 10: Tradeoff in different timescale rewards for the two user groups at Level 3. (a) For this group, long term reward of subscription renewal is aligned with medium term reward of weekly return rate (b) For this group, the long term reward is not aligned with the medium term reward.
At the third level, we simulate users with context L3 We use feedback modificution for Level 2 training get family of policies 11 [ a E {10, 1], [1, 0]3}. The parameterized feedback is given by Σ a. Rs, for two reward models R1, R2, such that R₁ = 4 and R2 min(T, 22). We use J = 0.8 as user group-specific threshold. This simulates the scenario where for one user group, optimizing the weekly return rate is beneficial, while for the other user group, optimizing only upto threshold T beneficial. simulate 3.3 across timescales of level 2 as
follows,
Σ
1.2 122
L3
Yuu
+(1 (1- - Tw) 1(activity preference)
where Tu (1,0) is the trade-off parameter.
Training Details For bandit policy æ at each level, we use a layer neural network with hidden dimension 256. We train using AdamW optimizer, with batch size of 256, learning rate le-4, for 4000 epochs.
MultiScale contexts. In this experiment, evaluate across groups from all three timescales. At each level, we simulate contexts belonging to two user groups
Figure shows the tradeoff in longer-term 1.2 reward and shorter term L1 reward for the two user groups individually. The context 4 at Level 2 belongs to these two groups. Similarly, Figure 10 shows the tradcoff in different timescale rewards for the two groups that belongs to at Level 3.
22

---

Long
190g
Long
Long
Medium
Short
Medrum
Short
Medium
Short
Medhim
Short
Level opt.
(MSBLL
opt
Level opt.
m5
Figure MultiScale contexts (x.1/ three timescales.
and rewards for groups in (a), (b), (c), and (d) across all
Policy 02
Noise
MSBL
12
01
03
F.2
04
05
1.2
0.05
0.82:10.13
0.66110.10
0.46110.3
0.68110.16
0.76.10.11
0.77.10.10
0.72.10.08
0.64:10.06
0.10
0.86110.08
0.66110.09
0.4810.10
0.6110.113
0.7710.12
0.7411.111
0.75
0.62:10.06
0.20
0.76:10.10
0.57
0.43+0.18
0.5910.10
0.6010.08
0,6110.11
0.5810.08
0.55+10.06
0.30
0.66-+0.12
0.54++0.6
0.34+0.17
0.49++0.12
0.49+0.12
0.55++0.33
0.53+0.04
0.53++0.02
0.40
0.58:10.06
0.53+10.02
0.37+00.9
0.45:10.12
0.51
0.51
0.524.0.01
0.51+10.01
0.50
0.62:10.08
0.47:10.03
0.3310.09
0.50.10.05
0.51
0.52.10.22
0.52.00011
0.51
Table 1: Level 2 Reward with varying noise features
In both cases, we find that MSBL policy achieves near-optimal long term rewards for each of the groups.
A Level 2 context 42 is drawn given a Level 3 context x For example, given that a user does not want be on the app every week (Level 3 preference), the user likes more diverse responses (Level preference). This results in 4 user groups across L2 and L3. Figure I shows the tradeoff in expected rewards across these groups. While there are user groups for whom policy that optimal in the medium term is also near-optimal in the long term (groups and 2 with overlapping red and blue lines), Level 3 opt. (MSBL) pareto dominates each of these policies for all the groups for the long term reward.
Robustness to noisy features. We also evaluate robustness of MSBL with varying noise in the user features with y E (0.05, 0.1, 0.2, 0.3, 0.4, 0.5) with default value in bold. Table shows that while the expected Level 2 rewards decrease with increasing noise in features, MSBL maintains the advantage of learning the interventions over other baselines across noise variations.
F.4 Recommender System
In the final setting, we use the KuaiRand dataset 12 to simulate two levels of short and long term feedback. This allows to evaluate based on real-world item, features, and their historical interactions, thereby evaluating for increasing action spaces. We use the simulator developed in [45] and modify for the contextual bandit setting. We randomly selected 1,771 users. There are 5659 items and their descriptionssfeatures in the dataset. At the lower level, each user logs T interactions, and transformer model uses item and user embeddings to predict relevance scores s. Action a represents a top-k selection of items for context z from the policy #41 arg max, s. Then, we simulate clicks according to a Bernoulli distribution of the relevance scores of selected items and micro-level reward as average clicks per user.
training dataset of size 14,265 and test
We form item groups based on the upload type of the video and user groups based on the users' activity level feature. At the upper level, each user group has unknown preference Pu,i for particular item group. We use Pu.i = 0.9 for the preferred user and item group pair. The user retention probability Pr is simulated this sefference-weightee fraction of items selected, given by p, = softmax (log(2 E; Pa, T,,,)) )), where n, is the number of items selected from group over
23

---

Slate Size
Slate Size 10
Slate Size 20
Slate Size 40
Long (Return Rates
19
*0-
68
NFDL
Short Ost.
88
101
10
18
Shart-term (Clicks)
Shart-term (Clicks)
Short-term (Clicks)
Short-term (Clicks) T
(a) 2 groups
Slote
Slate 10
Slate
Size
KEBL
Long Longtemm Serm (Retum Rate)
Shon
-8-
Grods
++-
16
26 Short-term (Clicks)
Short-term (Clicks)
10
26 Short-term (Clicks)
Short-term (Clicks)
(b) groups
HSR
Lane term (Return Rato)
H+
Orade
Short.term (Clicks)
Shart-torm (Clicks)
20
Short.torm (Clicks)
Shart-term (Clicks
25
(c) groups
Figure 12: Recommender system: Tradeoff between long term return rate and clicks by varying slate
size using the boost 12 to the relevance scores as policy modification across random seeds when the number of groups vary in (a) 2 groups (b) groups and (c) groups.
t1 (1, ... 5) timesteps of the lower level. The reward 1.2 representing the return rate is simulated as the inverse of the return day to the app.
For training the lower level policy, which is transformer model that uses input as the item and user embedding and predicts relevance scores, we use the user features and pre-processing following (45)
For the upper level bandit policy, we use "uf uueuuuueeeee active active_degree'' 'uf is tireessstemrrrr..e 'uf is video author', "uflfollow seslow_user__uuee range'.
user features. namely
'uf fans gaiss_user__uun paiiinuumaaaneee' with their definitions provided in [12].
Training Details For bandit policy 42 upper level, we raw user features into 16-dimensional vectors and a 4 layer neural network with hidden dimension 128. We train using AdamW optimizer, with batch size of 128, learning rate le-4, for 4000 epochs.
embedding module that embeds the
Increasing Level and Level 2 action space. Figure 12 shows the tradeoff with varying groups in (2,3, 4) and with varying Level I actions as the slate size top-k varies for & E {5, 10, 20, 40). We observe that the return rate for MSBL remains high across all sizes and for all groups Since we report average clicks per user, there is decrease in the short term (clicks) with increasing slate size. This experiment demonstrates that even with large micro action space A411 MSBL can effectively leverage the interventions to drive the system toward the long-term reward.
Updating policies after deployment. To study the changes in outcomes when the multiscale policies are updated asynchronously, we conduct the following experiment. We use 60% data to learn micro and macro policies and refer them as deployed policy. We include an additional 20% data of users and their corresponding interaction history. We update only the micro policy with the increased logged miero data, and leave the macro policy unchanged. The top row of Table 2 shows the % change relative to the deployed policy. The change in return rate under the updated micro policy

---

2 Groups
Groups
4 Groups
5 Groups
% Improv. updating
LT
ST
LT
ST
LT
ST
LT
ST
only micro policy
-1.18%
2.47%
-0.70%
2.61%
-0.07%
3.04%
4.88%
2.85%
micro & macro policy
0.67%
2.34%
4.90%
2.37%
2.12%
2.86%
5.66%
2.66%
Table 2: Improvement (relative to deployed policy) after policy updates. Short-term (ST) clicks highlighted when only micro policy is updated. Long-term (LT) return rate highlighted when macro policy is also updated.
2 Groups
3 Groups
4 Groups
LT
ST
LT
ST
LT
ST
w. new macro data
0.79 10.055
202.0 10.67
0.58 +0.009
203.3.10.53
0.50.10.07 0.54 205.1
203.8:10.81
w. updating only macro policy
0.84 +0001
201.9=0,70
0.61
203.4:+0.4
% Improv.
5.89%
-0.02%
6.26%
-0.09%
8.70%
-0.18%
Table 3: Improvement after only macro policy update.
and the clicks under the intervention of the deployed macro policy are reported. While the clicks consistently improve, the return rate deteriorate, macro policy with the logged data collected under the updated micro policy. We compare the % change relative the deployed policy and find that the long term return rate improves, as seen in the second row of Table 2 In conclusion, while updating the micro policy improves the short term outcome as expected, it can also affect the long term outcome, requiring an update to the macro policy.
for groups. Next, we update the
Next.we evaluate the long and short-term outcomes when new macro data is available, which simulate by changing the user preferences at the macro level in the environment. In the first row of Table 3 is change the macro policy while in the second row, the macro policy is updated using the new logged macro data and the micro policy is kept unchanged. We find that the long-term improves without much change to the short-term clicks under the intervention of boosts from this updated macro policy.
Robustness to errors in the micro policy. To evaluate the robustness with varying errors in the micro policy, we
Short Opt.
Long term (Retum Rate)
simulate perturbed policy #222 arg max; 8, where
d? mg²
noisy relevance scores sampled from (sss o²)
)) with
af?
o, E {0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 2.0). Figure 13 shows long term reward this noise varies for top- 10 selection with groups. We find that learning the macro policy is ro- bust for fairly large noise o₂ and only begins deteriorate for high values of a.. This experiment also demonstrates the advantage of learning micro policy to the long term return rate metric. In real-world deployment, the micro policy may be updated periodically, so this robustness errors makes the nested bandit learning practically appeal- ing.
0.6
df?
Micro Policy Noise
Figure 13: Long term return rate with varying noise a, micro policy varies in {0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 2.0).
25

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] " is generally preferable to "[No]" it is perfectly acceptable to answer " "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often
nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims and experimental findings referenced in the abstract and the intro- duction included in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
26

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specinf well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide theoretical motivation in Section 3 and its associated derivation in Section[E]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the setup for each experiments in Section and provide details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
27

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide the code repo link for the simulator described in Section F
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
sellli//guiess.e (owoeeobiiissiooonooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc cc/ http://lllgggg..eee/eeeeeeuueeee http:///oooosubmsssisooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
28

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide experiment setting/details in Section 3 and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report performance across multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emorr bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide information compute resources in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
29

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We believe work adheres to the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: These safeguard concerns do not apply our experimental domains. Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
30

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: include the information in Section Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessii thcode wwwccoo/aaaooooo datasat has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Not applicable to this paper
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects involved
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing or human subjects involved
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the method development in this research does involve LLMSs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tttp:://neuris... cttpeiiisscco///oooo /Cco/o/rnnnce//22///////// for what should or should not described.
32


Reviews:
[{"review_id": "VUePusQt7x", "reviewer": "Reviewer_mEmM", "summary": "This paper introduces a novel MultiScale Policy Learning framework for contextual bandits aimed at optimizing long-term objectives in interactive AI systems. The key insight is to address the mismatch between short-term interventions (e.g., clicks) and long-term outcomes (e.g., user retention) by learning policies across multiple interdependent timescales. The proposed MultiScale Off-Policy Bandit Learning (MSBL) leverages abundant short-term feedback to construct informative priors for higher-level policies, enabling more sample-efficient learning from sparse long-term signals. The framework is instantiated and evaluated on simulated tasks involving two to three levels of decision-making in both recommendation and conversational system scenarios.", "questions": "1) Is there a way to update the macro policies online as new macro-level data arrives, or is the learning strictly offline?\n2) Can the proposed multi-scale framework be extended to reinforcement learning beyond bandits?\n3) While the PAC-Bayes provides useful insight, it’s not empirically tested how much the prior advantage contributes to the macro-level improvement.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1) The problem of optimizing long-term objectives in AI systems is both timely and important, especially given the prevalence of short-term optimization in practice.\n2) The hierarchical treatment of feedback across multiple timescales is novel, providing a compelling approach to reconciling short-term actions with long-term outcomes.\n3) The experimental evaluation is thorough and well-structured, spanning diverse domains such as recommendation and conversational systems.\n\nWeaknesses\n1) While the PAC-Bayesian motivation provides valuable theoretical insight, the analysis remains somewhat informal. A more rigorous treatment—such as formal guarantees or generalization bounds for the full multi-level case—would strengthen the theoretical contribution.\n2) The experimental setup, though thoughtfully designed, is primarily based on synthetic simulators. While this is reasonable for initial validation, the absence of real-world datasets or deployment severely limits the practical applicability and external validity of the results.\n3) The framework presumes the existence of feedback clearly aligned to multiple levels (e.g., clicks, weekly returns, subscriptions), which may not be available or well-defined in some domains.", "was_revised": true, "final_justification": "Most of my concerns regarding the empirical performance have been addressed thus I will maintain my score.", "rebuttal": "We thank the reviewer for their time and effort for this valuable service.\n\n**We are glad that all four reviewers unanimously acknowledge the novelty of the method with high scores along the axis of Originality**. To quote, the reviewer  noted “The hierarchical treatment of feedback across multiple timescales is novel, providing a compelling approach to reconciling short-term actions with long-term outcomes.” Reviewer enko also noted “The proposed hierarchical separation is quite general and powerful in capturing a wide range of situations.”\n\nThe concerns raised are primarily clarifications. We address each of them below.\n\n> experiment setup on synthetic simulators\n\nThe reviewers largely agree that the paper tackles a significant, timely and an important problem. \nTo progress research in this direction, we used real world datasets/ setup in existing papers [1,4,5] for level 1 and simulate medium and long term feedback (following previous works [2,3,6]). For our simulator with three levels, we describe the design and modules in Figure 6 of Appendix and open source it as part of the paper.<!-- These are semi-synthetic experiments and the best available option in the absence of open source real world datasets for this problem.  -->\nPlease also refer to our response to reviewer enko under \"empirical evaluation\".\n\n[1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment (Yang et al'24)\n\n[2] Ranking with Long term constraints (Brantley et al'24)\n\n[3] Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF (Gao et al'25)\n\n[4] Anthropic’s helpful-harmless dataset\n\n[5] KuaiRec dataset for recommender systems (Gao et al'22)\n\n[6] LLM as judge\n\n> PAC Bayesian theoretical insights vs guarantees\n\nThanks for acknowledging the valuable theoretical insight of the PAC Bayes motivation. While we provide an upper bound for improvement in required training samples in eq (4), providing a practically meaningful formal guarantee is challenging given the complexity of the models involved. We see the value of the PAC Bayes analysis in validating the principle behind our model, not quantifying the outcome. \n\n> Updating MSBL policies as new data arrives\n\nThis is a great question. The macro policy is adaptable and can be updated when new data arrives. An on policy estimator (e.g., REINFORCE policy gradient) could be used but our work is centered on off-policy learning (OPL) setting. Below we describe the scenarios and provide empirical results with and without updating L2 policy (offline) as new macro data arrives.\n\nMSPL decouples the micro and macro policy learning so that they can be updated asynchronously. \n- If new macro level data arrives (e.g changing user preferences), the family of policies at the micro level do not need to be updated. We only need to update the macro policy that selects the optimal macro intervention according to the new macro data.\nThis is advantageous as compared to methods that reviewer z6c1 referred to as incorporating L2 learning as part of L1 learning policy. For such methods, one would need to update a single monolithic deployed policy if any new data arrives. MSPL's modular design allows decoupled learning between levels. \n\n    To demonstrate this empirically, we conduct an experiment for the last setup - two levels of large scale recommendation system with the MSBL policies deployed. We simulate a scenario such that the underlying user preferences change after deployment. In the below results the first row \"w. new macro data\" shows both long and short term reward without updating the L2 policy and the second row shows the expected rewards after L2 policy is updated according to the new macro level data.\n    Across different number of user groups we can see that while the micro level (clicks) remains mostly unchanged, **only** updating macro policy results in lifting the long term.\n    | 2 Groups | Long term (Return Rate) $\\uparrow$ | Short term (Clicks) $\\uparrow$|\n    |----------|----------|----------|\n    | w. new macro data    | 0.64$\\pm$ 0.01   | 239.62 $\\pm$ 0.68    |\n    | w. L2 policy updated    | 0.65 $\\pm$ 0.01    | 239.70 $\\pm$   0.62 |\n    ||||\n    | % Improv. w. updating L2    | $\\mathbf{1.37}$ %  | $0.03$ %   |\n    \n    | 3 Groups | Long term (Return Rate) $\\uparrow$ | Short term (Clicks) $\\uparrow$|\n    |----------|----------|----------|\n    | w. new macro data    | 0.48 $\\pm$ 0.12   | 240.50 $\\pm$ 0.76    |\n    | w. L2 policy updated    | 0.54 $\\pm$ 0.07    | 240.13 $\\pm$   0.41 |\n    ||||\n    | % Improv. w. updating L2    | $\\mathbf{14.46}$ %  | $-0.15$ %   |\n    \n    | 4 Groups | Long term (Return Rate) $\\uparrow$ | Short term (Clicks) $\\uparrow$|\n    |----------|----------|----------|\n    | w. new macro data    | 0.42 $\\pm$ 0.08   | 240.48 $\\pm$ 0.78    |\n    | w. L2 policy updated    | 0.45 $\\pm$ 0.06    | 240.42 $\\pm$   0.83 |\n    ||||\n    | % Improv. w. updating L2    | $\\mathbf{8.14}$ %  | $-0.02$%   |\n- On the other hand, new micro-level data (for instance with addition of items in catalog, updates required in the base LLM policy etc.) would require updating both micro and macro policy. \n\n- Finally, this framework elegantly scales to more than two levels when new data arrives. For instance, for three levels, if new data arrived for L2, only L2 and L3 policy need to be updated. The L1 policy would remain unchanged. Below, we show with an experiment on the second setup - a three level simulator that new L2,L3 data only affects the medium and long term and the L1 policy does not need to be updated.\n\n    | 2 Groups | Long term | Medium term  |Short term  |\n    |----------|----------|----------|----------|\n    | w. new L2, L3 data    | 0.88   | 0.85    |0.52   |\n    | w. L2, L3 policy updated    | 0.98   | 0.90   |0.52    |\n    |||||\n    | % Improv. w. updating L2 & L3   | $\\mathbf{11.59 }$ % | $\\mathbf{5.76}$ %  | $0.61$%  |\n> Extending to reinforcement learning beyond bandits\n\nThis is another great point. We note in limitations, line 354 that “extension to stateful policies would be an interesting future work”. We conjecture such an extension to be straightforward especially if dependencies at the lower level stay within the timestep of the next higher level. For example, in experiment 2 (conversational recommender simulator) for the lowest level, we already append the response of the L1 LLM policy at each timestep to the next context, so that the current context depends on the action taken in previous timestep.\nHowever, a more general and fully principled extension of MSPL to MDP policies requires careful consideration and challenges related to learning MDP policies with scarce data. \n\nRegarding \n> how much the prior advantage contributes to the macro-level improvement, \n\nTo evaluate the prior advantage from micro level learning, we conduct an experiment where we inject noise into the micro learning process and see how it affects macro level rewards for the last setup - two levels of large scale recommendation system. A noise of 0.4 refers to 40% randomness in micro action selection.\nBelow , we can see that the advantage of micro level when learned (w. 0.0 noise) is significant compared to a random micro policy (w. 1.0 noise)\n\n|Noise in Micro Policy | Long term (Return Rate) $\\uparrow$| \n|----------|----------|\n| 0.0   | $\\mathbf{0.82 \\pm 0.01}$| \n| 0.4|0.81 $\\pm$ 0.01 |\n| 0.6|0.79 $\\pm$ 0.01 |\n| 1.0 (random micro policy)    | $\\mathbf{0.72 \\pm 0.01}$ |"}, {"review_id": "bXWrWmakeN", "reviewer": "Reviewer_PY8n", "summary": "The paper addresses a critical challenge in interactive AI systems, where short-term feedback is abundant, but optimizing solely for it often harms long-term objectives. The authors propose the MultiScale Policy Learning framework to reconcile the difference in feedback timescales. Specifically, they present MSBL, which leverages plentiful short-term data to establish hierarchical priors for faster learning at higher, less data-rich levels. Empirical evaluations demonstrate performance improvements on tasks related to recommender and conversational systems.", "questions": "See weakness.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n(1) Effectively integrates feedback across multiple timescales, ensuring better alignment between short-term actions and long-term objectives.\n\n(2) Uses abundant short-term feedback to inform hierarchical priors, dramatically enhancing data efficiency for learning sparse, long-term objectives.\n\nWeakness:\n\n(1) This method heavily relies on accurate short-term feedback to construct hierarchical priors for long-term optimization. If the short-term feedback is biased or noisy, will the newly formed prior adversely affect the long-term optimization? If yes, does this significantly reduce its advantages compared to directly learning from short-term feedback?\n\n(2) Implementing and optimizing multi-level hierarchical policies introduces substantial computational complexity. What is the additional computational cost incurred compared to conventional single-level feedback learning? Is this increased cost justified by the performance improvements?", "was_revised": true, "final_justification": "Thanks for the author's detailed response. My concerns are addressed, and I will keep the positive score.", "rebuttal": "We thank the reviewer for their time and effort for this valuable service.\n\n**We are glad that all four reviewers unanimously acknowledge the novelty of the method with high scores along the axis of Originality**. \n\nThe concerns raised are primarily clarifications. We address each of them below.\n\n> How does errors/noise in short term feedback affect macro policy\n\nOur experiment in Figure 13 of Appendix looks at how noise in micro policy affects macro policy learning. We evaluate for varying levels of noise by perturbing the ranking scores.\nWhile the noise at micro level does affect long term optimization, we find that empirically macro learning is still robust to fairly large values of noise at micro level and maintains its advantage over the baselines (including the baseline of only learning from short term \"short term opt\"). \nPlease also see the response to reviewer mEmM under “how much the prior advantage contributes to the macro-level improvement”.\n\n> Computational cost of hierarchical learning\n- In order to compare the computation cost we analyze the samples required for directly optimizing long term feedback versus our approach of hierarchical learning in section 3. While MSPL learns a policy at each level, eq 4 shows that hierarchical learning would be beneficial to learning a single- level policy from an uniformed prior depending on the misalignment between different timescale feedback. As reviewer  z6c1 noted “By re-using micro-level knowledge as a Bayesian prior, MSBL needs dramatically fewer long-term samples to reach reliable macro policies—crucial when retention or renewal labels are rare.” \n- Aditionally, as noted in lines 234-236,  the macro action space (decoding strategies, reward weights etc.) is much smaller than that of the micro level (rankings, token action space) so that the macro policies are progressively smaller in size."}, {"review_id": "Oqvtrl0ffF", "reviewer": "Reviewer_z6c1", "summary": "The authors tackle a common dilemma in interactive systems such as recommender platforms and chat-bots: logs are rich in short-term feedback (clicks, single-turn ratings) but the retention, satisfaction, subscription renewal unfolds on a much slower time-scale. This paper introduce MultiScale Policy Learning, a hierarchical framework that: (1)learns micro-level policies on the plentiful fast feedback; (2)treats each macro action as selecting or modifying one of those micro policies, and (3) learns a macro-level policy—and potentially further levels—using the sparse long-term signals.", "questions": "See Weakness.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Advantages\n\n\n(1) By re-using micro-level knowledge as a Bayesian prior, MSBL needs dramatically fewer long-term samples to reach reliable macro policies—crucial when retention or renewal labels are rare.  \n(2) The algorithm is a lightweight wrapper around standard off-policy bandits; it supports multiple number of levels and both “policy-modification” and “feedback-modification” ways of generating the micro-policy family. This makes it easy to plug into existing ranking, LLM decoding or recommendation stacks.\n(3) Across chat, simulated and real-world recommender tasks, MSBL consistently lifts long-horizon objectives while keeping click/engagement drops small and showing resilience to noisy features or larger action spaces.\n\n\n\nWeaknesses:\n\n(1) Lack of empirical support for the core motivation. The central motivation of the paper—that there exists a substantial disconnect between the timescale of user interactions (e.g., click or watch-time) and long-term objectives (e.g., subscription renewal)—is intuitively plausible but insufficiently substantiated. The manuscript lacks quantitative evidence demonstrating how frequent or significant this misalignment is in practice? In many real-world applications, short-term metrics often correlate well with long-term outcomes, which the paper itself implicitly assumes when using L1 feedback to construct priors for L2 learning. \n\n(2) The proposed multi-scale learning framework introduces an explicit hierarchy to separate short-term and long-term objectives. However, in many bandit settings, long-term feedback can be modeled as delayed rewards and incorporated directly into the L1 learning process using well-established credit-assignment or temporal-difference methods. It is better to justify when or why a hierarchical design offers significant advantages over this simpler alternative. \n\n(3) The current work focuses solely on the offline (off-policy) setting, assuming access to logged data and sidestepping challenges related to exploration in dynamic environments. In practical deployments, online learning systems must balance exploration and exploitation, especially when optimizing long-term outcomes. It remains unclear how the proposed framework can be extended to the online setting or whether it is compatible with well-known exploration techniques such as UCB or Thompson sampling. \n\n(4) The related work section omits several relevant and recent contributions from the neural contextual bandit community, which tackle closely related problems such as exploration, long-term reward optimization, and hierarchical modeling. Notable examples include:\n\t•\tNeural contextual bandits with UCB-based exploration \n\t•\tEE-Net: Exploitation–Exploration Neural Networks in contextual bandits\n\t•\tGraph Neural Network Bandits", "was_revised": true, "final_justification": "After rebuttal, the authors have addressed the majority of my concerns, I have increased the score from 3 to 4.", "rebuttal": "We thank the reviewer for their time and effort for this valuable service.\n\n**We are glad that all four reviewers unanimously acknowledge the novelty of the method with high scores along the axis of Originality**. \nWe are glad that the reviewer found our proposed algorithm \"easy to plug into existing ranking, LLM decoding or recommendation stacks\" and noted that \"Across chat, simulated and real-world recommender tasks, MSBL consistently lifts long-horizon objectives while keeping click/engagement drops small and showing resilience to noisy features or larger action spaces.\"\n\nThe concerns raised are primarily misunderstandings. We clarify each of them and address them below.\n\n> Empirical support for core motivation\n\n- In recommender systems, over-optimization of clicks leading to lower user retention [1, 2], challenges in aligning short, medium and long session goals [3] and challenges with competing time horizon objectives [4] have been extensively reported. For instance, [4] describes time horizon objectives in Figure 1 (short term as click through rate, and long term as customer satisfaction) and states that “long-term and short-term objectives may be competing”. They discuss existing recommendation paradigms and conclude that “Such strategies that are successful in the short term may however be non-optimal or even detrimental in the long run.” \n\n- This misalignment between micro and macro objectives has been shown to be substantial [2,8]. Specifcially, Figure 2 of [2] shows the dcg metric for the micro/short term utility of micro actions (ranking) and the macro violation for the long term constraint violation. Similarly, [8] shows the tradeoff between utility and amortized exposure (collected over a time horizon T) as the macro goal. These works assume that the macro intervention is given while our work learns it via the macro policy.\n\n\n- In conversational systems, misalignment between token level metrics (e.g., perplexity) and multiple responses/document level metric (e.g., diversity in a paragraph) has been studied both qantitatively and qualitatively in [5, 6, 7]. \n\nAs a result, there is overwhelming evidence that short-term objectives can be substantially misaligned with long-term objectives across a wide range of applications. \n\nPlease also note that most reviewers found the problem important and timely. Reviewer mEmM noted “The problem of optimizing long-term objectives in AI systems is both timely and important, especially given the prevalence of short-term optimization in practice.”, reviewer PY8n noted \"The paper addresses a critical challenge in interactive AI systems, where short-term feedback is abundant, but optimizing solely for it often harms long-term objectives.\" and reviewer enko also noted “The problem being addressed is ubiquitous in practical applications, and so quite important.”\n\n> short-term metrics often correlate well with long-term outcomes, which the paper itself implicitly assumes when using L1 feedback to construct priors for L2 learning.\n\nThis is an important and subtle point. While short term metrics provide an important signal for long term outcomes, their unmitigated optimization is not necessarily aligned with long term goals (as we provided evidence above).\nIn our work, we provide a principled way of modifying the policy or the feedback so that while there may be a sacrifice in the short term metrics (e.g clicks, perplexity of a sentence) but the modified policy is more effective at optimizing upper level reward (e.g., fewer clicks by more aggressively pruning suspected click bait can lead to better weekly returns or transforming the short term rewards to steer for long term goals). See lines 189-192 and lines 206-208 in the main text.\n\n> when/why our method provides advantage over existing flat learning methods\n\n- Existing flat learning methods such as [2,9] consider optimizing for L1 actions only (rankings). They do not learn L2 interventions such as reward weights [10], ads [11], ad pacing [12], message notifications, decoding strategies [5] etc that are available in the system and can directly steer the system towards long term objectives. \n- The literature on delayed rewards such as [9] and the related works that the reviewer pointed out model the delayed reward of a micro level action. Crucially, they do not address the situation where the delayed long term reward cannot be attributed to a single micro level action, but is the result of multiple actions. For instance, diversity among T generated responses by an LLM policy depends on all the responses generated within the T horizon; similarly, a specific exposure target as a long term goal depends on rankings over the entire T horizon [2].\n- unlike the flat learning methods [2,9], our framework extends to more than two timescales of competing objectives.\nThese flat learning methods can in fact be plugged into a single level (usually the lowest level) in the MSPL framework.\n\n\n> Focus on off-policy\n\nIn this work we focus on off policy learning (OPL) using naturally collected logged bandit data. \n- OPL is of great practical relevance, as it enables improving the system without the risky, slow, and potentially unethical use of online exploration [13,14,15]. \n- Importantly, **access to logged bandit data is not considered an assumption but is naturally collected in most real setting [13,14,15]** . \n- While one could swap out the off-policy method with an on-policy method (e.g., REINFORCE policy gradient) and develop MSPL with online estimators, it is out of the scope of the current work. Instead we discuss the deployment of MSBL policies and their adaptation at inference time (as new data arrives).\n\n\n> Adapting to dynamic environments\n\nPlease refer to the response to reviewer mEmM on “Updating MSBL policies as new data arrives”. We provide clarification on how the different levels can be updated asynchronously (offline) and provide empirical results.\n\n> Exploration-exploitation tradeoffs in online systems\n\nAddressing exploration-exploitation tradeoffs is important but orthogonal to the focus of our work. \nWe described in the response under “Updating MSBL policies as new data arrives” how policies at different levels can be updated asynchronously. With that discussion as the context, one can also update MSBL policies incorporating exploration from existing UCB or thompson sampling works to balance this tradeoff.\nTo reiterate, centering our work on off-policy learning (OPL) has its own merits and practical relevance [15].\n\n> Related works\n\nWe will move the discussion on related works in this response and that in the appendix to the main text to incorporate these clarifications.\n\n[1] Focusing on the Long-term: It's Good for Users and Business (Hohnhold et al'15)\n\n[2] Ranking with Long term constraints (Brantley et al'24)\n\n[3] A Survey on Session-based Recommender Systems (Wang et al'21)\n\n[4] A survey on multi-objective recommender systems (Jannach et al'23)\n\n[5] Adaptive Decoding via Latent Preference Optimization (Dhiliawala et al'24)\n\n[6] Perplexity by PLM Is Unreliable for Evaluating Text Quality (Wang et al'23)\n\n[7] Escaping the sentence-level paradigm in machine translation (Post et al'24)\n\n[8] Controlling Fairness and Bias in Dynamic Learning-to-Rank (Morik et al'20)\n\n[9] Impatient Bandits (Zhang et al'24)\n\n[10] From Optimizing Engagement to Measuring Value (Milli et al'21)\n\n[11] Control-based Bidding for Mobile Livestreaming Ads with Exposure Guarantee (Zhang et al'22)\n\n[12] Smart Pacing for Effective Online Ad Campaign Optimization (Xu et al'15)\n\n[13] Counterfactual Risk Minimization: Learning from Logged Bandit Feedback (Swaminathan et al'15)\n\n[14] Off-Policy Evaluation for Large Action Spaces via Policy Convolution (Sachdeva et al'23)\n\n[15] Off-Policy Evaluation and Learning in Recommender Systems (Recsys 2021 Tutorial)"}, {"review_id": "e1WowFlrtG", "reviewer": "Reviewer_enko", "summary": "The paper considers the problem of mismatched proxy rewards and a natural tradeoff that occurs in most practical settings -- the sparsity of high fidelity long term reward and a more dense proxy reward that might be easier to optimize against. The authors propose viewing this as optimizing a multi-scale policy, where the lower level effectively provides a data driven prior that the sparse higher level policy can tractable guide using the limited feedback available. The authors motivate this suggestion using a simple intuition from PAC-Bayes. The paper proposes two concrete mechanisms for this hierarchical linkage:\nPolicy Modification: A base policy is first learned to optimize the short-term reward. The higher-level policy's actions then consist of applying modifications to this base policy, such as boosting the scores of certain items in a recommendation list.\nFeedback Modification: The higher-level action specifies weights on the objective of the lower-level policy. \n\nThe paper considers several experimental analyses and instantiations of the proposed high level idea, where they find that this does lead to plausible improvements over natural baselines. A limitation is the synthetic nature of the tasks studied, which the authors acknowledge clearly.  The paper's primary contribution is the \"MultiScale Policy Learning\"  framework itself—a new conceptual model for tackling long-term objectives. The algorithm presented, MSBL, uses standard off-policy contextual bandit learning at each level of the hierarchy. The technical innovation lies not in creating a new point-wise learning algorithm, but in the architecture that connects these standard components. The novelty is in the two proposed linkage mechanisms.", "questions": "- The definition of MSP appears like a product of two distributions when presented initially in Page 4 (section 4), but at this point, it is a bit unclear whether what is meant is conditional factorization. Could you clarify?\n\n- In Figure 3, I understand the superscript L2 indicating the level, but what are the action subscripts 1 and 2?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The problem being addressed is ubiquitous in practical applications, and so quite important.\n- The proposed hierarchical separation is quite general and powerful in capturing a wide range of situations. \n- The empirical validation is extensive. The paper evaluates three distinct and relevant tasks: a multi-turn conversation task, a simulated conversational recommender system, and a recommender system built on a video streaming benchmark dataset.\n\nWeaknesses\n- It is unclear how the proposed approach performs against strongly tuned baselines in real world systems, which reasonably optimize various levels of the stack independently at any each point in time. The paper's baselines, such as the \"Level 1 opt\" policy (optimizing only short-term reward) or a policy with random macro-level actions, appear designed to clearly demonstrate the benefit of the hierarchical learning structure. However, a sophisticated real-world system might employ a more complex baseline, perhaps by training separate models for different objectives and combining their outputs with a hand-tuned business logic layer.  While the authors propose that their framework can be seen as learning the macro-level interventions that might otherwise be fixed or hand-tuned, a direct comparison to a strong, practical baseline is missing (for good reasons, since the study is not performed on a production system).\n\n- Technical novelty limited (even if the conceptual contribution is significant), with standard off the shelf off policy learning being the workhorse underneath.", "was_revised": true, "final_justification": "Conceptually novel work with reasonable empirical evaluation considering the limits of the available open source datasets. Authors have also addressed most concerns during the rebuttal, so I raise my score to an accept.", "rebuttal": "We thank the reviewer for their time and effort for this valuable service.\n\n**We are glad that all four reviewers unanimously acknowledge the novelty of the method with high scores along the axis of Originality** and the reviewer noted “The proposed hierarchical separation is quite general and powerful in capturing a wide range of situations.” Reviewer mEmM also noted “The hierarchical treatment of feedback across multiple timescales is novel, providing a compelling approach to reconciling short-term actions with long-term outcomes.” \n\nThe concerns raised are primarily clarifications. We address each of them below.\n\n> empirical evaluation and tasks\n\nWe are glad that the reviewer noted “The empirical validation is extensive. The paper evaluates three distinct and relevant tasks: a multi-turn conversation task, a simulated conversational recommender system, and a recommender system built on a video streaming benchmark dataset.”\n\n\n- Regarding the synethtic nature of the tasks studied, we emphasize these are semi-synthetic experiments and the best available option in the absence of open source real world datasets for this problem. Below we detail how each of the tasks are formed from either real world data where possible (level 1) or simulated according to existing literature (for medium and long term feedback).\n    1. Our first experiment on multi-turn conversations is based on Anthropic’s helpful-harmless dataset [4] and 7b model similar to those in [1]. <!-- The difference from [1] is that [1] does not address or take into account the long term feedback.  -->\nTo obtain long term feedback on this task, we set up a user-agent multi turn conversation where the user is instantiated with a LLama-70B. This multi-turn setup is used in [3]. We let the conversation continue for up to 5-turns and obtain the long term feedback from LLama-70B with LLM as judge [6]. \n\n    2. The second task, where we built a simulator to demonstrate three levels for a conversational recommender system, is again based on the 7b model as in [1]. We model medium and long term feedback, similar to the long term constraints formulation in [2], where the long term constraints are only observed after T steps and the constraint satisfaction depends on all the actions taken during that time period. In [2], the authors used Kauirand dataset [5] and formed long term constraints as a function of exposure targets. In our work, we similarly form long term feedback based on user preferences. The difference from [2] is that while [2] assumes the long term intervention as given, we learn it with L2 policy. \n\n  3. The third task based on a real world video streaming app uses the dataset of engagements collected from real user interactions [5]. This experiment uses real user features and their interactions with a catalog of ~5K items for level 1. We formulate the long term feedback similar to that in [2].\n    \n        [1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment (Yang et al'24)\n\n        [2] Ranking with Long term constraints (Brantley et al'24)\n\n        [3] Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF (Gao et al'25)\n\n        [4] Anthropic’s helpful-harmless dataset\n\n        [5] KuaiRec dataset for recommender systems (Gao et al'22)\n\n        [6] LLM as judge\n\n\n\n- While we agree that deployment in a production system would be an interesting next step, our key contribution is the conceptualization of the new multi-scale formulation, which the reviewers acknowledge as interesting and important - and for which we provide both a PAC-Bayesian motivation and extensive empirical evaluation of semi-synthetic data. Whether this approach performs well on a particular industrial application cannot be a requirement for publication, and most NeurIPS papers do not provide such experiments.\n\n> strongly tuned baselines in real world systems\n\n\n- Heavily engineered production systems over many years can get excellent performance through manual tuning and complex sets of business rules for effective long term behavior. We kindly emphasize that is not the point of our research. The goal of our research is to provide a principled and general method for learning systems with effective long-term behavior for a variety of domains. To validate this, we provide both a PAC-Bayesian motivation and extensive empirical evaluation on semi-synthetic baselines. \n\n\n\n- In particular, we formed the baselines similar to other accepted papers in the community [1,2]. In [1] the authors provide baselines as the base LLM policy with and without their proposed module of predicting personalized temperature. [1] does not address or model the problem of long term objectives. In [2], the baselines consist of myopic controllers (short term optimizing baseline) and other proposed controllers that optimize for different formulations of long term constraints similar to our baselines for different types of macro interventions. \n\n    [1] To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO (Qiu et al'24)\n\n    [2] Ranking with Long term constraints (Brantley et al'24)\n\n> Conceptual vs technical novelty\n\nWe thank the reviewer for acknowledging the conceptual novelty of our proposed framework that is general and powerful for optimizing long term outcomes. The reviewer also clearly described our contributions of proposing two ways of building a family of policies at micro level (policy and feedback modifications). \n\nAs we noted in the conclusion, there are many other ways of instantiating the general MSPL framework with other algorithms (and potentially new estimators) as an interesting direction for future work. \n\n> Factorization of policies\n\nThanks for raising this important point and letting us clarify. In section 4, we introduce multiscale policies by factoring a monolithic policy space $\\Pi$  as $\\Pi^{L1} \\cdot  \\Pi^{L2}$. By this, we meant that we can decompose the policy space $\\Pi$ into decoupled and independent learning at each level.\n\nAs shown in Figure 2(b), learning a family of policy from the entire policy space at Level 1 ($\\Pi^{L1}$) consists of finding a set $\\hat{\\Pi}^{L1}= \\{\\hat{\\pi}^{L1}_{a^{L2}}; a^{L2} \\in \\mathcal{A}^{L2}\\}$. The entire policy space $\\Pi^{L1}$ can be viewed as consisting of different sets of $\\hat{\\Pi}^{L1}$.\nAt level 2, we find a policy $\\pi^{L2} \\in \\Pi^{L2}$ as shown in Figure 2 ( c ).\nThe factorization refers to learning a family of policy at L1 decoupled from learning $\\pi^{L2}$ at level 2.\n\nDuring inference as shown in Figure 2(a) the selection is conditional, where $a^{L2}$ is first selected from the learned $\\pi^{L2}(a^{L2}|x^{L2})$ and the lower level action is selected according to $\\pi_{a^{L2}}^{L1} (a^{L1}|x^{L2})$  \nThis does indeed make the overall action selection a conditional product of policies $\\pi^{L2}$ and $\\pi^{L1}_{a^{L2}}$.\n\nWe will add this clarification in section 4 of the main text.\n\n\n> Action subscripts in Figure 3\n\nThe action subscripts in Figure 3 stand for static L2 interventions, where $a^{L2} \\in$  [0.8(harml-), 0.2(helpf-)], [0.2(harml-), 0.8(helpf-)].\n$a^{L2}_1$ referes to the same L1 policy as other baselines but using the fixed L2 intervention of preference weights in [0.8(harml-), 0.2(helpf-)] , while $a^{L2}_2$ refers to using the fixed L2 intervention of preference weights in [0.2(harml-), 0.8(helpf-)]. We describe this in Appendix F.2 but will clarify it in the main text."}]

Author rebuttal:

Meta-review: In the space of interactive/online learning, the paper investigates the balance between optimizing for abundant, short-term feedback (e.g., clicks, engagement) and achieving desired long-term objectives (e.g., user retention, satisfaction). The authors identify the different timescales of interventions and feedback as a key obstacle and as a solution introduce the MultiScale Policy Learning (MSPL) framework. The framework leverages abundant short-term feedback to form hierarchical priors that accelerate learning from sparse long-term signals.  A PAC-Bayesian motivation is provided to show why such hierarchical priors lead to data efficiency. Empirical evaluations on simulated conversational tasks and recommender systems demonstrate consistent improvements on long-term metrics, with minimal sacrifice of short-term performance. The reviewers were leaning positive towards the paper and noted that the paper is timely and novel, offering a lightweight wrapper around standard bandits and demonstrating consistent gains on conversational and recommender tasks. However, reviewers also raised important concerns about reliance on simulators, robustness to noise, offline nature of the framework, lack of rigorous theoretical guarantees, and limited quantitative evidence for the claimed timescale misalignment. We thank the authors and the reviewers for engaging during the rebuttal period towards improving the paper and providing additional experiments (online policy updates and robustness to noisy micro policies). The clarification in rebuttal along with new results resolved many of the reviewers concerns, with some raising their scores. Overall, while limitations remain in theory and real-world validation, the work presents a compelling and original contribution and thus I lean recommending acceptance to NeurIPS given the significance of the problem and positive reviewer consensus.
Decision: Accept (poster)