Title: MultiScale Contextual Bandits for Long Term Objectives
Venue: NeurIPS 2025
Accepted: yes
Keywords: policy learning at multiple timescales

timescale of interventions vs feedback

off policy contextual bandits
Abstract: The feedback that AI systems (e.g., recommender systems, chatbots) collect from user interactions is a crucial source of training data. While short-term feedback (e.g., clicks, engagement) is widely used for training, there is ample evidence that optimizing short-term feedback does not necessarily achieve the desired long-term objectives. Unfortunately, directly optimizing for long-term objectives is challenging, and we identify the disconnect in the timescales of short-term interventions (e.g., rankings) and the long-term feedback (e.g., user retention) as one of the key obstacles. To overcome this disconnect, we introduce the framework of MultiScale Policy Learning to contextually reconcile that AI systems need to act and optimize feedback at multiple interdependent timescales. 
Following a PAC-Bayes motivation, we show how the lower timescales with more plentiful data can provide a data-dependent hierarchical prior for faster learning at higher scales, where data is more scarce. 
As a result, the policies at all levels effectively optimize for the long-term. We instantiate the framework with MultiScale Off-Policy Bandit Learning (MSBL) and demonstrate its effectiveness on three tasks relating to recommender and conversational systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of mismatched proxy rewards and a natural tradeoff that occurs in most practical settings -- the sparsity of high fidelity long term reward and a more dense proxy reward that might be easier to optimize against. The authors propose viewing this as optimizing a multi-scale policy, where the lower level effectively provides a data driven prior that the sparse higher level policy can tractable guide using the limited feedback available. The authors motivate this suggestion using a simple intuition from PAC-Bayes. The paper proposes two concrete mechanisms for this hierarchical linkage:\nPolicy Modification: A base policy is first learned to optimize the short-term reward. The higher-level policy's actions then consist of applying modifications to this base policy, such as boosting the scores of certain items in a recommendation list.\nFeedback Modification: The higher-level action specifies weights on the objective of the lower-level policy. \n\nThe paper considers several experimental analyses and instantiations of the proposed high level idea, where they find that this does lead to plausible improvements over natural baselines. A limitation is the synthetic nature of the tasks studied, which the authors acknowledge clearly.  The paper's primary contribution is the \"MultiScale Policy Learning\"  framework itself—a new conceptual model for tackling long-term objectives. The algorithm presented, MSBL, uses standard off-policy contextual bandit learning at each level of the hierarchy. The technical innovation lies not in creating a new point-wise learning algorithm, but in the architecture that connects these standard components. The novelty is in the two proposed linkage mechanisms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The problem being addressed is ubiquitous in practical applications, and so quite important.\n- The proposed hierarchical separation is quite general and powerful in capturing a wide range of situations. \n- The empirical validation is extensive. The paper evaluates three distinct and relevant tasks: a multi-turn conversation task, a simulated conversational recommender system, and a recommender system built on a video streaming benchmark dataset.\n\nWeaknesses\n- It is unclear how the proposed approach performs against strongly tuned baselines in real world systems, which reasonably optimize various levels of the stack independently at any each point in time. The paper's baselines, such as the \"Level 1 opt\" policy (optimizing only short-term reward) or a policy with random macro-level actions, appear designed to clearly demonstrate the benefit of the hierarchical learning structure. However, a sophisticated real-world system might employ a more complex baseline, perhaps by training separate models for different objectives and combining their outputs with a hand-tuned business logic layer.  While the authors propose that their framework can be seen as learning the macro-level interventions that might otherwise be fixed or hand-tuned, a direct comparison to a strong, practical baseline is missing (for good reasons, since the study is not performed on a production system).\n\n- Technical novelty limited (even if the conceptual contribution is significant), with standard off the shelf off policy learning being the workhorse underneath.\n\nQUESTIONS:\n- The definition of MSP appears like a product of two distributions when presented initially in Page 4 (section 4), but at this point, it is a bit unclear whether what is meant is conditional factorization. Could you clarify?\n\n- In Figure 3, I understand the superscript L2 indicating the level, but what are the action subscripts 1 and 2?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle a common dilemma in interactive systems such as recommender platforms and chat-bots: logs are rich in short-term feedback (clicks, single-turn ratings) but the retention, satisfaction, subscription renewal unfolds on a much slower time-scale. This paper introduce MultiScale Policy Learning, a hierarchical framework that: (1)learns micro-level policies on the plentiful fast feedback; (2)treats each macro action as selecting or modifying one of those micro policies, and (3) learns a macro-level policy—and potentially further levels—using the sparse long-term signals.\n\nSTRENGTHS AND WEAKNESSES:\nAdvantages\n\n\n(1) By re-using micro-level knowledge as a Bayesian prior, MSBL needs dramatically fewer long-term samples to reach reliable macro policies—crucial when retention or renewal labels are rare.  \n(2) The algorithm is a lightweight wrapper around standard off-policy bandits; it supports multiple number of levels and both “policy-modification” and “feedback-modification” ways of generating the micro-policy family. This makes it easy to plug into existing ranking, LLM decoding or recommendation stacks.\n(3) Across chat, simulated and real-world recommender tasks, MSBL consistently lifts long-horizon objectives while keeping click/engagement drops small and showing resilience to noisy features or larger action spaces.\n\n\n\nWeaknesses:\n\n(1) Lack of empirical support for the core motivation. The central motivation of the paper—that there exists a substantial disconnect between the timescale of user interactions (e.g., click or watch-time) and long-term objectives (e.g., subscription renewal)—is intuitively plausible but insufficiently substantiated. The manuscript lacks quantitative evidence demonstrating how frequent or significant this misalignment is in practice? In many real-world applications, short-term metrics often correlate well with long-term outcomes, which the paper itself implicitly assumes when using L1 feedback to construct priors for L2 learning. \n\n(2) The proposed multi-scale learning framework introduces an explicit hierarchy to separate short-term and long-term objectives. However, in many bandit settings, long-term feedback can be modeled as delayed rewards and incorporated directly into the L1 learning process using well-established credit-assignment or temporal-difference methods. It is better to justify when or why a hierarchical design offers significant advantages over this simpler alternative. \n\n(3) The current work focuses solely on the offline (off-policy) setting, assuming access to logged data and sidestepping challenges related to exploration in dynamic environments. In practical deployments, online learning systems must balance exploration and exploitation, especially when optimizing long-term outcomes. It remains unclear how the proposed framework can be extended to the online setting or whether it is compatible with well-known exploration techniques such as UCB or Thompson sampling. \n\n(4) The related work section omits several relevant and recent contributions from the neural contextual bandit community, which tackle closely related problems such as exploration, long-term reward optimization, and hierarchical modeling. Notable examples include:\n\t•\tNeural contextual bandits with UCB-based exploration \n\t•\tEE-Net: Exploitation–Exploration Neural Networks in contextual bandits\n\t•\tGraph Neural Network Bandits\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper addresses a critical challenge in interactive AI systems, where short-term feedback is abundant, but optimizing solely for it often harms long-term objectives. The authors propose the MultiScale Policy Learning framework to reconcile the difference in feedback timescales. Specifically, they present MSBL, which leverages plentiful short-term data to establish hierarchical priors for faster learning at higher, less data-rich levels. Empirical evaluations demonstrate performance improvements on tasks related to recommender and conversational systems.

STRENGTHS AND WEAKNESSES:
Strengths: 

(1) Effectively integrates feedback across multiple timescales, ensuring better alignment between short-term actions and long-term objectives.

(2) Uses abundant short-term feedback to inform hierarchical priors, dramatically enhancing data efficiency for learning sparse, long-term objectives.

Weakness:

(1) This method heavily relies on accurate short-term feedback to construct hierarchical priors for long-term optimization. If the short-term feedback is biased or noisy, will the newly formed prior adversely affect the long-term optimization? If yes, does this significantly reduce its advantages compared to directly learning from short-term feedback?

(2) Implementing and optimizing multi-level hierarchical policies introduces substantial computational complexity. What is the additional computational cost incurred compared to conventional single-level feedback learning? Is this increased cost justified by the performance improvements?

QUESTIONS:
See weakness.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel MultiScale Policy Learning framework for contextual bandits aimed at optimizing long-term objectives in interactive AI systems. The key insight is to address the mismatch between short-term interventions (e.g., clicks) and long-term outcomes (e.g., user retention) by learning policies across multiple interdependent timescales. The proposed MultiScale Off-Policy Bandit Learning (MSBL) leverages abundant short-term feedback to construct informative priors for higher-level policies, enabling more sample-efficient learning from sparse long-term signals. The framework is instantiated and evaluated on simulated tasks involving two to three levels of decision-making in both recommendation and conversational system scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1) The problem of optimizing long-term objectives in AI systems is both timely and important, especially given the prevalence of short-term optimization in practice.\n2) The hierarchical treatment of feedback across multiple timescales is novel, providing a compelling approach to reconciling short-term actions with long-term outcomes.\n3) The experimental evaluation is thorough and well-structured, spanning diverse domains such as recommendation and conversational systems.\n\nWeaknesses\n1) While the PAC-Bayesian motivation provides valuable theoretical insight, the analysis remains somewhat informal. A more rigorous treatment—such as formal guarantees or generalization bounds for the full multi-level case—would strengthen the theoretical contribution.\n2) The experimental setup, though thoughtfully designed, is primarily based on synthetic simulators. While this is reasonable for initial validation, the absence of real-world datasets or deployment severely limits the practical applicability and external validity of the results.\n3) The framework presumes the existence of feedback clearly aligned to multiple levels (e.g., clicks, weekly returns, subscriptions), which may not be available or well-defined in some domains.\n\nQUESTIONS:\n1) Is there a way to update the macro policies online as new macro-level data arrives, or is the learning strictly offline?\n2) Can the proposed multi-scale framework be extended to reinforcement learning beyond bandits?\n3) While the PAC-Bayes provides useful insight, it’s not empirically tested how much the prior advantage contributes to the macro-level improvement.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}