Title: Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bandits

Generative Models

Large Language Models

Mobile Health

Personalized Interventions

Sequential Decision Making
Abstract: Recent advances in generative artificial intelligence (GenAI) models have enabled the generation of personalized content that adapts to up-to-date user context. While personalized decision systems are often modeled using bandit formulations, the integration of GenAI introduces new structure into otherwise classical sequential learning problems. In GenAI-powered interventions, the agent selects a query, but the environment experiences a stochastic response drawn from the generative model. Standard bandit methods do not explicitly account for this structure, where actions influence rewards only through stochastic, observed treatments. We introduce generator-mediated bandit-Thompson sampling (GAMBITTS), a bandit approach designed for this action/treatment split, using mobile health interventions with large language model-generated text as a motivating case study. GAMBITTS explicitly models both the treatment and reward generation processes, using information in the delivered treatment to accelerate policy learning relative to standard methods. We establish regret bounds for GAMBITTS by decomposing sources of uncertainty in treatment and reward, identifying conditions where it achieves stronger guarantees than standard bandit approaches. In simulation studies, GAMBITTS consistently outperforms conventional algorithms by leveraging observed treatments to more accurately estimate expected rewards.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the notion of Generator-Mediated Bandits with Thompson Sampling. These bandit methods take into account a two stage problem architecture, where the bandit's output is fed to an LLM along with additional context, and the output of the LLM is what determines the reward. This is primarily motivated by personalized decision-making in mobile health contexts, where study participants are at the receiving end of text based intervention prompts. The proposed method, attributed to as GAMBITTS, takes this two stage problem into account by modeling the action as well as the reward generated - part of which is achieved by projecting the LLM's output into a lower dimension embedding. The paper provides regret bounds and experimental results wrt the methods proposed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well written!\n2. The methods proposed to model the LLMs output is very interesting and novel to me.\n\nWeaknesses and Questions:\n1. The paper talks about personalizing JITAIs or pJITAIs on line 39-40, yet none of the references (40 and 26) even mention or elaborate that term - they only talk about JITAIs. Can the paper please be edited to cite proper references?\n2. Line 269 - What are the definition of these dimensions? While some may be apparent like encouragement, it is difficult to ascertain what severity means. Also, what does it mean to have encouragement of 1 - does it mean the prompt is discouraging or just not encouraging (there is a subtle difference between the two). Related - Table F.3.3: What does -0.9 correlation mean between optimism and encouragement?\n3. The authors should be upfront that this paper assumes that there exists an LLM with nice properties like safety. I could only find a small one-liner wrt this in the Discussion section - while adoption of LLMs in these mHealth contexts are held back by the lack of safety guarantees.\n4. Some of the details wrt experimentation should be moved into the main paper. \n5. It would be interesting to see if the results are different with a different LLM. Especially for something like Figure 10.\n6. Appendix E.3 - How is one supposed to interpret the VAE score?\n7. Appendix E.3 - Why 1-10 scaling? Why not use low medium high instead? Given that messages are short in length, I personally do not expect much difference between messages of score 1 and of score 3 for a given prompt. The stochasticity combined with this gives a noisy Figure 10.\n8. Appendix E.3 - Continued from above, seems like the VAE just learnt noise. Why not instead use IRR to assign labels, and let the VAE learn those labels corresponding to where those messages fall under the given category instead? The problem currently is that because it is learning noise, the VAE score is extremely unreliable. \n9. Perception of a message is very \"human\", and it would differ across people. The entire point of Gen-AI is to deliver personalized intervention text. However using a single, population-level scoring does not reflect the heterogeneity in individual perceptions. The paper should address that, or acknowledge it.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Thompson sampling based algorithm for contextual bandits that interact with the environment through a generative model. The action taken by the algorithm is a prompt that is fed into the generative model. The response that is put into the environment is a sample from the generative model. They relate their approach to causal graphs in RL and provide theoretical guarantees for their approach under assumptions on the causal graph. They also empirically evaluate their approach and show that their method improves upon standard Thompson sampling empirically.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The problem tackled by this work is well-motivated and timely.\n- The introduction and problem motivation was very well-written\n- The algorithm seems relatively straightforward to understand and has both theoretical guarantees (under assumptions) and performs well empirically in simulations\n\n## Weaknesses\n- I found the “Problem Formulation and Notation” section 3 hard to follow. I feel it would benefit from a rewrite that discusses what all the variables are in a specific problem setup. For example, I am not sure exactly what the context $X_t$ is here (sometimes I thought this was the prompt). Then I wasn’t clear why the action space was finite (as I thought the prompt space could be very large). Should I be thinking about $A$ as a policy and $G$ as a sample from that policy? The general problem setup is still confusing for me.\n- The biggest weakness I see (which could be clarified) is why the action does not depend on the context $X$ in Figure 1. It seems like it is a contextual bandit problem, but the algorithm is not allowed to use the context to select actions, which seems odd.\n\nQUESTIONS:\n- I found the Z and Z* discussion sometimes a bit hard to follow. After reading more closely I think I understand it, but it might help to better clarify what are the assumptions of your algorithm and what is an assumption on the data generating environment. For example, in Algorithm 1, I spent a some time thinking about in the integration in step 3 what distribution of z was being integrated over Z or Z*. I’m pretty sure it is Z, but might help to make this more overt for future readers.\n- When I was reading, in step 3 of algorithm 1, isn’t the action space (the space of all possible prompts) huge? I feel there needs to be more acknowledgement of the computational challenges in the main text (even if you say you will discuss it further in a future section), as it made me think either this algorithm is not practical or I am misunderstanding what an example action space could be. Later in algorithm 1, there is X_t being used to select actions. Could you clarify this? Also is this $X_t$ or $X_t^G$?\n- Could you talk a bit more about what “context is sent to generator”? It appears this is not learned by the algorithm but pre-specified?\n\nLIMITATIONS:\nI feel the authors could have a more thorough discussion of the limitations of their algorithm, e.g., when it is appropriate to apply versus not. As well as discuss computational challenges of implementing their algorithm further.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn GenAI-powered interventions, the agent selects a query, but the environment experiences a stochastic response drawn from the generative model. Standard bandit methods do not explicitly account for this structure, where actions influence rewards only through stochastic, observed treatments. This paper introduces generator-mediated bandit - Thompson sampling (GAMBITTS), a bandit approach designed for this action/treatment split, using mobile health interventions with large language model-generated text as a motivating case study. GAMBITTS explicitly models both the treatment and reward generation processes, using information in the delivered treatment to accelerate policy learning relative to standard methods. The authors establish regret bounds for GAMBITTS by decomposing sources of uncertainty in treatment and reward, identifying conditions where it achieves stronger guarantees than standard bandit approaches. The empirical results show that GAMBITTS consistently outperforms conventional algorithms by leveraging observed treatments to more accurately estimate expected rewards.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe studied problem, generator-mediated bandits, is related to large language models (LLMs) and may receive attention in the online learning and LLM community.\n2.\tThe authors propose algorithmic instantiations for the GAMBITTS framework, and provide regret bounds for GAMBITTS.\n3.\tEmpirical results show the limitations of standard bandit algorithms and the performance gains achieved by the proposed GAMBITTS approaches.\n\nWeaknesses:\n\n1.\tThe technical novelty in algorithm design and theoretical analysis is unclear. It seems that this paper just applies Thompson sampling to the proposed generator-mediated bandit formulation.\n2.\tThe title mentioned “GenAI-Powered”, but it is unclear how the generator-mediated formulation or algorithm brings benefits to the decision-making process in theoretical results or experiments, compared to standard (no generator involved) bandit problem or standard Thompson sampling algorithm. The paragraph following Theorem 3 discusses the advantage when $d<<\\sqrt{CK}$. But this seems an advantage brought by a low-dimensional representation. How can we understand the benefits brought by generator for the decision-making or learning process?\n3.\tThe writing of this paper needs to be improved. The authors try to sell some big concepts like “GenAI”, but it is hard to understand the technical contributions of this paper.\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nPlease see the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNew contextual bandit model where the planner action is a query to an LLM which then goes to the environment (e.g., mHealth patient) potentially generating a reward. Relates to a broader literature where actions are stochastic, but with the ultra-high dimensionality of a language model. They give regret analysis and empirical results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nGAMBITTS framework seems to be new and relevant, will likely be relevant to broad set of researchers\nThey provide algorithms for solving as well as theoretical analysis\nThey provide empirical analysis on real-world derived datasets\nRelated work is sufficient, good connections to the literature on bandits with randomized actions, good connections to literature on natural language as an action space.\n\nWeakness\nMy main takeaway from this paper so far is that the most important thing a planner can do is learn a good projection from action space to embedding space before the policy execution/learning starts… which is a departure from the bandit part of the problem altogether. And further, the problem of learning the projection isn’t explored. Maybe this models some realistic scenario, but it does not capture what seem to be the richer parts of the problem from a methodological perspective. This is likely balanced by the value of introducing the formal model.\nEvaluation is somewhat lacking. There is no exploration of a case where the reward is a function of context, despite the choice to model it. What is the reward model of 6.3? Why is the contextual TS a “more natural” comparison, and if so, why not include it in the main text figures?\n\nQUESTIONS:\nWhat is the reward model of 6.3? \n\nWhy is the contextual TS a “more natural” comparison, and if so, why not include it in the main text figures?\n\nCan you comment on steps one might take toward learning the mapping from G to Z, in either an offline or online way? This will be key to adapting the framework to new domains where planners may have less informed priors on reward structures.\n\nCan you comment on the generality of GAMBITTS – other applications where, as you mention, there may be an interest in learning some randomized embedded action dimension… perhaps there is a connection to causal MDPs, where this is a known latent structure? https://proceedings.mlr.press/v177/lu22a/lu22a.pdf \n\nShould add related work on LLMs and bandits… much study on the LLM as the agent \nhttps://arxiv.org/html/2410.06238v1\nhttps://arxiv.org/pdf/2403.15371\n\nLIMITATIONS:\nyes, authors have a nice paragraph on limitations and future work", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GAMBITTS (Generator-Mediated Bandit–Thompson Sampling), a novel bandit framework that explicitly models environments where agent actions (queries/prompts) produce stochastic treatments (e.g., LLM-generated content), which in turn influence rewards (e.g., user response). Motivated by adaptive mobile health interventions, the authors formalize a generator-mediated causal structure, propose both fully online and partially online variants of Thompson Sampling algorithms (foGAMBITTS and poGAMBITTS), and offer theoretical regret bounds. They also propose ensemble extensions for nonlinear reward modeling and empirically demonstrate improved performance over standard bandits in simulations based on real-world mHealth studies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- GAMBITTS is well-motivated and grounded in causal reasoning, and both algorithmic and theoretical components are clearly articulated.\n- The regret bounds offer insightful decompositions of uncertainty from treatments and rewards.\n- While motivated by mHealth, the approach is broadly applicable to marketing, education, and other personalization domains involving LLMs.\n\n\nWeaknesses:\n- Performance critically depends on the quality of the treatment embedding (Z), yet embedding learning is left to future work. In practice, choosing or learning Z is nontrivial.\n- While simulations are well-designed, evaluation on real-world GenAI-intervention data is not yet feasible and leaves empirical validation incomplete.\n- The framework assumes the generator does not adapt or improve, which might limit its applicability in settings where models are updated over time.\n\nQUESTIONS:\nCould GAMBITTS be extended to incorporate uncertainty or feedback on the generator output quality (e.g., hallucination risk, toxicity)? In high-dimensional cases, could contrastive or self-supervised embedding learning methods be integrated to learn Z on the fly?\n\nLIMITATIONS:\nWhile the authors have discussed the limitations of the methods, an additional discussion on the limitations of practical applications would be beneficial to enhance the paper's strength.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}