Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventions
Authors: Marc Brooks
Gabriel Durham
Kihyuk Hong
Ambuj Tewari
Venue: NeurIPS 2025 poster ()
Keywords: Bandits
Generative Models
Large Language Models
Mobile Health
Personalized Interventions
Sequential Decision Making
Primary area: reinforcement_learning
TL;DR: We propose a bandit framework for GenAI-powered decisions where actions (queries) affect the environment only through stochastic model outputs, and introduce algorithms tailored to this setup.

Abstract: Recent advances in generative artificial intelligence (GenAI) models have enabled the generation of personalized content that adapts to up-to-date user context. While personalized decision systems are often modeled using bandit formulations, the integration of GenAI introduces new structure into otherwise classical sequential learning problems. In GenAI-powered interventions, the agent selects a query, but the environment experiences a stochastic response drawn from the generative model. Standard bandit methods do not explicitly account for this structure, where actions influence rewards only through stochastic, observed treatments. We introduce generator-mediated bandit-Thompson sampling (GAMBITTS), a bandit approach designed for this action/treatment split, using mobile health interventions with large language model-generated text as a motivating case study. GAMBITTS explicitly models both the treatment and reward generation processes, using information in the delivered treatment to accelerate policy learning relative to standard methods. We establish regret bounds for GAMBITTS by decomposing sources of uncertainty in treatment and reward, identifying conditions where it achieves stronger guarantees than standard bandit approaches. In simulation studies, GAMBITTS consistently outperforms conventional algorithms by leveraging observed treatments to more accurately estimate expected rewards.

Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventions
Mare Brooks'; Gabriel Durham'; Kihyuk Hong', Ambuj Tewari! Depaarmeent of Statisties, University of Michigan, Ann Arbor, MI (USA) (marcbr, gjdurham, kihyukh, tewaria) @umi ch, edu
Abstract
Recent advances in generative artificial intelligence (GenAl) models have enabled the generation of personalized content that adapts to up-to-date user context. While personalized decision systems often modeled using bandit formulations, the integration of GenAl introduces new structure into otherwise classical sequential learning problems. In GenAl-poweree interventions, the agent selects query, but the environment experiences a stochastic response drawn from the generative model. Standard bandit methods do not explicitly account for this structure, where actions influence rewards only through stochastic, observed treatments. We introduce generator-man mediated meedittto-mediated bundi--Thompson sampling (GAMBITTS), a bandit approach designed for this action/treatment split, using mobile health interventions with large language mooel-geneerted text as motivating case study. GAMBITTS explicitly models both the treatment and reward generation processes, using information in the delivered treatment to accelcrate policy learning relative to standard methods. We establish regret bounds for GAMBITTS by decomposing sources of uncertainty in treatment and reward, identifying conditions where it achieves stronger guarantees than standard bandit approaches. In simulation studies, GAMBITTS consistently outperforms conventional algorithms by leveraging observed treatments to more accurately estimate expected rewards.
Introduction
Bandit algorithms are ubiquitous in the design of personalized interventions as they provide flexible framework for learning from context and outcomes over time. Recently, researchers have begun integrating generative artificial intelligence (GenAl) models into these systems to enable richer forms of personalization in areas such as healthcare, e.g., patient care and mobile health (mHealth)
2 23), personalized education [24], music [48), and marketing (28). By generating rich, tailored content on-the-fly, GenAl integration expands the scope of personalization beyond what fixed libraries can offer, since predefined intervention options unable accommodate the full range of user contexts encountered in practice. While this real-time generative capability opens new possibilities for intervention design, it disrupts the action/reward feedback loop that underpins traditional bandit learning. Rather than directly acting on the environment, the agent selects a to large language model (LLM) and the environment receives the stochastic output generated in response to that query, which we refer to as the "treatment." " This separation between the agent's action and the delivered treatment introduces additional stochasticity into the intervention process and motivates the need for algorithms that explicitly account for the generative mechanism.
e.g., prompt
Broadly, focus online decision-make making problems where the treatment delivered the envi- ronment is a random function of the agent's 's action. As a motivating case study, we consider the use of LLMs in sequential mHealth interventions. Traditionally, mHealth systems deliver text-based
Equaa contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

support using data collected from smartphones and wearables, adapting to rapid fluctuations in user context, physiology, and behavior 33 50 Bandit algorithms have become workhorse for this
sequential personalization, with personalized Just-in-Time Adaptive Interventions (pJITAIs) serving
as a popular setting in which such methods are developed and evaluated [5 33 14
This framework for mHealth intervention has inspired countless recent advances in bandit algorithms; e.g., allowing partial pooling information across users [19 52 20], accommodating non-stationarity to adapt to shifting user behavior patterns [33, and integrating high-dimensional covariates [3]. While these advancements further optimize bandit learning within this intervention framework, such approaches are fundamentally limited by reliance on fixed library of text intervention options.
Dynamic message generation via LLMs offers path forward. Throughout our analyses, assume access to a generator whose responses are guaranteed to be safe and suitable for delivery (see Section 7 In this setting, the agent selects a query based on user context, submits both query and user context to the LLM, and delivers the generated text response as the treatment. The user then produces reward signal (e.g.. behavior or engagement). Crucially, while the agent selects the query, it cannot control the LLM's M's response.
This setup motivates bandit framework where the agent's action affects the environment only through a digh-dimensional, stochastic treatment. Standard bandit algorithms are not designed to exploit this indirect treatment mechanism, leading to inefficient learning by failing make use of the structure linking treatments to rewards. Section 3 more precisely defines this generator-mediated bandit (GAMBIT) framework.
We develop Thompson sampling- based framework tailored to this action/treatment split, incorporat- ing explicit models of the treatment and reward generation processes. This framework serves as the backbone of our approach due to its central role in PJITAI designs, which often rely on Thompson sampling both for its theoretical guarantees and for its compatibility with downstream causal analyses using logged data, a key consideration in mobile health research [33. We call the proposed approach mediated bandit- Thadm-n sampling (GAMBITTS).
GAMBITTS selects actions by reasoning over both the distribution of treatments induced by each query and the reward those treatments generate, using a two-stage sampling procedure that captures uncertainty across the intervention process. settings where the agent has simulation access the treatment- -eateeetnnggerraiigggegee mechanism, this structure enables offline estimation of the treatment model, accelerating online adaptation. To address the high dimensionality of treatments in generative settings (e.g.. LLM-generated text), GAMBITTS projects each observed treatment into a fixed low-dimensional representation, enabling scalable and sample-efficient learning.
This work responds to the core challenge of learning when actions yield stochastic, observed treat- ments. The primary contributions are: (a) the GAMBITTS framework for bandit learning with stochastic treatments, (6) algorithmic instantiations of GAMBITTS that vary in their approach to modeling the creatennt-eeneration process, ranging from fully online updates approaches that pretrain the treatment model, as well as ensemble-based variants that support nonlinear reward modeling, (c) theoretical regret bounds for GAMBITTS, including decompositions that isolate con- tributions from treatment and reward uncertainty, showing when modeling treatment generation leads to improved learning efficiency, and extending guarantees to flexible, nonlinear reward models, (d) empirical results demonstrating the limitations of standard methods and the performance gains achieved by GAMBITTS-bar approaches.
2 Related Work
In the bandit literature, researchers have illustrated how leveraging the causal structure underlying sequential interventions can improve policy learning 30 B5). Subsequent work has supported this conclusion, developing bandit algorithms wide variety of causal structures [57] [56]. These
approaches model a set of binary, intercomnected random variables X = {XI..... Xx1 } with general causal pathways to a reward Y. Actions are framed as interventions on subsets of X, and algorithms
are aimed at optimizing reward in settings where the action space corresponds to modifying causally
linked features that influence Y. In parallel, Sen et al. [2017] and Maiti et al. [2022] explore related
environments where unobserved confounding can impact bandit learning 45 38

---

While the works discussed above emphasize general causal structures with limited treatment represen- tations, our focus is the inverse: specific mediation structure, where actions influence reward through an observed stochastic intermediate, but with rich, high-dimensional treatments (i.e., the intermediate generative output). This designed mediation, where the effect of the agent's action operates through the generator's s stochastic response, parallels the instrumental variable (IV) framework, in which an
instrument affects outcomes only indirectly via stochastic channel [22]. Recent work has connected
IV estimation and bandit design in classical econometric settings [58] [ while related research has explored similar structures under noncompliance, where the realized action may differ from the one selected by the agent 49 29] Kallus [2018] makes this connection explicit through the instrument armed bandit framework [25]. Most recently, Zou et al. [2025] study mediated bandit
environments from psychometric perspective [59].
Appendix A discusses further connections and distinctions between our framework and the methods discussed above, along with recent work on LLMs and bandits and related, application- application-focused literature from the mHealth community.
3 Problem Formulation and Notation
We consider sequential decision making setting in which, at each time = ,...., T, an agent: (i) observes context X₃ E X C R& (ti) selects action A, E A. where |.A| = K N, (tii sends (A,, X²) to a generator, (iv) receives response G2 E G and delivers G2 to the environment, and (v) observes reward Y; R. The goal of the agent is to learn policy for choosing actions that maximizes cumulative expected reward over time.
As concrete example, consider an mHealth intervention designed to encourage physical activity. At time I, the agent: (i) observes user context/covariates X, (e.g., steps taken the previous day and current location), (fi) selects action A,, corresponding to query from finite set of options (e.g., "Please write a message that encourages the user to walk more; have attached the user's current location and previous day's 's step count"), (iii sends the query, along with the context, to an LLM, (iv) delivers the LLM's response, Gr, to the user, and (v) observes reward Y, (c.g., user's subsequent step count).
In our motivating example, G represents the output of an LLM, and the treatment space & consists of natural language text. In such GenAl settings, G is exceedingly high-dimensional, making it natural to posit the effect of G' on Y occurs through a lower-dimensional embedding Z' E 2" C Idd" which serves sufficient representation of the treatment for predicting reward. Recent work in high- high-dimensional causal inference supports this assumption, showing that such lower-dimensional embeddings can yield representations sufficient for treatment effect estimation (see, e.g., Veitch et al. [2019]) [53] Because 7" is not observed in practice, the analyst must instead specify working embedding ((C) = E Z C Rdd. We discuss strategies for constructing h in Section Furthermore, in practice, the agent may also restrict which components of X, are passed to the generator; we discuss this case Appendix B.I
X
A
G
Y
Z'
Figure 1:
Generator- Generator-Mediated Bandit Causal Structure
Dotted lines represent deterministic relationships. (ii Arrows from
Notes:
are omitted represent data- generating process rather than decision logic.
(ii) A: Action/Query; X: Full Context: G: Generated Response; Z": True Response Embedding:
Y: Reward.
3

---

We refer to the resulting learning setup generator-mediare bandit (GAMBIT) problem. This causal structure summarized in the directed acyclic graph (DAG) in Figure !
Methodology: Generator-Mediated Bandit-Thompson Sampling
As discussed above, seek Thompson sampling- based framework for our bandit agent, in line with its prevalence mHealth interventions. We observe two key conditional structures to model:
A, X, and E [Y X,
which correspond to the treatment and reward models. Modeling these two components comprises the two stages of the proposed Thompson sampler and induces the following parametrized environment:
Treatment model: Ze ~ f1 (2; A,, X1, 01)
Treatment prior: 01 ~ T1
Reward model: E [Y, Zz, X,] mg (Z X,; 02)
Reward prior: 0g 722
The construction of working treatment representation
l(() depends on the application domain. In marketing applications, for instance, G may represent Al-generated images, while in our moti- vating mHealth setting. G' corresponds to LLM-generated text, necessitating different approaches representation. For implementation. all algorithmic steps operate on the observed working em- bedding Z, not the unobserved benchmark Z". While the GAMBITTS framework is agnostic to the form of h, the choice can substantially influence empirical performance. The effectiveness of the GAMBITTS projection approach depends on how well Z preserves reward-relevant information; when Y 11 Z" Z, X, modeling Y Z,X X captures the full signal in Z (and thus that in as well). As the conditional dependence Y Z, X diverges from Y Z X, performance may degrade. explore this phenomenon empirically in Section and further discuss robustness to embedding misspecification in Appendix C In our motivating setting, many projection strategies have been 54].
proposed for natural language-baser interventions; see, e.g., T8 53
We propose two general approaches for GAMBITTS implementation: fully online variant that updates both treatment and reward models sequentially, and partially online variant that pretrains the treatment model using offline data and updates only the reward model during deployment. To improve flexibility and accommodate nonlinear reward relationships, we extend both variants with the ensemble-based sampling strategies introduced in Lu and Van Roy [2017] 36
The algorithms below focus on settings where the decision to intervene (e.g., ., send the user a text message) has already been made. In practice, mHealth interventions often include an initial decision of whether to send a message at all; GAMBITTS preliminary selection step, as discussed in Appendix B.2
be trivially extended to accommodate this
4.1 Fully Online Generator-Mediate Bandit- Sampling
Let D, = (X,, As, G., Y.) 1-1
s=1 represent the data collected up to time f. Algorithm below describes fully online approach for selecting action A, based D,-
Algorithm Fully Online GAMBITTS (FOGAMBITTS)
Inputs: Data D,, priors TT, n2, models f1, m2, current context Zt.
1: Derive posterior distributions Pı (0, Dz) and Pa (02 D.). 2: Sample 01, Pi (0, Dz) and 02 Pa (0z D.).
3: For each a E A, calculate
02, [Y [|,az]] - = m2 (2,21:
(1 ( d1.
4: Set A, arg max E0,,, Y x.))
e'EAA
The integral in Step 3 can often intractable in closed form, but can efficiently approximated via Monte Carlo approaches. Furthermore, because GAMBITTS selects agents via posterior sampling,

---

induces well-defined randomization probabilities that can be used for downstream causal inference or offline evaluation. Specifically, the probability of selecting action a at time t
Pi.e-
1
Pi (01 (2) (02 D.) d0
a = arg max E, 02 [Y [
a'EA
In settings where these probabilities are required for post hoc causal analyses (e.g., analysis of pJITAIs), the full set ( can be approximated retrospectively, after the study period, when additional computational resources are available A 33 Alternatively. if sufficient computing power is available deployment time, the agent may compute these probabilities online and sample from them directly. As with the integral in Step 3 the quantities Pi,a can be estimated via Monte Carlo sampling from the posterior distributions over and 0z.
4.2 Partially Online Generator-Mediater Bandit-Thompson Sampling
In settings where the agent has simulation access to the generator, or access to additional data sources, it may be possible to learn the treatment model independently of user interactions. For example, in our motivating application, an mHealth intervention designer could query an LLM multiple times without sending any generated text users.
This capability motivates partially online GAMBITTS (poGAMBITTS). The PoGAMBITTS framework exploits simulation access the generator to offload learning of the first-stage distribution, Z A, X to an offline phase. Empirically, this decomposition can substantially improve online performance, as discussed in Section
Here, we consider the generator to be random function, which the agent can query using prompt-context pairs (a. E A x X. In the PoGAMBITTS framework, the agent can obtain an offline approximation the treatment distribution fi by: (i) simulating Mofs prompt-context pairs aa (ii) using the generator to simulate G(a*,=*), with gf (a', x²)
{(a ........ (a denoting the observed output and of denoting its observed Z embedding, for = ,,...,,osf.. and (iii) taking forf A2; X,) to be the empirical conditional distribution observed in D == (a',x', ))), Marr
In cases where X is finite, the agent can repeatedly prompt the generator for each prompt-context pair (a, E A x and directly use the empirical distribution of A, X in DofS as foff When X is infinite, the agent must instead fit model to estimate the conditional distribution of Z A, based on D and use this estimate as the treatment model.
Algorithm 2 Partially Online GAMBITTS (PoGAMBITTS)
Inputs: Data D,, prior T2, model m2, current context Tt, pretrained model /
1: Derive the posterior distribution Pa (02 D.).
2: Sample 02 ~ P2 (02 Dz).
3: For each a E A. calculate
Es,, [Y a, [r] [ / m2 (2,01;02,1) pofs (2;a, xz)
4: Set A, = arg max Eg,,. [Y] xt) e'EAA
As in foGAMBITTS, the probability of selecting action at time can be written explicitly, here as
Pa (02 d02,
arg max Ea, [Y a', d',x:)
d'EA
which can computed or approximated retrospectively if randomization probabilities are needed for post hoc causal analyses. Again, we can approximate Eez, [Y A, a, Iz] via Monte Carlo.
5

---

Moreover, one may wish to blend the approaches in Algorithms and by using simulated offline data to form a posterior distribution over the treatment model parameters, P (0) Doff). During online deployment, the agent may either continue updating this posterior online, or fix it and sample throughout learning (avoiding the computational cost of full online 01 posterior updates).
4.3 Ensemble- Based GAMBITTS Approaches
Algorithms aandnssnnnnaa Ianduree explicit posterior updates for reward model parameters (02). restricting the class of models that can be feasibly used. This restriction can be particularly limiting for two reasons. First, flexible models are often desirable to avoid strong parametric assumptions on EE [ Z, X]. Second, when Z is an imperfect proxy for Z", flexible models may help capture residual information and improve the approximation of E Y Z", X] through m² (Z.X). X).
To address these challenges, we propose an ensemble-based posterior approximation strategy, drawing on the ensemble sampling framework introduced in Lu and Van Roy [2017] [36]. This approach maintains an ensemble of models to approximate the posterior distribution over parameters, making
posterior sampling tractable even for complex or nonlinear models. We present ensemble-based vari- ants of Algorithms and (((ens-fGGAMBTT)
and ens maa--OGGMMITTS, respectively) in Appendix D In Section 6 we evaluate the performance of ens aus-POAAMBITTS using multilayer pereeptron (MLP)
model for E Y X.
5 Theoretical Guarantees
This section explores regret guarantees for the GAMBITTS algorithms introduced in Section 4 We revisit the generator-medine mediated bandit learning problem from Section now from a theoretical perspective. Throughout,
assume finite context space (i.e., |X| < oo) and focus the correctly specified setting where the working treatment projection matches the true underlying treatment embedding (i.e., 7 Z*).
Under these assumptions, treatment and reward generation at time & follows a two-stage stochastic process. First, given context I; and action de, the generator samples intermediate treatment variable 2 E Rdd from a distribution g0, Sa.,a, (with density fi (z; xa, ag, 01)) supported on the ball (B( = f E R4 ||=||a < B}. Second, the agent receives noisy reward yr =m²(2, Iı; 82) me- where f/2 -s-suggaussinn noise. Here, A = {01, 022 are unknown parameters. We also assume that the mean reward distribution ma x; 02). under Z gdi is o -ubbaaussia for all (x, a) E X x A.
The goal of the agent to minimize the time-7' cumulative Bayesian regret, defined as
BRT = E
Σ
[mz(Z,X;;0)) X,, A7 E Ex,,A Ex..;
Ar, (Z, X A,, 01 A1)
where the inner expectations over Z and the outer expectations over X, 0,, 02, and agent actions A. The benchmark at denotes the Bayes-optimal action in context I2e defined by Ai := arg max 2 (m²(Z,21;; 02) 0). aEA
As baseline, we consider Thompson sampling agent that models the problem as a conventional multi-armed contextual bandit over X' x A, ignoring the generator-mediation structure. We refer to this as a "standard" Thompson sampling approach, as it captures natural modeling choice that treats rewards as direct consequence of actions. This serves point of comparison in both our theoretical and empirical results. As shown in Appendix [ its Bayesian regret satisfies
BRatandard 5 (01 + 02) VCKT)
While the regret bound for the standard Thompson sampling agent is tight (i.e., matching known lower bounds), it becomes unfavorable when either the context or action space is large. This motivates GAMBITTS approach, which can exploit shared structure the treatment space to support generalization across arms. We begin by analyzing PoGAMBITTS, as this approach aligns more closely with our motivating setting, then turn to the foGAMBITTS variant. Appendix presents the proofs of all results discussed below, along with additional theoretical results.

---

5.1 Regret Analysis for PoGAMBITTS
We begin by analyzing the regret of POGAMBITTS. This algorithm uses empirical conditional distribu- tions, with densities foff (2; c, a), that approximate the true treatment distributions cd, and define the estimated mean reward
m2 (2 02) = m2(2, 2;02) (2; a)dz.
PoGAMBITTS can be viewed as a Thompson sampling algorithm operating on the estimated mean reward function M2. More generally, we can consider other traditional bandit algorithms operating in the "partially online stochastic treatment" setting, where actions are selected based on an estimated reward model derived from empirical treatment distribution. Although actions chosen with respect to m2g rewards are generated according to the true mean m2, introducing a form of model misspecification. Nevertheless, we show that if the empirical treatment model is sufficiently close to the true distribution, then running no-regret contextual bandit algorithm on m2 still yields no-regret guarantees for the original problem.
In the following results, we let e 0 and consider empirical treatment models fo satisfying
KL ( a) (,2,,,01)) < for all (I, a) EX A.
c As shown in Appendix B under regularity conditions on Sx,:, for will satisfy 2 given poly (d,=-1) draws from simulator for each (ær a) E x
Theorem 1. Let foll be empirical model for the treatment distribution satisfying 2 Let Alg be
a contextual bandit algorithm that selects actions based on the estimated reward function fiz.. Then, running Alg in the partially online stochastic treatment setting gives T Tstep Bayesian regret BR with
BR s 0O (BRT +TVET)
Als
where BRG is the T-step Bayesian regret of Alg with respect to m2
Theorem I bounds the agent's 's true regret in terms of its regret under the estimated reward model induced by its misspecified treatment distribution. For POGAMBITTS with a linear reward model, we get the following result.
Corollary 1. Let foll satisfy 22. If m2 (z, zz; 0) is linear in E R then the Bayesian regret of POGAMBITTS (Algorithm
2 satisfies BR 50 (2dvT +
When ε I (a2d/T)², the regret simplifies to ① (2ddT)) This is sharper than the standard
Thompson sampling bound whenever (1+ VCK.
We now consider PoGAMBITTS with nonlinear reward model. In Theorem 2 the complexity measures dimr(FF, 7-2) and M(F 21 11x)) denote the eluder dimension and the covering number of F, respectively. Appendix provides formal definitions of these measures.
Theorem 2. Let goff satisfy (2). For any function class F with ma F, the Bayesian regret of PoGAMBITTS (Algorithm[2 2 satisfies
BRE
< <O ( --,, | + TVTT).
5.2 Regret Analysis for fOGAMBITTS
We now present regret guarantees for foGAMBITTS (Algorithm I in the linear reward setting, which applies when the agent does not have simulator access to the distributions (Ea.a (x.))xxAA satisfies
Theorem 3. If m2 (z, 24 6) is linear
IRd, the Bayesian regret of fOGAMBITTS (Algorithm)) i
BR
BR.oolin
S zō <0(0,CCT + o2dVT). +xxCTTT
7

---

The bound nearly matches the minimax lower bound of [[max VCKT o2 02 ddT)), presented in Appendix E Comparing Theorem Sannoeemaann and Equation [ shows foGAMBITTS achieves a sharper regret bound than standard Thompson sampling whenever d < VCK. When the working treatment representation is low-dimensional and well-specifted, the benefits from generalizing across arms outweigh the cost of estimating the treatment distribution. Moreover, the advantage grows when 02 >> 01, since generalizing across arms helps more when reward noise dominates. In addition, when an 1/VCK, the regret of foCAMBITTS becomes comparable to that of PoGAMBITTS. In such cases, where the generator sufficiently concentrated, simulator access offers little improvement.
Having established regret guarantees for the GAMBITTS algorithms,, we now turn to their empirical evaluation in Section 6
6 Simulation Results
To evaluate GAMBITTS-based algorithms, we designed simulation study modeled the 2023 Intern Health Study (IHS) which was aimed at supporting mental health among medical interns [40]. At each decision point, the agent (z) observes user context (location, recent step count), (ži) selects prompt from a finite list to submit to an LLM (Llama 3.1 8.0B), and (izi) delivers the generated text response the intervention [1]. The environment then generates a reward based select semantic dimensions of the generated text (optimism, severity, formality, clarity, and encouragement) The semantic dimensions were constructed by training me-dimensioonl variational autoencoders text generated to vary along each axis, yielding mappings from text to scores used the reward model (see Appendix G.3 for more details).
While no deployed JITAIs currently integrate L.LMs for real-time message generation (and thus no real-world dataset exists for this setting), calibrate simulation using empirical distributions from the 2023 IHS to model realistic conditional reward structures. Details on the compute resources, generative model, prompt design, and reward specification are provided in Appendix G with code and further documentation available on GitHub Additionally, Appendix includes further simulations exploring different first- and second-stage stage variance decompositions, varying the 2 dimensionality d, altering the level of simulation access available to POGAMBITTS, incorporating direct covariate influence reward, and evaluating performance under nonlinear data eetta-enerating mechanisms. All figures are based on 250 Monte Carlo runs per agent, with 95% confidence intervals shown.
6.1 Illustrative Example
We begin with a simple example where the outcome at time is given by Y, 32, 32.pplimism + Ety, with
zoptinism denoting the optimism score of the generated message and E, nD (0, As discussed above, parameters 3 and a are derived from IHS data. This setup isolates a single semantic dimension, where expect GAMBITTS to outperform standard Thompson sampling by leveraging shared reward structure. We compare GAMBITTS algorithms to both contextual and non-contextual standard Thompson sampling agents (StdTS: Contextual and StaTs, respectively); as shown in Figure 2 StåTS performs better here and is used as the baseline in subsequent simulations. As expected, GAMBITTS methods achieve lower regret than standard Thompson sampling, with PoGAMBITTS performing especially well in this setting.
Cumulative Regret
StdTS
StdTS:Contextual
foGAMBITTS
PoGAMBITTS
Time Period
Figure 2: Cumulative Regret Under Single-Dimension Reward Model
8

---

6.2 Embedding Misspecification
The assumption that the agent has correctly identified how G influences Y (i.e., that = Z*) quite strong. To assess the impact of misspecifying this mechanism, we revisit the illustrative setting from Section but evaluate performance when the agent uses linear model with an alternative we expect GAMBITTS performance to degrade as the working embedding departs from the true reward-relevan dimension (optimism).
embedding. As discussed in Section |
Cumulative Rogret
Carnulative Regret
Clarity
Encouragement
Formality
Optimism
Severity
40
-0
Time Period (1)
Time Period
(a) PoGAMBITTS
(b) foGAMBITTS
Figure 3: Cumulative Regret Under Treatment Embedding Misspecification
Figure confirms this pattern: GAMBITTS performs well when the working embedding is correlated with the true one, but deteriorates as the correlation declines (see Table Abble/G.3.4j) in Appendix G). In low-correlation settings, it can even underperform standard Thompson sampling, as policy learning fails when working treatment representations are misaligned with rewards.
6.3 Scaling Number of Arms in a More Complex Reward Structure
We next vary the number of treatment arms to Corollary D which predicts POGAMBITTS regret does not scale with K in linear settings. Furthermore, in this section and throughout Appendix F we turn to a more complex linear reward structure based on Zoptimism Zformality, and Z Z As we move to more complex eeta-generating environments, we introduce an ens- Ans-POAAMBTTSS agent with an MLP reward model.
Cumulativ Regret
Camolaatvv Ragnet
Cummulatvve Regret
Timee Period
Time Period (1)
Time Period
(a) K =
(b) K = 5
(c) K = 15
Cunulaaive Regret
Sumalative Regret
StdSS
ens-PoGAMBITTS foGAMBITTS
PoGAMBITTS
Time Period (1)
Time Period (1)
(d) K 30
(c) K 40
Figure 4: Cumulative Regret for Varying Across Sizes of Action Space
The simulations varying the size of the action set largely aligned with our expectations: the regret of the linear POGAMBITTS algorithm was stable across values K, and the regret of ens- POGAMBITSS increased slightly with K, but not substantially. This pattern is encouraging, as it suggests that

---

additional intervention options can be incorporated by increasing K without incurring large regret penalty, though computational cost scales linearly with K. foGAMBITTS performed noticeably worse in this setting, which is expected given that it must estimate the distribution of Z for each (x, a) pair. A more natural comparator for foGAMBITTS is the contextual bandit, and we examine this comparison in Appendix F
7 Discussion and Future Work
This paper introduces GAMBITTS for online learning in showing promise for synergistic relationship between classical bandit methods and modern advances in generative modeling. By formalizing settings in which actions (e.g., prompts) produce stochastic treatments (e.g., generated responses), this framework opens opportunities for personalization in domains such as marketing, mHealth, and education, where treatments cannot be easily enumerated and real-time generation can yield more deeply-tailored interventions.
bandit environments,
This work opens up a wide range of directions for future research. One immediate area concerns the structure and specification of the working treatment embedding. As shown in Section 6 GAMBITTS can still perform well under embedding misspecificatioo, and ficxible reward models can help mitigate this issue. Still, natural extension would be to learn the embedding online, (c.g., through online sufficient dimensionality reduction). Another direction involves moving beyond a static generator. While this manuscript assumes fixed model, fine-tuning the generator based observed outcomes could offer new opportunities for intervention design. Finally, this manuscript assumes that the agent always delivers the generated response the environment. While this may be appropriate in some applications. in others it may be essential to ensure that any content sent to users is safe. In such cases, intervention designers may wish incorporate safety checks content filters prior delivery. Incorporating these constraints into the generator-medieee mediated bandit framework may require new GAMBITTS variants and accompanying theoretical analysis.
10

---

Acknowledgments
The authors thank researchers at dßcenter for their comments in the early stages of this work; in particular: Daniel Almirall, Walter Dempsey, Mason Ferlic, Easton Huch, Wenchu Pan, Yao Song, and Shiyu Zhang. We also thank Jordy Berne, Liza Levina, and Abhiti Mishra for their comments on later drafts of the manuscript, as well as Elena Frank and Srijan Sen, director and principal investigator of the Intern Health Study, for their support and coordination. Lastly, Gabriel Durham and Ambuj Tewari acknowledge support from the NIH via grants DA039900 and P50DA054039, and 5R01 MH131617 and SU01 MH1 16056025, respectively.
References
[1] Jason Ansel, Edward Z. Yang. Horace He, Natalia Gimelshein, Animesh Jain, Michael Vozne- sensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, al. Pytorch Faster machine learning through dynamic Python bytecode transformation and graph compilation. In ASPLOS (2), pages 929-947, 2024.
[2] Mirza Mansoor Baig, Chris Hobson, Hamid Gholam Ehsan Ullah, and Shereen Afifi. Generative Al in improving personalized patient care plans: Opportunities and barriers towards its wider adoption. Applied Sciences, (4(23):10899 November 2024.
[3] Hamsa Bastani and Mohsen Bayati. Online decision making with high- digensinmns covariates. Operations Research, 68(1):276 294, January 2020.
[4] Audrey Boruvka, Daniel Almirall, Katie Witkiewitz, and Susan A. Murphy. Assessing time- varying causal effect moderation in mobile health. Journal of the American Statistical Associa- tion, 1 113((23): 11 July 2018
[5] Djallel Bouneffouf and Raphael Feraud. Survey: Multi-armed bandits meet large language models, 2025.
[6] Clément Chadebec, Louis J. Vincent, and Stéphanie Allassonnière. Pythae: Unifying generative autoencoders in Python a benchmarking use case, 2023.
[7] Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. InstructZero: Efficient instruction optimization for black-box large language models. In Proceedings the 41 International Conference on Machine Learning, volume 235, pages 6503-6518, PMLR, 21 21-27 Jul 2024.
[8] Zekai Chen, Po-Yu Chen, and Francois Buet-Golfouse. Online personalizing white-box LLMs generation with neural bandits. In Proceedings of the 5th ACM International Conference on Al in Finance, ICAIF '24 page 711-718. ACM, November 2024.
[9] Riccardo Della Vecchia and Dehabrota Basu. Stochastic online instrumental variable regression: Regrets for endogeneity and bandit feedback. Proceedings of the AAAI Conference on Artificial Intelligence, 16110-)-910-1160
April 2025.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, 2018.
[11] Naoki Egami, Christian J. Fong, Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart. How make causal inferences using texts. Science Advances, 8(42), October 2022.
[12] Khaled Eldowa, Nicolò Cesa- Dssa-Bianchi, Alberto Maria Metelli, and Marcello Restelli, Information capacity regret bounds for bandits with mediator feedback. J. Mach. Learn. Res., 25(1), January 2024.
[13] Lars Fritsche, Elena Frank, Laura Thomas, Amy Bohnert, and Srijan Sen. Implementing large language models for tailored mental health messaging in the COMPASS study, 2024.
[14] Susobhan Ghosh, Yongyi Guo, Pei-Yao Hung, Lara Coughlin, Erin Bonar, Inbal Nahum-Shani, Maureen Walton, and Susan Murphy, ReBandit: Random effects based online RI. algorithm for reducing cannabis use. IJCAI (U. S.), 2024:7278-7288 August 2024.

---

[15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The Llama 3 herd of models, 2024.
[16] David Haag. Devender Kumar, Sebastian Gruber, Mahdi Sareban, Gunnar Treff, Josef Niebauer, Christopher Bull, and Jan David Smeddinck. The last JITAI? The unreasonable effectiveness of large language models in issuing just-in-time adaptive interventions: Fostering physical activity in prospective cardiac rehabilitation setting, 2024.
[17] Jawad Hazime and Junaid Farooq. Evaluation of LLM powered agentic Al for solving multi-arm bandit problems. In 2025 IEEE International Conference on Omni-layer Intelligent Systems (COINS), page I 1-6. IEEE, August 2025.
[18] Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with natural language action space. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 1621-1630. Association for Computational Linguistics, 2016.
[19] Xinyu Hu, Min Qian, Bin Cheng, and Ying Kuen Cheung. Personalized policy learning using lon- gitudinal mobile health data. Journal of the American Statistical Association, 166533):410-420 August 2020.
[20] Easton Knight Huch, Jieru Shi, Madeline R Abbott, Jessica R Golbus, Alexander Moreno, and Walter H. Dempsey. RoME: A robust mixed-effects bandit algorithm for optimizing mobile health interventions. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024.
[21] Kosuke Imai and Kentaro Nakamura. Causal representation learning with generative artificial intelligence: Application texts treatments, 2024.
[22] Theodore J. Iwashyna and Edward H. Kennedy. Instrumental variable analyses. Exploiting natural randomness to understand causal mechanisms. Annals of the American Thoracie Society, 10(3):255-205 260, June 2013.
[23] Lorenzo J. James, Laura Genga, Barbara Montagne, Muriel A. Hagenaars, and Pieter M. E. Van Gorp. Toward scalable content generation for gamified mHealth interventions: The evaluation of LLM-generated goals on user engagement. In Entertainment Computing - ICEC 2024, pages 94-107, Cham, 2025. Springer Nature Switzerland.
[24] Jussi S. Jauhiainen and Agustín Garagorry Guerra. Generative Al and education: Dynamic personalization of pupils' school learning material with ChatGPT. Frontiers in Education, 9, November 2024.
[25] Nathan Kallus. Instrument-armen bandits. In Proceedings of Algorithmic Learning Theory, volume 83, pages 529-546, PMLR, April 2018.
[26] Jong-Min Kim. LIm-guided ensemble learning for contextual bandits with copula and gaussian process models. Mathematics, 13(15):2523, August 2025.
[27] Akshay Krishnamurthy, Keegan Harris, Dylan Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024.
[28] Nir Kshetri, Yogesh K. Dwivedi, Thomas H. Davenport, and Niki Panteli. Generative artifi- cial intelligence in marketing: Applications, opportunities, challenges, and research agenda. International Journal of Information Management, 75, April 2024.
(29) Branislav Kveton, Yi Liu, Johan Matteo Kruijssen, and Yisu Nie. Non-compliant bandits. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, page 138-1 1147, 2023.
[30] Finnian Lattimore, Tor Lattimore, and Mark D. Reid. Causal bandits: Learning good interven- tions via causal inference. In Advances in Neural Information Processing Systems 29 (NIPS 2016), page 1189-1197, 2016.
12

---

[31] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[32] Yingkai Li, Yining Wang, and Yuan Zhou. Nearly minimax-optimal regret for linearly parame- terized bandits. Conference on Learning Theory, pages 2173-2174, PMLR, 2019.
[33] Peng Liao, Kristjan Greenewald, Predrag Klasnja, and Susan Murphy, Personalized HeartSteps: A reinforcement learning algorithm for optimizing physical activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1):1-22, March 2020.
[34] Xiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. Use your INSTINCT: INSTruction optimization for LLMs using Neural bandits Coupled with Transformers. Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR.org. 2024.
[35] Ziyi Liu, Idan Attias, and Daniel M. Roy. Causal bandits: tivity, a reduction to linear bandits, and limitations around unknown marginals. In Proceedings of the 41st International Conference on Machine Learning. PMLR, 2024.
The Pareto optimal frontier of adap-
[36] Xiuyuan Lu and Benjamin Van Roy, Ensemble sampling. In Advances in Neural Information Processing Systems 30 (NIPS 2017), page 3260-3268, 2017.
[37] Yangyi Lu, Amirhossein Meisami, Ambuj Tewari, and Zhenyu Yan. Regret analysis of bandit problems with causal background knowledge. In Conference on Uncertainty in Artificial Intelligence, 2019.
[38] Aurghya Maiti, Vineet Nair, and Gaurav Sinha. A causal bandit approach to learning good atomic interventions in presence of unobserved confounders. In Proceedings of the Thirty- Eighth Conference Uncertainty in Artificial Intelligence, volume 180, pages 1328-1338. PMLR, August 2022.
[39] Inbal Nahum-Shani, Shawna N Smith, Bonnie J Spring, Linda M Collins, Katie Witkiewitz, Ambuj Tewari, and Susan A Murphy. Just-in-time adaptive interventions (JITAIs) in mobile health: Key components and design principles for ongoing health behavior support. Annals of Behavioral Medicine, 4466)444 December 2017.
[40] Timothy NeCamp. Srijan Sen, Elena Frank, Maureen A Walton, Edward L Ionides, Yu Fang, Ambuj Tewari, and Zhenke Wu. Assessing real-time moderation for developing adaptive mobile health interventions for medical interns: Micro-randomized trial. Journal of Medical Internet Research, 22(3): March 2020.
[41] Allen Nie, Yi Su, Bo Chang, Jonathan Lee, Ed Chi, Quoc Le, and Minmin Chen. EVOLvE: Evaluating and optimizing LLMs for in-context exploration. In Proceedings of the 42nd International Conference on Machine Learning, ICML'225, 2025.
[42] Ollama. Ollama (version 0.3.3). https:://olsse com 2024.
[43] Reid Pryzant, Dallas Card, Dan Jurafsky, Victor Veitch, and Dhanya Sridhar. Causal effects of linguistic properties. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2021.
[44] Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems 26 (NIPS 2013), 26, 2013.
[45] Rajat Sen, Karthikeyan Shanmugam, Murat Kocaoglu, Alex Dimakis, and Sanjay Shakkottai. Contextual Bandits with Latent Confounders: An NMF Approach. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54, pages 518-527. PMLR, 20-22 Apr 2017.
[46] Chengshuai Shi, Kun Yang. Zihan Chen, Jundong Li, Jing Yang, and Cong Shen. Efficient prompt optimization through the lens of best arm identification. In Advances in Neural Informa- tion Processing Systems 37 (NeurIPS 2024), 2024.
[47] Alcksandrs Slivkins. Introduction to multi-armed bandits, 2019.
13

---

[48] Oleg Stavitskii, Kyrylo Bulatsev, Philipp Petrenko, Dmitry Bezugly, Evgeny Gurzhiy, and Dmitry Evgrafov. System and method for creating personalized user environment. U.S. Patent 11,275,350 B2, 2022.
com httmm/pttent/11255500
[49] Andrew Stirn and Tony Jebara. Thompson sampling for noncompliant bandits, 2018.
[50] Brian Suffoletto. Deceptively simple yet profoundly impactful: Text messaging interventions to support health. Journal of Medical Internet Research, 26, August 2024.
[51] Ambuj Tewari and Susan A. Murphy. From Ads to Interventions: Contextual Bandits in Mobile Health, page 495-517. Springer International Publishing, 2017.
[52] Sabina Tomkins, Peng Liao, Predrag Klasnja, and Susan Murphy, IntelligemtPooling:: Practical Thompson sampling for mhealth. Machine Learning, (MN9))2685-2277, June 2021.
[53] Victor Veitch, Dhanya Sridhar, and David M. Blei. Adapting text embeddings for causal inference. Conference on Uncertainty in Artificial Intelligence, 2019.
[54] Jiangxing Wang, Jiachen Li, Xiao Han, Deheng Ye, and Zongqing Lu. Language model adaption for reinforcement learning with natural language action space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 1620-1634, Association for Computational Linguistics, 2024.
[55] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace's transformers:
State-of-th the-ar natural language processing. CoRR, 2019.
[56] Nuoya Xiong and Wei Chen. Combinatorial pure exploration of causal bandits. The 11th International Conference on Learning Representations, 2023.
[57] Akihiro Yabe, Daisuke Hatano, Hanna Sumita, Shinji Ito, Naonori Kakimura, Takuro Fukunaga, and Ken-ichi Kawarabayashi. Causal bandits with propagating inference. Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 5512-5520, PMLR, 10-15 Jul 2018
[58] Jingwen Zhang, Yifang Chen, and Amandeep Singh. Causal bandits: Online decision-making in endogenous settings. In NeurlPS 2022 Workshop on Causality for Real-world Impact, 2022.
[59] Lutong Zou, Ziping Xu, Daiqi Gao, and Susan Murphy, Causal directed acyelic graph-informed reward design. 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction describe the proposed meeerator-meditted bandit framework and the Thompson sampling-ben -basin algorithm (GAMBITTS) (these are presented in Sections [nd respectively). They also summarize the theoretical regret bounds and empirical performance improvements over standard baselines, aligning with the theoretical results in Section and the empirical results in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations related the reliance on a correctly specified working embedding and examine robustness to misspecification in Section 6.2 Additioanlly, as noted in Section []] the performance of PoGAMBITHI ITTS depends on the quality of the first-stage treatment approximation, and the theoretical results are derived under simplified conditions, including fixed context space and treatment model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
15

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We explain the assumptions for our main body results Section 5 We include proofs in Appendix E
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We include algorithmic details in Section + simulation specifications in Appendix[G] and we include the code necessary reproduce results in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
16

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: We provide code in the supplementary materials with instructions for repro- ducing the simulation results reported in Section 6 and Appendix F This code will be made publicly available on GitHub upon publication; the link is omitted here to preserve anonymity.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cca wwtpcccccccpccceeeeeeeeeeeeeeeee wetiiic/guieeee/eeeeeee / Caaeee//ooSSSbbisceeee ssiooioiccip for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the simulation setup and implementation details in Appendix G including data-generating mechanisms and hyperparameters. We also include code in the supplementary materials ensure results can be reproduced exactly.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
17

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: As discussed in Section 6
we report 95% confidence intervals for Monte
Carlo estimates of cumulative reward. These intervals are computed across 250 independent runs per agent and reflect uncertainty due to randomization in the environment and learning process.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We present this information in Appendix
G
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't it make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ottp
Answer: [Yes]
Justification: The research complies with the NeurIPS Code of Ethies. We have, the best of our ability, followed best practices in licensing, reproducibility. and responsible use of external models and datasets, and we address potential limitations and risks in Section 7
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
18

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes. We discuss broader social impacts Appendix H including potential benefits in areas like education and mHealth. We also note risks associated with deploying L..M-generated text in sensitive settings and highlight future directions for adapting the framework to mitigate those risks (see Section 7
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any pretrained models or datasets. However, discuss potential risks and implications of deploying algorithms like ours in sequential decision-making settings in Section
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
19

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We Llama 3.1 to generate text-based treatments in simulations and cite the official technical report. We used the model in accordance with its non-commercial research license, and all other external assets are properly credited and used within their stated terms.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces simulation framework and a pool of ALM-generated treatments. These are included in the supplementary materials along with code and documentation describing how they were generated and used in the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We did not conduct any experiments involving human subjects or crowdsoure- ing. However, some simulation parameters were based on prior published studies involving human participants, which we cite appropriately.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Some simulation parameters were derived from prior published studies with human subjects, which we cite. This paper does not involve new human subjects research and did not require IRB approval.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs are used to generate a pool of text-based treatments, which serve as the basis for our simulation experiments. While the LLM is not part of the learning algorithm itself, it plays a role motivating the problem setup (described Section] 3) and its use in the simulation experiments is described in Section and and Appendix G
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standdaddard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Further Related Work
Eldowa et al. [2024] study bandits with mediator feedback from information -etrmatiion-hhoretiic perspective, broadly considering settings where the effect of an action is funneled through a stochastic mediator Among works on bandits with mediator feedback, our approach is most closely related to those that draw connections to instrumental variables and noncompliance. Zhang et al. [2022] and Della Vecchia and Basu [2025] adopt an econometric perspective, developing OFUL-based algorithms for bandit problems with endogenous covariates. While both consider continuous treatments, they do not focus on high-dimensional treatment spaces or settings in which treatments can be generated offline Moreover, these works emphasize interpretable parameter estimation, particularly in the structural model of the reward, whereas our primary focus is on predictive modeling for policy learning. This distinction is important in generative settings, where treatments (e.g.. L.M--generatee text) are difficult to interpret directly, and causal inference used to support adaptive decision decision-making making rather than uncover underlying mechanisms.
12
without interacting with the environment [58 ?
Other related efforts include Stirn and Jebara [2018], which introduces Thompson sampling in the context of noncompliance, assuming a shared, discrete action and treatment space [49 Kveton et al. [2023] also study noncompliance and employ Thompson sampling-base -based approach, aiming to identify actions with the highest compliance-weighee mean reward, rather than modeling the full stochastic treatment- -eenretttnneerttionn process [29
The
bandit setup most closely resembles the noncompliance-inspieed instrument- armed bandit (IAB) framework of Kallus [2018], where each arm pull represents the choice of an instrument, and the mediated bandit environment of Zou et al. [2025] 22 [5]. One key departure in our setting is the presence of stochastic generator that produces the treatment, which leads to focus on questions of treatment representation, generator access, and nonlinear outcome modeling, areas not emphasized in the IAB other mediated bandit frameworks.
In contrast to Kallus' focus, our setting does not require the treatment space to match the action space A. While assuming shared action and treatment space is natural for modeling noncompliance, where the intended and realized actions are ideally aligned, our setting explicitly separates the two. Here, the distinction is fundamental rather than incidental, reflecting the design goal of producing personalized responses rather than enforcing direct action execution.
Like Zou et al. [2025]. we consider a general framework for learning in mediated bandit environments. However, motivated by the complex nature of generative outputs, we focus on flexible, potentially nonlinear models for both the mediator and the reward. In contrast, Zou et al. frame their approach more explicitly as a surrogate-reward method methoo@@emphasizinee emphasizing linear models and aiming for robustness in cases where the action may influence the reward directly, outside the mediation pathway
The works discussed above present algorithms that can be broadly understood instances of general noisy action framework, where the environment observes a stochastic transformation of the agent's chosen action | From this perspective, GAMBITTS represents a specific instantiation of noisy-action Thompson sampler. While GAMBITTS is motivated by the GAMBIT framework, with nigh-dimensional and continuous treatment spaces, taking h (as defined in Section (3) recovers algorithms suited to the econometric and psychometric mediated bandit environments discussed above. Further taking G A yields algorithms appropriate for noncompliance settings.
In addition to bandit methods with comparable structure, described above, another line of research has explored intersections between LLMs and bandits. Although these approaches address different statistical questions, they nonetheless relate to our setting through their shared focus on combining LLMs with bandit methods. Within this broader category, the most directly related line of work concerns using bandits for prompt selection, where prompts are treated as arms and evaluated based on task- specific performance. In contrast with our approach, much of this literature considers settings with deterministic rewards once the LLM output is observed, or with white-bor boxte-box&o access to the model
'Their analysis focuses discrete mediators and introduces We note that GAMBITTS can also be viewed in this way, as discussed in Section 5
ExP4-based algorithm.
'In our motivating application, the action affects the outcome only through the generated treatment, making full mediation a reasonable assumption.
'We use the term noisy action to emphasize that, from the environment's perspective, the observed treatment is noisy representation of the agent's 's intended action.
22

---

7 34 46
Another line of recent research examines LLMs themselves as the Deciisionmmaiigg
agents multi armed bandit [27 17 26 For a broader overview of work at the intersection of
LLMs and bandits, see Bouneffouf and Feraud [2025] 5
In the mHealth literature, researchers have recently begun exploring how LLMSs support interven- tion design, motivated by the the parallel rise of mobile interventions interventions and generative models. Haag al. [2024] compared the quality of JITAI intervention text generated by GPT-4 to that authored by both domain experts and non-experts. Their findings showed that LLM-generated content performed favorably compared with human- authored text [66. Additionally. James al. [2024] conducted randomized trial comparing LLM- and human-geneeee generated goals in a gamified mHealth application, evaluating their impact on participant engagement. The study found no signif- icant difference in engagement between the two groups, suggesting that LLMs may offer viable and scalable alternative for content generation in such settings (23). However, as the goals were pre-written and not personalized in real time; the study did not evaluate the use of LLMSs for on-the-fly message generation or adaptive personalization. Lastly, the HIS-COMPASS studies used L.LSs to generate JITAI intervention text in recent deployments. However, as in James et al. [2024], these messages were pre-written rather than generated on-the-fly [13]
B Practical Variations
B.1 Restricting Context for the Generator
As discussed in Section 3 in many practical applications, the agent may restrict which features of X₁ are passed to the generator, selecting subset X C X2 to include in the query. While some components of X; may be useful for outcome modeling. they may be appropriate for inclusion in the prompt sent to the generator. For example, variable like weight could improve reward prediction but would likely be excluded from the generator input for ethical design reasons. The corresponding causal structure is shown in Figure 5
X
XG
A
G
Y
Z
Figure 5: Generator-Mediated Bandit Causal Structure (Restricted Context in Query)
Notes: Dotted lines represent deterministic relationships. (ii) Arrows from
are omitted represent data geeer-enneenngggg process rather than
decision logic.
(iii) A: Action/Query; X: Full Context; X Context Sent to Generator; G: Generated Response;
Z" True Response Embedding: Y; Reward.
The methods and analyses in this manuscript change only superficially when X + Xx- In this case, the treatment model introduced in Section 4 fi (æ; X,, 01), is replaced by fi Ar, 01). Additionally, all algorithms that invoke f1 inherit this modification.
B.2 Optional Prompting Extension
As discussed in Section mHealth interventions frequently begin with a decision of whether to deliver any message all. GAMBITTS can be naturally extended incorporate this structure. this variant, primary agent first decides whether to act (e.g., to send message). A GAMBITTS- agent then operates only at the time points where the primary agent chooses to act. Algorithm outlines this extension.
23

---

Algorithm 3 GAMBITTS with Optional Prompting
Inputs: Full history D( GAMBITTS history D prompting policy 1() GAMBITTS policy p (2),, context 2q.
1: Sample R2 ~ (u(1) (D.", 2.) according to Agent
2: If R₂ then
3: Select query A, ~ vl2) (D.²) .x according to Agent 2.
4: Update ( D f(x,, Ar, V.)).
5: Update policy n1(2)
6: Update D(4) + (( U R.)
7: Update policy n
C Embedding Misspecification
As discussed Section 4 GAMBITTS performance hinges on the construction working embedding
Z = h(G),, with the agent modeling Y Z, X in place of Y G, X. Section 6.2 supports this notion, showing GAMBBITS is robust to mild misspecification but suffers under severe misspecification. With this phenomenon in mind, this appendix discusses two approaches aimed at improving robustness to embedding misspecification.
One option for promoting robustness is to construct a hedged reward model. In particular, given any initial embedding construct Z and model m2, an analyst can consider the modified form:
m2 (Ar, Zr, Xr:B, 02) )
Σ 8a LA + +22 Xx; 02).
When the goal is online learning the optimal action A, this structure offers a hedge against misspecification of Z and m2. In particular, if m2 fails to capture signal, its estimated contribution to m2 will shrink towards zero. In that case, the m2 model asymptotically behaves like a standard Thompson sampling agent that learns reward distribution over actions directly. Conversely, if Z and m2 are well-speciffed, then the action A, provides no additional information beyond what is already captured through m² (Z, X,; 0z). In this case, the estimated B terms will shrink to and ma will converge to m2. However, the agent must first learn that the embedding carries signal, potentially delaying efficient generalization early in learning.
Figure @ revisits the analysis presented in Figure of Section 6.2 now including hedged models. All hedged PoGAMBITTS agents perform similarly, as do all hedged f foGAMBITTS agents. While every hedged GAMBITTS variant outperforms standard Thompson sampling in this setting. the well- specified models no longer exhibit the performance advantage they had in the original (unhedged) analysis. This supports the hypothesis that hedging reduces downside risk but limits gains under well-specified embeddings.
Cummbeivv Rogeet
Curmlasive Ragret
SwTSS
Clarity
Encouragement
Formality
Optimism
Severity
(1)
(a) POGAMBITTS
(b) foGAMBITTS
Figure 6: Cumulative Regret Under Embedding Misspecification Original vs Hedged Approaches
Note: Dotted lines represent cumulative regret under original (unhedged) approaches.
24

---

A second option is to maintain set of candidate reward models, each potentially using different treat- ment embeddings. Periodically, the agent can evaluate these models based on predictive performance and switch to the one that best predicts observed outcomes. This allows the algorithm to recover from poor initial choices of Z (and mg more broadly) without requiring strong prior knowledge about which embedding will perform best.
Improving robustness to embedding misspecification also points to several directions for future work. For example, extending the candidate model approach by developing a corralling bandit, where master algorithm selects among reward models, could allow the agent adaptively shift across models and tailor the choice of embedding to the individual. Similarly, as discussed in Section 7 learning individualized treatment embeddings through methods like online sufficient dimensionality reduction would mitigate misspecification risk.
D Ensemble-Based Algorithms
As Lu and Van Roy [2017], given a second-stage reward model parametrized by 02, ensemble-based GAMBITTS approaches require initially drawing Mens sets parameters 0 Gahana from pre-determined initialization distribution. Algorithms/44an and tht then demonstrate how select action a, for = I.. As in Algorithms Monte Carlo.
T.
and an analyst may approximate the integrals below via
Steps and Algorithms 4 and (respectively) require updating model parameters as described in Lu and Van Roy [2017]. These updates involve perturbations of observed reward to promote exploration [36].
Algorithm 4 Ensemble-Based Fully Online GAMBITTS (ens-foGAMBITTS)
Inputs: Data Dr, prior TI, models fI, m2, current context It, reward model parameters
Mens
Om
m=1 1: Sample 81, Pi (8) D,)
2: Sample j ~ 1 ([1,.... Mens)).
3: For each E A, calculate
E D M2.
IY a, [a, = m2 0.21:02..) f1(2; a,, (2.
4: Set A, = arg max Eg aEA
By. [Y a, 21).
5: Given Te, v/),, update 0 2,,411...... 2.141.
Algorithm 5 Ensemble-Based Partially Online GAMBITTS (ens (ons-PoGAMBITTS)
Inputs: Data D., model m22 current context Ty, pretrained model fo reward model parameters
Mens
gi
m=1
1: Sample j ~ 11
2: For each E A, calculate
Egg [Y a, xx] = m2
3: Set A, == arg max By a
[Y a, vt].
4: Given (zt, v/), update 03 02, .
25

---

E Further Theoretical Discussion
In this appendix, we provide proofs for the results discussed in Section along with additional theoretical developments related to Thompson sampling in the GAMBITTS framework. The appendix organized as follows:
Appendix Proof of the regret upper bound presented in Section 5 for standard Thompson sampling in GAMBIT environment.
Appendix
1.2 Proofs of the regret upper bounds presented in Section for PoGAMBITTS-
based algorithms, as well as a bound on the number of samples needed for the offline (empirical) treatment distribution to approximate the true distribution.
Appendix E.3
Proof of the regret upper bound presented in Section 5 for foGAMBITTS.
Appendix
E.4
Statement and proof of regret lower bound for foGAMBITTS.
E.1 Upper Regret Bound for Standard Thompson Sampling
In this section, we present a Bayesian regret bound of a standard Thompson sampling algorithm that models the problem conventional multi-armed contextual bandit over X' A, ignoring the generator-mediation structure. Our analysis adapts the proof from Chapter 36.1 of Lattimore and Szepesvári [2020] 31.
Claim 1. The Bayesian regret of the standard Thompson sampling agent that models the problem as a aumti--rmmd contextual bandit over X x A bounded by
BRetandard O((ai + (2) log T + BCK)
where B is uniform bound the mean reward such that [m2 x; 0) = B.
Proof. Recall that the reward received when the agent selects action E A under context X' is given by m2 (2, x; 02) + where is o2-subgaussian and Z E I is drawn from the distribution e9 6 As discussed Section 5 we assume that my (Z, x; 02) is 01 su-subgausiinn. follows that the Sa..a
overall reward for a given context-action pair (z, a) is (01 + Ø2) -uubgausss....
We use m(x, a) to denote the mean reward for the context-action pair (z, i...
m(x a) = m2(2, x; (2; 01)dz.
Since we assume that the mean reward is bounded by [mz(2, z; 02) S B, it follows that [m a) S B.
Now, we adapt the Bayesian regret analysis for a standard Thompson sampling algorithm for multi- armed bandit settings the contextual multi-armed bandit setting where the context is assumed to be drawn i.idd every round.
Well, for each a E A. x E X and ( 1,...,77 T. define
2 log(1/8) IV Nz.a -
Ut(z, a) = clip clipp-BBB B,B) mr. 1) (0, (0))
where Nz. 1) - 1 1 =x,= at 1) denotes the empirical estimate of the reward of action under context based on observations up to time f 1. If Nz,a
the count of the pair (æ, a) up to time t and
= 0, we set mz.a 1) 0.
Let E be the event that, for all t = 1...., T and (x, E X x A, the empirical estimate satisfies
2log(1/8)
m a) m(a a) ml
(0) (0)+ 12))
NV 1)"
'Recall ca has density 11
26

---

Under this setup, the standard Hoeffding concentration bound gives P (E*) & 2CKT6 where C== X and [Al.
Let Fi = A V........ A,, Y,, X be the a-algebra generated by the interaction se- quence by the end of time (and the following context at time i + 1). Note that Uz a) is -m---uuaurralle. Using the fact that, under Thompson sampling, the conditional distributions of A arg max, m(X,, a) and A, given F are identical, obtain
X1,
T
BRy = E
(m(X,, 47) A())..A))
E [m A7) m(x,, At) Fr-1l
E
(m(X,, (X,,,, + E(V,(X,,))) m(X,, An))
On the event E the terms inside the expectation are bounded by 2BT. On the event E, the first sum negative and the second sum is bounded by
T
[ Σ (U, Ar) - (X(:)At))..
T
=1(F) ΣVΣΣ (Va m(x,a))
(=1 ZEX'AE
8log(1/8)
S ( + ΣEΣ
(X==zAAFa))
T Nz.a
REN aE
(T)
8log(1/8) ds
(01 + +22
ΣΣ
JEA aEA
(σ1 + (2)
ΣΣ
32N a, (T) log(1/0) (a) + 02) 32CKT 32CCTT log(
The result follows from choosing 8 , and applying the bound [Ea] < 2CKT6.
E.2
Upper Regret Bounds for POGAMBITTS
The contextual bandit environment of interest with the two-stage reward generating process fully specified by
E(1,, 02) (Px. ( ( {m2(2, x; Me)
where Px is the context distribution, <d is the distribution of the representation Z E Rd given the context-action pair (x, a). O2 is the parameter that governs the mean reward function m2 xz 02), and P. is the oz-subgaussian noise distribution.
In the analysis of PoGAMBITTS, we consider the Bayesian regret with 01 fixed to 07 and 0g is integrated over the distribution π2. We denote the environment with 0 fixed the true parameter 0* by
E(O₂) = E(0, 02) = Px,
00 Szia
rEx. {mz(z, x; 02) 2ex)
Theorem assumes access to Alg with the Bayesian regret bound of BR Alg under the approximate environment
ε(0²) = Pxx, [coff)
Jaex,0EA {mz(z, x; 02) (e)
where ei in the definition of E(22) replaced with its offline empirical estimate gaff (with density gofs (-|z, a)), and the same mean reward function m2(z, x;; 02) parameterized by 02 is used. To be
27

---

more precise, the Bayesian regret of the algorithm Alg under the approximate environment E(02) is defined as
T
[m2 X,) X,] Bey [mz(2, Xx] ]]
1R Alg EBgana
BE(02)
E Egors x1.4
where A arg max I8 cory
(2 Xx)
Xe, 02]: i.e. the optimal action in context X, under the
environment E(O2). Theorem shows that running the algorithm Alg under the environment (62) achieves a similar regret bound as long as the approximation & ( by E(62) is sufficiently close.
Theorem (Restated). Let pofs be an empirical model for the treatment distribution satisfying 2 Let Alg be contextual bandit algorithm that selects actions based on the estimated reward function m2g. Then, running Alg in the partially online stochastic treatment setting gives T T-step Bayesian
regret B Alg with
BR
BRA s 0o BRT Alg
+TVET)
Alg
where BRT
is the Tssepp Bayesian regret of Alg with respect to m22
We provide proof of Theorem Saareemmebblww below, which relies Pinsker's inequality.
Lemma (Pinsker's inequality). For any pair of measures P and Q on the same probability space (R,F), have
8(P.Q) = sup P(A) - Q(A) S V DKL (PIQ),
AEF
where DKL. denotes KL divergence in nats.
We now turn to the proof of Theorem 1
Proof of Theorem 7 Let pê, be the distribution of the reward given the context-action pair (I; a) v.a under the environment (62); i.e., the distribution of the random variable my [2 x;; 02) + + where o Z Ex.a and Pq. Similarly, let 2. be the distribution of the reward given the context-action pair (I, a) under the environment {(2); i.e., the distribution of the random variable m2 [2, o; 02) +
where Z coff and ~ Pay 8x,.0
(i.e., KL (1566) s 5), Recalling the assumption that fo satisfies 2 observe
KT.
p00 poz
= KT. (A(G(Z.)))1Ico((22..))
S KL (I(2,))IZ(Z.n)
= KL (C(Z))I(())
<
where 9(2,1)) m2(2, 02) 1) and L(W) denotes the law/distribution of random variable W. The first inequality is due to the data processing inequality.
The interaction between the algorithm Alg and the environment E(02) determines the distribution
of the sequence (X1, AI, Zi, YI...., XT, AT, ZT, YT). With slight abuse of notation, let P(02) be the probability measure the sequence generated by Alg under the environment E(02) and let
P(02) be the probability measure on the sequence under the environment E(02). Then, standard KL divergence decomposition results" for bandit environments gives
KL P (((2)) (((22))
ΣΣ ZEA' aEAA
Nz.a (T)K(
p0: 2,0
S ET,
EE.g Lemma 15.1 in Lattimore and Szepesvári [2020]
31
28

---

where (T) is the count of the pair (I, a) up to time T, as in Appendix E. Hence, the Bayesian regret of the algorithm Alg under the environment {(2) averaged over the prior distribution on 02 i
BRT
Es,
Er(O2)
E8 oi Ex114
(mz(2,Xx;;2))
o [m2 (m2(2,X(i02)) Ex
< Es,
E ((0a)
Σ
E [mz(2 Xe:
Ex2.a7
(P(02), P(02))
+ BT
T
< 1832
((x)))
Σ
B 5xei22
- -Bers, [m2
Xx: (x;
Sxe,A,
+ 2T 8 (XeiAA+ (5eeAA:552iA (Xiia)) + 2T ((8B)).((0)))
< Eag
E Bp(02)
Σ
(m2(2,Xi:2) ((; - -Bey! SX,.d,
Imm (Z, X1; Xx;
+ 27 B 1221 KT. (SxtAA:15Xx11) (5x..A)) + BTTBB KT. (P(B2)))((02))
S <BR Alg TB / 2(T + 1)e.
Here, we use the notations 8(X, Y) for the total variation between and Y, and At arg max E gors (mz Xx; 02) X,, 02), which is the optimal action in context X, under the en-
a vironment E(22). The first inequality follows from the fact that the absolute value of the expectant is bounded by B. The third inequality follows from Pinsker's s inequality (Lemma I The final inequality follows from the assumption that the algorithm Alg achieves Bayesian regret BR under the enviromment E(22,, averaged over 0g ~ T2. The result follows.
E.2.1 Linear Reward Function
Now, we use the reduction result in Theorem 1 obtain Bayesian regret bound for PoGAMBITTS under the environment E(2)). where the mean reward function m2(2, æ; 02) is linear in z. To apply the reduction, first establish Bayesian regret bound for the Thompson sampling algorithm under the environment & (02), assuming knowledge of fo.s.
This analysis relies on two standard results from the theory of linear contextual handits. The first is the concentration bound for the regression- begres estimate of the linear parameter:
Lemma 2 (Theorem 20.5 in 31) With probability at least - 8, we have, for all L t= 1,..., T, that
00
0.llv S 3
where B = 02/22log dlog(1 + T/d).
The second result technical lemma commonly used to bound the cumulative sum of upper confidence bounds in the analysis of optimism-based linear contextual bandit algorithms:
Lemma (Elliptical potential lemma). Let Vo Rdxdd be positive definite and a1,..., On E Rdd be
a sequence of vectors with ||a/]|2 < for all t 1,...,n. n, Furthermore, let V = Vo + D a.a.. Then,
n
det Vn det Vo
(T<(V) + nI.?
Σ (I^ (I (aoll?))) s 2lo
< 2d
ddeet
Here, use Eo, [W] to denote EdguTa [W].
29

---

Equipped with the two lemmas above, we now derive a Bayesian regret bound for Thompson sampling in the approximate environment {(2)).
Lemma Consider an environment (02) where the mean reward function m2 2; 02) (2, 02). The Thompson sampling algorithm run under this environment with a full knowledge of fofs, and prior T2 22, achieves Bayesian regret bound of
BRT S 0 (o2dv TlogT] (002d/TlooTT
Proof. We denote by V4.a Beoff [Z]. The Bayesian regret can be decomposed as:
E E (m ]
BRT
E
Egors (mz(2:xr;02)) Excint
E
(Wx.:: -xx.Ae 02)
T
E
Σ (Z - Ze, 02)
E
Σ (Mx.A 02)
+
(2. (Z. Σ [=1
d' 02)
where Aī = arg max, (Vxx..a 02) is the best action in context X, and Zz is a random variable sampled from cofs
Sx,,;; The second term above can be bounded by
I 5 )
Σ - 0O
B
Tlog B
which holds with probability at least 1 - 8 by the Azuma Hoemma-i inequality, since the se- quence {WVx.. Zi, (2,02) lT= forms a martingale difference sequence adapted the filtra- tion Fr = (02, X1, A1, Z1 .......... X,, A,, Zz Y,, X1+1). Choosing ō = $. we can bound The third term can be bounded similarly. To bound the first term, first define an upper confidence bound function U, 2 R
Et - 2i,00 02) <
U+(2) (2, 01)) #||=||==?
where V V+=1+E Z,Z, and 0, v, Σ Y.Z.. By the concentration inequality provided in
Lemma we are guaranteed that, for all t = l,... T. ||0 1102-0.llv I 3 with probability at least 1 - #
where we use 8 defined in the lemma with 8 4. Let E, be the event that ||0g and E n E. Then,
0 11-1ll1e-. S
E
Zr 7,,02)
=B
Σ
(7i Oz)
E
1 (B)
Σ 1 t=1
Ze, 02)
< 2 E
Σ (1) 02)
By the Thompson sampling algorithm, P
P implying
P p F.-1]= = P P(Z, = F. With this observation, the rest of the proof follows the
standard Bayesian regret analysis of linear contextual bandits (e.g., Chapter 36.3 in Lattimore and Szepesvári [2020]) B1I.
30

---

Since Uz(z) for any fixed & E 2 is Fi-1-measurable, E, [0((2))] = E,-1 (U((,) Therefore, follows that the second term the display above is bounded by
E (E))
Ze, 02)
= (E,) E [2, 02) - U.( + +U((Z)) -Z,, 02)
- 1(E,) E (Ux(Z;) - (2,, 02)
1 11 (F,) (Ze, (Z,,0--1 82) + S11 ||
s 2.3 Zel 1,llv
Combining the above with (E,) (Zx - Zz 02) S produces
(m,) (Z Z,,
28
- 28 TE
(1A Zo 12
$ 23 V 2dT log(1 + T/d)
where the second inequality follows from Schwwrtz and the last inequality from the elliptical potential lemma (Lemma 3 Combining the bounds yields
BR, < O (02dd/Toog")
as required.
Corollary Tin the main body provides Bayesian regret bound for PoGAMBITTS in the linear reward setting. The corollary follows directly by viewing the algorithm as an instance of Thompson sampling for a linear contextual bandit, and applying the reduction in Theorem along with the regret bound in Lemma 4
E.2.2 General Reward Function
In this section, we analyze the Bayesian regret of PoGAMBITTS for general reward functions. We consider function classes J4 that contains the mean reward function m2(2, x;; 02), and derive a regret bound that holds for any such class F. Our bound is expressed in terms of the eluder dimension of F, a complexity measure introduced by Russo and Van Roy [2013] (4) We begin by recalling the definition of the eluder dimension.
Definition 1 (e-dependence). A pair (2,x) E X X is e-dependent on pairs ((21,11) (n,),,))) C 2 X with respect to F any pair of functions m, m. F sat- isfying v VL (m( xx2) x,,,, S e also satisfies m(z,x) x) m(2,0)
Further, (z, x) is e-independent of (111,1)..... [n)] with respect to F if (2, x) is not e-dependent on ((11.11).
(Znz }.
The c-eluder dimension of function class F. dimmg c), is the length d of the longest sequence of pairs in x X such that, for some è' 2 e, every element is é' é'iindependent of its predecessors.
Definition (Eluder dimension 44
The Bayesian regret of an algorithm under the generator generater-mediated bandit framework measures the regret of the action A, taken in context X, against the best action A; As a first step toward bounding the Bayesian regret, we bound the regret of the embedding Z, generated by the action against the counterfactual embedding Zü that would have been generated by the best action. The analysis adapts the proof of Proposition in Russo and Van Roy [2013] (144). ), and expresses the bound in terms of wc. (Ze, Xx)) which measures the maximum variation across the reward functions the confidence set C, at (Ze, X,), where
wc(2,x) sup (m(2,x) - m' m2(2,)))
m.m
Where F. [W] := E. [W [.] for any random variable W. For F consisting of real-valued functions Z X R.
31

---

Lemma 5. Fix any sequence of confidence sets {Co)e=1 where C. S F is measurable with respect to
-:11
OX1, A,, Z1, ,,,,.,,,, A Z, 1-,,, X,). Then, for any L..... T. we have
E
i 02) (2( Xx;
E
(we, (2,, + 2B ((ma(s:0))
Proof.
For any E and x E X, define the upper and lower bounds
U1(2,z) == sup (m(2, x) m E C.). x)) inf (m(2,x) m E Ct).
Given any 82, if m2 + 02) E C₁, then
E [mi Xx; 02) m2 Zz, Xx; 02)]
s E Uo X1) - L, (Z, X1) + 2B 1 (ma(s:0))
s E ncc (Z X,) + 2B 1 (ma())1)) (U,(2) X₂) U, U,( X,))
By the definition of Thompson sampling, we have IP| [A = 9--1 P [A] = 1 implying
PIZ- G--1] [z -.
Since U2 is -:- measuraaa,,, E [U ,,) - (2, X,) 9 = 0. It follows that
[xz my (Z,, Xx: X2:
S E
Σ
(we, (Z,, X2) 2B (m22()))))7))
concluding the proof.
Russo and Van Roy [2013] (44)) bounds the sum of the errors we, Xc), where the confidence set C, centered at the least squares estimate mi where
m arg min
mZ( X,) YY)²
MEF
The bound, presented in Lemma of max
6 is in terms of the -f-eeuder dimension, where
inf {llm - m2lloo m1, m2 E F, m1 + m2}
Lemma 6 (Lemma in (4). (Be)t= nondecreasing sequence and C. {m E F
I|
m 12,E. S /.,,, then the following holds almost surely:
Σ wc. (Ze, X1) $ T + B. B-innn (dimg(F
af), (F,)) + 4v dimg of)
Russo and Van Roy [2013] ( 44
also shows that, the confidence width 3r is chosen as
a, (N, l1se) (N1 + 2at 8B
N
[12 ))
(1(a) = 80 log
80 log
where N(F, a. || |1,o) denotes the a-covering number of F with respect to || Il then the associated confidence set (centered the least squares estimate) contains the true mean reward function m2(-. ;2) with high probability. Lemma formalizes this claim.
Lemma (Proposition in [44)). For all ā and a >
t-1
C. = m E F: Σ
(m(Z,, - Y (a)
s=1
for all t = 1,..., T, then
P m2(:):02)
E
C, 02
2 1-26.
32

---

We next establish a Bayesian regret bound for POGAMBITTS in the general reward function setting. As first step, analyze the algorithm under the approximate environment (02). We assume that the given function class F is sufficiently rich to satisty of =T-
Lemma 8. Consider an environment ((2)) with mean reward function (2, 2; 02) E F for some function class F. The Thompson sampling algorithm run under this environment with a full knowledge of 1s and prior T2 on og achieves a Bayesian regret bound of
BRT S O (B dimæ (F.T + (F, (N
where the Bayesian regret is defined over the distribution T22 0z-
Proof. Well, the Bayesian regret for such a Thompson sampling agent can be decomposed as
BRT
E
E Boof [mz(Z,X(;0a) - E BEg!! [mz X,; (,; 82)1 Ax
E
(mzl Zi, 02: - (z, Xe: 82))
+ E
E coff Nein)
[mz(Z, Xx; 02)] m2;Z;xX1;88b)
T
E
Roots Neide
[m2(Z, (x; - m2 Xc:
E
- m2( (x:
+0 +O(BTTlogT
where the final inequality follows from the Azuma- Azefa-Honfiding inequality on the martingale difference sequences
and
{ {Eewr sx [m2 12, 02) - m2 (2,, Xci 02) T t=11' both of which are adapted to Gt (X1, A1, Z1, Y,, .......... Xxs Ar, Ze X(41). (2,
As before, we consider A := arg max E
Beff *xt.
X,, 0z] and we consider Z; sampled
from coff A.. Considering a fixed sequence of confidence sets Cr, as defined in Lemma 6 and applying Lemma 5 gives the following bound on the first term:
E
(mz (7, 02) m2 Xx; 62))
(we, (Z X,) + 2B 1 (ma((--2244.7))
0 B dimg (F, dimg (F, aF) 37 (af) T)
where the second inequality follows by Lemma and Lemma 7 with 0 = J. The desired bound
follows by plugging a of into the definition of (a) (Equation 3).
Finally, Theorem 2 directly follows by the reduction provided in Theorem 1 and the lemma above. Theorem 2 (Restated). Let raff satisfy (2). For any function class J. with i2 E F. the Bayesian regret of POGAMBITTS (Algorithm) 2) satisfee
BR s zō (o2Vdimg(F T-2) log log (en.hiimm(FFT-3))))227777.7.3 1b. + +TVET).
33

---

E.2.3 Bounding the Error in Offline Treatment Models
PoGAMBITTS requires access to an estimate f (:: a, .)) that E-close to the true treatment density fi(:a, 01) in KL divergence. Below, we consider the example where Sz.a is Gaussian for each (x, X' A, demonstrating that a sample-base based empirical estimate can meet this requirement. Lemma 9 (Multivariate Gaussian Distribution), Consider a multivariate normal distribution N (μ,, Σ) and let (Zi...., In be i.i.d. samples drawn from this distribution. Then, the estimated distribution (D, Σ) satisfies
KL (N (u, II (p. E) So
+ log(1/8)
Jamin (2:)
with probability least 1 - 8, where ği is the sample mean and E is the empirical covariance matrix.
Proof. The KL divergence has the following exact form:
KT. (N(u, (T,, 8.)
E (5-1))) + (7 5 11) d log det E. log det x
We now turn to bounding each term. standard concentration result for empirical covariance matrices, based on the matrix Bernstein inequality, gives:
d + log(1/0)
- Σ||
n
with probability at least 1 - 8, where || is the operator norm. Hence, the inverse E concentrates and it follows that
- 2
Amin (Σ)
d for n 2 (2 log(1/6) ) Amin (E)²
Hence, we can bound
$ 5-1ン) tr [ 5-1 -2 E)
tr
I d -5 E/
d + log (1/8)
<d+o
V
Aonin (2))
To bound the log-determinant term, observe that
log det
log det Σ = log det [ 1
1 - Σ- 121
E 1/2
( (5-E) -E- E E-1/21
$
d 11| Σ||
0 d
d + 1og(116))
Amin (I) V
Finally, we bound
(7- - 1))
T
E
( p1) < 17 11 1
< 0 + log( 106)
applying the vector Bernstein inequality to bound Combining all the bounds yields
|| ul12.
KL (N (n, Σ) IW (p, 8)) s 50
d
d + (18(
Xmin (2)
required.
34

---

E.3 Upper Regret Bound for FoGAMBITTS
In this section, we derive a Bayesian regret for foCAMBITTS in the environment E 82). The key step in the analysis is that the action selected by Thompson sampling has the same con- ditional distribution as the optimal action, given the agent's posterior. Formally, let F, (X1, A1, Z1, .i...., Xt, A,, 7,, Yt, X be the o-algebra generated by the interaction up to time L. Thompson sampling logic guarantees that the conditional distributions of Až and A, given Fi-1 are identical; i.e.,
P [[4
Fi-1] P [A Fi-1), almost surely.
We will use the following concentration inequality to control deviations martingale sequence in the proof of Theorem
Lemma 10 (Freedman's inequality). Let (Fu) be a filtration and let the sequence {Xx} of random variables satisfy E [X 0 and [X&] S R almost surely. Let S₃ = Ek=1 X; and Vi =
Σ E X Fs-1). Then, with probability at least 1 5, have, for all k. that
2R
S& S 22V log 2/0)
3 1og(2/0).
We now restate Theorem
3 and proceed with its proof.
Theorem 3 (Restated). If m2 (2) linear in z E Rd. the Bayesian regret of fOGAMBITTS
(Algorithm)) satisfies
BR/ I 2Ō =((VVCKT + o2dvT).
Proof of Theorem
We are interested in bounding the Bayesian regret defined
BRT = E
E
[m Xx; X,;02) - E E Ex..A.
[ma X1; 02)]
where A7 = arg max E
01
(2, Xx; 02) Xx: 0y] is the best action in context Xz. Under the
aEA
5x..a
linear reward model m2 (z, x; 02) = 02). we can write the Bayesian regret as
BRG
Σ (xx.,n; Ax.,A, 02)
where we define V/zaa
Eee [Z]. suppressing the superscript 0 for convenience.
Since the agent observes the embedding 2, and receives a noisy reward the form Y2 = (Ze, 02) +1k. it can estimate 02 by regressing Y, on Z.. Consider the following estimate for 0g:
0. - Y.Z.
where V; = 1 E.=1 7.7.J. Lemma guarantees ||0₂ 0ellv < 3 for all t. 1,..., T with probability at least 1 5, where 3, dependent on 6, is defined in the lemma. Let F, be the event that
||ey 110y-0,- 1 I 9, and let E == nT_ Eg, Setting 5 = 1/r in the definition of / gives IP [ 2 1 1 4. We can then bound the Bayesian regret under E and E separately, as follows.
BRg E
Σ (x..;; dx 02)
(E<) Σ
(Wx.i vxx.ns 02) + +R (E)
Wix.. Ux..A., O2)
T
2 E
1 (E) Σ -
2 E
1 (ma)) (Wx.n - (x.,d., 02) Σ
(5)
35

---

Before proceeding with the bound, we clarify the probability space governing the random variables X,, A,, Z, Y, We adopt the random table model (see Chapter 4.6 Lattimore and Szepesvári [2020]) and, conditional on 01. define the embedding (x, a) ~ ch independently for each = 1..... 7 and (I, a) X' x A B1. We define Ze (X, A,) to be the embedding observed by the agent. This random table model provides a convenient framework for the analysis that follows.
To proceed, we define the upper confidence bound function U2 : 2 ++ RR by
Uz(2) = (2, 0c-1) + 31
Then, under the event F,, it follows that, for all (ar, E x A. (Wr,a, U,(Zz(z, = (12 02) - (Z) -
(V',a az, 02) (Ze(a, a), O--1) - -2)
S (W'e 0z(2,)),02)
The inequality follows from using Cauchy-Schwarz to bound (Z 02 Ōz-1) S (Z,(2,a))l |002 - 0c-1llv_.. Furthermore, under the event Et, ||0g 0 S B. Sim- ilarly,
(z, - (W,a: B2) =
(2,,, Q)
(Ze(2r a), 0) a,,,,,)ll,,, - (W2,0 (2,
S (Z((z, a) 0), (r)))l
Since U.(-) and X, are measuuuurrre implies
(X,, 4i)) = Ft- = (U, = Fr-1).
and that
E [ A;)) = 1 (Z,,,, A))]
where, as before, E.-1 [+] denotes E Fc. Hence, recalling the definition 2, = Ze(Xes Ar).
E [LE)) (Vx,, vx 02)
-1, Er-1 [(Vx.,; 02) U,(ZZ(X, (1))))) A.)) (1xx 02)
1 ( (Wx.A: Zx, A7), 02) (Z+ vx..A. B2) + +231
Continuing the regret bound from and using B 1 (F.) (1xx,, V/x,, 02) S 2, observe
BRT 22+E E
B [L(E]] :x -xx
2 E
((E)) (UxmiA A7),
(a)
T
T
+ E
1 1(8,) (2: - Ux..A.9 02)
+ 23E
Σ C11zolly. A3)
(8)
(c)
To bound term (a), we express it as summation over (z, a):
1 (1x
Ax),,
ΣΣΣ REN tElt=l
((
1X Zz(x,,),02)) 02).
36

---

For any given context-action pair (z, a) E X x A, the sequence {2, defined by
(2 a) = 1 (E,) (X.=z.47=a) (Z( a) - 12 02)
a martingale difference sequence adapted to
G. (81,02, X1, A1, (Z1 (2, a)) Vi...., X,, A,, (2,(),))) V,, Xx+1, Ast1).
This follows because 1 1 and Ax== are --eassuurrrlee,, implying
E = (E,) (X.= (z a) Uz,a 9r-1), ,2)
(2 - 02) 9--1]
=0.
Furthermore, [Wa(2, a) 2 for all t= = 1,.... T, and
E [Wa(x,a)² 9 <
1 (x.= A;== Σ
[1Z( (x,a002)7 9
t=1
r
of
1 (x (x Σ
of (2,
where N7 a) ET= 1 =,)= as before. The first inequality follows from the assumption that mz Z, x: 02) 0 -subgaussian. Hence, Freedman's inequality (Lemma 10) coupled with union bound over (I, a) E X x A. guarantees that, with probability at least 1
Σ W,(z,,) & (( / (2N7(2,)) 20112N((00))
2CK 8
2CK 8
log
follows that
T
T Wi(z,a) a)
(EE))(x.A - -xX,, -
FEX aEA t=1
2CK
2CK
Σ REX
22NT(Z, a) log
+ log
8
3
8
2CK
4
(2CK 8
S son
2CKT log
CKCoo log
where the last inequality is by Cauchy-Schwarz and the fact that Σ. E Nr(xx a) = T. Similar analysis, with A, place Aj, yields the following bound on the term (b):
T
2CK
4 CK log
2CK
Σ LLEa)(Xx.. Ai), s on SC12CKTVV log
3
Finally, the last term (c) can be bounded using the elliptical potential lemma (Lemma
T
T
(3A A 7 [2)) 3
Σ
Z+
llve
1=1
T
53
T Σ
(IA
2z 1(==))
t=1
T
< 3
2d
1+
d
37

---

Combining the bounds above, we observe
2CK
(CK {
7
BRT 2
8CKTlog
CK log
+ +63
2dT 2dT log
1+ 1+7/ d
0o
CKT
(CA) à
o2d
Tlog 11 1477
[og
T ) [+
d
+ log
1 (8)
(CA)
CK log
The result follows.
E,4 Lower Regret Bound for foGAMBITTS
In this section, we establish minimax lower bound on the regret of foGAMBITTS.
Claim 2. The minimax regret of foGAMBITTS satisfies
Ry 2 max FoaxxoovCCO (oIVCKT, 02 log K k)).
Proof. To establish the lower bound, we construct instances which the agent cannot efficiently
resolve uncertainty, leading to provably high regret. We first show Rr 2 Ω 01 (oLVCTT)
Fix K, C' and d, and let A ..,..., K). X' (1....,C)) Let 02 (a), o,..., parameter for the linear reward model. Furthermore, choose the reward noise d2 = 0.
Rd be the
Let 0, E {AU { parameterize the distributions 22 such that, for each (x, a) E X x A,
Bernoulli on {ei, 0} with probabilities ( 2)
a # 01.
c0 Saya
A +
if 01,=²
for some A E [o, 1l Here, we use e to denote the first basis vector in Rd
Under this environment, the mean reward of each context-action pair is a o--subgaussian. Also, in context v, the mean reward of every action is + except for one optimal action 01.z. which gives mean reward! 1
2 (a) + A).
Because rewards in different contexts are generated from independent distributions, observations collected in one context carry no statistical information about any other context. Hence, from the learner's point of view, the problem decomposes into C' independent instances of standard K -armed bandits, each with horizon (1), (C), where N2(xx) denotes the number of rounds in which context appears.
For a context x, the regret lower bound over (x) rounds is cσ / K (x) for some constant e (see Chapter 15.2 in Lattimore and Szepesvári [2020] 31. Summing over and choosing Nr((x) C ensure the environment chooses contexts evenly, we obtain
T
Rr 2 Ω Σ CVNN(()) - -N -0 -
We now show that Rr 2 Ω (a2y dl log K). Consider an environment where ed, is deterministic and known the learner. This reduces standard contextual linear bandit environment with dimension d. Applying a known lower bound for this setting [32], we obtain
Rr 2 Ω (22 (02V/T d' log
Note that {Au ( here denotes the C-dimensional space with components elements of (AU (0)).
rather than the complement of {Au (0)).
38

---

Combining the two lower bounds yields
Rr 2 Ω max Jo (ax
required.
F Further Simulations
This appendix presents additional simulations designed probe specific factors influencing GAM- BITTS performance. Unless otherwise specified, all data-generating environments assume linear outcome model with optimism, formality, and severity predictors.
Variance Decomposition
Section and Appendix E discuss GAMBITTS performance terms of the relative variance treatment and outcome processes. In setup, the treatment is generated by LLM and fixed in advance, but we can control the residual variance the outcome- -enenne-eee model. This allows manipulate the ratio of treatment variance to outcome noise directly. Throughout the main simulations, we held this ratio fixed by setting σ1 = 02 12 In Figure 7 we increase 02 to explore how performance changes under high outcome noise. We scale d2 gradually, up the level observed when fitting the outcome- gennertvee model on held-out data from the 2023 IHS study.
Cumulatvv Regrst
Conrutative fl-grrt
Cummlaaive Regret
Time Period
Time (oriod
Time Periad
(a) d2=0.71
(b) 02=6.68
(c) 02=12.64
Cumulative Regret
Cumslaaiv Regret
StdTS
cns-PoGAMBITTS foGAMBITTS
PoGAMBITTS
Time Period ()
Time Pariod (1)
(d) 02=18.61
(e) 242.4.7
Figure Cumulative Regret Across Outcome Variances
Corollary 1 and Theorem 3 suggest that the performance advantage of linear GAMBITTS-Den algorithms over standard Thompson sampling increases with the outcome noise level a Figure shows that linear GAMBITTS baseed methods achieve lower regret across values of σ2; however, as 02 increases, all algorithms exhibit performance deterioration and begin to converge. One possible explanation that higher outcome noise reduces the overall signal-to-noiss ratio, making more difficult for any bandit agent to quickly learn a good policy. Future work may examine this phenomenon over longer time horizons better understand the asymptotic behavior of the competing methods.
We computed di from the generated text interventions used for outcome simulation; i.e., from the response_ct db described in Appendix G
39

---

F.2 Nonlinear Data-Generative Reward Model
Section 6.2 examined GAMBITTS performance under style embedding misspecification. Figure explores related question, focusing instead on misspecification the structure of the outcome model. In these experiments, all GAMBITTS agents used the correct style embeddings, but the true outcome-generating process E [Y X] followed an MLP structure. To assess how GAMBITTS agents respond this nonlinear specification, revisit the variance-scaling analysis from Figure 7 now under an MLP outcome model.
Cunulative Regres
Cimulativ Regest
Cunulative Kegget
(a) 02=0.71
(b) 02=6.66
02=12.64
Cumslative Rlgret
Cumulative Rtggess
StdTS
ens-PoGAMBITTS ToGAMBITTS
PoGAMBITTS
(d) 02=18.61
0==24.57
Figure 8: Cumulative Regret Across Outcome Variances: MLP Outcome Generative Model
In nonlinear data-generating environment, we expected ens oos-POGAMSITTS to outperform its linear counterparts, given its more flexible outcome model. However, in other simulation settings, ens ons-PoGAMBITTS exhibits volatile performance. Interestingly, the linear PoCAMBITTS agent per- forms relatively well, despite the misspecification of the outcome model. One possible explanation for this pattern, and for the broader volatility observed in ens PoCAMMMATS performance, is that, within the GAMBITTS framework (and in Thompson sampling more generally), the outcome model is used to support decision-making rather than to produce accurate predictions. As described in Appendix G.2 the neural network models counterparts, and may require far more data to converge. In contrast, while the linear models are misspecified, they may still capture sufficient structure to support effective decision-makee making.
substantially more parameterized than their linear
The volatility in ens -oSAMGGAM performance suggests that the approach holds promise, but is not panacea. Without tuning tailored to the specifics of the application, the agent may perform poorly. Future work could develop principled strategies for hyperparameter selection to make ens-poGAMBITTS more robust variation in the data-generating process.
F.3 PoGAMBITTS Simulator Access
As discussed in Section 4 partially online GAMBITTS variants use simulator access to construct poff an empirical estimate of the treatment- generating distribution. Figure 9 examines the sensitivity of POGAMBITTS and ens AaS-POAAMBITTS performance the number of samples per (z, a) pair used to construct fo
We expected the partially online GAMBITTS variants to perform better with increased access to the treatment- -eeeeenttnggenningggggg distribution. Figure 9 supports this hypothesis: cumulative regret decreases as the number of simulator draws increases. Interestingly, performance stabilizes after roughly 50 draws, suggesting that limited simulation access may sufficient to realize most of the benefits of the partially online approach.
40

---

Cumulative Regret
Cumulativv Reyrel
15 samples
50 samples
100 samples
500 samples
Complete
Time Period (1)
Time Period (1)
(a) PoGAMBITTS
(h) ens eos-POTGME
Figure Cumulative Regret Under Varying Levels of Simulator Access
F,4
Treatment Dimension
We also examine the effect of d. the dimensionality of the treatment embedding. To do so, introduce 15 additional style dimensions, supplementing the five used in Section 6 for a total of 20 dimensions: optimism, formality, encouragement, severity, clarity, humor, complexity, vision, detail, threat-level, urgency, politeness, personalization, conciseness, actionability, emotiveness, authoritativeness, authenticity, supportiveness, and gender-codedness
Canulative Regnet
Comolative Regnet
Canulazive Regret
Limee Period (t)
Time Period (0
Time Period
(a) d=3
(b) d=5
(c) d=10
Canulative Megret
Cumulativs Regiet 10
StdTS
ens-PoGAMBITTS foGAMBITTS
PoGAMBITTS
Time Period (1
Time Paricsl (9)
(d) d=15
(c) d=20
Figure 10: Cumulative Regret for Varying Treatment Emhedding Dimension
We anticipated that GAMBITTS agents would perform increasingly poorly relative to standard Thompson sampling as the embedding dimension d increased. Figure I0 confirms this trend for the foGAMBITTS agent. However, the linear PoGAMBITTS agent remains competitive even as d grows. As discussed in Appendix F2 the volatility in mas-POCAMBITTS performance further underscores the need for future work on robust hyperparameter tuning strategies within the GAMBITTS framework.
F.5 Direct Contextual Influence on Reward
The simulations in Sections 6.1 and 6.2 (along with Appendix C use reward models linear in ooptinism with Gaussian noise. As described in Section 6.3 the other data-generating environments
ITo select the dimensions used in Figures 10a 10e. response db. For each d, we used d-means clustering and randomly sampled one dimension from each cluster.
clustered the style dimensions by their correlation in
41

---

extend this structure to include Zophimism Zformality, and Zeeverity again with Gaussian noise This appendix considers a simulation designed evaluate GAMBITTS performance when rewards depend directly on context as well as semantic dimensions of GAMBITTS agents over standard Thompson sampling in our main experiments, we expect this advantage to persist under additional covariate influence.
14
[3
Given the substantial performance advantage
In this appendix, we extend the setup from Figure 4t by allowing rewards to depend directly on context. As described in Appendix G the simulated contextual variables were current location and steps taken the previous day. Specifically, we use the data getiaratnggnnnggg mode
Y,- gootimism
L zuplimism Bformality Z
Formality geeverity Zz
Zievenity
+ Buleps X yetepeprevday
Bluc Xpeedor + Et.
locELocs
Figure 11 reports results under correctly specified GAMBITTS agents. The figure supports the intuition that the advantage of GAMBITTS persists under direct covariate influence, showing little deviation from Figure 4b
Cumulative Regret
StdTS
StdTS:Contextual
ens-PoCAMBITTS
foGAMBITTS
PoGAMBITTS
1000
Time Period
Figure 11: Cumulative Regret Under Reward Model with Direct Context Influence
F.6
Scaling Number of Arms: Revisited
As observed in Section 6.3 foGAMBITTS underperforms standard Thompson sampling when the number of available arms, K, is large. However, the appropriate comparison is to contextual Thompson sampler, since foGAMBITTS incorporates covariate information through the treatment embedding. Moreover, Section 5 shows that foGAMBITTS is expected outperform standard Thompson sampling when > whereas the experiments in Section 6.3 assume investigate this further, examine foGAMBITTS performance with K 40 under varying levels of outcome noise.
02. To
WWith two exceptions: Appendix F.2 instead uses an MLP mean reward model, and Appendix F.4 varies the number of semantic dimensions in the mean reward model.
II all cases, rewards depend text indirectly through the semantic dimensions. Semantic dimensions functions of -produudd text. Although other data-generativn environments do not specify a direct effect of context on reward, context enters implicitly through the LLM query, which shapes the generated text and hence the semantic dimensions (described further in Appendix G
'Wee use Figure 4b as the baseline because it employs the default choices for (i) the number of arms, (ti) the set of style dimensions, and (iii) the reward noise level used throughout the experiments in this manuscript.
"Here Locs { ("home", "work", "a location other than home or work") and X ateppepreeyy was recoded to be 1, 2, 3, 4 if the simulated previous day's steps 0-4,999, 5,000-9,999, 10,000-15,000, 15,000, respectively.
than
19ehe standard contextual Thompson sampler continues underperform in this setting. Incorporating covariates requires
heavily parameterized model for the contextual Thompson sampling agent than for its non-contextual counterpart. When covariate effects are modest, this added complexity increases variance and can reduce performance (a known issue in contextual bandit learning [47). Furthermore, this simulation, the optimal action does not vary across context, limiting the potential gain from contextual information. Further work could examine GAMBITTS performance under different forms and strengths of contextual influence.
42

---

1730
Cumulative Regret
Curullttve Regret
1500
1230
000
500
4000
SX0
10003
2900
4001
5000
10803
Time Period (t)
Time Period (t)
(a) 02=0.71
(b) d2=12.64
300
Cumulative Regre!
1300
Std TS
Std TS: Contextual
foGAMBITTS
Time Period (t)
(c) 02-24.57
Figure 12: Cumulative Regret for foGAMBITTS with K 40 under Varying Reward Noise
Figure 12 supports the results in Section 5 foGAMBITTS performs increasingly well relative to standard Thompson sampling as grows. Moreover, even for small Ø2, foGAMBITTS outperforms contextual Thompson sampling agent. To better observe asymptotic behavior under high noise, we extend the time horizon to 10,000 rounds. Even so, cumulative regret does not appear to level off, suggesting that the agents are still learning the optimal policy. This persistent learning phase likely reflects the low signal-to-noise ratio estimated from the IHS study that generated the outcome parameters. Future research directions to address such scenarios could incorporate partial pooling methods (e.g.. Huch et al. [2020]) to share reward model information across and accelerate policy learning [20).
G Simulation Study Design
To evaluate algorithm performance a setting where ground truth available, we needed full knowledge of the data-generating process. This ruled out querying black-box language model during simulation runs, since doing so would obscure the underlying conditional expectations and make performance comparisons difficult interpret. Instead, simulated on-the-fly fly text generation by constructing a response database (referred to as response_db) for each prompt-context pair, we sampled 1, 1,000 responses from LLM and computed the style embeddings associated with each response, as detailed in Appendix G.3, At runtime, we drew from this pool to mimic ...M-ggnerrted responses, described in Algorithm
and
IIn particular, Llama with 8.0B parameters, 131,072 context length, 2,095 embedding length, and Q4 M quantization, accessed via Ollama (42.. Additionally, we re-ran select analyses using Qwen- based LLM. The results were consistent with those in the main manuscript, suggesting that the simulation conclusions sensitive to the choice of LLM, though further work would be needed establish this more formally.
43

---

We derived the outcome- outcome-generating model using data from the 2023 IHS study For each outcome model structure (linear neural network) and relevant style dimensions, we ht the corresponding model to the IHS data and used the estimated parameters to define the outcome surface. This provided a data-driven but controlled setting for comparing policies 21 We provide the exact parameters used in the data-gendn data-generating process on GitHub
20
Lastly, in our simulations, we included two context variables:
stepsprevday E (*0-4,999", "5,000-9,999", "10,000-15,000", "mmor than 15,000") and currloc E ("home", "work". "a location other than home or work").
These context variables varied uniformly across time points and were sampled independently of the action.
Algorithm 6 Pseudocode for Outcome Generation
Inputs: response style embedding functions f mean model structure
m2 (z, x; 02), data-generative outcome model parameters 0! conditional variance ož.
1: Given prompt and context randomly sample response y from response_ __b,, restricted to responses generated from r).
2: Compute style embedding vector 2 = [f(()) fD()))"
3: Simulate outcome by drawing ~ N (0, Ø2) and setting y = m2 (z,z; (US)) + e.
G.1 Prompt Specifications
To initialize the LLM environment for crafting intervention text, we provided the following system prompt:
You are tasked with writing single message to an individual. The message should be two to three sentences long and will be delivered the individual through mobile app. The intent of the message is to encourage the individual to take more steps, with the ultimate goal of improving their physical health. The app collects the following information on its users: whether they took 0-4,999, 5,000-9,999, 10,000-15,000, or more than 15,000 steps the previous day, and whether they are currently at home, at work, at another location.
To generate the intervention text used in our simulations, constructed prompts that explicitly manipulated specific stylistic dimensions. Each prompt was designed to target particular user context and asked the LLM to produce a message in which given dimension (e.g., optimism, formality) was either high or low. This approach yielded total of 40 prompts (one high and one low version for each of the 20 dimensions) attempting to promote alignment among the variation in the generated text with the interpretable axes used in the outcome model.
To generate simulation messages (for given contexts stepsprevday and currloc), used the following prompt structure:
The app provides the following context about the individual: they took stepaprevday steps the previous day and are currently at currloc. This context may be included in the message but does not need dominate it. Please make the message mildly <style polarity>. <dimension description>. Please do not include anything in your response other than the text of the message.
Here, <style polarity> refers to the high/low level of the target stylistic dimension (e.g., "op- timissie""pesssmisttee"" for the optimism dimension). <dimension description> is a descriptor
"e used the square root of the number of steps taken by the user on the day following the message as the reward. This transformation, rather than using step counts, yielded approximately normally distributed errors.
2'e one exception is our use of response_ct db and explored sensitivity to larger values of 02 in Appendix F.I
in the main simulations. We observed d from the variability in
44

---

of the stylistic quality, intended to guide the LLM toward the appropriate tone Unless otherwise noted (e.g.. Section 6.3), all experiments were run using five prompts available to the agent, chosen to be evenly distributed by expected outcome 23
G.2 Agent Specifications
We evaluate several bandit agents in Section and Appendix Unless otherwise noted, each agent type followed the base specifications described in the corresponding subsection below. Furthermore, all GAMBITTS based algorithms used a correctly specified mean outcome model (unless otherwise noted: e.g., Section 6.2). When possible, we used data from the 2022 IHS study to inform prior specification. For example, the average square root of daily steps was approximately 77, which guided our choice of prior for intercept terms.
G.2.1 Standard Thompson Samplers
The standard Thompson sampling agents modeled the reward for each action using normal likeli- hood, with a prior of the form ~ N (77, o.), where o² ~ unverse-Gammin 1, 10). The contextual Thompson sampling agents used the same model and priors, but indexed them by each context- pair (I, a).
G.2.2 POGAMBITTS Agent
Linear POGAMBITTS agents modeled the reward as Y Ni6 BT, for Z, Rd and E Raxdd Letting B == [Bo B7 E TRd+1, these agents placed the following priors on model parameters:
02 ~ N (Ho, Bo)
Po 77 ... 0] E R
B. = diag (0 -2 ,,,....,1 E p(d+1)x(d+1).
G.2.3
ens POOGABITSSSSSS Agent
Our implementation of ens POGAMMAASS uses PyTorch [1], with each neural network instantiated using Py PyTorch's s default initialization: weights and biases were drawn from a uniform distribution
u
where k the number of input features. The ensembles consisted of Mens 60
Vi Vh.
networks. Each network was single-layer feedforward model with 64 hidden units and ReLU
activation. Online training was performed with a batch size of 100 and a learning rate of 0.1. The ensembles maintained replay buffer of size 1,024. To approximate the expectation in Algorithm 4 we used 100 Monte Carlo samples. In each experiment, ens ens-poOAMBITTS training began after burn-in period of 100 steps, during which the neural networks did not update, allowing sufficient data to accumulate for batch-based optimization. Lastly, the agents used the data-generative d2 as the perturbation noise standard deviation.
G.2.4 foGAMBITTS Agent
foGAMBITTS agents used the same prior л2 on O2 as defined for the PoGAMBITTS agent Ap- pendix G.2.2 above. foGAMBITTS agents used a normal likelihood for Z E Rd with mean 01 Rdd The prior कi for 0, given below:
ΣS Inverse- Wishart (Ia,
0 ~ ~ (0,. Ex),
where Ia E Rddxd 022 E Rd, and d is the dimension of the latent treatment space Z.
For example, the following is the description of the optimism dimension: "Here, optimism is defined the degree of hopeful confident language in the message. A lower optimism level may include matter-of fact or pessimistic tone, while higher optimism level may include uplifting and buoyant language." E.gg, if an experiment included only three prompts, we selected the highest, lowest, and median prompt based on true expected value under the data getta-genraatig process.
45

---

G.3 Text Embedding Construction
We aimed to quantify LLM- M.MM-geerated responses along five stylistic dimensions: optimism, encourage- ment, formality, clarity, and severity For each dimension, d, we constructed mapping fa 9 - R. as described in Algorithm[Zjeele [/eloo.. This appendix details the construction of these embeddings and provides evidence that they capture the intended stylistic dimensions. This evidence is included primarily to aid interpretability of the simulation experiments. While high-quality embeddings are crucial in applied settings, they are not required for demonstrating the validity of the GAMBITTS approaches.
We learned each fa by prompting an LLM to generate a set of JITAI-style messages (brief behavioral messages aimed encouraging users to walk more, mirroring the setup of the IHS and HeartSteps Appendix G.3. below contains the exact prompt template used for this purpose. For each prompt, we asked the LLM to vary the degree which its response reflected particular stylistic dimension (d). The responses were intended to isolate variation along that dimension while holding other properties relatively constant. The goal was not to assign explicit scores or labels, but to induce embeddings in which each axis captured targeted semantic variation.
studies) BS 40
We then trained variational autoencoder (VAE) on the resulting text set, constraining the model to use a single latent dimension. The output of this latent space was used as the embedding fa(g) for each message g This process was repeated separately for each dimension. Algorithm 7 below provides pseudocode for this process.
Algorithm 7 Pseudocode for Style Dimension d Embedding Construction
Input: Style dimension label d, LLM, high-dimensional embedding model, VAE architecture, training size M
Mr Minain using the template presented in Appendix 1 1: Generate Mirain random prompts {p!] } i=1
2: For each prompt, pl, in Stepl 1 query the LLM with the prompt to receive r
3: For each response, re in Step 2, obtain a high-dimensional numerical embedding (c. (e by using the high-dimensional embedding model [55].
4: Given embeddings ej d,h ..... M train variational auto-encoder to obtain a dineeimmnsi
d
latent representation of the response 6 Call the resulting mapping fa-
As with the outcome generative model, we used Llama 3.1 as the LLM for embedding construction. Additionally, we used BERT as our high-dimensional embedding model [T] and set M 22, 000.
G.3.1 Embedding Generation Prompt Creation
To initialize the LLM environment, we provided a system prompt describing the context and purpose of the message generation task (e.g., encouraging physical activity through behavioral messages). This global instruction was held fixed across all prompt constructions and consisted of the following:
You are tasked with writing eleven separate messages an individual. Each message should be two to three sentences long and will be delivered to the individual through a mobile app. The messages should be independent and must not reference each other. The intent of each message is to encourage the individual to take more steps, with the ultimate goal of improving their physical health. The app collects the following information on its users: whether they took 0-4,999, 5,000-9,90 9,999, 10,000-15,000, or more than 15,000 steps the previous day, and whether they are currently home, work, or at another location.
To simulate messages that varied along a given stylistic axis, we used a template-based user prompt, shown below. The specific wording of the prompt varied by stylistic dimension; we present the version used for optimism as an illustrative example.
Along with the additional dimensions for the experiments in Appendix F44
We note that the messages used to train the VAE were distinct from those contained in the response_de database used for outcome simulation.
46

---

The app provides the following context about the individual: they took stepsprevday steps the previous day and are currently currloc. This context may be included the messages but should not dominate them.
The primary axis of variation in the messages should be optimism, with no intentional variation along other dimensions such as tone, length, or formality. For this task, create eleven messages where the optimism level varies as follows: Message should represent the least optimistic tone (optimism level 0). Message 11 should represent the most optimistic tone (optimism level 10). Messages between should gradually and evenly increase in optimism. Here, optimism defined as the degree of hopeful or confident language in the message. A lower optimism level may include more matter-of-fal fact pessimistic tone, while higher optimism level may include uplifting and buoyant language. Remember, low levels of optimism should have a pessimistic tone.
Please write each message on a separate line, and remember that the target length is to three sentences for each message. Do include anything in your response other than the text of the messages, Start every new message with its number. The optimism of the message should be independent of the specific information.
For this purpose, we randomly drew context variables stepsprevday and currloc uniformly from their possible values 26 We varied the user context across prompts to encourage the VAE to learn stylistic variation independently of context. For each style dimension d, we generated 2,000 responses (each consisting of eleven JITAI messages spanning the stylistic axis) yielding a total of 22,000 messages per dimension.
G.3.2 Text Embedding Results
After running Algorithm shows this relationship
compared the target input rating with the VAE output. Figure 13
10
Input Rating
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
VAE Score
Figure 13: Optimism Scores versus Prompted Rating
We find that the VAE captured some signal related to the intended tone of the message. While the relationship in Figure [3 is noisy, two caveats are worth noting. (i) Precisely quantifying variation along semantic dimension is likely domain-specifer challenge and, while important, outside the scope of this work. Our goal here is not to validate the embedding as a standalone
We used the same context variables for the embedding construction for outcome generation. Note that we removed two outliers with high VAE scores, one with rating ten and one with rating nine, which distorted the scale of the plot.
47

---

construct; nonetheless, alignment between prompted tone and latent score adds interpretability to the simulation setup. (ii) Some of the observed noise reflects inconsittencies LLM behavior: the model occasionally generates responses that do not align with the requested tone. For example, the following message was produced in response to a prompt with optimism rating 1, but appears highly optimistic: "You'n re on roll after yesterday's success keep up the pace and see where it takes you!" Larger LLMs or more carefully tuned unsupervised methods may improve the correspondence between prompted tone and latent score.
G.3.3 Embedding Construction Summary
Appendix G.3.1 presents the prompt instructions used for generating text that varied in optimism, and
Appendix G.3 shows how the resulting VAE scores relate to the specified input levels. As illustrated
in Figure 13 the VAE captures variation along the target semantic dimension, but its unsupervised nature means the sign the scores is arbitrary (for example, positive scores appear to align with pessimistic rather than optimistic text). Table summarizes the style definitions provided in the prompts for each of the five primary semantic dimensions and reports the styles of text the VAE associated with positive and negative scores 28
Table 1: Primary Semantic Styles, Prompted Definitions, and Interpretation of VAE Scores Style
Style Definition
VAE Score Interpretations
Negative Positive Optimistic Pessimistic
Optimism
"The degree of hopeful or confident language in the message. A lower optimism level may include more matter-of-fatt or pessimistic tone, while higher optimism level may include uplifting and buoyant language." "The degree of dire drastic language in the message. A lower severity level may include more lax, calm, gentle tone, while higher severity level may include more dark, intense, worrying and distressing language."
Severity
Severe
Lax
Formality
"The degree of which the message has an objective, academic, or professional tone. A lower formality level may include more personal, casual, or emotional tone and include colloquial language. A higher formality level may include more matter-of-fact, impersonal, professional and serious language."
Informal
Formal
Clarity
"The degree to which the intent of the message intelligibly communicated. A lower clarity level should be more vague and ambiguous in its language, while higher clarity level may include more precise, coherent, and intelligible instructions comments."
Clear
Unclear
Encouragement
"The degree of persuasion using positivity, contidence, and hope. A lower encouragement level may include a more depressed, dispirited or hopeless tone, while higher encouragement level may include more excited, heartening, and motivating language."
Discouraging Encouraging
G.3.4 Relationship between Style Embeddings
Table G.3.4 below shows the empirical correlation (within the response_ db dataset constructed for outcome generation) among the five stylistic embeddings discussed in Section 6.2
These associations were identified by manually examining sample of generated text and its corresponding
VAE scores.
48

---

Table Style Dimension Correlation Table
Optimism
Formality
Severity
Clarity
Encouragement
Optimism
1.000
-0.095
0.787
0.184
-0.960
Formality
-0.095
1.000
-0.568
-0.248
-0.121
Severity
0.787
-0.568
1.000
0.066
-0.644
Clarity
0.184
-0.248
0.066
1.000
-0.201
Encouragement -0.960
-0.121
-0.644
-0.201
1.000
Furthermore, we analyzed the structure of the relationship between the different style dimensions. Showoss pairwise scatterplots of style embeddings the five primary dimensions. Recall that, in addition to the five primary dimensions,
created 15 more for the experiments in Appendix [.44
The pairwise scatterplots for all 20 dimensions are shown Figure 15 These figures provide qualitative view of the dependencies among dimensions. However, the direction of the embeddings not directly comparable across dimensions; e.g., a high encouragement score corresponds to an encouraging tone, while high optimism reflects greater pessimism.
Clrntty
Fumalls
Opiiieen
Sevetty
Clartt
Encouragemeet
Fomulit
Oyciiiim
Severity
Figure 14: Pairwise Scatterplot of Five Primary Style Dimensions
49

---

Figure 15: Pairwise Scatterplot of All Style Dimensions
50

---

G.4 Computing Resources
The simulation of text-based treatments generated by Llama 3.1 was implemented on a high- performance compute cluster. Each node in the cluster included two 2.9 GHz Intel Xeon Gold 6226R processors, GB of allocated RAM, and a single NVIDIA A40 GPU with 48 GB of memory. VAE training was performed on similar setup, except each node was allocated 16 GB of RAM. The simulation studies were conducted on nodes equipped with two 3.0 GHz Intel Xeon Gold 6155 processors, 16 CPU cores, and between 16-32 GB of allocated RAM. Under these specifications, each experiment presented in Section 6 and Appendix (Feeuirrd approximately 50-70 minutes to complete.
H Broader Impact
As discussed in Section [ text-based sequential interventions are used across domains such as mobile health and education support behavioral change and improve outcomes. We view the GAMBIT framework and associated algorithms as a step toward more personalized decision-making in such settings. At the same time, personalization introduces risks (particularly in sensitive domains) where automatically generated content may influence individuals in subtle or unintended ways. In some applications, it may be inappropriate to deliver GenAl-generated content without human oversight safety review, We discuss these concerns and potential adaptations to the framework in Section 7 with the hope that variants of GAMBIT be developed address such challenges directly.
51


Reviews:
[{"review_id": "NfDjEMgqxG", "reviewer": "Reviewer_1gsV", "summary": "This paper introduces GAMBITTS (Generator-Mediated Bandit–Thompson Sampling), a novel bandit framework that explicitly models environments where agent actions (queries/prompts) produce stochastic treatments (e.g., LLM-generated content), which in turn influence rewards (e.g., user response). Motivated by adaptive mobile health interventions, the authors formalize a generator-mediated causal structure, propose both fully online and partially online variants of Thompson Sampling algorithms (foGAMBITTS and poGAMBITTS), and offer theoretical regret bounds. They also propose ensemble extensions for nonlinear reward modeling and empirically demonstrate improved performance over standard bandits in simulations based on real-world mHealth studies.", "questions": "Could GAMBITTS be extended to incorporate uncertainty or feedback on the generator output quality (e.g., hallucination risk, toxicity)? In high-dimensional cases, could contrastive or self-supervised embedding learning methods be integrated to learn Z on the fly?", "limitations": "While the authors have discussed the limitations of the methods, an additional discussion on the limitations of practical applications would be beneficial to enhance the paper's strength.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- GAMBITTS is well-motivated and grounded in causal reasoning, and both algorithmic and theoretical components are clearly articulated.\n- The regret bounds offer insightful decompositions of uncertainty from treatments and rewards.\n- While motivated by mHealth, the approach is broadly applicable to marketing, education, and other personalization domains involving LLMs.\n\n\nWeaknesses:\n- Performance critically depends on the quality of the treatment embedding (Z), yet embedding learning is left to future work. In practice, choosing or learning Z is nontrivial.\n- While simulations are well-designed, evaluation on real-world GenAI-intervention data is not yet feasible and leaves empirical validation incomplete.\n- The framework assumes the generator does not adapt or improve, which might limit its applicability in settings where models are updated over time.", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer 1gsV for their thoughtful comments on both the strengths of the paper and areas for improvement. We especially appreciated their questions about possible extensions (i.e., Points 4 and 5), which were interesting, insightful, and helped shape our thinking going forward.*\n\n# Point 1: $Z$ Embedding\n>*Performance critically depends on the quality of the treatment embedding (Z), yet embedding learning is left to future work. In practice, choosing or learning Z is nontrivial.*\n\nIndeed, performance does depend on quality of the treatment embedding ($Z$), as discussed in lines 126-129. Moreover, we show empirical evidence for this phenomenon in Figure 3 (Section 6.2), although these results suggest GAMBITTS can still outperform standard Thompson sampling under mild-to-moderate $Z$-misspecification.\n\nOur goal was to present an approach that is flexible with respect to embedding strategy, given that there is no consensus on how best to construct embeddings in settings with high-dimensional treatments such as text. To remain agnostic on this point, we did not include an extended discussion of embedding strategy. However, as discussed below, we agree this is an important applied concern and there are several minor modeling choices an analyst can make to improve robustness to embedding misspecification. We plan to add a *brief* appendix outlining these strategies.\n\nAs mentioned above, we agree that choice of $Z$ is nontrivial and of material concern for applied scientists implementing the algorithm. We outline two directions for addressing this: $(i)$ approaches within the scope of the current paper that aim to improve robustness to misspecification, and $(ii)$ broader directions for future work. We take each in turn.\n\n### $(i)$ Robustness Within the Current Framework\n\n#### $(i.1)$ Adding Prompt Indicator to Reward Model\nBelow line 121, we discuss a model for expected reward given treatment embedding and user context $(m_2(Z_t, X_t; \\theta_2))$. Given any embedding construct $Z$ and model $m_2$, consider the modified form:\n$$\\tilde{m}_2(A\\_t, Z\\_t, X\\_t; \\beta^., \\theta_2)=\\sum\\_{a\\in \\mathcal{A} }\\beta^a \\mathbb{I}(A\\_t=a) + m\\_2(Z\\_t, X\\_t; \\theta\\_2).$$\nWhen the goal is online learning of the optimal action $A$, this structure offers a hedge against misspecification of $Z$ and $m_2$. In particular, if $m_2$ fails to capture signal, its estimated contribution to $\\tilde{m}_2$ will shrink towards zero. In that case, the $\\tilde{m}_2$ model asymptotically behaves like a standard Thompson sampling agent that learns a reward distribution over actions directly. Conversely, if $Z$ and $m_2$ are well-specified, then the action $A_t$ provides no additional information beyond what is already captured through $m_2(Z_t, X_t; \\theta_2)$. In this case, the estimated $\\beta^.$ terms will shrink to zero and $\\tilde{m}_2$ will converge to $m_2$. However, the agent must first learn that the embedding carries signal, potentially delaying efficient generalization early in learning.\n\nWe revisited Figure 3 (Section 6.2) using hedged reward models (as described above). The table below reports point estimates and 95\\% Monte Carlo confidence intervals for cumulative regret at $t=1,000$ for the revised hedged versions for each agent (*a: poGAMBITTS; b: foGAMBITTS*).\n\n| Agent       | Hedged Cumulative Regret    |\n|:---------------------|:----------------------------|\n| Std TS               | 70.4 (65.3, 75.5)  |\n| Clarity (a)          | 53.4 (52.5, 54.4)  |\n| Encouragement (a)    | 53.1 (52.1, 54.0)  |\n| Formality (a)        | 53.2 (52.2, 54.2)  |\n| Optimism (a)         | 52.5 (51.5, 53.5)  |\n| Severity (a)         | 52.5 (51.5, 53.4)  |\n| Clarity (b) | 56.5 (55.6, 57.3)  |\n| Encouragement (b)    | 59.4 (58.5, 60.3)  |\n| Formality (b)        | 55.6 (54.7, 56.5)  |\n| Optimism (b)         | 60.3 (59.3, 61.2)  |\n| Severity (b)         | 59.7 (58.9, 60.6)  |\n\nAll poGAMBITTS agents perform similarly, as do all foGAMBITTS agents. While every GAMBITTS variant outperforms standard Thompson sampling in this setting, the well-specified models no longer exhibit the performance advantage they had in the original (unhedged) Figure 3. This supports the hypothesis that hedging reduces downside risk but limits gains under well-specified embeddings.\n\n\n\n#### $(i.2)$ Maintaining a Set of Candidate Reward Models\nA second approach is to maintain a set of candidate reward models, each potentially using different treatment embeddings. Periodically, the agent can evaluate these models based on predictive performance and switch to the one that best predicts observed outcomes. This allows the algorithm to recover from poor initial choices of $Z$ without requiring strong prior knowledge about which embedding will perform best.\n\n\n### $(ii)$ Future Directions\n\n#### $(ii.1)$ Online Embedding Learning\nA natural direction for future work is to learn individualized treatment embeddings online, providing a data-driven approach to specifying $Z$ (lines 317–321). While we highlight online sufficient dimensionality reduction as one possibility, other approaches may also be effective.\n\n#### $(ii.2)$ Corralling Bandit\nAn extension of the candidate model approach is to use a corralling bandit, where a master algorithm selects among reward models based on performance. This allows the agent to adaptively shift across models, potentially combining their strengths while minimizing regret.\n\n\n# Point 2: Evaluation on Real-World Data\n\n>*While simulations are well-designed, evaluation on real-world GenAI-intervention data is not yet feasible and leaves empirical validation incomplete.*\n\n\nWe agree that empirical validation on real-world GenAI interventions is important but currently infeasible, as such systems are not yet deployed at scale (line 270). Our simulations are grounded in realistic settings inspired by the 2023 Intern Health Study (lines 271–272), following a common mHealth development pipeline in which algorithms are first evaluated in simulation prior to deployment. For example, the reBandit algorithm was introduced and evaluated entirely in simulation in [Ghosh et al. 2024](https://www.ijcai.org/proceedings/2024/0805.pdf), then later deployed in the MiWaves pilot study. Similarly, the Oralytics algorithm was first developed and tuned using simulation-based reward design, then deployed in a real-world trial [Trella et al. 2023](https://pmc.ncbi.nlm.nih.gov/articles/PMC10457015/). In line with these trajectories, we view this manuscript as part of the foundational algorithmic development needed to support future deployment of GenAI-powered adaptive interventions.\n\n\n\n# Point 3: Generator Drift\n>*The framework assumes the generator does not adapt or improve, which might limit its applicability in settings where models are updated over time.*\n\nYes, we assume a static generator throughout (line 322). This reflects scenarios where models are siloed and fixed for privacy, regulatory, or engineering reasons. Of course, in many applications the generator may evolve over time. We agree that extending the framework to accommodate non-static generators is an important and interesting concern (lines 322-323). As before, we outline two directions for addressing this: (i) approaches within the scope of the current paper, and (ii) broader directions for future work.\n\n### $(i)$ Approaches Within the Current Framework\nWithin the current framework, there are two natural adjustments to better handle generator drift. First, in poGAMBITTS, the offline dataset used to estimate $f_1^{off}$ can be periodically refreshed by re-generating samples from the current generator. This provides a simple mechanism for accommodating changes in the treatment distribution without fully redesigning the learning algorithm. Second, in foGAMBITTS, the agent can weight recent observations more heavily when estimating the treatment model $f_1$, allowing the agent to adapt if the distribution of generated responses shifts over time.\n\n### $(ii)$ Future Directions\nIn some settings, we may want to deliberately adapt the generator over time to improve treatment quality. As noted in lines 322–323, one promising direction is online fine-tuning of the generator itself, using observed outcomes to improve the quality of the treatments it produces. In this framework, generator drift is not a challenge to be overcome, but can be an opportunity to develop more responsive and personalized intervention pipelines.\n\n\n\n# Point 4: Extensions for Feedback on Generator Output Quality\n\n>*Could GAMBITTS be extended to incorporate uncertainty or feedback on the generator output quality (e.g., hallucination risk, toxicity)?*\n\nYes: this is an immediate and important direction for extension (lines 323–328). To address safety concerns such as hallucination or toxicity, one approach is to integrate ideas from the learning-to-defer literature, allowing the system to block or reject generated texts deemed too risky. This would naturally connect to the cited work on noncompliant bandits, where the action taken (no text sent) may differ from the action proposed (send the text solicited by prompt $a$). Moreover, we believe such approaches could be extended to incorporate user feedback as well. \n\n\n# Point 5: Extensions for On-The-Fly Embedding Construction\n\n>*In high-dimensional cases, could contrastive or self-supervised embedding learning methods be integrated to learn Z on the fly?*\n\nYes, we see online learning of individualized treatment embeddings as a promising direction, as discussed in Point 1 and in lines 317–321. Our current focus has been on supervised approaches, such as sufficient dimensionality reduction. However, generator access opens the door to alternative strategies, including self-supervised and contrastive methods that do not rely directly on outcome signals. Exploring these approaches would be a natural focus for a future paper on embedding strategies."}, {"review_id": "AZuQRMD62U", "reviewer": "Reviewer_tZeG", "summary": "New contextual bandit model where the planner action is a query to an LLM which then goes to the environment (e.g., mHealth patient) potentially generating a reward. Relates to a broader literature where actions are stochastic, but with the ultra-high dimensionality of a language model. They give regret analysis and empirical results.", "questions": "What is the reward model of 6.3? \n\nWhy is the contextual TS a “more natural” comparison, and if so, why not include it in the main text figures?\n\nCan you comment on steps one might take toward learning the mapping from G to Z, in either an offline or online way? This will be key to adapting the framework to new domains where planners may have less informed priors on reward structures.\n\nCan you comment on the generality of GAMBITTS – other applications where, as you mention, there may be an interest in learning some randomized embedded action dimension… perhaps there is a connection to causal MDPs, where this is a known latent structure? https://proceedings.mlr.press/v177/lu22a/lu22a.pdf \n\nShould add related work on LLMs and bandits… much study on the LLM as the agent \nhttps://arxiv.org/html/2410.06238v1\nhttps://arxiv.org/pdf/2403.15371", "limitations": "yes, authors have a nice paragraph on limitations and future work", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nGAMBITTS framework seems to be new and relevant, will likely be relevant to broad set of researchers\nThey provide algorithms for solving as well as theoretical analysis\nThey provide empirical analysis on real-world derived datasets\nRelated work is sufficient, good connections to the literature on bandits with randomized actions, good connections to literature on natural language as an action space.\n\nWeakness\nMy main takeaway from this paper so far is that the most important thing a planner can do is learn a good projection from action space to embedding space before the policy execution/learning starts… which is a departure from the bandit part of the problem altogether. And further, the problem of learning the projection isn’t explored. Maybe this models some realistic scenario, but it does not capture what seem to be the richer parts of the problem from a methodological perspective. This is likely balanced by the value of introducing the formal model.\nEvaluation is somewhat lacking. There is no exploration of a case where the reward is a function of context, despite the choice to model it. What is the reward model of 6.3? Why is the contextual TS a “more natural” comparison, and if so, why not include it in the main text figures?", "was_revised": true, "final_justification": "Thank you to the authors for the additional clarity and results. I'm keeping my borderline accept score, as I'm finding myself with additional questions to each response (all about the evaluation) that are leading me to believe there's more to dig into than can be clarified in the rebuttal phase... e.g., we've gone through multiple rounds of additional simulation -- more here than just clarifying questions about the original submission. That said, I feel there's still enough here in the model and theory to accept, but I'm not willing to fight for this submission, as it could certainly benefit from more work on the evaluation.", "rebuttal": "# Point 1: Mapping from $G$ to $Z$\n## Point 1(a)\n>*The most important thing a planner can do is learn a good projection from action space to embedding space before the policy execution/learning starts*\n\nCertainly, the planner would benefit from a good projection from $G$ to $Z$, as evidenced by Figure 3 (Section 6.2). While these results show GAMBITTS can still outperform standard Thompson sampling under mild-to-moderate $Z$-misspecification, we agree that many applied scientists may wish to pursue modeling strategies robust to embedding misspecification. We briefly outline two directions for addressing this here: $(i)$ approaches within the scope of the current paper, and $(ii)$ broader directions for future work. **Furthermore, we refer readers to our response to Reviewer 1gsV Point 1 for a more in-depth discussion on these directions**, and note that we plan to include a brief supplement outlining these directions.\n\n\n\n### $(i)$ Robustness Within the Current Framework\n\n#### $(i.1)$ Adding Prompt Indicator to Reward Model\nBelow line 121, we discuss a model for expected reward $(m_2(Z_t, X_t; \\theta_2))$. Given an embedding construct $Z$ and model $m_2$, we consider the modified form:\n$$\\tilde{m}\\_2(A\\_t, Z\\_t, X\\_t; \\beta^., \\theta\\_2)=\\sum\\_{a\\in \\mathcal{A} }\\beta^a \\mathbb{I}(A\\_t=a) + m_2(Z\\_t, X\\_t; \\theta\\_2).$$\nThis helps the agent hedge against embedding misspecification when the goal is online learning of the optimal action $A$.\n\nWe revisited the simulation results in Section 6.2 using this hedged model. Brief summaries of these results can be found in our response to Reviewer 1gsV Point 1.\n\n\n#### $(i.2)$ Maintaining a Set of Candidate Reward Models\nA second approach is to maintain a set of candidate reward models, each potentially using different treatment embeddings. Periodically, the agent can evaluate these models based on predictive performance and switch to the one that best predicts observed outcomes. This allows the algorithm to recover from poor initial choices of $Z$ without requiring strong prior knowledge about which embedding will perform best.\n\n\n### $(ii)$ Future Directions\n\n#### $(ii.1)$ Online Embedding Learning\nAs noted in lines 317–321, a natural direction for future work is to learn individualized treatment embeddings online, providing a data-driven approach to specifying $Z$. \n\n#### $(ii.2)$ Corralling Bandit\nAn extension of the candidate model approach is to use a corralling bandit, where a master algorithm selects among reward models based on performance. This allows the agent to adaptively shift across models, potentially combining their strengths while minimizing regret.\n\n\n\n\n## Point 1(b)\n>*[Learning the mapping from $G$ to $Z$] is a departure from the bandit part of the problem altogether.*\n\n\nFrom an applied perspective, the goal of the embedding is to recover a model for $Y\\mid Z$ that approximates $Y\\mid G$, as discussed in lines 126-129. Broadly, this is fundamentally the outcome modeling central to Thompson sampling-based approaches. While this step becomes more challenging with high dimensional treatments (e.g., text), we do not view it as a departure from the bandit framework. Rather, the high-dimensional nature of the treatment (and subsequent inability to directly model $G$ to $Y$) introduces new modeling challenges and distinguishes our work from earlier mediated bandit approaches (see Appendix B).\n\n\n## Point 1(c)\n>*The problem of learning the projection isn’t explored.*\n\nWe aimed to propose a flexible framework that could accommodate the diversity of embedding strategies already in use (e.g., cited works on line 131). Accordingly, we focused on the development of the GAMBITTS model and intentionally kept the approach agnostic to embedding construction to ensure applicability across diverse techniques and modalities (e.g., text, sound, images).\n\nWith this in mind, a focused study on embedding strategy would be a valuable contribution in its own right. However, in the meantime, there are several minor modeling choices analysts can adopt to improve robustness to misspecification. As discussed in Point 1(a), we plan to add a *brief* appendix outlining these strategies.\n\n\n\n## Point 1(d)\n>*Can you comment on steps one might take toward learning the mapping from G to Z, in either an offline or online way?*\n\nAs noted in lines 122–125, the appropriate embedding strategy is highly dependent on the application area. In domains like mHealth, planners often have access to predecessor trials or related intervention data, which can support offline construction of a working embedding. For example, the personalized HeartSteps study ([26]) was a second iteration of a JITAI development trial, and prior data could plausibly be used to inform treatment representations.\n\nIn cases where such data are limited or the embedding is poorly aligned with outcomes, one can adopt more robust modeling choices as discussed in Point 1(a). Alternatively, embeddings may be adapted online, though the development of these online embedding strategies remains an important direction for future work (see Point 1(a) and lines 317–321).\n\n\n\n\n# Point 2: Evaluation\n## Point 2(a)\n>*Evaluation is somewhat lacking. There is no exploration of a case where the reward is a function of context, despite the choice to model it.*\n\nWe chose to omit context-dependent rewards in the main experiments to streamline the narrative and more directly highlight the benefits of incorporating observed mediation via text. However, we did run simulations where the reward depends on covariates, and the qualitative results were unchanged. As an illustrative example, we revisit the Figure 2 simulation (line 287) with reward depending on location, prior steps, and optimism. The table below reports point estimates and 95% Monte Carlo confidence intervals for cumulative regret at $t=1,000$. All GAMBITTS agents considered remain correctly specified under this setup.\n\n\n| Agent  | Original | With Covariates   |\n|:-------|:---------|:------------------|\n| Std TS: Contextual |    75.9 (74.0, 77.8)  | 76.4 (74.5, 78.2) |\n| Std TS |   70.4 (65.3, 75.5) |  71.3 (66.3, 76.4)  |\n| foGAMBITTS |  38.2 (37.8, 38.6)  |  34.2 (33.8, 34.5)  |\n| poGAMBITTS |  9.6 (9.1, 10.0)  |  8.8 (8.3, 9.2) |\n\nTo improve transparency and clarify this point, we will revise the supplement to include a subset of these simulations with covariate-dependent rewards.\n\n## Point 2(b)\n>*What is the reward model of 6.3?*\n\nIt is a linear reward model in optimism, formality, and severity. I.e.,\n$$Y_t = \\beta_0 + \\beta^{optimism}Z_t^{optimism} + \\beta^{formality}Z_t^{formality} + \\beta^{severity}Z_t^{severity} + \\varepsilon_t.$$\n\nExact values for all coefficients are located in the code supplement, and we will link to the GitHub repository when producing a de-anonymized draft.\n\n## Point 2(c)\n>*Why is the contextual TS a \"more natural\" comparison, and if so, why not include it in the main figures?*\n\nWe described contextual Thompson sampling (TS) as a more natural comparison to foGAMBITTS because both approaches employ separate modeling for each action-context pair $(a,x)$: foGAMBITTS by modeling treatment representations separately for each pair, and contextual Thompson sampling by modeling outcomes separately. However, we chose not to include contextual TS in the main figures because it consistently underperformed relative to standard TS in our simulations, likely due to the limited role of context in the reward. Since it was not a competitive baseline, we omitted it to keep figures focused, as noted in lines 283-285.\n\n# Point 3: Generality of GAMBITTS\n>*Can you comment on the generality of GAMBITTS – other applications where, as you mention, there may be an interest in learning some randomized embedded action dimension. Perhaps there is a connection to causal MDPs, where this is a known latent structure?*\n\nWe view any decision-making setup where prompts to a generative model yield high-dimensional treatments as falling naturally under the GAMBITTS framework. We discuss broad application potentials in lines 21-24. Other examples include LLM-guided robotic control (e.g., https://www.microsoft.com/en-us/research/articles/chatgpt-for-robotics/ and LLM-based protein design, where models generate novel proteins https://pmc.ncbi.nlm.nih.gov/articles/PMC10701588/#s4. GAMBITTS offers a potential tool for efficiently guiding such generation.\n\nThank you for noting the connection to causal MDPs, as in Lu et al. (2022). We see two core similarities:\n\n* A causal chain of the form action $\\to$ intermediate variable $\\to$ reward, where the agent cannot intervene directly on the intermediate.\n* Regret reduction by leveraging the fact that the intermediate lies in a lower-dimensional space than the action space.\n\nKey differences include the nature of the two spaces. Lu et al. consider a combinatorial action space on discrete interventions, while GAMBITTS operates over a large prompt space inducing high-dimensional treatments with low-dimensional structure. GAMBITTS also adopts a two-stage modeling strategy absent from Lu et al.'s framework.\n\n# Point 4: Related Work on LLMs\n>*Should add related work on LLMs and bandits, there is much study on the LLM as the agent.*\n\nWe viewed our primary methodological framing as grounded in mediated bandits and causal inference with text. This shaped our focus on connecting to literature on bandits with randomized actions and text as an action space, as noted by the reviewer. That said, we agree that recent work on integrating LLMs into bandit systems (particularly where the LLM acts as the agent) is relevant from an applied perspective. We will add a paragraph in Appendix B to discuss this line of work in more detail."}, {"review_id": "N44EP9L3zA", "reviewer": "Reviewer_SpBA", "summary": "In GenAI-powered interventions, the agent selects a query, but the environment experiences a stochastic response drawn from the generative model. Standard bandit methods do not explicitly account for this structure, where actions influence rewards only through stochastic, observed treatments. This paper introduces generator-mediated bandit - Thompson sampling (GAMBITTS), a bandit approach designed for this action/treatment split, using mobile health interventions with large language model-generated text as a motivating case study. GAMBITTS explicitly models both the treatment and reward generation processes, using information in the delivered treatment to accelerate policy learning relative to standard methods. The authors establish regret bounds for GAMBITTS by decomposing sources of uncertainty in treatment and reward, identifying conditions where it achieves stronger guarantees than standard bandit approaches. The empirical results show that GAMBITTS consistently outperforms conventional algorithms by leveraging observed treatments to more accurately estimate expected rewards.", "questions": "Please see the weaknesses above.", "limitations": "Please see the weaknesses above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe studied problem, generator-mediated bandits, is related to large language models (LLMs) and may receive attention in the online learning and LLM community.\n2.\tThe authors propose algorithmic instantiations for the GAMBITTS framework, and provide regret bounds for GAMBITTS.\n3.\tEmpirical results show the limitations of standard bandit algorithms and the performance gains achieved by the proposed GAMBITTS approaches.\n\nWeaknesses:\n\n1.\tThe technical novelty in algorithm design and theoretical analysis is unclear. It seems that this paper just applies Thompson sampling to the proposed generator-mediated bandit formulation.\n2.\tThe title mentioned “GenAI-Powered”, but it is unclear how the generator-mediated formulation or algorithm brings benefits to the decision-making process in theoretical results or experiments, compared to standard (no generator involved) bandit problem or standard Thompson sampling algorithm. The paragraph following Theorem 3 discusses the advantage when $d<<\\sqrt{CK}$. But this seems an advantage brought by a low-dimensional representation. How can we understand the benefits brought by generator for the decision-making or learning process?\n3.\tThe writing of this paper needs to be improved. The authors try to sell some big concepts like “GenAI”, but it is hard to understand the technical contributions of this paper.", "was_revised": true, "final_justification": "After reading the response and other reviewers' comments, I tend to weak accept.", "rebuttal": "*We thank Reviewer SpBA for their time and feedback. In our response, we seek to clarify the setting of our paper. Subsequently, we address Point 2 before Point 1.*\n\n# Point 2: Generator Integration\n>*The title mentioned “GenAI-Powered”, but it is unclear how the generator-mediated formulation or algorithm brings benefits to the decision-making process in theoretical results or experiments, compared to standard (no generator involved) bandit problem or standard Thompson sampling algorithm. The paragraph following Theorem 3 discusses the advantage when $d<<\\sqrt{CK}$. But this seems an advantage brought by a low-dimensional representation. How can we understand the benefits brought by generator for the decision-making or learning process?*\n\n\nThe integration of the generator is motivated by applied settings where generative models are increasingly used to deliver personalized sequential interventions (see lines 21–24). In such cases, the planner selects prompts, but the treatment actually delivered is the generator’s output. This motivates the GAMBIT framework, where the generator mediates the action-to-treatment mapping.\n\nOur goal is not to show that introducing a generator improves the learning process; indeed, it *complicates* the action-to-reward feedback loop. Rather, we show how, *in a GAMBIT environment*, explicitly incorporating the generator’s role (via GAMBITTS) can make learning feasible and efficient.\n\nThe theory compares standard Thompson sampling, which models the reward directly as a function of the prompt, with GAMBITTS, which accounts for the treatment generated in response. The referenced discussion around $d<<\\sqrt{CK}$ (lines 254-255) pertains to regret bounds for fully online GAMBITTS relative to standard Thompson sampling, illustrating when the generator-aware approach can provide tighter guarantees in a GAMBIT environment. Earlier theoretical results discuss this comparison in the context of partially online GAMBITTS.\n\n\n\n# Point 1: Technical Novelty\n>*The technical novelty in algorithm design and theoretical analysis is unclear. It seems that this paper just applies Thompson sampling to the proposed generator-mediated bandit formulation.*\n\nIndeed, standard Thompson sampling can be directly applied in the GAMBIT setting by directly modeling reward $Y$ as a function of action $A$. However, as introduced in lines 27-33, and formalized in lines 102-106, the GAMBIT environment induces a separation between the chosen action and delivered treatment. Here, the action ($A$) represents the query sent to the generator. Meanwhile, the treatment delivered to the environment is the generator's response ($G$). While the agent can directly control their action (generator query), they cannot directly control the generated response. In this environment, standard Thompson sampling ignores the mediation by $G$ and directly models $Y\\mid A$.\n\nUnlike standard Thompson sampling, GAMBITTS is designed specifically to handle this mediation by modeling the generator’s output and using that information in the decision-making process. Our formulation explicitly incorporates the generator mediation structure, allowing the agent to update beliefs over treatment effects rather than over prompt effects directly. Theoretical results (e.g., Theorems 1, 2, and 3, Corollary 1) and empirical comparisons in Section 6 and Appendix E show that GAMBITTS can outperform standard Thompson sampling under various data generative environments.\n\nLastly, as discussed throughout Appendix B, there is limited prior work on Thompson sampling in bandit settings with mediated feedback, particularly in the presence of continuous mediators. To our knowledge, no existing Thompson sampling-based methods address the high-dimensional setting induced by generator-mediation or account for the complexities introduced by large-scale generative models producing treatments. GAMBITTS fills this gap.\n\n\n# Point 3: Writing\n>*The writing of this paper needs to be improved. The authors try to sell some big concepts like “GenAI”, but it is hard to understand the technical contributions of this paper.*\n\nOur use of \"GenAI\" reflects the motivating applications, where generative models are already being used to deliver sequential interventions (see lines 21–24). The goal of this paper is not to propose GenAI as a solution, but to analyze decision-making in settings where such models are already in use. In this sense, the contribution is algorithmic: we develop a method tailored to a structure that arises naturally in these interventions. To our knowledge, there are no existing bandit algorithms specifically designed for this generator-mediated setting.\n\nRegarding the quality of the writing, Reviewer q3H4 noted that the introduction and problem motivation were well written, and Reviewer XiMn expressed the same view about the paper overall. If there are specific passages that were unclear, we would be happy to revise them for clarity."}, {"review_id": "DOVNaWySDn", "reviewer": "Reviewer_q3H4", "summary": "This paper presents a Thompson sampling based algorithm for contextual bandits that interact with the environment through a generative model. The action taken by the algorithm is a prompt that is fed into the generative model. The response that is put into the environment is a sample from the generative model. They relate their approach to causal graphs in RL and provide theoretical guarantees for their approach under assumptions on the causal graph. They also empirically evaluate their approach and show that their method improves upon standard Thompson sampling empirically.", "questions": "- I found the Z and Z* discussion sometimes a bit hard to follow. After reading more closely I think I understand it, but it might help to better clarify what are the assumptions of your algorithm and what is an assumption on the data generating environment. For example, in Algorithm 1, I spent a some time thinking about in the integration in step 3 what distribution of z was being integrated over Z or Z*. I’m pretty sure it is Z, but might help to make this more overt for future readers.\n- When I was reading, in step 3 of algorithm 1, isn’t the action space (the space of all possible prompts) huge? I feel there needs to be more acknowledgement of the computational challenges in the main text (even if you say you will discuss it further in a future section), as it made me think either this algorithm is not practical or I am misunderstanding what an example action space could be. Later in algorithm 1, there is X_t being used to select actions. Could you clarify this? Also is this $X_t$ or $X_t^G$?\n- Could you talk a bit more about what “context is sent to generator”? It appears this is not learned by the algorithm but pre-specified?", "limitations": "I feel the authors could have a more thorough discussion of the limitations of their algorithm, e.g., when it is appropriate to apply versus not. As well as discuss computational challenges of implementing their algorithm further.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The problem tackled by this work is well-motivated and timely.\n- The introduction and problem motivation was very well-written\n- The algorithm seems relatively straightforward to understand and has both theoretical guarantees (under assumptions) and performs well empirically in simulations\n\n## Weaknesses\n- I found the “Problem Formulation and Notation” section 3 hard to follow. I feel it would benefit from a rewrite that discusses what all the variables are in a specific problem setup. For example, I am not sure exactly what the context $X_t$ is here (sometimes I thought this was the prompt). Then I wasn’t clear why the action space was finite (as I thought the prompt space could be very large). Should I be thinking about $A$ as a policy and $G$ as a sample from that policy? The general problem setup is still confusing for me.\n- The biggest weakness I see (which could be clarified) is why the action does not depend on the context $X$ in Figure 1. It seems like it is a contextual bandit problem, but the algorithm is not allowed to use the context to select actions, which seems odd.", "was_revised": true, "final_justification": "I thank the authors for their detailed response. I feel their response has clarified many of the questions that I had and I have accordingly raised my score.", "rebuttal": "# Point 1: Clarity of Problem Formulation and Notation\n>*I found the “Problem Formulation and Notation” section 3 hard to follow. I feel it would benefit from a rewrite that discusses what all the variables are in a specific problem setup. For example, I am not sure exactly what the context $X_t$ is here (sometimes I thought this was the prompt). Then I wasn’t clear why the action space was finite (as I thought the prompt space could be very large). Should I be thinking about $A$ as a policy and $G$ as a sample from that policy? The general problem setup is still confusing for me.*\n\nWe begin by clarifying the notation.\n\n* $X_t$ - User context at time $t$. This could include any static user covariates (e.g., gender, race/ethnicity), as well as time-varying characteristics (e.g., steps taken the previous day, current location).\n* $A_t$ - The query sent to the generator at time $t$. This is the message the agent sends to the generator that requests a certain type of output (e.g. \"Please write a message that encourages the user to walk more.\").\n* $\\mathcal{A}$ - The set of possible queries to send to the generator. This set is finite and pre-specified.\n* $G_t$ - The generator's response. This is output of the generator given the query $A_t$ and context $X_t$.\n* $Z_t$ - The working low-dimensional embedding of the full generator output $G_t$. This is used for modeling $Y_t$.\n* $Y_t$ - The reward at time $t$.\n\nUsing LLM-assisted mHealth interventions as an illustrative example, a GAMBIT environment sees the following at time $t$:\n\n* The agent observes user context $X_t$ (e.g., demographic information and steps taken the previous day).\n* The agent selects query $A_t$ and sends it to the generator, along with the user information $X_t$. (E.g., \"Please write a message that encourages the user to walk more. The user is male and took 10,000 steps yesterday.\")\n* The LLM outputs a message $G_t$ (e.g., \"You're doing great! Let's see if you can take even more than 10,000 steps today!\"), which is then sent to the user.\n* The mHealth application records a user outcome (e.g., daily steps), which is stored as reward $Y_t$.\n\n\nWe are considering $\\mathcal{A}$ to be a finite set of prompts that are determined before the GAMBITTS algorithm is deployed; i.e., we are not assuming $\\mathcal{A}$ is the set of all possible prompts anyone could send to a generator.\n\nOne could think of $A$ as the chosen action and $G$ as a noisy representation of that action that is sent to the environment. In the GAMBIT framework, the agent can only exert direct control over $A$, not $G$ (even though the environment experiences $G$, not $A$). Subsequently, the policy concerns choices of $A$.\n\n\nLastly, we agree that adding a concrete example (as above) could help clarify Section 3. We plan to revise the manuscript accordingly.\n\n\n# Point 2: Action Dependence on Context\n>*The biggest weakness I see (which could be clarified) is why the action does not depend on the context $X$ in Figure 1. It seems like it is a contextual bandit problem, but the algorithm is not allowed to use the context to select actions, which seems odd.*\n \n\nIndeed, **GAMBITTS algorithms consider context at time $t$ ($x_t$) when selecting actions**. For example, Step 4 of Algorithm 1 (foGAMBITTS, line 149) states:\n$$\\textbf{Set } A\\_t = \\displaystyle\\text{argmax}\\_{a'\\in\\mathcal{A}} \\mathbb{E}\\_{\\theta_{1,t}, \\theta_{2,t}}\\left[Y\\mid a', x_t\\right].$$\n\nHere, the foGAMBITTS agent selects the action which maximizes expected reward (under the sampled $\\theta$ parameters from Step 2) *given* current context $x_t$. Furthermore, Step 4 of Algorithms 2 and 3 (poGAMBITTS and ens-foGAMBITTS, respectively), as well as Step 3 of Algorithm 5 (ens-poGAMBITTS) involve similar action selection. \n\nFigure 1, however, depicts the causal dynamics of the GAMBIT data-generating process, not the logic of the decision algorithm. The proposed GAMBITTS algorithms (which do incorporate context) are external to the causal model. Accordingly, we omit arrows from $X$ to $A$. We recognize that conventions may differ on this point and will add a note to the Figure 1 caption clarifying the absence of this arrow.\n\n\n\n# Point 3: $Z$ versus $Z^*$\n>*I found the Z and $Z^\\ast$ discussion sometimes a bit hard to follow. After reading more closely I think I understand it, but it might help to better clarify what are the assumptions of your algorithm and what is an assumption on the data generating environment. For example, in Algorithm 1, I spent a some time thinking about in the integration in step 3 what distribution of z was being integrated over $Z$ or $Z^\\ast$. I’m pretty sure it is Z, but might help to make this more overt for future readers.*\n \n\nIn the GAMBIT framework, the generator produces content $G$, which is delivered to the user, who then responds with reward $Y$. In practice, modeling $Y\\mid G$ is intractable due to the high dimensionality of $G$. To address this, we project $G$ into a low-dimensional space, taking $Z:=h(G)$, and model $Y\\mid Z$. This working model is used throughout for reward prediction and action selection.\n\nAs discussed in lines 126-131, GAMBITTS performance depends on how well $Y\\mid Z$ approximates $Y\\mid G$. Poor embeddings (where $Y\\mid Z, X$ is a bad proxy for $Y\\mid G, X$) can degrade performance relative to embeddings that capture more reward-relevant signal.\n\nWith this in mind, we introduce $Z^\\ast$ as a conceptual benchmark: the ideal low-dimensional embedding that captures all reward-relevant information from $G$. It is used solely in the theoretical analysis to characterize regret and establish performance guarantees.\n\nAll algorithmic steps (including the integration in Step 3 of Algorithm 1) operate on the observed working embedding $Z$, not $Z^\\ast$. Since the agent has no access to $Z^\\ast$, it plays no role in implementation.\n\nWe agree that clarifying this point can help avoid confusion, and will revise the introduction to Section 4 in response\n\n\n\n\n# Point 4: Size of Prompt Space\n>*When I was reading, in step 3 of algorithm 1, isn’t the action space (the space of all possible prompts) huge? I feel there needs to be more acknowledgement of the computational challenges in the main text (even if you say you will discuss it further in a future section), as it made me think either this algorithm is not practical or I am misunderstanding what an example action space could be. Later in algorithm 1, there is X_t being used to select actions. Could you clarify this? Also is this $X_t$ or $X_t^G$?*\n\nAs discussed in Point 1, we consider $\\mathcal{A}$ to be a finite set of prompts specified prior to deploying the GAMBITTS algorithm. However, as shown in Section 6.3 (lines 298-309), poGAMBITTS regret does not materially increase with the size of $\\mathcal{A}$. Thus, in settings with simulation access to the generator, the size of the action space can be quite large without sacrificing algorithmic performance. However, the computational cost does scale linearly with the number of arms. We agree this is worth noting and will revise Section 6.3 to include a comment on computational scaling.\n\nRegarding action selection: as discussed in Point 2, the full context $X_t$ is used to select actions. We discuss the distinction between $X_t$ and $X_t^G$ in our response to Point 5.\n\n\n\n\n# Point 5: Context Sent to Generator\n>*Could you talk a bit more about what “context is sent to generator”? It appears this is not learned by the algorithm but pre-specified?*\n\nWe take $X_t$ to represent the full observed context for the user at time $t$. This might include static variables (e.g., demographics) and time-varying ones (e.g., prior behavior). While some components of $X_t$ may be useful for outcome modeling, they may not be appropriate for inclusion in the prompt sent to the generator. For example, a variable like weight could improve reward prediction but would likely be excluded from the generator input for ethical or design reasons.\n\nTo reflect this, we consider interventions which send a subset of user context to the generator (denoted $X_t^G\\subseteq X_t$). We treat the choice of which features to include in $X_t^G$ as pre-specified, though learning this mapping online could be a promising direction for future work.\n\nThroughout the paper, we assume $X_t^G=X_t$ for simplicity and narrative clarity. The proposed algorithms require only superficial changes to accommodate $X_t\\neq X_t^G$ (namely, the treatment model introduced below line 121, $f_1(z; A_t, X_t, \\theta_1)$, becomes $f_1(z; A_t, X_t^G, \\theta_1)$). Since this distinction does not affect the main development, we agree it is clearer to move discussion of $X_t^G$ to the supplement and will revise the manuscript accordingly.\n\n\n# Point 6: Appropriate Applications\n>*I feel the authors could have a more thorough discussion of the limitations of their algorithm, e.g., when it is appropriate to apply versus not. As well as discuss computational challenges of implementing their algorithm further.*\n\nAs noted in Point 4, we will revise Section 6.3 to explicitly discuss how computational cost scales with the size of the action space. In response to Reviewer XiMn Point 3, we also plan to revise Section 1 to include a brief discussion of appropriate settings for algorithm deployment."}, {"review_id": "9CWOGUraAy", "reviewer": "Reviewer_XiMn", "summary": "The paper introduces the notion of Generator-Mediated Bandits with Thompson Sampling. These bandit methods take into account a two stage problem architecture, where the bandit's output is fed to an LLM along with additional context, and the output of the LLM is what determines the reward. This is primarily motivated by personalized decision-making in mobile health contexts, where study participants are at the receiving end of text based intervention prompts. The proposed method, attributed to as GAMBITTS, takes this two stage problem into account by modeling the action as well as the reward generated - part of which is achieved by projecting the LLM's output into a lower dimension embedding. The paper provides regret bounds and experimental results wrt the methods proposed.", "questions": "See above.", "limitations": "See above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written!\n2. The methods proposed to model the LLMs output is very interesting and novel to me.\n\nWeaknesses and Questions:\n1. The paper talks about personalizing JITAIs or pJITAIs on line 39-40, yet none of the references (40 and 26) even mention or elaborate that term - they only talk about JITAIs. Can the paper please be edited to cite proper references?\n2. Line 269 - What are the definition of these dimensions? While some may be apparent like encouragement, it is difficult to ascertain what severity means. Also, what does it mean to have encouragement of 1 - does it mean the prompt is discouraging or just not encouraging (there is a subtle difference between the two). Related - Table F.3.3: What does -0.9 correlation mean between optimism and encouragement?\n3. The authors should be upfront that this paper assumes that there exists an LLM with nice properties like safety. I could only find a small one-liner wrt this in the Discussion section - while adoption of LLMs in these mHealth contexts are held back by the lack of safety guarantees.\n4. Some of the details wrt experimentation should be moved into the main paper. \n5. It would be interesting to see if the results are different with a different LLM. Especially for something like Figure 10.\n6. Appendix E.3 - How is one supposed to interpret the VAE score?\n7. Appendix E.3 - Why 1-10 scaling? Why not use low medium high instead? Given that messages are short in length, I personally do not expect much difference between messages of score 1 and of score 3 for a given prompt. The stochasticity combined with this gives a noisy Figure 10.\n8. Appendix E.3 - Continued from above, seems like the VAE just learnt noise. Why not instead use IRR to assign labels, and let the VAE learn those labels corresponding to where those messages fall under the given category instead? The problem currently is that because it is learning noise, the VAE score is extremely unreliable. \n9. Perception of a message is very \"human\", and it would differ across people. The entire point of Gen-AI is to deliver personalized intervention text. However using a single, population-level scoring does not reflect the heterogeneity in individual perceptions. The paper should address that, or acknowledge it.", "was_revised": true, "final_justification": "I am satisfied with the author rebuttal and the clarifications presented. The authors have also offered to clarify a lot of my concerns in the paper. Hence, I am raising my score.", "rebuttal": "# Point 1: JITAIs vs pJITAIs\n>*The paper talks about personalizing JITAIs or pJITAIs on line 39-40, yet none of the references (40 and 26) even mention or elaborate that term - they only talk about JITAIs. Can the paper please be edited to cite proper references?*\n\nJust-in-Time Adaptive Interventions (JITAIs) are sequential decision frameworks designed to deliver support at moments of need. Personalized JITAIs (pJITAIs) form a subclass that use RL agents to learn decision rules online. [26] details the HeartSteps (V2) pJITAI; however, it does not explicitly use the term \"pJITAI.\" We will add a citation which uses the term (e.g., https://www.ijcai.org/proceedings/2024/0805.pdf).\n\n# Point 2: Dimension Interpretation\n>*Line 269 - What are the definition of these dimensions? While some may be apparent like encouragement, it is difficult to ascertain what severity means. Also, what does it mean to have encouragement of 1 - does it mean the prompt is discouraging or just not encouraging (there is a subtle difference between the two). Related - Table F.3.3: What does -0.9 correlation mean between optimism and encouragement?*\n\nAs described in Algorithm 7 (lines 1238-1246), we obtained the embeddings (for each style dimension $d$) using the following steps:\n\n* Generate prompts designed to elicit variation along style $d$, using a predefined template.\n* Query an LLM with each prompt to generate a text response.\n* Embed each response into a high-dimensional vector space using an embedding model (BERT).\n* Train a VAE on these embeddings to learn a 1D latent variable capturing variation along style $d$.\n\nThis mapping serves as the style embedding function $f_d$. As described in Algorithm 6 (below line 1171), we used these $f_d$ functions to generate style embeddings for the simulated JITAI messages. Correlations refer to the empirical correlation between the embedding values among all the simulated JITAI messages.\n\nTo elicit variation along a style dimension $d$, we crafted prompts that included a definition of $d$. For example, the prompt defining optimism appears on line 1258. Definitions for the remaining style dimensions are available in the code supplement (code/simulation_setup/utils.py); we will link to the corresponding GitHub repository in the main paper upon de-anonymization. When eliciting a style dimension, we aimed to span the full range of intensity, from strongly negative to strongly positive. E.g., for optimism, we prompted the LLM to generate pessimistic, neutral, and optimistic messages.\n\nWe note the unsupervised nature of the VAE means it does not guarantee alignment between the sign of the latent variable and the intensity of the style dimension. For instance, in the case of optimism, the VAE empirically assigned positive values to pessimistic messages. In contrast, for encouragement, positive values corresponded to more encouraging messages (and an encouragement value of 1 would mean a highly encouraging message). This also helps explain the observed negative correlation between optimism and encouragement. While this inconsistency is acceptable in our current setting, where embeddings are used as internal agent features, future work focused on embedding construction could aim to enforce consistent interpretability across style dimensions. \n\n# Point 3: LLM Safety\n>*The authors should be upfront that this paper assumes that there exists an LLM with nice properties like safety. I could only find a small one-liner wrt this in the Discussion section - while adoption of LLMs in these mHealth contexts are held back by the lack of safety guarantees.*\n\nWe agree earlier framing would help clarify the scope/focus of the paper. We will add a brief discussion in Section 1 to more clearly acknowledge the assumption that an LLM with desirable properties (e.g., safety) exists.\n\n# Point 4: Experimentation Details\n>*Some of the details wrt experimentation should be moved into the main paper.*\n \nWe agree that a summary of how the style dimension embeddings were generated, along with clearer references to the relevant appendix sections, would help frame the simulation study and improve accessibility. We will revise Section 6 accordingly.\n\n# Point 5: LLM Sensitivity\n>*It would be interesting to see if the results are different with a different LLM. Especially for something like Figure 10.*\n \nWe also ran simulations using prompts generated by Qwen, while keeping the original VAE-based embeddings trained on Llama-generated text. The results were similar to those in the main paper.\n\nWhen we plotted optimism ratings against the original VAE scores, the resulting curve closely matched Figure 10. This consistency across LLMs suggests that the VAE picked up a real signal, despite substantial noise. We see this as a promising direction for future work, particularly with custom VAEs and improved prompts to reduce noise.\n\n# Point 6: VAE Score Interpretation\n>*Appendix E.3 - How is one supposed to interpret the VAE score?*\n \nWe address this in our response to Point 2. The VAE score reflects the intensity and direction of a semantic dimension, but the sign is not consistently aligned across dimensions. For example, positive scores for optimism and encouragement correspond to pessimistic and encouraging messages (resp.). This inconsistency stems from the unsupervised nature of the VAE, which does not constrain the direction of the latent variable. In future work focused on embedding construction, we would take steps to ensure sign consistency.\n\n\n# Point 7: Scaling Approach\n>*Appendix E.3 - Why 1-10 scaling? Why not use low medium high instead? Given that messages are short in length, I personally do not expect much difference between messages of score 1 and of score 3 for a given prompt. The stochasticity combined with this gives a noisy Figure 10.*\n\nOur goal was to generate text spanning a wide range of intensities along each style dimension. Since we ultimately wanted a continuous embedding, we opted for a finer-grained 1–10 input scale rather than coarse labels like \"low,\" \"medium,\" or \"high.\" That said, the input score is only used to guide LLM generation, we discard it once the VAE embedding functions are trained.\n\nWe agree that LLMs are unlikely to consistently reflect subtle distinctions between nearby scores (e.g., 1 vs. 3), and this likely contributes to the noise observed in Figure 10. As discussed in lines 1272–1277, some of this noise stems from erratic LLM behavior. In future work focused more directly on embedding construction, additional fine-tuning and more targeted prompt engineering could help reduce this variability.\n\n\n# Point 8: VAE Reliability\n>*Appendix E.3 - Continued from above, seems like the VAE just learnt noise. Why not instead use IRR to assign labels, and let the VAE learn those labels corresponding to where those messages fall under the given category instead? The problem currently is that because it is learning noise, the VAE score is extremely unreliable.*\n\nAs discussed in our response to Point 5, the pattern (while noisy) persists across LLMs, which suggests the VAE is capturing *some* underlying signal. That said, we agree that the embeddings are not perfectly reliable. As noted in lines 1268-1272, precisely mapping text to a continuous style intensity is inherently challenging and likely requires domain-specific tuning. Developing more reliable embeddings is an important direction, but not the focus of this work. Our goal is to demonstrate the feasibility of the GAMBITTS framework, which relies on projecting high-dimensional treatments into a low-dimensional space. The alignment between prompted tone and VAE score helps interpret the example projection, despite the noise, but is not necessary to show the validity of the GAMBITTS framework at-large.\n\nWe agree that alternative strategies, such as using IRR-based labels or coarser scoring schemes (as the reviewer suggested in Point 7), are promising directions for future work, particularly for projects that foreground the embedding construction itself.\n\n# Point 9: Embedding Heterogeneity\n>*Perception of a message is very \"human\", and it would differ across people. The entire point of Gen-AI is to deliver personalized intervention text. However using a single, population-level scoring does not reflect the heterogeneity in individual perceptions. The paper should address that, or acknowledge it.*\n \nWe agree that accounting for heterogeneity in message perception is an important concern, especially in settings where the goal is to deliver personalized intervention text. As discussed in Point 1 of our responses to Reviewers 1gsV and tZeG, we consider two classes of approaches for robustness to embedding misspecification: $(i)$ approaches within the scope of the current paper, and $(ii)$ broader directions for future work. Many of these approaches address embedding heterogeneity as well, and we briefly describe these approaches below. Our response to Reviewer 1gsV contains more detailed descriptions.\n\n### $(i)$ Methods Within the Current Framework\n\n#### $(i.1)$ Maintaining a Set of Candidate Reward Models\n*Labelled i.2 in earlier responses.*\n\nMaintain a set of candidate reward models, each potentially using different treatment embeddings. The agent can periodically switch to the model with best predictive performance. This allows the algorithm to adaptively tailor embeddings to individuals, as different models may provide better fit for different users.\n\n### $(ii)$ Future Directions\n\n#### $(ii.1)$ Online Embedding Learning\nLearn treatment embeddings online, providing a data-driven approach to specifying $Z$ for each individual (lines 317–321).\n\n#### $(ii.2)$ Corralling Bandit\nAn extension of the candidate model approach is to use a corralling bandit, where a master algorithm adaptively selects among reward models.\n\nWe plan to include a *brief* discussion of these approaches in the supplement, as noted in our responses to Reviewers 1gsV and tZeG."}]

Author rebuttal:

Meta-review: The paper formalizes generator-mediated bandits and proposes Thompson-sampling algorithms that explicitly model the action→generated-treatment→reward pipeline, with ensemble extensions and explicit randomization probabilities for causal evaluation. It introduces a partially-online variant that pretrains the treatment model via generator simulation, which the authors argue improves online performance in mobile health scenarios.   Reviewers note that performance hinges on the quality of the treatment embedding, whose learning/choice is largely deferred.   They also point to limited real-world validation and the assumption of a static (non-drifting) generator as key weaknesses. To strengthen the paper, I recommend adding a crisp limitations/threats-to-validity section (when the proposed approach helps vs. fails), run ablations across embedding families/sizes with on-the-fly or corralling baselines, and report sensitivity to embedding misspecification.
Decision: Accept (poster)