Title: RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bandit Algorithms

Causal Inference

Supervised Learning

mHealth

Mixed-effects Modeling
Abstract: Mobile health leverages personalized and contextually tailored interventions optimized through bandit and reinforcement learning algorithms. In practice, however, challenges such as participant heterogeneity, nonstationarity, and nonlinear relationships hinder algorithm performance. We propose RoME, a **Ro**bust **M**ixed-**E**ffects contextual bandit algorithm that simultaneously addresses these challenges via (1) modeling the differential reward with user- and time-specific random effects, (2) network cohesion penalties, and (3) debiased machine learning for flexible estimation of baseline rewards. We establish a high-probability regret bound that depends solely on the dimension of the differential-reward model, enabling us to achieve robust regret bounds even when the baseline reward is highly complex. We demonstrate the superior performance of the RoME algorithm in a simulation and two off-policy evaluation studies.

RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions

Abstract

Mobile health leverages personalized and contextually tailored interventions optimized through bandit and reinforcement learning algorithms. In practice, however, challenges such as participant heterogeneity, nonstationarity, and nonlinear relationships hinder algorithm performance. We propose RoME, a Ro bust M ixedE ffects contextual bandit algorithm that simultaneously addresses these challenges via (1) modeling the differential reward with user- and time-specific random effects, (2) network cohesion penalties, and (3) debiased machine learning for flexible estimation of baseline rewards. We establish a high-probability regret bound that depends solely on the dimension of the differential-reward model, enabling us to achieve robust regret bounds even when the baseline reward is highly complex. We demonstrate the superior performance of the RoME algorithm in a simulation and two off-policy evaluation studies.

1 Introduction

Mobile health (mHealth) uses smart devices to deliver digital notification interventions to users. These notifications nudge users toward healthier attitudes and behaviors. Because mHealth applications can monitor and react to users and their environment, they offer the promise of personalized, contextually tailored interventions. In practice, achieving this promise requires bandit or reinforcement learning (RL) algorithms that can accurately learn mHealth intervention effects, including how they vary by user, over time, and based on context. In this regard, contextual bandit algorithms are appealing due to strong empirical performance in other settings, their ability to customize intervention decisions based on changing contexts, and their simplicity and extensibility relative to full RL algorithms.

One unique aspect of mHealth (Greenewald et al., 2017) is the presence of a 'do-nothing' action in which the mHealth app does not intervene. As mHealth rewards typically involve human decision making (e.g., whether the user chooses to exercise), the distribution of the corresponding baseline reward -the reward under the do-nothing action-typically evolves in a complex, nonstationary fashion as time progresses and contexts change. In contrast, the treatment effects -the difference in the expected value of the reward relative to the do-nothing action-tend to be much more stable

Madeline R. Abbott

Walter H. Dempsey

and can be adequately modeled using classical stationary models, such as linear models (Robinson, 1988).

Greenewald et al. (2017) introduced an action-centered contextual bandit algorithm with sublinear regret in this setting by replacing the observed reward, R t , with a 'pseudo-reward,' ˜ R t , where t indexes the time points. Denoting the binary action as A t ∈ { 0 , 1 } with A t = 0 corresponding to the do-nothing action, the pseudo-reward for A t can be written as

where π t is the (known) probability of treatment assignment from the Thompson-sampling algorithm. We took an additional step to derive an equivalent expression (second equality), which reveals that ˜ R t is proportional to an inverse-probability-weighted (IPW) estimator. Crucially, ˜ R t provides an unbiased estimate of the treatment effect regardless of the nonstationarity of the baseline outcome.

Although Greenewald et al. (2017)'s analysis shows sublinear regret in nonstationary settings, our simulations show that other methods can achieve lower regret over finite time horizons. The reasons include its failure to address treatment effect heterogeneity, pool information across users, and model the baseline outcome, which leads to highly variable pseudo-rewards and slow learning. We propose to generalize their method to overcome these limitations by

imposing hierarchical structure in the treatment effect model with shared fixed effects and random effects for users and time (i.e., a mixed-effects model),

efficiently pooling across users and time via nearest-neighbor regularization (NNR), and

denoising rewards with flexible supervised learning models via the debiased machine learning (DML) framework of Chernozhukov et al. (2018).

We name the resulting method RoME to highlight that it is a Ro bust M ixedE ffects algorithm. The primary contributions are (a) the RoME method, (b) a high-probability regret bound that relies solely on the dimension of the differential-reward model, which is typically much smaller than that of the complex baseline reward model, and (c) empirical comparisons demonstrating the superior performance of RoME in simulation and two real-world mHealth studies.

2 Related Work

Related work in statistics considers the estimation of treatment effects with longitudinal data. Cho et al. (2017) present a semiparametric random-effects method for estimating subject-specific static treatment effects. Qian et al. (2020) discusses the statistical challenges of applying linear mixedeffects models to mHealth data and shows that the resulting estimates may lack a causal interpretation.

In the bandit literature, several works address the challenge of heterogeneous users. Ma et al. (2011) introduced NNR in recommender systems using homophily principles-similar nodes are more likely to be connected than dissimilar ones (McPherson et al., 2001). Cesa-Bianchi et al. (2013) adapted NNR for bandit settings, improving regret. Subsequent improvements focused on scalability and algorithm modifications for stronger regret bounds (Vaswani et al., 2017; Yang et al., 2020).

Other bandit approaches explicitly address the longitudinal setting in which treatment effects may evolve over time. The intelligentpooling method of Tomkins et al. (2021) employs a Gaussian linear mixed-effects model with both user- and time-specific random effects. Hu et al. (2021) provides a related approach based on generalized linear mixed effects models. Unlike our approach, both methods require the (generalized) linear conditional mean model to be correctly specified.

Aouali et al. (2023) and Lee et al. (2024) also propose bandit algorithms with mixed effects; methodologically, however, their use of mixed effects differs from the above approaches and our own. Aouali et al. (2023) employs random effects to relate treatments within categories (e.g., movies within a genre), and Lee et al. (2024) considers the case in which a single, discrete action leads to a multivariate reward that is affected by a user-specific random effect. In contrast, the other approaches listed above (and our own) do not assume any relationships among the actions, and they associate each discrete action with a single scalar-valued reward in an iterative fashion.

Other work develops semiparametric methods that lead to sublinear regret without requiring correct specification of the conditional mean reward. Krishnamurthy et al. (2018) and Kim & Paik (2019) propose improvements to the approach of Greenewald et al. (2017) that lead to stronger regret bounds. Kim et al. (2021, 2023) develop a doubly robust approach for both linear and generalized linear contextual bandits. However, in contrast to Tomkins et al. (2021), Hu et al. (2021), and our approach, these semiparametric methods are not immediately applicable to longitudinal settings.

3 Setting

3.1 Problem Statement

Recognizing the connection between mHealth policy learning and contextual bandit methods, we now consider a contextual multi-armed bandit environment with one control arm (the do-nothing action) denoted by a = 0 and q non-baseline arms corresponding to different actions or treatments. Users are indexed by i = 1 , 2 , . . . and decision points are indexed by t = 1 , 2 , . . . . For each user i at time t , the algorithm observes a context vector S i,t ∈ S , chooses an action A i,t ∈ [ q ] := { 0 , . . . , q } , and receives a reward R i,t ∈ R . The differential reward describes the difference in expected rewards between choosing a non-baseline arm and the control arm, defined as ∆ i,t ( s, a ) := E [ R i,t | S i,t = s, A i,t = a ] -E [ R i,t | S i,t = s, A i,t = 0] .

We denote the observation history up to time t for user i as H i,t := ( S i 1 , A i 1 , R i 1 , . . . , S i,t ) . Actions are selected according to stochastic policies, π i,t : H i,t ×S → P ([ q ]) . We use π i,t ( a | s ) to denote the probability of action a ∈ [ q ] given context s ∈ S for a fixed (implicit) history. As in Greenewald et al. (2017), we assume that the probability of the control action is bounded:

Assumption 1. There exists 0 < π min , π max < 1 such that π min < π i,t (0 | a ) < π max for all i, t, a .

As noted in Greenewald et al. (2017), this assignment probability controls the number of messages that participants receive. It ensures that participants receive sufficient messages that they do not disengage, but not so many that they are overwhelmed or fatigued. We now define

the optimal arm (including control) and the optimal non-baseline arm, respectively. Respecting the restrictions in Assumption 1, we can now express the optimal policy, π ⋆ i,t , in two cases:

̸

These cases represent the appropriate boundary condition given the optimal action. We maximize the probability of the control action if A ⋆ i,t = 0 and minimize it if A ⋆ i,t = 0 , selecting the next-best action with the highest possible probability; we set the assignment probabilities for the remaining arms to zero. In Section 5.2, we define regret relative to this optimal policy, π ⋆ i,t .

Lastly, we consider a study design that progresses in stages , where users enter the study sequentially (Friedman et al., 2010), as illustrated in Figure 1. Staged recruitment serves two purposes in our analysis: it increases the fidelity of the theoretical setting by mimicking real-world recruitment strategies, and it allows us to estimate changes in the differential reward over time. Concretely, we first observe user i = 1 at time t = 1 . Then we observe users i = (1 , 2) at times t = (2 , 1) . Subsequent stages k involve users i ≤ k , each having had k -i +1 observations, respectively. Define O k = { ( i, t ) : i ≤ k, t ≤ k +1 -i } as the set of all observations in stage k . Appendix F provides a notation summary.

3.2 Doubly Robust Differential Reward

Following Greenewald et al. (2017), we assume that the differential reward is linear: ∆ i,t ( s, a ) = x ( s, a ) ⊤ θ i,t , where x ( s, a ) ∈ R d is a feature vector depending on the context and action. We allow the baseline reward, g t ( s ) , to be a nonlinear and nonstationary function of context and time. Combined, these two assumptions imply a partially linear model for the conditional mean reward:

Stage

t=1t=2t=3

.·.

t=k

t=T

Recruited

Not yet recruited

i=2

i=3

Subject

i=k

方

方

i=N

where δ a> 0 is an indicator of a non-baseline action. By incorporating both the linear predictor x ( s, a ) ⊤ θ i,t δ a> 0 and the nonlinear baseline g t ( s ) , we can maintain interpretability of the action effects while also allowing flexibility in specifying the observed rewards. The individual rewards are then realized according to the model

where we make the following standard assumption (Abeille & Lazaric, 2017) regarding ϵ i,t :

Assumption 2. The error ϵ i,t is conditionally mean zero (i.e., E [ ϵ i,t |H i,t ] = 0 ) and sub-Gaussian.

Assumption 2 ensures that large errors are sufficiently rare and that ˜ R t is proportional to an unbiased estimate of ∆ i,t ( s, a ) . However, ˜ R t suffers from high variance, which leads to suboptimal performance in practice. Our proposed algorithm, RoME, relies on an improved estimator, ˜ R f i,t ( s, ¯ a ) , that is also unbiased but achieves lower variance by denoising the rewards according to a working model, f i,t ( s, a ) , for the conditional mean r i,t ( s, a ) := E [ R i,t | S i,t = s, A i,t = a ] :

where ¯ a is a preselected non-baseline arm; i.e., we condition on A i,t ∈ { 0 , ¯ a } . The first component is a model-based prediction of the differential reward, and the second component is a debiasing term that uses IPW to correct for potential errors in the specification of f . This improved estimator is a form of pseudo-outcome, a common technique in the causal inference literature (Bang & Robins, 2005; Nie & Wager, 2021; Kennedy, 2023). We refer to it as a Doubly Robust Differential Reward because it produces an unbiased estimate of ∆ i,t ( s, a ) as long as either π i,t or f i,t is correctly specified (see proof in Appendix E.1); though, in practice, the benefit stems from (a) the robustness to misspecification of f (because π i,t is known) and (b) the variance reduction introduced by denoising the outcomes (see Remark 1 in Appendix E.2). In the following, we abbreviate ˜ R f i,t ( s, a ) as ˜ R f i,t , with the context and action implied.

4 Algorithm Components

We now construct ˜ R f i,t , introduce the weighted least-squares loss function used by RoME to estimate parameters, and use nearest-neighbor regularization to pool information across users.

4.1 Debiased Machine Learning

To construct ˜ R f i,t , we fit a working model f i,t for the conditional mean reward. Because ˜ R f i,t is robust to misspecification of f i,t , our regret bound in Theorem 1 does not require f i,t to be correctly specified. However, we show in Appendix E.2 that the following assumption results in a lower asymptotic variance bound for ˜ R f i,t .

Assumption 3. The working model, f i,t , satisfies | r i,t ( s, a ) -f i,t ( s, a ) | a.s. → 0 for all s , a .

Assumption 3 states that f i,t is a strongly consistent estimator of r i,t , but it does not impose any specific rate requirement on the convergence. Although our regret bound does not require Assumption 3, it does require two standard conditions within the DML framework: (a) Neyman orthogonality and (b) independence of f i,t and the observed rewards. The construction of ˜ R f i,t ensures that (a) holds, making ˜ R f i,t robust to misspecification of f . We can enforce (b) via sample-splitting techniques. We present two such options below.

Option 1: The first option exploits the fact that outcomes across different users are independent. Consequently, we can perform sample splitting as follows. Step 1 : Randomly assign each user i to one of J folds. Let W j denote the set containing user indexes for the j -th fold and let W ∁ j denote its complement. Step 2 : For each fold, use a supervised learning algorithm to estimate the working model for r i,t ( s, a ) denoted ˆ f ( j ) i,t ( s, a ) using W ∁ j . Step 3 : Construct the pseudo-outcomes using (1).

Option 2: Because we do not expect an adversarial environment in mHealth, we may be willing to assume that the errors, ϵ i,t , are independent across time. In this case, we can adapt the procedure given above by randomly assigning data to folds at the level of ( i, t ) . In this case, W j , contains the ( i, t ) pairs assigned to fold j . This option is more powerful than option 1 in that it enables us to learn heterogeneity across users in the nuisance function, f i,t , leading to greater variance reduction.

Lastly, we note two strategies that can reduce the computational demands of sample splitting. The first is to employ estimators that satisfy a leave-one-out stability condition, such as bagged estimators that use subsampling (Chen et al., 2022); these estimators eliminate the need to perform sample splitting. The second is to fit f i,t in an online fashion.

4.2 Estimation via Weighted Least Squares

Having formed ˜ R f i,t ( s, ¯ a ) , we now explain how to estimate the parameters, θ i,t ∈ R d , determining the differential reward, ∆ i,t ( s, a ) . We assume that θ i,t consists of three components:

The parameter θ shared is a fixed effect shared across all i, t . In contrast, θ user i and θ time t are random effects specific to a given user and time point, respectively. We place these parameters in a single column vector, θ , as vec ( θ shared , θ user 1 , . . . , θ user K , θ time 1 , . . . , θ time K ) , where K is the total number of stages. We then form a corresponding feature vector, ϕ i,t = ϕ ( x i,t ) ∈ R (2 K +1) d , where ϕ inserts x i,t into the positions corresponding to θ shared , θ user i , and θ time t with zeros in all other locations; i.e., ϕ ( x i,t ) = C ⊤ i,t x i,t , where

so that C i,t ∈ R d × (2 K +1) d . Wecould then estimate θ i,t via a weighted least squares (WLS) regression that minimizes the following objective function:

where ˜ σ 2 i,t = π i,t (0 | S i,t ) · { 1 -π i,t (0 | S i,t ) } . However, ℓ WLS ( θ ) is over-parameterized and does not take advantage of available network information. The next section modifies ℓ WLS ( θ ) by adding regularization, producing the final loss function used in RoME.

4.3 Nearest-Neighbor Regularization

We assume access to network information relating neighboring users and time points. These networks are defined by graphs G user = ( V user , E user ) and G time = ( V time , E time ) , where V denotes vertices and E , edges. In practice, these networks can be formed via known clusters (e.g., shared schools, companies, geographic locations) or similar covariates. In the case of G time, a reasonable choice is to construct a graph of neighboring sequential time points; i.e., t = 1 ↔ t = 2 ↔ t = 3 . . . . Because we expect neighboring users and time points to have similar parameters, we may be able to improve our estimates of θ by regularizing neighboring values of θ user i and θ time t toward each other. This can be accomplished by using an L 2 Laplacian penalty. We illustrate the idea using G user; the application to G time is similar.

First, we form the incidence matrix, Q , with the entry Q v,e corresponding to the v -th vertex (user) and the e -th edge. Denote the vertices of this edge as v i and v j with i > j . Q v,e is then equal to 1 if v = v i , -1 if v = v j , and 0 otherwise. The Laplacian matrix is then defined as L = QQ ⊤ ∈ R K × K . Similar to Yang et al. (2020), we form a network cohesion penalty across users as follows:

where Θ user := ( θ user 1 , . . . , θ user K ) ⊤ ∈ R K × d and L user is the Laplacian matrix for users. The penalty is small when θ user i and θ user j are close for connected users. We employ a shared regularization hyperparameter λ , for G user and G time, and include a standard L 2 penalty on θ with hyperparameter γ . The full penalization matrix V 0 ∈ R (2 K +1) d × (2 K +1) d is

where L user ⊗ := L user ⊗ I d , the Kronecker product of L user with I d ( L time ⊗ is defined similarly). We then adapt ℓ WLS ( θ ) to include a corresponding penalty term:

In Section 5, we show that (3) leads to a Thompson sampling algorithm for action selection.

5 Thompson Sampling Algorithm

We now describe the Thompson sampling procedure and provide a high-probability regret bound.

5.1 Algorithm Description

The minimizer of the weighted regularized least squares loss in (3) is ˆ θ = V -1 b , where

In analogy to Bayesian linear regression, the penalized and weighted Gram matrix, V , plays the role of a (scaled) posterior precision matrix under a multivariate Gaussian prior for θ . This motivates an algorithm in which we randomly perturb the point estimate, ˆ θ . Specifically, we define V -1 i,t :=

C i,t V -1 C ⊤ i,t and set ˜ θ i,t = C i,t ˆ θ + β i,t ( δ ) V -1 / 2 i,t η , where η is a mean-zero random variable and

In the above expression, Λ 0 i,t := C i,t V -1 V 0 V -1 C ⊤ i,t and K is the total number of stages. The value v is the sub-Gaussian factor for ˜ R f i,t , and ζ is defined in Remark 2 in Appendix E; both can be set to sufficiently large constants. The hyperparameter δ determines the probability with which the regret bound holds. The random deviate, η , is drawn from a distribution D TS satisfying the

technical conditions in Definition 1 in Appendix D. In practice, we recommend setting D TS to a multivariate Gaussian distribution or t-distribution to simplify calculations. Given ˜ θ i,t , we then compute ¯ A i,t = argmax a ∈ [ q ] \{ 0 } x ⊤ i,t ˜ θ i,t and randomly select A i,t = 0 with probability

or A i,t = ¯ A i,t with probability 1 -π i,t (0 | ¯ A i,t ) . The action selection is summarized in Algorithm 1.

Algorithm 1 Action selection

With probability π i,t (0 | A i,t ) , play action 0 ; otherwise, play action A i,t

Having detailed the action selection algorithm, we now summarize RoME in Algorithm 2. The algorithm includes an outer loop that iterates through the stages from k = 1 to k = K . Within each stage, we handle users sequentially, iteratively selecting actions according to Algorithm 1, observing rewards, and updating V and b accordingly.

Algorithm 2 RoME

5.2 Regret bound

We first define regret, Regret K , in terms of the following average across stages:

The inner sum calculates regret (the difference between the expected reward of the optimal action and the chosen action) weighted by the probability of the action, following Greenewald et al. (2017), for all users at stage k . We then average these values to account for the growing number of pairs ( i, t ) as the stages progress. The cumulative regret is the sum of these averaged values over all stages.

The rewards associated with the baseline action are excluded since ∆ i,t ( S i,t , 0) = 0 . Regret K is a form of pseudo-regret because it eliminates the randomness due to { ϵ i,t } K t =1 (Audibert et al., 2003). The regret bound requires additional technical assumptions:

Assumption 5. For all s and a , ∥ x ( s, a ) ∥ ≤ 1 .

Assumption 6. There exists a known value M ∈ R + such that | g t ( s ) | , | f i,t | ≤ M for all i , t , and s .

Assumption 7. Let e K := # E user K +# E time K denote the total number of edges. Then e K = O ( K ω ) for some ω ∈ [0 , ∞ ) .

Assumption 8. Let θ ⋆ k represent the vector of true parameter values at stage k = i + t -1 and V -1 i,t , the matrix V prior to decision point ( i, t ) . Then sup ( i,t ) ∈O K ∥ V 0 θ ⋆ k ∥ V -1 i,t = O p { log 3 / 4 ( K ) } .

Assumption 9. The matrix V i,t is bounded below as follows: V i,t ≻ α min( i, t ) I d for some α > 0 that may depend on d .

Assumption 5 is a standard assumption that simplifies the proof (Abeille & Lazaric, 2017). Assumption 6 ensures that the (estimated) baseline rewards are bounded. Assumption 7 ensures that the number of edges grows no faster than a polynomial rate. Assumption 8 limits the contribution of the random effects to the regret bound. Lastly, Assumption 9 ensures sufficient exploration of the context space. See the proof of Theorem 1 in Appendix E.3 for additional discusion of Assumptions 8 and 9. We now state the regret bound.

Theorem 1. Under Assumptions 1-2, 4-9, with probability at least 1 -δ , Regret is of order

The exact bound (including constants) and proof are given in Appendix E. Following Greenewald et al. (2017), we decompose the regret into two terms. The first term depends on a summation of terms of the form { π ⋆ i,t ( ¯ A ⋆ i,t | S i,t ) -π i,t ( ¯ A i,t | S i,t ) } x ( S i,t , ¯ A i,t ) ⊤ θ ⋆ i,t . We employ a novel technique based on Markov's inequality to bound this term (see Lemmas 12 and 13 in Appendix E).

We bound the second term using the high-level proof technique of Abeille & Lazaric (2017). The primary difficulty in bounding this term is that the classical theory for the estimation error of RLS estimates (Theorem 2 in Abbasi-Yadkori et al. (2011)) is not directly applicable to our setting due to the increasing dimensionality. Lemma 6 in Appendix E generalizes this theory to our setting. This Lemma produces a bound involving ∥ C i,t V -1 i,t V 0 θ ⋆ k ∥ V , which we subsequently bound i,t

by ∥ V 0 θ ⋆ k ∥ V -1 i,t in Lemma 7. This section of the proof requires Assumption 8 to ensure that the contribution of the new parameters to this bound is small enough that we incur no more than a logarithmic penalty.

√

The overall bound scales like ˜ O ( Kd 3 / 2 ) up to log factors, matching the standard rate given in Agrawal & Goyal (2012) and Abeille & Lazaric (2017) for linear Thompson sampling with fixed dimension. In particular, the bound depends solely on the dimensionality of the differential rewardnot the baseline reward, which may be much more complex.

6 Experiments

This section presents results from applying RoME in a simulated mHealth study with staggered recruitment and an off-policy comparison study for cardiac rehabilitation mHealth interventions. The simulations were implemented using Python and the results were generated using individual compute nodes with two 3.0 GHz Intel Xeon Gold 6154 processors and 180 GB of RAM. Case studies were implemented using R 4.2.2 and results were generated on a cluster composed of individual compute nodes with 2.10 GHz Intel Xeon Gold 6230 processors and 192 GB of RAM.

6.1 Competitor Comparison Simulation

In this section, we compare RoME to four competing methods in simulation. We implemented RoME using Option 2 with a bagged ensemble of stochastic gradient trees (Gouk et al., 2019; Mastelini et al., 2021) trained online via the River library (Montiel et al., 2021). We programmed the linear algebra operations using SuiteSparse (Davis & Hu, 2011) to take advantage of the sparse nature of the Laplacian matrices and ϕ i,t from Algorithm 2.

Our competing baselines are (a) Standard: Standard Thompson sampling for linear contextual bandits, (b) AC: the A ctionC entered contextual bandit algorithm (Greenewald et al., 2017), (c) IntelPooling: The intelligentpooling method of Tomkins et al. (2021) fixing the variance parameters close to their true values, and (d) Neural-Linear: a method that uses a pre-trained neural network to transform the feature space for the baseline reward (similar to the Neural Linear method of Riquelme et al. (2018)).

We compare these methods under three settings: Homogeneous Users, Heterogeneous Users, and Nonlinear. The first two involve a linear baseline model and time-homogeneous parameters, with the second setting having distinct user parameters. The third setting, designed to mirror the challenges of mHealth studies, includes a nonlinear baseline and both user- and time-specific parameters. For each setting, we simulate 200 stages following the staged recruitment regime depicted in Figure 1, and we repeat the full 200-stage simulation 50 times. Appendix A.1 provides details on the setup and a link to our implementation.

(a)HomogeneousUsers

(b)Heterogeneous Users

(c)NonlinearSetting

140

RoME

gret

Regr

120

100

IntelPooling

AC

Neural-Linear

gret

120

100

egret

Reg

140

120

100

ative

80

Standard

ative

80

Cumulative

80

Cumulat

60

60

60

40

Cumul

40

40

20

20

20

0

0

0

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

Stage

Stage

Stage

Figure 2 displays the average cumulative regret for each method. RoME and IntelPooling have similar performance in the Homogeneous Users setting, but RoME excels in the Heterogeneous Users and Nonlinear settings, with significantly lower cumulative regret by stage 200. This is expected as RoME can utilize network information and model nonlinear baselines in these settings. However, in the Homogeneous Users setting, where no network information is available and a linear baseline is used, RoME does not outperform IntelPooling.

Appendices A.2-A.5 offer more results, including additional method comparisons, a rectangular data array simulation, hyperparameter sensitivity analyses, and pairwise statistical comparisons. The latter show that RoME outperformed each competitor in at least 48 of 50 repetitions in the Nonlinear setting, indicating even higher statistical confidence than Figure 2 suggests. RoME substantially outperforms the competing algorithms in the additional comparisons and is several orders of magnitude faster than would be required in a standard mHealth study, producing over 20,000 decisions in as little as 95 seconds (see Table 2 in Appendix A.2).

6.2 Valentine Study Analysis Results

In this section, we compare RoME to the above algorithms via off-policy evaluation using data from the Valentine Study (Jeganathan et al., 2022), a prospective, randomized-controlled, remotely administered trial designed to evaluate an mHealth intervention to supplement cardiac rehabilitation for low- and moderate-risk patients. In the analyzed dataset, participants were randomized to receive or not receive contextually tailored notifications promoting low-level physical activity and exercise throughout the day. The left panel of Figure 3 shows the estimated improvement in the average reward over the original constant randomization, averaged over stages (K = 120) and participants (N=108). We see that RoME achieved the highest average reward; its average performance (across bootstrap replications) is higher than the 75th percentile of all other methods.

To further analyze the improvement of RoME relative to the other methods, we test whether the proposed algorithm significantly improves cumulative rewards using paired t-tests with one-sided alternative hypotheses. The null hypothesis ( H 0 ) for these tests is that the two algorithms being compared achieve the same average reward. The alternative hypothesis ( H 1 ) is that the algorithm listed in the column achieves higher average rewards than the algorithm listed in the row. The right of Figure 3 displays the p-values obtained from these pairwise t-tests. The dark shade of the last column indicates that the proposed RoME algorithm achieves significantly higher rewards than the other five competing algorithms ( p ≤ 0 . 01 for all pairwise comparisons). The results imply that RoME would have achieved a 3 . 5% increase in step count relative to the constant randomization policy that was

RelativeExpectedRewards

RoME

Unbiased Estimate of

0.06

0.04

Neural-Linear-

pvalue

0.02

IntelPooling

p≤0.01

p>0.05

0.00

AC

-0.02

Standard

-0.04

Standard

AC

IntelPoolingNeural-LinearRoME

AC

RoME

Standard

IntelPooling

Neural-Linear

actually implemented in this study. This effect translates to an increase of 140 steps per day, given a baseline of 1,000 steps per hour and four one-hour measurement windows. See Appendix B for additional implementation details.

We performed an additional off-policy comparison using data from the Intern Health Study (IHS) (NeCamp et al., 2020), as shown in Figure 11 in Appendix C.2. The results further demonstrate the competitive performance of the RoME algorithm, with the AC algorithm also showing comparable performance in this dataset. Further details on the analysis can be found in Appendix C.

7 Discussion

This paper introduces RoME, a robust mixed-effects contextual bandit algorithm for mHealth studies. RoME adapts DML and network cohesion penalties to dynamic settings, enabling researchers to efficiently pool information across users and over time. In addition to the methodological contribution, we also prove a high-probability regret bound in an asymptotic regime that involves a growing pool of users and time points. Our implementation of RoME is several orders of magnitude faster than would be required in practice and could easily be adapted for use in future mHealth studies.

We see several promising directions for improving RoME in future work. One such direction might consider improvements in the algorithm and theoretical analysis that enable us to replace Assumptions 8 and 9 with weaker and more primitive assumptions. In particular, these assumptions may not be plausible for q > 1 because most arm selections will be restricted to the same two arms (for large K ). Other interesting directions include data-adaptive strategies for hyperparameter selection and addressing long-term impacts of mHealth treatments, such as accumulating treatment fatigue. Future work could also consider computational improvements that would enable large-scale deployment of RoME, which would be especially relevant in nonclinical settings.

References

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems , 24, 2011.

Abeille, M. and Lazaric, A. Linear Thompson sampling revisited. Electronic Journal of Statistics , 11(2):5165 - 5197, 2017. doi: 10.1214/17-EJS1341SI. URL https://doi.org/10.1214/ 17-EJS1341SI .

Agrawal, S. and Goyal, N. Analysis of Thompson sampling for the multi-armed bandit problem. In Conference on Learning Theory , pp. 39-1. JMLR Workshop and Conference Proceedings, 2012.

Aouali, I., Kveton, B., and Katariya, S. Mixed-effect Thompson sampling. In International Conference on Artificial Intelligence and Statistics , pp. 2087-2115. PMLR, 2023.

Audibert, J.-Y., Biermann, A., and Long, P. Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica , 37(4):263-296, 2003.

Bang, H. and Robins, J. M. Doubly robust estimation in missing data and causal inference models. Biometrics , 61(4):962-973, 2005.

Cesa-Bianchi, N., Gentile, C., and Zappella, G. A gang of bandits. Advances in Neural Information Processing Systems , 26, 2013.

Chen, Q., Syrgkanis, V., and Austern, M. Debiased machine learning without sample-splitting for stable estimators. Advances in Neural Information Processing Systems , 35:3096-3109, 2022.

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal , 21(1):C1-C68, 01 2018. ISSN 1368-4221. doi: 10.1111/ectj.12097. URL https: //doi.org/10.1111/ectj.12097 .

Cho, H., Wang, P., and Qu, A. Personalize treatment for longitudinal data using unspecified randomeffects model. Statistica Sinica , pp. 187-206, 2017.

Davis, T. A. and Hu, Y. The University of Florida sparse matrix collection. ACM Transactions on Mathematical Software (TOMS) , 38(1):1-25, 2011.

Friedman, L. M., Furberg, C. D., and DeMetsR., D. L. (eds.). Machine Learning: An Artificial Intelligence Approach, Vol. I . Springer, New York, NY, 4 edition, 2010.

Golbus, J. R., Gupta, K., Stevens, R., Jeganathan, V. S. E., Luff, E., Shi, J., Dempsey, W., Boyden, T., Mukherjee, B., Kohnstamm, S., et al. A randomized trial of a mobile health intervention to augment cardiac rehabilitation. NPJ Digital Medicine , 6(1):173, 2023.

Gouk, H., Pfahringer, B., and Frank, E. Stochastic gradient trees. In Asian Conference on Machine Learning , pp. 1094-1109. PMLR, 2019.

Greenewald, K., Tewari, A., Klasnja, P., and Murphy, S. Action centered contextual bandits. Advances in Neural Information Processing Systems , 30, 2017.

Hu, X., Qian, M., Cheng, B., and Cheung, Y. K. Personalized policy learning using longitudinal mobile health data. Journal of the American Statistical Association , 116(533):410-420, 2021.

Jeganathan, V. S., Golbus, J. R., Gupta, K., Luff, E., Dempsey, W., Boyden, T., Rubenfire, M., Mukherjee, B., Klasnja, P., Kheterpal, S., et al. Virtual application-supported environment to increase exercise (VALENTINE) during cardiac rehabilitation study: Rationale and design. American Heart Journal , 248:53-62, 2022.

Kennedy, E. H. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics , 17(2):3008-3049, 2023.

Kim, G.-S. and Paik, M. C. Contextual multi-armed bandit algorithm for semiparametric reward model. In International Conference on Machine Learning , pp. 3389-3397. PMLR, 2019.

Kim, W., Kim, G.-S., and Paik, M. C. Doubly robust Thompson sampling with linear payoffs. Advances in Neural Information Processing Systems , 34:15830-15840, 2021.

Kim, W., Lee, K., and Paik, M. C. Double doubly robust Thompson sampling for generalized linear contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pp. 8300-8307, 2023.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Krishnamurthy, A., Wu, Z. S., and Syrgkanis, V. Semiparametric contextual bandits. In International Conference on Machine Learning , pp. 2776-2785. PMLR, 2018.

Lee, K., Paik, M. C., Oh, M.-h., and Kim, G.-S. Mixed-effects contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pp. 13409-13417, 2024.

Li, L., Chu, W., Langford, J., and Schapire, R. A contextual-bandit approach to personalized news article recommendation. WWW , pp. 661-670, 2010.

Ma, H., Zhou, D., Liu, C., Lyu, M. R., and King, I. Recommender systems with social regularization. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining , pp. 287-296, 2011.

Mastelini, S. M., de Leon Ferreira, A. C. P., et al. Using dynamical quantization to perform split attempts in online tree regressors. Pattern Recognition Letters , 145:37-42, 2021.

McPherson, M., Smith-Lovin, L., and Cook, J. M. Birds of a feather: Homophily in social networks. Annual Review of Sociology , 27(1):415-444, 2001.

Montiel, J., Halford, M., Mastelini, S. M., Bolmier, G., Sourty, R., Vaysse, R., Zouitine, A., Gomes, H. M., Read, J., Abdessalem, T., et al. River: Machine learning for streaming data in Python. 2021.

Nair, V. and Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning , pp. 807-814, 2010.

NeCamp, T., Sen, S., Frank, E., Walton, M. A., Ionides, E. L., Fang, Y., Tewari, A., and Wu, Z. Assessing real-time moderation for developing adaptive mobile health interventions for medical interns: Micro-randomized trial. Journal of Medical Internet Research , 22(3):e15033, 2020.

Nie, X. and Wager, S. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika , 108 (2):299-319, 2021.

Qian, T., Klasnja, P., and Murphy, S. A. Linear mixed models with endogenous covariates: Modeling sequential treatment effects with application to a mobile health study. Statistical Science , 35(3): 375, 2020.

Riquelme, C., Tucker, G., and Snoek, J. Deep Bayesian bandits showdown: An empirical comparison of Bayesian deep networks for Thompson sampling. arXiv preprint arXiv:1802.09127 , 2018.

Robinson, P. M. Root-n-consistent semiparametric regression. Econometrica: Journal of the Econometric Society , pp. 931-954, 1988.

Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., and Adams, R. Scalable Bayesian optimization using deep neural networks. In International Conference on Machine Learning , pp. 2171-2180. PMLR, 2015.

Tomkins, S., Liao, P., Klasnja, P., and Murphy, S. Intelligentpooling: Practical Thompson sampling for mhealth. Machine Learning , 110(9):2685-2727, 2021.

Vaswani, S., Schmidt, M., and Lakshmanan, L. Horde of bandits using Gaussian Markov random fields. In Artificial Intelligence and Statistics , pp. 690-699. PMLR, 2017.

Yang, K., Toni, L., and Dong, X. Laplacian-regularized graph bandits: Algorithms and theoretical analysis. In International Conference on Artificial Intelligence and Statistics , pp. 3133-3143. PMLR, 2020.

A Additional Details for Simulation Study

This appendix provides details regarding the setup of our simulation study, additional results not presented in the main paper, and results from additional simulations employing different competitors and/or simulation designs. We summarize the differences between the competing algorithms in Table 1.

A.1 Setup Details

The code for the simulation study is fully containerized and publicly available at https://github. com/eastonhuch/RoME . The simulation study involves a generative model of the following form for user i at time t :

Here S i,t = ( s 1 , s 2 ) ∈ R 2 is a context vector, with both dimensions iid ∼ U ( -1 , 1) . We set x ( s, a ) = a (1 , s 1 , s 2 ) . For simplicity, we set g to a time-homogeneous function. The specific nature of the function varies across the following three settings mentioned in Section 6.1:

Homogeneous Users: Standard contextual bandit assumptions with a linear baseline and no user- or time-specific parameters. The linear baseline is g ( S i,t ) = 2 -2 s 1 +3 s 2 , and the causal parameter is θ it = (1 , 0 . 5 , -4) such that the optimal action varies across the context space.

Heterogeneous Users: Same as the above but each user's causal parameter has iid N (0 , 1) noise added to it.

Nonlinear: The general setting discussed in the paper with a nonlinear baseline, user-specific parameters, and time-specific parameters. The base causal parameter and user-specific parameters are the same as in the previous two settings. The nonlinear baseline and timespecific parameter are shown in Figure 4.

We assume that the data are observed via a staged recruitment scheme, as illustrated in Figure 1. For computational convenience, we update parameters and select actions in batches. If, for instance, we observe twenty users at a given stage, we update our estimates of the relevant causal parameters and select actions for all twenty users simultaneously. This strategy offers a slight computational advantage with limited implications in terms of statistical performance.

For simplicity, we assume that the nearest-neighbor network is known and set the relevant hyperparameters accordingly. We took care to set the hyperparameters such that RoME performs a similar

amount of shrinkage compared to other methods, especially IntelPooling, which effectively uses a separate penalty matrix for users and time. To accomplish this, we used the same penalty matrices for RoME and IntelPooling, effectively generalizing the scalar γ penalty discussed in the main paper. We use 5 neighbors within the DML methods and set the other hyperparameters as follows: λ = 1 , δ = 0 . 01 , v = 1 , and ζ = 10 .

For the Neural-Linear method, we generate a 200 × 200 array of baseline rewards to train the neural network prior to running the bandit algorithm. Consequently, the results shown in the paper for Neural-Linear are better than would be observed in practice because we allowed the Neural-Linear method to leverage data that we did not make available to the other methods. This setup offers the computational benefit of not needing to update the neural network within bandit replications, which substantially reduces the necessary computation time.

Aside from the input features used, the Neural-Linear method has the same implementation as the Standard method. The Neural-Linear method uses the output from the last hidden layer of a neural network to model the baseline reward. However, we use the original features (the state vectors) to model the advantage function because the true advantage function is, in fact, linear in these features.

Our neural networks consisted of four hidden layers with 10, 20, 20, and 10 nodes, respectively. The first two employ the ReLU activation function (Nair & Hinton, 2010) while the latter two employ the hyperbolic tangent. We chose to use the hyperbolic tangent for the last two layers because Snoek et al. (2015) found that smooth activation functions such as the hyperbolic tangent were advantageous in their neural bandit algorithm. The loss function was the mean squared error between the neural networks' output and the baseline reward on a simulated data set. We trained our networks using the Adam optimizer (Kingma & Ba, 2014) with batch sizes of 200 for between 20 and 50 epochs. We simulated a separate validation data set to check that our model had converged and was generating accurate predictions.

Figure 5 compares the true baseline reward function in the nonlinear setting (left) to that estimated by the neural network (right). We see that the neural network produced an accurate approximation of the baseline reward, which helps explain the good performance of the Neural-Linear method relative to other baseline approaches, such as Standard.

We include the Neural-Linear method primarily to demonstrate that correctly modeling the baseline is not sufficient to ensure good performance. In the mHealth, algorithms should also be able to (a) efficiently pool data across users and time and (b) leverage network information. The Neural-Linear method satisfies neither of these criteria. Note that a neural network could be used to model the baseline rewards as part of our algorithm. Future work could consider allowing the differential rewards themselves to also be complex nonlinear functions, which could be accomplished by combining our method with Neural-Linear. We leave the details to future work.

1.0

8

b

0.8

20

0.6

0.4

6

(27

0.2

5

2

mWm

0.0

4

-0.2

3

-0.4

-0.6

2

-0.8

1

-1.0

0

-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

0

25

50

75

100

125

150

175

200

S1

t

A.2 Additional Method Comparisons

Our results in the main paper indicate that RoME outperforms existing methods, especially in the complex longitudinal setting that we are targeting. These positive results beg the question: What aspects of RoME contribute most to its superior performance?

In this section, we describe an additional simulation study designed to answer this question. In it, we compare RoME (as implemented in the main paper) to three additional comparison methods: RoME-BLM, RoME-SU, and NNR-Linear. Each comparison isolates particular aspects of our algorithm to understand its contribution to RoME's performance.

RoMEBLM is a version of RoME that employs B agged L inear M odels as the baseline supervised learning algorithm. Unlike the method implemented in the main paper (which uses bagged stochastic gradient trees), this baseline model is misspecified in the nonlinear setting; i.e., it cannot accurately model the nonlinear baseline reward function. Consequently, this comparison will allow us to gauge the impact of the supervised learning method used in our method. In particular, it should allow us to validate whether the algorithm is robust to baseline misspecification as the theory suggests.

RoMESU is a S ingleU ser version of our algorithm. Rather than having a separate θ i for each user, this method estimates a single advantage function that is shared across all users. It does, however, employ DML, time effects, and nearest-neighbor regularization (for the time effects). Consequently, this comparison will help us understand the impact of the user-specific parameters in our simulation.

NNR-Linear is similar to the existing network-cohesion bandit algorithms described in the main paper. It includes a distinct θ i for each user and regularizes them toward each other using a Laplacian penalty. It does not, however, use DML or time effects. As a result, this comparison isolates the impact of these two factors and should provide evidence that RoME is more broadly applicable than existing network-cohesion bandit algorithms.

Figure 6 displays the cumulative regret as a function of the stage for these methods across the three settings used in the main paper. We see that all methods perform similarly in panel (a), the setting with homogeneous users. However, in panel (b) we see that the regret achieved by RoME-SU begins to resemble a straight upward-sloping line because it cannot appropriately model the user-specific advantage functions.

In panel (c), we observe a much wider spread of performance, with RoME and RoME-BLM performing similarly and substantially outperforming the other methods. The fact that RoME and RoME-BLM perform similarly indicates that our algorithm is, in fact, robust to misspecification as expected. The comparison to NNR-Linear reveals that the DML and time effects-the new components of our method compared to existing network-cohesion bandit algorithms-do contribute meaningfully to RoME's superior performance. We found in additional simulations studies not reported here that the gap between NNR-Linear and our method widens as the magnitude of the time effects increases.

Table 2 displays the average computation time for each method across all three settings. Each run of the simulation produces 200 × 201 / 2 = 20 , 100 decisions. RoME and RoME-SU require the longest computation time at about seven minutes. RoME-BLM requires only 95-110 seconds, indicating

1.0

1.0

8

8

0.8

-

0.8

7

0.6

0.6

0.4

6

0.4

6

0.2

5

0.2

5

S

0.0

4

S

0.0

4

-0.2

-0.2

3

3

-0.4

-0.4

2

2

-0.6

-0.6

-0.8

1

-0.8

1

-1.0

0

-1.0

0

-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

S1

S1

(a)Homogeneous Users

(b)Heterogeneous Users

(c)NonlinearSetting

RoME

40

60

Regret

20

RoME-BLM

RoME-SU

Regret

Cumulative Regret

50

30

15

NNR-Linear

40

Cumulative

Cumulative

10

20

30

20

5

10

10

0

0

0

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

Stage

Stage

Stage

that the computational bottleneck for RoME is the ML model fitting. The required computation time for IntelPooling is about half that of RoME-BLM at 42-47 seconds. NNR-Linear requires 30-32 seconds, and the remaining methods run in less than one second. Because most clinical mHealth studies require fewer than 1,000 decisions per day, the speed of RoME is orders of magnitude faster than is required in practice. However, large-scale commercial deployment of RoME may require computational adjustments and approximations.

A.3 Pairwise Comparisons

Table 3 shows pairwise comparisons between methods across the three settings. The individual cells indicate the percentage of repetitions (out of 50) in which the method listed in the row outperformed the method listed in the column. The asterisks indicate p-values below 0.05 from paired two-sided ttests on the differences in final regret. The Avg column indicates the average pairwise win percentage.

RoME and RoME-BLM perform well across all three settings and the difference between them is statistically indistinguishable. These methods perform about equally well compared to IntelPooling and NNR-Linear in the first setting, but they perform better in the other two. In the Nonlinear setting in particular, RoME and RoME-BLM substantially outperform all other methods, and the pairwise differences are statistically significant at the 5% level.

A.4 Simulation with Rectangular Data Array

The main simulation involves simulating data from a triangular data array. At the 200-th (final) stage, the algorithm has observed 200 rewards for user 1, 199 rewards for user 2, and so on.

Homogeneous Users

Heterogeneous Users

Nonlinear

In this section, we simulate actions and rewards under a rectangular array with 100 users and 100 time points. Although we still follow the staged recruitment regime depicted in Figure 1, at stage 100 we stop sampling actions and rewards for user 1; at stage 101 we stop sampling for user 2; and so on until we have sampled 100 time points for all 100 users. Aside from the shape of the data array, the setup is the same for this simulation as for the main simulation.

(a)HomogeneousUsers

(b)Heterogeneous Users

(c)NonlinearSetting

60

RoME

60

60

IntelPooling

Regret

50

Neural-Linear

Regret

50

Regret

50

AC

Cumulative

40

Standard

Cumulative

40

Cumulative

40

30

30

30

20

20

20

10

10

10

0

0

0

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Time

Time

Time

(a)Homogeneous Users

(b)Heterogeneous Users

(c)NonlinearSetting

5

RoME

14

25

Regret

4

RoME-BLM

Regret

12

Regret

RoME-SU

20

10

3

NNR-Linear

Cumulative

Cumulative

8

Cumulative

15

2

6

10

1

4

5

0

2

-1

0

0

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Time

Time

Time

The cumulative regret for these methods as a function of time-not stage-is shown in Figure 7. Qualitatively, the results are nearly identical to those from the main simulation (compare to Figure 2). The differences in final regret are even larger in this simulation than they were in the main simulation, presumably because this study exhibits greater variation in time effects due to the balance across time.

Figure 8 displays comparisons similar to those shown in Figure 6 but for the rectangular simulation. We again see that the results are qualitatively similar to the main simulation. The methods perform similarly in the Homogeneous Users setting. RoME-SU performs poorly in the Heterogeneous Users setting, but the other methods perform similarly. Finally, RoME and RoME-BLM substantially outperform the other methods in the nonlinear setting. Again, the differences in performance are even larger in this simulation than in the main simulation.

Table 4 displays results from pairwise method comparisons, similar to those shown in Table 3. Again, the results are qualitatively similar to but slightly more exaggerated than those from the main simulation. In particular, in this simulation study, RoME and RoME-BLM outperform all other methods in 100% of repetitions in the Nonlinear setting.

Homogeneous Users

Heterogeneous Users

Nonlinear

A.5 Hyperparameter Sensitivity

In the simulation study, we chose hyperparameters for the fairest comparison possible, employing the same regularization parameters across methods. To assess robustness, we performed seven sensitivity analyses that alter hyperparameters for our methods while fixing those of the other methods; this approach gives the competing algorithms an advantage. Under these alternative hyperparameters, RoME and RoME-BLM still substantially outperform the other methods in the Nonlinear Setting.

Two of the sensitivity analyses led to no significant changes to Figure 2 and Table 3: setting δ = 0 . 05 and v = 10 . Below, we summarize the changes in the remaining analyses:

Rescaling γ by a factor of ten: This change resulted in RoME and RoME-BLM performing relatively better compared to the other methods in the Heterogeneous setting. In particular, RoME and RoME-BLM outperformed NNR-Linear in 70% and 72% of replications, compared to 56% and 52% using the original value of γ .

Rescaling λ by a factor of ten: Similar to the above results, this changed improved the performance of RoME and RoME-BLM compared to the other methods (especially RoMESU and NNR-Linear) in the Heterogeneous Users setting.

Adding (low and medium) noise to the nearest-neighbor network: RoME significantly outperformed RoME-BLM in the Nonlinear setting.

Increasing the number of neighbors to 10: The performance gap between the full RoME algorithms (RoME and RoME-BLM) and RoME-SU increased to 88%, 76% compared to 80% and 72% under the original configuration, presumably due to increased network cohesion.

The results for the λ sensitivity analysis are displayed in Table 5. Results for the remaining sensitivity analyses are available with our code.

B Additional Details for Valentine Study

Personalizing treatment delivery in mobile health is a common application for online learning algorithms. We focus here on the Valentine study, a prospective, randomized-controlled, remotely administered trial designed to evaluate an mHealth intervention to supplement cardiac rehabilitation for low- and moderate-risk patients (Jeganathan et al., 2022; Golbus et al., 2023). We aim to use smartwatch data (Apple Watch and Fitbit) obtained from the Valentine study to learn the optimal timing of notification delivery given the users' current context.

B.1 Data from the Valentine Study

Prior to the start of the trial, baseline data was collected from each of the participants (e.g., age, gender, baseline activity level, and health information). During the study, participants are randomized to either receive a notification ( A t = 1 ) or not ( A t = 0 ) at each of 4 daily time points (morning, lunchtime, mid-afternoon, evening), with probability 0.25. Contextual information was collected frequently (e.g., number of messages sent in prior week, step count variability in prior week, and pre-decision point step-counts).

Since the goal of the Valentine study is to increase participants' activity levels, we define the reward, R t , as the step count for the 60 minutes following a decision point (log-transformed to eliminate skew). Our application also uses a subset of the baseline and contextual data; this subset contains the variables with the strongest association with the reward. Table 6 shows the features available to the bandit in the Valentine study data set.

For baseline variables, we use the participant's device model ( Z 1 , Fitbit coded as 1), the participant's step count variability in the prior week ( Z 2 ), and a measure of the participant's pre-trial activity level based on an intake survey ( Z 3 , with larger values corresponding to higher activity levels).

At every decision point, before selecting an action, the learner sees two state variables: the participant's previous 30-minute step count ( S 1 , log-transformed) and the participant's phase of cardiac rehabilitation ( S 2 , dummy coded). The cardiac rehabilitation phase is defined based on a participant's

Homogeneous Users

Heterogeneous Users

Nonlinear

time in the study: month 1 represents Phase I, month 2-4 represents Phase II, and month 5-6 represents Phase III.

B.2 Justification of Assumption 4

The motivation for Assumption 4 arises from an exploratory analysis we performed using data from the Valentine study. We constructed the pseudo-reward for each observation as suggested by Equation (1) and then conducted an ANOVA test. The results show clear heterogeneity between users and across time, motivating the adoption of user- and time-specific random effects. The results of the ANOVA test are displayed in Table 7. Figure 9 shows the shape of the heterogeneity in the treatment effects over time.

0.1

Pseudo Outcomes

0.0

-0.1

-0.2

10

20

Weekin Study

B.3 Evaluation

The Valentine study collected the sensor-based features at 4 decision points per day for each study participant. The reward for each message was defined to be log (0 . 5 + x ) , where x is the step count of the participant in the 60 minutes following the notification. As noted in the introduction, the baseline reward (the expected step count of a subject when no message is sent), not only depends on the state in a complex way but is likely dependent on a large number of time-varying observed variables. Both of these characteristics (complex, time-varying baseline reward function) suggest using our proposed approach.

We generated 100 bootstrap samples and ran our contextual bandit on them, considering the binary action of whether or not to send a message at a given decision point based on the contextual variables S 1 and S 2 . Each user is considered independently and with a cohesion network, for maximum personalization and independence of results. To guarantee that messages have a positive probability of being sent, we only sample the observations with notification randomization probability between 0 . 01 and 0 . 99 . In the case of the algorithm utilizing NNR, we chose four baseline characteristics (gender, age, device, and baseline average daily steps) to establish a measure of 'distance' between users. For this analysis, the value of k representing the number of nearest neighbors was set to 5. To utilize bootstrap sampling, we train the Neural-Linear method's neural network using out-of-bag samples. The neural network architecture comprises a single hidden layer with two hidden nodes.

The input contains both the baseline characteristics and the contextual variables and the activation function applied here is the softplus function, defined as softplus ( x ) = log(1 + exp ( x )) .

We performed an offline evaluation of the contextual bandit algorithms using an inverse propensity score (IPS) version of the method from Li et al. (2010), where the sequence of states, actions, and rewards in the data are used to form a near-unbiased estimate of the average expected reward achieved by each algorithm, averaging over all users.

RelativeExpected Rewards

0.075

Unbiased Estimate of

RoME

0.050

pvalue

0.025

RoME-SU

p≤0.01

p>0.05

0.000

NNR-Linear

NNR-Linear

RoME-SU

RoME

RoME-SU

RoME

NNR-Linear

B.4 An ablation study

We further investigate the primary contributors to the algorithm's performance, we conducted a parallel ablation study using real-world data analysis. In this study, we compared RoME (as detailed in the main paper) with two additional methods for reference: RoME-SU and NNR-Linear. Each comparison focuses on specific elements of our algorithm to discern their impact on RoME's overall performance.

The differences between these methods are illustrated in Figure. 10. In this particular dataset, NNR plays a more significant role in driving the overall superior performance of our algorithm.

B.5 Inverse Propensity Score (IPS) offline evaluation

In the implemented Valentine study, the treatment was randomized with a constant probability p t = 0 . 25 at each time t . To conduct off-policy evaluation using our proposed algorithm and the competing variations of the TS algorithm, we outline the IPS estimator for an unbiased estimate of the per-trial expected reward based on what has been studied in Li et al. (2010).

Given the logged data D = { s t = s t , A t = a t , R t = r t } T t =1 collected under the policy p = { p t } T t =1 , and the treatment policy being evaluated π = { π t } T t =1 , the objective of this offline estimator is to reweight the observed reward sequence { R t } T t =1 to assign varying importance to actions based on the propensities of both the original and new policies in selecting them.

Lemma 1 (Unbiasedness of the IPS estimator) . Assuming the positivity assumption in logging, which states that for any given s and a , if p t ( a | s ) > 0 , then we also have π t ( a | s ) > 0 , we can obtain an unbiased per-trial expected reward using the following IPS estimator:

As mentioned in the previous section, we restrict our sampling to observations with notification randomization probabilities ranging from 0.01 to 0.99. This selection criterion ensures the satisfaction of the positivity assumption. The proof essentially follows from definition, we have:

Proof.

To address the instability issue caused by reweighting in some cases, we use a Self-Normalized Inverse Propensity Score (SNIPS) estimator. This estimator scales the results by the empirical mean of the importance weights, and still maintains the property of unbiasedness.

C Additional Details for the Intern Health Study (IHS)

To further enhance the competitive performance of our proposed RoME algorithm, we performed an additional comparative analysis using a real-world data set from the Intern Health Study (IHS) (NeCamp et al., 2020). This micro-randomized trial investigated the use of mHealth interventions aimed at improving the behavior and mental health of individuals in stressful work environments. The estimates obtained represent the improvement in average reward relative to the original constant randomization, averaging across stages (K = 30) and participants (N = 1553). The available IHS data consist of 20 multiple-imputed data sets. We apply the algorithms to each imputed data set and perform a comparative analysis of the competing algorithms. The results presented in Figure 11 shows our proposed RoME algorithm achieved significantly higher rewards than the other three competing ones and demonstrated performance comparable to the AC algorithm. These findings further support the advantages of our proposed algorithm.

C.1 Data from the IHS

Prior to the start of the trial, baseline data was collected on each of the participants (e.g., institution, specialty, gender, baseline activity level, and health information). During the study, participants are randomized to either receive a notification ( A t = 1 ) or not ( A t = 0 ) every day, with probability 3 / 8 . Contextual information was collected frequently (e.g., step count in prior five days, and current day in study).

We define the reward, R t , as the step count on the following day (cubic root). Our application also uses a subset of the baseline and contextual data; this subset contains the variables with the strongest association to the reward. Table 8 shows the features available to the bandit in the IHS data set.

At every decision point, before selecting an action, the learner sees two state variables: the participant's previous 5-day average daily step count ( S 1 , cubic root) and the participant's day in the study ( S 2 , an integer from 1 to 30 ).

C.2 Evaluation

We run our contextual bandit on the IHS data, considering the binary action of whether or not to send a message at a given decision point based on the contextual variables S 1 and S 2 . Each user is considered independently and with a cohesion network, for maximum personalization and

Table 8: List of features available to the bandit in the IHS. The features available to model the action interaction (effect of sending a mobile prompt) and to model the baseline (reward under no action) are denoted via a ' √ ' in the corresponding column, otherwise × .

independence of results. To guarantee that messages have a positive probability of being sent, we only sample the observations with notification randomization probability between 0 . 01 and 0 . 99 . For the algorithm employing NNR, we defined participants in the same institution as their own 'neighbors'. This definition enables the flexibility for the value of k , representing the number of nearest neighbors, to vary for each participant based on their specific institutional context. Furthermore, in our study setting, we assume that individuals from the same institution enter the study simultaneously as a group. Due to the limited access to prior data, we are unable to build the neural linear models as in the Valentine Study.

We utilized 20 multiple-imputed data sets and performed an offline evaluation of the contextual bandit algorithms on each data set. The result is presented below in Figure 11. Similar to Section B.4, here we also compared RoME (as detailed in the main paper) with RoME-SU and NNR-Linear. The differences between these methods are illustrated in Figure. 12. In this specific dataset, both NNR and DML exhibit comparable performance individually, but their combined effect significantly enhances the overall performance of the algorithm.

D Additional Details for Algorithm 1

This appendix briefly discusses two details of Algorithm 1. The first is efficient computation of V -1 . Because V is a large matrix, a full matrix inversion is expensive. Fortunately, however, we can dramatically reduce the necessary computational requirements because each additional ( i, t ) involves a rank-one perturbation to V . Consequently, we can apply the Sherman-Morrison formula to speed up computations. We leveraged this trick in our implementation of RoME for the simulation study by using efficient rank-one updates available in the SuiteSparse library (Davis & Hu, 2011).

The second detail is the requirements for the distribution, D TS :

Definition 1. D TS is a multivariate distribution on R d absolutely continuous with respect to Lebesgue measure which satisfies: 1. (anti-concentration) that there exists a strictly positive probability p such that for any u ∈ R d with ∥ u ∥ = 1 , P ( u ⊤ η ≥ 1) ≥ p ; 2. (concentration) there exists c , c ′ positive constants such that ∀ δ ∈ (0 , 1) , P ( ∥ η ∥ ≤ √ cd log( c ′ d/δ )) ≥ 1 -δ ; and 3. it possesses a finite second moment, E ∥ η ∥ 2 , where η ∼ D TS .

Relative Expected Rewards

0.04

Unbiased Estimate of

RoME

IntelPooling

0.02

pvalue

p≤0.01

AC

p>0.05

0.00

Standard

Standard

AC

IntelPooling

RoME

AC

Standard

RoME

IntelPooling

Relative Expected Rewards

0.04

Unbiased Estimate of

RoME

0.02

pvalue

RoME-SUI-

p≤0.01

p>0.05

0.00

NNR-Linear'

-0.02

NNR-Linear

RoME-SU

RoME

RoME-SU

RoME

NNR-Linear

While a Gaussian prior satisfies Definition 1, this approach allows us to move beyond Bayesian posteriors to generic randomized policies. As discussed in the main paper, we recommend setting D TS to a multivariate Gaussian distribution or multivariate t-distribution. These choices simplify computations because the probability computation in (5) simplifies to an evaluation of the CDF of either (a) a univariate Gaussian or (b) a univariate t-distribution. The mean and variance of the corresponding univariate distribution can easily be worked out using the moments of a linear combination of random variables.

E Regret Bound

E.1 Double Robustness of Pseudo-Reward

Going forward, we use the notation ∆ f i,t ( s, ¯ a ) := f i,t ( s, ¯ a ) -f i,t ( s, 0) to denote the prediction of the differential reward. In the computations in this section, we implicitly condition on H i,t and A i,t ∈ { 0 , ¯ a } .

Lemma 2. If either p = π or f = r

That is, the pseudo-reward is an unbiased estimator of the true differential reward.

Proof. Recall that

Case I: π 's are correctly specified

Then and

so that

Case II: f correctly specified

and

and

E.2 Preliminaries

Lemma 3. Let X be a mean-zero sub-Gaussian random variable with variance proxy v 2 and Y be a bounded random variable such that | Y | ≤ B for some 0 ≤ B < ∞ . Then XY is sub-Gaussian with variance proxy v 2 B 2 .

Proof. Recall that X being mean-zero sub-Gaussian means that

Now note that as desired.

Lemma 4. If X,Y are sub-Gaussian with variance proxies v 2 x , v 2 y , respectively, then αX + βY is sub-Gaussian with variance proxy α 2 v 2 x + β 2 v 2 y ∀ α, β ∈ R .

Proof. Recall the equivalent definition of sub-Gaussianity that X,Y are sub-Gaussian iff for some a, b > 0 and all λ > 0

Then

The following Lemma gives the sub-Gaussianity and variance of the difference between the pseudoreward and its expectation. We see that in the variance, all terms except those involving the inverse propensity weighted noise variance vanish as f i,t becomes a better estimate of r i,t .

Lemma 5. The difference between the pseudo-reward and its expectation (taken wrt the action and noise) is mean zero sub-Gaussian with variance

Proof. We need to show that it is sub-Gaussian and upper bound its variance. We write the difference as

Note that | r i,t ( s, A i,t ) | ≤ max( | r i,t ( s, ¯ a ) | , | r i,t ( s, 0) | ) ≤ M and | f i,t ( s, A i,t ) | ≤ max( | f i,t ( s, ¯ a ) | , | f i,t ( s, 0) | ) ≤ M . Thus, since ∣ ∣ 1 / { δ A i,t =¯ a -π i,t (0 | s ) } ∣ ∣ is upper bounded (because π i,t (0 | s ) ∈ [ π min , π max ] ), we have that r i,t ( s,A i,t ) -f i,t ( s,A i,t ) δ Ai,t =¯ a -π i,t (0 | s ) is bounded and thus (not necessarily mean zero) sub-Gaussian. Since ϵ i,t is sub-Gaussian, its denominator is bounded, and the

so that if | XY | > t , then | X | B > t . Thus by monotonicity

remaining terms are deterministic, the entire difference between the pseudo-reward and its mean is sub-Gaussian. Now

since E [ ˜ R f i,t | s, ¯ a ] is not random. Now we expand the first term on the rhs.

For the first term on the rhs of (8),

so that plugging this into (8), we have

and plugging this into (7) we obtain the variance.

Note that

and plugging this into (9),

as desired. Now note that

so that

Corollary 1. Let ˜ π := min( π min , 1 -π max ) and σ 2 := Var ( ϵ i,t ) . Then

Proof. Substitute σ 2 := Var ( ϵ i,t ) in Lemma 5 and apply the bounds M , π min , and π max to obtain

We now show that Assumption 3 results in a lower asymptotic variance bound.

Corollary 2. The asymptotic variance of ˜ R f i,t ( s, ¯ a ) is no greater than

Proof. By Assumption 3, the terms involving r i,t ( s, ¯ a ) -f i,t ( s, ¯ a ) and r i,t ( s, 0) -f i,t ( s, 0) converge almost surely to zero. Dropping these terms from the expression in Lemma 5, the asymptotic variance is then less than or equal to

Remark 1. Comparing v 2 1 and v 2 2 , we see that v 2 1 ≥ v 2 2 if

which must hold by the definition of ˜ π . Consequently, The effect of the DML is to lower the variance bound.

E.3 Derivation of Confidence Sets

To facilitate the developments below, we define the following quantities:

V 0 ,k , V i,t , b i,t , ˆ θ i,t : The quantities V 0 , V , b , ˆ θ := V -1 b in Algorithms 1 and 2 at decision point ( i, t ) .

O i,t : The set of user-time pairs up to but excluding ( i, t ) ; we can express O i,t as { (1 , 1) , (1 , 2) , (2 , 1) , . . . , ( i, t ) }\{ ( i, t ) }

θ ⋆ i,t := θ shared + θ user i + θ time t = C i,t θ ⋆ k ∈ R d .

ˇ θ i,t := C i,t ˆ θ i,t ∈ R d .

Λ 0 i,t := C i,t V -1 i,t V 0 ,k V -1 i,t C ⊤ i,t ∈ R ( d × d ) .

Λ + i,t := C i,t V -1 i,t [ ∑ ( j,u ) ∈O i,t ˜ σ 2 j,u ϕ { x ( S ju , A ju ) } ϕ { x ( S ju , A ju ) } ⊤ ] V -1 i,t C ⊤ i,t ∈ R ( d × d ) .

V i,t := (Λ 0 i,t +Λ + i,t ) -1 = ( C i,t V -1 i,t C ⊤ i,t ) -1 ∈ R d × d .

In the proof below, we allow the dimension of θ ⋆ k to increase as stages progress so that θ ⋆ k , b i,t , ˆ θ i,t ∈ R (2 k +1) d and V 0 ,k , V i,t ∈ R (2 k +1) d × (2 k +1) d . However, the proof is also applicable (with minor modifications) to settings where the final dimension of θ ⋆ k is known and set in advance (as in Algorithm 2).

In analogy to Bayesian linear regression, V i,t plays the role of a posterior precision matrix for θ ⋆ i,t . Similarly, V -1 i,t plays the role of a posterior covariance matrix. Assumption 9 ensures that V -1 i,t is 'decreasing' sufficiently fast. Proving that this assumption holds under more primitive conditions is an interesting area for future work. We conjecture that the following condition is sufficient provided that γ is set to a sufficiently large constant.

Assumption 10. Let O be a set of ( i, t ) pairs containing at least K ⋆ ∈ N elements, and let ω > 0 . Then the following ordering holds:

where # O denotes the cardinality of O .

This assumption ensures that we continue acquiring information about all elements of x i,t at a linear rate as we progress through stages. We now adapt the classical theory on RLS estimation error to our setting.

Lemma 6 (Adapted from Theorem 2 in Abbasi-Yadkori et al. (2011)) . Let ˇ θ i,t be the regularized least squares (RLS) estimate from Algorithm 1 for the i -th user at the t -th time point, corresponding to stage k := i + t -1 and let θ ⋆ i,t be the true parameter value. For any δ > 0 , with probability at least 1 -δ the estimates { ˇ θ i,t } ( i,t ) ∈O K satisfy

where v 2 is the variance proxy for the difference between the pseudo-reward and its mean.

Proof. Let m i,t = ˜ σ i,t ϕ { x ( S i,t , A i,t ) } and ρ i,t = ˜ σ i,t { ˜ R f i,t ( S i,t , ¯ A i,t ) -E ( ˜ R f i,t | S i,t , ¯ A i,t ) } . Further

and thus

Premultiplying, we can then obtain

Simplification yields

We can then apply the Cauchy-Schwarz inequality to bound the right-hand side.

Dividing by ∥ ˇ θ i,t -θ ⋆ i,t ∥ V i,t and applying the triangle inequality gives

We can show that

Combined with the fact that ρ i,t is sub-Gaussian with variance proxy v 2 , the above result allows us to apply Theorem 1 of Abbasi-Yadkori et al. (2011). Specifically, to match their notation (setting aside our definition of S i,t for the moment), let S i,t = C i,t V -1 i,t ξ i,t = ∑ ( j,u ) ∈O i,t C i,t V -1 i,t ρ j,u m j,u and

Adding this bound to ∥ C i,t V -1 i,t V 0 ,k θ ⋆ k ∥ V i,t completes the proof.

In the above inequality, we will need to bound ∥ C i,t V -1 i,t V 0 ,k θ ⋆ k ∥ V i,t in order to obtain useful confidence sets. To do so, we first define a new matrix, C i,t .

Definition 1. Let C i,t ∈ R 3 d × (2 k +1) d be a 3 × (2 k +1) block matrix where all blocks are d × d zero matrices except the blocks at entries (1 , 1) , (2 , 1 + i ) , and (3 , 1 + k + t ) which are equal to I d .

Note that C i,t selects the components of θ ⋆ k that are relevant to ( i, t ) : θ shared , θ user i , and θ time t . We also have the relationship C i,t = (1 ⊤ 3 ⊗ I d ) C i,t , which we leverage below.

Lemma 7.

Proof. To simplify the proof, we reorder the rows and columns of V i,t such that the rows and columns corresponding to θ shared , θ user i , and θ time t appear first. The matrices C i,t and C i,t are similarly reordered, resulting in C i,t = [ I 3 d 0 3 d, 2 d ( k -1) ] , C i,t = [(1 ⊤ 3 ⊗ I d ) 0 p, 2 d ( k -1) ] . We now partition V i,t as follows:

Using the 2 × 2 block-matrix inversion identity, it then follows that

where P := Z 11 -Z 12 Z -1 22 Z 21 .

Consequently, we have C i,t V -1 i,t = P -1 [ I 3 d -Z 12 Z -1 22 ] and C i,t V -1 i,t C ⊤ i,t = P -1 . Letting G = [ I 3 d -Z 12 Z -1 22 ] , we can write the squared norm as

where we have used the fact that P -1 / 2 (1 3 ⊗ I d ) { (1 ⊤ 3 ⊗ I d ) P -1 (1 3 ⊗ I d ) } -1 (1 ⊤ 3 ⊗ I d ) P -1 / 2 ⪯ I d because the former is a projection matrix and thus its maximum eigenvalue is one. We can further simplify and bound this quantity as follows:

Remark 2. We can apply Lemma 7 and Assumption 8 to bound ∥ C i,t V -1 i,t V 0 ,k θ ⋆ k ∥ V i,t . Assumption 8 implies that there exists a function ζ ( δ ) : (0 , 1) → R such that

with probability at least 1 -δ . This function corresponds with the hyperparameter ζ given in the main paper. The purpose of the max operation is to avoid having a bound of zero for K = 1 .

We confirmed in simulation that Assumption 8 assumption is reasonable. The left panel of Figure 13 provides empirical evidence that ∥ V 0 ,k θ ⋆ k ∥ V -1 i,t / log 3 / 4 ( k ) is bounded in probability in a simulation with values of k ranging from 1 to 200. We ran 400 repetitions with an intercept-only model. Similarly, the right panel suggests that sup ( i,t ) ∈O K ∥ V 0 ,k θ ⋆ k ∥ V -1 i,t / log 3 / 4 ( K ) is bounded as assumed. We conjecture that the following assumption is a sufficient condition (in combination with the other assumptions listed in the main paper) for Assumption 8 to hold.

4.5

Mean

4.5

Mean

4.0

80%Confidence

3/4

4.0

80%Confidence

50l/

601

3.5

3.5

3.0

3.0

2.5

≥(2)

2.5

dns

2.0

2.0

0

50

100

150

200

0

50

100

150

200

K

K

Assumption 11. The random effects θ user i , θ time t for i, t = 1 , 2 , . . . are distributed such that

So, in other words, it would be sufficient for the normalized sums above to converge in distribution. This effectively requires these normalized sums to be approximately mean zero. This requirement is reasonable given the presence of θ shared in the model.

We now bound the first term from Lemma 6.

Lemma 8. Assuming γ ≥ 1 , we have:

Proof. By definition, we have:

Taking the determinant of both sides:

Using the property that the determinant of a matrix is less than or equal to the product of its eigenvalues, and letting λ j ( · ) denote the j -th largest eigenvalue, we have:

Now letting σ 1 be the largest singular value of a matrix,

Since C i,t = (1 ⊤ 3 ⊗ I d ) C i,t , we have that

so that σ 1 ( C i,t ) 2 = 3 . Therefore,

and thus

Lemma 9.

Proof.

Lemma 10. Assuming γ ≥ 1 ,

Proof.

Now we need to upper bound λ max ( V i,t ) . Note that

This then gives us

Note that e k produces no more than a logarithmic contribution to the above bound due to Assumption 7. We now define a term, β i,t ( δ 1 , δ 2 ) , that corresponds with the term β i,t ( δ ) from the main paper, setting δ = 2 δ 1 with sufficiently large ζ . However, here we make explicit that the term depends on two probabilities, δ 1 and δ 2 .

Definition 2. Let β i,t ( δ 1 , δ 2 ) be defined as follows:

Note that by Lemma 6 and Assumption 8 we have ∥ ˇ θ i,t -θ ⋆ i,t ∥ V i,t ≤ β i,t ( δ 1 , δ 2 ) with probability at least 1 -δ 1 -δ 2 .

Lemma 11. Let δ ′ 1 = δ 1 / { K ( K +1) } . Then

with probability at least 1 -δ 1 / 2 -δ 2 . Further, sup ( i,t ) ∈O K β i,t ( δ ′ 1 , δ 2 ) ≤ β K ( δ 1 , δ 2 ) , where β K ( δ 1 , δ 2 ) is defined as

Proof. The first statement follows from applying Lemma 6 with a union bound. Note that the second term in Lemma 6 can be bounded uniformly using Lemma 7 and Assumption 8 as follows:

with probability at least 1 -δ 2 . The second statement follows from bounding the first term in β i,t ( δ ′ 1 , δ 2 ) using Lemma 10:

Lastly, we define several events and bound the probabilities with which they occur.

Proposition 1. Let

δ ′ 1 = δ 1 / { K ( K +1) }

Proof. First, Lemma 11 shows that Pr( ˆ E k ) ≥ 1 -δ 1 / 2 -δ 2 .

Next, applying the TS sampling distribution and ˜ θ i,t = ˇ θ i,t + β k ( δ ′ 1 , δ 2 ) V -1 / 2 i,t η i,t where η i,t is drawn i.i.d. from D TS we have

by the concentration property in Definition 1. A union bound then gives Pr( ˜ E K ) ≥ 1 -δ 1 / 2 . Applying a union bound to ˜ E K and ˆ E K yields the conclusion.

E.4 Treatment Probability Estimation Errors

The proof relies on bounding terms of the form { π i,t ( ¯ A i,t | S i,t ) -π ⋆ i,t ( ¯ A ⋆ i,t | S i,t ) } x ( S i,t , ¯ A i,t ) ⊤ θ ⋆ i,t . To simplify the proof, we denote these terms as ( π i,t -π ⋆ i,t ) x ⊤ i,t θ ⋆ i,t . We begin by defining two filtrations.

Definition 2. We define the filtration F S i,t as the information accumulated up to decision point ( i, t ) including the sampled context, that is, F S i,t = ( F 1 , σ ( S 1 , 1 , ¯ A 1 , 1 , A 1 , 1 , R 1 , 1 S 2 , 1 , . . . , S i,t )) . Similarly, we define the filtration F i,t as the information accumulated up to decision point ( i, t ) before sampling S i,t . Thus, F i,t can be expressed in the same form as that of F S i,t above, removing the term S i,t .

We now leverage F S i,t to conditionally bound ( π i,t -π ⋆ i,t ) x ⊤ i,t θ ⋆ i,t .

Lemma 12. Conditional on F S i,t , the term ( π i,t -π ⋆ i,t ) x ⊤ i,t θ ⋆ i,t is nonnegative and bounded as

Proof. We begin by conditioning on F S i,t so that x ⊤ i,t θ ⋆ i,t is nonrandom. There are three cases.

In this simple case, we have ( π i,t -π ⋆ i,t ) x ⊤ i,t θ ⋆ i,t = 0 .

In this case, the optimal policy is to apply the control arm with probability π ⋆ i,t = π min because the treatment arm has a positive expected effect on the reward. Because π i,t ∈ [ π min , π max ] by construction, we have π i,t ≥ π min = π ⋆ i,t , which implies that π i,t -π ⋆ i,t ≥ 0 , proving the first statement in the proof for this case.

Now the probability π i,t is based on the following probability calculation:

where the last two lines follow from Markov's inequality and Assumption 9. Working directly with the expectation, we can show that

This result implies that

Provided M i,t ≤ π min , we have ( π i,t -π ⋆ i,t ) x ⊤ i,t θ ⋆ i,t = 0 because π i,t = π ⋆ i,t = π min . Alternatively, when M i,t > π min , we have

Thus, in both cases we have

Case 3:

This case is essentially identical to Case 2 , except both π i,t -π ⋆ i,t and x ⊤ i,t θ ⋆ i,t are negative. The resulting bound is

Conclusion:

Combining the three cases yields

Lemma 13. Under event E K , the following bound holds for all ( i, t ) ∈ O i,t :

Proof. Let ( i ⋆ , t ⋆ ) denote the final decision point at stage K and let δ ′ 1 = δ 1 / { K ( K +1) } . Conditional on F S i ⋆ ,t ⋆ , Lemma 12 implies that

for all ( i, t ) ∈ O K because F S i,t ⊆ F S i ⋆ ,t ⋆ . We can then apply Lemma 11 to argue that

under event E K for all ( i, t ) ∈ O i,t .

E.5 Proof of Theorem 1

We first decompose the regret

We can bound the (absolute value of) the first term using Lemma 13:

under event E K .

The proof for the second term follows closely from Abeille & Lazaric (2017) with several adjustments. Assumption 5 implies that we only need to consider the unit ball X = { x ∈ R d : ∥ x ∥ ≤ 1 } . Then the second term of the regret can be decomposed into

where x ⋆ i,t = x ( S i,t , ¯ A ⋆ i,t ) and x i,t = x ( S i,t , ¯ A i,t ) . The first term is the regret due to the random deviations caused by sampling ˜ θ i,t and whether it provides sufficient useful information about the true parameter θ ⋆ i,t . The second term is the concentration of the sampled term around the true linear model for the advantage function.

Bounding R RLS ( T ) . We decompose the second term into the variation of the point estimate and the variation of the random sample around the point estimate:

The first term describes the deviation of the TS linear predictor from the RLS one, while the second term describes the deviation of the RLS linear predictor from the true linear predictor. The first term is controlled by the construction of the sampling distribution D TS , while the second term is controlled by the RLS estimate being a minimizer of the regularized cumulative squared error in (3). In particular, the first term will be small when the TS estimate concentrates around the RLS one, while the second will be small when the RLS estimate concentrates around the true parameter vector. Under event E K , we can bound R RLS ( K ) by leveraging Lemma 6 and decomposing the error via

Under event E K , we have

where γ K ( δ 1 , δ 2 ) := β K ( δ 1 , δ 2 ) √ cd log { K ( K +1) c ′ d/δ 1 } and β K ( δ 1 , δ 2 ) is defined in Lemma 11. Then, from Assumption 9, we have

under event E K . Using a similar derivation for the β K ( δ 1 , δ 2 ) case, the following inequality holds under event E K :

Bounding R TS ( T ) . Leveraging Abeille & Lazaric (2017, pages 6-7), Definition 1 lets us bound R TS ( K ) under the event E K :

We can bound the expectation under Assumption 9 as

Using a derivation similar to that of R RLS ( K ) , it then follows that

under event E K .

Overall bound. Composing the three bounds yields the following upper bound:

with probability at least 1 -δ 1 -δ 2 . With δ = 2 δ 1 and sufficiently large ζ , the bound holds with probability at least 1 -δ as stated in Theorem 1. The rate in the main paper follows by noting that

F Notation Guide

For convenience, we summarize below some key notation used in the main paper.

a = 0 : Control action

q : Number of non-baseline treatment arms

i = 1 , 2 , . . . : Index for users

t = 1 , 2 , . . . : Index for time points

S i,t ∈ S : Context vector observed for user i at time point t

A i,t ∈ { 0 , . . . , q } : Action chosen for user i at time point t

R i,t ∈ R : Reward observed for user i at time point t

r i,t ( s, a ) := E [ R i,t | S i,t = s, A i,t = a ] : Conditional mean for the observed reward given the state and context

∆ i,t ( s, a ) := r i,t ( s, a ) -r i,t ( s, 0) : Linear differential reward for any action a > 0 and state s

H i,t : History up to time point t for user i

π i,t ( a | s ) : Probability of action a ∈ [ K ] given context s ∈ S for a fixed (implicit) history

π min , π max : Lower and upper limits for π i,t (0 | s ) , respectively

A ⋆ i,t ∈ [ K ] : The optimal arm

¯ A ⋆ i,t ∈ [ K ] \{ 0 } : The optimal non-baseline arm

¯ A i,t ∈ [ K ] \{ 0 } : Potential non-baseline arm that may be chosen if the baseline arm is not chosen

O k = { ( i, t ) : i ≤ k , t ≤ k +1 -i } : The set of observed time points across all users at stage k of the sequential requirement setting

x ( s, a ) ∈ R d : Feature vector depending on the state and action

θ i,t ∈ R d : Vector of parameters that may depend on the user i and time point t . Later θ i,t is written as θ i,t = θ shared + θ user i + θ time t , where θ user i is the user-specific but time-invariant term and θ time t is a shared time-specific term

g t ( s ) : Baseline reward function (conditional mean) that is observed when users are randomized to receive no treatment

δ a> 0 : Indicator function that takes the value 1 if a > 0 and 0 otherwise

ϵ i,t : Conditionally mean-zero, sub-Gaussian error term

f i,t ( s, a ) : Working model for the true conditional mean r i,t ( s, a )

˜ R f i,t ( s, ¯ a ) : Pseudo-reward given state S i,t = s and potential arm ¯ a , which has the same expectation as the differential reward, ∆ i,t ( s i,t , ¯ a i,t ) ; this term is written as ˜ R f i,t in some points in the main paper, with the state and action implied

W j : The j -th fold, of J total folds for sampling splitting

ϕ i,t : The enlarged feature vector with x i,t placed in positions corresponding to θ shared , θ user i , and θ time t : ϕ i,t = ϕ ( x i,t ) = C ⊤ i,t x i,t

˜ σ 2 i,t = π i,t (0 | s i,t ) · { 1 -π i,t (0 | s i,t ) } : Weights used in the penalized regression estimation, which are inversely proportional to Var ( ˜ R f i,t )

C i,t : Matrix defined as [1 , 0 ⊤ i -1 , 1 , 0 ⊤ K -i + t -1 , 1 , 0 ⊤ K -t ] ⊗ I d

G user = ( V user , E user ) , G time = ( V time , E time ) : Graphs for the nearest-neighbor networks across users and time, respectively

Q: Incidence matrix where the element Q v,e corresponds to the v -th vertex (user) and e -th edge

L user , L time: Laplacian matrices for users and time, respectively

λ : Hyperparameter for Laplacian penalization

γ : Hyperparameter for ridge penalization

β i,t ( δ ) : Term used to scale confidence sets; see (4)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract offers a brief summary of the paper's motivation and contribution, while the introduction delves deeper into background details and highlights the key components of our proposed algorithm.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the Discussion section (Section 7), we reflect on limitations related to computation, modeling assumptions, hyper-parameter tuning, and practical implementation.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We present all assumptions clearly, each numbered for reference, and we provide the full set of assumptions in Theorem 1. Additionally, the proof is available in the Appendix, and within the main text, we indicate where the corresponding proof can be found.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In the manuscript, we provided a clear description of the proposed algorithm in Algorithm 2. Additionally, we elaborated on the key components of the algorithm in Section 4. Furthermore, in Section 6.1, we outlined the simulation setup, with detailed information available in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The primary simulation results can be entirely reproduced using the provided code on Github. However, the Valentine study data, used for our real-world analysis, is not publicly accessible.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Algorithm 2 requires data splits, and we devoted two paragraphs on Page 5 to discussing options for this split. Additionally, we clearly listed all of the hyperparameters involved in forming objective functions and provided guidance on how to make suitable choices for them.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Section 6.2, the experiment results include hypothesis testing, and we have clearly indicated the p-values in the Figure legend to denote statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: At the start of Section 6, we offer a paragraph outlining the computer resources required to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The author has obtained permission to access both the Valentine study and the Intern Health Study dataset. It's important to note that the experiments conducted in this paper are purely numerical simulations and offline evaluation of existing data, which do not involve any human research subjects or participants.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper is driven by the current challenges in mobile health policy learning, aiming to enhance policy learning to maximize the benefits of mobile interventions. As a result, we emphasize the societal impacts, focusing on how the algorithm has the potential to improve public health.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the associated papers that produced the code package and dataset in Section 6.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The Appendix includes a link to our GitHub repository that contains the code to reproduce our simulation study.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a robust contextual bandit algorithm to optimize personalized mobile health interventions. The proposed algorithm leverages Thompson sampling, mixed-effect models, debiased machine learning, and nearest-neighbor regularization techniques to address the problems of user and time heterogeneity, information pooling, and complex baseline rewards. The paper establishes a high-probability regret bound and demonstrates the algorithm’s effectiveness compared to existing methods through a simulation and two off-policy evaluation studies.\n\nSTRENGTHS:\n-\tThe paper proposes a method to effectively address three common issues in mobile health: treatment effect heterogeneity across users and time, the need to pool information across users, and the possibility of complex baseline rewards.\n-\tThe simulation and real data study sufficiently demonstrate the advantages of the proposed method in the presence of heterogeneous users and nonlinear baseline rewards, and discuss its limitations when users are homogeneous.\n-\tThe paper is well-organized and easy to follow.\n\nWEAKNESSES:\nThe proposed method combines several existing techniques, including Thompson sampling, mixed-effect models, debiased machine learning, and nearest-neighbor regularization. Could the authors elaborate on the specific challenges associated with the proposed method, whether in terms of methodology, computation, or theoretical proof?\n\nQUESTIONS:\n-\tIt is mentioned in the appendix that the nearest neighbor network is assumed to be known in the simulation study, and that the number of nearest neighbors k is set to 5 in the Valentine Study. Could the authors provide details on how the nearest neighbor network is defined in the simulation study and how the number of nearest neighbors is chosen in the Valentine Study? Is the algorithm's performance sensitive to the construction of the nearest neighbor network?\n-\tThe paper lists several feasible choices for the working model f_{i,t}. However, its construction in the simulation and real data analysis was not detailed. It would be helpful if the authors could discuss how the working model is chosen in practice.\n-\tAssumption 1 ensures that the randomization probability for action 0 is always positive, but it does not specify this for other actions. Is a positivity assumption required for all actions in the proposed method? If so, the authors should discuss how this is ensured for other actions.\n-\tComputational efficiency is crucial for online algorithms. It would be beneficial if the authors could provide the computation time of the proposed method, e.g., in the simulation study or the Valentine Study. How does it compare with the baseline methods? \n-\tIn the single-column vector \\theta, user-specific parameters are numbered from 1 to K. Should they instead be numbered from 1 to N, as in the \\Theta_{user} matrix? Besides, it seems that N has not been defined in the main text.\n\nLIMITATIONS:\nThe authors have discussed the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"A Robust Mixed-Effects Bandit Algorithm for Assessing Mobile Health Interventions\" introduces the DML-TS-NNR (Debiased Machine Learning Thompson Sampling with Nearest-Neighbor Regularization) algorithm. This novel contextual bandit algorithm is designed to address challenges in mobile health (mHealth) interventions, such as participant heterogeneity, nonstationarity, and nonlinearity in rewards. The algorithm incorporates user- and time-specific incidental parameters, network cohesion penalties, and debiased machine learning to flexibly estimate baseline rewards.\n\nSTRENGTHS:\n### Originality:\n\nThe DML-TS-NNR algorithm introduces a novel approach by combining debiased machine learning, network cohesion penalties, and Thompson sampling to address the unique challenges in mHealth interventions. The use of user- and time-specific incidental parameters and the flexible estimation of baseline rewards via debiased machine learning are innovative contributions that enhance the adaptability and robustness of the algorithm.\n\n### Quality:\n\nThe methodology is rigorously developed, with comprehensive theoretical analysis and detailed proofs provided for the high-probability regret bounds. Extensive experimental validation includes simulations and real-world mHealth studies, demonstrating the algorithm's effectiveness and practical viability.\n\n### Clarity:\n\nThe paper is well-structured, with clear explanations of the problem statement, related work, methodology, experimental setup, and results. Figures and tables effectively illustrate the proposed method and its performance improvements. Technical terms and concepts are explained thoroughly, ensuring accessibility to a broad audience, including those not deeply familiar with contextual bandit algorithms or mHealth interventions.\n\n### Significance:\n\nBy addressing critical challenges in mHealth, such as participant heterogeneity and nonstationarity in rewards, the DML-TS-NNR algorithm has significant implications for improving personalized treatment strategies and patient outcomes.\n\nWEAKNESSES:\n### Novelty:\n\nWhile the combination of debiased machine learning, network cohesion penalties, and Thompson sampling is innovative, a more detailed comparison with existing methods, particularly in terms of theoretical and practical advantages, would further highlight the unique contributions of the proposed DML-TS-NNR algorithm.\n\n\n### Experimental Validation:\n\nThe experimental validation primarily relies on controlled datasets. Including more diverse real-world testing scenarios, such as various health conditions or treatment modalities, would provide a more comprehensive assessment of the DML-TS-NNR algorithm's effectiveness and generalizability.\n\n\n### Technical Details:\n\nSome aspects of the algorithm, such as the optimization process for debiased machine learning and the derivation of the network cohesion penalties, could be explained in greater detail to enhance clarity and understanding. The choice of evaluation metrics and their suitability for different types of mHealth interventions could be discussed more extensively.\n\nQUESTIONS:\n1. Can the authors elaborate on why specific baseline methods (e.g., Standard Thompson Sampling, Action-Centered contextual bandit) were chosen for comparison? What advantages do these techniques offer for evaluating the effectiveness of the DML-TS-NNR algorithm in mHealth interventions?\n\n2. How does the DML-TS-NNR algorithm perform in more diverse and dynamic real-world settings, such as different health conditions or treatment modalities? Are there plans to test the approach in more varied environments, including chronic diseases or mental health applications?\n\n3. What potential challenges could arise regarding the scalability of the DML-TS-NNR algorithm for very large datasets or real-time applications? How does the algorithm handle computational and communication overhead in such scenarios?\n\n4. Can the authors provide more details on the optimization process for debiased machine learning and its impact on overall performance and accuracy of reward estimation?\n\n5. What are the potential ethical considerations or privacy concerns associated with the proposed method, particularly in using sensitive health data for mHealth interventions? How does the DML-TS-NNR algorithm address these concerns?\n\nLIMITATIONS:\n1. Potential challenges and mitigation strategies for deploying the DML-TS-NNR algorithm in real-world settings, particularly regarding data diversity and environmental variability.\n2. Discussing the ethical implications and privacy concerns related to continuous monitoring and intervention in high-stakes healthcare applications. This includes addressing the implications of using sensitive patient data and the potential impact of intervention decisions on different demographic groups.\n3. Further exploring the trade-offs between computational efficiency and intervention accuracy, particularly in scenarios where rapid decision-making is crucial for clinical outcomes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The authors propose a novel contextual bandit algorithm that addresses individual heterogeneity, nonstationarity, and nonlinearity of the reward function. This algorithm involves three distinct steps to manage these challenges.

STRENGTHS:
The current paper is easy to follow and addresses a complicated scenario that has not been frequently discussed in the literature.

The methods are well organized and the theoretical discussions are helpful.

WEAKNESSES:
The selection of tuning parameters plays a crucial role but is not discussed in the proposed method. It is unclear whether the promising performance of the proposed algorithm is due to the choice of a specific tuning parameter. Providing some robustness checks would enhance the soundness of the simulation experiments.

I also need some motivation for Assumption 4. For example, how well does the dataset under investigation support this assumption? In its current form, Assumption 4 appears to be arbitrary.

I also found the assumption of a known network structure to be quite restrictive, as network structures are typically unknown in practice. This limitation hinders the practicality of the proposed algorithm.

QUESTIONS:
I found the comments on page 4 about the doubly robustness concerning. As the methods discussed in Section 4.1 are all nonparametric estimators (hence, the model misspecification is not an issue), the benefit of incorporating DML is to improve estimation efficiency.

LIMITATIONS:
not noted.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel robust mixed-effects bandit algorithm, named \"DML-TS-NNR\", designed to optimize mobile health (mHealth) interventions. mHealth aims to deliver personalized and contextually tailored notifications to promote healthier behaviors. The proposed algorithm addresses key challenges in mHealth, such as participant heterogeneity, nonstationarity, and nonlinearity in rewards. The main contributions of the paper are:\n- Modeling Differential Rewards: Incorporates user- and time-specific parameters.\n- Network Cohesion Penalties: Uses penalties to pool information across users and time.\n- Debiased Machine Learning: Employs this technique for flexible baseline reward estimation.\n\nThe algorithm's high-probability regret bound is solely dependent on the differential reward model's dimension. The effectiveness of DML-TS-NNR is demonstrated through simulations and two off-policy evaluation studies.\n\nSTRENGTHS:\n- The integration of user- and time-specific incidental parameters for modeling differential rewards.\n- The novel application of network cohesion penalties and debiased machine learning in the context of mHealth interventions.\n- The empirical validation through simulations and real-world mHealth studies demonstrates the practical applicability and effectiveness of the proposed algorithm.\n - The paper is well-structured and clearly explains the problem, the proposed solution, and the results.\n - The algorithm addresses significant challenges in mHealth, potentially improving the effectiveness of personalized health interventions. By achieving robust regret bounds and superior empirical performance, the algorithm can contribute to advancements in personalized healthcare technologies.\n\nWEAKNESSES:\n- The algorithm's reliance on complex calculations, such as log-determinants and matrix inverses, may limit its scalability for large datasets.\n- The assumption of a known network with binary edges may not always be practical. Real-world networks can be more complex, and this assumption might limit the algorithm's applicability.\n\nQUESTIONS:\n- How robust is the algorithm to violations of the assumptions regarding the network structure and hyperparameters? Can the authors provide guidance on tuning these parameters in practice?\n- How can the algorithm be extended to consider long-term effects and treatment fatigue? Are there plans to address these aspects in future work?\n\nLIMITATIONS:\nWhile the authors have made strides in addressing some key limitations, such as computational demand and network assumptions, there are still areas that may need further attention:\n- The high computational demand due to log-determinants and matrix inverses is a limitation. Future work should explore more efficient computational methods or approximations.\n- The algorithm focuses on immediate rewards and does not consider long-term effects or treatment fatigue. Addressing these aspects is crucial for real-world applicability and effectiveness.\n- The need for correctly specified hyperparameters can be challenging in practice. Providing more robust methods for hyperparameter tuning and addressing potential misspecifications would improve the algorithm's robustness and ease of use.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}