Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning
Authors: Hongseok Namkoong
Samuel Daulton
Eytan Bakshy
Venue: Accepted by TMLR ()

Abstract: Thompson sampling (TS) has emerged as a robust technique for contextual bandit problems. However, TS requires posterior inference and optimization for action generation, prohibiting its use in many online platforms where latency and ease of deployment are of concern. We operationalize TS by proposing a novel imitation-learning-based algorithm that distills a TS policy into an explicit policy representation, allowing fast decision-making and easy deployment in mobile and server-based environments. Using batched data collected under the imitation policy, our algorithm iteratively performs offline updates to the TS policy, and learns a new explicit policy representation to imitate it. Empirically, our imitation policy achieves performance comparable to batch TS while allowing more than an order of magnitude reduction in decision-time latency. Buoyed by low latency and simplicity of implementation, our algorithm has been successfully deployed in multiple video upload systems for Meta. Using a randomized controlled trial, we show our algorithm resulted in significant improvements in video quality and watch time.

Published in Transactions on Machine Learning Research (02/2026)
Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning
Hongseok Namkoong
camkoong@@gss.cllumbee
Decision, Risk, and Operations Division Columbia Business School
Samuel Daulton
sdaulton@@meta.cmmm com
Meta
Eytan Bakshy
sdauttnn@meta..oo
Meta
Reviewed OpenReview:
https: http::////ppeeeeee.eeeeteee
forum? id= J8PrMuo
Abstract
Thompson sampling (TS) has emerged robust technique for contextual bandit problems. However, TS requires posterior inference and optimization for action generation, prohibiting its use in many online platforms where latency and case of deployment are of concern. We operationalize TS by proposing a novel imitation-learning--ass TS policy into an explicit policy representation, allowing fast decision-mas -aaiinnnn and easy deployment in mobile and server-based environments. Using batched data collocted under the imitation policy, our algorithm iteratively performs offline updates to the TS policy, explicit policy representation to imitate it. Empirically,
algorithm that distills a
and learns policy achieves performance comparable to batch TS while allowing more than an order of magnitude reduction in decision-time latency. Buoyed by low latency and simplicity of implementation, our algorithm has been successfully deployed in multiple video upload systems for Meta. Using a randomized controlled trial, we show our algorithm resulted in significant improvements in video quality and watch time. We release our implementation as open-source software here
imitation
1 Introduction
In the past decade, Thompson sampling (Thompson 1933) has emerged powerful algorithm for contextual bandit problems. The underlying principle is simple: action is chosen with probability proportional to it being optimal under the current. posterior distribution. Driven by the algorithm's strong empirical
performance (Scott 2010 Chapelle & Li [011 May & Leslie 2011h, many authors have recently established
rigorous performance guarantees (Kaufmann et al.] 2012 Agrawal & Goyal 2013a b Gopalan et al.) 2014
Honda Takemura 2014 Russo & Van Roy] [0014 Abeille et al.] 2017). Thompson sampling increas-
ingly being applied a broad range of applications including revenue management Ferrcira et al. 2018);
internet advertising Graepel et al. 2010 Agarwal et al. 2014 Schwartz et al. 2017 and recommendation
systems (Kwalee al. 2015).
Despite its conceptual simplicity and strong performance, Thompson sampling can be difficult to deploy in practice. Thompson sampling consists of two sleps: posterior sampling and oplimization. Posterior sampling requires evaluating a potentially large number of actions from a well-calibrated probabilistic model. Accurately calibrating uncortainty important for optimally trading off exploration and exploitation, and is critical to practical performance (Riquelme et al. 2018). Large-scale probabilistic machine learning models based deep networks show much promise they can adaptively learn good feature representations for

---

Published in Transactions on Machine Learning Research (02/2026)
uncertainty calibration (Wang & Yeung 2020 However, sampling from these probabilistic models can be demanding in terms of computation and memory. While approximate inference methods with better runtime characteristics exist, they often produce poorly calibrated uncertainty estimates that lead to poorer empirical performance (Riquelme et al. 2018). The second step, optimization, solves for a reward-optimizing action under the posterior sample. This can also be prohibitively expensive when the action space large or continuous. For example, an advertising platform that matches advertisers to users at each time period has to solve combinatorial optimization problems real-time in order to run Thompson sampling (Mas-Colell et al,, 1995).
For typical online platforms, low latency-real-time computational performance-is critical for user satisfaction and retention. The online nature of the computation required for Thompson sampling thus poses a substantive challenge to deploying it in large-scale internet services. These challenges are especially pronounced in resource- constrained mobile applications, ubiquitous modality for modern internet applications: as of 2018, an estimated 52.2% of worldwide web traffic was generated by mobile devices (Statistica) 2020). Mobile applications require decisions to be made in fast and memory-eflicient manner, and on-device decision- making is important to good user experience in domains such as adaptive video streaming (Mao et al. 2019
and social media ranking (Petrescu & Tas 2016). However, the majority of internet-conneeted mobile devices have limited memory, and utilize low-end processors that are orders of magnitude slower than server-grade devices [Bhardwaj et al., 2019 Wu et al. 2019). As affordable, compute-limited mobile devices are increasingly adopted in developing economies (Ricciardi 2019), the ability to deploy cutting-edge decision algorithms on diverse computing infrastructure important for democratization of technology and long term business growth.
Software development cost is another core practical consideration when implementing contextual bandit algorithms in large-scale online platforms. Long-term software development cost is commonly referred to as tech debt, which is incurred when a suboptimal, myopic development plan is followed in lieu of one that requires (sometimes much) higher initial effort, but less future work. Avoiding tech debt is critical to a reliable and scalable service (Sculley el al.j 2015 Ramasubbu Kemerer 2016 Banker el al. 2021), but contextual bandit systems are challenging due to their high complexity: they require temporal feedback loops consisting of different pipelines exploration, data logging, policy updates, and deployment (Agarwal et al., 2016). The online nature of the complex numerical routines required by Thompson sampling significantly exacerbate these practical difficulties. Real-time posterior sampling and action optimization leads the overall system to be cumbersome and hard to debug, posing challenges to reliable software development.
Example 1 (Video Transcoding): As our main real-world application, we study video uploads for large online platforms. Video is an increasingly popular medium on social networks, but uploading video is still a technically challenging problem, where limited bandwidth and compute capacity particularly problematic on mobile devices-leads to unsuccessful uploads. When a user requests a vidco be uploaded to a social media service, the service must choose the desired video quality (bitrate) for transcoding the video before uploading. Video needs to be optimally transcoded considering quality, and success of file upload. It is preferable to upload videos at high quality because it can lead to better viewer experience (if the viewer has sufficiently good network connection). However, higher quality videos have larger file sizes, making it more likely to fail to upload: larger files take longer time to upload, increasing the likelihood that the network connection to fail, or the user to grow frustrated and cancel the upload.
We are interested in an online platform who wish to make contextual decisions about how to optimally transcode video at upload time. Making such decisions quickly is critical for user satisfaction; low latency is particularly important for popular short-form videos uploaded on Tik-Tok, Snapchat, and Instagram, where videos are captured and uploaded frequently and in real-time. Although transcoding decisions needs to be made quickly in order to be responsive and keep the user engaged, most upload requests come from resourco-constrainee mobile devices. o
2

---

Published in Transactions on Machine Learning Research (02/2026)
Online
Offline
Video Upload Request
Offline Posterior Updates
Features
Offline Posterior Sampling Numerical
Distilled TS Policy
Optimization
Action
Imitation Learning
Transcoder
Transcoded Video Upload
Success?
Figure 1: An illustration of distilled TS on the example video uploads application, described in Example | Online action generation is performed asynchronously on resourco-constraincd mobile edge devicos whereas batched policy updates are performed offline on powerful backend servers.
The problem motivation goes beyond our main application. There are numerous examples of decision-makinn problems on online platforms where latency and system complexity are of central concern.
Example 2 (Advertising on third party systems): Every time user arrives to third party webpage (e.g. New York Times), the advertising platform (e.g. Google Ads) decides which ad to show in order to maximize conversion. Latency is important to good user experience (Agarwal et al., 2016), and curbing system complexity increases service reliability (Sculley et al. 2015).
Example 3 (Ranking): When a
user logs in, am internet service chooses a list of items to display to the user order to maximize revenue or engagement. Concrete examples include ranking news articles (Microsoft Network, MSN), products (online marketplaces like Amazon and Airbub), and content (Facebook and Linkedln feed). In all of these cases, latency is central to user satisfaction, but mobile edge devices and front-end servers are resource constrained (Agarwal et al. 2016). For instance, there has been work from Meta Facebook performing secondary ranking on device to avoid server communication latency and to only display content that has
been downloaded completely (Petrescu & Tas 2016 ,
Example (Personalized Pricing): As a customer enters virtual platform, the system generates a personalized price based on market conditions and user-specific contexts. Electronic commerce firms and airlines use price controls manage revenue (Talluri & Van Ryzin 2004 Den Boer 2015), and two-sided online marketplaces (e.g. Uber, Lyft, Airbub) dynamically prices on both sides of the market to reduce supply-demand imbalance. In both cases, latency is important for satisfactory experience.
Methodology Motivated by aforementioned challenges in implementing and deploying Thompson sampling online platforms, develop and analyze method that maintains explicit policy representation designed to imitate Thompson sampling. In order to avoid computationally demanding routines online, our algorithm simulates and imitates a Thompson sampling policy offline. An explicit policy representation can efficiently generate actions real-time even in large action spaces, without requiring real-time posterior inference or numerical optimization. An illustration of how this methodology can be applied to video transcoding (Example ! is provided in Figure This allows leveraging state-of-the-ae Bayesian models as Gaussian processes parameterized by deep neural networks and optimization solvers offline, while maintaining low
3

---

Published in Transactions on Machine Learning Research (02/2026)
latency on resource-constrained computing modalities such as low-end mobile devices During operation, actions can be generated efficiently from the distilled policy by sampling from parameterized distribution, allowing fast and asynchronous interaction with users. For example, recent engineering progress allows gencrating actions using an industrial-scale neural network model in 0.3880 milliscoonds (Coleman et al. 2017
By performing posterior updates and mimicking the behavior of Thompson sampling offline, we are able to move complex numerical routines from ceostrce- mobile devices to backend servers, and reduce long-term software development costs (tech debt). Such offine procedures using batched observations can be casily implemented using modern industry machine learning pipelines (Gauci et al.) 2018 Fujimoto et al., 2019). This allows leveraging the recent remarkable progress in machine learning software infrastructurr, such as engineering best practices and tools for reliable testing & deployment"
Our approach shifts computation from online latency-critical offline batch processing environments, and does not produce a universally dominating intervention. This trade-off is most useful when offline computation can be carried out powerful backend servers for a relatively extended period of time, whereas online computation must execute instantaneously resource-constraineed devices with strict latency requirements.
Practical impact Empirically, we evaluate our imitation algorithm on several benchmark problems and a real-world dataset for selecting optimal vidco transcoding configurations (Section In all of our experiments, our imitation algorithm performs as well as batch on-policy Thompson sampling in terms of cumulative regret, while reducing decision-time latency by an order of magnitude, Buoyed by low latency and simplicity of implementation showcased in our empirical benchmarking efforts, our imitation learning policics have been used in video upload systems across Meta products, which are leading social networking services. Our contextual policy tunes the bitrates for video uploads based on contextual features such as download bandwidth, device model, operating system, connection class (2G, 3G, 4G), country, and video features which include source resolution, bitrate, and file size.
To assess the impact of our algorithm, we ran internal randomized controlled trials (RCT) on each of the aforementioned products. We find our algorithm achieves significant improvements in video quality, which we measure using the fraction of videos with quality preserved at 1080p (high resolution). Our RCTs show up to 5x improvements over existing video upload policies on all surfaces.
The RCTs show significant increases in topline metrics that are of importance at the company level. Due to better video quality, we observed increased video watch times on multiple products: 1.1% on Facebook iOS Feed videos, 0.77% on Facebook Android Feed videos, 0.27% on Facebook Android Stories, 0.45% on Instagram Stories In addition, our contextual policies boosted interaction metrics on several products: increases in meaningful social interactions of 0.15% and 0. 0.14% on Facebook Android Stories and Facebook Android Feed, respectively, and an increase in interactions of 0.26% on Instagram Stories. All findings were significant at the 95% level.
Buoyed by these results, our contextual policy has been deployed across multiple product verticals including Facebook Feed, Stories, Reels and Instagram Stories and Reels. Our algorithm has been independently applied to both iOS and Android apps for all aforementioned products, and is reliably handling millions of uploads each day.
Theoretical contributions To understand the strong practical advantages we showcase, we take initial steps toward principled understanding of our imitation algorithm. Since our (batch) updates to the Thompson sampling policy are based on observations generated by the imitation policy, our algorithm emulates an off-policy version of Thompson sampling which may diverge from its on-policy counterpart. Due to its off- off-policy nature, an uninformed and pessimistic view of our procedure states that any initially small
1 Mooe generally, optimization can be challenge for non-Bayesian methods. Although outside of the scope of this paper, generalizing imitation framework to other policies will likely yield fruit in separating optimization from online action- generation.
As an example, dedicated top peer-reviowed conference for ML systems htttp:///llyy:: was recently established, and is undergoing rapid growth the forefront of academia and industry. This community focuses on improving the efficiency of ML. systems from an operational perspective.
4

---

Published in Transactions on Machine Learning Research (02/2026)
deviation between the imitation policy and Thompson sampling may cascade across time. Our main theoretical results (Section 5 preclude such possibility and ensure small deviations between the imitation policy and Thompson sampling do not magnify over time. Specifically, we show that our imitation policy enjoys Bayes regret similar to that of batch on-policy Thompson sampling, up to the sum of single-step imitation errors. We substantiate our performance guarantees in general modeling scenarios involving contextual Gaussian processes, where a cleverly initialized version of our algorithm (albeit impractical) achieves advantagcous Bayes regret (Section 5.2
Solving the imitation problem, or equivalently, finding the policy parameterization closest to Thompson sampling, only requires unsupervised contexts- without corresponding actions or rewards. On large- scale online platforms, unsupervised contexts are typically cheap and abundant, e.g,, the entire user database provides a wealth of such contexts. In Section [ we prove that each sigle-period imitation error term can be controlled-with a sufficiently rich imitation model-at at the rate (,/V/VN), where N is the number of supervised and unsupervised contexts. Combining this with our aforementioned regret bound in Soction 5 our imitation algorithm achieves Bayes regret comparable to batch on-policy Thompson sampling up to O(T /N) where T is the number of batched policy updates.
Despite the seemingly linear gap in Bayes regret, N is typically orders of magnitude larger than T in internet applications where we can utilize the database of users / entities. Typically, N is in the order of hundreds of millions; as of 2020, Facebook had 2.7 billion monthly active users; in our motivating video transcoding application, the service receives millions of video upload requests every day, providing an effectively unlimited number of unsupervised contexts. In contrast, the number of model updates (horizon T) is relatively small, in hundreds, due to complexities of policy deployment and nonstationary user behavior. In such practical problem instances, our imitation policy thus enjoys Bayes regret bounds comparable to that of batch on-policy Thompson sampling.
Paper Organization Section 2 reviews related work on Thompson sampling, approximate inference, and imilation learning. Section 3 formally introduces our distilled Thompson sampling algorithm. Section 4 presents empirical evaluations on benchmark problems and our real-world video transcoding application, demonstrating both regret performance and latency improvements. Sections 5 and 6 provide theoretical analysis: Section 5 shows that imitation error does not accumulate over time and relates algorithm's Bayes regret. to batch UCB algorithms,, while Section 6 establishes generalization guarantees for the imitation learning objective. Section 7 concludes with discussion and future directions.
2 Related work
There is a substantial body of work on Thompson sampling and its variants that use computationally efficient subroutines. We give a necessarily abridged overview of how our algorithm situates with respect to the extensive literature on bandits, approximate inference, and imitation learning.
A number of authors have showed that Thompson sampling achieves optimal regret for multi-armed ban- dits (Agraaal & Goyal 2012 2013a; Kaufmann et al. 2012 Honda & Takemura 2014 We refer the reader to the recent tutorial by Russo et al. (2018) and references therein for comprehensive overview. Agrawal & Goyal (2013b); Abeille et al. (2017) showed regret bounds for linear stochastic contextual bandits for a Thompson sampling algorithm with an uninformative Gaussian prior, and Gopalan et al. (2014) studied finite parameter spaces. Russo& & Van Roy (2014) established Bayesian regret bounds for Thompson sampling with varying action sets (which includes, in particular, contextual bandits); [Russo & Van Roy (2016) provides an information--heoretiic analysis that makes explicit the dependence on the prior (see also Bubeck & Eldan (2016)). We build the insights of Russo & Van Roy (2014), ; and show that our imitation algorithm retains the advantagcous properties of batch Thompson sampling, achieving (gap-independent) Bayes regret comparable to the best batch UCB algorithm.
Practical performance of Thompson sampling depends on having access to well-calibrated probabilistic predictions. Obtaining a balance between predictive accuracy, computational time, and memory requirements can be challenging in the context of large datasets with overparameterized models. Exact posterior sampling from even the simplest Gaussian linear models has time complexity of O(n2), where is the number of
5

---

Published in Transactions on Machine Learning Research (02/2026)
model p A common strategy used by some variational inference methods is to use mean-field approach where parameters are assumed to be independent (Blundell et al. 2015). This assumption can decrease sampling costs from (n²²) to O(z), where n is the number of parameters. However, [iqquelme et al. (2018) found that batch Thompson sampling using such approaches often leads to poor empirical performance.
When exact posterior inference is not possible, approximate inference methods can be used for posterior sampling. We refer the reader to Chapter of Russs et al. (2018)'s 's recent tutorial for discussion of approximation methods in relation to Thompson sampling. Bootstrapping (Eckles & Kaptein 2014 Osband 2016 Lu & Van Roy 2017) is a simple heuristic procedure that maintains multiple models to approximate et al. samples from the posterior distribution, although maintaining multiple models is often computationally
expensive. MCMC-based methods for approximate inference, and Hamilton Monte Carlo (HMC) (Neal 2011 in particular, are largely regarded as the "gold standard" for approximate Bayesian inference. HMC, and other MCMC-like approaches (e-g.; Chen et al. (2014); Welling & Teh (2011)) ) generate an arbitrary number of posterior samples for all parameters. While such algorithms permit rapid evaluation of postcrior samples (since the parameters are already sampled), they require substantial memory to store multiple samples of the parameters. Recent methods have also considered decomposing the covariance precision matrix into a diagonal and low-rank component (Zhang et al. 2018 Maddox et al. 2019). While this reduces computational complexity and memory costs relative to using the full covariance, sampling still incurs a time complexity of ((In+ + ())) where p is the rank of the covariance (or precision matrix) and p copies of the weights must. be stored.
By pre-computing and distilling Thompson sampling, our imitation learning framework allows the use of the most appropriate inferential procedure for the task at hand, rather than what is feasible to run in an online setting. In particular, the separation of online decision- making and offline computation allows the use of state-of-the-art Bayesian methods, such as those utilizing deep neural networks (Wang & Yeung 2020). While we restrict discussion to Thompson sampling in this work, the basic idea of offine imitation learning can be used to learn a explicit policy reprosentation of any complicated policy and allow operationalization at scale.
Imitation learning methods have received much altention recently, owing to their ability to learn complicated policies from expert demonstrations (Abbeel & Ng 2004 Ross Bagnell 2010 Ho & Ermon 20166). Our approach of minimizing the discrepancy between a parameterized policy and Thompson sampling can be viewed as an implementation of behavioral cloning (Ross & Bagnell 2010 Syed & Schapire 2010 Ross
lt al.] 2011]. Our imitation learning procedure resembles the "Bayesian dark knowledge" approach from Korattikara et al. (015)), which uses a neural network to approximate Bayesian posterior distributions. While most works in the imitation learning literature study reinforcement learning problems, we focus on the more limited contextual bandit setting, which allows us to show strong theoretical guarantees. We anticipate the growing list of works on imitation learning to be important in generalizing our imitation framework to the reinforcement learning (RL) setting. To account for time dependencies in state evolutions, both inverse RL approaches that directly model the reward (Abbeel & Ng. 2004 Syed & Schapire 2008). ) and the recent advances in generative adversarial imitation learning techniques (Ho & Ermon] 2016 Li ct al. 2017) show
promise in generalizing our imitation algorithm (behavioral cloning) to RL problems.
3 Distilled Thompson sampling
Reflecting typical operational scenarios on online platforms, we consider a batch (Bayesian) contextual bandit problem. The agent/ / deeiision-maker generates actions real-time as user requests come in asynchronousl,, and performs batched, infrequent updates to the policy, In what follows, we formally introduce imitation algorithm that makes it trivial to parallelize action generation over multiple computing nodes, even on each user's mobile device.
Let θ be the parameter space, and let ~ P be a prior distribution on e At each time t, the agent observes a context, takes an action, and receives a reward: we denote the context S, iid Pss, action A, E A, and reward
R, R.. We consider a well-specified reward model class {fo : Axxxxxxx x 5 R 0 (
fe(a, s) = E A, = a, S = s] for all a E A S. Thiis assumes the root decompasition of the covariance matrix has been cached, which incurs cost of ((33).
6

---

Published in Transactions on Machine Learning Research (02/2026)
Let H,- (S1, A1, A--1, Re-1) be the history of observations until time t. Assume that regardless of Hy for l' S the mean reward at time is determined only by the context-action pair
B 0, Hr s,
fo(a,s),
or equivalently, R. - (o St) t where e, is a mean zero i..d. noise.
At time 4, we denote by 7() the period before which the most recent policy update occurred. For example, for fixed batch size B
1.t=,.... B, if B 2B,
B+1
y(t)
(1)
2B +1
if 2 3B,
More generally, we allow time-varying batch sizes that are a priori unknown to the decision maker. We use (T(()) to denote the policy used at time that generates action A, based on the history Hg(()) available at the previous model update r(t): conditional on the history Hytty, we have A St TT(()-- |S), where abuse notation to suppress the dependence of Ty(() on the history Hygay- In the sequential (non-batch) setting, we simply have r(t) t.
The agent's objective is to maximize the cumulative sum of rewards by updating the policy Ty(1) based on observations. The regret the agent compares the agent's cumulative reward to the reward under the optimal action: for any fixed parameter value 0 e O, the (froquentist) regret for the set of policies (TE(
batches of context-action-revare
Regret (T, (T(tTT())NENN) = Σ
E
max fola, St) fo(A,, aEA
For simplicity, assume arg m So(a,s) is nonempty almost surely. We assume the agent's prior, P, is well-specified") key (standard) assumption that drives our subsequent analysis. Under the prior P over E θ the Bayes regret is simply the frequentist regret averaged over ~ P
T Σ 1=1
BayesRegret (T, (TTghee)) == Eo Regret (T, (x)) IEN, 0)]
Ban max fo St) fo so) aeA
Based on the history H*()), batch Thompson sampling plays an action according to the posterior probability of the action being optimal. The posterior probabilities are computed based on the prior P and previously obscrved context-action-
tuples. At time this is often implemented by
sampling from the posterior 0, ~ P(0 e Hya,, S.) and solving Åo E arg max So, (a, St). aeA
By definition, Thompson sampling enjoys the optimality property A. Hy St d Aj Hy(t), Sy where A E arg max fo(a, St) and is the true paramcter drawn from the prior P. Throughout, we assume A, H7(0), S, is independent of all else.
When the prior misspecified that the Thompson sampling policy uses Q instead of P, we have the equivalence noted Russo Van Roy (204))
by
Ee RRgret (T. (Txc) 0))
dP aQ
Regret (T, (Txte)heNN
(x)
where dP/dQ the Radon-Nikodyy derivative of P with respect Q. While misspecified priors incur substantially higher
regret LLu &
2016)
the worst-case, empirical evidence suggests Thompson sampling strong algorithm practice Scott
2010 Granmo
2010
Chapelle & T.I 2011 May Leslie 2011 Ferreira al. 2018 Graepel al. 2010 Agarwal al. 2014
Kawale al. 2015
Schwarta et 2017 Agarwal al. 2016

---

Published in Transactions on Machine Learning Research (02/2026)
To address challenges in implementing Thompson sampling real-time, we develop an imitation learning algorithm that separates online action generation from computationally intensive steps like posterior sampling and optimization. Our algorithm maintains an explicit policy representation that emulates the batch (off- policy) Thompson sampling policy by simulating its actions offtine. At decision time, the algorithm generates an action simply by sampling from the current policy representation, which is straight forward to implement and computationally efficient to run real-time. We summarize an idcalized form of our method in Algorithm 1 where conditional on the history Ha()) generated by the imitation policy
(x) 8) is the batch off policy Thompson sampling policy at time t.
(2)
This policy is different from the true, batch on-policy Thompson sampling since the inittaiio policy generales actions based on which rewards are observed. Nevertheless, we will show that our algorithm enjoys Bayes regret comparable to batch on-policy Thompson sampling.
At each time I, our algorithu observes a context St, and plays an action drawn from its explicit policy representation. Formally, we parameterize our policy x= (a s) with a model class m E M. For example, M can be a neural network that takes as input a context and outputs distribution over actions. We generate actions by sampling from the current policy A = (- St), which can be easily implemented to run with low latency on resource-constrained computing infrastructure such as mobile devices. The agent uses a batch of context-action-reward tuples to update its posterior on the parameter 0 E θ offtine. Although this step requires posterior inference that may be too burdensome to run real-time, our method allows running it offline on a different computing node, so that it does not affect latency. Using the updated posterior 0 P(. H())) the agent then simulates actions drawn by the Thompson sampling policy by computing the maximizer A((s) E arg max So, (a, 8), for a range of values E. Using these simulated context-action pairs, we learn an explicit policy representation that imitates the observed actions of the Thompson sampling policy.
(=(()()
Algorithm Imitating Batch Thompson Sampling
Input: prior on parameter space O, reward model class ())(:))), imitation policy model class {TT" m E MM, notion of distance D for probabilities Initialize m +- arg minme. Eg-ng (To, [s)]]
for t=t== 1 to 7" do
4: Obscrve S₁ sample A₃ ( (-1 St), receive R.
if frt+1a 1= 1 + then
6: Update model m + arg mi Esars (Tx(et)): T" [S)] offline
7: end if
8: end for
Dropping the time subscript to simplify notation, the imitation learning problem
minimize Es-Py [D (T, n" [S)]
(3)
mEM
learns a model E M minimizing a measure of discrepancy D (,, S) between the two distributions on A, conditional on the context S. As the imitation objective equation 13 cannot be computed analytically, we provide efficient approximation algorithms. To instantiate Algorithm we fix Kullback-Leibler (KL) divergence as the notion of discrepancy between probabilities and present finite-sample approximations based on observed contexts and simulated actions from the off-policy Thompson sampling policy 7,. For probabilities q² and q² on A such that q²,q² < for some o-finite measure on A, the KL divergence between q² and g² da' /do
is Dai (q* == LA log do (a)dv(a), where we use dg' and dy² q¹ and 4² with respect to v. For two policies n²' and n² we define
du dv to denote RtaonnNNkodie derivatives of
Dia ( 2 (S) = Da (7'(-1 S))
where we use x/, 2 to also denote their conditional densities over A.
8

---

Published in Transactions on Machine Learning Research (02/2026)
The imitation problem equation 8 with D D(:-155 = Du (+, (S) is equivalent to maximizing log likelihood
maximize Be ES (S.-s,,~==((15))) |oy m (A [S]]
(4)
me.
for simplicity. In the maximum likelihood estimation (MLE) problem equation 1 the data comprises of contexl-action pairs. First, contexts are generated under the marginal distribution 5 ~ Ps independent of everything else. Conditional on the context, actions are simulated from the batch off-policy Thompson sampling policy À ~ 7(- | ()). The MLE problem equation finds a model m E . maximizing the likelihood of observing actions generated by (T())-
In the following, we write B[-] = Ps-Ps.A-x(1S)
The imitation objective m + Ellog x² (A | [S]] involves an expectation over the unknown marginal distribution of contexts Ps and actions generated by the Thompson sampling policy 7(- | S). Although the expectation over S ~ Ps involves a potentially tigh-dimensional integral over an unknown distribution, sampling from this distribution is usually very cheap since the observations $ ~ Ps can be "unsupervised" in the sense that corresponding action/ action/reward are necessary. For example, it is common for online platforms to maintain a database of features S for all of its users. Using these contexts, we can solve the MLF problem equation 4 efficiently via stochastic gradient descent methods (Kushner & Yin 2003 Duchi 2018). In Section 6 we show that it is easy to solve the imitation problem equation 3 to high accuracy by using cheap unsupervised
contexts. In Section 5 we show that our imitation algorithm enjoys Bayes regret comparable to that of the batch on-policy Thompson sampling algorithm, up to the sum of single step imitation errors.
For continuous action spaces with a notion of geometry, it is sometimes natural to allow imitation policies to have slightly different support than the Thompson sampling policy. In this scenario, we can instantiate the abstract form of Algorithm | with Wasserstein distances as our notion of discrepancy D (t,- I The subsequent theoretical development for KL divergences has its analogue for Wasserstein distances, which we outline in Appendix A
4 Empirical evaluation
We study the performance of our imitation learning algorithm in terms of cunulative regret / reward and decision-time latency in a number of datascts. Our imitation lcarning algorithm achieves a significant reduction in latency on all problems and enjoys regret comparable to that of batch on-policy Thompson sampling, avoiding compounding of imitation error over time. Our experiments include real-world video upload transcoding application for an internet service receiving millions of video upload requests per day.
We include our open-sourced implementation as supplementary materials.
Datasets We compare our imitation algorithm alongside an array of benchmark mothods on four problom scenarios. For our first experiment, we study the wheel bandit problem, a synthelic problem constructed require significant exploration (Riquelme et al.] 2018). In this two-dimensional problem, there are 5 actions and rarcly seen contexts yield high rewards under one context-dependent action. We sample 10, 000 contexts for each trial. Specifically, two-dimensional contexts are sampled in the unit sphere with uniform probability. The first action al always has a mean reward of E[((s, 01)] = 1.2 independent of the context, and the mean rewards of the other actions depend on the context. If || < 6, then the remaining four actions are non-optimal with a mean reward of 1. If ||s||| > 6, then one of the remaining actions is optimal and determined by the sign of the two dimensions of 8 with a mean reward of 50. The remaining three actions all have a mcan reward of 1. All rewards are observed with zero-mean additive Gaussian noise with standard deviation a- 0.01. We set 8 = 0.95, which means the probability of sampling context on the perimeter ([|s||2 5) where one action yields large reward is 1 (0.95)² 0.0975 x 10%.
For our second problem, we design a contextual bandit problem from supervised classification task. The Mushroom UCI Dataset (mis)] 1987) contains 8,124 examples with 22 categorical features about the mushroom and labels indicating the mushroom is poisonous or not. At each time step, the forager decides whether to cat the mushroom or not and receives a small positive reward for cating safe mushroom, and a large negative reward for cating an unsafe mushroom. With equal probability, cating a poisonous mushroom lead to illness (r= -35) it may not harm the consumer (r = 5), while nonpoisonous mushroom always
9

---

Published in Transactions on Machine Learning Research (02/2026)
yields positive reward (r 5). The reward for abstaining is always 0. We sample 50, 000 contexts for each trial.
Next, we turn our attention to a more realistic healthcare scenario, pharamacological dosage optimiza- tion, where we wish to learn a good dosing policy for Warfarin. Warfarin is one of the most common anticoagulants (blood thinner), often prescribed to patients with atrial fibrillation to prevent strokes (Xiao) 2019). The optimal dosage varies considerably across genetic, demographic, and clinical differences (Bastani & Bayati 2015). The Warfarin dataset (Xiao] 2019) contains the optimal dosage of Warfarin for 4, 788 patients, which were found via trial and error by physicians. Using 17-dimensional context vector patient-specific demographics, medical history, and genetic markers, we construct contextual bandit benchmark where the action space is a uniformly discretized dosage levels, and rewards are given by absolute deviation from the optimal dosage. We present results for 20 discretized dosage levels, but as we shown in Section E we observe even bigger latency gains for 50 discretized dosage levels. We present results where we reshuffle contexts for each trial, but again find similar results when 50, 000 contexts are re-sampled each trial.
Finally, we focus on real-world video upload transcoding application, where we study video upload system for a leading social network platform receiving millions of upload requests on mobile devices (see Example The goal is to preserve high quality as much as possible while ensuring upload reliability constraints are met. We use a 38-dimensional context vector representing: (1) Video features: raw bitrate, resolution, file size, duration, codec; (2) Network features: connection type (WiFi/cellular), download bandwidth estimate, upload bandwidth estimate, signal strength; (3) Device features: device model category, operating system, available memory; (4) User features: country, historical upload success rate. There are actions corresponding to a unique (resolution, bitrate) pairs. The actions are ranked ordered in terms of quality: action i yields a video with higher quality than action j if and only if i 2 j. If successful, the reward for a successful upload is a positive and monotonically increasing function of the action. The reward for a failed upload is O.
We evaluate the performance of different contextual bandit algorithms using the unbiased, offline, policy evaluation technique proposed by [i et al.] (2011). The method evaluates a contextual bandit algorithm by performing rejection sampling on a stream of logged observation tuples of the form (S,, Rt) collected under a uniform random policy. Specifically, the obscrved tuple is rejected if the logged action does not match the action selected by the algorithm being evaluated. Our dataset contains 8 million observations logged under a uniform random policy. We evaluate each algorithm using the stream of logged data until each algorithm has "observed' 50, 000 valid examples.
Our ofline evaluation is not meant to suggest offline learning is a valid substitute for online learning algorithms. The cost of randomization and the high level of nonstationarity in the system makes online learning algorithms necessary. We use offline evalustions as an empirically rigorous scientific benchmark that supports and validates our methodological development. Our offline dataset is generated by particular vertical product, and provided the empirical evidence needed to invest significant resources in implementing the algorithm across multiple products. As the final evaluation, we ran randomized controlled study as described in the int roductiion and observed significant improvements in video quality and topline business metrics (watch time).
Algorithms and evaluation For all experiments, we consider models previously found to perform the best in a broad range of benchmark problems, as reported by Riquelme et al. (2018) in their extensive empirical experiments. Linear-TS uses an exact Bayesian linear regression to model the reward distribution for each action a independently. This policy evaluates the exact posterior under the assumption that the data for action were generated from the linear function: Ta s + where e (O) For each normal--invrre--gane -aammmmm-
action, we independently model the joint distribution, o(0,o2) P(O))3) distribution which allows for tractable posterior inference (scc Appendix E for closed form expressions). NeuralLinear-TSS models rewards using a neural network with two 100-unit hidden layers and ReLU activations, but discards the last linear layer and uses the last hidden layer (os) as the feature representation for a Lincar-TS policy. The neural network takes the context as input, and predicts the reward for cach action. The parameters of the neural network are shared for all actions and are learned independently of the Bayesian linear models. Bootstrap-NN--TS trains multiple neural networks on bootstrapped observations
10

---

Published in Transactions on Machine Learning Research (02/2026)
9000
8300
(300
600
7500
Cannielvv
2000
Canuleive
400
E200
300
8000
Unchomiaacco Nevral-Gentty
15
Uiifsmmaandan
NK-TS
3000
Greddy
TS-EL
5000
Liver-TS
Nesttiiinrr T5-11
Unear-13
Neurstitear11 15-I1.
Doontny N0-15-11
NeeniL
Dootrtap- T---
20000
30000
40000
50000
1000
2000
5000
(a) Cumulative regret on Mushroom dataset
(b) Cumulative regret Warfarin dataset
12000
11000
Running Average Rewards
1.00
Conalaativ Dogget
10000
1.03
UstoomBarddknn
Beebatp--N--S
1.02
Groedy Liase-13
Unea:
8000
NoxalLavu 15
Beetsaa NN IS IL
2000
6000
Unfienreeslsa Naund-Grody
Lisse-7S-1
1.00
5000
Lioer 15
NousaLisa 15
4000
NecraLinear IS
Booermp NN 15 IL
[L00]
6000
10000
20000
40000
00000)
(c) Cumulative regret Wheel bandit
(d) Hunning average of rewards for video transcoding
Figure 2: We report mean cumulative regret (or rumning average of rewards for video transcoding), alongside two standard errors over 50 trials (100 trials for the Wheel bandit, due to rarity of large rewards).
and randomly samples a single network to use for each decision. For all of the aforementioned TS policies, TS-IL denotes their imitated counterpart. We use a fully-connected neural network to parameterize the policy x" in the imitation learning problem equation 3 The policy representation has two hidden layers with 100 units each, hyperbolic tangent activations the hidden layers, and a soft-max activation the output layer to predict the conditional distribution P(als) for all a E A. We compare (batch) Thompson sampling and its imitation counterparts against two additional benchmarks: a random policy (UniformRandom) and a greedy policy that uses a feed-froward neural network to model rewards (Neural-Greedy)
Policies are updated every 1000 examples (except for the Warfarin problem, where we use update policies every 100 examples due to the small size of the dataset) and are initialized using a uniform random policy before the first batch update. Formally, the mapping 7(t) is specified in the definition equation [ with batch size B = 1000 or 100. We detail our hyperparameter choices in Section E. following extensive evaluations by Riquelme et al. 2018 we use their proposed settings for Thompson sampling.
In Figure 2 we show that each TS-IL method achieves performance comparable to its corresponding vanilla TS algorithm on all benchmark problems. We evaluate the cumulative performance at time steps along the entire learning curve, and observe that each TS-II. policy consistently matches its corresponding TS policy over time.
(Approsimate) Bayesian inference often requires substantial amount of compute and memory. We evaluate decision- time latency and time complexity for the specific models being considered, but note that the latency and complexity may be even greater under inference schemes not considered here. We define decision time lalency as the time required for a policy lo select. an action when it is queried. While BOOTSTRAP-NNY achieves low prediction latency, it requires storing many replicates the neural network and can significantly increase the memory footprint. On low-end mobile devices, such memory requirements can be prohibitive, limiting the applicability of methods based on bootstrapping; our imitation methods offer practical and effective alternative.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Decision-making latency in milliseconds. All latency measurements were made on a Intel Xeon E5-2680 v4 @ 2.40GHz CPU with 32-bit floating point precision. For each latency measurement, action generation is repeated 100K times and the mean latency and its 2-standard errors are reported.
MUSHROOM
WHEEL
VIDEO TRANSCODE
WARFARIN
UNIFORMRANDOM
0.040
(+0.000)
0.039
(+0.000)
0.040
(+0.000)
0.040 (+0.000)
NEURAL-GREEDY
0.242 (+0.001)
0.228
(+0.001)
0.231 (+0.001)
0.232 (+0.000)
LINEAR-TS
0.715 (=0.001)
1.142
(#0.001)
1.575 (+0.002)
3.963 (+0.002)
NEURALLINEAR-IIS
0.826 (=0.001)
1.492
(=0.001)
1.931 (=0.002)
4.814 (#0.004)
BOOTSTRAP-NN-TS
0.235
(+00001))
0.235
(+00001)
0.236 (1.0.001)
0.226 (+00.001)
LINEAR-TS-IL
0.184
(+00001)
0.178
(100.000
0.169 (100.000)
0.175 (400.000
NEURALLINEARR-TSIII
0.186
(+000000
0.179
(10..01)
0.169 (100000)
0.175 (100.00)
BOOTSTRAP-NN-TS-II.
0.190
(=0.001)
0.178 (+0.000)
0.175 (+0.000)
0.179 (+0.0011)
Table shows that the imitation policies (TS-IL) have significantly lower decision time latency compared to TS algorithms, often by over an order of magnitude on problems with larger action spaces (Warfarin and video upload transcoding). This is because generating drawing sample from the joint posterior P(O,, for each of the actions a, which is quadratic with respect to the context dimension for LINEAR-TS or the size of the last. hidden layer for NEURAI.. other hand, TS--LL IL simply requires a forward propagation through the policy network and a sample from multinomial sample, both of which are excoodingly cheap. In Section F we provide a detailed discussion of runtime and memory complexity, including those for alternative model choices.
action under the vanilla. TS policies requires
On the
Mobile Latency We report latency on server-grade hardware (Intel Xeon E5-2680) for controlled com- parison. On mobile devices, absolute latencies are higher but the relative improvement of our method is preserved amplified. Based internal benchmarking forward passes through our imitation network complete in milliseconds while Thompson sampling operations requiring matrix decompasitions can exceed at least an order of magnitude on low-end devices. Memory constraints on mobile devices (typically 2-4GB RAM) further favor our approach: storing a single imitation network versus multiple postcrior samples or bootstrap models. These practical constraints motivated our production deployment.
representative mobile devices (iOS and Android),
Randomized Controlled Trial Methodology Our production RCTs uses Meta's A/B testing platform, which uses a deterministic hashing function based off of unique user identifier and a unique identifier of the experiment to map subjects to conditions (Bakshy et al., 2014). As such, assignment is independent of any individual charactoristics or other treatments. Subjects were randomly assigned to treatment (distilled Thompson sampling) or control (existing rule-based upload policy) groups at the user level, ensuring that each user received consistent treatment throughout the experiment. Experiments ran for 2-4 weeks across more than millions of users per arm, providing sufficient statistical power detect the reported effect sizes.
5 Imitation controls regret
To understand the large practical gains we see in our numerical experiments and randomized controlled study, now provide some basic theoretical analyses. When the imitation policy generates actions (Algorithm D. the observations used to update the posterior are different from what the batch Thompson sampling policy would have generated. In this sense, our imitation algorithm does not emulate the batch on-policy Thompson sampling policy, but rather simply mimics its off-policy variant where posterior updales are based on the history generated by the imitation policy. In this section, we show how off-policy imitation is sufficient to achieve Bayes regret bounds available for batch on-policy Thompson sampling (Russo & Van Roy 2014), up to the sum of single-step imitation errors. In particular, our results guard against potential exponential compounding of errors that stem from imitating the off-policy variant of batch Thompson sampling.
12

---

Published in Transactions on Machine Learning Research (02/2026)
We show that minimizing the KL divergence equation g controls the Bayes regret of the imitation algorithm, justifying the the imitation learning loss equation 3 as valid objective. First, we relate the performance of our imitation policy with that of the batch off-policy Thompson sampler equation 2 and show batch off-policy Thompson sampling admits a Bayes regret dccomposition similar to that for on-policy Thompson sampling (Section p.1p. Building on this observation, we use similar proof techniques for proving Bayes regret bounds on batch on-policy Thompson sampling to provide guarantees for our imitation policy. We substantiate our results in scenarios where batch Thompson sampling is known to provide strong regret bounds (Section 5.2
5.1 Regret decomposition
Since our imitation learning problem equation 3 approximates batch off off-policy Thompson sampling. a pessimistic view is that any small deviation between the imitation and Thompson sampling policy can exacerbate over time. A suboptimal sequence of actions taken by the imitation policy may deteriorate the performance of the batch off-policy Thompson sampling policy equation 2 updated based on this data, compared to its on-policy counterpart updated based on data collected by itself. Since the imitation policy again mimics this batch off-policy Thompson sampler, this may lead to a negative feedback loop in the worst-case. Our analysis precludes such negative cascades when outcomes are avcraged over the prior P: the Bayes regret of the imitation policy is comparable to that of the best batch UCB algorithm, up to only the sum of expected discrepancy between the batch off-policy Thompson sampling policy and the imitation learner at cach period. In particular, imitation error at each period does not affect the Bayes regret lincarly in T as our worst-case intuition suggests, but rather only one-time approximation cost. The single-period imitation error can be controlled using cheap unsupervised contexts as we demonstrated in Section G
The Bayes regret suffcred under the batch off off-policy Thompson sampler is a counterfactual quantity as only the imitation policy interacts with the environment. Nevertheless, the fictitious quantity serves an important role in our analysis. Our starting point is that an batch off-policy Thompson sampler enjoys a Bayes regret decomposition similar to sequential, on-policy Thompson sampling, Since the off-policy nature of the policy does not affect the Bayes regret decomposition, are able to bound the Bayes regret of the batch off-policy Thompson sampler using proof techniques developed for sequential, on-policy Thompson sampling Ruusso & Van Roy 2014).
Before giving a formal result, we first summarize our approach, which builds on the insights of Russo & Van Roy 2014). We connect the performance of our imitation policy to that of batch off-policy Thompson sampling and in turn relate the latter method's Bayes regret to that of the best batch UCB algorithm. Since a similar approach also provides Bayes regret. bounds for batch on-policy Thompson sampling, our imitation policy enjoys comparable Bayes regret, up to the sum of single-perios imitation errors. Let Uz(; Hytey St) A + R be a sequence of batch upper confidence bounds, constructed using only data collected until the most recent batch Hy(n)- Let ABUCB be the action taken by the batch UCB policy (BUCB)
A ABUCB E arg max Ve(a; H7(t) St). aEA
Recalling the optimal action A E arg max fa, St), a typical argument for bounding the regret of a BUCB algorithm proceeds by noting that since (AABUCC H S)) 2 U St),
Se) - fa(A, BUCB S.)) S fo(A;, 8t) - U, (r(4 (1), S.) + U2 UA BUCB, (,)())) S:) - fa(ABBCB S.).
Taking expectations and summing over ,,...,T, T, BayesRegret (T, [T,BCB) hexx) is bounded by
S() - (A4; Hy S.)] + Σ EA T
T
BUCB HH St) - -oo So S.)).
If the upper confidence bound property holds uniformly over the actions so that Uz(a; Hy()) St) Z fo(a, St) for all a E A with high probability, the first term the above regret decomposition can be seen to be nonpositive. To bound the second term, a canonical proof notes each upper confidence bound not too far away from the population mean fo (ABUCB S..). Russo& & Van Roy 2214h's 's key insight was that (sequential on-policy)
13

---

Published in Transactions on Machine Learning Research (02/2026)
Thompson sampling admits analagous Bayes regret decomposition as above, but with respect any UCB sequence. This allows leveraging arguments that bound the (frequentist) regret of UCB algorithm to bound the Bayes regret of Thompson sampling. Since the Bayes regret decomposition for Thompson sampling holds for any UCB sequence, the performance of Thompson sampling enjoys Bayes regret guarantees of the best UCB algorithm.
By connecting the performance of our imitation policy to that of batch off-policy Thompson sampling, we show that similar Bayes regret decomposition can be leveraged despite its off- off-policy nature. Recall that we denote At ~ T-(t)(- St), the action generated by the batch off aff-policy Thompson sampler. See Section B for the proof of the following result.
Lemma 1. Let (Ty(0) HEN and Ue(:; Hy St) be any sequence of batch policies and UCBs (adapted to the history H,(r). If Efsupaea fola, S) L < 00;
Reyes (T, s
([fo(A, Si) - Ui(A;; H,,,, Si] +
Σ
(Au
(A; Ha,,, Si) fo(Ai, St)
1=1
(a): regret decomposition, for any batch UCB algorithm
+ L
R Dai (Fxce) 7,00 ss)]
(5)
(b): invilation error
The Bayes regret decomposition equation 5 shows that performance analysis of any batch UCB algorithm can characterize the regret of our imitation policy. In this sense, the imitation policy achieves regret comparable to the optimal batch UCB algorithm, up to the sum of single-period imitation errors. As we detail shortly in a general modeling scenario based on contextual Gaussian processes, term (a) can be bounded using canonical batch UCB proofs. Term (b) can be controlled by our imitation learning algorithm (Algorithm I and its empirical approximation seen in Section 6 Although this term scales as O(T/vN), we argue that the seemingly linear dependence on T is not of material concern. In large-scale internet applications, the number of unsupervised contexts N is very large as they can simply be read off of a databasc of user information (N sv 10 - 100M). The number of policy updates T is often orders of magnitude smaller (hundreds) in a typical product lifecycle due to operational challenges in deploying a policy. Thus, the term (b) can be made relatively small using big datasets and powerful overparameterized imitation models using the results in Scction d The fact that we are studying Bayes regret, as opposed to the frequentist regret, plays an important role in the above decompasition. We conjecture that in the worst-case, imitation error at any period (and consequently suboptimal exploration) can each linearly compound over time, leading to prohibitive quadratic dependence on T. It remains open whether specific problem structures can provably preclude such negative feedback loops uniformly over 6.
The main insight here that the Bayes regret analysis averages over the parameter drawn from the prior and under this averaging, the quality of the posterior whether it was updated using ou-policy or off-policy data- does not directly affect the regret decomposition. What matters is the current discrepancy between the imitation policy and the Thompson sampling policy at each time step, not the cumulative history of discrepancies. This is because Thompson sampling's 's optimality property (selecting actions according to their posterior probability of being optimal) holds regardless of how the posterior was formed. As a result, while off-policy data may lead to a posterior that is less informative than the on-policy counterpart, it does not systematically bias the Thompson sampling policy in a way that compounds crrors. Each poriod's imitation error contributes additively to the Bayes regret, rather than multiplicatively.
5.2 Regret bounds for contextual Gaussian processes
We now show concrete performance guarantecs for our imitation algorithm by using instance-independent (gap-independent) Bayes regret bounds for batch off- off-policy Thompson sampling. Despite its counterfactual nature, the decomposition equation 5 enables to control it using identical proof techniques for controlling
14

---

Published in Transactions on Machine Learning Research (02/2026)
the Bayes regret of batch on-policy Thompson sampling. This program allows us control over the term (a) in the decomposition equation
We consider a general setting where the mean reward function (a, (a s) can be modeled as sample path a Gaussian process, with potentially continuous action and context spaces. Formally, we assume that (a, 8) - fo 8) sampled from a Gaussian process on x S with mean function ju(a, 8) and covariance function (kernel)
E((a, s), (a', s')) = B((fo(a,s) ((a,s)) 8') p(a',
We assume that the decision maker observes rewards
R4 = fo(e,, E
where the noise 6, ld (0, are independent of everything else. Given these rewards, are interested in optimizing the function + (a, S₁) for cach observed context S, at time t.
Modeling mean rewards as a Gaussian process is advantageous since we can utilize analytic formulae to update the posterior at cach step. Since fo s) follows a Gaussian process, its posterior is also a Gaussian process with mean and variance is given by
()(a,s) : Bifo(as) Hd] = E, 8) (Kg + -0-1)-
(a, == Var(Jo(a, 8) H.) = E((a, 8). ()) - Ex(a,)) (Kass)([KK,,)) (A()))
where Ex(a, 8) = [E((A,, Sj), (a.,)))llssss-- K, = [A((A,, (Aj 1s: and R. = [Rg]
For scale applications, we can parameterize our kernels by a neural network and leverage the recently developed interpolations tochniques to perform offline posterior updates (Wilson Nickisch, 2015 Wilson
et al. 2015 2016).
We leverage regret bound techniques for batch UCB algorithms (Desautels et al. 2014 to bound the term (a) in the Bayes regret decomposition equation 5 This term is controlled by the maximal amount of information on the optimal action that can be gained after T time steps. Recall the definition of (conditional) mutual information between two random vectors
(( = Dia (Pay x Py) and I(2,Y W) = Dia (Pzr IP2w x Priw)
We define the maximal possible information gain after T time steps
[(xx fx)
Yr NC
where Rx (fo(x) + Extex and fx (fo(x))xex. For popular Gaussian and Matern kernels, Srinivas et al. 2012) has shown that the maximal information gain be bounded explicitly; summarize these bounds shortly.
Due to the batched nature of Algorithm ! we further need to control the maximal information gain in a single batch, assuming that the (timo-varying) batch size is uniformly bounded by some constant B.
Assumption A. Let y(t 7(t) < B for 1 1st5 T and let yh be constant satisfying
1 (Rx,5x Ryyn)) s log(ne) for all S
max x
(6)
xÇA
where Rx (f)(z) + xxx Ry(1) (RI...., Ry and fx (z)
For a compact action space A c Rd, term (a) in the decomposition equation is bounded by o ((og)) Our proof relies on the batch upper confidence bound
Ul(a; H4t), 8) (r + (iranuy(a,)) where Be 2olog(( =2oo((("rr))TTT)
(7)
15

---

Published in Transactions on Machine Learning Research (02/2026)
We use Ly to denote the (random) Lipschitz constant of the map + fo(a,
[o(a, So(a', s)]
b = sup sup sEB a.E'
||a - a'll1
Standard arguments from Gaussian process theory show [0[ < oo holds whenever u(-) and 21.+)) are times continuously differentiable (Ghosal et al. 2006 Theorem 5).
Theorem 1. For A [0, for some r > 0, let Assumplion hold. Assume that
C1 == sup |u(a, < co, == aEA.ES
I((a, (d', < oc,
a, a,
= and let == sup su fo fa s)²] as before. If E[ < co, there is a universal constant C >> such that
ReyesR (T₂ < (TE(L + Ce2 + Cdlog(rd)
c1 E[
c3
+ TMBYT log T + dl rd)
1/2
+ (I+ /228r)
E Dia (Tx(0) (T()) [s)]
log(1 o o-2)
See Section C.1 for the proof.
Bounds on >r To obtain concrete bounds the maximal information gain yr, focus the popular and flexible linear, Gaussian and Matern kernels
,x(x,²') = TTx, (I,xx')) exp (-II= x'113
212
22-v r" Br(r) where r Em(x, = T(u)"
V2v || x ||
where we used B() and T(-) to denote the Besel and Gamma functions respectively. To case notation, we let K denote the dimension of the underlying space, and define
My,, T) = x log
My 7) = (log 7)*+1, M) 7) == T FR log T.
We have the following bound on ?r for linear, Gaussian, and Matern kernels; the bound is a direct consequence
of Krause & Ong (0011 Theorem 2) and Srinivas et al. (2012 Theorem 5).
Lemma 2. Let A C Rd und S C RRd be convex and compuct. Lel the kernel E be given by E((a, 8), (a', =
Ex(a, a') + E'(s,s'). Then, YT = O (MT(AA,)) + MTss, + log T').
Bounds on ye To control the Bayes regret of batch off-policy Thompson sampling, it remains to control the per batch information gain Лß. Our development so far allows us to use techniques developed for on-policy Thompson sampling to bound this quantity. A naive bound for the per batch information gain T is πв I exp(2r:B), which can be prohibitively large in large batch scenarios. Towards tighter theoretical control, we use clever initialization scheme due Desautels et al. (2014). While we conjecture that batch Thompson sampling will perform well even without such a careful initialization scheme, we are unable to theoretically confirm the conjecture and leave it as future work.
We initialize our algorithm by targeting Timit users/conterd contexts who suffer the highest uncertainty in their reward. Considering the initialization index set E {-Timi + 1,...,0}, 0}, the posterior variance does not depend on previous rewards
s) = Var(fela,s) H.) Va(, s), (a, 8)) E)(a, (K, + -121-
16

---

Published in Transactions on Machine Learning Research (02/2026)
where s( =
(a, (o.s)) -) before engaging with the environment we can sequentially calculate
and K, [K((A,, (., Thus,
(A iiit , St E arg max (a, for t -Tint + 1,...,0 aEA,sES
We initially largel users/conlexts. Stait in the database with actions Ajuit for ( - 1, ,..... O. Using the history H, = {S,, A4, -- redefine Thompson sampling and Algorithm [ with initialization data. The following result shows that this initialization procodure controls the per batch information gain 1)B- For simplicity, we consider combinations of linear or Gaussian kernels and define d == max(d, d'). Recalling the batch upper confidence bound equation 7 the following result a direct consequence of Desautels et al. (2044 Lemma 4, Theorem 5); an analogous bound holds for Matern kernels, but we omit for brevity.
Proposition 2. Let the conditions of Theorem 1 hold and let ΣA, Es E (E,,Ig)- Consider the initialization procedure described the precious paragraph with Tinit periods. There is a consland C > 0 such that if we set Iin = C++1 B(log Był+), then
BayesRegret (T, (T(()EEN) Σ
Banp max fola, (A acA
0 cxxpd")) TT( + (L + Vo2Br)
E Dii (Tx(t): [s)]]
2
The first term bounds the Bayes regret of batch on-policy Thompson sampling: in comparison, sequential on-policy Thompson sampling (Krause & Ong 2011 achieves Bayes regret O (vār(log
Discussion of Assumptions Our theoretical results rely on several assumptions that merit discussion on their applicability. The Gaussian process assumption is standard in the Bayesian optimization literature and is reasonable when rewards are smooth functions of context-action pairs. In practice, the GP assumption serves more a theoretical vehicle; our empirical results (Section 4) demonstrate strong performance even when using linear and neural network-based models that may not strictly satisfy GP assumptions. More specifically, the smoothness conditions on kernels ensure bounded Lipschitz constants, which popular kernels including Gaussian (RBF), Matérn with 2, and polynomial kernels satisfy.
The theoretical initialization scheme targeting ensuring bounded per-batch information gain. In practice, uniform random initialization performs well (as shown in experiments) because real-world contexts naturally provide sufficient coverage. The gap between our thelretical initialization and practical random initialization remains an interesting open question.
contexts is primarily a proof technique
Our results require barring two types of misspecification: first, the imitation model class must contain the true Thompson sampling policy; second, the Thompson sampling policy's prior must be well-specified. While the former relatively mild given how well modern neural networks with sufficient capacity can approximate complex distributions, the second inherits all the non-robustness issues of Thompson sampling. More generally, our imitation approach will inherit any limitation of the original model class.
6 Generalization guarantees for imitation learning
In this section, we show that solving an empirical approximation of the imitation problem equation 4 can control the imitation objective. From results in the previous section, this in turn shows that the regret be controlled when we have many contexts. Given i.i.d. observations of (potentially unsupervised contexts) iid
S; ~ Ps, we solve the empirical approximation to the imitation problem equation 4
N
No log n (Aig S.).
m E arg max MEM
N
N
==1
j=1
17

---

Published in Transactions on Machine Learning Research (02/2026)
where we simulate actions from the batch off- off-policy Thompson sampling equation 2
As (0) | (t) j l...., No
for each context S,. Since actions can be simulated offtine in a parallel manner, we can efficiently generate a large number of actions No.
In what follows, we assume that our imitation model class is well-specified, so that there exists m E M satisfying -"" where omitted the subscript and denote मे * to case notation. This is often reasonable assumption we consider expressive model classes such nouparametric models involving reproducing kernel Hilbert spaces. With a well-specifeed imitation model, we prove with probability at least 1- 6,
1 N
1 (Compy + log 5 + VNN,
CompN.N.
Es-F, T,,
(9)
for some complexity measures Compy and Comp N.N. associated with the imitation model class M. Here, the notation S denotes inequality up to a universal constant. In typical internet applications, the number of unsupervised contexts N exceedingly large, and the imitation error equation g can be made vanishingly small.
The key challenges to showing the precoding result twofold: 1) the empirical procedure equation g employs non-ii.d. samples (S,, Aij), standard concentration results do not apply, and 2) the bound equation 9 scales with the "fast rate" 1/N, rather than the canonical parametric rate I/vN. To overcome the first challenge, proof carefully derives concentration incqualities for the two-step sampling process where nature generates S; ~ Ps, and for each S, we simulate A. ~ T(- S:) via posterior sampling. To prove the fast rate of convergence 1/N, we use an claborate localization- based proof approach (Bartlett et al. 2005) which exploits the fact that the complexity of the function class (s, a) + log " (a smaller on a neighborhood of the optimum m², compared over the entire model space m M.
iid
| s) may be substantially
To formalize our arguments, recall the standard notion of Rademacher complexity: for fixed EI,...; and ii.d. random signs E E {-1,1} (Rademacher variables) that are independent of the & the empirical Rademacher complexity of the class of functions 9 C {g E + R} is
(g) Eg
sup
Σ ELO(Et) gEg n
1
A function si IR+ - IR,, sub-root (Bartlett et al. 2005 if it nonnegative, nondecreasing, and r + (t)) VF is nonincreasing for all r > 0. This analytic notion guarantees that any non-constant sub-root function i continuous, and has a unique positive fixed point r* v'r*), where 2 ((r) for all r r*. Let th : RR+ R2, be sub-root upper bound on the localized Rademacher complexity
v)n 2 R( ((g g :[gg] $ s))].
(10)
(The localized Rademacher complexity itself is sub-rool.) Fixed points of th characterize uniform concentration guarantees; see Bartlett et al. (2005) and Koltchinskii 2006 for a detailed analysis of localized Rademacher complexitics.
The Rademacher complexity of the following set of functions controls the generalization performance of the empirical imitation model equation
G1 = 5 + E ( log #(A s) s) Th (A s)
m E m
(s(s) = {a + log n (a 8) E M}
G3 = {(a, s) + log n (a 8) m E M)-
We let ry be the unique fixed point of the sub-root function t/ satisfying the bound equation 10 for G = G1
r'y (x(r)) where (r)(r)
E :gg x1)]
18

---

Published in Transactions on Machine Learning Research (02/2026)
For any fixed context S, using ii.d. random signs Ey, we write
sup 1 N. mEM Na
RND2(s) = E.
Σ j=l
ej log Th (Ag)
For G3, using i.dd. random signs Eg we still write
N. Eig
RNN. 93 == E.
sup mEM
log π th (Aij |i:)
t=
j=l
Our main result in this section shows that the imitation error of the empirical solution equation g is Op (N-1 + N See Section D.1 for the proof.
Theorem 3. Let there erist m E M such that ā = -T²" Assume log -hh [a| S M for all a A, S.mEAN M. There is numerical constant C > 0 s.t. with probability at least 1 2
E Dia R nr [s)] sc (1
Mt N
V
N
sup E SES 1
Ry. (92(s)) Eji.N.
For liitt-dimeniional model classes with bounded VC-dimension, standard arguments bound the Rademacher
complexity terms in the above theorem (van der Vaart & Wellner 1996. Ch 2.6). Denoting by VC(.) the VC-dimension, we have
supses VC(G2(s)) N.
VC(gs) NN.
sup E nd RNN (G2(s)) S MV
and E[RNN,, (9a) S M
sES A ((())
Moreover, Corollary 3.7 of Bartlett al. (2055) implies that rty MVC Plugging these
bounds in Theorem 3 we obtain the previously claimed convergence rate equation
Due lo the generality of our localized Rademacher complexity approach, we can provide imitation guarantees for substantially larger and more expressive nonparametric model classes. We consider a reproducing kernel Hilbert space (RKHS) H defined over a kernel A: : : x E + R4 (Berlinet & Thomas Agnan 2004). For such nonparametric models, standard covering number bounds are loose (Kühn 2011), while localized arguments can still provide fast, concentration (Mendelson) (2003). Consider a RKHS with norm ||-I|x and evaluation kernel k(---). Mercer's theorem (Cristianini & Shawe- Tayler- 2004) states that the integral operator orthonormal decomposition in (2,
Te LE. (E, Th (M) = h ((, (E. (')dP(E') is compact, and have the cigenbasis expansion k(E,E)) Σド= (E)) (E') where 'ÿ are eigenvalues of T sorted in decreasing order and ø, give
Let ks S - R and KA A x A + R4 be kernels S and A respectively, and let us denote by By and BA the unit ball in the respective RKHS's. The kernels ks and A induce RKHS over functions S A formed with the kernel k((s,a), a), a()) = ks(s, s') + ka a'); we denote the unit ball in this space by BsxA- For simplicity, we assume that the function classes G1, G2(8,, and G3 belong in a unit ball appropriately defined RKHS's
G C Iss, G2(s) C BA for all s E S, G3 C BsxA
For SiKHS-based models, the rate of decay of the eigenvalues of The controls the rate of convergence in Theorem For example, eigenvalues of the popular Gaussian kernel k(g,g') = exp(- || - <'1l2) decay
exponentially fast X, ~ Mendelson 2003). Eigenvalues of kernel operators Tę for Sobolev spaces (Birman & Solomjak 1967 Gu 2002) decay polynomially fast Xς S j-28 where / > 1/ is the smoothness level. e.g., 1-dimension, the first-order Sobolev kernel k(E,E') min (E. where generates RKHS of Lipschitz
functions. We prove the below corollary in Section D.2
19

---

Published in Transactions on Machine Learning Research (02/2026)
Corollary 1. Assume supses ks, s) + supaE ka(a, a) S B for some B > 0. If the eigenvalues of Tke decay as As S 5- there numerical constant > 0 s.t. with probability at least 1 - =
T1 x $s
Mz + Vlg N + CMB NN.
2+1
E Dal
If the eigenvalues of Ths decay as X3 S j-2 for some B > 1/2, then there another numerical constant
C such that with probability at least 1 =
E Dui (
[s) s
Mz N
+ N
21 WOTT
+
MB
+ 1 NN
7 Discussion
In this paper, we used imitation learning to operationalize Thompson sampling, allowing it to scale to applications where latency and software complexity are of concern. We demonstrated that imitation learning provides a simple, practical, and efficient method with desirable regret properties. By distilling the Thompson sampling policy into casy-to-deploy explicit policy representations (c.g. neural networks), the-art Bayesian approaches to be used in contextual bandit problems. We hope that this work facilitates applications of modern deep barning-based Bayesian approaches to large-scale contextual bandit problems.
allow state-of-
Our algorithmic approach shifts computation from online latency-critical to offline batch processing envi- ronments, and is not meant to be taken as a universally dominating intervention. This trade-off is most useful when offline computation embarrassingly parallel and can leverage powerful backend servers, whereas online computation must execute on ressuree mobile devices with strict latency requirements. For example, in video transcoding deployment, N is chosen to be sufficiently large for accurate imitation (typically N = 10 contexts) while remaining computationally feasible for offline batch processing. With modern distributed computing infrastructure, the offline phase completes within minutes even for such sizes of N, which is acceptable given that policy updates occur infrequently (every 1000 observations in our experiments). In our deployment, policies are updated daily or weekly depending on traffic volume, providing ample time for offline computation. The batched nature of our algorithm naturally aligns with machine learning rnfrastructure designed for periodic model retraining.
While we have empirically evaluated two types of Bayesian models, framework is compatible with any type of probabilistic model. For example, practitioners may utilize domain knowledge to develop grey-box models (see e.g.. Schwartz et al. (2017)). Such models are simple to implement in probabilistic programming languages (Carpenter et al.J 2017 Bingham et al. 2018 Tran et al., 2018), but challenging and inefficient to deploy. Our imitation framework can allow ease of deployment for these models while maintaining a comparable level of performance. Although we restricted attention to contextual bandits problems, an interesting research direction is to extend these methods to combinatorial ranking problems (Cheung et al. Dimakopoulou et al. 2019), where computational savings of distillation may be even larger.
2018
While we focus on contextual bandits, our imitation framework naturally extends to sequential decision problems. The key challenge in reinforcement learning is that states evolve based on actions, introducing temporal dependencies that complicate off-policy learning. Several directions show promisc: behavioral cloning with trajectory-level imitation where we imitate entire rollouts from a Thompson sampling agent; inverse reinforcement learning approaches that learn reward functions from TS demonstrations; generative adversarial imitation learning that matches state-action distributions without explicit reward modeling. The theoretical analysis would require extending our regret decomposition to handle state dependencies, potentially building on recent work connecting Thompson sampling to posterior sampling for reinforcement. learning.
Broader Impact Statement Our work enables efficient deployment of Thompson sampling on resource- constrained devices, with demonstrated application in vidco upload systems serving millions of users. Our method allows sophisticated Docision-making algorithms to run on low-end mobile devices common in developing economies, potentially reducing the digital divide. Better video quality and upload reliability
20

---

Published in Transactions on Machine Learning Research (02/2026)
enhance communication, especially important for user-generated content platforms. By distilling complex policies into wight models, our approach may reduce energy consumption compared to running full Bayesian inference online.
On other other hand, contextual policies may perform differently across user subgroups. Users with older devices, slower connections, or regions with poor infrastructure might systematically receive lower-quality recommendations. We mitigate this by including device and network features in the context, allowing the policy to adapt appropriately rather than disadvantaging certain users. Ongoing monitoring of performance across groups is essential.
Our method requires collecting contextual features from user devices. We emphasize that the deployed system operates on aggregated, anonymized features and does not require access to video content itself. The imitation learning phase uses logged features without personally identifiable information.
The reported watch time improvements raise broader questions about zngagement-optimine algorithms. While better video quality generally beneficial, we acknowledge ongoing societal discussions about technol- ogy's role in attention capture.
References
Mushroom. UCI Machine Learning Repository, 1987. DOI: https///db..rr//124444//////
Picter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenly-Piee Firnt International Conference Machine Learning, pp. 1, 2004.
Mare Abeille, Alessandro Lazaric, et al. Linear thompson sampling revisited. Electronic Journal of Slatistics, 11(2):5165-5119) 2017.
Robert J. Adler and Jonathan E. Taylor. Random fields and geometry, volume 115. Springer, 2009.
Alekh Agarwal, Sarah Bird, Markus Cozowicz, Luong Hoang, John Langford, Stephen Lee, Jiaji Li, Dan Melamed, Gal Oshri, Oswaldo Ribas, et al. Making contextual decisions with low technical debt. arXiv preprinl XrXiv:1606 03966, 2016.
Deepak Agarwal, Bo Long, Jonathan Traupman, Doris Xin, and Liang Zhang. Laser: scalable response prediction platform for online advertising. In Proceedings of the 7th ACM international conference on Web search and data mining, pp. 173 182. ACM, 2014.
Shipra Agrawal and Navin Goyal. Analysis of Thompson sampling for the multi-armed bandit problem. In Proceedings of the Twenty Fifth Annual Conference on Computational Learning Theory, 2012.
Shipra. Agrawal and Navin Goyal. Further optimal regret bounds for thompson sampling, In Proceedings of the 16th International Conference on Artificial Intelligence and Statisties, pp- 107, 2013a.
Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In Proceedings of the 30th International Conference Machine Learning, pp. 127 135, 2013b.
Eytan Bakshy, Dean Eckles, and Michael S. Bernstein. Designing and deploying online field experiments.
Proceedings of the 23rd International Conference on World Wide Web, WWW '14, pp. 283 283-292, New York, NY, USA, 2014. Association for Computing Machinery.
Rajiv Banker, Yi Liang, and Narayan Ramasubbu. Technical debt and firm performance. Management Science, 6755:31174 3194, 2021.
Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local Rademacher complexities. Annals of Statistics, 33944:1497-1537, 2005.
Hamsa Bastani and Mohsen Bayati. Online maaksign-making with high-dimensionall covariates. SSRN Electronic. Journal, (11 2015. doi: 10.2139/ssmm.266 2661896..
21

---

Published in Transactions on Machine Learning Research (02/2026)
Alain Berlinet and Christine Thomas-Agnan. Reproducing Kernel Hilbert Spaces in Probobility and Statistics. Kluwer Academic Publishers, 2004.
Kartikeya Bhardwaj, Ching-Yr Yi Lin, Anderson Sartor, and Radu Marculescu. Memory- and communication- aware model compression for distributed deep learning inference on iot. ACM Trans. Embed. Comput. Syst., 18(5s), October 2019. doi: 0..1145/3358200
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep Universal Probabilisticc Programming. Journal of Machine Learning Research, 2018.
M. Birman and M. Solomjak. Piecewise polyyooial approximations of functions of the classes W Sbornik: Mathematics, 2(3):295-317, 1967.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference Machine Learning, pp. 1613 1113-1622,,21 2015.
Frédéric Bonnans and Alexander Shapiro. Perturbation Analysis of Oplimization Problems. Springer, 2000.
Sébastien Bubeck and Ronen Eldan. Multi-scale exploration of convex functions and bandit, convex optimiza- tion. In Conference on Learning Theory, pp. 583 589, 2016.
Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programiing language. Journal of statistical software, 76(1), 2017.
Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in Neural Information Processing Systems 24, pp. 2249-2257, 2011.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Proceedings of the 31st International Conference on Machine Learning, volune 32, 2014.
Wang Chi Cheung, Vincent Y. F. Tan, and Zixin Zhong. Thompson sampling for cascading bandits. CoRR, 2018.
Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. Dawnbench: An end-to-end deep learning benchmark and competition. Training, 100(101):102, 2017.
Nello Cristianini and John Shawe Taylor 2004.
Kernel Methods for Patlern Analysis. Cambridge University Press,
Arnoud V Den Boer. Dynamic pricing and learning: historical origins, current research, and new directions. Surveys in Operations Research and Management Science, 20(1):1-18, 2015.
Thomas Desautels, Andreas Krause, and Joel W Burdick. Parallelizing explorattoo-oxpplottaioon tradcoffs in gaussian process bandit optimization. Journal of Machine Learning Research, 15:3873-3922 2014.
Maria Dimakopoulou, Nikos Vlassis, and Tony Jebara. Marginal posterior sampling for slate bandits. Proceedings of the Fightt International Joint Conference on Artificial Intelligence, IJCAT-19, 2019.
John C. Duchi. Introductory lectures on stochastic convex optimization. In The Mathematics of Data, IAS/Park City Mathematics Series. American Mathematical Society, 2018.
Dean Eckles and Maurits Kaptein. Thompson sampling with the online bootstrap. ar preprint arXiv:1410.4009, 2014.
Kris Johnson Ferreira, David Simchi-Levi, and He Wang. Online network revenue management using thompson sampling. Operations Research, 66(6):1586-100)) 2018.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika. Chaudhuri and Ruslan Salakhutdlinov (eds.), Proceedings of the 36th Intervational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052 2062, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, Xiaohui Ye, Zhengxing Chen, and Scott Fujimoto. Horizon: Facebook's open source applied reinforcement learning platform. 1xx::1811.00000 Lcs. 2018.
Subhashis Ghosal, Anindya Roy, et al. Posterior consistency of gaussian process prior for nonparametric binary regression. Annals of Slalistics, 34(5):2413 2429, 2006.
Aditya Gopalan, Shie Mannor, and Yishay Mansour. Thompson sampling for complex online problems. In Proceedings of the 31st International Conference on Machine Learning, pp- 100-108, 2014.
Thore Graepel, Joaquin Quinonero Candela, Thomas Borchert, and Ralef Herbrich. Web-scale bayesian click- through rate prodiction for sponsored search advertising in microsoft's bing scarch engine. In Proceedings of the 27th International Conference on Machine Learning, 2010.
Che-Christoffer Granmo. Solving two-armed bernoulli bandit problems using bayesian learning automaton. International Journal of Intelligent Computing and Cybernetics, 3(2):207-234, 2010.
C. Gu. Smoothing spline ANOVA models. Springer, 2002.
Jonathan Ho and Stefano Ermon. Gencrative adversarial imitation learning. In Advances in Neural Information Processing Systems 29, pp. 4565 4573, 2016.
Junya Honda and Akimichi Takemura. Optimality of thompson sampling for gaussian bandits depends on priors. In Artificial Intelligence and Statistics, pp. 375 775-383,44 2014.
Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In International Conference on Algorithmic Learning Theory, pp. 199-213. Springer, 2012.
Jaya Kawale, Hung H Bui, Branislav Kveton, Long Tran-Thanh and Sanjay Chawla. Efficient thompson sampling for online matrix factorization recommendation. In Advances in Neural Information Processing Systems 15, pp. 1297-1305, 2015.
Vladimir Koltchinskii. Local Rademacher complexities and oracle inequalities in risk minimization. Annals of Statistics, 34(6):2593 2656, 2006.
Anoop Korattikara, Vivek Rathod, Kevin Murphy, and Max Welling. Bayesian dark knowledge. In Proceedings of the 28th International Conference on Newral Information Processing Systems Volume 2, NIPS'15, pp. 3438 3446, Cambridge, MA, USA, 2015. MIT Press.
Andreas Krause and Cheng S Ong. Contextual gaussian process bandit optimization. Advances in Neural Information Processing Systems 24, pp. 2447 2447-2455, 2011.
Thomas Kühn. Covering numbers of Gaussian reproducing kernel Hilbert spaces. Journal of Complerity, 27 (5):489 499, 2011.
H. J. Kushner and G. Yin. Stochastic Approzimation and Recursive Algorithmss and Applications. Springer, second edition, 2003.
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of taatexxtul-baandii based news article recommendation algorithms. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM '11, pp. 297-306. Association for Computing Machinery, 2011. doi: 10.1 00.1145/1135888 1935826. 1935878.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual
demonstrations. In Advances in Neural Information Processing Systems 30, pp. 3812-3822, 2017.
Che-Yu Yu Liu and Lihong Li. On the prior sensitivity of thompson sampling. In Internaiional Conference on Algorithmic Learning Theory, pp. 321-331133333 231-336.20 Springer, 2016.
Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. In Advances in neural information processing systems, pp. 3258 3266, 2017.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems 32, 2019.
Hongzi Mao, Shannon Chen, Drew Dimmery, Shaun Singh, Drew Blaisdell, Yuandong Tian, Mohammad Alizadeh, and Eytan Bakshy. Real-world video adaptation with reinforcement learning. In Newral Information Processing Systems Workshop on RL for Real Life, 2019.
Andren Mas-Colell, Michael Dennis Whinston, Jerry R Green, et al. Microeconomic theory, volume 1. Oxford university press New York, 1995.
Benedict C May and David S Leslie. Simulation studies in optimistic bayesian sampling in contextual-bandi. problems. Technical Report, Statistics Group, Department of Mathematics, University of Bristol, 11:02, 2011.
Shahar Mendelson. On the performance of kernel classes. Journal of Machine Learning Research, 4(Oct): 759-771, 2003.
Radford M Neal. Mome using hamiltonian dynamics. Hondbook of Markov Chain Monte Carlo, 2, 2011.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy, Deep exploration via bootstrapped dqn. In Advances in nearal information processing systems, pp. 4026-4034, 2016.
Alexander Petrescu and Sami Tas. Client side ranking to more efficiently show people stories in feed, Oct 2016. URL fnee shccceentlysssooooooll-eee-eeeee
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trendoo in Machine Learning, 11(5-6):355 607, 2019.
Narayan Ramasubbu and Chris F Kemerer. Technical debt and the reliability of enterprise software systems: A competing risks analysis. Management Science, 12(55)14877111 2016.
Vinny Ricciardi. Reddit comments, 2019. URL httee://bbogg..orobbbbboooooooor
hatpapaee.....ago.............aa 20with, hwrrg//teecccccaccco//////eeeeee
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown. In International
Conference on Learning Representations, 2018.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 627-635, 2011.
Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4):1221-122) 2014.
Daniel Russo and Benjamin Van Roy. An information-thcorrtie analysis of thompson sampling. Journal of Machine Learning Research, 1711):2442-2411 2016.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. Foundalions and Trendso in Machine Learning, 11(1):1-96, 2018.
Eric M Schwartz, Fric T Bradlow, and Peter $ Fader. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science, 36(4):500 522, 2017.
Steven L Scott. A modern bayesian look the multi-armed bandit. Applied Stochastic Models in Business and Industry, (6(6))639-665 2010.
David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems 28, pp. 2503 2511, 2015.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias W Seeger. information-theoree regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions Information Theory, 58(5):32500333 -2600, 2012.
Statistica. Percentage of all global web pages served to mobile phones from 2009 to 2018, 2020. URL
Umar Syed and Robert E Schapire. A game-theoretic approach apprenticeship learning. In Advances in Neural Information Processing Systems 22, pp. 1449 1456, 2008.
Umar Syed and Robert E Schapire. A reduction from apprenticeship learning to classification. In Advances in Neural Information Processing Systems 23, pp.
2010.
Kalyan Talluri and Garrett Van Ryzin. The theory and practice of revenue management, volume Springer, 2004.
William R. Thompson. On the likelihood that unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285 294, 1933.
Dustin Tran, Matthew W Hoffman, Dave Moore, Christopher Suter, Srinivas Vasudevan, and Alexey Radul. Simple, distributed, and accelerated probabilistic programming. In S. Bengio, H. Wallach, H. Larochelle, and R. Garnett (eds.), Advances in Neural Information Processing Systems
K. Grauman, N. 31, pp. 7598 7609. Curran Associates, Inc., 2018. URL http://ppaerraappp.c/ppppp//// distributed- ance- pwo.ssimiipooogannngg.....
Alexandre B. Tsybakov. Introduction to Nonporametric Estimation. Springer, 2009.
A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York, 1996.
Cedric Villani. Optimal Transport: Old and New. Springer, 2009.
Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019.
Hao Wang and Dit-Yan Yeung. survey on bayesian deep learning. ACM Computing Surveys Conneess(CSie (CSUR), (5):1-33 37, 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient. langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-11), pp. 681 688, 2011.
Andrew Wilson and Hannes Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). International Conference on Machine Learning, pp. 1775-1784, 2015.
Andrew Gordon Wilson, Christoph Dann, and Hannes Nickisch. Thoughts on massively scalable gaussian processes. Xiv: 15111018710 [es.LG], 2015.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial Intelligence and Statistics, pp. 370-378, 2016.
Carole-Jean Wu, David Brooks, Kevin Chen, Douglas Chen, Sy Choudhury, Marat Dukhan, Kim Hazelwood, Eldad Isaac, Yangqing Jia, Bill Jia, Tommer Leyvand, Hao Lu, Yang Lu, Lin Qiao, Brandon Reagen, Joe Spisak, Fei Sun, Andrew Tulloch, Peter Vajda, Xiaodong Wang, Yanghan Wang, Bram Wasti, Yiming Wu, Ran Xian, Sungjoo Yoo, and Peizhao Zhang, Machine learning al facebook: Understanding inference at the edge. In Proceedings 25th IEEE International Symposium on High Performance Computer Architecture, HPCA 2019, pp. 331-344, 3 2019. doi: 10.1109/HPPO 2019..0080
Hai Xiao. Online learning to estimate warfarin dose with contextual linear bandits. CoRR, 2019.
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. In International Conference on Machine Learning, pp. 5852-5861, 2018.
26

---

Published in Transactions on Machine Learning Research (02/2026)
A Imitation learning with Wasserstein distances
When actions can be naturally embedded in a contimuous space, we may want to measure closeness between the imitation and TS policy by incorporating the geometry of the action space A. In this section, we provide an alternative instantiation of the abstract form of Algorithm [ by using Wasserstein distances as the notion of discrepancy D (t, s). Our previous theoretical development for KT, divergences has direct analogues this setting, which we now briefly outline.
Given metric -( A, the Wasserstein distance between two distributions g' and q² on A is defined by the optimal transport problem
Dw .²)
inf vELlg" .44)
where .44) denotes the collection of all probabilities on A x A with marginals q' and q² (i.c., couplings). Intuitively, Dw (q',4") measures how much cost d(A, incurred by moving mass away from A ~ q² to A' q² in an optimal fashior Wasserstein distances encode the geometry of the underlying space A via the distance d. Unlike the KT, divergence DA (g+ll²)) that take value do whenever q² has support not contained in g², Wasserstein distance allows imitation policies to have different support than the Thompson sampling policy, which is more appropriate in continuous action spaces. To simplify notation, for two policies x² and x² we let
De (a² x2 1 = Dw (T (-))
When Algorithm 1
is instantiated with the Wasserstein distance as its notion of discrepancy D (c. De (s | (), the imitation learning problem equation 3 becomes
minimize Es-Ps [Dw (T, T S)] MEM
(11)
To solve the above stochastic optimization problem, we can again use stochastic gradient descent methods, where the stochastic gradient Vm Dw (T,(0); x" S) can be computed by solving an optimal transport problem. From Santorovich-Rubiisteen duality (see, for example, Villani (2009),, we have
Dw (TT(()==** TF 8)
sup E g.A-B
(AA (A)))(1)9(A)) g(a) g(a') s d(a,a') for all a, A).
(12)
where d(.,-) is the metric on A used to define Dr (: For discrete action spaces, the maximization problem equation 12 is linear program with O([4)) variables and constraints; for continuous action spaces, we can solve the problem over empirical distributions to approximate the optimal transport problem. We refer the interested reader to Peyré et al. (2019) for a comprehensive introduction to computational methods for solving optimal transport problems.
Letting 9* denote the optimal solution to the dual problem equation 12
the envelope theorem (or Danskin's 's 2000 Theorem 4.13)) implies that under simple regularity conditions
theorem; Bonnans & Shapiro
V Dw (TT()); Th 8) = Vn (1e)g
Assuming that an appropriate change of gradient and expectation is justified, can use the policy gradient trick to arrive at
V E (1) = (1n log x T" [A)s)) s)].
We conclude that for A ~ mm St),
g" (A)Vm log x (A 8.)
(13)
is stochastic gradient for the imitation problem equation II As before, can get lower variance estimates of the gradient by averaging the above estimator over many actions A ~ T" S)). Using these stochastic gradients equation
13 we can solve the imitation problem equation T efficiently.
Forr discrete action space, Dw (--)) can he defined with any symmetric matrix d(a,, satisfying d(as, oz) 2 with 0 iff
aj ajj and d(a;, az) d(a,, (4) + dlay, az) for any as ajj ag E
27

---

Published in Transactions on Machine Learning Research (02/2026)
We now show that the resulting imitation policy admits regret decomposition similar to Lemma for KL divergences. As a direct consequence of this decomposition, the regret bounds in Section 5.2 have their natural analogues with Wasserstein distances replacing KL divergences as the notion of discrepancy, though we omit them for brevity.
Lemma 3. Let Uc(:; Hat)) St) A + any upper confidence bound sequence that is measurable with respect Hy(t), St, As. If there is a Lo satisfying almost surely
[J((a So(a', 8)1 S Lad(a, a') for all 8 E E, E A,,
(14)
ReyesRegret (T, (ER))))
[BJo(A UG(;: H. H()) (.0)))))))) + Σ S
BlU H2),, - (e,
t=1
t=1
E Le (5,(0); Ty(0) St)
(15)
where Dw (-)-1)) is the Wasserstein distance defined with the metric d in the condition equation 14
Proof The proof mirrors that of Lemma 1 By the Kantorovich dual representation equation we have (A, St) - fo (Ac, St) 0, (n) So] < D (mx(t)) Tx(1) St).
Here, we have again used that A, H.()). S, and A, H,tt), S, are independent all else. Applying this
bound in the decomposition equation 16 and taking expectation over (H, St) on both sides and summing T, get the desired bound.
B Proof of Lemma 1
Conditional on (t) A, has the same distribution as A. Since Ui(a; H7(0), St) is a deterministic function conditional on (H+c), St), we have
(A,; St) H. S. (A;; ((),, H St).
We can rewrite the (conditional) instantenous regret as
E[fo(A;, (i) Hy(t). St]
-R(fo(Ai, S.) - U:( Hy(t); S.) H. St] (A,;
Hy,,,, St) B(U,,
[o(A,, H7(e) St] Haco) St]
= S.) - U.( Hyye), St) H Si] + (A,; H, S.) fo (A,, St)
+ ( Sz) fo(A,, S.) ) H.
(16)
We proceed by bounding the gap
St) - fo(A,, S:) Hy(),, S.]
(17)
using the KL divergence between (e) and Ty(e)- Recall Pinsker's inequality (Tybbkov
2009
|P Qlyy sup --1I-
Er(g(A)) - 1 (PIQ).
(P(Q).
From the hypothesis, Pinsker's incquality implics
E[fo(A,, S. S,) H, St] sup (a (. St) Tn(e) Salley
aEA
P (T,10, 7y()) S.)) sup (e (a)l a
28

---

Published in Transactions on Machine Learning Research (02/2026)
Here, we have used that A, (t)): St and A, ( independent of all clse.
Applying this bound in the decomposition equation 16 and taking expectation over (H-)))> St) on both sides and summing = ,, we get
T
BayesRegret (T, &
f Si) - U,(A7; (1), St)l + Σ ElUs(A;; Hot St) - fo(A,,
(ey,
t=1
E
sup fo( Silly 2 aEA
Dia (Tyct), Tx(1) (ss)
1=1
Applying Cauchy-Sehwarz inequality and noting that E Sola, S L, we obtain the final decomposition. o
C Proof of regret bounds
C.1 Proof of Theorem
In what follows, we abuse notation and let C' be a universal constant that changes line by line. We build on the batch UCB regret bound due to Desautels et (2014), defining the (batch) upper confidence bound
U,(a; Hy()))) (a 8) + v (ianx()(a,s)
with B, = (l From Borel-TI5 inequality (e.g., see (Adler Taylor 2009 ) we have
2 =
sup fo(a, s)² .EEA.ES
< <0.
We bound the first two terms in the regret decomposition equation 5
starting with the second term
Σ ElUUAAA; 81) fe(Ar, Σ
t=1
First, note that since (a, s) s vo,, Pinsker's incquality gives
(Ar, St) Hycoy, Si] - Elo,(o(Ac, Ha St]l S v02 | (.1 St) - Tx(0(( St)llTV
C2 Dia (ī-(t)) TT(e)) Sc).
We arrive at the interim bound
Σ (lu Hy)),, S:) fo( St)]
Br02
T
BT
B (e +
Di (Txa), TT(0) S.)].
(18)
2
By elementary calculation (e.g., see Desautels et al.
2014
Proposition 1)), we have
(a,) exp (I (f(s,a), (R.) R((()) S VTB. (1(a,)) s)
where the last. line follows from Assumption A Next, the following lemma due to Srinivas al. (012
29

---

Published in Transactions on Machine Learning Research (02/2026)
Lemma 4 Srinivas et al.
2012
Lemma 5.3)). For any sequence of Ae and St:
$
E Σ (A, Si)
27T
$ V
log(1 + t
Using the preceding two bounds, RHS of the inequality equation
18 can be further bounded by
r
E (Ac; Hato) Si) - fA, St)l
Cr
TTnBTR
Σ of (At,
+
E R[Dia (Zx00) (x()) [..)]
2
2TTBTT8T log(1 2)
Brez
B[DRi (Tx(0) Ty(e)) |s)]]
(19)
We now bound the first term in the decomposition equation a E A, there exists [a]: E A, such that ||a - [a]ell < 1/t4.
Let A be (1/t*)-cover of A, so that for any
T
[[fo(A) 5.) U,A Hn St)
Σ
E (ffo - fo([A]), SAI
(a)
([ S.) UG(AA]) 8.]]
(() 8t) - Ut(A2; S,)]
(c)
Using the definition of Ly, the first term (a) in the above equality is bounded by
Σ Blfo(A)) fo(lAt),, s B
Σ t=1
|14 - [At S <BL
Σ 1
CH|L
where we used the fact that Ar is coter of A. To bound the second term (b), use fo(a,s) Hy(t) ~ N(Mro) 5), o,n
[[Jo(a, Ua(a; H H) H Hn()) < [[([s((a,)) 8) - -U((a H H)) Hycel
-x(t) (a, 8) e < 2 A.
C2
(20)
22m
where we used 2 llg({AA)) S Br since [Ail < (1rd)) Hence, we obtain the bound
B(o((A))), - U.([A))) T ([L[A))),S))-(((())))))HH.. (0)) S.) S Σ Σ E[Jo(a, S.) Uz(a;
x
Ha H St)
C2
S
Cez
2xt
aEA.
where used the independence of S, and Hy(t), and the bound equation 20 To bound the third term (c), we show the claim
(a; (1) 8) - Ua(a'; H2(9) s EEL, Haw)] lla - d'll
(21)
so(c.,)2) Hxcoo 1) laa d'l
+ / Be 2F L, sup
ula, 8)2 + sup aEAsES
CEASES
30

---

Published in Transactions on Machine Learning Research (02/2026)
From the above claimed bound, it follows that
T
T
T
C1
L + c3
Σ E[U7([A7)) H. (t) U277; H 70,, (
-
23
1=1
t=1
I <BBLL,] + Colog()
B|y c3 /EL331
To show the bound equation 21 first note a + [[ s) Ha(() and + [[ s)² Ha(e are L 11
and y SuPREA,ES ( s) H Lipschitz respectively, for all S. Hence, 2200 (a. 8) is (c) + SUPREA,ES (fo(a, s) {)} H (1 -Lipschit. Noting that
o (a, 8) - or (d') (x(1) (a, 8) + (xt))aa,s))
(0,())(a -
S 10 o (a)
for any C O, taking the infiruum over c on the right hand side yields
(o s) ancola', 8) - / 2 (a, s) - (an)(a
3 ||a la-aa'll -aa a'lli 10(a,s)²) Hscs) 1)
< 2B L 2
sup
aEA.ES
which shows the bound equation 21
Collecting these bounds, we have shown that
Σ EEJ S.) - U1(A7; H, S.) s CE[Ls] + Ce2 + Cdlog(rd) (avElL (LL/ + ca c
T
(22)
Combining this with the bound equation 19 we obtain our result.
D Proof of generalization results
D.1 Proof of Theorem 3
We abuse notation and use C> > 0 to denote a numerical constant that changes value line to line. We use the following concentration guarantee using localized Rademacher averages.
Lemma 5 (Bartlett al. (2005) Thcorem 3.3)). For a class of functions with range [0, M), let r be
the unique posilive ftred point of the sub-root function vha salisfying the bound equation 70 Then, for i.i.d.
jid observations & ~ P. there is numerical constant C > 0 such that
Mz
CM
[l <
g(Ex) + 0(1 +1)
M , +
+
for all E 9 and 2 0
n
with probability al least 1 =
Notice that by Jensen inequality, we have
#(A SS,)
E A log nT (A [S.)
2 0 almost surely.
Applying Lemma 5 with the function class 9i and ' = 1/2, we have
E Dia (
ss] <
3
E f
log T
#(A Si)
+
"M
C
T
+
CMz
(23)
N
S.))
M
N
i=1
31

---

Published in Transactions on Machine Learning Research (02/2026)
In the rest of the proof, we bound the interim (uniform) approximation error
1
=(A S:) log xr <" S:)
1
* (A,s 1 S:)) T Si)
ZN.N. = sup meM
log
N
No
This is indeed sufficient for our purposes since the bound equation 23 implics
E Di T
|s)]
-
3 1
N
N.
log
(Ai St) TT' (Ai) S.)
+c ZN.N. +
rN +
Mz
2N
No
M
N
By the definition equation g of the empirical solution m and by virtue of having a well-specified model class M, the first term in the preceding bound is nonpositive
N
N, log
i(A THhh (Ais) SS))
S:)
N
N
[AAij | S.)
log
0. mm (Aij Ss))
N
N
N
1=1
Consider the Doob martingale Mu E[ZN.N.J. and
M₁ = EiNN,N. SI....., Syl for ISKSN,
martingale adapted to the filtration F& == (S Sy). Denote the martingale difference sequence D3 M M4-1 for k 2 1. Let 5% be an independent copy of Sg that is independent of all S, and let Abj ((- 5%) independent of everything other than Sy. We can write
}
N
(A S.)
N log
K (Ai) (.)) m (Ais (Si
DE =
sup mEM
E (A
log
S Se
TA 8:)
No
i=1
j=1
1
log =(À S.)
1
No
#(Aig (Ss)
IF
sup mEM
log
N
TA (s:)
N₂
TT (Aij (S:))
1
#(A') (%)
((A 5%
log T (A') 5%
log
S 1....... St
N
N₂
T (A'i)) 5%)
j=1
Independence of S,'s yields
1
No
#(A Sk))
*(Ar) Skx)
Del <
N
sup mEM
N
E
Am (-18)
log
(A Swh).
log T (Ass) Shh)
j=1
((A SQ))
*(A',) (5))
E (A
log TT (A' (.))
+ logg n (Al.) (5))
Sx
N₂
# 8)
a(Az 8)
S sup EE, S =A((((
sup mEM
E (
log T (A s)
log n (A, ()
N.
j=1
Nexi, we use slandard symmetrization result to bound the last display; see, for example, Chapter 2.3, van der Vaart & Wellner 1996) for a comprehensive treatment.
Lemma 6. f P, we have
E
sup
1
Σ (o(t)) Eig(E)))
< 4E,2, (9)
gEG
n
i=1
32

---

Published in Transactions on Machine Learning Research (02/2026)
Applying Lemma 6 to the bound on |D&] we conclude |Dxl S N supses E A, 1,(7(11)) is the function class
[RN. (G)(,))], where (g(s)
(2(s) = { m log
a(a {() Tm mm).
Note that Rv. (C]())) = RN. (92())). Then, Azuma- Hoeef-ng bound (Corollary 2.20, Wainwright (2019) yields
32: N
UNN & E[ZN,N.1 +
sup Rix (G2(s))
AES A, xx(0)
with probability at least 1- c=
It now remains to bound N.,,,,.J, for which we use symmetrization argument. Although (Si, Asj) not i..d. standard argument still applies, which outline for completeness. Denoting by (5), Aly) independent copies of (S4, Ang) again, we have
F(Ai) SS))
N
TAA S) n (A's (
E E = E
sup MEM
log
E
log
N
N
T (Mij S:)
N
N
N
N
#(Ais (S.)
क (A's) (S)
$ E
sup MEM
log
log
N
N
T.'h (Aij |S:)
T (A's) 5)
[=1
N
N
#(Aij] (S:)
5 (A 5
E
sup MEM
N
N
Gij
log x (Aij 5:)
log
n (4/ | (8))
i=1
<
RN,, (9a)).
Collecting these bounds, we obtain the desired result.
D.2 Proof of Corollary I
We use the following standard result that bound the Rademacher complexity of kernel models. Let k be a reproducing kernel on E, and let B be the unit ball in the RKHS H.
Claim 7. Let supçez suppezk(5,6) k(G,E) = B < co. Then, R. S v
Proof of Claim
For any fixed ,I,...,Em. En,
1
n
)
m
Rr (B)
Eg
sup KER
Σ Exk((,E))
E
Eck((,
Ei)
n2
12
74
2
1
n
B
<
Eg
Σ
(.
E2)
Σ |
12
vn"
Applying the claim to BA and and BSxA, we get
B VN
B
sup
(92())) (92(s))
S
and RNN, (92) VNN.
SES
To bound rs we the following result due to Mendelson
2003
33

---

Published in Transactions on Machine Learning Research (02/2026)
Lemma 8
MMendelson 2003
Theorem 2.1)). If X1 2 1/N, then for all 1/N
1 N
E Ri { B. E[K(S) Sr)
min
Consider the case where the spectrum of This decay exponentially
Vlog N
}
log N
log N
Vlog
N e
min
S
+ N
e
-
di
2
N
N
N
N
1=1
j=1
log N
log N
where we use S to denole inequality up to a numerical constant. polynomially decaying spectrum Ag S 28
We conclude ry S M
For
min mjin 29 r] R 23
28
+
00
23_1 2-8 dt x 28
1/28
20 1341
Solving for the fixed point, we get r x Mn
Collecting these bounds and plugging them into Theorem 3 we obtain the desired result.
E Experiment Details
Hyperparameters We use hyperparameters from Riquelme et al. (2018) follows. The NEURAL,REEDY, NEURAILLINFARTS methods fully-connected neural network with two hidden layers of containing 100 rectified linear units. The networks are multi-output, where each output corresponds for predicted reward under each action. The networks are trained using 100 mini-batch updates at each period to minimize the mean-squared error via RMSProp with an initial learning rate of 0.01. The learning rate is decayed after each mini-batch update according to an inverse time decay schedule with a decay rate of 0.55 and the learning rale reset the initial learning rate each update period. For BOOTSTRAP-NN-TS, we use 10 replicates and train each replicate with all observations as in Riquelme al. (0018).
The Bayesian lincar regression models used on the last lincar layer for NEURALLINEAR-TS, use the normal inverse gamma prior NIG(μ,, 0, a, = 3, 3, 3, Aa 0.257a). LINEAR-TS uses NIG = 0, a. 6, Ba A, = 0.25Ia) prior distribution.
The imitation models used by the IL mothods are fully-connected neural networks with two hidden layers of 100 units and hyperbolic langent activations. The networks use a Softmax function on the out outputs predict the probability of selecting each action. The networks are trained using 2000 mini-batch updates via RMSProp to minimize the KL-divergence between the predicted probabilities and the approximate propensity scores of the Thompson sampling policy TTS For each observed context S,, we approximate the propensity scores of the Thompson sampling policy (-(S;) using Na = 2048 Monte Carlo samples: (a - N Ennn, 1 (Ag = a} where Ai ~ n T' (-1S.). We use an initial learning rate of 0.001. learning rate is decayed every 100 mini-batches according to an inverse time decay schedule with decay rate of 0.05. In practice, the hyperparamcters of the imitation model can be optimized or adjusted at each update period by minimizing the KL-divergence on a held-out subset of the observed data, which may lead to better regret performance. We do not. use inverse propensity-weightingg on the observations, but we suspect that may it may further improve performance.
We normalize all numeric features to be in [0.1] and one-hot encode all categorical features. For the Warfarin dataset, we also normalize the rewards to be in [0.1].
34

---

Published in Transactions on Machine Learning Research (02/2026)
Iinformtaandom
NeitalLLnnse--T
Neural-Grendyy
Linear-S-II. Neural IS
8000
Linear-TS
6.00
Cume
4000
2000
10000
2000
300000
40000
50000
Figure 3: Cumulative regret on the Warfarin problem with 50 actions
Posterior Inference for Bayesian Linear Regression Linear-TS: For each action, We assume the data for action were generated from the linear function: Ta = s s + where ~ W(0,
~ 0a Oal0², NNH, o.Ea),
where the prior distribution is given by NIG Aa, aas Th observations of contexts Xa xd+x()+1) and rewards yo 1R we denote the joint posterior by P(O,, 0.) ~ NIG Aas Ga Ba), where
aas Br) and A, = Da1 the precision matrix. After
A = X Xa Aas
Po (AoMa + +X ya)
0g = a
na 2
B. = =B + 5 (yn Uo + 14e Aata p N..a).
Additional Results Warfarin 50 Actions Figure 3 shows the cumulative regret on Warfarin using
actions. The imitation learning methods match the cumulative regret of the vanilla. Thompson sampling methods.
F Time and Space Complexity
F.1
Complexity of Evaluated Methods
Table shows the decision-making time complexity for the methods used in our empirical analysis. The time complexity is equivalent to the space complexity for all evaluated methods.
NeuralGreedy The time complexity of NEURALGREEDY is the sum of matrix-vector multiplications involved in a forward pass.
Linear-TS The time complexity of LINEAR-TS is dominated by sampling from the joint posterior, which requires sampling from a multivariate normal with dimension d. To draw sample from the joint posterior p(0,o) at decision time, we first sample the noise level ø² ~ IG(a, and then sample 0|Ø² ~ ~ (H, Rather than inverting the precision matrix Σ A2A-1 we compute root decomposition (e.g. a Cholesky decomposition) of the d precision matrix A = LLT The root decomposition be computed once, with cost O(d),, after offline batch update and cached until the next batch update. Given L we sample directly by computing 0 = la + z, where
--I=C
(24)
and < - NV 1). Since L is upper triangular, Eqn. equation 24 can be solved using backward substitution in quadratic time: (
6The alternative approach of inverting the precision matrix to compute the covariance matrix Σ = A A-1, computing and caching its root decomposition = LrrL, and sampling Ø as Ø + Lv.C, where ç ( from the metrox-vvctor multiplication T.y..
lid \/ 1) also has time complexity of
35

---

Published in Transactions on Machine Learning Research (02/2026)
NeuralLinear-TS The time complexity of NEURALLINEAR-TS is the sum of forward pass up to the last hidden layer and sampling from multivariate normal with dimension hM, where has is the size of the last. hidden layer.
Imitation Learning The IL methods have the same time complexity as NEURAL ignoring the cost of sampling from multinomial with & categories.
F.2 Complexity Using Embedded Actions
An alternative modeling approach for the non-imitation methods is to embed the action with the context as input to the reward model.
NeuralGreedy Using an embedded action, the time complexity for a forward pass up the last layer is layer = (kdgl 1 Em=1 hmhm+1) because the input at decision time is k x d, matrix where the context is embedded with each of the k actions and the each context-action vector has dimension da- The time complexity of computing the output layer remains O(khar). The space complexity remains linear in the number of parameters, but it also requires computing temporary intermediate tensors of size k x hm for m = 1...M: Oduhh1 + EM-1 hmh + Em=1 khm).
Linear-TS Linear-TS, TS with an embedded action only requires using single sample of the parameters, which yields a complexity of to O kd,) for LINEAR-TS. The space complexity is also O kd,).
NeuralLinear-Tr For NEURALLINEAR TS the time complexity of computing the outputs given the last hidden layer is Oh + khar,, since only single sample of hM parameters is required for computed the reward for all actions. The space complexity for NEURALLINEAR- TS the sum the space complexities of NEURALGREEDY and LINEAR-TS.
Imitation Learning The computatiuonal cost of the IT. methods would be unchanged.
We choose empirically evaluate models without embedded actions because linear methods using embedded actions cannot model reward functions that involve non-linear interactions between the contexts and actions, whereas modeling each action independently allows for more flexibility. Riquelme et al. (0018) find that Thompson sampling using disjoint, exact linear bayesian regressions are strong baseline in many applications. Furthermore, Riquelme et al. (2018) observe that it is important to model the noise levels independently for each action.
F.3 Complexity of Alternative Methods
Alternative Thompson sampling methods including mean-field approaches, the low-rank approximations of the covariance matrix, and bootstrapping can also decrease the computational cost of posterior sampling. Mean-field approaches can reduce time complexity of sampling parameters from the posterior from quadratic O(n²) to linear ((n) in the number of parameters n However, assuming independence among parameters has been observed to result in worse performance in some scttings (Riquelme approximations of the covariance malrix allow for sampling parameters in O((n [[(n+11 1)p), where p is the rank of the approximate covariance, but such methods have a space complexity of O(m)) since they require storing copies of the parameters Zhang et al. (2018); Maddox et al. (2019). Bootstrapping also requires storing multiple copics of the parameters, so the space O(bn) where is the number of bootstrap replicates. However, bootstrapping simply requires a multinomial draw to select one set of bootstrapped parameters. All these methods require forward pass using the sampled parameters, and the time complexity is the sum of the time complexities of sampling parameters and the forward pass.
al.) 2018). Low-rank
We describe space complexity terms of the number of parameters n, so that do not make assumptions about the underlying model.
36

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Decision-making time complexity and space complexity for each method For methods relying fully-connected neural networks, the time complexity of forward pass to the last hidden layer is Cast-laaer = dh, + Em=l hm1 where d is the dimension of the context and hm is the number of units in hidden layer For BOOTSTRAP-NN-TE B denotes the number of bootstrap replicates.
METHOD
TIME COMPLEXITY
SPACE COMPLEXITY
NEURALGREEDY
O(CLAST-LAYER) O(khaz O(kd)
O(CLAST-LAYER + O(khar) O(ke² O[CLAST-LAYER) + (Kt22)) O(CLAST-LAYER B) O(khm B))
LINEAR-TS
NE
O(CLAST-LAYER) + (khar)
BOONSTRAP-NN-TS
O(CLAST-LAVER (khMM
IL
O(CLAST-LAVER + O(khas
37


Reviews:
[{"review_id": "7klkI272Tg", "reviewer": "Reviewer_Gu4r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an imitation-learning-based algorithm that distills a Thompson sampling (TS) policy into a lightweight neural network model, enabling efficient online decision making in mobile devices. The TS policy is updated using batched data collected under the imitation policy in an offline manner. The experimental evaluation demonstrates that the proposed method achieves comparable performance to batch TS and reduces the decision-making latency. In addition, the proposed method successfully deploys in the multiple video upload system for Meta, demonstrating its real-world applicability and impact. Furthermore, the paper presents the theoretical analyses that justify the effectiveness of the proposed algorithm on large-scale services.\n\n[Strengths]\n- This paper proposes a simple yet effective TS algorithm for efficient online decision making in mobile devices.\n- The effectiveness of the proposed algorithm is demonstrated through empirical evaluation on both synthetic and real-world problems.\n- The theoretical analysis justifying the proposed algorithm is also provided.\n\n[Weaknesses]\nI do not have any critical weaknesses to point out.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The effectiveness of the proposed method is well demonstrated through empirical evaluation. Additionally, the theoretical analyses justifying the proposed algorithm are provided. The paper is generally well-written.", "audience_yn": "Yes", "audience_explanation": "The proposed algorithm is relatively simple but powerful in real-world applications, and the theoretical analysis is also provided. Therefore, I believe that researchers and practitioners in machine learning and related fields would be interested in the findings of this paper.", "requested_changes": "I do not have any critical changes to suggest. However, I have a few minor comments below.\n\n- In Figure 2 (d), the label of the vertical axis should be \"Running Average of Rewards.\"\n\n- It might be better to add a brief explanation about the structure of this paper in the Introduction. Since this paper includes both empirical evaluation and theoretical analysis of the proposed algorithm, it would be helpful for readers to understand the organization of this paper.\n\n- In Table 2, the latency of the decision-making on Intel Xeon E5-2680, which is not a mobile device. It would be great if the authors report or discuss the latency on actual mobile devices. However, this is not a strict requirement.\n\n- On the last in the second paragraph of page 17, \"...entire model space $m \\in \\mathcal{M}$,\" should be \"...entire model space $m \\in \\mathcal{M}$.\" (The comma should be a period.)\n\n- I could not find the supplementary material, although the authors mentioned, \"We include our open-sourced implementation as supplementary materials.\" I would be great if the authors could share the link to the code repository after acceptance.", "broader_impact_concerns": "Although I do not have any critical concerns regarding the ethical implications of this work, it might be nice to include a Broader Impact Statement to discuss potential effects caused by operating large-scale systems, such as a video upload system, using the proposed algorithm, if possible.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "boLcZ8DXWX", "reviewer": "Reviewer_Wa3Q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# **Summary of Contributions**\n\nThis paper introduces *Distilled Thompson Sampling*, an imitation-learning framework designed to make Thompson Sampling (TS) operationally feasible in large-scale, latency-sensitive contextual bandit settings. The method relocates all posterior inference and optimization steps to an offline environment and learns a parametric policy that imitates TS, enabling fast action selection on resource-constrained devices. The principal contributions are summarized below.\n\nThe paper proposes a practical algorithm for offline simulation and imitation of batch TS. By repeatedly updating the posterior on backend servers and generating synthetic TS actions, the method trains an explicit policy representation that can be deployed online without requiring posterior sampling or numerical optimization. This design directly targets system-level constraints such as latency, memory usage, and engineering complexity.\n\nThe work demonstrates that the distilled policy can produce actions with extremely low latency—more than an order-of-magnitude faster than conventional TS—while maintaining cumulative regret competitive with batch TS across benchmark tasks, including Wheel, Mushroom, and Warfarin dosage. The approach is validated further through a large-scale real-world application in video transcoding, where the distilled policy improves quality and engagement metrics and is subsequently deployed across major products handling millions of daily video uploads.\n\nThe paper also provides theoretical analysis showing that the imitation policy achieves Bayes regret comparable to batch TS, with deviation controlled by per-period imitation error terms that do not accumulate over time. Finally, it establishes generalization guarantees for the imitation objective, showing that abundant unsupervised contexts enable accurate approximation of the TS policy.\n\n# **Key Strengths**\n\nThe paper presents a method with significant practical relevance, demonstrated both through controlled experiments and through deployment across multiple production surfaces. The proposed approach produces substantial latency reductions while preserving the performance characteristics of TS. The work offers a rigorous theoretical treatment, including a regret decomposition linking imitation error to Bayes regret, Gaussian process-based analyses, and convergence guarantees for the empirical imitation procedure. The system-level motivation is clear and well-supported, and the contribution addresses real barriers to implementing TS in industrial settings.\n\n# **Key Weaknesses**\n\nThe approach inherits the limitations of the underlying TS policy, including sensitivity to prior specification and uncertainty calibration. The theoretical results rely on assumptions such as well-specified Bayesian models, smooth kernels for Gaussian process analysis, and specialized initialization procedures, which may not hold universally. Although offline computation does not affect latency, the posterior-update and TS-simulation phases may be computationally demanding in practice. The off-policy nature of the imitation process introduces analytical complexity and may limit the generality of the theoretical guarantees outside the modeled settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission’s main empirical and theoretical claims are supported by evidence that is, in general, accurate, well-structured, and appropriate for the stated contributions. The empirical evaluation spans both standard contextual bandit benchmarks and a large-scale real-world application involving video transcoding. Across all synthetic and public datasets, the results consistently show that the distilled imitation policies match the cumulative regret of their corresponding Thompson Sampling variants while substantially reducing latency. The experimental design, including batching, hyperparameter choices, and evaluation metrics, is clearly described and aligns with precedents established in the literature.\n\nThe large-scale deployment evidence is particularly compelling. The authors describe improvements observed in internal randomized controlled trials, including increases in video quality, watch time, and user engagement across multiple platforms. While proprietary and therefore not independently reproducible, the details provided are sufficient to substantiate the practical relevance of the method and are consistent with standard reporting practices for industrial systems research.\n\nThe theoretical claims are supported by rigorous derivations and proofs. The regret decomposition is clearly presented, and the argument linking imitation accuracy to Bayes regret is logically sound and consistent with existing analyses of Thompson Sampling. The assumptions underlying the theoretical results are explicitly stated, and although some are strong (e.g., Gaussian process models, smooth kernels, availability of abundant unsupervised contexts), the theoretical contributions themselves are internally coherent and mathematically correct within that framework.\n\nTaken together, the combination of thorough empirical evaluation, carefully articulated deployment evidence, and detailed theoretical analysis provides convincing support for the claims made in the submission.", "audience_yn": "Yes", "audience_explanation": "The paper tackles a problem that is highly relevant to the TMLR community: enabling principled Bayesian decision-making algorithms to function effectively in large-scale, real-time environments. Thompson Sampling is widely studied in machine learning, reinforcement learning, and online decision-making. However, its direct deployment is often limited by the computational cost of posterior inference and the need for low-latency decisions. The proposed imitation-learning framework preserves the statistical strengths of Thompson Sampling while making it practical for deployment on real systems, including mobile and other resource-limited platforms. This represents a meaningful contribution for both researchers and practitioners.\n\nSeveral factors support its relevance to TMLR readers:\n\n1. The work provides a bridge between theory and practice by combining regret analysis with empirical validation and deployment results.\n2. The imitation-learning view of distilling Bayesian decision policies is general and can be applied to other learning problems, including reinforcement learning and settings with very large action spaces.\n3. The paper addresses a real challenge in the deployment of machine learning systems, namely how to operationalize bandit algorithms in environments with strict latency and system constraints.\n4. The results have clear implications for ML systems research and contextual bandit methodology, both of which are widely represented in the TMLR audience.\n\nGiven these factors, the findings are likely to be of interest to readers working on contextual bandits, Bayesian inference, imitation learning, online learning algorithms, and applied ML systems.", "requested_changes": "# **Critical**\n\n1. Clarify the assumptions used in the theoretical analysis: The regret bounds depend on specific modeling assumptions, including Gaussian process priors, smoothness of kernels, and a particular initialization procedure. A clearer explanation of when these assumptions hold in realistic settings, and how restrictive they are, would improve the transparency of the theoretical results.\n\n2. Provide more detailed intuition for the off policy analysis: The paper shows that imitation error does not accumulate over time, but the surrounding explanation is difficult to follow for readers who are not already familiar with this line of work. Additional intuition about why off policy imitation remains stable would be beneficial.\n\n3. Expand the discussion of offline computational cost: The method moves all posterior inference and optimization to an offline stage, but the submission provides limited detail about the cost of this process in practice. Since practical deployability is a central claim, a clearer description of the computational requirements and update frequency is necessary.\n\n# **Non Critical**\n\n1. Add a short discussion on model misspecification: Since the distilled policy inherits the behavior of the Thompson Sampling model, any misspecification in the Bayesian model can also affect the imitation learner. A brief discussion of this issue would strengthen the submission.\n\n2. Clarify the extent to which the method may generalize beyond contextual bandits: The paper mentions potential relevance to reinforcement learning, but it would be helpful to outline which elements of the current framework do or do not translate to sequential decision settings.\n\n3. Provide a concise summary of hyperparameters used for the imitation model: A small table or appendix listing neural architectures, optimizer settings, and training parameters would improve reproducibility.\n\n4. Offer additional qualitative detail for the video transcoding application: While proprietary constraints are understood, a clearer description of the structure of contexts, actions, and rewards in this setting would help readers appreciate the deployment environment.\n\n5. Improve the readability of some proofs: Several technical results, including those involving mutual information and regret decomposition, are presented in a dense manner. Minor editorial restructuring could make the proofs easier to follow.\n\n6. Discuss the behavior of the algorithm under irregular batch sizes: The framework assumes that batching is feasible and relatively stable. A short comment on how performance might be affected by highly variable batch sizes would add useful context.", "broader_impact_concerns": "The submission focuses on improving the practicality of Thompson Sampling by enabling low latency, large scale deployment through imitation learning. While the method itself is algorithmic and primarily technical in nature, several broader impact considerations arise from its intended use in real world systems.\n\nThe first concern involves the deployment of automated decision systems on resource constrained devices, including mobile phones. Such systems can influence user experience, content delivery, and resource allocation at large scale. The paper describes deployment in video upload systems and similar recommendations or ranking environments. These systems may affect user engagement patterns, consumption behavior, or visibility of certain types of content. Although the paper presents positive product level improvements, it would be beneficial to acknowledge that changes in algorithmic policies can have downstream effects on user behavior that are difficult to anticipate.\n\nThe second concern relates to fairness and potential disparate impact. Contextual bandit systems deployed across diverse user populations may behave differently for users in regions with limited bandwidth, older devices, or lower quality connectivity. Since the distilled policy is trained to imitate Thompson Sampling under specific contextual conditions, any bias in the underlying reward model, context features, or prior choices will transfer directly to the deployed policy. A brief reflection on fairness considerations, especially given global scale deployment, would strengthen the submission.\n\nA third concern involves privacy and data handling. The method assumes access to large amounts of unsupervised contextual data from user interactions. Although the paper does not manipulate or expose individual level information, the reliance on large internal datasets raises familiar privacy considerations. A short note clarifying that the method operates on aggregated feature representations without requiring additional data collection would help preempt misunderstandings.\n\nFinally, the approach is motivated partly by improvements in engagement metrics, such as watch time. While this is standard in many industry settings, there is a broader societal discussion on how engagement optimizing systems may contribute to addictive usage patterns. Although this paper does not introduce new mechanisms that amplify such effects, a brief acknowledgment that better system performance may influence user behavior would make the broader impact statement more complete.\n\nOverall, the work does not introduce direct ethical risks, but the scale and nature of the deployment environment warrants a short discussion of fairness, privacy, and behavioral effects.", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "7aCVuO4RYX", "reviewer": "Reviewer_SjvG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a distilled version of Thompson Sampling (TS), designed to deploy contextual bandit algorithms in low-latency, resource-constrained environments (e.g., mobile devices) leveraging imitation learning. The core method involves an architecture made up of:\n\n- Offline TS policy: A standard TS policy runs on backend servers, performing heavy posterior updates and optimization based on batched logs.\n\n- Distillation: Distillation: An explicit translation of the offline TS policy into a lightweight representation that enables fast action selection by eliminating the need for online posterior sampling and numerical optimization on low-latency environments.\n\n- Online imitation policy: An explicit policy is trained via Imitation Learning (minimizing KL divergence) to mimic the offline TS policy action distribution for real-time inference.\n\nThe authors provide a theoretical analysis decomposing the regret into the standard Bayesian regret plus an imitation error term. They establish general regret bounds comparable to batch TS and generalization bounds for the imitation step. Morevover, they empirically demonstrate that TS imitation policies achieve regret comparable to batch TS ones with significantly lower latency on benchmark datasets and report successful deployment in a large-scale video transcoding system.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are generally well-supported by a mix of theoretical proofs and empirical evaluations. In particular, the claim of \"order of magnitude reduction in decision-time latency\" is substantiated by Table 1, which compares the execution time of the distilled TS against standard TS approaches. Also, the claim that the imitation policy achieves performance \"comparable to batch TS\" is supported by Figure 2, where the cumulative regret curves of the imitation policies closely track their offline counterparts across multiple datasets.\nThis claim is supperted also by theoretical analysis regarding regret decomposition and generalization capabilities, which appear mathematically correct based on the derivations provided in the appendices.", "audience_yn": "Yes", "audience_explanation": "This paper bridges the gap between theoretical bandit literature and systems engineering, making it relevant to both ML practitioners and theoreticians. For the first group, the paper addresses a crucial point in industry, i.e., the difficulty of deploying principled exploration algorithms (like TS) on edge devices due to compute/latency constraints. The architectural pattern is highly applicable to recommendation systems and mobile applications. For the second group, the analysis of off-policy regret and the application of localized Rademacher complexity to bound the imitation error contribute to the understanding of how approximation errors propagate in sequential decision-making problems.", "requested_changes": "Clarify Offline Computational Cost: The paper emphasizes efficiency and low-latency of online inference, but the distillation step requires running the offline TS policy's inference on N contexts (where N is \"orders of magnitude larger\" than the batched policy updates T, and can be \"in the order of hundred of millions\"). The authors should explicitly acknowledge that while online latency is reduced, the offline compute budget increases significantly to generate the synthetic labels. A brief discussion or rough estimate of this offline cost trade-off is necessary to ensure the \"efficiency\" claim is not misleading, since the massive computational burden is merely shifted from the edge device to the server, rather than being eliminated.\n\nDiscuss Model Misspecification: The theoretical guarantees for imitation learning rely on the assumption that the \"imitation model class is well-specified\". In practice, a simple softmax policy cannot perfectly capture the geometry of a complex Bayesian posterior. The authors should add a remark acknowledging this gap and that irreducible approximation error (bias) exists and how it might impact the convergence rate claim.\n\nMore details on RCT results: While impressive, the RCT results are summarized in a way that is not reproducible and cannot be scientifically scrutinized. It is also not clear if we can attribute these results entirely to the new methodology or to exogenous or unmentioned factors.", "broader_impact_concerns": "The authors do not present a specific Broader Impact Statement. I do not see immediate critical ethical concerns requiring a halt to publication.", "additional_comments": "In general the paper is well-written and clear, presenting a complex architectural solution in an understandable manner. While I am not a core domain expert in contextual bandit theory, the analysis appears to be sound and convincing. The required revisions focus on strengthening the scientific rigor of the evidence and clarifying the systems architecture trade-offs.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)