Parallelizing Thompson Sampling

Abstract: Amin Karbasi, Vahab Mirrokni, Mohammad Shadravan

Reviews and author responses:


This paper considers Thompson sampling (TS) for standard and contextual bandit problems. In bandit problems, TS is known to have great performance both theoretically and empirically. Although these problems assume a fully adaptive setting where the algorithm updates itself at each time step, this may be too restrictive to run in practice. This paper considers a mini-batch setting and considers the batch Thompson sampling (B-TS) algorithm (Algorithm 1). In particular, the distribution-independent bound is not compromised by deriving exponentially large size of batches (Algorithm 1 and 2). The authors also analyzed the clipped version of TS (MOTS) for Gaussian rewards as well as contextual linear bandits (Algorithm 2). 

### After the discussion period
I think the contribution is weak, but okay to publish. I will keep the current rating but not strongly oppose to acceptance.   
In my view, this paper is marginally below the acceptance threshold. This paper is well-written and nice, but there is not a strong push for this paper, and given TS, Batched UCB, MOTS, and contextual Thompson sampling are well-known, the results here are somewhat incremental. Deeper results on the adaptivity, limitation on static make the paper theoretically interesting. 

- Remarks

Motivating applications: The paper exponentially increases the batch size and would be great if there are specific scenes where the proposed algorithms shine.

Is there any results on static batch (Can static batched bandit with O(log T) batch size achieve the same rate/impossibility of that?). Algorithm 1 and 2 are dynamic batching. Results/limitations on static make the paper more interesting.

Is there any way to fill the gap of logT samples and logT/(loglogT) sample lower bound by [Gao et al 2019]?

Can you characterize the difference between similar results on UCB with these results? 

- Minor remarks
p2 notation inconsistency (N log(T) and N log T) The paper addresses the limitations. The societal impact is not described as this paper is methodological paper. 

 We thank the reviewer for their detailed comments.

> In my view, this paper is marginally below the acceptance threshold. This paper is well-written and nice, but there is not a strong push for this paper, and given TS, Batched UCB, MOTS, and contextual Thompson sampling are well-known, the results here are somewhat incremental. Deeper results on the adaptivity, limitation on static make the paper theoretically interesting.

Thank you for the suggestions. Our paper is the first to achieve the problem-dependent optimal regret in the batch Thompson Sampling setting. We are not aware of any previous work. Indeed, the algorithm is simple (which is a plus) but the analysis is quite technical and needs a lot of care. We have carefully mentioned all the changes in the proofs that we need to make in order to have a rigorous result. We have also shown empirically (see Fig 1.b) that Batch Thompson sampling methods that use static batches, incur an order of magnitude higher regret even when the number of batches are larger (say 2 or 4 times higher). 

> Motivating applications: The paper exponentially increases the batch size and would be great if there are specific scenes where the proposed algorithms shine.

A simple application is A/B testing where the mechanism is usually under the designer's control. In this application, there are two clear procedures: non-adaptive and fully adaptive policies. For the sake of simplicity, let’s assume that each user takes 1 second to like/dislike. In a non-sequential setting, all the queries are asked in parallel among T users where half the population are given option A and the other half option B. Since everything is run in parallel it only takes one second to finish the whole experiment. The main issue is that half the population receive the suboptimal choice. In a fully sequential setting, users sequentially receive the option A or B, based on the previous responses. This experiment takes T seconds to finish. However, the benefit is that many users receive the better option. What we propose in this paper is a solution where the whole experiment takes $O(\log T)$ seconds while many users benefit from the guarantee of a fully adaptive policy. 

> Is there any results on static batch (Can static batched bandit with $O(\log T)$ batch size achieve the same rate/impossibility of that?). Algorithm 1 and 2 are dynamic batching. Results/limitations on static make the paper more interesting.

Our empirical results (Fig 1.b) show that  a static batched bandit with $O(\log T)$ batch size achieves much worse regret. We believe that in order to achieve the optimal problem-dependent regret bound, the batch sizes should be dynamic. 


> Is there any way to fill the gap of $\log T$ samples and $\log T/(\log\log T)$ sample lower bound by [Gao et al 2019]?

Currently, all the existing algorithms, the ones proposed in Gao et al 2019 and Esfandiari et al 2021, cannot close the gap. It is indeed a very intriguing question whether the gap can be closed or not (either by  a tighter lower bound or by a better algorithm). 

> Can you characterize the difference between similar results on UCB with these results?

Technically, the algorithm proposed in Gao et al 2019 is not UCB. It is an arm elimination algorithm that successively eliminates sub-optimal arms. For the optimal problem-dependent regret bound, our batch Thompson Sampling and their Batched Successive Elimination achieve the same regret bound in terms of $T$. Empirically, our algorithm achieves significantly lower regret. 

 We thank the reviewer for their detailed comments.

> My major concern here is the optimality of the results, especially the dependence of the regret on the number of batches. For example, in the MAB case, for the problem-dependent regret here is an extra N factor in the number of batches needed to achieve optimality; for the problem-independent bound, $O(N\log T)$ batches are needed instead of the $\Omega(\log\log T) $batches given in Gao et al., (2019). As such, I am not quite sure how to understand the advantage of the algorithm proposed (please correct me if I am wrong or misunderstood anything).

This is indeed a great question that needs further explanation in the paper. Note that our algorithm simultaneously achieves the optimum asymptotic/minimax regret and problem-dependent regret in $O(N\log T)$ number of batches. Indeed, it is impossible for any batch algorithm to achieve both types of regret in $O(\log\log T)$ number of batches as it violates the lower bound proposed in Theorem 3 (also see Corollary 2) in Gao et al., (2019). Notably, the $\Omega(\log\log T)$ batches given in Gao et al., (2019) does not achieve the optimal problem-dependent regret.  Any batch algorithm that aims to achieve both asymptotic/minimax regret and problem-dependent regret requires to have $O(\log T/\log\log T)$ number of batches. Our result is only $O(\log\log T)$ away from the lower bound in terms of $T$. Based on our empirical results, we believe that a linear dependence in $N$ for Thompson Sampling is unavoidable. Please see Fig 1 in the supplementary material. 


> Indeed, in general, it is hard to establish theoretical results for Thompson-Sampling-based algorithms, so it would help to illustrate why it is hard to establish tighter results in terms of the number of batches. On the other hand, Thompson-Sampling-based algorithms are often found to have a better performance numerically---so it may help to show more numerical evidence of the advantage of the algorithm.

As we mentioned earlier, our results in terms of the number of batches is quite tight.  Any batch algorithm that aims to achieve both asymptotic/minimax regret and problem-dependent regret simultaneously requires to have O(\log T/\log\log T) number of batches. Our result is only $O(\log\log T)$ away from the lower bound in terms of $T$. Moreover, currently there is no batch bandit algorithm that achieves the optimal problem-dependent regret bound in $O(\log T/\log\log T)$ number of batches. We have also provided empirical evidence regarding the performance of our batch Thomson Sampling method in Fig 1 of the paper. We showed, among other things, that B-TS achieves much better performance than fully sequential UCB. 


 We thank the review for their detailed remarks. 
> My main concern is that, given the existing research on Thompson sampling and batched stochastic bandits under the UCB algorithm, the novelty of the submission is marginal. The main difference between the author's B-TS in Algorithm 1 and the vanilla Beta Bernoulli TS lies in the doubling schedule (the "if... else" loop). The use of such a doubling schedule is not a new idea, for example it is covered in Exercise Questions 7.4, 7.5 in the book "Bandit Algorithms": https://tor-lattimore.com/downloads/book/book.pdf

We thank the review for the pointer. Indeed, UCB is long known to achieve low regret with $O(\log T)$ switches. However, this is not known at all for Thompson Sampling, for very good reasons. Indeed, if you keep the same action for the entire duration of the batch (as suggested by the reviewer), empirically, it results in much higher regret. Our doubling mechanism is quite different. It does not select the same action during a batch. Instead, it samples different arms based on the posterior distributions calculated from the previous batches. This is indeed crucial to achieve low regret with $O(\log T)$ batches and it is specifically designed for the Thompson Sampling. We can add the empirical result in the final version and we very much hope that the reviewer appreciates the subtlety and the difference between the two doubling mechanism methods. 

> Moreover, in Appendix A in the supplementary material for the analysis of B-TS, much of the technical content appears to be a straightforward corollary of Agrawal and Goyal 2017's TS paper. Altogether, the novelty in the algorithm design and analysis is below the bar for NeurIPS in my opinion. 

We believe that the simplicity of our algorithm and how it builds upon the previous work in a natural way is a plus. We intentionally kept the skeleton of the proofs similar to the previous work to emphasize where the differences lie and make it much simpler for the future work to build on our own. We do not feel that reinventing the wheel is a good idea. 

> Another major concern is that the authors only provide a regret bound for a specific value of batch size. Nevertheless, in existing works on batched bandits under UCB, a regret bound is provided for each batch size. Therefore, in comparison to these existing works, the generality of the results appears to be lacking.

The aim of this work is to minimize the number of batches while achieving low regret. It is quite well understood that this can only be done when the policy is dynamic and decides when to terminate the batch (as in our batch Thompson Sampling). There are certainly many concurrent work with the same goal, see for example “Double Explore-then-Commit: Asymptotic Optimality and Beyond”, COLT 2021 and ‘Almost Optimal Anytime Algorithm for Batched Multi-Armed Bandits”, ICML 2021. It is an interesting future direction to investigate the performance of Thompson Sampling when the number of batches is predetermined. Based on our empirical results, they achieve much higher regret. 

> in Theorem 4.6, the authors remark on the batch size used for achieving the $\tilde{O}(NT)$ bound, and they stated that their batch size is $O(N \log ⁡T)$. There seems to be a gap from the batch size $O(\log\⁡log ⁡T)$, which is known to be sufficient for achieving the $\tilde{O}(NT)$ regret bound (see Corollary in Gao et al 2019 https://arxiv.org/pdf/1904.01763.pdf for example)

We believe that the reviewer misunderstood the implications of the results by Gao et al 2019 in our setting. Theorem 4.3 of our paper shows that B-TS achieves the optimal problem-dependent regret in $O(N\log T)$ number of batches. Theorem 3 (Corollary 2) in Gao et al 2019 shows that to achieve the optimal problem-dependent regret bound, any batch algorithm has to have $O(\log T/\log\log T)$ number of batches. So in terms of $T$, our batch algorithm is only $O(\log\log T)$ away from the lower bound. Indeed, none of the existing algorithms, e.g., batch arm elimination or batch UCB, achieve the lower bound of $O(\log T/\log\log T)$ for the optimal problem-dependent regret bound. This point was mentioned in Remark 4.5 of our paper. 


 We thank the reviewer for the detailed remarks.

> The main weakness of this paper is the novelty and contribution part. In particular, the core contribution of this paper is the strategy of converting Thompson sampling into a batched algorithm, which seems novel and never done in the literature before. 

We believe that the simplicity of our algorithm and how naturally it builds upon the previous results is an advantage. We carefully mentioned all the nuances and difficulties in the proofs and how previous results should be modified. We intentionally kept the skeleton of the previous proofs as we do not need to reinvent the wheel. 

> For Remark 4.5, it is worth clarifying that the lower bound on the batch complexity (Ω(log⁡T/log⁡log⁡T)) proved in Gao et al., [2019] is for achieving the finite-time problem-dependent regret bound for multi-armed bandits. However, in order to achieve the problem-dependent asymptotic optimal regret as stated in Remark 4.5, it has been proved in [1] that constant batch complexity is sufficient for 2-armed bandits, and in [2] that a little more than O(log⁡log⁡T) batch complexity is sufficient. 
[1] "Double explore-then-commit: Asymptotic optimality and beyond." COLT, 2021. [2] "Almost Optimal Anytime Algorithm for Batched Multi-Armed Bandits." ICML, 2021.

We thank the review for bringing this concurrent work to our attention. We will cite them in the final version. 
Regarding [1], the authors only consider a 2-arm setting and asymptotic regret for the batch setting. As they clearly mention in their paper “For minimax and instance dependent regret bounds, Perchet et al. (2016) proved that any algorithm achieving the minimax optimality or instance dependent optimality will cost at least Ω(log log T ) or Ω(log T / log log T ) rounds respectively. How to extending our minimax/instance-dependent and asymptotic optimal Algorithm 3 to the batched bandit setting is an interesting open question.” Not only do we consider a different algorithm, namely Thompson Sampling, our results are also valid for the problem-dependent regret and almost achieve it. 
Regarding [2], the goal of this paper is to design batch algorithms for which the horizon T is known (previous work assumed that it is known). Even though this is an interesting result, it is orthogonal to our paper. Their proposed method and our Thompson Sampling strategy can be potentially combined to obtain an anytime algorithm for the batch multi-armed bandit. 

> For Algorithm 2, in Line 5, you need to keep track of the number of pulls of each arm. Does this mean the proposed algorithm does not work in the setting with infinite arms? In this setting, you may select different arms at each iteration and thus the number of arms selected could go to infinity.

The focus of this paper is in the finite arm setting. However, to extend the results to the infinite arm setting, we can potentially use the $\epsilon$-net idea proposed in “Regret Bounds for Batched Bandits”, AAAI 2021 (section 6.3). 


This paper studies the batched multi-armed bandit problem, where arms are pulled in a batch fashion and the rewards are only obtained at the end of the batch. The authors propose Thompson sampling based algorithms in the batch bandit setting, which enjoy logarithmic batch complexities. This improves their full sequential counterparts that have linear batch complexities. They proved that these algorithms with improved batch complexities also enjoy the same regret bounds as fully sequential algorithms. 
  This paper is well written and the algorithms are designed in a clear way. The proposed batched bandit strategy is general enough and shows its extension to different variants of Thompson sampling algorithms. The analysis seems to be solid and the results are correct based on my intuition (I did not check all the proofs and their correctness). 

The main weakness of this paper is the novelty and contribution part. In particular, the core contribution of this paper is the strategy of converting Thompson sampling into a batched algorithm, which seems novel and never done in the literature before. However, the other parts of the paper such as the minimax optimal thompson sampling (MOTS), MOTS-J for Gaussian rewards, and contextual TS are more or less a showcase of applying the proposed strategy to existing algorithms. This can also be seen from the proof analysis, where the majority of the proofs for these applications are similar to that in existing papers. 

For Remark 4.5, it is worth clarifying that the lower bound on the batch complexity ($\Omega(\log T/ \log \log T)$) proved in Gao et al., [2019] is for achieving the finite-time problem-dependent regret bound for multi-armed bandits. However, in order to achieve the problem-dependent asymptotic optimal regret as stated in Remark 4.5, it has been proved in [1] that constant batch complexity is sufficient for 2-armed bandits, and in [2] that a little more than $O(\log\log T)$ batch complexity is sufficient. 

[1] "Double explore-then-commit: Asymptotic optimality and beyond." COLT, 2021.
[2] "Almost Optimal Anytime Algorithm for Batched Multi-Armed Bandits." ICML, 2021. 

For Algorithm 2, in Line 5, you need to keep track of the number of pulls of each arm. Does this mean the proposed algorithm does not work in the setting with infinite arms? In this setting, you may select different arms at each iteration and thus the number of arms selected could go to infinity. 
 The main weakness of this paper might be that the algorithm design and analysis techniques are not very novel in the literature. I do not see negative societal impact directly related to this work.



The authors consider the Thompson sampling algorithm in the canonical stochastic bandit models, namely the N-armed bandit model and the linear contextual bandit model. They consider the case when the decision agent has to make the arm choice decisions in batches, and there are at most O(N log T) batches during the T-round horizon. The authors proposed batched versions of the TS algorithm by incorporating a doubling scheduling, and show that the batched version of the TS still achieves the state-of-the-art regret bounds by combining the analysis with existing works.   My main concern is that, given the existing research on Thompson sampling and batched stochastic bandits under the UCB algorithm, the novelty of the submission is marginal. The main difference between the author's B-TS in Algorithm 1 and the vanilla Beta Bernoulli TS lies in the doubling schedule (the "if... else" loop). The use of such a doubling schedule is not a new idea, for example it is covered in Exercise Questions 7.4, 7.5 in the book "Bandit Algorithms":

https://tor-lattimore.com/downloads/book/book.pdf

Moreover, in Appendix A in the supplementary material for the analysis of B-TS, much of the technical content appears to be a straightforward corollary of Agrawal and Goyal 2017's TS paper. Altogether, the novelty in the algorithm design and analysis is below the bar for NeurIPS in my opinion.

Another major concern is that the authors only provide a regret bound for a specific value of batch size. Nevertheless, in existing works on batched bandits under UCB, a regret bound is provided for each batch size. Therefore, in comparison to these existing works, the generality of the results appears to be lacking. 

Lastly, in Theorem 4.6, the authors remark on the batch size used for achieving the $\tilde{O}(\sqrt{NT})$ bound, and they stated that their batch size is $O(N \log T)$. There seems to be a gap from the batch size $O( \log \log T)$, which is known to be sufficient for achieving the $\tilde{O}(\sqrt{NT})$ regret bound (see Corollary in Gao et al 2019 https://arxiv.org/pdf/1904.01763.pdf for example) The author should have stated much more explicitly that, in existing research works on batched bandits under UCB, they provide a regret bound for every possible batch size $B\in \mathbb{N}$, whereas the authors only provide a regret bound for TS when the batch size is $O(N \log T)$.

This paper introduces a batched version of the Thompson Sampling algorithm. For the multi-arm bandit problem, within O(NlogT) batches, it is able to achieve a regret (both problem-dependent and problem-independent) that is the same order as the fully online TS; when substituting TS with MOTS, the batched-learning algorithm achieves the minimax optimal regret O(\sqrt{NT})., again within O(NlogT) batches. For the contextual-bandit problem, the batched-TS algorithm achieves the regret bound O(d^{3/2}\sqrt{T}) (up to logarithmic factors), again within O(NlogT) batches.
  The writing of this paper is clear and concise. One interesting point of this paper is that it proposes a batched algorithm based on dynamic grid design---note that in Gao et al., (2019), the lower bound is established w.r.t. dynamic grids but the optimal algorithm provided there in fact uses a static grid design. 

My major concern here is the optimality of the results, especially the dependence of the regret on the number of batches. For example, in the MAB case, for the problem-dependent regret here is an extra N factor in the number of batches needed to achieve optimality; for the problem-independent bound, O(NlogT) batches are needed instead of the \Omega(\log\logT) batches given in Gao et al., (2019). As such, I am not quite sure how to understand the advantage of the algorithm proposed (please correct me if I am wrong or misunderstood anything). 

Indeed, in general, it is hard to establish theoretical results for Thompson-Sampling-based algorithms, so it would help to illustrate why it is hard to establish tighter results in terms of the number of batches. On the other hand, Thompson-Sampling-based algorithms are often found to have a better performance numerically---so it may help to show more numerical evidence of the advantage of the algorithm.

 A discussion on the potential negative societal impact of this work should be added. 

Meta-review: While the problems studied in this paper look to be interesting, this paper has two weaknesses: (1) the bound $O(N\log T)$ batches is too large, compared with $O(\log T)$ or $O(\log T/\log\log T)$ using UCB or successive-elimination. The author's feedback appears to be attempting to minimize the importance of $N$ but only focus on $T$.   However, $N$ is often a significant consideration and can be very large in practice.  (2) It seems to be straightforward to combine the standard doubling method with Thompson sampling. Basically, if we can guarantee that the prior of each arm does not change by much in each round (which is true if we terminate a round when the number of pulls on a particular arm doubles), then we can easily mimic the proof of Agrawal&Goyal 2017 for (standard) Thompson sampling to obtain the regret bounds in the batched model.  In this regard, the novelty of this paper is limited.


Decision: accept