Title: Improved Bayes Regret Bounds for Multi-Task Hierarchical Bayesian Bandit Algorithms
Venue: NeurIPS 2024
Accepted: yes
Keywords: hierarchical Bayesian bandi/semi-bandit

multi-task bandit

Bayes regret bound

Thompson sampling algorithm

BayesUCB algorithm
Abstract: Hierarchical Bayesian bandit refers to the multi-task bandit problem in which bandit tasks are assumed to be drawn from the same distribution. In this work, we provide improved Bayes regret bounds for hierarchical Bayesian bandit algorithms in the multi-task linear bandit and semi-bandit settings. For the multi-task linear bandit, we first analyze the preexisting hierarchical Thompson sampling (HierTS) algorithm, and improve its gap-independent Bayes regret bound from $O(m\sqrt{n\log{n}\log{(mn)}})$ to $O(m\sqrt{n\log{n}})$ in the case of infinite action set, with $m$ being the number of tasks and $n$ the number of iterations per task. In the case of finite action set, we propose a novel hierarchical Bayesian bandit algorithm, named hierarchical BayesUCB (HierBayesUCB), that achieves the logarithmic but gap-dependent regret bound $O(m\log{(mn)}\log{n})$ under mild assumptions. All of the above regret bounds hold in many variants of hierarchical Bayesian linear bandit problem, including when the tasks are solved sequentially or concurrently. Furthermore, we extend the aforementioned HierTS and HierBayesUCB algorithms to the multi-task combinatorial semi-bandit setting. Concretely, our combinatorial HierTS algorithm attains comparable Bayes regret bound $O(m\sqrt{n}\log{n})$ with respect to the latest one. Moreover, our combinatorial HierBayesUCB yields a sharper Bayes regret bound $O(m\log{(mn)}\log{n})$. Experiments are conducted to validate the soundness of our theoretical results for multi-task bandit algorithms.

Improved Bayes Regret Bounds for Multi-Task Hierarchical Bayesian Bandit Algorithms

Abstract

Hierarchical Bayesian bandit refers to the multi-task bandit problem in which bandit tasks are assumed to be drawn from the same distribution. In this work, we provide improved Bayes regret bounds for hierarchical Bayesian bandit algorithms in the multi-task linear bandit and semi-bandit settings. For the multi-task linear bandit, we first analyze the preexisting hierarchical Thompson sampling (HierTS) algorithm, and improve its gap-independent Bayes regret bound from O ( m √ n log n log ( mn )) to O ( m √ n log n ) in the case of infinite action set, with m being the number of tasks and n the number of iterations per task. In the case of finite action set, we propose a novel hierarchical Bayesian bandit algorithm, named hierarchical BayesUCB (HierBayesUCB), that achieves the logarithmic but gap-dependent regret bound O ( m log ( mn ) log n ) under mild assumptions. All of the above regret bounds hold in many variants of hierarchical Bayesian linear bandit problem, including when the tasks are solved sequentially or concurrently. Furthermore, we extend the aforementioned HierTS and HierBayesUCB algorithms to the multi-task combinatorial semi-bandit setting. Concretely, our combinatorial HierTS algorithm attains comparable Bayes regret bound O ( m √ n log n ) with respect to the latest one. Moreover, our combinatorial HierBayesUCB yields a sharper Bayes regret bound O ( m log ( mn ) log n ) . Experiments are conducted to validate the soundness of our theoretical results for multi-task bandit algorithms.

1 Introduction

A stochastic bandit [26, 6, 27] is a sequential decision-making problem where at each round, an agent has to choose an action, and receives a stochastic reward without knowing its expected value. The gap between the cumulative reward of optimal actions in hindsight and the cumulative reward of agent is defined as regret . The goal is to minimize regret, through a combination of exploring different actions and exploiting those with high rewards in the past. Typical applications of bandit algorithms include news article recommendation [28], computational advertisement [20], and dynamic pricing [24]. For example, in news article recommendation, the agent must choose a news article for a user. The actions in this bandit setting are articles and the reward could be an indicator of a click from user.

When the agent has to solve multiple bandit tasks, many machine learning researchers resort to multi-task learning/meta-learning paradigm [8, 34] to benefit task adaptation. The existing works focused on the multi-task bandit problem can be categorized into three main groups: (1) The first group attempts to learn a low-dimensional representation shared by different bandit tasks, to derive a sharper cumulative regret bound than that derived by learning each task independently [19, 10]. (2) The second group leverages the similarity of contexts (e.g. the feature of actions) in bandit tasks to improve agent's ability to predict rewards in a new task [14, 36]. (3) The third group chooses to

maintain a meta-distribution over the hyper-parameters of within-task bandit algorithms (like TsallisINF [23], OFUL [9], and Thompson sampling [25, 7, 17]), and draws informative hyper-parameters from the meta-distribution for efficient regret minimization. Our work falls into the third group and formulates the problem of learning similar bandit tasks in a hierarchical Bayesian bandit model [17].

Specifically, in hierarchical Bayesian bandit setting, each bandit task is characterized by a task parameter. Different bandit task parameters are assumed to be independently and identically distributed according to the same distribution. At each round, the learning agent interacts with one or several bandit tasks, which correspond to the sequential and concurrent bandit settings respectively. Many existing works considered hierarchical Bayesian bandit problem, and proposed Thompson sampling [33] type algorithms to solve it [25, 7, 36]. The latest work [17] proposed hierarchical Thompson sampling (HierTS) algorithm and developed a gap-independent Bayes regret bound O ( m √ n log n log ( mn )) in the Gaussian linear bandit setting, where m is the number of bandit tasks and n the number of iterations per task. However, it is still unclear for us whether we can derive sharper regret bounds or how to extend hierarchical Bayesian bandit algorithms to the more general multi-task bandit setting.

In this work, we attempt to tackle the above two issues, by providing improved Bayes regret bounds for hierarchical Bayesian bandit algorithms in the multi-task Gaussian linear bandit and semi-bandit setting. Firstly, in the linear bandit setting, we improve the multi-task Bayes regret bound of HierTS to O ( m √ n log n ) in the case of infinite action set, strengthening the latest bound in [17, Thm 3] by a factor of O ( √ log ( mn )) . In the case of finite action set, we propose a novel hierarchical Bayesian bandit algorithm, named hierarchical BayesUCB (HierBayesUCB), that achieves the logarithmic but gap-dependent regret bound O ( m log ( mn ) log n ) under mild assumptions. All of the above regret bounds for linear bandit hold in both the sequential and concurrent setting. Secondly, we extend the aforementioned HierTS and HierBayesUCB algorithms to the multi-task Gaussian combinatorial semi-bandit setting. Concretely, our combinatorial HierTS algorithm attains comparable Bayes regret bound O ( m √ n log n ) with respect to the latest one in [7, Thm 6]. Moreover, our combinatorial HierBayesUCB yields a sharper but gap-dependent regret bound O ( m log ( mn ) log n ) . Extensive experiments in the Gaussian linear bandit setting are conducted to support our theoretical results.

Overall, our theoretical contributions are four-fold: (1) In the case of infinite action set, we provide a tighter Bayes regret bound O ( m √ n log n ) for HierTS. This bound improves the latest result by a factor of O ( √ log ( mn )) . (2) In the case of finite action set, we propose a novel HierBayesUCB algorithm, and provide gap-dependent logarithmic Bayes regret bound O ( m log ( mn ) log n ) for it. (3) We generalize the above regret bounds for linear bandit from sequential setting to the more challenging concurrent setting. (4) We extend both HierTS and HierBayesUCB algorithms to the more general multi-task combinatorial semi-bandit setting and derive improved Bayes regret bounds.

2 Related Work

Frequentist Regret Bounds for Stochastic Linear Bandit . In the frequentist stochastic bandit setting, we do not assume the bandit task parameter is sampled from a fixed distribution. The frequentist regret is thus for any fixed task parameter, without taking expectation over the distribution of task parameter. (1) In the case of finite action set: [5] for the first time investigated the stochastic linear bandit problem and proposed an algorithm with a frequentist regret of O ( √ dn log 3 / 2 n ) , where d is the dimension of action space and n is the number of rounds. [29] developed a new algorithm and improved the regret bound to O ( √ dn log n ) . [12] showed that the lower frequentist regret bound in the finite action set is Ω( √ dn ) . (2) In the case of infinite action set: Both [13] and [30] proposed algorithms that achieve O ( d √ n log 3 / 2 n ) regret. The regret bound was further improved in [1, 15] to O ( d √ n log n ) , by designing novel linear bandit algorithms or utilizing advanced martingale methods.

Bayes Regret Bounds for Bayesian Linear Bandit . In the Bayesian stochastic bandit setting, the Bayes regret is the expected cumulative regret whose expectation is taken over the draw of task parameter from a distribution. It is not difficult to see that the frequentist regret upper bound implies a Bayes regret upper bound, because the former holds for any task parameter. (1) When the action set is infinite: [30] showed that in the Gaussian linear bandit setting, the Bayes regret of any Bayesian bandit algorithm is lower bounded by Ω( √ n ) . [31] for the first time gave the Bayes regret bound of O ( d √ n log n ) for both Thompson sampling (TS) and BayesUCB [22] algorithms. Recently, [21] provided an improved Bayes regret O ( d √ n log n ) for TS algorithm with a concise proof. (2) When the action set is finite: [32] derived a tight regret bound of O ( √ dn ) for TS algorithm with

sub-Gaussian reward noise, via a novel information-theoretic approach. Recently, [3] developed a logarithmic Bayes regret bound O ( d 2 log 2 n ) for BayesUCB algorithm in the Gaussian bandit setting.

Frequentist Regret Bounds for Multi-Task Linear Bandit Problems . Under the representation learning paradigm, the frequentist regret bounds in [19, 10] for multi-task linear bandits scales as O ( m √ nk ) , where k is the dimension of low-dimensional representation. The expected frequentist regret upper bound for multi-task adversarial linear bandit in [23] is O ( m √ n log (1 + nV )) , with V being the similarity among multiple adversarial bandit tasks. Nevertheless, we should mention that all of these frequentist regret bounds for multi-task linear bandit problem are not tighter than Ω( m √ n ) .

Bayes Regret Bounds for Multi-Task Bayesian Linear Bandit/Semi-Bandit . The most related works to ours are [25, 7, 17], which provided hierarchical-type Thompson sampling algorithms for multi-task bandit and derived Bayes regret bounds in the Gaussian reward setting. We list these Bayes regret bounds in Table 1 for direct comparisons. Among them, the latest work [17] proposed the HierTS algorithm and obtained its regret bound of O ( m √ n log n log ( mn )) , [7] derived the first Bayes regret bound for multi-task hierarchical Bayesian semi-bandit algorithm. In this work, we provide for HierTS improved Bayes regret bound of O ( m √ n log n ) in Theorem 5.1. We also propose a novel HierBayesUCB algorithm that achieves logarithmic regret bound O ( m log n log ( nm )) . We finally extend HierTS and HierBayesUCB to the semi-bandit setting and derive improved theoretical results. Other works utilized action features or structure information to derive Bayes regret bounds for multi-task bandit [36, 37], e.g. the Bayes regret bound in [36] is O ( m √ n log n + m log 2 ( mn )) .

Hierarchical Bayesian Bandit Algorithms . Hierarchical Bayesian bandit algorithm was first proposed by [25] to solve multi-task bandit problems. More hierarchical-type Thompson sampling algorithms based on multi-task/meta learning frameworks were developed with improved theoretical guarantees and empirical performance [7, 17, 36, 37]. There also existed other works investigating hierarchical Bayesian bandit algorithms within the single-task bandit setting. For example, [16] extended the two-level hierarchical Bayesian bandit framework to the deeper multiple-level hierarchial Bayesian bandit framework. [2] generalized the single-effect-parameter HierTS algorithm (i.e. the action parameter is centered at a single latent variable) to the mixed-effect bandit framework where each action is associated with a parameter that depends upon one or multiple effect parameters.

3 Problem Setting

For any positive integer n , denote [ n ] = { 1 , 2 , ..., n } for brevity. For any square matrix M ∈ R d × d , denote λ 1 ( M ) , λ d ( M ) as its maximum and minimum eigenvalues respectively, denote κ ( M ) = λ 1 ( M ) /λ d ( M ) as its condition number. The action set A ⊆ B (0 , B ) ⊆ R d is assumed to be compact for some positive constant B > 0 , where B (0 , B ) is the closed ball centered at the origin. We use 〈 , 〉 to denote the inner-product between vectors, use w ( a ) or w a to denote the a -th element of vector w .

Single-Task Bandit . A stochastic bandit problem is characterized by an unknown parameter θ with an action set A . Each action a ∈ A under the bandit instance θ is associated with a reward distribution P ( ·| a, θ ) . The reward mean of action a under θ is denoted as r ( a ; θ ) = E Y ∼ P ( ·| a ; θ ) [ Y ] , and the optimal action under θ is denoted as A ∗ = arg max a ∈A r ( a ; θ ) . In the stochastic linear bandit setting, the mean reward of action a ∈ A is r ( a, θ ) = a glyph[latticetop] θ . In Bayesian bandit problem, we further assume that the task parameter θ is independently and identically distributed (i.i.d.) according to a task parameter distribution P ( ·| µ ∗ ) , which is characterized by an unknown hyper-parameter µ ∗ .

Single-Task Semi-Bandit . In the semi-bandit setting, the action set A = [ K ] is a set of finite items. A = { A ⊆ A : | A | ≤ L } is a family of subsets of A with up to L items, where L ≤ K . w ∈ R K is a weight vector. The weight of a set A ∈ A is defined as ∑ a ∈ A w ( a ) . We assume that the weights w are drawn i.i.d. from a distribution, and the mean weight is denoted as ¯ w = E [ w ] . Following previous work [38], we focus on the coherent case [39] which assumes that the agent knows a feature matrix Φ ∈ R K × d , such that ¯ w = Φ θ , where θ is the task parameter drawn from P ( ·| µ ∗ ) . The reward of a subset A ∈ A under the bandit instance θ is defined as r ( A ; θ ) = ∑ a ∈ A (Φ θ )( a ) = ∑ a ∈ A 〈 Φ a , θ 〉 , where Φ a is the transpose of the a -th row of matrix Φ . We further assume that ‖ Φ a ‖ ≤ B , ∀ a ∈ A .

Hierarchical Bayesian Multi-Task Bandit/Semi-Bandit . In this setting, the agent interacts with m tasks sequentially or concurrently. First, sample the hyper-parameter µ ∗ from a hyper-prior Q . Then, for each task s ∈ [ m ] , sample the task parameter θ s, ∗ independently from distribution P ( ·| µ ∗ ) . The learning process can be detailed as follows. At round t ≥ 1 , the agent interacts with a set

of tasks S t ⊆ [ m ] , takes a series of actions A t = ( A s,t ) s ∈S t , and receives a series of rewards Y t = ( Y s,t ) s ∈S t . In the bandit setting, Y s,t ∼ P ( ·| A s,t ; θ s, ∗ ) is a stochastic reward obtained by taking action A s,t in task s ∈ S t ; in the semi-bandit setting, Y s,t = { ˆ w s,t ( a ) } a ∈ A s,t is a series of stochastic rewards, where ˆ w s,t = ¯ w s + η s,t , ¯ w s = Φ θ s, ∗ , and η s,t is a K -dimensional random noise. The full hierarchical Bayesian bandit/semi-bandit model in the m -task learning setting is exhibited as follow:

Therefore, the goal of the agent in hierarchical Bayesian multi-task bandit/semi-bandit setting is to interact with m tasks efficiently and minimize the following cumulative multi-task Bayes regret :

where A s, ∗ = arg max a ∈A r ( a ; θ s, ∗ ) is the optimal action for task s ∈ [ m ] in the bandit setting, and A s, ∗ ∈ arg max A ∈ A r ( A ; θ s, ∗ ) is the optimal subset for task s ∈ [ m ] in the semi-bandit setting. The expectation is taken over µ ∗ , all task parameters ( θ s, ∗ ) s ∈ [ m ] , all actions ( A t ) t ≥ 1 , all stochastic rewards ( Y t ) t ≥ 1 . We further assume that the action set A is the same across different tasks for ease of exposition, and assume that the learning agent interacts with any task s ∈ [ m ] for at most n rounds for convenient comparison with exiting regret upper bounds for multi-task bandit/semi-bandit problem.

4 Algorithm

Denote H s,t =(( A s,glyph[lscript] , Y s,glyph[lscript] )) glyph[lscript]<t,s ∈S t as the history of all interactions of agent with task s ∈ [ m ] , and H t =( H s,t ) s ∈ [ m ] as the whole interaction history up to round t . We next introduce the specific form of Hierarchical Thompson Sampling (HierTS) and Hierarchical BayesUCB (HierBayesUCB) algorithms in the multi-task Bayesian linear bandit and semi-bandit settings, and instantiate these two algorithms to the multi-task Gaussian linear bandit (Algorithm 1) and semi-bandit (Algorithm 2) problems.

4.1 Hierarchical Thompson Sampling and Hierarchical BayesUCB

At round t , hierarchical Bayesian bandit algorithm samples a hyper-parameter µ t from the hyperposterior Q t defined as Q t ( µ ) = P ( µ ∗ = µ | H t ) , and then interacts with tasks S t ⊂ [ m ] . Next, we give details of bandit algorithms, and details of semi-bandit algorithms are deferred to Section 5.4.

Hierarchical Thompson Sampling . For any task s ∈ S t , HierTS samples task parameter θ s,t from the distribution P s,t ( θ | µ t ) glyph[defines] P ( θ s, ∗ = θ | µ ∗ = µ t , H s,t ) and takes the action A s,t = arg max a ∈A a glyph[latticetop] θ s,t , where P s,t ( θ | µ t ) is only conditioned on H s,t due to the independence between task parameter θ s, ∗ and other task histories. This process clearly samples bandit instance θ s,t from the true posterior P ( θ s, ∗ = θ | H t ) , which is equivalent to the form: ∫ P ( θ s, ∗ = θ, µ ∗ = µ | H t )d µ = ∫ P s,t ( θ | µ ) Q t ( µ )d µ , where P s,t ( θ | µ ) ∝ L s,t ( θ ) P ( θ | µ ) is the posterior probability, L s,t ( θ )= ∏ ( a,y ) ∈ H s,t P ( y | a ; θ ) is the likelihood function, P ( θ | µ ) is the prior probability by Bayes rule.

Hierarchical BayesUCB . For any task s ∈ S t in round t , HierBayesUCB computes the upper confidence bound U t,s,a = a glyph[latticetop] ˆ µ s,t + √ 2 log 1 δ ‖ a ‖ ˆ Σ s,t for any a ∈ A , where ˆ µ s,t and ˆ Σ s,t are the expectation and covariance of the distribution (i.e. P ( θ s, ∗ = θ | H t ) ) of θ s, ∗ conditioned on the history H t , and then takes action with the highest upper confidence bound : A s,t ← arg max a ∈A U t,s,a .

4.2 Multi-Task Gaussian Linear Bandit and Semi-Bandit

The hierarchical Gaussian environment is generated as follow. In the multi-task linear bandit setting: (1) µ ∗ ∼ N ( µ q , Σ q ) , (2) θ s, ∗ | µ ∗ ∼ N ( µ ∗ , Σ 0 ) , ∀ s ∈ [ m ] , (3) Y s,t | A s,t , θ s, ∗ ∼ N ( A glyph[latticetop] s,t θ s, ∗ , σ 2 ) , ∀ t ≥ 1 , s ∈ S t ; In the semi-bandit setting, the only difference lies in step (3) where Y s,t,a | A s,t , θ s, ∗ ∼ N ( 〈 Φ a , θ s, ∗ 〉 , σ 2 ) for any a ∈ A s,t . Here, µ q , µ ∗ , θ s, ∗ are d -dimensional vectors; Σ q , Σ 0 ∈ R d × d are positive semi-definite covariance matrices. In the above two settings, the reward noise can be regarded as N (0 , σ 2 ) . In the following theoretical analysis sections, we assume that all of µ q , Σ q , Σ 0 and σ are known by the agent to guarantee an analytically tractable posterior.

Concretely, using some basic algebraic computations in hierarchical Gaussian model (e.g. see [25, Appendix D]), we can obtain the closed-form hyper-posterior in round t as Q t ( µ ) = N ( µ ; ¯ µ t , ¯ Σ t ) , where the expectation ¯ µ t and the covariance matrix ¯ Σ t of Q t ( µ ) have the following explicit forms:

Algorithm 1 Hierarchical Bayesian Algorithms for Multi-Task Linear Bandit Setting Algorithm 2 Hierarchical Bayesian Algorithms for Multi-Task Combinatorial Semi-Bandit Setting

Here, in the bandit setting G s,t = σ -2 ∑ glyph[lscript]<t 1 { s ∈ S glyph[lscript] } A s,glyph[lscript] A glyph[latticetop] s,glyph[lscript] and B s,t = σ -2 ∑ glyph[lscript]<t 1 { s ∈ S glyph[lscript] } A s,glyph[lscript] Y s,glyph[lscript] ; in the semi-bandit setting G s,t = σ -2 ∑ glyph[lscript]<t 1 { s ∈ S glyph[lscript] } ( ∑ a ∈ A s,glyph[lscript] Φ a Φ glyph[latticetop] a ) and B s,t = σ -2 ∑ glyph[lscript]<t 1 { s ∈ S glyph[lscript] } ( ∑ a ∈ A s,glyph[lscript] Φ a ˆ w s,t ( a )) . After the hyper-parameter µ t is sampled from Q t ( µ ) , we sample task parameter θ s,t ∼ N ( θ ; ˜ µ s,t , ˜ Σ s,t ) for task s , where ˜ µ s,t = ˜ Σ s,t (Σ -1 0 µ t + B s,t ) is the posterior mean, ˜ Σ -1 s,t = Σ -1 0 + G s,t the posterior covariance matrix. Such posterior of a linear model is obtained with a Gaussian prior N ( µ t , Σ 0 ) and Gaussian observations ( Y s,glyph[lscript] ) glyph[lscript]<t,s ∈S glyph[lscript] by Bayes rule.

On the other hand, we also need to handle P ( θ s, ∗ = θ | H t ) . It is not difficult to see that, in the multi-task Gaussian linear bandit/semi-bandit setting, θ s, ∗ | H t is Gaussian and denoted as P ( θ s, ∗ = θ | H t )= N ( θ ; ˆ µ s,t , ˆ Σ s,t ) . According to Lemma B.1, ˆ µ s,t and ˆ Σ s,t have the following explicit forms:

5 Bayes Regret Bounds

In this section, we provide improved regret bounds of hierarchical Bayesian bandit algorithms for multi-task Gaussian linear bandit/semi-bandit problem. Concretely, we provide improved analysis for HierTS in the sequential linear bandit setting (Sections 5.1), propose a novel HierBayesUCB bandit algorithm with logarithmic regret guarantee (Section 5.2), develop regret bounds for these two algorithms in the concurrent linear bandit setting (Section 5.3), and finally extend these two algorithms to the semi-bandit setting (Section 5.4) with improved regret bounds. In the proof for our theoretical results, the most important step is to give an upper bound on the so-called posterior variance V m,n , which in the multi-task linear bandit setting is defined and upper bounded as follow:

Although the above bound on V m,n achieves the same order (w.r.t. m,n and d ) as that in the latest bound of [17, Sect B], our bound has a smaller multiplicative factor (see more details in Table 4). In the multi-task semi-bandit setting, the posterior variance is V m,n glyph[defines] E ∑ t ≥ 1 ∑ s ∈S t ∑ a ∈ A s,t ‖ Φ a ‖ 2 ˆ Σ s,t and can be bounded in a similar way. To finish the whole proof, our strategy consists of two main steps: (1) The first step is to transform the multi-task Bayes regret BR ( m,n ) into an intermediate regret upper bound that involves the posterior variance V m,n as the dominant term. (2) The second step is to bound V m,n with Eq. (4). Combining the results in steps (1) and (2) yields Bayes regret bound for multi-task hierarchical Bayesian bandit/semi-bandit algorithms. Detailed comparisons between our regret bounds and others in the bandit setting are shown in Table 1. Next, we define c 1 = σ 2 + B 2 λ 1 (Σ 0 ) , c 2 = σ 2 + B 2 λ 1 (Σ 0 )+ B 2 λ 1 (Σ q ) κ (Σ 0 ) to be used through the whole Section 5.

5.1 Improved Regret Bound for HierTS in the Sequential Bandit Setting

In the sequential bandit setting, |S t | = 1 . Then, conditioned on H t , it is not difficult to see that in Bayes regret, each term E [ θ glyph[latticetop] s, ∗ A s, ∗ -θ glyph[latticetop] s, ∗ A s,t | H t ] = E [ ( θ s, ∗ -ˆ µ s,t ) glyph[latticetop] A s, ∗ ∣ ∣ H t ] , and we use a novel Cauchy-Schwartz type inequality from [21, Prop 2] to bound E [ ( θ s, ∗ -ˆ µ s,t ) glyph[latticetop] A s, ∗ ∣ ∣ H t ] , leading to BR ( m,n ) ≤ E [ ∑ t ≥ 1 ∑ s ∈S t √ d E [( ( θ s, ∗ -ˆ µ s,t ) glyph[latticetop] A s,t ) 2 ∣ ∣ H t ] ] . Expand the expression in the right hand side of the above inequality, we then have BR ( m,n ) ≤ E ∑ t,s ∈S t √ dA glyph[latticetop] s,t ˆ Σ s,t A s,t ≤ √ dmn V m,n , reducing the Bayes regret bound to the posterior variance bound problem. Recalling Eq. (4) achieves our first improved Bayes regret upper bound in the sequential linear bandit setting.

Theorem 5.1 (Near-Optimal Sequential Regret) Let |S t | = 1 for any round t . Then in the multi-task Gaussian linear bandit setting, the Bayes regret upper bound of HierTS is as follow:

Our explanations for the above sequential regret bound are three-fold: (1) The term md √ nc 1 log (1 + n/d ) represents the regret bound for solving m bandit tasks, whose parameters θ s, ∗ are drawn i.i.d. from the prior distribution N ( µ ∗ , Σ 0 ) . Under this assumption, no task provides information for any other task, and hence this bound is linear in m . Similar observation was also pointed out by [25, 7, 17]. (2) The term d √ mnc 2 log ( 1+ m Tr(Σ q Σ -1 0 ) /d ) represents the regret bound for learning the hyper-parameter µ ∗ . Such bound is sublinear in m and is not a dominant term when m is large. (3) For a large m , the averaged Bayes regret bound across m tasks is of BR ( m,n ) /m = O ( d √ n log n ) , and strengthens the latest averaged bound O ( d √ n log n ) in [17, Thm 3] by a factor √ log n . Besides, since the lower Bayes regret bound for any Bayesian bandit algorithm is Ω( d √ n ) [30], our task-averaged Bayes regret bound is within O ( √ log n ) of optimality and hence is called 'Near-Optimal' sequential regret bound. We further make a detailed comparison between our regret bound in Theorem 5.1 and the regret bound [17, Thm 3] in the following remark.

Remark 5.1 (Improvements of Our Theorem 5.1 over the Latest One) Our sequential regret bound has two improvements over the latest one in [17, Thm 3, shown in Table 1]: (1) We remove the additional √ log ( mn ) factor in both the regret bound for solving m bandit tasks and the regret bound for learning the hyper-parameter µ ∗ . (2) In the regret bound for learning hyper-parameter µ ∗ , [17] has a multiplicative factor κ 2 (Σ 0 ) , whereas our multiplicative factor is κ (Σ 0 ) . Such improvement is achieved by using technical matrix analysis proposed in Lemma C.1. and explained in Remark A.1.

5.2 Logarithmic Regret Bound for HierBayesUCB in the Sequential Bandit Setting

In this section, we attempt to provide further improved Bayes regret bounds for hierarchical bandit algorithms in the sequential bandit setting. Because the task averaged Bayes regret bound in Theorem 5.1 is near optimal, it is not easy to derive improved Bayes regret bounds under the same assumptions. Therefore, we further assume that the action set A is finite, and propose a novel

hierarchical Bayesian bandit algorithm, named Hierarchical BayesUCB (HierBayesUCB), for multitask linear bandit problem. The pseudo-code of our proposed algorithm is shown in Algorithm 1.

Next, we introduce some necessary notations. Let ∆ s,t = θ glyph[latticetop] s, ∗ ( A s, ∗ -A s,t ) , ∆ s, min = min a ∈A\{ A s, ∗ } ( θ glyph[latticetop] s, ∗ A s, ∗ -θ glyph[latticetop] s, ∗ a ) , ∆ min = min s ∈ [ m ] ∆ s, min . For any glyph[epsilon1] > 0 , let ∆ glyph[epsilon1] min = max { glyph[epsilon1], ∆ min } . Define the event E s,t = {∀ a ∈ A : | a glyph[latticetop] ( θ s, ∗ -ˆ µ s,t ) | ≤ √ 2 log 1 δ ‖ a ‖ ˆ Σ s,t } . Then, analogous to [3], we decompose the Bayes regret BR ( m,n ) = E ∑ t ≥ 1 ∑ s ∈S t ∆ s,t into three terms: E ∑ t ≥ 1 ,s ∈S t ∆ s,t [ 1 { ∆ s,t ≥ glyph[epsilon1],E s,t } + 1 { ∆ s,t < glyph[epsilon1],E s,t } + 1 { ¯ E s,t } ] . We can bound the last two terms trivially with mn [ glyph[epsilon1] + 2 δ ( max t,s | ∆ s,t | ) · |A| ] . For the first term, we use the fact that A s,t | H t i.i.d. ∼ A s, ∗ | H t , as well as the Upper Confidence Bound (UCB) technique to reduce it to an intermediate upper bound ( ∑ t ≥ 1 ,s ∈S t ‖ A s,t ‖ 2 ˆ Σ s,t log 1 δ ) / min s,t | ∆ s,t | . Combining the upper bound over V m,n in Eq. (4), HierBayesUCB can achieve the following logarithmic Bayes regret bound in the sequential bandit setting (the logarithmic bound can be extended to the concurrent setting).

Theorem 5.2 (Logarithmic Sequential Regret of HierBayesUCB) Let |S t | = 1 for any round t , and the action set A is finite with |A| < ∞ . Then in the multi-task Gaussian linear bandit setting, for any δ ∈ (0 , 1) , glyph[epsilon1] > 0 , the Bayes regret BR ( m,n ) of HierBayesUCB is upper bounded by

We give more explanations for the above sequential regret in terms of the following five aspects: (1) If let δ = 1 / ( mn ) , glyph[epsilon1] = 1 / ( mn ) and ∆ min >> glyph[epsilon1] , the above sequential regret bound is of O ( log ( mn )( md log ( n d ) + d log ( m d )) ) . The term O ( md log ( mn ) log ( n d )) represents the regret bound for solving m bandit tasks and is linear in m . Such bound is sharper than the corresponding bound O ( md √ n log ( n d )) in our Theorem 5.1 by a multiplicative factor O ( √ log ( n/d ) /n log ( mn )) , which is less than 1 especially when m ≤ n . (2) The term O ( d log ( mn ) log ( m d )) represents the regret bound for learning the hyper-parameter µ ∗ , and its contribution to the Bayes regret bound can be negligible. Besides, this bound is sharper than the bound O ( d √ mn log ( m/d )) in our Theorem 5.1. (3) The averaged Bayes regret bound across m tasks can be regarded as BR ( m,n ) /m = O ( d log ( mn ) log n ) , which is logarithmic in n . Therefore, we call our regret bound as 'Logarithmic' sequential regret bound. Moreover, if there exists a fixed positive integer i << n , such that m ≤ n i , then our task-averaged Bayes regret BR ( m,n ) /m = O ( d E [ 1 ∆ glyph[epsilon1] min ] log 2 n ) matches the latest single-task Bayes regret bound in [3, Thm 5] and is remarkably similar to the frequentist regret O ( d ∆ -1 min log 2 n ) in [1, Thm 5] . (4) We can obtain sharper bounds by setting δ, glyph[epsilon1] as different values. For example, by setting δ = 1 /n , our regret bound becomes O ([ mnglyph[epsilon1] + m ] + log n ∆ glyph[epsilon1] min m log n ) , which is of order O ( m log 2 n ) if we set glyph[epsilon1] = 1 / ( mn ) and the gap ∆ min >> glyph[epsilon1] is large. (5) We also need to point out that, the Bayes regret bound in Theorem 5.2 scales with E [ 1 ∆ glyph[epsilon1] min ] . If the gap ∆ min ≤ 1 / ( mn ) , then ∆ glyph[epsilon1] min = 1 / ( mn ) and this may cause a large Bayes regret upper bound.

5.3 Improved Regret Bounds of HierTS and HierBayesUCB in the Concurrent Bandit Setting

In the concurrent bandit setting, there exists a positive integer L ≤ m , such that 1 ≤ |S t | ≤ L . The concurrent bandit setting is thus more challenging than the sequential bandit setting, because the agent in the concurrent setting needs to interact with multiple bandit tasks in parallel at each round t ≥ 1 , and the hyper-posterior Q t will not be updated until the end of round t . Therefore, we need to make an additional assumption on the action space A as follow to facilitate our theoretical analysis.

This assumption is also used in previous works [7, 17] for hierarchical Bayesian linear bandit. It indicates that ∑ d i =1 a i a glyph[latticetop] i is a positive definite matrix, and does not weaken the generality of our theoretical results. Actually, if R d is not spanned by actions in A , we can project A into a subspace where the assumption holds. We also need to modify the HierTS algorithm to let the agent take the basic actions { a i } d i =1 for the first d interactions in any task s ∈ [ m ] . This modification guarantees that the agent explores all directions within the task. Such exploration is very similar to the initialization method in UCB type K -arm bandit algorithms [6, 4], which choose to pull each arm in the first K rounds. Define c 3 = 1 + B 2 σ -2 κ (Σ 0 ) [ λ 1 (Σ 0 ) + σ 2 /β ] that will be used throughout the concurrent

setting. Then, analogous to the proof for Theorem 5.1, we bound √ mn V m,n with a more refined analysis, achieving the following improved Bayes regret bound for HierTS in the concurrent setting.

Theorem 5.3 Under Assumption 5.1, let 1 ≤ |S t | ≤ L for any round t ≥ 1 . Then in the multi-task Gaussian linear bandit setting, the Bayes regret BR ( m,n ) of HierTS is upper bounded by

The concurrent regret bound in Theorem 5.3 achieves almost the same order (w.r.t. m,n,d ) as the sequential regret bound in Theorem 5.1, but differs in two aspects: (1) The bound for learning m i.i.d. bandit tasks has an additional term Bmd √ λ 1 (Σ 0 +Σ q )( √ d + ‖ µ q ‖ ˆ Σ -1 s, 1 ) . This is due to the fact that we take the basic actions { a i } d i =1 first for each task s ∈ [ m ] in the modified HierTS algorithm. (2) The bound for learning the hyper-parameter µ ∗ has an additional multiplicative factor c 3 . This is the price for deriving regret bounds in the concurrent setting. Nevertheless, when compared with the latest concurrent regret bound in [17, Thm 4] for HierTS, our concurrent regret bound in Theorem 5.3 removes the √ log ( mn ) factor in both the regret bound for learning m bandit tasks and the regret bound for learning hyper-parameter µ ∗ . Detailed comparisons between different concurrent regret bounds for multi-task linear bandit setting are listed in Table 3. Furthermore, utilizing the proof strategy to demonstrate the logarithmic sequential regret for HierBayesUCB in our Theorem 5.2, we can analogously develop a logarithmic concurrent regret upper bound for HierBayesUCB algorithm, which is deferred to our Theorem C.2 in Appendix C due to the limited space of the main paper.

5.4 Improved Regret Bounds for HierTS and HierBayesUCB in the Semi-Bandit Setting

In this section, we extend the HierTS and HierBayesUCB algorithms to the multi-task Gaussian combinatorial semi-bandit setting. The pseudo-code of them is shown in Algorithm 2. Algorithm 2 is very similar to Algorithm 1 (i.e. the multi-task linear bandit algorithms), except that the combinatorial HierTS in Algorithm 2 uses the approximation/randomized algorithm ORACLE to solve combinatorial problem A ∗ ∈ arg max A ∈ A ∑ a ∈ A w ( a ) and denotes the solution as A ∗ = ORACLE( A , A , w ) . We adopt the ORACLE operator as in the seminal works [11, 38] to guarantee the efficiency of combinatorial HierTS semi-bandit algorithm. In this section, we only consider the sequential semi-bandit setting (i.e. |S t | = 1 ) for ease of presentation, and our results can be extended to the concurrent semi-bandit setting. Then, define c 4 = σ 2 + B 2 Lλ 1 (Σ 0 ) + B 2 λ 1 (Σ q ) κ (Σ 0 ) , we first derive the Bayes regret upper bound for combinatorial HierTS algorithm in the sequential semi-bandit setting.

Theorem 5.4 Let |S t | = 1 for any t ≥ 1 . Let c ≥ √ 2 ln ( nKBλ 1 (Σ 0 ) √ 2 π ) , then in the multi-task Gaussian semi-bandit setting, the Bayes regret upper bound of combinatorial HierTS is:

Detailed comparisons between different Bayes regret bounds for multi-task semi-bandit problem are listed in Table 2. We can see that, in our Theorem 5.4, both the regret bound O ( m √ n log n ) for learning m tasks and the regret bound O ( √ mn log m log n ) for learning hyper-parameter µ ∗ can achieve the same order (w.r.t. m and n ) when compared with the latest bound in [7, Thm 6]. Besides, our Bayes regret bound is logarithmic in the number K of items, whereas the Bayes regret bound in

[7, Thm 6] is sublinear in K . Therefore, our regret bound becomes sharper when the size of action set is very large, e.g. K>>L . Next, we derive a gap-dependent logarithmic multi-task Bayes regret bound for our proposed combinatorial HierBayesUCB algorithm in the sequential semi-bandit setting.

Theorem 5.5 Let |S t | = 1 for any t ≥ 1 . Then for any glyph[epsilon1] > 0 , δ ∈ (0 , 1) , in the multi-task Gaussian semi-bandit setting, the Bayes regret BR ( m,n ) of combinatorial HierBayesUCB is bounded by

In Theorem 5.5, if we set δ = 1 / ( mnK ) , glyph[epsilon1] = 1 / ( mn ) , and ∆ min >> glyph[epsilon1] , then the regret bound O ( m log n log ( mn ) ) for learning m tasks is logarithmic in n . Such bound is sharper than the latest one O ( m √ n log n ) in [7, Thm 6] for multi-task semi-bandit. The regret bound O (log m log ( mn )) for learning hyper-parameter µ ∗ is also sharper than that of O ( √ mn log m log n ) in [7, Thm 6]. Besides, since δ =1 / ( mnK ) , the whole Bayes regret bound is also logarithmic in the number K of items. Nevertheless, we should point out that our bounds hold for the multi-task semi-bandits with linear generalization, but [7] focuses on the multi-task K -arm semi-bandits without feature matrix Φ .

5.5 Technical Novelties for Deriving Improved Regret Bounds

In this section, we summarize our technical novelties in terms of the following three aspects:

(1) For the improved regret bound for HierTS in Theorem 5.1: our proof has three novelties: (i) We apply a novel Cauchy-Schwartz type inequality in Lemma A.2 to bound E [ ( θ s, ∗ -ˆ µ s,t ) glyph[latticetop] A s, ∗ ∣ ∣ H t ] ≤

(ii) Weuse a more technical positive semi-definite matrix decomposition analysis (i.e. our Lemma A.1) to reduce the multiplicative factor κ 2 (Σ 0 ) to κ (Σ 0 ) . (iii) Define a new matrix ˜ X s,t such that the denominator in the regret is σ 2 + B 2 λ 1 (Σ 0 ) , not just σ 2 , avoiding the case that the variance serves alone as the denominator. Such technical novelties are also listed explicitly in Table 4.

(2) For the improved regret bound for HierBayesUCB in Theorem 5.2 in the sequential bandit setting: our novelty lies in decomposing the Bayes regret BR ( m,n ) = E ∑ t ≥ 1 ∑ s ∈S t ∆ s,t into three terms:

and bounding the first term with a new method as well as the property of BayesUCB algorithm as

resulting in the final improved gap-dependent regret bound for HierBayesUCB as follows

(3) For the improved regret bounds for HierTS and HierBayesUCB in the concurrent setting and in the semi-bandit setting: besides the aforementioned technical novelties in (1) and (2) , the additional technical novelty lies in leveraging more refined analysis (e.g. using Woodbury matrix identity) to bound the gap between matrices ¯ Σ -1 t +1 and ¯ Σ -1 t (more details is shown in Lemma C.1 and Eq. (6)).

6 Experiments

In this section, we conduct experiments in the linear bandit setting to verify our theoretical results. Specifically, we show the influence of hyper-parameters (e.g. m,n,L ) to the multi-task Bayes regret of HierTS and HierBayesUCB, to validate the consistency between their regret bounds and practical performance. Besides, we compare the performance between our algorithms and other baselines, to show the effectiveness of hierarchical Bayesian bandit algorithms in the multi-task bandit setting.

0

2

4

6

8

10

Number of Tasks

m

0

50

100

150

200

Regret

Linear Bandit (

d

= 4,

σ

q

= 1.0,

L

=1)

n

=100

n

=200

n

=300

n

=400

(a) Regrets w.r.t. different

m

2

4

6

8

10

Number of Concurrent Tasks

L

0

100

200

300

400

Regret

Linear Bandit (

m

= 10,

σ

q

= 0.5)

d

=2

d

=4

d

=8

(b) Regrets w.r.t. different

L

0

100

200

300

400

Round

t

0

250

500

750

1000

Regret

Linear Bandit (

d

= 4,

m

= 10,

L

=5)

σ

q

=4

σ

q

=3

σ

q

=2

σ

q

=1

σ

q

=0.1

(c) Regrets w.r.t. different

σ

q

0

100

200

300

400

Round

t

0

200

400

Regret

Linear Bandit (

d

= 4,

m

= 10,

L

=5)

σ

0

=0.5

σ

0

=0.4

σ

0

=0.3

σ

0

=0.2

σ

0

=0.1

(d) Regrets w.r.t. different σ 0

0

100

200

300

400

Round

t

0

500

1000

Regret

Linear Bandit (

d

= 4,

m

= 10,

L

=5)

σ

=10

σ

=7

σ

=4

σ

=1

σ

=0.1

(e) Regrets w.r.t. different

σ

0

100

200

300

400

Round

t

0

100

200

300

Regret

Linear Bandit (

d

= 4,

σ

q

= 1.0)

OracleTS

TS

HierTS

HierBayesUCB

(f) Regrets of different algorithms

Experimental Setting . We follow the same experimental setting as that in [7, 17]. Concretely, we conduct linear bandit experiments with Gaussian reward. The synthetic problem is defined as follows. In most experiments, we set the number of total tasks as m = 10 , the dimension of action space as d = 4 , the number of concurrent tasks as L = 5 , the number of rounds as n = 200 m/L . We focus on the finite action space with |A| = 10 , and each action is sampled uniformly from [ -0 . 5 , 0 . 5] d . In hierarchical Bayesian model, we set the hyper-prior as zero-mean isotropic Gaussian distribution N ( µ q , Σ q ) = N ( 0 , Σ q ) , where Σ q = σ 2 q I d ; and set the task variance Σ 0 = σ 2 0 I d . Unless otherwise stated, we set σ q = 1 , σ 0 = 0 . 1 , σ 2 = 0 . 5 for each task in most experiments. We exhibit the regret performance of HierTS algorithm with respect to five hyper-parameters m,L,σ q , σ 0 , σ in Figure 1 (a)-(e) respectively. The regret performance of HierBayesUCB is shown in Figure 2 of Appendix F.

Besides, we compare HierTS/HierBayesUCB with other two TS type algorithms that do not learn the hyper-parameter µ ∗ in a hierarchical Bayesian model. The first baseline is the vanilla TS algorithm that samples task parameter θ s, ∗ from the marginal prior N ( µ q , Σ q +Σ 0 ) . The second baseline is an idealized TS algorithm that knows µ ∗ exactly and uses the true prior N ( µ ∗ , Σ 0 ) . We call the second baseline as OracleTS, since this TS algorithm accesses more information of µ ∗ than HierTS and vanilla TS algorithm. We show the regret performance of these four bandit algorithms in Figure 1 (f).

Experimental Results . From Figure 1, we can observe that: (1) In plot (a), the multi-task regret becomes larger with the increase of m and n , which is consistent with our regret upper bound in Theorems 5.1. (2) In plot (b), the regret increases with a higher dimension d . The number L of the concurrent tasks seems do not have a large impact on regret. (3) In plots (c)-(e), the regret decreases with a smaller variance (e.g. σ q , σ 0 and σ ) in hierarchical Bayesian model, validating the provable benefits of variance-reduction in regret minimization, which is revealed in our multi-task Bayes regret upper bounds. (4) The task-averaged regret of HierTS is tighter than that of single-task TS algorithm, empirically demonstrating the advantages of multi-task Bayesian bandit optimization paradigm over single-task bandit learning. (5) Our proposed HierBayesUCB achieves lower regret than HierTS.

7 Conclusions

This paper provides improved Bayes regret bounds for hierarchical Bayesian bandit algorithms in the multi-task Gaussian linear bandit and semi-bandit setting. For linear bandit problem: in the case of infinite action set, we strengthen the preexisting regret bound O ( m √ n log n log ( mn )) of HierTS to O ( m √ n log n ) by a factor of O ( √ log ( mn )) ; in the case of finite action set, we propose a novel HierBayesUCB algorithm that achieves logarithmic regret bound O ( m log ( mn ) log n ) under mild conditions. Our regret bounds in the bandit setting hold when the agent solves tasks sequentially or concurrently. Then, we extend the above HierTS and HierBayesUCB algorithms to the multi-task semi-bandit setting and derive improved regret bounds. The synthetic experiments further support our theoretical results. Our future work aims to extend our bounds to the sub-exponential bandit setting.

Acknowledgments and Disclosure of Funding

Jiechao sincerely appreciates the financial support from the People's Government of Guangzhou Municipality for his postdoctoral project. We thank all reviewers for their constructive suggestions to improve the quality of this paper. This work was supported in part by the National Key R&D Program of China (Grant No.2023YFF0725001), in part by the National Natural Science Foundation of China (Grant No.92370204), in part by the guangdong Basic and Applied Basic Research Foundation(Grant No.2023B1515120057), in part by Guangzhou-HKUST(GZ) Joint Funding Program (Grant No.2023A03J0008), Education Bureau of Guangzhou Municipality.

References

Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. In NeurIPS , pages 2312-2320, 2011.

I. Aouali, B. Kveton, and S. Katariya. Mixed-effect thompson sampling. In AISTATS , pages 2087-2115, 2023.

A. Atsidakou, B. Kveton, S. Katariya, C. Caramanis, and sujay sanghavi. Finite-time logarithmic bayes regret upper bounds. In NeurIPS , 2023.

J. Audibert and S. Bubeck. Minimax policies for adversarial and stochastic bandits. In Conference on Learning Theory (COLT) , 2009.

P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research (JMLR) , 3:397-422, 2002.

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47(2-3):235-256, 2002.

S. Basu, B. Kveton, M. Zaheer, and C. Szepesvári. No regrets for learning the prior in bandits. In NeurIPS , pages 28029-28041, 2021.

R. Caruana. Multitask learning. Machine Learning , 28(1):41-75, 1997.

L. Cella, A. Lazaric, and M. Pontil. Meta-learning with stochastic linear bandits. In ICML , pages 1360-1370, 2020.

L. Cella, K. Lounici, G. Pacreau, and M. Pontil. Multi-task representation learning with stochastic linear bandits. In AISTATS , pages 4822-4847, 2023.

W. Chen, Y. Wang, and Y. Yuan. Combinatorial multi-armed bandit: General framework and applications. In ICML , pages 151-159, 2013.

W. Chu, L. Li, L. Reyzin, and R. E. Schapire. Contextual bandits with linear payoff functions. In AISTATS , pages 208-214, 2011.

V. Dani, T. P. Hayes, and S. M. Kakade. Stochastic linear optimization under bandit feedback. In Conference on Learning Theory (COLT) , pages 355-366, 2008.

A. A. Deshmukh, Ü. Dogan, and C. Scott. Multi-task learning for contextual bandits. In NeurIPS , pages 4848-4856, 2017.

H. Flynn, D. Reeb, M. Kandemir, and J. Peters. Improved algorithms for stochastic linear bandits using tail bounds for martingale mixtures. In NeurIPS , 2023.

J. Hong, B. Kveton, S. Katariya, M. Zaheer, and M. Ghavamzadeh. Deep hierarchy in bandits. In ICML , pages 8833-8851, 2022.

J. Hong, B. Kveton, M. Zaheer, and M. Ghavamzadeh. Hierarchical bayesian bandits. In AISTATS , pages 7724-7741, 2022.

R. A. Horn and C. R. Johnson. Matrix Analysis . Cambridge University Press, 2012.

J. Hu, X. Chen, C. Jin, L. Li, and L. Wang. Near-optimal representation learning for linear bandits and linear RL. In ICML , pages 4349-4358, 2021.

S. Kale, L. Reyzin, and R. E. Schapire. Non-stochastic bandit slate problems. In NeurIPS , pages 1054-1062, 2010.

C. Kalkanli and A. Özgür. An improved regret bound for thompson sampling in the gaussian linear bandit setting. In IEEE International Symposium on Information Theory (ISIT) , pages 2783-2788, 2020.

E. Kaufmann, O. Cappe, and A. Garivier. On bayesian upper confidence bounds for bandit problems. In AISTATS , pages 592-600, 2012.

M. Khodak, I. Osadchiy, K. Harris, M. Balcan, K. Y. Levy, R. Meir, and Z. S. Wu. Meta-learning adversarial bandit algorithms. In NeurIPS , 2023.

R. D. Kleinberg and F. T. Leighton. The value of knowing a demand curve: Bounds on regret for online posted-price auctions. In Symposium on Foundations of Computer Science (FOCS) , pages 594-605, 2003.

B. Kveton, M. Konobeev, M. Zaheer, C. Hsu, M. Mladenov, C. Boutilier, and C. Szepesvári. Meta-thompson sampling. In ICML , pages 5884-5893, 2021.

T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6:4--22, 1985.

T. Lattimore and C. Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In International Conference on World Wide Web (WWW) , pages 661-670, 2010.

Y. Li, Y. Wang, and Y. Zhou. Nearly minimax-optimal regret for linearly parameterized bandits. In Conference on Learning Theory (COLT) , pages 2173-2174, 2019.

P. Rusmevichientong and J. N. Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research , 35(2):395-411, 2010.

D. Russo and B. V. Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

D. Russo and B. V. Roy. An information-theoretic analysis of thompson sampling. Journal of Machine Learning Research (JMLR) , 17:68:1-68:30, 2016.

W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25:285-294, 1933.

S. Thrun and L. Pratt. Learning to Learn . Kluwer Academic Publishers, 1998.

M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge University Press, 2019.

R. Wan, L. Ge, and R. Song. Metadata-based multi-task bandits with bayesian hierarchical models. In NeurIPS , pages 29655-29668, 2021.

R. Wan, L. Ge, and R. Song. Towards scalable and robust structured bandits: A meta-learning framework. In AISTATS , pages 1144-1173, 2023.

Z. Wen, B. Kveton, and A. Ashkan. Efficient learning in large-scale combinatorial semi-bandits. In ICML , pages 1113-1122, 2015.

Z. Wen and B. V. Roy. Efficient exploration and value function generalization in deterministic systems. In NeurIPS , pages 3021-3029, 2013.

APPENDIX

A Proofs for Regret Bound of HierTS in the Sequential Bandit Setting

We first give the following proposition to bound the posterior variance E ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˆ Σ s,t in the sequential setting. We choose to give the worst-case upper bound on ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˆ Σ s,t .

Proposition A.1 Let c 1 = σ 2 + B 2 λ 1 (Σ 0 ) , c 2 = σ 2 + B 2 λ 1 (Σ 0 ) + B 2 λ 1 (Σ q ) κ (Σ 0 ) , then ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˆ Σ s,t ≤ 2 mdc 1 log ( 1 + n d ) +2 dc 2 log (1 + m Tr ( Σ -1 0 Σ q ) d ) .

Proof. Note that ‖ A s,t ‖ 2 ˆ Σ s,t = A glyph[latticetop] s,t ( ˜ Σ s,t + ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t ) A s,t , then we bound ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˜ Σ s,t and ∑ t ≥ 1 ∑ s ∈S t A glyph[latticetop] s,t ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t A s,t respectively.

(1) Bounding ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˜ Σ s,t . Note that ˜ Σ s,t = (Σ -1 0 + G s,t ) -1 ≤ Σ 0 , then we have A glyph[latticetop] s,t ˜ Σ s,t A s,t ≤ B 2 λ 1 (Σ 0 ) < B 2 λ 1 (Σ 0 ) + σ 2 . Accordingly, we define a new matrix ˜ X s,t glyph[defines] (Σ -1 0 + 1 B 2 λ 1 (Σ 0 )+ σ 2 ∑ glyph[lscript]<t 1 { s ∈ S t } A s,glyph[lscript] A glyph[latticetop] s,glyph[lscript] ) -1 , and notice that ˜ X s,t ≥ ˜ Σ s,t = (Σ -1 0 + 1 σ 2 ∑ glyph[lscript]<t 1 { s ∈ S t } A s,glyph[lscript] A glyph[latticetop] s,glyph[lscript] ) -1 , and that ˜ X s, 1 = Σ 0 . Then recall c 1 = σ 2 + B 2 λ 1 (Σ 0 ) , we have

where the first inequality holds because the basic inequality x ≤ 2 log (1 + x ) , ∀ x ∈ [0 , 1] ; the third inequality holds due to the mean-value inequality ( ∏ d i =1 λ i ) 1 d ≤ ∑ d i =1 λ i d , for any λ i ≥ 0 ; the last inequality holds due to the fact that the agent interacts with each task s ∈ [ m ] at most n times.

Before bounding the remaining ∑ t ≥ 1 ∑ s ∈S t A glyph[latticetop] s,t ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t A s,t , we introduce the following lemma.

Lemma A.1 If the square matrices A > 0 , B ≥ 0 , then λ 1 [( ( I + AB )( I + BA ) ) -1 ] ≤ λ 1 ( A ) λ d ( A ) .

Proof. According to Theorem 7.6.1 in Page 485 of [18], there exists a non-singular matrix S , such that A = SS glyph[latticetop] , and B = S -glyph[latticetop] Λ S -1 , in which Λ ≥ 0 is a diagonal matrix. Then we have AB = S Λ S -1 , BA = S -glyph[latticetop] Λ S glyph[latticetop] . Therefore, applying Weyl's inequality we have

Remark A.1 (Smaller multiplicative factor than the latest one in [17]) The improvement lies in our sharper upper bound on λ 1 (Σ -1 0 ˜ Σ s,t ˜ Σ s,t Σ -1 0 ) , and detailed explanations are two-fold: (1) Previous work [17, Appendix B] directly used Weyl's inequality to upper bound

(2) Instead of directly using Weyl's inequality, we first propose Lemma A.1 which uses positive semi-definite matrix diagonalization technique to bound

Then we apply Lemma A.1 to upper bound

Therefore ¯ Σ t ≤ Σ q . Then applying Lemma A.1 and Weyl's inequality, we have

Meanwhile, we estimate the gap between matrix ¯ Σ -1 t +1 and matrix ¯ Σ -1 t as follow

where the second equality holds due to the Woodbury matrix identity, and the fifth equality holds due to the Sherman-Morrison formula. Then analogous to the proof for (1) Bounding ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˜ Σ s,t , recall c 2 = [ σ 2 + B 2 λ 1 (Σ 0 ) + B 2 λ 1 (Σ q ) κ (Σ 0 ) ] we have

where the second inequality holds due to Eq. (5). Combining (1) and (2) finishes the whole proof. glyph[square]

Remark A.2 Actually, we can replace the term Tr(Σ -1 0 Σ q ) in the above regret bound with O ( nλ 1 (Σ q )) , at the cost of a slightly larger regret upper bound, by bounding (Σ 0 + G -1 s,mn +1 ) -1 ≤ G s,mn +1 in the last but one step in the above (2) , instead of bounding (Σ 0 + G -1 s,mn +1 ) -1 ≤ Σ -1 0 . Specifically, we have the following estimation:

which is O (log ( mn )) , slightly larger than the regret bound of O (log ( m )) in our Proposition A.1.

Then we can begin proving our first Bayes regret bound for HierTS in the multi-task Gaussian linear bandit setting. We first give a lemma as follow, which is useful to prove our multi-task Bayes regret bound in the sequential setting.

Lemma A.2 (Proposition 2 in [21]) Let X 1 and X 2 be arbitrary i.i.d. R m valued random variables and f 1 , f 2 measurable maps such that f 1 , f 2 : R m → R d with E ‖ f 1 ( X 1 ) ‖ 2 2 , E ‖ f 2 ( X 1 ) ‖ 2 2 < ∞ , then | E [ f 1 ( X 1 ) glyph[latticetop] f 2 ( X 1 )] | ≤ √ d E [( f 1 ( X 1 ) glyph[latticetop] f 2 ( X 2 )) 2 ] .

Theorem A.1 (Theorem 5.1 in the main text). Let |S t | = 1 for all rounds t ≥ 1 . Then in the multi-task Gaussian linear bandit setting, the Bayes regret upper bound of HierTS is as follow:

Proof. Recall that H t = ( H s,glyph[lscript] ) glyph[lscript]<t,s ∈S glyph[lscript] is the history up to round t , then

where both the second and the fifth equality hold due to the independence between A s,t and θ s, ∗ conditioned on H t ; the first inequality holds by applying Lemma A.2 with functions f 1 ( y 1 , y 2 ) = y 1 , f 2 ( y 1 , y 2 ) = y 2 for any y 1 , y 2 ∈ R d , and the random variable X 1 = ( θ s, ∗ -ˆ µ s,t , A s, ∗ ) | H t , X 2 (with the second element as A s,t ) is the i.i.d. copy of X 1 ; the second inequality holds due to Jensen's inequality. Plugging the upper bound over E ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˆ Σ s,t in Proposition A.1 into the above result obtains the Bayes regret bound for HierTS. glyph[square]

B Proofs for Regret Bound of HierBayesUCB in the Sequential Bandit Setting

Proof. By definition, we have cov[ θ | µ, H ] = (Σ -1 0 + G ) -1 , E [ θ | µ, H ] = cov[ θ | µ, H ](Σ -1 0 µ + B ) where B = σ -2 ∑ n t =1 x t Y t . Hence cov[ θ | µ, H ] does not depend on µ . Then we have

On the other hand, because cov[ θ | µ, H ] does not depend on µ , E [cov[ θ | µ, H ] | H ] = cov[ θ | µ, H ] . In addition, since B is a constant conditioned on H , then according to [17, Lemma 2], we have the following result:

Theorem B.1 (Theorem 5.2 in the main text). Suppose the action set A is finite with |A| < ∞ . Let |S t | = 1 for all rounds t ≥ 1 . Then in the multi-task Gaussian linear bandit setting, the Bayes regret upper bound of Hierarchical BayesUCB is as follow:

In Theorem 5.2 in the main text, we replace √ d + √ 8 d ln 1 ζ in the right hand side of the above √

inequality with d for ease of exposition.

Proof. Define ∆ s,t = θ glyph[latticetop] s, ∗ A s, ∗ -θ glyph[latticetop] s, ∗ A s,t , the event E s,t = {∀ a ∈ A : | a glyph[latticetop] ( θ s, ∗ -ˆ µ s,t ) | ≤ √ 2 log 1 δ ‖ a ‖ ˆ Σ s,t } , and C t,s,a = √ 2 log 1 δ ‖ a ‖ ˆ Σ s,t . Then we can rewrite the multi-task Bayes regret BR ( m,n ) as the following equivalent form:

Then we will bound the three terms in the RHS of the above equality respectively.

(1) Bounding ∑ t ≥ 1 ∑ s ∈S t E [ ∆ s,t 1 { ∆ s,t ≥ glyph[epsilon1],E s,t } ] . Recall that U t,s,a = a glyph[latticetop] ˆ µ s,t + √ 2 log 1 δ ‖ a ‖ ˆ Σ s,t , then we have

where the first inequality holds due to the fact that U t,s,A s,t ≥ U t,s,A s, ∗ in the BayesUCB algorithm; the second inequality holds because when event E s,t occurs, θ glyph[latticetop] s, ∗ A s, ∗ ≤ U t,s,A s, ∗ the third inequality

holds due to the definition of U t,s,a ; and the last inequality due to the result in Proposition A.1. (2) Bounding ∑ t ≥ 1 ∑ s ∈S t E [ ∆ s,t 1 { ∆ s,t < glyph[epsilon1],E s,t } ] . We trivially have ∑ t ≥ 1 ∑ s ∈S t E [ ∆ s,t 1 { ∆ s,t < glyph[epsilon1],E s,t } ] ≤ mnglyph[epsilon1] .

First we give an upper bound of ∆ s,t = θ glyph[latticetop] s, ∗ ( A s, ∗ -A s,t ) . Using Schwartz's inequality, we have

Besides, we also have θ s, ∗ -µ q = θ s, ∗ -µ ∗ + µ ∗ -µ q ∼ N (0 , Σ 0 + Σ q ) = N (0 , ˆ Σ s, 1 ) , then E [ ‖ θ s, ∗ -µ q ‖ ˆ Σ -1 s, 1 ] ≤ √ E ‖ ˆ Σ -1 2 s, 1 ( θ s, ∗ -µ q ) ‖ 2 2 = √ d . According to [35, Exp 2.11], we have with probability 1 -ζ , ‖ θ s, ∗ -µ q ‖ ˆ Σ -1 s, 1 ≤ √ d + √ 8 d ln 1 ζ . Therefore, with probability 1 -ζ over the draw of { θ s, ∗ } s ∈ [ m ] , we have

where the last inequality holds because P ( ¯ E s,t ∣ ∣ H t ) ≤ 2 δ . Combining (1) , (2) and (3) , we achieve the final Bayes regret bound for any δ ∈ (0 , 1) , glyph[epsilon1] > 0 , ζ ∈ (0 , 1) :

C Proofs for Regret Bounds of HierTS and HierBayesUCB in the Concurrent Bandit Setting

Let C t = { s ∈ S t : λ d ( G s,t ) ≥ β/σ 2 } be the set of sufficiently-explored tasks at round t . We first give the following proposition to bound the posterior variance E ∑ t ≥ 1 ∑ s ∈S t 1 { s ∈ C t }‖ A s,t ‖ 2 ˆ Σ s,t in the concurrent setting. Analogous to the proof for Proposition A.1, we choose to give the worst-case upper bound on ∑ t ≥ 1 ∑ s ∈S t 1 { s ∈ C t }‖ A s,t ‖ 2 ˆ Σ s,t as follow.

Proposition C.1 Let c 1 = σ 2 + B 2 λ 1 (Σ 0 ) , c 2 = σ 2 + B 2 λ 1 (Σ 0 ) + B 2 λ 1 (Σ q ) κ (Σ 0 ) , c 3 = 1 + B 2 σ -2 κ (Σ 0 ) [ λ 1 (Σ 0 ) + σ 2 /β ] , then we have

Proof. Note that we have

On event { s ∈ C t } , the modified HierTS samples from the posterior and actually behaves the same as the original HierTS algorithm in Algorithm 1. Then we bound the two terms in the right hand side of the above equality respectively .

Similar to the proof for Theorem A.1, we have A glyph[latticetop] s,t ˜ Σ s,t A s,t ≤ B 2 λ 1 (Σ 0 ) + σ 2 and ˜ X s,t glyph[defines] (Σ -1 0 + 1 B 2 λ 1 (Σ 0 )+ σ 2 ∑ glyph[lscript]<t 1 { s ∈ S t } A s,glyph[lscript] A glyph[latticetop] s,glyph[lscript] ) -1 ≥ ˜ Σ s,t . Then we can analogously obtain

where the second inequality holds due to the fact that, if square matrix A ≥ B ≥ 0 , then det ( A ) ≥ det ( B ) , and |S t | ≤ m ; the last inequality holds because the agent interact with each task at most n times.

Analysis . The real difference of the proof for the concurrent regret from the sequential regret lies in bounding ∑ t ≥ 1 ∑ s ∈S t 1 { s ∈ C t } A glyph[latticetop] s,t ( ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t ) A s,t , because |S t | ≥ 1 and the result in Eq. (5) (which only holds for the case |S t | = 1 ) does not hold. To tackle this difference, we reduce the concurrent setting to the sequential setting. Let S t = { I tj } |S t | j =1 and define S t, 1: i = { I tj } i j =1 . Then at round t , let s = I ti and define ¯ Σ -1 s,t glyph[defines] Σ -1 q + ∑ z ∈S t, 1: i -1 (Σ 0 + G -1 z,t +1 ) -1 + ∑ z ∈ [ m ] \S t, 1: i -1 (Σ 0 + G -1 z,t ) -1 , we estimate the gap between ¯ Σ -1 s,t and ¯ Σ -1 t as follow:

Thus we can bound λ 1 ( ¯ Σ -1 s,t -¯ Σ -1 t ) and tackle ¯ Σ -1 s,t instead of ¯ Σ -1 t to reduce the concurrent setting to the sequential setting. We first give a useful lemma as follow.

Lemma C.1 For any fixed t ≥ 1 and i ∈ [ L ] , suppose λ d ( G s,t ) ≥ β/σ 2 and let s = I t,i . Then

Proof. Applying Weyl's inequality, we have

We first lower bound λ d ( ¯ Σ -1 t ) . According to Weyl's inequality, we have

where the last inequality holds because the tasks S t, 1: i -1 have been sufficiently explored. On the other hand, using our Lemma A.1 we can bound λ 1 ( ¯ Σ -1 s,t -¯ Σ -1 t ) as follow

Combining the above results, we have

Then, recall c 3 = 1 + B 2 σ -2 κ (Σ 0 ) [ λ 1 (Σ 0 ) + σ 2 /β ] , we can bound ∑ t ≥ 1 ∑ s ∈S t 1 { s ∈ C t } A glyph[latticetop] s,t ( ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t ) A s,t as follows:

where the second inequality holds due to Lemma C.1, the third and the fourth inequality hold in the same way as that in the proof of Bounding (2) in Proposition A.1. Combining the results in (1) and (2) finishes the whole proof. glyph[square]

Remark C.1 In the last step of proof for Lemma C.1, we bound

Thus our upper bound is independent of the number L of the concurrent tasks. Actually, ∀ i ∈ [ L ] :

the sharper bound 1 + B 2 σ 2 λ 1 (Σ 0 ) λ d (Σ 0 ) λ d (Σ -1 q ) L + 1 λ σ 2 /β is L -dependent. If λ d (Σ -1 q ) L << 1 λ 1 (Σ 0 )+ σ 2 /β , the influence of L to the regret may be large; otherwise, the influence of L to the regret may be negligible.

1 (Σ 0 )+

Next, we prove the concurrent regret bound for HierTS and HierBayesUCB.

Theorem C.1 (Theorem 5.3 in the main text). Let |S t | ≤ L ≤ m for all rounds t ≥ 1 . Then in the multi-task Gaussian linear bandit setting, the Bayes regret bound of HierTS is as follow:

Proof. Recall that C t = { s ∈ S t : λ d ( G s,t ) ≥ β/σ 2 } is the set of sufficiently-explored tasks at round t . Then, due to the modification of HierTS algorithm, we decompose Bayes regret BR ( m,n ) into two terms and bound them respectively:

(1) Bounding E ∑ t ≥ 1 ∑ s ∈S t 1 { s / ∈ C t } θ glyph[latticetop] s, ∗ ( A s, ∗ -A s,t ) . Similar to the proof for Theorem B.1 (3) , we have

and E [ ‖ θ s, ∗ -µ q ‖ ˆ Σ -1 s, 1 ] ≤ √ E ‖ ˆ Σ -1 2 s, 1 ( θ s, ∗ -µ q ) ‖ 2 2 = √ d . Recalling the independence between θ s, ∗ and actions A s,t yields

The last inequality holds because in the modified HierTS, event { s / ∈ C t } occurs at most d times for any task s ∈ [ m ] .

(2) Bounding E ∑ t ≥ 1 ∑ s ∈S t 1 { s ∈ C t }‖ A s,t ‖ 2 ˆ Σ s,t . It suffices to apply the upper bound in Proposition C.1.

Combining the upper bounds in steps (1) and (2) obtains the final Bayes regret bound for HierTS in the concurrent setting. glyph[square]

Theorem C.2 (Logarithmic Regret Bound for HierBayesUCB in the Concurrent Bandit Setting). Suppose the action set A is finite with |A| < ∞ . Let |S t | ≤ L ≤ m for all rounds t ≥ 1 . Then in the multi-task Gaussian linear bandit setting, the Bayes regret bound of HierTS is as follow:

Proof. Similar to the proof of Theorem C.1, we decompose the Bayes regret as BR ( m,n ) = E ∑ t ≥ 1 ∑ s ∈S t 1 { s / ∈ C t } θ glyph[latticetop] s, ∗ ( A s, ∗ -A s,t ) + E ∑ t ≥ 1 ∑ s ∈S t 1 { s ∈ C t } θ glyph[latticetop] s, ∗ ( A s, ∗ -A s,t ) . Then

we bound the first term with the proof for Theorem C.1 (1) , bound the second term with the proof for our Theorem B.1. Then with probability 1 -ζ over the draw of { θ s, ∗ } s ∈ [ m ] ,

Plugging the upper bound on ∑ t ≥ 1 ∑ s ∈S t E [ 1 { s ∈ C t }‖ A s,t ‖ 2 ˆ Σ s,t ] in Proposition C.1 into the right hand side of the above inequality finishes the whole proof. glyph[square]

D Proofs for Regret Bounds of HierTS and HierBayesUCB in the Semi-Bandit Setting

We also choose to give the worst-case upper bound on ∑ t ≥ 1 ∑ s ∈S t ∑ a ∈ A s,t Φ glyph[latticetop] a ˆ Σ s,t Φ a as follow.

Proposition D.1 Let c 1 = σ 2 + B 2 λ 1 (Σ 0 ) , c 4 = σ 2 + B 2 Lλ 1 (Σ 0 ) + B 2 λ 1 (Σ q ) κ (Σ 0 ) , then

Proof. Recall that G s,t = σ -2 ∑ glyph[lscript]<t 1 { s ∈ S glyph[lscript] } ( ∑ a ∈ A s,t Φ a Φ glyph[latticetop] a ) , B s,t = σ -2 ∑ glyph[lscript]<t 1 { s ∈ S glyph[lscript] } ( ∑ a ∈ A s,t Φ a ¯ w s ( a )) , ˜ Σ -1 s,t = Σ -1 0 + G s,t , ¯ Σ -1 t = Σ -1 q + ∑ s ∈ [ m ] (Σ 0 + G -1 s,t ) -1 . Then, analogous to the proof for Proposition A.1, we introduce the matrix ˜ X s,t glyph[defines] ( Σ -1 0 + 1 B 2 λ 1 (Σ 0 )+ σ 2 ∑ glyph[lscript]<t 1 { s ∈ S glyph[lscript] } ( ∑ a ∈ A s,t Φ a Φ glyph[latticetop] a ) ) -1 . We next bound ∑ t ≥ 1 ∑ s ∈S t ∑ a ∈ A s,t Φ glyph[latticetop] a ˜ Σ s,t Φ a and ∑ t ≥ 1 ∑ s ∈S t ∑ a ∈ A s,t Φ glyph[latticetop] a ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t Φ a .

(2) Bounding ∑ t ≥ 1 ∑ s ∈S t ∑ a ∈ A s,t Φ glyph[latticetop] a ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t Φ a . ∀ t ≥ 1 , s ∈ S t , ∀ a ∈ A s,t , we have Φ glyph[latticetop] a ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t Φ a ≤ B 2 λ 1 (Σ q ) κ (Σ 0 ) + LB 2 λ 1 (Σ 0 σ 2 . Meanwhile, define the matrix M glyph[defines] ( ˜ Σ 1 2 s,t Φ a 1 , ˜ Σ 1 2 s,t Φ a 2 , . . . , ˜ Σ 1 2 s,t Φ a | As,t | ) ∈ R d ×| A s,t | , we have ∑ a ∈ A s,t ˜ Σ 1 2 s,t Φ a Φ glyph[latticetop] a ˜ Σ 1 2 s,t = MM glyph[latticetop] . Using the Wely's inequality, we further have

Then we can estimate the gap between matrix ¯ Σ -1 t +1 and ¯ Σ -1 t as follow:

where the second and the sixth equality hold due to the Woodbury matrix identity. The proof for Eq. (6) is similar to the proof for Eq. (5), but requires more refined analysis (i.e. the first inequality in Eq. (6)) to estimate the lower bound of ¯ Σ -1 t +1 -¯ Σ -1 t . Then recall c 4 = σ 2 + B 2 Lλ 1 (Σ 0 ) + B 2 λ 1 (Σ q ) κ (Σ 0 ) for brevity, we can bound ∑ t ≥ 1 ∑ s ∈S t ∑ a ∈ A s,t Φ glyph[latticetop] a ˜ Σ s,t Σ -1 0 ¯ Σ t Σ -1 0 ˜ Σ s,t Φ a as follow:

where the second inequality holds due to Eq. (6).

glyph[square]

Lemma D.1 If a Gaussian random variable X ∼ N ( µ, σ 2 ) , then E [ X 1 { X ≥ 0 } ] = µ [ 1 -Φ G ( -µ σ ) ] + σ √ 2 π exp {-µ 2 2 σ 2 } . If further µ ≤ 0 , then E [ X 1 { X ≥ 0 } ] = σ √ 2 π exp {-µ 2 2 σ 2 } .

Theorem D.1 (Theorem 5.4 in the main text, Regret Bound of HierTS in the Semi-Bandit Setting). Let |S t | = 1 for any t ≥ 1 . Let c ≥ √ 2 ln ( nKBλ 1 (Σ 0 ) √ 2 π ) , c 1 = σ 2 + B 2 λ 1 (Σ 0 ) , c 4 = σ 2 + B 2 Lλ 1 (Σ 0 )+ B 2 λ 1 (Σ q ) κ (Σ 0 ) , then in the multi-task Gaussian semi-bandit setting, the Bayes regret bound of combinatorial HierTS is:

Proof. Note that ¯ w s = Φ θ s, ∗ , then define g ( A,θ ) = ∑ a ∈ A 〈 Φ a , θ 〉 for brevity, we have the following result:

Define upper confidence bound U t,s ( A ) = ∑ a ∈ A [ 〈 Φ a , ˆ µ s,t 〉 + c √ Φ glyph[latticetop] a ˆ Σ s,t Φ a ] , where c is a constant to be specified. Notice that A s, ∗ | H t i.i.d. ∼ A s,t | H t and U t,s ( · ) is a deterministic function, thus

E [ U ( A ∗ ) | H ] = E [ U ( A ) | H ] . Then we can decompose Bayes regret BR ( m,n ) as follow:

where the second equality holds because of the mutual independence between A s,t | H t and θ s, ∗ | H t , and E [ˆ µ s,t -θ s, ∗ | H t ] = 0 ; the last inequality holds due to the Jensen inequality. Then applying

Proposition D.1 to bound E ∑ t ≥ 1 ∑ s ∈S t ∑ a ∈ [ A s,t ] Φ glyph[latticetop] a ˆ Σ s,t Φ a , we can obtain

Combining the above results finishes the whole proof.

glyph[square]

Theorem D.2 (Theorem 5.5 in the main text, Regret Bound of HierBayesUCB in the Semi-Bandit Setting). Let |S t | = 1 for all rounds t ≥ 1 . Let c 1 = σ 2 + B 2 λ 1 (Σ 0 ) , c 4 = σ 2 + B 2 Lλ 1 (Σ 0 ) + B 2 λ 1 (Σ q ) κ (Σ 0 ) , Then for any glyph[epsilon1] > 0 , δ ∈ (0 , 1) , ζ ∈ (0 , 1) , in the multi-task Gaussian semi-bandit setting, the Bayes regret upper bound of combinatorial HierBayesUCB is as follow:

In Theorem 5.5 in the main text, we replace √ d + √ 8 d ln 1 ζ in the right hand side of the above inequality with √ d for ease of exposition.

Proof. Define the event E s,t = {∀ a ∈ A : | Φ glyph[latticetop] a ( θ s, ∗ -ˆ µ s,t ) | ≤ √ 2 log 1 δ ‖ Φ a ‖ ˆ Σ s,t } , and the upper confidence bound U t,s ( A ) = ∑ a ∈ A 〈 Φ a , ˆ µ s,t 〉 + √ 2 log 1 δ ‖ Φ a ‖ ˆ Σ s,t . Let ∆ s,t = g ( A s, ∗ , θ s, ∗ ) -g ( A s,t , θ s, ∗ ) , then we decompose the Bayes regret into three parts as follow:

where the first and the second inequality hold due to the definition of the upper confidence bound U t,s ( A s,t ) , the fourth inequality holds due to the Cauchy-Schwartz inequality. Utilizing the upper bound on ∑ t ≥ 1 ∑ s ∈S t ∑ a ∈ A s,t ‖ Φ a ‖ 2 ˆ Σ s,t in Proposition D.1 completes the proof for the first part.

Note that θ s, ∗ -µ q ∼ N (0 , ˆ Σ s, 1 ) , and E [ ‖ θ s, ∗ -µ q ‖ ˆ Σ -1 s, 1 ] ≤ √ E ‖ ˆ Σ -1 2 s, 1 ( θ s, ∗ -µ q ) ‖ 2 2 = d . Then according to [35, Exp 2.11], we have with probability 1 -ζ , ‖ θ s, ∗ -µ q ‖ ˆ Σ -1 s, 1 ≤ √ d + √ 8 d ln 1 ζ Therefore, with probability 1 -ζ over the draw of { θ s, ∗ } s ∈ [ m ] , we have

where the first inequality holds due to the Schwartz inequality. Then we have with probability 1 -ζ :

Combining the results in (1) (2) and (3) finishes the whole proof.

E Technical Overview and Limitations of this Work

In this section, we explain our technical novelties for deriving near-optimal sequential regret bound for HierTS and logarithmic sequential regret bound for HierBayesUCB as follow, when compared with the latest bound in [17] (More detailed explanations can be found in Table 4):

(1) The Technical Overview for Deriving Near-Optimal Regret Bound in Theorem 5.1 . The biggest novelty lies in bounding each term E [ ( θ s, ∗ -ˆ µ s,t ) glyph[latticetop] A s, ∗ ∣ ∣ H t ] in Bayes regret BR ( m,n ) . Existing work [17] chose Cauchy-Schwartz inequality to directly bound ( θ s, ∗ -ˆ µ s,t ) glyph[latticetop] A s, ∗ ≤ ‖ θ s, ∗ -ˆ µ s,t ‖ ˆ Σ -1 s,t ‖ A s, ∗ ‖ ˆ Σ s,t , used UCB technique to bound ‖ θ s, ∗ -ˆ µ s,t ‖ ˆ Σ -1 s,t (which caused an additional multiplicative factor log 1 δ ), leveraged the fact that A s, ∗ | H t i.i.d. ∼ A s,t | H t to transform

glyph[square]

∑ t,s ‖ A s, ∗ ‖ ˆ Σ s,t into V m,n , and obtained an intermediate regret upper bound √ mn V m,n log (1 /δ ) . Instead of using UCB technique, our Theorem 5.1 applies a novel Cauchy-Schwartz type inequality

(i.e. Lemma A.2) to bound E ( θ s, ∗ -ˆ µ s,t ) glyph[latticetop] A s, ∗ ≤ √ d E ( ( θ s, ∗ -ˆ µ s,t ) glyph[latticetop] A s,t ) 2 ≈ √ d ‖ A s,t ‖ ˆ Σ s,t , and finally achieves the regret bound √ mn V m,n , removing the √ log (1 /δ ) factor. Besides, when bounding the posterior variance V m,n , we use a different matrix analysis to prevent variance terms (e.g. σ 2 , λ 1 (Σ 0 ) ) solely appearing in the denominator of regret bound. Moreover, we employ a matrix decomposition technique (in our Lemma A.1) to reduce the multiplicative factor κ 2 (Σ 0 ) in [17, Theorems 3-4] to κ (Σ 0 ) in our bounds (see more details in Table 4).

(2) The Technical Overview for Deriving Logarithmic Regret Bound in Theorem 5.2 . To obtain sharper sequential regret bound than the near-optimal regret bound in Theorem 5.1, our Theorem 5.2 chooses the Bayes regret decomposition strategy shown above Theorem 5.2, uses UCB technique to bound the first term in the regret decomposition as ∑ t,s E ∆ s,t 1 { ∆ s,t ≥ glyph[epsilon1],E s,t } ≤ V m,n log 1 δ , and finally combines the upper bound on posterior variance V m,n in Eq. (4) to achieve a logarithmic Bayes regret upper bound of (log 1 δ ) md log n d .

Nevertheless, we also need to point out the limitations of our Bayes regret bounds:

The Limitations of the Multi-Task Bayes Regret Bounds . Honestly speaking, our regret bounds have two main limitations, because they are: (i) Not advantageous when compared with single-task regret bound. This is because our Bayes regret bounds (e.g. O ( m √ n log n ) in Theorem 5.1) for hierarchical Bayesian bandit problem are almost the same as the summation of regret bounds of learning m Bayesian bandit task independently. This is also the limitation of existing bounds in this field (see [25, 7, 17]). (ii) Unable to shed more light on the advantages of multi-task bandit optimization. The existing regret bound O ( m √ nk ) for multi-task representation which demonstrated that multi-task regret bound can be smaller for learning a low-dimensional representation (i.e. k << d ) than the regret bound of O ( m √ nd ) for learning each task independently, and the existing regret bound O ( m √ n log (1 + nV )) for multi-task adversarial linear bandit which proved that the regret bound decreases with more similarity (i.e. smaller V ) among bandit tasks. Our hierarchical Bayesian bandit model has assumed that different bandit instances are sampled the same metadistribution, and hence fails to reveal the influence of task similarity to the multi-task Bayes regret.

Remark E.1 (The Underlying Causes for the Limitation of Multi-Task Bayes Regret Bound.)

The underlying causes for the shortcoming of the multi-task Bayes regret bound is that the upper bound on the posterior variance V m,n = E ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˆ Σ s,t may be not tight enough. Detailed explanations lie in the following three aspects:

(1) Recall that in the proof for our Theorem 5.1, we can upper bound the multi-task Bayes regret as BR ( m,n ) ≤ √ mnd √ V m,n . Then in Proposition A.1 we use a purely algebraic technique to bound the posterior-variance V m,n ≤ O ( m log n ) , resulting in the final Bayes regret bound of

which is almost the same as the summation of the regret bounds for learning m bandit tasks independently. Therefore, if we can upper bound the posterior-variance V m,n with a bound that is sublinear with respect to m and logarithmic w.r.t. n (e.g. a bound of O ( √ m log n ) ), then the final Bayes regret bound will be much sharper. The upper bounds on the posterior-variance V m,n in existing works (e.g. see [25, 7, 17] in our Table 1 in the main text) are also obtained via purely algebraic techniques and are not sharp either (or even worse).

(2) In the proof for Proposition A.1, we only give the worst-case upper bound on ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˆ Σ s,t via purely algebraic technique (thus leading to a worst-case upper bound on the posterior variance

E

∑

t

1

s

∈S

t

A

s,t

ˆ Σ

s,t via purely algebraic techniques, ignoring the expectation over the randomness of A s,t and ˆ Σ s,t . Therefore, we may achieve sharper regret bound by considering the expectation over the randomness in the posterior variance.

(3) To derive a sharper and meaningful upper bound on the posterior-variance V m,n = E ∑ t ≥ 1 ∑ s ∈S t ‖ A s,t ‖ 2 ˆ Σ s,t , we need to consider other bounding technique like concentration inequality, or more technical matrix analysis, to achieve an upper bound that is sublinear w.r.t. the

∑

‖

‖

2

).

Such worst-case upper bound is obtained

number m of tasks and sublinear w.r.t. the number n of iterations per task. Only in this way can we obtain a multi-task regret bound o ( mn ) that is sublinear w.r.t. m and sublinear w.r.t. n .

(4) On the other hand, we also consider finding the lower bound of posterior-variance V m,n to show that our upper bound on V m,n is tight, or finding the lower bound of multi-task Bayes regret BR ( m,n ) to show that our multi-task Bayes regret upper bound could not be improved. This serves as one of our ongoing research directions.

F Additional Experiments and Computer Resources

0

2

4

6

8

10

Number of Tasks

m

0

50

100

150

Regret

Linear Bandit (

d

= 4,

σ

q

= 1.0,

L

=1)

n

=100

n

=200

n

=300

n

=400

(a) Regrets w.r.t. different

m

2

4

6

8

10

Number of Concurrent Tasks

L

0

50

100

150

Regret

Linear Bandit (

m

= 10,

σ

q

= 1.0)

d

=8

d

=4

d

=2

(b) Regrets w.r.t. different

L

0

100

200

300

400

Round

t

0

50

100

150

Regret

Linear Bandit (

d

= 4,

m

= 10,

L

=5)

σ

q

=5

σ

q

=4

σ

q

=3

σ

q

=2

σ

q

=1

(c) Regrets w.r.t. different

σ

q

0

100

200

300

400

Round

t

0

100

200

300

Regret

Linear Bandit (

d

= 4,

m

= 10,

L

=5)

σ

0

=0.5

σ

0

=0.4

σ

0

=0.3

σ

0

=0.2

σ

0

=0.1

0

100

200

300

400

Round

t

0

250

500

750

1000

Regret

Linear Bandit (

d

= 4,

m

= 10,

L

=5)

σ

=10

σ

=7

σ

=4

σ

=1

σ

=0.1

0

100

200

300

400

Round

t

0

100

200

300

400

Regret

Linear Bandit (

d

= 4,

σ

q

= 0.5)

OracleTS

TS

HierTS

HierBayesUCB

(d) Regrets w.r.t. different

σ

0

(e) Regrets w.r.t. different

σ

(f) Regrets of different algorithms

Experimental Results . From Figure 2, we have the similar observations as that in Figure 1: (1) In plot (a), the multi-task regret of HierBayesUCB becomes larger with the increase of m and n , which is consistent with our regret upper bound in Theorems 5.2. (2) In plot (b), the regret increases with a higher dimension d , and increases with a larger number L of the concurrent tasks. (3) In plots (c)-(e), the regret decreases with a smaller variance (e.g. σ q , σ 0 and σ ) in hierarchical Bayesian model, validating the provable benefits of variance-reduction in Bayes regret minimization. (4) The task-averaged regret of our proposed HierBayesUCB is smaller than that of HierTS, and such improvement becomes larger with the increase of σ q (when compared with σ q = 1 . 0 in Figure 1 (f)).

Computer Resources . Our implementations are based on Python. We run all bandit algorithms on a platform with 8 NVIDIA RTX 6000 GPUs and 2 AMD EPYC 7543 Processors. Each GPU has 48G memory, and each CPU has 64 cores. The CUDA version is 12.1, the Python version 3.7.16, the matplotlib version 3.5.3, and the tensorflow version 1.15. The source code for reproducing all experimental results of HierTS and HierBayesUCB is provided in the supplementary material.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: see Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: see Section E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: see assumptions in Section 5, and see proof sketch in Section E.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: see implementation details in Section 6 and Section F, and see the source code in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: see the source code in our supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: see the implementation details in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: see our Figures 1-2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: see details in Section F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: this paper is a purely theoretical paper and has no negative social impact. Besides, we release the source code to implement our proposed algorithms in the supplementary material.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: there is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: the paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript discusses a multi-task learning problem and proposes a hierarchical Bayesian (HB) bandit approach. In this approach, the agent maintains a meta-posterior distribution over the hyperparameters of the within-task bandit problems. In the HB bandit, each bandit task is characterized by a task parameter. The paper examines both the sequential and concurrent settings for two types of bandit problems: 1) Gaussian Linear bandit and 2) Semi Bandit. In particular, for both sequential and concurrent bandit in the Gaussian Linear bandit setting:\n1. The paper improves upon the existing gap-independent Bayes regret bound of HierTS from O(m \\sqrt{n \\log n  \\log (mn)} ) to O(mn\\log n) for infinite actions set where m is the number of tasks and n is the number of iterations per task. \n2. The paper also extends the HierTS and proposes a new algorithm for finite action sets, HierBayesUCB, with gap-dependent O(m \\log n  \\log (mn)) regret bound. \n\nMoreover, the paper extends the HierTS and HierBayesUCB for multi-task Gaussian combinatorial semi-bandit setting and derives regret bounds of O(m \\sqrt{n} \\log n) and O(m \\log (mn \\log n)).\n\nSTRENGTHS:\nThis paper's contribution is important and novel because:\n1. It improves upon the existing regret bound for HeirTS in the infinite action setting\n2. Proposes a new algorithm to solve the finite action case. \n3. They also extend these algorithms to semi-bandit settings with SOTA regret bounds, which is novel.\n\nWEAKNESSES:\nMany parts of the paper can be significantly improved: \n\t\n1. [Novelty] I don’t think we need the last paragraph in the introduction. Instead, the authors should briefly describe the technical novelty they have introduced to improve the HierTS regret bound for both the sequential and concurrent bandit over the existing works. Why was the previous analysis not sufficient?   What is their novel and non-trivial contribution beyond BayesUCB? Please broadly discuss the novelties summarized in the appendix in the main paper, as this is your main contribution that led to improved bounds. Also, discuss the novelty in the techniques beyond using the improved inequality from [21].\n\n2. 140-141: Why do we need this assumption of at least n-round interaction, and how is it ensured in the algorithm?\n\n3. Alg1 and 2:  How do the algorithms choose the set \\mathcal{S}_t of the tasks at each time t? Even the task sequence is also not input to the algorithm. It is not clear to me.\n\n\n4. What is \\hat \\mu and \\hat \\simga in Alg 1 (HierBayesUCB)?  It is not defined yet. I found it to be defined later in (3).\n\n5. display (4): If the order of V_m,n is the same as derived in [17] up to multiplicative factors, how does it help to improve the regret bound?  I cannot find the intuition to understand what improved the bound. The authors must discuss this in the main part of the paper.\n\nQUESTIONS:\n1. L228-229: Please elaborate more on how you have achieved this.\n\nMinor:\n\n- Single task semi-Bandit: L 121-122—The authors should mention in the introduction and abstract that they consider coherent cases of semi-bandit problems.\n\n- L53: Latest\n\n- L143:  Fix  ‘=‘\n\n- L159: insert ‘and’\n\nLIMITATIONS:\nThe nature of the work is theoretical.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper improves the Bayesian regret bound for hierarchical Bayesian bandit algorithms in multi-task bandit and semi-bandit settings. Firstly, it improves the gap-independent bound by a factor of $\\mathcal{O}(\\sqrt{\\log(mn)})$ for infinite action set, $m$ being number of tasks and $n$, the number of iterations per task. For finite action set, the authors propose and analyze HierBayesUCB algorithm and analyze its gap-dependent Bayesian regret. Finally, the paper extends these algorithms to multi-task combinatorial semi-bandit setting.\n\nSTRENGTHS:\nThe paper deals with an interesting hierarchical bandit/semi-bandit setting, where the agent needs to optimize its policy for several tasks simultaneously based on semi-bandit feedback. For the Gaussian setting, the paper provides Bayesian regret bounds that either improve upon previous results, or provide new results (e.g., HierBayesUCB in the combinatorial setting).\n\nWEAKNESSES:\nThe main weakness of the paper is the requirement of Gaussian distributions for each layer of the hierarchy including the noise. While such an assumption facilitates the development of closed-form posteriors to ease the learning task, it is highly restrictive since in many cases, the noise is not Gaussian (assuming sub-Gaussian would be better, many classes of noise distributions including uniform, truncated Gaussian or Rademacher could be accommodated). This should at least be mentioned in the abstract since the current abstract gives the sense of a more general theory being developed in the paper.\n\nQUESTIONS:\nHere are a few questions regarding the paper:\n1. In the case of several tasks, what is the benefit of assuming a hierarchical structure rather than dealing with each task independently? In Bayesian hierarchical models, this kind of shared structure often allows borrowing of strength across the different groups (tasks here), so that the overall rate could be better -- in this case, the overall regret is linear in $m$, which seems to be similar to that of dealing with the tasks independently. Can the authors shed more light on this. \n2. Is it possible to use this result to derive frequentist regret bounds (either high probability bounds or in expectation), given either some true $\\theta_{s}^*$. To keep the tasks exchangeable, one can alternatively assume a true $Q^*$ and $\\{\\theta_s^*\\}$ as latent variables. Furthermore, in the Gaussian case, is the lower bound $\\Omega(d\\sqrt{n})$ tight in this hierarchical (shared) setting (note that this lower bound is for a linear bandit learning a single task, while in this case, it is not immediately obvious whether or not the shared structure across the tasks could help in the regret bound)\n3. I am concerned with the Gaussian assumptions required for all the theoretical results. While the distributions for $Q$ and $\\theta_{s}^*|\\mu_*$ are fine to assume Gaussian (equivalent to a Gaussian prior on $(\\mu_{1}^*,\\dots,\\mu_{m}^*)$ with dependence structure), the issue is with the Gaussian noise, which is restrictive to apply the method to broader range of problems. Unlike this work, Thompson sampling for linear bandits have been studied where the Gaussianeity assumption in the likelihood is only taken for algorithmic convenience, whereas the theoretical results only require milder assumptions like sub-Gaussianeity (e.g., [Agrawal and Goyal]). Furthermore, the paper does not deal with the case where $\\mu_q, \\Sigma_q, \\Sigma_0, \\sigma$ are unknown -- although the experiments place further priors on them, but this essentially makes the model a mixture, whereby the theoretical results are no longer valid. \n4. The term $\\mathcal{V}_{m,n}$ is the posterior variance of which distribution?\n5. Regarding novelty in technical contribution, it seems the application of the Cauchy Schwarz inequality from [C. Kalkanli and A. Özgür.] (line 204) results in the reduction of the extra factor of $\\sqrt{\\log(mn)}$ from both Theorems 5.1 and 5.3, which is the main improvement in the paper. For Theorem 5.2, it seems like the fact that $|\\mathcal{A}|<\\infty$ is the key to reducing the $\\sqrt{n}$ factor in the averaged Bayes regret (compared to Theorem 5.1), otherwise, trivially $\\Delta_{s,\\min}=0$ for compact $\\mathcal{A}$ -- however, the additional term of $\\log(m)$ is intriguing, can the authors comment on this? The last comment in line 267 shows that if $\\Delta_{\\min}$ is small, then this bound is much worse, scaling as $m^{3/2}$ (also the additional $\\sqrt{\\log(mn)}$ reappears).\n6. Can the authors comment on the technical difference to remove the polynomial dependence of $K$ to logarithmic in Theorem 5.4 for the combinatorial case?\n7. The bound in Theorem 5.5 is stronger than that in [S. Basu, B. Kveton, M. Zaheer, and C. Szepesvári.] in the case $\\Delta_{\\min}$ is large, otherwise the comment in line 266 would result in a bound worse -- is this correct?\n8. Algorithms 1 and 2: Apart from $Q$, the update step for $Q_{t+1}$ also requires the various other likelihood and model parameters (assumed known) like $\\mu_q, \\Sigma_q, \\sigma$ which should be included as input. Also, $\\delta$ is an input parameter for the HierBayesUCB algorithm. \n9. The *ORACLE* used in the semi-bandit setting: In practice, this is a combinatorially hard problem (e.g., the work by Chen, Wang and Yuan considers a $(\\alpha,\\beta)-$approximate oracle for efficiency). No simulation examples were shown to demonstrate the performance and more importantly, the computational complexity, for this important step in both HierTS and HierBayesUCB algorithms.\n10. What are the values of $m, n$ in Figure 1(f) -- it seems like HierTS surprisingly performs very poorly, in fact, with higher $T$, vanilla TS might have better regret than HierTS. Any insights as to why this is the case? This is also related to my first question.\n\nOther small comments: In line 153, since $t$ is mentioned as a suffix in $P_{s,t}$, it might be better to use $\\mu_t$, i.e. $\\mathbb{P}(\\theta_{s}^*=\\theta|\\mu_*=\\mu_t, H_{s,t})$. Also, $\\hat{\\mu}_{s,t}$ in line 160 is not defined till the following page (equation 3).\n\nLIMITATIONS:\nSee the questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies the multi-task Gaussian linear bandit and semi-bandit problems. It uses a Bayesian approach to maintain a meta-distribution over the hyper-parameters of within-task parameters. It provides an improved regret bound for the for multi-task for HierTS algorithm in the case of infinite action set. For the same setting, it proposes HierBayesUCB which is the UCB counterpart of HierTS. The authors extend their algorithm to the concurrent multi-task and combinatorial semi-bandit settings as well.

STRENGTHS:
Contribution: The paper improves upon prior works in several directions; it proposes tighter regret bounds for HierTS, extends multi-task bandit algorithms to combinatorial semi-bandit setting. 

Presentation: the paper is very well-organized, it clearly defines the problem, pseudo-code of the algorithms are clearly presented, and the experimental results are extensive.

WEAKNESSES:
The paper is lacking real-world dataset experiments. This could help assess the algorithm's robustness w.r.t model misspecification and gauge the theoretical results consistency w.r.t assumptions violation.

QUESTIONS:
No questions.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16

comments: SUMMARY:
This paper revisits the learning problems of (multi-task) Bayesian linear bandit/semi bandits, which is interesting. The improvement of the Bayesian regret bound for the multi-task Bayes regret bound of HierTS is very marginal. For the remaining presented upper bounds, there are still near-optimal up to some $\log$ factors from the regret lower bounds. I do not check the proofs in the appendix in detail, but it seems they are correct and the proofs are well written.

In the introduction section, the description
 “The gap between the cumulative reward of optimal actions in hindsight and the cumulative reward of agent 24 is defined as regret” seems not to be aligned with the learning problems to be solved. For me, I think we still use pseudo rewards. So, we may not care about the optimal actions in hindsight. Instead, we care about the action with the highest mean given a problem instance. 

Questions:

I have a question, when using the notion of Bayesian regret, why is there an instance-dependent gap? For the UCB-based algorithms, I am feeling that it is also possible to have an instance-dependent regret bound using the notion of frequentist regret. Why not do it in that way?



In Line 53, there is a typo. I guess it should be “latest” instead of “latext”.

Can you rotate Table 4? It is hard for readers who use a desktop.

STRENGTHS:
see the first box

WEAKNESSES:
see the first box

QUESTIONS:
see the first box

LIMITATIONS:
Frequentist analysis may be more interesting, but I kind of know the reasons why not to use frequentist regret as the combination of TS and frequentist regret needs to have anti-concentration bounds, which are always hard to develop.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-16