Title: Beyond task diversity: provable representation transfer for sequential multitask linear bandits
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bandits

multi-task

meta-learning

representation learning

online learning

task diversity
Abstract: We study lifelong learning in linear bandits, where a learner interacts with a sequence of linear bandit tasks whose parameters lie in an $m$-dimensional subspace of $\mathbb{R}^d$, thereby sharing a low-rank representation. Current literature typically assumes that the tasks are diverse, i.e., their parameters uniformly span the $m$-dimensional subspace. This assumption allows the low-rank representation to be learned before all tasks are revealed, which can be unrealistic in real-world applications. In this work, we present the first nontrivial result for sequential multi-task linear bandits without the task diversity assumption. We develop an algorithm that efficiently learns and transfers low-rank representations. When facing $N$ tasks, each played over $\tau$ rounds, our algorithm achieves a regret guarantee of $\tilde{O}\big (Nm \sqrt{\tau} + N^{\frac{2}{3}} \tau^{\frac{2}{3}} d m^{\frac13} + Nd^2 + \tau m d \big)$ under the ellipsoid action set assumption.
This result can significantly improve upon the baseline of $\tilde{O} \left (Nd \sqrt{\tau}\right)$ that does not leverage the low-rank structure when the number of tasks $N$ is sufficiently large and $m \ll d$. We also demonstrate empirically on synthetic data that our algorithm outperforms baseline algorithms, which rely on the task diversity assumption.

Beyond Task Diversity: Provable Representation Transfer for Sequential Multi-Task Linear Bandits

Abstract

We study lifelong learning in linear bandits, where a learner interacts with a sequence of linear bandit tasks whose parameters lie in an m -dimensional subspace of R d , thereby sharing a low-rank representation. Current literature typically assumes that the tasks are diverse , i.e., their parameters uniformly span the m -dimensional subspace. This assumption allows the low-rank representation to be learned before all tasks are revealed, which can be unrealistic in real-world applications. In this work, we present the first nontrivial result for sequential multitask linear bandits without the task diversity assumption. We develop an algorithm that efficiently learns and transfers low-rank representations. When facing N tasks, each played over τ rounds, our algorithm achieves a regret guarantee of ˜ O ( Nm √ τ + N 2 3 τ 2 3 dm 1 3 + Nd 2 + τmd ) under the ellipsoid action set assumption. This result can significantly improve upon the baseline of ˜ O ( Nd √ τ ) that does not leverage the low-rank structure when the number of tasks N is sufficiently large and m ≪ d . We also demonstrate empirically on synthetic data that our algorithm outperforms baseline algorithms, which rely on the task diversity assumption.

1 Introduction

Recommendation systems that interact with customers to promote the best items for each user have been widely adopted around the world. These interactions are often sequential and can be modelled as linear bandit problems [Abe and Long, 1999, Dani et al., 2008, Rusmevichientong and Tsitsiklis, 2010, Abbasi-Yadkori et al., 2011], where the characteristics of items can be represented as context vectors, and a user's preference for an item (i.e., reward) can be modelled using a linear combination of the context of the item. Even though the problem is typically high-dimensional, different users may exhibit similar preferences, leading to a low-dimensional underlying reward structure.

Motivated by this observation, there has been growing interest in representation learning within the context of linear bandits. For instance, in the item recommendation example, each session of interaction with a user can be seen as a linear bandit task, and similarity across tasks can be captured by the existence of a global feature extractor that applies to all problem instances.

Formally, we consider a problem where the learner sequentially faces N d -dimensional linear bandit tasks, each with horizon τ , with a key assumption that the reward predictors of the N tasks, θ 1 , . . . , θ N , lie in an m -dimensional linear subspace of R d . The goal of the learner is to minimize their meta (pseudo-)regret, which is the sum of regret across all tasks (see Equation (1) below), by exploiting the shared subspace structure.

One naive approach is to solve each task independently using a base algorithm (such as LinUCB [Abbasi-Yadkori et al., 2011] or PEGE [Rusmevichientong and Tsitsiklis, 2010]); this approach, which we will henceforth refer to as the individual single-task baseline ), would yield an upper bound on the meta-regret of 0 ˜ O ( Nd √ τ ) . On the other hand, had the shared m -dimensional subspace been known beforehand, one would only need to estimate each reward predictor's projection onto the subspace; this leads to a meta-regret of ˜ O ( Nm √ τ ) . In this work, we focus on the setting where N and τ are large and m ≪ d , the regime in which representation transfer learning would be beneficial.

Despite rich results for multi-task linear bandits in the parallel setting [e.g. Yang et al., 2020, Hu et al., 2021, Yang et al., 2022, Cella et al., 2023], progresses on multi-task bandits in the sequential setting have been relatively sparse. This can be attributed to the additional challenge of meta-exploration : in addition to exploration in each bandit learning task, one also needs to determine when (in which tasks) to acquire more information on the shared m -dimensional subspace representation. This is in contrast to the parallel setting, where algorithms that treat all tasks equally can achieve a near-optimal regret through a reduction to low-rank linear bandits [Hu et al., 2021, Jang et al., 2021a].

Under the assumption that the action sets are well-conditioned ellipsoids, Qin et al. [2022] design an efficient algorithm with a meta-regret of ˜ O ( Nm √ τ + dm √ τN ) . However, it relies on an additional key assumption that the tasks are 'diverse' in the m -dimensional subspace: more formally, for any subsequence of tasks S , the m -th eigenvalue of the task parameters' covariance matrix 1 | S | ∑ n ∈ S θ n θ ⊤ n is bounded away from zero (see Tripuraneni et al. [2021] for a related assumption in the supervised regression setting). However, this task diversity assumption is hard to verify and may not even hold in practice. Therefore, we raise the question:

Can we design sequential multi-task bandit algorithms with provable low meta-regret without strong assumptions on task parameters, especially on task diversity?

In this paper, we answer this question positively. Under mild assumptions that the action sets are wellconditioned ellipsoids and all task parameters have norms upper and lower bounded by constants, we design an algorithm with a meta-regret of ˜ O ( Nm √ τ + N 2 3 τ 2 3 dm 1 3 + Nd 2 + τmd ) 1 , providing the first nontrivial result for sequential multi-task linear bandit without the task diversity assumption. To the best of our knowledge, prior to our work, no regret bounds better than that of the individual single-task baseline (i.e., o ( Nd √ τ ) ) were known for this setting.

Our algorithm, BOSS , is based on a reduction to a bandit online subspace selection problem. Specifically, for each new task n , our algorithm chooses a subspace, represented by its canonical orthonormal basis ˆ B n , to approximate the ground-truth subspace B and guide exploration. To address the challenge of meta-exploration, as choosing different ˆ B n 's leads to learning the projections of θ n onto different subspaces/directions, our algorithm is designed to randomly meta-explore in some tasks and use them to learn B . Empirically, we demonstrate the effectiveness of our algorithm in a simulated adversarial environment where the task diversity assumption does not hold.

1.1 Related work

Parallel representation transfer for multi-task linear bandit. The parallel setting where the learner interacts with N tasks simultaneously in each round was initially studied by Yang et al. [2020]. For the finite action setting, under some distributional assumptions on the action set in each round, they provide a total regret lower bound of Ω ( N √ mτ + √ mdτN ) and an algorithm that matches this up to logarithmic factors. For the infinite action setting, they provide a lower bound for the problem of Ω ( Nm √ τ + d √ mτN ) , which holds even under the task diversity assumption. Under the same task diversity assumption and in the infinite action set setting 2 , Yang et al. [2022] present an algorithm with a regret guarantee of ˜ O ( Nm √ τ + d √ mτN ) .

For general action spaces, Hu et al. [2021] provide a regret guarantee of ˜ O ( N √ mdτ + d √ τNm ) , albeit using a computationally inefficient algorithm. They also provide an extension for multi-task linear reinforcement learning under the assumption of low inherent Bellman error.

Unlike previous approaches, Cella et al. [2023] relaxes the requirement to know the subspace rank m and the need for the task diversity assumption. Assuming that the action sets are finite and drawn from a specific distribution, by using trace-norm regularization for estimating task parameters and taking actions in a greedy fashion, they provide a regret upper bound of ˜ O ( N √ mτ + √ dmτN ) that matches the lower bound from Yang et al. [2020] up to logarithmic factors.

Sequential representation transfer for multi-task linear bandit. Compared with the parallel setting, the sequential setting is more challenging, where the learner only interacts with one task at a time; hence, even after many tasks, the reward predictors of the seen tasks may not span the underlying m -dimensional subspace. Qin et al. [2022] avoid this challenge by assuming a task diversity assumption, i.e., any large enough subset of tasks span the underlying m -dimensional subspace in a well-conditioned manner. With this, they provide a meta-regret guarantee ˜ O ( Nm √ τ + dm √ τN )

which nearly matches a lower bound of Ω ( Nm √ τ + d √ mτN ) . They also extend their analysis to a nonstationary representation setting where the global feature extractor can change over segments of tasks. Yang et al. [2022] study the sequential setting with an additional assumption that ∥ θ n ∥ = 1 for all tasks; however, it appears that there may be a non-trivial oversight in the analysis. 3

Bilaj et al. [2024] study a related setting where the task parameters are i.i.d. sampled from a distribution with high variances from a m -dimensional subspace and with low variances in the orthogonal directions. They provide a meta-regret guarantee of at least ˜ O ( N √ τ ( d -m ) log ( 1 + m λ A min ( d -m ) )) , where λ A min is the smallest eigenvalue of the empirical covariance matrix of the actions taken. Since a linear bandit algorithm is expected to converge to pull the optimal arm at the end; λ A min may be constant when the action space is fixed. Thus, this bound can be as large as ˜ O ( Nd √ τ ) . For a quick reference, see Table 1 for a comparison between our work and most related works. See also Appendix A for further discussions on related work.

2 Problem setup

We consider a sequence of linear bandit tasks of the same length τ , described by the task parameters θ 1 , · · · , θ N ∈ R d chosen by an environment such that they satisfy Assumption 1. The learner solves N tasks sequentially. In task n , for each time step t = 1 , . . . , τ , the learner chooses an action A n,t from the action set A that satisfies Assumption 2 and receives a reward r n,t = A ⊤ n,t θ n + η n,t , where

Algorithm 1 Meta-Exploration procedure

1: Input: Task index n , exploration length τ 1 (a multiple of d ) 2: for i ∈ [ d ] do 3: Let A n,t = λ 0 e i for t = u ( i -1) + 1 , · · · , ui , where u = τ 1 d 4: end for 5: for time step t ← 1 , · · · , τ 1 do 6: Take action A n,t and receive the reward r n,t 7: end for 8: Compute ˆ θ n := argmin θ ∈ R d 1 τ 1 ∑ τ 1 t =1 ( ⟨ A n,t , θ ⟩ -r n,t ) 2 9: for time step t ← τ 1 +1 , · · · , τ do 10: Take action A n,t ← arg max a ∈A 〈 a, ˆ θ n 〉 11: end for

η n,t is independent, mean-zero 1 -sub-Gaussian noise. The learner then moves on to task n +1 and repeats the same learning process.

Assumption 1 (Low-rank representation) . Let m<d . For task parameters θ 1 , · · · , θ N , there exist (i) a global feature extractor B ∈ R d × m with orthogonal columns and (ii) vectors w 1 , . . . , w N ∈ R m , such that θ i = Bw i for all i ∈ [ N ] .

Given a semi-orthonormal matrix U ∈ R d × m (i.e., U ⊤ U = I m ), denote by U ⊥ ∈ R d × ( d -m ) a matrix whose columns constitute an orthonormal basis of the orthogonal complement of span( U ) , where we break ties in dictionary order 4 . We also denote the i -th column vector of U by U ( i ) .

Following Rusmevichientong and Tsitsiklis [2010] and [Qin et al., 2022], we also assume that the action set A is an ellipsoid and the task parameters have bounded ℓ 2 norms:

Assumption 2 (Linear bandits with ellipsoid action sets) . The action set A := { x ∈ R d : x ⊤ M -1 x ≤ 1 } is an ellipsoid, where M is a symmetric, positive definite matrix. In addition, there exist constants θ min and θ max ≤ 1 such that for all tasks n ∈ [ N ] , θ min ≤ ∥ θ n ∥ 2 ≤ θ max .

We define the expected pseudo-regret for task n as R n τ := τ · max a ∈A a ⊤ θ n -E [∑ τ t =1 A ⊤ n,t θ n ] , and the meta-regret for all N tasks as

The learner's goal is to sequentially interact with each task in a way that minimizes its meta-regret.

3 Algorithm

Unlike in the parallel setting or the sequential setting with a task diversity assumption, here, the learner cannot directly learn the global feature extractor B . Instead, it needs to reason with the uncertainty about B learned from the seen tasks.

High-level idea of our approach. To simultaneously learn B online and utilize our (imperfect) knowledge of it, we solve the sequential multi-task bandit problem using a bi-level approach:

At the lower level, for each task n , the learner has the option of invoking two base algorithms: one performs naive exploration that does not incorporate our knowledge on B , using a variant of fulldimensional linear bandit algorithm (PEGE [Rusmevichientong and Tsitsiklis, 2010], Algorithm 1); the other tries to incorporate a learned subspace ˆ B as prior knowledge to get reduced regret (Algorithm 2).

Algorithm 1 and 2 can be viewed as performing meta-exploration and meta-exploitation respectively: Algorithm 1, while ignoring the low-dimension property of the tasks, produces unbiased estimators

Algorithm 2 Meta-Exploitation procedure

1: Input: Task index n , exploration length τ 2 (a multiple of m ), and the subspace orthonormal basis ˆ B n ∈ R d × m

2: for i ∈ [ m ] do

3: Let A n,t = λ 0 ˆ B n ( i ) for t = u ( i -1) + 1 , · · · , ui , where u = τ 2 m

4: end for

5: for time step t ← 1 , · · · , τ 2 do

6: Take action A n,t and receive the reward r n,t

7: end for

8: Compute ˆ w n := argmin w ∈ R m 1 τ 2 ∑ τ 2 t =1 (〈 A n,t , ˆ B n w 〉 -r n,t ) 2

9: Let ˆ θ n := ˆ B n ˆ w n

10: for time step t ← τ 2 +1 , · · · , τ do

11: Take action A n,t ← arg max a ∈A 〈 a, ˆ θ n 〉

12: end for

of θ n that helps learn the task subspace B ; Algorithm 2 allows the learner to achieve a much lower regret when the subspace ˆ B approximately contains θ n ; however, using it may slow down the learning of B .

At the upper level, the learner has two decisions to make for each task n : (1) choosing between meta-exploration and meta-exploitation; (2) choosing a subspace ˆ B n to use if performing metaexploitation. To this end, we propose Algorithm 3, which aims at making these decisions in a feedback-driven way to ensure low meta-regret.

We now elaborate on each level in more detail.

The lower level. As mentioned above, Algorithm 1 is for meta-exploration. When invoked in task n , it can achieve two goals simultaneously: obtaining an unbiased estimate of θ n , while maintaining a reasonable regret guarantee for task n . For the first τ 1 steps (line 3 to 6), the learner takes actions { λ 0 e i } d i =1 that span the action space A , where e i is the i -th canonical vector of R d and λ 0 = √ λ min ( M ) is a constant factor that ensures λ 0 e i ∈ A (recall Assumption 2).

Then, the learner estimates task parameter ˆ θ n in line 8 and acts greedily for the rest of the task (line 10). We summarize its guarantee (originally due to Rusmevichientong and Tsitsiklis [2010]) as follows:

Lemma 3. Fix τ 1 to be a multiple of d . Suppose Algorithm 1 is run on task n with the exploration length τ 1 . Then, there exists some constants c 1 , c 2 > 0 (that depend on λ 0 , θ max , θ min , and M ) such that:

The regret on task n is bounded as R n τ ≤ c 1 · ( τ 1 + τ · d 2 τ 1 ) =: C info ;

We defer the proof of this lemma to Appendix D. Lemma 3 reveals a tradeoff between metaexploration and regret minimization for task n : if τ 1 is larger (say, closer to τ ), ˆ θ n estimates θ n more accurately; however, this may yield a worse bound on R n τ .

On the other hand, Algorithm 2 is for meta-exploitation. It takes a subspace (represented by its orthonormal basis ˆ B ) as input, and when invoked in task n , it can achieve a lower regret guarantee than Algorithm 1 when the subspace contains vectors that closely approximate θ n . Instead of exploring R d , the learner only explores the subspace induced by ˆ B n (lines 3 and 6). Then, the learner estimates the low-dimensional task parameter ˆ w n in line 9 and acts greedily for the rest of the task (line 11). We summarize its guarantee (originally due to Yang et al. [2020]) as follows:

Lemma 4. Fix τ 2 to be a multiple of m . Suppose Algorithm 2 is run on task n with input subspace ˆ B n and the exploration length τ 2 . Then, there exists some constant c > 0 (that depends on λ 0 , θ max , θ min ,

Algorithm 3 BOSS : Bandit Online Subspace Selection for Sequential Multitask Linear Bandits. The full algorithm 4 is in Appendix A.

1: Input: Task length τ , number of task N , task dimension d , subspace dimension m , and exploration rate p . 2: Initialize: An uniform distribution D 0 over the set of experts E ε in Definition 5 3: for n ∈ [ N ] : do 4: Randomly draw ˆ B n from D n 5: With probability p : Z n = 1 , otherwise Z n = 0 6: if Z n = 1 then 7: Exploration procedure in Algorithm 1 8: Update the distribution D n +1 with the EWA algorithm [Freund and Schapire, 1997], with learning rate η = ln 2 9: else 10: Exploitation procedure in Algorithm 2 with ˆ B n 11: end if 12: end for

and M ), such that the regret on task n is bounded as:

Specifically, if ∥ ˆ B ⊤ n, ⊥ θ n ∥ 2 ≤ 2 α , then R n τ ≤ 4 c ( τ 2 + τ · ( m 2 τ 2 + α 2 )) , where α is defined in Lemma 3.

Lemma 4 reveals the opportunistic nature of Algorithm 2: if θ n is perfectly contained in the subspace spanned by ˆ B n , ∥ ˆ B ⊤ n, ⊥ θ n ∥ = 0 and the regret bound is R n τ ≤ O ( τ 2 + τ · m 2 τ 2 ) , which can be as low as O ( m √ τ ) ; on the other extreme, ∥ ˆ B ⊤ n, ⊥ θ n ∥ can be as large as ∥ θ n ∥ in the worst case, which yields a trivial linear regret bound. Thus, its regret guarantee hinges on good choices of subspace ˆ B n as input. See Appendix E for the proof of Lemma 4.

The upper level. For the upper level, we propose Algorithm 3 that decides (1) when to perform meta-exploration and (2) the subspace to use if performing meta-exploitation.

For (1), for each task, the learner chooses to explore the subspace with probability p (line 6) or exploit with the online subspace estimate ˆ B n (line 9).

For (2), we propose to choose { ˆ B n } N n =1 that can optimize the following cost function online 5 :

The motivation behind this definition of C n is as follows: according to Lemma 4, C n ( ˆ B n ) is (up to constant) an upper bound of the regret of the learner at task n , were the learner to invoke Algorithm 2 using ˆ B n for this task. Therefore, if we can guarantee ∑ N n =1 C n ( ˆ B n ) to be small, then using Algorithm 2 for all tasks yields a small meta-regret.

An immediate challenge in directly optimizing C n defined in Eq. (2) is its dependence on unobserved quantity θ n . This challenge is further complicated by the following: (i) at best, we observe ˆ θ n 's that are α -approximations of θ n (e.g. in those meta-exploration tasks, see Lemma 3); (ii) in metaexploitation tasks, we do not have guarantees on how close ˆ θ n is to θ n ; note the difference in the definitions of ˆ θ n in meta-exploration and meta-exploitation tasks, respectively.

To address the challenge (i), we propose to optimize the following surrogate cost function for C n :

In Lemma 11 of Appendix C, we show that, with high probability, ˜ C n is an upper bound of C n (when ˆ θ n is close to θ n as guaranteed by Lemma 3 in meta-exploration rounds).

With this modification of the optimization objective, challenge (ii) persists: ˜ C n is only a valid high-probability upper bound of C n for all meta-exploration rounds n ; to ensure this upper bound property, we propose to optimize the cost:

where we introduce the importance weighting multiplier Z n p . Our key observation is that, although ¯ C n ( B ) is no longer an upper bound of C n ( B ) , the upper bound property holds in-expectation ; to see this, observe that for any fixed B ,

here, the first equality is from the definition of ¯ C n ( B ) ; the inequality (here, ≳ indicates greater than up to a negligible constant) uses the above-mentioned property that when Z n = 1 , with high probability, ˜ C n ( B ) ≥ C n ( B ) (see Lemma 3 and Lemma 11); the second equality is from the fact that Z n ∼ Ber( p ) .

Following the online learning literature, we propose to use the exponential weight algorithm (EWA, aka Hedge) [Freund and Schapire, 1997] to optimize { ¯ C n ( B ) } online. Ideally, we would like to run EWA with the B = { B : B ∈ R d × m s.t. B ⊤ B = I m } , the set of all m -dimensional subspaces; however, this is impossible because |B| is infinite. So instead, we propose to run EWA with the expert set defined as an ε -cover of B :

Definition 5. E ε is said to be a ε -cover over the set of B in the principal angle sense, if:

Definition 5 is motivated by the well-known fact that ∥ B ⊤ ⊥ B ′ ∥ F is the Frobenius norm of the sine of the principal angle matrix between subspaces spanned by B and B ′ . In Appendix C.1 we show how to construct E ε and its size |E ε | ≤ ( √ dm/ε ) O ( dm ) . In subsequent discussions, we will assume that BOSS uses such an E ε .

Define a constant shift and scaling of ¯ C n , ℓ n ( B ) := p C miss [ ¯ C n ( B ) -C hit Z n p ] ; we note that any regret guarantee over sequence { ℓ n } immediately translates to a regret guarantee over sequence { ¯ C n } . By the construction of the expert set E ε , sequence { ℓ n } is realizable with high probability: there exists some B ε ∈ E ε such that ∑ N n =1 ℓ n ( B ε ) = 0 ; this allows EWA to achieve a constant regret guarantee, summarized as follows:

Lemma 6. Let ε = α = c 2 d √ ln d δ τ 1 (with c 2 defined in Lemma 3) and δ = 1 N 2 , where c is a constant in Lemma 13. Then, assuming that τ ≫ d 2 , Algorithm 3 chooses a sequence of subspaces { ˆ B n } over the expert set E ε , defined in Definition 5, such that:

Note that the cumulative cost bound of { ˆ B n } has two terms: the benchmark term N C hit and the regret bound term C miss log |E ε | p . The benchmark term represents the best-case regret bound one can

achieve if, for all task n , the chosen subspace ˆ B n can well approximate θ n (i.e. ∥ ˆ B ⊤ n, ⊥ ˆ θ n ∥ ≤ α ) On the other hand, the regret-bound term depends on a few important factors: first, the cost decreases when the meta-exploration probability p increases - this matches our intuition that a larger p gives more frequent feedback to learn about θ n , allowing the learned ˆ B n to adapt faster to θ n ; second, the cost depends logarithmically on the size of the expert set ln |E ε | , which is standard in the online learning from expert advice literature; third, the cumulative cost depends on the range of the instantaneous costs C miss .

Remark 1. The subspace selection game in the upper level resembles the partial monitoring problem [see Lattimore and Szepesvári, 2020, Chapter 37], where the learner does not directly observe the loss for its actions but receives signals from an environment according to an observation matrix. Here, in our subspace selection problem, the learner does not directly observe the chosen subspace's cost for most tasks, except when they choose to explore the subspace B . Unlike the traditional partial monitoring problem, here, the observation would be ˆ θ n , and the cost C n ( B ) depends on the chosen subspace B and θ n . This means that the cost matrix has an infinite number of columns (one for each B ) and the observation depends on the actions of the learner and the environment in a stochastic fashion, unlike a deterministic dependence in the original partial monitoring setting.

4 Performance Guarantees

We bound the meta-regret of Algorithm 3 in Theorem 7:

Theorem 7. With exploration probability p = min ( ( 2 m √ τ N ) 2 3 , 1 ) , by choosing ε = α = c 2 d √ ln d δ τ 1 (with c 2 defined in Lemma 3) , where δ = 1 N 2 , τ 1 = d · ⌊ min ( d √ τ p , τ ) /d ⌋ , τ 2 = m · ⌊ √ τ ⌋ , the meta-regret of the BOSS algorithm is bounded by:

In the meta-regret bound (5), we view the first two terms as the main terms and the last two as 'burn-in' terms. The first term, Nm √ τ is the cumulative regret bound of the oracle baseline, i.e. the idealized algorithm that takes advantage of the extra knowledge of B to achieve a O ( m √ τ ) regret for every task. The second term, N 2 3 τ 2 3 dm 1 3 is the main overhead for learning the representation B ; it grows sublinearly in N , and as a consequence, is dominated by the first term when the number of tasks N is very large (specifically, N ≫ d 3 √ τ m 2 ). Compared with multi-task regret bounds in the parallel setting [Yang et al., 2020, Hu et al., 2021], our dependence on N is admittedly weaker; nevertheless, to our knowledge, Theorem 7 is the first nontrivial result in the sequential setting without task diversity assumptions, which has not been studied before in [e.g., Qin et al., 2022].

For the burn-in terms, the Nd 2 term can be interpreted as a constant d 2 regret overhead per task; the τmd term can be interpreted as the learner sacrificing a constant number of tasks ( md tasks) to learn a good representation B . Observe that, in the less favourable situation where N < md or τ < d 2 , the burn-in terms would lead to regret bounds worse than the trivial Nτ .

Comparison with the individual single task baseline. Recall that the individual single-task baseline has a meta-regret of O ( Nd √ τ ) ; our meta-regret guarantee improves over this baseline when τ ≫ d 2 and N ≫ m √ τ . We leave broadening the parameter regimes when our guarantee outperforms the individual single-task baseline as an important open problem.

Comparison with lower bounds. Qin et al. [2022] showed a lower bound for the problem: Ω ( Nm √ τ + d √ mτN ) . We can see that there still exists a gap between our upper bound in Theorem 7 with this, and the gap is bigger than other solutions with task diversity assumption such as Qin et al. [2022]; we speculate that this is a price we pay due to not making any assumptions on task diversity.

We now sketch the proof of Theorem 7 below.

Proof sketch. Denote the pseudo-regret for task n as ˆ R n τ := τ max a ∈A ⟨ θ n , a ⟩ -∑ τ t =1 ⟨ θ n , A n,t ⟩ .

We decompose R τ as follows:

where the first two equalities are by the definition of R τ and algebra; the third equality uses the law of total expectation; the inequality uses Lemmas 3 and 4 to bound the first and second terms, respectively; the fourth equality is due to the definition of C info and Lemma 6; the last equality is by algebra.

The meta-regret of Algorithm 3 follows from the choices of τ 1 , τ 2 , and p - specifically, τ 2 balances the last two terms, whereas τ 1 and p aims at balancing the first three terms subject to the constraint that τ 1 ≤ τ and p ≤ 1 - see Appendix G for the remaining details.

Adaptivity to problem parameters. Algorithm 3 requires the knowledge of N and m , the total number of tasks and the dimensionality of the subspace underlying the task parameters. Below, we show that knowledge of N can be relaxed.

We can relax the need to know N by using the doubling trick on BOSS . Specifically, in phase i , we can run our algorithm with the assumption that there are 2 i total tasks in this phase. The modified algorithm has a meta-regret guarantee that is within a constant of the algorithm that knows N . This implicitly gives an adaptive setting of meta-exploration probability p that is decaying over time.

For m , the requirement can be relaxed to knowing an upper bound of m . Removing this knowledge requires a change of approach, such as low-rank matrix optimization, as in Cella et al. [2023] or additional assumption, as in Bilaj et al. [2024]. Cella et al. [2023] is in the parallel setting, which is not applicable here, and Bilaj et al. [2024]'s guarantee can be as large as O ( Nd √ τ ) as discussed in section 1.1.

5 Experiments

In this section 6 , we compare the performance of our BOSS algorithm with the baselines on synthetic environments. The algorithms we evaluate include:

PEGE : independently solves each task using the PEGE algorithm [Rusmevichientong and Tsitsiklis, 2010]

PEGE-oracle : The 'oracle baseline' that only uses PEGE on the true subspace B , for all tasks

SeqRepL : our implementation of [Qin et al., 2022], in which ˆ B n is estimated with SVD and the tasks for meta-exploration are chosen deterministically at round n = i ( i +1) 2 for i = 1 , 2 , · · · .

BOSS-no-oracle : Algorithm 3 with E ε set of 100,000 experts drawn uniformly at random from B .

BOSS : Algorithm 3 with E ε set as 100,000 experts drawn uniformly at random from B , plus the ground truth expert B . This algorithm is a better approximation of the original BOSS (Algorithm 3) since there exists B ε ∈ E ε such that ∥ ( B ε ) ⊤ ⊥ B ∥ F ≤ ε .

The setting is ( N,τ,d,m ) = (4000 , 500 , 10 , 3) . The environment reveals a new subspace dimension at tasks 1, 2501, and 3501. In this experiment, we assume A is a unit sphere, i.e., M = I d . At each task n , denote by B n ∈ R d × m n the subspace basis that the environment used to generate θ n , where m n is incremented when n = 1 , 2501 , 3501 . θ n is chosen in the following way: θ n = λ 1 B n w n for some w n ∼ Unif( S m n -1 ) and λ 1 ∼ Unif([0 . 8 , 1]) is a random scaling factor to ensures Assumption 2, where θ min = 0 . 8 , θ max = 1 .

The hyper-parameters p, τ 1 , τ 2 , and α of all algorithms, where it applies, are tuned. The error bands in the figures indicate ± 1 standard deviation computed over 5 independent runs.

Figure 1a clearly shows the linear dependency of the cumulative regret on N . Observe that BOSS and its variants outperform both the independent PEGE and the SeqRepL baselines. It is also clear that the gap between BOSS-no-oracle and BOSS exists because the expert set B ε used in this experiment does not cover the true B , since even with ε = d √ ≈ 0 . 5 , the theoretical size of the expert set is

τ 1 |E ε | = ( √ dm/ε ) dm ≈ 11 30 in this experiment setting which is much larger than the expert set size used in BOSS-no-oracle .

In Figure 1b, we plot ∥ ˆ B ⊤ n, ⊥ B n ∥ F , which measures the closeness of ˆ B n, ⊥ to B n . When the environment reveals a new subspace dimension at tasks 1, 2501, and 3501, all algorithms' estimation ˆ B n require updates and converge after a while. Even though BOSS-no-oracle has a worse estimation of ˆ B n compared to SeqRepL , it achieves a better regret due to having a better estimation of ˆ θ n as shown in Figure 1c.

(a)

(b)

(c)

700000

BOSS

BOSS-no-oracle

1.2

BOSS

BOSS-no-oracle

1.0

1.2

BOSS

BOSS-no-oracle

600000

PEGE

1.0

PEGE-oracle

0.8

Cum Regret

500000

400000

0.8

0.6

0.6

W

300000

0.4

0.4

200000

0.2

0.2

100000

00

0.0

0

0.2

0

500

1000

15002000

2500

3000

3500

4000

0

500

1000

1500

2000

2500

3000

3500

4000

0

500

1000

1500

2000

2500

3000

3500

4000

# of task

# of task

# of task

6 Discussion and Future Work

We study the problem of sequential representation transfer in multi-task linear bandit, where the task parameters are allowed to be chosen adversarially online. Our BOSS algorithm achieves the regret guarantee of ˜ O ( Nm √ τ + N 2 3 τ 2 3 dm 1 3 + Nd 2 + τmd ) without using the task diversity assumption as in previous works. This opens up many promising avenues for future work. Statistically, it would be good to design an algorithm that performs no worse than the individual single-task baseline's performance in all parameter regimes. In addition, BOSS utilizes the special structure of fixed, ellipsoid-shaped action spaces to obtain useful information for meta-exploration, extending the algorithm and guarantees to general and time-varying action spaces is an important direction. Practically, it would also be nice to design parameter-free variants of BOSS that do not require knowing m ahead of time. Furthermore, BOSS requires maintaining an exponentially large number of experts in B ε ; in the future, we would like to develop more computationally-efficient algorithms. Lastly, it would be interesting to study relaxations of Assumption 1 (all task parameters lie exactly in a m -dimensional linear subspace), similar to Bilaj et al. [2024] or the changing subspace setting of Qin et al. [2022].

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Naoki Abe and Philip M Long. Associative reinforcement learning using linear probabilistic concepts. In ICML , pages 3-11. Citeseer, 1999.

Javad Azizi, Thang Duong, Yasin Abbasi-Yadkori, András György, Claire Vernade, and Mohammad Ghavamzadeh. Non-stationary bandits and meta-learning with a small set of optimal arms. Reinforcement Learning Journal , 5:2461-2491, 2024.

Maria-Florina Balcan, Keegan Harris, Mikhail Khodak, and Zhiwei Steven Wu. Meta-learning adversarial bandits. arXiv preprint arXiv:2205.14128 , 2022.

Steven Bilaj, Sofien Dhouib, and Setareh Maghsudi. Meta learning in bandits within shared affine subspaces, 2024.

Leonardo Cella, Karim Lounici, Grégoire Pacreau, and Massimiliano Pontil. Multi-task representation learning with stochastic linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 4822-4847. PMLR, 2023.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. In COLT , volume 2, page 3, 2008.

Murat A Erdogdu and Mert Vural. Csc 2532 winter 2024: Statistical learning theory, lecture 5. https://erdogdu.github.io/csc2532/lectures/lecture05.pdf , 2024.

Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences , 55(1):119-139, 1997.

John C Gower and Garmt B Dijksterhuis. Procrustes problems , volume 30. OUP Oxford, 2004.

Jiachen Hu, Xiaoyu Chen, Chi Jin, Lihong Li, and Liwei Wang. Near-optimal representation learning for linear bandits and linear rl. In International Conference on Machine Learning , pages 4349-4358. PMLR, 2021.

Kyoungseok Jang, Kwang-Sung Jun, Se-Young Yun, and Wanmo Kang. Improved regret bounds of bilinear bandits using action space analysis. In International Conference on Machine Learning , pages 4744-4754. PMLR, 2021a.

Kyoungseok Jang, Kwang-Sung Jun, Se-Young Yun, and Wanmo Kang. Improved regret bounds of bilinear bandits using action space analysis. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4744-4754. PMLR, 18-24 Jul 2021b. URL https://proceedings. mlr.press/v139/jang21a.html .

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Yuzhen Qin, Tommaso Menara, Samet Oymak, ShiNung Ching, and Fabio Pasqualetti. Non-stationary representation learning in sequential linear bandits. IEEE Open Journal of Control Systems , 1: 41-56, 2022.

Paat Rusmevichientong and John N Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research , 35(2):395-411, 2010.

Matus Telgarsky. Deep learning theory lecture notes. https://mjt.cs.illinois.edu/dlt/ , 2021. Version: 2021-10-27 v0.0-e7150f2d (alpha).

Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In International Conference on Machine Learning , pages 10434-10443. PMLR, 2021.

Trung Vu. Matrix perturbation and davis-kahan theorem. https://trungvietvu.github.io/ notes/2020/DavisKahan , 2020.

Jiaqi Yang, Wei Hu, Jason D Lee, and Simon Shaolei Du. Impact of representation learning in linear bandits. In International Conference on Learning Representations , 2020.

Jiaqi Yang, Qi Lei, Jason D Lee, and Simon S Du. Nearly minimax algorithms for linear bandits with shared representation. arXiv preprint arXiv:2203.15664 , 2022.

A Further discussion on related work

Meta learning bandit problems. Balcan et al. [2022] analyzed a harder problem in the Sequential setting by dealing with tasks generated by an adversary, and each task is an adversarial linear bandit problem. By using an online mirror decent base algorithm, they provided a guarantee of ˜ O ( min 1 τ ≤ ϵ ≤ 1 √ τ N ( d ˆ V ϵ √ τ + ϵτ ) + N 3 4 τ 2 d ) , where ˆ V ϵ measures the proximity of the optimal parameters of all N tasks. Balcan et al. [2022]'s results also applies to the sequential meta-learning with adversarial K -arms bandit, which is further investigated by Azizi et al. [2024]. Their bandit meta-learning with a small set of optimal arms setting is analogous to the sequential multi-task representation transfer in linear bandit. The task diversity assumption is equivalent to the Azizi et al. [2024]'s Efficient Identification assumption; both require a gap to separate the noise from the problem's parameters. Our BOSS algorithm also has some high-level similarities with their E-BASS algorithm.

Bilinear bandits. The bilinear bandit problem described by Jang et al. [2021b] studies a setting where the learner has (potentially time-varying) sets of left actions and right actions available. It can take a left action x L and a right action x R and receive reward r = x ⊤ L Θ x R + η . This reduces to our setting when the action set of the left action is the task descriptor { e i } i =1 , ··· ,N and the task's parameters have a low-rank structure. When applying their approach to our setting, their guarantee is ˜ O ( Nd √ τ + N 3 2 √ dτ ) , which is worse than independently using a classical algorithm, such as PEGE, for each task. This is due to the fact that Jang et al. [2021b]'s solution does not exploit the low-rank and the left action structures.

In Table 2, we comprehensively compare the settings and assumptions of the previous works for the multi-task bandit representation transfer problem.

B Additional information on Algorithm 3

C Additional details related to Section 3

We first provide more details on the construction of the expert set used in Section 3.

To construct E ε , an ε -cover of B in the principal angle sense (Definition 5) , we do the following:

Construct E ε 2 F , a proper ε 2 -cover over ( B S , ∥ · ∥ F ) , where B S is defined in Definition 8 (see below). Since B ⊂ B S , E ε 2 F is an also improper ε 2 -cover over ( B , ∥ · ∥ F ) . 7

Construct E ε from E ε 2 F and show that it is a proper ε -cover over ( B , ∥ · ∥ F )

Show that E ε is a proper ε -cover over ( B , ∥ · ⊤ ⊥ ·∥ F ) .

We provide the details in Subsection C.1.

Definition 8. E ε 2 F is said to be an ε 2 -cover over the set of B S = { B : ∥ B ∥ F ≤ √ m } in the Frobenius norm sense, if:

After defining the expert set, to use the EWA algorithm, we need to use a surrogate expert's cost ˜ C n defined in Equation (3), which is a high probability upper-bound of the true cost C n ( B ) following Lemma 11. To construct ˜ C n , we need to introduce the definition of α -covering in Definition 9.

Definition 9. A subspace (represented by its orthonormal basis) B ∈ R d × m is said to α -approximately cover vector ϕ ∈ R d , if ∥ B ⊤ ⊥ ϕ ∥ 2 ≤ α .

Note: the specific choice of B ⊥ does not affect the validity of this definition - e.g., ∥ B ⊥ ϕ ∥ 2 are always the same regardless of the specific choice of B ⊥ , as long as its columns form a orthonormal basis of span( B ⊥ ) : for any two choices of B ⊥ (denoted by B ⊥ , 1 , B ⊥ , 2 , respectively) there exists orthonormal V ∈ R ( d -m ) × ( d -m ) such that B ⊥ , 1 = B ⊥ , 2 V . Therefore,

The following lemma justifies that all valid choices of B ⊥ are equivalent up to a ( d -m ) × ( d -m ) orthogonal transformation:

Lemma 10. Let W be a k -dimensional subspace of R d . Let B, ˆ B ∈ R d × k be matrices whose columns form an orthonormal basis of W . Then, there exists an orthogonal matrix V ∈ R k × k such that ˆ B = BV .

Proof. Since B is a basis of W , there exists some V such that ˆ B = BV . Since V ⊤ V = V ⊤ ( B ⊤ B ) V = ( BV ) ⊤ ( BV ) = ˆ B ⊤ ˆ B = I , V is an orthogonal matrix.

Next, we justify the use of ˜ C n as a surrogate cost for the true cost C n in Lemma 11, which requires Remark 2.

Remark 2. By the observation that

B α -approximately covers ϕ if and only if there exists some θ in span( B ) that is α -close to ϕ . As a result:

If θ n ∈ span( B ) and ∥ θ n -ˆ θ n ∥ ≤ α , B also α -covers ˆ θ n ;

If B α -covers ˆ θ n and ∥ θ n -ˆ θ n ∥ ≤ α , B also 2 α -covers θ n .

Lemma 11. If ∥ ˆ θ n -θ n ∥ ≤ α , ˜ C n is an upper bound of C n . Equivalently,

Proof. Recall that

First, observe that ˜ C n ( B ) ≥ C hit for any B . Then, we conduct a case analysis:

Case 1: ∥ B ⊤ ⊥ θ n ∥ 2 ≤ 2 α . In this case, C n ( B ) = C hit ≤ ˜ C n ( B ) .

Case 2: ∥ B ⊤ ⊥ θ n ∥ 2 > 2 α . In this case, C n ( B ) = C miss . We now show that ˜ C n ( B ) = C miss . Indeed,

where the first inequality is by triangle inequality; the second inequality is by the definition of this case and that ∥ B ⊤ ⊥ ( ˆ θ n -θ n ) ∥ 2 ≤ ∥ ˆ θ n -θ n ∥ ≤ α .

In summary, in both cases, C n ( B ) ≤ ˜ C n ( B ) .

To ensure that there exists some B ∈ E ε that can α -cover all ˆ θ n , we need to choose ε by following Lemma 12.

Lemma 12. By choosing ε ≤ α , there exists some B ε ∈ E ε such that, for all n , B ε α -approximately covers θ n .

Proof. For any n ∈ [ N ] , θ n ∈ span( B ) . Hence, θ n = P B θ n . Thus,

Hence, assuming that θ max ≤ 1 , by setting ε ≤ α , we have that min U ∈E ε ∥ ϕ -UU ⊤ ϕ ∥ 2 ≤ α .

C.1 The expert set E ε : construction and size

First, we construct the surrogate set E ε 2 F , a proper ε 2 -cover over ( B S , ∥ · ∥ F ) . Following Erdogdu and Vural [2024], we construct it by discretizing the hyper-cube that contains the ball E ε 2 F . Then |E ε 2 F | ≤ ( 4 m √ d ε ) dm = ( dm ε ) O ( dm ) . Note that for any B ∈ B , ∥ B ∥ F = ∥ vec( B ) ∥ 2 = √ m ; therefore, B ⊂ B S , and thus E ε 2 F is an improper ε 2 -cover of B .

Then, we follow a procedure similar to [Telgarsky, 2021, Remark 15.3] to construct E ϵ as follows:

where U ( b ) := argmin B ∈B ∥ B -b ∥ F . In words, E ϵ is the collections of 'nearest neighbors' of E ϵ in B . By the definition of E ϵ , E ϵ ⊂ B and |E ϵ | ≤ |E ε 2 F | ≤ ( dm ϵ ) O ( dm ) .

For the rest of the subsection, we will show that our construction of E ϵ satisfies Definition 5. Lemma 13.

E ε is a proper ε -cover of ( B , ( A,B ) ↦→∥ A -B ∥ F ) .

E ε is a proper ε -cover of ( B , ( A,B ) ↦→∥ A ⊤ ⊥ B ∥ F ) .

Proof. For the first item, we consider any B ∈ B . We would like to show that there exists some element C in E ϵ such that ∥ C -B ∥ F ≤ ε .

We find element C as follows: first, by definition of E ε 2 F , there exists element b ∈ E ε 2 F such that ∥ B -b ∥ F ≤ ε 2 . We define C = U ( b ) . Therefore, ∥ C -b ∥ ≤ ∥ B -b ∥ F ≤ ε 2 . Thus:

Hence, E ε is a proper ε -cover over ( B , ∥ · ∥ F ) .

We now prove the second item. It suffices to show that, for A,B ∈ B , ∥ A ⊤ ⊥ B ∥ 2 F ≤ ∥ A -B ∥ 2 F .

Following [Vu, 2020], for semi-orthogonal matrices A,B ∈ B , we relate ∥ A ⊤ ⊥ B ∥ F to the orthogonal Procrustes problem in [Gower and Dijksterhuis, 2004],

With the solution R ∗ = A ⊤ B ( B ⊤ AA ⊤ B ) -1 2 at the optimum,

We have ∥ A ⊤ ⊥ B ∥ 2 F ≤ ∥ AR ∗ -B ∥ 2 F ≤ ∥ A -B ∥ 2 F .

D Proof of Lemma 3: Regret of Meta-exploration Tasks

We first restate Lemma 3.

Lemma 3. Fix τ 1 to be a multiple of d . Suppose Algorithm 1 is run on task n with the exploration length τ 1 . Then, there exists some constants c 1 , c 2 > 0 (that depend on λ 0 , θ max , θ min , and M ) such that:

The regret on task n is bounded as R n τ ≤ c 1 · ( τ 1 + τ · d 2 τ 1 ) =: C info ;

With probability ≥ 1 -δ , ∥ ˆ θ n -θ n ∥ ≤ c 2 · ( d √ ln d δ τ 1 ) =: α .

Proof. For the first item, following [Rusmevichientong and Tsitsiklis, 2010, Lemma 3.4], we have

where c 0 is a constant that depends on λ 0 and M . By [Yang et al., 2020, Lemma 17], we have that max a ∈A ⟨ a -A n,t , θ n ⟩ ≤ J ∥ θ n -ˆ θ n ∥ 2 / ∥ θ n ∥ , where A n,t = argmax a ∈A 〈 a, ˆ θ n 〉 and J = λ max ( M ) √ λ min ( M ) = λ max ( M ) λ 0 . Thus,

The proof of the first item is concluded by taking c 1 = max( λ 0 θ max , Jc 0 θ min ) .

For the second item, recall that u = τ 1 d , and A n, 1 , . . . , A n,τ 1 are constructed as follows: for each i ∈ [ d ] and t ∈ { u ( i -1) + 1 , · · · , ui } , A n,t = λ 0 e i .

Since ˆ θ n := argmin θ 1 τ 1 ∑ τ 1 t =1 ( ⟨ A n,t , θ ⟩ -r n,t ) 2 , by the closed-form solution of ordinary least squares, we have ˆ θ = ( A ⊤ A ) -1 A ⊤ r , where

and r n := ( r n, 1 , · · · , r n,τ 1 ) . Observe that

Let η n := ( η n, 1 , . . . , η n,τ 1 ) . We have

where both (a) and (b) follow from Eq. (6).

It now suffices to show that ∥ ∥ ∥ ˆ θ n -θ n ∥ ∥ ∥ 2 = ∥ ∥ ∥ 1 uλ 2 0 A ⊤ n η n ∥ ∥ ∥ 2 ≤ O ( d √ log( d/δ ) τ 1 ) with probability at least 1 -δ . To this end, observe that by the construction of A n ,

Since for each n and t , η n,t is zero-mean and 1-sub-Gaussian, by [Lattimore and Szepesvári, 2020, Corollary 5.5], for any i ∈ [ d ] , we have

Let δ = dδ ′ and

Then, by the union bound, F n happens with probability at least 1 -δ . Under the event F n ,

and the proof of the second item is complete.

E Proof of Lemma 4: Regret of Meta-exploitation Tasks

We first restate Lemma 4.

Lemma 4. Fix τ 2 to be a multiple of m . Suppose Algorithm 2 is run on task n with input subspace ˆ B n and the exploration length τ 2 . Then, there exists some constant c > 0 (that depends on λ 0 , θ max , θ min , and M ), such that the regret on task n is bounded as:

Specifically, if ∥ ˆ B ⊤ n, ⊥ θ n ∥ 2 ≤ 2 α , then R n τ ≤ 4 c ( τ 2 + τ · ( m 2 τ 2 + α 2 )) , where α is defined in Lemma 3.

Proof. Similar to the approach in [Rusmevichientong and Tsitsiklis, 2010], define J = λ max ( M ) √ λ min ( M ) = λ max ( M ) λ 0 . By [Yang et al., 2020, Lemma 17], we have max a ∈A ⟨ a -A n,t , θ n ⟩ ≤ J ∥ θ n -ˆ θ n ∥ 2 / ∥ θ n ∥ , where A n,t = argmax a ∈A 〈 a, ˆ θ n 〉 . Thus,

To bound the second term, we use Lemma 14 (subspace-informed estimation). We have

It follows that

In addition, when ∥ ˆ B ⊤ n, ⊥ θ n ∥ 2 ≤ 2 α ,

We now present Lemma 14 used in the proof above for subspace-informed estimation; see also [Qin et al., 2022, Lemma 2] and [Yang et al., 2020, Lemma 18].

Lemma 14 ( Subspace-informed estimation ) . Suppose Algorithm 2 is run on task n with the exploration length τ 2 , then E ∥ ˆ θ n -θ n ∥ 2 ≤ m 2 λ 2 0 τ 2 + ∥ ˆ B ⊤ n, ⊥ θ n ∥ 2 .

Proof. Without loss of generality, we assume that τ 2 is a multiple of m . Since A n,t = λ 0 ˆ B n ( i ) , i ∈ [ m ] , and each action repeats ⌊ τ 2 /m ⌋ times for t ≤ τ 2 , we have ∑ τ 2 t =1 A n,t A ⊤ n,t = τ 2 λ 2 0 m ˆ B n ˆ B ⊤ n . Thus,

Then, the OLS estimator is given by

where the first equality uses the closed-form solution of OLS; the second equality is by the definition of r n,t ; and the other equalities follow from algebraic manipulations. Now, we have

For s 1 , we have

For s 2 , we have

Hence,

F Proof of lemma 6: Regret of the subspace selection game

Lemma 6. Let ε = α = c 2 d √ ln d δ τ 1 (with c 2 defined in Lemma 3) and δ = 1 N 2 , where c is a constant in Lemma 13. Then, assuming that τ ≫ d 2 , Algorithm 3 chooses a sequence of subspaces { ˆ B n } over the expert set E ε , defined in Definition 5, such that:

Proof. Recall the guarantee of EWA from Freund and Schapire [1997]:

Theorem 15 (Freund and Schapire [1997]) . For any sequence of loss vectors ℓ 1 , · · · , ℓ T and the initial weights of the experts are w i 1 = 1 /n for all i ∈ [ n ] and γ ∈ (0 , 1) , the EWA algorithm with learning rate η = -ln(1 -γ ) generates { p t } T t =1 such that its expected loss is bounded by

specifically, if at each round we choose i t ∼ p t ,

Applying Theorem 15 with T = N , expert set E ε , i t = ˆ B n , loss functions ℓ n ( B ) = p C miss [ ˜ C n ( B ) Z n p -C hit Z n p ] , n ∈ [ N ] , B ∈ E ϵ , and the baseline expert i = B ε , we get:

Using the basic fact that -ln(1 -x ) x ≤ 1 + x for x ∈ (0 , 1 2 ] we have that, for γ ∈ (0 , 1 2 ] ,

Define F n = { Z n = 0 ∨ ( Z n = 1 ∧ ∥ ˆ θ n -θ n ∥ ≤ α ) } and F = ∩ N n =1 F n . Then, when all of our estimations ˆ θ n are accurate, we have: I ( F ) ∑ N n =1 ℓ n ( B ε ) = 0 for ℓ n ( B ) = p C miss [ ¯ C n ( B ) -C hit Z n p ] and B ε ∈ E ε .

By the definition of F n , we have: F c n = { Z n = 1 ∧ ∥ ˆ θ n -θ n ∥ > α } . Thus, by Lemma 3, P ( F c n ) = P ( ∥ ˆ θ n -θ n ∥ > α | Z n = 1) P ( Z n = 1) ≤ pδ , and therefore P ( F ) ≥ 1 -∑ N n =1 P ( F c n ) = 1 -Npδ . Since Algorithm 3 chooses the learning rate for EWA as η = log(2) , thus, γ = 1 -exp( -η ) = 0 . 5 , we have

Then,

Thus,

where, for the last inequality, we use the observation that E [ C n ( ˆ B n ) · I ( F c n ) · Z n p ] ≤ C miss p E [ I ( F c n )] ≤ C miss p P ( F c n ) ≤ C miss δ ≤ C miss N 2 , since P ( F c n ) ≤ pδ ≤ p N 2 . Hence,

Substituting C hit and C miss in Equation (2) to complete the proof.

G Theorem 7: meta-regret guarantee

Theorem 7. With exploration probability p = min ( ( 2 m √ τ N ) 2 3 , 1 ) , by choosing ε = α = c 2 d √ ln d δ τ 1 (with c 2 defined in Lemma 3) , where δ = 1 N 2 , τ 1 = d · ⌊ min ( d √ τ p , τ ) /d ⌋ , τ 2 = m · ⌊ √ τ ⌋ , the meta-regret of the BOSS algorithm is bounded by:

Remainder of the Proof of Theorem 7. Recall that in the proof sketch of Theorem 7 (Section 4), we have proved that

Now, by the choice of τ 2 = m · ⌊ √ τ ⌋ ,

Wewant to tune the parameters p, τ 1 to minimize the meta-regret subjected to the constraint: p ∈ [0 , 1] and τ 1 ∈ [0 , τ ] .

The meta-regret is:

H Additional experiment result

H.1 Adversarial environment for SeqRepL 's deterministic exploration schedule

×106

6

BOSS

PEGE

PEGE-oracle

BOSS-no-oracle

1.2

1.4

0.8

1.0

BOSS-no-oracle

BOSS

5

1.0

Cum Regret

3

0.8

0.6

BOSS

BOSS-no-oracle

0.4

°g-

1

0.6 -

0.4

2

0.2

0.2

0.0

0

0.2

F 0°0

0

1000

2000

3000

4000

5000

6000

0

1000

2000

3000

4000

5000

6000

0

1000

2000

3000

4000

5000

6000

of task

of task

of task

H.2 When the Task Diversity assumption is satisfied

×106

3.5

3.0

BOSS

PEGE.oracle

BOSS-no-oracle

PEGE

1.50

1.25

BOSS

BOSSnouoracle

0.8

BOSS

BOSS-no-oracle

2.5

1.00

0.6

Cum Regret

2.0 -

1.5

0.50

0.75

1°

0.4

F 0°T

0.25

0.2

0.5

0.00

0.0

0.25

F 0°0

0

1000

2000

of task

3000

4000

5000

6000

0

1000

2000

# of task

3000

4000

5000

6000

0

1000

2000

f of task

3000

4000

5000

6000

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the abstract, the end of Section 1 (and Section 4 and 5 for elaboration) on a summary of the contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section 4 and Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the necessary assumptions and proof are shown in the main paper and Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We will publish the code in the future, including all the hyper-parameters to reproduce the results in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code for our paper can be found at https://github.com/ duongnhatthang/BOSS .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment information is provided in Section 5 and the code with all the parameters can be found at https://github.com/duongnhatthang/BOSS .

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: This is shown in Section 5, where we provide a standard deviation error over five randomized experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The experiment is relatively small and simple, and the paper's main focus is the theoretical analysis.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and found no violation.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work focus on the theoretical analysis and poses no significant societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks. Our dataset is synthetic.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All authors and citations are mentioned, to the best of our knowledge.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We don't provide any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not require IRB Approvals or Equivalent for Research with Human Subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the existing work on multi-task linear bandits where the task parameters lie in a low rank subspace. Specifically, this paper assumes $N$ $d$-dimensional linear bandits with parameters lie in an $m$-dimensional subspace. For such a setting, the classical approaches yield a regret linear in $N$ and $d$. This paper then proposed a new algorithm that improves the regret with mild assumptions of well-conditioned ellipsoids of the tasks and bounded task parameters. The algorithm is based on a reduction to a bandit online subspace selection problem. The key idea of the proposed approach is bi-level. At the lower level, the algorithm performs either an meta-exploration algorithm or a meta-exploitation algorithm for each of the tasks. At the upper level, the learner either chooses exploration/exploitation or chooses a subspace to use for exploitation. The effectiveness of the proposed algorithm is verified empirically on synthetic adversarial settings.\n\nSTRENGTHS:\n**Significance**: This paper proposed a new algorithm for multi-task linear bandit with provable low regret without making strong assumptions on task parameters. The sequential setting studied is more challenging than parallel settings due to the slow revelation of the underlining subspace. Compared with a few related existing works, this paper does not need assumptions such as task diversity.\n\n**Quality**: this paper is in general good quality. The algorithm is well described.\nThe theoretical part of the work is clearly formulated. Thought the experiments are synthetic, the result still verifies the effectiveness of the proposed algorithm.\n\n**Clearity**: The design of the algorithm and intuition behind the design is clearly explained.\n\nWEAKNESSES:\nThere is no major weakness in the paper. I do have some questions regarding the following:\n\n**1**. This paper assumes knowledge of the parameters corresponding to the task number, the dimension, and the number of rounds, which is not ideal. \n\n**2**. [Originality] The algorithmic design and the guarantee is closely related to a a few works, including the PEGE by [Rusmevichientong and Tsitsiklis 2010] (related to lemma 3 in this paper), the analysis in Yang et al [2020] (related to lemma 4), EWA algorithm (related to lemma 5). Therefore, the technical novelty of the proposed algorithm and analysis is not very obvious to me.\n\nQUESTIONS:\nPlease see weaknesses.\n\n\nAlso, I am curious about the following:\n\n1. This paper assumes knowledge to a few hyper-parameters. Among these, I wonder whether the assumption on the prior knowledge of the $\\tau$ parameter can be relaxed with certain adaptive design. Could the author elaborate on why knowledge is necessary for the current algorithm to work?\n\n2. Is there a corollary or simple extension that can readily extend the current result to the parallel setting?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies lifelong learning in linear bandits and designs a two-level algorithm with provable low regret without task diversity assumptions. The paper assumes the tasks share a low-rank representation and provides a regret upper bound.\n\nSTRENGTHS:\nThe paper uses a two-level approach to solve this problem, which is novel. A theoretical guarantee is given with numerical experiments performed.\n\nWEAKNESSES:\nThere are some statements in the theorems that seem wrong and the upper bound provided does not seems tight. See Questions for details.\n\nQUESTIONS:\n1) In line 109, for a matrix $U$, isn't $U_{\\perp}$ not unique? This should be defined as a set or stated as one of the matrices.\n2) The Assumption 2 is stated as linear bandits. I didn't get the reason for this title.\n3) In line 3 in Algorithm 1, it should be $\\mu=\\frac{\\tau_1}{d}$\n4) The statement in Lemma 4 seems problematic. As stated in 1), the matrix $\\hat B_{n,\\perp}$ is not unique, hence the RHS is a matrix-dependent bound. Thus, the condition $||\\hat B_{n,\\perp} \\theta_{n}||_2\\leq 2\\alpha$ is also dependent on the choice of the matrix. This discussion is missing in the paper. This will also affect the definitions in (2) and (3), and the following results.\n5) In line 173, the definition of the expert set $\\mathcal{E}^{\\epsilon}$ is in the Appendix, which makes the paper not self-contained.\n6) What does the EWA algorithm do in Algorithm 3? Which part of the results is directly from the EWA algorithm?\n7) Overall, the paper provides the regret bounds of order  $\\tilde{\\mathcal{O}} (N m\\sqrt{\\tau} + N^{\\frac{3}{2}} \\tau^{\\frac{2}{3}} d m^{\\frac{1}{3}} + \\tau m d)$ in equation (5),  times of interactions. Compared to the baseline, which is $\\tilde{\\mathcal{O}} (N d \\sqrt{\\tau})$, I didn't see a clear improvement in the regret/ Firstly, the regret scales linear with $\\tau$ when fix $N$, which is known to be suboptimal in linear bandit. At least some truncation can be done in the first $md$ tasks to get rid of the linear term. This term cannot be hidden from the abstract from my point of view. Secondly, to achieve similar upper bounds, one needs $N\\geq md$ and $d^2\\leq\\tau\\leq (\\frac{N}{m})^2$, which means the algorithm can only learn for a finite time and the tightness discussion is missing the paragraph Comparison with lower bounds with only a conjecture.\n7) The author answer Yes to the Question about Open access to data and code in the Checklist, but the code is not released during the submission. And I am curious how Figure 1B is generated.\n9) The labels on the y-axis in the figures need to be fixed.\n\nLIMITATIONS:\nThe regret upper bounds has a hugh gap to the lower bound.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a strategy for the multi task linear bandit problem, where the tasks are assumed to share a low rank representation, which is ought to be learnt via a new meta learning strategy, where meta exploration and exploitation needs to be balanced. The low rank representation is learnt via optimizing a newly constructed loss function over an epsilon-net. The tasks appear sequentially to the learner and the authors explicitly do not make any assumption on the task diversity of their setting.\nThe authors provide a meta regret bound for their algorithms and synthetic data results.\n\nSTRENGTHS:\nThe paper is well written and easy to follow\n\nOriginal idea for solving the multi-task/meta learning setting with low rank representation\n\nThe assumption on the action set is weaker than in comparable works (constant action set vs iid action set)\n\nWEAKNESSES:\nThe EWA algorithm for the meta learning procedure should not just be referenced but actually appear in the paper (or at least in the supplementary)\n\nThe presentation of the actual meta learning could be made clearer. For that matter, definition 7 and 8 should be moved to the main paper so it becomes clearer what the uniform distributions D_n are supposed to be.\n\nThe experiments would benefit if there were another baseline to compare to (Cella et al or Bilaj et al as they appear in the related work section)\n\nThe plot labels of figure 1 b) and c) should have latex style writing\n\nQUESTIONS:\nIn Figure 1a) it looks like the Boss algorithm performs worse after a new dimension for the subspace is incremented at n= 2501, does that mean that the dimension of the subspace was falsely estimated?\n\nat n=1 the the low rank should be estimated as m=1 as well, which makes it hard to minimize $B^T_{n+1,\\perp}\\theta_{n+1}$, (Since the n+1th task parameter might be outside the subspace selected for the first n tasks), Intuitively, the algorithm should work best if at least m task parameters are already bias-less estimated for a m-dimensional low rank structure. Could you explain how you mitigate this issue?\n\nShouldn't a bound on the term $||B_{\\perp}-\\hat{B}_{n+1,\\perp}||$ or a term that showcases the estimation error on the representation,  appear in the final regret?\n\nDoes p need to be constant? after enough tasks explored, p could gradually vanish. I see in the final bound that $p \\sim 1/\\sqrt[3]{N^2}$, which is still constant.\n\nLIMITATIONS:\nAside from the assumptions on their setting made, I did not see any limitations explicitly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of representation transfer in a sequential multi-task linear bandit problem. The main objective in the paper is to remove the task diversity assumption made in Qin et al (2022), which places a constraint on any subsequence of tasks observed. The paper proposes an algorithm that for each task performs exploration with a constant probability, and uses the exponential weighted average algorithm to choose the exploration probability over a large set of arm vectors. The paper proves an upper bound on the cumulative regret without making the task diversity assumption and demonstrate the performance in an experiment.\n\nSTRENGTHS:\n* The paper presents the results in an engaging and clear manner and the algorithms proposed are intuitive.\n* Removal of the task diversity assumption appears to be significant advance.\n\nWEAKNESSES:\n* The problem setting emphasizes the sequential nature of the tasks. Specifically, the algorithm needs to collect all samples from $\\theta_1$ task before moving on to $\\theta_2$, i.e., the tasks cannot be interleaved. It would be helpful to outline some situations where this aspect of the problem setup is important.\n* The algorithm needs to know the value of $m, N, \\tau$ to choose the exploration probability. The proposed algorithm performs better than the \"individual task baseline\" only under a specific parameter regimes (see line 242).\n* The theoretically valid size of the set of experts is extremely large for reasonable parameter values (lines 408 and 284). Is that one of the reasons why BOSS only performs better when we have very large number of tasks?\n* There is a gap to the known lower bound, which is suggested to be due to the task diversity assumption not being met.\n\nQUESTIONS:\n* If possible, it would be nice to demonstrate the impact of the task diversity assumption. Specifically, could we construct a simulated environment where the assumption is not met by a significantly large fraction of subsequences, and the performance of BOSS is better than SeqRepL Qin et al (2022) ? Maybe your experiment already demonstrates that but I missed it. Could you clarify?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}