Title: Network Lasso Bandits
Venue: NeurIPS 2024
Accepted: no
Keywords: contextual bandits

multi-task learning

clustering

network lasso

graph total variation
Abstract: We consider a multi-task contextual bandit setting, where the learner is given a graph encoding relations between the bandit tasks. The tasks' preference vectors are assumed to be piecewise constant over the graph, forming clusters. At every round, we estimate the preference vectors by solving an online network lasso problem with a suitably chosen, time-dependent regularization parameter. We establish a novel oracle inequality relying on a convenient restricted eigenvalue assumption. Our theoretical findings highlight the importance of dense intra-cluster connections and sparse inter-cluster ones. That results in a sublinear regret bound significantly lower than its counterpart in the independent task learning setting. Finally, we support our theoretical findings by experimental evaluation against graph bandit multi-task learning and online clustering of bandits algorithms.

1

Abstract

We consider a multi-task contextual bandit setting, where the learner is given a graph encoding relations between the bandit tasks. The tasks' preference vectors are assumed to be piecewise constant over the graph, forming clusters. At every round, we estimate the preference vectors by solving an online network lasso problem with a suitably chosen, time-dependent regularization parameter. We establish a novel oracle inequality relying on a convenient restricted eigenvalue assumption. Our theoretical findings highlight the importance of dense intra-cluster connections and sparse inter-cluster ones. That results in a sublinear regret bound significantly lower than its counterpart in the independent task learning setting. Finally, we support our theoretical findings by experimental evaluation against graph bandit multi-task learning and online clustering of bandits algorithms.

1 Introduction

Online commercial websites aim to properly recommend their products to their customers, and the performance of these recommendations depends on the knowledge of users' preferences. Unlike traditional collaborative-filtering-based methods [Su and Khoshgoftaar, 2009], such knowledge is initially unavailable. Therefore, the online recommender systems need to recommend various items to the users and observe their ratings to explore their preferences. At the same time, the recommender system should be able to recommend items that attract users' attention and receive high ratings by exploiting the learned knowledge. The contextual bandits frameworks [Li et al., 2010] have been popularly used to formalize and address this exploration-exploitation trade-off.

However, the classical form of contextual bandits [Li et al., 2010, Chu et al., 2011, Abbasi-Yadkori et al., 2011] ignores the availability of social networks amongst users and solves the problem for each user separately. Consequently, such algorithms have some drawbacks when applied to problems with a large number of users. First, such a large number hinders the computational efficiency of such algorithms. Second, the partial feedback of the bandit settings exposes the algorithms to have weak estimations and impair their decision-making ability [Yang et al., 2020]. Consequently, to improve bandit algorithms' performance for large-scale applications, structural assumptions that link the different users are usually integrated within bandit algorithms [Cesa-Bianchi et al., 2013, Gentile et al., 2014, Li et al., 2019, Herbster et al., 2021].

The papers of Cesa-Bianchi et al. [2013], Yang et al. [2020] attempt to integrate the prior knowledge of social networks into their contextual bandit algorithms. Both papers proposed UCB-style algorithms and exhibited the importance of using the social network graph to achieve lower regrets using

Laplacian regularization. Consequently, both methods promote smoothness among the preference 33

vectors of users in order to transfer the collected information between them. However, the Laplacian 34

regularization does not account for the smoothness heterogeneity introduced by a piecewise constant 35

behavior over the graph [Wang et al., 2016]. On the other hand, algorithms of online clustering of 36

bandits [Gentile et al., 2014, Li et al., 2019] start from a graph and gradually add or remove edges to 37

form clusters as connected components. However, their clustering can cause overconfidence in the 38 constructed clusters, potentially leading to error accumulation. 39

In this paper, we assume access to a graph encoding relations between bandit tasks, and that the task 40 parameter vectors are piecewise constant over the graph. That means that tasks form clusters. We 41 propose an algorithm that integrates the prior knowledge of the piecewise constant structure to update 42 tasks rather than finding the clusters explicitly. That way, we mitigate the limitations mentioned 43 above: the piecewise constant smoothness is naturally integrated into our regularizer, and we do not 44 estimate the clusters so our algorithm does not suffer from overconfidence drawbacks. 45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

More precisely, we provide the following contributions

We analyze an instance of the Network Lasso problem [Hallac et al., 2015], where every vertex's preference vector is estimated using data generated during the interaction between users and the bandit. We provide the first oracle inequality in this setting and link it to fundamental quantities characterizing the relation between the graph and the true preference vectors of the users. Our result relies on our novel restricted eigenvalue (RE) condition, which we assume for our setting. This result is of independent interest and can be applied to independently generated data as a special case.

We prove how the empirical multi-task Gram matrix of the data inherits the RE condition from its true counterpart. Both this result and the previous one depend on the sparsity of inter-cluster connections and the density of intra-cluster ones.

We provide a regret upper bound for our setting. Our bound highlights the advantage of our algorithm in high dimensional settings, and for large graphs.

We support our theoretical findings by extensive numerical experiments on simulated data that prove the advantage of our algorithm compared to other approaches used for online clustering of bandits.

The rest of the paper is organized as follows. Section 2 discusses the relation of our work to the literature. We formulate our problem and state some of our assumptions in Section 3, then we present our bandit algorithm in Section 4. We analyze the problem theoretically in Section 5, and finally, we demonstrate its practical interest via numerical experiments in Section 6.

2 Related work

Lasso contextual bandits To address the high dimensional setting for linear bandits, several multiarmed bandit papers solve a LASSO [Tibshirani, 1996] problem under different assumptions [Bastani and Bayati, 2019, Kim and Paik, 2019, Oh et al., 2021, Ariu et al., 2022]. They all rely on a previously established compatibility or RE condition [Bühlmann and van de Geer, 2011], that they adapt to the non-i.i.d case. Such assumptions were also used in the multi-task setting by Cella and Pontil [2021] with a Group Lasso regularization [Yuan and Lin, 2006], and to impose a low rank structure on the task preference vectors in Cella et al. [2023]. In our case, we provide a novel oracle inequality, rather than just generalize an existing one to the non-i.i.d setting, with a newly introduced RE assumption.

Clustering of bandits Sequentially clustering bandit tasks was introduced in Gentile et al. [2014] with CLUB algorithm. In CLUB, starting with a fully connected graph, an iterative graph learning process is performed, where edges between users are deleted if their preference vectors are significantly different. As a result, any connected component is seen as a cluster and only one recommendation per cluster is developed. In another work, Li et al. [2019] generalize the setting of Gentile et al. [2014] and address its limitations via including merging operations in addition to splitting. In contrast to these approaches, the algorithm in Nguyen and Lauw [2014] groups users via K-means clustering, and the algorithm in Cheng et al. [2023] relies on hedonic games for online clustering of bandits. Furthermore, Yang and Toni [2018] make use of community detection techniques on graphs to find user clusters. Gentile et al. [2017] study the clustering of the contextual bandit problem where their proposed algorithm, named CAB, adaptively matches user preferences in the face of constantly evolving items. Our work fundamentally differs from the previous ones on two aspects. First, we assume access to a graph encoding relations between users, which is more informative than a complete graph. Second, we do not keep track of a model for each cluster, but rather we integrate a prior over

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

the graph via a graph total variation regularizer that enforces a piecewise constant behaviour for the estimated preference vectors.

Multi-task learning Several contributions assume some underlying structure that links the bandit tasks. In Cella and Pontil [2021], task preference vectors are assumed to be sparse and to share their sparsity support, implying that they lie in a low-dimensional subspace with dimensions aligning with the canonical basis vectors. This idea is further generalized in Cella et al. [2023], where the tasks are assumed to be confined to an arbitrary unknown low-dimensional subspace. That work improves upon Hu et al. [2021] by not requiring the knowledge of the small dimenson of the task space. The underlying structure linking tasks can also be a graph encoding relations between them [Cesa-Bianchi et al., 2013, Yang and Toni, 2018], which is our case. However, while they assume smoothness as a prior, we assume piecewise constant behavior.

3 Problem setting

We consider a linear bandit setting, with a finite number of tasks representing users in a recommendation system for example. For each task the agent has to choose among K arms, each associated to a d -dimensional context vector. All interactions over a horizon of T time steps. We further assume that we have access to an undirected graph G = ( V , E ) , with vertex set V representing the tasks and edge set E encoding the relationships between them. We identify the vertex set V with the set of vertex indices [ |V| ] . Thus, we consider E to be a subset of V 2 , where every edge ( m,n ) ∈ E has weight w mn > 0 , with m < n . The tasks' preference vectors are denoted by { θ m } m ∈V ⊂ R d verifying ∥ θ m ∥ ≤ 1 ∀ m ∈ V , which we concatenate as row vectors into matrix Θ ∈ R |V|× d . The latter represents a graph vector signal, assumed to be piecewise constant over G .

At a round t ∈ N ⋆ , a user m ( t ) ∈ V is selected uniformly at random and served an arm with context vector x ( t ) from a finite action set A ( t ) ⊂ R d with size K , depending on their estimated preference vector ˆ θ m ( t ) ( t ) ∈ R d . We assume the expected reward to be linear, with an additive, σ -sub-Gaussian noise conditionally on the past. Formally, denoting by F 0 the trivial sigma-algebra, and for all t ≥ 1 , by F t the sigma-algebra generated by history set { m (1) , x (1) , y (1) , · · · , m ( t ) , x ( t ) , y ( t ) , m ( t +1) } , the received reward y ( t ) is given by y ( t ) = 〈 θ m ( t ) ( t ) , x ( t ) 〉 + η ( t ) , where η ( t ) is F t -measurable and

At the end of a round t , all preference vectors are updated into a new estimation ˆ Θ ( t ) while leveraging the structure of graph G , formally by solving the following optimization problem:

where ∥·∥ denotes the Euclidean norm for vectors. The performance of our policy is assessed by the expected regret over the T interaction rounds for all tasks:

where x ⋆ ( t ) ∈ arg max ˜ x ∈A ( t ) 〈 θ m ( t ) , ˜ x 〉 .

The Optimization problem in (2) is an instance of the Network Lasso [Hallac et al., 2015]. Other 122 instances of the same type were studied by Jung et al. [2018], Jung and Vesselinova [2019], Jung 123 [2020]. The objective is characterized by its second term that, while being just the Laplacian 124 regularization without squaring the norms, promotes a piecewise constant behavior rather than 125 smoothness. For real-valued signals ( d = 1 ), this regularization has been extensively studied for 126 image and graph signal denoising, for the problem of trend filtering on graphs [Wang et al., 2016]. 127 According to Wang et al. [2016], that regularization better adapts to the heterogeneity of smoothness 128 of the signal and induces a cluster structure in the data: similar users will not only have similar 129 models but the same model, which offers a compression of the overall model over the graph. Note 130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

that our setting is cluster agnostic; our algorithm does not aim to learn the cluster structure explicitly but to exploit it implicitly using the total variation semi-norm as regularization. The latter's strength is controlled via a time-dependent regularization coefficient α ( t ) , which we will express later in the analysis.

We formalize our assumption on the context generation as follows.

Assumption 1 (i.i.d action sets) . Context sets {A ( t ) } T t =1 are generated i.i.d. from a distribution p over R K × d , such that ∥ x ∥ ≤ 1 ∀ x ∈ A ( t ) ∀ t ≥ 1 .

In addition to the i.i.d assumption, we assume more regularity.

Assumption 2 (Relaxed symmetry and balanced covariance) . There exists a constant ν ≥ 1 such that for all X ∈ R K × d , p ( -X ) ≤ νp ( X ) . Furthermore, there exists ω > 0 , such that for any permutation ( a 1 , · · · , a K ) of [ K ] , for any i ∈ { 2 , · · · , K -1 } , and for any w ∈ R d , we have

This assumption was introduced in Oh et al. [2021], and has already been used in a multi-task setting by Cella et al. [2023]. Parameter ν controls the skewness, as ν = 1 corresponds to a symmetric distribution. ω decreases with increasing positive correlation between arms. It verifies ω = O (1) for multi-variate Gaussians and uniform distributions over the unit sphere [Oh et al., 2021]. The piecewise constant behaviour of the graph signal Θ is formalized in the next assumption.

Assumption 3 (Piecewise constant signal) . There exists a partition P of V , such that for any cluster C ∈ P , signal Θ is constant on C , and the graph obtained by taking the vertices in C and the edges linking them is connected.

Assumption 3 basically states that the true preference vectors are clustered and that the given graph induces the cluster structure. It is required for our approach to be beneficial, as we will detail in the analysis section. For the sake of clarity, we defer the statement of other technical assumptions to Section 5.

4 Algorithm

Our policy in Algorithm 1 follows a greedy arm selection rule in a multi-task setting, in the same vein as those presented in Oh et al. [2021], Cella et al. [2023]. Indeed, as pointed out in Oh et al. [2021], exploration is implicitly incorporated into regularization parameter α ( t ) 's time dependence. It has the following expression

where the set of time steps a task m has been selected up to time t is denoted by T m ( t ) .

5 Analysis

This section provides the main steps of the analysis. One of the paper's contribution lies in finding an oracle inequality of the network lasso problem given a restricted eigenvalue condition holding for the true multi-task Gram matrix. In this regard, the next major challenge and contribution is to show that the empirical multi-task Gram matrix, estimated in the algorithm, satisfies the restricted eigenvalue condition. We start by proving an oracle inequality for the estimation error of Θ , assuming that the condition given by Definition 2 is verified by the empirical data Gram matrix. Then, we prove that the latter assumption actually holds with high probability given that true multi-task Gram matrix satisfies it. Our final contribution in this work is the establishment of a regret bound for our algorithm.

5.1 Notation and technical assumptions

We provide additional notations required for the analysis. We denote by ∂ P the set of all edges in 171 E connecting vertices from different clusters from partition P (Assumption 3), and we call it the 172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

boundary of P . Thus, ∂ P c , the complementary set of ∂ P , is formed by edges connecting vertices of the same cluster. The total weight of the boundary, i.e. the sum of its edges' weights, is referred to as w ( ∂ P ) . Given a signal Z ∈ R |V|× d , we denote by Z P the signal obtained by setting row vectors of Z to their mean-per-cluster value w.r.t. P . For any edge subset I ∈ E , we denote the following norms: ∥·∥ F as the Frobenius norm, ∥ z ∥ M = √ z ⊤ Mz as the weighted norm of vector z ∈ R d induced by matrix M ∈ R d × d and ∥ Θ ∥ I := ∑ ( m,n ) ∈ I w mn ∥ θ m -θ n ∥ as the total variation semi-norm of Θ ∈ R |V|× d over I . Thus, the regularization term of Problem (2) is equal to ∥ Θ ∥ E . Also, we define the incidence matrix B I ⊂ R |E|×|V| restricted to I ⊆ E to be null except at rows with index i ∈ I corresponding to edge ( m,n ) , where it equals w mn ( e m -e n ) , where e m is the m th canonical basis vector of R |V| . We define A V ( t ) := diag ( X 1 ( t ) ⊤ X 1 ( t ) , . . . , X |V| ( t ) ⊤ X |V| ( t ) ) ∈ R d |V|× d |V| , and subsequently the empirical multi-task Gram matrix up to time step t is given by 1 t A V ( t ) . The following definition introduces quantities related to the clusters defined by partition P , with crucial roles that we will elucidate throughout the analysis.

Definition 1 (Cluster content constants) . Let C ∈ P be a cluster.

We denote by ∂ v C the inner boundary of C , i.e. the vertices of C that are connected to its complementary. We define the inner isoperimetric ratio of C as ι G ( C ) := | ∂ v C| |C| .

By abuse of notation, we denote as B C the incidence matrix restricted to edges linking vertices of C , its associated Laplacian matrix by L C := B ⊤ C B C , and its pseudo-inverse by L † C . The topological centrality index of node m ∈ C w.r.t C is equal to ( L † C ) -1 mm . We define the topological centrality index of C by c G ( C ) := min m ∈C ( L † C ) -1 mm .

The inner isoperimetric ratio of a cluster measures how many 'interior' nodes a cluster contains, in the sense that they are not connected to its complementary. It is at most equal to the isoperimetric ratio for weightless graphs as the size of the inner boundary is at most equal to that of the edge boundary, the latter being connected to the algebraic connectivity via the Cheeger inequality [Cheeger, 1970].

The topological centrality index measures the overall connectedness of a vertex in a network and indicates how robust a node is to edge failures [Ranjan and Zhang, 2013]. Also, it can be tied to electricity spreading in a network according to Van Mieghem et al. [2017]. We refer the interested reader to the two previously mentioned works for a detailed account of the properties of the topological centrality index. In the appendix, we show that for binary weights graphs the minimum topological centrality index is at least equal to the algebraic connectivity theoretically and experimentally, where we showcase that the difference between the two can be significant.

To proceed, we will need the following definition that introduces several notations to reduce the 204 clutter. 205

Definition 2 (Restricted Eigenvalue (RE) condition and norm) . Let { M i } |V| i =1 ⊂ R d × d be a set of 206 positive semi-definite matrices. We say that the matrix M V := diag( M 1 , · · · , M |V| ) verifies the 207 restricted eigenvalue condition with constants κ ≥ 0 and ϕ > 0 if 208

where S is the cone defined by:

and the RE semi-norm is defined by ∥ Z ∥ RE := ∥ ∥ Z P ∥ ∥ F ∨ (1 -κ ) + ∥ ∥ ∥ B † ∂ P B ∂ P Z ∥ ∥ ∥ .

To interpret the previous definition, we point out that the sum on the right-hand side of Definition 2 can be written as ∥ ∥ vec( Z ⊤ ) ∥ ∥ M V , where vec denotes the operation of stacking a matrix's columns vertically. As a result, the condition is analogous to requiring that M V is invertible with minimum eigenvalue ϕ 2 , but weaker since it holds only for signals Z ∈ S and for the ∥·∥ RE norm. This requirement has the same form as the compatibility assumption for the Lasso [Bühlmann and van de Geer, 2011, Oh et al., 2021] or the restricted strong convexity assumption [Cella et al., 2023].

We further make the following assumption on the true multi-task Gram matrix:

Assumption 4 (RE condition for the true multi-task Gram matrix) . For k ∈ [ K ] , let Σ k := E [ x k x ⊤ k ] be the Gram matrix of the k th context vector's marginal distribution, let Σ V be the true multi-task Gram matrix of the context vector generating distribution, given by

We assume that Σ V verifies RE condition (Definition 2) with some problem dependent constants κ ∈ [ 0 , 1 2 w ( ∂ P ) min C∈P √ c G ( C ) ) and ϕ > 0 .

This assumption is common to make for Lasso-like bandit problems [Oh et al., 2021, Ariu et al., 2022, Cella et al., 2023]. We will later show that it can be transferred to empirical multi-task Gram matrix.

5.2 Oracle inequality

This section is dedicated to provide a bound on the estimation error of the Network Lasso problem given in Equation (2) at a particular step t of Algorithm 1.We assume fixed design, meaning that the context vectors are given and fixed, and we are not concerned by their randomness (due to the context generating distribution), nor by the randomness of their number for each user (due to random selection at each time step).

For a time step t , we deliver the oracle inequality controlling the deviation between the estimated preference vectors ˆ Θ ( t ) and the true ones Θ . For the sake of simplicity, we provisionally assume that the RE condition holds for the empirical multi-task Gram matrix A V ( t ) .

Theorem 1 (Oracle inequality) . Assume that the RE assumption holds for the empirical multitask Gram matrix with constants κ ∈ [ 0 , 1 2 w ( ∂ P ) min C∈P √ c G ( C ) ) and ϕ > 0 . Suppose that max m ∈V |T m ( t ) | ≤ bt for some b > 0 . Then, with a probability at least 1 -δ ( t ) , we have

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

where 237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

where 269

The proof of the previous theorem mainly relies on a decomposition of the estimation error signal into two parts: one is the projection of the error onto its mean per cluster value, that is, every node within the same cluster is mapped to the mean estimation error of its cluster. The second part of the decomposition is simply the residual part i.e. the deviation from the mean per cluster value, which is related to the incidence matrices of each cluster. The probabilistic statement comes from a high probability bound on the Euclidean norm of an empirical vector process associated with our problem, using a generalization of the Hanson-Wright inequality to the subgaussian case [Hsu et al., 2012, Theorem 2.1]. Compared to the bound of Jung [2020, Theorem 1], we bound a norm of the estimation error rather than just the total variation semi-norm. Additionally, the bound exhibits different behavior depending on whether κ > 1 . Indeed, due to the expressions of a 1 ( Θ , G ) and a 2 ( Θ , G ) , in the case where κ > 1 , the bound significantly decreases with the products w ( ∂ P ) min C∈P √ ι ( C ) and w ( ∂ P ) max C∈P c G ( C ) -1 2 , which are both small enough for dense intra-cluster edge links and sparse inter-cluster ones. However, when κ < 1 , the w ( ∂ P ) term might dominate if it is moderately large, and its effect can only be mitigated via a small subgaussianity constant σ or a large enough RE condition constant ϕ .

5.3 RE condition for the empirical multi-task Gram matrix

To establish the oracle inequality, we assumed that the RE condition holds for the empirical multi-task Gram matrix. The goal of this section is to prove this holds with high probability. To this end, we use the same strategy as in Oh et al. [2021], Cella et al. [2023]. We prove that on the one hand, given the empirical multi-task Gram matrix inherits the RE condition from its adapted counterpart since it concentrates around it. On the other hand, we prove that the adapted Gram matrix verifies the RE condition due to Assumption 1, 2 and 4 made on the context generation distribution.

Theorem 2 (RE condition holding for the empirical multi-task Gram matrix) . Under assumptions 2 and 4, let t ≥ 1 , and let κ, ϕ be the constants from Assumption 4. Assume that max m ∈V |T m ( t ) | ≤ bt . Then, for any γ ∈ ( 0 , ( 1 + a 2 ( G , Θ )+(1 -κ ) + √ 2 w ( ∂ P ) a 1 ( G , Θ ) ) -2 ) , the empirical multi-task Gram matrix verifies the RE condition with constants κ and ˆ ϕ , with

with a probability at least equal to 1 -6 d |V| exp ( -3 γ 2 ˜ ϕ 4 (min C∈P (˜ c G ( C ) ∧ ˜ c G ( C ) 2 ) t 6 b +2 √ 2 γ ˜ ϕ 2 ) , where ˜ ϕ := ϕ √ 2 νω and ˜ c G ( C ) := c G ( C ) ∧ |C| ∀C ∈ P .

The proof follows the same approach as in Oh et al. [2021], Cella et al. [2023]; we prove that the RE condition transfers from the true multi-task Gram matrix to its adapted counterpart V V ( t ) , defined as follows:

This transfer relies on the work of Oh et al. [2021, lemma 10]. The other step of the proof is showing 270 that the empirical multi-task Gram matrix and V V ( t ) become close to each other with high probability 271 after sufficiently many time steps, the respective distance between the two is measured with a matrix 272 norm induced by the RE semi-norm and the restriction to set S (Definition 2). The bound showcases 273 a dependence on min C∈P c G ( C ) ∧|C| , which is of the same order as |C| for a fully connected cluster 274 with vertices C . It is also clear that with a higher minimum centrality of a cluster, the probability of 275 satisfying the RE condition increases. 276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

5.4 Regret bound

To bound the regret, we bound the expected instantaneous regret for each round t ≥ 1 . This bound relies on the oracle inequality holding and on the RE condition being satisfied for the empirical Gram matrix, both with high probability. These two conditions are ensured and Theorem 1 and Theorem 2.

Theorem 3 (Regret bound) . Let the mean horizon per node be T = T |V| . Let min C∈P √ c G ( C ) √

going asymptotically to infinity and max C∈P ι G ( C ) going asymptotically to zero as well as max C∈P √ ι G ( C ) w ( ∂ P ) and w ( ∂ P ) min C∈P √ c G ( C ) going asymptotically to zero. Under assumptions1 to 4

and κ < 1 , the expected regret of the Network Lasso Bandit algorithm is upper bounded as follows:

Our regret is mainly formed of two parts. The first one is the sublinear time-dependent term and represents the bulk of horizon dependence. Interestingly, it does not depend on the dimension, which is a consequence of using the concentration inequality from Hsu et al. [2012]. Interestingly, it decreases as the topological centrality index grows with the graph size, which proves the importance of intra-cluster high connectivity.

The second significant term comes from ensuring the RE condition for the empirical multi-task Gram matrix, and can be interpreted as the number of time steps necessary for it to hold, as pointed out by Oh et al. [2021]. It has a logarithmic dependence in the graph size and in the dimension, which is a characteristic of regret bound of the "lasso type". Also noteworthy is that the regret grows with log( d ) only in the time-independent term, making our policy useful in high-dimensional settings.

6 Experiments

We provide experiments to showcase the effect on the problem's parameters on our algorithm's performance as well as highlighting its advantageous performance compared to other algorithms. At each time step, the algorithm solves the network lasso problem (2) via a primal-dual algorithm used in Jung [2020].

We compare our algorithm to several baselines of the literature. On the one hand, baselines relying on a given graph, GOBLin [Cesa-Bianchi et al., 2013] and GraphUCB [Yang et al., 2020] that use the Laplacian to smooth the preference vectors. On the other hand, we consider online clustering of bandits baselines, namely CLUB [Gentile et al., 2014] and SCLUB [Li et al., 2019]. Since these latter approaches start with a fully connected graph, we provide them the known graph for a fair comparison. As a sanity check, we also compare the independent task learning case with LinUCB (LinUcbITL) where each task is solved independently, and to the case of a LinUCB agent for each cluster (LinUcbOracle). The graph used is generated using stochastic block models in order to ensure that the generated graph induces a cluster structure, where an edge is constructed with probability p within clusters and q between clusters.

Experimentally, we found that normalizing the adjacency matrix, that is we utilize the following

1

normalized edges: w mn = √ deg( m ) deg( n ) , where deg( m ) denotes the degree of node m , yields significantly better results. Indeed, such a normalization makes the algorithm focus more on edges between low-degree nodes, which improves the propagation of the collected information within the graph. In all experiments we have set α 0 = 0 . 1 .

Our results clearly showcase an improvement compared to the other baselines. Apart from the oracle that has complete knowledge of all clusters from the beginning, our policy performs significantly better than the rest beyond the error margins, covering one standard deviation at ten repetitions. We

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

0

500

1000

1500

2000

2500

3000

0

200

400

600

800

1000

CLUB

GOBLin

GraphUCB

LinUcbITL

LinUcbOracle

NetLasso

SCLUB

(a)

|V|

= 100

, d

= 20

, p

= 0

.

4

, q

= 0

.

1

0

200

400

600

800

1000

0

100

200

300

400

500

CLUB

GOBLin

GraphUCB

LinUcbITL

LinUcbOracle

NetLasso

SCLUB

(b)

|V|

= 100

, d

= 10

, p

= 0

.

5

, q

= 0

.

1

0

500

1000

1500

2000

2500

3000

0

100

200

300

400

500

CLUB

GOBLin

GraphUCB

LinUcbITL

LinUcbOracle

NetLasso

SCLUB

(c)

|V|

= 50

, d

= 80

, p

= 0

.

8

, q

= 0

.

2

0

1000

2000

3000

4000

5000

0

250

500

750

1000

1250

1500

1750

2000

CLUB

GOBLin

GraphUCB

LinUcbITL

LinUcbOracle

NetLasso

SCLUB

(d)

|V|

= 200

, d

= 20

, p

= 0

.

5

, q

= 0

.

05

provide results for up to |V| = 500 nodes showing the effective transfer of knowledge within the graph.

7 Conclusion and future perspectives

In this work, we proposed a multi-task bandit framework that solves the case where the task preference vectors are piecewise constant over a graph. To this end, we used the Network Lasso policy to estimate the task parameters, which bypasses explicit clustering procedures. We showed a sublinear regret bound and as a byproduct, we proved a novel oracle inequality that relies on the small size of the boundary as well as on the high value of the topological centrality index of each node within its cluster. Our experimental evaluations highlight the advantage of our method, especially when either the number of dimensions or nodes increases.

Due to the technical similarity of our problem with the Lasso, a natural extension would be to extend it to a thresholded approach, in the same vein as [Ariu et al., 2022]. Another possible extension would be to use regularization with higher order total variation terms that impose a piecewise polynomial signal on a graph, as explained for scalar signals in Wang et al. [2016], Ortelli and van de Geer [2019].

References

Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

Ariu, K. Abe, and A. Proutiere. Thresholded Lasso Bandit. In Proceedings of the 39th International Conference on Machine Learning , pages 878-928. PMLR, 2022.

Bastani and M. Bayati. Online Decision Making with High-Dimensional Covariates. Operations Research , 2019. doi: 10.1287/opre.2019.1902.

Basu, B. Kveton, M. Zaheer, and C. Szepesvari. No Regrets for Learning the Prior in Bandits. In Advances in Neural Information Processing Systems , 2021.

Bilaj, S. Dhouib, and S. Maghsudi. Meta learning in bandits within shared affine subspaces. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics . PMLR, 2024.

Borge-Holthoefer, A. Rivero, I. García, E. Cauhé, A. Ferrer, D. Ferrer, D. Francos, D. Iniguez, M. P. Pérez, G. Ruiz, et al. Structural and dynamical patterns on online social networks: the spanish may 15th movement as a case study. PloS one , 6(8), 2011.

Bühlmann and S. van de Geer. Statistics for high-dimensional data . Springer Series in Statistics. Springer, Heidelberg, 2011. ISBN 978-3-642-20191-2.

Cella and M. Pontil. Multi-task and meta-learning with sparse linear bandits. In Uncertainty in Artificial Intelligence . PMLR, 2021.

Cella, A. Lazaric, and M. Pontil. Meta-learning with stochastic linear bandits. In Proceedings of the 37th International Conference on Machine Learning . PMLR, 2020.

Cella, K. Lounici, G. Pacreau, and M. Pontil. Multi-task representation learning with stochastic linear bandits. In International Conference on Artificial Intelligence and Statistics , 2023.

Cesa-Bianchi, C. Gentile, and G. Zappella. A gang of bandits. Advances in neural information processing systems , 26, 2013.

Cheeger. A lower bound for the smallest eigenvalue of the laplacian. Problems in analysis , 1970.

Cheng, C. Pan, and S. Maghsudi. Parallel online clustering of bandits via hedonic game. In International Conference on Machine Learning , pages 5485-5503. PMLR, 2023.

Chu, L. Li, L. Reyzin, and R. Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics . JMLR Workshop and Conference Proceedings, 2011.

Dong, D. Thanou, M. Rabbat, and P. Frossard. Learning graphs from data: A signal representation perspective. IEEE Signal Processing Magazine , 2019.

Easley, J. Kleinberg, et al. Networks, crowds, and markets: Reasoning about a highly connected world , volume 1. Cambridge university press Cambridge, 2010.

Fontan and C. Altafini. On the properties of laplacian pseudoinverses. In 2021 60th IEEE Conference on Decision and Control (CDC) . IEEE, 2021.

Gentile, S. Li, and G. Zappella. Online clustering of bandits. In International Conference on Machine Learning , pages 757-765. PMLR, 2014.

Gentile, S. Li, P. Kar, A. Karatzoglou, G. Zappella, and E. Etrue. On context-dependent clustering of bandits. In International Conference on machine learning , pages 1253-1262. PMLR, 2017.

Hallac, J. Leskovec, and S. Boyd. Network lasso: Clustering and optimization in large graphs. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 387-396, 2015.

Herbster, S. Pasteris, F. Vitale, and M. Pontil. A gang of adversarial bandits. Advances in Neural Information Processing Systems , 34, 2021.

Hsu, S. Kakade, and T. Zhang. A tail inequality for quadratic forms of subgaussian random vectors. Electronic Communications in Probability , 17, 2012.

Hu, X. Chen, C. Jin, L. Li, and L. Wang. Near-optimal representation learning for linear bandits 382 and linear rl. In International Conference on Machine Learning . PMLR, 2021. 383

384

385

Jung. Networked Exponential Families for Big Data Over Networks. IEEE Access , 8, 2020. ISSN 2169-3536.

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

Jung and N. Vesselinova. Analysis of network lasso for semi-supervised regression. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 380-387. PMLR, 2019.

Jung, N. Tran, and A. Mara. When Is Network Lasso Accurate? Frontiers in Applied Mathematics and Statistics , 3, 2018. ISSN 2297-4687.

G.-S. Kim and M. C. Paik. Doubly-robust lasso bandit. Advances in Neural Information Processing Systems , 32, 2019.

Kveton, M. Konobeev, M. Zaheer, C.-w. Hsu, M. Mladenov, C. Boutilier, and C. Szepesvari. Meta-thompson sampling. In International Conference on Machine Learning . PMLR, 2021.

Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Li, W. Chen, and K.-S. Leung. Improved algorithm on online clustering of bandits. arXiv preprint arXiv:1902.09162 , 2019.

McPherson, L. Smith-Lovin, and J. M. Cook. Birds of a feather: Homophily in social networks. Annual review of sociology , 27(1):415-444, 2001.

E. Newman. Modularity and community structure in networks. Proceedings of the national academy of sciences , 103(23):8577-8582, 2006.

T. Nguyen and H. W. Lauw. Dynamic clustering of contextual multi-armed bandits. In Proceedings of the 23rd ACM international conference on conference on information and knowledge management , pages 1959-1962, 2014.

Nourani-Koliji, S. Bilaj, A. R. Balef, and S. Maghsudi. Piecewise-stationary combinatorial semi-bandit with causally related rewards. arXiv preprint arXiv:2307.14138 , 2023.

M.-H. Oh, G. Iyengar, and A. Zeevi. Sparsity-Agnostic Lasso Bandit. In Proceedings of the 38th International Conference on Machine Learning , pages 8271-8280. PMLR, 2021.

Ortelli and S. van de Geer. Synthesis and analysis in total variation regularization. arXiv preprint arXiv:1901.06418 , 2019.

Peleg, N. Pearl, and R. Meir. Metalearning linear bandits by prior update. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics . PMLR, 2022.

Ranjan and Z.-L. Zhang. Geometry of complex networks and topological centrality. Physica A: Statistical Mechanics and its Applications , 2013.

Su and T. M. Khoshgoftaar. A survey of collaborative filtering techniques. Advances in artificial intelligence , 2009, 2009.

Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology , 1996.

Tropp. Freedman's inequality for matrix martingales. Electronic Communications in Probability , 16:262 - 270, 2011.

Van Mieghem, K. Devriendt, and H. Cetinay. Pseudoinverse of the laplacian and best spreader node in a network. Physical Review E , 2017.

Y.-X. Wang, J. Sharpnack, A. J. Smola, and R. J. Tibshirani. Trend filtering on graphs. Journal of Machine Learning Research , 17(105):1-41, 2016. URL http://jmlr.org/papers/v17/ 15-147.html .

427

428

429

430

431

432

433

434

Yang and L. Toni. Graph-based recommendation system. In 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP) , pages 798-802. IEEE, 2018.

Yang, L. Toni, and X. Dong. Laplacian-regularized graph bandits: Algorithms and theoretical analysis. In International Conference on Artificial Intelligence and Statistics , pages 3133-3143. PMLR, 2020.

Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology , 2006.

A Some helper results

Proposition 1 (Bounds on norms of matrix products) . Let M ∈ R m × n and N ∈ R n × p . Then 435

440

441

442

443

Fourth inequality We have 444

445

Proposition 2 (Decomposition of a signal over a graph) . For any C ∈ P 446

Proof. 436

First inequality For any q ∈ [1 , ∞ ] , we have: 437

438

Second inequality We have 439

Third inequality We have

Elements of ( i, j ) entry of matrix NN ⊤ is the inner product 〈 e ⊤ i N , e ⊤ j N 〉 . Hence, we have

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

463

464

465

466

467

468

469

470

471

472

473

474

Let Z ∈ R |V|× d be a graph signal. Let us denote by Z C the signal obtained from Z by setting rows of vertices outside of C to zeros, and let Z |C ∈ R |C|× d be the signal obtained from Z C by removing the rows of vertices outside of C . Also, let B |C ∈ R |E C |×|C| be the matrix obtained by taking B C , and removing rows of edges that link C to its outside, and the resulting null columns. It is clear that

where J C = 1 C 1 ⊤ C |C| , Q C = B † C B C ∀C ∈ P and Q ∂ P c := B † ∂ P c B ∂ P c .

While ∑ C∈P J C projects each entry of a graph signal onto the mean vector value of its respective cluster, its residual Q ∂ P c can be interpreted as the projection onto the respective entries deviation from its cluster mean value.

Proof. Since the proof of the first point is trivial, we directly treat the second point. Denoting B † |C the pseudo-inverse of B |C it is a well-known linear algebra result that the matrix Q |C := B † |C B |C is the projector onto the null space of B |C . Since C is connected, the null space of B |C is unidimensional, and is generated by vector 1 |C| ∈ R |C| having only ones as coordinates. Since the projector into that nullspace is J |C| := 1 |C| 1 |C| |C| , we deduce that

where in the last line, Q C := B † C B C . Consequently, we have 462

To prove the second point, we recall that B ∂ P c is the incidence matrix obtained by setting rows corresponding to edges in ∂ P to zero. In other words, B ∂ P c is the incidence matrix of the graph after removing the boundary edges, and having exactly |P| connected components. Hence, B ∂ P c has a null space spanned by the set { 1 C } C∈P , and the orthogonal projector onto this null space is ∑ C∈P J C . Combining this fact with the fact that Q ∂ P c is the projector onto the orthogonal of the null space of B ∂ P c , we arrive at the second point.

Proposition 3 (On the minimum topological centrality index of a graph vertex) . Let G be a connected graph with incidence matrix B and vertex set size N , and let L := B ⊤ B . Let c ( G ) denote the minimum value of inverses of diagonal element of L † , called its minimum topological centrality index. Also let a ( G ) be its algebraic connectivity, defined as the minimum non null eigenvalue of L . Then

c ( G ) = ∥ L ∥ -1 ∞ , ∞ .

c ( G ) ≥ a ( G ) .

If G is weightless, then c ( G ) ≤ N 2 .

N -1 475

Let Q C := B † C B C . Then

Proof. Since L is PSD, L † is PSD and hence ∥ ∥ L † ∥ ∥ ∞ , ∞ is equal to the maximum diagonal entry of 476 L † . Taking the inverse proves the first point. Also, this implies that 477

where we used the fact that ∥·∥ ∞ , ∞ ≤ ∥·∥ for matrices. This proves the second point of the 478 proposition. 479

For the last point, assume G is weightless, let L comp be the Laplaciane of complete graph built on the 480 vertices of G . Then we have L comp = N ( I N -J N ) , where J is the square matrix of dimension N 481 having 1 /N as entries. From Fontan and Altafini [2021, Lemma 4], we have 482

which has diagonal elements 1 N -1 N 2 .

̸

On the other hand, L ≼ L comp Hence, by Fontan and Altafini [2021, lemma 4] we have for any u = 0

This implies that the maximum diagonal entry of L † is at least equal to that of L † comp , i.e. to 1 N -1 N 2 . Taking the inverse of that entry finishes the proof.

B Proofs of the different claims

B.1 Additional notation

The regularization term can be written more compactly using the incidence matrix of the graph B ∈ R |E|×|V| corresponding to an arbitrary orientation under the following form

where the ∥·∥ 2 , 1 norm denotes the sum of the L 2 norms o the rows of a matrix. 1 We provide notations that we use in the proofs of the different statements, in order to reduce the clutter. We define E := ˆ Θ -Θ as the error signal, and its rows by { ϵ m } |V| m =1 .

While ∑ C k =1 J C projects each entry of a graph signal onto the mean vector value of its respective cluster, its residual Q ∂ P c can be interpreted as the projection onto the respective entries deviation from its cluster mean value.

Let η m be a vector, vertically concatenated by noise terms of rewards received by node m , then we define K ∈ R |V|× d as the matrix of vertically concatenated row vectors η ⊤ m X m .

B.2 Oracle inequality

In this section, we present all intermediary theoretical results leading to Theorem 1 stating the oracle inequality. To reduce the clutter, we omit the dependence on t of several quantities. For instance, we write α and ˆ Θ instead of α ( t ) and ˆ Θ ( t ) .

Lemma 1 (A first deterministic inequality) . Let t be a time step. We have

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

Proof. By optimality of ˆ Θ , we have 506

On the one hand, given a user index m ∈ V , and since by definition of the observed rewards we have 508 we have for the least squared terms 509

where we used the fact that y m = X m θ m + η m , which holds by definition of the observed rewards. 510 Summing over the users, and using the definition of K , we have 511

On the other hand, we have for the estimated preference vectors 512

For the true ones, and for any C ∈ P , let E C denote the edges linking the nodes of set of nodes C . It is 513 clear that ∂ P c = ⋃ C∈P E C as a disjoint union, hence 514

where the last equality holds due to the cluster assumption. 515

Hence, we have 516

517

518

519

520

521

522

523

524

525

where the first inequality holds due to the triangle inequality, and the last one since ∥ Θ ∥ ∂ P c = 0 . Combining Equations (16) to (18), we obtain the result of the statement.

In the proof for the oracle inequality, we utilize projection operators on the graph signal, that we define as followed:

While ∑ C k =1 J C projects each entry of a graph signal onto the mean vector value of its respective cluster, its residual Q ∂ P c can be interpreted as the projection onto the respective entries deviation from its cluster mean value.

Lemma 2 (Bounding the error restricted to the boundary) . The total variation of E restricted to the boundary verifies

Proof. The proof relies on a decomposition of the ∥ E ∥ ∂ P term from Proposition 2. We have 526

where E P is obtained by setting the error signal on every cluster to its mean.

527

For the first term on the right-hand side, let us denote by ϵ C the value of any row of E P belonging to 528 cluster C , which is equal to the mean of errors E over that cluster. Also, we denote by ( E P ) ∂ P the 529 signal obtained from E P by setting its rows corresponding to nodes that are not adjacent to any edge 530 in the boundary ∂ P to zeros. Also, let ∂ v C denote the inner boundary of set of nodes C ,i.e. nodes of 531 C that connect it to its complementary. Then it holds that: 532

For the second term, we have 533

Theorem 4 (Theorem 2.1 of Hsu et al. [2012]) . At time step t , let A ∈ R b × t where b ∈ N ∗ , and let 535 v ∈ R t be a random vector such that for some σ ≥ 0 , we have 536

where 541

Proof. We recall that K ∈ R t × d is the matrix obtained by stacking the row vectors η ⊤ m X m vertically. 542 On the one hand, we have 543

where X V := diag( X 1 , · · · , X |V| ) ∈ R t × d |V| . 544

From Equations (24) and (25), we can apply Theorem 4 to matrix X V and random vector η , which 546 implies that with a probability at least 1 -δ , we have 547

548

549

550

551

Proposition 4 (Probabilistic inequality) . With a probabability at least 1 -δ , we have 552

where we used the equalities ∥ X V ∥ F = ∑ m ∈V Tr( A m ) , ∥ X V ∥ 2 = max m ∈V ∥ A m ∥ and ∥ ∥ X V X ⊤ V ∥ ∥ 2 F =

∥ ∥ X ⊤ V X V ∥ ∥ 2 F = ∑ m ∈V ∥ A m ∥ 2 F . To arrive the the statement of the theorem, we use the fact that the context vectors have Euclidean norms of at most 1 .

Then for any δ ∈ (0 , 1) , we have with a probability at least 1 -δ : 537

Lemma 3 (Empirical process bound) . Let X m ∈ R |T m |× d denotes the matrix of collected context 538 vectors for task m ∈ V , then, given collected context matrices { X m } m ∈V , for any δ ∈ (0 , 1) we 539 have with probability of at least 1 -δ : 540

Proof. The proof is a combination of the results of Lemmas 1 to 3. We have 554

where the last line is an application of Lemma 2. Grouping the terms by the type of norm applied to 555 E finishes the proof. 556

557

558

559

Theorem 1 (Oracle inequality) . Assume that the RE assumption holds for the empirical multitask Gram matrix with constants κ ∈ [ 0 , 1 2 w ( ∂ P ) min C∈P √ c G ( C ) ) and ϕ > 0 . Suppose that max m ∈V |T m ( t ) | ≤ bt for some b > 0 . Then, with a probability at least 1 -δ ( t ) , we have

where 560

Proof. Using the previously established results, we obtain 561

where the last inequality holds for any β > 0 , and is a consequence of the property that uv ≤ u 2 + v 2 2 562 for any u, v ∈ R . 563

As a result, we can bound the norm of Q ∂ P c E as follows: 564

We can also bound the norm of E P as follows: 565

The result is then obtained by combining Equations (30) and (31) along with using the fact that 566 E = E P + Q ∂ P c E and the expressions of a 1 ( Θ , G ) and a 2 ( Θ , G ) , and bounding α δ ( t ) as follows: 567

B.3 Inheriting the RE condition from the true to the empirical data Gram matrix

B.3.1 From the adapted to the empirical multi-task Gram matrix

Lemma 4 (Bounding a quadratic form using projections) . Let M 1 , · · · , M p ∈ R d × d be symmetric matrices, and let J := 1 p 11 ⊤ , and Q = I -J . Then, for any Z ∈ R p × d with rows { z i } p i =1 , we have:

568

569

570

571

572

Proof. We have 573

where we used the fact that z i -¯ z = Z ⊤ e i -Z ⊤ Je i = Z ⊤ Qe i . 574

Let us now examine every term on the right-hand side of Equation (32). For the first term, we have 575

For the second term, we have 576

Finally, for the last term, we have 577

Combining Equations (33) to (35) yields the result. 578

579

We also define an operator norm that is induced by the ∥∥ RE introduced in Definition 2.

580

581

582

Definition 3 ((RE, S )-induced operator norm) . Let { M m } m ∈V ⊆ R d × d be symmetric matrices associated to the graph nodes V , and let M V := diag ( M 1 , · · · , M |V| ) ∈ R d |V|× d |V| . For any cluster C ∈ P , let the cluster mean and mean of squares associated to those matrices be given by

The RE-induced operator norm of M V is defined as 583

584

585

586

587

588

589

B.3.2 Linking the adapted to the empirical Gram

We first start by establishing that given the closeness of two PSD matrices in a certain sense, the RE condition can be transferred between them.

Proposition 5 (Restricted spectral norm) . Let Z ∈ R |V|× d verifying

Let { M m } m ∈V ⊆ R d × d be symmetric matrices associated to the graph nodes V , and let M V := diag( M 1 , · · · , M |V| ) ∈ R d |V|× d |V| . Then we have:

Proof. For any cluster C , we denote by B C the incidence matrix obtained by setting the rows of B 590 outside the edges linking nodes in C to null vectors. The latter's nullspace is the span of the vector 1 C 591 having coordinates 1 at nodes in C and zeros elsewhere. Hence, the projector onto the orthogonal of 592 1 C is Q C := B † C B C . 593

On the one hand, for any signal Z ∈ R |V|× d we have 594

Hence, by the proposition's assumptions, Z verifies 595

From Lemma 4, we have 596

where we used Equation (9). 597

This allows us to bound every term in Equation (38). For the second term on the right-hand side, we 598 have 599

As for the third term, we have 600

Consequently, denoting v = a 2 ( G , Θ ) + (1 -κ ) + ∥ B ∥ 2 , 1 a 1 ( G , Θ ) , and combining Equations (38) to (40), 601 we obtain 602

603

which finishes the proof.

Proposition 6 (Inheritance of a RE condition from a close matrix) . Assume that the matrix V V 604 verifies the RE condition with constant ϕ > 0 , and that ∥ ∥ ∥ ∥ A V t -V V ∥ ∥ ∥ ∥ op , RE ≤ γϕ 2 for some 605

606

Proof. From Proposition 4, we know that 607

609

610

611

Then, for any u, v > 0 , 612

613

614

615

Corollary 1. Let { N ( τ ) } t τ =1 by a sequence of matrices of dimension d 1 × d 2 , adapted to filtration {F τ } t τ =1 . Let { t i } N i =1 an increasing sequence with elements in [ t ] for some N ≤ t . Consider the sequence { M ( n ) } N τ =1 of random matrices defined by

Then { M ( n ) } N n =1 is a martingale adapted to the filtration {F t n } N n =1 . 616

Moreover,if ∥ N ( τ ) ∥ ≤ b ∀ τ ∈ [ t ] for some b > 0 , then we have 617

Proof. We denote E [ ·|F s ] as E s [ · ] for any s ∈ N . Also, let C ( s ) := E s -1 [ N ( s )] , which is 618 F s -1 -measurable by construction. We have for any n ∈ [ N ] , 619

where the first equality is due to the tower rule since F t n -1 ⊂ F t n -1 . Also, we have for any τ ≥ 1 620

Hence N ( τ ) -C ( τ ) is integrable for any τ ≥ 1 . This shows that M ( n ) is a sequence of partial sums 621 of matrix martingale differences, hence it is a matrix martingale. 622

The second part of the corollary statement is a consequence of Theorem 5. The boundedness of 623 the sequence of martingale differences has already been established above. To verify the second 624 requirement of the theorem, let us compute bounds on the norms of W col and W row from Theorem 5. 625 Notice that the two matrices are equal since the difference sequence matrices N ( t s ) are symmetric. 626

Theorem 5 (Matrix Freedman Inequality, Tropp [2011]) . Consider a matrix martingale { M ( t ) } t ≥ 1 with dimension d 1 × d 2 . Let { N ( t ) } t ≥ 1 be the associated difference sequence. Assume that for some A > 0 , we have ∥ N ( t ) ∥ ≤ A ∀ t ≥ 1 almost surely. Define for any t ≥ 1 :

Hence, for any n ∈ [ N ] , we have 627

By Theorem 5, we have for any u > 0 628

where the last line holds because we showed that the inequality ∥ W col ( N ) ∥ ≤ Nb 2 holds almost 629 surely. 630

Proposition 7 (Concentration of the empirical multi-task Gram matrix around the adapted one) . Let 631 t ≥ 1 , b > 0 . Then we have: 632

where 633

Proof. For γ > 0 , let us define 634

where ∆ V is block diagonal matrix formed by { ∆ m } m ∈V . We also define ∆ C and ∆ 2 C in the same 635 pattern of Definition 3. We can express the complementary of this event as the disjunction of a finite 636

number of events as follows: 637

The first and third event can be bounded by considering the sequence xx ⊤ ( τ ) adapted to the filtration 638 {F τ } , verifying ∥ ∥ xx ⊤ ( τ ) ∥ ∥ ≤ . 639

Bounding the probability of the first event Let C ∈ P be a cluster. By definition, we have 640

We will apply Corollary 1 for the sequence of time indices in C , i.e. ⋃ m ∈V T m ( t ) . Hence |C| ∆ C is a 641 martingale sequence, and we have 642

Bounding the probability of the third event Let m ∈ V be a task index. We apply Corollary 1 for 643 the sequence of time steps in T m ( t ) . We have 644

is a martingale sequence, hence 645

Bounding the probability of the second event Let C ∈ P be a cluster, and let us denote e m the 646 m th canonical vector of R |C| . We have 647

where the last equality holds since m ( τ ) is measurable w.r.t. F τ -1 . We will apply the Corollary 1 to 648 the set of time steps ⋃ m ∈C T m ( t ) and the adapted sequence e ⊤ m ( τ ) ⊗ xx ( τ ) of matrices in R d × d |C| . 649 Hence we have 650

Union bound We conclude the result of the statement via a union bound using Equation (61). 651

Proposition 8 (Concentration of the empirical multi-task Gram matrix around the adapted one, 652 simplified) . propEmpCovConcentrationSimplified Let t ≥ 1 , b > 0 . Assume that max m ∈V |T m ( t ) | ≤ 653 bt . Then we have: 654

where ˜ c G ( C ) := c G ( C ) ∧ |C| ∀C ∈ P . 655

Proof. The proof will rely on simple calculus inequalities. Hence, let u = min C∈P c G ( C ) , v = 656 min C∈P |C| , f = 3 γ 2 , g = 6 b, h = 2 √ 2 γ , which are all positive. Then, we have 657

where we used the fact that functions of the form x ↦→ x β 1 x + β 2 for positive β 1 , β 2 are increasing on 658 R + . 659

660

661

As a final step, we use the inequality (1 ∧ x ) f f +(1 ∧ x ) g ≥ x ∧ 1 f + g taken for x = u ∧ v , we apply the exp( -· t ) function and we use the result of Proposition 7, we deduce the result.

662

663

664

665

666

B.3.3 From the true to the adapted Gram matrix

For all of the proofs in this subsection, we follow an approach similar to that of Oh et al. [2021]. In particular, we use their Lemma 10.

Theorem 6 (Lemma 10 of Oh et al. [2021]) . Under Assumption 2 on the context generating distribution, let t ≥ 1 . We have for any θ ∈ R d :

Proposition 9 (RE condition from the true to the adapted Gram matrix) . Under Assumption 2, for 667 any t ≥ 1 , the adapted Gram matrix V V ( t ) verifies the compatibility condition with constants κ and 668 ϕ √ 2 νω . 669

Proof. For t ≥ 1 , we have 670

Let m ∈ V . We have 671

Now, let Z ∈ S , where S is defined with constant κ of Assumption 4. Then 672

which finishes the proof. 673

Theorem 2 (RE condition holding for the empirical multi-task Gram matrix) . Under assumptions 2 674 and 4, let t ≥ 1 , and let κ, ϕ be the constants from Assumption 4. Assume that max ∈V |T ( t ) | ≤ bt 675

verifies the RE condition with constants κ and ϕ , with 677

m m . Then, for any γ ∈ ( 0 , ( 1 + a 2 ( G , Θ )+(1 -κ ) + √ 2 w ( ∂ P ) a 1 ( G , Θ ) ) -2 ) , the empirical multi-task Gram matrix 676 ˆ

with a probability at least equal to 1 -6 d |V| exp ( -3 γ 2 ˜ ϕ 4 (min C∈P (˜ c G ( C ) ∧ ˜ c G ( C ) 2 ) t 6 b +2 √ 2 γ ˜ ϕ 2 ) 678 ˜ ϕ := ϕ √ 2 νω and ˜ c G ( C ) := c G ( C ) ∧ |C| ∀C ∈ P . 679

, where

Proof. For the sake of readability, let ˜ ϕ = ϕ √ 2 νω the compatibility constant of the adapted Gram 680 matrix, according to Proposition 9. Then: 681

≤ P [ A V t satisfies the RE condition with constant κ and ˆ ϕ ] (by Proposition 6) , (70)

B.4 Regret bound 683

Lemma 5 (Concentration of the fraction of observations per task) . lemma Assume that |V| ≥ 2 . Then 684 for δ ∈ (0 , 1) , we have with a probability at least 1 -δ : 685

Proof. We have |T m ( t ) | := ∑ t τ =1 [ m ( τ ) = m ] , where ∀ t, ∀ m ∈ V , P [ m ( t ) = m ] = 1 |V| , meaning that the binary variable [ m ( t ) = m ] follows a Bernoulli distribution B ( 1 V ) . Then, the random variable X t := [ m ( t ) = m ] -1 |V| has mean 0, variance 1 |V| (1 -1 |V| ) , and verifies | X t | ≤ 1 -1 |V| since |V| ≥ 2 . As a result, via the Bernstein inequality, we have for any m ∈ V , and for any w ≥ 0 ,

For the right-hand side to hold with a probability at most δ ∈ (0 , 1) , it is sufficient to have 686

Hence, and via a union bound, we get 687

The result is obtained by adjusting the value of δ . 688

689

690

691

Theorem 3 (Regret bound) . Let the mean horizon per node be T = T |V| . Let min C∈P √ c G ( C ) going asymptotically to infinity and max C∈P √ ι G ( C ) going asymptotically to zero as well as max C∈P √ ι G ( C ) w ( ∂ P ) and w ( ∂ P ) min C∈P √ c G ( C ) going asymptotically to zero. Under assumptions1 to 4

692

693

694

695

696

697

698

699

and κ < 1 , the expected regret of the Network Lasso Bandit algorithm is upper bounded as follows:

Proof. For any time step t , we will define a list of good events under which the Oracle inequality and the RE condition for the empirical multi-task Gram matrix both hold with high probability. Then, we will use those bounds to sum up over time steps until horizon T .

Good events We formalize these requirements as three families of time-depending "good" events.

G pro ( t ) is the event that the mean of the empirical process bounded by α ( t ) up to a constant c , which is equivalent to saying that it converges:

· G sel ( t ) is the event that the number of selections of all tasks is bounded by its expected value up 700 to a small constant ρ ( t ) 701

G RE ( t ) is the event that the empirical multi-task Gram matrix 1 t A V ( t ) satisfies the RE condition. 702

Event G pro ( t ) is the most straightforward to cover since our bound on the empirical process given in 703 Lemma 3 holds with a probability of at least 1 -δ ( t ) , thus: 704

where we included the time dependency on δ ( t ) in contrast to the previous section. This way we 705 emphasize to adjust δ ( t ) after each round, to guarantee a sub linear regret bound. The probability of 706 event G sel ( t ) can be determined using Bernstein's inequality: 707

From Lemma 5 we can select ρ ( t ) = 2 √ t |V| log |V| δ sel ( t ) + 4 3 log |V| δ sel ( t ) as well as P [ G sel ( t ) c ] ≤ δ sel ( t ) . 708

B.4.1 Instantaneous regret decomposition 709

Now, given the event probabilities, we condition the instantaneous regret r ( t ) on the good events at a 710 time t > t 0 . We have for its expectation: 711

where we used the worst case bound r ( t ) ≤ 2 if any one of the good events does not hold. 712

Bounding the regret Inserting our results of the event probabilities, the oracle inequality and the 713 decomposition of the expected instantaneous regret in Equation (76) and bounding the sum over 714 rounds, yields the final result. Thus, we start by bounding the sum over the first term i.e. the expected 715 regret in case all good events hold: 716

Taking the result of our oracle inequality in Theorem 1, we point out that only α ( t ) is time dependent 717 such that the rest of the terms can be pulled outside the sum: 718

where 719

Weupper bounded the sum with an integral i.e. ∑ T t =1 f ( t ) ≤ ∫ T 0 f ( t ) dt for monotonically decreasing 720 functions f ( t ) in the last inequality. Also b is the bound on the concentration of the fraction of 721 observation per task provided by Lemma 5. For t 0 = √ |V| we find by inserting the result to Lemma 5 722 for all t > t 0 : 723

Finally we bound the sum over the instantaneous regret term for the bad events: 724

By construction, we have max( P [ G pro ( t ) c | G sel ( t )] , P [ G sel ( t ) c ]) ≤ δ ( t ) = 1 t 2 . Hence, 725

As for the RE condition event, letting A := 3 γ 2 min C∈P (˜ c G ( C ) ∧ ˜ c 2 G ( C )) 6 b +2 √ 2 γ , we have for any t 0 ≥ 1 726

where in the last line, we used the inequality exp( A ) ≥ A +1 . Hence, for any u > 0 , choosing 727

implies that ∑ T t = t 0 P [ G RE ( t ) c | G sel ( t )] ≤ u . Before we continue with the regret bound, we need to 728 find an appropriate bound on f ( G , Θ ) ˆ ϕ 2 . Given our result in Theorem 1 and assuming that κ > 1 , we 729 get: 730

The first big O notation is obtained due to the fact that for for large min C∈P √ c G ( C ) and small 731 max C∈P √ ι G ( C ) the denominator term i.e. ˆ ϕ 2 behaves like 1 -γ , which leaves the numerator 732 dominating the rest of the term. Now, we simply have to insert all our results into the sum of 733 instantaneous regrets: 734

where we set u = 1 2 A in the third inequality. 735 736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

C Additional related work

Homophily and modularity in social networks Given the large number of users on social networks, one may be able to learn their preferences more quickly by leveraging the similarities between them. This idea relies on the notion of homophily in social networks McPherson et al. [2001], Easley et al. [2010]. In modelling social networks, users' preferences relationships are encoded in a graph, where neighboring nodes are users with similar preferences. This graph can be known a priori or it can be inferred from previously collected feedback Dong et al. [2019]. Exploiting this information and integrating them into bandit algorithms can lead to a significant increase in performance Yang et al. [2020]. Indeed, the knowledge of user relations allows the algorithm to tackle the data sparsity issue that is inherent to bandit settings.

Another fundamental point that can be used for integration of information from social networks is that, social networks show large modularity measures Newman [2006] Borge-Holthoefer et al. [2011]. This implies that we have high density of edges within clusters and low density of edges between clusters. As a result, users can be clustered based on the graph topology and a preference vector can be learned for each cluster, substantially reducing the dimensionality of the problem. In other words, discovering the clustering structure of users can reduce the computational burden of large social networks. Consequently, there have been attempts in exploiting the clustered structures of social networks in bandit algorithms Gentile et al. [2014], Nguyen and Lauw [2014], Yang and Toni [2018], Li et al. [2019], Nourani-Koliji et al. [2023], Cheng et al. [2023].

Bandit meta-learning In contrast to the multi-task setting, meta learning deals with sequentially arriving tasks that have to be learnt and generalizing the gained information to improve performance for future tasks. Here, as in the multi-task setting, it is assumed that the tasks share some common structure that is ought to be learnt and exploited. In the work of Bilaj et al. [2024] it is assumed that the tasks were sampled from a common distribution such that they are concentrated around an affine subspace, which is learnt through PCA algorithm. The resulting projection matrices could then be exploited to improve learning for new tasks in an adapted UCB and Thompson sampling approach. Other lines of work are Cella et al. [2020], Kveton et al. [2021], Basu et al. [2021], which learns the mean of the distribution under the assumption that the covariance of the prior is known or Peleg et al. [2022] which generalizes this assumption and attempts to learn the covariance as well.

D Additional experimental details

D.1 About experiments of the main paper

The experiments have been conducted with an intel i7 CPU with 12 2.6 GHz cores and 32 GB of RAM. The two experiments with the highest number of tasks (200) and dimension (80) take about 8 hours, parallelized over the 12 cores.

To generate clusters, we generate |P| variables v ii ∈P from the uniform distribution, then we use them to construct a categorical distribution with probabilities proportional to e v i . These probabilities defines the cluster proportions.

D.2 Solving the Network Lasso problem

We implement the Primal-Dual algorithm proposed in Jung [2020] to solve the Network Lasso problem but we do not vectorize the matrices (in the sense of stacking their columns into a vector), which speeds up computation.

D.3 Algebraic connectivity vs topological centrality index

Given two fully connected graphs weightless G 1 and G 2 with size 100 each, we progressively link them by edges, we construct the Laplcian L of the resulting graph G . We measure the minimum topological centrality index min 1 ≤ i ∈ 200 ( L † C ) -1 ii , and the algebraic connectivity, i.e. the minimum non-null eigenvalue of L .

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

fraction of all edges linking the two complete graphs

0

20

40

60

80

100

120

140

algebraic connectivity

minimum topological centrality index

Clearly, the minimum topological centrality index grows faster than the algebraic connectivity in 783 this case, and seems to saturate at some level that is reached in a linear progress by the algebraic 784 connectivity. 785

786

D.4 Limitations

The first limitation of the paper is the restriction to the setting of i.i.d generated action sets. This 787 restriction is common to all papers relying on Lasso-type optimization objectives [Bastani and Bayati, 788 2019, Oh et al., 2021, Cella and Pontil, 2021, Ariu et al., 2022, Cella et al., 2023]. Also, we do not 789 provide a lower bound for the regret, a challenge that we let for future work. Besides, our optimization 790 problem is not strongly convex, which can be mitigated by adding a squared L 2 norm regularization. 791 However, such an addition would probably drastically change the theoretical analysis. 792

793

D.5 Broader Impacts

As our method can be applied to transfer knowledge between users of a recommender system, it has 794 the potential to improve their overall experience by learning their preferences quickly. However, one 795 must be careful with the strength of the integrated prior knowledge as it can lead to an adverse effect 796 of slowing down the learning process. 797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The piecewise stationarity on a graph assumption is mentioned in Section 3 and formalized in Assumption 3. Theorem 1 states the oracle inequality, and Theorem 3 provides the regret bound after using the result of Theorem 2. Experiments are carried out at Section 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Appendix D.4 is dedicated to such discussion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state the main assumptions in an Assumption environment. Full proofs are available in the supplementary material and some proof ideas in the main material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code is included as a zip-file and all results are reproducible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use only simulated data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: While training and testing sets are not part of our paper as it is about bandit algorithms, we provide the source of the optimization algorithm we use to solve Equation (2).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are provided in the plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Computation resources used are mentioned in Appendix D.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our results are theoretical, and their potential harm largely depends on their application.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: It is provided in Appendix D.5 in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Only simulated data are used.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All relevant work is properly cited and code is provided to reproduce the 1045 results. 1046

1047

1048

1049

1050

1051

1052

1053

1054

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

1081

Justification: [NA]

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

1098

Justification: [NA]

1099

1100

1101

1102

1103

1104

1105

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

1106

1107

1108

1109

1110

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to use network Lasso to learn a multi-task bandit problem with given network structure. More specifically, the network structure has pre-defined unknown clustering structure, where within each cluster all the bandit tasks share the same model. The authors propose a bandit algorithm that can learn and provide a sublinear guarantee. The key difference of this paper with GOBLin in Cesa-Bianchi et al., 2013 is that this paper uses a network Lasso (or something like a group Lasso) penalty while GOBLin use a ridge penalty.\n\nSTRENGTHS:\nEven though I think the authors overclaimed their contributions, which I will state below, I still feel it's meaningful to discuss network Lasso and design a bandit algorithm based on certain network structure, given the limited literature on multitask bandit. Compared to previous network bandit literature such as Cesa-Bianchi et al., 2013, this paper characterizes the network structure in more detail.\n\nWEAKNESSES:\n1. I think the authors need to provide more real-world examples to show why their network structure (and the correpondingly induced network Lasso bandit algorithm) is pratical, instead of stating their algorithm is good because it provides piecewise constant property in constrast to smoothness of GOBLin in Cesa-Bianchi et al., 2013. The key assumption in this paper is that the network structure is given, and the network can be splitted into connected clusters, within which the task parameters are the same. Can authors find or describe a couple of pratical examples/datasets where such a network exists, given that bandit is a very pratical problem?\n\n2. The literature review that compare with the previous literature are not very accurate and sufficient IMO. For example, the authors mention Gentile et al., 2014 and Li et al., 2019 can cause overconfidence in constructing clusters. However, these algorithms do not have prior information about clusters such as a given network and thus they have to learn the clusters conditioned on the task similarities. In that sense, these algorithms are more pratical because oftentimes in practice network information is lacking. Here one should also add a related reference Context-Based Dynamic Pricing with Online Clustering by Miao et al., 2022. There are also robust multitask bandit algorithms (e.g., Multitask Learning and Bandits via Robust Statistics by Xu and Bastani, 2024) that can also solve the network bandit problem if the network structure follows certain assumptions; Multi-Task Learning for Contextual Bandits by Deshmukh et al., 2017 discuss a multitask bandit problem but use kernal based method. I suggest the authors add a more detailed literature review to discuss their paper's connection with these current multitask bandit algorithms. \n\n3. Typically, greedy algorithm (the method proposed in this paper is greedy too due to Assumption 2) has better performance in bandit simulations compared to UCB-based algorithm (see Bastani and Bayati, 2020). Therefore, it's not a fair comparison in Figure 1, where all benchmarks are UCB algorithms. I think it's necessary to add the following benchmarks: OLS-Bandit or Lasso-Bandit (Bastani and Bayati, 2020) without task sharing, and Cella et al., 2023 which use low-rank structure to do multitask bandit, and also a few others mentioned above as a benchmark to show that the network structure indeed helps, fixing the difference due to UCB and greedy algorithms.\n\n4. I feel it's a false claim that the regret bound in Theorem 3 \"doesn't depend on the dimension\" and it's due to the concentration inequality from Hsu et al., 2012. Intuitively, the regret bound should depend on the dimension unless one assume that the number of tasks in a cluster is d-dependent so the regret bound is smaller compared to a typical single bandit regret bound. I think the reason why here the bound seems to be d-independent is because the dimension $d$ is hiden in the problem-dependent parameter $\\phi$. Since the authors assume the context $x$ has norm 1, the minimum eigenvalue of $E[xx^\\top]$ should scale as $1/d$, and hence $\\phi^2$. I don't think a typical tail inequality in Hsu et al., 2012 can improve the bound regarding the context dimension. \n\n5. I think the asymptotic assumptions in Theorem 3 is incompatible with the finite-sample analysis in a typical bandit analysis and looks weird. I suggest the authors keep the isoperimetric ratio and centrality index as part of the regret bound (instead of forcing them out using the asymptotic assumptions), even though it might add an additional T-linear term due to the misspecification error caused by the inter-cluster edge connections. I feel that's the case because in the extreme case when each cluster has size 1, there will be misspecification error penalizing tasks connected the inter-cluster edges towards each other. I think it's totally fine to have such non-sublinear terms to provide a more comprehensive understanding of the limit of such network structures. \n\nI am willing to raise my rating if the authors can solve my questions and concerns.\n\nQUESTIONS:\nSee my points above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors work under the multi-task contextual bandit settings by representing the task correlations through the graph structure. To solve this problem, authors propose an algorithm that utilizes a linear regression formulation with Lasso constraint in terms of the node connectivity. Theoretical analysis as well as experiments against several baselines are presented to demonstrate the effectiveness of proposed method.\n\n-\tThe paper is generally well-written, with crisply clear descriptions of required assumptions, and the proposed solution is intuitive and well-motivated.\n\n-\tGood empirical performances. Authors compare proposed algorithm with several clustering of bandits baselines, showing the effectiveness of the proposed method. The performance gain over existing methods is impressive. \n\n-\tNovel theoretical analysis roadmap. Overall, the theoretical analysis pipeline is novel and the looks promising to me. With the additional introduced RE assumption, authors are able to improve the regret bound to $\\tilde{O}(\\sqrt{\\bar{T}})$ instead of the vanilla time horizon.\n\n-\tMy major question is regarding the numerous assumptions required for the theoretical analysis. For instance, in Assumption 1, authors assume the candidate arms across different arounds are generated i.i.d. from a fixed distribution. This is different from existing clustering of bandits works, where the candidate arm contexts in each round is conditioned on previous observed arms. In this case, assumption 1 is somehow deviates from the actual applications of recommender systems where the candidate arms of each round are refined along with more information collected from the environment.\n\n-\tFor the experiments, authors have compared against multiple clustering of bandit works. In this case, it would be good if authors can include additional discussions comparing your theoretical outcomes with those of existing clustering of bandits works, which can offer more intuitive comparison with existing approaches.\n\nSTRENGTHS:\nPlease see my comments above.\n\nWEAKNESSES:\nPlease see my comments above.\n\nQUESTIONS:\nPlease refer to my questions above.\n\nLIMITATIONS:\nPlease see my comments above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the multi-task bandit problem using graph information. The given graph represents the relationships between tasks. Assuming that the preference vector of clustered tasks is constant, the problem is formulated as a network lasso problem to estimate the lasso estimator. \nA modified restricted eigenvalue condition, commonly used in high-dimensional statistics, is defined to derive the oracle inequality for the network lasso estimator on non-i.i.d. data. The oracle inequality of the proposed network lasso estimator is derived under the assumption that the true multi-task Gram matrix satisfies the adapted RE condition.\nBased on the derived oracle inequality, a greedy-type algorithm is presented, achieving $\\sqrt{T}$ regret. Numerical experiments support the theoretical performance of the proposed algorithm.\n\nSTRENGTHS:\n- The proposed algorithm efficiently learns task preference vectors by using graph information that encodes relationships between tasks. Specifically, it employs a network lasso estimator under the assumption that preferences within clustered tasks are constant, demonstrating its effectiveness in high-dimensional contexts.\n- The paper adapts the restricted eigenvalue condition from high-dimensional statistics to the graph-based multi-task bandit setting. Based on the adapted RE condition, they established oracle inequality for network lasso estimator and showed that the proposed algorithm achieved $\\sqrt{T}$ regert even though I haven't verified every proof in detail.\n- The algorithm's performance seems robust even as the number of tasks and dimensions increase.\n\nWEAKNESSES:\n- Since I'm not very familiar with the graph-based multi-task bandit setting, it may be that the concepts explaining the restricted eigenvalue condition (Def 2) are too heavy. It would be helpful to include comparisons or examples from existing RE conditions in high-dimensional statistics or high-dimensional contextual bandits to improve understanding.\n\nQUESTIONS:\n1. You mentioned that the proposed algorithm is cluster-agnostic, but if a graph representing the relationships between tasks is given, can't we identify which tasks are clustered? Compared to Gentile et al. (2014), it seems that the problem setting uses additional information. If the same information were given to Gentile et al. (2014), what advantages does the proposed algorithm have compared to not needing clustering?\n2. Is the estimation error in the oracle inequality (Theorem 1) dimension($d$)-independent?\n3. Can the restricted eigenvalue condition be transformed into a compatibility condition?\n4. What is the definition of $\\overline{Z}_P$?\n5. The regret bound of high-dimensional contextual bandit using the Lasso estimator is proportional to the sparsity level (e.g., usually denoted by $s_0$) instead of the feature dimension $d$. What does sparsity correspond to in the network lasso instance, and how does it appear in the regret bound?\n\nLIMITATIONS:\nThe authors have well-addressed the limitations in Appendix D.4 and further research directions in Section 7.\n\nThe content discussed in this paper appears to have little to no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a multi-task contextual bandit algorithm that leverages a graph structure to model relationships between tasks. The algorithm assumes that the preference vectors of the tasks are piecewise constant over the graph, forming clusters. By solving an online network lasso problem with a time-dependent regularization parameter, the algorithm estimates the preference vectors, achieving a sublinear regret bound lower than independent task learning. Theoretical findings are supported by experimental evaluations against other graph bandit and online clustering algorithms.\n\nSTRENGTHS:\n(1) The paper introduces a approach by incorporating graph structures to model relationships between tasks.\n\n(2) The algorithm is supported by comprehensive theoretical analysis, including a oracle inequality and a regret bound.\n\n(3) Extensive experiments validate the proposed method, showing that it outperforms existing baselines in terms of cumulative regret, highlighting its practical applicability and effectiveness.\n\nWEAKNESSES:\n(1) The problem setting and algorithm presented are primarily adaptations of existing works, such as Oh et al. [2021]. The main difference is the inclusion of a graph matrix in the user preference vector, but this is not the first algorithm to incorporate a graph in contextual bandits, limiting the overall novelty.\n\n(2)  The i.i.d. assumption in contextual bandits is quite strong. Even in clustering approaches like CLUB, a conditional i.i.d. assumption is used. The current regret upper bound complexity is \\(\\sqrt{VT}\\). There should be special cases where the algorithm can improve over \\(V\\) to demonstrate a more significant advantage.\n\n(3)  Since 2019, there have been many more works on clustering in bandits. The authors should conduct a broader survey to include these more recent works and relevant baselines. Using SCLUB, which is considered outdated, as a baseline, limits the comprehensiveness and relevance of the comparative analysis.\n\nQUESTIONS:\nWhat are the unique challenges in regret analysis with Lasso regularization compared to L2 regularization?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-19"}