Title: Asynchronous Multi-Agent Reinforcement Learning with General Function Approximation
Venue: NeurIPS 2024
Accepted: no
Keywords: Reinforcement Learning

Markov Decision Processes

Contextual Bandits

Learning Theory

Federated Learning

Multi-agent

General Function Approximation

Machin Learning
Abstract: We study multi-agent reinforcement learning (RL) where agents cooperate through asynchronous communications with a central server to learn a shared environment. Our first focus is on the case of multi-agent contextual bandits with general function approximation, for which we introduce the Async-NLin-UCB algorithm. This algorithm is proven to achieve a regret of $\tilde{O} (\sqrt{T \dim_E(\mathcal{F}) \log N(\mathcal{F})})$ and a communication complexity of $\tilde{O} (M^2 \dim_E(\mathcal{F}))$, where $M$ is the total number of agents and $T$ is the number of rounds, while $\dim_E(\mathcal{F})$ and $N(\mathcal{F})$ are  the Eluder dimension and the covering number of function space $\mathcal{F}$ respectively. We then progress to the more intricate framework of multi-agent RL with general function approximation, and present the Async-NLSVI-UCB algorithm. This algorithm enjoys a regret of $\tilde{O} (H^2 \sqrt{K \dim_E(\mathcal{F}) \log N(\mathcal{F})})$ and a communication complexity of $\tilde{O} (H M^2 \dim_E(\mathcal{F}))$, where $H$ is the horizon length and $K$ the number of episodes. Our findings showcase the provable efficiency of both algorithms in fostering collaborative learning within nonlinear environments, and they achieve this with minimal communication overhead.

1

Abstract

We study multi-agent reinforcement learning (RL) where agents cooperate through asynchronous communications with a central server to learn a shared environment. Our first focus is on the case of multi-agent contextual bandits with general function approximation, for which we introduce the Async-NLin-UCB algorithm. This algorithm is proven to achieve a regret of ˜ O ( √ T dim E ( F ) log N ( F )) and a communication complexity of ˜ O ( M 2 dim E ( F )) , where M is the total number of agents and T is the number of rounds, while dim E ( F ) and N ( F ) are the Eluder dimension and the covering number of function space F respectively. We then progress to the more intricate setting of multi-agent RL with general function approximation, and present the Async-NLSVI-UCB algorithm. This algorithm enjoys a regret of ˜ O ( H 2 √ K dim E ( F ) log N ( F )) and a communication complexity of ˜ O ( HM 2 dim E ( F )) , where H is the horizon length and K the number of episodes. Our findings showcase the provable efficiency of both algorithms for collaborative learning within nonlinear environments and minimal communication overhead.

1 Introduction

Multi-agent reinforcement learning (RL) is an important paradigm in RL, and has been successfully 16 applied to real-world tasks such as robotics [Williams et al., 2016, Liu et al., 2019, Ding et al., 2020, 17 Liu et al., 2020, Na et al., 2022], games [Vinyals et al., 2017, Berner et al., 2019, Jaderberg et al., 18 2019, Ye et al., 2020], and control systems [Bazzan, 2009, Yu et al., 2014, 2020, Min et al., 2022, Xu 19 et al., 2023]. By learning cooperatively, agents benefit from sharing learning experiences, enabling 20 them to collectively enhance their decision-making capabilities. This collaborative process is usually 21 accomplished through the utilization of a central server, whose task is to aggregate local data and 22 deliver feedback for the agents. 23 There has been an excellent line of work establishing provably efficient algorithms for multi-agent 24 bandits and RL. However, most existing works are restricted to the synchronous setting, where com25 munications between all agents and the server must happen simultaneously. This is impractical since 26 in many scenarios the availability of agents may vary and be unpredictable. Ideally, communication 27 should be allowed to happen asynchronously to offer the agents more flexibility. He et al. [2022] and 28 Min et al. [2023] studied this setting respectively for linear contextual bandits and linear Markov 29 Decision Processes (MDPs), both of which assumes linearity in the environment, and introduced 30 algorithms with low regret and communication cost. Yet the linear function class is quite limited, and 31 does not encompass practical reinforcement learning scenarios where nonlinearity is prevalent. 32 To address the aforementioned drawback, in this work, we tackle environments with general function 33 approximation, broadening the applicability of the algorithm to more realistic and complex scenarios. 34 Wefirst delve into multi-agent contextual bandits with general function approximation, where multiple 35 agents interact with homogeneous environments in parallel to solve a common objective. Notably, 36 the communication protocol is designed to be flexible and asynchronous, allowing agents to initiate 37

communication with the server and acquire new policy functions whenever the need arises. The 38

primary objective is to minimize total regret while reducing communication cost as much as possible. 39

Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.

We propose an algorithm Async-NLin-UCB, which adapts a fully asynchronous communication 40

protocol, and leverages various methods for tackling nonlinear function approximation. Despite the 41

flexibility of communication, our algorithm performs almost as well as a single agent, in terms of a 42

regret that is mostly independent of the number of agents and a low communication cost. 43

We then progress to multi-agent RL with general function approximation under similar requirements 44 and objectives. We propose an algorithm named Async-NLSVI-UCB based on Least-Squares Value 45 Iteration (LSVI) to learn the underlying Markov decision processes (MDPs), which demonstrates 46 similar advantages with provably low regret and communication cost. 47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

Our main contributions are summarized in the following:

For asynchronous multi-agent nonlinear contextual bandits, we propose the algorithm Async-NLin-UCB, which enjoys an ˜ O ( √ T dim E ( F ) log N ( F ) + dim E ( F )) regret and an ˜ O ( M 2 dim E ( F )) communication complexity, where dim E ( F ) and N ( F ) are respectively the Eluder dimension and the covering number of function space F .

For asynchronous multi-agent nonlinear MDPs, we propose the algorithm Async-NLSVI-UCB, which enjoys an ˜ O ( H 2 √ K dim E ( F ) log N ( F ) + H 2 dim E ( F )) regret and a communication complexity of ˜ O ( HM 2 dim E ( F )) .

At the core of our algorithm, we design a communication criterion in order to tackles the challenges posed by both asynchronous communication and the nonlinearity of function approximation. To guarantee a low communication cost, we propose a low switching communication criterion that allows the agent to trigger communication rounds.

We carefully design our download content from server to local agents, which consist only of decision and bonus functions, with no mention of any specific historical data. This effectively protects user data against exposure by disallowing local users from obtaining the data of others.

Notation. We use lower case letters to denote scalars. We denote by [ n ] the set { 1 , . . . , n } . For two positive sequences { a n } and { b n } with n = 1 , 2 , . . . , we write a n = O ( b n ) if there exists an absolute constant C > 0 such that a n ≤ Cb n holds for all n ≥ 1 . We use ˜ O ( · ) to further hide the polylogarithmic factors. For two non-negative integers a, b satisfying a < b and a sequence { s i } indexed by integers i , we use s [ a : b ] to denote the subsequence { s a , s a +1 , · · · , s b } .

2 Related Work

2.1 Multi-Agent Bandits

First, there is a multitude of previous work on distributed or federated multi-armed bandits and stochastic linear bandits [Liu and Zhao, 2010, Szorenyi et al., 2013, Landgren et al., 2016, Chakraborty et al., 2017, Landgren et al., 2018, Martínez-Rubio et al., 2019, Sankararaman et al., 2019, Wang et al., 2020a,c, Zhu et al., 2021, Huang et al., 2021]. For the more realistic setting of contextual bandits, most previous work are within the scope of linear contextual bandits with synchronized communication. Korda et al. [2016] introduced two novel distributed confidence ball (DCB) algorithms for linear bandit problems in peer-to-peer networks. Wang et al. [2020c] considered both P2P and star-shaped communication, achieving near-optimal regret and low communication cost that is largely independent of the time horizon in their algorithm DisLinUCB. Dubey and Pentland [2020] proposed FedUCB, an algorithm focusing on differential-privacy. Li and Wang [2022] first considered an asynchronous communication protocol and proposed the algorithm Async-LinUCB with near-optimal regret, yet the algorithm contains a download step for all agents triggered by the central server. Their results are flexible and contains a parameter to control the trade-off between regret and communication cost. He et al. [2022] improved the setting to a fully asynchronous communication, proposing the algorithm FedLinUCB with near-optimal regret of ˜ O ( d √ T ) and low communication cost of ˜ O ( dm 2 ) , comparable to the benchmark in singleagent contextual linear bandits [Abbasi-Yadkori et al., 2011]. We consider the same communication protocol in our results. A summary of these results along with ours can be found in the first four rows of Table 1.

2.2 Multi-Agent RL

Multi-agent reinforcement learning is decidedly more challenging than contextual bandits. There is also a vast literature on this setting, with many works discussing different aspects of multi-agent RL

98

99

100

101

102

103

104

105

Narrowing it down to multi-agent RL with function approximation, the benchmark is the LSVI-UCB algorithm in the single-agent setting [Jin et al., 2020], with an ˜ O ( d 3 / 2 H 2 √ K ) regret. Dubey and Pentland [2021] proposed CoopLSVI for multi-agent linear MDPs, which requires a synchronized communication through central server, and proves a regret of ˜ O ( d 3 / 2 H 2 √ MK ) . They also extended their result to the heterogeneous setting. Min et al. [2023] considered the fully asynchronous setting and introduced the Async-Coop-LSVI-UCB algorithm, with a ˜ O ( d 3 / 2 H 2 √ K ) regret not dependent on the number of agents M , as well as a low communication cost. A summary of these results along with ours can be found in the last three rows of Table 1.

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

2.3 General function approximation

Reinforcement learning with general function approximation extends the well-studied case of linear MDPs to more general classes of MDPs, and has gained a lot of traction in recent years [Wang et al., 2020b, Jin et al., 2021, Foster et al., 2023, Du et al., 2021, Agarwal and Zhang, 2022, Agarwal et al., 2023]. Previous works focus on different measures of complexity for the function classes, for example the Bellman rank proposed by Jiang et al. [2017], the Bellman Eluder dimension introduced in Jin et al. [2021], the Decision-Estimation Coefficient in Foster et al. [2023], and generalized Eluder dimension in Agarwal et al. [2023]. Our work considers the Eluder dimension with the introduction of uncertainty estimators D 2 , which has been widely utilized to establish results in RL with general function approximation [Agarwal et al., 2023, Zhao et al., 2023, Ye et al., 2023, Di et al., 2023].

3 Preliminaries

In this section, we introduce the formal definition of both multi-agent nonlinear contextual bandits and MDPs and some related concepts, and discuss the asynchronous communication protocol.

3.1 Multi-Agent Contextual Bandits with General Function Approximation

We assume a global action set A that is known to all agents. At each round t ∈ [ T ] , a single arbitrary agent m t ∈ [ M ] is chosen to participate. The agent receives a contextual decision set A t ⊆ A and chooses from the set an action a t ∈ A t to perform, and subsequently receives a random reward r t .

The assumption of general function approximation is that the reward is generated according to 123

where f ∗ is the ground truth objective function, and η t is a random noise variable. We assume the the objective function lies within a known function class F . In addition, we also make the following assumptions regarding the function class and noise variables, which are standard assumptions for contextual bandits [Abbasi-Yadkori et al., 2011, He et al., 2022]:

Assumption 3.1. Suppose the following conditions hold for the contextual bandits environment:

For any f ∈ F and a ∈ A , | f ( a ) | ≤ 1 ;

η t is R -sub-Gaussian conditioned on data history: E [ e λη t ∣ ∣ a 1: t , m 1: t , r 1: t -1 ] ≤ exp( R 2 λ 2 / 2) , ∀ λ.

Learning Objective. The primary goal of contextual bandits is to minimize the cumulative regret

Notice that this summation is across all time steps does not depend on agent participation order, as should be the case for the resulting regret bound. To achieve this goal, agents are allowed to communicate with the server to upload their interaction history and update their policy. The secondary learning objective is to reduce communication overhead. We will explain the communication protocol further in Section 3.4.

3.2 Multi-Agent Episodic MDPs with General Function Approximation

We consider episodic MDPs, which are a classic family of models in reinforcement learning [Sutton and Barto, 2018]. It is characterized by the following elements, which we assume to be homogeneous across all agents: a state space S , an action space A , the horizon length H , transition probability functions P = { P h ( ·|· , · ) } H h =1 and reward functions { r h ( · , · ) } H h =1 ) . Similar to the bandit case, for each episode k = 1 , · · · , K , a single agent m = m k is chosen to participate. An episode k begins with an initial state s k 1 , which is drawn from an unknown fixed distribution. Then for steps h = 1 , · · · , H , the participating agent m selects an action a k h based on the observed state s k h . After each action, the agent receives a reward r k h = r h ( s k h , a k h ) , where r h : S × A → R is the reward function at step h . Here for the sake of convenience, we assume the reward function to be deterministic, but it is not difficult to generalize our result to stochastic rewards. We also assume r h ( s, a ) ∈ [0 , 1] for all ( s, a ) ∈ S × A without loss of generality. The environment then transitions to the next state according to s k h +1 ∼ P h ( ·| s k h , a k h ) , where P h is the transition probability at step h . The episode terminates when r H is observed.

The strategy an agent employs to interact with the environment is called the agent's policy , which can be described by a set of decision functions π = { π h } H h =1 , where π h : S → A is the decision function at level h , mapping the current state to an action to select.

Value Functions. For any policy π = { π h } , we define Q -value functions and V -value functions:

where the expectation is taken over the trajectory ( s 1 , a 1 , · · · , s h , a h ) , determined by the transition probability functions P and policy π . The optimal strategy π ∗ is the maximizer of the value functions:

We also have optimal value functions Q ∗ h := Q π ∗ h and V ∗ h := V π ∗ h , which satisfy Bellman equations 155

Function Approximation. We approximate Q -value functions with function classes {F h } H h =1 , which contain real value functions with domain S × A . One basic assumption is that Q ∗ h ∈ F h for all steps h ∈ [ H ] . Now with the convention that functions at level H +1 are uniformly zero, i.e., f H +1 = 0 , we define the Bellman operator T h :

and we expect T h to map any function in F h +1 to a function in F h , i.e., T h F h +1 ⊆ F h . This is called the completeness assumption, which is a fundamental assumption in RL with general function approximation [Wang et al., 2020b, Jin et al., 2021].

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

Learning Objective. The primary goal in multi-agent MDPs is to minimize the cumulative regret over K episodes

where π m,k is the policy of agent m = m k at round k , while the secondary objective is to minimize the communication cost.

3.3 Eluder Dimension and Covering Number

To measure the complexity of the learning objective, Russo and Van Roy [2013] first proposed the concept of Eluder dimension, which we define below.

Definition 3.2 ( ϵ -dependence) . For a function class F on domain D , a point z ∈ D is ϵ -dependent on Z ⊆ D if, for any f 1 , f 2 ∈ F satisfying √ ∑ z ′ ∈Z ( f 1 ( z ′ ) -f 2 ( z ′ ) ) 2 ≤ ϵ , it must hold that | f 1 ( z ) -f 2 ( z ) | ≤ ϵ . Accordingly, z is ϵ -independent of Z if it is not ϵ -dependent on Z .

Definition 3.3 (Eluder dimension) . The ϵ -Eluder dimension dim E ( F , ϵ ) is the length of the longest sequence of elements in D satisfying that, for some ϵ 0 > ϵ , each element is ϵ 0 -independent of the set consisting of its predecessors.

It has been demonstrated that the Eluder dimension roughly corresponds to regular dimension concepts in linear and quadratic cases [Russo and Van Roy, 2013], and that the Eluder family is strictly larger than the generalized linear class [Li et al., 2022]. Note that our Eluder definition can be applied to either the contextual bandit case with D = A or the MDPs case with D = S × A .

We also introduce covering number for function classes [Wainwright, 2019] in the following:

Definition 3.4 (Covering number) . An ϵ -cover of F is any subset F ϵ ⊆ F such that for any f ∈ F , there exists f ′ ∈ F ϵ that ∥ f -f ′ ∥ ∞ ≤ ϵ . The covering number of F , denoted by N ( F , ϵ ) , is the minimal cardinality of its ϵ -cover.

3.4 Communication Protocol

We consider a star-shaped communication model [He et al., 2022, Min et al., 2023], where the agents communicate through a central server to collaborate. To ensure asynchronous communication, we mandate that all communications must be initiated by a participating agent. Specifically, at the end of a time step / episode, the agent will decide whether or not to trigger a communication round. If so, the agent uploads its local data history and receives some global data for future decision making. The communication cost is the total number of communication rounds initiated by the agents.

One variability is the form of global data that the communicating agent downloads from server. It may be tempting to have the server send all its stored trajectories to the agent for future decision making, but this will unnecessarily expose other agents' data to the current participating agent. We will come back to this issue and our solution in Section 4.2.

4 Multi-Agent Contextual Bandits

In this section, we introduce the Asynchronous Nonlinear UCB (Async-NLin-UCB) algorithm designed for multi-agent contextual bandits with general function approximation, and provide a theoretical result for its regret and communication cost.

4.1 Algorithm: Async-NLin-UCB

Algorithm 1 takes as input the total number of time steps T , regularization parameter λ , communication parameter α and exploration radii { β t } T t =1 .

In the algorithm, there are some variables that go through different versions as t progresses through 1 , · · · , T . For clarity, here we give them an extra subscript t to denote the version of that variable before (not included) the least squares calculation on Line 12 at round t .

Throughout the learning process, the server maintains a global history set Z ser t that stores actionreward pairs ( a, r ) ∈ A× [0 , 1] , initialized on Line 2 and updated only during communication rounds. Each local agent m maintains a decision function f m,t for taking action, a bonus function b m,t for checking communication criterion, and a local data history set Z loc m,t , all initialized on Line 3. Each step of Algorithm 1 contains two parts: local exploration and server updates.

Part I: Local Exploration. At step t a single agent m = m t is active (Line 5). It receives a decision set, finds the greedy action according to its decision function f m,t , receives a reward, and updates its local dataset Z loc m,t (Lines 5 - 7).

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

Algorithm 1 Async-NLin-UCB

1: Input: total number of rounds T , parameters λ , α , β t for t = 1 , . . . , T .

2: Server init: Set Z ser = ∅ .

3: Local init: For all m ∈ [ M ] , set f m = 1 , b m = B A ( ∅ , F ; λ, β 0 ) and Z loc m = ∅ .

4: for t = 1 , . . . , T do

5: Agent m = m t ∈ [ M ] is active.

6: Receive decision set A t ⊆ A and take action a t ∈ argmax a ∈ D t f m ( a ) and receive reward r t .

7: Update local history Z loc m = Z loc m ∪ { ( a t , r t ) } .

8: if switch condition (4) is met then

9: Send new data Z loc m to server.

10: on server :

11: Update Z ser = Z ser ∪ Z loc m .

12: Calculate ̂ f according to (5) and the bonus function b = B A ( Z ser , F ; λ, β t ) .

13: Send ̂ f + b and b to agent m .

14: end of server

15: Agent m receives decision and bonus functions f m = ̂ f + b , b m = b , then set Z loc m = ∅ .

16: end if

17: end for

After exploration, the agent checks if the switch condition is true using its bonus function:

where t ′ is the last time step when agent m communicated with the server. If so, the agent initiates a communication round and uploads its local data (Line 9), prompting the server to begin global policy updates. We will discuss the reasons behind this switch condition in Section 4.2.

Part II: Server Updates. After receiving a new local data history from an agent, the server merges the data into its global dataset Z ser t (Line 11), and calculate a function ̂ f t +1 ∈ F which minimizes the sum of squares error according to the current dataset Z ser t (Line 12):

The next step is to obtain a bonus function b t +1 from the oracle B A from Definition 4.1 (Line ?? ). We discuss the specifics of this construction in detail up next in Section 4.2. Finally, the server sends the optimistic value function ̂ f t +1 + b t +1 and the bonus function b t +1 back to agent m for future exploration and updates; agent m also resets its local data history to an empty set (Lines 13 and 15).

4.2 Uncertainty Estimators and Bonus Functions

In this section, we introduce uncertainty estimators and bonus functions, and give a detailed explanation for our communication criterion (4). Most of these apply to the MDPs setting as well.

Uncertainty Estimators. First we define the uncertainty estimator of new data a against data history Z , which is considered in many works on bandits and RL with general function approximation [Gentile et al., 2022, Agarwal et al., 2023]:

here λ is the regularization parameter, F is a function class. Intuitively, the uncertainty estimator measures the difference between functions on new data a against the difference on historical data Z . Switch Condition Based On Uncertainty Estimators. The determinant-based criterion is a common technique used in contextual bandits and RL with linear function approximation to reduce policy switching or communication cost [Abbasi-Yadkori et al., 2011]. For nonlinear function approximation, one can use uncertainty estimators to formulate a new form of switch condition:

where we use Z new t and Z old t to denote newly accumulated data and old historical data. This criterion has a similar function as the determinant-based criterion in linear settings. Parameter α controls communication frequency: smaller α indicates more frequent communication, more accurate decision functions and smaller regret, thus implying a trade-off between regret and communication cost.

Bonus Function Oracle. Next, we introduce bonus functions obtained through oracles that approximate the uncertainty estimators.

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

Definition 4.1 (Bonus Function Oracle B D ) . Given domain D , the oracle B D ( Z, F ; λ, β ) takes the following as inputs: a dataset Z consisting of a series of data points ( z, e ) , where z ∈ D and e is some additional data content; function class F with functions f : D → R ≥ 0 ; regularization parameter λ and exploration radius β . It returns a function b ∈ W D : D → R ≥ 0 satisfying for any z ∈ D that

where C B is an absolute constant.

Remark 4.2 . Similar bonus function oracles have been proposed in previous works (Definition 3 in Agarwal et al. [2023]). The accessibility of these oracles is also supported by previous works that proposed methods to compute bonus functions [Kong et al., 2023, Wang et al., 2020b]. In this definition, we leave the domain and data format to be variable so the oracle can be applied to both contextual bandits and MDPs. For bandits, the domain is A , and the data format has z = a and e = r .

The first property of the bonus function guarantees the optimism of decision functions ̂ f t +1 + b t +1 (see Lemma 6.1 for MDPs or Lemma A.2 for bandits), while the second property links bonuses to uncertainty estimators.

Switch Condition Based On Bonus Functions. If we try to adapt the switch condition (7) in our setting, a local agent will require access to historical data Z old t to calculate uncertainty estimators D 2 λ, F ( a ; Z old t ) . For multi-agent learning, this dataset consists of the collective data from all agents, and giving local agent access is a clear violation of data privacy. Our solution is to let local agents download bonus functions and set communication criterion to (4), using bonus functions instead of uncertainty estimators.

Decision Functions Based On Bonus Functions. Another benefit of introducing the bonus function is evident from our exploration method in line 6. A common practice for nonlinear RL algorithms is to construct confidence sets of functions during policy update, and find the optimal function within the confidence sets during exploration [Agarwal et al., 2023, Ye et al., 2023]. However, in a multi-agent setting, this would involve the download of confidence sets, which is impractical due to the complex nature of function classes. With the bonus function, local agents need only download the decision function from the server for future exploration, which for contextual bandits is simply ̂ f t +1 + b t +1 .

4.3 Theoretical Results

Our main results for Algorithm 1 are summarized in the following theorem, which provides a regret upper bound and communication complexity order. √

Theorem 4.3. By taking γ = O (1 /T ) , β t = C β, 1 ( λ + RC ( M,α ) log(3 MN ( F , γ ) /δ ) ) and C ( M,α ) = √ 1 + Mα ( √ 1 + Mα + M √ α ) , the regret of Algorithm 1 within T rounds is O ( √ T ˜ β 1 √ (1 + Mα ) dim E log( T/ min { 1 , λ } ) + (1 + Mα ) dim E log 2 ( T/ min { 1 , λ } ) ) , where we abbreviate dim E := dim E ( F , λ/T ) ; the total communication complexity is )

Remark 4.4 . When reduced to linear contextual bandits, where dim E ( F , λ/T ) = ˜ O ( d ) and log N ( F , γ ) = ˜ O ( d ) , our result on regret correspond exactly to Theorem 5.1 of He et al. [2022], except for an extra 1 + Mα term in the communication cost, an unimportant term when taking α = 1 /M 2 that comes from the complication of communication cost analysis in nonlinear settings.

5 Multi-Agent Reinforcement Learning

In this section, we introduce the Asynchronous Nonlinear Least Squares Value Iteration UCB (Async-NLin-UCB) algorithm for multi-agent MDPs with general function approximation, and a corresponding theoretical result.

5.1 Algorithm: Async-NLSVI-UCB

To better represent the elements in the datasets, we sometimes use o h to represent the tuple ( s h , a h , r h , s h +1 ) and z h to represent ( s h , a h ) when there is no confusion. Similar to the bandit case, we give some variables an extra subscript k here for clarity, which denotes the version of the variable before (not included) Line 14 at episode k .

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

The server maintains global historical datasets Z ser k,h containing sequences of tuples ( s h , a h , r h , s h +1 ) , initialized in Line 2. Each local agent m maintains optimistic value functions { Q m,k,h } H h =1 , bonus functions { b m,k,h } H h =1 , and local datasets { Z loc m,k,h } H h =1 , all initialized in Line 3.

Each episode k of Algorithm 2 also consists of the two parts local exploration and server updates. Part I: Local Exploration. At step k an agent m = m k is active (Line 5). It interacts with the environment by executing the greedy policy according to { Q m,k,h } H h =1 , obtaining a trajectory { ( s k h , a k h , r k h , s k h +1 ) } H h =1 , which is then stored into the local historical datasets Z loc m,k,h (lines 6 - 9). After exploration, the agent checks for the following switch condition: there exists h ∈ [ H ] so that

where k ′ is the last communication round for m . If so, the agent triggers communication (Line 11). Part II: Server Updates. After receiving new data, the server merges it with its global datasets Z ser k,h (Line 13) and calculates value function estimates { Q k +1 ,h } H h =1 and { V k +1 ,h } H h =1 using LSVI. Suppose we already have Q - and V -value function estimates Q k +1 ,h +1 and V k +1 ,h +1 at level h +1 . We solve the least squares problem for ̂ f h to minimize the Bellman error (Line 16):

We now also define the uncertainty estimator of a new pair of data z = ( s, a ) against data history Z with normalization parameter λ and function class F as

Similar to the bandits setting, the uncertainty can be approximated with the bonus function acquired from an oracle B S×A in Definition 4.1. In this case, the domain D = S × A , and the data format corresponds to z = ( s, a ) and e = ( r, s ′ ) . Despite these definitions not depending on the step h , we expect the parameters z, Z, F to always come from same step h . Finally, we allow the bonus function classes W h = W h, S×A to vary between different levels.

After calling oracle for b k +1 ,h (Line 16), we can obtain value function estimates (Line 17):

̂ a ∈A Iterating through h = H, · · · , 1 , the server calculates a set of updated Q -value functions { Q k +1 ,h } H h =1 and bonus functions { b k +1 ,h } H h =1 , and send them back to agent m for future exploration and updates (lines 19 and 21).

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

5.2 Theoretical Results

We summarize the regret and communication cost of Algorithm 2 in the following theorem:

where we abbreviate dim E := dim E ( F , λ/K ) ; the total communication complexity is

Remark 5.2 . This result when reduced to linear MDPs correspond well to Theorem 5.1 in Min et al. [2023]. Taking α = 1 /M 2 , we get a regret of ˜ O ( H 2 √ K dim E log N + H 2 dim E ) and a communication cost of ˜ O ( HM 2 dim E ) , where N = max h { N ( F h , γ ) , N ( W h , γ ) } .

6 Proof Sketch

In this section, we provide an outline for the proof of Theorem 5.1, while a more detailed proof can be found in Appendix B, and the full versions of the following lemmas are in Appendix B.1.

6.1 Regret Upper Bound

For the regret upper bound, the first lemma establishes optimism of value function estimates.

Lemma 6.1. Taking β k,h as in Theorem 5.1, with probability at least 1 -δ , for all k , z ∈ S × A and h ∈ [ H ] , |T h Q k +1 ,h +1 ( z ) -̂ f k +1 ,h ( z ) | ≤ b k +1 ,h ( z ) .

This allows us to decompose regret into a sum of bonuses:

The sum of bonuses is equal to the sum of uncertainty up to a constant, which we bound in the following lemma corresponding to the elliptical potential lemma [Abbasi-Yadkori et al., 2011].

Lemma 6.2. Define universal datasets as Z all = { o k } k ′ ∈ [ k ] . Then we have for any h ∈ [ H ] :

Careful examination exposes a problem: the uncertainty D λ, F ( z ; Z ser k,h ) corresponding to bonuses are based on server data Z ser k,h instead of universal data Z all k,h . The next lemma bridges this gap:

λ, F k,h λ, F k,h With these, we can deduce the regret bound from (12).

6.2 Communication Cost

For communication cost, we employ an epoch segmentation scheme, which defines N epochs segmented by episodes { k i } N i =1 , with k i being the smallest episode satisfying

This is a generalization of epoch segmentation based on doubling determinants in linear settings, yet the lack of determinant in the nonlinear case dramatically increases its complexity. Intuitively, switch condition (8) suggests an agent must gather a substantial amount of data to trigger communication, yet a careful analysis according to (13) yields a maximum of M + C/α communication rounds within one epoch. With this we only need an upper bound for the number of epochs N . This is derived by summing (13) over all epochs, then using Lemma 6.1 and Lemma 6.3 to bound the left hand side.

7 Conclusions

Wepropose the algorithms Async-NLin-UCB and Async-NLSVI-UCB to tackle multi-agent nonlinear contextual bandits and MDPs with asynchronous communication. We prove that our algorithms enjoy low regret and communication cost, which are comparable to previous results.

Our algorithms employ a communication criterion that allows the agents to trigger communication rounds, effectively controlling communication cost while promoting the asynchronous protocol.

Moreover, we carefully design the contents of server download to guard against data exposure.

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24:2312-2320, 2011.

Alekh Agarwal and Tong Zhang. Model-based RL with optimistic posterior sampling: Structural conditions and sample complexity. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Alekh Agarwal, Yujia Jin, and Tong Zhang. Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 987-1063. PMLR, 12-15 Jul 2023.

Ana LC Bazzan. Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multi-Agent Systems , 18(3):342-375, 2009.

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Mithun Chakraborty, Kee Yuan Peh Chua, Sanmay Das, and Brendan Juba. Coordinated versus decentralized exploration in multi-agent multi-armed bandits. In IJCAI , 2017.

Alfredo V Clemente, Humberto N Castejón, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. arXiv preprint arXiv:1705.04862 , 2017.

Qiwei Di, Heyang Zhao, Jiafan He, and Quanquan Gu. Pessimistic nonlinear least-squares value iteration for offline reinforcement learning. arXiv preprint arXiv:2310.01380 , 2023.

Guohui Ding, Joewie J Koh, Kelly Merckaert, Bram Vanderborght, Marco M Nicotra, Christoffer Heckman, Alessandro Roncone, and Lijun Chen. Distributed reinforcement learning for cooperative multi-robot object manipulation. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 1831-1833, 2020.

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in rl. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2826-2836. PMLR, 18-24 Jul 2021.

Abhimanyu Dubey and Alex Pentland. Provably efficient cooperative multi-agent reinforcement learning with function approximation. arXiv preprint arXiv:2103.04972 , 2021.

Abhimanyu Dubey and AlexSandy' Pentland. Differentially-private federated linear bandits. Advances in Neural Information Processing Systems , 33:6003-6014, 2020.

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning , pages 1407-1416. PMLR, 2018.

Dylan J. Foster, Sham M. Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making, 2023.

Claudio Gentile, Zhilei Wang, and Tong Zhang. Fast rates in pool-based batch active learning, 2022.

Jiafan He, Tianhao Wang, Yifei Min, and Quanquan Gu. A simple and provably efficient algorithm for asynchronous federated contextual linear bandits. In Advances in Neural Information Processing Systems , 2022.

Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representations , 2018.

Ruiquan Huang, Weiqiang Wu, Jing Yang, and Cong Shen. Federated linear contextual bandits. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E. Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1704-1713. PMLR, 06-11 Aug 2017.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR, 2020.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Federated reinforcement learning with environment heterogeneity. In International Conference on Artificial Intelligence and Statistics , pages 18-37. PMLR, 2022.

Dingwen Kong, Ruslan Salakhutdinov, Ruosong Wang, and Lin F. Yang. Online sub-sampling for reinforcement learning with general function approximation, 2023.

Nathan Korda, Balázs Szörényi, and Shuai Li. Distributed clustering of linear bandits in peer to peer networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48 , ICML'16, page 1301-1309. JMLR.org, 2016.

Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations , 2022.

Peter Landgren, Vaibhav Srivastava, and Naomi Ehrich Leonard. On distributed cooperative decision-making in multiarmed bandits. In 2016 European Control Conference (ECC) , pages 243-248, 2016.

Peter Landgren, Vaibhav Srivastava, and Naomi Ehrich Leonard. Social imitation in cooperative multiarmed bandits: Partition-based algorithms with strictly local information. In 2018 IEEE Conference on Decision and Control (CDC) , 2018.

Chuanhao Li and Hongning Wang. Asynchronous upper confidence bound algorithms for federated linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 6529-6553. PMLR, 2022.

Gene Li, Pritish Kamath, Dylan J Foster, and Nati Srebro. Understanding the eluder dimension. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 23737-23750. Curran Associates, Inc., 2022.

Boyi Liu, Lujia Wang, and Ming Liu. Lifelong federated reinforcement learning: a learning architecture for navigation in cloud robotic systems. IEEE Robotics and Automation Letters , 4(4):4555-4562, 2019.

Dianbo Liu, Vedant Shah, Oussama Boussif, Cristian Meo, Anirudh Goyal, Tianmin Shu, Michael Mozer, Nicolas Heess, and Yoshua Bengio. Stateful active facilitator: Coordination and environmental heterogeneity in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2210.03022 , 2022.

Dongfang Liu, Yiming Cui, Zhiwen Cao, and Yingjie Chen. Indoor navigation for mobile agents: A multimodal vision fusion model. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2020.

Keqin Liu and Qing Zhao. Distributed learning in multi-armed bandit with multiple players. IEEE Transactions on Signal Processing , 58(11):5667-5681, 2010. doi: 10.1109/TSP.2010.2062509.

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems , 30, 2017.

David Martínez-Rubio, Varun Kanade, and Patrick Rebeschini. Decentralized cooperative stochastic bandits. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems . Curran Associates, Inc., 2019.

Yifei Min, Tianhao Wang, Ruitu Xu, Zhaoran Wang, Michael Jordan, and Zhuoran Yang. Learn to match with no regret: Reinforcement learning in markov matching markets. In Advances in Neural Information Processing Systems , 2022.

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

Yifei Min, Jiafan He, Tianhao Wang, and Quanquan Gu. Cooperative multi-agent reinforcement learning: Asynchronous communication and linear function approximation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 24785-24811. PMLR, 23-29 Jul 2023.

Seongin Na, Tomáš Rouˇ cek, Jiˇ rí Ulrich, Jan Pikman, Tomáš Krajník, Barry Lennox, and Farshad Arvin. Federated reinforcement learning for collective navigation of robotic swarms, 2022.

Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296 , 2015.

Jiaju Qi, Qihao Zhou, Lei Lei, and Kan Zheng. Federated reinforcement learning: techniques, applications, and open challenges. arXiv preprint arXiv:2108.11887 , 2021.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Abishek Sankararaman, Ayalvadi Ganesh, and Sanjay Shakkottai. Social learning in multi agent multi armed bandits. Proc. ACM Meas. Anal. Comput. Syst. , 3(3), 2019.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Balazs Szorenyi, Robert Busa-Fekete, Istvan Hegedus, Robert Ormandi, Mark Jelasity, and Balazs Kegl. Gossipbased distributed stochastic bandit algorithms. In Proceedings of the 30th International Conference on Machine Learning , 2013.

Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 , 2017.

Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, and Mingyi Hong. Multi-agent reinforcement learning via double averaging primal-dual optimization. Advances in Neural Information Processing Systems , 31, 2018.

Martin Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . 02 2019. ISBN 9781108498029.

Po-An Wang, Alexandre Proutiere, Kaito Ariu, Yassir Jedra, and Alessio Russo. Optimal algorithms for multiplayer multi-armed bandits. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 4120-4129. PMLR, 26-28 Aug 2020a.

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6123-6135. Curran Associates, Inc., 2020b.

Yuanhao Wang, Jiachen Hu, Xiaoyu Chen, and Liwei Wang. Distributed bandit learning: Near-optimal regret with efficient communication. In International Conference on Learning Representations , 2020c.

Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference on Robotics and Automation (ICRA) , pages 1433-1440. IEEE, 2016.

Ruitu Xu, Yifei Min, Tianhao Wang, Michael I Jordan, Zhaoran Wang, and Zhuoran Yang. Finding regularized competitive equilibria of heterogeneous agent macroeconomic models via reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 375-407. PMLR, 2023.

Chenlu Ye, Wei Xiong, Quanquan Gu, and Tong Zhang. Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and Markov decision processes. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 39834-39863. PMLR, 23-29 Jul 2023.

Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems , 33:621-632, 2020.

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955 , 2021.

Shuai Yu, Xu Chen, Zhi Zhou, Xiaowen Gong, and Di Wu. When deep reinforcement learning meets federated learning: Intelligent multitimescale resource management for multiaccess edge computing in 5g ultradense network. IEEE Internet of Things Journal , 8(4):2238-2251, 2020.

Tao Yu, HZ Wang, Bin Zhou, Ka Wing Chan, and J Tang. Multi-agent correlated equilibrium q ( λ ) learning for coordinated smart generation control of interconnected power grids. IEEE transactions on power systems , 30 (4):1669-1679, 2014.

Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Networked multi-agent reinforcement learning in continuous spaces. In 2018 IEEE conference on decision and control (CDC) , pages 2771-2776. IEEE, 2018a.

Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning , pages 5872-5881. PMLR, 2018b.

Heyang Zhao, Jiafan He, and Quanquan Gu. A nearly optimal and low-switching algorithm for reinforcement learning with general function approximation, 2023.

Zhaowei Zhu, Jingxuan Zhu, Ji Liu, and Yang Liu. Federated bandit: A gossiping approach. Proc. ACM Meas. Anal. Comput. Syst. , 5(1), 2021.

Hankz Hankui Zhuo, Wenfeng Feng, Yufeng Lin, Qian Xu, and Qiang Yang. Federated deep reinforcement learning. arXiv preprint arXiv:1901.08277 , 2019.

517

Impact Statement

Our work has the potential to enhance cooperative learning systems across diverse fields. By 518 introducing algorithms that enable efficient collaboration among agents with minimal communication 519 overhead, our research paves the way for advancements in distributed systems, including robotics, 520 traffic management, and distributed sensor networks. This could lead to more adaptive, efficient, 521 and scalable systems capable of tackling complex problems in dynamic environments, ultimately 522 contributing to technological progress and societal well-being. 523

As far as we can tell, there is hardly any negative social impact from our work, mainly because we do 524 not include experiments apart from our theoretical analysis. 525

A The Bandit Case: Proof of Theorem 4.3 526

Before we begin the analysis of Algorithm 1, we reiterate and add some notations for clarity and convenience. Define the data collected by agent m that has already been uploaded to the server by round t as Z up m,t , and the universal data at round t as Z all t . Apart from these we also have from the algorithm the datasets Z loc m,t and Z ser t . It is not difficult to check that they satisfy the following relation:

Furthermore, when t is not a communication round, we also have

and when it is a communication round that

which will be useful in our proof of Lemma A.1 and B.1 in Section C.1.

Next, we assume that at rounds 0 = t 0 < t 1 < · · · < t L < t L +1 = T +1 , the participating agent communicates with the server, where t 0 and t K +1 are dummy rounds. The subscripts will be denoted as l = 1 , · · · , L in the future.

Wenowdescribe a participant reordering trick for our asynchronous multi-agent setting, which we will use multiple times in the proof. The basic idea is that, as long as the communication order remains the same, and for any given agent, the number of rounds between two consecutive communication rounds remains the same, one can switch the episodes around and change the order of agent participation to a certain degree. For example, we may assume that m t = m t l for all t ∈ ( t l -1 , t l ] by reordering the participants, which means all participation of any given agent happens immediately before a certain communication round; as another example, we may assume m t = m t l -1 for all t ∈ [ t l -1 , t l ) , which means all participation happen immediately after communication rounds. It should be noted that one needs to be careful when utilizing this argument, since switching the participation order changes the values of t l and many associated elements, so applying this trick twice in succession would lead to contradictions.

For a dataset Z , we define the Z -norm on function set F as ∥ f ∥ 2 Z := ∑ ( a,r ) ∈ Z f 2 ( a ) for any f ∈ F . Then we have the shortened notation

Finally, we define the confidence set of functions at round t +1 as: 542

which is a common construction in reinforcement learning.

A.1 Auxiliary Lemmas

In this section we present some auxiliary lemmas that will be used in the proof of Theorem 4.3. Note these lemmas correspond well to the lemmas presented in 6, only that these are for the contextual bandit case. The proofs for these lemmas can be found in Section C.

Lemma A.1. For any t ∈ [ T ] , m ∈ [ M ] and f 1 , f 2 ∈ F , as long as agent m does not communicate with the server at time step t , we have

Furthermore, for any t ∈ [ T ] and f 1 , f 2 ∈ F ,

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

543

544

545

546

547

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

and as a corollary, for any a ∈ A ,

This lemma describes the discrepancy between different datasets. Crucially, it provides a worst case 548 ratio between uncertainty measured on the server dataset and universal dataset. This is an important 549 tool for bridging between the different uncertainty estimators in the following proofs. The proof can 550 be found in Section C.1. 551

Lemma A.2. By taking γ = O (1 /T ) and

with C β, 1 = 6 , where C ( M,α ) := √ 1 + Mα + M √ α , we have f ∗ ∈ F t +1 for all t ∈ { t l } L l =1 with probability at least 1 -δ . As a corollary, we also have | f ∗ ( a ) -̂ f t +1 ( a ) | ≤ b t +1 ( a ) for any a ∈ A t and t ∈ { t l } L l =1 .

This is the central optimism lemma present in all provably efficient reinforcement learning literature. It states that the confidence function set contains the ground truth function f ∗ with high probability, and in our case, that the decision function ̂ f t + b t is optimistic. With this, we define the good event E T = { f ∗ ∈ F t +1 , ∀ t ∈ { t l } L l =1 } . Then according to A.2, P ( E T ) ≥ 1 -δ . The proof can be found in Section C.2.

Lemma A.3. The sum of squared uncertainty estimators of new data over all historical data can be bounded as follows with some absolute constant C D :

This lemma corresponds to the elliptical potential argument from the linear setting [Abbasi-Yadkori et al., 2011]. In the nonlinear setting, this lemma essentially reveals the relationship between the sum of Eluder-like confidence quantities and the Eluder dimension. The proof can be found in Section C.3.

A.2 The Epoch Segmentation Scheme

In this section, we introduce an epoch segmentation scheme, which is needed for both the regret and communication cost proofs presented in the next two sections. It is a generalization of the epoch segmentation scheme based on doubling determinant in the linear bandits / MDPs setting [He et al., 2022, Min et al., 2023], but the lack of a Gram matrix (used for linear regression) in the nonlinear case complicates matters significantly.

We segment the entire run of t = 1 , · · · , T into N epochs as follows. Define iteratively 0 = l 0 < l 1 < · · · < l N ≤ L as

where for a given l in the summation, m = m t l ′ is the participating agent at t l ′ . In the iterative 570 process, if the above minimum does not exist, simply define N = i -1 and end the process there. 571 Correspondingly, the i -th epoch is defined by the time steps [ t , t ) . 572

The following sections will make use of this epoch scheme as befit their needs, but here we shall give 573 an upper bound for the total number of epochs N . Based on the definition of l , we have for any 574

′ l i -1 l i i

l i -1 ≤ l < l i that 575

for any a , and so 577

which gives λ + ∥ f 1 -f 2 ∥ 2 Z ser t l ≤ 2 ( λ + ∥ f 1 -f 2 ∥ 2 Z ser t l i -1 ) for any f 1 , f 2 ∈ F . Then we have 576

and summing over i = 1 , · · · , N -1 that: 578

If we apply the participant reordering trick and let m t = m t l for all t ∈ ( t l -1 , t l ] and l ∈ [ L ] , we get 579 Z ser t l \ Z ser t l -1 = { ( a t , r t ) } t l t = t l -1 +1 , and so applying Lemma A.1 and Lemma A.3, we get 580

which gives the order of total number of epochs: 581

Notice that the participant reordering trick is only used to bound the number of epochs, which itself 582 does not depend on the specific order of participation. This is crucial since it suggests this reordering 583 does not change anything essential, and is in fact not necessary for the proof - it just made the proof 584 easier to read. Therefore we can still reorder participants as we see fit in other parts of our proof. 585

586

587

588

589

590

591

592

593

A.3 Proof of Regret Upper Bound

Now we are ready to prove the first part of Theorem 4.3 concerning the regret upper bound. We begin by applying the participation reordering trick to assume, without loss of generality, that the same agent is active within the rounds [ t l , t l +1 -1] , i.e. m t l = m t l +1 = · · · = m t l +1 -1 . Under this assumption, we have t 1 = 1 .

Let a ∗ t := argmax a ∈ D t f ∗ ( a ) be the best arm at time t . Then by Lemma A.2, f ∗ ( a ∗ t ) ≤ ( ̂ f m t ,t + b m t ,t ) ( a ∗ t ) ≤ ( ̂ f m t ,t + b m t ,t ) ( a t ) , where the second inequality is due to the choice of a t at round t . Hence we get

where the first inequality is due to | f | ≤ 1 from Assumption 3.1, and the second inequality again 594 uses Lemma A.2. We first bound the second term here using the epoch scheme in Section A.2. We 595 start by converting the bonus term to uncertainty: 596

Now consider the episodes in an epoch i , specifically { t l i -1 , t l i -1 +1 , · · · , t l i } . For any l i -1 < l < l i , 597 since Z ser t l i -1 ⊆ Z ser t l -1 , we can deduce that 598

where the second inequality is borrowed from (15) from Section A.2. Therefore continuing from 599 (18), 600

Now combine this result with the first term in (17) and use again (18), we get 601

According to Lemma A.1 and Lemma A.3, the term 602

combining this with (16), we get 603

604

605

606

607

A.4 Proof of Communication Cost

In this section we prove the second part of Theorem 4.3, by calculating the communication complexity. First, for each communication round t l , assume the last time before t l when the agent m t l communicated with the server was t l ′ , then

Now employing the epoch segmentation scheme from section A.2, for the i -th epoch consisting of 608 the time steps [ t l i -1 , t l i ) , we have the inequality 609

For m ∈ [ M ] , assume the agent m communicated with the server a total of n m times within [ t l i -1 , t l i ) . Then except for the first of these communication rounds, for each l ∈ [ l i -1 +1 , l i -1] with m t l = m , there exists l ′ ∈ [ l i -1 , l ) with m t l ′ = m , thus we have Z up m,t l ⊃ Z up m,t l ′ +1 = Z ser t l ′ ⊃ Z ser t l i -1 . With this we have the corresponding term

therefore 610

Notice that ∑ M m =1 n m = l i -l i -1 is the number of communication rounds within [ t l i -1 , t l i ) , hence summing over i the total number of communication rounds is upper bounded by N ( M + C 2 B /α ) . Combine this with (16), we have the total number of communication rounds throughout the algorithm is

611

612

613

614

615

B The MDPs Case: Proof of Theorem 5.1

Similar to the bandit case, we define Z loc m,k,h , Z up m,k,h , Z ser k,h , and Z all k,h to be the local, uploaded, server and universal data, with corresponding subscripts of agent m ∈ [ M ] , episode k ∈ [ K ] , h ∈ [ H ] .

Suppose at rounds 0 = k 0 < k 1 < · · · < k L < k L +1 = T +1 , the participating agent communicates with the server, where k 0 and k L +1 are dummy rounds.

For a dataset Z h in the MDPs setting, we again define the Z h -norm on function set F h as ∥ f ∥ 2 Z := ∑ o h ∈ Z f 2 ( z h ) for any f ∈ F . As a reminder, the tuples o h = ( s h , a h , r h , s h +1 ) and z h = ( s h , a h ) . Then we have the shortened notation

Finally, we define the confidence set of functions at round k +1 and step h as: 616

617

B.1 Auxiliary Lemmas

In this section we present some auxiliary lemmas that will be used in the proof of Theorem 5.1. These 618 lemmas are generalizations / restatements to the lemmas presented in 6, and their detailed proofs can 619 be found in Section C. 620

Lemma B.1 (Restatement of Lemma 6.3) . For any k ∈ [ K ] , m ∈ [ M ] , h ∈ [ H ] and f 1 , f 2 ∈ F , as long as agent m does not communicate with the server at episode k , we have

Furthermore, we have for any k ∈ [ K ] and f 1 , f 2 ∈ F ,

and as a corollary, for any z = ( s, a ) ∈ S × A ,

Similar to Lemma A.1, this lemma provides a worst case ratio between uncertainty measured on the 621 server dataset and universal dataset. The proof can be found in Section C.1. 622

623

624

625

626

627

628

629

630

631

632

633

Lemma B.2 (Restatement of Lemma 6.1) . By taking γ = 1 / ( C γ KH ) with C γ ≥ 20 , as well as

with C β, 2 = 12 for all k ∈ [ K ] and h ∈ [ H ] , where N h ( γ ) = N ( F h , γ ) · N ( F h +1 , γ ) · N ( W h +1 , γ ) , we have with probability at least 1 -δ that T h Q k +1 ,h +1 ∈ F k +1 ,h for all k ∈ { k l } L l =1 with probability at least 1 -δ . As a corollary, we also have |T h Q k +1 ,h +1 ( s, a ) -̂ f k +1 ,h ( s, a ) | ≤ b k +1 ,h ( s, a ) for any ( s, a ) ∈ S × A , k ∈ { k l } L l =1 and h ∈ [ H ] .

This is the central optimism lemma. It states that the Bellman operator of Q -value function at level h + 1 is within the confidences set at level h . The conclusion immediately gives the optimism inequality T h Q k +1 ,h +1 ( s, a ) ≤ Q k +1 ,h ( s, a ) , which we will use at the start of the regret upper bound prove. The proof of the lemma can be found in Section C.2.

With this, we define the good event E T = {T h Q k +1 ,h +1 ∈ F k +1 ,h , ∀ k ∈ { k l } L l =1 , h ∈ [ H ] } . Then according to Lemma B.2, P ( E T ) ≥ 1 -δ .

Lemma B.3. For some absolute constant C D , the following holds for all level h ∈ [ H ] :

This lemma is essentially the same as Lemma A.3. It reveals the relationship between the sum of 634 Eluder-like confidence quantities and the Eluder dimension. The proof can be found in Section C.3. 635

B.2 The Epoch Segmentation Scheme 636

In this section, we introduce the epoch segmentation scheme for MDPs, which is again needed for 637 both the regret and communication cost proofs presented in the next two sections. All of this is 638 quite similar to the bandit case in Section A.2, but the introduction of multiple levels h ∈ [ H ] does 639 complicate things a bit. 640

We segment the entire run of episodes k = 1 , · · · , K into N epochs as follows. Define iteratively 0 = l 0 < l 1 < · · · < l N ≤ L as

where for a given l ′ in the summation, m = m k l ′ is the participating agent at k l ′ . In the iterative 641 process, if the above minimum does not exist, simply define N = i -1 and end the process there. 642 Correspondingly, the i -th epoch is defined by the episodes [ k l i -1 , k l i ) . 643

The following sections will make use of this epoch scheme as befit their needs, but here we shall give 644 an upper bound for the total number of epochs N . Based on the definition of l i , we have for any 645 l i -1 ≤ l < l i that 646

which gives λ + ∥ f 1 -f 2 ∥ 2 Z ser k l ,h ≤ 2 ( λ + ∥ f 1 -f 2 ∥ 2 Z ser k l i -1 ,h ) for any h ∈ [ H ] and f 1 , f 2 ∈ F h . Then 647 we have 648

for any h ∈ [ H ] and z h ∈ S × A , and so 649

and summing over i = 1 , · · · , N -1 that: 650

If we apply the participant reordering trick and let m k = m k l for all k ∈ ( k l -1 , k l ] and l ∈ [ L ] , we 651 get Z ser k l ,h \ Z ser k l -1 ,h = { o k h } k l k = k l -1 +1 , and so applying Lemma 6.3 and Lemma 6.2, we get 652

which gives the order of total number of epochs: 653

654

655

656

657

658

659

B.3 Proof of Regret Upper Bound

In this section, we prove the first half of Theorem 5.1, which gives an upper bound for the cumulative regret of Algorithm 2.

Using the participant reordering trick, assume without loss of generality that the same agent is active within the rounds [ k l , k l +1 -1] , i.e. m k l = m k l +1 = · · · = m k l +1 -1 . Under this assumption, we have k 1 = 1 .

.

Wefirst prove via induction that Q ∗ h ≤ Q m,k,h for any m ∈ [ M ] , k ∈ [ K ] and h ∈ [ H +1] . This holds true for h = H +1 trivially since both value functions at H +1 are uniformly 0 . Suppose we already have Q ∗ h +1 ≤ Q m,k,h +1 , we have from Lemma B.2 that for the last communication round k ′ for agent m , the server functions satisfy T h Q k ′ +1 ,h +1 ( s, a ) ≤ ̂ f k ′ +1 ,h ( s, a ) + b k ′ +1 ,h ( s, a ) = Q k ′ +1 ,h ( s, a ) Couple this with the fact that Q m,k,h = Q k ′ +1 ,h , we can prove that

which finishes the induction process. 660

Now let a k ∗ h := argmax a ∈A Q ∗ h ( s k h , a ) be the best action at time t , then V ∗ h ( s k h ) = Q ∗ h ( s k h , a k ∗ h ) ≤ 661 Q m,k,h ( s k h , a k ∗ h ) ≤ Q m,k,h ( s k h , a k h ) , where the second inequality is due to the choice of a k h at round 662 k . Hence we get 663

where the second equality uses the Value-decomposition Lemma from Jiang et al. [2017], the second 664 inequality uses again Lemma B.2, and from the third inequality onward we let k ′ be the last time 665 agent m communicated with the server. 666

We now bound the second term here using the epoch scheme in Section B.2. We start by converting 667 the bonus term to uncertainty: 668

Nowconsider the episodes in an epoch i , specifically { k l i -1 , k l i -1 +1 , · · · , k l i } . For any l i -1 < l < l i , 669 since Z ser k l i -1 ,h ⊆ Z ser k l -1 ,h , we can deduce that 670

where the second inequality is borrowed from (21) from Section B.2. Therefore continuing from 671 (24), 672

Now combine this result with the first term in (23) and use again (24), we get 673

According to Lemma 6.3 and Lemma 6.2, the term 674

Now with γ = O (1 /KH ) , we have 675

therefore, with C ( M,α ) = 1 + Mα + M α and the upper bound for number of epochs N in 676 (22), we have 677

678

679

680

681

682

683

684

where ˜ β 2 = C β, 2 [ √ λ + HC ( M,α ) log ( HMN ( F , γ ) N ( W , γ ) /δ ) ] is the choice of β k,h in the algorithm.

B.4 Proof of Communication Cost

Next up, we calculate the communication complexity of Algorithm 2 and prove the second half of Theorem 5.1. For each communication round k l , assume the last time before k l when the agent m = m k l communicated with the server was k l ′ , then by the communication rule there exists h l ∈ [ H ] such that ∑ o h l ∈ Z loc m,k l ,h l b 2 k l ′ ,h l ( z h l ) / ( β 2 k l ′ ,h l + λ ) ≥ α ,

Next we will make use of the epoch segmentation scheme in Section B.2. For the i -th epoch consisting 685 of the time steps [ k l i -1 , k l i ) , we have the inequality 686

For m ∈ [ M ] , assume the agent m communicated with the server a total of n m times within [ k l i -1 , k l i ) . Then except for the first of these communication rounds, for each l ∈ [ l i -1 +1 , l i -1] with m k l = m , there exists l ′ ∈ [ l i -1 , l ) with m k l ′ = m , thus we have Z up m,k l ,h ⊃ Z up m,k l ′ +1 ,h = Z ser k l ′ ,h ⊃ Z ser k l i -1 ,h for all h ∈ [ H ] . With this we have

therefore 687

688

689

690

691

692

693

694

695

696

Notice that ∑ M m =1 n m = l i -l i -1 is the number of communication rounds within [ k l i -1 , k l i ) , hence summing over i the total number of communication rounds is upper bounded by N ( M +4 C 2 /α ) . Combine this with the result in (22), we have the total number of communication rounds throughout the algorithm is

C Proof of Auxiliary Lemmas

In this section we prove all the auxiliary lemmas in Section A.1 and Section B. 1 . Note that some of these lemmas are very similar in nature, for which we will only give the proof for the version for the MDPs case, and briefly remark on the version for the bandit case.

C.1 Proof of Lemma A.1 and Lemma B.1

Here we prove Lemma B.1 in detail. The proof for Lemma A.1 is very similar, and so we will only give a short remark on how to apply this to the bandit case.

Proof of Lemma B.1. First, for an episode k ∈ [ K ] and agent m ∈ [ M ] such that m does not communicate with the server at episode k (either m is not participating or k is not a communication

round), from the communication criterion we have 697

where k ′ is the last communication round for agent m . This means that for any f 1 , f 2 ∈ F h , (1 /α ) ∥ f 1 -f 2 ∥ 2 Z loc m,k,h ≤ λ + ∥ f 1 -f 2 ∥ 2 Z ser k ′ ,h . Observing that Z ser k ′ ,h ⊂ Z ser k,h = ⋃ M m ′ =1 Z up m ′ ,k,h proves the first conclusion that

Second, for any f 1 , f 2 ∈ F h , from the above conclusion we have for any k ∈ [ K ] \{ k l } L l =1 that 698

and when k = k l for some l ∈ [ L ] , we have alternatively 699

̸

̸

Either way, we can deduce for any k ∈ [ K ] that

Finally, from the above we immediately have 700

701

702

703

704

705

706

707

Remark C.1 . Notice that this prove does not depend on the multi-level structure of episodic MDPs, but is a direct result of the communication criterion and protocol. This means the proof can be converted to the bandit case of Lemma A.1 without any essential changes: simply change episode k into time step t , disregard all mentions of level h , and consider z = a instead of z = ( s, a ) .

C.2 Proof of Lemma A.2 and Lemma B.2

We begin with the proof of Lemma A.2, which is an almost direct application of Lemma D.3.

Proof of Lemma A.2. We invoke Lemma D.3 with ϵ 0 = 0 , then with probability at least 1 -δ , for all t ∈ { t l } L l =1 ,

if we let γ = O (1 /T ) be sufficiently small and take ˜ β 1 = C β, 1 [ √ λ + 708 RC ( M,α ) log(3 MN ( F , γ ) /δ ) ] with C β, 1 = √ C ERM = 6 . Thus taking β t = ˜ β 1 , accord709 ing to the definition of F t +1 , this directly implies f ∗ ∈ F t +1 . 710 With this, since the bonus function satisfy

711

712

which is based on the first property of the bonus oracle in Definition 4.1, by taking f 1 = ̂ f t +1 and f 2 = f ∗ we get for any a ∈ A that b t +1 ( a ) ≥ | f ∗ ( a ) -̂ f t +1 ( a ) | , which finishes the proof.

Next we prove Lemma B.2, which is more challenging and requires an analysis on the least squares 713 value iteration method. 714

Proof of Lemma B.2. Take F h +1 ,γ as a γ -cover of F h +1 , and W h +1 ,γ as a γ -cover of W h +1 . Select 715 ¯ f k +1 ,h +1 ∈ F h +1 ,γ ⊕ W h +1 ,γ so that ∥ Q k +1 ,h +1 -¯ f k +1 ,h +1 ∥ ∞ ≤ ¯ ϵ := (1 + β k +1 ,h +1 ) γ . For 716 o h = ( s h , a h , r h , s h +1 ) , define the corresponding y h = r h + V k +1 ,h +1 ( s h +1 ) and ¯ y h = r h + 717 sup a ∈A ¯ f k +1 ,h +1 ( s h +1 , a ) . Let 718

Then we have 719

Now notice that E ¯ y h = T h ¯ f k +1 ,h ( s h , a h ) , and the difference ¯ y h - T h ¯ f k +1 ,h ( s h , a h ) is bounded 720 in [ -H,H ] , hence we may apply Lemma D.3 with f ∗ = T h ¯ f k +1 ,h , r t = ¯ y h , R = H , ϵ 0 = 2¯ ϵ 721 and δ = δ/ 3 HN ( F h +1 , γ ) · N ( W h +1 , γ ) , taking a union bound over ¯ f ∈ F h +1 ,γ ⊕ W h +1 ,γ and 722

h ∈ [ H ] , we have 723

where N h ( γ ) = N ( F h , γ ) · N ( F h +1 , γ ) · N ( W h +1 , γ ) . By taking γ = 1 / ( C γ KH ) with sufficiently large absolute constant C γ (for example, C γ = 20 ), the second and third terms within the bracket above are both less than (1 / 2) β k +1 ,h +1 , and hence we can easily prove via induction on h that the above is no greater than ˜ β 2 , where

C.3 Proof of Lemma A.3 and Lemma B.3

In this section we prove Lemma B.3 in detail. The proof for Lemma A.3 is very similar, and so we will again only give a short remark on how to apply this to the bandit case.

Proof of Lemma B.3. We fix the level h ∈ [ H ] throughout the proof. For an index set K 0 ⊆ [ K ] , we denote Z ( K 0 ) := { z k h : k ∈ K 0 } .

First, let n = ⌈ log( K/λ ) / log 2 ⌉ , and we divide the set of episodes K = [ K ] into n + 1 disjoint episode sets as follows. For any 1 ≤ l ≤ L and k l ≤ k < k l +1 , let

and define L k : S × A → R as L k ( z ) = ( ¯ f k, 1 ( z ) -¯ f k, 2 ( z ) ) 2 . Now we define K ι := { k ∈ K : L k ( z k h ) ∈ (2 -ι -1 , 2 -ι ] } for ι ∈ { 0 , 1 , · · · , n -1 } and K n := { k ∈ K : L k ( z k h ) ∈ [0 , 2 -n ] } . We note that for k ∈ K n , L k ( z k h ) ≤ λ/K .

724

725

726

727

728

729

730

731

732

733

Now define the mapping τ : [ K ] → [ K ] , such that for any k ∈ [ K ] , τ ( k ) is the last episode when 734 agent m k communicated with the server (not including k ). We will bound ∑ k ∈K ι D 2 λ, F h ( z k h ; Z all h,k -1 ) 735 for ι ∈ { 0 , · · · , n -1 } . 736

For a fixed ι ≤ n -1 , we now decompose K ι = ⋃ n ι +1 j =1 K ι j , where n ι = ⌈ |K ι | / dim E ( F h , 2 -ι -1 ) ⌉ . 737 We start off each set K ι j = ∅ , and fill them up gradually by iterating through k ∈ K ι one by one 738 in increasing order to decide which subset K ι j should k belong to. Specifically, we define j ( k ) to 739 be the smallest index j < n ι such that is z k h is 2 -( ι +1) / 2 -independent of Z ( K ι j ) , and assign k to 740 the set K ι j ( k ) . If such a j does not exist, we simply let j ( k ) = n ι +1 assign k to K ι n ι +1 . Finally 741 after the assignment process, we define K ι j,k = K ι j ∩ [ k ] for any k ∈ [ K ] . Then we have the 742 elements added into K ι j ( k ) -1 ,k form a sequence where each data corresponding to a new member 743 is 2 -( ι +1) / 2 -independent of the old members, and so there are no more than dim E ( F h , 2 -ι -1 ) 744 members within each of them. Moreover, for all k ∈ K ι that z k h is 2 -( ι +1) / 2 -dependent on each of 745 Z ( K ι 1 ,k ) , · · · , Z ( K ι j ( k ) -1 ,k ) . 746

Nowfor any k ∈ K ι by the definition of K ι , we have ( ¯ f k, 1 ( z k h ) -¯ f k, 2 ( z k h ) ) 2 ≥ 2 -ι -1 . This combined with the 2 -ι -1 -dependencies imply that for each j ′ = 1 , · · · , j ( k ) -1 , ∥ ¯ f k, 1 -¯ f k, 2 ∥ 2 Z ( K ι j ′ ,k ) ≥ 2 -ι -1 .

Notice that Z ( K ι j ′ ,k ) ⊂ Z all h,k -1 for any j ′ ∈ [ j ( k ) -1] , and that Z ( K ι j ′ ,k ) for j ′ ∈ [ j ( k ) -1] are disjoint, therefore

It follows that 747

where the first inequality uses the definition of K ι . Summing over k ∈ K ι , we have 748

where we used the relation ∣ ∣ K ι j ∣ ∣ ≤ dim E ( F h , 2 -ι -1 ) and the definition of n ι in the second inequality. 749 Additionally, for ι = n we also have 750

and so finally we sum over ι = 0 , · · · , n to get 751

752

753

754

755

756

757

758

759

760

761

762

where the final step makes the assumption that λ = O (1 / log K ) , in which case it holds with some absolute constant C D .

Remark C.2 . Again, this prove does not depend on the multi-level structure of episodic MDPs. In fact, it only relies on the Eluder dimensionality of F h . This means the proof can be converted to the bandit case of Lemma A.3 without any essential changes: simply change episode k into time step t , disregard all mentions of level h , and consider z = a instead of z = ( s, a ) .

D Technical Lemmas

In this section, we provide a technical concentration lemma that serves as the core of our results. For one, this lemma is based on the following concentration inequality:

Lemma D.1. For a sequence of random variables { Z t } t ∈ N adapted to the filtration {S t } t ∈ N and function f ∈ F , for any λ > 0 , with probability at least 1 -δ , for all t ∈ N , we have

The proof for this lemma can be found under Lemma 4 of Russo and Van Roy [2013]. Apart from 763 this, we need yet another basic concentration lemma: 764

Lemma D.2. Suppose { η t } T t =1 is a sequence of conditional R -sub-Gaussian random variables satisfying E [ e µη t ∣ ∣ H t -1 ] ≤ exp ( R 2 µ 2 / 2 ) , where H t -1 denotes all history before time t , with probability 1 -δ , we have

A proof of this lemma can be found under Lemma G.2 of Ye et al. [2023]. With this, we can prove 765 the following lemma characterizing the accuracy of least squares solution. Even though we need 766 this lemma for both bandit and RL settings, we will follow the notations presented in multi-agent 767 contextual bandits. Detailed explanation of how this translates to multi-agent MDPs can be found in 768 Section C.2. 769

LemmaD.3. Suppose we have a sequence of inputs { ( a t , r t ) } T t =1 that follow the rule r t = f ∗ ( a t )+ η t for some ground truth f ∗ ∈ F , with η t being conditionally R -sub-Gaussian:

We also have server datasets Z ser t at different time steps, collected following the communication 770 protocol in our settings. Note that strictly speaking, the conditions under which η t is sub-Gaussian 771 should also include the former participants m 1: t , but we will omit this dependency for convenience. 772 Consider ̂ f ser t +1 , the approximate ERM solution to the least squares problem: 773

Then abbreviating N = N ( F , γ ) and taking C ERM = 36 , with probability at least 1 -δ , 774

Proof of Lemma D.3. Let F γ be a γ -cover of the function class F with respect to the infinity norm ∥ · ∥ ∞ . For f ∈ F and ( a t , r t ) for some t ∈ [ T ] , let

Since r t = f ∗ ( a t ) + η t , we can write ϕ ( f, a t , r t ) as 775

Since η t is R -sub-Gaussian conditional on Z all t -1 , a t , we have for any positive parameter µ that 776

Using Lemma D.1, we have with probability at least 1 -δ/ 3 , for all f ∈ F γ and t ∈ [ T ] , 777

where µ all > 0 is a parameter we will determine later. 778

On the other hand, if we consider any local agent m , when m t = m , we have η t is R -sub-Gaussian 779 conditional on Z up m,t -1 ∪ Z loc m,t -1 and a t , i.e. all the data agent m has received from the environment 780 up to this point. Thus we have for any µ > 0 that 781

Then again using Lemma D.1 and taking summation on Z loc m,t , with probability at least 1 -δ/ 3 , the 782 following holds for any m ∈ [ M ] : 783

where µ loc > 0 is a parameter we will determine later. 784

Taking the summation of (27) for all m ∈ [ M ] and combining (26), while observing that Z ser t = 785 Z all t ∖⋃ M m =1 Z loc m,t , we get 786

From Lemma A.1, we have λ + ∥ f -f ∗ ∥ 2 Z all t ≤ (1+ Mα ) ( λ + ∥ f -f ∗ ∥ 2 Z ser t ) ⇔∥ f -f ∗ ∥ 2 Z all t ≤ Mαλ + 787 (1 + Mα ) ∥ f -f ∗ ∥ 2 Z ser t . Plugging this inequality into the above and letting µ all = 1 / 8 R 2 (1 + Mα ) 788 and µ loc = 1 / 8 R 2 Mα , we get 789

Now for ̂ f ser t +1 , there exists ˜ f ∈ F γ such that ∥ ˜ f -̂ f ser t +1 ∥ ∞ ≤ γ . Using Lemma D.2, this gives us the 790 following with probability at least 1 -δ/ 3 : 791

where we used the basic inequality √∑ ( a + b ) 2 ≤ √∑ a 2 + √∑ b 2 in the first inequality and 792 used the property of ̂ f ser t +1 in the second inequality. Finally, taking a union bound and combining this 793

with (28), we have with probability at least 1 -δ , 794

where the first inequality uses again ∥ ˜ f -̂ f ser t +1 ∥ ∞ ≤ γ , and it can be verified that the last inequality 795 holds when C ERM ≥ 36 . 796

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We accurately summarize our contribution and main results in the abstract, and elaborate further on motivation and main techniques in our introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: All assumptions made for our theoretical analysis are present and stated clearly within the main paragraphs of our paper. Discussions on the applicability of these assumptions are also included.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are listed in the main paper, while a very detailed and sound proof is displayed in the appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Our theoretical paper does not present any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Our theoretical paper does not present any experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Our theoretical paper does not present any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our theoretical paper does not present any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our theoretical paper does not present any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have thoroughly reviewed the NeurIPS Code of Ethics, and believe our work conforms fully the the Code.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Impact Statement before appendices

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our theoretical paper does not present any experimental results, and thus does not feature such data and models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our paper does not include any such assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our theoretical paper does not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our theoretical paper does not present any experimental results.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our theoretical paper does not present any experimental results.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
In this paper, the authors study multi-agent reinforcement learning where agents cooperate through asynchronous communications with a central server to learn a shared environment. They consider the following two settings: multi-agent contextual bandits with general function approximation, and multi-agent RL with general function approximation. For both settings, they propose provably efficient algorithms with low regret and low communication complexity.

STRENGTHS:
1. The problem of asynchronous MARL with general function approximation is interesting and important.

2. This paper is the first to consider the setting with general function approximation. The results are solid and the proof looks good to me.

3. For both settings, the authors propose provably efficient algorithms. The results generalize previous results under the linear setting.

WEAKNESSES:
1. It seems that part of the techniques is from previous results, such as the bonus function oracle. It will be helpful if there is a section discussing technical novelty.

2. It seems that the setting is closely related to low switching RL and RL with delayed feedback. It will be interesting if the authors could briefly discuss about the connections.

3. For the communication complexity bound in theorem 5.1, should it be $/\alpha$ instead of $\alpha$? In addition, why not choose $\alpha=1/M$ in both theorems? In this way, the communication cost can be improved. (Please correct me if I misunderstood anything) 

4. Line ?? in line 214 of page 6. Please correct it.

QUESTIONS:
Please refer to the weaknesses above.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose two algorithms for asynchronous communication in multi-agent reinforcement learning with generalized value function approximation: Asynchronous-NLin-UCB for context bandit scenarios and Asynchronous-NLSVI-UCB for episodic MDP scenarios. These algorithms achieve near-optimal regret with low communication complexity. The authors theoretically show the trade-off between regret and communication complexity.\n\nSTRENGTHS:\n- The authors provide a detailed background on the related literature concerning regret, communication complexity, and the presence of asynchronous update, which is greatly helpful in understanding the contributions of the proposed algorithms. \n- The theoretical foundations and proofs regarding the communication criterion are important and interesting. Also, the trade-off between regret and communication complexity via parameter $\\alpha$ offers valuable insights. \n- The approach of receiving decisions and bonus functions from a central server instead of historical data is intuitive and appears crucial from a privacy perspective.\n\nWEAKNESSES:\nWhile I have studied general value function approximation, I do not have research background in this field for multi-agent scenarios. Therefore, my critique may not have captured the weaknesses of this paper. \nI’m open to revising my score based on the authors' responses.\n\nAs far as I know, MARL often adopts the Centralized Training Decentralized Execution(CTDE) framework to avoid the action space growing exponentially with the number of users. However, it is unclear whether the proposed scenario follows \"decentralized execution\". Agents are supposed to execute based on partial observations in a decentralized manner, but the proposed approach appears to involve a central server consistently during execution. If the proposed scenario is inconsistent with CTDE, I would be interested to hear from the authors what the distinct advantages or necessity of this scenario is.\n\nTypos:\n\n- Line 214: Reference to the label is not correctly written.\n- Theorems 4.3 and 5.1: $\\tilde{\\beta}$ is not properly defined in the statement, and $\\beta_t$ should be fixed to $\\tilde{\\beta}$.\n- Theorem 5.1: Total communication complexity should be fixed to $O((1+M\\alpha)^2 / \\alpha)$.\n\nQUESTIONS:\nA simple question: Why is the order of the Eluder dimension of regret reported as $O(\\sqrt{\\text{dim}_E})$ instead of $O(\\text{dim}_E)$? The regret in Line 677 shows similar results to numerous other papers, but is written in $\\sqrt{O(\\text{dim}_E)}$ despite the fact that $O(\\text{dim}_E)$ dominates. For the linear MDP case, [1] presents a lower bound of $\\Omega(dH\\sqrt{T})$, and considering $\\text{dim}_E = \\tilde{O}(d)$, the reported results seem lower than the known lower bound. Many papers report similar results as the authors, but am I missing something?\n\n[1] Zhou Dongruo, Quanquan Gu, and Csaba Szepesvari. \"Nearly minimax optimal reinforcement learning for linear mixture Markov decision processes.\" Conference on Learning Theory. PMLR 2021.\n\nLIMITATIONS:\nThe requirement for global state instead of partial observation may limit the practical applicability of the proposed methods.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper studies the asynchronous multi-agent bandit and RL problem with general function approximation (measured by Elude dimension). The main contribution is to establish $\tilde{O}(\sqrt{\text{dim} T})$ regret bound with $\tilde{O}(M^2 \text{dim})$ communication complexity.

STRENGTHS:
This paper is well written and the contribution is solid.

WEAKNESSES:
I think the major concern is non-optimal complexity bounds. Although it seems unreasonable to ask for a matching upper\&lower regret bound for the contextual bandit problem, the part about RL could be possibly improved (at least, the dependence on $H$ is not tight). Also I am curious that what is the current best lower bound for the communication cost to reach an $\sqrt{T}$ regret bound. It would be an interesting problem to study the exact trade-off between the communication cost and regret.

A minor concern might be about the technical novelty given previous methods on measuring the uncertainty.

QUESTIONS:
What is the meaning of "fully asynchronous" in table 1?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the distributed federated contextual bandit and federated reinforcement learning (FRL) in the presence of a trusted server. In both problems, nonlinearity and asynchronous communications are explored. Similar algorithms for contextual bandit and FRL that encourage exploration via bonus functions are proposed. Finite-time convergence results in terms of regrets are established for both algorithms and communication complexities are also characterized.\n\nSTRENGTHS:\n* This paper studied the asynchronous federated learning problem where only one agent is activated to sample data and infrequently communicate with server.\n* The trigger-based communication is an interesting approach in multi-agent or multi-learner problems.\n\nWEAKNESSES:\n* The clarity of some of the important quantities are not well defined or explained. For example, \n1)\tIn the sample complexity result of Theorem 4.3, $\\tilde{\\beta}_1$ is used. However, it was not defined. It’s unclear what this notation is referring to. Similarly, in Theorem 5.1, $\\tilde{\\beta}_2$ is used.\n2)\tThe oracle for to compute bonus term bk+1,h is crucial in understanding the algorithms. However, it was not very well-explained or shown anywhere in the main paper.\t\n3)\tThe two sentences from Line 300 to Line 302 are confusing. Please clarify them.\n* Typos:\n1)\tAn extra closing parenthesis appeared in Line 141.\n2)\tLine 214, ?? -> 12\n3)\tIn Line 1 of algorithm 3, $k=[K]$ -> $k\\in [K]$.\n4)\tIn Line 154, the trajectory should be $(s_h, a_h, \\cdots, s_H, a_H)$.\n\nQUESTIONS:\n* What if there are more than one agents are to be activated instead of just having one agent at each time t?\n* How is an agent chosen to be activated in Line 5 of both Algorithm 1 and 2?\n* What is the oracle to compute the bonus term?\n* What are the terms $\\tilde{\\beta}_1$ and $\\tilde{\\beta}_2$?\n* Please clarify the two sentences from Line 300 to Line 302.\n\nLIMITATIONS:\nPlease see weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}