Title: Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Randomized Exploration

Deep Reinforcement Learning
Abstract: We present the first study on provably efficient randomized exploration in cooperative multi-agent reinforcement learning (MARL). We propose a unified algorithm framework for randomized exploration in parallel Markov Decision Processes (MDPs), and two Thompson Sampling (TS)-type algorithms, CoopTS-PHE and CoopTS-LMC, incorporating the perturbed-history exploration (PHE) strategy and the Langevin Monte Carlo exploration (LMC) strategy respectively, which are flexible in design and easy to implement in practice. For a special class of parallel MDPs where the transition is (approximately) linear, we theoretically prove that both CoopTS-PHE and CoopTS-LMC achieve a $\widetilde{\mathcal{O}}(d^{3/2}H^2\sqrt{MK})$ regret bound with communication complexity $\widetilde{\mathcal{O}}(dHM^2)$, where $d$ is the feature dimension, $H$ is the horizon length, $M$ is the number of agents, and $K$ is the number of episodes. This is the first theoretical result for randomized exploration in cooperative MARL. We evaluate our proposed method on multiple parallel RL environments, including a deep exploration problem (i.e., $N$-chain), a video game, and a real-world problem in energy systems. Our experimental results support that our framework can achieve better performance, even under conditions of misspecified transition models. Additionally, we establish a connection between our unified framework and the practical application of federated learning.

Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning

Abstract

We present the first study on provably efficient randomized exploration in cooperative multi-agent reinforcement learning (MARL). We propose a unified algorithm framework for randomized exploration in parallel Markov Decision Processes (MDPs), and two Thompson Sampling (TS)-type algorithms, CoopTS-PHE and CoopTS-LMC, incorporating the perturbed-history exploration (PHE) strategy and the Langevin Monte Carlo exploration (LMC) strategy respectively, which are flexible in design and easy to implement in practice. For a special class of parallel MDPs where the transition is (approximately) linear, we theoretically prove that both CoopTS-PHE and CoopTS-LMC achieve a ˜ O ( d 3 / 2 H 2 √ MK ) regret bound with communication complexity ˜ O ( dHM 2 ) , where d is the feature dimension, H is the horizon length, M is the number of agents, and K is the number of episodes. This is the first theoretical result for randomized exploration in cooperative MARL. We evaluate our proposed method on multiple parallel RL environments, including a deep exploration problem ( i.e., N -chain), a video game, and a real-world problem in energy systems. Our experimental results support that our framework can achieve better performance, even under conditions of misspecified transition models. Additionally, we establish a connection between our unified framework and the practical application of federated learning.

1 Introduction

Multi-Agent Reinforcement Learning (MARL) has emerged as a potent tool with wide-ranging applications in diverse fields including robotics [23, 54], gaming [74, 92, 84], and numerous realworld systems [10, 25, 85]. This is particularly evident in cooperative scenarios, where MARL's effectiveness is enhanced through both direct and indirect communication channels among agents. This requires MARL algorithms to adeptly and flexibly coordinate communications to optimize the benefits of cooperation. One of the classical challenges in MARL is balancing exploration and exploitation so that agents effectively utilize existing information while acquiring new knowledge. Recent literature highlights the intricacies of this balance, focusing on cooperative exploration strategies [27] and dynamic exploitation tactics [68]. Achieving this equilibrium is crucial for the practical deployment of MARL systems in real-world scenarios, where unpredictability and the need for rapid adaptation are prevalent [27, 14, 55].

Optimism in the Face of Uncertainty (OFU) is a popular strategy to address the explorationexploitation problem [1]. OFU strategy leads to numerous upper confidence bound (UCB)-type algorithms in contextual bandits [18, 1, 50], single-agent reinforcement learning [36, 76], and more recently multi-agent reinforcement learning [24, 56]. These algorithms compute statistical confidence regions for the model or the value function, given the observed history, and perform the greedy

policy with respect to these regions, or upper confidence bounds. Though UCB-based methods give out strong theoretical results, they often have poor performance in practice [61, 60]. For example, Wang et al. [76] demonstrates that computing the confidence bonus necessitates advanced sensitivity sampling and the expensive computation makes the practical applications inefficient. It is worth noting that UCB is mostly constructed based on a linear structure [18, 36]. NeuralUCB is a notable attempt at a nonlinear version while it is infeasible in terms of computational complexity [94, 82].

Inspired by Thompson Sampling (TS) [73], posterior sampling for reinforcement learning (RL) [7, 95] involves maintaining a posterior distribution over the parameters of the Markov Decision Processes (MDP) model parameters. Although conceptually simple, most existing TS methods require the exact posterior or a good Laplacian approximation [83]. Recently, there have been advancements in randomized exploration with approximate sampling. One important method is perturb-history exploration (PHE) strategy, which involves introducing random perturbations in the action history of the agent [45, 47, 32]. This randomized exploration approach diversifies the agent's experience, aiding in learning more robust strategies in environments with uncertainty and variability. Another effective method is Langevin Monte Carlo (LMC) method [83, 33, 31, 42, 58, 34]. Notably, Ishfaq et al. [33] maintains the simplicity and scalability of LMC, making it applicable in deep RL algorithms by approximating the posterior distribution of the Q function.

Despite the aforementioned advancements of randomized exploration in bandits and single-agent RL, there remains a scarcity of research on randomized exploration within cooperative MARL, which motivates us to present the first investigation into provably efficient randomized exploration in cooperative MARL, with both theoretical and empirical evidence. We specifically focus on the applicability in parallel MDPs, aiming to facilitate faster learning and to improve policy optimization with the same state and action spaces, allowing for leveraging similarities across MDPs. We theoretically and empirically demonstrate that randomized exploration strategies can be extended to the multi-agent setting and the benefit of randomized exploration instead of UCB can be significant from single-agent to multi-agent setting.

In summary, our contributions are as follows:

We propose a unified algorithm framework for learning parallel MDPs, and apply two TS-related strategies PHE and LMC for exploration, which leads to the CoopTS-PHE and CoopTS-LMC algorithms. Unlike conventional TS, which suffers from sampling errors due to Laplace approximation and expensive posterior computation [66, 46], our proposed algorithms only require adding standard Gaussian noises to the dataset (CoopTS-PHE) or the gradient (CoopTS-LMC) when performing Least-Square Value Iteration, which is efficient in computation and avoids sampling bias due to the Laplace approximation. Notably, both algorithms are easily implementable which are more practical than UCB-based algorithms in deep MARL.

When reduced to linear parallel MDPs, we theoretically prove that both CoopTS-PHE and CoopTSLMC with linear function approximation can achieve a regret bound ˜ O ( d 3 / 2 H 2 √ M ( √ dMγ + √ K )) with communication complexity ˜ O ( ( d + K/γ ) MH ) , where d is the feature dimension, H is the horizon length, M is the number of agents, K is the number of episodes for each agent, and γ is a parameter controlling the communication frequency. When γ = O ( K/dM ) , our algorithms attain ˜ O ( d 3 / 2 H 2 √ MK ) regret with ˜ O ( dHM 2 ) communication complexity. This result matches the best communication complexity in cooperative MARL [56], and the best regret bounds for randomized RL in the single-agent setting ( M = 1 ) [32, 33]. A comprehensive comparison with baseline algorithms on episodic, non-sationary, linear MDPs is presented in Table 1.

We further extend our theoretical analysis to the misspecified setting where both the transition and reward are approximately linear up to an error ζ and the MDPs could be heterogeneous across agents, which is a generalized notion of misspecification [36]. We theoretically prove when ζ = O (√ d/MK ) , the cumulative regret for CoopTS-PHE matches the result in the linear homogeneous MDP setting. Simultaneously, when ζ = O (√ 1 /MK ) , the cumulative regret for CoopTS-LMC matches the result in the linear homogeneous MDP setting. This result indicates that CoopTS-PHE has a slightly higher tolerance on the model misspecification than CoopTS-LMC.

We conduct extensive experiments on various benchmarks with comprehensive ablation studies, including N -chain that requires deep exploration, Super Mario Bros task in a misspecified setting, and a real-world problem in thermal control of building energy systems. Our empirical evaluation demonstrates that our randomized exploration strategies outperform existing deep Q -network

(DQN)-based baselines. We also show that these strategies in cooperative MARL can be adapted to the existing federated RL framework when data transitions are not shared.

2 Preliminary

In parallel Markov Decision Processes (MDPs), M agents interact independently with their respective discrete-time MDPs, sharing the same but independent state and action spaces. Each agent might have its unique reward functions and transition kernels. Specifically, for agent m ∈ M , the associated MDP is defined by the tuple MDP( S , A , H, P m , r m ) . Here S and A are the state and action spaces respectively, H is the horizon length, P m = { P m,h } h ∈ [ H ] and r m = { r m,h } h ∈ [ H ] are the sets of transition kernels and reward functions. For step h ∈ [ H ] , P m,h ( ·| s, a ) is the probability measure over the next state given current state-action pair ( s, a ) , r m,h : S × A → [0 , 1] is the deterministic reward function. The policy π m = { π m,h } h ∈ [ H ] is a sequences of decision rules where π m,h : S → A is the deterministic policy at step h .

For agent m ∈ M , given any policy π and transition P , to evaluate the policy effectiveness in the m th MDP, we define value function V π m,h ( s ) := E π [ ∑ H h ′ = h r m,h ′ ( s m,h ′ , a m,h ′ ) | s m,h = s ] and Q function Q π m,h ( s, a ) := E π [ ∑ H h ′ = h r m,h ′ ( s m,h ′ , a m,h ′ ) | s m,h = s, a m,h = a ] for any ( h, s, a ) ∈ [ H ] ×S×A The optimal policy is defined as π ∗ m , and we denote V ∗ m,h ( s ) = V π ∗ m m,h ( s ) . For each k ∈ [ K ] , at the beginning of episode k , each agent m ∈ M receives the initial state s k m, 1 chosen arbitrarily by the environment. For each step h ∈ [ H ] in this episode, each agent m observes its current state s k m,h , selects an action a k m,h based on policy π k m,h , receives a reward r m,h ( s k m,h , a k m,h ) , and then transitions to the next state s k m,h +1 based on the transition probability measure P m,h ( ·| s k m,h , a k m,h ) . The reward defaults to 0 when the episode terminates at step H + 1 . The goal of agents is to minimize the cumulative group regret after K episodes, which is defined as

3 Algorithm Design

In this section, we first present a unified algorithm framework for conducting randomized exploration in cooperative MARL. Then we introduce two practical randomized exploration strategies.

3.1 Unified Algorithm Framework

A unified algorithm framework is presented in Algorithm 1, where each agent executes Least-Square Value Iteration (LSVI) in parallel and makes decisions based on collective data obtained from communication between each agent and the server. Before we describe the details of our algorithm, we first define notations about the datasets stored on each agent's local machine and the server.

Index notation We define k s ( k ) (denoted as k s when no ambiguity arises) as the last episode before episode k where synchronization happens. For episode k and step h , we define three datasets:

By definition, U ser h ( k ) is the dataset that is shared across all agents due to the latest synchronization at episode k s . U loc m,h ( k ) is the unique data collected by agent m since episode k s . Then U m,h ( k ) is the total dataset available for agent m at the current time. Let K ( k ) = | U m,h ( k ) | be the total number of data points. For the simplicity of notation, we also re-order the data points in U m,h ( k ) , and rename the tuple ( s τ m,h , a τ m,h , s τ m,h +1 ) as ( s l , a l , s ′ l ) such that we have U m,h ( k ) = ⋃ K ( k ) l =1 ( s l , a l , s ′ l ) . In fact, this can be done by the following one-to-one mapping

Therefore, we use indices ( s, a, s ′ ) ∈ U m,h ( k ) and l ∈ [ K ( k )] interchangeably for the summation over set U m,h ( k ) .

Algorithm interpretation At a high level, each episode k in Algorithm 1 consists of two stages. The first stage (Lines 3-15) is parallelly executed by all agents and the second stage (Lines 16-24) involves the communication among agents and the server.

In the first stage (Lines 3-15) of Algorithm 1, each agent m operates in two parts. The first part (Line 6) updates estimated Q functions { Q k m,h } H h =1 through LSVI with a randomized exploration strategy (Algorithm 2 or Algorithm 3, which will be introduced in Section 3.2). In particular, given the estimated value functions V k m,h +1 ( · ) = max a ∈ A Q k m,h ( · , a ) at step h +1 , we perform one step

robust backward Bellman update to obtain V k m,h ( · ) at step h . And we initialize V k m,H +1 ( · ) to be 0 (Line 5). In the second part (Lines 7-14), after obtaining the estimated Q functions, in each step h we execute the greedy policy with respect to Q k m,h and collect new data points which are added to the local dataset U loc m,h ( k ) (Lines 8-10). Then we verify the synchronization condition (Lines 11-13). In this paper, we mainly use three types of synchronization rules. (1) We can synchronize every c episode where c is a user-defined constant, which is easy to implement in practice. (2) We can also synchronize at the episode of b 1 , b 2 , ..., b n , with b representing the base of the exponential function. This is guided by the intuition that agents require more transitions urgently at the early learning stages. (3) Additionally, if we have a feature mapping ϕ ( s, a ) : S × A → R d , based on (3.1), we define the following empirical covariance matrices.

We synchronize as long as the following condition is met:

where γ is a communication control factor. In our experiments, we try all three rules and compare their performance, which is discussed in detail in Appendix K.1.

The second stage (Lines 16-24) is executed only when the synchronization condition is satisfied. First, all the agents upload their local transition set U loc m,h ( k ) , i.e., the newly collected local data after the last synchronization, to the server. Then, the server gathers all information together in U ser h ( k ) and sends it back to each agent. Finally, each agent resets the local transition set U loc m,h ( k ) ←∅ . Now agent m can access the dataset U m,h ( k ) = U ser h ( k ) ⋃ U loc m,h ( k ) , which contains the historical data of all agents up to last synchronization and its local dataset.

3.2 Randomized Exploration Strategies

When we update the model parameter and estimate Q functions in Algorithm 1 (Line 6), we use exploration strategies to avoid suboptimal policies. Previous work adopted Upper Confidence Bound (UCB) exploration in the linear function class [24, 56] to estimate the Q function { Q k m,h } H h =1 . Although UCB-based methods come with strong theoretical guarantees, they often perform poorly in practice [16, 61, 60]. Moreover, UCB requires precise computation of the confidence set, which is usually hard to be implemented beyond the linear structure. In contrast, randomized exploration strategies offer more robust performance, flexibility in design, ease of implementation, and do not require a linear structure.

Weapproximate the Q functions with the following function class F = { f w : S×A → R | f w ( s, a ) = f ( w ; ϕ ( s, a )) } , where w ∈ R d is the parameter and ϕ ∈ R d is a feature mapping associated with state-action pairs. Now we define the loss function for estimating the Q functions.

l =1 where r l h = r h ( s l , a l ) , ϕ l = ϕ ( s l , a l ) , and L is a user-specified loss function.

Perturbed-History Exploration The first strategy we use in Algorithm 1 is called the perturbedhistory exploration [45, 47, 32], displayed in Algorithm 2. We refer to the resulting algorithm as CoopTS-PHE. In particular, we optimize the following randomized loss function, where we add random Gaussian noises to the rewards and regularizer in (3.4).

where ϵ k,l,n h i.i.d ∼ N (0 , σ 2 ) , ξ k,n h ∼ N ( 0 , σ 2 I ) , and n ∈ [ N ] . Then we obtain the following perturbed estimated parameter

Note that we repeat the above steps for n = 1 , . . . , N to obtain independent copies of parameters, which is referred to as the multi-sampling process [32, 33]. Then we obtain the estimated Q function Q k m,h based on Line 7 in Algorithm 2. Finally, by maximizing Q k m,h over action space A , we obtain the estimated value function V k m,h .

Algorithm 2 Perturbed-History Exploration

Langevin Monte Carlo Exploration Next we introduce the Langevin Monte Carlo exploration strategy [83, 33] in Algorithm 3, which stems from the Langevin dynamics [67, 8, 19, 81, 96]. Combining it with Algorithm 1 leads to our second proposed algorithm, CoopTS-LMC. Specifically, we update the model parameter iteratively. For iterate j = 1 , . . . , J k , the update is given by

where L k m,h is defined in (3.4), ϵ k,j,n m,h ∈ R d is a standard Gaussian noise, η m,k is the learning rate, and β m,k is the inverse temperature parameter. We similarly use the multi-sampling trick to obtain N independent estimators and estimate Q function Q k m,h by truncation based on Line 10 in Algorithm 3.

Algorithm 3 Langevin Monte Carlo Exploration

4 Theoretical Analysis

4.1 Homogeneous Parallel Linear MDPs

We provide theoretical analyses of our algorithms in the linear structure under the assumption of linear function approximation and linear MDP setting. We first present the definition of linear MDPs.

Definition 4.1 (Linear MDP [36]) . An MDP( S , A , H, P , r ) is a linear MDP with feature map ϕ : S × A → R d , if for any h ∈ [ H ] , there exist d unknown measures µ h = ( µ 1 h , ..., µ d h ) over S and an unknown vector θ h ∈ R d such that for any ( s, a ) ∈ S × A ,

Without loss of generality, we assume that for all ( s, a ) ∈ S × A , ∥ ϕ ( s, a ) ∥ ≤ 1 and max {∥ µ h ( S ) ∥ , ∥ θ h ∥} ≤ √ d.

Throughout the analyses in this section, we assume the homogeneous parallel MDPs setting where all agents share the same linear MDP defined in Definition 4.1. We also provide the results when the MDPs across agents are approximately linear and heterogeneous, which is deferred to Section 4.2 due to the space limit. Under the linear MDP assumption, it is known that the Q -function admits a linear form [36, Proposition 2.3]. Consequently, we choose the loss function L in (3.4) to be the l 2 loss and approximate the Q function in the linear function class f ( w ; ϕ l ) = w ⊤ ϕ l .

Now we first present the regret bound for CoopTS-PHE.

Theorem 4.2. Under Definition 4.1, choose L to be l 2 loss and linear function class f ( w ; ϕ l ) = w ⊤ ϕ l in (3.4). In CoopTS-PHE (Algorithm 1+Algorithm 2), let N = ˜ C log( δ ) / log( c 0 ) where ˜ C = ˜ O ( d ) and c 0 = Φ(1) , Φ( · ) is the cumulative distribution function (CDF) of the standard normal distribution. Let λ = 1 and 0 < δ < 1 . Under the determinant synchronization condition (3.3), we obtain the following cumulative regret

with probability at least 1 -δ .

Remark 4.3. When we choose γ = O ( K/dM ) in the synchronization condition (3.3), the cumulative regret of CoopTS-PHE becomes ˜ O ( d 3 / 2 H 2 √ MK ) , which matches the result of UCB exploration [24]. When M = 1 , the regret becomes ˜ O ( d 3 / 2 H 2 √ K ) , which matches the existing best randomized single-agent result [32, 33]. Note that if there is no communication at all and agents act independently, with the same number of learning rounds (or samples), the cumulative regret becomes ˜ O ( M · d 3 / 2 H 2 √ K ) . By incorporating communication, our regret bound in Theorem 4.2 is lower than that of the independent setting by a factor √ M . A similar strategy called rare-switching update with a determinant synchronization condition has also been adopted in parallel bandit problems [69, 15].

Similarly, we have the following result for CoopTS-LMC.

Theorem 4.4. Under Definition 4.1, choose L to be l 2 loss and linear function class f ( w ; ϕ l ) = w ⊤ ϕ l in (3.4). In CoopTS-LMC (Algorithm 1+Algorithm 3), let N = ¯ C log( δ ) / log( c ′ 0 ) where c ′ 0 = 1 -1 / 2 √ 2 eπ and ¯ C = ˜ O ( d ) . Let 1 / √ β m,k = ˜ O ( H √ d ) for all m ∈ M , λ = 1 , and 0 < δ < 1 . For any episode k ∈ [ K ] and agent m ∈ M , let the learning rate η m,k = 1 / ( 4 λ max ( Λ k m,h )) , the update number J k = 2 κ k log(4 HKMd ) where κ k = λ max ( Λ k m,h ) /λ min ( Λ k m,h ) is the condition number of Λ k m,h . Under the determinant synchronization condition (3.3), we have

with probability at least 1 -δ .

Remark 4.5. Note that CoopTS-PHE and CoopTS-LMC have the same order of regret. Hence the discussion in Remark 4.3 also applies to CoopTS-LMC. We would also like to highlight that our results are the first rigorous regret bounds for randomized MARL algorithms.

From the perspective of technical novelty, our analysis of randomized MARL algorithms is different from that of UCB-based algorithms [24] because the model prediction error here contains randomness, causing a more complex probability analysis and an additional approximation error. We would also like to point out that in proofs for both CoopTS-LMC and CoopTS-PHE we use a new ε -covering technique to prove that the optimism lemma holds for all ( s, a ) ∈ S×A instead of just the state-action pairs encountered by the algorithm, which is essential for the regret analysis. This was ignored by previous works [13] and its follow-up works [93, 33] that use the same regret decomposition technique. Furthermore, the multi-agent setting and the communications from synchronization in our algorithms also significantly increase the challenges in our analysis compared to randomized exploration in the single-agent setting [32, 33].

Next we present the communication complexity of Algorithm 1 with synchronization condition (3.3).

Lemma 4.6. The total number of communication rounds between the agents and the server in Algorithm 1 is bounded by CPX = ˜ O (( d + K/γ ) MH ) . Moreover, the total number of transferred random bits only has a logarithmic dependence on the number of episodes K .

Remark 4.7. We provide a refined analysis in Appendix C to get this improved result based on that of [24], which studied the same communication procedure as ours. When we choose γ = O ( K/dM ) , the communication complexity reduces to ˜ O ( dHM 2 ) , which only has a logarithmic dependence on the number of episodes K . Additionally, we provide a rigorous analysis to show that the algorithm only needs to communicate logarithm number of random bits throughout the learning process.

Note that Min et al. [56] studied the asynchronous setting where only one agent is active in each episode, giving out the regret ˜ O ( d 3 / 2 H 2 √ K ) with the communication complexity ˜ O ( dHM 2 ) . It is interesting to see that our algorithm, though in the synchronous setting, has the same communication complexity as the asynchronous variant. This implies that the asynchronous algorithm can only circumvent current communication by delaying it to the future but does not decrease the communication complexity. In fact, the synchronous setting can learn the policy better in our work, which is indicated by comparison of the average regret (the cumulative regret divided by the total number of samples used by the algorithm) in Table 1. By achieving a matched communication complexity, we find that synchronous and asynchronous settings have their own advantages and cannot replace each other. This phenomenon can help us better understand the properties of these two communication schemes.

4.2 Misspecified Setting

In this part, we extend our theoretical analysis to the misspecified setting. In this setting, the transition functions P m,h and the reward functions r m,h are heterogeneous across different MDPs, which is slightly more complicated than the homogeneous setting. Moreover, instead of assuming the transition and reward are linear, we only require each individual MDP is a ζ -approximate linear MDP [36] where both the transition and reward are approximately linear up to an controlled error ζ .

Definition 4.8 (Misspecified Parallel MDPs) . For any 0 < ζ ≤ 1 , and for any agent m ∈ M , the corresponding MDP( S , A , H, P m , r m ) is a ζ -approximate linear MDP with a feature map ϕ : S × A → R d , for any h ∈ [ H ] , there exist d unknown (signed) measures µ h = ( µ (1) h , . . . , µ ( d ) h ) over S and an unknown vector θ h ∈ R d such that for any ( s, a ) ∈ S × A , we have

where ∥ · ∥ TV is the total variation norm, for two distributions P 1 and P 2 , we define it as: ∥ P 1 -P 2 ∥ TV = 1 2 ∑ x ∈ Ω | P 1 ( x ) -P 2 ( x ) | . Without loss of generality, we assume that ∥ ϕ ( s, a ) ∥ ≤ 1 for all ( s, a ) ∈ S × A , and max { ∥ µ h ( S ) ∥ , ∥ θ h ∥ } ≤ √ d for all h ∈ [ H ] and m ∈ M .

Remark 4.9. Note that our misspecified setting defined in Definition 4.8 is a generalized notion of misspecification in [36]. Moreover, our misspecified setting is also more general and cover the small heterogeneous setting mentioned in [24]. The triangle inequality can easily be used to derive small heterogeneous setting from our misspecified setting, but not vice versa.

Next we state our regret bound for CoopTS-PHE in the misspecified setting.

Theorem 4.10 (Misspecified Regret Bound for CoopTS-PHE) . In CoopTS-PHE (Algorithm 1+Algorithm 2), under Definition 4.8 and determinant synchronization condition (3.3), with the same initialization with Theorem 4.2, we obtain the following cumulative regret

with probability at least 1 -δ .

Remark 4.11. When we choose ζ = O (√ d/MK ) , the cumulative regret becomes ˜ O ( d 3 2 H 2 √ M ( √ dMγ + √ K )) . This matches the result of Theorem 4.2 in the linear MDP setting.

Similarly, we can have the following result for CoopTS-LMC.

Theorem 4.12 (Misspecified Regret Bound for CoopTS-LMC) . In CoopTS-LMC (Algorithm 1+Algorithm 3), under Definition 4.8 and determinant synchronization condition (3.3), with the same initialization with Theorem 4.4 except that 1 / √ β m,k = ˜ O ( H √ d + H √ MKdζ ) , we obtain the following cumulative regret

with probability at least 1 -δ .

Remark 4.13. When ζ = O (√ 1 /MK ) , the cumulative regret becomes ˜ O ( d 3 2 H 2 √ M ( √ dMγ + √ K )) . This matches the result of Theorem 4.4 in the linear MDP setting. By comparing Theorems 4.10 and 4.12, we find the result of CoopTS-LMC has an extra √ d factor worse than that of CoopTS-PHE, causing the chosen ζ in CoopTS-PHE has an extra √ d order over that in CoopTS-LMC. This indicates that CoopTS-PHE has better performance tolerance for the misspecified setting.

5 Experiments

In this section, we present an empirical evaluation of our proposed randomized exploration strategies ( i.e., CoopTS-PHE and CoopTS-LMC) with deep Q -networks (DQNs) [57] as the core algorithm on varying tasks under multi-agent settings compared with several baselines: vanilla DQN, Double DQN [28], Bootstrapped DQN [62], and Noisy-Net [26]). Given that all experiments are conducted under multi-agent settings unless explicitly specified as a single-agent or centralized scenario, we denote CoopTS-PHE as "PHE" and CoopTS-LMC as "LMC" in both experimental contexts and figures. Note that we run all our experiments on Nvidia RTX A5000 with 24GB RAM. The implementation of this work can be found at https://github.com/panxulab/MARL-CoopTS

0

500

1000

1500

2000

Training Episodes per Agent

0

2

4

6

8

10

PHE

LMC

DQN

Bootstrapped DQN

NoisyNet DQN

DDQN

(a) m=2

0

500

1000

1500

2000

Training Episodes per Agent

0

2

4

6

8

10

PHE

LMC

DQN

Bootstrapped DQN

NoisyNet DQN

DDQN

(b) m=3

0

200

400

600

800

1000

1200

1400

1600

Training Episodes per Agent

0

100

200

300

400

500

600

700

800

900

PHE

LMC

DQN

Bootstrapped DQN

NoisyNet DQN

DDQN

(c) Mario (Parallel)

0

200

400

600

800

1000

1200

1400

1600

Training Episodes per Agent

0

100

200

300

400

500

600

700

800

900

PHE

LMC

DQN

Bootstrapped DQN

NoisyNet DQN

DDQN

(d) Mario (Federated)

5.1 N -chain

The N -chain [62] comprises a sequence of N states denoted as { s l } N l =1 . Assuming the existence of m agents, all initiating their trajectories from s 2 , this study explores the dynamics of their movement within the chain. At each time step, agents face the decision to move either left or right. Notably, each agent incurs a nominal reward of r = 0 . 001 upon reaching state s 1 , while a more substantial reward of r = 1 is obtained upon reaching the terminal state s N . The illustration of N -chain environment is shown in Appendix K.1. With a horizon length of N +9 , the optimal return is 10 . We consider N = 25 with the communication among agents in Figure 1 following the synchronization approach in Algorithm 1. In Figure 1(a), we show that PHE and Bootstrapped DQN result in higher average episode return among all agents while LMC can also eventually converge to a similar reward.

Upon increasing the number of agents to m = 3 , we show in Figure 1(b) that our randomized exploration methods outperform all other baselines. Notably, the fluctuation in PHE is observed to be less pronounced against LMC. This observation lends support to our theoretical framework regarding performance tolerance in the misspecified setting, as detailed in Section 4.2. The complete results for N -chain and ablation studies can be found in Appendix K.1.

5.2 Super Mario Bros

Environmental heterogeneity, arising from various sources, is a prevalent challenge in practical scenarios. In Section 4.2, we illustrate the extension of homogeneous parallel MDP to the misspecified

setting. In the Super Mario Bros task [74], we examine a scenario where four agents, denoted as m = 4 , engage in learning within distinct environments. Despite these environments sharing the same state space S , action space A , and reward function, their characteristics are different described in Appendix K.2. The primary objective of the Super Mario Bros task is to train an agent capable of advancing as far-right and rapidly as possible without collisions or falls. Utilizing preprocessed images as input states, agents aim to select optimal actions from a set of 7 discrete actions.

Figure 1(c) visually depicts that both randomized exploration strategies outperform other baselines in cooperative parallel learning. Notably, We observe that the superiority of LMC gets significant against PHE unlike the results in N -chain in Figures 1(a) and 1(b). In the case of PHE, Gaussian noise is introduced to the reward before applying the Bellman update, which can be viewed as a method empirically approximating the posterior distribution of the Q function using a Gaussian distribution. However, it is crucial to note that in practical scenarios, unlike the N -chain setting, Gaussian distributions may not always provide an accurate approximation of the true posterior of the Q function [33]. Here, transitions are shared among the four agents whenever the synchronization condition in (3.3) is met. We also conducted extra experiments in this task extending our proposed method to federated learning shown in Figure 1(d) with details in Appendix K.2.

5.3 Thermal Control of Building Energy Systems

Finally, we assess the efficacy of our randomized exploration strategies through their application to a practical task within a sustainable energy system: BuildingEnv, as outlined in [85]. BuildingEnv is designed to manage the heating supply in a multi-zone building, which involves addressing realworld physical constraints and accounting for environmental shifts over time. The objective is to meet user-defined temperature specifications while simultaneously minimizing overall electricity consumption. We defer the environment details to Appendix K.3.

With the availability of different cities in varying weather types, we conduct experiments on multiple cities in parallel and share their data following Algorithm 1 for each exploration strategy. During the evaluation, we deploy those trained policies to the environment of each city/weather respectively. We include all methods as well as random action in Figure 2 for a fair comparison. Specifically, we sample action randomly from action space for random action. We display the distribution of the return with probability density in violin plots, indicating that our PHE and LMC can perform better with a higher mean. Additional results for other cities can be found in Appendix K.3.

6 Conclusion

We proposed a unified algorithm framework for provably efficient randomized exploration in parallel MDPs. By combining this unified algorithm framework with two TS-type randomized exploration strategies, PHE and LMC, we obtained two algorithms for parallel MDPs: CoopTS-PHE and CoopTSLMC. These two algorithms are both flexible in design and easy to implement in practice. Under the linear MDP setting, we derived the theoretical regret bounds and communication complexities of CoopTS-PHE and CoopTS-LMC. This is the first result for randomized exploration in cooperative MARL, matching the best existing regret bounds for single-agent RL [32, 33]. We also extended our theoretical analysis to the misspecified setting. Our experiments on diverse RL parallel environments verified that randomized exploration improves the balance between exploration and exploitation in both homogeneous and heterogeneous settings. Future research directions includes extending our randomized exploration algorithm to fully decentralized or federated learning settings. Additionally, developing a more communication-efficient algorithm to reduce the substantial communication costs in the general function class setting is another potential direction.

Random

PHE

LMC

DQN

BootstrappedNoisyNet

DDQN

Algorithm

50

40

30

20

10

Daily Return

Acknowledgments

We would like to thank the anonymous reviewers for their helpful comments. HH and MP were supported in part by the ONR under agreement N00014-23-1-2206, AFOSR under the award number FA9550-19-1-0169, and by the NSF under NAIAD Award 2332744 as well as the National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks, Grant CNS-2112562. WWand PX were supported in part by the National Science Foundation (DMS-2323112) and the Whitehead Scholars Program at the Duke University School of Medicine. The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agency.

References

Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011. 1, 56, 57

M. Abeille and A. Lazaric. Linear thompson sampling revisited. In Artificial Intelligence and Statistics , pages 176-184. PMLR, 2017. 18

M. Abramowitz and I. A. Stegun. Handbook of mathematical functions with formulas, graphs, and mathematical tables , volume 55. US Government printing office, 1968. 57

P. Agrawal, J. Chen, and N. Jiang. Improved worst-case regret bounds for randomized leastsquares value iteration. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 6566-6573, 2021. 17

S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning , pages 127-135, 2013. 60

S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013. 18

S. Agrawal and R. Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. In Advances in Neural Information Processing Systems , volume 30, pages 1184-1194, 2017. 2, 17

D. Bakry, I. Gentil, M. Ledoux, et al. Analysis and geometry of Markov diffusion operators , volume 103. Springer, 2014. 6

N. A. Bakshi, T. Gupta, R. Ghods, and J. Schneider. Guts: Generalized uncertainty-aware thompson sampling for multi-agent active search. In IEEE International Conference on Robotics and Automation (ICRA) , pages 7735-7741. IEEE, 2023. 17

A. L. Bazzan. Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multi-Agent Systems , 18:342-375, 2009. 1

D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of Operations Research , 27(4):819-840, 2002. 17

C. Boutilier. Planning, learning and coordination in multiagent decision processes. In Theoretical Aspects of Rationality and Knowledge , 1996. 17

Q. Cai, Z. Yang, C. Jin, and Z. Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020. 7, 22

G. Chalkiadakis and C. Boutilier. Coordination in multiagent reinforcement learning: A bayesian approach. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems , pages 709-716, 2003. 1

J. Chan, A. Pacchiano, N. Tripuraneni, Y. S. Song, P. Bartlett, and M. I. Jordan. Parallelizing contextual bandits. arXiv preprint arXiv:2105.10590 , 2021. 7

O. Chapelle and L. Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems , 24, 2011. 5, 18

Y. Chen, P. Dong, Q. Bai, M. Dimakopoulou, W. Xu, and Z. Zhou. Society of agents: Regret bounds of concurrent thompson sampling. Advances in Neural Information Processing Systems , pages 7587-7598, 2022. 17

W. Chu, L. Li, L. Reyzin, and R. Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011. 1, 2

A. S. Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. Journal of the Royal Statistical Society Series B: Statistical Methodology , 79(3): 651-676, 2017. 6

C. Dann, M. Mohri, T. Zhang, and J. Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in neural information processing systems , 34, 2021. 17

M. Dimakopoulou and B. V. Roy. Coordinated exploration in concurrent reinforcement learning. Proceedings of the 35th International Conference on Machine Learning , pages 1271-1279, 2018. 17

M. Dimakopoulou, I. Osband, and B. V. Roy. Scalable coordinated exploration in concurrent reinforcement learning. Advances in Neural Information Processing Systems , pages 4219-4227, 2018. 17

G. Ding, J. J. Koh, K. Merckaert, B. Vanderborght, M. M. Nicotra, C. Heckman, A. Roncone, and L. Chen. Distributed reinforcement learning for cooperative multi-robot object manipulation. In Proceedings of the 2020 International Conference on Autonomous Agents and Multiagent Systems, AAMAS , pages 1831-1833. ACM, 2020. 1

A. Dubey and A. Pentland. Provably efficient cooperative multi-agent reinforcement learning with function approximation. arXiv preprint arXiv:2103.04972 , 2021. 1, 3, 5, 7, 8, 17, 19, 22

Y. Fei and R. Xu. Cascaded gaps: Towards logarithmic regret for risk-sensitive reinforcement learning. In International Conference on Machine Learning , pages 6392-6417. PMLR, 2022. 1

M. Fortunato, M. G. Azar, B. Piot, et al. Noisy networks for exploration. In International Conference on Learning Representations , 2018. 9, 17, 58

J. Hao, T. Yang, H. Tang, C. Bai, J. Liu, Z. Meng, P. Liu, and Z. Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain. IEEE Transactions on Neural Networks and Learning Systems , 2023. 1

H. V. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double qlearning. In Annual AAAI Conference on Artificial Intelligence (AAAI) , 2016. 9, 58

E. Hillel, Z. S. Karnin, T. Koren, R. Lempel, and O. Somekh. Distributed exploration in multi-armed bandits. Advances in Neural Information Processing Systems , 26, 2013. 17

R. A. Horn and C. R. Johnson. Matrix analysis . Cambridge university press, 2012. 56

T. Huix, M. Zhang, and A. Durmus. Tight regret and complexity bounds for thompson sampling via langevin monte carlo. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 8749-8770. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/huix23a.html . 2

H. Ishfaq, Q. Cui, V. Nguyen, A. Ayoub, Z. Yang, Z. Wang, D. Precup, and L. Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning , pages 4607-4616. PMLR, 2021. 2, 3, 5, 6, 7, 10, 17, 56, 57

H. Ishfaq, Q. Lan, P. Xu, A. R. Mahmood, D. Precup, A. Anandkumar, and K. Azizzadenesheli. Provable and practical: Efficient exploration in reinforcement learning via langevin monte carlo. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=nfIAEJFiBZ . 2, 3, 6, 7, 10, 17, 22, 56, 59

H. Ishfaq, Y. Tan, Y. Yang, Q. Lan, J. Lu, A. R. Mahmood, D. Precup, and P. Xu. More efficient randomized exploration for reinforcement learning via approximate sampling. Reinforcement Learning Journal , 3:1211-1235, 2024. 2, 3

M. Jafarnia-Jahromi, R. Jain, and A. Nayyar. A bayesian learning algorithm for unknown zero-sum stochastic games with an arbitrary opponent. In International Conference on Artificial Intelligence and Statistics , pages 3880-3888. PMLR, 2024. 17

C. Jin, Z. Yang, Z. Wang, and M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR, 2020. 1, 2, 3, 7, 8, 17, 56

H. Jin, Y. Peng, W. Yang, S. Wang, and Z. Zhang. Federated reinforcement learning with environment heterogeneity. In International Conference on Artificial Intelligence and Statistics , pages 18-37. PMLR, 2022. 64

T. Jin, P. Xu, J. Shi, X. Xiao, and Q. Gu. Mots: Minimax optimal thompson sampling. In International Conference on Machine Learning , pages 5074-5083. PMLR, 2021. 17

T. Jin, P. Xu, X. Xiao, and A. Anandkumar. Finite-time regret of thompson sampling algorithms for exponential family multi-armed bandits. Advances in Neural Information Processing Systems , 35:38475-38487, 2022.

T. Jin, X. Yang, X. Xiao, and P. Xu. Thompson sampling with less exploration is fast and optimal. In International Conference on Machine Learning , pages 15239-15261. PMLR, 2023. 17

T. Jin, H.-L. Hsu, W. Chang, and P. Xu. Finite-time frequentist regret bounds of multi-agent thompson sampling on sparse hypergraphs. In Annual AAAI Conference on Artificial Intelligence (AAAI) , 2024. 17

A. Karbasi, N. L. Kuang, Y. Ma, and S. Mitra. Langevin thompson sampling with logarithmic communication: Bandits and reinforcement learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 15828-15860. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/ v202/karbasi23a.html . 2, 17

R. M. Kretchmar. Parallel reinforcement learning. In The 6th World Conference on Systemics, Cybernetics, and Informatics , 2002. 17

N. Kuang, M. Yin, et al. Posterior sampling with delayed feedback for reinforcement learning with linear function approximation. Advances in neural information processing systems , 2023. 17

B. Kveton, C. Szepesvari, M. Ghavamzadeh, and C. Boutilier. Perturbed-history exploration in stochastic multi-armed bandits, 2019. 2, 5

B. Kveton, M. Zaheer, C. Szepesvari, L. Li, M. Ghavamzadeh, and C. Boutilier. Randomized exploration in generalized linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 2066-2076. PMLR, 2020. 2, 18

B. Kveton, M. Zaheer, C. Szepesvari, L. Li, M. Ghavamzadeh, and C. Boutilier. Randomized exploration in generalized linear bandits. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 2066-2076. PMLR, 26-28 Aug 2020. URL https://proceedings.mlr.press/v108/kveton20a.html . 2, 5

P. Landgren, V. Srivastava, and N. E. Leonard. On distributed cooperative decision-making in multiarmed bandits. In 2016 European Control Conference (ECC) , pages 243-248. IEEE, 2016. 17

L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010. 60

L. Li, Y . Lu, and D. Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017. 1

T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems , volume 2, pages 429-450, 2020. 17

Z. Li, Y. Li, Y . Zhang, T. Zhang, and Z.-Q. Luo. Hyperdqn: A randomized exploration method for deep reinforcement learning. In International Conference on Learning Representations , 2022. 17

J. Lidard, U. Madhushani, and N. E. Leonard. Provably efficient multi-agent reinforcement learning with fully decentralized communication. In 2022 American Control Conference (ACC) , pages 3311-3316. IEEE, 2022. 17

B. Liu, L. Wang, and M. Liu. Lifelong federated reinforcement learning: a learning architecture for navigation in cloud robotic systems. IEEE Robotics and Automation Letters , 4(4):4555-4562, 2019. 1

Z. Liu, J. Zhang, Z. Liu, H. Du, Z. Wang, D. Niyato, M. Guizani, and B. Ai. Cell-free xl-mimo meets multi-agent reinforcement learning: Architectures, challenges, and future directions. IEEE Wireless Communications , 31(4):155-162, 2024. 1

Y. Min, J. He, T. Wang, and Q. Gu. Cooperative multi-agent reinforcement learning: asynchronous communication and linear function approximation. In International Conference on Machine Learning , pages 24785-24811. PMLR, 2023. 1, 2, 3, 5, 8, 17

V. Mnih, K. Kavukcuoglu, D. Silver, et al. Human-level control through deep reinforcement learning. Nature , 518:529-533, 2015. 9, 58, 65

A. Mousavi-Hosseini, T. Farghly, Y. He, K. Balasubramanian, and M. A. Erdogdu. Towards a complete analysis of langevin monte carlo: Beyond poincaré inequality, 2023. 2

T. Nguyen-Tang and R. Arora. On sample-efficient offline reinforcement learning: Data diversity, posterior sampling and beyond. Advances in neural information processing systems , 2023. 17

I. Osband and B. Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International conference on machine learning , pages 2701-2710. PMLR, 2017. 2, 5, 17

I. Osband, D. Russo, and B. Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26, 2013. 2, 5, 17

I. Osband, C. Blundell, A. Pritzel, and B. V. Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016. 9, 17, 58

I. Osband, B. Van Roy, and Z. Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning , pages 2377-2386. PMLR, 2016. 17

I. Osband, J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning. Advances in neural information processing systems , 31, 2018. 17

S. Qiu, Z. Dai, H. Zhong, Z. Wang, Z. Yang, and T. Zhang. Posterior sampling for competitive rl: Function approximation and partial observation. Advances in neural information processing systems , 2023. 17

C. Riquelme, G. Tucker, and J. Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. In International Conference on Learning Representations , 2018. 2, 18

G. O. Roberts and R. L. Tweedie. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli , pages 341-363, 1996. 6

C. Rojas-Córdova, A. J. Williamson, J. A. Pertuze, and G. Calvo. Why one strategy does not fit all: a systematic review on exploration-exploitation in different organizational archetypes. Review of Managerial Science , 17(7):2251-2295, 2023. 1

Y. Ruan, J. Yang, and Y. Zhou. Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 74-87, 2021. 7

D. Russo. Worst-case regret bounds for exploration via randomized value functions. Advances in neural information processing systems , 32:14410-14420, 2019. 17

M. Strens. A bayesian framework for reinforcement learning. In International Conference on Machine Learning , pages 943-950. PMLR, 2000. 17

M. E. Taylor and P. Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(56):1633-1685, 2009. 17

W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933. 2, 17

J.-J. Tsay, C.-C. Chen, and J.-J. Hsu. Evolving intelligent mario controller by reinforcement learning. In International Conference on Technologies and Applications of Artificial Intelligence , pages 266-272, 2011. 1, 10

R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018. 56

R. Wang, R. R. Salakhutdinov, and L. Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135, 2020. 1, 2

Y. Wang, J. Hu, X. Chen, and L. Wang. Distributed bandit learning: Near-optimal regret with efficient communication. In International Conference on Learning Representations , 2020. 17

Z. Wang and M. Zhou. Thompson sampling via local uncertainty. In International Conference on Machine Learning , pages 10115-10125. PMLR, 2020. 17

Q. Xie, Y. Chen, Z. Wang, and Z. Yang. Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Proceedings of Thirty Third Conference on Learning Theory , volume 125, pages 3674-3682. PMLR, 2020. 17

W. Xiong, H. Zhong, C. Shi, C. Shen, and T. Zhang. A self-play posterior sampling algorithm for zero-sum markov games. In International Conference on Machine Learning , pages 2449624523. PMLR, 2022. 17

P. Xu, J. Chen, D. Zou, and Q. Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. Advances in Neural Information Processing Systems , 31, 2018. 6

P. Xu, Z. Wen, H. Zhao, and Q. Gu. Neural contextual bandits with deep representation and shallow exploration. In International Conference on Learning Representations , 2021. 2

P. Xu, H. Zheng, E. V. Mazumdar, K. Azizzadenesheli, and A. Anandkumar. Langevin monte carlo for contextual bandits. In International Conference on Machine Learning , pages 2483024850. PMLR, 2022. 2, 6, 17, 18

D. Ye, G. Chen, W. Zhang, S. Chen, B. Yuan, B. Liu, J. Chen, Z. Liu, F. Qiu, H. Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems , 33:621-632, 2020. 1

C. Yeh, V. Li, R. Datta, J. Arroyo, N. Christianson, C. Zhang, Y. Chen, M. Hosseini, A. Golmohammadi, Y. Shi, Y. Yue, and A. Wierman. Sustaingym: A benchmark suite of reinforcement learning for sustainability applications. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track . PMLR, 2023. 1, 10, 65, 66

C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y. Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. 17

C. Yu, X. Yang, J. Gao, J. Chen, Y. Li, J. Liu, Y . Xiang, R. Huang, H. Yang, Y . Wu, and Y . Wang. Asynchronous multi-agent reinforcement learning for efficient real-time multi-robot cooperative exploration. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS , pages 1107-1115. ACM, 2023. 17

A. Zanette, D. Brandfonbrener, E. Brunskill, M. Pirotta, and A. Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics , pages 1954-1964. PMLR, 2020. 3, 17

K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Ba¸ sar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning , volume 80, pages 5872-5881. PMLR, 2018. 17

W. Zhang, D. Zhou, L. Li, and Q. Gu. Neural thompson sampling. In International Conference on Learning Representations , 2021. 60

Y. Zhang, G. Qu, P. Xu, Y. Lin, Z. Chen, and A. Wierman. Global convergence of localized policy iteration in networked multi-agent reinforcement learning. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 7(1):1-51, 2023. 17

Y. Zhao, I. Borovikov, J. Rupert, C. Somers, and A. Beirami. On multi-agent learning in team sports games. arXiv preprint arXiv:1906.10124 , 2019. 1

H. Zhong and T. Zhang. A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes. Advances in Neural Information Processing Systems , 36, 2023. 7

D. Zhou, L. Li, and Q. Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning , pages 11492-11502, 2020. 2, 60

Y. Zhou, J. Li, and J. Zhu. Posterior sampling for multi-agent reinforcement learning: solving extensive games with imperfect information. In International Conference on Learning Representations , 2019. 2, 17

D. Zou, P. Xu, and Q. Gu. Faster convergence of stochastic gradient langevin dynamics for non-log-concave sampling. In Uncertainty in Artificial Intelligence , pages 1152-1162. PMLR, 2021. 6

A Related Work

Cooperative Multi-Agent Reinforcement Learning Cooperative MARL is closely intertwined with the domain of multi-agent multi-armed bandits, exemplified by decentralized algorithms featuring communication across a network or hypergraphs [48, 91, 41] and distributed settings [29, 77]. Cooperative MARL manifests primarily in two categories: multi-agent MDPs [12, 89, 79, 24] and parallel MDPs [11, 24, 53, 11, 56]. In the realm of cooperative multi-agent robotics, the former is employed to formulate optimal multi-agent policies across the distributed system [86, 87]. On the other hand, homogeneous parallel MDPs leverage inter-agent communication to expedite learning processes [43]. Additionally, heterogeneous parallel MDPs establish connections to heterogeneous federated learning [51] and exhibit improved generalizability in transfer learning scenarios [72].

We focus on parallel MDPs in this paper, where agents interact with the environment simultaneously to tackle shared challenges within extensive and distributed systems [43]. Recently, Dubey and Pentland [24] proposed the Coop-LSVI algorithm, extending the LSVI-UCB algorithm [36] in singleagent RL to MARL with linear MDPs. In a parallel RL setting with asynchronous communication, Min et al. [56] builds upon Coop-LSVI while relinquishing compatibility with heterogeneous MDPs. Meanwhile, Lidard et al. [53] focuses on fully decentralized multi-agent UCB Q -learning in a tabular setting, maintaining polynomial space complexity even as the number of agents increases. However, it is worth noting that neither of the previous works [24, 56] in non-tabular cooperative MRAL provides experimental validation for the efficacy of their proposed communication strategies. The gap arises from their reliance on LSVI-UCB as the core algorithm, wherein optimism is instantiated through UCB. Empirical evidence suggests that UCB-based approaches tend to underperform in practical scenarios [61, 60, 33]. Moreover, the computational demands of LSVI-UCB become untenable due to the necessity of recurrently computing the feature covariance matrix for updating the UCB bonus function. On the other hand, distributed applications of parallel MDPs in TS-based concurrent RL algorithms have been explored [21, 22, 17]. Specifically, Dimakopoulou and Roy [21] proposed a tabular model learning method based on seed sampling for coordinated exploration. This approach was further generalized to address intractable state spaces in [22] and supported by a Bayesian regret bound in [17]. However, none of these studies consider the communication complexity associated with efficient cooperative strategies. Therefore, randomized exploration in this work is critical to make these algorithm designs practical.

Randomized Exploration The roots of randomized exploration, particularly TS, can be traced back to its success in bandit problems [73]. Randomized exploration strategies can typically exhibit superior performance in practical applications due to avoidance of early convergence to suboptimal actions [38-40]. Furthermore, these strategies demonstrate robustness in the face of noise and uncertainty, particularly within non-stationary environments [78, 9]. This success has extended to Langevin Monte Carlo Thompson Sampling (LMCTS), which has been applied to various domains, including linear bandits, generalized linear bandits, and neural contextual bandits [83]. The exploration of posterior sampling techniques in RL has gained prominence, building upon the foundation laid by TS [71, 7]. Randomized Least-Square Value Iteration (RLSVI) is an approach that leverages random perturbations to approximate the posterior, with frequentist regret analysis applied under the tabular MDP setting [63], inspiring subsequent works focusing on theoretical analyses aimed at improving worst-case regret under tabular MDPs [70, 4], with extensions to the linear setting [88, 32, 20]. In addition to theoretical advancements, several practical algorithms have been proposed based on RLSVI to approximate posterior samples of Q functions in deep RL. These approaches involve ensembles of randomly initialized neural networks [62, 64] and noise injection into the parameters of the neural network [26, 52]. With the success of LMCTS [83] in bandit domains, the exploration of randomized methods has expanded to alternative approaches like LMC in tabular RL [42] and linear MDPs with neural network approximation [33]. Further works delve into the realm of random exploration from the perspectives of delayed feedback [44] and offline RL [59].

While posterior sampling demonstrates superiority in various contexts, its theoretical foundations in the multi-agent setting remain underexplored. Existing research predominantly focuses on two-player zero-sum games, considering both Bayesian [95, 35] and frequentist regrets [80, 65]. There is no existing work studying randomized exploration for cooperative multi-agent settings.

B Instantiation of the Proposed Algorithms in the Linear Function Class

In this section, we specifically discuss our TS-related algorithms in the linear structure, which is under the assumption of linear function approximation and linear MDP setting.

Recall from the loss function in (3.4), here we choose L to be l 2 loss and linear function class f ( w ; ϕ l ) = w ⊤ ϕ l . By solving this least-square regression problem, we obtain the unperturbed regression estimator ̂ w k m,h . In the linear setting, we have the closed-form solution

where Λ k m,h and b k m,h are defined as follows

A natural way of doing randomized exploration is to add a noise N ( 0 , σ 2 ( Λ k m,h ) -1 ) to ̂ w k m,h and get the estimated parameter ¯ w k m,h . Then we can construct estimated Q function Q k m,h ( · , · ) = min { ϕ ( · , · ) ⊤ ¯ w k m,h , H -h + 1 } + . We call this method as CoopTS, which is aligned with other linear TS algorithms [6, 2]. In what follows, we theoretically show that our proposed algorithms are equivalent or approximately converge to the CoopTS algorithm in the linear function approximation setting.

For CoopTS-PHE (Algorithm 1+Algorithm 2), let the function approximation in (3.5) be linear and choose L to be the squared loss. By solving this least-square regression problem, we obtain the perturbed regression estimator ˜ w k,n m,h in CoopTS-PHE. The following proposition conveys that CoopTS-PHE is actually equivalent to CoopTS.

Proposition B.1 (Equivalent to CoopTS) . The output ˜ w k,n m,h by CoopTS-PHE is equivalent to adding a Gaussian vector to the unperturbed regression estimator ̂ w k m,h , i.e., ˜ w k,n m,h = ̂ w k m,h + ζ k,n m,h , where ζ k,n m,h ∼ N ( 0 , σ 2 ( Λ k m,h ) -1 ) .

For CoopTS-LMC (Algorithm 1+Algorithm 3), let function approximation in (3.4) be linear and choose L to be l 2 loss to get the loss function. Then after finishing the LMC update, we get the estimated parameter w k,J k ,n m,h and construct the model approximation of Q function. The following proposition conveys that the distribution of w k,J k m,h converges to the posterior distribution of Thompson Sampling exploration. The proof of this proposition is given in [83].

Proposition B.2 (Approximately equivalent to CoopTS [83]) . If the epoch length J k in Algorithm 3 is sufficiently large, the distribution of w k,J k m,h converges to Gaussian distribution N ( w k , β -1 ( Λ k ) -1 ) .

Propositions B.1 and B.2 indicate that the results of our two randomized exploration strategies are closely related to CoopTS. As we have mentioned above, in CoopTS, the estimated parameter ¯ w k m,h is sampled from the normal distribution N ( ̂ w k m,h , σ 2 ( Λ k m,h ) -1 ) . However, in practice, this sampling is often executed in this way: we sample β ∼ N ( 0 , I ) first, then we calculate ¯ w k m,h = ̂ w k m,h + σ ( Λ k m,h ) -1 2 β and obtain the estimated parameter. Nevertheless, computing ( Λ k m,h ) -1 2 can be computationally expensive, often requiring at least O ( d 3 ) operations with the Cholesky decomposition, making it impractical for high-dimensional machine learning challenges. Additionally, the Gaussian distribution used in Thompson Sampling may not effectively approximate the posterior distribution in more complex bandit models than the linear MDP due to their intricate structures.

Moreover, as pointed out by recent work [16, 66, 46, 83], the Laplace approximation-based Thompson Sampling exhibits a constant approximation error in the estimation of the posterior distribution. Therefore, it necessitates a careful redesign of the covariance matrix to ensure effective performance.

Advantages of PHE and LMC As mentioned above, computing ( Λ k m,h ) -1 2 can be computationally expensive. However, Perturbed-History exploration and Langevin Monte Carlo exploration can avoid this. For PHE, by only adding i.i.d random Gaussian noise to perturb reward and regularizer, its performance will be equivalent to TS. For LMC, by only performing noisy gradient descent, we can do the randomized exploration, resulting in similar performance compared with TS. Additionally, these two methods can easily be implemented to general function class while Thompson Sampling usually cannot be generalized except for the linear setting. In summary, these two methods are both flexible in design and easy to implement in practice.

Communication cost We emphasize that agents can just send compressed statistics to the server under the linear setting, which can largely reduce communication cost. In the linear function class, we can calculate the closed-form solution of the regression problem (B.1). In this case, when synchronization process is met, all the agents will only need to send their calculated local statistics loc Λ k m,h and loc b k m,h to help solve the regression problem. This communication cost is much smaller because Λ is only a d × d matrix and b is only a d -dimensional vector, where d is the feature dimension in linear MDP assumption. This can also avoid privacy disclosure through communications.

Nevertheless, in the general function class setting, our proposed algorithms still require sharing all the collected datasets, which will cause relatively large communication cost. Additionally, in Appendix K.2, we also propose a federated setting algorithm Algorithm 4. In this setting, instead of sharing collected datasets, agents can just share the weight of the collected estimated Q functions, which can largely reduce the communication cost.

C Analysis of the Communication Complexity of Algorithm 1

The proof of the communication complexity is largely inspired by that in [24]. However, we provide a refined analysis here, and thus obtain an improved communication complexity ˜ O ( dHM 2 ) , in contrast with the ˜ O ( dHM 3 ) complexity in their paper. We also discussed this in Remark 4.7 and showed that our result matches that of a recently proposed asynchronous algorithm. Moreover, we do a careful calculation of the total number of transferred random bits and show it only has a dependence on the number of episodes K .

Proof of Lemma 4.6. Weassume σ = { σ 1 , . . . , σ n } as the synchronization episodes, where σ i ∈ [ K ] , we also denote σ 0 = 0 . To bound the number of synchronization n , we separate σ into two parts with an undetermined term α

Then we have n = | I 1 | + | I 2 | . Note that

Then we have | I 2 | < K/α . Then note that

Define Λ K h = ∑ m ∈M ∑ K k =1 ϕ ( z k m,h ) ϕ ( z k m,h ) ⊤ + λ I where z k m,h = ( s k m,h , a k m,h ) . On the other hand, we have

where the first inequality holds due to the trivial fact that A ≼ B ⇒ det( A ) ≤ det( B ) , the second inequality follow from Lemma J.2 and the fact that ∥ ϕ ( · ) ∥ 2 ≤ 1 . Combine (C.1) and (C.2), then we have | I 1 | ≤ dα/γ log(1 + MK/d ) . Finally, we choose α = K/d , then we have

When one synchronization occurs, communications between agents and the server will occur M times because we have M agents in total. Recall from Lines 16-24 in Algorithm 1, also note that in one synchronization episode, communications will happen H times between every agent and the server. Finally, the upper bound of communication complexity is

Next we consider the total number of transferred random bits. We first calculate the communication bits per round. Under the linear setting, we can calculate the closed-form solution of the regression problem ˆ w k m,h = ( Λ k m,h ) -1 b k m,h , where

Note that l ∈ [ K ( k )] is equivalent to ( s, a, s ′ ) ∈ U m,h ( k ) , and the index set U m,h ( k ) consists of U ser h ( k ) and U loc m,h ( k ) . Therefore, the empirical covariance matrix Λ k m,h and the vector b k m,h can be decomposed into the summation of the local matrices and vectors on each agent. When the synchronization occurs, agents just need to send their local statistics loc Λ k m,h and loc b k m,h to the server to help solve the regression problem on each agent.

For the local empirical covariance matrix loc Λ k m,h

this is the summation of up to K d × d matrices. Note that ∥ ϕ ( s, a ) ∥ ≤ 1 , thus it is easy to see that the entries of each matrix, namely, ϕ ( s l , a l ) ϕ ( s l , a l ) ⊤ , are bounded by 1 . Therefore, the entries in loc Λ k m,h are bounded by K . For each entry in this matrix, it suffices to use O (log K ) bits to communicate between the server and the agent. Thus in each round, O ( d 2 log K ) bits are needed to send the matrix loc Λ k m,h .

For the local vector loc b k m,h

this is a d -dimensional vector. Note that r h is bounded by 1 , V k m,h +1 is bounded by H and is linear with ϕ by definition, which indicates we only need to communicate a d -dimensional vector ¯ w k m,h to obtain V k m,h +1 . Similar to the above analysis, in each round, O ( d log( K ( H +1))) bits are needed to send the vector loc b k m,h .

Therefore, the total bits of communication still only has a logarithmic dependency on the number of episodes K . This completes the proof.

D Proof of the Regret Bound for CoopTS-LMC

The general framework for CoopTS-LMC and CoopTS-PHE is closely similar. To make the article more concise, we first prove CoopTS-LMC completely, which is a bit more complicated. Then we can simplify the following similar proof for CoopTS-PHE in Appendix G.

D.1 Supporting Lemmas

Before deriving the regret bound for CoopTS-LMC, we first provide the necessary technical lemmas for our regret analysis. Note that the loop (Line 3-9) in Algorithm 3 is to do multi-sampling for N times. To simplify the notations, we eliminate the index n before Lemma D.7 because the previous lemmas have nothing to do with multi-sampling.

Definition D.1 (Model prediction error) . For any ( m,k,h ) ∈ M× [ K ] × [ H ] , we define the model error associated with the reward r h ,

Definition D.2 (Filtration) . For any ( m,k,h ) ∈ M× [ K ] × [ H ] , we define the filtration F m,k,h as F m,k,h = σ ( {( s τ n,i , a τ n,i )} ( n,τ,i ) ∈M× [ k -1] × [ H ] ⋃{( s k n,i , a k n,i )} ( n,i ) ∈ [ m -1] × [ H ] ⋃{( s k m,i , a k m,i )} i ∈ [ h ] ) .

Proposition D.3. In Algorithm 3, the parameter w k,J k m,h satisfies the Gaussian distribution N ( µ k,J k m,h , Σ k,J k m,h ) , where mean vector and the covariance matrix are defined as

where A i = I -2 η m,i Λ i m,h for i ∈ [ k ] .

Lemma D.4. For any ( m,k,h ) ∈ M × [ K ] × [ H ] , the unperturbed estimated parameter ̂ w k m,h satisfies

Lemma D.5. Let λ = 1 in Algorithm 3. For any fixed 0 < δ < 1 , with probability at least 1 -δ 2 , for any ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

Lemma D.6. Let λ = 1 in Algorithm 3. For any fixed 0 < δ < 1 , with probability at least 1 -δ , for any ( m,k,h ) ∈ M× [ K ] × [ H ] , we have

Lemma D.7. Let λ = 1 in Algorithm 3. For any fixed 0 < δ < 1 , with probability at least 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] , we have

where C δ = [ 1 2 log( K +1) + log ( 2 √ 2 KB δ/ 2 NMHK H ) +log 3 δ ] 1 / 2 and B δ is defined in Lemma D.6.

Lemma D.8. Let λ = 1 in Algorithm 3. Under Definition 4.1, for any fixed 0 < δ < 1 , with probability at least 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

Lemma D.9 (Error bound) . Let λ = 1 in Algorithm 3. Under Definition 4.1, for any fixed 0 < δ < 1 , with probability at least 1 -δ -δ 2 , for any ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

where C δ is defined in Lemma D.7.

Lemma D.10 (Optimism) . Let λ = 1 in Algorithm 3 and c ′ 0 = 1 -1 2 √ 2 eπ . Under Definition 4.1, for any fixed 0 < δ < 1 , with probability at least 1 -|C ( ε ) | c ′ 0 N -2 δ where |C ( ε ) | ≤ (3 /ε ) d , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for all ( s, a ) ∈ S × A , we have

where α δ = √ MK ( 2 H √ d + B δ/NMHK ) .

Remark D.11. Here we point out that in our proofs for both CoopTS-LMC and CoopTS-PHE, we use a new ε -covering technique to prove that the optimism lemma holds for all ( s, a ) ∈ S × A instead of just the state-action pairs encountered by the algorithm, which is essential in applying this lemma to bound the term E π ∗ [ l k m,h ( s m,h , a m,h ) | s m, 1 = s k m, 1 ] in (D.2) in the regret analysis. This was ignored by previous works [13, 33] that use the same regret decomposition technique in the single-agent setting.

The following lemma gives the upper bound of self-normalized term summation in the multi-agent setting, which is first introduced by Lemma 9 in [24]. To make our analysis complete, we give out the proof in the Appendix E.9 where we make some necessary modifications compared with Lemma 9 in [24].

Lemma D.12. Let Algorithm 2 run for any K > 0 , M ≥ 1 , and γ as the communication control factor. Define Λ K h = ∑ m ∈M ∑ K k =1 ϕ ( s k m,h , a k m,h ) ϕ ( s k m,h , a k m,h ) ⊤ + λ I , then we have

The following lemma shows that we can decompose the regret of Algorithm 2 into three different components. The proof of this lemma closely resembles Lemma 4.2 in [13] for the single-agent setting. When we fix the agent m ∈ M , it is totally same as Lemma 4.2 in [13].

Lemma D.13. [13, Lemma 4.2] Define the operators and the following terms:

Then we can decompose the regret into the following form:

D.2 Regret Analysis

In this part, we give out the proof of Theorem 4.4, the regret bound for CoopTS-LMC.

Proof of Theorem 4.4. Based on the result from Lemma D.13, we do the regret decomposition first

Next, we will bound the above three terms respectively.

Bounding Term (i) in (D.2) : for the policy π k m,h , we have

This is because by definition π k m,h is the greedy policy for Q k m,h .

Bounding Term (ii) in (D.2) : note that 0 ≤ Q k m,h ≤ H -h + 1 ≤ H , based on (D.1), for any ( m,k,h ) ∈ M× [ K ] × [ H ] , we have | D m,k,h, 1 | ≤ 2 H and | D m,k,h, 2 | ≤ 2 H . Note that D m,k,h, 1 is a martingale difference sequence E [ D m,k,h, 1 |F m,k,h ] = 0 . By applying Azuma-Hoeffding inequality, with probability at least 1 -δ/ 3 , we have

Note that D m,k,h, 2 is also a martingale difference sequence. By applying Azuma-Hoeffding inequality, with probability at least 1 -δ/ 3 , we have

K

∑

k

∑

m

∈M

=1

H

∑

h

=1

By taking union bound, with probability at least 1 -2 δ/ 3 , we have

Bounding Term (iii) in (D.2) : based on Lemmas D.9 and D.10, by taking union bound, with probability at least 1 -|C ( ε ) | c ′ 0 N -2 δ ′ -MHK ( δ ′ + δ ′ 2 ) , we have

D

m,k,h,

2

2

√

2

MH

3

K

log(6

/δ

)

.

The first inequality follows from Lemma D.10, the second inequality follows from Lemma D.9, the third inequality follows from Lemma D.12, the last inequality holds due to Lemma J.2 and the fact that ∥ ϕ ( · ) ∥ 2 ≤ 1 .

Here we choose ε = dH √ d/MK/α δ ′ = ˜ O ( √ 1 /dHM 3 K 4 N ) and choose 1 √ β K = 20 H √ dC δ ′ + 16 3 , we have

occurs with probability at least 1 -|C ( ε ) | c ′ 0 N -2 δ ′ -MHK ( δ ′ + δ ′ 2 ) .

We set δ ′ = δ/ 12( MHK +1) and choose N = ¯ C log( δ ) / log( c ′ 0 ) where ¯ C = ˜ O ( d ) , then we have

Combining Terms (i)(ii)(iii) together: Based on (D.3), (D.4) and (D.5). By taking union bound, we get that the final regret bound for CoopTS-LMC is ˜ O ( dH 2 ( dM √ γ + √ dMK )) with probability at least 1 -δ .

E Proof of Supporting Lemmas in Appendix D

E.1 Proof of Proposition D.3

Recall from Algorithm 3, the LMC update rule is

where we have ∇ L k m,h ( w k,j -1 m,h ) = 2 ( Λ k m,h w k,j -1 m,h -b k m,h ) . Plug in the above formula, then we can calculate that

where the third equality follows from iteration. Denote that A i = I -2 η m,i Λ i m,h . Moreover, we choose the step size such that 0 < η m,i < 1 / ( 2 λ max ( Λ i m,h )) . Thus we have

where the first equality holds because b k m,h = Λ k m,h ̂ w k m,h and w k -1 ,J k -1 m,h = w k, 0 m,h , the third equality follows from the fact that I + A + ... + A n -1 = ( I -A n )( I -A ) -1 , and the fourth equality holds because of iteration.

Note that ϵ i,J i -l m,h ∼ N ( 0 , I ) , based on the property of multivariate Gaussian distribution, we have w k,J k m,h ∼ N ( µ k,J k m,h , Σ k,J k m,h ) . Then we can directly get the mean vector

Next we will calculate the covariance matrix Σ k,J k m,h . For simplicity, we define M i = √ 2 η m,i β -1 m,i A J k k ... A J i +1 i +1 , thus we have

Thus we get the covariance matrix Σ k,J k m,h ,

where the third equality follows from the fact that I + A + ... + A n -1 = ( I -A n )( I -A ) -1 . Here we complete the proof.

E.2 Proof of Lemma D.4

Proof. Note that ̂ w k m,h = ( Λ k m,k ) -1 b k m,h , we can calculate that

where the first inequality follows from Lemma J.3, the second inequality is due to 0 ≤ V k m,h ≤ H -h +1 , 0 ≤ r h ≤ 1 and ∥ ϕ ( s, a ) ∥ ≤ 1 , the third inequality follows from Lemma J.4, and the last inequality holds because K ( k ) = ( M -1) k s + k -1 ≤ Mk .

E.3 Proof of Lemma D.5

Proof. We separate the error into two terms and bound them respectively,

Bounding Term I 1 in (E.1) : by Cauchy-Schwarz inequality, we have

By choosing η m,k ≤ 1 / (4 λ max ( Λ k m,h )) for all k and m , then we have

Recall the definition of Σ k,J k m,h in Proposition D.3. By choosing β m,i = β K for all i ∈ [ k ] and m ∈ M , then we have

where the first inequality follows from (E.2). By the definition of Λ i m,h and Woodbury formula, we have

where φ is a matrix with the dimension of d × n , n is the number difference of ϕ ( s l , a l ) between ( Λ i m,h ) -1 and ( Λ i +1 m,h ) -1 ( i.e. we concatenate all ϕ ( s l , a l ) into the matrix φ ). Note that n ≤ M , then we have

where ∥·∥ F is Frobenius norm and the last inequality is due to ∥ Λ -1 2 X ∥ 2 F = tr( X ⊤ Λ -1 X ) and (E.2). Thus we have

Using the inequality a 2 + b 2 ≤ a + b for a, b > 0 , we get

Note that ( Σ k,J k m,h ) -1 / 2 ( w k,J k m,h -µ k,J k m,h ) ∼ N ( 0 , I d ) . By the Gaussian concentration property, we have

Then we have

Bounding Term I 2 in (E.1) : Recall from Proposition D.3, we have

Then we can get

In Algorithm 3, we choose w 1 , 0 m,h = 0 and ̂ w 1 m,h = ( Λ 1 m,h ) -1 b 1 m,h = 0 . Thus we have I 21 = 0 . To bound term I 22 , we use the inequalities in (E.2) and Lemma D.4, we have

Thus we get

Substituting (E.3) and (E.4) into (E.1), with probability at least 1 -δ 2 , we have

Here we choose η m,j = 1 / (4 λ max ( Λ j m,h )) and set κ j = λ max ( Λ j m,h ) /λ min ( Λ j m,h ) , then we have ( 1 -2 η m,j λ min ( Λ j m,h ) ) J j = (1 -1 / 2 κ j ) J j .

We want to have (1 -1 / 2 κ j ) J j < ϵ , it suffices to choose J j such that

Note that 1 / 2 κ j ≤ 1 / 2 , we have log(1 / (1 -1 / 2 κ j )) ≥ 1 / 2 κ j because e -x > 1 -x for 0 < x < 1 . Therefore, we only need to pick J j ≥ 2 κ j log(1 /ϵ ) .

Also note that 1 ≥ ∥ ϕ ( s, a ) ∥ ≥ √ λ ∥ ϕ ( s, a ) ∥ ( Λ i m,h ) -1 and tr ( φ ⊤ ( Λ i m,h ) -1 φ ) ≤ M due to the fact that n ≤ M . By setting ϵ = 1 / (4 HMKd ) and λ = 1 , we obtain

where the second inequality follows from ∥ ϕ ( s, a ) ∥ ( Λ k m,h ) -1 ≥ 1 / √ K ( k ) + 1 ∥ ϕ ( s, a ) ∥ ≥ 1 / √ MK ∥ ϕ ( s, a ) ∥ , the fourth inequality follows from ∑ k -1 i =1 ϵ k -i -1 = ∑ k -2 i =0 ϵ i < 1 / (1 -ϵ ) ≤ 4 / 3 . Finally we have

This completes the proof.

E.4 Proof of Lemma D.6

Proof. Recall that w k,J k m,h ∼ N ( µ k,J k m,h , Σ k,J k m,h ) . Let ξ k,J k m,h = w k,J k m,h -µ k,J k m,h ∼ N ( 0 , Σ k,J k m,h ) , thus we have

Bounding ∥ ∥ µ k,J k m,h ∥ ∥ in (E.6) : Based on Proposition D.3, we have

where the second inequality holds from Lemma D.4, the third inequality follows from the fact that rank ( A J k k . . . A J i +1 i +1 ( I -A J i i )) ≤ d and ∥ X ∥ 2 ≤ ∥ X ∥ F ≤ rank( X ) ∥ X ∥ 2 where ∥ X ∥ 2 = σ max ( X ) .

Recall that in Lemma D.5, we set J j ≥ 2 κ j log(1 /ϵ ) where κ j = λ max ( Λ j m,h ) /λ min ( Λ j m,h ) , ϵ = 1 / (4 HMKd ) and λ = 1 , thus we get

Bounding ∥ ∥ ξ k,J k m,h ∥ ∥ in (E.6) : Note that ξ k,J k m,h ∼ N ( 0 , Σ k,J k m,h ) , using Gaussian concentration Lemma J.5, we have

Recall from Proposition D.3, we have

where the inequality holds due to Lemma J.6. Recall from (E.2) that, when η m,k ≤ 1 / (4 λ max ( Λ k m,h )) for all k and m , we have A J i i ≼ (1 -2 η m,k λ min ( Λ k m,h )) J j I , set λ = 1 , then we obtain

Similarly, we have I -A 2 J i i ≼ ( 1 -1 2 2 J i ) I , then we get

Also, based on ( I + A i ) -1 ≼ 2 3 I , we have

Note that λ max (( Λ i m,h ) -1 ) ≤ 1 , we have

Combine the above results together and choose β m,i = β K for all i ∈ [ K ] and m ∈ M , we have

Then we have

Combine above results together: with probability at least 1 -δ , we have

This completes the proof.

E.5 Proof of Lemma D.7

Proof. Based on Lemma D.6, for any fixed n ∈ [ N ] , with probability at least 1 -δ , for any ( m,k,h ) ∈ M× [ K ] × [ H ] , we have

By taking union over n, m, k, h , we have for all ( m,k,h ) ∈ M × [ K ] × [ H ] and for all n ∈ [ N ] , with probability 1 -δ/ 2 , we have

Based on Lemma J.7 and Lemma J.9, we have that for any ε > 0 and δ > 0 , with probability at least 1 -δ/ 2 ,

Here we set λ = 1 , ε = H 2 √ 2 k , with probability at least 1 -δ/ 2 , we have

By applying union bound between (E.7) and (E.8), and define that C δ = [ 1 2 log( K +1) + log 3 δ + log ( 2 √ 2 KB δ/ 2 NMHK H )] 1 / 2 , finally we obtain that for all ( m,k,h ) ∈ M× [ K ] × [ H ] ,

with probability at least 1 -δ .

E.6 Proof of Lemma D.8

Proof. We denote the inner product over S by ⟨· , ·⟩ S . Based on P h ( ·| s, a ) = 〈 ϕ ( s, a ) , µ h ( · ) 〉 S in Definition 4.1, we have

Here the last equality uses P h ( ·| s, a ) = 〈 ϕ ( s, a ) , µ h ( · ) 〉 S again. Then we can separate the following error into three parts,

Here the first equality holds due to (E.9). We now provide an upper bound for each of the terms in (E.10).

Bounding Term (i) in (E.10) : using Cauchy-Schwarz inequality and Lemma D.7, with probability at least 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

Bounding Term (ii) in (E.10) : we first note that

where the first and fourth equality holds due to the definition r h ( s, a ) = 〈 ϕ ( s, a ) , θ h 〉 from Definition 4.1, the third equality uses the definition of Λ k m,h . Next we can obtain that

where we use the fact that λ max (( Λ k m,h ) -1 ) ≤ 1 /λ and ∥ θ h ∥ ≤ √ d from Definition 4.1. By Combining (E.12) and (E.13), we obtain

Bounding Term (iii) in (E.10) : we have

where the second inequality holds due to the fact that λ max (( Λ k m,h ) -1 ) ≤ 1 /λ , the third inequality uses the fact that V k m,h +1 ≤ H and the last inequality follows from ∥ µ h ∥ ≤ √ d in Definition 4.1.

Combine Terms (i)(ii)(iii) together: combine (E.11), (E.14) and (E.15), then set λ = 1 , with probability at least 1 -δ , we get

This completes the proof.

E.7 Proof of Lemma D.9

Proof. Recall from Definition D.1,

Bounding Term I 1 : based on Lemma D.5, for any fixed n ∈ [ N ] , for any ( m,h,k ) ∈ M× [ H ] × [ K ] and for any ( s, a ) ∈ S × A , with probability at least 1 -δ 2 , we have

By taking union bound over n , we have for all n ∈ [ N ] , with probability 1 -δ 2 , we have

This indicates, for any ( m,h,k ) ∈ M × [ H ] × [ K ] and ( s, a ) ∈ S × A , with probability at least 1 -δ 2 , we have

Bounding Term I 2 : based on Lemma D.8, with probability at least 1 -δ , for any ( m,h,k ) ∈ M× [ H ] × [ K ] and ( s, a ) ∈ S × A , we have

Combine the two result above, by taking union bound, with probability at least 1 -δ -δ 2 , for any ( m,h,k ) ∈ M× [ H ] × [ K ] and ( s, a ) ∈ S × A , we have

This completes the proof.

E.8 Proof of Lemma D.10

Proof. Recall from Definition D.1,

Note that

Note that ∥ ϕ ( s, a ) ∥ ( Λ k m,h ) -1 ≤ √ 1 /λ ∥ ϕ ( s, a ) ∥ ≤ 1 for all ϕ ( s, a ) . Define C ( ε ) to be a ε -cover of { ϕ | ∥ ϕ ∥ ( Λ k m,h ) -1 ≤ 1 } . Based on Lemma J.8, we have |C ( ε ) | ≤ (3 /ε ) d .

First, for any fixed ϕ ( s, a ) ∈ C ( ε ) , based on the results in Proposition D.3, we have that ϕ ( s, a ) ⊤ w k,J k ,n m,h ∼ N ( ϕ ( s, a ) ⊤ µ k,J k m,h , ϕ ( s, a ) ⊤ Σ k,J k m,h ϕ ( s, a ) ) for any fixed n ∈ [ N ] . Now we define

Consider the numerator of Z k :

Based on Lemma D.8, with probablity at least 1 -δ , we have

From (E.4), we have

Recall the proof of Lemma D.5, we set η m,j = 1 / (4 λ max ( Λ j m,h )) , J j ≥ 2 κ j log(1 /ϵ ) , then we have for all j ∈ [ K ] , (1 -2 η m,j λ min ( Λ j m,h )) J j ≤ ϵ , set ϵ = 1 / 4 HMKd and λ = 1 , we have

So, with probablity at least 1 -δ , we have

Consider the denominator of Z k : recall from the definition of Σ k,J k m,h from Proposition D.3, then we have

where we used the fact that 1 2 I ≼ ( I + A k ) -1 . Then we have

By the definition of Λ i m,h and Woodbury formula, we have

where φ is a matrix with the dimension of d × n , n is the number difference of ϕ ( s l , a l ) between ( Λ i m,h ) -1 and ( Λ i +1 m,h ) -1 (i.e. we concatenate all ϕ ( s l , a l ) in to the matrix φ ). Note that n ≤ M , we have

where ∥·∥ F is Frobenius norm and the last inequality is due to ∥ Λ -1 2 X ∥ 2 F = tr( X ⊤ Λ -1 X ) and (E.2). Therefore, we have

Similar to the proof of Lemma D.5, note that tr ( φ ⊤ ( Λ i m,h ) -1 φ ) ≤ M , when we choose J j ≥ 2 κ j log(3 kM ) , we have

where we used the fact that λ min (( Λ k m,h ) -1 ) ≥ 1 /kM and ∥ ϕ ( s, a ) ∥ ( Λ k m,h ) -1 ≥ 1 / √ kM ∥ ϕ ( s, a ) ∥ . Therefore, according to (E.17) and (E.18), with probablity at least 1 -δ , it holds that

This completes the proof.

which implies | Z k | < 1 when 1 √ β K = 20 H √ dC δ + 16 3 .

Till now we have proved that for any fixed ϕ ( s, a ) ∈ C ( ε ) and for all ( m,h,k ) ∈ M× [ H ] × [ K ] , for any fixed n ∈ [ N ] , with probablity at least 1 -δ , we have

By taking union bound over n ∈ [ N ] , with probablity at least 1 -δ , we have

where c ′ 0 = 1 -1 2 √ 2 eπ . Therefore, for any fixed ϕ ( s, a ) ∈ C ( ε ) and for all ( m,h,k ) ∈ M× [ H ] × [ K ] , with probability at least (1 -δ ) ( 1 -c ′ 0 N ) > 1 -δ -c ′ 0 N , we have

Next for any ϕ = ϕ ( s, a ) , we can find ϕ ′ ∈ C ( ε ) such that ∥ ϕ -ϕ ′ ∥ ( Λ k m,h ) -1 ≤ ε . We define ∆ ϕ = ϕ -ϕ ′ . Recall from Definition 4.1, we have

where w k m,h = θ h + 〈 µ h , V k m,h +1 〉 S . Note that max {∥ µ h ( S ) ∥ , ∥ θ h ∥} ≤ √ d and V k m,h +1 ≤ H -h ≤ H , thus we have

Then we define the regression error ∆ w k m,h = w k m,h -w k,J k ,n m,h . Thus we have

By triangle inequality, with probability at least 1 -δ , we have

Denote α δ = √ MK ( 2 H √ d + B δ/NMHK ) . Then, for all ( m,h,k ) ∈ M × [ H ] × [ K ] , with probability at least 1 -δ , we have

Recall from (E.19), by taking union bound, with probability at least 1 - |C ( ε ) | c ′ 0 N -2 δ , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for all ( s, a ) ∈ S × A , we have

Finally, with probability at least 1 -|C ( ε ) | c ′ 0 N -2 δ , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for all ( s, a ) ∈ S × A , we have

E.9 Proof of Lemma D.12

Proof. For simplicity, we denote ( s k m,h , a k m,h ) as z k m,h . Then we consider the following mappings ( ν M , ν K ) : [ MK ] → [ M ] × [ K ] ,

where we set ν M ( τ ) = M if M | τ . Next, for any τ ≥ 0 , we define

We denote σ = { σ 1 , . . . , σ n } as the synchronization episodes, where σ i ∈ [ K ] , we also denote σ 0 = 0 . Then we separate the episodes k = 1 , . . . , K into two groups based on the following condition,

Note that the left inequality always holds due to ¯ Λ σ i -1 h ≼ ¯ Λ σ i h and the trivial fact that A ≼ B ⇒ det( A ) ≤ det( B ) . Then we define that I 1 = { k ∈ N + , k ∈ [ σ i -1 , σ i ) , ∀ i ∈ [ n ] | (E.20) is true } and I 2 = { k ∈ N + , k ∈ [ σ i -1 , σ i ) , ∀ i ∈ [ n ] | (E.20) is false } , then [ K ] = I 1 ∪ I 2 ∪ { K } . For any k ∈ [ σ i -1 , σ i ) and k ∈ I 1 , note that ¯ Λ σ i -1 h ≼ Λ k m,h ≼ ¯ Λ k h ≼ ¯ Λ σ i h , thus for any m ∈ M , we have

where the first inequality follows from Lemma J.12, the second inequality follows from the trivial fact that A ≼ B ⇒ det( A ) ≤ det( B ) , and the final inequality holds because k ∈ I 1 . Then we will bound the summation for k ∈ I 1 and k ∈ I 2 respectively.

where the first inequality follows from Cauchy-Schwarz inequality, the second inequality holds due to (E.21), the final equality follows from Lemma J.1 and Λ K h = ∑ m ∈M ∑ K k =1 ϕ ( s k m,h , a k m,h ) ϕ ( s k m,h , a k m,h ) ⊤ + λ I .

For any interval [ σ i -1 , σ i ) , define ∆ i = σ i -σ i -1 -1 , we calculate that

where the last inequality follows from the synchronization condition (3.3).

Define R h = ⌈ log ( det( Λ K h ) det( λ I ) )⌉ , note that σ n ≤ K , then we can find that

We can claim that I 2 has at most R h synchronization episodes, otherwise

which causes the contradiction. Thus I 2 has at most R h intervals, then we get

Finally, we can bound the total summation,

This completes the proof.

F Proof of the Regret Bound for CoopTS-LMC in Misspecified Setting

In this section, we prove the regret bound for CoopTS-LMC in the misspecified setting. The regret analysis, the essential supporting lemmas and their corresponding proofs are almost same as what we have presented in Appendix D and Appendix E. Here we mainly point out the differences of proof between these two settings.

F.1 Supporting Lemmas

Definition F.1 (Model prediction error) . For any ( m,k,h ) ∈ M× [ K ] × [ H ] , we define the model error associated with the reward r m,h ,

Lemma F.2. Let λ = 1 in Algorithm 3. Under Definition 4.8, for any fixed 0 < δ < 1 , with probability at least 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

where C δ is defined in Lemma D.7.

Proof of Lemma F.2. Recall from Definition 4.8, we have

where the first inequality follows from Cauchy-Schwarz inequality, the second inequality follows from the fact that ∥ V k m,h +1 ∥ ∞ ≤ H and P 2 , ∥ P 1 -P 2 ∥ TV = 1 2 ∑ x ∈ ω | P 1 ( x ) -P 2 ( x ) | = 1 2 ∥ P 1 -P 2 ∥ 1 for two distributions P 1 and P 2 , note that here we regard distribution as infinite dimensional vector, the third inequality follows from Definition 4.8. Define ∆ m, 1 = P m,h V k m,h +1 ( s, a ) -ϕ ( s, a ) ⊤ 〈 µ h , V k m,h +1 〉 S , thus | ∆ m, 1 | ≤ 2 Hζ . Then we have

Based on (F.1), we can separate the following error into four parts,

We now provide an upper bound for each of the terms in (F.3).

Bounding Term (i) in (F.3) : same as (E.11) in Appendix E.6, with probability at least 1 -δ , we have

√

Bounding Term (ii) + Term (iv) in (F.3) : define ∆ m, 2 = r m,h ( s, a ) -ϕ ( s, a ) ⊤ θ h , then we have | ∆ m, 2 | ≤ ζ due to Definition 4.8. Next we have

where the third equality uses the definition of Λ k m,h . By Combining (F.5) and (E.13) in Appendix E.6, we obtain

Then we calculate that

where the second inequality follows from Cauchy-Schwarz inequality and the fact that | ∆ m, 1 + ∆ m, 2 | ≤ | ∆ m, 1 | + | ∆ m, 2 | ≤ 2 Hζ + ζ ≤ 3 Hζ , the third inequality holds because of CauchySchwarz inequality, and the last inequality holds because K ( k ) ≤ MK and Lemma J.4. Substitute (F.7) into (F.6), we have

Bounding Term (iii) in (F.3) : same as (E.15) in Appendix E.6, we have

Combine all the terms in (F.3) together: by using triangle inequality in (F.3), we combine (F.4), (F.8) and (F.9), then set λ = 1 , with probability at least 1 -δ , we get

This completes the proof.

Lemma F.3 (Error bound) . Let λ = 1 in Algorithm 3. Under Definition 4.8, for any fixed 0 < δ < 1 , with probability at least 1 -δ -δ 2 , for any ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

where C δ is defined in Lemma D.7.

Proof of Lemma F.3. We do the same process as that in Appendix E.7, and we have

Bounding Term (i): based on (E.16), for any ( m,h,k ) ∈ M× [ H ] × [ K ] and ( s, a ) ∈ S × A , with probability at least 1 -δ 2 , we have

Bounding Term (ii): based on Lemma F.2, for all ( m,h,k ) ∈ M× [ H ] × [ K ] and ( s, a ) ∈ S × A , we have

Combine the two result above, by taking union bound, with probability at least 1 -δ -δ 2 , we have

This completes the proof.

Lemma F.4 (Optimism) . Let λ = 1 in Algorithm 3 and c ′ 0 = 1 -1 2 √ 2 eπ . Under Definition 4.8, for any fixed 0 < δ < 1 , with probability at least 1 -|C ( ε ) | c ′ 0 N -2 δ where |C ( ε ) | ≤ (3 /ε ) d , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for all ( s, a ) ∈ S × A , we have

where α δ = √ MK ( 2 H √ d + B δ/NMHK ) .

Proof of Lemma F.4. This proof is similar to the proof in Appendix E.8, we just prove the part that for fixed ϕ ∈ C ( ε ) . Recall from Definition F.1,

Note that

Here we define

where ∆ m, 1 = P m,h V k m,h +1 ( s, a ) -ϕ ( s, a ) ⊤ 〈 µ h , V k m,h +1 〉 S , ∆ m, 2 = r m,h ( s, a ) -ϕ ( s, a ) ⊤ θ h . Based on the results in Proposition D.3, we have that ϕ ( s, a ) ⊤ w k,J k ,n m,h ∼ N ( ϕ ( s, a ) ⊤ µ k,J k m,h , ϕ ( s, a ) ⊤ Σ k,J k m,h ϕ ( s, a ) ) , for any fixed n ∈ [ N ] . When | Z k | < 1 , by Gaussian concentration Lemma J.10, we have

Consider the numerator of Z k :

Bounding Term I 1 in (F.10) : recall the proof of Lemma F.2, we do the almost same error decomposition as (F.3) with the only difference of adding term (∆ m, 1 +∆ m, 2 )

We now provide an upper bound for each of the terms in (F.11).

Bounding Term (i) in (F.11): almost same as (E.11) in Appendix E.6 with the only difference between P h and P m,h , with probability at least 1 -δ , we have

Bounding Term (ii) + Term (iv) in (F.11) : we do the same calculation as that in the proof of Lemma F.2, based on (F.5), we have

By Combining (F.13) and (E.13) in Appendix E.6, we obtain

Then we calculate that

where the first inequality follows from Cauchy-Schwarz inequality and the fact that | ∆ m, 1 +∆ m, 2 | ≤ 3 Hζ , the second inequality holds because of Cauchy-Schwarz inequality, and the last inequality holds because K ( k ) ≤ MK and Lemma J.4. Substitute (F.15) into (F.14), we have

Bounding Term (iii) in (F.11) : same as (E.15) in Appendix E.6, we have

Combine all the terms in (F.11) together: by using triangle inequality in (F.11), we combine (F.12), (F.16) and (F.17), then set λ = 1 , with probability at least 1 -δ , we get

Bounding Term I 2 in (F.10) : same as the proof in Appendix E.8, we have

So, with probability at least 1 -δ , we have

Consider the denominator of Z k : same as the proof in Appendix E.8, with (E.18), we have

where we used the fact that λ min (( Λ k m,h ) -1 ) ≥ 1 /k and ∥ ϕ ( s, a ) ∥ ( Λ k m,h ) -1 ≥ 1 / √ k ∥ ϕ ( s, a ) ∥ 2 . Therefore, according to (F.18) and (F.19), with probability at least 1 -δ , it holds that

which implies | Z k | < 1 when 1 √ β K = 20 H √ dC δ +12 Hζ √ MKd + 16 3 .

Now we have already proved that, for any fixed n ∈ [ N ] , with probability at least 1 -δ , we have

By taking union bound over n ∈ [ N ] , with probablity at least 1 -δ , we have

where c ′ 0 = 1 -1 2 √ 2 eπ . Finally, with probability at least (1 -δ ) ( 1 -c ′ 0 N ) , for all ( s, a ) ∈ S × A , we have

Till now we have completed the proof of fixed ϕ ∈ C ( ε ) . Follow the proof in Appendix E.8, we can get the final result.

F.2 Regret Analysis

In this part, we give out the proof of Theorem 4.12, the regret bound for CoopTS-LMC in the misspecified setting.

Proof of Theorem 4.12. This proof is almost same as the proof in Appendix D.2. We do the same regret decomposition (D.2) and obtain the same bound for Term(i) (D.3) and Term(ii) (D.4). Next we bound Term (iii) with new lemmas in the misspecified setting.

Bounding Term (iii) in (D.2) : based on Lemma F.3 and Lemma F.4, by taking union bound, with probability at least 1 -|C ( ε ) | c ′ 0 N -2 δ ′ -MHK ( δ ′ + δ ′ 2 ) , we have

The first inequality follows from Lemma F.4, the second inequality follows from Lemma F.3, the third inequality follows from Lemma D.12, the last inequality holds due to Lemma J.2 and the fact that ∥ ϕ ( · ) ∥ 2 ≤ 1 , the last equality follows from 1 √ β K = 20 H √ dC δ ′ +12 Hζ √ MKd + 16 3 , which we define in Lemma F.4.

The probability calculation is same as that in Appendix D.2. By combining Terms (i)(ii)(iii) together, we get that the final regret bound for CoopTS-LMC in misspecified setting is

with probability at least 1 -δ . Here we finish the proof.

G Proof of the Regret Bound for CoopTS-PHE

Before getting the regret bound for CoopTS-PHE, we first present some essential technical lemmas required for our analysis.

G.1 Supporting Lemmas

Proposition G.1. The difference between the perturbed estimated parameter ˜ w k,n m,h and unperturbed estimated parameter ̂ w k m,h satisfies the Gaussian distribution,

where ̂ w k m,h = ( Λ k m,h ) -1 ( ∑ ( s l ,a l ,s ′ l ) ∈ U m,h ( k ) [ r h + V k m,h +1 ( s ′ l )] ϕ ( s l , a l ) ) is the unperturbed estimated parameter.

Next we will define some good events that hold with high probability to help prove the critical lemmas in this section.

Lemma G.2 (Good events) . For any fixed 0 < δ < 1 , with some constant c > 0 , we define the following random events

where c 1 = c √ log( dNMKH/δ ) . Then the event G ( M,K,H,δ ) occurs with probability at least 1 -δ .

Lemma G.3. Let λ = 1 in Algorithm 2. For any fixed 0 < δ < 1 , conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] , we have

where we define D δ = [ 1 2 log( K +1) + log ( 6 √ 2 K (2 H √ MKd + c 1 σ √ d ) H ) +log 1 δ ] 1 / 2 .

Lemma G.4. Let λ = 1 in Algorithm 2. Under Definition 4.1, for any fixed 0 < δ < 1 , conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

Lemma G.5 (Optimism) . Let λ = 1 in Algorithm 2 and set c 0 = Φ(1) . Under Definition 4.1, conditioned on the event G ( M,K,H,δ ) , with probability at least 1 -|C ( ε ) | c N 0 -δ where |C ( ε ) | ≤ (3 /ε ) d , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for all ( s, a ) ∈ S × A , we have

where A δ = c 1 σ √ d +5 H √ dD δ = ˜ O ( Hd ) .

Lemma G.6 (Error bound) . Let λ = 1 in Algorithm 2. Under Definition 4.1, for any fixed 0 < δ < 1 , conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

where c 2 = ˜ O (1) .

G.2 Regret Analysis

In this part, we give out the proof of Theorem 4.2, the regret bound for CoopTS-PHE.

Proof of Theorem 4.2. Based on the result from Lemma D.13, we do the regret decomposition first

Next, we will bound the above three terms respectively.

Bounding Term (i) in (G.2) : for the policy π k m,h , we have

This is because by definition π k m,h is the greedy policy for Q k m,h .

Bounding Term (ii) in (G.2) : note that 0 ≤ Q k m,h ≤ H -h + 1 ≤ H , based on (D.1), for any ( m,k,h ) ∈ M× [ K ] × [ H ] , we have | D m,k,h, 1 | ≤ 2 H and | D m,k,h, 2 | ≤ 2 H . Note that D m,k,h, 1 is a martingale difference sequence E [ D m,k,h, 1 |F m,k,h ] = 0 . By applying Azuma-Hoeffding inequality, with probability at least 1 -δ/ 3 , we have

Note that D m,k,h, 2 is also a martingale difference sequence. By applying Azuma-Hoeffding inequality, with probability at least 1 -δ/ 3 , we have

By taking union bound, with probability at least 1 -2 δ/ 3 , we have

Bounding Term (iii) in (G.2) : conditioned on the event G ( M,K,H,δ ′ ) , based on Lemma G.6 and Lemma G.5, by taking union bound, with probability at least 1 -|C ( ε ) | c N 0 -δ ′ -MHKδ ′ , we have

The first inequality follows from Lemma G.5, the second inequality holds due to Lemma G.6, the third inequality follows from Lemma D.12, the last inequality holds due to Lemma J.2 and the fact that ∥ ϕ ( · ) ∥ 2 ≤ 1 .

Here we choose ε = dH √ d/MK/A δ ′ = ˜ O ( √ d/MK ) . Conditioned on the event G ( M,K,H,δ ′ ) , we have

with probability at least 1 -|C ( ε ) | c N 0 -δ ′ -MHKδ ′ . Based on Lemma G.2, the event G ( M,K,H,δ ′ ) occurs with probability at least 1 -δ ′ . Therefore, (G.5) occurs with probability at least

We set δ ′ = δ/ 6( MHK +2) and choose N = ˜ C log( δ ) / log( c 0 ) where ˜ C = ˜ O ( d ) , then we have

Combining Terms (i)(ii)(iii) together: Based on (G.3), (G.4) and (G.5). By taking union bound, we get that the final regret bound for CoopTS-PHE is ˜ O ( dH 2 ( dM √ γ + √ dMK )) with probability at least 1 -δ .

H Proof of Supporting Lemmas in Appendix G

H.1 Proof of Proposition G.1

Proof. Based on (3.5), we can calculate that

where ̂ w k m,h = ( Λ k m,h ) -1 ( ∑ ( s l ,a l ,s ′ l ) ∈ U m,h ( k ) [ r h + V k m,h +1 ( s ′ l )] ϕ ( s l , a l ) ) is the unperturbed estimated parameter. Since ϵ k,l,n h ∼ N (0 , σ 2 ) , for l ∈ [ K ( k )] , based on the property of Gaussian distribution, we have

Since ξ k,n h ∼ N ( 0 , σ 2 I ) , we can calculate the covariance matrix of the second term in (H.1),

It is obvious that the mean of the second term in (H.1) is 0 . Thus, we have

This completes the proof.

H.2 Proof of Lemma G.2

Proof. Recall that in Proposition G.1, we have

By Lemma J.10, for fixed n ∈ [ N ] , with probability at least 1 -δ , we have

By applying union bound over N samples, we have

Now we define c 1 = c √ log( dNMKH/δ ) , and we define the event

Thus for any fixed m , h and k , the event G k m,h ( ζ , δ ) occurs with a probability of at least 1 -δ/MKH . By taking union bound over all ( m,h,k ) ∈ M× [ H ] × [ K ] , we have

This completes the proof.

H.3 Proof of Lemma G.3

Proof. Based on the result in Lemma D.4, for any ( m,h,k ) ∈ M× [ H ] × [ K ] , we have

By recalling the construction of Λ k m,h , it is trivial to find that λ min ( Λ k m,h ) ≥ λ . Conditioned on the event G ( M,K,H,δ ) , we have

Then by triangle inequality, for all n ∈ [ N ] , we obtain the upper bound

Based on the result from Lemma J.7 and Lemma J.9, we have that, for any ε > 0 , and for all ( m,k,h ) ∈ M× [ K ] × [ H ] , with probability at least 1 -δ , we have

Here we set λ = 1 , ε = H 2 √ 2 k , with probability at least 1 -δ , we have

where we define D δ = [ 1 2 log( K +1)+log ( 6 √ 2 K (2 H √ MKd + c 1 σ √ d ) H ) +log 1 δ ] 1 / 2 . Here we finish the proof.

H.4 Proof of Lemma G.4

Proof. This proof is almost same as the proof of Lemma D.8 in Appendix E.6. The only difference is the Term (i) in (E.10). Here based on Lemma G.6, conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , we have

Finally, conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

∣ ∣ ∣ ϕ ( s, a ) ⊤ ̂ w k m,h -r h ( s, a ) -P h V k m,h +1 ( s, a ) ∣ ∣ ∣ ≤ ( 3 H √ dD δ + H √ d + √ d ) ∥ ϕ ( s, a ) ∥ ( Λ k m,h ) -1 ≤ 5 H √ dD δ ∥ ϕ ( s, a ) ∥ ( Λ k m,k ) -1 . Here we finish the proof.

H.5 Proof of Lemma G.5

Proof. Recall from Definition 4.1, we have

where w k m,h = θ h + 〈 µ h , V k m,h +1 〉 S . Note that max {∥ µ h ( S ) ∥ , ∥ θ h ∥} ≤ √ d and V k m,h +1 ≤ H -h ≤ H , thus we have

Then we define the regression error ∆ w k m,h = w k m,h -̂ w k m,h . For any ( m,h,k ) ∈ M× [ H ] × [ K ] and any ( s, a ) ∈ S × A , we have

where the last inequality holds because | r h | ≤ 1 and V k m,h +1 ≤ H -h , this indicates r h ( s, a ) + P h V k m,h +1 ( s, a ) = ϕ ( s, a ) ⊤ w k m,h ≤ H -h +1 . Note that ∥ ϕ ( s, a ) ∥ ( Λ k m,h ) -1 ≤ √ 1 /λ ∥ ϕ ( s, a ) ∥ ≤ 1 for all ϕ ( s, a ) . Define C ( ε ) to be a ε -cover of { ϕ | ∥ ϕ ∥ ( Λ k m,h ) -1 ≤ 1 } . Based on Lemma J.8, we have |C ( ε ) | ≤ (3 /ε ) d .

First, for any fixed ϕ ( s, a ) ∈ C ( ε ) , we have

Use the property of Gaussian distribution, we obtain

By taking union bound over n ∈ [ N ] , we obtain

By applying union bound over C ( ε ) , with probability 1 -|C ( ε ) | c N 0 , for all ϕ ∈ C ( ε ) , we have

Then, for any ϕ = ϕ ( s, a ) , we can find ϕ ′ ∈ C ( ε ) such that ∥ ϕ -ϕ ′ ∥ ( Λ k m,h ) -1 ≤ ε . Define ∆ ϕ = ϕ -ϕ ′ , we have

Conditioned on the event G ( M,K,H,δ ) , we have

For any vector x ∈ R d , we have

where the third equality holds due to the definition of Λ k m,h . We set x = Λ k m,h ∆ w k m,h . By using Cauchy-Schwarz inequality, we have

This indicates that with probability at least 1 -δ , for all ( m,h,k ) ∈ M× [ H ] × [ K ] , we have

where the second inequality holds because of Lemma G.3. Then for all ( m,h,k ) ∈ M× [ H ] × [ K ] , with probability at least 1 -δ , (H.4) becomes

Now we choose σ = ˜ O ( H √ d ) and guarantee that σ > 5 H √ dD δ ≥ ∥ ∥ ∆ w k m,h ∥ ∥ Λ k m,h , this is achievable through calculation. Define A δ = c 1 σ √ d + 5 H √ dD δ = ˜ O ( Hd ) . Then, for all ( m,h,k ) ∈ M× [ H ] × [ K ] , with probability at least 1 -δ , we have

Recall from (H.3), by taking union bound, with probability at least 1 - |C ( ε ) | c N 0 -δ , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for all ( s, a ) ∈ S × A , we have

Finally, recall from (H.2), we have, with probability at least 1 -|C ( ε ) | c N 0 -δ , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for all ( s, a ) ∈ S × A , we have

This completes the proof.

H.6 Proof of Lemma G.6

Proof. Recall the definition of model prediction error in Definition D.1, we get

Based on Lemma G.4, conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

Conditioned on the event G ( M,K,H,δ ) , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for any ( s, a ) ∈ S × A , we have

Combine (H.5) and (H.6), then use σ defined in Lemma G.5. Conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for any ( s, a ) ∈ S × A , we get

where c 2 = ˜ O (1) . Here we completes the proof.

I Proof of the Regret Bound for CoopTS-PHE in Misspecified Setting

In this section, we prove the regret bound for CoopTS-PHE in the misspecified setting. The regret analysis, the essential supporting lemmas and their corresponding proofs are very similar to what we have presented in Appendix G and Appendix H. Here we mainly point out the differences of proof between these two settings.

I.1 Supporting Lemmas

Lemma I.1. Let λ = 1 in Algorithm 2. Under Definition 4.8, for any fixed 0 < δ < 1 , conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

where D δ is defined in Lemma G.3.

Proof of Lemma I.1. This proof is almost same as the proof of Lemma F.2, with the only difference in bounding Term(i) in (F.3). Here (F.4) becomes

Finally we can get the desired result.

Lemma I.2 (Optimism) . Let λ = 1 in Algorithm 2 and set c 0 = Φ(1) . Under Definition 4.8, conditioned on the event G ( M,K,H,δ ) , with probability at least 1 -|C ( ε ) | c N 0 -δ where |C ( ε ) | ≤ (3 /ε ) d , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for all ( s, a ) ∈ S × A , we have

where A δ = c 1 σ √ d +5 H √ dD δ = ˜ O ( Hd ) .

Proof of Lemma I.2. This proof is similar to the proof in Appendix H.5. In the previous part, we have defined

where | ∆ m, 1 | ≤ 2 Hζ and | ∆ m, 2 | ≤ ζ . Thus we have

where w k m,h = ⟨ µ h , V k m,h +1 〉 S + θ h . Then we define ∆ w k m,h = w k m,h -̂ w k m,h . For any ( m,h,k ) M× [ H ] × [ K ] and any ( s, a ) ∈ S × A , we have

In Appendix H.5, we have proved that with probability at least 1 -|C ( ε ) | c N 0 -δ , for all ( m,h,k ) ∈ M× [ H ] × [ K ] and for all ( s, a ) ∈ S × A , we have

Substitute it into (I.2), we can get the final result.

Lemma I.3 (Error bound) . Let λ = 1 in Algorithm 2. Under Definition 4.8, for any fixed 0 < δ < 1 , conditioned on the event G ( M,K,H,δ ) , with probability 1 -δ , for all ( m,k,h ) ∈ M× [ K ] × [ H ] and for any ( s, a ) ∈ S × A , we have

where c 2 = ˜ O (1) is same as that in Lemma G.6.

Proof of Lemma I.3. Similar to the proof in Appendix H.6, using (H.6) in Appendix H.6 and (I.1), we have

where c 2 = ˜ O (1) is same as that in Lemma G.6. Here we completes the proof.

I.2 Regret Analysis

In this part, we give out the proof of Theorem 4.10, the regret bound for CoopTS-PHE in the misspecified setting.

Proof of Theorem 4.10. This proof is almost same as the proof in Appendix G.2. We do the same regret decomposition (G.2) and obtain the same bound for Term (i) (G.3) and Term (ii) (G.4). Next we bound Term (iii) with new lemmas in misspecified setting.

Bounding Term (iii) in (G.2) : conditioned on the event G ( M,K,H,δ ′ ) , based on Lemma I.3 and Lemma I.2, by taking union bound, with probability at least 1 -|C ( ε ) | c ′ 0 N -δ ′ -MHKδ ′ , we have

The first inequality follows from Lemma I.2, the second inequality holds due to Lemma I.3, the third inequality follows from Lemma D.12, the last inequality holds due to Lemma J.2 and the fact that ∥ ϕ ( · ) ∥ 2 ≤ 1 , and again we choose ε = dH √ d/MK/A δ ′ = ˜ O ( √ d/MK ) .

The probability calculation is same as that in Appendix G.2. By combining Terms (i)(ii)(iii) together, we get that the final regret bound for CoopTS-PHE in misspecified setting is

with probability at least 1 -δ . Here we finish the proof.

J Auxiliary Lemmas

Lemma J.1. [1, Lemma 11] Let { X t } ∞ t =1 be a sequence in R d , V is d × d positive definite matrix and define ¯ V t = V + ∑ t s =1 X s X ⊤ s . Then, we have that

Further, if ∥ X t ∥ 2 ≤ L for all t , then

and finally, if λ min ( V ) ≥ max ( 1 , L 2 ) then

Lemma J.2. [1, Lemma 10] Suppose X 1 , X 2 , . . . , X t ∈ R d and for any 1 ≤ s ≤ t , ∥ X s ∥ 2 ≤ L . Let ¯ V t = λ I + ∑ t s =1 X s X ⊤ s for some λ > 0 . Then,

Lemma J.3. [32, Lemma D.5] Let A ∈ R d × d be a positive definite matrix where its largest eigenvalue λ max ( A ) ≤ λ . Let x 1 , ..., x k be k vectors in R d . Then it holds that

Lemma J.4. [36, Lemma D.1] Let Λ t = λ I + ∑ t i =1 ϕ i ϕ ⊤ i , where ϕ i ∈ R d and λ > 0 . Then it holds that

Lemma J.5. [33, Lemma D.1] Given a multivariate normal distribution X ∼ N ( 0 , Σ ) , we have,

Lemma J.6. [30] If A and B are positive semi-definite square matrices of the same size, then

Lemma J.7. [36, Lemma D.4] Let { s i } ∞ i =1 be a stochastic process on state space S with corresponding filtration {F i } ∞ i =1 . Let { ϕ i } ∞ i =1 be an R d -valued stochastic process where ϕ i ∈ F i -1 , and ∥ ϕ i ∥ ≤ 1 . Let Λ k = λ I + ∑ k i =1 ϕ i ϕ ⊤ i . Then for any δ > 0 , with probability at least 1 -δ , for all k ≥ 0 , and any V ∈ V with sup s ∈S | V ( s ) | ≤ H , we have

where N ε is the ε -covering number of V with respect to the distance dist( V, V ′ ) = sup s ∈S | V ( s ) -V ′ ( s ) | .

Lemma J.8. [75, Covering number of Euclidean ball] For any ε > 0 , N ε , the ε -covering number of the Euclidean ball of radius B > 0 in R d satisfies

Lemma J.9. Let V denote a class of functions mapping from S to R with the following parametric form

where the parameter w n satisifies ∥ w n ∥ ≤ B for all n ∈ [ N ] and for all ( x, a ) ∈ S × A , we have ∥ ϕ ( x, a ) ∥ ≤ 1 . Let N V ,ε be the ε -covering number of V with respect to the distance dist ( V, V ′ ) = sup s ∈S ∣ ∣ V ( s ) -V ′ ( s ) ∣ ∣ . Then

Proof. Consider any two functions V 1 , V 2 ∈ V with parameters { w n 1 } n ∈ [ N ] and { w n 2 } n ∈ [ N ] respectively. Then we have

Let N w ,ε denote the ε -covering number of { w ∈ R d | ∥ w ∥ ≤ B } . Then, Lemma J.8 implies

For any V 1 ∈ V , we consider its corresponding parameters { w n 1 } n ∈ [ N ] . For any n ∈ [ N ] , we can find w n 2 such that ∥ w n 1 -w n 2 ∥ ≤ ε , then we can get V 2 ∈ V with parameters { w n 2 } n ∈ [ N ] . Then we have dist( V 1 , V 2 ) ≤ max n ∈ [ N ] ∥ w n 1 -w n 2 ∥ ≤ ε . Thus, we have,

This completes the proof.

Lemma J.10. [3] Suppose Z is a Gaussian random variable Z ∼ N ( µ, σ 2 ) , where σ > 0 . For 0 ≤ z ≤ 1 , we have

And for z ≥ 1 , we have

Lemma J.11. [32, Lemma D.2] Consider a d -dimensional multivariate normal distribution N ( 0 , A Λ -1 ) where A is a scalar. Let η 1 , η 2 , . . . , η N be N independent samples from the distribution. Then for any δ > 0

where c is some absolute constant.

Lemma J.12. [1, Lemma 12] Let A , B and C be positive semi-definite matrices such that A = B + C . Then we have that

̸

r

1000

S1

S2

SN-1

SN

r=1

K Additional Experimental Details

We conduct comprehensive experiments investigating the exploration strategies for DQN under a multi-agent setting. For all the Q networks in our experiments, we use ReLU as our activation function. Given that all experiments are conducted under multi-agent settings unless explicitly specified as a single-agent or centralized scenario, we denote our methods: CoopTS-PHE as "PHE" and CoopTS-LMC as "LMC" in experimental contexts and figures. In addition to our methods, the baselines we selected are either commonly used (DQN [57], DDQN [28]) or with competitive empirical performance (Bootstrapped DQN [62], NoisyNet DQN [26]). Both Bootstrapped DQN and NoisyNet DQN are randomized exploration methods. Bootstrapped DQN uses finite ensembles to generate the randomized value functions and views them as approximate posterior samples of Q -value functions. NoisyNet DQN injects noise into the parameters of neural networks to aid efficient exploration. For those figures which aim to compare among different m agents within a single plot, we use Total Episodes to indicate the total number of training samples for a direct comparison. Note that the shaded areas on all figures represent the standard deviation.

K.1 N -chain

We commence by presenting the comprehensive results for N = 25 in Figure 4, illustrating that our randomized exploration methods exhibit greater suitability in realistic scenarios characterized by an increasing number of agents. This superiority is particularly evident under two potential circumstances: (1) where there are more limitations on computation or data access from each source in the real world, and (2) when parallel learning from multiple sources can significantly enhance runtime efficiency.

Subsequently, we provide a more comprehensive study to investigate the exploration capabilities facilitated by parallel training. Preliminary experiments are conducted with a reduced state space, specifically considering N = 10 . The study aims to investigate exploration capabilities across varying agent counts, specifically within the set m ∈ { 1 , 2 , 3 , 4 } .

We list the details of all swept hyper-parameters in N -chain for PHE and LMC in Table 2 and Table 3 respectively. Specifically, PHE is trained with reward noise ϵ k,l,n h = 10 -2 and regularizer noise

0

500

1000

1500

2000

Training Episodes per Agent

0

2

4

6

8

10

PHE

LMC

DQN

Bootstrapped DQN

NoisyNet DQN

DDQN

(a) m=2

0

500

1000

1500

2000

Training Episodes per Agent

0

2

4

6

8

10

PHE

LMC

DQN

Bootstrapped DQN

NoisyNet DQN

DDQN

(b) m=3

0

500

1000

1500

2000

Training Episodes per Agent

0

2

4

6

8

10

PHE

LMC

DQN

Bootstrapped DQN

NoisyNet DQN

DDQN

(c) m=4

ξ k,n h = 10 -3 in (3.5) and LMC is trained with β m,k = 10 2 and in (3.7) and optimized by Adam SGLD [33] with α 1 = 0 . 9 , α 2 = 0 . 999 and bias factor α = 0 . 1 . The final hyper-parameters used in N -chain are presented in Table 4.

Performance Consistency with Varying m In the investigation detailed in Figure 5, we explore parallel learning without inter-agent communication. Note that the x -axis implies the total training episodes from m agents. Consequently, while multiple agents engage in simultaneous policy learning, each agent independently formulates its policies without the exchange of transition information. The discernible trend in this scenario is that an increase in the number of agents sharing the total episodes results in a slower rate of policy learning. Notably, despite this temporal discrepancy, all learning trajectories eventually approximate convergence towards the optimal dashed line.

0

500

1000

1500

2000

2500

3000

Total Episodes

2

4

6

8

10

m = 1, N = 10

m = 2, N = 10

m = 3, N = 10

m = 4, N = 10

(a) PHE

0

500

1000

1500

2000

2500

3000

Total Episodes

0

2

4

6

8

10

m = 1, N = 10

m = 2, N = 10

m = 3, N = 10

m = 4, N = 10

(b) LMC

0

250

500

750

1000

1250

1500

Total Episodes

0

2

4

6

8

10

centralized, n = 10

no communication(2 agents), n = 10

linear(2 agents), n = 10

constant(2 agents), n = 10

exponential(2 agents), n = 10

(a) m=2

0

500

1000

1500

2000

Total Episodes

0

2

4

6

8

10

centralized, n = 10

no communication(3 agents), n = 10

linear(3 agents), n = 10

constant(3 agents), n = 10

exponential(3 agents), n = 10

(b) m=3

0

500

1000

1500

2000

2500

3000

Total Episodes

0

2

4

6

8

10

centralized, n = 10

no communication(4 agents), n = 10

linear(4 agents), n = 10

constant(4 agents), n = 10

exponential(4 agents), n = 10

(c) m=4

0

250

500

750

1000

1250

1500

1750

Total Episodes

0

2

4

6

8

10

centralized, n = 10

no communication(2 agents), n = 10

linear(2 agents), n = 10

constant(2 agents), n = 10

exponential(2 agents), n = 10

(d) m=2

0

500

1000

1500

2000

Total Episodes

0

2

4

6

8

10

centralized, n = 10

no communication(3 agents), n = 10

linear(3 agents), n = 10

constant(3 agents), n = 10

exponential(3 agents), n = 10

(e) m=3

0

500

1000

1500

2000

2500

3000

Total Episodes

0

2

4

6

8

10

centralized, n = 10

no communication(4 agents), n = 10

linear(4 agents), n = 10

constant(4 agents), n = 10

exponential(4 agents), n = 10

(f) m=4

0

200

400

600

800

1000

Total Episodes

0

2

4

6

8

10

LMC

PHE

NeuralTS

NeuralUCB

LinTS

LinUCB

(a) m=2

0

500

1000

1500

2000

Total Episodes

0

2

4

6

8

10

LMC

PHE

NeuralTS

NeuralUCB

LinTS

LinUCB

(b) m=3

0

500

1000

1500

2000

2500

3000

Total Episodes

0

2

4

6

8

10

LMC

PHE

NeuralTS

NeuralUCB

LinTS

LinUCB

(c) m=4

Different Synchronization Conditions To further demonstrate the efficiency of parallel learning with communication, we compare different synchronization conditions in Section 3.1. Specifically, we denote synchronization (1) in every constant step as constant , (2) following exponential function as exponential , and (3) based on (3.3) as linear . To have a fair comparison among different synchronization conditions, we firstly record the empirical number of synchronization via linear condition in average, and then we consider constant value for constant condition and select proper base b for exponential condition with a similar number of synchronization. Figure 6 illustrates that any synchronization condition can improve learning efficiency but still with centralized learning as an upper bound. Note that the x -axis implies the total training episodes from m agents.

Performance Compared with Bandit-inspired Methods Since one of our proposed random exploration strategies, PHE is a variant of approximation TS, it is fair for us to investigate the performance of other exploration methods from bandit algorithms with the integration of DQN. We mainly compare both TS and UCB under neural network ( i.e., NeuralTS [90] and NeuralUCB [94]) and linear ( i.e., LinTS [5] and LinUCB [49]) settings. We show that a performance gap exists between linear approaches and other neural-based methods even in a small-scale exploration problem with N = 10 in Figure 7. Note that the x -axis implies the total training episodes from m agents.

32_2

32_3

64_2

64_3

Neural Network

5

10

15

20

25

30

Computation Time (sec)

PHE

LMC

DQN

Bootstrapped DQN

NoisyNet DQN

DDQN

NeuralTS

NeuralUCB

0

1000

2000

3000

4000

5000

6000

Training Episodes per Agent

0

2

4

6

8

10

inv temp 1e0

inv temp 1e2

inv temp 1e4

inv temp 1e6

inv temp 1e8

(a) m=1 (centralized), N=25

0

500

1000

1500

2000

2500

3000

Training Episodes per Agent

0

2

4

6

8

10

inv temp 1e0

inv temp 1e2

inv temp 1e4

inv temp 1e6

inv temp 1e8

(b) m=2 (no communication), N=25

Computational Time We have demonstrated that both NeuralTS and NeuralUCB exhibit convergence to performance levels comparable to our proposed randomized exploration strategies ( i.e., PHE and LMC) when considering the case of N = 10 with m = 4 under the synchronization condition ( linear ), as outlined in (3.3). However, we argue that the scalability of both methods is limited due to their associated computational costs. To substantiate this assertion, we conduct experiments across all methods including DQN baselines with N = 10 and m = 4 over 10 4 steps with varying neural network sizes, such as [32 , 32 , 32] , which signifies three layers with 32 neurons in each layer. Importantly, the length of the chain N has no bearing on the running time.

In Figure 8, we show the computational time of all methods under different neural network sizes. The solid lines represent the average computational time over 10 random seeds and the shaded area represents the standard deviation. We observe that NeuralTS and NeuralUCB have heavy running time consistently with varying network sizes. Although the computation time of LMC is still higher than other remaining approaches, we observe that it maintains a similar computation time with different neural network sizes, which can still be scaled up to more complex problems with larger neural networks.

Hyper-parameter Tuning of Inverse Temperature β m,k Subsequently, we scale the problem to N = 25 . Given the extended horizon, the demand for exploration intensifies, leading us to conduct

0

1000

2000

3000

4000

5000

6000

Total Episodes

0

2

4

6

8

10

m = 1, N = 25, full buffer

m = 1, N = 25, half buffer

m = 1, N = 25, less buffer

m = 2 (no communication), N = 25, full buffer

m = 2 (no communication), N = 25, half buffer

m = 2 (no communication), N = 25, less buffer

(a) PHE

0

1000

2000

3000

4000

5000

6000

Total Episodes

0

2

4

6

8

10

m = 1, N = 25, full buffer

m = 1, N = 25, half buffer

m = 1, N = 25, less buffer

m = 2 (no communication), N = 25, full buffer

m = 2 (no communication), N = 25, half buffer

m = 2 (no communication), N = 25, less buffer

(b) LMC

0

1000

2000

3000

4000

5000

6000

Total Episodes

0

2

4

6

8

10

m = 1 (centralized), N = 25

m = 2 (no communication), N = 25

m = 2 (linear), N = 25

m = 2 (constant), N = 25

m = 2 (exponential), N = 25

(a) PHE

0

1000

2000

3000

4000

5000

6000

Total Episodes

0

2

4

6

8

10

m = 1 (centralized), N = 25

m = 2 (no communication), N = 25

m = 2 (linear), N = 25

m = 2 (constant), N = 25

m = 2 (exponential), N = 25

(b) LMC

hyper-parameter tuning for the inverse temperature parameter β m,k in LMC, as illustrated in Figure 9. It is crucial to note that the efficacy of learning is significantly influenced by the exploration capacity in both centralized learning and parallel learning without communication. Our observations reveal a discernible gap between centralized and parallel learning, a departure from the pattern observed in Figure 5. We posit that the disparity may stem from issues associated with the replay buffer size in off-policy RL algorithms. Specifically, when the replay buffer exhausts its capacity for new transitions, the incoming transition replaces the oldest one.

Hyper-parameter Tuning of Buffer Size Therefore, we present a performance comparison between a solitary agent ( m = 1 ) and a scenario involving two agents ( m = 2 ) in Figure 10 with different buffer sizes. Full buffer and half buffer indicate the replay buffer's capacity to store the complete set and half of the transitions during training, respectively. We observe that the learning process is more efficient with less buffer size in a centralized setting because having an excessively large replay buffer may potentially impede the efficiency of the learning process. Furthermore, the gap between centralized setting and paralleling learning still exists among different buffer sizes. Therefore, we focus on the setting of less buffer size with different synchronization conditions in Figure 11. We conclude that linear condition results in competitive performance in both PHE and LMC in the N -chain problem and we report all exploration strategies with linear condition in

0

1000

2000

3000

4000

5000

6000

Total Episodes

0

2

4

6

8

10

m = 1 (centralized), prioritized

m = 2 (no communication), prioritized

m = 2 (linear), prioritized

m = 1 (centralized)

m = 2 (no communication)

m = 2 (linear)

(a)

SuperMarioBros-1-1-

v0

(b)

SuperMarioBros-1-2-

v0

(c)

SuperMarioBros-1-3-

v0

(d)

SuperMarioBros-1-4-

v0

MARIO

000000

X00

WORLD

1-1

TIME

390

MARIO

000000

xO0

WORLD

TIME

862

000000

MARIO

x00

WORLD

1-3

TZNE

MARIO

000000

x00

WORLD

1-4

TIME

287

宁

Section 5.1. Note that the x -axis in Figure 10 and Figure 11 represent the total training episodes from m agents.

Ablation Study of Sampling Mechanism To reduce the reward gap, we adopt a better sampling mechanism in the replay buffer with prioritized experience replay (PER). In Figure 12, parallel learning without inter-agent communication can increase reward with PER, where the x -axis represents the total training episodes from m agents. However, centralized learning with PER improves faster convergence with similar performance and the trends for linear condition curves are similar. Therefore, the gap between centralized and parallel learning without communication is reduced with PER. Note that the main experimental results in Figure 1 are based on standard experience replay because standard sampling in linear condition has similar performance against PER with faster training time.

K.2 Super Mario Bros

While cooperative parallel learning enhances training efficiency through data sharing, challenges emerge when handling data from devices capturing images or audio due to privacy concerns in realworld applications. In response, our approach extends randomized exploration strategies to a federated reinforcement learning framework as shown in Algorithm 4, from Algorithm 1, which incorporates parameter synchronization among Q neural networks rather than relying on the conventional practice of sharing agents' transitions in Line 14 in Algorithm 4. Note that the synchronization follows the format as in Algorithm 1 to update Q functions with horizon h ∈ H . However, in practice, we can directly update the weight of the neural network to reduce the communication cost.

The training process unfolds within a federated reinforcement learning framework, wherein local updates and global aggregations are iteratively executed [37]. Specifically, each agent iterates through multiple local updates of its value function, followed by server-mediated averaging of these functions across all agents, constituting a form of parameter sharing. Note that the transitions are not accessible among agents, leading us to directly synchronize all agents with parameter sharing every constant local iteration instead of synchronization condition in (3.3). We use the same architecture for all the experiments in the Super Mario Bros task with the preprocessed images as the input states and 7 discrete actions in action space.

Particularly, we construct 3 convolutional neural network layers with width [32, 64, 32], followed by 2 fully connected layers with the output of action space in the Q network. The detailed hyper-parameters for Super Mario Bros task are presented in Table 5.

K.3 Thermal Control of Building Energy Systems

BuildingEnv encompasses the regulation of heat flow in a multi-zone building to sustain a desired temperature setpoint. We focus on one pre-defined building called "office small" in different cities with varying weather types, i.e., Tampa (Hot Humid), Tucson (Hot Dry), Rochester (Cold Humid), and Great Falls (Cold Dry). Each episode is designed to span a single day, comprising 5-minute time intervals (H = 288, τ = 5/60 hours).

Hyper-parameter

PHE

LMC

DQN

Bootstrapped

DQN

Noisy

DQN

DDQN

Discount Factor

λ

0.99

0.99

0.99

0.99

0.99

0.99

Learning Rate

η

k

3

×

10

-

3

3

×

10

-

3

3

×

10

-

3

3

×

10

-

3

3

×

10

-

3

3

×

10

-

3

Relu

Relu

Relu

Relu

Relu

Relu

Output Activation

Linear

Linear

Linear

Linear

Linear

Linear

No Update

J

k

1

8

1

1

1

1

No Target Networks

2

1

1

4

1

1

32

32

32

32

32

32

NN size

64

×

64

64

×

64

64

×

64

64

×

64

64

×

64

64

×

64

Random

PHE

LMC

DQN

BootstrappedNoisyNet

DDQN

Algorithm

50

40

30

20

10

Daily Return

(a) Tampa (hot humid)

Random

PHE

LMC

DQN

BootstrappedNoisyNet

DDQN

Algorithm

50

45

40

35

30

25

20

15

Daily Return

(b) Tucson (hot dry)

Random

PHE

LMC

DQN

BootstrappedNoisyNet

DDQN

Algorithm

110

100

90

80

70

60

50

Daily Return

(c) Rochester (cold humid)

Random

PHE

LMC

DQN

BootstrappedNoisyNet

DDQN

Algorithm

200

150

100

50

0

Daily Return

Observation Space The state at time step t , denoted as s ( t ) ∈ R M +4 , encompasses the temperatures T i ( t ) of each zone, where i ∈ M , along with four additional properties: Q GHI ( t ) , ¯ Q p ( t ) , T G ( t ) , and T E ( t ) . Specifically, Q GHI ( t ) represents the heat gain from solar irradiance, ¯ Q p ( t ) denotes the heat acquired from occupant activities, while T G ( t ) and T E ( t ) signify the ground and outdoor environment temperatures, respectively.

Action Space The continuous version of the action a ( t ) ∈ [ -1 , 1] M controls the heating of M zones. However, since our randomized exploration strategies use DQN [57] as the backbone, we adopt the multi-discrete action space defined in [85], which is a vector of action spaces. Then we convert the multi-discrete action space to a single discrete action space with action mapping.

Reward Function The primary objective is to minimize energy consumption while ensuring the maintenance of temperature within a specified comfort range. Therefore, the reward is penalized with both temperature deviations and HVAC energy consumption as follows:

where T target ( t ) = [ T target 1 ( t ) , T target 2 ( t ) , ..., T target M ( t )] are the target temperatures and T ( t ) = [ T 1 ( t ) , T 2 ( t ) , ..., T M ( t )] are the actual zonal temperatures. The parameter β is the trade-off between the energy consumption and temperature deviation penalties.

We execute experiments following the united framework in Algorithm 1, synchronizing every constant number of steps across diverse weather conditions in varying cities. The hyper-parameters we used are in Table 6. Subsequently, we evaluate the performance of all methods in distinct cities respectively, as illustrated in Figure 14. Notably, our proposed random exploration strategies demonstrate a consistently higher mean return across all cities. However, it is worth highlighting that DQN in Figure 14(c) and Noisy-Net in Figure 14(d) exhibit lower returns compared to random actions. This outcome can be attributed to the discrete action space configuration [85]. In addition, we observe that maintaining thermal control of buildings is more challenging in cold weather conditions compared to hot weather conditions.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose our algorithm in Section 3 with main theoretical analysis in Section 4. We also support our proposed method with empirical results in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss that the limitation of theory is hard to generalize to deep RL setting and the communication cost will also be higher in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We present theoretical analyses of our algorithms in the linear structure under the assumption of linear function approximation and linear MDP setting in Section 3. We provide comprehensive proof from Appendix B to Appendix J, including the analysis of the communication complexity, proof of the regret bound for two proposed methods, misspecified setting, and all supporting lemmas.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe experiment settings and environments with the main experiment results in Section 5 and mention the hardware we run on. In addition, the used architectures and hyper-parameters are provided in Appendix K. We also release our code in supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide our code in supplementary materials and in the link in Section 5. Specifically, we provide scripts and access to the datasets/environments we use.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We present the goals, main settings, and the datasets/environments in Section 5. Due to the space limit, the full training details are provided in Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We present our learning curves in the figures with shaded areas, representing the standard deviation over varying random seeds. In addition, violin plots can display the distribution of the return with probability density.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We indicate the related description in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We do review and follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Reinforcement Learning. This work does not have any potential negative social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not include and release new data, so the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited and the original paper that produced the code package and dataset. In addition, we also provide the URL in the README.md.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper is mainly for proposing provable and practical algorithms without any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper is mainly for proposing provable and practical algorithms without any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the first result on provably efficient randomized exploration in cooperative multi-agent RL. This paper focuses on parallel MDP where the transition kernel assumes an approximately linear structure. To this end, two Thompson-sampling-type algorithms are propose which leverages the perturbed-history exploration and Langevin Monte Carlo exploration strategies, respectively. When applied to linear MDPs, both algorithms provaly achieve $\\tilde{\\mathcal{O}}(d^{3/2} H^2 \\sqrt{MK})$ regret bound with $\\tilde{\\mathcal{O}} (d H M^2)$ communication complexity, where $H$ is the horizon length, $M$ is the number of agents and $d$ is the parameter dimension, marking the first-ever non-trivial theoretical results for randomized exploration in cooperative multi-agent RL. To evaluate the proposed methods, experiments on multiple parallel environments, including $N$-chain, a video game, and an energy system control problem, are conducted. The results show the effectiveness of the proposed algorithms, even under certain misspecified transitions.\n\nSTRENGTHS:\n**Significance**: \n\n**1.** The paper proposed the first meaningful theoretical result for randomized exploration in multi-agent cooperative RL. \n\n**2.** The algorithms are not only theoretically meaningful, but also easy to implement and has various advantages such as computational efficiency and avoidance of sampling bias.\n\n**Quality**: this paper is high-quality. \n\n**1.** The theory part of this paper is very solid.\nFirst a unified framework is introduced which has wide applicability and can incorporate various specific settings. Then detailed application to linear MDP with theoretical guarantees is given. The paper further considers the misspecified setting where the transition kernel is slightly misspecified in a certain way, which can be common in practice. This helps extend the applicability of the proposed method.\n\n**2.** The experimental result of this paper is extensive. Results on both video games and realistic energy control are provided.\n\n**Clearity**: This paper is clearly written.  \n\n**1.** The theory of this paper is very clear, with the help of well-chosen terms and notations, unambiguous definitions and clear description of theorems. \n\n**2.** A detailed table comparing all necessary related works is provided, helping readers quickly grasp the pros and cons of the proposed methods.\n\n**3.** In section 3.1 and 3.2, a thorough description of the interpretation behind the PHE and LMC strategy is given. The rationale behind the synchronization rules is explained. \n\n**4.** For all the experiments in section 5, the necessary implementation detail is given.\n\nWEAKNESSES:\nI do not detect any technical or major weakness in the paper. I think this paper does make a novel theoretical contribution to a meaningful problem in RL. It is also well-written.\n\nQUESTIONS:\nI do not have specific confusion regarding the major content of this paper since it is clearly written. Still I am interested in the following and any thoughts from the authors would be great.\n\nThis paper reports theoretical guarantees on linear MDPs. I am just wondering if randomized exploration in cooperative RL can be extended to richer MDP classes. It seems that the current algorithm design is very suitable for linear MDPs. However, the possibility and path of extending to other MDP classes is less clear.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies provably efficient randomized exploration in parallel MDPs setting. The authors consider the linear setting, and propose two Thompson Sampling-style algorithms and establish the regret guarantees and communication cost. After that, they also conduct some experiments for evaluation.\n\nSTRENGTHS:\nThe authors contribute some new algorithms in this cooperative learning setting, and derive their regret bounds. The paper writing is also easy to follow. The experiments results are provided to verify the performance of algorithms.\n\nWEAKNESSES:\n1. I'm not convinced that the setting considered in this paper should be called \"multi-agent RL\". I think it would be better to regard it as a multi-task RL setting (or maybe consider the terminology the authors used in paper: the \"parallel MDP setting\"), because the transition and reward functions for each agent here does not depend on the behavior of other agents, which I believe the key feature of MARL setting is missing.\n\n2. The comparison with previous work is not precise and misleading:\n\n    Line 210-211: \"... matches the existing best single-agent results ...\", which is not true. [1] considers a more general linear function approximation seeting and its regret only have $O(d)$ dependence on the dimension. I would also suggest to include the comparison with it in Table 1.\n\n3. The setting requires more description. It seems to me there are two objectives: regret and communication complexity. It is unclear to me how the authors decide to trade-off them. I would suggest the authors to make it more clear. For the current version, it is unclear whether the regret bounds or the communication cost are optimized to optimal (or one of them are optimal while the other is better than previous).\n\n    Besides, it seems that the agents can feel free to decide when to synchronize with the other agents to optimize the communication cost, which seems not practical in most of the times. \n\n4. The authors motivated the randomized exploration by pointing out that the UCB style algorithms can be computational intractable beyond the linear setting. However, the objective in Eq. (3.5) and update rule Eq.(3.7) seem not easy to be generalized beyond linear setting. Besides, all the theoretical results are also limited in the linear setting. So the motivation of this paper seems not convincing to me.\n\n\n[1] Zanette et al., Learning Near Optimal Policies with Low Inherent Bellman Error\n\nQUESTIONS:\n1. Can you highlight a bit more about the technique novelty of the proposed methods?\n\n2. In Page 26, line 877, can you explain in details why the inequality holds?\n\n3. In Page 28, line 891, I didn't find the definition of $w^{1,0}, \\hat{w}^{1,0}, \\Lambda^1, b^1$ in Algorithm 3. Can you explain it?\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

comments: SUMMARY:
This paper considers randomized exploration in a multi-agent reinforcement learning setting called parallel MDPs. Two Thompson sampling-type algorithms are provided with a regret bound and communication complexity bound. The algorithms are empirically validated in multiple environments.

STRENGTHS:
* The paper is well-written and easy to follow.
* Theoretical analysis is provided and it can match the performance of the state-of-the-art results with the potential to generalize to deep RL.
* Experiment validation is provided.

WEAKNESSES:
* The technical novelty and contribution seem limited. Can authors elaborate on the challenges in the analysis?

QUESTIONS:
* What is the tradeoff between using perturbed history exploration and Langevin Monte Carlo exploration?

LIMITATIONS:
I didn't see potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates multi-agent reinforcement learning in cooperative scenarios. The main contribution is the extension of randomized exploration methods, including perturbed-history exploration and Langevin Monte Carlo exploration, to the multi-agent cooperative setting. The authors offer a regret analysis for the linear MDP case and present empirical results to validate the proposed method.\n\nSTRENGTHS:\n-  Although the results of this paper are not entirely new, as they extend randomized exploration methods from the single-agent to the cooperative multi-agent setting, the authors effectively highlight the technical challenges and their contributions (line 227-236).\n- The algorithm proposed in this paper have better communication complexity than the UCB-type algorithm in the synchronous setting.\n-  Extensive experiments are conducted to validate the effectiveness of the proposed method, which is an advantage for a theoretically oriented paper.\n\nWEAKNESSES:\n- About the contenders in experiments: It is unclear to me how the contenders DQN, Double DQN, and others were implemented. Are they running independently in multi-agent environments, with the average reward reported? If not, since Algorithm 1 provides a unified framework for parallel MDPs, the empirical comparison might be more complete if other contenders are also equipped with the synchronized steps. Besides, It is unclear why the performance of Bootstrapped DQN deteriorated significantly in the $N$-chain problem. \n\n- Discrepancy between theory and empirical results: As shown by Theorem 4.3 and Theorem 4.4, the average performance of the proposed method improves with the growth of $M$. However, in Figure 1 and Figure 4 in the appendix, a slower convergence rate is observed when $m$ is larger. Additionally, the scaling of the x-axis in Figure 1(a) and Figure 1(b) is different, which can be misleading since the convergence rate in Figure 1(b) is actually slower than in Figure 1(a).\n\n- Parameter setting: It appears that setting the threshold $\\gamma$ requires knowledge of $K$. How should this parameter be set in practice?\n\nOverall, this paper makes solid progress in developing randomized algorithms for cooperative multi-agent RL. However, I am concerned about the discrepancy between theory and experiment. I would raise my score if these concerns are adequately addressed.\n\nQUESTIONS:\n- Could you provide a more detailed explanation for the configuration of the contenders?\n- Could you provide a more detailed explanation for the discrepancy between the theoretical guarantees and the experimental results?\n- How should the parameter $\\gamma$ be set in practice?\n\nLIMITATIONS:\nI do not find the negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}