Society of Agents: Regret Bounds of Concurrent Thompson Sampling
Authors: Yan Chen
Perry Dong
Qinxun Bai
Maria Dimakopoulou
Wei Xu
Zhengyuan Zhou
Venue: NeurIPS 2022 Accept (2022)

Abstract:  We consider the concurrent reinforcement learning problem where $n$ agents simultaneously learn to make decisions in the same environment by sharing experience with each other.
 Existing works in this emerging area have empirically demonstrated that Thompson sampling (TS) based algorithms provide a particularly attractive alternative for inducing cooperation, because
 each agent can independently sample a belief environment (and compute a corresponding optimal policy) from the joint posterior computed by aggregating all agents' data , which induces diversity
 in exploration among agents while benefiting shared experience from all agents. 
 However, theoretical guarantees in this area remain under-explored; in particular, no regret bound is known on TS based concurrent RL algorithms. 
 
 In this paper, we fill in this gap by considering two settings.
 In the first, we study the simple finite-horizon episodic RL setting, where TS is naturally adapted into the concurrent setup by having each agent sample from the current joint posterior at the beginning of each episode. We establish a $\tilde{O}(HS\sqrt{\frac{AT}{n}})$ per-agent regret bound, where $H$ is the horizon of the episode, $S$ is the number of states, $A$ is the number of actions, $T$ is the number of episodes and $n$ is the number of agents.  
 In the second setting, we consider the infinite-horizon RL problem, where a policy is measured by its long-run average reward. Here, despite not having natural episodic breakpoints, we show that by a doubling-horizon schedule, we can adapt TS to the infinite-horizon concurrent learning setting to achieve
 a regret bound of $\tilde{O}(DS\sqrt{ATn})$, where $D$ is the standard notion of diameter of the underlying MDP and $T$ is the number of timesteps.  Note that in both settings, the per-agent regret decreases at an optimal rate of $\Theta(\frac{1}{\sqrt{n}})$, which manifests the power of cooperation in concurrent RL.

Society of Agents: Regret Bounds of Concurrent
Thompson Sampling
Yan Chen
Perry Dong2*
Qinxun Bai Maria Dimakopoulou"
Wei Xu 'Spotify
Zhengyuan Z6t UC Berkeley
Arena Technologies
Duke SHorizon Robotics NYU Stern
yan. chen@duke. edu perxydong@berkelly, edu
{qinxun ( wei Shorizon... ai tnimaaoopouluu@@potffyyyooo com
zzhou0stern nyu. edu
Abstract
We consider the concurrent reinforcement learning problem where agents simul- taneously learn make decisions the same environment by sharing experience with each other. Existing works in this emerging area have empirically demon- strated that Thompson sampling (TS) based algorithms provide a particularly attractive alternative for inducing cooperation, because each agent can indepen- dently sample belief environment (and compute a corresponding optimal policy) from the joint posterior computed by aggregating all agents' data which induces diversity in exploration among agents while benefiting shared experience from all agents. However, theoretical guarantees in this remain under-explored; in particular, no regret bound is known on TS based concurrent RL algorithms.
In this paper, we fill in this gap by considering two settings. In the first, we study the simple moite-horizon episodic RL setting, where TS naturally adapted into the concurrent setup by having each agent sample from the current joint posterior
at the beginning of each episode. We establish Ó(HS, AT per-agent regret bound, where H is the horizon of the episode, S is the number of states, A is the number of actions, 7' is the number of episodes and n is the number of agents. In the second setting, we consider the infinite-horizon RL problem, where policy is measured by its long-run average reward. Here, despite not having natural episodic breakpoints. we show that by doubling- -orizzon schedule, we can adapt TS to the infinite-horizon concurrent learning setting to achieve regret bound of where D is the standard notion of diameter of the underlying MDP and is the number of timesteps. Note that in both settings, the per-agent regret decreases at an optimal rate of O which manifests the power of cooperation in concurrent RL.
Introduction
In concurrent reinforcement learning (26.. many agents interact with (different instances of) the same environment simultaneously and share data with each other order to jointly learn an optimal policy for the environment. This learning scenario has important applications for large-scale automated sequential decision making systems. For instance, in recommendation systems I (e.g. movie recommendation, personalized ads recommendation. personalized news/content recommendation), company often has a large number of servers - sometimes located in different geographical regions that serve the entire population of the customer base. As such, incoming customer is randomly routed to a server (i.e. an agent), which learns to make recommendation decisions in conjunction with
Equaa contributions. i Correspondence Zhengyuan Zhou (zCarena-ad at.enna-iii.coom com).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

all the other servers, and the performance judgement is made on the entire set of servers. Another example is in concurrent personalized promotions, where each agent actively learning to provision personalized promotions on a product, along with other agents, each of which is performing the same personalized promotions task on similar product. As surprising as it sounds, this example has already been done in practice, where Arena Technologies a cutting-edge enterprise Al solution provider, deploys sophisticated reinforcement learning agents to provision personalized promotions/pricing to an ensemble of beverages and other consumer packaged goods. A third example which exists as concept but not yet happened- of concurrent RL is controlling a team of autonomous vehicles, which, succinctly summarized in 19], concerns a scenario where "each agent manages single vehicle, and again, the agents learn from each other as data is gathered. The goal could be to optimize a combination of metries, such as fuel consumption, safety, and satisfaction of transportation objectives. Exploratory actions play an important role, and structured diversity of experience may greatly accelerate learning." " "This application is particularly relevant and presents an unique opportunity for the emerging era of autonomous driving. Other concurrent RI. applications include robotics ([T), (26) and biology( ((2))
An important opportunity in the concurrent RL setup -in comparison to the traditional single-agent RL setup lies in coordinated exploration: since those agents can share their own learning experience with others, the team as a whole should achieve better overall regret performance (or equivalently better per-agent regret performance) if they intelligently coordinate the exploration efforts. In particular, different agents may want to diversify their exploratory actions so as to quickly reduce the team's 's uncertainty over the landscape of the environment in which they all operate. By sharing the resulting data with each other, there is hope that the team would-in a environment-interactinn frame much less than that from single-agent setting- acchive the same level of performance.
time
When it comes to exploration, a substantial amount of literature has been devoted to designing (provably) efficient algorithms in the single-agent RL setting. One prominent class of algorithms, based on upper confidence bounds, is known as UCRL (and its variants); (see references citing them and cited by them. The UCRL family of algorithms embodies the important philosophy of "optimism in face of uncertainty" in their design and enjoys optimal theoretical regret guarantees (along with elegant analysis) in various settings, including both finite-horizon episodic RL and infinite-horizon average-rewand reward RL. As such, it is perhaps not surprising that UCRL algorithms the first family of algorithms that have been adapted to the concurrent RL settings [11] and [23). In particular, concurrent UCRL has been analyzed for the sample complexity performance measure (i.e. how many samples needed to learn an c-optimal policy) under both finite action space setting and infinite action space setting. More specifically, [I]] provided a high-probability bound of Ō S + SAn) ) for the sample complexity with agents interacting in the environment. Their algorithm was extended from MBIE(ssee (28)), with a single agent performing concurrent RL across set of anlinitehhorizor MDPs. The results there show that with sharing samples from copies of the same MDP a linear speedup in the sample complexity of learning can be achieved. But no regret bound was derived there for concurrent RL. [33 designed algorithm which extended concurrent exploration to continuous state-action spaces and allowed smooth generalization between concurrent MDPs under infinite horizon setting. Likewise, the authors also explored the scenario where one single agent interacts with concurrent MDPs at the same time. Additionally, they define an error metric, called TCE (total cost of exploration), and showed an Ó( + (d)) bound for TCE, where. (dd) is constant representing the covering number of the MDP. This is sufficient for an Ó( (d)) bound for sample complexity.
and many
However, these from the critical visadvantage of no coordinated exploration: since the upper confidence bounds computed by aggregating all agents' data are the same the entire team, each agent would follow the exact same policy, thereby yielding no diversity in exploration. This point has been observed in 9 8) which has advocated for designing concurrent RL algorithms based on Thompson sampling 20 22
guarantees notwithstanding, concurrent UCRL algorithms suffer
(29). [14] [17.118) also known posterior sampling reinforcement learning (PSRL). Unlike UCRL, PSRL manifests the philosophy of "probabilistic optimism in face of uncertainty". where the agent is uncertain about the transition probabilities and reward functions, and hence at the start of each episode, the agent would sample an MDP from the current posterior distribution based on data
"See http::/wwww.rem--a.commm for more information.

---

gathered up to that time and makes decisions according to the policy that is optimal with respect to the sampled MDP. [6] and [22] have provided good motivating examples for empirical evidence.
In fact, not only PSRL. is well-known to outperform UCRL empirically in single-agent RI. setting, it also by its very design lends itself well to concurrent RL. settings: each agent can sample from a posterior distribution and explores accordingly, which induces diversity in a coordinated and S proposed sophisticated Thompson sampling variants that
way. Building on this insight, [9] demonstrated good empirical performance. Despite these remarkable empirical results, to date, no theoretical guarantee has been provided on any concurrent Thompson sampling algorithm, simple or sophisticated. Our goal in this paper is to make initial inroads on the theoretical front and attempt to prove the first bounds on simple-but butte-rurtttaalll concurrent PSRL algorithms.
1.1 Our Contributions
We establish per-agent regret guarantees for concurrent PSRL algorithms in two settings: one in finite-horizon episodic setting and the other in infinite-horizon average reward setting. In the first setting, TS is naturally adapted into the concurrent setup by having each agent sample from the current joint posterior at the beginning of each episode. We establish a Ö(HSV AT' per-agent regret bound, where H is the horizon of the episode, S is the number of states, A is the number of actions, is the number of episodes and n is the number of agents. This regret bound further improves
to Ö(H V tight(see [6].
(AAT) under the special Dirichlet prior. Note that when n = 1, both regret bounds are
FL
(1I)) These two regret bounds further highlight that the per-agent regret decreases at an optimal rate of O( (t))) and hence manifests the power of cooperation in concurrent RL: as a
(and Ô(H /n5 AT) under Dirichlet prior), which is minimax optimal, because that is the optimal regret performance (up to log factors) achievable by single-agent for nT episodes that can adapt at each episode
team, the total regret achieved is D(SVVAA
In the second setting, we consider the infinite-horizon RL problem, where policy is measured by its long-run average reward. Here, despite not having natural episodic breakpoints, we show that by doubling-horizon schedule, we can adapt TS to the infinite-horizon concurrent learning setting achieve regret bound of Ö(DSVATH)
where D is the standard notion of diameter of the underlying MDP and T the number of timesteps. Here, we again see that the per-agent regret decreases at an optimal rate of O the team of agents from efficient concurrent reinforcement learning. Due to space limitation, we'll 'll include simulation plots in the appendix.
Vn ( (L) and the fact that single learning trajectory does not prevent
2 Problem Formulation
We consider the Bayesian regret bound of concurrent Thompson Sampling of Markov decision process in finite-horizon episodic setting and infinite-horizon setting. In both settings, we provide bounds on the general prior distributions and Dirichlet prior distributions for concurrent Thompson Sampling of the MDPs.
2.1 Finite- Horizon Episodic Setting
Formally, we consider the problem of learning to optimize random (parallel) finite-horizon episodic
Markov decision process M = (S,A, R2" P* H, p) over repeated episodes of interaction. Here S} S = {1, S} is the state space, A = 2, .., A} is the action space, H is the horizon of each episode, and p is the initial state distribution for each episode. At the start of each episode k(k > 1). each agent will receive the history across all the agents before period k. And at the start of episode 1, all the agents don't have any information regarding the history. For every period episode, each agent p observes state sh E S, selects action of E A, receives a reward ry ~ R transitions new state Shi1 (' d.). At time agent p chooses policy T², which is
h = 1,2,..., H within
( (sf df), where the reward functions Th are assumed to be in [0, 1], and
That is, one can equivalently think of the concurrent setting as a single agent making decisions but only being able to adapt every time episodes. As such, the team's 's performance as whole in 7 episodes will never be better than for the single agent in ref' episodes.
3

---

mapping from the information that she has, i.. Hi, to action E A chosen by agent p. For each agent p. MDP M and policy T², the state-action value function for each period h is defined as Q.P,, (s, a) = E MLكم=,, FM ay)s,, = 8, 4/ = al. where FM (s, a) = Blrlr ~ R (s, a)]. The subscript T. indicates that the actions agent p over periods h l..... H are chosen according to policy π². Let VA AP.A (s) := Q
TM E arg max, V-M ". (s) for all x (s, (s,,s)). We say policy x² is optimal for MDP M if and = 1.2....... H.
Let H4 = 1p=1 denote the history of observations made by time (prior to and including time step 1) across all the agents p = 1, 1.2.... ....., where HP" is the history of agent p by time step (include time (), with t= 1, 2, ...... T and T is the total time horizon. So (k (k-18 H) + / is the time step at period within episode k. We use bb to denote the state of agent p at time 1 = (k - 1)H + h, i.e. episode of functions, each mapping H(K-11H1 (i.c. the history of observations prior to episode k) to a probability distribution T& (H(R-1)H) over policies, from which each agent samples a policy Tkp for the k-th episode. The regret for agent p in episode k is DEp == Exes ((s) x- x-,1 (s) - -V
k period h. The RL algorithm
consider here is deterministic sequence {TT : k =
The V- (x) where x² nM"
overall regret in episode k is As == E p (V)
The regret incurred up time T for policy is Regret (T, n, M*) = =
(T D The Bayesian expected regret for M distributed according to the prior & is defined as BayesRegret n, σ) E|Regret((, n, .M MMM Φ)
2.2 Infinite-Horizon Setting
We follow the setting in 3 and consider the case where different agents can cooperate. As in 3 we consider communicating MDPs with finite diameter. We first define 'diameter', 'communicating" and other related terms.
Definition 2.1. A deterministic policy x : S + A is mapping from state space S to action space A. Definition 2.2. Diameter ((M) of MDP M is defined as the minimum time required to go from one state to another in the MDP using some deterministic policy: D(M) = maXe4e's S - Here T T is the expected number of steps it takes to reach state s' when starting from state s and using policy и.
Definition 2.3. An MDP is communicating and only if it has finite diameter.
Definition 2.4. The gain (s) of a policy π, from starting state $1 s, is defined as the infinite horizon undiscounted average reward given by X (=(s) = Blimr + E r(st, I (s:)) 81 = s). where 8t is the state reached at time f. by executing policy n.
Lemma 2.1. (Optimal gain for communicating MDPs). For a communicating MDP M with diameter D we have: 1. ((24) Theorem 1.2. Theorem 8.3.2) The optimal gain X is state independent and is achieved by a deterministic stationary policy x i.e. there exists a deterministic policy " such that X' = max max, (s') = X= (s), Vs e S; 2. (15), Theorem 4). The optimal gain X satisfies the following equations X minners max,, r(s, a) P h2 max, r(s,a)
PT ha hgg Vs where the vector h = (he)ses satisfies the following optimality equation h² e = max.g.A a) + PT h.Tgh"). max, hig min, h D.
h*). Here h² is referred to as the bias vector of MDP M satisfying:
Given the above definitions and results, we now formally define the Reinforcement Learning problem consider here. The process proceeds in time steps / 1, 1,2, ... T. For agent p = 1,2 2, ...... n, each starts from a state of at time step 1 = 1. In the beginning of each time step f, agent p takes an action df A and observes the reward rf)), where s/ is the state of agent p at time t, and also the next state 8441 ~ Ps d²). And again the reward function is assumed to be in [0. 1]. Here and P are the reward function and the transition model respectively for communicating MDP M with diameter D. Each learning agent knows the state-space S, the action space A, as well as the rewards r(s, a), Vs E S. E A, for the underlying MDP, but not the transition model P or the diameter D. The agent can use the past observations across all the agents to learn the underlying MDP model and decide future actions. The goal for each agent is to maximize total reward Ei= r(sf, a²), or equivalently minimize the total regret over a time horizon T,, defined as R(T, TnX* Ep=1 Σt r(s,,, d²), where X" is the optimal gain of MDP M.

---

Unlike the mnite-horizon case, we assume that we know the reward functions as in B Since without loss of generality, as long as the reward functions are bounded, we can have the same bound on the regret. This is because the unknown transition probability is the main source of difficulty in the analysis of the regret bound.
Remark 2.1.
Notations We use S to denote the state space, where [S] = S, and A to denote the action space, with |A| A. There are n agents interacting with the environment. We use r(s, a) denote the reward related to state-action pair (s, a). In finite horizon case, H is the horizon of each episode, the initial state distribution for each episode. sy is the state of of agent p at time step t, and of is the action that agent p takes at time step t, where
In finite horizon case, we use P(s, a) to refer to the transition probability vector related to state-action pair (s, a), and use P..e under the infinite horizon case. The policy x is mapping from the information set that agent has access to, to an action a in the action space. Under finite horizon, we use Sy to indicate the state of agent p in episode k period h, and o to indicate the action that agent p takes in episode k period h. We use p* to denote the transition probability of the true MDP, X to denote the optimal gain of the true MDP under infinite horizon case. X is the optimal gain of the sampled MDP in epoch under infinite horizon case. D is the upper bound of the diameter of the MDP considered. For finite horizon case, H." denotes the information gathered by agent p before time step t, and Hi = - H is the history of observations prior to time step across all agents. For infinite horizon case, FK is the information of agent p within and before epoch kt and Fk = 1 FP is the information within and before epoch l across all agents.
1.2.....
Pinite-Horizon Episodic Case
Algorithm Here we consider concurrent version of the PSRL in (21]] At the beginning of each episode, each agent samples an MDP independently, where each one computes the optimal policy for their sampled MDP, and proceeds with this policy during the rest of the episode. Then they gather and update information at the end of each episode (See Algorithm [ There are n agents interacting in the environment, each episode is of length H, and the number of total time steps is T. So there are K = [T/H] episodes in the environment.
Intuition for Ō Val per-agent regret bound We obtain an Õ(VTn) Bayesian regret bound
for finite-horizon case, with Ö(HS for general prior distribution, and Ö(HvSATH) for Dirichlet prior distribution, where is the number of agents interacting in the environment. Note that in the single-agent case, previous literature shows such regret bounds, (Ō(SVAD) AT)) for general prior in [16] and Ö(VSAT) for Dirichlet prior in (1).. This indicates that when n agents interactively learn in the same unknown environment over a time horizon of T, the bound scales in non-linear way as if a single-agent learn over time horizon of T" n,T, which suggests the optimality of our bounds here with respect to the number of agents n. This is also a reflection of the power of concurrent learning within concurrent RL.
Improvement of Bayesian Regret Under Dirichlet Prior Distribution The key analysis from O(SSATT)) to (JH/SATmm follows the technical notes in [2I] where we borrow a technical lemma on -aussian-Dirichlet concentration bound (I9)). With this, are able to show that the regret contribution from the transition probability estimate can concentrate at a rate independent of S. Finally use Sub-Gaussia tail bound to make the improvement. The proof idea follows [21].
Decomposition of the Regret:
Note that we have
BayesRegret(T, n, Ø)
E Reggett, n. M M'M dj
[2/H]
E
p(s) (V (V - VAV. (s)) ~ Φ)
k=1
SES
And we can write M . (5) VMM'(s)
VM (s)
V MEp (3) + +V VMMa (s)
VM
Tep.1
A
5

---

Algorithm 1: Concurrent PSRL
Data: prior distribution d.H6 = 0, vp m https:/www.overreer wwwpeereeeeeeooooo/ee/////////ed
for episode k = 1,2, 1....... do
for p = 1 do
Sample MDP Map (1-HH(x11)
Compute Mkpp E arg max, y:
for period h=l,...,H H..... do
take action dig = lkp (8gh ht; update HP ( 11-1)) ( (k-1))+h-11 U ( Lgo rhio s 1
H.
end end
HAH = -p=1 HP HK Update posterior distribution (((HH)));
end
Note that conditioned upon any data H4H-1)H the true MDP M and the sampled Map are iden- tically distributed. This means that B[A S 0 for all k, Therefore, we just need to bound > [T/H Epm kp ( For the convenience of notation, write kh valke
Now we rewrite the regret via the Bellman operator, with (x) = rxp(x)) fx(x)) (x) (Pep(x)) P=(x))² Ep
k.
E[A
M(K-1) = [[(kkp - (=k))) + Pap(=1a) V V -P* M(x=-jnr kp
cone
[F[(kkp (*2()))) (Php Pa(xaa)) E((V Viz2 P*(ahil)
H
H (1bp( 8t (x(1))
E
Σ ((Ppp((<h)) 1
H
< E w
(chh))
+
Σ wh ( (1
h=1
h=1
3.1 Bayesian Regret Bound under General Prior Distribution
Theorem 3.1. (Bayesian Regret Bound Under General Prior Distribution) Let M* be the true MDP distributed according to any prior distribution d. If the number of agents n satsifies OSS2 AT log(S (nnT)),, Then the Bayesian regret for Algorithm [I) is bounded by
E [Regre((T.,,,,n)) n, ó.)) O(HSVATM
Proof Sketch: We first define confidence set M2 for each episode k as in [. so that this set contains both sampled MDP Mj and true MDP M with high probability. Then we decompose Akp into AM, MR, M E M&) + Atp [MM & Mr M &f MMM]]. Then we the fact that when condition on history H(k-1)H1 we have [[1 & M ((((d- [[1[M & M& HH(-1) ((16), so that [1M $ M₁ or M & MM)]] is bounded by 2HP(M* & Ma). As for the first part, namely A M E M&), we found that within the confidence intervals, Akp can be controlled with the desired upper bound. And for A [1(M, & M₁ or M $ M&)].. since it's bounded by 2HP & MMs), we just need show that PMMM & M&) is small, which indeed the case. We decompose the regret into two parts, with one part negative and the other part positive, So we focus on bounding the positive part. Then we rewrite the positive upper bound via the Bellman operator, where part is related to the concentration of reward estimate, and the other part related to the concentration of transition probability estimate, i.c. BayesRegret(TT, n, Φ) = V.M (s)) d)
Reggret," MMM Ø) 3E1 2
[T/H|
Σ E| (V (s) (s)

---

(x) And we can write V (s) - Taps? x1
Tepsl (s)
debky Tap.
(s)
V (s). A
Conditioned
App
A
upon data before the start of episode k, H( the true MDP M and the sampled Mr identically distributed. This means that E 4p4 Σ [T/M]] k=1 Ep El kp
I for all k, Therefore, we just need to
kpl
kp ( For the convenience of notation, we write V.kp = Vrrdap mep.h
bound
Now we rewrite the regret via the Bellman operator, with (x) (x) fx(x), w. wf (x) (Pkp(a)) - Pa(x)) (oo))
R, V B| / (1%+,) + Ei= ww (x ( ())()))) Based upon this, we prove that with high probability, both the true MDP and the sampled MDPs are in the confidence sets constructed here. And also conditional upon the event that both true MDP and sampled MDPs are in the confidence set, the total regret can be controlled.
Then by some simple derivation we find that (
S
3.2 Bayesian Regret Bound under Independent Dirichlet Prior Distribution
Theorem 3.2. (Bayesian Regret Bound Under Dirichlet Prior Distribution) Let M* be the true MDP distributed according to prior ó with independent Dirichlet prior over transition probabilities. Then the Bayesian regret for Algorithm ( is bounded by
BayesRegret n, d, n) S Ó(HVSATH)
Proof Sketch: From [21]. under Dirichlet prior, by a union bound at each 2 = (s, a) on P(s.a)
and rss, a), we have that for each p. with probability at least nf
H
H _ 2(H + +
20og AnT)
E A=1
(( + aam HHH S
m 1,
h=1
By pigeonhole principle, when we over all the i and p. the above inequality according to the number of times that each state-action pair were visited. For those state-action pairs which were visited less than twice, since the reward function is within [0, 1]. the sum of these parts are upper bounded by 2SA + 1. Lemma (D.5) guarantees that the regret from transition probability estimation error can be bounded at rate independent of S. and Lemma (D.4) further provides a bound for the dominating Gaussian variable on the Dirichlet distribution. this way we can get an improved bound of ((H from the case under general prior distribution. We leave the proof in the appendix.
decompose the right hand side of
4 Infinite-Horizon Case
Algorithm Algorithm 2 combines the idea of Thompson sampling and double epoch for infinite horizon case. Similar to the finite-horizon episodic case, the agents sample MDP independently at the beginning of each epoch. The difference that unlike the finite-horizon case where each episode is of the same length, the infinite horizon adapt a "double epoch" " strategy ([4), [3), where we stop the current epoch and move on to the next one immediately as long as the number of visits to some state-action pair at least doubles. As in umite-horizon case, we can also get O vi per-agent Bayesian regret bound as well. For the similar reason as in the finite-horizon case, here we also have an improvement from D(SAATT) to Ó(VSAT) AAT)) if we condition on Independent Dirichlet prior distribution instead of on some general prior.
From Finite-Horizon To Infinite Horizon: Doubling Epoch [15] points out the necessity have "phases" in infinite-horizon problem. In aefinite-horizonn case, if the reward of one state is slightly larger than the reward in the current state, the MDP tends to switch to the other state very easily, But for finite-horizon setting, an optimistic algorithm would eventually stop considering switches because eventually, the loss for switching would be assessed to be higher than the potential gain from switching, For this reason, recalculating the optimistic policy in each round may lead to the algorithm to switch states frequently, which can lead to linear regret in the end. As (3), we borrow the idea of doubling epoch [4], where the next phase starts once the number of visits to some agent's state-action pair least doubles. Since the total number of visits to any state-action pairs can be no more than nf" the number of epochs is bounded by some constant times SAA log(nf").
7

---

4.1 Bayesian Regret Bound Under General Prior Distribution
Algorithm 2: Concurrent Infinite-Horizz Posterior Sampling MDP
Data:
State space S, Action space .A, starting state vector across all agents $1 E SP, reward function time horizon T, FP () for p E [n].
for epochs k = 1,2,... do
Sample transition probability vectors: For each s, a generate sample probability vectors Q as follows:
Posterior sampling: use samples from the posterior sampling:
2h ~ o(
Compute policy #*: as the optimal gain policy for the sampled MDP A7 constructed using sample set {Q E S, E A).
Execute policy #
for time steps = Tk, Th 1, ...,, until break epoch do for agent p= 1.2. ..... do
(This block can be implemented in parallel) Play action o. = R (s
Observe the transition the next state s
Update Fh.h = = U {sp of r. s411
end
N o. break epoch.
2
TM for some p. (i.e. for some (s,, af)), then set Th41 L ++1 1 and oL
FR Up_ FP
Update posterior distribution ((-1Fa);;
end end
Theorem 4.1. (Bayesian Regret Bound Under General Prior Distribution) For any communicating
MDP M with states, actions, diameter D. Then for T 2 AoSSAlo""(S (SATn)). Then
BayesRegre(TT.MM.n Ö(DSVATT)
Proof Sketch: We first write the total regret as the sum of regrets over all the epochs. We focus on bounding the regret within any fixed epoch. Then we decompose the difference between the optimal gain of the true MDP and the reward functions as the sum of two parts: one part is the difference between the optimal gain of the true MDP and the optimal gain of the sampled MDP; the other part is the difference between the optimal gain of the sampled MDP and the reward function at some state-action pair. Then finally when we do a sum over all the epochs, we can get the desired regret bound. As defined before, RTT,M,, of MDP M. Algorithm
TnA EØ= Ei r(s,, af".. X" is the optimal gain (2) proceeds in epochs = 1,2, K, where K < SA log(T"). Now we Top where R2 is the regret within epoch kt So we can write R2 as R4 = E E [X -=p = Σne Σn [(** )x) + (A rrogaa)), )} where A the optimal gain of the sampled MDP 1 From Lemma (2.1) we know that for any state 8, optimal policy Ty for communicating MDP M action गu (s), A = Tea + pT h ha-
define RA = (Th-1 7k)m -NI111 - Σp
where P.o. Q. So we have

---

T4-1-
15
7411- n
Σ
Σ (ye rgg
Σ
Σ
Pef.at Petao + Pof.of 1,p)
p=1
p=1
7++1-
To41-1
Σ
Σ
Σ
P P.j.ot i
p=1
p=1
7++11
Tot1
E
Σ
Pst Ptt
Σ
Σ
Ps.fot -
1.2
4.2 Bayesian Regret Bound Under Dirichlet Prior
Theorem 4.2. (Bayesian Regret Bound Under Dirichlet Prior) Let M be the true MDP distributed according to prior é with any independent Dirichlet prior over transition probabilities. Then the Bayesian regret for Parallel Infinite-Horizon posterior sampling MDP algorithn 2 bounded by
BayesRegret(T, M, Φ) O(DVSATT)
Proof Sketch: We use Lemma G.2) to prove the concentration bound between sampled transition
probabilities and the expectation of the true probabilities conditional on the current data. And note that one key property of Thompson sampling is that conditional upon the current data, the true MDP and the sampled MDP are identically distributed.
4.3 Worse-case Bound Under Dirichlet Prior
Theorem 4.3. (Worst-case Infinite-Horizon Regret Bound Under Dirichlet Prior) With high probability, Algoithm 5 (provided in Appendix B) is bounded by
- ( ATn
5 Simulation
In this section, we describe our empirical study of concurrent Thompson Sampling on Markov Decision Process with a Dirichlet transition function and normal reward function for both finite- horizon and infinite-horizon cases. In particular, we train the concurrent Thompson Sampling agents on randomly sampled MDPs and evaluate the regrets of these agents the end of each episode/epoch on fixed number of environments sampled from the ground truth distribution. The simulation settings are moved to appendix due to page limit. Some of the experiment plots are shown in Figured ab.
lb
2a and (2h).
6 Conclusion
We have extended PSRL from a single agent multiple agents, where the agents can concurrent with each other to learn in parallel and share data with each other. We consider settings: both finite- horizon episodic MDP and infinite-horizon MDP. For state space size S, action space size A, total
accumulated time horizon T. episode horizon H, we show a Ö(HS AT per-agent Bayesian regret bound under general prior distribution; while especially for independent Dirichlet prior distribution, we show that tighter bound can be achieved Ö(H SAT Under infinite horizon case, for the
general prior distribution, we achieve Ö(DS
per-agent Bayesian regret bound; and for the
infinite-horizon case under dirichlet prior distribution, we obtain a Õ(D) ATS Bayesian regret bound, with D being the diameter of MDP considered here. We have provided theoretical upper bound for concurrent learning under posterior sampling RI setting, and the multiplicative constants in the bound were just properly chosen but not tuned in optimistic way. For future work, it would

---

Concurrent PSAL Finite Horizon Setting Km30 Hm75
5-20 A-10 H--1
800
H-75
5-5 A-3 K-20 It-36
8001sartini
200/s tin
500
400tscrtini
75/sgrtint
100
Segjet
60
200
60-
40-
100
20
20
62
100
Numbber Agents
(a) K=30, H=75
(b) K=20, H=10
Figure 1: Bayesian regrets for Finite-Horizon Concurrent TS with Dirichlet Prior for Transition Probabilities and Gaussian prior for rewards. Solid curves are Bayesian regrets of method vs number of concurrent agents. Dashed curves are reference curves given by our regret bounds O
Concurrent
Setting
Concument PSRL Setting 5-10 T-2000
190
5=10 Aw5 1=1000
5-10 T-2000
1235omt(
325/sgrtin)
100
70
Deges
40-
30-
10
Number
Agents
10
20
25
50
Number Agents
(a) S=10. A=S, T=1000
(b) S=10, A=5, T=2000
Figure 2: Bayesian regrets for Infinite-Horizon Concurrent TS with Dirichlet Prior for Transition Probabilities and Gaussian prior for rewards. Solid curves show Bayesian regrets of our method vs number of concurrent agents. Dashed curves are reference curves given by our regret bounds O 1
be interesting to include communication ((2) 30 and functional approximation concurrent
learning. One could evaluate this algorithm under model mis-specificatioon ((33)). It is also valuable explore model-free and worst-case bound settings 332 255 (7). Another extension is to
look at heterogeneous reward structure across agents, and compare the dynamic under the same reward structure setting for both PSRL and UCRL.
Acknowledgement
This work was supported in part by NSF award CCF-2106508. Zhengyuan Zhou acknowledges the generous support from Horizon Robotics faculty research fellowship and New York University's Center for Global Economy and Business faculty research grant. We would like thank Pratap Ranade and Engin Ural for inspiring discussions on society of agents that have helped shape and push an ambitious vision of this research agenda, for which this work is only initial step.
10

---

References
[1] Alckh Agarwal, Sarah Bird, Markus Cozowicz, Luong Hoang, John Langford, Stephen Lee, Jiaji Li, Dan Melamed, Gal Oshri, Oswaldo Ribas, et al. Making contextual decisions with low technical debt. arXiv preprint 1606.00396 2016.
[2] Mridul Agarwal, Bhargav Ganguly, and Vaneet Aggarwal. Communication efficient parallel reinforcement learning. In Uncertainty Artificial Intelligence, pages 247-256. PMLR, 2021.
[3] Shipra Agrawal and Randy Jia. Posterior sampling for reinforcement learning: worst-case regret bounds. arXiv preprint arXiv: 00040, 2017.
[4] Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning, Advances in neural information processing systems, 21, 2008
[5] Peter L Bartlett and Ambuj Tewari. Regal: A regularization based algorithm for reinforcement learning in weakly communicating mdps. arXiv preprint 2205.6611, 2012.
[6] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems, 24, 2011.
[7] Christoph Dann, Mehryar Mohri, Tong Zhang, and Julian Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems, 34:12040-12051, 2021.
[8] Maria Dimakopoulou, Ian Osband, and Benjamin Van Roy. Scalable coordinated exploration in concurrent reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
[9] Maria Dinakopoulou and Benjamin Van Roy. Coordinated exploration in concurrent reinforce- ment learning. In International Conference on Machine Learning, pages 1271-1279. PMLR, 2018.
[10] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on roboties and automation (ICRA), pages 3389-3396. IEEE, 2017.
[11] Zhaohan Guo and Emma Brunskill. Concurrent pac Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2015.
[12] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. In The collected works of Wassily Hoeffding, pages 409-426. Springer, 1994.
[13] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 29(2)2009-232, 2002.
[14] Michael Jong Kim. Thompson sampling for stochastic control: The finite parameter case. IEEE Transactions on Automatic Control, 62(12):6415-6422, 2017.
[15] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[16] Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.
[17] Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems, 27, 2014.
[18] Ian Osband and Benjamin Van Roy. Near-optimal reinforcement learning in factored mdps. Advances in Neural Information Processing Systems, 27, 2014.
[19] Ian Osband and Benjamin Van Roy. Gaussian-dirichlet posterior dominance in sequential learning. arXiv preprint arXiv: 26, 2017.
[20] Ian Oshand and Benjamin Van Roy. On optimistic versus randomized exploration in reinforce- ment learning, arXiv preprint
2017.

---

[21] Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International conference on machine learning, pages 2701-2710. PMLR, 2017.
[22] Ian Osband, Benjamin Van Roy, Daniel Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res., 20(124):1-62, 2019.
[23] Jason Pazis and Ronald Parr. Efficient pac-optimal exploration in concurrent, continuous state mdps with delayed updates. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[24]
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[25] Daniel Russo. Worst-case regret bounds for exploration via randomized value functions. Advances in Neural Information Processing Systems, 32, 2019.
[26] David Silver, Leonard Newnham, David Barker, Suzanne Weller, and Jason McFall. Concurrent reinforcement learning from customer interactions. International conference on machine learning, pages 924-932. PMLR, 2013.
[27] Sam Sinai, Richard Wang, Alexander Whatley, Stewart Slocum, Elina Locane, and Eric D Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.0214), 2020.
[28] Alexander Strehl and Michael L. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1 -131,, 2008.
[29] Malcolm Strens. A bayesian framework for reinforcement learning. In ICML, volume 2000, pages 943-950, 2000.
[30] Yuanhao Wang, Jiachen Hu, Xiaoyu Chen, and Liwei Wang. Distributed bandit learning: Near-optimal regret with efficient communication. arXiv preprint arXiv:1904.06300 2019.
[31] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the deviation of the empirical distribution. Newleet-Puckane Labs, Tech. Rep, 2003.
[32] Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics, pages 1954-1964. PMLR, 2020.
[33] Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834-857, 2022.
12


Reviews:
[{"review_id": "uQf4-mSv9NG", "reviewer": "Reviewer_yBVn", "summary": "This work extends PSRL to the concurrent multi-agent setting. The authors discuss the finite horizon and infinite horizon settings, establishing a $1/\\sqrt{n}$ per-agent regret, which is minimax optimal under a Dirichlet prior. For the infinite horizon case, the authors use a doubling schedule. In both settings the authors discuss results under general priors and under Dirichlet priors.", "questions": "Please see review above", "limitations": "The authors didn't discuss negative societal impact (I'm unsure how relevant they are either).\nRegarding limitations of their work, I didn't find a clear discussion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Paper provides an extension of PSRL to the concurrent case, whereas previous work was based on UCRL-type algorithms.\n- Proofs are provided for the theoretical results. The proofs seem correct, though clarity could be greatly improved.\n- Please add the assumption that $r \\in [0,1]$, even though it does not affect generality of the solution.\n- I recommend stating all formal lemmas used from other papers in the appendix. This will greatly increase overall clarity of the proofs.\n- It is hard to follow the proofs of the thoerems in the appendix. I recommend going over them too add some more explanations, and possibly divide them into a few more lemmas (if possible).\n- Please provide an explicit algorithm in the appendix, where bonuses and posterior definitions are provided.\n- Please find a way to put the simulations plots in the front matter - they should not be put in the appendix if they are referenced directly in the paper.\n- How do the results in this paper generalize to the frequentist regret setting?\n- The contribution of this paper is clear, but I am on the fence w.r.t. the overall strength of this paper. Particularly, theoretical aspects of the paper are standard and not surprising. The improvement the authors make over prior work is unclear. Overall, I worry that the extension the authors make is trivial w.r.t. current work.\n- Simulations in this paper are very weak. There are many interesting aspects to test, especially in the frequentist setting, which is usually the relevant setting in real-world problems.\n\nComments regarding proof of Theorem 3.1 (Appendix):\n\n- Line 519: where did this inequality come from?\n- Lines 525-526: I understand what the authors wanted to say, but please rephrase this statement to make it more percise and clear.\n- The use of lines 525-529, including Equations B.7 - B.10 is unclear in the final statement. \n- It is unclear what statements the authors are using in the final equations of the proof.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "rXTOHJdJMlq", "reviewer": "Reviewer_vsWP", "summary": "This paper studies the Bayesian regret bound of Concurrent Thompsom Sampling, where n agents interact with the environment simultaneously. The existing works are confined to the optimism-in-face-of-uncertainty principle (UCB). This paper presents the first attempt to analyze posterior sampling based algorithms, both in the episodic and infinite-horizon setting. \n\n", "questions": "See weakness part", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1 This paper provides the first line of theoretical analysis for TS-based algorithm in the concurrent episodic/infinite-horizon RL setting. \n2 While the analysis techniques seem to follow from the seminal works of Benjamin Van Roy, the resulting bounds are good as they are minimax-optimal.\n3 The theoretical guarantee is verified by simulation results, which is appreciated as empirical study has been largely ignored in the RL theory community.\n\nWeakness:\n1 Comparison to existing works. The paper motivates the use of TS by some empirical works, showing that TS can have advantage as agents can determine their policies individually, while UCB-type algorithms tend to use the same policy. (Line 72-77) This is a good motivation for the use of TS. However, it seems that the analysis of this paper does not verify this potential advantage over UCB-type algorithms theoretically. In particular, the paper does not contain a comparison between them. Are they all minimax-optimal? What is the advantages of TS-based algorithms over a UCB-based one?\n\n2 Simulation result. While it is good to see that the paper contains simulation result, the experiments are not satisfactory as they do not include the UCB-type algorithms as baseline. While I understand that a thoroughly comprehensive comparison between UCB and TS in theroy may need more involved analysis, I strongly recommend the authors to conduct simulations to verify the effectiveness of TS. \n\nI am willing to raise my score if my major concerns could be addressed.\n---------------------- update after rebuttal\nI have read the responses from the authors and increase the score to support the acceptance of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Cy6B1R9y3dS", "reviewer": "Reviewer_qFko", "summary": "The paper considers a concurrent RL setup, where a set of agents share experience when individually interacting in the same environment, and optimizing the same reward function. Thompson Sampling (TS) based exploration is considered and corresponding regret bounds are derived for the finite and infinite horizon setting. These bounds extend the TS guarantees for the single-agent RL case to the concurrent RL setup. Moreover, simple simulations are performed showing the computed regrets are of the same order of the obtained bounds.", "questions": " Even if agents have different reward functions, I believe the power of cooperation should still be visible because they interact with the same environment. In such a case:\n- I expect some of the guarantees to go through, but how would the overall results change?\n- UCRL (or similar) approaches should ensure increased exploration w.r.t. when agents have the same reward. Do you think there is a right measure (in the rewards' space) to quantify this?", "limitations": "I don't foresee potential negative societal impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have appreciated the motivations for the concurrent RL setup provided in the introduction, and I think the considered problem is interesting and relevant. Moreover, the paper is nicely written and the derived theoretical results are sound both intuitively and technically.\nHowever, I do have a concern: \n\n- I think most (if not all) of the claims made are under the assumption that the agents reward functions are the same?  This should be clarified better in the introduction. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6ktfbUfiwh", "reviewer": "Reviewer_WDrE", "summary": "This paper studies efficient Thompson sampling algorithm for concurrent RL. They mainly consider efficient learning in episodic setting and infinite-horizon setting. For both settings, they provide efficient learning algorithm for general prior and Dirichlet prior. Their regret bounds show that $\\sqrt{n}$ speed up is possible when there are $n$ agents sharing the information and exploring the environment simultanously.", "questions": "It would be helpful for evaluation if the authors could clearly explain the technical difficulties of concurrent PSRL and the methods they used to tackle them. ", "limitations": "The authors have explained several limitations in the conclusion sections.Other possible directions for improvements are listed as follows:\n- This paper does not tackle the communication cost in concurrent learning. It would be interesting to design efficient learning algorithms with efficient communication (e.g. [1,2]).\n- There are also works studying efficient PSRL with function approximation. I wonder whether $\\sqrt{n}$ speed-up is also possible in the function approximation setting.\n\n[1] Agarwal et al. Communication efficient parallel reinforcement learning.\n\n[2] Wang et al. Distributed Bandit Learning: Near-Optimal Regret with Efficient Communication.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nTo the best of my knowledge, this is the first work studying efficient Thompson sampling algorithm in concurrent setting. The algorithm and analysis is a natural extension of the previous work in the single-agent setting. The paper is well-written and the theoretical results are solid and reasonable.\n\nWeakness:\n\nThe authors fail to explain the technical difficulties of concurrent Thompson sampling compared with the single-agent setting. It seems that the algorithm and analysis almost directly follows that in [1,2]. Since the main contribution of this work is their theoretical improvement, I wonder whether there is anything novel in the algorithmic design and the proof technique.\n\n\n\n[1] Osband et al. Why is posterior sampling better than optimism for reinforcement learning?\n\n[2] Agrawal et al. Posterior sampling for reinforcement learning: worst-case regret bounds.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept