Federated Ensemble-Directed Offline Reinforcement Learning
Authors: Desik Rengarajan
Nitin Ragothaman
Dileep Kalathil
Srinivas Shakkottai
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Reinforcement Learning
Offline Reinforcement Learning
Federated Learning
Primary area: reinforcement_learning
TL;DR: A novel federated offline reinforcement learning algorithm

Abstract: We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected datasets generated according to different unknown behavior policies. Na\"{i}vely combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies.  In response, we develop the Federated Ensemble-Directed Offline Reinforcement Learning  Algorithm (FEDORA), which distills the collective wisdom of the clients using an ensemble learning approach.  We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on a mobile robot. We provide our code and a video of our experiments at \url{https://github.com/DesikRengarajan/FEDORA}.

Federated Ensemble-Directed Offline Reinforcement Learning
Desik Rengarajan' Nitin Ragothaman Department of Electrical and Computer Engineering, Texas A&M University
Dileep Kalathil Srinivas Shakkottai
Abstract
We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn high-quality control policy only using small pre-collected datasets generated ac- cording different unknown behavior policies. Naïvely combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies. In response, develop the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FE- DORA), which distills the collective wisdom of the clients using ensemble learning approach. We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the com- bined data pool, in various complex continuous control environments and real- world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on mobile robot. We provide our code and video of our experiments at httth:::/// //g thub. www DeeeeeeeaaaaaaaaaEaaDDD.
Introduction
Federated learning an approach wherein clients learn collaboratively by sharing their locally trained models (not their data) with federating agent, which periodically combines their models and returns the federated model to the clients for further refinement [10 30). Federated learning
has seen recent success in supervised learning applications due to its ability generate well-trained models using small amounts of data at each client, while preserving privacy and reducing the usage of communication resources. There has also been interest in federated learning for online reinforcement learning (RL), wherein clients learn via sequential interactions with their environments and federating learned policies across clients [2 are often infeasible, and each client might only posses pre-collected operational data generated according to a client-specific behavior policy. The fundamental problem of federated offine RI. is on how to learn the optimal policy only using such offline data collected by heterogeneous policies at clients, without actually sharing any of the data.
However, such online interactions with real-world systems
Offline RL algorithms [17, such as CQL [15] and TD3-BC [5] offer an actor-critic learning approach that only utilizes existing datasets at each client. However, in our case, this approach taken across many small datasets at clients will produce an ensemble of policies of heterogeneous (unknown) qualities across the clients, along with their corresponding critics of variable accuracy. We will see that naively federating such offline RL trained policies and critics using standard federation approach, such as FedAvg (20] can lead a policy that even worse than the constituent policies. We hence identify the following basic challenges of federated offline RI.: (i) Ensemble heterogeneity: Heterogeneous client datasets will generate policies of different performance levels. It is vital to capture the collective wisdom of this ensemble of policies, not average them. (ii) Pessimistic value
Cooresponding author. Email: desik. 290gmai1. com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

computation: Offline RL employs pessimistic approach toward computing the value of actions poorly represented in the data to minimize distribution shift (and so reduce the probability of taking these actions). However, federation must be ambitious in extracting the highest values as represented in the ensemble of critics (and so promote high-value actions). (iii) Data heterogeneity: As with other federated learning, multiple local gradient steps based on heterogeneous data at each client between federation rounds may lead to biased models. We must regularize local policies to reduce such drift.
In this work, we propose Federated Ensemble-Directed Offline RL Algorithm (FEDORA), which collaboratively produces a high-quality control policy and critic function. FEDORA estimates the performance of client policies using only local data (of unknown quality) and, at each round of federation, produces weighted combination of the constituent policies that maximizes the overall objective, while regularizing by the entropy of the weights. The same approach is followed to federate client critics. Following the principle of maximum entropy in this manner produces both federated policies and critics that extract the collective wisdom of the ensemble. In doing so, it constructs federated policy and a critic based on the relative merits of each client policy an ensemble learning manner. FEDORA ensures optimism across evaluation by the federated and local critic at each client and so sets ambitious targets to train against. It addresses data heterogeneity by regularizing client policies with respect to both the federated policy and the local dataset. Finally, FEDORA prunes the influence of irrelevant data by decaying the reliance on a dataset based on the quality of the policy can generate. To the best of our knowledge, other work systematically identifies these fundamental challenges of offline federated RL, or designs methods to explicitly tackle each of them.
We develop a framework for implementing FEDORA either on single system or over distributed compute resources. We evaluate FEDORA on a variety of MuJoCo environments and real-world datasets and show that it outperforms several other approaches, including performing offline RL on pooled dataset. We also demonstrate FEDORA's excellent performance via real-world experiments on a TurtleBot robot ①.
2 Related Work
Offline RL:The goal of offline RL is learn a policy from a fixed dataset generated by a behavior policy [17]. One of the key challenges of the offline RL approach is the distribution shift problem where the state-action visitation distribution of learned policy may be different from that of the behavior policy which generated the offline data. It is known that this distribution shift may lead to poor performance of the learned policy [77. A common method used by offline RL algorithms to tackle this problem is to learn a policy that is close to the behavior policy that generated the data via regularization either on the actor or critic (5] 7 16 14 33 Some offline RL algorithms perform weighted versions of behavior cloning or imitation learning on either the whole or subset of the
dataset [1 22 3 38 37 propose data rebalancing methods designed to prioritize nighly-rewarding
transitions that can be augmented to offline RL algorithms to alleviate the distribution shift issue for heterogeneous data settings.
Federated Learning: (20) introduced FedAvg, a federation strategy where clients collaboratively learn a joint model without sharing data. generalized version of FedAvg was presented in (25 A key problem in federated learning is data heterogeneity wherein clients have non non-identically distributed data, which causes unstable and slow convergence
[8] To tackle the issue of data heterogeneity, [18) proposed FedProx, a variant of FedAvg, where a proximal term is introduced reduce deviation by the local model from the server model.
Federated learning has recently been extended to the online RL setting. [12 analyzed the performance federated tabular Q-learning. [23] combined traditional online RL algorithms with FedAvg for multiple applications. Some works propose methods to vary the weighting scheme of FedAvg according to performance metrics such as the length of a rally in the game of Pong (21) or average return in the past 10 training episodes [19] to achieve better performance or personalization. [32) proposed method to compute weights using attention over performance metrics of clients such as average reward, average loss, and hit rate for an edge caching application. {8 used a transformer encoder to learn contextual relationships between agents in the online RL setting. [9] proposed an alternative approach to federation where reward shaping is used to share information among clients. [3 proposed a KI. divergence-basee regularization between the local and global policy to address the issuc of data heterogeneity in an online RL setting.
Federated Reinforcement Learning:

---

In the offline RL setting, [39 propose federated dynamic treatment regime algorithm by formulating offline federated learning using a multi-site MDP model constructed using linear MDPs. However, this approach relies on running the local training to completion followed by just one step of federated averaging. Unlike this work, our method does not assume linear MDPs, which a limiting assumption in many real-world problems. Moreover, we use the standard federated learning philosophy of periodic federation followed by multiple local updates. To the best of our knowledge, ours is the first work to propose general federated offline RL algorithm for clients with heterogeneous data.
3 Preliminaries
Federated Learning: The goal of federated learning is to minimize the following objective,
F(0) = E.-p [F)
(1)
where 0 represents the parameter of the federated (server) model, F; denotes the local objective function of client i, and P is the distribution over the set of clients N. The FedAvg algorithm [20] is a popular method to solve Eq. (T) in a federated way. FedAvg divides the training process into rounds, where at the beginning of each round f, the server broadcasts its current model le to all the clients, and each client initializes its current local model to the current server model. Clients perform multiple local updates on their own dataset D; to obtain updated local model 0 The server then averages these local models proportional to the size of their local dataset to obtain the server model 0c-1 for the next round of federation,
W W 17
W
D D
0++1
W
=
|D|
=
D.--
Reinforcement Learning: We model RL using the Markov Decision Process (MDP) framework denoted tuple A, R, P. 14), where S is the state space, A is the action space, R A I the reward function, and P : AxS -> [0, state s' by taking action in state s, y is the discount factor, and Jı is
denotes the transition probability function that gives
the probability of transitioning the distribution of the initial state 8p- A policy æ is a function that maps states to actions (deterministic policy) or states to distribution over actions (stochastic policy). The goal of RL is to maximize the infinite horizon discounted reward of policy ग, defined as J(x) = P,, 12 which is the expected cumulative discounted reward obtained by executing policy x. The state- action value function (or Q function) of policy ı at state and executing action is the expected cumulative discounted reward obtained by taking action in state s and following policy π thereafter: Q* (s, a) = E E 7 R(st. ee) s, ao = a).
= R[(s,, at)],
Offline Reinforcement Learning: The goal of offline RL is learn policy æ only using a static dataset D of transitions (s. a, r, s') collected using a behavior policy Tb without any additional interactions with the environment. Offline RL algorithms typically utilize some kind of regularization with respect to the behavior policy to ensure that the learned policy does not deviate from the behavior policy. This regularization is done to prevent distribution shift, significant problem in offline RL, where the difference between the learned policy and behavior policy can lead to erroneous Q-value estimation of state-action pairs not seen in the dataset [16 17].
Our approach is compatible with most offline RL algorithms, such as CQL [5] or TD3-BC [5). We choose TD3-BC for illustration, motivated by its simplicity and its superior empirical performance in benchmark problems. The TD3-BC algorithm is a behavior cloning (BC) regularized version of the TD3 algorithm 6 The policy in TD3-BC is updated using linear combination of TD3 objective and behavior cloning loss, where the TD3 objective ensures policy improvement and the BC loss prevents distribution shift. More precisely, the TD3-BC objective can be written
arg max Up (TT,
(3)
where (T( E, DDD [QQ (s) - (T(s) a)²].
(4)
and X is hyperparameter that determines the relative weight of the BC term.
3

---

4 Federated Offline Reinforcement Learning
In real-world offline RL applications, data is typically obtained from the operational policies of multiple agents (clients) with different (unknown) levels of expertise. Clients often prefer not to share data. We aim to learn the optimal policy for the underlying RL problem using only such offline data, without the clients knowing the quality of their data, or sharing it with one another or the server. Furthermore, neither the clients nor server have access to the underlying model or the environment. We denote the set of clients as N. Each client E N has the offline dataset D; {(8j, ajz Ty 6/) <²) generated according to a behavior policy x.. We assume that the underlying MDP model P and reward function -(--)) are identical for all the clients, and the statistical differences between the offline datasets Dj are only due to the difference in behavior policies X.! used for collecting the data.
In a standard federated learning algorithm such as FedAvg, each client performs multiple parameter updates before sending its parameters to the server. It is known that performing multiple local updates in federated learning can reduce the communication cost significantly without compromising on the optimality of the converged solution [10 (3).. In federated offline RL, since each client has to perform multiple steps of policy evaluation and policy update using its local offline data D; it is reasonable to consider a client objective function that is consistent with a standard offline RL algorithm objective. We choose the objective function used in the TD3-BC algorithm
(5), i.e., UD, given Eq. 3 as the client objective function. Our choice is motivated by the simplicity of the TD3-BC objective function
and its empirical success variety of environments. Similar to the standard federated learning objective given in Eq. (I. can now define the federated offline RL objective as
W
U(Tee))
Σ
w W,UD, (Tea))
(5)
where w; are weights to be determined.
One approach to leveraging experiences across users without sharing data would be to combine existing federated learning techniques with offline RL algoritmms. Is such nuive federation strategy sufficient to learn an excellent federated policy collaboratively? Furthermore, is federation even necessary? In this section, we aim to understand the challenges of federated offline RL with the goal of designing an algorithmic framework to address these challenges.
We start by illustrating the issues in designing a federated offline RL algorithm. We consider the Hopper environment from MuloCo (28). with [M| = 10, |D,| = 5000, and we use the data from the D4RL dataset [4. However, instead of using the data generated by the same policy for all clients, we consider the setting where five clients use the data from the hopper-expert--2 dataset (which was generated using a completely trained (expert) SAC policy) and five clients use the data from the dataset (which was generated using a partially trained (medium) policy achieving only third of the expert performance). The clients and the server unaware of the quality (expert or medium) of the data. Fig. [ shows the performance comparison of multiple algorithms, where the mean and the standard deviation are calculated over 4 seeds.
hopper-mediunt
Combining All Data (Centralized)): Combining data and learn- ing centrally is the ideal scenario in supervised learning. How- ever, as seen in Fig. T performing centralized training over combined data generated using different behavior policies in offline RL can be detrimental. This consistent with (36 that proves that pooling data from behavior policies with different expertise levels can exacerbate the distributional shift between the learned policy and the individual datasets, leading to poor performance. Similar deterioration due to combining data has also been observed in other offline RL literature [5] [6]. We also explore centralized algorithms with data re- balancing. and observer that FEDORA is still superior (See Appendix [B.9).
Contrullind
Indivikall
Fediic
2400
1405
2009
Figure 1: Performance comparison of federated and centralized offline RL algorithms.
We would like to further emphasise that combining the data from all clients is a hypothetical base in the federated setting, data is distributed amongst clients and cannot be combined.
Individual Offline RL: Here, agents apply offline RL to their own datasets without collaborating
with others. In Fig. observe that clients with either expert medium data do not learn well and

---

exhibit a large standard deviation. This observation may be attributed to no client having sufficient data to learn a good policy.
Naïve Federated Offline RL: A simple federation approach is to use the offline RL. objective as the local objective and apply FedAvg (Eq. (2). However, offline RL algorithms typically comprise two components an actor and a critic. It is unclear priori which components should be federated, so we conduct experiments where we federate only the actor (Fed-A) or both the actor and the critic (Fed-AC). Surprisingly, these naïve strategies result in federated policies that perform worse than individual offline RL, as seen in Fig.
4.1 Issues with Federated Offline RL
Our example illustrates several fundamental issues that must be addressed while designing viable federated offline RL algorithms, including:
Ensemble Heterogeneity: Performing offline RL over heterogeneous data yields a set of policies of different qualities. It is crucial to leverage the information contained in these varied policies rather than simply averaging them. However, federation after a single-step local gradient at each client using weights in the manner of FedAvg, W, = [Dil/| M [Dal. is equivalent to solving the offline
RL problem using the combined dataset of all clients [30 This approach leads to poor performance
due to the resulting distribution shift, as shown in Fig. How should we optimally federate the
ensemble of policies learned by the clients?
2. Pessimistic Value Computation: Most offline RL algorithms involve pessimistic term with respect the offline data for minimizing the distribution shift. Training client critic using only the local data with this pessimistic term could make it pessimistic towards actions poorly represented in its dataset but well represented in other clients' data. How do we effectively utilize the federated critic along with the locally computed critic to set ambitious targets for offtine RL at each client?
3. Data Heterogeneity: Federated learning calls for performing multiple local gradient steps at each client before federation to enhance communication efficiency. However, numerous epochs would bias client's local model its dataset. This client drift effect is well known in federated (supervised) learning and could lead to policies that are not globally optimal. In turn, this could cause the federated policy's 's performance to be worse than training locally using only the client's data, seen in Fig. How should we regularize local policies to prevent this?
5 FEDORA Design Approach
We desire develop Federated Bassemble-Directer Offline RL Algorithm (FEDORA) that addresses the issues outlined in Section | in a systematic manner. Three fundamental requirements drive our approach. First, the clients jointly possess ensemble of local policies of different (unknown) qualities, and the server must leverage the collective knowledge embedded in this ensemble during federation. Second, the quality of these policies must be assessed using ensemble of crities that depend on local data for policy evaluation. Finally, after each round of federation, clients must update their local policies via offline RL utilizing both their local data and the received federated policy.
Maximizing the federated offline reinforcement learning (RL) objective in Eq. (5) using FedAvg would set weights as in Eq. (2) i.e., each client's contribution is weighted by the size of its dataset. This is is equivalent to solving the offline RL problem using the combined dataset of all clients. However, such an approach exacerbates the distribution shift problem that affects offline RL algorithms. leading to poor performance, This issue has been verified analytically and empirically in (36). We illustrated this phenomenon in Fig. I where offline RL over pooled data resulted in a sub-optimal policy. The
recommendation in B6 is to share data conservatively by identifying which samples are likely to result in policy improvement. However, we cannot share any of the data across clients in the federated offline RL setting.
Our solution is to follow the principle of maximum entropy to choose weights that best represent the current knowledge about the relative merits of the clients' policies. Here, the weights are prevented from collapsing over a few clients that have the best current performance by adding an entropy
5

---

regularization over the weights with temperature parameter 3 resulting in the following objective:
IA
(Teed)
WrUDD (Tedd)
Σ
wy log wi.
(6)
3
2=1
We can then show using a Lagrange dual approach that this objective is maximized when
BUD, (Miat)
W/3 Σ W @UD, (Tee)
Based on these soft-max type of weights suggested by the entropy-regulatee objective, we now design FEDORA accounting for each of the three requirements indicated above.
In what follows, (1.b) denotes the policy of client in round t of federation after k local policy
update steps. Since all clients initialize their local policies the federated policy the beginning of each round of federation, (f.0) * for each client i. We also denote oद = =; (t.K) where is the maximum number of local updates. Since all clients initialize their local critics to the federated critic,
we can similarly define ②, (t.bk) Q (e Viea. and Q = Q!!.K) for the local critic.
5.1 Ensemble-Directed Learning over Client Policies
We first require means of approximating Up, (Tiea) in order determine the weight w, of client as shown in Eq. 7 We utilize the performance of the final local policy J = E,wD, [Q] n/ r a proxy for UD, (Ted).. Here, Q is
which also characterizes the relative performance at client the local critic function at round after K local updates. It is hard to directly obtain such an unbiased local critic O offline RI., since we do not have access to the environment for executing the policy and evaluating its performance. Our approach toward computing Qi and xf are described later. The accuracy of the local estimates Jí are highly dependent on the number of data samples available i, and so in the usual manner of federated averaging, we need to account for the size of the dataset [D,] while computing weights. We thus have client weights and federated policy update as
eBJt Dri
I
1+1 Fed
Σ
W/T!
(8)
MMA D.
5.2 Federated Optimism for Critic Training
The critic in our algorithm plays major roles. First, offline RL for policy updates each client requires policy evaluation using local data. Second, policy evaluation by the critic determines weight w/ of the local policy client i for ensemble learning during each round of federation. We desire local critic at each client that can utilize the knowledge from the ensemble of critics across all clients while also being tuned the local data used for policy evaluation.
critic based on offline data suffers from extrapolation errors as state-action pairs not seen in the local dataset will be erroneously estimated, greatly impacting actor-crite style policy updates in federated offline RL. Since the federated policy is derived from the set of local policies, it may take actions not in any client's is local dataset. This problem is exacerbated when the local policy the beginning of each communication round is initialized to the federated policy. We introduce the notion of federated optimism to train local critics, wherein critics leverage the wisdom of the crowd and are encouraged to be optimistic. We achieve this federated optimism via two steps.
First, we use an ensemble-directed federation of the critics, where the local critic of client at round t weighed according to its merit to compute the federated critic as
Σ
Q
Such regropy-regularizzed averaging ensures that the critics from clients with good policies signifi- cantly influence the federated critic.

---

FEDORA
Fedi AC-Prox
Fed AC
FediA
HDAFL
Centralized
HalfCheetah
Hopper
Walker2D
10000
5000
Server Reward
3030
6000
2030
3000
4000
2000
1000
0
200
4.0
800
8:0
1000
200
4.00
800
800
1000
200
400
6:00
800
1060
Communication Round
Communication Round
Communication Round
Figure 2: Evaluation of algorithms on different MuJoCo environments.
Second, for the local critic update, we choose the target value as the maximum value between the local critic and the federated critic, given by @((.() (s, a) max (Q.thk) (s, Qia,,,, Qia((s,)) a)): where Q (s, a) is the target value of state and action the tt round of federation after & local critic updates. This ensures that the local critic has an optimistic (but likely feasible) target seen by the system. Using this optimistic target in the Bellman error, update the local critic as
of (lb++)) = arg min E ((sanns)==,((( [(r )( -d.
(10)
Q
where a' (1.b)) In practice, we obtain ((*k+())) after a single gradient update.
5.3 Proximal Policy Update for Heterogeneous Data
While essential in order ambitious estimates, an optimistic critic might erroneously estimate the value (!..)) Therefore, regularizing the local policy update w.r.t. both the local data and the federated policy is crucial. For regularization w.r.t. to the local offline data, use the same method as in the TD3-BC algorithm and define the local loss function Liccal (T)) (s,,a)_D, Q (s, T (s))
(T(s) - J. We then define the actor loss Lactor in Eq. (II). where the second term is regularization w.r.t. to the federated policy. The local policy is updated using Lactors
Lactor = (T) + T-E (( - (s)){, (,+1 arg min Lactor
(11)
5.4 Decaying the Influence of Local Data
FEDORA combination of local data loss and proximal term for its policy update Eq. (T). However, the local data loss might hamper the updated policy's performance since the local dataset may be generated according to non-expert behavior policy. Hence, client must decay the influence of its local data if it is reducing the performance of the updated policy by lowering the influence of Liocal Lactor- To do so, first evaluate the performance of the federated policy using the federated critic and local data at round t. For this evaluation, we use the proxy estimate ylod.! RE,D: (s. (ffa(s))),, We compare this value with the performance of the updated policy, JR, which is obtained using the updated critic. This difference provides us with an estimate of the improvement the local data provides. We decay the influence of L by factor 8 if fledd.t 2 J
We summarize FEDORA in Algorithm | and 2
We would like to emphasize that in our algorithm, as in any offline RI setting, the clients server do have access the environment the MDP. Further, would like to point out that the clients not aware of the quality of data they possess.
7

---

Algorithm Outline of Client i's Algorithm
Algorithm 2 Outline of Server Algorithm
1: function cliienteeettrre lea Pica)
1: Initialize Tie Qiod
2: Ti
(t.D) Fied Q,tx00 = Qiae
2: for / E 1El... 1.... do
3: for 1 S.
K do
3: Send Tied and Of toiE
4: Update Critic by one gradient step w.r.t.
4: Sample M₁ C.
Eq. (TO)
5: for i E Nu do
5: Update Actor by one gradient step w.r.t.
6: i.train _liintteletttteeee (ii Qisa) (Client side)
Eq. (11)
7: end for
6: end for
8: Compute m and Q for clients in Me
7: Decay Liocal by ō if Jlalt 2
using Eq. (8) and (9) respectively.
8:
end function
9: end for
6 Experimental Evaluation
We conduct experiments to answer three broad questions: (i) Comparative Performance: How does FEDORA perform compared to other approaches with client data generated by heterogeneous behavior policies?, (ii) Sensitivity to client updates and data quality: How does the performance depend on the number of local gradient steps at clients, the randomness the available number of agents for federation, and the quality of the data at the clients?, and (iii) Ablation: How does the performance depend on the different components of FEDORA? We implement FEDORA over the Flower federated learning platform (2) which supports learning across devices. We also provide simulation setup that can be executed on a single machine (See Appendix(A)
The local objective of all clients follows TD3-BC (Eq. 3) The server performs FedAvg over the actor's parameters, whereas each client learns the critic locally. (ii) Fed-AC: The local objective of all clients follows TD3-BC and the server performs FedAvg over the parameters of both the actor and the critic. (iii) Fed-AC-Prox:: We add proximal term to Fed-AC, which has been shown to help in federated supervised learning when clients have heterogeneous data (18). (iv) Heterogeneous Data-Aware Federated Learning (HDAFL) We extend HDAFL (35] to the offline RL setting by dividing the actor network into generic and speni-sp parts and then federating only the generic part during each round. (v) Centralized: We perform offline RL (TD3-BO BC) over the pooled data by combining the data present in all clients.
Baselines: We consider the following baselines. (i) Fed-A:
6.1 Experiments on Simulated Environments
We focus on a scenario where clients are collaboratively learning to solve the same task, but the behavior policies used to collect data for each client could differ. We run experiments with the number of clients [ = 50, with each client having a local dataset of size [D,] = 5000. Of these 50 clients, 25 are provided with data from the D4RL 4 expert dataset, while the other 25 are provided with data from the D4RI. medium dataset. The clients (and the server) are unaware of the quality of their datasets. Further, both the client and server do not have access the environment. We choose [ = 20 clients at random to participate each round f of federation. The server obtains weights from clients in [W] and computes the federated weight "fed r++1 and Q
Experimental Setup:
For each plot, we evaluate the performance with four different seeds. We evaluate the performance of FEDORA A and baselines over three MuJoCo tasks: Hopper, HalfCheetah, and Walker2D. During round of federation, each client performs 20 epochs of local training in all algorithms, which is roughly 380 local gradient steps in our experimental setup.
Comparative Performance of FEDORA: In Fig. 2 plot the cumulative episodic reward of the server/federated policy during each round of communication/icceratiii
We observe that FEDORA outperforms all federated baselines and achieves performance equivalent to or better than centralized training. Furthermore, the federated baselines fail to learn a good server policy even after training for many communication rounds and plateau at lower levels compared to FEDORA, emphasizing that the presence of heterogeneous data hurts their performance.
To understand the effect of data coming from multiple behavior policies centralized training, we consider scenario where 50 clients with datasets of size [D,| 5000 participate in federation, with 25 clients having expert data and the other 25 having random data, i.c., data generated from
8

---

epoch
eppeht
spochs
15% cletts
35% ciants 50% Clents
20% ecpert espert
epochs
20 spochs
20% clents ++- 40% dients
30% expett ++- 57% expert
Varying local epachs
Varying porticipating clients
Varying the number of expert dients
35:00
3500
Servar Boward
3000
3000
3000
2300
23:00
2300
2000
2303
1500
1500
2000
1000
1000
500
250
300
1200
250
730
1000
250
750
1000
Communicaton Round
Communication Round
Communication Round
Figure 4: Effect of varying the number of (a) local gradient steps, (b) participating clients in each round, and (c) expert clients in FEDORA.
random policy. From Fig. compared to FEDORA. This observation highlights the fact that performing centralized training with data collected using multiple behavior policies can be detrimental.
notice that combining data of all clients deteriorates performance as
Start Targe:
Behavior
Behavior
Stort
Fed Prox
Fed-A HDAFL
Bahavier or
Bahav
Target rEDORA
ted AC
0.50
HEAPL
0.75
0.2%
30
0.30
3.00
Secarr Reward
0:25
40.25
Rebot
3.75
192
4.50
-1.00
-0.35
45
02
100
-550-6225 0:00 0.25 255 5.55 1.00 1,25 1.50
-051-6.25 3.00 6.25 0.50
3.75 3.00 1.25 1:50 Robot pose O.))
Rebet pose (m)
(c) Comparison of FEDORA with
(a) Trajectories hehavior policies (b) Trajectories of learned policies federated baseline algorithms
Figure 5: Evaluation of FEDORA and other federated baselines for mobile robot navigation task in the presence of an obstacle.
Sensitivity to Client Updates and Data Quality: We study the sensitivity of FEDORA to client update fre- quency and data quality in the Hopper environment the same setting in Fig. 2 Increasing the number of local training steps can improve communication efficiency, but is detrimental under heterogeneous data due to client drift [1] In Fig. 4a), we study the effect of varying the number of local training epochs. We observe that in- creasing the number of epochs leads to faster learning, emphasizing that FEDORA erogeneous data. Not all clients may participate in every round of federation due communication/compute con- straints. In Fig 4b), we study the effect of the fraction of clients participating in federation. We observe that FEDORA is robust towards variations in the fraction of clients during federation. Finally, in Fig. [cc) we study the effect of data heterogeneity by varying the percentage of clients with expert datasets. We observe that FEDORA performs well even when only 20% of the total clients have expert-quality data. We present several ablation studies and additional experiments in appendix B
HallCheetah
Hopper
WalkerZD
12033
5000
4000
2500
21000
2000
2500
2000
3000
2000
2000
Figure 3: Comparison of FEDORA and centralized training with heterogencous data.
effectively learn with het-
6.2 Real- World Experiments on TurtleBot
We evaluated the performance FEDORA on TurtleBot [1. two-wheeled differential drive robot (Fig. 5) to collaboratively learn control policy to navigate waypoints while avoiding obstacles using offline data distributed across multiple robots (clients). This scenario relevant to several real-world applications, such as cleaning robots in various houses, which aim to collaboratively

---

learn control policy to navigate and avoid obstacles using data distributed across different robots. Collaborative learning is essential, because single robot might not have enough data to learn from or have encountered adequately different scenarios. Additionally, federated learning overcomes the privacy concerns associated with sharing data among the robots.
We collect data in the real-world using four behavior policies with varying levels of expertise Fig. [5 a))) We train over 20 clients for 100 communication rounds, each consisting of 20 local epochs (see Fig. ()) Fig. () shows the trajectories obtained by the learned policies of different algorithms in the real-world, and only FEDORA is able to successfully reach the target by avoiding the obstacle. We provide more details Appendix [C] We provide a video of our experiments
at pattps://githhe com Des htwp://eiiiooggrajaaaaaDD.D
Figure 6: Turtle- Bot3 Burger.
7 Conclusion
We presented an approach for federated offline RL, accounting for the heterogeneity in the quality of the ensemble of policies that generated the data at the clients. We solved multiple challenging issues by systematically developing a well- well-performing ensemble-directed approach entitled FEDORA. which extracts the collective wisdom of the policies and critics and discourages excessive reliance on irrelevant local data. We demonstrated its performance on several simulation and real-world tasks.
8 Ethics Statement and Societal Impacts
In this work, we introduce a novel algorithm for federated offline reinforcement learning. The domain of federated offline RL offers the potential for widespread implementation of RL algorithms while upholding privacy by not sharing data, as well as reducing the need for communication. Throughout our study, no human subjects or cuman--generatee data were involved. As result, we do not perceive any ethical concerns associated with our research methodology.
While reinforcement learning holds great promise for the application in socially beneficial systems, caution must be exercised when applying it to environments involving human interaction. This caution arises from the fact that guarantees in such scenarios are probabilistic, and it is essential to ensure that the associated risks remain within acceptable limits to ensure safe deployments.
Limitations and Future work
In this work, examine the issue of Federated Offline RL. We make the assumption that all clients share the same MDP model (transition kernel and reward model), and any statistical variances between the offline datasets Moving forward, we aim to broaden this to cover scenarios where clients have different transition and reward models. To achieve this, we plan extend ideas from offline meta RL to the federated learning scenario. Furthermore, we plan to explore personalization in federated offline RL as an extension to our research. We also believe that our approach may also be useful in the context of federated supervised learning, especially when the data is sourced from varying qualities, and we intend formally investigate this in the future separate line of work.
due to differences in the behavior policies used to collect the data.
10 Acknowledgement
This work was supported in part by NSF Grants CNS 2312978, ECCS 2038963, ARO Grant W911NF-19-1-0366 and ESF-CCARER--ECN-225555 or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsoring agencies.
Any opinions, findings, and conclusions
Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing.
10

---

References
[1] Robin Amsters and Peter Slacts. Turtlebot as a robotics education platform. In Robotics in Education: Current Research and Innovations, pages 170-181. Springer, 2020.
[2] Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, and Nicholas D Lane. Flower: A friendly federated learning research framework. arXiv preprint 2Xiv:2007.143900 2020.
[3] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 13:18333-18363 2020.
[4] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint vrXiv:2004.072199 2020.
[5] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 14:20132-2011 2021.
[6] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596, 2018.
[7] Scott Fujimoto, David Meger, and Doina Precup. Off-pooicc deep reinforcement learning without exploration. In International conference machine learning, pages 2052-2062. 2019.
[8] Liam Hebert, Lukasz Golab, Pascal Poupart, and Robin Cohen. Fedformer: Contextual federation with attention in reinforcement learning. arXiv preprint arXiv:2205. 136977 2022.
(9] Yiqiu Hu, Yun Hua, Wenyan Liu, and Jun Zhu. Reward shaping based federated reinforcement learning. IEEE Access, 9:67259-6722 67267, 2021.
[10] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1-22 1-100, 2021.
[11] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143, 2020.
[12] Sajad Khodadadian, Pranay Sharma, Gauri Joshi, and Siva Theja Maguluri. Federated rein- forcement learning: Linear speedup under markovian sampling. International Conference on Machine Learning, pages (0997-11057) 2022.
[13] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit lelaamnig. arXiv preprint arXiv:2110.06/60 2021.
[14] Aviral Kumar, Justin Fu, Matthew Soh. George Tucker, and Sergey Levine. Stabilizing off- policy q-learming via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[15] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforeement learning. Advances in Neural Information Processing Systems, 33:1179. 1191, 2020.
[16] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179- 1191, 2020.
[17] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01645 2020.
[18] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 4500, 2020.

---

[19] Hyun-Kyo Lim, Ju- Bo-B Kim, Ihsan Ullah, Joo-Seong Heo, and Youn- Han. Federated reinforcement learning acceleration method for precise control of multiple devices. IEEE Access, 9776296-76366, 2021.
[20] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication--fficcae learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282, 2017.
[21] Chetan Nadiger, Anil Kumar, and Sherine Abdelhak. Federated reinforcement learning for fast personalization. 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pages 123 IEEE, 2019.
[22] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weiggteed regresssin Simple and scalable off-policy reinforcement learning. arXiv preprint 1xXiv:19/0000177,,0000 2019.
[23] Jiaju Qi, Qihao Zhou, Lei Lei, and Kan Zheng, Federated reinforcement learning: techniques, applications, and open challenges. arXiv preprint arXiv:2108. 11887, 2021.
[24] Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu. Shengkai Huang. Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. arXiv preprint arXiv:2102.00711 00714, 2021.
[25] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Leurning Representations, 2021.
[26] John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference Machine Learning, volume 37, pages 1899-1897, 2015.
[27] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-orientte deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as a "Launchpad", 2022.
[28] Emanuel Todorov, Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In JEEE/RSJ international conference on intelligent robots and systems, pages 5026- 5033, 2012.
[29] José R Vázquez-Cantelle Cantell, Sourav Dey, Gregor Henze, and Zoltán Nagy Citylearn: Standard- izing research in multi-agent reinforcement learning for demand response and urban energy management. arXiv preprint arXiv:2012. 10504, 2020.
[30] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al- Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. field guide to federated optimization. arXiv preprint arXiv: 107.069777 2021.
[31] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 2018
[32] Xiaofei Wang, Ruibin Li, Chenyang Wang. Xiuhua Li, Tarik Taleb, and Victor CM Leung. Attention-weighted federated deep reinforcement learning for device-to-device assisted het- erogeneous collaborative edge caching. IEEE Journal on Selected Areas in Communications, 39(1):154-169, 2020.
[33] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, arXiv preprint 19Xiv:1911.11382 2019.
[34] Zhijie Xie and Shenghui Song. Fedkl: Tackling data heterogeneity in federated reinforcement learning by penalizing kl divergence. IEEE Journal on Selected Areas in Communications, 120::1222212222 2023.
[35] Lixuan Yang, Cedric Beliard, and Dario Rossi. Heterogeneous data-aware federated learning. arXiv preprint arXiv:2011.0639) 2020.
12

---

[36] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:1 1501-11516, 2021.
[37] Yang Yue, Bingyi Kang, Xiao Ma, Gao Huang, Shiji Song, and Shuicheng Yan. Offline prioritized experience replay. arXiv preprint arXiv:2306.05412 2023.
[38] Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint arXiv:22/0.09241, 2022.
[39] Doudou Zhou, Yufeng Zhang, Aaron Sonabend-V W. Zhaoran Wang, Junwei Lu, and Tianxi Cai. Federated offline reinforcement learning. arXiv preprint a7X5: 2022.
13

---

Appendix
We present several results and details in the appendix that illustrates the performance of FEDORA. These include details of our experimental setup (Appendix A). additional experiments studying different components FEDORA and illustrating its performance different settings (Appendix and details of our real-world experiments using TurtleBot (Appendix C
A Experimental Setup
We the PyTorch framework program the algorithms in this work, based on a publicly-available TD3-BC implementation. The actor and the critic networks have two hidden layers of size 256 with ReLu non-linearities. We discount factor of 0.99, and the clients update their networks using the Adam optimizer with a learning rate of 3 x 10-1 For training FEDORA, we fixed the decay rate å 0.995 and the temperature 3 = 0.1. TD3-BC trains for 5 x 10 time steps in the centralized setup. The batch size is 256 in both federated and centralized training.
Algorithm Implementation:
The training data for clients are composed of trajectories sampled from the D4RL dataset. In situations where only a fraction of the clients partake in round of federation, uniformly sample the desired number clients from the entire set.
Federation Structure: We implement FEDORA over the Flower federated learning platform (2). which supports learning across devices with heterogeneous software stacks, compute capabilities, and network bandwidths. Flower manages all communication across clients and the server and permits us to implement the custom server-side and client-side algorithms of FEDORA easily. However, since Flower is aimed at supervised learning, it only transmits and receives a single model at each federation round, whereas we desire to federate both policies and critic models. We solve this limitation by simply appending both models together, packing and unpacking them at the server and client sides appropriately.
While "FEDORA-DO FlloerrFlowerr is an effective solution for working across distributed compute resources, we also desire simulation setup that can be executed on a single machine. This approach sequentially executes FEDORA at each selected client, followed by a federation step, thereby allowing us to evaluate the different elements of FEDORA in an idcalized federation setup.
Each run on the MuJoCo environments (as in Fig. 2 takes around 7 hours complete when run single machine (AMD Ryzen Threadripper 3960X 24-Core Processor, 2x NVIDIA 2080Ti GPU). This time can be drastically reduced when run over distributed compute using the Flower framework.
Compute Resources:
B Additional Experiments
B.1
Importance Individual Algorithm Component
FediA
4. 13) with optimistic critic
weighted actor
decay
Fed-WC
[4] with proximal term dacay [FEDORA]
optimistic
FEDORA
weightad
proximal
3500
3500
Beword
3000
Server Reward
3000
2500
23:00
0.000
2000
000
2000
1500
06.90
1000
0001
1000
5000
0.
500
00000
500
0000
0001
0000
(a) Effect of sequentially adding
(b) Effect of removing individual algorithm components from FEDORA
one algorithm component at a time
Figure 7: Ablation Studies.
14

---

We perform an ablation study to examine the different components of our algorithm and understand their relative impacts on the performance of the federated policy. We use the experimental framework with 10 clients and the Hopper environment described in Section 4 and plot the performance of the federated policy with mean and standard deviation over 4 seeds. The ablation is performed in two ways: (a) We build up FEDORA starting with Fed-A, the naïve method which federates only the actor, and add one new algorithm component at a time and evaluate its performance. (b) We exclude one component of FEDORA at a time and evaluate the resulting algorithm.
We observe in Fig. 7a that using rrioity-weighee averaging of the client's policy to compute the federated policy (Eq. ( ), and an optimistic critic (Eq. (9) (IO)) significantly improves the performance of the federated policy. This consistent with our intuition that the most important aspect is extracting the collective wisdom of the policies and critics available for federation, and ensuring that the critic sets optimistic targets. The proximal term helps regularize local policy updates (Eq. 11I by choosing actions close to those seen in the local dataset or by the federated policy. Additionally, decaying the influence of local updates enables the local policy to leverage the federated policy's 's vantage by choosing actions not seen in the local dataset.
From Fig. 7b we observe that removing priority- wrighte actor from FEDORA causes the steepest drop in performance, followed by the optimistic critic. Again, this is consistent with our intuition on these being the most important effects. Excluding the proximal term and local decay also results in a reduction in server performance along with a greater standard deviation.
B.2 Ablation of Decaying mechanism on Walker Environment
We study the effect of decaying the influence of local data (5.4) in the Walker2D environment in Figure [8] Although the decaying mechanism seems to give only a small improvement in Figure 7 which pertains to experiments on the Hopper environment, we observe that it provides a significant improvement in the Walker2D environment.
FEDORA with Decay
FEDORA without Decay
Walker2D
FEDORA with Decay
X FEDORA without Decay
5000
Walker2D
Server Reward
4000
Server Reward
4000
3000
3000
2000
2000
1000
Communication
0
(a) Training curve
(b) Peak performance
Figure 8: Ablation study of decaying mechanism on Walker2d environment (setting similar to Fig 7)-
B.3 Hypterparameter sweep
In Fig. rulrun FEDORA for different values of 3, which is the temperature parameter of federation.
We consider a scenario similar to Fig. [ where we consider the Hopper-v2 environment with 10 clients having DiI 5000 participating in federation, where 5 clients have expert data, and 5 clients have medium data. When 3 = 0, it boils down to a uniform weighting scheme. where the quality of data present in each client is not considered during federation. As 9 + oo it tends to max weighting scheme, where the federated policy is the same as an individual client's policy with the highest quality data.
In Fig. 9b we run FEDORA for different values of the decay parameter d. The decay parameter controls the influence of the local data in Laster by decaying the influence of Lieat In Fig. |b] we consider a scenario similar to Fig. [ where we consider the H2pper-v2 environment with 10 clients having [D = 5000 participating in federation, where 5 clients have expert data, and 5 clients have medium data. We run all the algorithms for 750 rounds of federation. We observe that FEDORA is robust to the variations in the values of 8.
15

---

0.1
0.01
1
0.0
0.05
0.5
5
Varying the temperature parameter
0.995
0.97
0.98
+
0.99
3500
3000
3000
Server Reward
2500
2500
2000
2000
1500
1500
1000
1000
500
500
250
500
750
1000
Communication Round
250
500
750
Communication Round
(a) Varying /
(b) Varying
Figure Hyperparameter sweep
B.4 Analysis of Client Performance
expert
medium
HalfCheetah
Hopper
Walker2D
0.10
0.100
Clent Welght
0.075
0.08
0.075
0.050
0.050
0.04
0.025
0.02
0.025
0.000
0.000
200
600
800
1000
200
400
600
800
1000
0
200
600
800
1080
Communication Round
Communication Round
Communication Round
(a) Client ratio
expert
- medium
HalfCheetah
Hopper
Walker2D
1.0
1.0
1.0
Clent Docay
0.8
0.8
0.8
0.6
0.4
0.
Communication Round
Communication Round
Communication Round
(b) Client decay
Figure 10: Analysis of client performance during federation. The average the performance metric computed across expert and medium clients participating in given round of federation.
We train FEDORA on MuloCo environments using a setup similar to Section where 20 out of the 50 clients are randomly chosen to participate in each round of federation. Our goal is to analyze the contribution of clients with expert data and those with medium data to the learning process. As before, the clients and the algorithm
unaware of the data quality.
We plot the mean weights wi across the expert and medium dataset clients participating in a given round of federation in Fig. [Oal We observe that the weights of medium clients drop to 0, while the weights of expert clients rise to 0.1. This finding emphasizes the fact that clients are combined based on their relative merits.
In Fig. 10b we plot the mean of the decay value associated with Lieca across participating expert and medium dataset clients (Section55.4) The decay of both sets of clients drops training progresses. A reduction in decay occurs each time the local estimate of the federated policy's performance J
16

---

is greater than the estimated performance of the updated local policy J. A decreasing decay implies that the federated policy offers a performance improvement over local policies more often as the rounds advance. Thus, training only on local data is detrimental, and participation in federation can help learn a superior policy.
B.5 Federated Offline RL experiments with CityLearn
Real-world environments often have large state space and are stochastic in nature. We run federated experi- ments on CityLearn [29 assess the effectiveness of FEDORA on such large-scale systems. CityLLear is an OpenAl Gym environment with the goal of urban-scale energy management and demand response, modeled on data from residential buildings. The goal is to reshape the aggregate energy demand curve by regulating chilled water tanks and domestic hot water, two modes of thermal energy storage in each building. The energy demand of residential buildings changes as communities evolve and the weather varies. Hence, the controller must update its policy periodically to perform efficient energy manage- ment. Federated learning would allow utilities that communities in close proximity to train a policy collaboratively while preserving user data privacy, motivating the use of FEDORÁ for this environment.
FEDORA
Fad
Centralized
Fed AC Prox
Fed
50000
40300
Server
30300
20000
30
100
125
130
Communication Round
Figure 11: Evaluation of algorithms on CityLeern.
In our experiments, we have 10 clients with 5000 training examples such that they all participate in 150 rounds of federation. The training data for the clients is obtained from NeoRL, offline RL benchmark [24] 5 clients each have data from the CityLearn High and CityLearn Low datasets, which collected by SAC policy trained to 75% and 25% of the best performance level, respectively. During each round of federation, each client performs 20 local epochs of training. The server reward at the end of each federation round is evaluated online and shown in Fig. II We observe that FEDORA A outperforms other federated offline RI. algorithms as well as centralized training, which learns using TD3-BC on the data aggregated from every client. These findings indicate that FEDORA can perform well large-scale stochastic environments.
B.6 Effect of multiple behavior policies and proportion of clients participating in federation
In this section, we study the effects of clients having data from multiple behavior policies for varying proportions of clients participating in federation. We consider a scenario with 50 clients having D; 5000 in the Hopper-v2 environment where,
Varying participsting clients
3500
12 clients have expert data (samples from policy trained to completion with SAC.).
2500
2000
1500
12 clients have medium data (samples from a policy trained to approximately 1/3 the performance of the expert).
750
1002
Communication Round
Effect of varying the number of participating clients in each round on FEDORA
Figure 12:
14 clients have random data samples from a randomly initialized policy).
12 clients have data from the replay buffer of a policy trained up to the performance of the medium agent.
We run FEDORA by varying the the percentage of clients participating in each round of federation. We observe that the FEDORA is fairly robust to the fraction of clients participating in federation even when the fraction as low as 20%.
B.7 Variable Size Datasets
17

---

In Fig. 13 we run FEDORA with clients having variable dataset sizes and compare it with FEDORA with a fixed dataset size.
FEDORA
FEDORA Vor Dutaset
2320
FEDORA Var Dataset: We consider a scenario where we have 10 clients in the Hopper-v2 environment. clients have the expert dataset with dataset sizes 4000, 5000, 6000, 7000, 8000 and 5 clients have the medium dataset with dataset sizes 4000, 5000, 6000, 7000, 8000.
0530
2000
1530
1000
FEDORA: We consider the scenario described in Fig 1 with constant dataset size of 5000.
200
300
150
Communication Hound
Figure 13: Variable Dataset Size
From Fig. 13 we can observe that FEDORA is robust can handle variable dataset sizes.
B.8 Centralized training with other Offline RL algorithms
We consider scenario similar to the one in Fig. for the Hopper- v2 environment with 50 clients, having |P;] = 5000 participating in federation, where 25 clients have expert data, and 25 clients have random data. We compare the performance of different Offline RI. algorithms over the pooled data with FEDORA. The algorithms we choose are Conservative Q-Learning for Offline Reinforcement Learning (CQL) [15] and Offline Reinforcement Learning with Implicit Q-Learning (IQL) [3]
Agaltni
Figure 14: Comparison
whose implementations are obtained from the CORL library [27]. We can observe from Fig. [4 that pooling data from different behavior policies
with different Offline RL algorithms
affects both offline RL algorithms.
B.9 Data Rebalancing
T00-2C.10
FEDORA
We consider a scenario similar to Fig. I where we consider the Hopper- v2 environment with 10 clients having |Dr] 5000 participating in federation, where 5 clients have expert data, and 5 clients have medium data. We compare the performance of TD3-BC, TD3-BC with data rehalancing (TD3-BC_RB) 38, 37], and FEDORA. From Fig. 15 we can notice that the addition of data rebalancing does help the performance of offline RI. algorithms when data is collected using multiple behavior policies. We also notice that the performance of TD3-BC with data rebalancing does not match the performance of FEDORA. We hypothesise that this could be due to the superior weighting mechanism employed by FEDORA. A. and that data rebalancing cannot completely solve the distribution shift issue caused by
1435
Figure 15: Comparison
with Offline RL with data rebalancing
data coming from multiple behavior policies.
B.10 Different Weighing Mechanisms
We consider scenario similar to Fig. [ where we con- sider the Hopper-v2 v2 environment with IO clients having [D,| 5000 participating in federation, where 5 clients have expert data, and 5 clients have medium data. We all the algorithms for 750 rounds of federation. We compare the performance of FEDORA with two addi- tional baselines in which change only the weighing mechanism of FEDORA to be based on average reward of the dataset at each client. (1.) FEDORA RAvg: We
FEDORA FEDORA
FEDORA,
35.00
3000
Server Roward
2500
1500
combine clients based on the average reward in their dataset. We choose the weights of federation of client wt ELeN, Rs where R2 corresponds to the
500
R.
250
500
750
Communication Round
erage reward of client i's dataset and N₁ is the set of clients participating in federation at round (2.) FE-
Figure 16: Comparison with different weighing mechanism based on average reward in the dataset
DORA RC: We extend the weighing scheme proposed
18

---

in 33 to the federated setting, In this scenario, we choose the weights of federation for client i, here R& corresponds to the average reward of client i's W) Sicx, Ps Pa Where P; Hmase-Ren Memnn
Pi
Ri Ruin
dataset, Rmin = min R4, and Rmax = maxeNN, R4
From Fig. 16 we notice that FEDORA outperforms both baselines. We observe that weighing based on average reward of the datasets does not yield good results, this can attributed to the fact that the average rewards in the dataset do not vary much. For instance, the average reward of clients with the expert dataset is 3.6, while that of the medium dataset is 3.11. Thus combining based solely on the average reward boils down to using uniform weighing scheme. We observe that extending the approach proposed in 38 does help, but this weighing scheme still inferior to that of FEDORA.
FEDROA's weighing scheme is superior as it combines the policies based on the performance of the learned policy from the dataset, rather than the average reward in the dataset. In other words, FEDORA weighs dataset from which better performing policy can be learnt higher.
C Details of World Robot Experiments
C.1 Demonstration Data Collection
We train four behavior policies of varying levels of expertise using TRPO [26] on a custom simulator for mobile robots described in section C2
The first policy is capable of waypoint navigation but collides with obstacles. The second policy can reach waypoints while avoiding obstacles present one fixed position. The third policy has not fully generalized to avoiding obstacles at various positions. Finally, the fourth policy can navigate to the goal without any collision. We execute the behavior policies in the real-world by varying the waypoint (target location) and location of the obstacle gather demonstration data, which we then use to train FEDORA and other baselines. Each client has a dataset consisting 300 data points collected using single behavior policy. After training. we test the learned policies in the real-world TurtleBot to ascertain its feasibility.
C.2 Simulator Design
We develop a first-order simulator for mobile robots using the OpenAl Gym framework, which enables the training of RL algorithms. The robot's pose is represented by its X- and Y-coordinates in a 2D space and its orientation with respect to the X- X-axis, B. The pose is updated using differential drive kinematics
Ft+1 = + At v cos O2
Mt+1 3/1 - At usin O,
(12)
4.41 = O2 + At w,
where (Iz, Vr, 02) is the pose at time f, v and w are the linear and angular velocity of the robot respectively, and At is time discretization of the system.
The simulator uses a functional LIDAR to detect the presence of obstacles. We simulate the LIDAR using discrete representation of the robot and obstacles in its immediate environment. For each scanning direction around the LIDAR, we use Bresenham's line algorithm to generate a path com- prising of discrete points. The simulator determines LIDAR measurements by counting the number of points along each path, starting from the robot and continuing until it encounters and obstacle or reaches the maximum range.
The reward function is designed to encourage effective waypoint navigation while preventing colli- sions. We define boundary grid that extends for im beyond the start and the goal positions in all directions. The reward function at time t for navigating to the goal position (Igg Mg) is chosen to be
+00,,
if [x+ x,و S thresh and 1y. Ug S thresh if robot outside boundary robot collides otherwise
10,
R₂
100,
(c.t.ef + a.t.e; + h.e)) + Σ lidart,
(13)
where c.t.e, is the cross-track error, a.t.e; the along-track error, h.e, is the heading error, lidar, is the array of LIDAR measurements at time and thresh is the threshold error in distance, chosen as
19

---

0.1m. Let the L-2 distance to the goal and the heading to the goal at time t be dī and 0g respectively, Then, have de
(Tg Ft 1
(yg
yb)
09 tan
Ds 1/1
c.t.e,
dß sin
(0g
2x -2
0e);
(14)
a.t.e; Ir'
+ 18g Vol.
h.er 0f A:.
C.3 Mobile Robot Platform
We evaluate the trained algorithms on a Robotis TurtleBot3 Burger mobile robot M. open-source differential drive robot. The robot has a wheel encoder- pose estimation system and is equipped with RPLIDAR-AI LIDAR for obstacle detection. We use ROS as the middleware to set up communication. The robot transmits its state (pose and LIDAR information) over wireless network to computer, which then transmits back the corresponding action suggested by the policy being executed.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings. It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be
20

---

used reliably to provide closed captions for online lectures because it fails to handle technical Jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
See Appendix
and code httpe://ggthhbe.cco//bDelsssgggg
FEDORA
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
21

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
See code httd://////ggttuooooo////essssss
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis..co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//iips.cc/publl wttpcccpppppccceeeeee..ee..ccltt
wtt.c//pppp.ggudeeeeeeeeeeeeeeee guwdes/ http///oodubiisiinii.. sosoooooicssi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See section
6
Appendix
A
and code in https://gi com/
DesikRengaraje an/ FEDORA
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
22

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
23

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Code at http:///tteegggggc.cco////////aa and Appendix A

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See code at otttp:///ggtthhhhcmm///eeeeeeegp
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
25

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "TYNoqwNsax", "reviewer": "Reviewer_hv6n", "summary": "This paper presents the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA), a novel approach for collaborative learning of high-quality control policies in a federated offline reinforcement learning (RL) setting. The paper identifies key challenges in federated offline RL, including ensemble heterogeneity, pessimistic value computation, and data heterogeneity. To address these issues, FEDORA estimates the performance of client policies using only local data and, at each round of federation, produces a weighted combination of the constituent policies that maximize the overall offline RL objective, while maximizing the entropy of the weights. Besides the core idea, FEDORA also performs data pruning", "questions": "My questions are stated above.", "limitations": "Yes, limitations are adequately discussed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This is a novel work proposing the first federated offline RL algorithm in the general case (without assuming linearity). The paper is very well written with clear motivations and detailed discussions on the insufficiency of existing, naive approaches. \n\n2. The experiments are also very thorough and convincing with experiments ranging from simple 2D environments to high-dimensional continuous control problems. The algorithm is also tested on a real-world robot platform, which is very impressive given the density of algorithmic contributions in the paper.", "weaknesses": "1. \"Collect wisdom\" can be replaced by more rigorous exposition. Same goes with \"ambitious targets\". \n\n2. The number of communication rounds needed for FEDORA to converge is still quite high. \n\n3. Given how well the algorithm does, some sort of theoretical analysis could further strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive endorsement of our work. We are happy to know that the reviewer finds our work novel, our experiments extensive and our paper well written. Below we address the concerns of the reviewer. \n\n*1. \"Collect wisdom\" can be replaced by more rigorous exposition. Same goes with \"ambitious targets\".*\n\n**Response:** We thank the reviewer for their suggestion, we will incorporate this in the final version of the paper.\n\n*2. The number of communication rounds needed for FEDORA to converge is still quite high.*\n\n**Response:** The number of communication rounds that FEDORA takes depends on factors such as the complexity of the problem and the number of local epochs performed during each round of federation. We believe that the number of communication rounds can be reduced by increasing the number of local epochs performed in each round of federation.  \n\n*3. Given how well the algorithm does, some sort of theoretical analysis could further strengthen the work.*\n\n**Response:**  Thank you, we are indeed working on this problem. Providing a theoretical guarantee for our FEDORA algorithm is a challenging problem that requires technical analysis of multiple complicated parts corresponding to offline policy evaluation, pessimistic estimation, ensemble-style quality-based federation, dealing with heterogeneous data, and analyzing these components jointly to get the final performance bound.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UBTqYwAk3j", "reviewer": "Reviewer_EkXX", "summary": "The authors identify fundamental challenges for Federated Offline Reinforcement Learning and present Fedora, an approach that tackles each of them. They perform extensive evaluation of the approach on Mujoco and real-world datasets showing improved performance over existing work.", "questions": "* How far do the authors perceive that this model can be pushed? i.e. the assumption that all clients have the same MDP is restrictive but understandable for a first set of experiments.\n\n* Have any experiments been run using D4RL-random datasets? It would be interesting to see whether this collapses learning.\nWith regards to FEDORA outperforming centralised training I think a deeper discussion on this would be useful. \n\n* What is the main reason for this? Heterogenous data though previous work has successfully mixed datasets: https://arxiv.org/abs/2106.06860", "limitations": "Limitations should be explicitly stated. I feel that the authors could give a more balanced view of the algorithm by not only showing strengths but also assessing the limits of the work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written and, importantly, the code has been shared. The authors run extensive experiments. The work is novel and the notion of federated optimism is particularly interesting. Federated offline RL is an important research area with vast real-world applicability. The algorithm has been shown to be robust to diverse/ heterogenous client datasets. It is also commendable that the approach was tested on a real-world robot.", "weaknesses": "No theoretical guarantees have been given for the algorithm though it does build upon foundational work.  I believe that the authors should explicitly discuss limitations/ opportunities for future work in the paper. It is important for the algorithm pseudocode to be included in the main material as is the norm in such papers. I believe that there are perhaps many experiments included the main paper meaning that the discussion/ hypotheses for results is somewhat diluted. \nAnother minor issue is that the figures are placed very far away from where they are referred to in text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and are happy to note that they find our work novel, our experiments extensive, and our paper well-written. Below, we address their concerns and hope that they consider increasing their score.\n\n*1. No theoretical guarantees have been given for the algorithm though it does build upon foundational work.*\n\n**Response:** Thank you for your comment. Providing a theoretical guarantee for our FEDORA algorithm is indeed a challenging problem that requires technical analysis of multiple complicated components, including offline policy evaluation, pessimistic estimation, ensemble-style quality-based federation, dealing with heterogeneous data, and analyzing these components jointly to derive the final performance bound.\n\nWe, however, emphasize that, to the best of our knowledge, ours is the first paper on federated offline deep RL with an algorithm that performs really well in a variety of settings. Our design is analytically driven, identifies each issue of learning from the ensemble of policies, and builds up the algorithm methodically one step at a time corresponding to each analytical insight. We sincerely believe that our rigorous empiricism-driven approach is valuable based on its algorithmic contributions. \n\n*2. I believe that the authors should explicitly discuss limitations/ opportunities for future work in the paper.*\n\n**Response:** We discuss the limitations and future directions of our research in Appendix E. For the final submission, we will move this discussion to the main part of the paper to make it more accessible and prominent.\n\n\n*3. It is important for the algorithm pseudocode to be included in the main material as is the norm in such papers...Another minor issue is that the figures are placed very far away from where they are referred to in text.*\n\n**Response:** We pushed the pseudocode to the appendix due to space constraints. However, for the final version, we will move the pseudocode back to the main text, as we will have an extra page for the final submission. Additionally, we will adjust the placement of figures to be closer to the relevant text.\n\n\n*4. How far do the authors perceive that this model can be pushed? i.e. the assumption that all clients have the same MDP is restrictive but understandable for a first set of experiments*\n\n**Response:** We plan to extend FEDORA to a meta federated learning setting, wherein we can learn with clients having different transitions and reward functions. This extension is discussed in Appendix E, and we aim to explore this direction in future work. \n\n*5. Have any experiments been run using D4RL-random datasets? It would be interesting to see whether this collapses learning. With regards to FEDORA outperforming centralized training I think a deeper discussion on this would be useful.*\n\n**Response:** Yes, we have run experiments using the D4RL random dataset and compared it with centralized training (See Figure 3 in Section 6.1). We also conduct experiments with clients having different datasets (including random datasets) in Appendix C.6. \n\n*6. What is the main reason for this? Heterogeneous data though previous work has successfully mixed datasets*\n\n**Response:** The use of heterogeneous datasets in centralized offline RL is a significant challenge. One reason for the drop in performance when pooling data from behavior policies with different expertise levels is that it can exacerbate the distributional shift between the learned policy and the individual datasets, leading to poor performance [1].\n\n\n**References**\n\n[1] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn.\nConservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:11501–11516, 2021.484", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j092eJ518S", "reviewer": "Reviewer_EGVU", "summary": "This paper proposed the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm. The combination of offline RL and federated learning is interesting in addressing the training data insufficiency issue due to small pre-collected datasets.", "questions": "1. Some technical details need to be explained. For example, the ensemble learning and its role.\n2. The novelty of this paper needs further clarification, and what is the main difference between this proposed method and existing studies? It seems that there is only a simple combination of two technologies.\n3. Numerically, the authors could consider comparing their method with more baselines. There are some studies on federated learning for offline RL.", "limitations": "1. What is the technical drawback of the proposed method? E.g., the effectiveness of the agent weight by ensemble approach\n2. Does this proposed method work for other RL algorithms?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The originality of this paper is relatively good, since the proposed Federated Ensemble-Directed Offline Reinforcement Learning Algorithm is effective in offline reinforcement learning. The quality and clarity are also clear, and this paper is actually well-written. The significance of this paper is obvious, because offline reinforcement learning is important in real-world scenarios.", "weaknesses": "1. Some technical details need to be explained. For example, the ensemble learning and its role.\n2. The novelty of this paper needs further clarification, and what is the main difference between this proposed method and existing studies? It seems that there is only a simple combination of two technologies.\n3. Numerically, the authors could consider comparing their method with more baselines. There are some studies on federated learning for offline RL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are delighted to know that the reviewer finds our work original, our problem significant, and our paper well written. Below, we address the reviewer's concerns and hope they will consider increasing their score. \n\n*1. Some technical details need to be explained. For example, the ensemble learning and its role.*\n\n**Response:** The idea of ensemble learning in our algorithm is to use the data distributed among different clients to learn a federated policy collectively. In Section 4.1, we have mentioned that `ensemble heterogeneity' is one of the key challenges in federated offline RL. In our work, we proposed an ensemble approach to overcome this challenge. Specifically, our approach uses the performance of the local policies as a proxy to weigh each client's contribution to the federated policy. This ensures that policies from clients with higher performance data have a greater influence on the federated policy. We have explained our approach in Section 5.1, including the mathematical equation that translates our idea to an actual algorithmic step. In addition to the experimental results given in Section 6 that show the superior performance of our FEDORA algorithm, we have also included additional ablation experimental results in Appendix C.1 which show the significance of our ensemble method w.r.t. other ingredients of FEDORA. Please let us know if there are any specific aspects that need to be elaborated further. \n\n*2. The novelty of this paper needs further clarification, and what is the main difference between this proposed method and existing studies? It seems that there is only a simple combination of two technologies.*\n\n**Response:** We respectfully disagree with the reviewer's comment that our proposed algorithm is ``is only a simple combination of two technologies''. We emphasize that a simple combination of federated learning and offline RL is insufficient, as we have explained in our paper (see Fig. 1 and the explanation there). Significant algorithmic innovations are necessary to overcome some unique challenges of federated offline RL, as we have explained in Section 4.1. Our contributions include four key innovations: $(i)$ Ensemble-Directed Learning over Client Policies (Section 5.1), $(ii)$ Federated Optimism for Critic Training (Section 5.2), $(iii)$ Proximal Policy Update for Heterogeneous Data (Section 5.3), and  $(iv)$ Decaying the Influence of Local Data (Section 5.4).  We have given detailed experimental evidence on the superior performance of this method, see Section 6 and Appendix. Moreover, we have done ablation experiments that show the importance of each of these proposed innovations. \n\n\n*3. Numerically, the authors could consider comparing their method with more baselines. There are some studies on federated learning for offline RL*\n\n**Response:** We have demonstrated the superior performance of our FEDORA algorithm against four different baseline algorithms through simulation experiments, see Section 6.1 and Appendix C. We have also evaluated the performance of FEDORA in the real-world using TurtleBot, a two-wheeled differential drive mobile robot, see Section 6.1, and compared it with the same baseline algorithms. We have also included a video of this real-world demonstration.  We sincerely believe that these experiments and real-world demo clearly show the superior performance of FEDORA against the standard baselines. We would appreciate further guidance on specific baselines the reviewer would like us to include in our comparisons. \n\n*4. What is the technical drawback of the proposed method? E.g., the effectiveness of the agent weight by ensemble approach*\n\n**Response:** We address our limitations in Appendix E. We make the assumption that all clients have the same MDP model (transition kernel and reward model), and any statistical variances between the offline datasets are due to differences in the behavior policies used to collect the data. In future works, we aim to broaden this to cover scenarios where clients have different transition and reward models.\n\nReading the second remark about the effectiveness, please note that we have already included detailed ablation experiments to analyze the effectiveness of different components of our algorithm, see Appendix C1 and C2. \n\n*5. Does this proposed method work for other RL algorithms?*\n\n**Response** Indeed! The FEDORA framework that we propose is general and can work with any actor-critic-based offline RL algorithms. We have mentioned this in our paper, please see lines 126-128.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### Joint Response

We would like to express our gratitude to all the reviewers for their time and feedback. We are delighted that the reviewers recognize the novelty of our work (hv6n, EkXX, EGVU), find our paper well-written (hv6n, EkXX, EGVU), and appreciate the comprehensiveness of our experiments (hv6n, EkXX). Below, we provide detailed responses to their queries. We look forward to a productive discussion during the reviewer-author period.

Meta-review: A summary of the strengths and weaknesses based on the reviews and rebuttal (including follow-up discussion and that among the reviewers) is provided below:


**STRENGTHS**

1. This paper proposes a novel federated offline RL algorithm in the general case (without assuming linearity).

2. The paper is well-written.

3. The experiments are extensive and compelling and include that of a real robot platform.


**WEAKNESS**

The paper lacks a theoretical performance analysis of the algorithm, for which the authors acknowledge would be challenging due to multiple components being involved. **However, the reviewers think that the technical contributions of this work (listed under the strengths) outweigh this weakness.**

As a minor comment, I like to suggest that the authors include other references on federated reinforcement learning, especially those published in the 3 flagship ML conferences.

The authors are encouraged to revise their main paper based on the above feedback and that of the reviewers, as well as based on their rebuttal (e.g., limitations).
Decision: Accept (poster)