Federated Ensemble-Directed Offline Reinforcement Learning

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:50:01

Abstract: We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected datasets generated according to different unknown behavior policies. Na\"{i}vely combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies.  In response, we develop the Federated Ensemble-Directed Offline Reinforcement Learning  Algorithm (FEDORA), which distills the collective wisdom of the clients using an ensemble learning approach.  We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on a mobile robot. We provide our code and a video of our experiments at \url{https://github.com/DesikRengarajan/FEDORA}.

Main paper: 
 introduction: 
 Federated learning is an approach wherein clients learn collaboratively by sharing their locally trained models (not their data) with a federating agent, which periodically combines their models and returns the federated model to the clients for further refinement (Kairouz et al., 2021; Wang et al., 2021) . Federated learning has seen recent success in supervised learning applications due to its ability to generate well-trained models using small amounts of data at each client, while preserving privacy and reducing the usage of communication resources. There has also been interest in federated learning for online reinforcement learning (RL), wherein clients learn via sequential interactions with their environments and federating learned policies across clients (Khodadadian et al., 2022; Nadiger et al., 2019; Qi et al., 2021) . However, such online interactions with real-world systems are often infeasible, and each client might only posses pre-collected operational data generated according to a client-specific behavior policy. The fundamental problem of federated offline RL is on how to learn the optimal policy only using such offline data collected by heterogeneous policies at clients, without actually sharing any of the data.Offline RL algorithms (Levine et al., 2020) , such as CQL Kumar et al. (2020b) and TD3-BC (Fujimoto & Gu, 2021) offer an actor-critic learning approach that only utilizes existing datasets at each client. However, in our case, this approach taken across many small datasets at clients will produce an ensemble of policies of heterogeneous (unknown) qualities across the clients, along with their corresponding critics of variable accuracy. We will see that naïvely federating such offline RL trained policies and critics using a standard federation approach, such as FedAvg (McMahan et al., 2017) can lead to a policy that is even worse than the constituent policies. We hence identify the following basic challenges of federated offline RL: (i) Ensemble heterogeneity: Heterogeneous client datasets will generate policies of different performance levels. It is vital to capture the collective wisdom of this ensemble of policies, not average them. (ii) Pessimistic value computation: Offline RL employs a pessimistic approach toward computing the value of actions poorly represented in the data to minimize distribution shift (and so reduce the probability of taking these actions). However, federation must be ambitious in extracting the highest values as represented in the ensemble of critics (and so promote high-value actions). (iii) Data heterogeneity: As with other federated learning, multiple local gradient steps based on heterogeneous data at each client between federation rounds may lead to biased models. We must regularize local policies to reduce such drift.In this work, we propose Federated Ensemble-Directed Offline RL Algorithm (FEDORA), which collaboratively produces a high-quality control policy and critic function. FEDORA estimates the performance of client policies using only local data (of unknown quality) and, at each round of federation, produces a weighted combination of the constituent policies that maximizes the overall objective, while regularizing by the entropy of the weights. The same approach is followed to federate client critics. Following the principle of maximum entropy in this manner produces both federated policies and critics that extract the collective wisdom of the ensemble. In doing so, it constructs a federated policy and a critic based on the relative merits of each client policy in an ensemble learning manner. FEDORA ensures optimism across evaluation by the federated and local critic at each client and so sets ambitious targets to train against. It addresses data heterogeneity by regularizing client policies with respect to both the federated policy and the local dataset. Finally, FEDORA prunes the influence of irrelevant data by decaying the reliance on a dataset based on the quality of the policy it can generate. To the best of our knowledge, no other work systematically identifies these fundamental challenges of offline federated RL, or designs methods to explicitly tackle each of them.We develop a framework for implementing FEDORA either on a single system or over distributed compute resources. We evaluate FEDORA on a variety of MuJoCo environments and real-world datasets and show that it outperforms several other approaches, including performing offline RL on a pooled dataset. We also demonstrate FEDORA's excellent performance via real-world experiments on a TurtleBot robot (Amsters & Slaets, 2020) . We provide our codebase, several experimental results and a video of the robot experiments in the supplementary material. 
related work: 
 Offline RL:The goal of offline RL is to learn a policy from a fixed dataset generated by a behavior policy (Levine et al., 2020) . One of the key challenges of the offline RL approach is the distribution shift problem where the state-action visitation distribution of learned policy may be different from that of the behavior policy which generated the offline data. It is known that this distribution shift may lead to poor performance of the learned policy (Levine et al., 2020) . A common method used by offline RL algorithms to tackle this problem is to learn a policy that is close to the behavior policy that generated the data via regularization either on the actor or critic (Fujimoto & Gu, 2021; Fujimoto et al., 2019; Kumar et al., 2020a; 2019; Wu et al., 2019) . Some offline RL algorithms perform weighted versions of behavior cloning or imitation learning on either the whole or subset of the dataset (Wang et al., 2018; Peng et al., 2019; Chen et al., 2020) . Yue et al. (2022; 2023) propose data rebalancing methods designed to prioritize highly-rewarding transitions that can be augmented to offline RL algorithms to alleviate the distribution shift issue for heterogeneous data settings. A key problem in federated learning is data heterogeneity wherein clients have non-identically distributed data, which causes unstable and slow convergence (Wang et al., 2021; Karimireddy et al., 2020; Li et al., 2020) . To tackle the issue of data heterogeneity, Li et al. (2020) proposed FedProx, a variant of FedAvg, where a proximal term is introduced reduce deviation by the local model from the server model. Federated Reinforcement Learning: Federated learning has recently been extended to the online RL setting. Khodadadian et al. (2022) analyzed the performance of federated tabular Q-learning. Qi et al. (2021) combined traditional online RL algorithms with FedAvg for multiple applications. Some works propose methods to vary the weighting scheme of FedAvg according to performance metrics such as the length of a rally in the game of Pong (Nadiger et al., 2019) or average return in the past 10 training episodes (Lim et al., 2021) to achieve better performance or personalization. Wang et al. (2020) proposed a method to compute weights using attention over performance metrics of clients such as average reward, average loss, and hit rate for an edge caching application. Hebert et al. (2022) used a transformer encoder to learn contextual relationships between agents in the online RL setting. Hu et al. (2021) proposed an alternative approach to federation where reward shaping is used to share information among clients. Xie & Song (2023) proposed a KL divergence-based regularization between the local and global policy to address the issue of data heterogeneity in an online RL setting.In the offline RL setting, Zhou et al. (2022) propose federated dynamic treatment regime algorithm by formulating offline federated learning using a multi-site MDP model constructed using linear MDPs. However, this approach relies on running the local training to completion followed by just one step of federated averaging. Unlike this work, our method does not assume linear MDPs, which is a limiting assumption in many real-world problems. Moreover, we use the standard federated learning philosophy of periodic federation followed by multiple local updates. To the best of our knowledge, ours is the first work to propose a general federated offline RL algorithm for clients with heterogeneous data. 
preliminaries: 
 Federated Learning: The goal of federated learning is to minimize the following objective,F (θ) = E i∼P [F i (θ)] ,(1) where θ represents the parameter of the federated (server) model, F i denotes the local objective function of client i, and P is the distribution over the set of clients N . The FedAvg algorithm (McMahan et al., 2017) is a popular method to solve Eq. (1) in a federated way. FedAvg divides the training process into rounds, where at the beginning of each round t, the server broadcasts its current model θ t to all the clients, and each client initializes its current local model to the current server model. Clients perform multiple local updates on their own dataset D i to obtain an updated local model θ t i . The server then averages these local models proportional to the size of their local dataset to obtain the server model θ t+1 for the next round of federation, asEQUATION)Reinforcement Learning: We model RL using the Markov Decision Process (MDP) framework denoted as a tuple (S, A, R, P, γ, µ), where S is the state space, A is the action space, R : S ×A → R is the reward function, and P : S×A×S → [0, 1] denotes the transition probability function that gives the probability of transitioning to a state s ′ by taking action a in state s, γ is the discount factor, and µ is the distribution of the initial state s 0 . A policy π is a function that maps states to actions (deterministic policy) or states to a distribution over actions (stochastic policy). The goal of RL is to maximize the infinite horizon discounted reward of policy π, defined as J(π) = E π,P,µ [ ∞ t=0 γ t R(s t , a t )], which is the expected cumulative discounted reward obtained by executing policy π. The stateaction value function (or Q function) of a policy π at state s and executing action a is the expected cumulative discounted reward obtained by taking action a in state s and following policy π thereafter:Q π (s, a) = E π,P [ ∞ t=0 γ t R(s t , a t )|s 0 = s, a 0 = a].Offline Reinforcement Learning: The goal of offline RL is to learn a policy π only using a static dataset D of transitions (s, a, r, s ′ ) collected using a behavior policy π b without any additional interactions with the environment. Offline RL algorithms typically utilize some kind of regularization with respect to the behavior policy to ensure that the learned policy does not deviate from the behavior policy. This regularization is done to prevent distribution shift, a significant problem in offline RL, where the difference between the learned policy and behavior policy can lead to erroneous Q-value estimation of state-action pairs not seen in the dataset (Kumar et al., 2020a; Levine et al., 2020) .Our approach is compatible with most offline RL algorithms, such as CQL Kumar et al. (2020b) or TD3- BC Fujimoto & Gu (2021) . We choose TD3-BC for illustration, motivated by its simplicity and its superior empirical performance in benchmark problems. The TD3-BC algorithm is a behavior cloning (BC) regularized version of the TD3 algorithm (Fujimoto et al., 2018) . The policy in TD3-BC is updated using a linear combination of TD3 objective and behavior cloning loss, where the TD3 objective ensures policy improvement and the BC loss prevents distribution shift. More precisely, the TD3-BC objective can be written asEQUATION)and λ is a hyperparameter that determines the relative weight of the BC term. 
federated offline reinforcement learning: 
 In real-world offline RL applications, data is typically obtained from the operational policies of multiple agents (clients) with different (unknown) levels of expertise. Clients often prefer not to share data. We aim to learn the optimal policy for the underlying RL problem using only such offline data, without the clients knowing the quality of their data, or sharing it with one another or the server. Furthermore, neither the clients nor server have access to the underlying model or the environment. We denote the set of clients as N . Each client i ∈ N has the offline dataset D i = {(s j , a j , r j , s ′ j ) mi j=1 } generated according to a behavior policy π b i . We assume that the underlying MDP model P and reward function R(•, •) are identical for all the clients, and the statistical differences between the offline datasets D i are only due to the difference in behavior policies π b i used for collecting the data. In a standard federated learning algorithm such as FedAvg, each client performs multiple parameter updates before sending its parameters to the server. It is known that performing multiple local updates in federated learning can reduce the communication cost significantly without compromising on the optimality of the converged solution (Kairouz et al., 2021; Wang et al., 2021) . In federated offline RL, since each client has to perform multiple steps of policy evaluation and policy update using its local offline data D i , it is reasonable to consider a client objective function that is consistent with a standard offline RL algorithm objective. We choose the objective function used in the TD3-BC algorithm (Fujimoto & Gu, 2021) , i.e., U Di given in Eq. ( 3), as the client objective function. Our choice is motivated by the simplicity of the TD3-BC objective function and its empirical success in a variety of environments. Similar to the standard federated learning objective given in Eq. ( 1), we can now define the federated offline RL objective asEQUATIONwhere w i are weights to be determined.One approach to leveraging experiences across users without sharing data would be to combine existing federated learning techniques with offline RL algorithms. Is such a naïve federation strategy sufficient to learn an excellent federated policy collaboratively? Furthermore, is federation even necessary? In this section, we aim to understand the challenges of federated offline RL with the goal of designing an algorithmic framework to address these challenges.We start by illustrating the issues in designing a federated offline RL algorithm. We consider the Hopper environment from MuJoCo (Todorov et al., 2012) , with |N | = 10, |D i | = 5000, and we use the data from the D4RL dataset (Fu et al., 2020) . However, instead of using the data generated by the same policy for all clients, we consider the setting where five clients use the data from the hopper-expert-v2 dataset (which was generated using a completely trained (expert) SAC policy) and five clients use the data from the hopper-medium-v2 dataset (which was generated using a partially trained (medium) policy achieving only a third of the expert performance). The clients and the server are unaware of the quality (expert or medium) of the data. Fig. 1 shows the performance comparison of multiple algorithms, where the mean and the standard deviation are calculated over 4 seeds. Combining All Data (Centralized): Combining data and learning centrally is the ideal scenario in supervised learning. However, as seen in Fig. 1 , performing centralized training over combined data generated using different behavior policies in offline RL can be detrimental. This is consistent with Yu et al. (2021) that proves that pooling data from behavior policies with different expertise levels can exacerbate the distributional shift between the learned policy and the individual datasets, leading to poor performance. Similar deterioration due to combining data has also been observed in other offline RL literature (Fujimoto & Gu, 2021; Kumar et al., 2020a) .Individual Offline RL: Here, agents apply offline RL to their own datasets without collaborating with others. In Fig. 1 , we observe that clients with either expert or medium data do not learn well and exhibit a large standard deviation. This observation may be attributed to no client having sufficient data to learn a good policy.Naïve Federated Offline RL: A simple federation approach is to use the offline RL objective as the local objective and apply FedAvg (Eq. ( 2)). However, offline RL algorithms typically comprise two components -an actor and a critic. It is unclear a priori which components should be federated, so we conduct experiments where we federate only the actor (Fed-A) or both the actor and the critic (Fed-AC). Surprisingly, these naïve strategies result in federated policies that perform worse than individual offline RL, as witnessed in Fig. 1 .Our example illustrates several fundamental issues that must be addressed while designing viable federated offline RL algorithms, including:1. Ensemble Heterogeneity: Performing offline RL over heterogeneous data yields a set of policies of different qualities. It is crucial to leverage the information contained in these varied policies rather than simply averaging them. However, federation after a single-step local gradient at each client using weights in the manner of FedAvg,w i = |D i |/| |N | i=1 |D i |, is equivalent to solving the offline RL problem using the combined dataset of all clients (Wang et al., 2021) . This approach leads to poor performance due to the resulting distribution shift, as shown in Fig. 1 . How should we optimally federate the ensemble of policies learned by the clients? 2. Pessimistic Value Computation: Most offline RL algorithms involve a pessimistic term with respect to the offline data for minimizing the distribution shift. Training a client critic using only the local data with this pessimistic term could make it pessimistic towards actions poorly represented in its dataset but well represented in other clients' data. How do we effectively utilize the federated critic along with the locally computed critic to set ambitious targets for offline RL at each client?3. Data Heterogeneity: Federated learning calls for performing multiple local gradient steps at each client before federation to enhance communication efficiency. However, numerous epochs would bias a client's local model to its dataset. This client drift effect is well known in federated (supervised) learning and could lead to policies that are not globally optimal. In turn, this could cause the federated policy's performance to be worse than training locally using only the client's data, as seen in Fig. 1 . How should we regularize local policies to prevent this? 
fedora design approach: 
 We desire to develop a Federated Ensemble-Directed Offline RL Algorithm (FEDORA) that addresses the issues outlined in Section 4 in a systematic manner. Three fundamental requirements drive our approach. First, the clients jointly possess an ensemble of local policies of different (unknown) qualities, and the server must leverage the collective knowledge embedded in this ensemble during federation. Second, the quality of these policies must be assessed using an ensemble of critics that depend on local data for policy evaluation. Finally, after each round of federation, clients must update their local policies via offline RL utilizing both their local data and the received federated policy.Maximizing the federated offline reinforcement learning (RL) objective in Eq. (4) using FedAvg would set weights as in Eq. (2), i.e., each client's contribution is weighted by the size of its dataset. This is is equivalent to solving the offline RL problem using the combined dataset of all clients. However, such an approach exacerbates the distribution shift problem that affects offline RL algorithms, leading to poor performance. This issue has been verified analytically and empirically in Yu et al. (2021) . We illustrated this phenomenon in Fig. 1 , where offline RL over pooled data resulted in a sub-optimal policy. The recommendation in Yu et al. (2021) is to share data conservatively by identifying which samples are likely to result in policy improvement. However, we cannot share any of the data across clients in the federated offline RL setting.Our solution is to follow the principle of maximum entropy to choose weights that best represent the current knowledge about the relative merits of the clients' policies. Here, the weights are prevented from collapsing over a few clients that have the best current performance by adding an entropy regularization over the weights with temperature parameter β resulting in the following objective:EQUATIONWe can then show using a Lagrange dual approach that this objective is maximized whenEQUATION)Based on these soft-max type of weights suggested by the entropy-regularized objective, we now design FEDORA accounting for each of the three requirements indicated above.In what follows, π (t,k) i denotes the policy of client i in round t of federation after k local policy update steps. Since all clients initialize their local policies to the federated policy at the beginning of each round of federation, π (t,0) i = π t fed for each client i. We also denoteπ t i = π (t,K) i, where K is the maximum number of local updates. Since all clients initialize their local critics to the federated critic, we can similarly defineEQUATIONQ t i = Q (t,K) ifor the local critic.We first require a means of approximating U Di (π fed ) in order to determine the weight w i of client i as shown in Eq. ( 6). We utilize the performance of the final local policyJ t i = E s∼Di [Q t i (s, π t i (s))], which also characterizes the relative performance at client i, as a proxy for U Di (π fed ). Here, Q t i is the local critic function at round t after K local updates. It is hard to directly obtain such an unbiased local critic Q t i in offline RL, since we do not have access to the environment for executing the policy and evaluating its performance. Our approach toward computing Q t i and π t i are described later. The accuracy of the local estimates J t i are highly dependent on the number of data samples available at i, and so in the usual manner of federated averaging, we need to account for the size of the dataset |D i | while computing weights. We thus have client weights and federated policy update asEQUATIONThe critic in our algorithm plays two major roles. First, offline RL for policy updates at each client requires policy evaluation using local data. Second, policy evaluation by the critic determines weight w t i of the local policy at client i for ensemble learning during each round t of federation. We desire a local critic at each client that can utilize the knowledge from the ensemble of critics across all clients while also being tuned to the local data used for policy evaluation.A critic based on offline data suffers from extrapolation errors as state-action pairs not seen in the local dataset will be erroneously estimated, greatly impacting actor-critic style policy updates in federated offline RL. Since the federated policy is derived from the set of local policies, it may take actions not seen in any client's local dataset. This problem is exacerbated when the local policy at the beginning of each communication round is initialized to the federated policy. We introduce the notion of federated optimism to train local critics, wherein critics leverage the wisdom of the crowd and are encouraged to be optimistic. We achieve this federated optimism via two steps.First, we use an ensemble-directed federation of the critics, where the local critic of client i at round t is weighed according to its merit to compute the federated critic asEQUATIONSuch entropy-regularized averaging ensures that the critics from clients with good policies significantly influence the federated critic.Second, for the local critic update, we choose the target value as the maximum value between the local critic and the federated critic, given byQ(t,k) i (s, a) = max Q (t,k) i (s, a), Q t fed (s, a) , where Q(t,k) i (s, a)is the target value of state s and action a at the t th round of federation after k local critic updates. This ensures that the local critic has an optimistic (but likely feasible) target seen by the system. Using this optimistic target in the Bellman error, we update the local critic asEQUATIONwhere a ′ = π (t,k) i. In practice, we obtain Q (t,k+1) i after a single gradient update.While essential in order to set ambitious estimates, an optimistic critic might erroneously estimate the value of Q(t,k) i . Therefore, regularizing the local policy update w.r.t. both the local data and the federated policy is crucial. For regularization w.r.t. to the local offline data, we use the same method as in the TD3-BC algorithm and define the local loss functionL local (π) = E (s,a)∼Di [-Q (t,k) i (s, π(s)) + (π(s) -a) 2 ].We then define the actor loss L actor in Eq. ( 10), where the second term is a regularization w.r.t. to the federated policy. The local policy is updated using L actor ,EQUATIONFEDORA uses a combination of local data loss and a proximal term for its policy update Eq. ( 10). However, the local data loss might hamper the updated policy's performance since the local dataset may be generated according to a non-expert behavior policy. Hence, a client must decay the influence of its local data if it is reducing the performance of the updated policy by lowering the influence of L local in L actor . To do so, we first evaluate the performance of the federated policy using the federated critic and local data at round t. For this evaluation, we use the proxy estimateJ fed,t i = E s∼Di [Q t fed (s, π t fed (s))].We compare this value with the performance of the updated policy, J t i , which is obtained using the updated critic. This difference provides us with an estimate of the improvement the local data provides. We decay the influence of L local by a factor δ if J fed,t i ≥ J t i . We summarize FEDORA in Algorithm 1 and 2. Compute π t+1 fed and Q t+1 fed for clients in N t using Eq. ( 7) and (8) respectively. 9: end forAlgorithm 1 Outline of Client i's Algorithm 1: function train_client(π t fed , Q t fed ) 2: π (t,0) i = π t fed , Q (t,0) i = Q t fed 3: for 1 ≤ k < K 
experimental evaluation: 
 We conduct experiments to answer three broad questions: (i) Comparative Performance: How does FEDORA perform compared to other approaches with client data generated by heterogeneous behavior policies?, (ii) Sensitivity to client updates and data quality: How does the performance depend on the number of local gradient steps at clients, the randomness in the available number of agents for federation, and the quality of the data at the clients?, and (iii) Ablation: How does the performance depend on the different components of FEDORA? We implement FEDORA over the Flower federated learning platform (Beutel et al., 2020) which supports learning across devices. We also provide a simulation setup that can be executed on a single machine (See Appendix A).Baselines: We consider the following baselines. (i) Fed-A: The local objective of all clients follows TD3-BC (Eq. 3). The server performs FedAvg over the actor's parameters, whereas each client learns the critic locally. (ii) Fed-AC: The local objective of all clients follows TD3-BC and the server performs FedAvg over the parameters of both the actor and the critic. (iii) Fed-AC-Prox: We add a proximal term to Fed-AC, which has been shown to help in federated supervised learning when clients have heterogeneous data (Li et al., 2020) . (iv) Heterogeneous Data-Aware Federated Learning (HDAFL) We extend HDAFL (Yang et al., 2020) to the offline RL setting by dividing the actor network into generic and client-specific parts and then federating only the generic part during Comparative Performance of FEDORA: In Fig. 2 , we plot the cumulative episodic reward of the server/federated policy during each round of communication/federation. We observe that FEDORA outperforms all federated baselines and achieves performance equivalent to or better than centralized training. Furthermore, the federated baselines fail to learn a good server policy even after training for many communication rounds and plateau at lower levels compared to FEDORA, emphasizing that the presence of heterogeneous data hurts their performance. To understand the effect of data coming from multiple behavior policies on centralized training, we consider a scenario where 50 clients with datasets of size |D i | = 5000 participate in federation, with 25 clients having expert data and the other 25 having random data, i.e., data generated from a random policy. From Fig. 3 , we notice that combining data of all clients deteriorates performance as compared to FEDORA. This observation highlights the fact that performing centralized training with data collected using multiple behavior policies can be detrimental.Sensitivity to Client Updates and Data Quality: We study the sensitivity of FEDORA to client update frequency and data quality in the Hopper environment in the same setting as in Fig. 2 . Increasing the number of local training steps can improve communication efficiency, but is detrimental under heterogeneous data due to client drift (Karimireddy et al., 2020) . In Fig. 4 (a), we study the effect of varying the number of local training epochs. We observe that increasing the number of epochs leads to faster learning, emphasizing that FEDORA can effectively learn with heterogeneous data. Not all clients may participate in every round of federation due to communication/compute constraints. In Fig. 4 (b), we study the effect of the fraction of clients participating in federation. We observe that FEDORA is robust towards variations in the fraction of clients during federation. Finally, in Fig. 4 (c) we study the effect of data heterogeneity by varying the percentage of clients with expert datasets. We observe that FEDORA performs well even when only 20% of the total clients have expert-quality data. We present several ablation studies and additional experiments in appendix B. We evaluated the performance of FEDORA on TurtleBot (Amsters & Slaets, 2020), a two-wheeled differential drive robot (Fig. 5 ) to collaboratively learn a control policy to navigate waypoints while avoiding obstacles using offline data distributed across multiple robots (clients). This scenario is relevant to several real-world applications, such as cleaning robots in various houses, which aim to collaboratively learn a control policy to navigate and avoid obstacles using data distributed across different robots. Collaborative learning is essential, because a single robot might not have enough data to learn from or have encountered adequately different scenarios. Additionally, federated learning overcomes the privacy concerns associated with sharing data among the robots. We collect data in the real-world using four behavior policies with varying levels of expertise ( Fig. 6(a) ). We train over 20 clients for 100 communication rounds, each consisting of 20 local epochs (see Fig. 6 (c)). Fig. 6 (b) shows the trajectories obtained by the learned policies of different algorithms in the real-world, and only FEDORA is able to successfully reach the target by avoiding the obstacle. We provide more details in Appendix C. A video of our experiment and code is provided in supplementary material. We discuss limitations and societal impact of our work in Appendix D. 
conclusion: 
 We presented an approach for federated offline RL, accounting for the heterogeneity in the quality of the ensemble of policies that generated the data at the clients. We solved multiple challenging issues by systematically developing a well-performing ensemble-directed approach entitled FEDORA, which extracts the collective wisdom of the policies and critics and discourages excessive reliance on irrelevant local data. We demonstrated its performance on several simulation and real-world tasks. 
 
 Appendix: We present several results and details in the appendix that illustrates the performance of FEDORA. These include details of our experimental setup (Appendix A), additional experiments studying different components of FEDORA and illustrating its performance in different settings (Appendix B), details of our real-world experiments using a TurtleBot (Appendix C), and discussion on limitations, societal impact and future work (Appendix D).Algorithm Implementation: We use the PyTorch framework to program the algorithms in this work, based on a publicly-available TD3-BC implementation. The actor and the critic networks have two hidden layers of size 256 with ReLu non-linearities. We use a discount factor of 0.99, and the clients update their networks using the Adam optimizer with a learning rate of 3 × 10 -4 . For training FEDORA, we fixed the decay rate δ = 0.995 and the temperature β = 0.1. TD3-BC trains for 5 × 10 5 time steps in the centralized setup. The batch size is 256 in both federated and centralized training.The training data for clients are composed of trajectories sampled from the D4RL dataset. In situations where only a fraction of the clients partake in a round of federation, we uniformly sample the desired number of clients from the entire set.Federation Structure: We implement FEDORA over the Flower federated learning platform (Beutel et al., 2020) , which supports learning across devices with heterogeneous software stacks, compute capabilities, and network bandwidths. Flower manages all communication across clients and the server and permits us to implement the custom server-side and client-side algorithms of FEDORA easily. However, since Flower is aimed at supervised learning, it only transmits and receives a single model at each federation round, whereas we desire to federate both policies and critic models. We solve this limitation by simply appending both models together, packing and unpacking them at the server and client sides appropriately.While 'FEDORA-over-Flower' is an effective solution for working across distributed compute resources, we also desire a simulation setup that can be executed on a single machine. This approach sequentially executes FEDORA at each selected client, followed by a federation step, thereby allowing us to evaluate the different elements of FEDORA in an idealized federation setup.Compute Resources: Each run on the MuJoCo environments (as in Fig. 2 ) takes around 7 hours to complete when run on a single machine (AMD Ryzen Threadripper 3960X 24-Core Processor, 2x NVIDIA 2080Ti GPU). This time can be drastically reduced when run over distributed compute using the Flower framework.We perform an ablation study to examine the different components of our algorithm and understand their relative impacts on the performance of the federated policy. We use the experimental framework with 10 clients and the Hopper environment described in Section 4, and plot the performance of the federated policy with mean and standard deviation over 4 seeds. The ablation is performed in two ways: (a) We build up FEDORA starting with Fed-A, the naïve method which federates only the actor, and add one new algorithm component at a time and evaluate its performance. (b) We exclude one component of FEDORA at a time and evaluate the resulting algorithm.We observe in Fig. 7a that using priority-weighted averaging of the client's policy to compute the federated policy (Eq. ( 7)), and an optimistic critic (Eq. ( 8) -( 9)) significantly improves the performance of the federated policy. This is consistent with our intuition that the most important aspect is extracting the collective wisdom of the policies and critics available for federation, and ensuring that the critic sets optimistic targets. The proximal term helps regularize local policy updates (Eq. ( 10)) by choosing actions close to those seen in the local dataset or by the federated policy. In this section, we study the effects of clients having data from multiple behavior policies for varying proportions of clients participating in federation. We consider a scenario with 50 clients having D i = 5000 in the Hopper-v2 environment where,• 12 clients have expert data (samples from a policy trained to completion with SAC.).• 12 clients have medium data (samples from a policy trained to approximately 1/3 the performance of the expert).• 14 clients have random data ( samples from a randomly initialized policy).• 12 clients have data from the replay buffer of a policy trained up to the performance of the medium agent.We run FEDORA by varying the the percentage of clients participating in each round of federation. We observe that the FEDORA is fairly robust to the fraction of clients participating in federation even when the fraction is as low as 20%. We train four behavior policies of varying levels of expertise using TRPO Schulman et al. (2015) on a custom simulator for mobile robots described in section C.2. The first policy is capable of waypoint navigation but collides with obstacles. The second policy can reach waypoints while avoiding obstacles present at one fixed position. The third policy has not fully generalized to avoiding obstacles at various positions. Finally, the fourth policy can navigate to the goal without any collision. We execute the behavior policies in the real-world by varying the waypoint (target location) and location of the obstacle to gather demonstration data, which we then use to train FEDORA and other baselines. Each client has a dataset consisting of 300 data points collected using a single behavior policy. After training, we test the learned policies in the real-world on a TurtleBot to ascertain its feasibility.We develop a first-order simulator for mobile robots using the OpenAI Gym framework, which enables the training of RL algorithms. The robot's pose is represented by its X-and Y-coordinates in Under review as a conference paper at ICLR 2024 a 2D space and its orientation with respect to the X-axis, θ. The pose is updated using differential drive kinematics x t+1 = x t + ∆t v cos θ t y t+1 = y t + ∆t v sin θ t θ t+1 = θ t + ∆t ω,where (x t , y t , θ t ) is the pose at time t, v and w are the linear and angular velocity of the robot respectively, and ∆t is time discretization of the system.The simulator uses a functional LIDAR to detect the presence of obstacles. We simulate the LIDAR using a discrete representation of the robot and obstacles in its immediate environment. For each scanning direction around the LIDAR, we use Bresenham's line algorithm to generate a path comprising of discrete points. The simulator determines LIDAR measurements by counting the number of points along each path, starting from the robot and continuing until it encounters and obstacle or reaches the maximum range.The reward function is designed to encourage effective waypoint navigation while preventing collisions. We define a boundary grid that extends for 1m beyond the start and the goal positions in all directions. The reward function at time t for navigating to the goal position (x g , y g ) is chosen to be We evaluate the trained algorithms on a Robotis TurtleBot3 Burger mobile robot (Amsters & Slaets, 2020) , an open-source differential drive robot. The robot has a wheel encoder-based pose estimation system and is equipped with an RPLIDAR-A1 LIDAR for obstacle detection. We use ROS as the middleware to set up communication. The robot transmits its state (pose and LIDAR information) over a wireless network to a computer, which then transmits back the corresponding action suggested by the policy being executed.R t =        +100, if |x t -In this work, we examine the issue of Federated Offline RL. We make the assumption that all clients share the same MDP model (transition kernel and reward model), and any statistical variances between the offline datasets are due to differences in the behavior policies used to collect the data.Moving forward, we aim to broaden this to cover scenarios where clients have different transition and reward models. To achieve this, we plan to extend ideas from offline meta RL to the federated learning scenario. Furthermore, we plan to explore personalization in federated offline RL as an extension to our research. We also believe that our approach may also be useful in the context of federated supervised learning, especially when the data is sourced from varying qualities, and we intend to formally investigate this in the future as a seperate line of work. In Fig. 15 run FEDORA for different values of β, which is the temperature parameter of federation. We consider a scenario similar to Fig. 1 , where we consider the Hopper-v2 environment with 10 clients having |D i | = 5000 participating in federation, where 5 clients have expert data, and 5 clients have medium data. When β = 0, it boils down to a uniform weighting scheme, where the quality of data present in each client is not considered during federation. As β → ∞ it tends to a max weighting scheme, where the federated policy is the same as an individual client's policy with the highest quality data. 


Decision: A summary of the strengths and weaknesses based on the reviews and rebuttal (including follow-up discussion and that among the reviewers) is provided below:


**STRENGTHS**

1. This paper proposes a novel federated offline RL algorithm in the general case (without assuming linearity).

2. The paper is well-written.

3. The experiments are extensive and compelling and include that of a real robot platform.


**WEAKNESS**

The paper lacks a theoretical performance analysis of the algorithm, for which the authors acknowledge would be challenging due to multiple components being involved. **However, the reviewers think that the technical contributions of this work (listed under the strengths) outweigh this weakness.**

As a minor comment, I like to suggest that the authors include other references on federated reinforcement learning, especially those published in the 3 flagship ML conferences.

The authors are encouraged to revise their main paper based on the above feedback and that of the reviewers, as well as based on their rebuttal (e.g., limitations).

Reviews: review: paper_summary: paper_summary: This paper presents the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA), a novel approach for collaborative learning of high-quality control policies in a federated offline reinforcement learning (RL) setting. The paper identifies key challenges in federated offline RL, including ensemble heterogeneity, pessimistic value computation, and data heterogeneity. To address these issues, FEDORA estimates the performance of client policies using only local data and, at each round of federation, produces a weighted combination of the constituent policies that maximize the overall offline RL objective, while maximizing the entropy of the weights. Besides the core idea, FEDORA also performs data pruning
strength_weakness: strength_weakness: 1. "Collect wisdom" can be replaced by more rigorous exposition. Same goes with "ambitious targets". 

2. The number of communication rounds needed for FEDORA to converge is still quite high. 

3. Given how well the algorithm does, some sort of theoretical analysis could further strengthen the work.
questions: questions: My questions are stated above.
limitations: limitations: Yes, limitations are adequately discussed.
paper_summary: paper_summary: The authors identify fundamental challenges for Federated Offline Reinforcement Learning and present Fedora, an approach that tackles each of them. They perform extensive evaluation of the approach on Mujoco and real-world datasets showing improved performance over existing work.
strength_weakness: strength_weakness: No theoretical guarantees have been given for the algorithm though it does build upon foundational work.  I believe that the authors should explicitly discuss limitations/ opportunities for future work in the paper. It is important for the algorithm pseudocode to be included in the main material as is the norm in such papers. I believe that there are perhaps many experiments included the main paper meaning that the discussion/ hypotheses for results is somewhat diluted. 
Another minor issue is that the figures are placed very far away from where they are referred to in text.
questions: questions: * How far do the authors perceive that this model can be pushed? i.e. the assumption that all clients have the same MDP is restrictive but understandable for a first set of experiments.

* Have any experiments been run using D4RL-random datasets? It would be interesting to see whether this collapses learning.
With regards to FEDORA outperforming centralised training I think a deeper discussion on this would be useful. 

* What is the main reason for this? Heterogenous data though previous work has successfully mixed datasets: https://arxiv.org/abs/2106.06860
limitations: limitations: Limitations should be explicitly stated. I feel that the authors could give a more balanced view of the algorithm by not only showing strengths but also assessing the limits of the work.
paper_summary: paper_summary: This paper proposed the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm. The combination of offline RL and federated learning is interesting in addressing the training data insufficiency issue due to small pre-collected datasets.
strength_weakness: strength_weakness: 1. Some technical details need to be explained. For example, the ensemble learning and its role.
2. The novelty of this paper needs further clarification, and what is the main difference between this proposed method and existing studies? It seems that there is only a simple combination of two technologies.
3. Numerically, the authors could consider comparing their method with more baselines. There are some studies on federated learning for offline RL.
questions: questions: 1. Some technical details need to be explained. For example, the ensemble learning and its role.
2. The novelty of this paper needs further clarification, and what is the main difference between this proposed method and existing studies? It seems that there is only a simple combination of two technologies.
3. Numerically, the authors could consider comparing their method with more baselines. There are some studies on federated learning for offline RL.
limitations: limitations: 1. What is the technical drawback of the proposed method? E.g., the effectiveness of the agent weight by ensemble approach
2. Does this proposed method work for other RL algorithms?

Comments: title: Response noted
comment: Thanks for the authors' efforts in the detailed response. My concern has been addressed. I will raise my score to weak accept.
Thank you for engaging with the review. I think a brief discussion of the limitations should be in the main paper. Please ensure that all other promised changes are made: I think that my comments can be used to move around some important content into the main paper. If the concerns are addressed then I will stick to my original score.