SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Setup: We consider three MuJoCo tasks [44] with the MIT License, which have continuous state spaces. Specifically, a Swimmer-v4 task with small state and action spaces, a Hopper-v4 task with middle state and action spaces, and a Humanoid-v4 task with large state and action spaces are considered as described in Table 3. Policies are parameterized by fully connected multi-layer perceptions (MLPs) with settings in Table 4. We follow the practical settings in TRPO with line search [38], and in stable-baselines3 [35] with generalized advantage estimation (0.95) [39] and the Adam optimizer [18] in our implementation. Convergence performances are measured over 10 runs with random seeds from 0 to 9. The solid lines in Figure 2 and 3 are averaged results, and the shadowed areas are confidence intervals with  $95\%$  confidence level. We use PyTorch [32] to implement deep neural networks and RL algorithms. The tasks are trained on NVIDIA RTX 3080 GPU with 10 GB of memory.
Performance metrics: We consider performance metrics as follows:
1. Communication overhead: the data size transmitted from each agent;
2. Rewards: the average trajectory rewards across the batch collected at each iteration;
3. Convergence: rewards versus iterations during the training process.
We first evaluate the influence of the number of federated agents. In Figure 2, with different numbers of agents, we test the convergence performances of standard average FedNPG in (12) with  $\mathcal{O}(d^2)$  communication complexity and FedNPG-ADMM in Algo. 1 with  $\mathcal{O}(d)$  communication complexity at each iteration. The x-axis denotes the number of iterations in federated learning. Both algorithms converge faster, have lower variance, and achieve higher final rewards when more agents are engaged to collect trajectories. Compared to the standard average FedNPG, FedNPG-ADMM does not only reduce communication complexity, but also keeps convergence performances. The hyperparameter and MLP settings are described in Table 4. We summarize the final reward results with standard deviations in Table 2. FedNPG-ADMM achieves similar final rewards compared to the standard average FedNPG. It also works with slightly high variance when only one agent is engaged.
In Figure 3, we compare the performances of the standard average FedNPG, FedNPG-ADMM, and first-order FedPPO algorithms. The number of federated agents  $N$  is fixed to 8. PPO clipping parameter is set as 0.2. FedNPG methods outperform FedPPO in these tasks with faster convergence and higher final rewards. It is noticeable that FedNPG-ADMM has similar convergence rates and achieves slightly higher final rewards than the standard average FedNPG for the Swimmer-v4 task, while it becomes slightly lower for the Humanoid-v4 task. Generally, there is no significant difference in convergence rates after ADMM approximation. The communication overhead is measured by the number of transmitted parameters with double precision in each agent. FedNPG-ADMM keeps the communication overhead as the first-order methods, while FedNPG has much higher costs. The ADMM method reduces the cost by 4 orders of magnitude in the Swimmer-v4 task and about 6 orders of magnitude in the Humanoid-v4 task.
(a) FedNPG (Swimmer-v4)
(b) FedNPG-ADMM (Swimmer-v4)
(c) FedNPG (Hopper-v4)
(d) FedNPG-ADMM (Hopper-v4)
Figure 2: Reward performances of standard average FedNPG and FedNPG-ADMM on MuJoCo tasks, where  $N$  is the number of federated agents. Top: Swimmer-v4, Bottom: Hopper-v4. Left: FedNPG with  $\mathcal{O}(d^2)$  communication complexity, Right: FedNPG-ADMM with  $\mathcal{O}(d)$  communication complexity.
Table 2: Final rewards in federated settings.
# Agents	1	2	4	8
Swimmer-v4	FedNPG	111.9 ± 5.4	119.5 ± 3.4	122.1 ± 3.6	124.8 ± 2.4
FedNPG-ADMM	109.4 ± 5.9	123.6 ± 10.3	127.2 ± 2.2	128.5 ± 1.9
Hopper-v4	FedNPG	1644 ± 396	2468 ± 426	2458 ± 171	2736 ± 158
FedNPG-ADMM	1473 ± 383	2384 ± 371	2507 ± 230	2719 ± 173
(a) Swimmer-v4 Rewards
(b) Humanoid-v4 Rewards
(c) Swimmer-v4 Overhead
(d) Humanoid-v4 Overhead
Figure 3: Comparisons of FedPPO, standard average FedNPG, and FedNPG-ADMM on MuJoCo tasks, where the number of federated agents  $N$  is 8 and the communication overhead is measured by the transmitted bytes in each agent. Left: Swimmer-v4 task, Right: Humanoid-v4 task, Top: Reward performances, Bottom: Communication overhead.
Table 3: Description of the MuJoCo environment.
Tasks	Agent	Action Dimension	State Dimension
Swimmer-v4	Three-link swimming robot	2	8
Hopper-v4	Two-dimensional one-legged robot	3	11
Humanoid-v4	Three-dimensional bipedal robot	17	376
Table 4: Hyperparameter and MLP settings.
Hyperparameter	Setting
Task	Swimmer-v4	Hopper-v4	Humanoid-v4
MLP	64 × 64	128 × 128	512 × 512 × 512
Activation function	ReLU	ReLU	ReLU
Output function	Tanh	Tanh	Tanh
Penalty (ρ)	0.1	0.1	0.01
Radius (δ)	0.01	0.01	0.01
Discount (γ)	0.99	0.99	0.99
Timesteps (T)	2048	1024	512
Iterations (K)	1 × 103	2 × 103	3 × 103
Learning rate	3 × 10-4	3 × 10-4	1 × 10-5
In Figure 4, we test performances with agent selection. In the Swimmer task, we randomly select  $75\%$  and  $50\%$  of agents in each iteration, and the performances only drop slightly (final rewards drop less than  $6\%$ ). Thus, our proposed method is robust for agents with disconnection in practice.
Figure 4: Reward performances of FedNPG-ADMM on the Swimmer-v4 task with agent selection. In each iteration, the server randomly selects  $100\%$ ,  $75\%$ , and  $50\%$  of agents for the aggregation.
6 Conclusion & Discussion
In this paper, we proposed a novel federated natural policy gradient (NPG) algorithm that estimates global directions using ADMM (FedNPG-ADMM). Our ADMM-based gradient updates significantly reduce communication complexity from  $\mathcal{O}(d^2)$  to  $\mathcal{O}(d)$ , where  $d$  is the number of model parameters. It thus enables second-order policy gradient methods to be used for large-scale federated reinforcement learning problems. We proved that our FedNPG-ADMM algorithm achieves stationary convergence without requiring more samples as compared to standard FedNPG. Furthermore, we empirically showed the improved performances of FedNPG-ADMM in the MuJoCo environments as compared to FedNPG and the first order methods.
Overall, our proposed FedNPG-ADMM algorithm provides a scalable and efficient solution for large-scale federated reinforcement learning problems. Our contributions include a novel direction on policy-based FedRL, a new algorithm, reduced communication complexity, and convergence analysis. We believe that our approach can have a significant impact on a wide range of real-world applications, where large-scale distributed reinforcement learning with communication and privacy constraints is critical.
Limitations: (1) While the communication complexity improvement is loosely tied to the privacy advantage, exploring deep connections with differential privacy improvement is open. (2) Partial agent participation is not studied under this framework, in which communication complexity can be further reduced. (3) In experiments, it would be more persuasive to extend the number of federated agents to a larger scale.
We thank all reviewers for their invaluable feedback that helped us significantly strengthen the work. C. Brinton acknowledges financial support from NSF CNS-2146171 and DARPA D22AP00168-00. Han Wang and James Anderson acknowledge financial support from NSF grants ECCS-2144634 and ECCS-2231350.
[1] Agarwal, A., Kakade, S.M., Lee, J.D., Mahajan, G.: On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research 22(1), 4431-4506 (2021)