Title: Towards Federated Reinforcement Learning Free of Problem-Parameter-Based Tuning
Primary area: reinforcement learning
Keywords: Federated reinforcement learning, adaptive optimization, adaptive stepsize, environment heterogeneity

Abstract: Federated reinforcement learning (FRL) has emerged as a promising framework for decentralized decision-making in privacy-sensitive environments. However, its practical deployment is often impeded by the need for extensive hyperparameter tuning, a challenge further exacerbated by non-stationary data distributions and environment heterogeneity across agents. To overcome these limitations, we propose a \textbf{problem-parameter-free FRL framework} that eliminates manual stepsize selection through a novel integration of adaptive strategies and momentum-based updates. To further address environment heterogeneity, we introduce carefully designed control variates at both the client and server levels. Building on this framework, we develop two algorithms: \textbf{PFedPG-VR}, which integrates an adaptive scheme into variance-reduced policy gradient updates, and \textbf{PFedPG-HA}, which refines the approach using Hessian-aided corrections. Through rigorous theoretical analysis, we prove that both algorithms achieve state-of-the-art convergence rates, with a sample complexity of $\mathcal{O}(\epsilon^{-3})$ and a communication complexity of $\mathcal{O}(\epsilon^{-2})$. In addition, they enjoy linear convergence speedups with respect to the number of agents and local update steps at each global round. Notably, our methods eliminate the reliance on environment heterogeneity bounds required in previous FRL approaches, significantly broadening their applicability. Extensive experiments on benchmark FRL tasks further demonstrate the superior performance and robustness of our proposed algorithms compared to existing baselines.