Model-Free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approach
Authors: Han Wang
Leonardo Felipe Toso
Aritra Mitra
James Anderson
Venue: Accepted by TMLR ()

Abstract: We study a model-free federated linear quadratic regulator (LQR) problem where M agents with unknown, distinct yet similar dynamics collaboratively learn an optimal policy to minimize an average quadratic cost while keeping their data private. To exploit the similarity of the agents' dynamics, we propose to use federated learning (FL) to allow the agents to periodically communicate with a central server to train policies by leveraging a larger dataset from all the agents. With this setup, we seek to understand the following questions: (i) Is the learned common policy stabilizing for all agents? (ii) How close is the learned common policy to each agent's own optimal policy?  (iii) Can each agent learn its own optimal policy faster by leveraging data from all agents? To answer these questions, we propose the federated and model-free algorithm FedLQR. Our analysis overcomes numerous technical challenges, such as heterogeneity in the agents’ dynamics, multiple local updates, and stability concerns. We show that FedLQR produces a common policy that, at each iteration, is stabilizing for all agents. Moreover, we prove that when learning each agent's optimal policy, FedLQR achieves a sample complexity reduction proportional to the number of agents M in a low-heterogeneity regime, compared to the single-agent setting.

Published in Transactions on Machine Learning Research (02/2026)
Model-Free Learning with Heterogeneous Dynamical Sys- tems: A Federated LQR Approach
Han Wang
han2786@@olumbicccc
Deportment of Electrical Engineering Columbia Universily
Leonardo F. Toso
Comaadoo.tsso@oooammmiioo eduu
Department Electrical Engineering Columbia University
Aritra Mitra
amdrro2@ncsu edu
Department of Electrical and Computer Engineering North Carolina State University
James Anderson
james. wommes.onderoon@ooummbiccc edu
Department of Electrical Engineering
Columbia Universily
Reviewed on OpenReview: https: optpr/vveeeeeeeeee/e net/ forum? id= WSRQeCUc3g
Abstract
We study a model-free federated linear quadratic regulator (LQR) problem where M agents with unknown, distinct yet similar dynamics collaboratively learn an optimal policy to minimize an average quadratic cost while keeping their data private. To exploit the similarity of the agents' dynamics, we propose to use federated learning (FL) to allow the agents to periodically communicate with central server to train policies by leveraging larger dataset from all the agents. With this sctup, we seek to understand the following questions: (i) Is the learned common policy slabilizing for all agents? (ii) How close is the learned common policy to each agent's own optimal policy? (iii) Can each agent learn its own optimal policy faster by leveraging data from all agents? To answer these questions, we propose the federated and model-free algorithm FedLQR. Our analysis overcomes numerous technical challenges, such heterogeneity in the agents' dynamics, multiple local updates, and stability concerns. We show that FedLQR produces common policy that, at each iteration, is stabilizing for all agents. Morcover, we prove that when learning cach agent's optimal policy, FedLQR achieves a sample complexity reduction proportional to the number of agents M in a low- heterogeneity regime, compared to the single-agent setting.
1 Introduction
There has been significant progress in the application of model-free reinforcement learning (RL) methods to fields such as video games (Mnih et al., 2015) and robotic manipulation (Rajeswaran et al., 2017; Levine et al., 2016; Tobin et al., 2017). In particular, RL has recently been used in the fine-tuning of pretrained foundation models (Ziegler et al., 2019). Although RL has shown impressive results in simulation, it often suffers from poor sample complexity, thereby limiting its ellectiveness in real-workd applications (Dulac-Arnold el. al., 2019). To resolve the sample complexity issue and accelerate the learning process, federated learning (FL) has emerged as a popular paradigm (Konečný et al., 201 2016a; McMahan et al., 2017), where multiple similar agents collaboratively learn common model without sharing their raw data. The incentive for collaboration arises from the fact that these agents "similar" in some sense and hence end up learning "superior"

---

Published in Transactions on Machine Learning Research (02/2026)
Server
Kn+1
1nn
K MM,
Agent 1
Agent
Agent M
System (A,,B1)
System M (AM, BM)
Figure 1: Illustration of the heterogeneous federated LQR setup involving M agents, where the i-th agent interacts with its own LTI system characterized by the pair (A,, B,) for i E [M]. The agents communicate via central server in communication rounds n .,, and upload their locally computed control gain matrices (K1)) hepan in each round n. The server averages the agents' gain matrices to compute global control policy Kn+1 that is broadcast to all agents to initiate the (n + 1)-th round.
model than if they were to learn alone. In the RL setting, Federated Reinforcement Learning (FRL) aims to learn a common value function (Wang et al., 2024b) or produce better policy from multiple RL agents interacting with similar environments. In the survey paper by Qi et al. (2021), FRL has empirically shown great success in reducing the sample complexity for autonomous driving (Liang et al., 2022), IoT devices (Lim et al., 2020), and resource management in networking (Yu et al., 2020).
Lately, there has been lot of interest in applying RL techniques to classical control problems such as the linear quadratic regulator (LQR) problem (Anderson & Moore, 2007). In the standard control setting, the dynamical model of the system is known and one seeks to obtain a controller that stabilizes the closed-loop system and provides optimal performance. RL approaches such as policy gradient (Williams, 1992; Sutton et al., 1999) (which we pursue here) differ in that they are "model-free", i.e., a control policy obtained despite not having access the model of the dynamics. Despite the lack of convexity even simple problems, policy gradient (PG) methods have been shown to be globally convergent for certain structured settings such the LQR problem (Fazel et al., 2018). While this is promising, a major challenge in applying PG mot is that in general, one does not have access to eract deterministic policy gradients. Instead, one relies estimating such gradients via sampling based approachos. This typically leads to noisy gradients that can suffer from high variance. As such, reducing the variance in PG estimates to achieve "good performance" may end up requiring several samples.
Motivation. The main premise of this paper to draw on idcas from the FL litcrature to alleviate the high sample-complexity burden of PG methods (Agarwal al., 2019; Wang al., 2019; Liu al., 2020), with the focus being on model-free control. As motivating example, consider fleet of identical robots produced by the same manufacturer. Each robot can collect data from its own dynamics and learn its own optimal policy using, for instance, PG methods. Since the fleet of robots shares similar dynamics, and more data can potentially lead to improved policy performance (via more accurate PG estimates), it is natural to ask: Can a robot accelerate the process of learning its own optimal policy by leveraging the data of the other robots in the fleet? The answer is not as obvious as one might expect since in reality, it unlikely that any two robots will have evactly the same underlying dynamics, i.e., heterogeneity in system dynamics is inevitable. The presence of such heterogeneity makes the question posed above both interesting and non- trivial. In particular, when the heterogeneity across agents' dynamics is large, leveraging data from other agents might degrade the performance of a single agent. Indeed, large heterogeneity may make it impossible to learn a common stabilizing p Moreover, even when such a stabilizing policy exists, it may deviate from each agent's 's local optimal policy, rendering poor performance and discouraging participation in the FL
1 See Section 5 for more details the underlying intuition and necessity behind the low heterogeneity regime.
2

---

Published in Transactions on Machine Learning Research (02/2026)
process. Thus, to understand whether more data² helps or hurts, it is crucial to characterize the effects of heterogeneity in the lederated control setting.
With this aim in mind, we study a multi-agent model-free LQR problem based on policy gradient methods. Specifically, there are M agents in our setup, each with its own distinct yet similar linear time-invariant (LTT) dynamics. Inspired by the typical objective FL,, our goal is to find a common policy which can minimize the average of the LQR costs across agents.* In particular, we address the following questions.
Q1. Is this common policy stabilizing for all the systems? If so, under what conditions?
Q2 How for is the learned common policy from each agent's 's locally optimal policy?
Q3. Can an agent use the common policy as an inilial guess to fine-tune and learn ils own optimal policy faster (I.e., with fewer overall samples) than if it acted alone?
There are several challenges to answering the above questions. First, even for the single agent setting, the policy gradient-baser LQR problem is non-convex, and requires fairly intricate analysis (Fazel et al,, 2018). Second, key distinction relative to standard federated supervised learning stems from the need to guarantee stability at every averaging iteration - this problem is amplified in the hetcrogencous multi- agent scenario we consider. It remains an open problem to design an algorithm ensuring that policies are simultaneously stabilizing for each distinct system. Third, to reduce the communication cost, FL. algorithms rely on the agents performing multiple local update steps between successive communication rounds. When agents have non-identical loss functions, these local steps lead a "client-drift" effect where each agent drifts towards its own local minimizer (Charles & Konečný, 2020a; 2021a). While several works in FL have investigated this phenomenon (Li et al., 2020; Khaled et al., 2019a; 2020; Li et al., 2019a; Karimireddy et al., 2020; Pathak& & Wainwright, 2020a; Wang et al., 2020a; Acar et al., 2021; Gorbunov et al., 2021; Mitra et al., 2021; Mishchenko et al., 2022: Laguel et al., 2021), the effect of "client-drift" on stability remains completely uncxplored. Unless accounted for, such drift effects can potentially lead to non-stabilizing controllers.
Challenges:
Our Contributions: In response to the above challenges, we propose a policy gradient method called FedLQR to solve the (modee-based and model-free) federated LQR problem, and provide a rigorous finite- time analysis of its performance that accounts for the interplay between system heterogeneity, multiple local steps, client-drift effects, and stability. Our specific contributions in this regard are follows.
Iterative stability guarantees. We show via a careful inductive argument that under suitable require- ments on the level of heterogeneity across systems, the learning rate schedule can be designed to ensure that FedLQR provides a stabilizing controller at every iteration for all systems. Theorem 1 provides a proof in the mosel-based setting, and Theorem provides the model-free result.
Bounded policy gradient heterogeneity in the LQR problem. We prove in Lemma 3 that, for each pair of agents i E [M], the policy gradient direction (in the model-based setting) of agent is close to that of agent j, if their dynamics are similar (i.e., Definition 1). This is the first result to observe and characterize this bounded gradient heterogeneity phenomenon in the multi-agent LQR setting.
Quantifying the gap between FedLQR's output and each system's optimal policy. Building on Lemma 3, we prove that when the agents' dynamics are similar, the common policy returned by FedLQR is close to each agent's optimal policy; see Theorem In other words, we can leverage the federated formulation to help each agent find its own optimal policy up some accuracy depending on the level of heterogeneity. Moreover, we prove that FedLQR finds a controller that is close to the average's cost optimal solution up to heterogeneity bias (Corollary 1). Our work is the first to provide a result of this flavor.
Linear speedup. As our main contribution, we prove that in the model-free setting, FedLQR converges to a solution that is in a neighborhood of each agent's 's optimal policy, using M- -imes fewer samples relative to when each agent just uses its own data (see Theorem 2). The radius of this neighborhood captures the level of heterogeneity across the agents' dynamics. The key implication of this result is that in a low-
"In accordance with both FL & FRL frameworks, the agents in our problem do not exchange their private data (c.g., rewards, states, etc.). Instead, each agent only transmits its policy gradicnt. While one can certainly consider other FL objective functions, we study the average LQR cost formulation here since comprehensive analysis this setting missing in the current. literature.
3

---

Published in Transactions on Machine Learning Research (02/2026)
heterogeneity regime, FedLQR (in the model-free setting) reduces the somple-compecnity by factor of M w.r.t. the centvalized selling (Fazel et al., 2018; Malik et al. 2019), highlighting the benefit of collaboration. Simply put, FedLQR enables each agent to quickly find an approzimate locally optimal policy; as in standard FL (Collins et al., 2022), the agent can fine-tune this policy based on its own data.
In summary, we provide a new theoretical framework that quantitatively characterizes the interplay between the price of heterogeneity and the benefit of collaboration for model-free control.
2 Related Work
There has been a line of work that explores various RL algorithms for solving the model-free LQR problem (Fazel et al., 2018; Malik et al., 2019; Hambly et al., 2021; Mohammadi et al., 2021; Gravell et al., 2020; Jin et al., 2020; Ju et al., 2022). However, their analysis is limited to the singlo-agent sctting. Most recently, Ren et al. (2020) solve the model-free LQR tracking problem in a federated manner and achieve a linear convergence speedup with respect to the number of agents. However, they consider a simplified setting where all agents follow the some dynamics. As such, the stability analysis of Ren et al. (2020) follow from arguments for the centralized setting. In sharp contrast, to establish the linear speedup for FedLQR, need to address the key technical challenges arising from the effect of heterogeneity and local steps the stability of distinct systems. This requires new analysis tools that we develop. For related work on multi-agent RL (that do not specifically look at the control setting) we point the reader to Lin et al. (2021); Zhang el al. (2021) and the references therein. Below we highlight the relevant work related to our problem setting.
Federated Learning (FL): In this work, we employ the foderated learning (FL) paradigm to facilitate collaborative learning among systems without the need to share raw data with other participants or a server (Konečný et al., 2016a; McMahan et al., 2017; Konečný et al., 2016b; Bonawitz al., 2019). Despite FL being a relatively recent approach, it has already gained significant attention and boasts wealth of literature.
Federated averaging (FedAvg) stands as the pioneering and most widely adopted algorithm in FL. Originally proposed in McMahan et al. (2017), FedAvg has demonstrated its effectiveness in ho- mogencous settings (Stich, 2018; Wang & Joshi, 2021; Spiridonoff et al., 2020; Reisizadeh et al., 2020; Haddadpour et al., 2019) where all participating clients aim to minimize the same objective function. However, ensuring convergence guarantees for FedAvg becomes notably more challenging in the presence of heterogencity (Khaled et al., 2019b; 2020; Haddadpour & Mahdavi, 2019; Li et al., 2019b), thus necessitating additional assumptions on gradient and Hessian dissimilarity bounds (Li et al., 2019b; Li & Orabona, 2019; Khaled et al., 2019b; Karimireddy et al., 2020). This difficulty arises primarily due to a "client-drift" effect, which inherent to the FedAvg algorithm and has a detrimental impact on its convergence performance (Charles & Konečný, 2020b; 2021b). As result of the challenges posed by FedAvg, several alternative algorithms have been proposed to address its limitations. Notable examples include FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2020), FedSplit (Pathak & Wainwright, 2020b), FedDR (Tran Dinh et al., 2021), FedADMM (Wang et al., 2022a). FedLin (Mitra et al., 2021), and S-Local -SVRG (Gorbunov et al., 2021). Each of them introduces unique techniques and modifications to the original FedAvg algorithm, aiming to enhance convergence guarantees while handling communication cost concerus, statistical heterogeneity, client dropout, and sample complexity more effectively.
Applying federated learning (FL) to control systems introduces novel rescarch direction that comes with its own set of challenges. Control systems exhibit unique characteristics, such as non-iid and non-isotropic data, as well as system instability, which arise due to the dynamic nature of the systems. These characteristics pose specific challenges when attempting to leverage data from multiple systems for tasks such as system identification (Wang et al., 2022b) or control synthesis (Ren al., 2020).
Although Ren et al. (2020) address the model-free LQR tracking problem in multi-agent setting, it focuses on a significantly simpler scenario where all agents follow identical dynamics (i.e., homo-
'Throughout this paper, we use the terms "centralized" and "single-agent" interchangeably.
4

---

Published in Transactions on Machine Learning Research (02/2026)
gencous). In contrast, our work introduces new analysis techniques to achieve linear speedup when dealing with heterogeneous systems and multiple local updates per communication round.5
The policy gradient (PG) approach is a fundamental component of the success of reinforcement learning (RL) and plays a crucial role in policy optimization (PO). This approach directly optimizes the policy improve system-level performances through gradient ascent steps. The concept of policy optimization has been influential in RL (Sutton et al., 1999) with some well-known algorithms such REINFORCE (Williams, 1992), trust-region policy optimization TRPO (Schulman et al., 2015), actor-critic methods (Konda & Tsitsiklis, 1999), and proximal policy optimization PPO (Schulman et al., 2017). We highlight an important difference between standard MDP models and control models in RL. In control, one requires the policy to provide closed-loop stability, iee, all trajectories of the system must converge for a given policy. In contrast, there are no analogous stability concerns in finite state-action MDP settings studied in RL.
Policy Gradient (PG):
The extensive body of literature on policy optimization for reinforcement learning (RL) and its adaptability to the model-free setting paves the way for leveraging policy gradient methods in the pursuit of learning optimal control policies for classical control problems (Hu et al., 2022; Perdomo et al., 2021). Despite the non-convex nature of the formulation involved in policy gradient methods, recent work (Fazel et al., 2018; Malik et al., 2019; Hambly et al., 2021; Mohammadi et al., 2021; Gravell et al., 2020; Jin et al., 2020; Ju et al., 2022; Perdomo et al., 2021; Lamperski, 2020; Toso et al., 2024b; 2025) has demonstrated global convergence in solving the model-free LQR problem via policy gradient moth in different settings. This global convergence is achieved due to certain properties of the quadratic cost function inherent in the LQR problem (e.g., smoothness and gradient dominance) as established in Fazel et al. (2018). In contrast to the aforementioned work, which exclusively focus on the centralized control setting, our paper offers convergence guarantees for the multi-agent setting. In this context, each agent follows different dynamics, thereby distinguishing it from the simpler scenario proposed in Ren et al. (2020).
Federated Reinforcement Learning (FRL): The flexibility of policy gradient methods in the model-free RL setting has paved the way for a relatively recent. research direction known as federated reinforcement learning (FRL), which aims to address practical implementation challenges of RL through the use of federated learning (Qi et al., 2021). FRL focuses on learning a common value function (Wang et al., 2024b; Fabbro et al., 2023) improving the policy by leveraging multiple RL agents interacting with similar environments (Woo et al., 2023; Wang et al., 2024a). The empirical evidence presented in the survey paper (Qi et al., 2021) demonstrates the significant success of FRL in reducing sample complexity across various applications such as aulonomous driving (Liang et al., 2022), LoT devices (Lim et al., 2020), resource management in networking (Yu et al., 2020), and communication efficiency (Gatsis, 2022). The FRI. litcrature can be broadly groupod into two categories: (i) The homogeneous setting, where all agents interact with the same MDP (Khodadadian et al., 2022; Woo et al., 2023; Lan et al., 2023; Shen et al., 2023; Liu & Olshevsky, 2023; Fabbro et al., 2023; Tian et al., 2024; Salgia & Chi, 2024; Dal Fabbro et al., 2025), and (ii) the heterogeneous setting, where the agents' MDPs can potentially differ in their reward functions and/or probability transition kernels (Jin et al., 2022; Xie & Song, 2023; Wang et al., 2024b; Zhang et al., 2024; Wang et al., 2024a; Zhu et al., 2024; Mangold et al., 2024; Labbi et al., 2024; 2025). In particular, federated variants of policy gradient methods are analyzed in Xic & Song (2023); Lan ct al. (2023); Wang et al. (2024a); Zhu et al. (2024); Labbi et al. (2025); Zhang et al. (2025). It is important to note that all the papers in FRL discussed above consider MDPs with finite state and action spaces, where there is no notion of stability. In sharp contrast, our work considers federated control problem involving dynamical systems with continuous state-action spaces, where we need to tackle the challenge of finding a common, stabilizing policy for all the agents at every iteration.
"In the time since this work first made available ouline it has been used develop new results on multi-task LQR design, including studics on meta-learning linear quadratic regulators (Toso al., 2024c; Pan al., 2025; Aravind et al., 2024), asynchronous federated LQR design (Toso et al., 2024a; Zhao al., 2025), learning LQR, controllers from proxy systems (Ye et. al., 2024), and domain randomization for the TQQR problem (Fufinami et al., 2025).
5

---

Published in Transactions on Machine Learning Research (02/2026)
3
Notation
Given a set of matrices { we denote ||S||max == max; || and ||S == min, [|Stt1] All vector norms are Euclidean and matrix norms are spectral, unless otherwise stated.
4 Problem Setup
Classical control approaches aim to design optimal controllers from a well-defined dynamical system model. The model-based T.QR problem well understood and admits a convex solution. In this work, we consider the LQR problem but in the model-free setting. Moreover, we consider a federated model-free LQR problem in which there are M agents, each with their own distinet but "similar" dynamics. Our goal is to collaboratively learn optimal controller that minimizes average quadratic cost. We seek to characterize the optimality of our solution as function of the "difference" across the agent's dynamics. In what follows, we formally describe our problem of interest.
Federated LQR: Consider a system with M agents. Associated with each agent a linear time invarint (LTI) dynamical system of the form
2 ++1 = 200 24 + (0 zo ~ D, for all i [M],
where [M] := 1, ..... M, and A R B(0 E Xn. We assume each initial state To is randomly generated from the same distribution D. In the single-agent setting, the optimal LQR control policy is known to be linear and static. We denote the policy by u, Kttt where for each agent, Kद solves
KJ = arg min K
(K)(K) =
Qui + +u (8) Ru(
s.1.
L++1
(4) + B( u " = -K A
(i)
xo D,
(1)
where Q X", and R E X", are known positive definite matrices. In federated setting, the objective is to find an optimal common policy {u] to minimize the average cost of all the agents (K) = 1 Ei=1 (K) without knowledge of the system dynomies, i.e., (A(0), B()). As mentioned above, classical results (Anderson & Moore, 2007) from optimal control theory show that, given the system matrices A B(i) Q and R, the optimal policy can be written as linear function of the current state. Thus, consider common policy of the form ut -K The objective of the federated LQQ problem can be written as:
K arg min I Cavz( =
Σ
E
Σ
( Qui
(6) + +u
Ru
M
1=1
1=0
(1) S.L. 4+1 = 100 + B00
(1)
" =
(8)
K Ktt
I ~ D.
(3)
(2)
The rationale for finding K" is as follows. Intuitively, when all agents have similar dynamies, K* will be close to each KY. Thus, K" will serve to provide good common initial guess from which each agent then fine-tune/ personaaees (using only its own data) to converge ezactly to its own locally optimal controller K². The key here is that the initial guess K can be obtained quickly by using the collective data of all the agents. We will formalize this intuition in Theorem
We make the standard assumption that for each agent, ((i)) stabilizable. In addition, we make the following assumption on the distribution of the initial state:
Assumption 1. Given f > 0, for any i [Ad], the initial state T
D and distribution D satisfy
E 00 Ea (1) 0, E 1000 1
(1) 2
(T) > plla,, and || (8) || < H almost surely.
We quantify the heterogeneity in the agent's dynamics through the following definition:

---

Published in Transactions on Machine Learning Research (02/2026)
Definition 1. (Bounded system heterogeneity) There erist positive constants a and eg such that
max [1 - | S Se,, and max [I - AJE
B0 S 62.
We assume that E and Eg are finite. Similar bounded heterogeneity assumptions are commonly made in FL (Karimiroddy et al., 2020; Khaled et al., 2020; Reddi et al., 2020). However, unlike typical FL works where one directly imposes helerogeneity assumptions on the agents' gradients, in our setting, we need to carefully characterize how heterogeneity in the system parameters (A(), B()) translates to differences in the policy gradients; see Lemma 3.
On the meaning of system similarity: The notion of heterogeneity introduced in Definition 1 measures similarity in terms of parameter deviations between system matrices. It is important to emphasize that similarity in the system parameters (A(", B() does not in general imply similar closed-loop behavior (see Propositions and 2). Even a small perturbation in system parameters can produce significant differences in the set of stabilizing controllers, and may not guarantee the existence of common stabilizing controller across agents. Recent work (Stamouli et al., 2025) has proposed refined similarity mcasure based on bisimulation-lype metrics that quantify closedd-lopp heterogeneity rather than parameter differences, and the main results of our paper extend to their setting.
Before providing our solution to the federated LQR problem, we first recap existing results on model-free LQR in the single-agent setting.
When there is only one agent, i.e., M = 1, let us denote the system matrix as (A, B) and likewise, the cost functional by C(K). If (A, B) is known, the optimal controller K can be computed by solving the discrete-time algebracc Riccati equation (ARE) (Anderson & Moore, 2007).
Single-agent setting:
Strikingly, Fazel et al. (2018) show that policy gradient methods can find the globally optimal LQR policy K* despite the non-convexity of the problem. The policy gradient of the LQR problem can be expressed as:
VC(K) = Σm = 2 B -
PKA) ΣK.
where Px the positive definite solution to the Lyapunov equation: Pk = Q+ [O+KTTG KT RK (A- BK) P (A- BK), EK == (R+ BT BK K B PKA, and Σx := Exo- DE: I. The policy gradient method K K-nK--vev VC((K) will find the global optimal LQR policy, i.c., K-K", provided that EzywD [x is full rank and an initial stabilizing policy is used. When the model is unknown, the analysis technique employed by Fazel et al. (2018) is to construct near-exact gradient estimates from reward samples and show that the sample complexity of such a method is bounded polynomially in the parameters of the problem.
In contrast to the single-agent setting, the heterogeneous, multi-agent scenario we consider here is consid- erably more difficult to analyze. First, designing an algorithm satisfying the iterative stability guarantees becomes complex lask. Second, since each agent in the system has its own unique dynamics and gradient estimates, it can be difficult to aggregate these directions in a manner that ensures the updating direction moves toward the average optimal policy K Nonetheless, in the sequel, we will overcome these challenges and provide finite-time analysis of FedLQR.
5 Necessity of the Low Heterogeneity Requirement
In our main theorems, we require certain bounds on the parameters 61 and c that define the heterogencity of the M dynamical systems we work with. Here, we point out that, unlike standard federated learning settings, these bounds are necessary for convergence. From a control and dynamical systems viewpoint, these bounds are perhaps intuitive: if the systems are too different, then there no reason to believe there exists common stabilizing controller, i.e., there is no solution to the problem (2). In what follows, we will formalize this point. To do so, let us define an "instance' of our FedLQQ problem via parameter M that characterizes the number of agents geents/systems and the set of corresponding system matrices (A B(0) (()))e 6
Jie[M].
6 Although technically the cost. matrices Q and R are also part FedL.Q problem formulation, they are needed to establish the necessity of a low- hww-heterogeneity requirement. such, we do include them here.
7

---

Published in Transactions on Machine Learning Research (02/2026)
We now prove couple of simple impossibility results. Our first result shows that even when the imput matrices are identical across agents, heterogeneity in the state transition matrices can lead to the non- existence of simultaneously stabilizing controllers, thereby rendering the FedLQR problem infeasible.
Proposition 1. There exists an instance of the FedLQR problem with M = 2 and O, such that if > 2, then it is impossible to find a common state-feedback gain K that simultancously stabilizes both systems.
Proof: Consider an instance with just two scalar systems defined by:
2+1 o.r (1) + u/ and x++ = -ax (2) + u (2)
for some a > O. By simple inspection, note that in this case 61 = 2a and 62 = 0. Thus, 61 > = a > 1. Now for a controller ug = -kx to stabilize both systems, the spectral radius conditions are la k| <
and a + k| < 1. Trivially, there exists no gain & that satisfies both these requirements when a > 1.
To complement the above result, we now show that the effect of heterogencity is not just limited to the state iransition matrices. In particular, even when the stale transition matrices are identical across agents, (arbitrarily small) heterogeneity in the input matrices can also lead to the non-existence of simultaneously stabilizing control gains. We formalize this below.
Proposition 2. There exists an instance of the FedLQQR problem with M and 0, such that if > 0, then it is impossible to find common state-feedback gain K that simultancously stabilizes both systems.
Proof: Consider instance with two scalar systems defined by:
(1) x 4 _ (1) + Bu.") (1) and (2) Jz+1 z2 BBg -B
(2)
for some B. By simple inspection, note that in this E1 0 and 62 = 23. Thus, 62 > 0 + 8 > 0.
Now for a controller " ko, to stabilize both systems, the spectral radius conditions are || 3k] <1 and + [Kk < 1. Trivially, there exists no gain k that satisfies both these requirements when 0. This concludes the proof.
The above example suggests that in certain settings, we can tolerate no heterogeneity whatsoever in the input matrices. More generally, the main take-home message from this section is that the requirement of a llow-heterogeneity regime" is fundamental to the problem and not merely an artifact of our analysis. We formally define a low heterogeneity regime in the sequel.
6 The FedLQR algorithm
In this section, we introduce our algorithm FedLQR, formally described by Algorithm 1, to solve for K* in (2). First, we impose the following assumption regarding the algorithm's 's initial condition Ko:
Assumption 2. We can access an initial stabilizing controller, Ko, which stabilizes all systems ((A(), B() i.e., the spectral radius p(A()) B Ko) <1 holds for all i E [M].
Algorithm description: At a high level, FedLQR follows the standard FL algorithmic template: server first initializes a global policy, Ko, which it sends to the agents. Each agent proceeds to execute multiple PG updates using their local data. Once the local training is finished, agents transmit their model update to the server. The server aggregates the models and broadcasts an averaged model to the clients. The process repeats until a termination criterion is met. Prototypical FI. algorithms that adhere to this structure include, for instance, FedAvg (Khaled et al., 2020) and FedProx (Li et al., 2020).
With this template in mind, we now dive into the details: FedLQR initializes the server and all agents with K-(i) = K2- - a controller that stabilizes all agent's dynamics." In each round n, starting from a common global policy Kn, each agent i independently samples ni trajectories from its own system at each local ileration and performs approximate policy gradient updates using the zeroth-order optimization procedure (Fazel
TTo establish notation, x(() refers to the local controller associated to agent i after rounds of averaging, and after 7th local iteration of policy gradient updates.
8

---

Published in Transactions on Machine Learning Research (02/2026)
et al., 2018) which denote ZO; see line 7. For clarity, we present the explicit steps of using the zeroth- order method to estimate the true gradient in Algorithm 2, which will be discussed shortly. Between every communication round, each agent updates their local policy L times. Such an L is chosen to balance between the benefit of information sharing and the cost of communication. After L local iterations, cach agent i uploads its local policy difference (t) (line 10) to the server. Once all differences are received, the server averages these differences (A (line 12) to construct a new global policy Kn41- The whole process is repeated N times.
Algorithm 1 Model-free Federated Policy Learning for the LQR (FedLQR)
1: Input: initial policy Ko: local step-size in and global step-size 'g.
2: Initialize the server with Ko and fl
3: for n = :,....N N -
do
4: for each system E [M] do
5: for 1 0.... I. do
6: Agent i initializes k(] n.0 = Kn Agent i estimates VC (K = 20 20(K1)))) () and updates local policy as
7:
k-0i nl+1 = K n (
8:
9: end for
10: send A (1) n.L Kn back to the server
11: end for
12: Server computes and broadcasts global model Kn+1 Kn + 37 EM A.
13:
end for
Zeroth-order optimization (Conn et al., 2009; Nesterov & Spokoiny, 2017) provides a method of optimization that only requires oracle access to the function being optimized. Here, we briefly describe the details of our zeroth-order gradient estimation step* in Algorithm 2. To obtain a gradient estimate at given policy K, we sample trajectories from the i-th system n, times. At each time s, we use the perturbed policy K2 (line 3) and a randomly generated initial point Do 'D to simulate the i-th closed-loop system for 7 steps. Thus, we can approximately calculate the cost by adding the stage cost from the first T time steps on this trajectory (line 4), and then estimating the gradient as in line 6.
Discussion of Assumption 2: Assumption 2 is commonly adopted in the LQR (Fazel et al., 2018; Dean et al., 2020; Agarwal et al., 2019; Ren al., 2020) and robust control literature (Boyd al., 1994; Lu al., 1996; Doyle et al., 1989). In addition, there exist efficient ways to find such stabilizing policy Ko; (Boyd et al., 1994) that addresses the model- setting, while (Lamperski, 2020; Perdomo et al., 2021; Zhao et al., 2022; Toso et al., 2025) each addresses the problem in the single-agent model-free setting. Moreover, it is well-known that the sample complexity of finding an initial stabilizing policy only adds a logarithmic factor to that for solving the single-agent LQR problem (Zhao et al., 2022). More recently, Fujinami et al. (2025) have shown that solving a sequence discounted LQR problems using the average (a) cost yields a common stabilizing controller for all systems under low heterogeneity conditions.
Choice of the number of local updates: The parameter L controls the standard communication, drift trade-off encountered in federated optimization. Executing more local steps before aggregation reduces communication frequency, but at the cost of amplifying the deviation between the aggregated update and the true global gradient direction. Conversely, choosing a smaller L keeps the local iterates closer to the global descent direction and improves agreement with the centralized update, albeit requiring more communication rounds. In our setting, this trade-off becomes particularly delicate because drift wetw local controllers directly affects stability. We refer the reader to Lemmata 10 and 11 for additional details on how the number of local updates enters our analysis.
Challenges in FedLQR analysis: Although FedLQR is similar in spirit to FedAvg (T. et al., 2019b; McMahan et al., 2017) (in the supervised lcarning sctting), it is significantly more difficult to analyze the convergence of FedLQR for the following reasons.
8See Appendix A.5 for more details zeroth-order optimization.
9

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm 2 Zeroth-order gradient estimation (zO)
Input: K, number of trajectories na: trajectory length T; smoothing radius r, dimension ng and Bur system index i.
2: for s I,..,,, The do
3: Sample policy K2 = K + Us, with Ug drawn uniformly at random over matrices whose (Frobenius) norm is r.
4: Simulate the i-th system for 7 steps starting from To ~ 'D using policy K,, Let Õ2 be the empirical estimate: c.=Et. Ei= Ct where G = x (Q Kg RR,) 24 on this trajectory.
5: end for
6: Return the estimate:
AC(K)= = 1 Σx Ra Ö,U.
First, the problem we study is non-convex. Unlike most existing non-convex FL optimization re- sults (Karimireddy et al., 2020) which only guarantee convergence to stationary points, our work investigates whether FedLQR can find a globally optimal policy.
Second, standard convergence analyses in FL (McMahan et al., 2017; Karimireddy al., 2020; Wang et al., 2020b; Li et al., 2020) rely "bounded gradlent-heterggenitty assumption. For the LQR problem, it is not clear priori whether similar bounded policy gradient dissimilarity still holds. In fact, this is something we prove in Lemma 3.
Third, the randomness in FL usually comes from only one source: the data oblained by each agent are drawn ii.d. from some distribution; we call this sample randomness. However, in FedLQR, there are randomness in the initial condition, and randomness from the smoothing matrices that show up in the gradient estimation process outlined in Algorithm 2. To reason about these different forms of randomness (that are intricately coupled), we provide a careful martingale-basel analysis.
Finally, we need to determine whether the solution given by FedLQR is meaningful, i.c., to decide whether the policy generated at each (local and global) iteration will stabilize all the systems.
To tackle these difficulties, we first define a stability region in our setting comprising of M heterogencous systems as:
Definition 2. (The slabilizing set) The slabilizing set is defined as (B) == : g()) where
(() = (x: C() -(0 ((0 S (ko)))
C0
As in Malik et al. (2019), (B) is defined as the interscction of sub-level sets containing points K whose cost gap is at most B times the initial cost gap for all systems. We drop the dependence B when it is clear from the analysis and context. It was shown in Hu et al. (2022) that this is compact set. Each sub-level set corresponds to cost gap to agent i's optimal policy K;, which is at most B times the initial cost gap C0 (Ko) - (K)). Note that 3 can be any positive finite constant. Since any finite cost function indicates that K is stabilizing controller, we conclude that any K E (B) stabilizes all the systems. Following from Assumption 2, there exists a constant / such that (8) is nonempty. Morcover, it worth remarking that the LQR cost function in the single-agent setting coercive. That is, the cost acts as a barrier function, ensuring that the policy gradient update remains within the feasible stabilizing set (B). By defining the stabilizing set gD as above, the cost function (()(()) retains its cocrciveness on g ((8) for the federated setting considered in this paper. We also note that for our per-iteration stability analysis in Lemma 10, B is properly set and kept. fixed over iterations.
In order to solve the federated LQR problem and provide convergence guarantoes for FedLQR, we first need to recap some favorable propertics of the LQR problem in the single-agent setting that enables PG to find the globally optimal policy.
10

---

Published in Transactions on Machine Learning Research (02/2026)
7 Background on the centralized LQR using PG
In the singlo-agent setting, it was shown that policy gradient methods (i.c., model-froe) produce the global optimal policy despite the LQR problem being non-convex (Pazel et al., 2018). We summarize the properties that make this possible and which we also exploit in our analysis.
Lemma 1. (Local Cost and Gradient Smoothness) Suppose K is such that || K|| & ha(K) 00. Then, the cost and gradient function satisfy:
(K') C(K) heost (K)ss -|], K|1, IVC VC(K)) S 11 grad (K)]|A|) and IVC VC(K)ll hgrnal (K] Al]
respectively, where ha (K), (Kos((K) and hgraa((K are positive scalors depending on C(K). Lemma 2. (Gradient Domination) Let K* be an optimal policy. Then,
|E
-
[VC(K)
44 (R)
holds for any stabilizing controller K, i.e., any K satisfying the spectral radius p(A - BK) < 1.
For simplicity, we skip the explicit expressions in these lemmas for ha(K), heest (eoos(K); and Igraa
functions of the parameters of the LQR problem. Interested readers referred to the appendix for full details. With Definition 2 of the stabilizing set in hand, we can define the following quantities:
Tigrad= =- sup (ggra(K), Heost == sup heost (K), and ha Keg
inf ha(K).
Keg
KEG"
With these quantities, we can transform the local properties of the LQR problem discussed in Lemmas 1-2 into properties that hold over the global stabilizing set G For convenience, we use letters with with (under and over) bars such as hgrad to denote the global parameters. We are now ready present our main results of FedLQR in the next section.
8
Main results
To analyze the performance of FedLQR in the model-free case, we first need to examine its behavior in the model-based case. Although this is not our end goal, these results are of independent interest.
8.1
Model-based setting
When (4(4),(()) are available, exact gradients can be computed, and the Z0 estimation scheme is no longer needed. In this case, the updating rule of FedLQR (Algorithm 1, line 12) reduces to
M L-1
Kn+1 = Kn ML
17
ΣΣ VC(K)) n.11
where ry == Ll Intuitively, if two systems are similar, i.e., satisfy Assumption 1, their exact policy gradient directions should not differ too much. We formalize this intuition as follows.
Lemma 3. (Policy gradient heterogeneity) For any i,j E [M] and K g", we have:
[[VC((()) - VCC) S E1h + (K),
(3)
where (K) and (22ee(K) are positive bounded functions depending on the parameters of the LQR problem."
For simplicity, we write ld) Na function of only K since only K changes during the iterations while other parameters
remain fixed.
11

---

Published in Transactions on Machine Learning Research (02/2020)
g
G(1)
Ki
g (3)
1K2
g
Kn,f
(2)
Figure 2: Illustration of the dynamics of the FedLQR algorithm for sctting with M 3 agents. Recall from Definition 2 that G represents the set of stabilizing policies for the i-th agent, and go = n== is the common stabilizing set. In a low-heterogeneity regime, the agents' policy gradients are "close" to one another (as formalized by Lemma 3), causing the sequence/trajectoon of policies (controllers) generated by FedLQR to also evolve close to each other during the course of the algorithm. These trajectories (K())
P== g
are illustrated by the black arrows. Crucially, initializing these trajectories within go causes them to always remain within go i.e., the set go forward-invarion for the trajectories generated by FedLQR fact established rigorously in Lemma 10. This forward invariance property the key to showing that the policies generated by FedLQR remain stabilizing for all systems.
By Lemma 3 (the proof of which is deferred to Appendix A.4), if K belongs to a bounded set, the right-hand hand side of Eq. (3) is of the order O(e + 2)). In other words, the exact gradient direction of agent can be well-approximated by the gradient direction of agent j when the heterogeneity constants 61 and 22 are small. This justifies why it is beneficial to use other agents' data under the low-heterogeneity setting. Moreover, can immediately conclude that the exact update direction of our FedLQR algorithm is also close to each agent's 's policy gradient direction based on Lemma 3. This fact is crucial for analyzing the convergence of FedLQR since we can map the convergence of FedLQR to that of the centralized LQR problem (with only one agent). However, Lemma 3 alone is not sufficient to provide the final guarantees since we still need to consider the impact multiple local updates i.e., L >>> (Algorithum 1, line 5) and stability concerns with heterogeneous systems- i.e., is p(A(') BB KN) for all agents E [M] and all n E [N]? Nevertheless, by overcoming these difficulties, we establish the convergence of FedLQR in the model-based setting follows:
Theorem 1. (Optimality in each agent's 's cost function) When the heterogeneity level satisfies
(EI het + ²2 < hi there exist constant step-sizes Ta and mn such that FedLQR enjoys the follow- ing performance guarantees over N rounds:
(((K))-C((33) < 1
Mipo mmin (R)
-(()))))((((()) + Cani,1 X B(EI,E2),
- -||=x]
with (((1,62) == o (R) (Ea <2hhet) where hhe == sup 1.2( min(nz, and Cund,1 is a universal constant. Moreover, we have Kn E g for all n 0, ... NN
Main Takeaways: Theorem reveals that the output K₂ of FedLQR can stabilize all M systems at each round n. However, FedLQR can only converge to a ball of radius 6(E1)62) around each system's optimal controller Kü, regardless of the choice of the step-sizes. The term ((G,, (2) captures the effect of heterogencity
10 notation 13 a positive scalar depending on the parameters of the LQR, problem; see Appendix A.4.2 for full details.
12

---

Published in Transactions on Machine Learning Research (02/2026)
and becomes zero when each agent follows the same system dynamics, Le., 61 = E2 O. When there is no heterogeneity, the convergence rate matches the rate of the centralized setting (Fazel et al., 2018) up to a constant factor. But, since there is no noise introduced by the zeroth-order gradient estimate, there is no expectation of obtaining a benefit from collaboration. Nonetheless, understanding the model-basod setting provides valuable insights for exploring the model-free setting. The proof of Theorem given in Appendix A.4.2. We are now ready to provide the convergence guarantees for FedLQR with respect to solving Eq. (2) the average cost.
Corollary 1. (Optimality in average cost function) When the heterogeneity level satisfies"l (E + 22 I Keey after N rounds, FedLQR enjoys the following optimality gap in average cost function across all M agents:
N
(1 1
mh o min! (R) max; [Xx]1] ||
Caug(Kx (Nx) - Corg( S
sup (C")(K)) -C(0 + Cuni,1 x B(ei, 62). sE(M)
The main message conveyed by Corollary 1 is that FedLQR can converge to a ball around the average optimal controller K* with linear convergence rate. The size of the ball depends the system heterogeneity level, i.., and 62. Combining Theorem and Corollary 1, we infer that FedLQR not only approximates each system's optimal controller KJ but also approximately converges toward the average optimal controller K* when the underlying M systems are close (in the sense of Definition 1). The primary distinction between converging to Ki and K* lies in the linear convergence rate. Compared to converging to K2; where the linear converge rate depends only on system i's parameter ||<; ||. the rate in converging to K* depends on all systems' parameters, i.e., [||=x;1] N [0 See Appendix A.4.3 for a comprehensive proof.
How to ensure FedLQR's stability? We bricfly discuss proof technique for ensuring the iterative stability guarantees. The main idea is to leverage an inductive argument. We start from stabilizing global policy K, E GO. We aim to show that the next global policy Ka+1 is stabilizing; see Fig. 2 for an illustration. This is achieved by demonstrating that Kn+1 can reduce each system's cost function compared to Kn. To achieve this goal, we take the following steps: (1) at each itcration, initiate from the globally stabilizing controller computed at the previous ilerate, (2) determine a small global step-size such that inequalities in Section 7 can be applicd; (3) use Lemma to provide a descent direction to reduce each system's cost 1
function; (4) bound the drift term ML Em El=o (1(( - K, Step (4) can be accomplished using a small local step-size nn such that each local policy is a small perturbation of the global policy Kn-
On the role of control variates for local drift mitigation: Control-variate techniques such as proposed in SCAFFOLD (Karimireddy et al., 2020) are designed to correct the optimization drift that arises in federated optimization under non-llD) data by making each local update an approximately unbiased estimator of the global gradient. However, emphasize that such methods do not eliminate the intrinsic gap between the minimizer of the averaged LQR cost. Cavg( and the individual optima K2 when the underlying dynamical systems differ. This limitation reflects a fundamental distinction between control-variate methods in classical FL and their applicability to our setting of dynamical systems. In standard FL, bias correction can ensure that the global iterates closely follow an "idea" " gradient-descent step on a strongly convex objective, enabling the use of gradient-domination inequalities to guarantee fast. convergence. In contrast, for heterogeneous LQR systems, the averaged cost (av does not, in general, correspond to the LQR cost of any single "averaged" dynamical system, and hence, one cannot directly appeal to pradient-domination properties like in the optimization setting to achieve fast linear rates. An important exception arises when all agents share identical dynamics but differ only through their cost matrices: in that case, Cavg reduces to the LQR cost associated with (A, B, Q, It), where Q = (1/M) Exe[M] Q. and π = (1/M) EiE[M] R., making gradient- domination applicable and mitigating the local drift as discussed in Zhu et al. (2024).
Equipped with these model-based results, we are ready to present our main results of the model-free setting,
The notation 13 is a positive scalar depending on the parameters of the LQR problem; see Appendix A.4.2 for full details.
13

---

Published in Transactions on Machine Learning Research (02/2026)
8.2 Model-free setting
We now analyze FedLQR's convergence in the model-free setting, where the policy gradient steps are approx- imately computed using peroth-order optimization (Algorithm 2), without knowing the true dynamics, i.e., 1 B are not available and so VC (K(() can't be directly computed. The key point in this sctting is to bound the gap between the estimated gradient and the true gradient. In the centralized setting (Fazel et al., 2018), the gap can be made arbitrarily accurate with enough trajectory samples ns; sufficiently long trajectory length т, and small smoothing radius
We aim to achieve sample complexity reduction for each agent by utilizing data from other similar but non-identical systems with the help of the server. This presents a significant challenge, as averaging gradient estimates from multiple agents may not necessarily reduce the variance even for homogeneous systems due to the high correlation between local gradient estimates. This challenge is compounded in our case as the gradient estimates are not only correlated but also come from non-ide con-identical systems. As a result, the variance reduction and sample complexity reduction for the FedLQR algorithm is not obvious priori. After addressing these challenges using a partingale-based analysis, we show that one can establish variance reduction for our setting well. This is formalized in the next result:
Lemma 4. (Variance Reduction) Suppose the smoothing radius and trajectory length from Algo- rilhm 2 salisfy S hr (9) and 2 hr (Immmu) respeclively. 12 Moreover, suppose the sumple size sulisfies 13
6 H² $
haample, trunc
MT P
ng
(4)
ML
Then, when Kn E Go with probability 1- 6, the estimated gradients satisfy:
M 1-1
1 MT
ΣΣ 1=
-
-V0))))))))))))
Se.
F
We prove this result and provide the definition of the parameters of bammple in Appendix A.6. The most important information conveyed by our variance reduction lemma is that each agent at each local step only needs to sample MT. fraction of samples required in the centralized setting. Notably, this lemma plays important role in showing that FedLQR can help improve the sample efficiency. Equipped with Lemnma 4, we now present the main convergence guarantees for FedLQR:
1
Theorem 2. (Model-free) Suppose the trajectory length satisfies T 2 ft re' the smoothing radius A||=x
s satisfies r S N. (. and the sample size of each agent n. satisfies Eq. (4) with e' m When
the heterogeneity level satisfies (erhher + e2hh,et) S Whet then, given any (0, 1), with probability there exist constant step-sizes My and m, which are independent of d', such that FedLQR enjoys the following performance guarantees:
1. (Stability of the global policy) The global policy al each round is slabilizing, i.e., K E GO;
2. (Stability of the local policies) The local policies satisfy K(} go for all i and I;
||=x; ||
log (1000)))))-0000033)) rounds, we have
(Convergence rate) After N > qhl mmp
(K()))
S (2), (2)
(5)
where (2) is defined in Theorem 1.
12ehe notation he, he, heample, trune and W in Lemma 4 and Theorem 2 are polynomial functions of the LQR problem, depending For simplicity, we defer their definition the appendix. Fo convenience comparison with existing literature, we use the same notation in Fazel et. al. (2018).
14

---

Published in Transactions on Machine Learning Research (02/2026)
This theorem establishes the finite-time convergence guarantees for FedLQR. The first two points in Theorem 2 provide the iterative stability guarantees of FedLQR, i.e., the trajectories of FedLQR will always stay inside the stabilizing set G The third point implies that when heterogeneity is small, ie., E2) is negligible, FedLQR converges to each system's optimal policy with a linear speodup w.r.t. the number of agents M, which we discuss further next.
Discussion: For a fixed desired precision e, we denote N to be the number of rounds such that the first term e' in Eq (5) is smaller than In what follows, we focus analyzing the total sample complexity of FedLQR for each agent, which can be calculated by N x ngg. Note that N, in our case, is in the same order as the centralized setting. However, in terms of the sample size requirement at each local step, it is only ML = of that needed in the centralized setting, as presented in the variance reduction Lemma 4. Therefore, in a low- hewerogeergyyyt regime, where ((E1, is negligible, our FedLQR algorithm reduces the sample complexity of learning the optimal LQR policy by Ō (( of the centralized setting (Fazel et al., 2018; Malik et al., 2019). 11 Specifically, FedLQR improves the sample cost required by each agent from ( to Ō( Ö(d)))) up to small heterogeneity bias term. This result is highly desirable since the number of agents in FL is usually large; leading to a significant speedup due to collaboration.
It is important to mention that our results also capture the cost of federation embedded in the term B(c1, That is when two systems exhibit significant differences from each other, leveraging data across them may not be beneficial in finding a common stabilizing policy that applics to both. In summary, Eq. (4)-(5) provide an explicit interplay between the price of heterogeneity and the benefit of collaboration. The trade-off in Theorem 2 is explored in the simulation study presented in the next section.
9 Numerical Results
The following section describes the experimental setup and results for FedLQR in the model-free setting!5.
9.1
System Generation
Numerical experiments are conducted to illustrate and evaluate the effectiveness of FedLQR (Algorithm 1). The simulations involve different and unstable dynamical systems described by discrete-time linear time- invariant (LTI) models, as in (4), where each system has n₂ = 3 states and n, = 3 inputs. To generate different systems while respecting the bounded heterogeneity assumption (Assumption 1), the following steps are followed:
1. Given nominal system matrices (Ao. Bo), generate random variables 7 ~ ((0 and 12 u(0,e) e2), Vi E [M], with and e2 being predefined dissimilarily parameters.
The random variables generated above are combined with modification masks Z, E 3 and Zz E 33x3 to generate the different systems matrices (A(), B(0) for all e [M].
3. The systems (A (()) for 0 < M are then constructed by perturbing the nominal systems
according to: A = Ao + 71 Z, and B60 Bo + 7/ Z2, where Z, and Z2 are defined in step 2.
B(J) 4. The nominal matrices are included in the set of generated systems (A)) (Ao, Bo).
In particular, we consider
1.20
0.50
0.40
Ag =
0.01
0.75
0.30
B = Is. Q= Q= and
2 -
0.10
0.02
1.50
14 Fazel al. (2018), the sample complexity of policy gradient with one-point zeroth-order estimation O( this was later improved to O by Malik al. (2019). We compare results to the refined analysis in Malik al. (2019).
15 Code can be downloaded from https://gåthub. http:///ghhub.cccc//dadderr//e/e con/jjd-andersso/Feeel..
15

---

Published in Transactions on Machine Learning Research (02/2026)
Single system (M-1)
0,
Multiple systems (M-10)
0.30 6g 0.15
Multiple systems (M=50)
1.5, E2 0.65
ghiKc cllipe
6 10-3.
x 10-1
4x 10-1
3x
1000
2000
Number global iterationn
Number global iterations (n)
(a) Varying M with = 0.5.
(b) Varying 61, <2, with M 10.
Figure 3: Gap between the current and optimal cost with respect to the number of global iterations.
for the nominal system matrices and cost matrices respoctively. The optimal controller for the nominal system (4(1), B()) is
1.0056
0.4293
0.3570
KT
0.0262
0.6239
0.2657
0.1003
0.0298
1.2960
and was obtained by solving the discrete algebraic Riccati equation (DARE).
9.2 Algorithm Parameters
For the gradient cstimation step in the zeroth-order algorithm (Algorithm 2), we set the initial state for cost computation as a random sample from a standard normal distribution, denoted as D _ N la), for all systems E [M]. Additionally, we consider n. = 5 trajectories, where each trajectory has a rollout length of T 15, and we set the smoothing radius r = 0.1 for the zeroth-order gradient estimation.
Throughout our simulations, we consider the following initial stabilizing controller Ko 1.62Is (Line in Algorithm 1). Note that although the control action u,") = =Ko²? (i) may not be optimal for any of the M systems. For example, the suboptimality of Ky applicd to the nominal system is evidenced by its cost of (K)) = 18.4049, compared to the optimal cost of (²2) (KG) = 9.5220, when computed from an initial (1) [1 Ij and time horizon T = 500. However, it is important to note that Ko is still stabilizing state 20
all M systems. Note that we will use K₂ as the initial controller for all of the experiments in this paper.
9.3 Experiments
To assess the performance of FedLQR, we evaluate the normalized gap between the current cost (()((M)) of the nominal system when using the common stabilizing controller K, and its corresponding optimal cost (Kt)))) This metric is represented as (C)(K))-C(((() for each global iteration n E [N]. In our experi- ments, we set the step sizes as n2 = x 10-2, with an adaptive decrease of 0.05% per global iteration, and = 10-4 Further details regarding other parameters, such as the number of systems M, heterogeneity levels 62) are provided in the figures and the subsequent discussion.
(()(A))
Figures ba present the normalized distance between the current cost associated with the common stabi- lizing controller and the optimal cost for the nominal system, plotted with respect to the number of global iterations. These figures demonstrate the impact of varying the number of systems M and the heterogeneity parameters (e1, 62) on the convergence and performance of Algorithm 1.
In Figure 3-(a), we specifically investigate the effect of the number of systems M participating the collaboration compute a common controller K on the convergence of our algorithm. In this analysis, we set the heterogeneity parameters é 0.5 and 62 0.5 and consider modification masks Zn Z2 la-
16

---

Published in Transactions on Machine Learning Research (02/2026)
The figure reveals noticcable reduction in the gap between the current and optimal cost the number of participating systems M increases. This numerical result aligns with our theoretical findings, which indicate that the number of samples required to achieve reliable estimation for the cost function's gradient can be scaled down with the number of systems participating in the collaboration. Consequently, as the number of systems involved increases, there considerable reduction the gap between the common computed controller and the optimal one.
Figure 3-(b) illustrates the influence of the heterogeneity parameters (c1, 62) on the convergence rate of Algorithm 1. In this analysis, set the mumber of systems M = 10, and the modification masks Z = diag( [3.5 1 0.1)) and Z2 = diag((1.5 0.1 1)). Consistent with our thcoretical findings, we observe that an increase in the dissimilarity among the systems results in a siguificant gap between the common and optimal controller. This discrepancy arises due to the additive effect of system heterogeneity on the convergence rate of our algorithm, as elaborated in Theorem
10 Conclusions and Future Work
We investigated the problem of learning common and optimal LQR policy with the objective of minimizing an average quadratic cost. The primary focus of this paper was to thoroughly examine and provide com- prehensive answers to the following questions: (i) Is the learned common policy stabilizing for all agents? (ii) How close is the learned common policy to each agent's own optimal policy? (iii) Can each agent learn its own optimal policy faster by leveraging data from all agents? To address these questions, we proposed a federated and model-free approach, FeaLQR, where M heterogencous systems collaborate to learn a com- mon and optimal policy while keeping the system's data privale. Our analysis tackles numerous technical challenges, including system heterogeneity, multiple local gradient descent updates, and stability. We have demonstrated that FedLQR produces common policy that stabilizes all systems and converges to the optimal policy (Theorem 2) of each agent to a seterrogeneit-inndcer bias term Furthermore, FedLQR achieves a reduction in sample complexity proportional to the number of participating agents M (Lemma 4). We have also provided numerical results to effectively showcase and evaluate the performance our FedLQR approach in freel-free setting. Future work will address the assumption of requiring full-state information to ex- tend results to the Linear Quadratic Gaussian (T.QG) problem in federated setting. We are currently investigating data-driven and system-theoretie metrics for heterogencity, well as personalization-baat methods to mitigale the impact of heterogeneily on the performance of the proposed approach.
Acknowledgments
The authors thank the anonymous reviewers for their thorough and constructive feedback, which significantly improved this work. Leonardo F. Toso is funded by the Center for AI and Responsíble Financial Innovation (CAIRFI) Fellowship and by the Columbia Presidential Fellowship. James Anderson is partially funded by NSF grants ECCS 2144634 and 2231350 and the Columbia Center of AI Technology in collaboration with Amazon.
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXio preprint arXiv:2111.04233, 2021.
Naman Agarwal, Elad Hazan, and Karan Singh. Logarithmic regret for online control. Advonces in Neural Information Processing Systems, 32, 2019.
Brian DO Anderson and John B Moore. Optimal control: linear quadratic methods. Courier Corporation, 2007.
Ashwin Aravind, Mohammad Taha Toghani, and César Uribe. A moreau envelope approach for lqr meta- policy estimation. In 2024 IEEE 63rd Conference on Decision and Control (CDC), pp. 415-420. IEEE, 2024.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Francis Bach and Vianney Perchet. Highly-smooth zero-th order online optimization. In Conference Learning Theory, pp. 257-283. PMLR, 2016.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloc Kiddon, Jakub Koncčný, Stefano Mazzocchi, Brendan McMahan, et al. Towards foderated learning scale: System design. Proceedings of machine learning and systems, 1:374-388 -784-388,. 2019.
Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan. Linear matria inequalities in system and control theory. SIAM, 1994.
Zachary Charles and Jakub Konečný. On the outsized importance learning rates in local update methods. arXiv preprint ar arXi:20077.006 0087,, 2020a.
Zachary Charles and Jakub Konečný. On the outsized importance of learning rates in local update methods. arXiv preprint XrXi:22007.08875 2020b.
Zachary Charles and Jakub Koncčný. Convergence and Accuracy Trade-Offs in Federated Learning and Meta-
Learning. In International Conference on Artificial Intelligence and Stalistics, pp- 2575 2583. PMLR, 2021a.
Zachary Charles and Jakub Konečný. Convergence and accuracy trade-offs in federated learning and meta- lcarning. In International Conference on Artificial Intelligence and Statistics, pp. 2575-2583. PMLR, 2021b.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Fodavg with fine tuning: Local updates lead to representation learning. orXiv preprint arXiv:2205. 13692, 2022.
Andrew R Conn, Katya Scheinberg, and Lais N Vicente. Introduction to derientive-free oplimization. STAM, 2009.
Nicolò Dal Fabbro, Aritra Mitra, Robert W Heath, Luca Schenato, and George Pappas. Finite-time analysis of over-the-air federatod td learning. IEEE Transactions on Wireless Communications, 2025.
Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, 20(4):633 -699, 2020.
John C Doyle, Keith Glover, Pramod Khargonekar, and Bruce Francis. State-space solutions to standard H2 and Ho control problems. JEEE Transactions on Automatic Control, 34(8):831-84) 1989. doi: 10.1109/9.29425.
John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, (1())27788-288 2015.
Gabriel Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint ar arXiv:1904, 12901, 2019.
Nicolò Dal Fabbro, Aritra Mitra, and George J Pappas. Federated td learning over finite-rate crasure channels: Linear speedup under markovian sampling. arXiv preprinl X3x:02305008144 2023.
Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International conference on machine learning, pp. 1467 1476. PMLR, 2018.
Tesshu Fujinami, Bruce D Lee, Nikolai Matni, and George J Pappas. Policy gradient for lqr with domain randomization. arXiv preprint arXi::2003. 24871, 2023.
Konstantinos Gatsis. Federated reinforcement learning at the edge: Exploring the commnictto tradeoff. In 2022 Europeon Control Conference (FCC), pp- 1890-1895, TEFF, 2022.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. Local SGD: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics, pp. 3556 -564. PMI.R, 2021.
Benjamin Gravell, Peyman Mohajerin Esfahani, and Tyler Summers. Learning optimal controllers for linear systems with multiplicative noise via policy gradient. IEEE Transactions on Aulomalic Control, 66(11): 5283-52988 2020.
Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint 1rX0x:1910.144 14425, 2019.
Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck Cadambe. Local sgd with periodic averaging: Tighter analysis and adaptive synchronization. Advances in Neural Information Processing Systems, 32, 2019.
Ben Hambly, Renyuan Xu, and Huining Yang. Policy gradient methods for the noisy lincar quadratic regulator over finile horizon. SIAM Journal Control and Oplimizalion, 59(5):3359 3391, 2021.
Bin Hu, Kaiqing Zhang, Na Li, Mchran Mcsbahi, Maryam Fazel, and Tamer Başar. Towards thcoretical foundation of policy optimization for learning control policies. arXiv preprint arXiv:2210004810, 2022.
Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Federated reinforcement learning with environment heterogeneity. In International Conference on Artificial Intelligence and Statistics, pp. 18 18.37. PMLR, 2022.
Zeyu Jin, Johann Michael Schmitt, and Zaiwen Wen. On the analysis of model-free methods for the linear quadratic regulator. arXiv preprint wrXiv:200700361,, 2020.
Caleb Ju, Georgios Kotsalis, and Guanghui Lan. A model-free first-order method for linear quadratic regulator with ((1/e) sampling complexity. arXiv preprint arXi::221.00088, 2022.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143. PMLR, 2020.
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. First analysis of local gd on heterogeneous data. arXiv preprint arXiv:1909.04715, 2019a.
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. First analysis of local gd on heterogencous data. arXio preprint arXiw:1909.0477 04715, 2019b.
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogencous data. In International Conference on Artificial Intelligence and Statistics, pp. 4519-4529. PMLR, 2020.
Sajad Khodadadian, Pranay Sharma, Gauri Joshi, and Siva Thoja Maguluri. Federated reinforcement learn- ing: Linear speedup under markovian sampling. In International Conference on Machine Learning, pp. 10997-11057. PMLR, 2022.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Dis- tributed machine learning for on-device intelligence. arxie preprint orXiv:1610. 02527, 2016a.
Jakub Konečný, Н Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Foderated learning: Strategics for improving communication efficiency. arXio preprint urXiv:1610.05.p00 2016b.
Safwan Labbi, Daniil Tiapkin, Lorenzo Mancini, Paul Mangold, and Eric Moulines. Federated ucbvi: Commuiiatiinn-fffiii foderated regret minimization with heterogeneous agents. ar preprint arXiv:2400 10. 10.22008, 2024.
19

---

Published in Transactions on Machine Learning Research (02/2026)
Safwan Labbi, Paul Mangold, Daniil Tiapkin, and Eric Moulines. On global convergence rates for federated policy gradient under heterogeneous environment. arXiv preprinl or Xri:22505. 2025.
Yassine Laguel, Krishna Pillutla, Jerôme Malick, and Zaid Harchaoui. A superquantile approach to federated learning with heterogencous devicos. In 2021 55th Annual Conference on Information Sciences and Systems (CISS), pp. 1-6. IEEE, 2021.
Andrew Lamperski. Computing stabilizing linear controllers via policy iteration. In 2020 59th IEEE Con- ference on Decision and Control (CDC), pp. 1902-1907. IEEE, 2020.
Guangehen Lan, Han Wang, James Anderson, Christopher Brinton, and Vaneet Aggarwal. Improved commu- nication efficiency in federated natural policy gradient via ADMM-based gradient updates. arXiv preprint arXiv:2310.190 19807, 2023.
Sergey Levine, Chelsoa Finn, Trevor Darrell, and Picter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):133-13373, 2016.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXio:1907.0218) 2019a.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang, On the convergence of fedavg on non-iid data. orXiv preprint arXio:1907.02119, 2019b.
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd international conference artificial intelligence and Statistics, pp- 083-992. PMT.R, 2019.
Xinle Liang, Yang Liu, Tianjian Chen, Ming Liu, and Qiang Yang. Federated transfer reinforcement learning for autonomous driving. In Federated and Transfer Learning, pp. 357-371. Springer, 2022.
Hyun-Kyo Lim, Ju-Bong Kim, Joo-Scong Hoo, and Youn-Hee Han. Federated reinforcement learning for training control policies on multiple iot devices. Sensors, 20(5):1359, 2020.
Yiheng Lin, Guannan Qu, Longbo Huang, and Adam Wicrman. Multi-agent reinforcement learning in stochastic networked systems. Advances in Neural Information Processing Systems, 34:7825-788 77377,77777 2021.
Rui Liu and Alex Olshevsky. Distributed TD (0) with almost communication. IEEE Control Systems Letters, 7:2892 2897, 2023.
Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems, 33: 2020.
W-M Lu, Kemin Zhou, and John C Doyle. Stabilization of uncertain linear systems: An 1ft approach. IEEE Transactions on Automatic Control, 41(1):50 65, 1996.
Dhruy Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter Bartlett, and Martin Wainwright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. In The 22nd international conference on artificial intelligence and statistics, pp. 2916-2925, PMLR, 2019.
Paul Mangold, Sergey Samsonov, Safwan Labbi, Ilya Levin, Reda Alami, Alexey Naumov, and Eric Moulines. Scafflsa: Taming heterogeneity in federated linear stochastic approximation and ud learning. Advonces in Newral Information P Processing Systems, 2024.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaisc Aguera y Arcas. Communication-offiien learning of deep networks from decentralized data. In Artificial intelligence and slatistics, pp. 1273-1222221 1282. PMLR, 2017.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Konstantin Mishchenko, Grigory Malinovsky, Scbastian Stich, and Peter Richtárik. ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! arXiv preprint or (xXx:2202009357 2022.
Aritra Mitra, Rayana Jaafar, George J Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogencity and sparse gradients. Advances in Newral Information Processing Systems, 34:14606 14619, 2021.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et. al. Human-level control through deep reinforcement learning. nature, 518(7540):529 -3333 2015.
Hesameddin Mohammadi, Armin Zare, Mahdi Soltanolkotabi, and Mihailo R Jovanović. Convergence and sample complexity of gradient methods for the modcl-free linear-quadratic regulator problem. JEEE Transaclions on Aulomalic Control, 67(:)34435 2450, 2021.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17:527-566, 2017.
Yunian Pan, Tao Li, and Quanyan Zhu. Model-agnostic meta-policy optimization via zeroth-order estimation: A linear quadratic regulator perspective. arXiv preprint ar arXiv:2003.0033 2025.
Reese Pathak and Martin J Wainwright. FedSplit: An algorithmic framework for fast federated optimization. arXiv preprint Xix:20005. 05238, 2020a.
Reese Pathak and Martin J Wainwright. Fedsplit: An algorithmic framework for fast federated optimization. Advances in neural information processing systems, 33:7057 7066, 2020b.
Juan Perdomo, Jack Umenberger, and Max Simchowitz. Stabilizing dynamical systems via policy gradient methods. Advances in Neural Information Processing Systems, 34:29274-29286, 2021.
Boris T Polyak. Introduction to optimization. optimization software. Inc., Publications Division, New York, 1:32, 1987.
Jiaju Qi, Qihao Zhou, Lei Lei, and Kan Zheng. Federated reinforcement learning: Techniques, applications, and open challenges. arXiv preprint Xix::2100. 1188, 2021.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 700.10097.. 2017.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00200 2020.
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fed- paq: A communication-effiieee federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics, pp. 2021-2031. PMLR, 2020.
Zhaolin Ren, Aoxiao Zhong, Zhengyuan Zhou, and Na Li. Federated LQR: Learning through sharing. arXiv preprint crXic:2011.018155l, 2020.
Sudeop Salgia and Yuejic Chi. The samplecommmncation complexity tradc-off in federated q-lcarning. Advances in Newral Information Processing Systems, 2024.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. ar Xiv preprint Xiv: 0ae:1707.0633 707.06347, 2017.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Han Shen, Kaiqing Zhang, Mingyi Hong, and Tianyi Chen. Towards understanding asynchronous advantage actor-critic: Convergence and linear speedup. IEEE Transactions on Signal Processing, 2023.
Artin Spiridonoff, Alex Olshevsky, and Ioannis Ch Paschalidis. Local sgd with a communication overhead depending only on the number of workers. arXiv preprint S55::2006.025883 2020.
Charis Stamouli, Leonardo F Toso, Anastasios Tsiamis, George J Pappas, and James Anderson. Policy Gradient Bounds Multitask LQR. arXiv preprinl arXiv:2509. 19266, 2025.
Sebastian Stich. Local sgd converges fast and communicates little. arXiv preprint orXiv:1005.097 2018
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Haoxing Tian, Ioannis Ch Paschalidis, and Alex Olshevsky, One-shot averaging for distributed TD (X) under markov sampling. IEEE Control Systems Letters, 2024.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23 30. IEEE, 2017.
Leonardo F Toso, Han Wang, and James Anderson. Asynchronous heterogeneous linear quadratic regulator design. In 2024 IEFE 63rd Conference Decision and Control (CDC), pp. 801-808. IEEE, 2024a.
Leonardo F Toso, Han Wang, and James Anderson. Oracle complexity reduction for model-free lqr: A stochastic variance-reduced policy gradient approach. In 2024 American Control Conference (ACC), pp. 4032-4037. IEFF, 2024b.
Leonardo F Toso, Donglin Zhan, James Anderson, and Han Wang. Meta-learning linear quadratic regulators: A policy gradient maml approach for the model-free LQR. arxiv preprint urXiv:240111453, 2024c.
Leonardo F Toso, Lintao Ye, and James Anderson. Learning stabilizing policies via an unstable subspace representation. arXiv preprint arXiv:2505.nm 01348, 2025.
Quoe Tran Dinh, Nhan H Pham, Daung Phan, and Lam Nguyen. Feddr-randomizdd douglas-rachford splitting algorithms for nonconvex federated composite optimization. Advances in Neural Information Processing Systems, 44:30326-30333 2021.
Joel Tropp.
Freedman's inequality for matrix martingales. Electron. Commun. Probab., 2011.
Han Wang, Siddartha Marella, and James Anderson. FedADMM: A federated primal-dual algorithm allowing partial participation. In 2022 IEEE 61st Conference on Decision and Control (CDC), pp. 287 -94.. IEEE, 2022a.
Han Wang, Leonardo F Toso, and James Anderson. FedSysID: A federated approach sample-offciient system identification. Xixiv preprint orXie:2211.14490 2022b.
Han Wang, Sihong He, Zhili Zhang, Fei Miao, and James Anderson. Momentum for the win: Collabora- tive federated reinforcement learning across heterogencous environments. In International Conference on Machine Learning. PMLR, 2024a.
Han Wang, Aritra Mitra, Hamed Hassani, George J. Pappas, and James Anderson. Federated TD learning with linear function approximation under environmental heterogeneity. Transactions on Machine Learning Research, 2024b. ISSN 2835-8856. URL https://openrevieee het//form?ieehhddegytt
Jianyu Wang and Gauri Joshi. Cooperative sgd: unified framework for the design and analysis of local- update sgd algorithms. The Journal of Machine Learning Research, 22(1) (2)179970 -9970999778,
9758, 2021.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Informalion. Processing Systems. 33, 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 7611-7623, 2020b.
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global opti- mality and rates of convergence. aivi preprint arxi:19999.0133 2019.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, pp. 5 5-32, 1992.
Jüin Woo, Gauri Joshi, and Yuejie Chi. The blessing of heterogeneity in lederated Q-learning: Linear speedup and beyond. In International Conference on Machine Learning, pp. 37157-377 37216. PMLR, 2023.
Zhijie Xie and Shenghui Song. Fedkl: Tackling data heterogeneity in federated reinforcement learning by penalizing kl divergence. IEEE Journal Selected Areas in Commundcations, 41(4):1227-1242, 2023.
Lintao Ye, Aritra Mitra, and Vijay Gupta. On the convergence of policy gradient for designing a linear quadratic regulator by leveraging a proxy system. In 2024 IEEE 63rd Conference on Decision and Control (CDC), pp. 6016 6021. IEEE, 2024.
Shuai Yu, Xu Chen, Zhi Zhou, Xiaowen Gong, and Di Wu. When deep reinforcement learning meets federated learning: Intelligent multitimescale resource management for multiaccess edge computing in 5g ultradense network. IEEE Internet of Things Journal, 8(4):2238 2251, 2020.
Chenyu Zhang, Han Wang, Aritra Mitra, and James Anderson. Finite-time analysis of on-policy heteroge- neous federated reinforcement learning, In The Twelfth International Conference on Learning Represen- tations, 2024. URL attpss//oppenevieet met/forru/idddDeev@PP...
Haochen Zhang, Zhong Zheng, and Lingzhou Xue. Gap-Dependent Bounds for Federated Q-learning. arXiv preprint ar X5v: 02859, 2025.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321-384 2021.
Feiran Zhao, Xingyun Fu, and Keyou You. On the sample complexity of stabilizing linear systems via policy gradient methods. ar preprint drXiw:2205.14335 2022.
Feiran Zhao, Xingyu Sha, and Keyou You. Asynchronous parallel policy gradient methods for the linear quadratic regulator. IEEE Transactions on Automatic Control, 2025.
Feng Zhu, Robert W Heath, and Aritra Mitra. Towards fast rates for federated and multi-task reinforcement learning. In 2024 IEEE 63rd Conference on Decision ond Control (CDC), pp. 2658-2663. IEEE, 2024.
Daniel M Ziegler, Nisan Stiemnon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- liano, and GeofTrey Irving. Fine-tuning language models from human preferences. ar preprinl arXiv:1909.0885 2019.
23

---

Published in Transactions on Machine Learning Research (02/2026)
A Appendix
A.1 Appendix Roadmap
This appendix is organized as follows. Sections A.2 and A.3 present auxiliary norm inequalities and lemmas that play key role in proving the main results of this paper. The proof of our main results in the model- based setting is provided in Section A.4, while Section A.6 is dedicated to the corresponding results in the model-frec sctting. Additional details on the zeroth-order optimization method are provided in Section A.5.
A.1.1 Notation Recap
For convenience we briefly recap and summarize our notation. We use ||]] denote the maximum spectral norm taken over the family of matrices ..... (()....,((M))) All norms for matrices and vectors are spectral and Euclidean respectively, unless otherwise stated. The integer sequence ,N, is denoted as [N]. The spectral radius of square matrix is denoted by p(-).
Table Notations
Symbol
Meaning
M
number of systems
L
number of local updates (counter: I)
N
number of rounds of averaging (counter: m)
Kn
averaged controller al round n
K;
optimal controller for system (A(1), B(i)
K
controller for system after / local iterations and n averaging rounds
A.2 Useful Norm Inequalities
Given any two matrices A, B of the same dimensions, for any & 0, we have
||A + BI < (1 +) + (1+ |
(6)
Given any two matrices A, B of the same dimensions, for any E > 0, we have
(A, B) < |
2 BI
(7)
This inequality goes by the name of Young's inequality.
Given m matrices Ai...., An of the same dimensions, the following is simple application of Jensen's 's inequality:
Σ
A
< m
||A, 2
1=1
m
m
Σ
A
< m
A| 1
(8)
P
24

---

Published in Transactions on Machine Learning Research (02/2026)
Given any two vectors 2, y R for any constant <> 0, we have
| + vll I (1 + 112111 + 1 () 11v11?
(9)
Given any two vectors 2; y E R", for any constant Ç > O, we have
S |110|1 + |lyl1? (x.y) -
1 2 2
(10)
A.3 Useful Lemmas and Constants
Lemma 5. For each E [M], we have that:
(C (C((K)
(K)(K)
||
(11)
O miin (Q)
p
Proof. The proof of this lemma is detailed in (Fazel et al., 2018, Lemma 13).
Lemma (Uniform bounds for (K)(K) and [|]]] For each agent i [M), the gradient (C( and ||]] can be bounded as follows:
[V) s IVC S (i(K) and |||]|| S h2(
where hu (K), and (K) are some positive scalars depending on the function C(K).
Proof. In this lemma, h2()), and (2( are the functions defined as:
| ll max (Cmux (K) Cinin (K))
ho(K):= ==
h2(K): = Canax (M)ax()))o('))
h2(K): ho(K) + +IIBT PK max
Tmin mmin(Q)
J (R)
where == max;
R + B(O) ((0) BGi
By using (Fazel el al., 2018, Lemma 13), we have
[K)] < Tr (20) (T) 2 < |>
Tr E E
(K)
Tr
E (
min (Q)
By (Fazel et al., 2018, Lemma 11), obtain
R BGT p(0)
P B
(C (6) (K;))
Tr (ER E
which proves the first claim:
CO((K)
R B p(t) B())
(CU(() (( (K;))
(CC(()) <
mmin (Q)
R+ B (()) B(O)
(Cmax (K) Cmin (K))
Cmax K)
max
Guin (Q)
25

---

Published in Transactions on Machine Learning Research (02/2026)
On the other hand, by exploiting (Fazel et al., 2018, Lemma 11) can also write
R B p
B(o)
|| R B P B(o) K
1
|| [R (()) pk B(o)) ||
minu (R)
(1( R B(i) T p(i B(i) - B(O p(ù 100)
B p ((5))
m R)
B
p(i) A K
miin (R)
mmnn (R)
T ER Ir L
B p A
Omiul (R)
minu R
(C)) - C(i) (Kz))
+ B ( K
B
B plo A
/ (R)
min (R)
which completes the proof for the second claim.
It is worth noting that the local cost and gradient smoothness in Lemma 1 not only hold for the single- agent setting but also hold for the multi-agent setting. Moreover, will make use of the following matrix concentration incquality.
Definition (Matrix Martingale). Consider random process (Y k k= 0, with matrices Ys of fonile dimension, we say that the process is martingale if Y, O, E Y and B|| oo, where E is the expectation conditioned on filtration Fk-1-
Lemma 7. (Rectangular Matriz Freedman (Tropp, 2011)). Consider a matrix martingale (Yx k- = 0,1,2.... whose values are matrices with dimension d1 x d2, and denote the difference sequence as (Xg- = Yv Yk-1 k 1,2.3....J Assume that the difference sequence is uniformly bounded:
| S R almost surely for k 1,2,3.....
Define two predictable quadratic variation processes for this martingale:
k E5-1 (XyX3)
Wool co =
and
j=1
W Wrow &
D
E5-1 (X}x,)
for k 1,2,3....
5=1
Then, for all 2 0 and o² > O,
-22/2
P 3k
Y& t and max (l]Woo WWl alll s o²] (d1 + dz) exp
02 RR/33/
A.3.1 Proof of Lemma
Proof. In this proof, we aim to show
(009 - c(0)
S Neaet (aonn(K)) (K)l K]],
VCO (K') - VCl0 (x))
$ Agrad (K)] and
VCl (K') - (C)(())
S &ggadd (K]|A|
26

---

Published in Transactions on Machine Learning Research (02/2026)
hold for all agents i [M], and K' satisfying | - K|| S ha(K) <0.. defined as
The term ha( is the polynomial
Gwin (Q)
ha(K)= B| max Caa (K) (IIA BK] max + +1)
the term heost (K) and hggaad (K) are defined
[11X1]+ (k( hoost (K) = 4 Tr (Eo) maax (K)
Cinax (K) Jo (Q)
+ | max ||| (114 BK mmax
mu0 (Q)
2
Cmax (K)
Bggaad (K) =44
[| + || (IIA||max + |1 (1] + ha(K)))
mmnn (Q)
heust (K) mmuxx (K)
Cmuxx (K)
+ || |
Tr (Eo)
max
1
Cmax (K)
2
max (IIA BK BKIlmax + 1) ho(K).
+8
Trin (Q)
1
For the single-agent (i.e., M = 1) setting, the proof is explained in detail in (Fazel et al., 2018, Lemma 24 and 25). For the multi-agent. setting (i.e., M > 1), we can complete the proof by taking the maximum over the clients i E [M] of all the system-dependent parameters, such as [B]] max-
A.3.2 Proof of Lemma 2
Proof. For the single-agent (i.e., M 1) setting, the proof is explained in (Fazel et al., 2018, Lemma 11). For the mulli-agent setting (i.e., M 1), it is easy to see that
Ex
max; Σx; (CC)(K)
(K) (K?)
(K)LE
4y.2omin R
4112 A (R)
holds for any stabilizing controller and any agent i [M].
A.4 The model-based setting
We first introduce the following operators on a symmetric matrix X,
B(0 T ( (X) = - KY' (4 B -B00 {))KK T]
- (X) = (A B0 BO -
(12)
We also define the induced norms of T and F as
[xx(X]] |||||
[x
TK sup X
|| sup
|||
Lemma 8. When (AtO - B(0) K) has spectral radius smaller than 1, we have
T = (I-
holds for each i E [M].
Proof. The proof is detailed in (Fazel et al., 2018, Lemma 18).
27

---

Published in Transactions on Machine Learning Research (02/2026)
Lemma 9. If 16
T
k
F
K
-
F2
2 !
(13)
holds for any system i.j E [M]] then we have
( - (x))
(x)
7x
10
|
F
(x)
F/22 -/
Proof. Define A 1- 1- and B = = - In this case A -xx² and (A B)- = =Tx Hence, the
condition T (
F
Fk
S 1
translates the condition ||A-1|| | s 1 2"
2
First, observe that
-A-1-(A4 (A- (x) (- (1-
() (4 (x)
= (1- (Tx (x),
(14)
where f g denotes the composition fg(x))) Since
(1 -1- (C-A-1oB) = -1-
we have:
[1-- o B)
11 s1+ |A- B|| || (1-AA--BB)
S <1+ 2
1 1
[1
(15)
Now rearranging terms in Eq. (15), obtain | o |
2. Therefore, we have
|| -1- B3) A-1 o Bo (1-A-- B) -1
$ || |B8|| (1 B)
< 2 || ||8|,
and so
|| -1- . B) bB 2 |8| = 1 5k!
(16)
Then, we have
( - - (x)
(4 - (A-
(1 (1-A-1) 1)
(x)
(x)
2
0
2 - F |
where (a) is due to Eq4( and (b) is due to E1.) This completes the proof of Lemma 9.
16This Jemma has similar flavor that of Lemma 20 (Fazel al,, 2018). It worthwhile to mention that the inequality (13) imposes certain conditions on heterogeneity. Note that the constant 1 can be changed into any finite constant. Thus, this heterogeneity requirement can be subsumed by that in (2.(21).
28

---

Published in Transactions on Machine Learning Research (02/2026)
A.4.1
Proof of Lemma
Proof. First, we know that (C)) and (K) are given by,
VC 2E 10 > -k and (K)= -K
where,
E (R+ B p
B B BBOTTA)
and
(8)
K
2
D
1t 21
1=0
Thus, we can write,
(K]]]
= - -2 >
EK
|
I1).
+
91
B2
From Eq. (11) we can upper bound ||>*||| as follows:
c (K)
min (Q)
With the definition of BK RK + BOIT p B K - BGT pk) A(d,
A we can use triangle inequality to write,
- |R||| || + |
|||| [ K|| + | ()) P (11
[B
(C
S ||
([ (I)
where |1PK2 < (R)( from Eq. (11), with μ min (X
#
With the notation that introduced previously, write
Cmax R) muin (Q)
B1 = | S Σx maxx
and,
|| ||Exx|maxx S ||||||||||| + |max Cmax (K)
82 = 11 <
(1 max + ||A|/max),
where (m) == max, (K).
Next will derive upper bound for |al E
Upper bound for | EQ ||: We can first use the definition of E and EQ to write,
E -E(-) BUIT Pk (A B() B p(U (A(1) B((K)
-BB(OT pp2 (A()) -B B(ST ((() (A() B()) B()) pL0 (AU) BBOK)
BBOT (2(((A()) - B(OK) BBOTT pk3 (AU) - B(OK) + BBO) pk] (AC) -B()K).
29

---

Published in Transactions on Machine Learning Research (02/2020)
Then, by using triangle inequality, we obtain the following expression:
| - k || - B - K) - B B (AL)
Bb
11
|| B K (A B K) BBT (A()) BB) K)
H2
p B p (AC) - B(O K) BOTT p() (AU) - B(O K) 11
Incorporating the heterogeneity bounds from assumption 1 gives
|//l|| < || [|] [( + e2]
to which we apply the max-norm definition to arrive at
|| || max (el + + |maxx
(17)
Similarly, we can also derive upper bounds for || and || [|Hl] as follows,
||22|| < ||] Pk 111 BB) || max |IP pL0 I1IA - BK max
(18)
and
|aal| S e2 14 B(t) x|||||
S E2
BK]
max || Imaxx
(19)
To bound H2, need to derive an upper bound for |Ikk) pk2 For this purpose, have that for any fixed system i E [M]
1|PK
Th ( (Q
(R) The (Q+ RK)
K
Thus, by using Lemma 9, we can write,
||PK pk
||
Tx
k
- | [Q
where ||T (1)
|| <
(K)
Camax (K) Fmin(Q) (detailed in Lemma 17 of (Fazel et al., 2018)). With the following upper
umin (Q)
bound for
F K
F
|1 - -(2) (X0)| = 11(A() BB -BOK)) K) B K)
-AA - () K)
S 2(E1 + +=2|A(()||X| BKllaa
we have
Cmaxx (max(()) 2
|| 1 pk. <
Omin (Q) (E1 + KTD|IA - BK] mmx (1QQ1 + ||||||]
(20)
Plugging in Eq. (20) into He and adding the upper bounds of H1 (Eq. 17), H2 (Eq. 18) and H3 (Eq. 19) together, we have
| ERRI s E2,
30

---

Published in Transactions on Machine Learning Research (02/2026)
where 91 is linear in c1 E2 and polynomial in the remaining problem data. Specifically,
[|B]| maxe (max (K)
1 Cmax(() (IIA BK |K||ma)) (I1Q11 [[[111]]19) [[[])]]][]])]))]]]]]]]]}}}}}}}))
91 gi 62, K)= =
Omin' (Q)i
+ c2 [|| [ [
1 +
Cmaxx (K)
(11A- BK mmax) (I1@1I + ||]|+|
H
Omin mmin(Q)ps
+| BK] mmax)
In what follows, we will derive an upper bound for
|>
K
Upper bound for
| Σ ||: From the previous definitions in Eq.(12) and Lemma 9, we have,
|>
5 = 1 (Eo) - THe (Eo)l
s T 2
Fle - F2 || Coll
2 miin (Q)Hk
Cimxx
)
4
+2
BK| mmx Eool
(4) T
where Eg = E
Thus, we have the following upper bound for 11> =/ |1,
E.[''| S g2(e1, K) ||
with,
(1( Tmiu (Q){1) (4]IA - BK| max | [ool + <2l K||
CCmax (omax(K)
Cmax (K)
g2(e1, 6,, K) =
4 m QQ) (4|1 - BK mmax [Eo
Therefore, we can finally write an upper bound for ||VC(0(K) - VCV) which
[[VC(( vCV)(K)) S k2)
where,
f(e,, E2, K) K),
= (2, K) + 3292 (c, E2, K)).
After some rearrangement, we have that
f(1,, k)= = = hher (K) + e, heet (K),
where heet = his + has and a = has + has, and
B| max (Cmax K))²
Cmax (K)
hif
1 1
min (Q) (IIA BK 1mlxa) (11211 +|[[[||*]])))
mmin (Q)zz
h25 2
Cuaxx (K)
(4]1A BK |mxx EEo l),
14
mmin (Q)
has 2 |||||||| + ||B]| max Cmax (K)
max + ||A|| (14xmmmxx))))
| max C (K)
C Cmax (K)
x
1
(IIA - BK max)) (11211 + ||||||| ||||||
Miiu (Q)p²)
+1|1
BK||maxk
har = 8 (1/2]//11
|B]mmaxx Omax (K)
Cmax (K)
[||B] max + ||A||
|||||
1|A BK max [Sol
1
mmin (Q)11)
31

---

Published in Transactions on Machine Learning Research (02/2026)
A.4.2 Proof of Theorem 1
In this theorem, we consider the setting where (e hiht + e2l S hie with
mmn (R) (
Mhet
min ME[M]
(21)
4||Ex] min (nx, Nu,
Outline: To prove Theorem 1, we first introduce some lemmas: Lemma 10 establishes stability of the local policies; Lemma 11 provides the drift analysis; Lemma 12 quantifies the per-round progress of our FedLQR algorithm. As a result, we are able to present the iterative stability guarantees and convergence analysis of FedLQR in the model-based setting.
Lemma 10. (Stability of the local policies) Suppose Kn E go. If the local step-size satisfies mn S min LA } } and the heterogeneity level satisfies (ELh + egh S hhee then K E go holds for h1 ana
all i E [M] and I E [L].
Proof. Since Kn E go, based on the local Lipschitz property in Lomma 1, we have:
(k() (b. (Kn) (Kn) - ( K( Kn)) + Agraad (Kn)
(Ka) [ K_lli
s s-(vCú)(k,) vC() (k)) Agraa
m (VC()(
(22)
holds for any i,j E [M, |n [VV Sha S <ha(Kn); which holds when
n V.VC ( (nnn(K,)) s ndi v hA
where (a) comes from Lemma and (b) holds because of the requirement on in the statement of the lemma. We note that although hgraa (Kn) depends on Kn, our analysis is always restricted to the common stabilizing sublevel set (3),, on which, Agraa(K, (Kn) is bounded from above by some hgaad < oo (see (Φ) p)-local smoothness property in Malik et al. (2019)).
Following the analysis in Eq (22), have
(K(()) - (K)(Kn)) S s- (FC)))))))))))((
VCLO(K.))
m (ro)) (C.))
21
hgrad (Kn)
In VCC 12
(23)
2
Now Ti can be bounded as
Ti < m (C()(K))
VC VC (K.)
m / mm/miinnnn. ne
V0()((K.))
VC(l -VC()(K))
(n. n
VCU) (K,))
| +
(24)
where (c) is due to Lemma 3. Plugging in the upper bound of Ti into (22), we have:
cli (((()) - (k)(K,)) S -m ( (VC))(K)) vCli (C.)..
+ my / VOL) (aa + egThhee + Igradd (n))
m VKO)
32

---

Published in Transactions on Machine Learning Research (02/2026)
-m(voou) vol)(kn)) + m fn nu /atpe////////oeoo ( (s + t2gh
+ + (k))\/n (C (Kn))
- haraad (kraa((K))) (/ 11:
2 m (VC))(K.)) (Kn)) + TM min(uz, nun (CU) (E (ei + e22
+ M (K.) (Km)) /| m hgrad (K, min finffnn (nx, (cr c2hhor)
_2 -m comvvccl (Kn))(Kn), (06)(KK)) "
[[( || + m min min nu H (ri + c2 )²
+ np hggrad |oll(Kn) || + nh hgraad min(n:: nu nu (EL + e2hhict)
= --m -mm (mco()) (KO)( + O + ((hhraa)) VCU)
+ (m + hgrrad)) min na Heahteet + c2hheet)
(0) " (C (k))) + 2m min nu Mut(erhhor + E2 eghnet )
which implies
(k)( -60 - (K*) S
(h)
1 2n min (R)
(c60((K) -(()
Exg
+ 2m min nu Heither + t22h
(25)
where (d) is due to Eq. (8); (e) is due to Lemma 3; (f) is due to (q.(1 with ç= 1: (g) is due to the choice of step-size such that ní hirrad 2, which holds when m I Ilgead and (h) is due to Lemma 2 and the fact that Kn E go. If 6 and are small enough that
p2omin (R) (Ko) (K)(
her + c2hhhen22 < min
jE[M]
||Ex;| min nur
we have that
-c(((K))))))))) -C)
holds for any j E [M].
The above inequality implies [(] g as long as K,, E go. Then we can use the induction method to obtain that x(! go since [[] E go. As result, an identical argument can be used from K! to K Therefore, by repeating this step for L times, we have that all the local polices {(} E go holds for all i E [M] and I 1,... L, when the global policy Kn go
Lemma 11. (Drift term analysis) If Tn S min { {- the local policy and global policy can be bounded as follows Vi [M] and I E [Z]:
Thames 2- L(3h 1[3hoast and K₂ E go, the difference between
log
[((0 Kn 1| S 2m 2/
2n (K.)
7y
Proof. We have
(( Kn 1-
(() n Kn - VC
n.l-1
33

---

where, for (a), we used the fact that (1 + 2)7+1 < 1 - 2x(T + 1) holds for x < log2 In other words, log? Published in Transactions on Machine Learning Research (02/2026)
[ al-1 Kn 1= -2m
((VC
n.l- Kn))
| n, n,1-1 1.0-->11?
K mi-1 - Kn 1.= -211 [(VC - K n Kn)
2-
2m (< (VC()((K))
K - kn,)| + n |V 1111111)
- 10 - Kn
1-- +2m |C(0 (111) - (K()
2²
K K. - Kn
F
+ 2m VC (Km))
|1_1
[
1 K + 1/ 11VOO n. 11)
(0)
ni-1
0
Km
2
221thgrad (Ka)
kr(0)
-kn
k(()
n.-1
Kn
F
+ 71 VC
1,-
+ m | ni
Kn
1
+
(K 11-1)) 1p
F
5 (1 + 2 gradd (Kn) + m)) |
Kn
2 (m + 2n1²
VC
VC
1/2
+ 2m VC 10) (K(10 n11-1) VC(((KK)) 1|2
(6)
S (1 + 22piigraa [kn) + n/)) || | n.1-1
Kn 22
+ (n + 2nd \V
F
+ 2n nyrad (Kn) ( Kn 1
h2
F
(0)
2 (1 + 2m + m + 21/ grua)) || | p 1ii-1-np
Kn 1| + (n + 2n/))
(4)
1 (1 + 3ndigrad + n)
(<(0 n
Kn 1, + 2m +2ml|/ [VC))(
(26)
where (a) and (b) are due to Lemma 1; (c) is due to the fact that Kn E go; (d) is due to the choice of step-size such that 2n lirdd I miggrad and 2nd I m. which hold when m I minf 2hgrad 2). Therefore, we have
| | - Kn 1/ S (1 + 3 + m)) 1000
2
(Km)) 1
Kn
+
+m
F.
1-1
(1 + 3mpdgradd + =m)
Kn
1|2 +2 Σ
(1 + 3mp Tigaa + m)) m (0C((())))
1=
=0
(1 + 3rhhgrad + m)) - 2:11
( 2
+ 3rphgrad + 11) - 1
2 1 + 1(31p + m7) 1
((()
31ograd + 1
s 2 VC (Kr)) ||2
(1 + 3mthgrad + m)' < 1 1(3mhgara + m) holds when 3nphgrad + m S I i.c., when m S
lug2
]
-13hgma-1))
Lemma 12. (Per round progress) Suppose Kn Go. If we choose the local step-size
1 min 111 2
1 hA log2 2
[hhradd 2' hn L(3h gred + 1) 80 grad
34

---

Published in Transactions on Machine Learning Research (02/2026)
choose 1 = 1/2 minn h/ 111 1, 2 J, and the global slep-size as fg Imm
2 then, for all E [M, it holds that
3kyma
a (R)
c0
- -C(0 S
(Kn) - (c() + 3n min (nx; nu } (eahhet + eghhoet? (27)
[x: max
Proof.
(a) - S (K,), - Kn)) + Agraa
- -n 11?
2
(C6 ) Mgraa((nn)
VCLO
77
77
ML
MI
j=1
j=1 1=0
M
(Kn),
VC (K - (x())))
- - ( /|2
ML
1=0
hgraa((K))
7
M
I-11
VCU)(K (1) 2
2
MT.
j=1
1=0
--(VC
M
L-1
17
(
+ (CC)
(C() (VC(
ML
1=0
1 (K,)) 1|2
+ Agrad (Kn)
T
M
(
ML
j=1
M
L-1
v((
(CU((K - - 1|
ML
j=l
1=0
M
+ 7
Σ
(C(l (| (k,))
M
5=1
M L-1
VC (Km)) 1|2-
+ Agraad (Ka)
" ML
77
j=l
Agroad (aaa(Ki (Kn)
M L-1 ΣΣ 1=1
|1 - Kn 1111
m|
(000
F
ML
2
17
M
voll(Kk.) - vc(( -vCt ||=
17 1 [k,))
+
F
M
j=1
+ hgraad (Kn),
377
M
1-
vCO)(K))) - (KL))
2ML
1=0
3n 31 Mgradd (k,))
35
(Kx)
VK()(Kn)) - vC(((()))) +
(0(0((K))
+
2M
2
1=1
(4) 1 VC (K)))
+
nhg grad
x((0)
Kn
1
ML
F
7
(K)))
3n
hgradd
M
-
+
17
4
F
2
M
F
J=1
7
VC (Km))
+
3r7 grad
ΣΣ
k(())
Kn
|| (k(0
+ 7
2ML
1=
35

---

Published in Transactions on Machine Learning Research (02/2026)
(4) 3n (KO)(K))) |
hn grad
M
- -
+ ng M
(K,)l
+ 2n miy: nu (e + c2h 2
(e) 37
- 8
(Kn)
+
10 grad
Σ
M
VCl) -CC
Mg
10r
2 |/-
grad
(KO))
2n min The (erhhee + e2hheet
f1
7 VC+ 2 F
+ 3r min n. n. (e + e2Thhe)) 2
nu min (R)
(C (K,) - C (KG)) + ma ny Het + e2h
Σx;
In the above steps, (a) is due to the choice of step-size such that
M ΣΣ T S nh ha.
[k K = ML :=1
holds when 17 S As For (b), we use the Lipschitz property of the gradient (Lomma 1) in the first line, and use
h.
(10( with <= = 2 in the second line, and for the third and forth lines use Eq. (8); (c) is due to Lemma 1
and 3n) Agrnd < 3; (d) is due to Lemma Lemma 11 and the choice of step-size such that 3n Baah $ S 77
(e) is due to E8); and for (f), we use the fact that 10y" S E S n, which holds when VR S 80
We use the gradient domination property (Lemma 2) in the last equality.
With this lemma, we are now ready to provide the convergence guarantees for FedLQR in the model-based setting.
Proof of the iterative stability guarantees of FedLQR: Here we leverage the method of induction to prove FeaLQR's itcrative stability guarantecs. First, we start from an initial policy Kg E go. At round n, we assume Kn E G. According to Lemma 10, we can show that all the local policies K-(0) n, go. Furthermore,
by choosing the step-sizes properly in Lemma 12, we have that
(10(1)))) ((1) (Kn) < mp R (K() -(()) (KT))
Σx:
+ 3n mn nu HeIR + t22 )
for any E [M]. Then, for any fixed system i E [M], with (G + cahifee )2 < hieer we have that
-C(KK))))C((K)
< 1 1 mmiu (R)
mmia
-C())
|Ex;
+ 3/ mnin(nay nu (E + +
Mp mmiu R
-CO)) (K7))
|xx;
mpe min (R) (C(O((K) - ((,))
|EK;
< (1) (Ko)
(Ko) - ((,))
36

---

Published in Transactions on Machine Learning Research (02/2026)
With this, easily see that the global policy Kn+1 the next round n + 1 is also stabilizing, i.c., Kat1 E go by using the definition of go (Definition 2). Therefore, we can complete proving FedLQR's s iterative stability property by inductive reasoning.
Proof of FedLQR's convergence: From Eq-(27), we have
T/l m (R) -
- =
(Kn)
[xx;
31) min (Ray nur (Ei Aeet + 22htee)?
Under the assumptions in Lemma 12, FedLQR thus enjoys the following convergence guarantee after N rounds:
N (K)) - (K()()))
(KN) < 1 (1- 1714 Tll"mmin( ( P
|Xx;1
(nz, na ||E;
+
(el (e + 61 Eihhet)
pommia (
" -||Ex;
Thus, we finish the proof Theorem with Cuni.1 = 12 and Blei, 62) = apt (2)
(e + EI Eihheet?
A.4.3 Proof of Corollary 8.3
Proof. From Theorem 1, we have that
s 1 n/2 mmin(()) )
(K6) (KI))
|2x]
3 minn(ne not || (c + hiheett
+ p
holds for all i E [M]. Based on the fact that K; is system i's optimal LQR controller, i.e., (0( s C(O(K)) we have that
n/ (0)))) N
((K)))C((K)))
(Ko)
<
|2x=
min(nz, nut || |l
+
(EI + eghleet
(28)
12o (R)
We can finish the proof of Corollary by averaging Eq. (28) across all M systems.
37

---

Published in Transactions on Machine Learning Research (02/2026)
A.5 Zeroth order optimization
To prepare for the model-free setting where the controllers only have access to the system's trajectories, we first quickly recap the basic idea behind zeroth-order optimization. Say our goal is to minimize a loss function f(x), where x E R When one has access to exact deterministic gradients of this loss function via an oracle, the standard approach for minimization would be to query the gradient oracle at each iteration, and run gradient descent. Concretely, one would run the following iterative scheme: 2t+1 = 2t ' [ where ry is a suitably chosen Baamiing-rat/sstep--e history, there has also been growing interest in understanding the behavior of derivative-free (zeroth-order) methods that can only query function values, as opposed to the gradients. Two immediate reasons (among many) for studying zeroth-order optimization are as follows: (i) in practice, one may only have access to a black-box procedure that cannot evaluate gradients; and (ii) computing gradients might prove to be too patpaaationnll-xxppnsysys
While such first-order optimization schemes have a rich
Given two or more function evaluations, the basic idca behind zeroth-order algorithms is construct an estimate of the true gradient for evaluating and updating model parameters. For instance, a typical zeroth- order scheme with single-point function evaluation would take the following form (Polyak, 1987):
[f(xz + m.2) - f(xe)
Z++1 = Zt 17
He
In the expression above, {m3 is the cearning-rate sequence, { is a sequence typically choson in a way such that 1t + O, and u is a random vector distributed uniformly over the unit sphere. For details about the convergence of zeroth-order optimization algorithms such as the one above, we refer the interested reader to (Nestcrov & Spokoiny, 2017; Duchi et al., 2015; Bach & Perchet, 2016).
We now turn to brielly describing the model-free setup for our LQR problem. (Fazel et al., 2018) propose a zeroth-order-based algorithm BaaaahmnAAgggaa (Algorithm 1 in (Fazel et al., 2018)) to compute an estimation VC(K) and Σx for both (C(K) and Σx, for a given K. Algorithm in (Fazel et al., 2018) exploits a multiple- basod tcchnique that uses a Gaussian perturbed cost function (i.c., producing a Gaussian smoothing function) to estimate VC(K) from cost function perturbed values. That is, given the cost function C(K), we can define its perturbed function as,
(,) BUwB, IC + U)]
where IB, is the uniform distribution over all matrices with Frobenius norm at most and U is random matrix with proper dimension and generated from B,, For small the smooth cost CK) is a good approximation to the original cost C(K). Due to the Gaussian smoothing, the gradient has a particularly simple functional form (Gravell et al., 2020):
VC, = nn (00,,,, + U)U]. ,2
Therefore, this expression implies straightforward method to obtain an unbiased estimate of (K,, through obtaining the infinite-horizon rollouts. However, in practice, we can only obtain the fnite-horizon rollouts to approximate the gradient. Thanks to (Fazel et al., 2018), they showed that the approximation error of the exact gradient can be reduced to arbitrary accuracy if the number of sample trajectories n, and the length of each rollout 7 are sufficiently large, and the smoothing radius r is small enough.
We also emphasize that the zeroth-order estimator obtained from (K) is a biased approximation of the true gradient VC(K), with the bias arising from the smoothing step. The magnitude of this bias is controlled by the choice of the smoothing radius r: smaller values of r yield closer approximation to the original cost landscape at the cost of higher variance. In our analysis, make the estimation error sufficiently small by selecting r, the number of samples n,, and the rollout horizon 7 appropriately.
It is important to emphasize that our results can be readily extended to the two-point zeroth-order estima- tion scheme, where the estimation variance can be reduced, as discussed in Malik et al. (2019). However, addressing the two-point zeroth-order estimation is beyond the scope of this work, and we refer the reader to (Toso et al., 2024a) for an instantiation of the results in the asynchronous aggregation setting under the two-point gradient estimation scheme.
38

---

Published in Transactions on Machine Learning Research (02/2026)
A.6 The model-free setting
For notational brevity we rewrite (K) as (K) where
VCO(K)= = (K) 1
ReRa ((.)()) (K (K+ +U. U.O
n4 s=1
r²
and introduce two new gradient-basod terms:
V'C((() =
nx Bu ((.)i
K + U U.!.
n
p²
Nx
(K) =
nx (K3)
(K+ U(!) USO,
p2
where ((0,67)
K + +!!))
EE ti
(
Qui
(i)
wi
(8) Ruf")
with
2
(K
D (2)
((l((t) (K + +U!")
= FB ROOND ND Ei-1
(a!²T Qui + ui
(B)
(1)
()) (i)
Rut
and
c K (K+Ug (K+ = B= B,
( Q07 (1) + u =RTTRu
t=0
A.6.1 Auxiliary Lemmas
Lemma 13. (Approzimating (K) and EN with finite horizon) Suppose K is such that (K) is fonite. Define the finite horizon estimates,
(..0r) K
E
(1) ( 24 xz
and (K) =E E
x
(e)T Qx
(0)
+ u ( Ru:
for all systems i E [M] Now, let be arbitrarily small constant such that
n (C() (K))² n2. (Cmax (K))2
t 2 (e(e)) = max
(u(()min(Q)) M
Ep(a min (2)
such that
E> (T) - x01 SE.
If
T2 (11))) |
T 2 (2(() = max {E[M]
Ny (Cmax (1Q)) + +|| |||]|||| K113)
we have
(() Sc
where (K) max (K).
Proof. The proof for this lemma is detailed in (Fazel et al., 2018, Lemma 23).
39

---

Published in Transactions on Machine Learning Research (02/2026)
Lemma 14. (Estimating (K) with finitely mony infinite-horizon rollouts) Given arbitrary tolerance and probabilily 6, suppose the radius v salisfies
Cmax hooss
S hr ( min
ghored S
and the number of samples n, satisfies,
(2:0) 80 min min(nz nax)
nr + nl
n 2 Nsample
log
e²
5
o = (2 Craxx
+
(
+ hig)
Then with a high probability of at least 1 8, the estimate
(K) = 1 nl (1i)) (K + U2!0) U.!)
x2
satisfies
|[CC (K)]))) Sc
for any system e [M] and K go.
Proof. The proof for this lemma is detailed in Lemma B.6 of (Gravell et al., 2020). It is worthwhile to mention that, in (Gravell et al., 2020), the number of samples satisfies
80 min(n na))
Imin(n... na))
RSc
n2 + 12
n4 2
log
22
22
/ min (Nry ny))
8
T2
with R 2n Cauax + + h1 In the analysis throughout the paper, only keep the dominant term Ti in Nay since Ti is in the order O(e while T2 is in the order O(e-1).
By taking the maximum over K inside g", we make the local parameters become the global parameters, e.g., Caax == MEE (K).
Lemma 15. (Estimating VC() (K) with finitely many finite-horizon rollouts): Given an arbitrary tolerance and probability 6, suppose that the smoothing radius r satisfies,
hr (1) =min min { has Tresst 4h 4hgraed
Crux
and the brajectory length T salisfies
re
)
4n (Cmax (K))² (1121 ||| |||
T 2 h
requomin (Q)²
According to Assumption 1, the distribution of the initial states satisfies x D and 2
s H almost
(6) (i) surely. Thus, for any given realization 20,8 of xi and for any system E [M] have
T
2
(
To.e
S H,
0..
(6) 17 notation "0, denates s-th sample of the initial from i-th system.
40

---

Published in Transactions on Machine Learning Research (02/2026)
As result, the summation over the finite-time horizon
(T) (1) T F
Qu 4t
H Ru
S C(i)
(K UK))
Furthermore, suppose the namber of samples ns satisfies
n2 Asample,trune ( H22 =11
320 min(n nw)
nx + Tbu
log
1
22
8
where
of = ne H On
2
+ +
T/
then, with a high probability of at least 1 - 8, the estimated gradient
VC =
nz ((3),(T) K + U(0) US! U
p2
salisfies
(VC VV)0 sc
for any system e [M] and K go.
Proof. The proof for this lemma detailed Lemma B.7 of (Gravell al., 2020). As Lemma 14, we only keep the dominant term in the requirement of sample size ng. By taking the maximum over K inside go, all the local parameters inside the polynomials such as (5) become global parameters.
A.6.2 Proof of Lemma
Proof. For our subsequent analysis, we will use FY to denote the filtration that captures all the randomness up to the I-th local step in round n. We have
ML
M
(x.)
ML
ML
1-1
V'CO(K - ( F
ML
M I.
||-
MT
1=0
1s
Next, will bound Ti, T2, and Ts, respectively.
41

---

Published in Transactions on Machine Learning Research (02/2026)
Bounding T2: From the proof of Lemma B.7 in (Gravell et al., 2020), have
M T-1
- -
T
-
(29)
ML
holds long T 2 1r,
Bounding Ts: To precede, we bound T3 as
1 M
T3
Σ
-VO
ML
1 L-1
-
vol
ML
Blas term
M
(C) F
(30)
ML
1=0
Variance term 2
where cCC (K())
(C + [
= Bulg -B,
For the bias term 1 since the smoothing radius S hr (i), we have that
- VCO(K))) (
S hgraa ( (x)))r S hgradr I
(31)
4
For the variance term, simplicity, we denote
2 we will exploit the matrix Freedman inequality (Lemmn 7) to bound it For
c
①
1
vC.0 (x))
q a== Σ o
ML
Then, we have
M
1-11
- (k.)
Σ
ML
1=0
Next, aim to prove the following claims:
Claim I: Y = El er is a martingale w.r.t JEL1 for = 1, . and e = Em el is a martingale
(1)
difference sequence.
Note that E( = VC. tf ())) Then we can easily have Ejes] = 0 for = 0, L Fr-1] = Y. since Y, Y + + In other words, : Σl 2i=o e is martingale
Proof:
As result, we have B|,,
w.r.t Fh for t 1.... L 1.
Claim II:
E eteT
< ME where of is as defined in Lemma 14.
Proof: From Lemma B.7 in (Gravell et al., 2020), we can write
ot
E
(AT 4
FC_
2
6
FC
n, M
42

---

Published in Transactions on Machine Learning Research (02/2026)
and based this fact, we have
? )
M
JラL |
B aee
FL
B
Σ
( (
デー + Σ
(3)
Ti)T E
Fi-1
Tyy MT2
===0
where use the fact that T4 0 identical argument holds for E e
( because ej and el are independent, if we conditioned on FT. An
F
Define Wostt Ei=o E
e
Fi
and W Wrowe = Ei-o B e FL
then have
of
ot
[ S M,
I Wrow,el S
n,ML
Claim III: ||e/|| S Rs n,L
where R nn Cmax + 2 + h1.
Proof: From Lemma B.7 (Gravell al., 2020), we have |le, 11 S
R. ML
With this fact, we have
Rs n L
|eall -
i
With Claim I, II and Claim III and the matrix Freedman inequality (7), we have, for all 0,
of
P 3 2 0: Amax (Yi) 2 and max { (Woo all W .113 S
n. ngML
2/2
< (n= + nu)) exp
(32)
of ML
R 3n, L
+
Therefore, rephrasing (32), if
320 min (nz, nu)
32LR2 v min (Nay Ru))
M ML + n.))
11. 2
4
log
(33)
MLe
12Te
8
T
have that
- 1 MT. 11 ΣΣ
1
4 (k)) -
|| e
(34)
holds with probability 1- 8. As we discussed in Lemma 14, we only keep the dominant term T3 in the requirement of the sample size n, (as in E3))) Because T3 is in the order O(c-2) while Ta in the order note that by making
OEE-)). Moreover, note that, as an alternative to imposing the restriction O(1MM), one can apply the Freedman lemma (Lemma 7) to ML terms rather than to terms.
(L1MM), Ts when compared to T3 is negligible. We also
In summary, if n, 3202 min(n..nu)
ML(n+tne))
Asummple ( xfr
M L
(c(0 - VC. (k:1) S
2
=
(35)
ML
1=
43

---

Published in Transactions on Machine Learning Research (02/2026)
holds with probability 1 1 8.
As result, we have 73 & holds with probability 1 when r hy (9) and n, heample (i. follows, we will provide upper bound on the term Ti.
In what
Bounding Ti: We follow the same analysis of bounding 2 in T3 to bound Ti. Different from the filtration we define in analyzing 2 we need to define new filtration F where F JP U UP and UP {Uni I Note that U is the sigma-field generated by the randomness of all random sml,,,,
smoothing matrices U.() from all the systems at the n-th global iteration and I-th local iteration. Replacing of (33) with into of and Be with Re, we have that
M
(
c
T = ML
(36)
1=0
holds with probability 1 1 when
hampple trunc (a. ML
H²a
320 min (nz nu)
ML( + nu))
n1. 2
log
ML.²
MT.
Combing the upper bound of Ti (3g,(36)), T3 (Eq.(29)) and T3 (3q. and (35)), we have
M L-1
ML (CO((K))
(COO
S + T2 T
when the trajectory length 5 satisfies 2 h
the smoothing radius satisfics r S h, (4) and the
Inguu
size samples satisfies n, 2 max hamplectone we complete the proof of Lemma 4.
(1.7 #2
haample (i miz.
heample trone ( mT.. #²
Thus,
ML
ML
ML
A.6.3 Proof of Theorem 2
Outline: To prove Theorem 2, we first introduce some lemmas: Lemma 16 establishes stability of the local policies; Lemma 17 provides the drift analysis; Lemma 18 quantifies the per-round progress of our FedLQR algorithm. As result, we are able to present the iterative stability guarantees and convergence analysis of FedLQR in the model-free setting.
Lemma 16. (Slability of the local policies) Suppose Kn, Go and the heterogeneily level salisfies (e (erhhet e2h2 < hih where hi is defined in (21). If the local step-size m satisfies
hAN H (hi + Vē)
Mh S min
9h 9hgrad
the smoothing radius satisfies
minset (Ko)
}
V2
x min
.ss hr
hoost
the trajectory length satisfies t 2 hr Ang
and the reumber of the sample size satisfies
Ve H2 ''' ) hsample
Ve 8 }
n4e 2 max
hsampie,
18 Here the index denote thth sample. Note that each local iteration 1, we need generate the random smoothing matrices ny times.
44

---

Published in Transactions on Machine Learning Research (02/2026)
where we choose fixed error tolerance to be
3p omin (R) (Ko) -CO))((((K)))) ::= min JE[M]
5||Ex;| ||
then with probability 1 - 6,, where 5 E (0.1), K Go holds for all i E [M] and I = 0, 0, , 1.
Proof. For any i,j E [M], according to the local Lipschitz property in Lemma 1, we have that (K) (Kn) S (VCl)(Kn), [(] - Kn)
+ (Kn)
k(1 - K,ll/ (Local lipschitz)
2
= -- (VVO))(K)) ( + (grad(kn))
(K))) F²
2
holds if If (K)O hha S (K)). Note that this inequality holds when ni satisfies
n (KV
Ru (00,6)) (Knn + U1!) 1
=T]
11g
2
BeBu d(0) (Kn + U.-0) Us!)
11
118
i=1
MH2
11
H2
- (K)
m:112
(VC) + +
11
TH (In (I++Ve)
(37)
19 (a) is due to Lemma 15; according to Lemma 14, (b) holds with high probability, when the number The last inequality follows from the uniform upper gradient
of the sample size satisfies n, 2 hammpl
2: -)
bound in Lemma 6. Then we can easily conclude that |Im (, s hs holds when vn S
A
Following the analysis in Eq (22), we have
( (V)(Kn)) S 71 - mm (VCC (K.)
VCL)(Kn))
T
I (cl) - vc()
+ (grad(k.)) 2
n (kT
where Ti can be upper bounded as
T s VoC)
VCC (VO)(())
VCL) -
F
For sake of the notation, ignore the dependence the local iteration and global iteration n when index U/ in this part.
45

---

Published in Transactions on Machine Learning Research (02/2026)
S m V mnn na}
VCL) (eahhet + Eghnects
where we use the policy gradient heterogencity bound in Lemma 3 and the fact that K E go. We can bound T2 as follows
T3 S T (K,) VCli Vē,
F
s m
where it holds with probability 1 1-8. Here we use the Cauchy-Schwarz incquality in the first inoquality, and the second inequality is due Lemma 15 since n, 2 hsample,trune
vE 8, H² the smoothing radius satisfies
#
S hr vE and the length of trajectories satisfies T 2 hr
rvVE
Amznu
Plugging the upper bounds of Ti and T2 in Eq (23), we have:
-C0(
(Ui(K c(, (Kn) (1) 7
2
VCU) (Kn)
+ 171 V m nu
VCU)(K,) (G + t2gheet
V 3 3/grad(K.)) Vi
(C())(K)))
(C(0 -VC
+ 7
3h Angrad((
K m
VC.O())) VC (CCO((K)))
2
3/ 3/ (
(6)
-m (OW) || + m / /mninnz:nal nul (KW)
(ea + E2h1ae)
V 3hgraami
m VC()((K)) /|
Th min fag, nut (ei Whet + te2
+ 3hgradhh
(CC F
2
2
where (a) follows from Eq.(8)) (b) follows from the same reasoning as we bound Ti and To and the fact that
Kn E go If we choose the local step-size satisfies mn S Dharad i... shamand
<
2 we have
2
(U((( (K)(Kn) 2 -m (Ol
VCUi
2
+
F
2
VCU)((K))
+ 3m min nut (cathhot + o2 )²
m
31p
+ m 7
+
2
6
2
6
+ in min nad
(eI + (2ther)
)2+
11
(KL)(()))
||2
6
6
M VA
Th
1 + VCCL (Km)
5mp minfn (n. nud
(Ea + h +
3mk
3 E
2
3
2mp (R)
(K) + 5mp min ( (1z,, nuz
5mn E,
(E, ho + h2
|Σgş||
3
where (a) follows from the Young's inequality in (9); and (b) follows from the gradient domination in
Lemma 2.
Therefore, if the heterogeneity satisfies (c, Mhet + 12 )2 < T then we have 3 (R) (CU)(Ko) cU (K)
The + + < min
]EMM]
5 5||Eg] min ini((na. n
46

---

M Published in Transactions on Machine Learning Research (02/2026)
Since the error tolerance
3p min (R)
min JE[M]
5||_x;
we have
- (K))
1 (1
2111e² min (R)
(KL) -C())))(()))))
[x::
mpp mmin 2pp20min (R) (Ko)) - (K)(K
+ 2
amin (12) (Ko) (k)(()))
||Sx3]
||>x;]
(0)
1 2mp o (R)
- - + 20pu2 (Ko)
[xx;]
= (K)) - (K),
where we use the fact that Kn go in (a). The above inequality implies KM] E G with high probability when Kn E g Then we can use the induction method to obtain that K go, since KM E go.
By repeating this step for L times, we have that all the local polices K E go holds for all i E [M] and 1= 0,0,, ... L 1, when the global policy Kn goo
Lemma 17. (Drift term analysis) Suppose Kn E Go. 1f m S min { "hhgnss 1
the remmber of the I(3hgmet2) {
log2
sample size n, satisfies
1/ 5 H 4 T
somplle trunc
n1 2
ML
the smoothing radius satisfies r S hr
v and the length of trajectories satisfies
TIVE
h
given any 8 E (0, 1), the difference between the local policy and global policy can be bounded by
[ Kn 1|= 2. [ (CC (K₂) 1|2 M 27
[CC)
fg
holds, with probability 8, for all i [M] and / -0,1, 1. Proof.
x-(0 Kn 1
(1)
Kn - - (x
1
n.1-1
(K - Kn 1|- -2m
[< -ka)
+ | ( )
[ - Kn 1|2 |² - 22 ( - (x(()((())) ( k-(1) - -k,)
2m ((VC((( (1/10) (1))1))) - VCli( 1210 - KKn)| - 2m (VCC (0) K.,)
+ +1|10010 (1 2
47

---

Published in Transactions on Machine Learning Research (02/2026)
(0)
|
n1-1 - K2ll2² -Kol 2m - 1.
(1)
K - -Ka)
+ 2m VC (K 11 (KG)) /
1 n-111 1n.11 + 2m (K(()k)))) || 1
1 / (K (8)
112
+
n,1
(0)
2 - - Kn
|² F
-
x(( - - -Ka)
2011
+ 2nchgrad (nn) )
||
Kn
|||||1100
-Kul + m ml((((((km)) 1|- + '/| x -m|xx000 n,1-1 Kn 2
?
n.1-1
+ M (1) (K
1
(6)
(38)
where we use Cauchy schwarz inequality for (a); and for (b), we use Eq. (9).
Following the analysis in (3.) we have
[110 - Kn 12 p
2
S (1+ 2mthhrad (Kn) + +m))
n - Kn
- m (Km)
F
2m ((EC() (K() - 10(0((1)))) (1)--1)); ((1)
n.1-1 - K,,) 11
12
+ 1/
(0)
2 (1 + 211 grad ( (Kn) + m1) k2100 n Kn
+ m|| (KC()(K)))
2
F
+ 2m
V0 (K (6)
( VC
K n1 Kn |rl
+ + (C 11-11) -
VC (K 11)
||-
2
2
(C (x())
11,-11
(6)
2 (1 + 2m Agraad (nn) +m) [x(0] - K₂
2
(
+ m
F
F
+ m1 4c ( (1 (1)
2
2 (1) (0-11 - Kn
+ 711
F
+ 2m
(K -
VC
|
n.
F
+ 4
(C(0(( (1 VC (K₂) )
-
1/3
+ 4n
(C(0((m)) (Kn)
2
F.
F
(c)
3 (1 - 2m hgrad (Kn) + +m) |x100 - Kn
|,
2
+ (m + 4n/²
VC(0 2
+ m 4c ( (1) m1111) V (x(0)( (1)
(1)
|/
2
k(0 n.1-1 - Kn 2
1))
+ 171
F
+ 2// 4C
VC n. 11)
(1()
2
+
4n h ggrnaa ( (K₂))²
k-(i) n - Kr 12
1|
F
@ (1 + 2m /grae kn))= + 2m + 4n h hgrad ()
Kn
1
+ (m + 4n7) VC ( (Kn) (K,))
p,1-1
p
+ (m + m FC10
(1)1) VC(11) (3(1)
(()11))
|
F
+ (m + +m?)) VC 2
< (1 + 21 + 2m + 447 12 (
| 1
K
r
+ (m + 2n ( 10-1) VC
2
where we use Cauchy-Schwarz incquality and (6) for (a); for (b), we use (6. and (8); for (c), we use the gradient smoothness lemma in Lemma 1 and for (d), we use the fact that K₂ go.
48

---

Published in Transactions on Machine Learning Research (02/2026)
From Lemma 15, we can bound Tj term as follows:
T = (o00 (1 - VC R.1-1 $ MLe,
heample, trune
where it holds with probability 1- hr and the length of trajectorics satisfies 1
8, since n
the smoothing radius satisfies S
ML
rvC
Hence, we obtain
[0 - Kn
2
S (1 + 21phgrad + 2n + 4n graa)))
| Kn
+ (n + 4n/)
(Km)) 2
F
+ (n + 2m)) Me
(a)
(1 3mphgrad 2m)) [(() n1111
K2
1
F (C())(k))
2TH
+22
(1 3mphgrad
2m)
K
Kn
||
-0
+ 2m
Σ
(1 + 3mp
+ 2m1))
(C((((K)))
+Le
1=0
(1 + 31phhgrad
+ 2m
$ 2m
(C(0
+ MLC
(1 + 33phhgrd
+ 2(11)
(1 + 31phgrad + +2111)
2 (K,)))))))
1.- MLE
<2 x
3hgrad +
F
(b)
< x
1 + 1(3m1) l + 21/1) -
VC(0(())
1 + MLE
2
3h 3hgrad +2
1|- + ML
where (a) is due to the choice of local step-size which satisfies 2mphrrad + 2m + 4n7 S 331phgrad + 2m and 1
11 For (b), we used the fact that (1+x) I 1 2z 1) m7. + 2m I m7. + 4n < 2m,, i.c., m I min { Ahyendd is
holds for x < 1og2 In other words, (1 + 3mpggad + 2m)) = 1(3mihrrad + 2m)) when 3nihgrad + 2m S log
2
i... M7 Σ L(3h 22)
Lemma 18. (Per round progress) Suppose Kn, E go. If we choose the local step-size as
BAH
1 log
M11 = 2 min
1 (11 + Ve) 9h grad" 4' L(3h + 2) 256 grad
with slep-size = Langg-
minf
Ba H2 (Ve
1, 32h J. and the smoothing radius 20
minse[M] (Ko)
Ve
r S min
.hshhr hr
heost
4
min (Ko)
}.
The exact requirement of r r & min
Here, without loss generality, we This can be done because the ervor tolerance is usually small, and so
hA by
hr
heaet
drop the br
term from the min expression.
hr
1/ 4
<
hr
Vi
holds. The ussumptions on F and n, follow similarly.
49

---

Published in Transactions on Machine Learning Research (02/2026)
Tye where the trajectory length satisfies T 2 h [n
and the number of the sample size satisfies
V8 8 H²
Asample, trune
4 I
n2
ML
then with probability 1- 8, for any small (0, 1), the FedLQR algorithm provides the following convergence guaranlee:
(()) - 1 nu m (Rt)
(Kn)
(c() + 21/e
+ 2m the J (e2 (cahher h2hhet)
(39)
Proof. For any i E [M], according to the local Lipschitz property in Lemma 1, we have that
(VC) (Kn): Knt1 Kn) Airrad (K))
(K)(k))) ((0)
[kn+1
Kn 1i
2
= =- (V(C)))))))
M 1-1
vCC)( )
+ (gn))(Kn)
7
M
L-1
(CU)
12
(40)
ML
ML
F²
j=1
holds when MI Σك= Ei VCUI
Shs S hK(.)). Following the same analysis as E3.( this
inequality holds when
haH
[ C(+) (Ko)))
S
r min
hs J.
122 (h1 ve)
Reost
Following the analysis in Eq. we have
C² (Kn+1)) (Kn)
L-1 (KO)
M L-1
(Kn), ML
Σ
Σ
VCl
ML
1=0
1=
M
1-1
(Kn), 1
(CU)(K))) - vc(l((K))
ML
j=1
1=0
M
(Kn)
"
vCL)(K)) -VC00()))) - 1000000
M-
j=1
h Agrad (Kn) (Kn)
v
2
ML
(0)
2 1/
VO (Km))
(CW) -
M
(o - (kN))
VC
vol) - -VCk)))))()()) ||- - (K.)))
+
Agrad (Kn)
2
ML
50

---

Published in Transactions on Machine Learning Research (02/2026)
M
L- 1
7
(K()
/ + 1
1
Σ
(CO)( - -V
|
x1)
4
ML
j=1
1=0
1
VC
(Km))
10
1|2 +
2mhgrad 2kn)²
M 1-11
x((i
Kn 1
8
ML
n.1
j=1 1=0
"
VCl
VCUI Ch 1p
+
4
M
F
n (k.)))
?
Agradd ( kn)
7
M L-1
4ロしح (k())) 12 1=0
2
ML
F²
where (a) is due to Schwarz incquality; and (b) is due to Cauchy-Schwara Schwarz inequality and (q)
Moreover, we have
(c(0))
(6) "
[K.)
||2 +
1
(CO)(K - vC() -V())))))))) 1|
ML
+
VC
(K))))
(0)
1/: +
2nhhrraa ( 2
M L-1
x(0]
K. 1|3
8
ML
M
+ 4
(Km))
+
"
VCL)(K,) - (k() ||?
M
p
1=1
| (Kn)
1|2
+ hgrad (Kn)
L-1
ECli (H(
((1
2
2
ML
ne.!
j=1
1=0
(
31 (K()
2
+ +
grad
Σ
M
1/-
2
+ MLE
8
Mg
+ minnnn,, na hee + h2aa + Ayradd ( (Kn)
M L-1
(41)
2
ML
1=0
where (b) follows from the gradient Lipschitz property in Lemma 1; and (c) follows from the policy gradient heterogencity property in Lemma 3, Lemma 4 and Lemma 17.
Following the analysis in (41),,) we have
ClO (10(1)))) C0 (Kn)
(d) 37 (C(( || + me +
Σ
M
[||
(CW)())))) + ML
8
M
+ y min, nu ) (eather
+ anh2eet) )2
4s7 hiraad
M L-1
( - vC-0
1
2
ML
F
1=
4n
2 hrrad
Σ
M
L-1 (OW)((()) - VCh. ||
Σ
2
ML
#=1 1=0
4ny² 2 hggaad
Σ
MM
(CO)(K.)) - vCLk(k)))
/| + Baghar
2M
1=1
51

---

Published in Transactions on Machine Learning Research (02/2026)
- - 8
37 VC (K>)) + (m) + 2n/h ) + h grad
Σ
M
|| Vo (Kn)) 1| + +M
1=1
1|2
+ (17 + hraa)) min (nz; Mu (rather + +22l )2 + 217 hrand
VCC0
41 hgrad
1
M L-1
(x VCL) (Km))
+
ML
j=1
5
3n
| (Km)) /|2 + (17 + 2n hgrad)e
21j"hgrau hgraad
8
417² hn2Rarrad grad
M
Σ
+MLL] + (n7 + 211 minfn min(ng.naN(aiihh. c2hihet) )2
M
i=1
4 grad
ΣΣ
Kl] Kn 1,- F
2ML
1=0
3/
+ 211" T gradd
|VC10 (Kn))
2 + (17 + 21) 6
p
8
417 2 grad + 4n h2
grad
VCL)) (Kn) || MLc
M
nu + (17 + 2n7² 2ny"hgradl Ma Meahhet + +22
(42)
where (d) is due to (8); (c) is due to variance reduction property in Lemma 4 and policy gradient heterogencity in Lemma 3; (f) is due to gradient Lipschitz property in Lemma 1; (g) is due to drift term analysis in Lemma 17.
Continuing the analysis in (42), we have that
3m
(k)))))))))) 1|2 + (n (m++2 hgvaae
(10 (Kn) <
+ 2n7² haraa)
8
21 grad + +177 722
ggadd Σ
M
VC
(Km) (kn) ||- 2
+ +MC
+ (17 + hrna) min fry, nen Pe Heihhet + eiihhta)
(a)
311 21 highhd
((C0)())))) 1|2- + (n+ + 21 hgraade
2ny grad
+ 417 grad
M
2|
((()))))))) - (K))(K)) || +2 +2|
(k)))))) + MLE
ng M
1=1
+ (1) + 2n7 2 graa) min Mu nu Hei + +EIThee) )2
(b)
3m 2ry ggadd +
8T grad + TT grad
S
n9
41 2 grad + 4n 2h hraad
+
17 + 27²
hgraa
+
ML
ng
2 grad
+ 2 Th2
+
11 + 2n
Tigrad
+
grad
min(n nu nu Heihhet + e2
fl
(c)
2 VCt
P + 2ne + 2n min((p nu Kea + hh
52

---

Published in Transactions on Machine Learning Research (02/2026)
(d) nu omi R
(Kn) (K7) + 2nee + min nu (ea + egh
(43)
Ex;
where (a) is due to (q) ; (b) is duc to policy gradient heterogeneity in Lemma 3; and (c) is due to the choice of step-size such that 3y
8 hgrad + 81 t8y R
S and
8 grad + 81 112 grad
17 + hgrad
Σ 21,.
1l9
which holds when v S minf 32 1) and 1n1 S 256 grnd Lemma In conclusion, we have that
for (d) we use the gradient domination lemma
- <
1- nu mmi (1t) (m)n((t)
(c())(K)) (((( + 21/e
+ 2m mu (E2 hhet c2hiert
holds when the step-size, smoothing radius, trajectory length, and sample size satisfy the requirements mentioned above and those in Lemma 16 and Lemma 17.
With this lemmna, are now ready to provide the convergence guarantees for the FedLQR under the model- free setting.
Proof of the iterative stabilityy guarantees of FedLQR: Here, we leverage the method of induction to prove FedLQR's iterative stability guarantees. First, we start from an initial policy Ko E g", At round n, we assume Kn E go According to Lemma 16, we have that all the local policies K the hepot of in Lemma 18, we have that
E go. Furthermore, frame
(1 (ul"omin())) (K()) - (KT)) (C()))((((((K)) + 2n
(Kn+))
|
+ 2m fiminnnggin nu (E2 + e2h
Since (Ei hhet + +E2h < Ti
have
C())) 1 nu Tupaomin (R)
(C() (((() + 27e
q120min (R)
(c(i)(Ko) (Ko) -c())
|xx;
(Ko) ((,)),
where (a) follows from the fact that can be arbitrarily small by choosing a small smoothing radius, sufficient long trajectory length, and enough samples.
With this, we can easily have that the global policy Kn+1 at the next round + is also stabilizing, i.e., Knt1 E go. Therefore, we can finish proving FedLQR's iterative stability property by inductively reasoning.
Proof of FedLQR's convergence: From E3.(3 we have
-C(K)-))--(((3)))) s 1- mp2 minn R
(C (KT)) 2ne + 2m min(ne, ny (eihea <27
[xx;
53

---

Published in Transactions on Machine Learning Research (02/2026)
Using the above inequality recursively, FedLQR enjoys the following convergence guarantee after N rounds:
( = mp min (R)
N
2 2||2k] μ²² o (R)
(cO) (KT)) +
||>n; ;
2 min me nud |xx; (cI +2h
O (R)
) the smoothing radius satisfies S R1 (4) where
Suppose the trajectory length satisfics h
6' 4
[in (Ko)
( )
min
hss hr
4
Brost
and the number of the sample size of each agent n, satisfies
5 H²a (4
heample,trune
ML
T2g
MT.
Exz with a (R)
Caui,4 Exxe nu ommn (R)
log (k)) our FedLQR algorithm enjoys the
When the number of rounds following convergence guarantec:
n712 mulamin((t))
- -C6K =
-(0)))))))())) (K))) (Co) -
+ 2
[xx
2 min Nuy || (er e2hiher)?
11²t 1i20min (R)
mi nu |><x; ||
S é' +
(eit + e2h
122 (R)
which completes the proof.
54


Reviews:
[{"review_id": "Nf0aOGI6nz", "reviewer": "Reviewer_gAUU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies a federated reinforcement-learning setting where $M$ agents, each possessing a distinct but similar linear dynamical system, collaborate to learn a common linear-quadratic regulator (LQR) controller without revealing their raw data. The authors propose a new federated policy-gradient algorithm named $\\textbf{FedLQR}$. Each agent performs multiple local updates (using zeroth-order gradient estimates) and then transmits their local model to a central server, which averages them and re-broadcasts a global policy.  The proposed theoretical analysis focuses on three central questions:  (i) Does the common policy remain stabilizing for all systems at every iteration?  (ii) How close is the learned global policy to each agent’s own optimal policy? (iii) Can agents reduce sample complexity by collaborating?. The authors develop a complete finite-time analysis to address these questions. They show that under a low-heterogeneity regime,  $\\textbf{FedLQR}$ maintains stability across all systems, converges linearly to a neighborhood of each agent’s optimal controller, and achieves an $\\mathcal{O}(1/M)$ per-agent improvement in sample complexity in the model-free setting. \n\n**Strengths**\n-  The paper unifies federated learning and control theory in a principled manner. The heterogeneity issue in federated LQR has not been rigorously analyzed before; this is the first rigorous theoretical treatment of federated, model-free control under heterogeneity.\n\n- The theoretical results are fully justified. The proofs are structured into auxiliary lemmas that make them accessible despite their complexity.\n\n- The impossibility results in Section~5 convincingly establishes that the “low heterogeneity” condition is necessary and not an artifact of the analysis. The simple counterexamples illustrating the failure of stabilization under high heterogeneity are well-founded and perfectly illustrate the requirement of low heterogeneity.\n\n - The analysis links physical system heterogeneity to policy-gradient heterogeneity bounds. This bridge allows reusing tools from single-agent LQR analysis in the federated setting.\n\n-  Combining the variance-reduction lemma with the stability argument, the paper establishes the first provable linear-speedup in sample complexity ($\\mathcal{O}(1/M)$) for model-free control with heterogeneous systems. This is an impactful theoretical advance in federated reinforcement learning.\n\n-  The related work section does an excellent job of situating the paper within three relevant research areas: (i) Federated Learning, (ii) Policy-Gradient LQR, and (iii) Federated Reinforcement Learning. This clear positioning highlights how the paper bridges these distinct literatures and underscores its novelty\n\n**Weaknessess**\n\n-  Assumption~2 requires access to an initial controller $K_0$ that simultaneously stabilizes all systems. While this assumption is standard in theoretical LQR analyses, it might pose a practical obstacle in the heterogeneous, model-free setting. The paper only briefly refers to single-agent works capable of identifying such stabilizing policies, without discussing how such initialization might be achieved or approximated in the federated heterogeneous case.\n    \n- The experimental section is minimal and restricted to synthetic, low-dimensional (3-state, 3-input) systems. Although the setting is clean and controlled, its small scale limits the ability to assess the robustness and effectiveness of FedLQR under realistic conditions. A more comprehensive evaluation would substantially strengthen the empirical evidence.\n    \n- There appear to be inaccuracies in the mathematical statements. In particular, the definition of $\\epsilon'$ in Theorem 2 seems inconsistent: based on the first inequality on page 53, $\\epsilon'$ should likely be defined as\n\n  $$ \\epsilon' = \\sqrt{\\frac{16 \\| \\Sigma_{K_i^{\\star}} \\|}{\\mu^2 \\sigma_{\\min}(R)}\\,\\epsilon}.\n    $$\n\n  Furthermore, in Lemma 18, the bound involving the heterogeneity parameters appears to contain a typographical error. The term\n    $2\\eta \\min \\{ n_x, n_u \\} (\\epsilon_1 h_{het}^1 + \\epsilon_1 h_{het}^2)^2$\n    should instead read $2\\eta \\min \\{ n_x, n_u \\} (\\epsilon_1 h_{\\text{het}}^1 + \\epsilon_2 h_{\\text{het}}^2)^2.$\n    These corrections are minor but should be addressed.\n\n-  While the analysis clearly characterizes the heterogeneity-induced bias term, the practical impact of this bias on learning performance is not well explored. In particular, when heterogeneity increases, this term may significantly degrade convergence and stability. The authors do not discuss why variance-corrected or control-variate methods such as $\\texttt{Scaffold}$[1] are not employed to mitigate this effect, leaving open whether such approaches could reduce the bias without compromising stability guarantees.\n\n- The introduction of the universal constant $c_{\\text{uni},2}$ in Theorem~2 seems unnecessary, as its value is explicitly given as $2$. If this constant is fixed, it could be substituted directly to simplify the statement and improve readability.\n\n[1] Karimireddy et al, SCAFFOLD: Stochastic Controlled Averaging for Federated Learning, ICML", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The theoretical claims are carefully proven, with all assumptions stated explicitly and supported by intuition and examples. The numerical results qualitatively match the theoretical trends (improvement with larger $M$, degradation with increased heterogeneity). While empirical validation is limited, the theoretical development is sufficiently rigorous and complete to justify the main claims.", "audience_yn": "Yes", "audience_explanation": "Yes. The paper lies at the intersection of federated learning, reinforcement learning, and control theory, three areas of strong current interest to the TMLR community. The results advance our understanding of model-free control in the heterogeneous federated setting, providing novel mathematical tools that could inspire further work in both theory and applications.", "requested_changes": "-  The authors should justify how Assumption~2 can be satisfied in practice, for instance by extending or adapting the initialization techniques proposed in [1]. Alternatively, they could explicitly acknowledge that finding such a common stabilizing controller is non-trivial in heterogeneous, model-free settings and leave this as an open direction for future work.\n\n-  Address minor inconsistencies noted in Theorem 2 and Lemma 18 regarding the definition of $\\epsilon'$ and the heterogeneity term $(\\epsilon_1 h_{\\text{het}}^1 + \\epsilon_2 h_{\\text{het}}^2)^2$. \n\n- The bias term can greatly degrade learning. A well-known technique to learn a common optimal policy without suffering from bias is the SCAFFOLD-type method. Can the authors add a paragraph on why it is more advantageous to use their methods rather than SCAFFOLD-type methods?\n\n- Can the authors add the following missing references in federated reinforcement learning:\n(i)Zhang et al, 2025, Gap-Dependent Bounds for Federated Q-learning, ICML\n(ii) Labbi et a,l 2025, Federated UCBVI: Communication-efficient federated regret minimization with heterogeneous agents, AISTATS \n\n\n[1] Zhao et al, 2022,  Convergence and Sample Complexity of Policy Gradient Methods for Stabilizing Linear Systems , IEEE", "broader_impact_concerns": "There are no evident ethical or societal risks associated with this work, as it is primarily theoretical in nature.", "additional_comments": "Overall, I find the work technically sound, well written, and of clear interest to the TMLR audience; I therefore support its acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "b89Q6UC9Jy", "reviewer": "Reviewer_cDQa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes FedLQR, a federated learning framework for linear quadratic regulation (LQR) across multiple linear time-invariant systems with distinct yet similar dynamics. Building on the policy gradient (PG) approach to model-free control, the authors develop a federated variant that allows agents to collaboratively learn a common feedback policy while keeping their data private. The goal is to exploit inter-agent similarity to accelerate convergence and improve sample efficiency compared to learning individually. Theoretical analysis establishes conditions under which the federated policy remains stabilizing for all agents, quantifies its deviation from each agent’s optimal controller (in terms of the cost incurred), and proves a reduction in sample complexity proportional to the number of participating agents. \n\nThe article is very well-written and sufficiently motivated; the literature review for the state of the art has also been performed thoroughly. All theoretical results are clearly stated, along with the underlying assumptions and supporting results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in the contribution have been established rigorously through theoretical analysis with clear and sufficient proofs. The same has also been demonstrated through numerical experiments.", "audience_yn": "Yes", "audience_explanation": "The article will be of particular interest to researchers working at the intersection of optimization, machine learning, and control theory, as it connects federated learning methodologies with classical LQR control problems. Moreover, as highlighted in the motivation section, the work is also relevant to the robotics community, especially in scenarios involving multi-robot systems or distributed control, where agents share similar but not identical dynamics.", "requested_changes": "1. The convergence guarantees in Theorem 1 and Corollary 1 are expressed in terms of the cost difference relative to the optimal value. The results do not provide explicit statements about the behaviour of the optimizers. The authors briefly acknowledge this limitation after Corollary 1, but the explanation is vague — the phrase “...M systems are close” lacks formal clarity. It would be helpful if the paper explicitly clarified whether this “closeness” refers to the bounded heterogeneity defined in Definition 1 or to a stronger notion that can be derived from system-theoretic properties. A more precise statement would strengthen the interpretation of the convergence results.\n\n2. The authors establish via Propositions 1 and 2 that mere similarity in terms of small norm differences between system matrices does not necessarily ensure similar closed-loop behavior or the existence of a common stabilizing policy. However, this important insight appears relatively late in the paper. Since the question of whether “similar” systems can share a stabilizing policy naturally arises early, it would improve readability to mention this fact, perhaps in the Introduction or Motivation.\n\n3. The paper emphasizes that FedLQR achieves global convergence, in contrast to the first-order stationary point guarantees typically seen in federated learning literature. While this statement is valid for the single-agent LQR problem—where the gradient-dominant property ensures global optimality of policy gradient methods—it is less clear that this property holds once multiple heterogeneous systems are considered jointly. When the objective becomes the average of the individual agents’ costs, the overall function may no longer satisfy the same gradient domination structure. Therefore, is it correct to state that a gradient-based algorithm can reach the global optimiser? It seems that the suboptimality introduced by nonconvexity and heterogeneity is effectively absorbed into the residual “ball” term (whose radius depends on the heterogeneity parameters).\n\n4. The analysis relies on the existence of a common initial stabilizing controller that stabilizes all participating systems (Assumption 2). While this assumption is standard in LQR literature, it is quite strong in a federated setting involving multiple heterogeneous systems. It would be valuable if the authors could discuss practical approaches to obtain such a controller, both in model-based and model-free contexts.", "broader_impact_concerns": "No concerns related to ethical implications were identified. The work is theoretical and methodological in nature, with experiments conducted on synthetic dynamical systems. There is no indication of ethical risks such as misuse of data, privacy violations, or societal harms.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jptCdZTBlT", "reviewer": "Reviewer_i6JD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the Linear Quadratic Regulator (LQR) problem in a federated setting with M agents. Each of the agents have potentially different dynamics. The objective of the paper is to study the convergence under such heterogeneous conditions and to quantify the impact of heterogeneity on the final error and the rate of convergence. The paper studies this for the class policy gradient algorithms that aim to learn a common policy for all the agents.\n\nThe authors show that under both model-based and model-free approaches their proposed algorithm is stable and converges. The authors establish the rate of convergence and show that the final error has a bias term that depends on the heterogeneity among the models. They also demonstrate that under the model-free setting, their proposed approach offers a linear speedup in the error demonstrating the benefit of collaboration.\n\nLastly, the authors corroborate their theoretical results with empirical studies and show that their proposed algorithm performs well under synthetic settings.\n\nStrengths:\n- One of the key strengths of this paper is that it well-written. It was refreshing to see a nicely written paper. There are some minor typos here and there but nothing that a careful read can't fix.\n- The paper clearly demonstrates how heterogeneity affects convergence and its role in the final error bounds. This provides a clear understanding of the impact of heterogeneity and hence helps understand when collaboration can be beneficial for learning.\n\nWeakness: \n- I think this paper falls a bit short on the novelty front. Typically in a theory paper like this, the main contribution comes from either the algorithm or the analysis. Clearly, the algorithm is a standard FedAvg template. In terms of analysis, it relies heavily on existing results from Fazel et al 2018 and Gravell et al 2020 and techniques in FL algorithms that are largely standard by now. I am not saying that there is no novelty at all. The paper seems more applying some well-known techniques to a different problem setting. Also, at times the authors seemed to have overclaimed their contributions (I have outlined it in the next section).", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, I have gone through the proofs and they look correct to me (except for some minor comments outlined below).", "audience_yn": "Yes", "audience_explanation": "Yes, I think the paper is relevant to TMLR audience. It deals with Federated Learning in LQR systems and is potentially interesting to both the control and FL communities.", "requested_changes": "I will add both suggestions and questions that I have to this section. I feel all of them are critical at this point.\n\n1.  I am a bit confused about definition 2. From what I understand there has to be a $\\beta$ dependence in the definition. Either $\\mathcal{G}^{(i)}$ has to actually be defined as $\\mathcal{G}^{(i)}(\\beta)$, i.e., a function of a $\\beta$. Or the definition has to be amended to $\\exists$ a $\\beta > 0$ such that the condition holds. I believe the authors are going for the first option, but it is not totally clear from the text following the paragraph. I would suggest the authors to be explicit about it. I would refer to it as $\\mathcal{G}_0(\\beta)$ going forward. I believe the second option is not the correct way to define it.\n2.  The claim \"...which only guarantee convergence to stationary points, our work investigates whether FedLQR can find a globally optimal policy.\" is a bit misleading in my opinion. Assumption 2, which states that you have a \"good\" $K_0$ to start with is akin to assuming that start \"close enough\" to the global optimizer. The closeness is determined by $\\beta$. Furthermore, Lemma 2 is akin to the PL condition in optimization literature, under which convergence to global optimum has been discussed.\n3. How is \"sample randomness\" in your setting different from the combination of \"initial condition randomness\" and \"randomness from the smoothing matrices that show up in the gradient estimation process\". From what I understand, you refer \"sample randomness\" to the uncertainty in gradient estimates. In your case, the uncertainty of gradient estimate is just the uncertainty its estimation which is a combination of initial condition and Alg 2.  In other words, if you had an oracle that gave you a \"sample\" of your gradient, then your problem reduces to the \"sample randomness\" scenario. The only difference that now that your oracle is not a blackbox and you outline how it works and hence also characterize its uncertainty. Yet again, this seems to be a bit misleading, or at least overclaiming.\n4. It is important to specify that $\\hat{\\nabla}C(K)$ is a *biased* estimated of the true gradient and the bias comes from the smoothing steps. While the error bound that is adopted from (Gravell et al., 2020) takes care of the bias term, there is no mention of the same in this paper. This also lies at the heart of the choice of $r$, which has not been discussed much in the paper.\n5. Can you elaborate how Eq 22 comes about? I know it is standard bound when then gradient is $L$-Lipschitz. However, in such cases the Lipschitz constant is independent of $x$, the decision variable or argument of the function. That is no longer true for this particular case. The correct way to derive that would be to go by a Talyor series expansion, which reads as $f(x) = f(x_0) + f'(x_0)(x-x_0) + f''(x_1)(x- x_0)^2/2$, which $x_0$ is the point around which we perform the expansion and $x_1$ is some point between $x$ and $x_0$. Typically, since we have $x$ independent bound, we have $|f''(x_1)| \\leq L$ and we obtain the final result. However, in case how does this bound come about? It might possible to replace it with $\\overline{h}_{grad}$ but for that we need to ensure that the point at which the residual is calculated is within $\\mathcal{G}_0(\\beta)$. This is a relation that has been used multiple times so this question applies everywhere.\n6. The claim on page 32 that $K\\_{n,1}^{(i)} \\in \\mathcal{G}\\_0$ is again a bit ambiguous. In particular, the equation $C^{(j)}(K_{n,1}^{(i)}) - C^{(j)}(K_n) \\leq C^{(j)}(K_0) - C^{(j)}(K_j^*)$ implies that if $K\\_n \\in \\mathcal{G}\\_0(\\beta)$, then $K_{n,1}^{(i)} \\in \\mathcal{G}_0(\\beta + 1)$, i.e., the $\\beta$ changes. There are several points to be noted here. Firstly, this clearly shows why the definition in remark #1 is critical. Secondly, it is incorrect to keep allowing growing $\\beta$'s across iterations because you want the iterates to remain bounded in a certain, *fixed* set to complete your convergence arguments. With that said, it should not be difficult to fix this, as long as you invoke the correct value of $\\beta$ in Eqn 25. This will automatically force a bound on $\\beta$. This will circle back to the point I raised in remark #2 on how $\\beta$ affects the initial condition.\n7. Eqn 33 as currently written is incorrect. The term $T_6$, with $ML\\epsilon$ in the denominator, should have just $L\\epsilon$ in the denominator. There are two ways to fix this. First, you restrict $\\epsilon < 1/M$, which would yet again make $T_5$ the dominant term. Second, you carefully apply the Freedman to $ML$ terms instead of $L$, and you will have $M$ in the denominator in $T_6$ as well.\n8. Can the authors elaborate on the choice of $L$? I could not find any discussion on how it chosen or how it affects convergence or communication costs? Typically in FL, there is a limit on how large $L$ can be because it affects the drift terms.\n9. The final bound in Corollary 1 (and equivalently in the model-free case) is quite similar to that of strongly convex and smooth functions in optimization literature. Is it possible to obtain $\\mathcal{O}(1/N)$ convergence bounds with no bias terms, similar to the optimization literature. Such a result would definitely add value to this paper.", "broader_impact_concerns": "I think there is no particular need for a broader impact section for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)