FedECADO: A Dynamical System Model of Federated Learning
Authors: Aayushya Agarwal
Gauri Joshi
Lawrence Pileggi
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Gradient Flow
Equivalent Circuit Optimization
Primary area: optimization

Abstract: Federated learning harnesses the power of distributed optimization to train a unified machine learning model across separate clients. However, heterogeneous data distributions and computational workloads can lead to inconsistent updates and limit model performance. This work tackles these challenges by proposing FedECADO, a new algorithm inspired by a dynamical system representation of the federated learning process. FedECADO addresses non-IID data distribution through an aggregate sensitivity model that reflects the amount of data processed by each client. To tackle heterogeneous computing, we design a multi-rate integration method with adaptive step-size selections that synchronizes active client updates in continuous time. Compared to prominent techniques, including FedProx, FedExp, and FedNova, FedECADO achieves higher classification accuracies in numerous heterogeneous scenarios.

FedECADO: A Dynamical System Model of Federated Learning
Aayushya Agarwal
Gauri Joshi Larry Pileggi
Abstract
local clients. Neglecting the impact of heterogeneity can lead to inconsistent models with limited performance (Wangl et al. (020)) However, previous research has often studied federated learning in the context where local clients have identical learning rates and an IID) data distribution (Kairouz) ct al. 2021 Wang et al. (021). As a result, new techniques for aggregating local updates require careful consideration that synchronize heterogeneous client updates for model consistency while being computationally efficient without extensive hyperparameter tuning.
Federated learning harnesses the power of dis- tributed optimization to train a unified machine learning model across separate clients. However, heterogeneous data distributions and computa- tional workloads can lead to inconsistent updates and limit model performance. This work tack- les these challenges by proposing FedECADO, a new algorithm inspired by a dynamical system representation of the federated learning process. FedECADO addresses non-lID data distribution through an aggregate sensitivity model that re- flects the amount of data processed by each client. To tackle heterogeneous computing, we design a multi-rate integration method with adaptive step- size selections that synchronizes active client up- dates in continuous time. Compared to prominent techniques, including FedProx, FedExp. and Fed- Nova, FedECADO achieves higher classification accuracies in numerous heterogeneous scenarios.
In this work, we design a new consensus algorithm that addresses the challenges of non-lID data distribution and heterogeneous computing in federated learning. We design heuristics inspired by an equivalent circuit model of dis- tributed optimization (Agarwal and Pileggi (023) named ECADO (Equivalent Cireuit Approach for Distributed Op- timization) which represents the trajectory of the global model's state variables by an ordinary differential equa- tion (ODE) and whose steady state coincides with the local optimum. These methods draw inspiration from circuit sim- ulation decomposition methods to address key challenges in federated learning. Our main contributions are:
1. Introduction
Aggregate Sensitivity Model: A first-order sensitivity model of cach client update provides faster conver- gence in the consensus step and reflects the fraction of data for non-IID data distributions amongst clients.
Federated learning collaboratively trains machine learning models across distributed compute nodes, each equipped with a distinct local dataset and computational capabilities (McMahan et al. 2017; Kairouz et al., 2021). Employing techniques from distributed optimization, federated learning samples update from individual clients and calculates aggregate update to refine the global model. This paradigm yields several advantages, such as improving data privacy by keep data at the edge node, improving model generaliz- ability via collaboration with other nodes, and optimizing efficiency when training large datasets.
Multi-rate integration with adaptive step sizes: Lo- cal updates from clients with heterogeneous computa- tional capabilities are synchronized in continuous- time using a multi-rate numerical integration. Using proper- ties of numerical accuracy, we propose an aggregation method with adaptive step sizes for faster convergence.
The key novelty of our work is the introduction of circuit- inspired modeling and simulation principles to federated learning. FedECADO builds on the distribution framework (Agarwal and Pileggi 2023) and introduces new methodolo- gies (multi-rate integration and aggregate sensitivity model) to address challenges unique to federated settings including intermittent client availability, heterogeneous client com- putational capabilities, and non-lID data distributions. By reinterpreting these challenges through circuit theory, we are able to use well-estar establissee techniques from distributed circuit simulation to handle heterogeneity. This provides a
Nevertheless, compared to distributed optimization, feder- ated learning encounters unique challenges of non-IID data distributions and varying computation capacities amongst
'Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, USA. Correspondence to: Aayushya Agarwal <aayushya wwwrww.cm..cdus....
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FedECADO: A Dynamical System Model of Federated Learning
new perspective federated learning for robust and scalable training.
mulation of optimization is inspired by work on control sys-
tems (Behrman 1998, Attouch and Cominetti 1996 Polyak
and Shcherbakoy 2017; Wilson et al., 2021) and circuit
Compared to prominent methods including FedProx, Fed- Exp, FedRS and FedNova, our combined approach demon- strates faster convergence and achieves higher classification accuracies for training deep neural network models in nu- merous heterogeneous computing scenarios.
simulation principles (Agarwal et al., [023]] A Agarwal and Pileggi 2023) to design new optimization algorithms. This approach was previously applied to distributed optimization in ECADO (Agarwal and Pileggi 2023), but was not suit- able for federated learning because assumed full client participation. Our work specifically extends ECADO's ideas to address the distinct challenges posed by federated learn- ing. The key distinctions between our work and ECADO are the introduction of an aggregate sensitivity model to account for non-III data distributions among clients and a multi-rate integration mechanism to synchronize client updates from heterogeneous client computation.
2. Related Work
Federated learning has received significant attention in ad- dressing challenges related to heterogeneous computation and communication overhead. A comprehensive survey of these methods is available in (Zhu et al., 2021; Kairouz et al.] 202T Wang et al. 202T). We specifically focus on methods that target heterogeneous client computation and non-IID
3. Background on Circuit-Inspired Optimization
data distributions.
Federated learning methods traditionally rely on discrete iterative algorithms, such as FedAvg (Li et al. 2020) which uses SGD for client training and averages the updated results in each consensus step. While FedAvg offers theoretical guarantees, its performance often limited in settings with heterogeneous computation and non-Ill) data distribution (Zhao et al. 2018). To address these limitations, several modifications have been proposed, including adjustments to the SGD update using state information from the central agent (Acar et al.
In federated learning. n distributed edge devices collectively train global model. Each device, i, has local dataset, D;, and coordinates with the central server to update the param- eters of a global machine learning model, represented by a vector x. Due to communication and privacy constraints, the raw local data is not transferred to the central server; instead only model updates or gradients are shared.
Each client device trains localized model, where the local objective function fi(x) is the empirical risk function with respect to the local dataset D,, defined
2021: Pathak and Wainwright 2020 2020 Li et al. 2020) and a Newton-like Karimireddy et al.
update in FedDANE (Li et al. 2019a), One notable method, FedProx (Li et al. 2020), penalizes client updates deviating far from the central agent step. Other variations of FedProx
((x: fi(x) -e(x;;)
(I)
include the following (Acar et al. 2021 Li et al. 2019b).
SED.
where E is sample index and {(x; E) is the sample loss func- tion. The central server seeks to minimize global objective, which the sum of the local objectives:
Certain federated learning methods have improved conver- gence rates by introducing new step-size routines (Li et al. 2019b Malinovsky et al. 2023 Charles and Konečný 2020) or incorporating momentum into client updates (Das al.
n
2022 Xu and Huang 2022 Khanduri et al., (022)) Adaptive step size selections, as seen in Yogio FedADAM, SCAF-
mün f(x) where f(x) = B()).
(2)
FOLD Karimireddy et al. 2020) and FedAdaGrad (Reddi
ctt al. 2020), as well as FedExp (Jhunjhunwala et al. 2023
To tackle the challenges of heterogeneous computing and non-lID data distribution, we design a federated learning algorithm inspired by an equivalent eircuit (HC) model of the federated learning process. This EC model builds on the framework introduced in (Agarwal and Pileggi 2023). which employs circuit-based approach distributed opti- mization. In the EC approach, the solution trajectory of the global objective is analyzed as a continuous-ting ordinary differential equation (ODE), referred to as gradient flow:
have also been explored; however, these methods generally do not address heterogeneous computational challenges. A notable method, FedNova (Wang et al., 2020), specifically addresses heterogeneous computation and data scenarios in federated learning by modifying the gradient update to compensate for variations in client local computation.
Our work adopts a new continuous-time formulation of the federated learning process, where the challenges of hetero- geneous computation and non-lII data distributions are seen as updates occurring in parallel in continuous time. This enables us to design new methods based on concepts from dynamical system processes. Our continuous-time for-
x = V (x)
(3)
n
Vf((x), x(0) = Xx
(4)
2

---

FedECADO: A Dynamical System Model of Federated Learning
ODEs of the central agent and local agents according to
x 0
20100
O 1
Kirchhoff's current law (KCL) of the EC as follows:
(x1
(t) Σ1( (2)
(5)
(t) + x,(1) + Vfx( -0
(6)
01000
X2
Inspired by the current-voltage relationship of an inductor, the flow variables, Ig, couple the central agent to the local state variables according to:
(x2)
LI (t) - (c(t) -010-xxp1
(7)
L
The flow variable, I2 represents the cumulative error be- tween the central agent state, xg, and local state, X;, over the simulation window [0, 4). This acts an integral controller for the dynamical system and achieves second-order effect for faster convergence to steady-state (Agarwal and Pileggi 2023). At steady-state (i.e., critical point in the optimization function), the system reaches an equilibrium where the flow variables, I7, are stationary, indicating that x, = X; for all i. The settling time for the continuous-time response of the flow variables, I2, is influenced by the hyperparameter and can be tuned to provide fast convergence shown in AAgaraa and Pileggi 2023).
20000
X
(xn)
The federated learning process is modeled as an equivalent circuit, where node voltages represent state vari- ables, x,, and gradients, f. are voltage-controlle current sources. Using circuit insights, the gradient flow equations (4 are modified by introducing inductor (with an inductance of L.) between the central agent state, Xc, and the state of each sub-problem, x, The resulting gra- dient flow equations 5).6.77 are mapped to the equivalent circuit shown.
Figure 1:
The modified ODEs - describes a set of differential equations with the states of all local subproblems, x,, implic- itly coupled. In federated learning, the differential equations are solved over set of distributed compute nodes. To de- couple the eircuit equations (i.e., ODEs), an iterative Gauss- Seidel (G-S) method separates each subproblem from the central agent by treating the intermediate flow variable as constant from the prior iteration. Analyzing the distributed computation as a G-S enables us to study the continuous- time convergence of the full set of ODEs without the effect of discrete updates due to client participation.
where Xo are the initial conditions. At steady-state, the gradient flow 4 reaches a point where x = 0, implying fx() = as per 33. Thus, the steady-state aligns with a critical point of the objective function (2).
Directly solving the gradient flow equations, 3 to steady- state reveals similar convergence characteristies to those of SGD (Agarwal and Pileggi 2023). To improve conver- gence, new insights are derived by translating the ODE into an equivalent circuit (EC) model, motivating physics- based optimization techniques that can effectively address the challenges in heterogeneous computation. In the EC model depicted in Figure [ the node voltages correspond to the state variables for the central agent and local sub- problems, while the local gradients, (x;;), are repre- sented by voltage-controlled current sources. By applying principles from circuit analysis and simulation, we create cireuit-inspired algorithms that shape the solution trajectory and select appropriate step sizes. The connection between circuit analysis and the ODEs is provided in Appendix A
The G-S process separately solves each client independently and subsequently communicates updates the coupling vari- ables at each iteration. During the (A: + 1)-th iteration of G-S, a client first simulates its local sub-problem (i.e., gradient flow) as follows:
x++1(1) VI (1))+++++ (1)= 0,
(8)
where I² is the intermediate flow variable that couples the local client ODE to the central agent. In the G-S iteration, I is modeled as a constant from the previous G-S iteration (as indicated by I ). The differential equation is solved using a numerical integration method over a time window of (4o, 4)). For example, a Forward Euler integration (equivalent to gradient descent (Agarwal al. 2023) solves for the state at each discrete time point:
To separate the central agent state, x,, from the local state of each subproblem, X,, in the EC model, the gradient flow equations (3) are first modified by introducing an in- termediate flow variable (representing an inductor current in Figure 1 Ig. The flow variables, It, interact with the
x ++1 +1+(+A))) x x1)+(()) (()-A(VV((() ( (9)
3

---

FedECADO: A Dynamical System Model of Federated Learning
where At is the time step (or learning rate).
data distributions.
During each iteration of G-S, the local subproblem is simu- lated for number of time steps, corresponding to a number of iterations denoted as e₁. Afterwards, active clients com- municate their local states to the central agent. Using the local client updates, the central agent then updates the flow variables, Iř, and central agent state, Xe, according to the following ODE:
4.1. Aggregate Sensitivity Model
In federated learning, non-III data distributions can result in inconsistent models if not addressed during the consensus step. To represent the non-llD data distributions, we account for these variations in the following joint optimization with non-lID data distributions:
min f(x) x
(12)
x (t) = I (t)
(10)
1=1
f(x) Σ Pi)i(x)
(13)
LI (t) - x k+l (t) x (t).
(11)
where P. scales the contribution of f((x) to the overall ob- jective based on the size of local dataset, D;, relative to the total dataset size, D, defined as:
In this update, the state variables of each sub-problem is represented by a constant value, + which effectively models the sub-problem for a given time period.
p D.I
(14)
The entire circuit then progresses in time towards its natural steady-state. Through this perspective, we design new fed- erated learning algorithms that aims to efficiently simulate the circuit equations (10)([I) to the critical point of the objective function in the federated learning setting of hetero- geneous client computation and non-IID data distributions.
The gradient flow equations for the federated learning ob- jective with non-lID data distributions (13 is:
x x(1) P..V (x(((x(()),
x(0) Xo-
(15)
Using the modified gradient flow based on the EC in Figure the set of ODEs describing the circuit are:
4. FedECADO
The EC model depicted in Figure physical analogy to the underlying optimization problem. However, simulat- ing the EC model for federated learning to a steady-state encounters unique challenges due to heterogeneous client computation and non-IID datasets. While prior work (Agar wal and Pileggi 2023p leveraged insights from the EC model to address homogeneous distributed optimization problems, these approaches are not directly applicable to federated learning and are susceptible to model inconsistencies, as highlighted in (Wang et al., 2020). Specifically, (Agarwal and Pileggi 2023) simulates the circuit model assuming full client participation and uses a global step learning rate consistent for all clients and the central server. However, heterogeneous computation in federated learning creates asynchronous simulation timescales between clients because each client model is trained using a different learning rate and number of epochs.
X Σ1
(16)
IEC
(7,()) + (t) + + V (x; = 0
(17)
LI, (t) = x. x.()) (t).
(18)
where the local gradients are scaled by the relative dataset size, Ps. in (7).
However, the constant-value model of each client assumes that a change in the central agent state, x.. does not influence x; for the nime-period. We improve the consensus updates by modeling the first-order effect of each sub-problem due to changes in the -tatibbariable, Xc, using an aggregate sensitivity model of each client's 's subproblem, denoted by G This reflects the varying data distributions and the relative dataset size, pi, of each client in the consensus step. Incorporating the first-order sensitivity model has been proven improve the convergence of the G-S process as we can better capture the coupled interactions between clients and adds a proportional controller to the dynamical system to improve the convergence rate of G-S (Agarwal and Pileggi 2023 DDartu and Pileggi 1998).
We introduce FedECADO, a new algorithm that leverages circuit-based insights from the BC model in Figure [ to address the challenges posed by heterogeneous client com- putation and non-IID data distributions in federated learning. Our focus is on deriving the aggregation step for the cen- tral server, which processes updates from each active client based on (6). Our approach presents a multirate integra- tion method that synchronizes client updates to account for varying client computational capabilities. Additionally, we propose an aggregate sensitivity approach to model non-Ill)
The aggregate sensitivity model represents a circuit concept known as Thevenin impedance of a client sub-cireuit and defined for the EC in Figure I as follows:
G 0x,
(19)
4

---

FedECADO: A Dynamical System Model of Federated Learning
Using
we evaluate the sensitivity model, Gi as
4.2. Multi-Rate Integration for Heterogeneous Computation
a (x( + Ox
Gh Pt 0xi
x,,
(20)
The aggregate sensitivity model adjusts the ODE equations to accelerate the convergence of the Gauss-Seidel (G-S) pro- cess in (3 24). To address the challenges posed by hetero- geneous client computations, we introduce a technique that synchronizes and simulates the central agent ODEs (23)- 24). for efficient model convergence in federated learning.
The sensitivity model in (2) includes partial of a time- derivative term. Assuming a BE step for the ODE, we can numerically evaluate G/ as:
During each round of communication, a set of active clients transmit their most recent updates the central agent. The central agent then numerically solves the ODEs describing the dynamics of the central agent states (33 240.
1 Gh At +xxx)(()).
(21)
We apply
The derivation of Gh is provided in (Agarwal and Pileggi 2023). However, calculating the Hessian at each G-S itera- tion is bottleneck for computation and communication. To reduce the computation load of evaluating the Hessian, V² f. for individual datapoints, FedECADO introduces a constant aggregate sensitivity, G which is derived by averaging the Hessian across subset of datapoints. Employing con- stant value to approximate the Hessian, H V2 fi, defines constant sensitivity model for each client:
a Backward Euler (BE) integration step to solve for the states. The BE step is numerically stable and improves the convergence rate of the distributed optimization process (Agaraal and Pileggi 2023). The BE step solves for the states at time + At:
x (1+ (1+AA) At)
X (1) A Σ 1
(1+ (25)
++1 (t+ At) 12- u+1 (t)
1 = At
1
G
+ +
(22)
Al ++1 (t+ At) (I2 (t+ Gt) (26)
L
Gh extends the work in (Agarwal and Pileggi 2023) for the federated learning setting as captures the relative dataset size, Pe, within the first-order sensitivity. Notably, in fed- erated settings with non-IlI data, each client's 's local loss function exhibits a distinct landscape shaped by its data distribution. The Hessian captures the local curvature and is used to estimate the first-order sensitivity of a client's s model to changes in the global parameters. This enables the central agent anticipate how aggregation steps will influence individual client updates.
+x (1 At) 1 (t A0)O )).
However, the BE step in (66 assumes a globally syn- chronous timescale, where all clients are simulated with the same time step, At, for the same number of epochs. This assumption is not applicable to federated learning, where the subset of actively participating clients, C2 E C, exhibit a varying step-size, At, and number of epochs, é,.
FedECADO tackles this issue by introducing a multirate in- tegration method grounded in continuous perspective of federated learning. We recognize that in continuous time, each active client simulates its local ODE (6) for a unique time window, T;:
This sensitivity model can be periodically updated to re- assess each client's 's first-order response, balancing the trade- off between communication and computation. Intuitively, this approach suggests that a client with a larger local dataset will produce higher G thus having a greater influence on the central agent's 's state updates.
T
(27)
The linear sensitivity model is then incorporated into the G-S process as follows:
where Arf is the learning rate for the client i during an epoch k.
12
(t)
(23)
This insight builds upon the equivalence between discrete step-sizes and time steps, At,, resulting in each client es- sentially simulating its local sub-problen for e; time steps (i.e., number of epochs). For instance, a client with local learning rate of 10 and 3 epochs simulates its local ODE for T: = 3 x 10-3 seconds.
I = x --t1-(Tt t1 (1) +x (1) - (1) (1)=x
(24)
The relative values of each client's 's linear sensitivities drive the central agent states toward certain client updates.
The continuous timee perspective shows that each active client simulates its local ODE on a distinct timescale and
5

---

FedECADO: A Dynamical System Model of Federated Learning
communicates its final state, (T,),, to the central agent, leading asynchronous update. Figure 2 illustrates this issue of asynchronous updates from three active clients. Note, requiring synchronous timescale is vital for conver- gence, as all clients must reach steady state simultaneously.
operator 288 This operator addresses the challenges posed by asynchronous client updates, which can otherwise lead to model inconsistencies and poor performance. Without the operator, [(x; + (t), (t),+ the central agent would be forced to incorporate asynchronous client states directly, leading to disparate timescales within the coupled system. This would prevent the central agent and local clients from syn- chronously reaching steady state, which has been estab- lished by Remark 1 as necessary condition to converge to stationary point in the objective function.
Convergence to critical point for the central agent is achieved when all clients simultaneously reach a steady state.
Remark I:
From a continuous- point of view, Remark 1 illustrates the importance of maintaining uniform timescale with each sub-circuit to simultaneously achieve a global steady-
FedECADO solves for the central agent states in 29 30 using a numerically stable BE integration method as follows:
Inspired by asynchronous distributed circuit simulation (White and Sangiovanni- Vincentelll 2012), FedECADO introduces a multi-rate integration scheme designed to ad- dress asynchronous local updates in federated learning. This scheme effectively synchronizes the client updates to ensure accuracy and consistency of the central model.
n (T + At) (31)
x (r + 26) = x (T) A
L
At
I (T + At) = 12 (t) +
(x
k+l (T + At)
L
- -(Tz
k+1 (T - A/JG + [(x k+1 (0), T + At) (32)
During each communication round, the multi-rate integra- tion scheme begins by collecting the latest updates from all active clients, (T,) for all i E C.. along with each client's simulation runtime, denoted by T, Communicat- ing the simulation time of each client is essential for syn- chronizing local updates at the central client server and adds minimal computation and communication costs. Next, FedECADO solves for the central client states a syn- chronous timescale at intermediate timepoints. To synchro- nize the client updates. we employ a linear interpolation and extrapolation operator, [(x;(1)) (t), r), that estimates client states, (t), at an intermediate time point, T, defined as:
-I + AA) ))
This results in the following set of linear equations that determine the central agent states at the time-point, 7:
1 Arath
- +
7 ret+ (T+At)
0
1+ AlOM
- +
1/ k1 (T+A
0
1
(T+
AAt
AAt
- -rxf+* (1),))+) (na (1x)+(0))77 (1)gg
A4
L
((x+(1)=)) = (12) - x, (11)
(T - 41 + (41), (28)
t2 - t
(33)
Note, At represents the learning rate for the central agent and is independent from the client learning rate. To estab- lish the convergence properties of the multi-rate integration using the linear interpolation and extrapolation operator, T(.), we prove that each central agent step in ( is a contraction mapping that progressively moves the central agent states toward a stationary point.
where (t2) and x; (t)) represent known state values at time points 12 and f1, respectively.
This constructs a synchronous timescale for the central agent to evaluate its state variables over time window E [fo. to + max(T;)], where to is the latest time point in the previous communication round and max(T;) is the largest simulation time window amongst active clients. The central agent states are now governed by the following ODEs using the operator, r(-):
Theorem 4.1. The operator I'xx, r), defined in (28). syn-
chronizes local client updates and at each evaluation of the central agent states via the FedECADO consensus step in 33 is contruction mapping towards a stutionary point.
x xt+ Σ 1 (T) 1=1
(29)
The proof of Theorem 4.1 is provided in Appendix B
LIL (7)= x k+1 (T) - (I, (r)
1.2.1. SELECTING CENTRAL AGENT STEP-SIZE
r(x ++ (1),)) - - (
(30)
During each communication round, we solve the central agent ODEs 22) (30) using a BE integration. The BE in- tegration is a stable numerical method that approximates the central agent state at time points, 7 E [to, to + max(T:))
where [(x; k+ ++ (t), 7) calculatte the client states estimated at time using the linear interpolation and extrapolation
6

---

FedECADO: A Dynamical System Model of Federated Learning
X
Interpolated
Final state
Extrapolated state
t
T1
T2
T3
T1
T2
T3
Figure 2: Heterogeneous computation among three clients leads to simulation for different time windows (T1, T2. Tz). The final states (x1 (T x2 (72), (T3)) are communicated to the central agent, resulting in asynchronous updates.
Figure FedECADO proposes a multi-rate integration that evaluates the central agent step intermediate time points by linearly interpolating and extrapolating client states to the synchronized time point.
over a time-step, At. We propose adaptively selecting At using numerical accuracy properties of the BE integration.
Algorithm Adaptive Time Stepping Method Input: L > 0, > 0, Alo 0
The accuracy of the BE step for the central agent ODEs (29) and (30b) can be measured by local truncation error (LTE) derived in (Pillage 1998). The LTE for determining the central agent state, BB. from 29 is estimated as:
1: At + Alo
do while max([EBE]) S
3:
hi
4:
At
mar(IcBBll At
Compute x (T+ At), I2 (T using 33
k-1
At
n
(7)
Σ L (T
(t)
At)
6:
Evaluate EBE = |= Ggel using 34
BE
7:
Return At
(34)
The LTE of the BE integration step for evaluating the flow variables from (30). denoted E/g is estimated as:
5. Experiments
We evaluate FedECADO's performance by training multiple models distributed across multiple clients. The FedECADO workflow shown in Algorithm@iin Appendix We bench- mark our approach against established federated learning methods designed for heterogeneous computation, includ- ing FedProx (Li et al. 2020), FedNova (Wang et al.) (2020). FedExp (Jhunjhunwala et al. 2023). FedDecorr (Shi et al.) 2022) and FedRS (Li and Zhan 2021 Our experiments focus on two key challenges: non-IID data distribution and asynchronous client training. We then demonstrate the scal- ability of FedECADO on larger models with both non-lll) data distributions and asynchronous training in Section D In these scenarios, FedECADO achieves higher classifica- tion accuracy, thus demonstrating its efficacy for real-world federated learning applications.
=EE = 2L
At
[xx k (t) I #t1 (t)G +x, (t)+(())-
E (1)G - (x + (1 + At) I #+
(1+ (1)
+ (t+ + At) I (t At)G '11 (35)
where h is the sensitivity model derived in
To accurately capture the ODE trajectory, we adaptively select the time step to guarantee that the accuracy of the BE integration step in (32) remains within a specified tolerance, 8. At each iteration, a backtracking line-search style method (shown Algorithm[]] []) selects a step-size, At, ensure the following accuracy condition is satisfied:
ma S 5,
(36)
5.1. Non-1ID Data Distribution
where EB = [ Egl
We evaluate FedECADO's performance by training a VGG model (Simonyan and Zisserman 2014) on the non-IID CIFAR-I0 (Krizhevsky et al.] 2009) dataset distributed across 100 clients. To model realistic scenarios, we set an active participation ratio of 0.1, meaning only 10 clients actively participate in each communication round. The data distribution adheres to non-IID Dirichlet distribu- tion (Dir1 The specific dataset size, |D,], for each client is predetermined according to the Dirichlet distribu- tion before training and used to precalculate the average sensitivity model proposed in (2).. In these experiments,
The adaptive time step selection in Algorithm I is initiated by a time step, Ato > 0, which can be selected as a constant hyperparameter from the previous communication round. Then back-tracking line search adjusts At ensure that the LTE is bounded by 5. Note that convergence in continu- ous time guarantees that there exists At > 0 that satisfies the BE accuracy condition which ensures that the Algorithm is bounded (Agarwal and Pileggi 2023). Although Ato is a hyperparameter, it does not affect convergence but can influence the number of iterations in Algorithm 1

---

FedECADO: A Dynamical System Model of Federated Learning
Classification Acc. (%)
FedECADO
FedNova
FedProx
FedExp
FedDecorr
FedRS 46.1 (4.2)
Mean (Std.)
57.8 (3.6)
48.9 (2.9)
44.3 (3.2)
45.3 (4.7)
45.3 (4.7)
Table 1: Classification accuracies for training VGG-11 model on CIFAR-10 dataset distributed across 100 with Dirichlet data distribution for 100 epochs
Classification Acc. (%) Mean (Std.)
FedECADO 72.2 (3.8)
FedNova 67.6 (5.2)
FedProx 61.4 (4.8)
FedExp 64.6 (3.8)
FedDecorr 60.3 (5.1)
FedRS 69.3 (2.8)
Table 2: Classification accuracies for a VGG-11 model trained on CIFAR-10 dataset across 100 clients with each client learning rate and epochs set by
38
Classification Acc. (%)
FedECADO
FedNova
FedProx
FedExp
FedDecorr
FedRS
Mean (Std.)
81.3 (4.8)
69.5 (5.1)
70.6 (4.1)
68.6 (10.2)
61.3 (5.5)
72.3 (5.8)
Table 3: Classification accuracies for training ResNet34 model on CIFAR-100 dataset distributed across 100 clients with Dirichlet data distribution and random learning rates for 200 epochs.
the average sensitivity model is not updated during training.
classification accuracy. Note, due to the IID data distribu- tion, the improvement solely attributed to the multi-rate integration because Gth is identical for each client.
Using each method, we train for 100 epochs, examining the classification accuracy at each step. As illustrated in Figure f. FedECADO achieves the highest classification accuracy throughout the training process (with an improvement of 7% compared to FedNova and 13% compared to FedProx).
FedECADO's improvement is further demonstrated across multiple runs, where the learning rate and number of epochs are randomly selected according to (37p. 38 As shown in Table 2 FedECADO achieves a higher mean classification accuracy and the low variance indicates that it performs well across range of client settings.
This demonstrates the efficacy of its aggregate sensitivity model in adapting to data heterogeneity.
To test FedECADO's robustness, we repeat the experiment 20 times with random data partitioning sampled by Dirichlet distribution. Table shows the mean and standard deviation (std) of each methods' classification accuracies after 100 epochs. FedECADO exhibits the highest mean accuracy with low variance, demonstrating its effectiveness across diverse data distributions.
5.3. Scaling FedECADO for heterogeneous FL
To showcase the effectiveness of our method in heteroge- neous settings, we evaluate FedECADO on larger ResNet- 34 model trained on CIFAR-I00 dataset. In this setup, we study the efficacy of our methodology where the data is distributed according
non-IlI Dirichlet distribution and each client is assigned a random learning rate defined in (37) The result of these experiments are shown in Table 3 where FedECADO further demonstrate its scalability and efficacy in heterogeneous settings as compared to the state-of-the-art optimizers.
5.2. Asynchronous Computation
In this experiment, we evaluate the performance the multi- step integration proposed Section 4.2 We train the VGG- 11 model (Simonyan and Zisserman 2014) on a CIFAR-10 dataset (Krizhevsky et al. (2009) for 100 epochs across 100 clients with an IID data distribution. However, each client exhibits a different learning rate, lra, and number of epochs, e, whose values are sampled by uniform distribution:
Scaling to Other Datasets and Models: We demonstrate FedECADO's ability scale across additional datasets, including Sentiment140 and TinylmageNet, using diverse models such as ResNet-18, ResNet-34, and LSTM in a heterogeneous setting. Appendix D presents the results, comparing FedECADO against FedNova, FedProx, FedExp, FedDecorr, and FedRS.
Ir ~ U110= 10 10
(37)
C ~ U[1 10).
(38)
Figure 4p highlights the training loss and classification accuracy for a single random sample of ir; and using FedECADO, FedNova, and FedProx.
FedECADO achieves runtime compa- rable to the baseline methods, as shown in Appendix D
Runtime Analysis:
FedECADO's multi-rate integration synchronizes the up- dates of active clients at each communication round, result- ing faster convergence toward a steady-state and higher
Comparison with ECADO: FedECADO addresses the challenges of heterogeneous federated learning overlooked by ECADO (Agarwal and Pileggi 2023) and achieves
8

---

FedECADO: A Dynamical System Model of Federated Learning
higher classification accuracies shown in Appendix
Zachary Charles and Jakub Konečný. On the outsized im- portance of learning rates in local update methods. arXiv preprint arXiv:2007.00875 2020.
6. Conclusion
We introduce a new federated learning algorithm, FedECADO, inspired by a dynamical system of the under- lying optimization problem, which addresses the challenges of heterogeneous computation and non-III data distribution. To handle non-IID data distribution, FedECADO constructs an aggregate sensitivity model that is integrated into the central agent update for more accurate model adjustments. To address heterogeneous computation in federated learning, FedECADO employs a linear interpolation and extrapola- tion algorithm that synchronizes client updates at each com- munication round. The central model state is then evaluated using a new multi-rate integration, which adaptively selects step-sizes based on numerical accuracy, thus guaranteeing convergence to critical point. We demonstrate the effi- cacy of FedECADO through distributed training of multiple DNN models across diverse heterogeneous settings. Com- pared to prominent federated learning methods, FedECADO consistently achieves higher classification accuracies, under- scoring its effectiveness training distributed DNN models with varying client capabilities and data distributions.
Florentin Dartu and Lawrence T Pileggi. Teta: Transistor- level engine for timing analysis. In Proceedings of the 35th annual Design Automation Conference, pages 595- 598, 1998.
Rudrajit Das, Anish Acharya, Abolfazl Hashemi, Sujay Sanghavi, Inderjit S Dhillon, and Ufuk Topcu. Faster non- convex federated learning via global and local momentum. In Uncertainty in Artificial Intelligence, pages 196-506. PMLR, 2022.
Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapola- tion. arXiv preprint Stti::2301.06004 2023.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends in machine learning, 14(1-2):1-210, 2021.
Impact Statement
Sai Praneeth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for fed- erated learning. In International conference on machine learning, pages 5132-5143, PMLR, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Prashant Khanduri, Pranay Sharma, Haibo Yang, Mingyi Hong, Jia Liu, Ketan Rajawat, and Pramod Varshney. Stem: A stochastic two-sided momentum algorithm achieving near-optimal sample and communication com- plexities for federated learning. Advances in Neural In- formation Processing Systems, 34:6050-600 6061, 2021.
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regular- ization. arXiv preprint arXiv:2111.04263. 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Aayushya Agarwal and Larry Pileggi. An equivalent cir- cuit approach to distributed optimization. arXiv preprint arXiv:2305. 14607, 2023.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar San- jabi, Ameet Talwalkar, and Virginia Smithy. Feddane: federated newton-type method. In 2019 53rd Asilomar Conference Signals, Systems, and Computers, pages 1227-1231. IEFF, 2019a.
Aayushya Agarwal, Carmel Fiscko, Soummya Kar, Larry Pileggi, and Bruno Sinopoli. An equivalent circuit work- flow for unconstrained optimization, 2023.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar San- jabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-4500 2020.
Hedy Attouch and Roberto Cominetti. A dynamical ap- proach to convex minimization coupling approximation with the steepest descent method. Journal of Differential Equations, 128(2):519-55 540. 1996.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data, arXiv preprint 1xXiv:197..02/108 2019
William Behrman. An efficient gradient flow method for unconstrained optimization. stanford university, 1998.

---

FedECADO: A Dynamical System Model of Federated Learning
Xin-Chun Li and De-Chuan Zhan. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD conference knowledge discovery & data mining, pages 995-1005, 2021.
Ashia C Wilson, Ben Recht, and Michael Jordan. A lya- punoy analysis of accelerated methods in optimization. Journal of Machine Learning Research, 22(113):1-34, 2021.
An Xu and Heng Huang. Coordinating momenta for cross- silo federated learning. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 36, pages 8735- 8743, 2022.
Grigory Malinovsky, Konstantin Mishchenko, and Peter Richtárik. Server-side stepsizes and sampling without replacement provably help in federated optimization. In Proceedings of the 4th International Workshop on Dis- tributed Machine Learning, pages 85-104, 2023.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non- iid data. arXiv preprint arXiv:1806.00530 2018
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273- 1282. PMLR, 2017.
Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Feder- ated learning on non-iid data; A survey, Neurocomputing. 465:371-390, 2021.
Reese Pathak and Martin J Wainwright. Fedsplit: An algorithmic framework for fast federated optimization. Advances in neural information processing systems, 33:
7057-7066, 2020.
Lawrence Pillage. Electronic Circuit & System Simulation Methods (SRE). McGraw-Hill, Inc., 1998.
Boris Polyak and Pavel Sheherbakov. Lyapunov functions:
An optimization theory perspective. PaperOsnprnnnl.ine 50(1):7456-7411 2017. 2017.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garreut Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
Yujum Shi, Jian Liang, Wenqing Zhang, Vincent YF Tan, and Song Bai. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning.
arXiv preprint arXiv:2210.00220 2022.
Karen Simonyan and Andrew Zisserman. Very deep con- volutional networks for large-scale image recognition. arXiv preprint /409. /556, 2014.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Ad- vances in neural information processing systems, 33: 7611-7623, 2020.
Janyy Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen An- drew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint ar arXiv:2107.06917, 2021.
Jacob K White and Alberto L Sangiovanni- Vincentelli. Re- laxation techniques for the simulation of vlsi circuits. 2012.
Sangiovanni-Vinceeelll
10

---

FedECADO: A Dynamical System Model of Federated Learning
A. Background on Circuit Formulation
Federated learning trains a global model by aggregating updates from distributed clients, each training on local datasets. The update to the global model can be written as:
N
Ek+1
fxx))
(39)
where a is the learning rate and (1x) is the local gradient from client As a -> this update can be modeled as a continuous-time gradient flow:
(tt) Σ Vf((((t)).
(40)
This defines an ordinary differential equation (ODE) that assumes each client computes its local gradients using global state, (t). However, in federated settings, each client maintains its own local model state that is periodically communicated to the central agent. To address this, (Agarwal et al. 2023) introduced continuous-time dynamical model that couples global and client states via auxiliary flow vector, 1 (f), resulting in the following dynamical system:
C
d dl
+ (c(
Σ (1) = 0,
(41)
i=1
L.İ((1) = =x(()) - ri((), (1) = (1) v.((()))),
(42)
(43)
where (1) the global model, v1( is the state of client i, and the hyperparameter L, controls how strongly each client coupled to the central agent.
The dynamical system is modeled by ordinary differential equation (ODE) that models the continuous-ting evolution of the client and central agent state variables. These state variables converge to steady-state which coincides with the stationary point of the global objective in (39p. By viewing the federated learning setting an ODE, this reframes the federated learning method as simulating the ODE to its steady-state. However, in general, maintaining accurate and efficient
simulation of ODEs is challenging.
To efficiently simulate the ODEs to their steady-state, we adopt methods from cireuit simulation, which has demonstrated robust methodologies capable of scaling to billions of transistor devices. We model the dynamical system an electrical circuit in Figure !
In the equivalent circuit model, node voltages represent the model states at each component of the system: the global state, (.). is represented by the voltage at the central server node, while each client's local state, sr)()), is modeled by the voltage at a corresponding node. These nodes are connected using an electrical device known as an inductor to represent the flow of information between clients and the server.
The dynamics of each client captured by capacitor, electrical component that stores energy and resists sudden changes in voltage. In this context, the capacitor captures the continuous-time evolution of the client's model. The current-voltage behavior of the capacitor is governed by:
Ic (t),
(44)
where Ie the total current into the capacitor and d; is the rate of change of the client's state. This reflects how the client integrates incoming signals update its model parameters.
Each client node is then connected to the central node via inductor, component that resists changes in current and introduces momentum-like dynamics into the system. In this equivalent circuit model, the inductor is connected between the central agent node and a client node. Therefore, the voltage across each inductor is given by (t( - (t).,. representing the difference the client and global state-variables. The inductor then captures the accumulation of difference between the global and local states over time, and is represented by the following current-voltage relation:
(t) (x(()) T.TT((),,
(45)

---

FedECADO: A Dynamical System Model of Federated Learning
where L, is the inductance, and r (4) is the current flowing from the server client The inductor effectively damps the interaction between client and server, mitigating sharp transitions.
The behavior of the overall system is governed by Kirchhoff's s Current Law (KCL), fundamental principle in circuit theory that states that the total current entering node must equal the total current leaving it. Applying KCL to each node in the equivalent circuit, we observe that : (1) the current from the central server capacitor equals to the sum from all client branches sum, (2) at each client, the current from the inductor equals the sum of the client capacitor current and gradient- induucdd currents, faze). This correspondence between circuit behavior and federated optimization dynamics forms the foundation for our iimulation-driven approach.
B. Proof of Theorem 4.1
Proof. The convergence proof of FedECADO relies on the following assumptions for each local objective function, f(x).
Assumption 1. (Boundedness) C and infxeR" fxx > - RR.oe for some > O.
Assumption 2. (Coercive) f is coercive (i.e., liml f(x) = +20)
Assumption 3. (Lipschitz and bounded gradients): for all x, R". [Vf(x) vf(y)) S L|x vyll. and [V S B for some B > 0.
In order to analyze the convergence of the FedECADO consensus step (33) using the interpolation/extrapollttoo operator 28 we represent all the state variables of the central agent, including the central agent state x, and the flow variables I
as vector X = [T P. '. ". x.:l. The ODE for the central agent state in FedECADO, as defined by equations 31 and
can be generalized as follows:
(()) xx),
(46)
where g(X is defined
Σ 1 (mm) k++ (Tm) (tx, Tm)
g(X)
(47)
x
Here, Tm represents the discretized time point indexed by m. Furthermore, generalize the BE integration of the central
agent ODE 33
ALr(g(X>
(Ti), (T)) :.))..)))
(48)
where the operator, pl-), is defined as
+
Aage
0
Ac
0
1+
Aig
A
P=
L.
(49)
0
0
A
At
At
and (.), is defined as
- -I(xft (Ti), Tml + 111 (Tm) G&hh
-(x (T2)) Fm) 12 (Tm Gthh
(50)
Note, the operator, p(-), can be inverted to evaluate the central agent states:
x x Atp - (g(('(((( (T..). (T.).
(T)
[(x (Ti) Tm)), [(c (Ti). ..)).
(51)
The continuous-tin ODE of ij = The proof of convergence is provided in (Agarwal and Pileggi 2023). In this analysis, we study the multi-rate discretization of the ODE to ensure the Gauss-Seidel process of solving the coupled system converges toward the steady
converges to a stationary point characterized by x, = x; for all i E C and
12

---

FedECADO: A Dynamical System Model of Federated Learning
each iteration. This proof uses the analysis for multirate waveform relaxation for circuit simulation from (White
and Sangiovanni- Vincentelli 2012). This analysis is based proving that the central agent step is contraction mapping towards the steady state (i.e., the stationary point of the global objective function).
The multi-rate integration uses the linear operator, r(-), to interpolate and extrapolate state variables at intermediate time points. Two important properties of the linear operator are the following:
1. Given two signals, y() and z(1):
+ +
2. Given a signal yt),, and scalar,
=(ay(t),))=c((( cI(((t),
To prove convergence of the multi-rate integration step, we employ continuous-time 3 > 0 norm defined as:
Iyll m, 8t [ Vi E CC]
(52)
Under certain conditions, (51) is a contraction mapping on the norm. To prove this relation, we evaluate the difference between two series, {X* (mm) and (y* (Tm)), as follows:
(X k+1 (Tm)) - (y x+ (Tm)) Atp ( [(x] (Ti) Tm), (X))
-tp (T))) (Ti), [ (()) Tm),..) (53)
Exploiting the linearity the operators, (() and p. leads to the following:
{x (mm)} - (X x+1 (Tm)) }} = Atp "tpp'oll(X') (Tm),
(Tm)) r(x; 1 (Ti). Tm)) F(x (Tz) Tm))
-g(Y* Ply (Ti), Tm), T(y ( (T2), m.)...)) (54)
The BE operator p (-) can expanded into a series of summations, as shown Appendix B.1 using the following equation:
(X++1(Tm) - -(Y = At Σ vlo( (Ti), Tm mmm (Ti), T),...
g(Xx*((), (Ti) T), Iyy (Ti), r) ..)) (55)
where is scalar that determines the weight of the past state values in the numerical integration method.
To prove that the difference between the series is a contraction mapping, the following two lemmas are useful.
Lemma B.I. Given two sequences, (X()) (Y))) if (T.)) > (TT,) Vi, then T(X( > ((Y(T:)::) T). Further- more, if X (T.) K Vi where K constant value, then ((X( =
Lemma B.2. The norm the following series of X is bounded according to the following:
M
max
Tm-
1- 1-
81 [XX, mm)
(56)
[0.7]
1=0
where M equal max; Ihal..
Proof of the two lemmas is provided in Appendixes B.2 and B.3 Using Lemma B.l and the Lipschitz constant of V we can bound (55) as follows:
B.2
(xk+1 - -fyn) (y)+1(7))
hal
Σ
yk+ 5m-1)) + Σ
At, T(X} - Y² Fm-iill
1=0
jm1+1
(57)
13

---

FedECADO: A Dynamical System Model of Federated Learning
where Lig is the Lipschitz constant of the ith row of g with respect X,. Using the triangle inequality, we formulate this follows:
( S
Σ
AI Les
hal
- Tm 1) +
L
bill( yj
1)}-
1=0
j=++1
(58)
Multiplying both sides by e- and taking the maximum over the time window, [0, T). results in the following:
m
maxe (()+ (y* (7)) -
Al, Lag max e
hill hxllT(X
yj Tm Tm-11)
[0.T]
[077]
==1
m Σ 1
Σ
At,L max e 88
h - .m- (59)
10,27
J 1-
where T max(T.). Using Lemma
B.2
we conclude the following:
MAt, -AA
max [0.7]
36 (x (T))
Σ
Lig
1x
1l
1=1
MAL: BBr
Σ
Lij
|x - 1
(60)
j=i+1
where || is the 3 norm defined in (66).
Assuming all time steps, At; >> 0 are positive and a B > 0, then there exists scalar, 8 >> 0 such that:
MAt;
8
Σ
Lij.
(61)
BAT.
==
Using the definition of we conclude the following:
max 81 (X)+
(x)++(T)
8 6|x
ykt1lla
+ 8||**
(62)
[0.T]
Because the value of 8 holds for all time-steps (indexed by i), then:
(()++ - (r)+)
S
5|| 51|x
yktil|a
+ ø||** - r (63)
which can be rewritten as:
8
| - yk+ || I
1-8 | - ykl
(64)
This proves that for a value such that 1 < 1, the multirate integration scheme, p (.), is a contraction mapping whereby the series (x)+ and (y*++(rr) converges to a stationary point.
Note, the rate of the contraction mapping, determined by ェ-3 is not affected by the ratio of client step-sizes, At, This enables clients to take vastly different step sizes, with accuracy considerations imposed by a Local Truncation Error (LTE). The LTE estimates the error the approximation of the numerical integration and a key measure of the accuracy at any iteration.
To prove convergence to stationary point of the global objective function, we consider the contraction mapping between two series, (X*) and {X*), where X* is the state at the stationary point (i.e., g(X*) = 0). The difference between the series is diminished according to the contraction mapping:
|| X -
1 | 1 X*1
(65)
8 This implies that for At ensuring I-8 1, the FedECADO update asymptotically converges to stationary point, with convergence rate determined by 1-8
5
14

---

FedECADO: A Dynamical System Model of Federated Learning
B.1. Expanding Backward Euler Operator
Given general ODE
(t)) = (2,
(66)
with an initial known state of =(0), the state of z time point, Tm determined by solving the following:
2(mm) = z(Tm-1) +
s=((T),
(67)
The integral on the right-hand side generally does not have a closed form solution and approximated using a generalized numerical integration method:
=(mm) = =(Tm-11) Σ ki mt),
(68)
where ky is a scalar used to weight the contribution of the state at time 7, This expression can be further generalized:
2(Tm) =
+00),
(69)
2=0
where n weights the contribution of past states.
We can apply this form for the Backward-Eulde integration step of the ODE (46 which is generally written as follows:
(Tm) = (Tm-11) + Almg( X (Tm, (Tm, (T(())))
(70)
where the index m represents the iteration of Backward Euler steps taken. An equivalent representation is the following:
X (Tm))=
Σ (m (()).
(71)
This expresses the latest state ((mm) summation of previous values of g(-).
B.2. Proof of Lemma B.1
The proof of Lemma B.1
a direct consequence of the linearity of the operator, Г. Consider two sequences, X(r) and (()), which evaluated time points / and l2, where by:
x(t1) > ((h) x(22) > ((2))-
(72)
(73)
Applying the linear operator, T(, for a time point [4, 12], is defined as follows:
T(X,7) = X(12) - X(h) (T - (1) + X(ti)
(74)
t2 - t
r(Y, Y(t2) - Y (ta)
(r -1) Y(t))
(75)
- t
Because X > Y at time points t1 and t2, observe the following:
((22) ((t)) (r (t) +
Y(42) - -(t)) t2 - t
(T (1) Y(ti)
(76)
f2 t
thereby proving that T(X, 7) > r)Y,)). Expanding this proof to multiple evaluated time points T,, we note that X(T.) > Vi, then T(X,7) T(V,7).
15

---

FedECADO: A Dynamical System Model of Federated Learning
B.3. Proof for Lemma B.2
The proof of Lemma B.2
reconstructed in the following from
(White and Sangtovanmi-Vincentllll 2012 ].
From the definition of the 3-norm, that:
X (Tm-t))l maxe 33tm
xx (7m-1)ll+
(77)
m
which can be upper- upper-bounded bounded using the triangle inequality as:
XT(Tmmi)lle I max 9.tm
Σ hall (Tm-
(78)
1=0
Multiplying ( (equal 1) into the right-hand side of the equation above leads to
Σ 1=0
n Tm (mm- S maxe e 9Atm hal (1 8(mm-ppmm00000m- -)ar || X(m-
(79)
Because e (-8(m-1)a (T -I)l S |X (Tm 7mm- then
m
m
(Tm-)))) I max
8Ahm
(80)
m
1=0
Suppose Iyr upper-bbuuded by M, then the inequality becomes
m
Σ
X (Tm (1m- v
le 95mm] X (fm
(81)
1=0
Because e -35m > and Ei=o é 8Am < Eis
1== e
-35m then the inequality is as follows:
m
nX (m M
[[]]| (Tm-
(82)
The infinite series can be directly computed as:
e -
3Am
(83)
1 c e-AAt*
1=0
where the upper bound is as follows:
e -
Σ
nX (mx(Tm-))ll 3 M
-BDtm
[|
(84)
1=0
which proves Lemma B.2
C. FedECADO Algorithm
The full workflow for the FedECADO algorithm is shown in Algorithm 2 The algorithm begins by initializing the state values in Steps 1-4 and precomputing the constant sensitivity model, Gea. for all clients in Step 5. The hyperparameters for the algorithm are 7, > and the local truncation error tolerance, 0.
For each epoch, FedECADO begins by simulating the set active clients, C, E C, in step 10, for number of epochs, e,. The client ODE is solved by using numerical integration selected by the user (further details are provided in (Agarwal et al.) In step 11 of Algorithm 2 we use Porward-Fule integration to simulate the local ODE.
2023
16

---

FedECADO: A Dynamical System Model of Federated Learning
The active clients then communicate their final states, ( + Tj), and simulation time, Tr the central agent server, which evaluates its states intermediate time points (steps 12-16). First the central agent estimates the active client state values at time point, т, using the operator, I',,, r). Then after selecting time step, At, that satisfies the accuracy conditions in (34) (35), the central agent solves for its states at the proceeding time point, T + At, in step 15. The central agent server progresses through time (performing steps 13-14), until has simulated the maximum client simulation time window determined by max(T;)
Note, for a given time-step, At, the central agent LU-factorizes the left hand matrix Step 16 of Algorithm that any subsequent central agent steps with the same step size only requires forward-backwant substitution to solve for the central agent states. In practice, once an appropriate step size, At, is selected that satisfies the accuracy conditions in (35 it is infrequently updated. As result, re-using the same LU-factor provides computational advantage across multiple central agent evaluations and improves the overall runtime performance of FedECADO.
so
(34
Algorithm FedECADO Central Update
Input: Vf().x(0),) O, 0
I: - (00)
2: x; + x(0)
3: I," +0
4: 0
5: Precompute G Vi E C
6: do while |
0
7:
8:
X
9:
Parallel Solve for active client states, x (1 T.)Vi E C.. by simulating:
10: for epochs:
(t+ At. x
(t))) (t) At,I (t)
12:
for E [t,t+ max(T:))
13:
Select At according to Algorithm |
14:
Evaluate active client states at timepoint r: T(x r) Vi E Ca
Solve for x (T + At), I (r + At) according (33
16:
7 + AL
17:
Return X
D. Scaling the FedECADO Algorithm
The previous experiments highlight the individual contribution of the proposed aggregate sensitivity model and multi-rate integration on distributed training. In this experiment, we study the impact of both methods to address the challenges of federated learning across multiple datasets and models.
We demonstrate the effectiveness and scalability of FedECADO by training multiple models (ResNet 34, ResNet18 and LSTM) on datasets including CIFAR-100, TinyImageNet and Sentiment 100, distributed among 100 clients using a Dirichlet allocation. In this setup, only 10% of clients participate in each training round, and each client is assigned random learning rate defined in equation 337 We train larger ResNet-34 model CIFAR-100 dataset distributed across 100 clients with both non-lID data distribution as well as heterogeneous learning rates and numbers of epochs, determined by (37 Tables demonstrates the efficacy of FedECADO and larger and varied datasets and models.
Figure showcases FedECADO's s advantage over FedNova and FedProx in training the ResNet-34 model. Our approach achieves a lower training loss, indicating more efficient convergence, as well as higher classification accuracy after 100 epochs (4.6% higher than FedNova and 8.6% higher than FedProx).
17

---

FedECADO: A Dynamical System Model of Federated Learning
Classification Accuracy (%)
70
80
Classification Accuracy
60
70
50
60
50
30
40
FedECADO
FedECADO
20
30
FedNova
FedNova
FedProx
FedProx
20
20
40
60
80
100
20
40
60
80
100
Iterations
Iterations
Figure 4: The training loss and classification accuracy for VGG-11 model trained a CIFAR-10 dataset across 100 clients with (a) non-IID Dirichlet distribution and identical learning rates, and (b) each client's learning rate and number of epochs is randomly determined by (37) 38)
Classification Acc. (%)
FedECADO
FedNova
FedProx
FedExp
FedDecorr
FedRS
58.9
45.4
40.7
47.6
41.9
44.3
Table 4: Classification accuracies for training a ResNet-18 on TinylmageNet dataset distributed across 100 clients with Dirichlet data distribution and random learning rates for 60 epochs.
Testing Errors
FedECADO
FedNova
FedProx
FedExp 78.4
FedDecorr
FedRS
79.9
77.1
78.6
79.2
77.4
Table 5: Testing errors for training a LSTM model [R2] on Sentiment| 40 dataset distributed across clients with Dirichlet data distribution and random learning rates for 10 epochs.
Classification Accuracy (%
5
100
FedECADO
FedNova
Training Loss
4
FedProx
80
3
60
2
40
20
FedECADO
FedNova
FedProx
0
20
40
60
80
100
20
40
60
80
100
Iterations
Iterations
Figure Scaling FedECADO to train ResNet34 model on CIFAR-100 dataset distributed on 100 clients with heterogeneous computation (non-IID data distribution and asynchronous updates.
18

---

FedECADO: A Dynamical System Model of Federated Learning
E. Runtime Analysis
Despite evaluating the central agent multiple times for multi-rate integration, FedECADO's clock time for centralized updates is comparable those of FedNova and FedProx (only 1% slower than FedNova and 2.4% slower than FedProx). The complete comparison of runtime analysis shown in Table The main computational cost of Algorithm 2 occurs on the subset of active clients during each communication round, which is significantly smaller than the total client base. The resulting BE matrix 333 is relatively small, with dimensions of Raxe, where is the size of the active client list. Additionally, we pre-compute the LU matrix of in 33 to minimize the computation time for central agent evaluations.
Furthermore, a potential bottleneck for FedECADO's runtime can be attributed to the adaptive server step sizes routine in Algorithm This routine is initiated by the step size from the prior communication round and in practice, does not require multiple iterations to satisfy the LTE conditions in equation (3 and 35).
FedECADO
FedNova
FedProx
FedExp
FedDecorr
FedRS
Normalized Runtime
1.06
1.0
1.02
1.04
1.02
1.0
Table 6: Normalized per-epoch runtime (normalized to FedProx) of centralized server step for training VGG model on CIFAR-10 dataset distributed across 100 clients.
F. Comparison with ECADO
FedECADO uses the distributed optimization method from ECADO AAgarall and Pileggi 2023) as basis to derive insights for the unique challenges of federated learning including heterogenous dataset distributions and varying client computational capabilities (i.e., learning rates). However, ECADO is not equipped to handle the heterogenous challenges of federated learning, and prone to model drifts, an issue demonstrated (Wang al. 2020). Nonetheless, the unique perspective of equivalent circuit model of the federated learning process allows us derive intuitive methodologies. For example, the challenge of heterogeneous client learning rates clearly identified challenge of asynchronous communications in the continuous-ting representation. From the circuit perspective which views the distributed computation splitting large circuit, the immediate solution to this problem is using an intepolation//etrapllattoo operator to synchronize client computations, as described in Section
As result of these improvements, we observe that FedECADO improves the convergence ECADO in the federated learning setting. The difference is shown in Table 7 where we observe significant improvement in performance when training distributed CIFAR-100 dataset with non-Ill) data distributions and varying client learning rates.
Classification Acc.
FedECADO
ECADO
Mean (std) %
81.3 (4.8)
76.5 (2.1)
Table 7: Classification accuracies for training a ResNet-34 model on CIFAR-100 dataset distributed across 100 clients with Dirichlet data distribution and random learning rates for 200 epochs.
19


Reviews:
[{"review_id": "ZZKB37bXuF", "reviewer": "Reviewer_jW9j", "summary": "This paper explores the interpretation of federated learning in dynamical systems and adapts the ECADO algorithm to federated learning, proposing the FedECADO algorithm. It uses a physical equivalent circuit model to explain the federated learning process and targets optimizations for non-IID data and client asynchronous updates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments.\n\n---\n\n## **Novelty of FedECADO**\n\n FedECADO builds on the distributed optimization method,  ECADO, and introduces key innovations (multi-rate integration and chord-based modeling) to address the unique challenges in federated learning. Unlike in distributed optimization, where the worker nodes are homogeneous and always available, we tackle heterogeneous client computation capabilities where computational nodes are not available simultaneously and exhibit varying local learning rates and non-IID datasets. We believe the novelty of our approach lies in establishing a connection between federated learning and circuit methods, which offers a new perspective on these challenges. \nFor example, we re-interpret heterogeneous client learning rates as sub-circuits simulating for different time-periods. This allows us to take inspiration from circuit methods for distributed simulation. While advancements in general optimization have made the connection with circuits to inspire fast-convergent methods [R1]-[R3], FedECADO is the first work that introduces circuit knowledge to federated learning. This provides insights that are easily attained from the circuit model and simulation practices.\n\n---\n\n## **Aggregate Sensitivity Model**  \n \nCould the reviewers clarify their specific concerns about aggregate model sensitivity?\n\nWe would like to emphasize that in federated learning with non-IID data, each client’s local loss function has a unique solution landscape shaped by its data distribution.  The Hessian captures the curvature of the loss function at the client’s specific operating point and local training data. We use the Hessian to estimate the first-order sensitivity of a client’s local model to changes in the global model parameters. This sensitivity is used to anticipate how aggregation steps will affect individual client states. To approximate a representative Hessian for clients, we sample multiple data points and compute an averaged Hessian. Future work in FedECADO can also extend this by approximating the Hessian via [R4]-[R6]\n\n---\n\n## **Multi-Rate Integration**  \nCould the reviewers clarify their concern regarding multi-rate integration?\n\nTo clarify, we are not considering asynchronous client updates where clients begin with different global model versions. Instead, using the circuit model, we interpreted the challenge of heterogeneous client learning rates as clients simulating their local models for different time scales. To remedy this, the central server performs multi-rate integration by accepting the latest client updates, similar to FedAvg. Then, rather than directly aggregating these updates as in FedAvg, FedECADO first applies a linear interpolation/extrapolation operator to synchronize the simulation time scales across clients. The central agent step is then computed using a Backward-Euler integration step. This approach has a process similar to the aggregation step in FedAvg, but adds the additional linear operation, $\\Gamma(\\cdot)$, to address the challenge of heterogeneous client computation, and uses a Backward-Euler step to maintain numerical stability.\n \nAdditionally, in FedECADO, each client also receives the updated global state as shown in Algorithm 2 in the main text. \n\n---\n\n## **Improving the background on circuit-based framework**  \n \nAs part of improving the readability, we will include a section in the appendix that provides a deeper background on ODEs and circuit analysis.\n\n---\n\n## **Formatting**  \n\nThank you for the suggestion. In the following revision, we will improve the layout of Tables 4 and 6 as well as the equations.\n\n---\n\n## **Update on Table 1 and Optimization Plots**\nThank you for pointing out the inconsistency in Table 1. This was an unfortunate type in the previous version and is updated to reflect true classification accuracy as shown in the response for Reviewer xyRN.\n\n---\n[R1] Boyd, S., Parshakova, T., Ryu, E. and Suh, J.J., 2024. Optimization algorithm design via electric circuits. Advances in Neural Information Processing Systems, 37, pp.68013-68081.\n\n[R2] Agarwal, A., Fiscko, C., Kar, S., Pileggi, L. and Sinopoli, B., 2022. ECCO: Equivalent Circuit Controlled Optimization. arXiv preprint arXiv:2211.08478.\n\n[R3] Yu, Y. and Açıkmeşe, B., 2020. RC circuits based distributed conditional gradient method. arXiv preprint arXiv:2003.06949.\n\n[R4] Elsayed, M., Farrahi, H., Dangel, F. and Mahmood, A.R., 2024. Revisiting scalable hessian diagonal approximations for applications in reinforcement learning. arXiv preprint arXiv:2406.03276.\n\n[R5] Elsayed, M. and Mahmood, A.R., 2022. Hesscale: Scalable computation of hessian diagonals. arXiv preprint arXiv:2210.11639.\n\n[R6] Yao, Z., Gholami, A., Keutzer, K. and Mahoney, M.W., 2020, December. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data) (pp. 581-590). IEEE", "claims_and_evidence": "The proposed FedECADO method aims to solve the classic issues of non-IID data and client asynchronous updates in federated learning.", "methods_and_evaluation_criteria": "This paper uses the Aggregate Sensitivity Model to address the non-IID data distribution issue in federated learning. However, as mentioned in lines 32 and 233 of the paper, the Aggregate Sensitivity Model only reflects differences in the number of client datasets and does not address optimization for data distribution differences. \n\nFurthermore, the Multi-rate integration with adaptive step sizes, used to address asynchronous client updates, has limited applicability, especially for scenarios where model updates are not done locally but are aggregated on the server, with clients only accepting the global model parameters provided by the server. \n\nAdditionally, the core idea of the method is derived from ECADO, which limits its novelty.", "theoretical_claims": "I am not familiar with EC-related work, so I cannot determine if the process of using EC to model federated learning in this paper is correct, but it generally makes sense.", "experimental_designs_or_analyses": "Around line 366, this paper claims that Table 1 demonstrates that FedECADO has the highest average accuracy. However, this claim clearly contradicts the actual results shown in Table 1.", "relation_to_broader_scientific_literature": "Using equivalent circuits to explain the federated learning is a novel perspective.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "As noted earlier, the overall clarity of the paper could be improved.", "other_comments_or_suggestions": "• The formatting on the first page is incorrect.\n\n• The layout of Table 4 and Table 6, as well as most of the formulas, can be improved.", "questions_for_authors": "How does the Aggregate Sensitivity Model module address the issue of non-IID data distributions in federated learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0pRbaNVH7I", "reviewer": "Reviewer_xyRN", "summary": "This paper proposes a federated variant of ECADO (Agarwal and Pileggi, 2023). ECADO is an equivalent circuit approach to distributed optimization. ECADO consists in  reconstructing a distributed optimization problem in terms of circuit principles, and finding the critical points of the equivalent circuit model using a distributed Gauss-Seidel (G-S) process.  \n\nThe main contribution of this paper is the multi-rate numerical integration for heterogeneous computation, inspired by asynchronous distributed circuit simulation (White and Sangiovanni-Vincentelli, 2012). This is meant to address asynchronous local updates in federated learning resulting from from clients with heterogeneous computational capabilities. \n\nThe paper provides numerical simulation illustrating the performance of the proposed FedECADO approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. \n\n ---\n## **Novelty of FedECADO**\n\nRegarding novelty, FedECADO is inspired by distributed circuit simulation, where sub-circuits are independently simulated and recombined using waveform relaxation (White and Sangiovanni-Vincentelli, 2012). Our approach extends waveform relaxation by incorporating inductors and constant sensitivity models to accelerate convergence. The key innovation lies in establishing a connection between federated learning and circuit simulation, allowing simulation principles to be adopted to federated learning.\n\n\nFedECADO extends the circuit model of distributed optimization presented in (Agarwal and Pileggi, 2023) by introducing multi-rate integration and chord-based modeling to address the unique challenges of federated learning. Unlike in distributed optimization, where the worker nodes are homogeneous and always available, federated learning can have heterogeneous client computation capabilities where computational nodes are not available simultaneously and exhibit varying local learning rates. The multi-rate integration method is a key innovation of our work that can handle these new challenges. Importantly, the circuit model provides a new perspective to these challenges (such as framing heterogeneous client learning rates as asynchronous communication), which enables the development of intuitive and effective heuristics. FedECADO provides the first bridge between these fields, enabling new research directions where physical principles and simulation methods can drive federated learning solutions.\n\n ---\n## **Optimization Plots and Update on Table 1**\n\nTo the reviewer’s request, we have added the plots for training CIFAR-10 in the following link (in addition to the supplementary):\n  https://drive.google.com/file/d/1Ye1AiKFR2c8Xz2Tmps6x387WG8l0xlte/view?usp=share_link\n \nWe also would like to address the inconsistency between the text and Table 1. We have an unfortunate typo in the previous version and have have updated Table 1 (specifically FedRS, FedDecorr, and FedExp) to match the given plots.\n \n\n| Classification Acc. (%) | FedECADO   | FedNova    | FedProx    | FedExp     | FedDecorr  | FedRS      |\n|-------------------------|------------|------------|------------|------------|------------|------------|\n| Mean (Std.)             | 57.8 (3.6) | 48.9 (2.9) | 44.3 (3.2) | 45.3 (4.7) | 45.3 (4.7) | 45.3 (4.7) |\n\n\n The convergence plots for this CIFAR-10 example is provided in the link below:\n\n\nhttps://docs.google.com/document/d/e/2PACX-1vTamA9cixTmdcZecNeVjy7PZ9i-NuQi9c53d0zG2wPzjTg2tuoF4K_aGFIVZgLt096D0189JNK235ht/pub\n \n---\n## **Background on circuit details**\n\nWe understand that the inspiration for FedECADO (namely circuit-based ODEs) is not within the realm of the community. We plan on adding additional supplementary information that gives a primer on circuit physics and simulation in the appendix.", "claims_and_evidence": "The claim that FedECADO outperforms other federated optimization techniques is not well-supported. For example, FedRS, FedExp, and FedDecorr significantly outperform FedECADO on  CIFAR-10 dataset (Table 1). Moreover, FedECADO does not significantly outperform FedRS in Table 2. \n\nThe paper does not show the convergence curves for all compared methods.", "methods_and_evaluation_criteria": "Evaluation settings are standard in the context of federated learning. One could argue that more datasets/models are needed, but I think that the paper has already a sufficient number of datasets/models is enough. \n\nI argue that in this optimization-focused paper, providing convergence curves, in addition to final accuracy, is beneficial. Moreover, it would be helpful to illustrate the performance of the proposed method on a synthetic dataset, where the characteristics of the optimization problem are adjustable.", "theoretical_claims": "The only real theoretical claim of the paper is Theorem 4.1. I did not check the correctness of the proof, but the result sounds intuitively correct.", "experimental_designs_or_analyses": "See \"Methods And Evaluation Criteria\"", "relation_to_broader_scientific_literature": "The paper discusses relevant federated optimization papers.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "My main concern about this paper is its relative lack in terms of novelty. The paper is in many parts mirroring (Agarwal and Pileggi, 2023). The contribution of the paper is in Section 4.2. This contribution is an heuristic heavily inspired by (White and Sangiovanni-Vincentelli, 2012). \n\nI find that the paper is not well-written, and it was not easy to read through and to follow it without reading (Agarwal and Pileggi, 2023). The paper did not do a good job in summarizing and explaining (Agarwal and Pileggi, 2023), which is expected given that (Agarwal and Pileggi, 2023) is not a popular paper, and the reader is not supposed to know it a-priori. \n\nFedECADO has a 6% computational increase over FedRS (Table 6).", "other_comments_or_suggestions": "I think it would be better to use $I_{i}^{k}$, instead of $I_{L}^{i^k}$.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rghwqLVx9f", "reviewer": "Reviewer_uRW5", "summary": "This paper considered the federated learning problem, and focused on addresses the challenges from heterogeneous data distributions and computational workloads. To address these challenges, this paper proposed FedECADO, which is the first algorithm that leverages the idea of a dynamical system representation of the federated learning process. The performance of FedECADO is compared with various state-of-the-art methods such as FedProx, FedNova, FedExp, FedDecorr and FedRS.\n\n-----------------update after rebuttal---------------\n\nMy concerns were addressed by the authors and I maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments. \n\n---\n \n## **Hessian Approximation**\n \nIn federated learning with non-IID data distribution, each client’s local loss function is a function of its unique dataset. The Hessian captures the curvature of each loss function at the client’s specific operating point. As a result, different tasks, domains and scenarios will create unique solution landscapes whose curvature can be measured using the Hessian. We use this Hessian to measure how sensitive a client’s model is to changes from the global model during the central aggregation step.  This sensitivity helps the server anticipate the impact of model aggregation on individual clients. \n\n\nWe approximate a constant Hessian for each client by sampling multiple data points and averaging their respective Hessians.  Future work will improve this approach by using prior work (such as [R2]-[R4]) to improve efficiency and accuracy.\n\n\n ---\n \n## **Ablation Study**\n\nIn this work, the server step-sizes are modeled as time-steps for a Backward-Euler integration. These step sizes are adaptively selected based on the local truncation error, a measure of how closely we are tracking the designed ordinary differential equation. As a result, we do not select the step sizes, but rather select the tolerance for the discretization error due to Backward-Euler integration, denoted as $\\delta$ in equation (36). We find that the value of $\\delta$ does not provide much difference in the final simulation accuracy. We have added a figure, shown in [R1] where $\\delta$ is changed by three orders of magnitude (assuming a fixed client step size) for training on CIFAR-10 dataset. From the figure, we notice that the convergence plots remain well-behaved regardless of the choice of $\\delta$. Future work will look at designing the value of $\\delta$ based on approximating the Lipchitz constant of the gradient-flow updates. \n \n---\n\n## **Communication Cost**\n\nThe advantage of FedECADO in terms of communication is that we can handle client updates with heterogeneous client learning rates with only an additional scalar constant being communicated at each epoch. This scalar constant, $\\Delta T$, measures the amount of time each client is simulated for. \n  \n \n---\n\n## **Dynamic Clients**\n\nAdding and removing clients during the federated learning process can definitely be mapped to FedECADO's circuit simulation framework. We can model this behavior similarly to the switching of transistors in circuits, where transistors are added in series with clients and are “turned on or off” to connect or disconnect clients from the central agent. Circuit simulators, with their deep history in simulating discrete events, are well-suited for this type of modeling.\n\n ---\n\n[R1] https://drive.google.com/file/d/1OZRDtok0Ou6RdELJ2Eb13N7_4TJaU-J3/view?usp=share_link\n \n[R2] Elsayed, M., Farrahi, H., Dangel, F. and Mahmood, A.R., 2024. Revisiting scalable hessian diagonal approximations for applications in reinforcement learning. arXiv preprint arXiv:2406.03276.\n\n[R3] Elsayed, M. and Mahmood, A.R., 2022. Hesscale: Scalable computation of hessian diagonals. arXiv preprint arXiv:2210.11639.\n\n[R4] Yao, Z., Gholami, A., Keutzer, K. and Mahoney, M.W., 2020, December. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data) (pp. 581-590). IEEE", "claims_and_evidence": "Several claims have been made in the paper. Some examples are as follows: \n\n- Claim: The proposed FedECADO uses multi-rate integration to handle heterogeneous client computation\n- Evidence: This is validated via experiments. For example, in Table 2, it shows that when clients have varying learning rates and epochs, the accuracy can be improved. \n\n- Claim: The convergence of FL is improved by FedECADO in non-iid settings.\n- Evidence: Again, this is validated via experiments. For example, in Table 1, it shows that FedECADO outperforms the considered baselines in different datasets.", "methods_and_evaluation_criteria": "The methods applied in this paper is novel. It should be one of the first work to propose a dynamical system formulation of federated learning. \n\nThe evaluation of the proposed algorithm is appropriate as in many existing works in this area, e.g., using accuracy as a performance metric on CIFAR-10/100 datasets, comparisons with state-of-the-art baselines (e.g., FedProx, FedNova, FedExp, FedDecorr and FedRS).", "theoretical_claims": "Theorem 4.1 is the main theoretical claim in this work, i.e., it shows that FedECADO is a contraction mapping and hence it ensures convergence.\n\nI checked the proofs in Appendix A, which seems correct.", "experimental_designs_or_analyses": "The experimental designs are reasonable and similar to many existing works in the federated learning domains.", "relation_to_broader_scientific_literature": "This paper is related to federated learning, which has been extensively studied over the past decade.", "essential_references_not_discussed": "This paper covers most of the relevant works.", "other_strengths_and_weaknesses": "Strengths: \n- The methods applied in this paper is novel. It should be one of the first work to propose a dynamical system formulation of federated learning. \n- FedECADO is proved to be a contraction mapping and ensuring convergence.\n- Extensive experimental results were provided to validate the performance of FedECADO\n\nWeakness: \n- The design of FedECADO or its performance gain relies on approximating Hessians. This is somehow heuristic. Will this be easily generalized over different tasks/domains/scenarios? \n- The ablation study on hyperparameters can be improved, e.g., how to tune the step sizes?", "other_comments_or_suggestions": "- Communication cost is another important metric in federated learning settings. What's the advantage of FedECADO over baselines from this perspective? \n- Another real-world setting is that the clients/agents in federated learning are dynamic, i.e., some new agents may join the system, while some may leave the system. Can FedECADO handle this setting?", "questions_for_authors": "- Communication cost is another important metric in federated learning settings. What's the advantage of FedECADO over baselines from this perspective? \n- Another real-world setting is that the clients/agents in federated learning are dynamic, i.e., some new agents may join the system, while some may leave the system. Can FedECADO handle this setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jva0xkVsMj", "reviewer": "Reviewer_hJyM", "summary": "This work addresses the inherent challenges of heterogeneous data distributions and computational resource disparities in FL by introducing FedECADO, a novel algorithm inspired by a continuous-time ODE theoretical framework for understanding federated optimization process. Extensive empirical studies have demonstrated the effectiveness of the proposed method.\n\n## update after rebuttal:\nI decided to maintain my score since the current version of this paper is difficult to follow for readers without prior knowledge and thus requires further revision.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments and hope the following addresses their concerns.\n \n---\n\n## **Continuous-time ODE**\n\nThe key innovation of FedECADO over the framework proposed by Agarwal and Pileggi (2023) is its introduction of circuit- and simulation-based techniques to address the unique challenges of heterogeneous computation in federated learning. While the previous framework was designed for traditional distributed optimization, assuming homogeneous and always-available worker nodes, FedECADO is specifically designed to handle heterogeneous clients with varying computational capabilities, learning rates, non-IID data distributions, and availability.\n\nAt the core of FedECADO is a continuous-time ODE model, which reframes the challenges of heterogeneous client learning rates as a distributed simulation problem. In this model, each client's sub-circuit is defined by local loss function and dataset. By modeling federated learning as a continuous-time ODE, it becomes apparent that client drift due to heterogeneous client computation (where each client has different learnings) is a result of simulating each client for different time-scales. \n \n\nBuilding on this insight, FedECADO maps the federated learning process to an equivalent circuit to enable the use of circuit simulation to handle the challenges of asynchronous distributed updates. Additionally, this circuit model introduces a new way to model the client sensitivity using a Thevenin impedance, a concept well-established within circuit literature. As part of the novelty of FedECADO, we were able to integrate this sensitivity into the central agent step to improve the convergence rate of the overall federated learning process. \n \n\nThe mapping to continuous-time ODEs and circuit models is outside the realm of this community, however, brings new a perspective to address the challenges in federated learning. In this regard, we will add a background section to the supplementary material that covers ordinary differential equations and an understanding of circuit theory to provide the reader with more appreciation on how we derived our methodology. \n \n---\n\n## **Dataset size**\n\nThe value of $p_i$ refers to the relative number of samples of each client’s training set. The relative weighting allows us to ensure clients with larger datasets have greater influence on the central agent server. \n \n ---\n\n## **Computational Cost of (33)**\n\nThe main computational cost of performing FedECADO’s central agent step involves computing the Backward-Euler integration step in equation (33). We leverage a constant sensitivity model for each client to ensure that the left-hand side matrix in equation (33) is constant over the simulation. This allows us to pre-LU factor the large matrix prior to training process. During the training process, each epoch of the central agent step performs a forward-backward substitution of the pre-computed LU factor. This helps reduce the computational complexity of FedECADO as well as the overall runtime. The difference in the computational runtime is highlighted in Appendix D.", "claims_and_evidence": "The authors could elaborate on the necessity of employing the proposed continuous-time ODE analysis framework to derive algorithms aimed at addressing challenges related to data and computational heterogeneity.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "1. The paper can be challenging to follow at times due to its reliance on prerequisites in physics and ODE. It would be beneficial if the authors could provide more intuitive explanations of their theoretical framework and devised methods to enhance accessibility for a broader audience.\n\n2. The authors could discuss the essential distinctions in adapting the theoretical framework proposed in (Agarwal and Pileggi, 2023) to the scenario of partial client participation considered in this paper, including the associated technical challenges and how the novel theoretical tools provided in this work addresses them.", "experimental_designs_or_analyses": "One of the key contributions of the proposed method is addressing the challenge of heterogeneous distributions in FL. While the non-IID data splitting discussed in this paper primarily focuses on label distribution skew, it would be valuable to explore whether the proposed method can also be evaluated under scenarios involving feature distribution skew.", "relation_to_broader_scientific_literature": "This paper introduces a theoretical framework based on circuit theory and continuous-time ODE analysis, offering the community a novel perspective for understanding the federated optimization process.", "essential_references_not_discussed": "This paper have discussed sufficient references.", "other_strengths_and_weaknesses": "**Strengths:** This paper presents a novel and motivated theoretical framework based on circuit theory an ODE to analyze the federated optimization process, offering an intriguing and insightful perspective.\n\n**Weaknesses:** The paper heavily relies on circuit theory and continuous-time ODE analysis, which may be less accessible and less reader-friendly for those unfamiliar with these fields.", "other_comments_or_suggestions": "The diagrams in Figures 2 and 3 are somewhat roughly illustrated. Specifically, the blue and green curves extend beyond the vertical axis.", "questions_for_authors": "1. In Equation (15), could the authors clarify which client's relative dataset size the notation $p_i$ corresponds to?\n\n2. Does the calculation of solving Equation (33) in the proposed method incurs additional computational costs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers tend to accept this paper (scores 3, 3, 3, 3) and reach a consensus on the following aspects:
1. This paper provides a new approach to federated learning built on ECADO (Agarwal and Pileggi, 2023), which is an equivalent circuit approach to distributed optimization.
2. This paper is difficult to follow as it lacks sufficient self-contained understandable background knowledge of (Agarwal and Pileggi, 2023).

Reviewer jW9j and Reviewer xyRN have concerns on the novelty of FedECADO, worrying that FedECADO may be a straightforward application of ECADO (Agarwal and Pileggi, 2023) to the FL setting. The authors clarified in the rebuttal:
> Unlike in distributed optimization, where the worker nodes are homogeneous and always available, federated learning can have heterogeneous client computation capabilities where computational nodes are not available simultaneously and exhibit varying local learning rates.   

Despite the new perspective to applying physical simulation principles to solve federated learning problems, all reviewers found it extremely challenging to follow as it heavily relies on prerequisites in physics and ODE. I’m inclined to recommend acceptance on the condition that the authors will provide more intuitive explanations of their theoretical framework and make the paper self-contained.
Decision: Accept (poster)