NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel
Authors: Gabriel Thompson
Kai Yue
Chau-Wai Wong
Huaiyu Dai
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Decentralized Federated Learning
Neural Tangent Kernel
Primary area: general_machine_learning

Abstract: Decentralized federated learning (DFL) is a collaborative machine learning framework for training a model across participants without a central server or raw data exchange. DFL faces challenges due to statistical heterogeneity, as participants often possess data of different distributions reflecting local environments and user behaviors. Recent work has shown that the neural tangent kernel (NTK) approach, when applied to federated learning in a centralized framework, can lead to improved performance. We propose an approach leveraging the NTK to train client models in the decentralized setting, while introducing a synergy between NTK-based evolution and model averaging. This synergy exploits inter-client model deviation and improves both accuracy and convergence in heterogeneous settings. Empirical results demonstrate that our approach consistently achieves higher accuracy than baselines in highly heterogeneous settings, where other approaches often underperform. Additionally, it reaches target performance in 4.6 times fewer communication rounds. We validate our approach across multiple datasets, network topologies, and heterogeneity settings to ensure robustness and generalization. Source code for NTK-DFL is available at https://github.com/Gabe-Thomp/ntk-dfl}{https://github.com/Gabe-Thomp/ntk-dfl

NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel
Gabriel Thompson' Kai Yue
Chau-Wai Wong 12 Huaiyu Dai
Abstract
model aggregation. In the popular federated averaging (Fe- dAvg) algorithm (McMahan et al., 2017), clients perform multiple rounds of stochastic gradient descent (SGD) on their own local data, then send this new weight vector to a central server for aggregation. As FL gains popularity in both theoretical studies and real-world applications, numer- ous improvements have been made address challenges, including communication efficiency, heterogeneous data distributions, and security concerns (Sattler et al., 2020; Li et al., 2020b; Zhu et al., 2019). To handle the performance degradation caused by data heterogeneity, many works have proposed mitigation strategies for FedAvg (Karimireddy et al., 2020; Li et al., 2020b). Notably, some researchers have introduced the neural tangent kernel (NTK), replacing the commonly-use SGD in order to improve the model convergence (Yu et al., 2022; Yue et al., 2022).
Decentralized federated learning (DFL) is col- laborative machine learning framework for train- ing model across participants without central server or raw data exchange. DFL faces chal- lenges due to statistical heterogeneity, as partici- pants often possess data of different distributions reflecting local environments and user behaviors. Recent work has shown that the neural tangent ker- nel (NTK) approach, when applied to federated learning in centralized framework, can lead to improved performance. We propose an approach leveraging the NTK to train client models the decentralized setting, while introducing a synergy between NTK- WTse- evolution and model aver- aging. This synergy exploits inter-client model deviation and improves both accuracy and con- vergence in heterogeneous settings. Empirical results demonstrate that our approach consistently achieves higher accuracy than baselines highly heterogeneous settings, where other approaches often underperform. Additionally, reaches tar- get performance in 4.6 times fewer communica- tion rounds. We validate our approach across multiple datasets, network topologies, and hetero- geneity settings to ensure robustness and general- ization. Source code for NTK-DFL is available at https://gituub..om//bbbbe hhtp:/htttboooooooooooooooonnnta
Despite these advancements, the centralized nature of tradi- tional FL schemes introduces the possibility for client data leakage, computational bottlenecks at the server, and high communication bandwidth demand (Kairouz et al., 2021). Decentralized federated learning (DFL) has been proposed solution to these issues (Martínez Beltrán et al., 2023). In DFL. clients may communicate with each other along an undirected graph, where each node represents client and each edge represents a communication channel between clients. While DFL addresses some of the issues inherent to centralized FL., both frameworks grapple with the chal- lenge of statistical heterogeneity across clients. Although mixing data on central server could readily resolve this issue, transmitting raw, private training data from clients in- troduces privacy concerns, making FL and DFL approaches good candidates to address this challenge (Yuan et al., 2024). This paper focuses on the following research question: How can we design DFL approach that effectively addresses statistical heterogeneity?
1. Introduction
Federated learning (FI.) is machine learning paradigm in which multiple clients train global model without the explicit communication of training data. In most FL scenar- ios, clients communicate with a central server that performs
We propose a method that exploits the NTK to evolve weights. We denote this paradigm NTK-DFL. Our approach combines the advantages of NTK-based optimization with the decentralized structure of DFL. The NTK-DFL weight evolution scheme makes use of the communication client Jacobians, allowing for more expressive updates than tra- ditional weight vector transmissions and improving per-
'Electrical and Computer Engineering. NC State Uni- versity "Secure Computing Institute, NC State Univer- sity, Raleigh, USA. Correspondence Chau-Was Wong <chauwai. wong nesu.edu>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
formance under heterogeneity. Complementing this NTK- based evolution, utilize a model averaging step that ex- ploits inter-client model deviation, creating global model with much better generalization than any local model. We demonstrate that NTK-DFL. maintains high performance even under aggressive compression measures. Through reconstruction attack studies, we also analyze how this com- pression affects data privacy. The contributions of this paper are threefold.
as the case of many hospitals training a model for tumor classification with local, confidential images (Shiri et al., 2022). They may also aim to train a personalized model for each client to perform better on the local data distribu- tion. For example, different groups of smartphone users may use different emojis and would benefit from a personalized model (Tan et al., 2023). Our method focuses on training a nigh-performing global model that generalizes well across all clients, offering improved convergence and resilience to data heterogeneity compared to existing DFL methods.
1.
The proposed NTK-DFL method achieves convergence with 4.6 times fewer communication rounds than existing approaches in heterogencous settings. To the best of our knowledge, this is the first work leveraging weight evolution for decentralized federated training.
Neural Tangent Kernel (NTK) NTK has primarily been used for the analysis of neural networks (Golikov et al., 2022), though it has recently seen use in the training of neural networks for FL (Yue et al., 2022). Introduced by Jacot et al. (2018), shows that the evolution of an infinitely wide neural network converges to kernelized model. This approach has enabled the analytical study of models that are well approximated by this infinite width limit (Liu et al., 2020). NTK has also been extended to other model types, such as the recurrent neural network (Alemohammad et al., 2021) and convolutional neural network (Arora et al., 2019). We instead use the linearized model of the NTK approxi- mation as tool for weight evolution. Some studies have explored the integration of NTKs with FL. For instance, Huang et al. (2021) applied the NTK analysis framework to study the convergence properties of FedAvg, while Yu et al. (2022) extended NTK applications beyond theoretical analysis by training a convex neural network. Moreover, Yue et al. (2022) replaced traditional SGD-based optimiza- tion with NTK-based evolution in federated setting, where clients transmit Jacobian matrices to a central server that performs weight updates using NTK.
2. The effective synergy between NTK-based evolution and DFL demonstrates superior resilience to data heterogene- ity with model averaging.
3. The NTK-DFL aggregated model exhibits robust perfor- mance across various network topologies, datasets, data distributions, and compression measures. This perfor- mance is further supported by theoretical bounds demon- strating improved convergence rates.
2. Related Work
Federated Learning (FL) FL was introduced by McMa- han et al. (2017) as a machine learning approach that enables training a model on distributed datasets without sharing raw data. It attempts to address key issues such as data privacy, training on decentralized data, and data compliance for more heavily regulated data (e.g., medical imaging) (Zhang al., 2021). Despite its advantages, the centralized topology of FL introduces several challenges. These include potential privacy risks at the central server, scalability issues due to computational bottlenecks, and high communication over- head from frequent model updates between clients and the server (Mothukuri et al., 2021).
3. Proposed Method: NTK- Decentralized Federated Learning
3.1. Problem Statement
We begin with a brief overview of centralized FL. The goal of centralized FL is to train a global model w across M clients with their private, local data D; { ) where N₂ is the number of training examples of the ith client. FL algorithms aim to nu- merically solve the sample-wise optimization problem of min F(ur), where F(w) M EM (,F,(w) and P((w) , Ejnn ((w: y:.))
Decentralized Federated Learning (DFL) DFL aims to eliminate the need for a central server by connecting clients in a fully decentralized topology. Sun et al. (2021) adapted the FedAvg approach of multiple local SGD itera- tions to the decentralized setting. Our NTK-DFL method may be viewed as building on this foundation, using the neural tangent kernel for more effective weight updates. Dai al. (2022) proposed method of DFL where each client possesses their own sparse mask personalized to their specific data distribution. Shi et al. (2023) employed the sharpness-aware minimization optimizer to reduce the in- consistency of local models, whereas we tackle this issue through per-round averaging and final model aggregation. DFL approaches can aim to train one global model, such
In the decentralized setting, an omnipresent global weight w is not available to clients in each communication round. Rather, each client possesses their own model w; that is trained in the update process. Following related DFL work (Shi et al., 2023; Sun et al., 2021), we seek global model w that benefits from the heterogeneous data stored locally across clients and generalizes hetter than any indi-
2

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
possesses its own aggregated weight in m as well as a set of aggregated weights w from each of their neigh- bors E A The ith client computes the Jacobian of f(X;; u (1)) with respect to the neighboring model parame- ters wy (k) using its local data X, We denote this neighbor- specific Jacobian as
clients compute 2 (k)
inter-diiet weight exchange
excbange of avveraged weights (k)
client&
neighhbrrs Je N7.
J (k)
clients compute for neighbors
4
clients evolive weights via NTKs
[
(k1
1
H
18
20
clients consiruct H
Ixx:;
J = [wx;
(2)
:
J y, S(C;; ub!
(K+1
Each client sends every neighbor their respective Jacobian J(H true label Y,, and function evaluation f(,;; uo ). Note the order of the indices in the Jacobian: the ith client sends J an evaluation on the ith client's data and its
20 m
④
Figure I. NTK-DFL process: ① Clients exchange weights, @ Average weights with neighbors, & Compute and exchange Ja- cobians, labels, and function evaluations. @ Construct local NTK and evolve weights [Eg. (5a)]. This decentralized approach en- ables direct client collaboration and NTK-driven model evolution without central server.
neighbors' weights. contrast, the ith client receives J (AA from each of its neighbors, an evaluation on its neighbors' data and the ith client's 's weights. Algorithm 3 in Appendix A describes this process.
j.
vidual client model wi. A global or aggregated model may take the form u- x EM N,w,, where N = EM Nz. M
Weight Evolution After all inter-client communication is completed, the clients begin the weight evolution phase of the round (see Algorithm 4 in Appendix A). Here, all clients act in parallel as computational nodes. Each client possesses their own Jacobian tensor J as well as their neighboring Jacobian tensors J/ for each j E (,((
Notation Formally, we have set of clients C (L..........,.! J. Each client is initialized with its weight w (0) E Rd, where d is the size of the parameter vector and the superseript in w.
(0) denotes the initial communication round. Model training is done in series of communi- cation rounds denoted k E {1, 2,.,,.KK, K}. Let the graph at round be g(k) = (C, E where (()) is the set of edges representing connections between clients. Further- more, the neighborhood of client i at round denoted (( = {j I (i,j) E E. This graph is specified before each communication round and can take an arbitrary form.
We denote the tensor of all Jacobian matrices possessed by the ith client at round h as i matrices from the set (A) U (J(12)
(k) which is composed of
j. l e stacked along the third dimension. We denote the matrix of true labels and function evaluations stacked in the same manner as Vr and f(X)), respectively. Explicitly, we have J.h) R N. Mixdy xd (A) E R xd2, and f(x;) E RNxds, Ad- ditionally, N; = N₁ + + jENf' N, represents the total number of data points between client i and its neighbors, and d2 is the output dimension.
3.2. Proposed NTK-DFL
Figure 1 illustrates the proposed NTK-DFL method. We describe its key components below.
From here, each client performs the following operations to evolve its weights. First, compute the local NTK H. (k) from the Jacobian tensor J(() using the definition of NTK:
Per-round Parameter Averaging At the beginning of each communication round k each client both sends and receives weights-clien i sends its model w0!h (k) all neigh- bors E ! and receives w
(lk) from all neighbors. Each
(k) m,
1 1 (. (M) (Im)) (K) JM (ar
client then aggregates its own weights with its neighbors' weights to form a new averaged weight as follows:
d2
Each element of the NTK is a pairwise Frobenius inner product between Jacobian matrices, where the indices m and correspond to the mtth and nth data points, respec-
w (k)
N,+ N,+ SEN( N,, (N:wo! (k) +
Σ
N) (L)
SEN?
tively. Second, using ((()), the client evolves their weights as follows (see Appendix B for more details):
The client must then send this aggregated weight w. back to all neighbors. This step enables each client to construct a local NTK, comprised of inner products of Jacobians from both neighboring clients and its own Jacobian. See Algorithm 2 in Appendix A for details.
al ( (K) g (krtt (2.)) = (1-e
(4)
+ * (())
(()) (x.).
Local Jacobian Computation At this point, each client
We unroll gradient steps to find the weight (k.1) wt w0(h
as follows:
3

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
We also provide results on a reconstruction attack performed using client Jacobians in Appendix F.
w
(k.()
(R)
(k.t) 1,17
+ =
(5a)
4. Theoretical Analysis
R (k.0)
7
(1())) - (x
(5b)
In this section, we derive a convergence bound for NTK- DFL by analyzing the behavior of the average client weight M
d2
==0
vo (k) 1 Σ (*) similar to Sun et al. (2021); Shi et al.
(k:f) for timestep
Third, the client selects the weight w;
(2023). Unlike DFedAvg, the NTK-DFL bound includes a key additional dependence on the number of local iterations T in its main term. The bound captures the unique advan- tage that NTK-DFL gains by using much larger T values compared to other methods, improving convergence. The analysis also captures key factors such as the relationship between the spectral gap and model convergence, impact of data heterogeneity, and NTK approximation error.
with the lowest loss according to the evolved residual (k+).0) (( (X:) - y/ j. This is used as the new weight w. for the next communication round.
Final Model Averaging Throughout the paper, we study the convergence of the final averaged model ((kk) Ni EM w. (k) In the decentralized setting, clients would average all K client models after all training is completed. This may be done through fully- cully-connected topology, se- quential averaging on a ring topology, or in a secure, cen- tralized manner. The average may also be carried out over a subset of clients. In practice, we observe that the aggregated model (x(*) generally performs better than any individual client model w, (k) We study the impact of the order of client averaging on model performance with a client selec- tion algorithm and show the results in Figure Each client that opts in to model averaging contributes a portion of its data to a global validation set before training begins. Our client selection algorithm selects clients in the order of their accuracy on the validation set. We will demonstrate that in the practical setting, with a proper selection of clients, not all nodes must opt into final model averaging in order for the aggregated model to benefit from improved convergence. We note a difference between model consensus, often dis- cussed in the DFL literature (Savazzi al., 2020; Liu et al., 2022), and the proposed final model averaging approach. Model consensus refers to the eventual convergence of all client models to a single, unified model over numerous com- munication rounds. In contrast, our approach implements final model averaging as a distinct step performed after the completion of the training process.
Definition 4.1. (DFL objective). We are interested in mini- mizing the global loss [(w) across all clients, defined as
M
N >e Awx; Dig. (6)
((u)- = Σ New (u), Li(w)= N. 1m
where {(-) is a sample-wise loss applied to client data.
Definition 4.2. (The gossip/mixing matrix). [Definition 1,
(Sun et al., 2021)] The matrix M = [ma.y] E [0, 1]mmxmm is assumed to satisfy the following properties: (i) (Graph) If
+ j and (t,5) É V. then ma,j = O; otherwise, ma. > O. (ii) (Symmetry) The matrix is symmetrie, i.e. M M (ii) (Null space property) The null space of I M is spanned by the all-ones vector, null(I M) = span 1). (iv) (Spectral property) The matrix satisfies I 2 M > -I. The eigenvalues of M satisfy 1 = A^ (M) > |22 2
2 Am (M)), and the spectral gap is defined as (1 X)² E (0. 11. where == max AAg [M], |, Am (M)).
Assumption 4.3. (Standard DFL assumptions). We assume that VL, is Lipschitz continuous. We bound the variance of client gradients with o and the client gradient norm with B. We note that these assumptions are relatively mild and common in the DFL. literature. Additional details can be found in Appendix G.
Lastly, while communication overhead and memory effi- ciency are not the primary focus of this paper, we briefly note a technique to address potential memory constraints in NTK-DFF implementations. For scenarios involving dense networks or large datasets, we introduce Jacobian batching. This approach allows clients to process their local datasets in smaller batches, reducing memory complexity from ((dd))))
Assumption 4.4. (Approximation error of NTK gradient). For any client E {1, (1.2...., JM. the difference between the NTK gradient and the true gradient of the loss is bounded by ŌNTK, i.c.,
O(N:d2d/m) m1). where mi is the num- ber of batches. We also study communication efficiency, where NTK-DFL is resilient to compression measures such as top-k sparsification and random projections. This en- ables significant reductions in communication costs without significantly compromising convergence. A thorough dis- cussion of overhead can be found in Appendices D and E.
CNCK (w) V.((w))] S &
for all E Rdd We note that an explicit formulation of ŚNIK is possible given assumptions of simplified model (Huang et al., 2021; Yue et al., 2022) (see Lemma G.9 in Appendix G). In our analysis, we treat this as a constant in order to remain model-agnostic.
4

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
Theorem 4.5. Consider the average weight over M clients
(Xiao al., 2017), FEMNIST (Caldas et al., 2019), and MNIST (Lecun al., 1998). Each dataset contains C' = 10 output classes. For Fashion-MNIST and MNIST, data het- erogeneity has been introduced in the form of non-llD parti- tions created by the symmetric Dirichlet distribution (Good, 1976). For each client, a vector qi ~ Dir((a) is sampled, where qi E JRC is confined to the (C - 1) staanadd simplex such that >j j=1 Pi = 1. This assigns a probability distribu- tion over labels to each client, creating heterogeneity in the form of label-skewness. For smaller values of a, client possesses a distribution concentrated in fewer classes. We test over range of a values in order to simulate different degrees of heterogeneity. In FEMNIST, data is split into shards based on the writer of each digit, introducing hetero- geneity in the form of feature-skewness. For the model, we use a two-layer multilayer perceptron with a hidden width of 100 neurons for all trials.
M Eiz w; for round k- E {1.....K) and suppose the previously stated assumptions hold. Assuming that the learning rate satisfies 0 <
i (k) = 1.
<1= S I. we have
2) -64 ]
VC ( )I²
min ISASK
KT(T,n)
+ T, .g. δNTK) + (1, T, og: ŌNIK: A), (8)
where the constants are defined as
(T, : 1/T - a(n, og, ŌNTK) = -ل 21/T [L 1)
32/12 32-1
L + 1), (9a)
+ ÖNTX B} (9b)
[(1, T,g ŌNTK, - 512n $1 L33 (1ill + 1)
2 to o B²). (9c)
Corollary 4.6. Let the learning rate satisfy O(1 KT). Using similar assumptions Theorem 4.5 in Appendix G, we have the following convergence rate for NTK-DFL:
Network Topologies A sparse, time-variant k-regular graph with k 5 was used the standard topology for experimentation, where for each communication round k, new random graph g()) with the same parameter i is created. Various values of к were tested to observe the effect of network density on model convergence. We also experimented with various topologies to ensure robustness to different connection settings. We used a network of 300 clients throughout our experiments.
VC (C((0())) )0²
5 ((((1)) -C C.
min ISKSK
VKT T(RTI ogg)
(T(NrrKB + 62TX + og))
VK T + 9 + B²) (1 2)2K
K
+ ((ŌTK). (10)
Baseline Methods We compare our approach to various state-of of-thhe-rt baselines in the DFL setting. These include D-PSGD (Lian et al., 2017), DFedAvg, DFedAvgM (Sun et al., 2021), DFedSAM (Shi et al., 2023), and DisPFL (Dai et al., 2022). We also compare with the centralized baseline NTK-FL (Yue et al., 2022). The upper bound NTK-FL would consist of client fraction of 1.0 where the server constructs an NTK from all client data each round, which is infeasible due to memory constraints. Instead, we conducted a comparison following Dai et al. (2022), which we include in Appendix C.1. We also include details on baseline hyperparameters in Appendix C.2.
From the bound above, we can see that the main conver- gence term can be improved when we increase the number of local iterations, T. However, as expected, we cannot arbitrarily increase T without inducing a greater error in the terms ŌNTK. B, and gg. Compared to bounds in Sun et al. (2021), O (vx VK
of+B² ). the first term og + (T-A)==7=)
to the right of the inequality highlights improved conver- gence derived from the ability to select a value of T that to 2 orders of magnitude larger than that of other DFL methods. Intuitively, the bound tightens with increasing spectral gap (1 2) which is associated with mixing speed. We note that there is an irreducible convergence floor from the NTK approximation term ÔNTK. Obtaining an explicit bound on this term can be done through specific assumptions about the loss function and the model itself. For instance, as detailed in Lemma G.9, assumptions about the smooth- ness of f/ and bounded per-client residual norm lead to ÓNTX
Performance Metries We evaluate the performance of the various DFL approaches by studying the aggregate model accuracy on global, holdout test set. This ensures that we are measuring the generalization of the aggregate model from individual, heterogeneous local data to a more rep- resentative data sample. Our approach is in line with the goal of training a global model capable of improved gener- alization over any single local model (Section 3.1), unlike personalized federated learning where the goal is to fine- tune a global model to each local dataset (Tan et al., 2023). When evaluating the selection algorithm in Figure 6, we split the global test set in a 50:50 ratio of validation to test data. We use the validation data to sort the models based on their accuracy, and report the test accuracy in the figure.
+ B²)), thus becoming reducible.
5. Experiments
5.1. Experimental Setup
Datasets and Model Specifications Following Yue et al. (2022), we experiment on three datasets: FaShion-MNIST
5

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
Method
IID
a =
0.5
a = 0.1
Test Accuracy
0.85
DFedAvg
31
DFedAvgM
23
43
86
0.80
NTK-DFL (Ours)
DFedSAM
DFedSAM
200+
DFedAvg
DisPFL
DisPFL
43
87
200+
0.75
DFedAvgM
D-PSGD
D-PSGD
53
125
0
20 Communication Round
10
10
20
NTK-DFL
12
17
18
Communication Round
Figure 2. Convergence of different methods Aishion-MNISS for (left) highly non-llI with 0.1 and (middle) IID settings. (Right) The table displays the communication rounds required to reach 85% test accuracy on improvement in NTK-DFL convergence baselines for more heterogeneous settings.
We observe increased
0.86
0.85
Performance
Test Accuracy
Gap
0.84
0.83
DF (Ours)
NTK-DFL (Ours)
DFedAvg
DFedAvg
0.82
DFedAvgM
DFedAvgM
DisPFL
DisPFL
0.81
DFedSAM
DFedSAM
D-PSGD
D-PSGD
0.80
2
3
4
0.1
0.2
0.3
0.4
0.5
Number of Neighbors
Heterogeneity Level
Figure 3. Performance of NTK-DFL (left) sparsity level and (right) heterogeneity level (smaller more heterogeneous). NTK-DFL outperforms the baselines and the gains are stable as the factors vary.
5.2. Experimental Results
heterogeneity levels, the choices of the topology, and weight initialization scheme. Figure 3 illustrates the test accuracy of KNTKDFL and other baselines as functions of the sparsity and heterogeneity levels, respectively. We observe a mild increase in convergence accuracy with decreasing sparsity. NTK-DFL. experiences stable convergence across hetero- geneity values a ranging from 0.1 to 0.5. The left plot reveals that NTK-DFL consistently outperforms baselines by 2-3% across all sparsity levels. The right plot demon- strates NTK- resilience to data heterogeneity- while baseline methods' performance deteriorates with decreas- ing a, NTK-DFL maintains stable performance. In Figure of Appendix C, we evaluate NTK-DFL across a range of network topologies and find that it performs consistently well across different connection structures with the same sparsity level. Additionally, Figure 11 illustrates the impact of a dynamic network topology on NTK-DFF convergence. The dynamic topology accelerates convergence compared to the static topology, likely due to improved information flow among clients. Figure demonstrates effect of weight initialization on NTK-DFL performance. While random per- client initialization slightly slows convergence compared
Test Accuracy & Convergence Our experiments demon- strate the superior convergence properties of NTK-DFL compared to baselines. Figure illustrates the convergence trajectories of NTK-DFL and other baselines on Fashion- MNIST. We see that NTK-DFL convergence benefits are enhanced under increased heterogeneity. Under high het- erogeneity with 0.1, NTK-DFL establishes a 3-4% accuracy lead over the best-performing baseline within just five communication rounds and maintains this advantage throughout the training process. Additionally presented are the number of communication rounds necessary for con- vergence to 85% test accuracy, where NTK-DFL consis- tently outperforms baselines. For the a 0.1 setting, NTK-DFL achieves convergence in 4.6 times fewer com- munication rounds than DFedAvg, the best performing baseline. Figure 10 in Appendix C demonstrates simi- lar convergence advantage for NTK-DFL on both feature- skewed FEMNIST and label-skewed MNIST datasets.
Factor Analyses for NTK-DFL We evaluate NTK-DFL's performance over various factors, including the sparsity and
6

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
0.86
0.85
0.84
Test Accuracy
Test Accuracy
0.80
0.82
0.75
0.80
NTK-DFL
0.70
0.78
DFedAvgM
IID
DFedAvg
0.65
a' 0.5
0.76
D-PSGD
a = 0.1
0.00
0.01
0.02
0.03
0.04
0.60
Inter-Client Model Deviation
5
10
15 20
25
Communication Round
Figure 5. Relationship between inter-client model deviation and final test accuracy Fashion-MNIST. Each point represents trial with distinct hyperparameters. The plot reveals a positive correlation between model deviation and accuracy, suggesting that higher deviation may henefit model averaging in DFL. to a certain extent. Notably, the NTK-DFL. appprach demonstrates hoth higher accuracy and greater model deviation compared to other methods.
Figure 4. Performance gains of model averaging on convergence, trained Fashion- -NNSS.. Solid lines correspond to accuracy of the aggregated global model, whereas dotted lines correspond to the mean accuracy across client models. NTK-DFL's aggregated model maintains high performance, whereas mean client accuracy declines significantly with increased heterogencity.
the selection algorithm for the final model aggregation. The "best-to-worss selection algorithm highly effective in a highly heterogeneous setting with a = 0.1. It significantly outperforms a random averaging order and the lower-bound averaging order, which requires the fewest clients to be av- eraged to achieve the same level of accuracy. In practical deployments, final model aggregation has implications in fully decentralized setting. For example, when training starts to converge, clients may need to connect denser topology to shortlist neighbors with higher validation accu- racies for final model aggregation.
to uniform initialization, NTK-DFL exhibits robustness to these initialization differences.
Gains Due Final Model Aggregation Figure demon- strates the dramatic effect of final model aggregation on final test accuracy. Though the individual client models decrease in accuracy as the level of heterogeneity increases, the fi- nal aggregated model remains consistent across all levels of heterogeneity (as seen in Figures 2 and 3). In the most heterogencous setting a = 0.1 that we tested, the difference between the mean accuracy of each client and the aggregated model accuracy is nearly 10%, shown in Figure A sim- ilar phenomenon is observed in Figure 9 of Appendix C as the client topology becomes more sparse. For the same heterogeneity setting with sparser topology of k = 2, the difference between these accuracies is nearly 15%. Though the individual performance local client models may suffer under extreme conditions, the inter-client model deviation (see detailed definition in Appendix C.3) created by such unfavorable settings is exploited by model averaging to cuperate much of that lost performance. Figure suggests that inter-client model deviation enhances the performance of model averaging in DFL. While extreme dissimilarity in model weights would likely result poor performance of the averaged model, we observe that a moderate degree of deviation can be beneficial. For example, this is seen in Li et al. (2020a) with the tuning of the "proximal term" yu that regulates this degree of client diversity in model up- dates. We posit that the NTK-based update steps generate more advantageous level of deviation compared to baseline approaches, contributing to improved overall performance.
Per-round Averaging A blltion Study In Figure 7, perform an ablation study in which we remove the per-round parameter averaging that is a part of the NTK-DFL process. Here, clients forego the step of averaging their weight vec- tors with their neighbors during each communication round. Instead, clients compute Jacobians with respect to their orig- inal weight vector and send these to each of their neighbors (see Algorithm in Appendix A). A massive distribution shift can be seen in the figure, where the distribution in the ablated setting is clearly skewed into lower accuracies. In contrast, NTK-DFL with per-round averaging demonstrates a much tighter distribution around higher mean accuracy, effectively eliminating the long tail low-performing mod- els. Per-round averaging NTK-DFL serves as a stabilizing mechanism against local model drift, safeguarding clients against convergence to suboptimal solutions early in the training process. In other words, client collaboration in the form of per-round averaging with neighbors ensures that no client lags behind in convergence. This is a partic- ularly valuable feature in decentralized federated learning scenarios, where maintaining uniformity across diverse
Selection Algorithm
Figure 6 demonstrates the results of

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
0.85
0.86
0.80
0.84
Test Accuracy
Test Accuracy
0.75
0.82
Histogram of individual acc
0.70
0.80
Best to worst
0.65-
0.78
Worst to best Mean Shuffled
0.60
0.76
95%
0
20
40
80
0.55
Number of Clients Averaged
3
6
9
Communication Round
Figure Final model test accuracy on Bushion-MNIS" vs. the num- ber of clients averaged for highly heterogeneous setting with a = 0.1. The histogram shows the distribution of individual client model accuracies. Three client selection criteria tested: pro- posed high-to-low (red), random (green), and low-to-high (blue).
Figure 7. Distributions of individual client model accuracy vs. the communication round for Fashion-MNISTT. The proposed scheme (red) conducts per-round averaging among neighbors, whereas the ablated setup (hlue) does Per-round averaging reduces the skewness of the model performance distribution.
set of clients with heterogeneous data is major challenge (Martínez Beltrán al., 2023).
final model accuracy. Our method combines the expressive- ness of NTK-based weight evolution with decentralized architecture, allowing for efficient, collaborative learning without a central server. We reduce the number of commu- nication rounds needed for convergence, which may prove advantageous for high-latency settings or those with heavy encoding/decoding costs.
Communication and Memory Overhead While over- head is not the primary focus of our work, we propose several strategies to allow for control over these factors. To mitigate memory overhead, we experiment with Jaco- bian batching, where clients divide their data into batches each round and perform an NTK-DFL update per batch. As shown in Figure 13 of Appendix E, test accuracy per communication round improves with increasing batch size. To address communication overhead, we explore datapoint subsampling, where clients compute Jacobians with respect to only a fraction of their data each round. Figure 14 of Appendix illustrates the expected accuracy drop from this approach- natural trade-off for reduced communica- tion. Finally, evaluate compression techniques reduce NTK-DFL's communication volume further. Figure 15 of Appendix E compares the convergence of NTK-DFL with different compression methods, while Figure 16 presents convergence in terms of bits communicated. We leave read- ers to Appendix E for detailed discussions.
There are promising unexplored directions for NTK-DFL. For instance, extending the algorithm training models such as CNNs, ResNets (He et al., 2016), and transform- ers (Vaswani et al., 2017), possibly making use of new NTK methods suited for modern architectures (Arora et al., 2019; Tirer et al., 2022; Yang. 2019). Additionally, fu- ture research could explore the application of NTK-DFL to cross-silo federated learning scenarios, particularly in domains such healthcare, where data privacy concerns and regulatory requirements often necessitate decentralized approaches (Guo et al., 2025). Lastly, NTK-DFL may serve as useful paradigm for transfer learning applications in scenarios where a single, centralized source of both compute and data not available.
6. Conclusion and Future Work
Impact Statement
In this paper, we have introduced NTK-DFL, novel ap- proach to decentralized federated learning that leverages the neural tangent kernel to address the challenges of statis- tical heterogeneity in decentralized learning settings. Our work extends NTK-based training to the decentralized set- ting, while discovering unique synergy between NTK evolution and decentralized model averaging that improves
This paper presents work aimed advancing the field of machine learning. Our research has several potential so- cietal consequences, none of which we believe need to be specifically highlighted in this context.
8

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
Acknowledgments
Huang, B., Li, X., Song, Z., and Yang, X. FL-NTK: A neu- ral tangent kernel-based framework for federated learning analysis. In International Conference on Machine Learn- ing. pp. 4423 4434. PMLR, 2021.
This work was supported in part by the US National Sci- ence Foundation under grants SaTC-2340856 and ECCS- 2203214, and the ECE Undergraduate Research Program and the Caldwell Fellows Program at the NC State Univer- sity. The views expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, pp. 1880-8589, Red Hook, NY, USA, 2018.
References
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14:1-210 2021.
Alemohammad, S., Wang, Z., Balestriero, R., and Baraniuk, R. The recurrent neural tangent kernel. In International Conference on Learning Representations, 2021.
Alistarh, D., Hoefler, T., Johansson, M., Konstantinov, N., Khirirat, S., and Renggli, C. The convergence of sparsi- fied gradient methods. In Advances in Neural Information Processing Systems, volume 31, 2018.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averag- ing for federated learning. In International Conference on Machine Learning, pp. 5132-5143 PMLR, 2020.
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R., and Wang, R. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, 2019.
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1 1) 11:2878 23244, 1998.
Caldas, S., Duddu, S. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A. LEAF: A benehmark for federated settings. In Advances in Nearal Information Processing Systems, 2019.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogencous networks. In 3rd MLSys Conference, 2020a.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 450, 2020b.
Dai, R., Shen, I.., He, F., Tian, X., and Tao, D. DisPFL:: Towards communication-effiieee personalized federated learning via decentralized sparse training. Interna- tional Conference on Machine Learning, 2022.
Lian, X., Zhang, C., Zhang, H.,, Hsieh, C.-J,, Zhang, W., and Liu, J. Can decentralized algorithms outperform central- ized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Golikov, E., Pokonechnyy, E., and Korviakov, V. Neural tangent kernel: A survey, 2022.
Liu, C.,, Zhu, L.,, and Belkin, M. On the linearity of large non-linear models: When and why the tangent kernel is constant. In Advances Neural Information Processing Systems, volume 33, pp. 15954-15964, 2020.
Good, J. J. On the application of symmetric Dirichlet dis- tributions and their mixtures to contingency tables. The Annals of Statistics, 4(6), 1976.
Liu, W., Chen, L., and Zhang, W. Decentralized federated learning: Balancing communication and computing costs. IEEE Transactions on Signal and Information Processing over Networks, 1-13,, 2022.
Guo, P., Wang, P., Zhou, J., Jiang, S., and Patel, V. M. Enhancing MRI reconstruction with cross-silo federated learning. In Li, X., Xu, Z., and Fu, H. (eds.), Federated Learning for Medical Imaging, The MICCAI Society book Series, pp. 1555177. Academic Press, 2025.
Martínez Beltrán, E. T., Pérez, M. Q., Sánchez, P. M. S., Bernal, S. I.., Bovet, G., Pérez, M. G., Pérez, G. M., and Celdrán, A. H. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges. JEEE Communications Surveys; Tutorials, 25 (4):2983-3013, 2023.
Hc, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
9

---

NTK-DFL; Enhancing DFL in Heterogencous Settings via NTK
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efffcient learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282. PMLR, 2017.
Yu, Y., Wei, A., Karimireddy, P., Ma, Y., and Jordan, M. TCT: Convexifying federated learning using boot- strapped neural tangent kernels. In Advances in Neural Information Processing Systems, 2022.
Mothukuri, V., Parizi, R. M., Pouriyeh, S., Huang, Y., De- hghantanha, A., and Srivastava, G. A survey security and privacy of federated learning. Future Generation Computer Systems, 15:611-640, 2021.
Yuan, L., Wang. Z., Sun, L., Yu, P. S., and Brinton, C. G. De- centralized federated learning: A survey and perspective. IEEE Internet of Things Journal, 2024.
Yue, K., Jin, R., Pilgrim, R., Wong, C.-W., Baron, D., and Dai, H. Neural tangent kernel empowered federated learn- ing. In International Conference on Machine Learning, 2022.
Sattler, F,, Wiedemann, S., Müller, K.-R., and Samek, W. federated learning
Robust and communication-efficee from non-i..d. data. IEEE Transactions on Neural Net- works and Learning Systems, 3109):340-33113, 2020.
Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., and Gao, survey on federated learning, Knowledge Showledge-Based Based Systems, pp. 106775, 2021.
Savazzi, S., Nicoli, M., and Rampa, V. Federated learning with cooperating devices: A consensus approach for mas- sive IoT networks. IEEE Internet of Things Journal, 7(5):
4641 4654, 2020.
Zhu, L., Liu, Z., and Han, S. Deep leakage from gradients. Advances in Neural Information Processing Systems, volume 32, 2019.
Shi, Y., Shen, L., Wei, K., Sun, Y., Yuan, B., Wang, X., and Tao, D. Improving the model consistency of decentral- ized federated learning. In International Conference on Machine Learning, 2023.
Shiri, I., Vafaei Sadr, A., Amini, M., Salimi, Y., Sanaat, A., Akhavanallaf, A., Razeghi, B., Ferdowsi, S., Saberi, A., Arabi, H., Becker, M., Voloshynovskiy, S., z, D., Rahmim, A., and Zaidi, H. Decentralized distributed multi-institutionn PET image segmentation using fed- erated deep learning framework. Clin Nucl Med, 47(7); 606-617, (66-617,30 Jul 2022.
Sun, T., Li, D., and Wang, B. Decentralized federated averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4))428944301 2021.
Tan, A. Z., Yu, H., Cui, L., and Yang, Q Towards person alized federated learning. IEEE Transactions on Neu- ral Networks and Learning Systems, 34(12)95577-603, 2023.
Tirer, T., Bruna, J., and Giryes, Kernel-baas based smoothness analysis of residual networks. 2nd Mathematical and Scientific Machine Learning Conference, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information
Processing Systems, volume 30, 2017.
Xiao, H.,
Rasul, K., and Vollgraf, R. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms, 2017.
Yang, G. Wide feedforward or recurrent neural networks of any architecture are Gaussian processes. Advances in Neural Information Processing Systems, volume 32, 2019.
10

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
A. NTK-DFL Algorithms
Algorithm 1 Consolidated Federated Learning Process
Require: A set of clients C
Initialize weights w, for each client i.
2: for each communication round
K do
3: Initialize graph structure (K (C, E specifying the neighbors A7(h) for each client i.
4: Execute Algorithm 2 for Per-Round Parameter Averaging
5: Execute Algorithm 3 for Local Jacobian Computation and Sending
6: Execute Algorithm 4 for Weight Evolution
7: end for
Algorithm 2 Per-Round Parameter Averaging
Require: For each client i, set of neighbors A/(²) and initial weights us!
1: for each client i C in parallel do
2: Send un! (h! to all neighbors j E A,!!
3: Receive vn (k) from all neighbors E An(k)
Nc+ SEN!") Ny (New! (k) + Ejevge (
4: w
5: Send aggregated weight w; (k) back to all neighbors e A7th)
6: end for
Algorithm 3 Local Jacobian Computation and Sending Jacobians
Require: Each client knows neighbors N7(h) and has access to local data X, and the aggregated weights w from
each neighbor <v?(k)
for each client C in parallel do
2: Compute the Jacobian (k) =
=v. wi (k)) using the client's own aggregated weight wt and local data X..
3: for each neighbor j E N do
4: Compute the Jacobian J (8) 1.j
(X,; fXx; w
) using the neighbor's aggregated weight wy (k) and client's local
data X,
5: Send J (k)
true label Y,, and function evaluation f(X,; (h)) to neighbor j.
6: end for
7: end for
B. Additional Details on Weight Evolution
In implementation, computing the matrix exponential ( in (4) to evolve weights can be computationally expensive.
In practice, the weights are evolved according to the more general differential equation from which (4) is derived, reliant
upon the linearized model approximation f fXX,; w. (k.10)
(0 (h.0) + V (x (k.0)) (w (k.0)
(t.hh.0) w
(k.0)) The
differential equation is follows
f(x;; w - - H C,C.
(11)
Here, is the loss function. For example, for a half qqaane error (MSE) loss, term on the right becomes the residual matrix v,C f (X;;; 17/h ((:0)) y, During weight evolution, client evolves their neighboring function evaluation from the initial condition f(;;; ;; in ((:0)) to the time-evolved f wo (*.))) using differential equation solver and the differential equation above. To implement (5a), we use a process similar to Yue et al. (2022) where the initial client residual is evolved over series timesteps specified by the user. For user-specified timesteps, the loss at that time is found using the evolved residual. Then, the best- persfreefonnng weights are evolved using the left side (5a) and selected for the next communication round.

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
Algorithm 4 Weight Evolution
Require: Each client i has access local data X; and initial weights w0!h) and knows its neighbors A/(A)
1: for each client i C after intra-client communication do
2: Compute local Jacobian tensor (k) and receive J from each neighbor j
3: Construct tensor J.
(k) from { J (J (k)
1 E }
4: Compute local NTK H (1) using J (R):
5: for each data point pair (Iem Fn)) do
6: [ mn, <- d2 (J (8) (Im), J. (k) (zm))r
7: end for
8: for each timestep t 1 T do
9:
(k:0 (X:) + (I e R:
(k)
+ e (R)
N.
(x.)
10: W (k,t)
d2
(J
(k)
R (k.1)
11: end for
12: Select ;
1,00 t- w.
(k.+) with the lowest loss given the residual f(k.tl) (X,, - -y:
13: end for
C. Additional Experimental Details
C.1. Baselines
NTK-FL is the only centralized baseline that we compare with. We choose a per-round client fraction that ensures that the busiest node in the centralized setting is no busier than the busiest decentralized setting. By busier, we mean the degree of the node or the number of clients communicating with it. We note that NTK-FL is not an upper bound in this case due to the comparison being founded on node busyness, which disadvantages a centralized approach where all communication happens through a single, centralized node. Evaluating NTK-FL in the same setting as the table in Figure 2, NTK-FL converges to threshold accuracy in 73, 85, and 180 communication rounds for heterogeneity settings IID, respectively. D-PSGD (Lian et al., 2017) is one of the first decentralized, parallel algorithms for distributed machine learning that allows nodes to only communicate with neighbors. DFedAvg (Sun et al., 2021) adapts FedAvg to the decentralized setting, and DVedAvgM makes the use of SGD--based momentum and extends DFedAvg. Both use multiple local epochs between communication rounds, like vanilla FedAvg. DFedSAM (Shi et al., 2023) incorporates the SAM algorithm (Foret et al., 2021) into the DFL process. DisPFL (Dai et al., 2022) is personalized federated learning approach that aims to train global model and personalize it each client with a local mask. In order to make the comparision fair, we report the accuracy of the global model on our test set.
0.5, and a 0.1,
C.2. Hyperparameters
We perform a hyperparameter search over each baseline and select the hyperparameters corresponding to the best validation accuracy. We use the 0.1 Fashion-MNIST test accuracy at communication round 30 as the metric for selection. This is done because the majority of comparisons take place on Fashion-MNIST in the non-IID setting. For D-PSGD, we use a learning rate of 0.1, and a batch size of 10 (local epoch count is defined to be one in this approach). For DFedAvg, we use learning rate of 0.1, a batch size of 25, and 20 local epochs. For AveM we use a learning rate of 0.01, a batch size of 50, 20 local epochs, and a momentum of 0.9. For DisPFL, we use a learning rate of 0.1, batch size of 10, and 10 local epochs. Following Dai et al. (2022), we use the sparsity rate of 0.5 for DisPFL. For DFedSAM, we began with the parameters suggested Shi et al. (2023). After a hyperparameter search, we found that a radius p = 0.01, 17 = 0.01, momentum of 0.99, learning rate decay of 0.95, weight decay of 5 10-4 local epochs, and a batch size of 32 yielded the best performance. Note that used single gossip step per round for all approaches in order maintain fair comparison. As for the NTK-DFL, we use learning rate of 0.01 and search over values / E {100, 200, 800) during the weight evolution process. For the baseline compression methods, we experimented with quantization as well as top-k sparsification (see Appendix E). We found that the baseline methods, which were maddient-based cannot withstand the aggressive sparsification applied to NTK-DFL. Therefore, we used only quantization [such as in Sun et al. (2021)) and selected the most effective value to facilitate a fair comparison.
12

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
C.3. Inter-Client Model Deviation
We plot the relationship between the accuracy and the inter-client model deviation among NTK-DFL clients in Figure where the inter-client model deviation is defined for set of weight vectors wI..... WM R as follows:
d
M
M M
V
Σ
wis
wz)²
wis
(12)
d 1=1
where it captures per-parameter variation in averaged sense.
C.4. Additional Experimental Results
0.85
0.8
Test Accuracy
Test Accuracy
Cluster
0.80
0.7
Regular
Erdos-Renyi Ring
0.75
Aggregated Model
0.6
Line
Individual Client Models
0
10
15
2
3
6
Communication Round
Number of Neighbors
Figure 8. Convergence of NTK-DFL across different dynamic topologies, trained Fashion-MNNI MNIST. NTK-DFL is evaluated with a clustered graph (in yellow) with 5 neighbors per client, a 5 regular graph (in blue), Erdos-Renyi random graph with five mean neighbors (in green), a ring topology (in red), and a line topology (in purple). We observe that NTK-DFL demonstrates steady convergence across diffcrent topology classes. For the ring line topologies, convergence
Figure NTK-DFL. model accuracy as function neighbor count trained on Fashion-MNIST. Notably, the aggregated model accuracy across NTK-DFL DFL clients (in blue) remains consistent, even network sparsity varies. This stability persists despite signifi- cant decline mean individual client test accuracy (in yellow) as the number of neighbors decreases.
slower due to sparser graph of 2 rather than 5 neighbors client.
0.95
Test Accuracy
0.90
0.85
NTK-DFL (Ours)
DFedAvg
0.80
DFedAvgM
D-PSGD
DisPFL
0.70
20
25
20
25
15
20
Communication Round
Communication Round
Communication Round
(a) FEMNIST sreauree-skewd)
(b) MNIST (a 0.05)
(c) MNIST (a 0.1)
Figure Convergence various methods on heterogeneous datasets: (a) FEMNIST, (b) Non-IID MNIST (a 0.05), and (c) Non-IID
MNIST (a = 0.1). NTK-DFL consistently outperforms all baselines.
D. Discussion of Communication Trade-offs
Federated learning methods balance two key communication dimensions: the volume of data transmitted per round and the total number of communication rounds required. NTK-DFL transmits more data round than traditional approaches such as DFedAvg by sharing Jacobian matrices rather than gradients or weights, but requires fewer total rounds. We emphasize that the number communication rounds a key practical consideration. Below, we highlight scenarios where minimizing is especially beneficial:
13

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
0.86
0.86
0.84
0.84
Test Accuracy
Test Accuracy
0.82
0.82
0.80
0.80
0.78
0.78
NTK-DFL
NTK-DFL
0.76
0.76
DFedAvg
DFedAvg
0.74
0.74
5
10
15 20
25
5
10
15 20
25
Communication Round
Communication Round
Figure 11. The effect of static vs. dynamic topology on NTK-DF (Fashion-MNISS) a 0.1). Solid lines correspond dynamic topology. whereas dotted lines correspond to static topology. Both methods benefit from the dynamic topology NTK-DFL out- performs DFedAvg under both topologies. Other baselines are not drawn but perform similarly to DFedAvg.
Figure 12. The effect of different vs. identical weight initialization (Fashion-MNIST,
0.1). Solid lines correspond to the same weight initialization for all clients, whereas dotted lines correspond different initialization. The convergence of NTK-DFL affected less than that of DFedAvg. Other baselines not drawn but perform similarly to DFedAvg.
Large per-round communication latency, where fewer rounds can significantly reduce overall training time (e.g.. compression or encryption of weights/gradients, heavy preprocessing of input data).
Limited device availability, in which fewer rounds allow more efficient training when devices are intermittently available.
High bandwidth applications, where ample network bandwidth (c.g., gigabit home internet) can accommodate a large data volume for each communication round, making the number communication rounds the dominant factor in training efficiency.
Synchronization delays, where each round must wait for all devices to complete computation, with the slowest device bottlenecking progress, thus making the number of communication rounds an important factor.
E. Overhead Mitigation Strategies
While analysis of memory and communication overhead are not a central theme of this paper, we include strategies to mitigate both forms of overhead for practical deployment, thorough analysis of optimization and parallelization is out of the scope of this work and we leave it to future research.
E. Jacobian Batching
We introduce Jacobian batching to address potential memory constraints NTK-DFL implementations. For scenarios involving dense networks or large datasets, clients can process their local datasets in smaller batches, reducing memory complexity from O(N:d2d) to O(Nd2d//m (m)) where mi the number of batches. Clients compute and transmit Jacobians for each batch separately, evolving their weights multiple times per communication round. This approach effectively trades single large NTK H N for m. smaller NTKs H,, N/m, x//m, that form block diagonals of H, where N represents the total number of data points between client i and its neighbors N, While some information is lost in the uncomputed off-diagonal entries of H, this mitigated by the increased frequency of NTK evolution steps. Figure 13 demonstrates this phenomenon, where an increasing batch number my actually leads to improved convergence. This complexity reduction enables clients connect denser network for the same memory cost.
E.2. Use of the Clustered Topology
In the traditional NTK-DFL approach, each client must compute Jacobians on their own data with respect to the weight vector every neighbor, as NTK construction requires all Jacobians be evaluated the same point weight space. This
14

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
Test Accuracy
0.850
Test Accuracy
0.850
0.825
0.825
m1 2
M2 = 2
0.800
0.800
mi 4
M2 = 4
0.775
mi = 6
0.775
M2 = 6
mi =
M2 8
0.750
0.750
10
20
30
10
20
30
Communication Round
Communication Round
Figure 13. Test accuracy of NTK-DFI. communication round for various Jacobian batch numbers m., with higher my values denoting more batches (Fashion-MNIS) general, counterintuitive increase in test accuracy with an increased number batches.
Figure 14. Test accuracy of NTK-DFL. vs. communication round for sampling divisors m2 (Fashion-MNIST, a: = 0.1). Different from Jacobian batching. only a 1/me fraction of client data is selected each communication round. We observe slight decrease in test accuracy with increased 7z..
0.1). We observe a
implies that Jacobian computation scales linearly with the number of neighbors. However, a (dynamic) clustered topology reduces this burden: after the initial weight synchronization step, all clients within cluster share the same (aggregated) weight, so each client only needs to compute single set of Jacobians. These Jacobians then be reused for all neighbor interactions within the cluster. Furthermore, the weight evolution step can be offloaded to single designated client, or distributed across clients by sharing intermediate weights during the weight unrolling process (5b). As a result, both communication and computation overheads become independent of the number of neighbors, assuming each round's evolution is handled by single client per cluster. We illustrate the performance of NTK-DFL under this clustered topology in Figure 8 of Appendix C, which shows convergence comparable that of regular graph with the same average degree.
E.3. Communication Cost
Compared to traditional weight-based approaches that communicate a client's parameters w, each round, NTK-DFL utilizes Jacobian matrices enhance convergence speed and heterogeneity resilience. This tensor has memory complexity O(Ndgd)) where N, denotes the number of data points between client and its neighbors Nr, d is the model parameter dimension, and d2 is the output dimension. We propose the following strategies to improve the communication efficiency of NTK-DFL while maintaining convergence properties in heterogeneous settings.
Data Subsampling We introduce an approach where clients sample 1/mz fraction of their data each round for NTK evolution. Clients follow the protocol described in Section 3.2, but exchange Jacobian matrices of reduced size. As demonstrated Figure 14, moderate values m yield light performance degradation, validating this communication reduction strategy.
Jacobian Compression We employ several techniques reduce Jacobian tensor dimensionality. First, apply top-k sparsification, zeroing out elements with the smallest magnitude (Alistarh al., 2018). The remaining nonzero values are quantized to 5 bits. Additionally, we introduce a shared random projection matrix P E R xd, generated from common seed, creating projections Z; = X,P that reduce input dimension from dj to d'. This combination of techniques maintains convergence properties while significantly reducing communication costs. Note that similar compression schemes applied to weight-based approaches lead to significant degradation performance (Yue et al., 2022). Figure 15 illustrates the relative differences communication load for different combinations of the techniques above, with sparsification of 0.5, quantization to 6 bits, sampling of m2 5, and projection to d', 200 for the full optimization eurve.
Figure 16 compares the convergence of NTK-DFL with the baseline methods in terms of communication volume. NTK-DFL uses the compression methods detailed above, as well as a clustered topology (Section E.2) to reduce communication overhead. The communication-on opttimize NTK-DFL converges in fewer rounds than the baselines. However, with more expressive updates, uses greater communication volume. This enforees the idea that NTK-DFL is especially useful in scenarios where convergence in fewer rounds is important, such as those outlined in Appendix E.
15

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
0.85
0.86
Test Accuracy
0.84
0.80
Test Accuracy
0.82
D-PSGD
0.75
0.80
DFedAvg
Sparse
DFedSam
0.70
Sparse + Sample
0.78
DFedAvgM
Sparse Sample + Project
0.76
NTK-DFL
0.65
10
102
103
104
10
20
30
40
50
60
Communication Volume (MB)
Communication Load (MB)
Figure 16. Test accuracy vs. communication volume for NTK-DFL and baselines Freshion-MNISTT the use of compression techniques and clustered topology, NTK-DFL performs similarly in term of communication vol- ume. It is slightly outperformed by more communication efficient algorithms, though they need about 5 times as many communication rounds to converge.
Figure Comparison of NTK-DFL variants with progressive com- munication optimizations (Fashion-MNISTT, pling and projection technique provides compounding reductions communication load compared sparsification alone, while the fully optimized variant demonstrates significantly lower communication requirements comparable test accuracy.
0.1). With
0.1). Data sam-
F. Reconstruction Attack
While privacy preservation is not the primary focus of this work, we conduct a brief analysis of data privacy in NTK-DFL. Following the reconstruction attack method Zhu (2019), we evaluate the feasibility of reconstructing client data from transmitted Jacobian matrices under varying compression levels. Our experiments range from basic top-k sparsification with sparsity 0.25 to combined sparsification with random projection to dimension d', 200. Figure 17 illustrates that client data reconstruction becomes increasingly difficult when random projection is additionally applied to the Jacobian matrices.
Original Image
Sparsification
d' 500
d' 400
d 300
d' 200
Figure 17. Reconstruction attack of client data from Jacobian matrices for various levels of compression. For the image corresponding
sparsified matrices (the middle image of the first row), no random projection is done. We observe the ability to reconstruct a very noisy version of client data. For the other images, we use sparsification and random projection dimension d. We observe inability reconstruct client data when the random projection is additionally applied.
16

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
G. Mathematical Analysis
Below, we include more detailed assumptions that are used throughout the proof of the lemmas presented below and the subsequent proof of Theorem 4.5, We note that the assumptions listed below are relatively mild and common in DFL settings (Sun al., 2021; Shi al., 2023). Assumption G.1. (Lipschitz smoothness). The function L; differentiable, and its gradient VL, is L--Lipschitz continuous for all E (1. m), i...
[VL(ww) - -V)))l - L|/ -v/,
(13)
for all w, v Rd. Assumption G.2. (Bounded variance). The global variance of the gradient for functions {c,),=1 is jointly bounded, i.e..
m [VL(ww V og
(14)
m
for all w E Rd. This quantity is directly related to data heterogeneity among clients.
Assumption G.3. (Bounded gradients). For any i E (1, (1. m} and w RR, the gradient of the local loss function is bounded as [w)lw) < B, for some constant B > 0.
We present the following lemmas to aid in the analysis. First, we present Lemma G.4 that is common in DFL literature relating to the spectral properties of the mixing matrix. The following lemmas bound the mean divergence of the client weights over local update steps i (Lemma G.5), the variance of client weights (Lemma G.7), and the mean gradient norm over clients (Lemmma G.8). Lastly, we bound the error term δNIK (Lemma G.9).
Lemma G.4. [Following Lian et al. (2017)) For any k E z+, the mixing matrix M E Rm satisfies the inequality
|MM - Pllop S X?,
(15)
where the parameter X is defined as
x== max (22 1mm (M))-
(16)
Here, the spectral norm of a matrix A is denoted by ||A and the matrix P is given by
11T
R
(17)
m
where 1 is the all-one vector [1, 11,,,..., 1| m
Lemma G.S. Assume that Assumptions G.3 and 4.4 hold. Let (k.0) denote the model weights at round l and timestep f, and let the weight evolution be governed by the NTK-DFL approach. Then, it follows that
(k.++) 12. < 17 (B + ÖNTK)
(18)
for < k K.
Proof. [Following Sun et al. (2021)) We begin with the NTK update step
(k,++1)
(k.t)
(X, w. (T. (X,, Y,, w (f.t)). N (R)
w
w
(19)
where (X,, (x,, Y.) is the (possibly transformed) residual over the data between client and its neighbors j E N,. The specific r; depends on the loss of choice, and we provide general derivation that is loss-agnostic. Define the NTK-approximatte gradient of the local loss function L; with respect the model weights w; using the Jacobian evaluated = 0, we obtain
(1.t+1)
(k.t)
(k:t)) ANT (wo (ND (b))))
Wt u
(20)
17

---

NTL:DFL; Enhancing DFL in Heterogencous Settings via NTK
where
V w CocKTT(KU (w. (1)
1 Vwf w0 TT. (X,,, 3r. 20 (
(21a)
Nith)
A (wLi(u((it)) v. (2o (e0
(21b)
Using Assumptions G.3 and 4.4 and taking the norm:
||20 (k++1)
, (k:0) S (B + 5NTK);
(22)
therefore completing the proof.
Lemma G.6. Given the stepsize < 17 S SLT" assume w (k.3) and th) are generated by the decentralized NTK-FL algorithm for all i E 21, m). Assumptions G.I, G.2, and 4.4 hold, it follows that
1
m
m
m
(k.4)
w, 10(2.0011 1 S 16 8NTX + t0g + m
|VL(w (8) )12
(23)
m
for 1 K.
Proof. [Following Sun et al. (2021))
Decomposing the difference into global variance, the NTK difference, and the difference between = 0 and t,
w (R.+1) - w (x.00 (het)) w (2.0) 17 A NTK (w (k:0) + (V (2o (h.0)) - VZ.( (wo(t)))))
(h.1)
(VLC, (t.0)) (l(20((.0)) + vL((o
(24)
We consider two terms such that mm (t (h,+11)
(k:00) 1|2 < 1+ 11, using the Cauchy inequality and Assumptions G.2 and 4.4
wi
I (1+ ) w
(k..)
w
(k.0)
II 8712 (NIX (oo (. +2 (VV (1))
(25)
From here, rewriting term II using Assumption G.I, we have
w (k,(+1)
- wf || s (1 - + 8872 20 w (2/0001 + 22 + +oz) -T7211/<(( (1.0 )1.
(k.4)
(26)
Given an appropriately small step size, we simplify 1 + د yields
87 < 1+ 7 Solving the recursive relationship
w (k,()
w
1
Σ
Σ
(14
og)) + V[ (
(27a)
777
m
TRT + og
1
[V[ (4:00 |
(27b)
with the recursive sum Ete
- < 27 for T 2 10. Thus, we have completed the proof.
Lemma G.7. Suppose the stepsize satisfies 0 < 1) S SI7. and let w (k.0) be the model weights produced by the NTK-DFL algorithm for all clients E {1.2...., m). Under Assumptions G.2 and 4.4 the following bound holds:
2, 6NTK og + m
[V (5.00 112
w
(k.0)
w (10011
(28)
m
(1 - X)2
for all 1 S < K.
18

---

NTL-DFL; Enhancing DFL in Heterogencous Settings via NTK
Proof. [Following Sun et al. (2021) and Shi et al. (2023)) With a slight abuse of notation, Y = [w] (k.T) the mixing process as
w22 w22
(R.2)
(A.T) Wm
and X = (K.0) 222
(k.0)
wm 1
we may write
= x
= MX* -²
(29)
where
: M(X M(X* Y*).
(30)
We note that the order of averaging and updating weights does not affect this proof, as it relies on the spectral gap of the (possibly time-varying) mixing matrix, which does not change over time. We can write recursively
R-1 ΣM EM--1-SC
x = M
(31)
J=0
We note the following property of the mixing operation:
PM MP=
(32)
From Lemma G.4 we have that ||M Pllop S A Left-multilyinng both sides of (31) and using the mixing property (32),
we obtain
k-1
k-1 PÇ = - E
Px = PX
Σ
(33)
j=0
j=0
with the initialization X Then, we are led
k-1
|X
PX
Σ (P - M -
Σ P
1=
Σ
x
(34)
1=0
With the Cauchy inequality, we obtain
k-
k-
12
<
Σ
Σ
Σxers
Σ
x
(35)
1=0
1=0
and we can compute that
1< s ||M] ||X 1 < |X Y1
(36)
Applying Lemma G.6, for j.
m
x Y1 < 2
8NIK + +o² 02+ + m Σ
VV( 11
(37)
m
Finally, we complete the proof with
- x -
( + og + 1 Σm
/C(W (4) 1P?]
<
(38)
(1-
Lemma G.8. Using the result from Lemma G.7, obtain the following bound for the average squared gradient norm:
1
112 1112
2L2C1n2 (1- A)2
+ (
(39)
where Ci = 167 + + B²).
19

---

NTK-DFL; Enhancing DFL in Heterogencous Settings via NTK
Proof.
We expand the expression below by adding difference term:
VL(w )1
VL(w,) ( -V((("))))(() + (
(40)
Using ||a + b112 v 2|Ia||2 2 + 2|1b| we have
V x || m Σ
2/ VL( (1
(41)
m
With Assumption G.I, we can bound the gradient difference as
(1) )2 2L
1w (k) w + 2 VL (
(42)
1=1
Finally, we substitute the result from Lemma G.7 and use Assumption G.3 to finish the proof:
(V ( I|2 32L?T (RNTK + og B²) + 2 (1)
(43)
(1
Lemma G.9. Assuming appropriately small step size (such as is chosen in Theorem 4.5), Lipschitz continuity for Si(;;; (k)))) and a bounded mean residual norm across all clients, i.e. N. Σ [X; Y; W))ll S we can bound the error term in Assumption 4.4 as follows:
NTK S (6 B²).
(44)
Proof.
We can express ΑNTK as follows
A NTK Vw Cw (t:0)) ww (wo (1:0))
(45a)
(R) (V w/! ( V (Xi, w ( )r: (Xi, Vi, w ().
(45b)
Using the Lipschitz assumption listed above, where the Lipschitz constant for is L, have
ONTK S L3 w
(k.1)
(4.00 12,2
(46)
Now, substituting the result from G.6
6NTX I L I (6 +gg) + T6 ,
(47)
Solving for I
12,2 B²)
1 1 16 Rs 16, B2))
(48)
where the second inequality comes from assuming step size 17 such that, for a large communication round K, 16, << Thus, the proof is complete.
Proof of Theorem 4.5.
We begin with
W (k+1)
(b)
( io (k)
(49)
due to the fact that the application of the mixing matrix does not change w. Using the NTK-DFL weight update step, noting that the average weight at round and time step is written as (k) (k.0),
T
T
to (k.T)
(k)
17
V NTK (w (kiw)
7
Σ
ΣΣC
+ A (kia))
(50)
(,NTK
2=
20

---

NTK-DFL; Enhancing DFL in Heterogencous Settings via NTK
Assumption G.1 gives us
)(w(ktt)) S ((w(1)) + (VL(((()) w(++1) -w(()) + L [ - -w11)
_ 2
(51)
For the first inner product term, we have
(V (VL(w(()), (A++1) 25 (4)) (TV (TVL( (20(A (x))) V [n -
-w
(52a)
TV (k) 17 + (0.T) (A)
--
(52b)
T
7 T
- (
(V)
(K)
(z)
17
(L,(w, (1.))) ) +NNIK (zo(())))
(52c)
T
T
i=1t=1
(>))
(V
( - (ci( A (20 (2o((h011))
(52d)
T
m
T
() vc((20(h())) ))
V VL
+ ITTKK
(52e)
m
Using Assumption G.1, we have
(V((w())), (k+t)) 20 (k)) -n (8)
1
(b)
w.
(k.0 + nITTTTB
(53)
m
Using Young's Inequality S a² 2a + 2. write the norm in terms of the squared norm where set:
1w w (8.2)) |,
= 1/L,
(54)
which yields the bound:
T/7 (L)))) 2 nL2T 1
nLT ((
(k) (htt
1
(55)
2
2
i=1
With Jensen's 's inequality and Lemma G.6:
mLT ()
m (k)
(k) - w (n(ttt1) || s T
(
+ nT.TT
m
(k.tt - w (1)
(56a)
2
2
m
171
i=
T 2
(+)) nT.TT
26 (
og
VVL( (k)) 112)
(56b)
Substituting this bound above, the inner product term is bounded by:
rf 2
(V)(w(()) (0 (1+1) 2 (k)) - )
(1)
+ 877 1278
82TH + +02
v VLw (2) 112
+ +TTbN (57)
m
For the second term, we have
L (0
2 (k+1)
L - (1))
[k(1)
<
w
(R.T)
w
(58)
2
2
2
21

---

NTK-DFL: Enhancing DFL in Heterogencous Settings via NTK
Using results from Lemma G.6
L
- 8 QNTK og Σ v (( (1)
(59)
From here, we can write (51) as
nI'
((unlkt)) - ((20(1)) S 2
[V 8172T22 1)(6 + +zz) + 1/T B
m
+
7 VL )0) (60)
Substituting Lemma G.8, obtain
((w((+1)) ((( (() s off 16 T (V [[
+ + + IITTNxKB
L11 +1]
(1 1)2 (61)
Taking the sum over K rounds, we obtain
TT - s K
[ 167
min I<RSK
+
X)2 (62)
By moving terms around, we can finally prove the theorem below:
C (w)(1)) - C (20(k+1)
min IEkcK
VL
(b))))
|| <
K
DT 2 16 16n2T= 1)
8T (1..T + 1 (RRTK + + + T/TTT B 8T 1)-
-I-XP²
(63)
nT
2 16nTTT
Wrapping this up constants, we can rewrite the above inequality as
VC [1² < )[ -c]
min S
TT T, 6NTK) + B(n, T, 8NTK. A);
(64)
KT(T,
where
{(T,n) = -/T -32 L(1TL+
(65a)
a(1. Tg ÖNTK) 161,TT2L (nLT + 1) (62TH + +ogg) + 2nT B
(65b)
(T,
8(7, T. o.. 8NIK, A) 4 3 (1/T'L + 1)(T +oz 132)
(65c)
(1 -1- ()3
22


Reviews:
[{"review_id": "Srn8Jw9Bz2", "reviewer": "Reviewer_cjuK", "summary": "The paper proposes NTK-DFL, a decentralized federated learning method that uses the Neural Tangent Kernel (NTK) for weight evolution, replacing SGD with Jacobian-based updates. It integrates per-round parameter averaging and final model averaging to address statistical heterogeneity. Experiments show the improved accuracy of the proposed method in heterogeneous settings (α = 0.1), and robustness across datasets (Fashion-MNIST, FEMNIST, MNIST) and topologies. A theoretical convergence bound is provided, showing dependence on local iterations (T) and spectral gap.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer cjuK\n\n1. **“NTK-based approaches have explicit constraints in modeling neural networks (linear layers), which makes the proposed method hard to incorporate into modern architectures (CNN, Transformers).”**\n\nWe appreciate this important point. Indeed, our current fully-connected model was chosen specifically to study NTK-DFL’s effects clearly without additional complexity. However, several recent works have successfully extended NTK frameworks to CNNs, ResNets, and Transformers (Yang, 2019). Incorporating these approaches represents a promising avenue to generalize our method to modern neural architectures. We will clarify this future direction explicitly in the conclusion of the revised manuscript. \n>“...  possibly making use of NTK methods suited for modern architectures (Arora et al., 2019; Yang, 2019).”   \n\n2. **“Sharing prior distribution (true labels for all local data samples) can incur potential privacy concerns.”**\n\nWe agree with the reviewer that sharing true labels across clients could introduce privacy risks, however, the introduced risks are comparable to those introduced by gradient updates in FedAvg-style methods. We had actually tested privacy leakage due to sharing true labels in our submitted manuscript with a reconstruction-attack analysis under different mitigation strategies (Fig. 17, Appendix E). We found that under compression alone, reconstructions are significantly noisy, and when combined with random Gaussian projection, reconstruction becomes effectively impossible. \n\n3. **“The claim that the proposed method reduces the number of communication rounds to reach a target accuracy could mislead readers into assuming it improves overall communication efficiency… For a fair comparison, the method should be evaluated based on the total communication overhead required to achieve the target accuracy, rather than just the number of rounds.”**\n\nWe will update the manuscript to explicitly highlight communication trade-offs in NTK-DFL. Specifically, while our method significantly reduces the number of rounds needed to reach target accuracy, each round involves higher communication overhead due to Jacobian exchange. Please see ***Response 2 to Reviewer oV9C*** for a clear comparison of communication cost versus the test accuracy for NTK-DFL and DFedAvg, the best-performing baseline. For a detailed description of settings where the communication rounds are an important metric, see ***Response 1 to Reviewer Zq37***. \n\n4. **“Technical novelty is limited because the NTK-based update rule has been proposed in previous works and its validity for mitigating data heterogeneity also has been shown.”**\n\nWe will again kindly direct the reviewer to our ***Response 2 to Reviewer Zq37***.\n\n5. **“It would be good to include the results of the incorporation of several compression techniques (sparsification) with other baselines.”**\n\nWe plan to update Figure 15 to include a comparison of compression techniques with other baselines. Notably, gradient-based DFL methods perform significantly worse under the aggressive compression used in NTK-DFL. For example, DFedAvg reaches only ~55% test accuracy when both sparsification and quantization are applied, although it maintains better performance when subjected to quantization alone. Conversely, NTK-DFL is able to withstand severe quantization and compression while only sacrificing 1-2% performance degradation over the same number of communication rounds.\n\n6. **“Why do model variance and final test accuracy on Fashion-MNIST show positive correlations? Can you provide more detailed discussion or analysis for that relation? How can we define moderate or severe dissimilarity between client weights?”**\n\nWe thank the reviewer for raising this insightful question about the relationship between model variance and test accuracy, as well as the criteria for defining weight dissimilarity. Model averaging benefits from a diverse set of client solutions that generalize better when averaged together than any single model. Furthermore, weight dissimilarity can be quantified in many ways. For example, in Sahu et al. (2018), the norm of the weight difference is used as a metric to tune the FL updates. A mathematical formulation of “moderate” vs. “severe” dissimilarity between client weights may be grounds for a paper itself, and therefore we leave it to future work. We plan to update the end of the section “Gains Due to Final Model Aggregation” to include the discussion of this phenomenon based on our response above. \n\n**References**\n- Anit Kumar Sahu et al. (2018). Federated Optimization in Heterogeneous Networks. \n- Greg Yang. (2019). Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.", "claims_and_evidence": "The claims—faster convergence, heterogeneity resilience, and robustness—are supported by empirical results and theoretical bounds (Theorem 4.5). The evidence is clear, with detailed experiments across multiple settings. However, the proposed methods have several weaknesses. First, NTK-based approaches have explicit constraints in modeling neural networks (linear layers), which makes the proposed method hard to incorporate into modern architectures (CNN, Transformers). Second, sharing prior distribution (true labels for all local data samples) can incur potential privacy concerns. The claim that the proposed method reduces the number of communication rounds to reach a target accuracy could mislead readers into assuming it improves overall communication efficiency. In reality, it transmits significantly more data per round than naive FedAvg. For a fair comparison, the method should be evaluated based on the total communication overhead required to achieve the target accuracy, rather than just the number of rounds.", "methods_and_evaluation_criteria": "The NTK-DFL method is appropriate for DFL, targeting heterogeneity via NTK and averaging. Evaluation of standard datasets (Fashion-MNIST, FEMNIST, MNIST) with Dirichlet-based heterogeneity and global test accuracy as a metric aligns with FL research.", "theoretical_claims": "I reviewed Theorem 4.5 and Corollary 4.6 proofs (Appendix F) without a fully rigorous check, but they appear mathematically correct under the stated assumptions (Lipschitz continuity, bounded variance, NTK error bound).", "experimental_designs_or_analyses": "The design—comparing NTK-DFL to baselines across heterogeneity levels, topologies, and sparsity—is sound. However, not the accuracy over communication rounds, but over total communication overheads is more appropriate for the fair comparison.", "relation_to_broader_scientific_literature": "First NTK-based DFL: Claims to be the first to apply NTK-based weight evolution in a decentralized setting, achieving 4.6x fewer communication rounds than baselines in heterogeneous settings.", "essential_references_not_discussed": "I believe the key papers are discussed in the main papers.", "other_strengths_and_weaknesses": "Technical novelty is limited because the NTK-based update rule has been proposed in previous works and its validity for mitigating data heterogeneity also has been shown.", "other_comments_or_suggestions": "As mentioned in previous sections, it would be better to compare the accuracy over the total communication overheads for the fair comparison. It would be good to include the results of the incorporation of several compression techniques (sparsification) with other baselines.", "questions_for_authors": "- Why do model variance and final test accuracy on Fashion-MNIST show positive correlations? can you provide more detailed discussion or analysis for that relation? how can we define moderate or severe dissimilarity between client weights?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HTlQKsGauO", "reviewer": "Reviewer_oV9C", "summary": "The paper proposes to integrate NTK training in DFL. Numerical results show NTK-DFL achieves higher accuracy under the same level of communication round.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer oV9C\n1. **\"It is hard to imagine a scenario where [the NTK-DFL regime] has any benefit in terms of communication overhead.\"**\n\nPlease see ***Response 1 to Reviewer Zq37*** for a detailed discussion.\n\n2. **“Authors should show acc vs. communication bit curve instead of acc vs. communication round curve for all benchmark algorithms.”**\n\nWe note that communication rounds are also an important metric in certain cases (e.g., encoding delays, high-bandwidth applications, etc.) as outlined in ***Response 1 to Reviewer Zq37***. \nTaking into account the reviewer’s suggestion, we plan to add a figure in Appendix E comparing communication volume across baseline algorithms. We plan to update Figure 16 in Appendix E to include a comparison of communication volume with other baselines. We provide a clear comparison of communication costs between our method and the next best-performing baseline in the table below.   \n\n\n*Comparison of NTK-DFL and DFedAvg convergence across communication volume thresholds.* \n| Threshold Communication Volume (MB) | Test Accuracy NTK-DFL | Test Accuracy DFedAvg | Comm. Rounds NTK-DFL | Comm. Rounds DFedAvg |\n|-------------------------------------|------------------------|------------------------|-----------------------|-----------------------|\n| 10                              \t| 79.2               \t| 81.2               \t| 4                 \t| 28                \t|\n| 20                              \t| 82.4               \t| 83.1               \t| 9                 \t| 58                \t|\n| 30                              \t| 83.5               \t| 84.3               \t| 14                \t| 87                \t|\n| 40                              \t| 84.2               \t| 84.8               \t| 19                \t| 116               \t|\n| 50                              \t| 84.6               \t| 85.2               \t| 24                \t| 146               \t|\n\nWith the original compression methods from Appendix E and a clustered topology (see ***Response 3 to Reviewer Qo5R***), NTK-DFL performs comparably to DFedAvg, as illustrated in the table above. More specifically, NTK-DFL suffers a small 0.5-1% test accuracy degradation in exchange for converging in 6x fewer communication rounds. DFedAvg is shown with a regular topology but performs similarly to a clustered one ($\\< 0.1$% difference in test accuracy at 1000 comm. rounds). Only the best-performing baseline, DFedAvg, is shown; results for other baselines will be added in the updated manuscript.\n\n3. **\"In the implementation, it seems that the authors use the eigen-decomposition of $H$ to evaluate the exponential map. This is very unconventional in deep learning, where people usually use gradient descent. A running time comparison would be needed to inform the practitioners.\"**\n\nNo, eigendecomposition on the matrix $H$ is not required. Instead, the weights are evolved using a differential equation solver (Chen, 2018) according to the more general differential equation:\n$$ \\frac{d}{dt} f\\left(\\mathbf{X}_i; \\bar{\\mathbf{w}}_j^{(k, t)}\\right) = -\\eta \\mathbf{H}_j^{(k)} \\nabla_f \\mathcal{L}.$$\nTo perform weight evolution, the client evolves the initial residual matrix over a series of timesteps specified by the user. For user-specified timesteps, the loss at that time is found using the evolved residual. Then, the best-performing weights are calculated and selected for the next communication round. Analogous to local epochs in gradient-based federated learning methods, the runtime scales as $O(t)$, where $t$ is the number of local evolution steps in the linearized NTK region. This process had been described in depth in Appendix C of the original manuscript. \n\n4. **\"There are some typos in Algorithm 4: Weight Evolution.\"**\n\nWe thank the reviewer for noticing these typos. We will correct the notation and relevant equations.\n\n**References**\n- Chen, R. (2018). Torchdiffeq: PyTorch Implementation of Differentiable ODE Solvers. https://github.com/rtqichen/torchdiffeq. Accessed Mar 26, 2025", "claims_and_evidence": "The claims are well supported.", "methods_and_evaluation_criteria": "The benchmark datasets and the algorithms are relevant for comparison.", "theoretical_claims": "The theorem 4.5 looks correct.", "experimental_designs_or_analyses": "Authors should compare communication overhead in bits (or MB) for all benchmark algorithms.\n\nLet's consider three scenarios.\n\n1. Scenario 1: DFL regime. Client $i$ only shares the $d$-dimensional gradient of model parameters with the neighbors in each round. Communication complexity on each edge is $O(d)$.\n\n2. Scenario 2: NTK-DFL regime. Client $i$ shares the Jacobian $J_{i,j}^{(k)}$ with the neighbor $j$ in each round. Communication complexity from $i$ to $j$ is $O(d N_i d_2)$\n\n3. Scenario 3: Data sharing regime. Client $i$ shares all its data with neighbor $j$ at the beginning of training. Communication complexity from $i$ to $j$ is $O(N_i d_2)$.\n\nIn terms of communication complexity, 2 is worse than 3. In terms of convergence, 2 is worse than 1, according to Theorem 4.5. One might argue that 2 is better than 3 because Jacobian sharing might protect privacy after the Jacobian sparsification techniques discussed in supplementary material E. \n\n\nTherefore, it is hard to imagine a scenario where 2 has any benefit in terms of communication overhead. To show 2 is better than 1 and 3 in terms of communication complexity, authors should show acc vs. communication bit curve instead of acc vs. communication round curve for all benchmark algorithms. The results are only seen in Figure 16 in the supplementary material, which does not show an advantage of 2 over 1.\n\nAlso, in the implementation, it seems that the authors use the eigen-decomposition of $H$ to evaluate the exponential map. This is very unconventional in deep learning, where people usually use gradient descent. A running time comparison would be needed to inform the practitioners.", "relation_to_broader_scientific_literature": "NTK is an interesting theoretical topic in deep learning literature, and so is DFL.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "There are some typos in Algorithm 4: Weight Evolution.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l2rHUw1Ga3", "reviewer": "Reviewer_Qo5R", "summary": "This paper studies decentralized federated learning and leverages the neural tangent kernel to improve performance and convergence under heterogeneous settings. The proposed method is evaluated on three public datasets and shows improved performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer Qo5R\n\n1. **\"More recent literature should be included in the discussion.\"**\n\nWe thank the reviewer for this suggestion. In the originally submitted manuscript, we had included comparisons with DFedSAM (Shi et al., 2023), which is, to our knowledge, the most recent, relevant DFL baseline for our comparisons. Additionally, we plan to update our manuscript by referencing more recent DFL papers such as Yuan et al. (2024) and Guo et al. (2025) throughout the introduction and related work sections. We also plan to cite recent personalized FL works for context but do not include experimental comparisons due to differing problem settings.\n\n2. **\"The motivation for leveraging NTK in DFL needs to be justified.\"**\n\nOur motivation for leveraging NTK in DFL arises directly from the challenge of statistical heterogeneity, which is even more pronounced in DFL than in centralized FL. The expressiveness of NTK-based approaches offers a natural solution to address this heterogeneity, making them particularly suitable for decentralized settings. This motivation was highlighted in the introduction of the original manuscript, where we discussed the previously demonstrated robustness of NTK-based methods to statistical heterogeneity in centralized settings (Yu et al., 2022; Yue et al., 2022).\n\n\n3. **\"The proposed method needs to calculate the Jacobians for each neighbor, which suffers a significant computational cost.\"**\n\nWe share the reviewer’s concern regarding the computational cost of calculating Jacobians for each neighbor. To mitigate this issue, we had outlined several tactics in Appendix D of the original manuscript to reduce overall computation, including data subsampling and Jacobian batching.  \n\nTo further address this concern, we plan to offer another mitigation strategy. When clients are arranged in a clustered topology—where all clients in a cluster share a common aggregated weight—each client only needs to compute a single set of Jacobians per round. This significantly reduces per-client computational cost. After we ran additional experiments, we saw that NTK-DFL maintains its test accuracy under this topology, demonstrating that this optimization does not degrade performance. Additionally, the weight evolution step can be delegated to a single representative client within each cluster, further alleviating the computational burden on the remaining clients. Together, these strategies **decouple computation and communication overhead from the number of neighbors**. We plan to add a detailed description of this approach in Appendix D.2.\n\n4. **\"What are the specific challenges when applying the NTK to the decentralized FL setting compared to FL?\"**\n\nThe unique challenges introduced by applying the NTK to decentralized FL are as follows. First, there is increased communication overhead, as clients communicate with their neighbors rather than a central server. Second, computational costs rise, since clients must compute Jacobians with respect to their own weights, as well as each of their neighbors’ weights. The original approach by Yue et al. (2022) uses the full batch of client data each round, which can induce memory constraints. To address these challenges, we had introduced in the submitted manuscript Jacobian batching and datapoint subsampling to reduce memory and communication overhead, finding that NTK-DFL is resilient to aggressive compression.\n\n5. **“The network topologies need to be clearly specified”**\n\nWe had specified the specific baseline topology that we used for experiments in the original manuscript under *Network Topologies*: \n>“A sparse, time-variant $\\kappa$-regular graph with $\\kappa=5$ was used as the standard topology for experimentation…”\n\n\n6. **\"Is this method applicable to different network topologies? […] what would be the results with a fixed topology?\"**\n\nIn the original manuscript, we had studied the performance of NTK-DFL across the ring topology, d-regular topology, and Erdos-Renyi random topology in Figure 8 of Appendix B. We also studied the effect of various levels of topological sparsity in Figure 3. Lastly, we had performed an experiment on a fixed d-regular topology in Figure 11 from Appendix B and compared it with DFedAvg (the next best performing baseline). Both models suffered slight performance degradation, though NTK-DFL still outperforms DFedAvg. We point out that the time-varying topology has been used in recent DFL work, e.g., Shi et al. (2023).  \n\n**References**\n- Guo, P. et al. Chapter 13 - enhancing mri reconstruction with cross-silo federated learning. 2025.\n- Shi, Y. et al. Improving the model consistency of decentralized federated learning. ICML, 2023.\n- Kai Yue, et al. Neural tangent kernel empowered federated learning. ICML, 2022.\n- Yuan, L., et al. Decentralized federated learning: A survey and perspective. IEEE IOT Journal. 2024.\n- Yu, et al. Convexifying federated learning using bootstrapped neural tangent kernels. Neurips, 2022.", "claims_and_evidence": "The claims are supported by method design and experimental validations.", "methods_and_evaluation_criteria": "The proposed method and evaluation make sense in general but lack some comparisons regarding the computational cost and different topologies.", "theoretical_claims": "The theoretical claims and proofs seem to be correct.", "experimental_designs_or_analyses": "The experimental design and analysis are sound in general.", "relation_to_broader_scientific_literature": "This paper contributes to the general federated learning community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strength**\n- Decentralized federated learning with statistical heterogeneity is a challenging topic.\n\n- The theoretical analysis shows better convergence than DFedAvg.\n\n- The proposed method shows better performance than the compared methods.\n\n\n**Weakness**\n- The literature discussed in this paper was mainly published around 2022; more recent literature should be included in the discussion.\n\n- The motivation for leveraging NTK in DFL needs to be justified.\n\n- The proposed method needs to calculate the Jacobians for each neighbor, which suffers a significant computational cost.\n\n- The network topologies need to be clearly specified, and different topologies (e.g., line, ring) should be explored in experiments.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- What are the specific challenges when applying the NTK to the decentralized FL setting compared to FL? Please further elaborate.\n- Is this method applicable to different network topologies? In addition, it may not be true in the real-world to have topology changes every communication round, what would be the results with a fixed topology?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HOQr2sYDC8", "reviewer": "Reviewer_Zq37", "summary": "The paper proposes NTK-DFL, a decentralized federated learning method that uses the neural tangent kernel (NTK) to mitigate the effect of data heterogeneity in the clients. The authors prove the convergence guarantee of the proposed method and show that it outperforms existing methods such as DFedAvg in the numerical experiments.\n\n## update after rebuttal\nAs the authors claim, the proposed method might be effective in some settings, but I feel that such situations are rare in FL scenarios. Regarding novelty, I acknowledge that this paper is the first work to extend NTK-FL to decentralized scenarios, but the algorithm is very similar to the central one. I think this paper is borderline, and I have decided to maintain my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer Zq37\n\n1. **\"The proposed method incurs a high communication cost because clients send Jacobian matrices to their neighbors. This is undesirable since communication cost is often a bottleneck in decentralized federated learning.\"**\n\nWe acknowledge the reviewer’s concern regarding communication overhead. Indeed, NTK-DFL inherently transmits more data per communication round compared to more traditional approaches such as FedAvg, due to the sharing of Jacobian matrices rather than gradients or weights alone. \nHowever, we emphasize that the number of communication rounds remains an important practical metric, particularly in scenarios involving:\n\n- **Large per-round communication latency**, where fewer rounds can significantly reduce overall training time (e.g., compression or encryption of weights/gradients, heavy preprocessing of input data).\n- **Limited device availability**, in which fewer rounds allow more efficient training when devices are intermittently available.\n- **High bandwidth applications**, where ample network bandwidth (e.g., gigabit home internet) can accommodate  a large data volume for each communication round, making the number of communication rounds the dominant factor in training efficiency.\n- **Synchronization delays**, where each round must wait for all devices to complete computation, with the slowest device bottlenecking progress, thus making the number of communication rounds an important factor.\n\nWe do not claim to be a silver bullet for federated learning. Instead, we provide an alternative method for when minimizing communication rounds is critical. NTK-DFL can also be deployed in conjunction with gradient-based approaches, reducing the number of communication rounds in the early stages of training.\n\nIn addition, for application scenarios where NTK-DFL is more appropriate while communication overhead is a major concern, we had also provided several mitigation tactics in the submitted manuscript to improve communication efficiency, including datapoint subsampling and compression techniques, e.g., sparsification and quantization. These tactics have shown substantial reductions in communication costs, allowing for trade-offs between expressiveness and overhead (see Appendix D.2). Also, please see ***Response 3 to Reviewer Qo5R*** for a description of how we **decoupled communication and computation overhead from scaling with the number of neighbors**.\n\n2. **\"Novelty is limited since the proposed method is a straightforward extension of NTK-FL to the decentralized setting.\"**\n\nWhile our method builds on NTK-FL, we are the first to extend it to the decentralized federated learning (DFL) setting, where the lack of a central aggregator makes the adaptation nontrivial; i.e., not all FL approaches work in DFL. Surprisingly, we also found that combining NTK-based weight evolution with decentralized model averaging yields an exciting synergy: it promotes beneficial inter-client variance and markedly improves generalization in heterogeneous settings. Lastly, we propose strategies for overhead minimization such as Jacobian batching, data subsampling, and the use of a clustered topology that are not present in NTK-FL.\n\n3. **\"What is the main difficulty in extending NTK-FL to the decentralized setting?\"**\n\nPlease see ***Response 4 to Reviewer Qo5R***.\n\n4. **\"I would like to see a clear comparison of the communication and computational costs between the proposed method and the baselines.\"**\n\nPlease see ***Response 2 to Reviewer oV9C*** for a clear comparison of communication cost versus the test accuracy for NTK-DFL and DFedAvg, the best-performing baseline. \nRegarding computational cost, computing the Jacobian in the NTK-based approach “does not incur additional client computational overhead compared to [D]FedAvg, since calculating the Jacobian tensor enjoys the same computation efficiency as computing aggregated gradients” (Yue et al., 2022). While the decentralized setting requires each client to compute Jacobians for all neighbors, adopting a clustered topology (as described in ***Response 3 to Reviewer Qo5R***) **prevents both the computational and communication burden of the Jacobians from scaling with the number of neighbors**. Lastly, analogous to local epochs in gradient-based approaches, our runtime for weight evolution scales as $O(t)$, where $t$ represents the number of local evolution steps.\"\n\n5. **\"at stage 1 in Fig. 1 seems to be a typo of $w_i^{(k)}$.\"**\n\nWe thank the reviewer for their careful attention to detail. We have fixed the notation.\n\n**References**\n- Kai Yue, Richeng Jin, Ryan Pilgrim, Chau-Wai Wong, Dror Baron, and Huaiyu Dai. Neural tangent kernel empowered federated learning. In Proceedings of the 39th International Conference on Machine Learning, Jul 2022.", "claims_and_evidence": "Claims are supported by theoretical results or numerical experiments.", "methods_and_evaluation_criteria": "The proposed method incurs a high communication cost because clients send Jacobian matrices to their neighbors. This is undesirable since communication cost is often a bottleneck in decentralized federated learning.", "theoretical_claims": "I did not rigorously check the correctness of the proofs.", "experimental_designs_or_analyses": "The experimental setting is reasonable.", "relation_to_broader_scientific_literature": "The proposed method is a decentralized extension of NTK-FL proposed by Yue et al. While the extension appears straightforward, the theoretical analysis may be non-trivial.", "essential_references_not_discussed": "Nothing in particular", "other_strengths_and_weaknesses": "- strength\n    1. The proposed method provably mitigates the effect of data heterogeneity in the clients.\n    2. The numerical experiments are detailed.\n- weakness\n    1. Communication cost is larger than existing methods since clients send Jacobian matrices to neighbors.\n    2. The computational cost at each client is also large.\n    3. Novelty is limited since the proposed method is a straightforward extension of NTK-FL to the decentralized setting.", "other_comments_or_suggestions": "- $\\bar{w}_i^{(k)}$ at stage 1 in Fig. 1 seems to be a typo of $w_i^{(k)}$.", "questions_for_authors": "1. What is the main difficulty in extending NTK-FL to the decentralized setting?\n2. I would like to see a clear comparison of the communication and computational costs between the proposed method and the baselines.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper applies the NTK in FL, to guide the client models via a synergy of NTK evolution and FL aggregation. It is shown that the proposed method can scale well through different datasets, topologies and heterogeneities.
Decision: Accept (poster)