DICE: Data Influence Cascade in Decentralized Learning
Authors: Tongtian Zhu
Wenhao Li
Can Wang
Fengxiang He
Venue: ICLR 2025 Poster (2025)
Keywords: Decentralized Learning
Data Influence
Data Valuation
Contribution Attribution
Incentive Mechanism
Primary area: interpretability and explainable AI
TL;DR: We introduce DICE, the first framework for quantifying data influence in fully decentralized learning. Our analysis shows that data influence cascades through the communication network like ripples, shaped by data, communication, and loss curvature.

Abstract: Decentralized learning offers a promising approach to crowdsource data consumptions and computational workloads across geographically distributed compute interconnected through peer-to-peer networks, accommodating the exponentially increasing demands. However, proper incentives are still in absence, considerably discouraging participation. Our vision is that a fair incentive mechanism relies on fair attribution of contributions to participating nodes, which faces non-trivial challenges arising from the localized connections making influence ``cascade'' in a decentralized network. To overcome this, we design the first  method to estimate Data Influence CascadE (DICE) in a decentralized environment. Theoretically, the framework derives tractable approximations of influence cascade over arbitrary neighbor hops, suggesting the influence cascade is determined by an interplay of data, communication topology, and the curvature of loss landscape.DICE also lays the foundations for applications including selecting suitable collaborators and identifying malicious behaviors.
Project page is available at https://raiden-zhu.github.io/blog/2025/DICE.

Published as a conference paper at ICLR 2025
DICE: DATA INFLUENCE CASCADE IN DECENTRAL- IZED LEARNING
Tongtian Zhu, Wenhao Li & Can Wang Zhejiang University (rai den, wenhao-li, wcan)ezju edu.
Fengxiang He University of Edinburgh F. HeHedd ac.uk
ABSTRACT
Decentralized learning offers promising approach crowdsource data consump- tions and computational workloads across geographically distributed compute interconnected through peer-to-peer networks, accommodating the exponentially increasing demands. However, proper incentives are still in absence, considerably discouraging participation. Our vision is that fair incentive mechanism relies on fair attribution of contributions to participating nodes, which faces non-trivial challenges arising from the localized connections making influence "cascade" in a decentralized network. To overcome this, we design the first method to estimate Data Influence CascadE (DICE) in a decentralized environment. Theoretically, the framework derives tractable approximations of influence cascade over arbitrary neighbor hops, suggesting the influence cascade determined by an interplay data, communication topology, and the curvature of loss landscape. DICE also lays the foundations for applications including selecting suitable collaborators and identifying malicious behaviors. Project page is available at & DICE.
INTRODUCTION
Large language models (LLMs) have seen remarkable progress in recent years (Guo et al., 2025; OpenAl, 2024; Dubey et al., 2024; DeepMind, 2024; Anthropic, 2024), surpassing human key benchmarks (Maslej et al., 2024). The compute scaling, highlighted by Ho et al. (2024) as a major reason of the successes, is estimated by Epoch AI to increase four to five times annually in cutting- edge models (Sevilla & Roldán, 2024). This dramatic computational demand requires substantial financial investments; for example, training OpenAl's GPTT4 requires approximately $78 million in compute costs (Maslej et al., 2024). Such exorbitant expenses are far beyond the affordability of most smaller players, making tech giants increasingly dominant.
Currently, large-scale training and inference processes are primarily performed in expensive data centers. Decentralized training, echoing swarm intelligence (Bonabeau al., 1999; Mavrovouniotis et al., 2017), offers a cost-efficient alternative avenue by crowd-sourcing computational workload decentralized compute nodes (Yuan al., 2022: Jaghouar et al., 2024). One notable example showcasing decentralized computing's 's computational potential is the Bitcoin eco-system which virtually distributes jobs requiring instantaneous 16 GW power consumption (CCAF, 2023).
Despite profound potential advantages, contributing to decentralized training incurs non-negligible costs for participants, raising a natural question: What motivates edge participants to engage in decentralized training? Game theory suggests that when appropriate incentives exist, self-interested (rational) players can be keen to contribute for socially desirable outcomes. is thus essential to design proper incentive mechanism to unleash the collectively massive computational potential of decentralized nodes. Our vision is that such incentive mechanism relies on accurate quantification of contributions from players. This leads the following problem:
How to quantify individual contributions in decentralized learning?
Quantifying the contributions (or "influence") to the learned model has been well studied in the centralized paradigm (Koh & Liang, 7017a; Pruthi et al., 2020). However, it is still largely untouched in understanding and measuring data influence in fully decentralized environments. Unlike centralized

---

Published as a conference paper at ICLR 2025
Stem
Higher data influense direct node
Lower data influence distant node
Figure 1: A semantic visualization of influence cascade in decentralized learning with ResNet- 18 on CIFAR-10 10. The illustration depicts a 16-node communication topology (see Figure D.25), where node sizes represent DICE-E influence scores (see Theorem 2). Influence originates from the stem node and propagates through the network, weakening over distance, akin to "ripples in water". This highlights how data contribution extends beyond local nodes, shaped by the communication topology.
scenarios, where data influence is computed on single model statically, decentralized learning relies on localized, indirect communications on perhaps sparse network the influence of data on one node impacts its own model, propagates to its neighbors through iterative parameter exchanges, and cascades to multi-hop neighbors. We term this mechanism as cascading influence (see Figure 1). Existing data influence estimators tailored for centralized settings assume that influence is computed within a single model and do not account for its recursive propagation through parameter exchange. As result, they are not applicable to decentralized learning, where data influence extends beyond direct neighbors to multi-hop neighbors.
To address this challenge. propose Data Influence CascadE (DICE), the first work for measuring the influence in a decentralized learning environment. We make contributions as follows:
DICE introduces the concept of ground-truth data influence for decentralized learning, integrating direct and indirect contributions to capture influence propagation across multiple hops during training (see Definition 3).
Conceptual contributions:
Theoretical contributions: Building on this foundation, derive tractable approximations of ground-trutt DICE for an arbitrary number of neighbor hops, establishing foundational framework systematically characterize the flow of influence across decentralized networks. These theoretical results uncover, for the first time, that data influence in decentralized learning extends beyond the data itself and the local model, as seen centralized training. Instead, a joint product of three critical factors: the original data, the topological importance of the data keeper, and the curvature information intermediate nodes mediating propagation (see Theorem 2).
We anticipate that our DICE framework will pave the way for novel incentive mechanism designs and the establishment of economic opportunities for decentralized learning, such as data and parameter markets. DICE also holds significant potential to address critical challenges of identifying new suitable collaborators, and detecting free-riders. We envision these applications will contribute to scalable, autonomous, and reciprocal decentralized learning eco-system.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Data Influence Estimation. As high-quality data becomes increasingly critical in modern machine learning (Hoffmann et al., 2022; Penedo et al., 2023; Li et al.. 2023; Villalobos et al., 2024), understanding its influence has emerged key research direction (Sorscher et al., 2022; Grosse et al., 2023). Data influence estimation quantifies the contribution of training data to model predictions (Chen et al.. 2024a; Ilyas et al., 2024). supporting incentive mechanisms and applications in few-shot learning (Park et al., 2021), dataset pruning (Yang et al., 2023), distillation (Loo et al., 2023). fairesss (Li & Liu, 2022), machine unlearning (Sekhari et al., 2021), explainability (Koh & Liang, 2017b; Grosse et al., 2023), and security (Demontis et al., 2019; Hammoudeh & Lowd, 2022).
Existing methods fall into static and dynamic categories. Static approaches. including retraining- based methods (e.g., leave-one-ous out (Cook, 1977), Shapley value (Shapley, 1953), Datamodels (Ilyas et al., 2022)) and one-point methods (e.g., influence functions (Koh Liang, 2017b)), estimate influence post-training. While theoretically grounded, these methods cannot characterize dynamic influence during training. Dynamic approaches address this limitation by tracking model parameter evolution (Charpiat et al., 2019). Notable methods include TracIn (Pruthi et al., 2020) and Run Data Shapley (Wang et al., 2024), which average gradient similarities over time. Recent advances (Nickl al., 2023) leverage memory- memory-perturbalte perturbation equations extend dynamic influence estimation to various optimization algorithms. For a more detailed background, please refer to Appendix A.1.
However, existing methods primarily foeus centralized training. To the best of our knowledge, the most closely related work is by Terashita & Hara (2022), who propose a decentralized hyper-gradient method and offer novel insights into using dyper-gradients compute data influence. Nevertheless, their estimation method is static and cannot capture the influence cascade in decentralized training. In contrast, our framework, DICE, is specifically designed for fully decentralized environments, providing fine-graine fine-grained characterization of influence propagation unique to these settings.
Incentivized Decentralized Learning. Most existing incentive mechanisms for collaborative learning are designed for federated learning (Zeng et al., 2021). For instance, Wang et al. (2023b) propose an Incentive Collaboration Learning (ICI.) framework to promote collaboration. Their focus is on mechanism design rather than the precise quantification of individual contributions. In federated learning, the Shapley value has been effectively utilized to quantify participant contributions (Jia et al., 2019; Ghorbani & Zou, 2019; Wang al., 2019; 2020). Our approach differs fundamentally in two key aspects: first, we focus on fully decentralized settings without central servers, although framework supports federated learning scenarios (see Algorithm 1); second, our work considers influence cascade between participants, an completely new perspective that has not been explored in existing literature. Regrading decentralized learning,
only aware of the work by Yu et al. (2023) presenting a blockchain-based incentive mechanism for fully decentralized learning. However, their mechanism relies on smart contracts and differs from ours.
NOTATIONS AND PRELIMINARIES
This section introduces notations and essential preliminaries for decentralized learning. This work focuses the most studied form of decentralized learning: data parallelism with only peer-level communication. For more detailed background, please refer to Appendix A.2.
We consider a general personalized distributed optimization problem over connected graph G = (V, E), where V represents the set of participants and denotes the communication links between them. The participants collaboratively minimize weighted sum of local personalized objectives (T. Dinh et al., 2020; Hanzely & Richtárik, 2020):
min (O.ER"))eeN
L( &
Σ Lu KEV
where 9R 2 0 with Σkev gs = 1, and each local objective (. = EZ,ND, [L(::: xx)] is defined by the expectation over the local data distribution Dr. Empirical risk minimization involves
3

---

Published as a conference paper at ICLR 2025
optimizing the sample average approximation:
(( = Σ
qu (Ok))
where (xx)
L(O Zk.)
T&k ==1
KEV
Here, ng is the number of samples in participant k, and {zk,
12 are drawn from Dx-
Communication
Local training en Ovio, A3i
Local training
Participunt
Communication
Participants
(a) Server-based Learning
(b) Decentralized Learning
Figure 2: A comparative illustration of server-based learning versus decentralized learning.
Decentralized learning aims to minimize the global objective ((0) Σ kEv (Ox) with only local computations and gossip communications among neighboring participants (Tsitsiklis et al., 1986; Nedic & Ozdaglar, 2009). The communication protocol is governed by weighted adjacency matrix W E [0, 1]mxn, where Wk.j J 2 O represents the strength of connection from participant j to participant with Wa.j if (k,j) E E. This matrix characterizes the communication topology. thereby defining how information propagates through the network (see Figure A. In this paper, W designed to be row-stochastic, satisfying プ Wk.j 1 for all V Decentralized learning alternates between local optimization and gossip- based parameter aggregation, shown below:
Algorithm Decentralized Learning with Flexible Gossip and Optimization
Require: G EV, {00)kvv. hKev, optimizer Ok, number of communication rounds T, and mixing matrix distributions W (vt [T')
1: for = 1 to T do in parallel for all participants k V
2: Local Update:
3: Sample =1 DR, update parameters with optimizer Oę: 0² Ox
4: Gossip Averaging:
5: Send 0x + to Wi, > 0} and receive 0 from {j Wkd 0}.
6: Sample W W perform gossip averaging: 0 + ΣjeN(( Wk, 0
End for
Remark 1. Algorithm provides flexible framework for decentralized learning with arbitrary optimizers and randomized gossip. A special case of this framework is decentralized stochastic gradient descent (DSGD) (Yuan et al., 2016a; Lian et al., 2017; Koloskova al., 2020), where the
optimizer O₁ performs simple stochastic gradient step: 0² g+1 0 V'VIL(O =f), with sff being the learning rate. Another notable special case FedAVG (MeMahan et al., 2017), which corresponds to the standard server-based learning setting, where central server collects and averages model updates from all participants each round. Mathematically, this equivalent to using a fully connected and uniform mixing matrix in Algorithm 1, i.c., W K 111" (see Figure 2 for visual comparison and its connection to decentralized learning). Therefore, the framework is applicable to both federated and decentralized learning paradigms, although our primary focus remains on fully decentralized learning without central server.
'The weighted adjacency matrix W is typically assumed to be doubly-stochasie The row-stochastic assumption weaker, yet the convergence of decentralized SGD still guaranteed (Yuan al., 2019; Xin al., 2019).
4

---

Published as a conference paper at ICLR 2025
4 DATA INFLUENCE CASCADES
In this section, we introduce DICE, comprehensive framework for measuring data influence in decentralized environments. Subsection 4. introduces the ground-truth influence measures designed for decentralized learning and Subsection 4.2 provides their dynamic gradient- gradient-based estimations.
4.1 GROUND- TRUTH INFLUENCE IN DECENTRALIZED LEARNING
To ensure logical and coherent flow, we first introduce the fundamental concepts of data influence in centralized settings and then discuss the significant challenges involved in extending these ideas to decentralized environments.
conventional centralized setups, the influence of an individual data instance can be assessed by evaluating the counterfactual change in learning performance through leave-one-out retraining (LOO) (Cook, 1977), defined as follows:
Definition (Leave-one-out 1 Influence).
Lo z') = 2/) - L(0
where z denotes the training data instance under influence assessment, z' the loss-evaluating instance, 0t and 0r_ are the models trained on the entire dataset S and S \ {z}, respectively.
Intuitively, Equation (3) quantifies the influence of z by its individual impact on test loss reduction. smaller LOO value indicates a significant contribution learning, which aligns with the concept that the data influence is reflected in its ability to enhance model performance. LOO influence is often considered as the "gold standard" for evaluating how well influence estimators approximate the ground-truth influence in the data influence literature (Koh & Liang, 20170 Basu et al., 2021).
However, extending LOO to decentralized scenarios introduces non-trivial challenges due to the distributed nature and the localized connections in decentralized learning, reflected in Equation (2) and Algorithm 1. centralized setups, the core idea LOO is to link data influence to variations in loss parameter outcomes. In contrast, decentralized learning systems involve multiple participants sharing model parameters through inter-participant communications. As a result, alterations in model parameters caused by data-level modification propagate throughout the whole network.
natural way to measure the influence one participant in such collaborative environments is through evaluating its contribution to the whole community (Wang et al., 2020; Yu et al., 2020), which aligns with the customer-cemt centric principle (Drucker, 1985) in determining value". In decentralized learning, when a participant transmits its training assets (e.g., model parameters gradients) to neighboring participants-ale akin to offering product-the recipients derive possible utility from these training assets and may provide reciprocal feedback, such as sharing their own assets in return. This dynamic positions the neighbors as "customers", thereby entrusting them with the rights to determine the value of the assets provided by the supplier. With these insights in mind, we recognize that assessing data influence in decentralized scenarios is far more complex, summarized below:
Key observations: In decentralized learning,
I) neighbors who serves as customers hold the rights to determine data influence; 2) data influence is not static but spreads across participants through gossips during training.
Unfortunately, existing static estimators only calculate the loss change after training and thus cannot characterize the dynamic transmission of data influence within the whole decentralized learning community. Based on the above discussion, we posit that "gold-standard" influence measure in decentralized scenarios should satisfy the following requirements:
Quantify community-level influence: Measure the impact of training data instances on the collective
utility of the community.
Depend on training dynamics: Measure the influence based on the training process to characterize the propagation of influence decentralized networks.
Peter Drucker's customer-centric value principle from Innovation and Fatrepreneurship states, "Quality in product service what the supplier puts what the customer gets out
5

---

Published as a conference paper at ICLR 2025
In the following, we introduce the ground -ruth influence measures tailored to the requirements of decentralized environments, termed as the ground-trune trunt inftuence cascade (DICE-GT).
Definition 2 (One-hop Ground-truth Influence). The one-hop DICE-GT value quantifies the influence of data instance 2 from participant j on loss-evaluating instance z' within itself and its immediate neighbors. Formally, for given participant j E V:
(1) (DICE-G(=J qi
Σ q (L(ORt)
- L(0 L(0
REN (1)
direct marginal contribation of 2] in
indirect marginal contribation z] one-hop neighbors
where 0 denotes the updated model parameters of after training on =j at iteration f (see Algo-
rithm 1). For each one-hop out-neighbor & -Noul ()), o denotes the averaged model parameters
after receiving updated parameters (0,
+ | > 0} influenced by zJ, while 0 represents the A/,z
K(= = EiENa, 0 + W model parameters of k without the influence from =3, i.e.
The economic intuition behind the DICE-GT value is that captures both the direct marginal contribution of data instance to itself and its subsequent impact immediate neighbors. Specially, the first term L(=';0, 2 ( 03) captures the inter-node direct influence of training data instance z) on the test loss change at node j. which corresponds to the TracInIdeal influence in (Pruthi et al., 2020) designed for centralized scenarios. The second term EKENSH (L(z'; 0 L(z'; 0 Rra)))
measures the intra-node influence unique in decentralized Icarning, which aggregates the indirect influences on all one-hop neighbors, i.e., direct neighbors, of node j.
In decentralized learning environments, data influence propagates not only to immediate neighbors but also to multi-hop neighbors through the communication topology. To characterize this multi-hop influence, we extend the ground-truth influence cascade measure arbitrary r-hop neighbors.
Definition 3 (Multi-hop Ground-trute -trth Influence). The multi-hop DICE-GT value quantifies the cumulative influence of a data instance z on loss-evaluating instance z' across all nodes within r-hop neighborhoods of participant j. Formally, for given participant j E V:
(r) DICE--T (2),,²) -())==²))
=(0)
Σ
qk (L(O)t;;)) L(00T2):2))
REVE ())
where Aodal (j) denotes the of s-hop out-neighbors of (please refer Appendix A.3 for details of high-order neighbors). Here 0 and 0++s represents the parameters of node k at iteration A\=55
when the influence from 2) included and excluded, respectively.
Analogous to Definition 2, the first term captures the direct influence of data z on the loss at node j. The subsequent summation aggregates the indirect influences on all multi-hop neighbors up to steps away from node ja. The reason to measure test loss change at the / + step is that the impact of z)
propagating to k E Andaa (j) requires steps. This layered formulation accounts for the multi-hop cascading effects through the network up to the specified order r.
4.2 DYNAMIC GRADIENT-BASER ESTIMATIONS
To meet the second aforementioned requirement of decentralized learning, design dynamic gradient-based estimators for DICE-GT, called the influence cascade estimations (DICE-E).
We note that the future effect of 2f its lncal model 8 over 2.... iterations included within the indirect influence terms, node inherently considered part arbitrar--onler neighbar.

---

Published as a conference paper at ICLR 2025
Proposition (Approximation of One-hop DICE-GT), The one-hop DICE-GT value (see Defini- tion 2) can be linearly approximated follow:
[ = 9i VI(0); ",,"
(0))
E
D Wh v
2)),
KEAM (
where
O The proof is included in Appendix C.I.
Additivity. The one-hop DICE-E influence measure additive over training instances. Specifically, for a mini-batch Bj from participant j, the total influence is the sum of individual influences:
T (B), x')
Σ E
(2).
(4)
The additivity provides guarantees for efficient computation of DICE-E score for large mini-batches. batches.
We can then extend the influence approximation multi-hop neighbors in decentralized learning and show how the influence of a data instance cascades over the decentralized learning network.
Theorem 2 (Approximation of r-hop DICE-GT). The r-hop DICE-GT influence I(() (see Definition 3) can be approximated follows:
(
Tric (2),)
Σ 1=
Σ
n'qr qs,
W kg
VL z¹) R
communication graph-related term
test gradicnt
P
X
II (1 11--1" 1/ H(O 1: =
2k.
A, 21)
curvature
optimization related term
03. ho j. Here p(e) denotes the set of all sequences where (,(0), 23) O,(0j, (ki...., kp) such that k; E EAull (k._1) for ......P..... (see Definition A.7) and H H( 2(*) is the Hessian matrix of L with respect to O evaluated at 0 and data =k, +t+s For the cases when p= and p = 1, the relevant product expressions are deñned as identity matrices, thereby ensuring that the r-hop DICE-E remains well-defined. Full proof is deferred to Appendix C.3.
Multi-hop DICE-E characterizes the cascading effects of data influence through multiple "layers" of communication. In this context, the influence of data instance from participant, can propagate through a sequence intermediate nodes, reaching participants that are p hops away.
Influence Dynamics: Exponential Decay and Topological Dependency. Theorem 2 demonstrates that the multi-hop influence data instance zj² is governed by the product of communication weights I[ s 1" ks 1 and Hessian-related terms 2 (I- H 1+--1). This indicates that data influence in decentralized learning depends on the curvature of intermediate nodes and decays exponentially with each additional hop. Nodes with higher topological importance (e.g., node j with large Σj = W, propagate their data influence more widely and with greater impact on global utility (see Figure 1). These characteristics underscore the interplay between the original data, the loss landscape curvature, and communication topology in shaping data influence.
4.3 PRACTICAL APPLICATIONS
In idealized scenarios, participants may seek to estimate the influence their high-order neighbors on their local utility improvement. In practice, one-hop DICE-E emerges more suitable choice
7

---

Published as a conference paper at ICLR 2025
due to its computational efficiency. Based on Proposition 1, we derive the peer-level contribution, which we refer to as the proximal inffuence.
Definition 4 (Proximal Influence). The proximal influence of a data instance 2/ from participant on participant & at iteration is defined as follows:
DICER (2), -n VL(0 ;
(5)
This term quantifies the influence of the data instance =; from participant j on the loss reduction experienced by its immediate neighbor k. Importantly, under the information sharing protocol defined in Algorithm 1, participant & has access to gk- Wh.j. k.j- V VI( 25), and V VI =')..* Therefore, each participant can compute the proximal contributions of its neighbors. The proximal influence can be utilized in the following scenarios:
Collaborator Selection, In decentralized learning, local data remains private and only local parameter communication is permitted. The absence of central authority complicates the problem of selecting the most suitable neighbors with high-quality data. Fortunately, DICE offers a mechanism for participants efficiently estimate the contributions of their neighbors with proximal influence. By assessing the proximal influence of their neighbors, participants can identify the potential collaborators that have the most significant positive impact on their learning process.
To ensure reciprocal collaboration (Gouldner, 1960; Fehr & Gächter, 2000; Sundararajan & Krichene, 2023), participants can compute reciprocity factors, which evaluate the mutual balance of influence.
Definition (Reciprocity Factors). The reciprocity factor is defined two forms:
1. Proximal Reciprocity Factor: The reciprocity factor between participants j and k at iteration t is
2k W VL( :²) LVV'L((O) 2
R2
(6)
Ij Wt VL(0) -1;;)) 22) (L 2f)
2. Neighborhood Reciprocity Factor:
To evaluate reciprocity at the community level, the neighbor- hood reciprocity factor for participant at iteration is defined as:
ExENSY 9 Wh VL =² VL x/)) R² = EiENTY W W., VL 21 VL(O): 2/)
The proximal reciprocity factor measures the balance of influence between two participants, with val- ues near unity indicating equitable mutual contributions. Significant deviations suggest an imbalance, helping participants refine their collaboration strategies. The neighborhood reciprocity factor extends this concept to participant's 's local community, evaluating the balance between influence inflow and outflow. These metrics support participants in adjusting their engagement and aids the community in managing membership, such admitting new members excluding underperforming participants.
EXPERIMENTS
This section presents the experimental results, with implementation details outlined in Appendix D.
Influence Alignment We evaluate the alignment between one-hop DICE-GT (see Definition 2) and its oirst-order approximation, one-hop DICE-E (see Proposition 1). One-hop DICE-E ( =²) computed as the sum of mample DICE-E within the mini-bateh B thanks to the additivity (see Equation (4)). DICE-GT (DICE-GT") =²) )') is calculated by measuring the loss reduction after removing By from node j at the 1-th iteration. As shown Figure 3, each plot contains 30 points,
"In decentralized learning literature, it common for each participant to share only local parameters with its neighboring participants (Lian et al.. 2017: Koloskova et al., 2020). We note that sharing local gradients with neighbors maintains the decentralized learning paradigm and does significantly compromise privacy, participant can reconstruct the gradients neighbors using the shared parameters (Mrini al., 2024).
8

---

Published as a conference paper at ICLR 2025
CNN Exponential Graph
CNN Ring Graph
MLP Exponential Graph
MLP Ring Graph
loss
Pire ar.
Predicted diff in loss
Predicted diff In loss
Predicted diff in loss
Predicted dirf in loss
Figure 3: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on 32-node ring graph. Each node uses 512-sample subset of Tiny ImageNet. Models are trained for epochs with batch size of 128 and a learning rate of 0.1.
with each point representing the result of a single comparison of the ground-truth and estimated influence. We can observe from Figure that DICE-E closely tracks DICE-GT under different settings. The alignment becomes even stronger on simpler data set including CIFAR-10 10 and CIFAR-100, as detailed in Appendix D.2. These results demonstrate that DICE-E provides a strong approximation of DICE-GT, achieving consistent alignment across datasets (CIFAR-10, 10, CIFAR-100 and Tiny ImageNet) and model architectures (CNN and MLP). Further validation this alignment is provided in Appendix D.2 corroborate the robustness of one-hop DICE-E under changing batch sizes, learning rates, and training epochs.
Anomaly Detection DICE identifies malicious neighbors, referred to as anomalies, by evaluating their proximal influence, which estimates the reduction in test loss caused by single neighbor. A high proximal influence score indicates that neighbor increases the test loss, negatively impacting the learning process. In our setup, anomalies are generated through random label flipping or by adding random Gaussian noise to features, please kindly refer to (Zhang et al., 2024). Figure illustrates that the most anomalies (in red) are readily detectable with proximal influence values. Additional results in Appendix D.3 further validate the reliability of this approach.
ResNet-18 CIFAR-10
CNN on CIFAR-10
MLP on CIFAR-10
Proximal Influence
0001
48000
-1:04
Figure& 4: Anomaly detection exponential graph with 32 nodes. Each node uses 512-sample
subset of CIFAR-10. Models are trained for 5 epochs with a batch size of 128 and learning rate of 0.1. 32-node exponential graph, each participant connects with 5 neighbors, where the neighbor in red is set as an anomaly by random label flipping, while the other four are normal participants.
Influence Cascade The topological dependency of DICE-E in our theory reveals the "power asymmetries" (Blau, 1964; Magee Galinsky, 2008) decentralized learning. To support the theoretical finding, we examine the one-hop DICE-E values of the same batch on participants with vastly different topological importance. Figure 1 illustrates the one-hop DICE-E influence scores of an identical data batch across participants during decentralized training of a ResNet- 18 model on the CIFAR-10 10 dataset. Node sizes represent the one-hop DICE-E influence scores, quantifying how single batch impacts other participants in the network. The dominant nodes (e.g., those with larger outgoing communication weights in W) exhibit significantly higher influence, as shown in Appendix D.4. These visualizations underscore the critical role of topological properties in shaping data influence in decentralized learning, demonstrating how the structure of the communication matrix W determines the asymmetries in influence.

---

Published as a conference paper at ICLR 2025
6 CONCLUSION AND FUTURE WORK
In this paper, we introduce DICE, the first comprehensive framework for quantifying data influence in fully decentralized learning environments. By modeling influence propagation across multiple hops, DICE reveals how local data contributions extend beyond immediate neighbors to reach -on-anjacent neighbors. Mathematically, DICE formalizes how data influence cascades through the communication network, uncovering for the first time the intricate interplay between original data, communication topology, and the curvature of the optimization landscape shaping data influence.
Future Work. Beyond its theoretical contributions, DICE holds significant potential for practical applications. By identifying influential contributors, DICE provides foundation for designing fair incentive schemes that ensure equitable attribution of contributions, thereby promoting broader participation in decentralized learning. Moreover, DICE could contribute to the development of decentralized markets, including data markets (Huang al., 2023), parameter markets (Fallah et al., 2024), and compute markets (Kristensen et al., 2024), which serve as fundamental building blocks of broader decentralized learning ecosystem.
ACKNOWLEDGMENT
This work supported by the National Natural Science Foundation of China (No. 62476244), ZJU- China Unicom Digital Security Joint Laboratory and the advanced computing resources provided by the Supercomputing Center of Hangzhou City University.
REFERENCES
Youssef Allouah, Anastasia Koloskova, Aymane El Firdoussi, Martin Jaggi, and Rachid Guerraoui. The privacy power of correlated noise decentralized learning. In Proceedings of the 4lst International Conference Machine Learning, volume 235, pp. 1115-1143, 2024.
Anthropic. Introducing claude 3.5 sonnet. Anthropic News, 2024.
URL https: //www.
anthropic. www.coo///eeeeee www.///w///aa/ee/eeeeeeeoo 3-rale---555 3-5-sonnet..
Samyadeep Basu, Xuchen You, and Soheil Feizi. On second-order group influence functions for black-box predictions. In Proceedings of the 37th International Conference on Machine Learning. 2020.
Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations, 2021.
Peter M. Blau. Exchange and power in social life. 1964.
Eric Bonabeau, Marco Dorigo, and Guy Theraulaz. Swarm Intelligence: From Natural to Artificial Systems. Oxford University Press, 1999.
Marco Bornstein, Tahseen Rabbani, Evan Z Wang, Amrit Bedi, and Furong Huang. SWIFT: Rapid decentralized federated learning via wait-free model communication. The Eleventh International Conference Learning Representations, 2023.
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 568- Association for Computational Linguistics, 2023a.
Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. Distributed inference and fine-tuning of large language models the internet. In Advances in Neural Information Processing Systems, 2023b.
Ying Cao, Zhaoxian WU, Kun Yuan, and Ali H Sayed. On the trade-off between flatness and optimization in distributed leamnnn. arXiv preprint arXiv:2406.2000 2024.
10

---

Published as a conference paper at ICLR 2025
CCAF.
Cambridge bitcoin electricity consumption index (CBECT). cbeci, 2023.
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. Advances in Neural Information Processing Systems, 2019.
Samprit Chatterjee, R. Dennis Cook, and Sanford Weisberg. Residuals and influence in regression. 1982.
Daiwei Chen, Jane Zhang, and Ramya Korlakai Vinayak. Unraveling the impact training samples. In ICLR Blogposts 2024, 2024a. URL httpss//ccclllbboooooo wwtposttsggghuut. github. oappp/44eeggi.iaaaaaaaaaeeeeaaaa httpgo/g/..iiiiiioo///o//imiimmp
https://iclr-
Lesi Chen, Haishan Ye, and Luo Luo. An efficient stochastic algorithm for decentralized nonconvex- strongly-concave minimax optimization. International Conference Artificial Intelligence and Statistics, 2024b.
Xuxing Chen, Minhui Huang, Shiqian Ma, and Krishna Balasubramanian. Decentralized stochastic bilevel optimization with improved per-iteration complexity. Proceedings of the 40th Interna- lional Conference on Machine Learning, volume 202, pp. 4641-4671, PMLR, 2023.
R. Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1): 15-18, 1977.
Edwige Cyffers, Aurélien Bellet, and Jalaj Upadhyay. Differentially private decentralized learning with random walks. In Proceedings of the 41 International Conference on Machine Learning. volume 235, pp. 9762 9783, 2024.
Google DeepMind. Introducing gemini 2.0: our new ai model for the agentic era. The
Keyword, 2024. URL https: /bllg./////o./o htw.loogggggleeeeecoloooo... google-gemint wtteee-eeiiiii------eeeeeee ai-uudaae aaiuuddate-decmmeerr 2atiideerr2224///0 20242 20024/gemmii-225 flash.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX Security Symposium (USENIX Security 19), 2019.
Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models. arXiv preprint arXiv:2311.08100 2023.
Peter F. Drucker. Innovation and Entrepreneurship. Perennial Library, 1985.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Mathieu Even, Anastasia Koloskova, and Laurent Massoulie. Asynchronous SGD on graphs: unified framework for asynchronous decentralized and federated optimization. In Proceedings of The 27th International Conference Artificial Intelligence and Staristics, 2024.
Alireza Fallah, Michael I Jordan, Ali Makhdoumi, and Azarakhsh Malekian. On three-layer data markets. arXiv preprint arXiv:2402.09697, 2024.
Ernst Fehr and Simon Gächter. Fairness and retaliation: The economics of reciprocity. Journal of Economic Perspectives, 14(3):159-881, 2000.
Shaopeng Fu, Fengxiang He, and Dacheng Tao. Knowledge removal in sampling based bayesian inference. In International Conference on Learning Representations, 2022.
Hongchang Gao and Heng Huang. Fast training method for stochastic compositional optimization problems. Advances in Neural Information Processing Systems, 34:25334-2553 2021.
11

---

Published as a conference paper at ICLR 2025
Hongchang Gao, Bin Gu, and My T. Thai. On the convergence of distributed stochastic bilevel optimization algorithms over a network. In Proceedings of The 26th International Conference on Artificial Intelligence and Staristics, volume 206, pp. 9238 9281. PMLR, 2023.
Anissa Gardizy and Amir Efrati. Microsoft and OpenAl plot $100 billion stargate Al supercom- puter. The Information, 2024. URL https: / www. the nfrrat in coom/rrticll mi www.ooo-tannddooooo openai -plot - 00-000000000 bi htt-bblll------rattteeee a watpeeeaa-suuuurssuee supercomputer..
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Advances in Neural Information Processing Systems, 2020.
Alvin W. Gouldner. The norm of reciprocity: A preliminary statement. American Sociological Review, 25(2):161-17,, 1960.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage. Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308. 03296, 2023.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. Proceedings of the 37th International Conference on Machine Learning, 2020.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprins arXiv:2501.12444 2025.
Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. FastIF; Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, 2021.
Mert Gurbuzbalaban, Yuanhan Hu, Umut Simsekli, Kun Yuan, and Lingjiong Zhu. Heavy-tail phenomenon in decentralized sgd. arXiv preprint arXiv:2205.0666 2022.
Zayd Hammoudeh and Daniel Lowd. Identifying a training-set attack's 's target using renormalized influence estimation. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022.
Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: a survey. Machine Learning, 1 113(5)23311-444 24003, 2024.
Frank Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346))383-33) 1974.
Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint XXX::2002.005166, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, 2016.
Lie He, Sai Praneeth Karimireddy, and Martin Jaggi. Byzantine-robus: decentralized learning via clippedgossip. arXiv preprint arXiv:2202.01545, 2022.
Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. Algorithmic progress language models. arXiv preprint arXiv:2403.0581 055812 2024.
12

---

Published as a conference paper at ICLR 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, 2022.
Tzu-Heng Huang, Harit Vishwakarma, and Frederic Sala. Train 'n trade: Foundations of parameter markets. In seirth- Conference on Neural Information Processing Systems, 2023.
Andrew Ilyas, Sung Min Park. Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Data- models: Understanding predictions with data and data with predictions. In Proceedings of the 39th Internationaa Conference on Machine Learning, 2022.
Andrew Ilyas, Kristian Georgiev, Logan Engstrom, and Sung Min (Sam) Park. Data attribution at scale, 2024. URL https: / /ml data inttorial.org.. ICMI 2024 Tutorial.
Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lueas Atkins, Maziyar Panahi, Charles Goddard, et al. Intellect-1 technical report. arXiv preprint arXiv:2412.0113 152, 2024.
Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2021.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song. and Costas J. Spanos. Towards efficient data valuation based on the shapley value. In Proceedings of the Twenty-Secone International Conference on Artificial Intelligence and Statistics, 2019.
Heasung Kim, Hyeji Kim, and Gustavo De Veciana. Clustered federated learning via gradient-based based partitioning. In Proceedings of the 41 International Conference on Machine Learning, 2024.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017a.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, 2017b.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In International Conference on Machine Learning. 2020.
Lingjing Kong, Tao Lin, Anastasia Koloskova, Martin Jaggi, and Sebastian Stich. Consensus control for decentralized deep learning. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Jesper Kristensen, David Wender, and Carl Anthony. Commodification of compute. arXiv preprint wwxii:2406.11261, 2024.
Alex Krizhevsky, G Hinton, al. Learning multiple layers of features from tiny images (tech. rep.). University of Toronto, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
Batiste Le Bars, elree Bellet, Mare Tommasi, Erick Lavoie, and Anne-Marie Kermarrec. Refined convergence and topology learning for decentralized sgd with heterogeneous data. In Proceedings of The 26th International Conference on Artificial Intelligence and Statisties, volume 206, pp. 1672-1702. PMLR, 25-27 Apr 2023.
Batiste Le Bars, Aurélien Bellet, Mare Tommasi, Kevin Scaman, and Giovanni Neglia. Improved stability and generalization guarantees of the decentralized SGD algorithm. Proceedings of the 4 st International Conference Machine Learning, 2024.
13

---

Published as a conference paper at ICLR 2025
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25 2324, 1998.
Peizhao Li and Hongfu Liu. Achieving faimess at no utility cost via data reweighing with influence. In Proceedings of the 39th International Conference on Machine Leurning. 2022.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint (rXiv:2309.05633. 2023.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu, Can decentralized study for decentralized parallel stochastic
algorithms outperform centralized algorithms? gradient descent. In Advances in Neural Information Processing Systems, 2017.
Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning, 2018.
Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund. Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. Consent crisis: The rapid decline of the ai data commons. arXiv preprint arXiv:2407. 14933, 2024.
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In Proceedings of the 40th International Conference on Machine Learning, pp- 22649-22674, 2023.
Cassio G. Lopes and Ali H. Sayed. Diffusion least-mean squares over adaptive networks: Formulation and performance analysis. IEEE Transactions on Signal Processing, 56(7). 2008
Yucheng Lu and Christopher De Sa. Optimal complexity in decentralized training. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Joe C Magee and Adam D Galinsky. Social hierarchy: The self-reinforcing nature of power and status. Academy of Management Annals, 2(1):351-398, 2008
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint 0rXv:22002.10619 2020.
Enrique Tomás Martínez Beltrán, Mario Quiles Pérez, Pedro Miguel Sánchez Sánchez, Sergio López Bernal, Gérôme Bovet, Manuel Gil Pérez, Gregorio Martínez Pérez, and Alberto Huertas Celdrán. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and chal- lenges. IEEE Communications Surveys & Tutorials, 25(33:29833301
2023.
Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Brynjolfsson, John Etchemendy. Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark. The AI index 2024 annual report. Technical report, AI Index Steering Committee, Institute for Cenaan- AI, Stanford University, 2024.
Michalis Mavrovouniotis, Changhe Li, and Shengxiang Yang. A survey of swarm intelligence for dynamic optimization: Algorithms and applications. Swarm and Evolutionary Computation, 33: 1-17, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.
Abdellah Mrini, Edwige Cyffers, and Aurélien Bellet. Privacy attacks in decentralized learning.
In Proceedings of the 41st International Conference on Machine Learning, 2024.
Giorgi Nadiradze, Amirmojtaba Sabour, Peter Davies, Shigang Li, and Dan Alistarh. Asynchronous decentralized sgd with quantized and local updates. Advances in Neural Information Processing Systems, 2021.
Angelia Nedi'e and Alex Olshevsky. Distributed optimization over time-varying directed graphs. volume 60, pp. 601-615. IEEE, 2014.
14

---

Published as a conference paper at ICLR 2025
Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1):48-65 61, 2009.
Peter Nickl, Lu Xu, Dharmesh Tailor, Thomas Möllenhoff, and Mohammad Emtiyaz E Khan. The meemory-perturaatunn equation: Understanding model's sensitivity to data. In Advances Neural Information Processing Systems, 2023.
OpenAI. Learning to reason with llms. OpenAl Blog, 2024. URL https:////ppe com/ index/learning-tone rectoorresnnwiihheee whtoowwwttt-t- 1mms/.
OpenAl. Announcing the stargate project. announcing-thee sta pergate-projecteo 2025.
https::/openni..com contnd/e//cc/////x/////
Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balanced loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, 2020.
Dominic Richards et al. Graph-dependent implicit regularisation for distributed stochastic subgradient descent. Journal of Machine Learning Research, 2020.
David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning internal representations by error propagation. MIT Press, 1986.
Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. SWARM parallelism: Training large models can be surprisingly communication-effcccee In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 29416-29440. PMLR, 2023.
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 32(8):3710-377 37-22 2021.
Ali H. Sayed. Adaptation, Learning, and Optimization over Networks. Now Publishers, 2014.
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Advances in Neural Information Processing Systems, 2021.
Jaime Sevilla and Edu Roldán. Training compute of frontier ai models grows by 4-5x per year, 2024. URL https: //ep::///epchaaaaa wwwpaaii.rrroloooooo 4- 5x pear
hewkiing-compute-ee -oooo---oooiirer---ii-diiisee-
Lloyd S. Shapley. value for n-person games. In Contributions to the Theory of Games, Volume II,
chapter 17. Princeton University Press, 1953.
Li Shen, Yan Sun, Zhiyuan Yu, Liang Ding, Xinmei Tian, and Dacheng Tao. On efficient training of large-scale deep learning models. ACM Comput. Surv,, 57(3), 2024.
Abhishek Singha, Charles Lua, Gauri Guptaa, Ayush Chopraa, Jonas Blanca, Tzofi Klinghoffera, Kushagra Tiwarya, and Ramesh Raskara. A perspective on decentralizing ai. 2024.
15

---

Published as a conference paper at ICLR 2025
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances Neural Information Processing Systems, 2022.
Tao Sun, Dongsheng Li, and Bao Wang. Stability and generalization of decentralized stochastic gradient descent. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Mukund Sundararajan and Walid Krichene. Inflow, outflow, and reciprocity in machine learning. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Canh T. Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau
envelopes. In Advances in Neural Information Processing Systems, 2020.
Yuki Takezawa, Ryoma Sato, Han Bao, Kenta Niwa, and Makoto Yamada. Beyond exponential
graph: Communication--ffiienn topologies for decentralized learning via finite-time convergence. In Thirly-seventh Conference Neural Information Processing Systems, 2023.
Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. D2: Decentralized training over decentralized data. In International Conference Machine Learning. PMLR, 2018.
Naoyuki Terashita and Satoshi Hara. Decentralized hyper-gradin computation over time-varying directed networks. arXiv preprint arXiv:2210.02129, 2022.
Tsitsiklis, D. Bertsekas, and M. Athans. Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE Transactions on Automatic Control, 31(9):003-12,, 1986.
Vladimir Vapnik and Alexey Chervonenkis. Theory of Pattern Recognition. Nauka, Moscow, 1974.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? Limits of LLM scaling based buman-generated data. In Proceedings of the 4lst International Conference on Machine Learning. 2024.
Thijs Vogels, Lie He, Anastasiia Koloskova, Sai Praneeth Karimireddy, Tao Lin, Sebastian U Stich, and Martin Jaggi. Relaysum for decentralized deep learning on heterogeneous data, Advances in Neural Information Processing Systems, 34:28004-28015,
2021. 2021.
Thijs Vogels, Hadrien Hendrikx, and Martin Jaggi. Beyond spectral gap: The role of the topology in decentralized learning. Journal of Machine Learning Research, 24(355):1-31 2023.
Guan Wang, Charlie Xiaoqian Dang, and Ziye Zhou. Measure contribution of participants in federated learning. In 2019 IEEE International Conference on Big Data (Big Data), pp. 2597-2604, 2019.
Jiachen T Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. Data shapley in one training arXiv preprint arXiv:2406.1 11011, 2024.
Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. CocktailSGD: Fine-tuning foundation models over 500Mbps networks. Proceed- ings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 36058-36076 PMLR, 2023a.
Tianhao Wang, Johannes Rausch, Ce Zhang, Ruoxi Jia, and Dawn Song. A Principled Approach to Data Valuation for Federated Learning, pp. 153-167. Springer International Publishing, 2020.
Xinran Wang, Le, Ahmad Faraz Khan, Jie Ding, and Ali Anwar. A framework for incentivized collaborative learning, arXiv preprint 23X5v:2305.170:: 2023b.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In Proceedings the 41st International Conference on Machine Learning, pp. 54104-54132, 2024.
Wenhan Xian, Feihu Huang, Yanfu Zhang, and Heng Huang. A faster decentralized algorithm for nonconvex minimax problems. Advances in Neural Information Processing Systems, 34: 25865-25877 2021.
16

---

Published as a conference paper at ICLR 2025
Ran Xin, Chenguang Xi, and Usman A. Khan. Frost- fastt-fatti row-stochastic optimization with uncoor- dinated step-sizes. EURASIP Journal on Advances in Signal Processing, 2019(1):1, 2019.
Jie Xu, Wei Zhang, and Fei Wang. sgd:) Asynchronous decentralized parallel stochastic gradient descent with differential privacy, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. In The Eleventh International Conference on Learning Representations, 2023.
Shuoguang Yang, Xuezhou Zhang, and Mengdi Wang. Decentralized gossip- stochastic bilevel optimization over communication networks. Advances in Neural Information Processing Systems, 35:238 252, 2022.
Haoxiang Ye and Qing Ling. Generalization error matters in decentralized learning under Byzantine attacks. IEEE Transactions on Signal Processing, 2025.
Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, and Wotao Yin. Exponential graph is provably efficient for decentralized deep training. In Advances in Neural Information Processing Systems, 2021.
Han Yu, Zelei Liu, Yang Liu, Tianjian Chen, Mingshu Cong. Xi Weng, Dusit Niyato, and Qiang Yang. A fairness-aware incentive scheme for federated learning. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2020.
Haoxiang Yu, Hsiao- Yuan Chen, Sangsu Lee, Sriram Vishwanath, Xi Zheng, and Christine Julien. idml: Incentivized decentralized machine learning. arXiv preprint arXiv:2304. 05354, 2023.
Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao. Beidi Chen, Percy Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 2022.
Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization, 26(3):1835-185555 2016a.
Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization, 1835)11855-1854, 2016b.
Kun Yuan, Bicheng Ying, Xiaochuan Zhao, and Ali H. Sayed. Exact diffusion for distributed opti- mization and learning- 67(3):708-723, 2019.
Algorithm development. IEEE Transactions on Signal Processing,
Liangqi Yuan, Ziran Wang, Lichao Sun, Philip S. Yu, and Christopher G. Brinton. Decentralized federated learning: A survey and perspective. IEEE Internet of Things Journal, pp. 1-1, 2024.
Shahryar Zehtabi, Jun Han, Rohit Parasnis, Seyyedali Hosseinalipour, and Christopher G Brin- ton. Decentralized sporadic federated learning: A unified algorithmic framework with convergence guarantees. In The Thirteenth International Conference Learning Representations, 2025.
Rongfei Zeng, Chao Zeng, Xingwei Wang, Bo Li, and Xiaowen Chu. A comprehensive survey of incentive mechanism for federated learning. arXiv preprint arXiv:2 15406, 2021.
Chang Zhang, Shunkun Yang, Lingfeng Mao, and Huansheng Ning. Anomaly detection and defense techniques in federated learning: comprehensive review. Artificial Intelligence Review, 57(6): 150, 2024.
Miaoxi Zhu, Li Shen, Bo Du, and Dacheng Tao. Stability and generalization of the decentralized stochastic gradient descent ascent algorithm. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023a.
Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, and Dacheng Tao. Topology- aware generalization of decentralized SGD. In International Conference on Machine Learning. PMLR, 2022.
17

---

Published as a conference paper at ICLR 2025
Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao. Decentralized SGD and average- direction SAM are asymptotically equivalent. In Proceedings of the 40th International Conference Machine Learning, 2023b.
18

---

Published as a conference paper at ICLR 2025
A BACKGROUND
A.1 1
DATA INFLUENCE
Data Influence Estimation. As high-quality data becomes increasingly critical in modern machine learning (Hoffmann et al., 2022; Penedo et al., 2023; Li et al., 2023; Longpre et al., 2024; Villalobos et al., 2024), understanding the influence of data has emerged as a crucial research direction (Sorscher al., 2022; Grosse al., 2023; Xia et al., 2024). Data influence estimation quantifies the contribution of training data to model predictions (Chen et al., 2024a; llyas et al., 2024). It enables fair attribution of data source contributions, serving as the foundation for incentive mechanisms. Besides serving as an incentive, data influence has been extensively applied in various machine learning domains, including few-shot learning (Park et al., 2021), dataset pruning (Sorscher al., 2022; Yang et al., 2023), distillation (Loo et al., 2023), faimess improving (Li & Liu, 2022), machine unlearning (Guo et al., 2020; Sekhari et al., 2021), explainability (Koh & Liang, 201 Han et al., 2020; Grosse et al., 2023), as well as training-set attacks (Demontis et al., 2019; Jagielski et al., 2021) and defenses (Hammoudeh & Lowd, 2022).
Data influence estimators are broadly categorized into static and dynamic approaches Specifically, static approaches include both retraining- based and one-point methods. Retraining- based methods, such as leave-one-out (Cook, 1977), Shapley value (Shapley, 1953), and Datamodels (Ilas et al., 2022), assess data influence by retraining the model on (subsets of) the training data. These methods offer conceptually straightforward computation of data influence and are grounded in theoretical foundations, but they are often computationally expensive due to the requirement for retraining.
In contrast, one-point influence methods approximate the effect of retraining using single trained model. A well-established one-point method is the canonical influence function (Koh & Liang, 2017b), developed from statistics (Hampel, 1974: Chatterjee al.. 1982). which examines how infinitesimal perturbations of a training example affect the empirical risk minimizers (ERM) (Vapnik incorporate higher-order
& Chervonenkis, 1974). The influence function has been extended information (Basu et al., 2020) and scaled for larger models (Guo et al., 2021; Schioppa et al., 2022), including LLMs (Grosse et al., 2023). Fu al. (2022) extend influence function Bayesian inference. While these static influence measures have elegant theoretical foundations, they are limited in characterizing how training data influences the training process.
Alternatively, dynamic methods enhance influence estimation by considering the evolution of model parameters across training iterations (Charpiat et al., 2019). Notable examples in this category include TracIn (Pruthi et al., 2020) and In-Run Data Shapley (Wang et al., 2024), which track the influence of training data points by averaging gradient similarities over time. The practicality of dynamic influence estimators is demonstrated by their applications in improving training processes in modern setups (Xia et al., 2024). Recently, Nickl et al. (2023) adopt a novel anemory-perturbenn equation framework to derive dynamic influence estimation of model trained under different centralized optimization algorithms, including SGD, RMSprop and Adam.
However, the existing statie and dynamic influence estimation methods primarily target centralized scenarios, and little progress has been made in analyzing data influence in fully decentralized environments. To the best of our knowledge, the only closely related work is by Terashita & Hara (2022), who proposed decentralized hyper-gradient method and provided novel insights on applying hyper-gradients to compute centralized formulation of data influence. Nevertheless, their estimation method is static and cannot capture the influence cascade arising from gossip communication during decentralized training. In contrast, our framework, DICE, is specifically designed for fully decentralized environments, allowing it to provide fine-grained characterization of the unique influence cascade inherent in these settings.
A.2 DECENTRALIZED LEARNING
Currently, large-scale training and inference processes are primarily performed in expensive data centers. Decentralized training, echoing swarm intelligence (Bonabeau al., 1999; Mavrovouniotis
Tyypcallly, data influence estimators classified retraining- and baaeed methods (Hammoudeh & Lowd, 2024). For enhanced logical coherence in this paper, group retraining- methods and one-point gradient-based techniques under the static category. This classification does not contradict conventional categorizations.
19

---

Published as a conference paper at ICLR 2025
et al., 2017), offers a cost-efficient alternative avenue by crowd-sourcing computational workload to geographically decentralized compute nodes, without the control of central servers (Yuan et al., 2022; Borzumov et al., 2023b; Jaghouar al., 2024). One notable example showcasing decentralized computing's 8 computational potential is the Bitcoin eco-system which virtually distributes jobs requiring instantaneous 16 GW power consumption (CCAF, 2023)- this has been triple of the estimated 5 GW of the world's largest planned cluster for AI (Gardizy & Efrati, 2024; OpenAI, 2025).
In the following, we provide an overview of the algorithmic and theoretical advancements in de- centralized learning. While our discussion touches on several notable contributions, it is far from exhaustive. For a more comprehensive survey, we refer readers to Martínez Beltrán et al. (2023); Singha et al. (2024); Yuan et al. (2024).
Algorithmic Development of Decentralized Learning. The advancement of decentralized learning algorithms has been driven by the need for communication-effiiiee optimization methods in practical distributed learning scenarios. These algorithms have adapted to accommodate dynamic network structures (Nedi'c c & Olshevsky, 2014; Koloskova et al., 2020; Ying et al., 2021; Takezawa et al., 2023), asynchronous communication (Lian et al., 2018; Xu et al., 2021; Nadiradze et al., 2021: Bornstein et al., 2023; Even et al., 2024), data heterogeneity (Tang et al., 2018; Vogels et al., 2021; Le Bars et al., 2023), and Byzantine adversaries (He et al., 2022; Ye & Ling, 2025). Furthermore, their applicability has extended beyond conventional optimization problem to more complex problem formulations, including compositional (Gao & Huang, 2021), minimax (Xian et al., 2021; Zhu et al., 2023a; Chen al., 2024b), and bi-level (Yang et al., 2022; Gao et al., 2023; Chen et al., 2023) optimization problems. Additionally, privacy concerns in decentralized learning are also critical, with efforts focusing on differentially privacy (Cyffers et al., 2024; Allouah et al., 2024) and data reconstruction attacks (Mrini et al., 2024).
Theoretical Development of Decentralized Learning. In terms of optimization, earlier works on decentralized optimization (Nedic & Ozdaglar, 2009; Sayed, 2014; Yuan et al., 2016b; Lian et al., 2017) lay the groundwork for understanding convergence. Lu & De Sa (2021) present systematic framework for federated and decentralized learning by categorizing decentralization into three distinct layers. Koloskova et al. (2020) unify synchronous decentralized gradient descent algorithms across various communication topologies, while Even et al. (2024) extend this framework to accommodate asynchronous scenarios. Building on these efforts, Zehtabi et al. (2025) further develops existing frameworks to consider the sporadicity of both communications and local computations. Regarding generalization, Richards et al. (2020) establishes generalization bounds of decentralized SGD in convex settings via uniform stability. Sun et al. (2021) extend this non-convex settings, revealing an additional (()) dependence on graph topology, though later empirical studies suggest this gap might be overstated (Kong et al., 2021). To refine this, Zhu et al. (2022) introduce a Gaussian weight difference assumption, improving the p dependence to O((1 ))²). Le Bars et al. (2024) further show that in convex settings, the generalization error of local models in decentralized SGD matches that of standard SGD, while in non-convex settings, decentralization primarily affects worst-case generalization. To explain previously unexplained phenomena in decentralized learning (Kong et al., 2021; Gurbuzbalabaa et al., 2022; Vogels et al., 2023), Zhu et al. (2023b) later link decentralized SGD to random sharpness-awart minimization, uncovering flatness bias in decentralized training. Complementing this, Cao et al. (2024) further analyze the flatness properties of DSGD and its role in escaping local minima.
Decentralized Training of Foundation Models. DT-FM (Yuan et al., 2022) introduces tasklet scheduling for training Transformers in decentralized settings with low-bandwidth networks, opti- mizing resource utilization in distributed environments. SWARM Parallelism (Ryabinin et al., 2023) enhances scalability through fault-tolerant pipelines and dynamic node rebalancing. CocktailSGID (Wang et al., 2023a) combines decentralization with sparsification and quantization enhances com- munication efficiency in fine-tuning LLMs. On the inference side, Petal (Borzunov al., 2023a) leverages swarm parallelism to amortize inference costs across heterogeneous resources. Recently, Intellect (Jaghouar et al., 2024) built on Diloco (Douillard et al., 2023) has employed a combination of data parallel and model parallel to collaboratively train large models with up to billions of param- eters. For a comprehensive overview of large-scale deep learning training, including data, model architecture, optimization strategies, budget constraints, and system design, see Shen et al. (2024).
The following figure presents a comparison between server-based learning and decentralized learning.
20

---

Published as a conference paper at ICLR 2025
Server
Communication
Local trining of -ovog.e
Local traininng
Participant
Communication
Partiipantss
(a) Server-based Learning
(b) Decentralized Learning
Figure A.l: A comparative illustration of server-based learning versus decentralized learning.
We summarize some commonly used notions regarding decentralized training as follows:
Definition A.1 (Doubly Stochastic Matrix). Let (V, E) represent decentralized communication topology. where V is the set n nodes and the set of edges. For any g = (V, E), the doubly stochastic gossip matrix W [W R=x* is defined the edge set & and satisfies:
if # l: and (j, k) & &, then Wj = 0; otherwise, W > 0. W₁, E [0, for all j, k. and Σx Wk.j = E3 W =1
Intuitively, the doubly stochastic property ensures balanced flow information during gossip communication, common assumption in decentralized learning literature. However, in the scenarios we consider, participants may occupy different roles within the network. Influential nodes might have higher outgoing weights, i.e., Ej=1 W,,
1.
To accommodate such cases while still ensuring the convergence of decentralized SGD (Yuan et al., 2019: Xin al., 2019), we introduce relaxed condition:
Definition A.2 (Row Stochastic Matrix). Let = (V, denote a decentralized communication topology, where V is the set of n nodes and & is the set of edges. For any G = (V,E), the row
stochastic gossip matrix W = [W E gaxx is defined on the edge set & and satisfies:
if + k and (j, k) & E, then W = 0; otherwise, W, 0.
W j,A E [0, for all k. and E, Wk.j 1.
The weighted adjacency matrix W in Algorithm can vary across iterations, resulting in time-varying collaborations among participants. Additionally, FedAVG (McMahan et al., 2017) is special case of Algorithm where the averaging step is performed globally. This demonstrates that our framework accommodates decentralized learning with dynamic communication topologies and is applicable to both federated and decentralized learning paradigms, even though the primary focus is on fully decentralized learning without central servers.
A.3 MULTI-HOP NEIGHBORS
In graph theory, the concept of neighborhoods is fundamental for understanding the structure and dynamics of graphs. To ensure a coherent and comprehensive flow Section 4, we provide formal definitions of multi-hop neighborhoods.
The adjacency matrix serves as a powerful tool for representing and analyzing the structure of graph. Multi-hop hopp neighbors can be precisely defined using the adjacency matrix.
Definition A.3 (Adjacency Matrix). The adjacency matrix A of graph G (V,E) is an square matrix (where = (V) defined by:
{
if kj, E otherwise.
(A.1)
21

---

Published as a conference paper at ICLR 2025
The adjacency matrix enables the determination of r-hop neighbors through matrix exponentiation. Specifically, the (j. -entry of A", denoted as (A"))k. corresponds to the number of distinct paths of length r from node to node k.
Definition A.4 (r-hop Neighbor via Adjacency Matrix), The set of r-hop neighbors is formally defined using the adjacency matrix A as:
xO - - EV (Ar)st > 0 and Vs < (A*) =
(A.2)
This definition indicates that there exists least one path of length connecting nodes j and k; and no shorter path exists between them.
Multi-hop neighbors can also be defined via the shortest path length between two nodes.
Definition A.5 (Shortest Path Length). In connected graph G (V,E), the shortest path length d(j,k) k) between nodes j E V and & E V is the minimum number of edges that must be traversed to travel from j to k.
Building upon this, the of r-hop neighbors defined as follows:
Definition A.6 (r-hop Neighbor via Shortest Path Length). For any node j E V and a positive integer 2 the set of r-hop neighbors, denoted by (f), consists of all nodes that are distance of exactly r from node j. Formally,
NO (k E V ([
(A.3)
where d(j,k) k)) represents the shortest path length between nodes j and the graph G.
Furthermore, an alternative perspective on r-hop neighborhoods involves characterizing them through sequences of nodes, which provides formal framework aligned with influence propagation in decentralized learning.
Definition A.7 (r-hop Neighbor via Node Sequences). For any node E V and positive integer 2 let p(r) denote the set of all sequences (ki,..., k,) such that for each 1, .....y J, the node k. is out-neighbor of kg-1, with ko j. Formally,
pfr) ( k,) lig E
EV (kg-1) for 1....,r
This definition ensures that each node the r-hop neighborhood is reachable from node j through a sequence of consecutive immediate out-neighbors within steps.
This sequence-based characterization of r-hop neighborhoods provides granular understanding of the pathways through which influence or information can propagate within the network, complement- ing the previous definitions based on adjacency matrices and shortest path lengths.
B DISCUSSIONS
B.1 PRACTICAL APPLICATIONS OF DICE
Decentralized Machine Unlearning. As concerns about data privacy and the right be forgotten increase, the ability to remove specific data contributions from trained model becomes important (Guo et al., 2020; Sekhari et al., 2021). 1). In decentralized settings, retraining the model from scratch is often impractical for edge users with limited compute. The proximal influence measure enables participants to estimate the impact of removing a particular data instance from neighbor. For example, by assessing the influence of 2 on neighbors, participants can adjust their local models to mitigate the effects of 2 without requesting full retraining of the whole decentralized learning system. This approach facilitates efficient and targeted unlearning procedures, avoiding costly system-wide retraining while respecting individual data privacy requests.
22

---

Published as a conference paper at ICLR 2025
B.2 ADDITIONAL RELATED WORK
Clustered Federated Learning. Clustered Federated Learning (CFL) addresses the challenge of data heterogeneity by grouping clients with similar data distributions and training separate models for each cluster (Mansour et al., 2020; Ghosh et al., 2020; Sattler et al., 2021; Kim et al., 2024). Gradient-based CFL, methods (Sattler et al., 2021; Kim et al., 2024) use client gradient similarities to form clusters, with Sattler et al. (2021) employing cosine similarity to recursively partition clients after convergence and Kim et al. (2024) dynamically applying spectral clustering to organize clients based gradient features during training. These methods effectively capture direct, peer-to-peer gradient relationships to cluster clients with similar data-generating distributions. Both gradient- CFL and the one-hop DICE estimator (see Proposition 1) utilize gradient similarity information. However, CFL is inherently limited to local interactions, as its gradient similarity metrics are confined to pairwise relationships. In contrast, DICE extends far beyond this scope by quantifying the propagation of influence across multiple hops in a decentralized network. Mathematically, Theorem 2 highlights how DICE generalizes peer-level gradient similarity into a non-trivial extension for decentralized networks. This includes incorporating key factors including network topology and curvature information, enabling deeper understanding of how influence flows through the whole decentralized learning systems. A promising future direction is to explore the potential of DICE-E E as a more advanced high-order gradient similarity metrie for effectively clustering participants in decentralized federated learning.
C PROOF
C.1 PROOF OF PROPOSITION
Proposition 1 (Approximation of One-hop DICE-GT). The one-hop DICE-GT value (see Defini- tion 2) can be linearly approximated as follows:
DICEn(E)) VI(00="'TA,,000
Σ
Ok W ;0) 2/),
REACH (
(C.1)
where = -0 - The proof is given below.
Proof.
Recall from Definition 2 that the one-hop DICE-GT is defined by
(1)) 2 = 45 (L(O)t ;²)) -((0)==))))))
Σ T
- :
KEA (
We proceed by applying a first-order Taylor expansion to each term.
First term: Using Taylor expansion, write
b - L(03; =V((0)); T -09)
Under the new update rule, we have
0 = 00,0=,,00,,..
so that
0 - 0 = 0, 05.
Thus, the first term is approximated by
1(0)) -(())=²))
(0,,00) -0;)
23

---

Published as a conference paper at ICLR 2025
Second term: For each Nall (j), similarly have
L(
VL(OKt
(
(0t+1
0 k/
By the gossip averaging step in the algorithm, we have
0tt
E
k.
(k)
0 [
W 0
Σ
W 0
(ENG(R)
It then follows that
0++ [
W (0/ -0;) -0:)
= -W/y (0,(0,,))) 03)
Thus, the second term becomes
1( :2) 20) ~ W VL(O (0,(0))))))))))))
23) 07)
Combining the Approximations: Substituting the above approximations into the definition of DICR.GT yields
((1)
(1))) 93 VL(0):=') -00
Σ 9 WE, VL(O :2' (0,(0), -0)) kEA
This completes the proof.
C.2 PROOF OF TWO-HOP DICE-E APPROXIMATION
Proposition (Approximation of Two-hop DICE-GT). The two-hop DICE-GT influence I (22)
(fj, 2²) (see Definition 3) can be approximated as
DICEE (2), 2²) I(1)
Σ
Σ
W't Wt, VL(O -n"
n1* (=)))
KENO) (1) (E)
(C.2)
where 4, x5)
0) 0%. and (0 ; denotes the Hessian matrix of L with
respect 0.4 evaluated at =
Proof. We begin from the definition in Definition 3 where the two-hop DICE-GT influence is given by
IP (z),
Σ Σ qi -
READ (J) IEAD (k)
Subtracting the one-hop influence from both sides yields
(22) 12) x') - - (11) ID(ce.on3)) =
Σ
Σ
- L(0)t))2²))
kE (f) IEAS (R)
24

---

Published as a conference paper at ICLR 2025
A first-order Taylor expansion gives
L(0 2 (( VL(0)+ :2)) T (0,tt2
Next, the update rule in Algorithm implics that
0,+2=
0 Lm
and similarly for 0 A2
Since the influence of zJ reaches only via intermediate nodes, we may write
0
Σ
W
(0) ++ 3
RENTE ())
At an intermediate node k the local update in iteration t gives
= n [ =
+
Therefore, the difference between the actual and perturbed updates is
(I- (
; (0 0
~
m1=;)
At node k. the difference between the parameters updated with and without the influence from 2] is given by
0t1 0++1 1/
Wh, (0 - -1/ - W 4,(0),
00))
Hence, we obtain
nnj (1- ( x1) W 4,(0), 25).
Substituting back into the expression for node I, we have
0
Σ
w
Why (I (I- +1
2/).
REA (3)
Plugging this into the Taylor expansion for the loss difference and multiplying by q yields
2(0++2 L (0/t)3:=²) z')
~ - ( T
Σ
W Wh, (1- (( A 2/).
kEA (
Finally, summing over all intermediate nodes and multiplying by q. we obtain
I (z), x²) ((1)
(
FD)C 2')
Σ Σ
n' with WE, =/ (I n1- (1-1""-(00 (, 21).
RENUD (1) (E (k)
This completes the proof.
25

---

Published as a conference paper at ICLR 2025
C.3 PROOF OF THEOREM 2
Theorem (Approximation of Multi-hop DICE-GT). The r-hop DICE-GT value (see Definition 3) be approximated
P II
(()) DOjC, =- Σ
Σ
of qs,
Wi+o-1 Aoiks-1
VI(O
p= (p)
I 1 H(Ot ; = (0)
(C.3)
where
A O0 O0 z3) -0
where Ko = j. p(() denotes the set of all sequences (A1,..., kp) such that Ks E Nool (kg-1) for ,,..., P (see Definition A.7) and H (OO =) is the Hessian matrix of L with respect to O evaluated at f and data z(+ For the cases when p = and p the relevant product expressions are defined as identity matrices, thereby ensuring that the r-hop DICE-E remains well-defined.
Proof. From the definition in Definition 3, the r-hop influence is
TOiCE (z), p=0 (p)
Σ
yhp (1(O))
Here the = O term (with ku = j) corresponds to the direct influence on node j. For any p 2 1, define the incremental influence
Az AT DiCE- (=), IbfCROOT z¹).
Thus,
ADDIC.GT (2). (2),=)))==
Σ
ako [<Ottee;*] L( x0
(p)
A first-order Taylor expansion gives
1( 2') L( ko (zt 2')
R2 (L( 2²) Tottp 0 kp
kp
Our goal is express the parameter change A & tt perturbation from node j.
0h in terms of the propagated ko
According to the gossip update Algorithm 1, for any node hp we have
Wh+e-1
2+p-3
(k,)
Σ
Wh+e-1
0 (tp- m/zf
\z!
(k,
Since only the predecessor k--1 is affected by the perturbation from 25. we obtain
w c x koiko ( 1 ky 112))
At node kp-1, using the local update rule,
0 -+--
Oh (0hto-) _+p-1 1) ,
kp-1
26

---

Published as a conference paper at ICLR 2025
the difference can be written
-+ 1
x+pp (0(pp-1
0ctp-1
kp-1 Ro
1125
kp-1
Kp- 125)
-
(V ( +ttp1) (103)-VV41
(I(Atto =(tp-1))
Rp-1 k2
kp-1 127 kp--1
A further first-order Taylor expansion approximates
VL(O Ro
=k )
VL(O 1 H
0.+p-11
Ky
1/291 2k,
Ap
Ro 1)27)
Thus,
O Whte-1 (1- H²tp-1) O Octp-11 12-1)
By recursively unrolling this relation from down to 8 = 1, we deduce
kg
0h Eplis
Racke-1
[I 1| (((0-11)
(0/+2
0 kz)25)
s=2
At the base level, the local update node gives
0+
o++1 -nf
W Wh. (0),
b1\=
Hence,
gh+ 6p kp
0
[I
Wete-1 kgika-l
( + Helmittto H1tto-)) (0,0),
Substituting this back into the Taylor expansion for the loss difference, we have
AT(e) DICE cr z))
Σ
1f qu.
fiwwter ku.ko-
(L(ORt?:2')
(ks (N
I - -/ (--ntt---Httt--) (
20))) ),13,
II
x
Summing over p to (with the 0 term accounting for the direct influence at node j) yields
(2C. 23) -- Σ
Σ
n¹ qu,
Wh++-1 ky.ks
p=0
(kI....k))erfo)
II
I 17 + 1- (00
x
=k
This concludes the proof.
D ADDITIONAL EXPERIMENTS
D.I DETAILS OF EXPERIMENTAL SETUP
Computational Resources. The experiments are conducted on a computing facility equipped with 80 GB NVIDIA & AM GPUs.
We employ the vanilla mini-batch Adapt- Tee--hhhe---ommme Communicate version of Decentralized SGD (Loppes & Sayed, 2008), Algorithm 1) with commonly used network topologies (Ying et al., 2021) train
27

---

Published as a conference paper at ICLR 2025
three-layer MLPs (Rumelhart et al., 1986), three-layer CNNs (LeCun et al., 1998), and ResNet- (He et al., 2016) on subsets of MNIST (LeCun et al., 1998), CIFAR-10. 10, CIFAR-I00 (Krizhevsky et al., 2009), and Tiny ImageNet (Le & Yang. 2015). The number of participants (one GPU participant) is set to 16 and 32, with each participant holding 512 samples. For sensitivity analysis, evaluate the stability of results under hyperparameter adjustments. The local batch size is varied as 16, 64, and 128 per participant, while the learning rate is set as 0.1 and 0.01 without decay. The code will be made publicly available.
D.2 INFLUENCE ALIGNMENT
In this experiments, evaluate the alignment between one-hop DICE-GT (see Definition 2) and its first-order approximation, one-hop DICE-E (see Proposition 1). One-hop DICE-E (15 =²) computed as the sum of one-sample DICE-E within the mini-batch BY thanks to the additivity (see Equation (4)). DICE-GT (DICE-GT") ) is calculated by measuring the loss reduction after removing 3 from node the f-th iteration. In the following Figures, each plot contains 30 points. with each point representing the result of a single comparison of one-hop DICE-GT and the estimated influence DICE-E. Strong alignments of DICE-GT and DICE-E are observed across datasets (CIFAR-10, 10, CIFAR-100 100 and Tiny ImageNet) and model architectures (CNN and MLP).
CNN on CIFAR-10
CNN on CIFAR-100
MLP on CIFAR-10
MLP on CIFAR-100
Bes to
Actuml
Actusl
Predicted diff in less
Predicted diff in loss
Predicted diff in loss
Predicted diff in loss
Figure D.I: Alignment between hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on a 16-node exponential graph. Each node uses a 512-sample subset of CIFAR 10 or CIFAR-100. Models are trained for 5 epochs with batch size of 128 and learning rate of 0.1.
CNN on CIFAR-10
CNN CIFAR-100
MLP CIFAR-10
CIFAR-100
Actual dit losa
Actual diff in loss
Predicted diff in loss
Predicted diff In loss
Prodicted diff loss
Predicted diif In loss
Figure D.2: Alignment between hne-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on a 32-node exponential graph. Each node uses a 512-sample subset of CIFAR- or CIFAR-100. Models are trained for 5 epochs with batch size of 128 and learning rate of 0.1.
We conduct additional sensitivity analysis experiments to evaluate the robustness of DICE-E under varying hyperparameters, including lcarning rate, batch size, and training epoch. These results demon- strate that DICE-E provides strong approximation of DICE-GT, achieving consistent alignment across datasets (CIFAR- 10 and CIFAR-100) and model architectures (CNN and MLP) under different batch sizes, learning rates, and training epochs.
D.2.1 SENSITIVITY ANALYSIS ON BATCH SIZE
We conduct experiments to evaluate the robustness of DICE-E under varying batch sizes.
28

---

Published as a conference paper at ICLR 2025
CNN on CIFAR-10
CNN on CIFAR-100
MLP on CIFAR-10
MLP on CIFAR-100
Actual ditf loss
Ap 8' Boan
Predicted diff in less
Predicted diff in less
Predicted ditf in loss
Predicted diff in loss
Figure D.3: Alignment between one-hop hop DICE-GT (vertical axis) and DICE-E (horizontal axis) 32-node ring graph. Each node uses a 5 112-sample subset of CIFAR-10 or CIFAR- 100. Models are trained for epochs with batch size of 16 and learning rate of 0.1.
CNN on CIFAR-10
CNN on CIFAR-100
MLP on CIFAR-10
MLP on CIFAR-100
aee
Actual der in lozs
in
dirf
Predicted diff
Predicted diff
Predicted diff
Predicted diff
Figure D.4: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) 32-node ring graph. Each node uses a 512-sample subset of CIFAR-10 or CIFAR-100. Models are trained for epochs with a batch size of 64 and a learning rate of 0.1.
CNN on CIFAR-10
CNN on CIFAR-100
MLP on CIFAR-10
MLP on CIFAR-100
Actual ditt in loss
Actunl diff in losss
Actuel dief in loss
In
All
8
Pradicted
Pradicted
Pradicted
Predicted
Figure D.5: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on 32-node ring graph. Each node uses 512-sample subset of CIFAR- 10 or CIFAR-100. Models are trained for epochs with a batch size 128 and learning rate of 0.1.
D.2.2 SENSITIVITY ANALYSIS ON LEARNING RATE AND THE NUMBER NODES
We also condcut experiments to evaluate the robustness DICE-E under varying learning rates and the number of nodes.
29

---

Published as a conference paper at ICLR 2025
CNN on CIFAR-10
CNN on CIFAR-100
MLP on CIFAR-10
MLP on CIFAR-100
Actual diff loss
Actual ditf loss
Predicted diff In loss
Predicted diff in loss
Predicted diff in loss
Predicted diff in loss
Figure D.6: Alignment between one-hop hop DICE-GT (vertical axis) and DICE-E (horizontal axis) 16-node ring graph. Each node uses a 512-sample subset CIFAR-I0 or CIFAR- -IFAR-100. Models are trained for epochs with batch size of 64 and learning rate of 0.1.
CNN on CIFAR-10 10
CNN on CIFAR-100
MLP on CIFAR- 10
MLP on CIFAR-100
Actual dift loes
Actual dRr
Actual
shll
Predicted diff loss
Predicted diff loss
Predicted diff
Predicted diff lows
Figure D.7: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) 16-node ring graph. Each node uses 5 312-sample subset of CIFAR-10 10 or CIFAR-100. Models are trained for epochs with a batch size of 64 and a learning rate of 0.01.
CNN on CIFAR-10
CNN on CIFAR-100
MLP on CIFAR-10
MLP on CIFAR-100
Actual dtt in loss
Actual in
Actual in
Actual diff
Predicted
Predicted diff
Predicted diff
Predicted diff
Figure D.8: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on 32-node ring graph. Each node uses $1 512-sample subset of CIFAR-10 10 or CIFAR-100. Models are trained for epochs with a batch size of 64 and a learning rate of 0.1.
CNN CIFAR-10
CNN CIFAR-100
MLP on CIFAR-10
MLP on CIFAR-100
Actual dilf losa
Predicted loss
Predicted
Predicted
Figure D.9: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on a 32-node ring graph. Each node uses 5 512-sample subset of CIFAR- CIFAR-100, Models are trained for epochs with a batch size of 64 and a learning rate of 0.01.
D.2.3 SENSITIVITY ANALYSIS ON TRAINING EPOCHS
We conduct experiments to evaluate the robustness of DICE-E under varying training epochs.
30

---

Published as a conference paper at ICLR 2025
with
with Size
Size
MLP Size
Actual ditr in losa
ft
Predicted diff in loss
Predicted diff in loss
Predicted diff In loss
Predicted diff in loss
Figure 10: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on 16 and 32-node exponential graphs. Each node uses a 3192-sample subset of Tiny ImageNet. Models are trained for 10 epochs with batch size of 128 and a learning rate of 0.1.
CNN with Size 16
CNN with Size 32
MLP with Size 16
MLP with Size 32
Actual dirf in loss
Actual dirt in loss
Actual eirr in lass
dier in
m
Predicted
Predicted diff loss
Predicted diff
Predicted diff
Figure D.11: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on 16 and 32-node exponential graphs. Each node uses a 8192-sample subset of Tiny ImageNet. Models are trained for 20 epochs with batch size of 128 and a learning rate of 0.1.
CNN with 16
CNN with Size 32
MLP with Size
with Size
gadt.
in
Actual diet
Actual diff
Predicted diff loss
Predicted diff loss
Predicted diff in less
Predicted diff in less
Figure D.12: Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on a 16 and 32-node exponential graph. Each node uses 8192-sample subset Tiny ImageNet. Models trained for 30 epochs with batch size of 128 and learning rate of 0.1.
D.3 ANOMALY DETECTION
We can also use the proximal influence metric to effectively detect anomalies. Specifically, anomalies are identified by observing significantly higher lower proximal influence values compared to normal data instances. In our setup, anomalies are generated through random label flipping or by adding random Gaussian noise to features. The following Figures illustrates that the most anomalies (in red) is detectable with proximal influence values.
D.3.1 RANDOM LABEL FLIPPING
We can conclude from these experiments that anomalies introduced through random label flipping readily detectable by analyzing their proximal influence.
31

---

Published as a conference paper at ICLR 2025
ResNet-18 CIFAR-10
CNN CIFAR-10
MLP CIFAR-10
Proximal infruence
690
-833
+340
Figure D.13: Anomaly detection on exponential graph with 32 nodes. Each node uses a 512-sample
subset of CIFAR-10. Models are trained for 5 epochs with a batch size of 16 and a learning rate of 0.1. 32-node exponential graph, each participant connects with 5 neighbors, where the neighbor in red is set an anomaly by random label flipping, while the other four are normal participants.
ResNet-18 CIFAR-10
on CIFAR-10
CIFAR-10
Proximal influence
54M
300
081
-0001
0351
Figure D.14: Anomaly detection exponential graph with 32 nodes. Each node uses a 512-sample
subset of CIFAR-10. Models are trained for 5 epochs with batch size of 64 and a learning rate of 0.1. In 32-node exponential graph, each participant connects with 5 neighbors, where the neighbor red is set as an anomaly by random label flipping, while the other four are normal participants.
ResNet-18 on CIFAR-10
CNN on CIFAR-10
MLP on CIFAR-10
Proximal Influence
E000
-2.00
+3000
Figure D.15: Anomaly detection on exponential graph with 32 nodes. Each node uses a 512-sample
subset of CIFAR-10. Models are trained for 5 epochs with a batch size of 128 and learning rate of 0.1. 32-node exponential graph, each participant connects with 5 neighbors, where the neighbor in red is set an anomaly by random label flipping, while the other four are normal participants.
D.3.2 FEATURE PERTURBATIONS
We can conclude from Figure 1.19, Figure D.20 and Figure D.21 that most anomalies introduced through adding zero-mean Gaussian noise with high variance are readily detectable by analyzing their proximal influence, which significantly deviates from that of normal data participants.
32

---

Published as a conference paper at ICLR 2025
ResNet-18 CIFAR-10
CNN CIFAR-10
CIFAR-10
Proximal influence
630
0.03
Figure 1.16: Anomaly detection on exponential graph with 32 nodes. Each node uses a 512-sample subset of CIFAR-10. Models are trained for 5 epochs with batch size of 16 and learning rate of 0.01. In a 32-node exponential graph, each participant connects with neighbors, where the neighbor red is as an anomaly by random label flipping, while the other four are normal participants.
ResNet-18 CIFAR-10
CNN CIFAR-10
CIFAR-10
Proximal Influence
0.84
0.80
0.00
-0083
Figure 1.17: Anomaly detection on exponential graph with 32 nodes. Each node uses a 512-sample
subset of CIFAR-10. Models are trained for 5 epochs with batch size 64 and a learning rate of 0.01. In 32-node exponential graph, each participant connects with 5 neighbors, where the neighbor red is set as an anomaly by random label flipping, while the other four are normal participants.
ResNet-18 on CIFAR-10
CNN on CIFAR-10
MLP on CIFAR-10
Proxinal Influence
X00
080
0.64
-082
Figure 1.18: Anomaly detection on exponential graph with 32 nodes. Each node uses a 512-sample subset of CIFAR-IO. Models are trained for epochs with a batch size of 128 and learning rate of 0.01. In 32-node exponential graph, each participant connects with neighbors, where the neighbor red is set as an anomaly by random label flipping, while the other four are normal participants.
D.4 INFLUENCE CASCADE
D.4.1 ONE-HOP INFLUENCE CASCADE
The topological dependency of DICE-E our theory reveals the "power asymmetries" (Blau, 1964; Magee & Galinsky, 2008) in decentralized learning. To support the theoretical finding, we examine the one-hop DICE-E values the same batch on participants with vastly different topological importance. Figure illustrates the one-hop DICE-E influence scores of an identical data batch across participants during decentralized training of a ResNet-18 model on the CIFAR- 10 dataset. Node sizes represent the one-hop DICE-E influence scores, quantifying how single batch impacts other participants in the network. The dominant nodes (e.g.. those with larger outgoing communication weights in W) exhibit significantly higher influence, shown in Figure and further detailed
33

---

Published as a conference paper at ICLR 2025
ResNet-18 on CIFAR-10
CNN CIFAR-10
MLP on CIFAR-10
ARA
AAM
-4.0
-4.0
Figure 1.19: Anomaly detection on exponential graph with 32 nodes. Each node uses a 512-sample subset of CIFAR-10. 10. Models are trained for 5 epochs with a batch size of 128 and a learning rate of 0.1. 32-node exponential graph, each participant connects with 5 neighbors, where the neighbor in red is set as anomaly by adding zero-mean Gaussian noise with variance equals 100 on each feature, while the other four are normal participants.
ResNet-18 on CIFAR-10
CNN on CIFAR-10
MLP on CIFAR-10
0.95
3.04
0.50
-0.0
Figure D.20: Anomaly detection exponential graph with 32 nodes. Each node uses 512- subset of CIFAR-10. Models are trained for 5 epochs with a batch size 64 and a learning rate of 0.01. In 32-node exponential graph, each participant connects with neighbors, where the neighbor in red is set as an anomaly by adding zero-mean Gaussian noise with variance equals 100 on each feature, while the other four are normal participants.
ResNet-18 on CIFAR-10
CNN on CIFAR-10
MLP on CIFAR-10
-043
+0.0
Figure D.21: Anomaly detection on exponential graph with 32 nodes. Each node uses a 512-sample
subset of CIFAR-10. Models are trained for 5 epochs with a batch size of 16 and a learning rate of 0.1. In 32-node exponential graph, each participant connects with 5 neighbors, where the neighbor in red is set as an anomaly by adding zero-mean Gaussian noise with variance equals 100 on each feature, while the other four are normal participants.
in Figure D.23 and Figure D.24. These visualizations underscore the critical role of topological properties in shaping data influence decentralized learning, demonstrating how the structure the communication matrix W determines the asymmetries in influence.
To better observe and showcase the "influence cascade" phenomenon, we design a communication matrix with one "dominant" participant (node 0), two "subdominant" participants (nodes 7 and 10), and several other common participants. Figure D.22 (Left) visualizes the communication topology. where node sizes indicate out-degree, reflecting their influence, and edge thickness represents the strength of communication links. Node 0 stands out as the dominant participant with the largest size, while nodes 7 and 10 serve as subdominant intermediaries. Figure D.22 (Right) complements this by
34

---

Published as a conference paper at ICLR 2025
showing the adjacency matrix W as a heatmap, where the color intensity highlights the magnitude of connection strengths, with the dominant participant exhibiting strong outgoing links across the network. Together, these visualizations highlight the hierarchical structure and asymmetries in the communication matrix, crucial for understanding topological influences in decentralized learning.
20
OO
00
OO
8.3
30
RX
63
Figure D.22: Left: Visualization of the communication topology used in Section 5, where each node represents participant, and edges indicate communication links. Node sizes are proportional to their out-degree (sum of outgoing edge weights), reflecting their communication influence within the community. Edge thickness corresponds to the strength connection (i.e., weight), with directional arrows capturing the flow of information between participants. Self-loops omitted for simplicity. Right: Heatmap representation of the weighted adjacency matrix W used in Section 5, where each entry WE quantifies the communication strength from participant to k. The color intensity represents the magnitude of the weights.
Figure D.23: Visualization of one-hop influence cascade during decentralized trainingg with MLP on MNIST (left) and CIFAR-10 (right) under designed communication matrix (see Figure D.22). The thickness of edges represents the strength of communication links (i.e., weights in W), while node sizes correspond the relative one-hop DICE-E influence scores (see Proposition 1) computed for the same data batch across different participants. The numerical labels on the nodes indicate the corresponding participants, aligning with the participant indices Figure D.22.
D.4.2 MULTI-HOP INFLUENCE CASCADE
To better illustrate the communication structure underlying the influence cascade phenomenon in multi-hpp decentralized learning (see Figure 1), following the setup in Appendix D.4.1, with the only
35

---

Published as a conference paper at ICLR 2025
Figure D.24: Visualization of one-hop influence cascade during decentralized trainingg with ResNet- 18 on CIFAR-I0 (left) and 100 (right) under a designed communication matrix (see Fig- ure D.22). The thickness of edges represents the strength of communication links (i.e., weights in W). while node sizes correspond to the relative one-hop DICE-E influence scores (see Proposition 1) computed for the same data batch across different participants. The numerical labels the nodes indicate the corresponding participants, aligning with the participant indices Figure D.22.
modification being the of different mixing matrix. This modification is specifically designed to refine the visualization for better geographic representation, making the spatial relationships of decentralized participants more apparent. The heatmap Figure D.25 visualizes the corresponding mixing matrix (i.e., weighted adjacency matrix) W for the 16-node topology, where each entry Wj, represents the communication strength from participant j to k. The color intensity encodes the magnitude of these weights, with warmer colors indicating stronger connections.
64
67
4.5
6.5
RO
6.1
Figure D.25: Heatmap representation of the weighted adjacency matrix W used in Figure 1, where each entry WEy quantifies the communication strength from participant j to k. The color intensity represents the magnitude of the weights.
36


Reviews:
[{"review_id": "LgX6P36CeB", "reviewer": "Reviewer_kSES", "summary": "This paper proposes the DICE framework for measuring the cascading propagation of data influence\nin decentralized learning networks. Decentralized learning enables large-scale model training\nthrough distributed computation, yet the lack of effective incentive mechanisms can lead to unfair\ncontributions and malicious behavior among nodes. The DICE framework introduces data influence\ncascades (DICE-GT and DICE-E), which respectively measure the direct and indirect influence of data\nwithin the network, addressing the limitations of existing data influence measurement methods in\ndecentralized environments. Experiments validate the consistency and accuracy of DICE across\nvarious network topologies and demonstrate its potential in practical applications like anomaly\ndetection and collaborator selection", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The DICE framework is the first to systematically measure the cascading propagation of data\ninfluence in decentralized learning environments, providing an effective method to assess data\ncontributions among nodes and filling a gap in data influence evaluation within decentralized\nnetworks.\n2. The experiments cover different network topologies (such as ring and exponential graphs) and\ndatasets (such as MNIST, CIFAR-10, and CIFAR-100), validating the applicability and consistency\nof the DICE framework across various scenarios.\n3. The DICE framework provides accurate contribution measurement, laying the foundation for\ndesigning fair and effective incentive mechanisms in decentralized learning systems, with the\npotential to foster equitable collaboration within decentralized networks.", "weaknesses": "1. Figure 1 lacks legend information, making it difficult to understand.\n2. The performance differences of the DICE framework under different parameters (such as learning\nrate, batch size, etc.) have not been thoroughly discussed. It is recommended to add parameter\nsensitivity experiments to demonstrate the impact of different parameter selections on the\nperformance of the DICE framework, thereby enhancing its practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RGEMLpGspN", "reviewer": "Reviewer_f3R7", "summary": "The paper proposes a method for quantifying the impact of data points in decentralized machine learning settings. The influence is measured not only at immediate neighbors but the entire network. This method can be useful for machine unlearning  or to develop new incentive mechanisms.", "questions": "1) Please motivate the approach with practical use-cases. \n2) Please discuss link with clustered federated learning, in particular techniques that use gradients to cluster clients. \n3) Please provide all necessary details to replicate the results.\n4) Please evaluate the impact of batch size (smaller and larger values), to show the scalability of the technique and its robustness in showing the compatibility among clients.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  The paper is well-organized, with clear definitions, figures, and explanations that make the methods and results easy to follow.\n- The paper provides a solid theoretical framework, supported by rigorous proofs and analyses.", "weaknesses": "- Need for more details about the practical use of this technique: While the authors use LLMs as one of the examples in the introduction, it might not be the best example to use in this case. It hard to see how this research addresses a practical problem or application that has real-world significance, or how this framework would be relevant for practitioners.\n- Link with other papers that use gradient to cluster clients should be added, particularly interesting and relevant in the collaborator choice part.  \n- Experiments seem non-exhaustive and many details are missing to replicate the experiments. For instance, no indication on what the anomaly is vs normal client. This is particularly important when using gradients. I expect that the framework would perform differently if the anomaly is label flipping vs if it was noisy features. Additionally, evaluation of the impact of batch size would be particularly important for both scalability and compatibility among clients.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34ipQ3RW1f", "reviewer": "Reviewer_4yzV", "summary": "The paper proposes DICE as a framework for measuring data influence cascades in decentralized environments. The framework explains how data influence propagates through the communication network, emphasizing the interaction between the original data and the network structure in shaping data influence within decentralized learning. The experimental results show that the first-order approximation of the “gold standard” for evaluating data influence in decentralized environment can approaching the truth, and this framework can used for detecting mislabeled anomalies.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper summarizes previous work on measuring data influence and highlights the gaps in applying these methods to distributed scenarios.\n2. This paper proposes a sound “gold standard” and its first-order approximation to quantify individual contributions in decentralized learning.", "weaknesses": "1. The experiments are weak, and Section 5.3 is unfinished.\n2. The notation η^t in Theorem 1 is previously appears as η_t in Algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)