CYCle: Choosing Your Collaborators Wisely to Enhance Collaborative Fairness in Decentralized Learning
Authors: Nurbek Tastan
Samuel Horváth
Karthik Nandakumar
Venue: Accepted by TMLR ()

Abstract: Collaborative learning (CL) enables multiple participants to jointly train machine learning (ML) models on decentralized data sources without raw data sharing. While the primary goal of CL is to maximize the expected accuracy gain for each participant, it is also important to ensure that the gains are fairly distributed: no client should be negatively impacted, and gains should reflect contributions. Most existing CL methods require central coordination and focus only on gain maximization, overlooking fairness. In this work, we first show that the existing measure of collaborative fairness based on the correlation between accuracy values without and with collaboration has drawbacks because it does not account for negative collaboration gain. We argue that maximizing mean collaboration gain (MCG) while simultaneously minimizing the collaboration gain spread (CGS) is a fairer alternative. Next, we propose the CYCle protocol that enables individual participants in a private decentralized learning (PDL) framework to achieve this objective through a novel reputation scoring method based on gradient alignment between the local cross-entropy and distillation losses. We further extend the CYCle protocol to operate on top of gossip-based decentralized algorithms such as Gossip-SGD. We also theoretically show that CYCle performs better than standard FedAvg in a two-client mean estimation setting under high heterogeneity. Empirical experiments demonstrate the effectiveness of the CYCle protocol to ensure positive and fair collaboration gains for all participants, even in cases where the data distributions of participants are highly skewed. The code can be found at https://github.com/tnurbek/cycle.

Published in Transactions on Machine Learning Research (08/2025)
CYCle: Choosing Your Collaborators Wisely to Enhance Collaborative Fairness in Decentralized Learning
Nurbek Tastan
nurbbe..taston@@maauu..cco
Mohamed bin Zayed University of Artificial Intelligence (MBZUAT)
Samuel Horváth
Mohamed bin Zayed Universiby of Artificial Intelligence (MBZUAT)
Karthik Nandakumar
nandakum Sresuluummmmmmcoo edu
Michigon State University (MSU)
Mohomed bin Zayed University of Artificial Intelligence (MBZUAI)
Reviewed on OpenReview: https: //per///eee/..w....mmmmmm net/ formm? id= id=yggNaLOggfH
Abstract
Collaborative learning (CL) enables multiple participants to jointly train machine learning (ML) models on decentralized data sources without raw data sharing. While the primary goal of CL is to maximize the expected accuracy gain for each participant, it is also important to ensure that the gains are fairly distributed: no client should be negatively impacted, and gains should reflect contributions. Most existing CL met hods require central coordination and focus only on gain maximization, overlooking fairness. In this work, we first show that the existing measure of collaborative fairness based on the correlation between accuracy values without and with collaboration has drawbacks because does not account for negative collaboration gain. We argue that maximizing mean collaboration gain (MCG) while simultaneously minimizing the collaboration gain spread (CGS) is a fairer alternative. Next, we propose the CYCle protocol that enables individual participants in a private decentralized learning (PDL) framework to achieve this objective through a novel reputation scoring method based on gradient alignment between the local cross-entropy and distillation losses. We further extend the CYCle protocol to operate on top of gossip-based decentralized algorithms such as Gossip-SGD. We also theoretically show that CYCle performs better than standard FedAvg in a two-client mean estimation setting under high heterogeneity. Empirical experiments demonstrate the effectiveness of the CYCle protocol to ensure positive and fair collaboration gain for all participants, even in cases where the data distributions of participants are highly skewed. The code can be found at https/////gttttt...oo///rrrr////
1 Introduction
Collaborative learning (CL) refers to a framework where several entities can work together by pooling their resources to achieve a common machine learning (MI.) objective without sharing raw data. This approach offers particular advantages in domains like healthcare and finance that require access to extensive datasets, which can be difficult to acquire for any single entity due the costs involved or privacy regulations such
HIPAA (Centers for Medicare & Medicaid Services 1996) and GDPR (European Parliament & Council of the European Union| 2016 Albrecht 2016). Federated learning (FL) (McMahan et al. 2017) is specific case of CL that enables multiple entities to collectively train a shared model by sharing their respective model parameters or gradients with a central server for aggregation. However, FL methods can lead to information updates with an untrusted third-party server that can potentially carry out reconstruction attacks (Zhu et al. 22199: Zhao et al.) (202)). Furthermore, most CL algorithms assume equal contributions from all participants. When this assumption is violated, there is
Icakage associated with sharing

---

Published in Transactions on Machine Learning Research (08/2025)
Participunt
Eee Clcot elCopiaa BLCooi
Letn ElGue C.CoMIN B(CoiiN !
II
Total distillation loss computation Dt.
D
Purticipant P.
Participunt Ps
Reputation Scoring
finuk)
Va, Ace,
Va. Vo.Con
1E 18) CDLinn BlAD! "CEy
ElCouia CuCui (lC EoB,...
Adaptive Sharing
D
Pa senis updates with prohability T(u.)
Participant Px
Participant Ps
Figure 1: Illusraaioon of the proposed Choose Your Collaborators Wisely (CYCle) protocol for Private Decentralized Learning (PDL).
little incentive for collaboration because the gains are not distributed fairly. To avoid these pit.falls, a fully decentralized learning algorithm that fairly rewards the contributions of individual participants is required.
Recent works have attempted to tackle the challenge of collaborative foirness (CF) in FL settings. Shapley value (SV) (Shapley et al. 1953) is a common choice for estimating participants' marginal utility, but its computation and communication costs are high. To mitigate this problem, variants of SV have been proposed in (Wang et al., 2020 Kumar et al. 2022: Xu et al., [021 Tastan et al., 2024). But these methods only focus on FL with central server. Existing private decentralized learning (PDL) algorithms such CaPriDe learning (Tastan & Nandakumar 2023 CaPC (Choquetto- Caepc et al. 2021 and Cronus (Chang al.
2019) emphasize confidentiality, privacy, and utility, while ignoring collaborative fairness.
In this work, we aim to bridge this significant gap by designing the CYCle protocol for knowledge sharing in the PDL sctting. We focus on three goals: maximizing the mean accuracy gain across all participants, ensuring that no participant sulfers performance degradation due to collaboration, and that the accuracy gains are evenly distributed. To achieve these goals, we make the following contributions:
We analyze the collaborative fairness metric based on the correlation coefficient between the contri- butions of participants and their respective final accuracies, and show that it fails in cases where the collaboration gain is negalive.
We introduce the CYCle protocol that regulates knowledge transfer among participants in PDL framework based on their reputations to achieve better collaborative fairness. The proposed repulation scoring scheme uses gradient alignment between cross-entropy and distillation losses to accurately assoss relative contributions made by collaborators in PDL.
We extend the CYCle protocol to operate seamlessly on top of gossip- based decentralized algorithms such as Gos cnabling fair collaboration.
We theoretically study the CYCle protocol in the context of mean estimation between two participants and show that it outperforms FedAvg in the presence of data heterogencity.
2 Related Work and Background
Several challenges associaled with CL have been addressed recently, including privacy (Dwork et al. 2014
Choquette- Choo et al.] 2021 Chang et al. 2019 Tastan & Nandakumar 2023 Kairouz et al. 2021 Erlingsson
al. 2020: McMahan al. 2018), confidentiality (Tastan & Nandakumar
2023 Choquetto- Choo al. Chang al. (2019). communication efficiency (McMahan al. (2017), robustness [Lakshminarayanan
2021
et al.
2017 Athalye al. 2018 Bagdasaryan et al. 2020), and fairness
[u et al.
2021
Zhou et al.
2

---

Published in Transactions on Machine Learning Research (08/2025)
2021
Lyu et al. 2020 Most existing works focus only on FL algorithms with central orchestration such as Fed Avg (McMahan et al. 2017). Due to the potential harm caused by gradient leakage in FT. (Zhao et al. 2020
Zhu et al. 2019) which can result in the disclosure of sensitive user data, there is a growing interest in decentralized learning algorithms that do not require contralized orchestration. Focus has been
mainly on ensuring confidentiality, privacy, and utility in decentralized learning (Tastan & Nandakumar
2023 Crrquctte-Choo et al. 2021; Chang et al.] 2019). While frameworks like Cronus [Chang et al. 2019) offer robustness and privacy in decentralized settings, they do not address fairness, which central to our work. To our knowledge, CYCle is the first to explicitly target collaborative fairness in decentralized learning.
To achieve collaborative fairness in CL, it is critical to evaluate marginal contributions of participants (Jia let al. 2019 Xu et al.) 2021 Shi et al.] 2022 Jiang et al., 2023 Tastan et al.] 2024 Donahue & Kleinberg, 2021 Because data distributions are often heterogeneous, some clients may contribute more valuable data,
yet vanilla. FL gives all participants the same global model, discouraging collaboration and hindering adoption. While Shapley Value (SV) can be usod for data valuation, SV computation is expensive and hard to employ (Shapley et al., 1953). In CGSV (Xu et al.) 2021), cosine similarity wetween local and global parameter updates is used to approximate the SV. The same cosine similarity approach is used in RFFL (Xu & Lyu 2021) for reputation scoring. Kumar ct al. (2022) used logistic regression models as proxies for client data and utilized an ensemble of them to approximate the SVs.
Compared to prior work that either assumes centralized orchestration or overlooks fairness, CYCle introduces a fully decentralized protocol that ensures fair collaboration by regulating knowledge transfer based on participant reputations. It departs from correlation- based fairness metrics that fail under negative collaboration gain and instead proposes metric specifically designed for fair collaborative learning. CYCle extends naturally to Gossip-SGD, and we show both empirically and theoretically that it achieves better alignment between contribution and reward, particularly under data heterogeneity.
2.1 Preliminaries
Notations. Let Ma X + y be a supervised classifier parameterized by 0, where X S Rd and y = {1,2.-.. } denote the input and label spaces, respectively, d is the imput dimensionality, and M is the number of classes. Let MM : X' + 2 denote a mapping from the input to the logits space (2 c RR) and or be a softmax function (with temperature parameter T) that maps the logits into a probability distribution p=or(e oz(2) over the M classes. The sample is uvlly assigned to the class with the highest probability. Given an imput sample æ e and its ground truth label y E y, let LCe denote the cross-entropy (CE) loss based the prediction (MM()()) and y. We denote LDLN as the pairwise distillation loss between two models Me; and Ma,- For example, the distillation loss (DL) could be computed as KL divergence between (TMMe (a)) and or(Ma,(a)) Note that other types of distillation losses (Gou et al. 2021 can also be used in lieu of KL divergence.
Decentralized Learning. Let N be the number of collaborating participants and N = 21, NJ. We assume that each participant Pa, n N has its own local training dataset D, and hold-out validation set D,, Based on the local training set, each participant can learn a local ML model MM, by minimizing the local cross-entropy loss LCE. The validation accuracy of this standalone local model Ma, on D, is denoted as B,. The goal in a typical FL algorithm is to build a single global model Ma. that has a better accuracy than the standalone accuracy of all the local models. In contrast, the goal of each participant in decentralized learning is to obtain a better local model Møg, which has a higher local validation accuracy An after collaboration. In other words, each participant Pn aims to maximize its collaborative gain (CG), which is defined as Gn- = (An Ba). This is typically achieved by minimizing the following objective:
C. = LCE. + AnL
(1)
where CDL., denotes the total distillation loss of P., and An > 0 is a hyperparameter that controls the relative importance of the cross-entropy and distillation losses. Note that the term DL in the above formulation is
used generically (without restricting to any specific loss function), and it intuitively captures the differences
between the local model of participant and the models of all its collaborators. The total DL of P, can in
3

---

Published in Transactions on Machine Learning Research (08/2025)
P1
Weighted combination of cross-entropy and distillation losses is used to update the model Ma.
Participant Pi
P(..)
CCB, ExL,
L1 LOB, + A1LDL,
D1
((P))))
Ebs (Exe, (E(CnLo))
D1 : (21
Mo.
LDL.
N -1
Participant Pa
Eneryption- friendly pairwise distillation losses be securely computed in enerypted domain
M
E(21,2)
E(C ((Ckin))
Participant Pa
E(21,)
E(CRLqs))
Figure Illustration of confidential and private decentralized (CaPriDe) learning framework for N participants, showing the learming process for only participant Pı. Here, D1 (A1.34 = 1511 |Pah is the local data of Pı, (24) denotes the collection of enerypted unlabeled samples of Pi, and P(1.)) and =(1.)) are the prediction probabilities and logits obtained by applying model Mæ, of participant Pi to XT. Black dashed line represents exchange of encrypted data between participants in cach round, and blue dashed line denotes a single transfer at the beginning of the protocol. d decryption method. Loe and LDL represent the cross-entropy and distillation losses, respectively. In the context of our paper, LDLLM = LKL, d d( (ExtoL)))
turn be computed by aggregating the pairwise distillation losses follows:
CDL. ΣxENT(n) (n
(2)
where LDL is the pairwise DL between Pn and Pk; and A(,,)) are the weights assigned to the pairwise
losses between different participants.
Private Decentralized Learning (PDL). In the PDL framework, the total DL is computed in privacy- preserving way without leaking the local data of participants. For example, CaPC lcarning Choquette-Chor el al. 2021 leverages secure multi-parly computation, homomorphic encryption, and differential privacy (DP) to securely estimate CDL,, However, CaPC learning requires a semi-trusted third-party privacy guardian to privately aggregate local predictions, and the pairwise distillation losses are not accessible. contrast, CaPriDe learning (Tastan & Nandakumar 2023) utilizes fully homomorphic encryption (FHE) to securely compute approximation of the pairwise DI. losses. However, all existing PDL algorithms lack mechanism to assess the contributions of different participants within the CL framework. Hence, A(n,k) is usually set to [W-1] and An λo, V E A. Ao is typically determined through hyperparameler search. We refer to the above sccnario as vanilla PDL (VPDL).
CaPriDe Learning. CaPriDe learning framework (Tastan Nandakumar 2023 as described above, uses knowledge distillation approach and homomorphic encryption enable secure and private computations. Figure 2 illustrates how CaPriDe learning works. Let E(X,) = (Ex,,.)) (P.l be the collection of encrypted input samples of participant P... Here, the encryption is based on an FHE scheme with the public key of P,.- In CaPriDe learning, each party initiates the collaboration by publishing (X,,) to the other participants. Let k), and ()) denote the logits vector and probability vector obtained when model Mo, of Pe is applied
4

---

Published in Transactions on Machine Learning Research (08/2025)
the input sample Jj.; ie., the jth training sample of Px. Assume that at round 11, <t < of the ionabooaaioo,, the current model of Pn is Mos Then, Pn performs one forward pass on its own training set Dr to obtain the predictions (P). )/nl To enable knowledge transfer between participants, CaPriDe learning makes use of the knowledge distillation (KD) approach (Hinton et al., 2015), where a student model learns to mimic the predictions of a teacher model. In CaPriDe learning, there is no designated teacher model, and mutual KD between multiple peer models is used for knowledge transfer. Let DDL..)) denote the pairwise distillation loss between predictions of Pr, and Pk, which is defined as:
Dnl
CDL
Σ
DDL (Ps.i))))))P((())))) CKL. d (8 (ExIc..))
(3)
where DDL(PI.P2 P2)
denotes the mimic distance between two predictions P1 and pa and
iPal
[Dal
E (2),(n))
CKL_ = Σ (Pilmm) log P..(mm)) E (Extum)) = Σ & (pplmm))
(4)
T
j=1
where T temperature parameter and a is encryption-friendy approximate KL divergence as per CaPriDe learning (Tastan & Nandakumar 2023
3 Towards a Better Fairness Metric
In Lyu et al.
2020 Xu & Lyu 2021
collaborative fairness (CF) is defined follows:
Definition (Collaborative Fairness). In a federated system, a high-contribution participant should be rewarded with better performing local model than a low-contribution participant. Mathematically, fairness can be quantified by the correlation coefficient between the contributions of the participants and their respective final model accuracies.
In the above CF Definition | the contribution of a participant is typically evaluated based on its standalone local accuracy (B,). Thus, the CF metric attempts to ensure that participants with higher standalone accuracy receive a model with higher final accuracy (An) compared to that received by other participants. However, the problem with this approach Pi and P2 with standalone accuracy of 60% and 80%, respectively. Suppose that after collaboration, P1 (P2) receives model with an accuracy of 70% (70.1%). In this scenario, the system is considered perfectly fair (with a fairness measure of 1) according to the above CF definition, because P2 receives a model with higher accuracy compared to Pı. However, it is obvious that only P1 benefits from this collaboration and achieves a CG of 9 = 10,, whereas P2 gains nothing. Strictly speaking, Pi has a negative CG of G2 = -9.9%, but it can always dump the collaboratively trained model and use its own standalone model to achieve zero CG. This scenario cannot be considered fair, because the participant with the lower contribution benefits significantly without helping the participant with higher contribution.
be illustrated using a simple example. Consider two participants
The problem with CF quantification in Definition 1 is that it overlooks the scenario where the collaboration gain is negative. This is usually not the case in FL with iid settings, because all the participants have similar standalone accuracy, and the accuracy of the collaboratively learned model is much higher than any of the individual standalone accuracies. However, in extreme non-iid settings (which where collaborative fairness is most nceded!), the problem of negative CG can be encountered often. Hence, there need for a better metric to quantify collaborative fairness.
One possible solution is to measure the rewards in terms of CG rather than final model accuracy. Fairness is then quantified by the correlation between participant contributions (B,) and their CGs (G..) However, this metric is problematic: participants with high standalone accuracy are harder to improve. Going back to the earlier example, if P1 improves from 60% to 70%, P2 has to improve its accuracy from 80% to more than 90% to achieve a fairness score of 1. While it much casier to satisfy the first condition, is hard to meet the second condition using any CL algorithm. Hence, the fairness metric based on the correlation between standalone accuracy and CG is likely to be negative for most cases, indicating poor fairness.
5

---

Published in Transactions on Machine Learning Research (08/2025)
A better alternative is to minimize the probability that the CG is negative zero, i.e. P(gn 0) should be small. Let Jr [[] and v /Var denote the mean and standard deviation of the collaboration gains, respectively. Intuitively, this is possible only when I is large positive value, and if is small. Mathematically, using the well-known Chebyshev single-tail inequality, we can show that P(gn Σ 0) is bounded by 1ン (I+(n/2)9J) provided μ > 0. Thus, minimizing P(gn 0) requires maximizing μ and minimizing v. Let the mean collaboration gain (MCG) and the collaboration gain spread (CGS) be defined as the sample mean and standard deviation of CG across participants, ie.,
N
Σ Gn. MCG
1 N
CGS
Σ Gn, - MCG)². R=1
(5)
(N
n=1
Hence, an ideal decentralized learning scheme should maximize MCG (achieve better utility), while at the same time minimize CGS (ensure that all participants benefit. fairly). This minimizes the likelihood of a participant being negatively impacted by the collaboration (having a negative CG).
4 Proposed CYCle Protocol
Algorithm CYCle (t, k, Vo, CCE.)
Assumptions and Scope. In this work, we restrict ourselves to the PDL framework pre- sented in Eqs. D and 2 We also assume cross-
Input: Momentum factor
1: Φ + Is Update Available from PE?
2: Reputation Scoring
silo setting, where the number of participants is relatively small (N < 10) to ensure the prac- tical feasibility of PDL. Our method requires a PDL algorithm that can compute pairwise DL in a privacy-preservnn way. While we em- ploy CaPriDe learning (Tastan & Nandakumar 2023) as the baseline method for PDI. in this work, other alternatives plete description of the bascline PDL method and discussions about its convergence, commu- nication efficiency, or privacy guarantees are beyond the scope of this work.
3: == then
CDL(
else
LDIa Get DL from Pe.
if (t mod R) 0 then
st cos( Vo., CCE. Vo,, (DM(N)))
8:
2
also be used. Com-
h(s)
10:
((m.A) f- or ir1 (n,
+ a))
11: end if
12: end
13: Adaptive Sharing
Rationale. To maximize its individual CG (and hence maximize MCG), each participant must intuitively give more importance to dis- tillation losses corresponding to other reliable participants, while de-emphasizing distillation losses corresponding to unroliable participants.
14: if (t mod R) then
15: Share predictions with Pe
16: else
17:
+ Uniform (()
18:
S T(k.n) then
Share predictions with Pk 20: end if
19:
To minimize CGS, each participant must penal-
ize other unreliable participants (who do not contribute to their own learning) by sharing less knowledge with them. To achieve both these goals, we need to solve the following two sub-problems: (i) Reputation Scoring (RS): How to efficiently evaluate the reliabille biliityyy////i /rtpptepppuoonnnooe T(n,A) of participant Pa in the context of collaborative learning of model Mg: belonging to P,?, (ii) Adaptive Sharing (AS): How to regulate knowledge transfer among participants based on the reputation score to achieve collaborative fairness?.
21: end if
22: return (F/m..)) (DL(M)))
4.1 Reputation Scoring
Several reputation scoring methods have been proposed in the FL literature (Xu Lyu 2021 Zhang et al., 2021). These reputation scores are generally used by the FL server either for client selection or for weighted aggregation. In contrast FL, where global model is learned by a central server, each participant builds its local model in PDL. Hence, it is not possible to compute single reputation score for PDL participant.

---

Published in Transactions on Machine Learning Research (08/2025)
Instead, each participant has its own local estimate of the reputation of other participants. Hence, the same participant may be assigned different. reputation scores by their collaborators.
The proposed reputation scoring is based on the intuition that the gradient alignment between local CE loss and pairwise DL a strong indi- cator of the utility of P& to Pn. For example, if the two gradients (either because the data distributions of Pk and Pn do not have any overlap or because Pk is malicious), attempting to learn from Pk is likely to harm the learning of Pn. rather than being beneficial. On the other hand, if there is per- feet alignment, the two gradicnts reinforce cach other and accelerate the model learning at Pn- In fact, if KL divergence is used for pairwise 7:
Algorithm Private Decentralized Learning (PDL) based on Proposed CYCle Protocol
Input: Number of participants N, maximum commuicatiin rounds Imaz and learning rate 1j
completely misaligned
Assumption: Each Pn has D, = [2 MjnnJ=1 PPal Ma; + Local training Pn, VnEW for each round t= 0, 0, .tmax do
3:
for each participant E do
4:
LCE, + Compute cross-entropy loss of Px Vor.,E. t Compute the gradient of foE. for each participant & E . [n) do (r/m.)) LDL(.ss) + CYClo(t, k. VLCE.)
5:
6:
DL computation, the gradients based on the local CE loss and pairwise DL will be closely aligned only when the predictions made by Pk on P, 's data closely match the ground-trute la- bels available with P.. This is because CE loss measures the "distance" between or (Me, (ar) and y, while DL loss mcasures the "distance" between (MM,, (a)) and ot oMMa, (a:)). These losses will lead to aligned gradients only when ør (Ms. (œ)) is close to This justifies our choice of using the gradient alignment between CE loss and pairwise DL as the basis for reputation scoring.
end for
M(.K) & +((mk) LDI.. + >EEeA(n) A(m.)) LDLLaN)
10:
11:
Cu t LCEn + do LDL.
0h t -- fnn 13: end for
12:
14: end for
The reputation a participant Pk from the perspective of Pn (denoted as r(n,))) is computed in two steps. First, the gradient (mis)alignment metric (m,,)) computed as:
1-
cosos cos (Vo, LCE., Ve,, (DLLNN))
((,
(6)
2
where CoE, and Ve, LDL:.., are the gradients of cross entropy and distillation losses, respectively, with respect to the current parameters 0n of the participant Pn,, n E N, and k (M). Note that ((,,)) + 0 indicates better gradient aligmment. We empirically observed that when the gradient alignment $(n,) is below a threshold Topt, collaboration with P& is mostly beneficial to Pa- This is because as long as the predictions of PA are closer to the ground-truth at Pa, learning from Pa will help Pn. In contrast, when 8(m,)) above a threshold Tmaz, collaboration with Ps becomes harmful to Pn. Based on these observations, we compute the reputation score by applying soft clipping function to ((, Specifically, the reputation score in the current round ((n,k) = ())(,,))), where h is defined as:
h(s)) max (D, (1,
Tmaz
Topt Tiaax
To minimize variations due to stochasticity, the reputation score is updated using momentum factor, iee. rt (m,)) =ar((_)) + (1 (1- a))))....)) where 0 < a is the momentum hyperparameter and r(m,)) is the final reputation score in round t (t 0). When t = 0, r/n,,) f(n,A)- Heneeforth, we drop the index t for convenience. A higher reputation score T(n.k) implies that updates from Pa are useful for model learning at Pn- Hence, Pn can directly utilize T(a,k) to weight (A(m,a)) the pairwise distillation loss. While it is possible to perform reputation scoring and dynamically update the weights A(n,k) after every collaboration round, it can also be performed periodically (after a fixed number of rounds) minimize computational costs.
4.2 Adaptive Sharing
Once the reputation scores are computed, Pn can use T(n.)) to choose its collaborators wisely. Specifically, Pn, can decide if it needs "to share or not to share" the distillation loss LD with Pt. To be precise, P..
7

---

Published in Transactions on Machine Learning Research (08/2025)
computes and sends CDL..., to Pk with probability '(a.k). This adaptíve sharing mechanism plays key role in ensuring collaborative lairness. In order to receive updates from Pa in the current collaboration round, participant Pk needs to maintain a high reputation score with Px. This, in turn, requires the sharing of updates that are useful for model learning at Pn in the previous round.
The above reputation-based adaptive sharing scheme has the following advantages: (1) Tt incentivizes Pe to share honest and useful updates with Pa. Sharing noisy or malicious updates will hurt Pa's s reputation score and hence, dampen PA 's ability to learn from Pn. (2) If Pa finds that Ps has low reputation score, it will send updates less frequently to Pk. thereby saving valuable computational and communication resources. Note that in the PDL framework, privacy-reeeervig DL loss computation is often computationally expensive. However, there is one limitation in the proposed protocol. If Pn, is malicious, it can deviate from the protocol and stop sending updates to Pk even though T(n.k) is high. While this can be addressed by penalizing participants (by reducing their reputation score) for not sharing updates, such an approach is not desirable because will force participants to keep sharing updates even they are not benefiting from the collaboration. Hence, we go for a compromise solution, where all the parties are forced to share updates after every R collaboration rounds, and the reputation scores are recalibrated based on these responses.
4.3 Theoretical Comparison with FedAvg via Mean Estimation
To illustrate the theoretical benefit of CYCle over traditional FL approaches such as FedAvg (McMahan
et a.l. 2017), consider simple mean estimation problem with two clients (N = 2), each mininizing
(w( (w- (w-ee 02)² using limited local samples. Let 0, >V(0,, and 0g (V2.. denote the empirical estimates of cach client, where 72 2² /N is the estimation variance.
In Fed,Ag, the clients compute the global model as w (0) + /2) However, as shown in prior work (Cho let al.] 2022), the probability that this shared model outperforms standalone estimate exponentially suppressed with increasing data heterogencity:
22
P (Fa(w)<F((OA) < 2 2xx
S
(8)
572
where 38 = ((0) 02) quantifies heterogeneity. This shows that FedAvg performs poorly when the client distributions are dissimilar.
In contrast. CYCle uses reputation-weighted aggregation: w w= (1 - r/2) 01 + +r/2- 02 where the reputation score depends the empirical distance between 0, and 02.
Theorem 4.1. The probability that the model obtained through CYCle's collaborative aggregation improves upon slandalone model is lower bounded as
P (FF((w) < FL(())) 2 exp
4
Unlike FedAvg, this bound is independent of heterogeneity 76. highlighting CYCle's robustness to client diversity. Full derivations and proofs are provided in Appendix A
4.4 Extension to Gossip- SGD
CYCle can be integrated with decentralized optimization algorithms such as Gossip-SGD (Boyd et al. 2006 Koloskova et al. 2019) by modifying the communication mechanism through dynamic, reputation-driven mixing matrix. In classical Gossip-SGD, each client averages model updates with neighbors using a static mixing matrix. We replace this matrix with that evolves based on alignment scores between gradient directions, capturing the degree of collaborative benefit.
Formally, during each communication round, each node computes the pairwise cosine similarity with its neighbors' gradients and maps the resulting score using a softmax function controlled by a hyperparameter B. The updated mixing matrix W is then used to sample binary interaction mask Bernoull((W), ensuring that communication occurs only with beneficial peers. This mechanism balances collaboration and robustness by probabilisticall encouraging high-alignment links while discouraging noisy exchanges.
8

---

Published in Transactions on Machine Learning Research (08/2025)
We detail the full algorithm and its integration into Gossip-SGD in Appendix where we also describe the mapping functions and sampling strategy. Empirical results on the Fed-ISTC2019 dataset confirm that this extension maintains fairness and improves utility under realistic, non-i..d. data conditions.
5 Experiments
Datasets. CIFAR-10 (Krizhevsky et al. 2009) is dataset consisting of 60000 images of size 32 x 32 pixels, categorized into 10 classes with 6000 images per class. It has 50000 training and 10000 test samples. CIFAR-100 dataset [Krizhevsky et al. 2009) shares similarities with the CIFAR-10 dataset, but it consists of 100 classes with 600 samples per class. The training and test sets are divided in the same way as in CIFAR-10. The CIFAR-100 dataset used with data augmentation (random rotation (up to 15 degrees), random crop, and horizontal flip). Fed-ISIC2019 (Ogier du Terrail al. 2022) (from FLamby benchmark) is multi-class dataset of dermoscopy images comprising 23, 247 images with 8 different melanoma classes and high label imbalance (ranging from 49% to less than 1% for class 0 to 7). The dataset is designed for 6 clients based on the centers. Train/test split is: (9930/2483), (3163/791), (2691/672)) (1807/452) (655/ (14)), (351/88).
5.1 Experimental Setup
Baseline approaches. We consider several existing FL algorithms baselines: vanilla FL. based on FedAvg
(FedAvg) [McMahan et al. 2017), FL based cosine gradient Shapley value (CGSV) (Xu et al.
2021
collaborative fairness in FL (CFFL) (Lyu et al. 2020), and robust and fair FL (RFFL) (Xu & Lyu
2021
The above methods are designed for FL with central orchestration. As mentioned earlier, the baseline VPDL
method is based on CaPriDe learning (Tastan & Nandakumar 2023). We also evaluate standalone (SA)
accuracy, where each participant trains its ML model on its local dataset without any collaboration with others. Furthermore, we report the results for these baseline methods on CIFAR-10 and CIFAR-100. We do not compare against Cromus similar docentralized methods, as they focus robustness under adversarial conditions and omit fairness evaluation. Since our goal is to promote fair collaboration, instead benchmark against fairness-aware FI. methods. We leave integration with robustness-centric frameworks for future work.
Participant data splitting strategies. We implement three types of strategies to split the training dataset among the participants: (i) homogeneous, (ii) heterogeneous, and (iii) imbalanced dataset sizes (denoted as 'imbalanced'.. A homogeneous setting refers to the case where each participant gets an equal number of data points per class. In contrast, the heterogeneous method assigns varying number of data points to each participant, based on Dirichlet(6) distribution. The parameter 5 reflects the degree of non-IID characteristies within the dataset, where smaller values of 5 lead to more heterogeneous setting, while larger values tend towards an IID setting. To determine the specific allocation, we sample ~~ Dir (6) and assign a fraction Pm, of the lotal data samples to Pn. We employ these settings for CIFAR-10 and CIFAR-100 with the following mumber of participants: N = 2,5, 10. In all our experiments, we utilize all training samples (e.g., all 50000 samples of CIFAR-10 and CIFAR-100). In the case of the imbalanced data distribution, we utilize a custom function that relics on parameters and m, where g determines the proportion of data points received by each of the m chosen participants. The remaining (N - m) participants then share the remaining data among themselves. We explore this setting with N = 5 participants and consider the following configurations: (i) imbalanced (w = 0.8, m = 1), (ii) imbalanced (x = 0.35, m = 2). and (iii) imbalanced -==00 0.6, m = 1). For example, in the last case, one participant holds 60% of the data, while the remaining 4 participants get 10% of the data each.
Evaluation metrics. We assess the accuracy of the trained models based on their mean validation accuracy (MVA) and mean collaboration gain (MCG) across all participants. To evaluate collaborative fairness, we use the collaboration gain spread (CGS). Reputation scores are reported in matrix form to determine the relative contribution of each client.
5.2 Experimental Results
We start by benchmarking existing algorithms on the CIFAR-10 dataset. Table summarizes the results for different splitting strategies that involve N participants. Our key findings are follows:
9

---

Published in Transactions on Machine Learning Research (08/2025)
Table Performance comparison on CIFAR-10 dataset: Validation accuracy evaluated with N participants. The top section of the table presents the performance of our proposed framework compared the FedAvg algorithm. The bottom part of the table compares the collaboration gain and fairness of our proposed CYCle algorithm with existing works (rows 6-9). We use MVA (t), MCG (f) and CGS (4) as eval.
metrics.
Setting
Homogeneous
Dirichlet (0.5)
Imbalanced (0.8, !)
Imbalanced (0.35,
Imbalanced (0.6, 1)
Metric
MVA
MCG
MVA
MCG
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
FodAvg
90.60
7:20
0.48
88.76
21.40
4.31
90.17
26.35
14.51
90.34
13.12
9.61
90.16
9.11
VPDL
1.58
0.71
3.58
78.71
75.43
1.35
2.45
CYCle
86.33
2.93
0.32
76.93
9.57
2.07
69.26
5.44
2.56
81.12
3.89
2.65
76.72
2.83
1.38
CFFL
62.65
2.21
0.87
49.04
2.49
2.35
58.66
3.48
4.39
65.08
9.99
9.95
65.20
14.35
9.79
RFFL
61.50
1.05
0.95
18.52
1.96
1.63
56.27
1.08
2.45
58.51
3.43
7.16
51.35
0.50
6.74
CGSV
63.27
2.83
0.94
64.45
7.90
3.40
56.96
0.78
9.32
61.99
6.91
11.34
61.34
10.49
11.23
CYCle
71.95
11.51
0.58
51.21
4.65
1.11
61.80
6.62
2.79
68.26
13.17
7.12
64.93
14.08
6.22
Standalone
FedRvg
VPDI
CYCle
P2
80
90
Homogenneuss
Heterogeneous (-Dir(0.5)
Imbalanced (0.8,
Imbalanced (0.6,
Imbalanced (0.35, 2)
Figure 3: Per-participant performance comparison on CIFAR-10 dataset: N participants.
Validation accuracy evaluated with
We assess the performance of three algorithms FedAvy, VPDI, and CYCle using the ResNet18 architecture. These results shown in Figure 3 and in the upper section of Table [ (rows 3-5). FedAvg has better MCG compared to other methods due to its centralized FL approach, which involves aggregating and sharing full model parameters, resulting in good overall utility. However, when comes to collaborative fairness, FedAvg falls short of being fair to participants who contribute a larger amount of data. In particular, the CYCle approach exhibits significantly lower CGS values compared to the FedAvg algorithm in the imbalanced settings. Additionally, our method consistently provides positive collaboration gains all participants in various settings. This is not the case for FodAvg VPDL, which is evident in the outcomes for Pi in the imbalanced (0.8, and (0.6, settings.
For comparison with other fairness-aware FL algorithms, we utilize custom CNN architecture used in Xu et al. (2021). This necessary because CGSV works better on simpler architectures. The results are shown in Fig. 4 and the bottom part of Tab. (rows 6-9). Our method has better comparable CG values when compared to CFFL, RFFL, and CGSV in most of the scenarios. In terms of fairness, we surpass other approaches by achieving the lowest CGS values in all but one scenario: the imbalanced (0.8, 1) setting, where it is on par with RFFL. Notably, CYCle consistently achioves positive gains for all participants, whercas other methods dograde performance for the most contributing participant (see Figure
Next, we assess the performance of our approach on the CIFAR-10 and CIFAR-100 datascts with different numbers of participants (N E (5, 10}) and data splitting scenarios (Table 2 In all settings, we consistently achieve a positive collaboration gain for all participants, and the CYCle method consistently has lower CGS compared VPDL, demonstrating improved fairness across participants. Furthermore, we observe that when integrated with Gossip-SGD (DSGD), CYCle maintains this trend: while the baseline DSGD achieves high MCG values due to aggressive mixing, it often results in large CGS values, indicating significant disparity
10

---

Published in Transactions on Machine Learning Research (08/2025)
Standalone
CFFL
RFL
COSV
CYCl
70
70
70
Homogeneous
Heterogeneous (-Dir(0.51)
Imbalanced
Imbolanced
Imbalanced (0.35.
Figure Per-paaticipant performance comparison on CIFAR-10 dataset using custom CNN architecture from Xu et al. h021 Validation accuracy evaluated with N participants.
Table 2: Collaboration gain and spread (MCG, CGS) CIFAR-10 and CIFAR-100 datasets and the given parlition stralegies for N (5, 10) using proposed algorithm. DSGD refers to the Gossip-SGD algorithm.
Dataset
CIFAR-10
CIFAR-100
N
5
5
10
Split
Method
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
VPDL
84.98
1.58
0.71
71.01
1.22
L.58
56.74
7.58
1.20
40.87
10.42
2.48
CYCle
86.24
2.84
0.37
73.66
4.57
0.71
37.56
8.40
0.69
41.17
10.72
1.15
Homageneons
DSGD
84.05
8.86
0.16
80.57
13:75
0.48
56.45
19.28
0.19
49.79
23.95
0.38
CYCle
84.22
9.03
0.16
79.84
13.02
0.30
56.07
18.90
49.41
23.57
0.23
VPDL
74.27
6.91
2.31
50.64
1.24
2.24
46.36
10.51
2.59
27.70
3.41
2.80
CYCle
76.93
9.57
2.13
53.82
4.42
1.98
47.98
12.13
0.68
28.47
4.18
1.07
Dirichlet (0.5)
DSGD
81.83
22.98
3.54
77.16
28.26
4.92
50.16
21.74
2.84
44.38
24.12
2.11
CYCle
81.14
22.30
2.60
75.82
26.92
3.71
49.61
21.19
0.95
43.91
23.65
1.16
VPDI
81.26
3.38
5.51
5.00
56.31
12.43
2.31
9.71
3.16
CYCle
83.13
8.48
37.61
13.73
1.97
Dirichlet (2.0)
DSGD
83.66
12.64
0.97
79.86
18.25
2.29
55.53
21.19
0.97
50.21
25.89
0.61
CYCle
83.48
12.45
0.68
79.64
18.03
1.87
56.36
22.01
0.89
50.59
26.27
0.57
VPDI.
83.32
2.39
1.94
72.37
4.54
3.96
58.18
9.80
1.38
41.86
10.03
2.92
CYCle
84.80
3.86
1.15
74.83
7.00
3.72
59.67
11.29
1.08
41.92
10.09
1.38
Dirichlet (5.0)
DSGD
83.72
10.02
0.56
80.38
15.65
1.17
56.06
20.02
0.54
50.33
24.94
0.67
CYCle
83.85
10.15
0.33
80.46
15.73
0.77
55.87
19.83
0.48
49.62
24.24
0.43
among participants. In rastrast, CYCle significantly reduces CGS without compromising MCG, highlighting its ability to rogulate knowledge flow more equitably in decentralized and peer-to-peer settings.
Reputation scoring visualization. Figure 5 visualizes the reputation scores computed three collabora- tion rounds: / 0, 25, 75 using heatmap plots. In Figure 5a] where participants have imbalanced data (with Pi having 80% of the data), method can already identify distinct patterns of each participant's data value from the beginning of the collaboration. The heatmap figure emphasizes that higher importance is given to
Round
Round 25
Round 75
Round
Round 25
Round 75
1.0
1.0
0.59/0.60
0.66
0.31 0.39..33
0.228 0. 0.23
0.88
0.67
0.59
0.70
0.27 0.28
0.22
0.14
0.34
0.14
0.930.933
0.70777
01..54
0.65
0.37
0.15 0.15
0.15
0.95
0.93
0.73
0.47
1.00
0.96
1.00
0.93
0.52
0.44
m
00
0.94
0.97
0.88
1.00
55
0.55
1.00
0.28
0.33
0.30
1.00
0.89
0.92
1.00
3.00
0.95
1.00
1.00
0.65
0.61
02
0.2
00
0.85
0.86
5.06
86
1.00
54
0.54
0.63
.000
0.
32
38
0.40.3
1.00 0.99
0.79
0.77
77
0.95
0.93
0.60
0.58
6.82
8.80
0.36
0.330
0.0
0.0
(a) Imbalanced (0.8, 1) setting
(b) Imbalanced (0.35, 2) setting
Figure 5: Heatmap visualization of reputation scores for imbalanced settings when N 5.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Pi by all the other participants. As the collaboration progresses, the reputation of P1 intensifies further, while the reputation scores of other participants decrease. Toward the end of the collaboration, the values converge to a state in which participants with an equal amount of data receive equal reputation scores. A similar pattern can be observed in Figure 5b where the first two participants possess larger amounts of data.
While the first five rows of the healmap figures depict the raw reputation scores calculated using Equation 7 the sixth row at the bottom represents the average reputation scores aggregated over all the rounds.
Table 3: Validation accuracies of participants (P1, .... Pg)
Filo rate 0.2
Fliprate 0.5
Flip rate 1.0
0.63
063
0.73
0.00
0.70
0.66
0.664
0.00
0.740.56.74
0.00
with P3 having varying rates of label flipping. The last col-
0.20
2.0
1.0
0.85
0.590.7
0.8
umn refers to the average accuracy of honest participants
(P1, Pa). The corresponding figure with reputation
scores: Fig.
0
0.66
0.700.61
0.4
0:50
0:44
0.48
0.18
0.05
0.00
0.00
0.00
0.00
0.2
Setting
P1
Pa
Pa
P4
Ps.
avK
0.70
0.65
0.67
0.72
2.001
0.66
0.55
0
0.61
0.00
0.53
0.53
0.57
0.57
0.00
Standalone (B,)
83.42
83.67
83.15
$3.13
$3.63
83.34
Plip 0.0
$6.12
87.03
86.33
85.62
86.10
86.28
Figure 6: Heatmap of reputation scores when Ps undergoes label flipping. Higher flip rates lead to
Flip rase 0.2
84.95
84.5
84.5
86:45
00.95
85.10
Flip 0.5
84.25
84.65
84.65
63.25
55.80
Flip 1.0
83.9
84.5
83.96
86.1
10.75
84.36
Pg being rapidly identified malicious by others.
Free Rider Study, In this study involving N = participants with homogencous data splitting scenario, we simulate varying degrecs of label flipping at participant Pg. We tested the label flipping rales of 0.2, 0.5, and 1.0. From Figure G it is evident that honest participants (participants PI,...,PP P4) quickly discern the anomaly with Pss detecting it as a potential threat. Consequently, they assign a reputation score of 0.0 to Ps, indicating a cessation of updates to this participant. On the other hand, Pg accurately recognized that the other participants were beneficial when the label flipping rate at 0.2, but this positive assessment drops to zero for the case when the label space of Pa becomes completely different.
Pearson Carn: 0.9577, p-val: 0.0026
Balanced Accuracy
0.6
0.4
0.2
SA
CYCle
0.0
P1
P2
P3
P4
P5
P6
Participants
Figure 7: Per-participant accuracy under CYCle (Fed-ISIC2019).
Table 3 aligned with Figure 6 details the values of Bn and A.. It reveals that, in the presence of flipped labels, the collaboration gain declines, reflecting a loss of a participant that holds 20% of the data.
Evaluation on Real-World Medical Dataset (Fed-ISIC2019).
Table 4: CYCle vs. Gossip-SGD on MCG/CGS
To validate our framework in real-world heterogeneous settings, we evaluate CYCle on the non-IID Fed-ISIC2019 dataset (Ogier du Terrail et al. 2022). This benchmark features complex inter-client shifts in class distributions and image types.
( (Fed-ISIC2019).
Method MCG
CGS
Gossi--SGD (Complete)
30.26
15.61
CYCle
23.79
10.94
Figure 7 shows the per-client validation accuracy using our CYCle
extension on top of Gossip-SGD. Despite differences client data sizes and class coverage, CYCle maintains high fairness across participants, as evidenced by strong correlation between client contribution and reward (p 0.9577). Table confirms that CYCle achieves the lowest CGS while preserving collaborative utility.
The complete experimental setup, alternate topologies, and detailed visualizations (including reputation dynamics and data distribution) are provided in Appendix G.3
6 Conclusion
This paper proposed novel reputation-basel adaptive sharing algorithm to promote collaborative fairness, specifically tailored for decentralized learning scenarios. By leveraging gradient alignment, CYCle dynamically adjusts knowledge transfer based on participants' contributions, ensuring positive and equitable collaboration gains. We further extended the protocol to operate scamlessly over gossip-basod decentralized lcarning algorithms. Through theoretical and empirical analyses, this work has illustrated the algorithm's superior capabilities in achieving collaborative fairness.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Broader Impact Statement
Privacy Protection Measures. CYCle inherits the secure knowledgg-sharring protocol of the CaPriDe learning framework (Tastan & Nandakumar 2023 which uses fully homomorphie encryption (FHE) to safeguard sensitive data. Specifically, each client only transmits encrypted prediction logits to peers, never raw data (c.g. medical images from Fed-ISIC2019) or model parameters. These encrypted outputs are used solely to compute a local distillation loss on peer-held datasets, and they are never decrypted by any party. This ensures that all sensitive information remains protected throughout the training process.
Abuse Risk and Mitigation. A common concern in collaborative learning is the potential for knowledge monopolization, where dominant participants attempt suppress or filter contributions from less performant peers. CYCle directly addresses this risk via its similarity-basod, bidirectional collaboration mechanism. Specifically, clients benefit most when they are aligned with others (as defined by gradient similarity), ) which discourages unilateral exclusion. Furthermore, the soft contribution mapping function h(s) makes it suboptimal for any client to reject collaboration entirely, as doing so reduces thoir own update quality. This design promotes a balanced and inclusive exchange of knowledge.
Acknowledgments
This material is partly based on work supported by the Office of Naval Research N00014-24-1-2166.
References
Jan Philipp Albrecht. How the gdpr will change the world. Eur. Data Prot. L. Rev., 2:287, 2016.
Anish Athalye, Nicholas Carlini, and David Wagner, Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International conference machine learning, pp- 274-283. PMLR, 2018.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor foderated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-20992 2948. PMLR, 2020.
Stephen Boyd, Arpita Ghosh, Balaji Prabhakar, and Devavrat Shah. Randomized gossip algorithms. IEEE transactions on information theory, 52(())2508-22)) 25300, 2006.
Centers for Medicare & Medicaid Services. The Health Insurance Portability and Accountability Act of 1996 (HIPAA). Online at http://www.cn/ wwt.:/cw.......o....o.. wowgg/oo/hia///a/aaaa 1996.
Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint ar Xrx:::1912. 11279, 2019.
Yae Jee Cho, Divyansh Jhunjhunwala, Tian Li, Virginia Smith, and Gauri Joshi. To federate or not to federate: incentivizing client participation in federated learning. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NewrIPS 2022), 2022.
Yac Jee Cho, Divyansh Jhunjhunwala, Tian Li, Virginia Smith, and Gauri Joshi. Maximizing global model appeal in federated learning. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL
https://openrevier
Christopher A. Sroqueete-Chooo Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Pa- pernot, and Xiao Wang. CaPC Learning: Confidential and Private Collaborative Learning. International Conference on Learning Representations, 2021. URL
wtt.://wwww..eeetooormrmaaadaa.. WMVq
Kate Donahue and Jon Kleinberg. Models of fairness in federated learning. arXiv preprint Xr 2021.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trendo0 in Theoretionl Computer Science, 9(3-4):211-405 2014.
Üffrr Erlingsson, Vitaly Feldman, Πya Mironov, Ananth Raghunathan, Shuang Song, Kunal Talwar, and Abhradeep Thakurta. Encode, shuffle, analyze privacy revisited: Formalizations and empirical evaluation. arXiv preprint ar -xxx2200.006111 2020.
European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council, The General Data Protection Regulation IaaiiattionnnGGDiSI ('GDPR'), 2016. URL https:
//data.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-1888 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015, URL 02531
bttps://aaxxivoor/abbbe
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1167-1176. PMLR, 2019.
Meirui Jiang, Holger R. Roth, Wenqi Li, Dong Yang, Can Zhao, Vishwesh Nath, Daguang Xu, Qi Dou, and Ziyue Xu. Fair federated medical image segmentation via client contribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16302 16311, June 2023.
Shutong Jin, Zhen Gu, Guangyan Li, Donglong Chen, Cetin Kaya Kog, Ray CC Cheung, and Wangchen Dai. Efficient key-switching for word-type fhe and gpu acceleration. Cryptology ePrinl Archive, 2024.
Peter Kairouz, Ziyu Liu, and Thomas Steinke. The distributed discrete gaussian mechanism for federated learning with secure aggregation. In International Conference on Machine Learning, pp. 5201-5212. PMLR, 2021.
Jongmin Kim, Wonseok Choi, and Jung Ho Ahn. Cheddar: A swift fully homomorphic encryption library for cuda gpus. arXiv preprint 2077:2447.18055, 2024.
Anastasia Koloskova, Sebastian Stich, and Martin Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In International Conference on Machine Learning, pp. 3478-3487. PMLR, 2019.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Scbastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pp.
PMLR, 2020.
Alex Krizhevsky, GeofTrey Hinton, et al. Learning multiple layers of features from tiny images. Toronlo, ON, Canada, 2009.
Sourav Kumar, A Lakshminarayanan, Ken Chang, Feri Guretno, Ivan Ho Mien, Jayashree Kalpathy-Crann Cramer, Pavitra Krishnaswany, and Praveer Singh. Towards more efficient data valuation in healtheare federated learning using ensembling. In Distributed, Collaborative, and Federated Learning, and Affordable AI and Healtheare for Resource Diverse Global Health: Third MICCAT Workshop, DeCaF 2022, and Second MICCAI Workshop, FAIR 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18 and 22, 2022, Proceedings, pp. 119-129. Springer, 2022.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Lingjuan Lyu, Xinyi Xu, Qian Wang, and Han Yu. Collaborative fairness in federated learning. Federated Learning: Privacy and Incentive, pp. 189 204, 2020.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication- efficient. learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp- 1273 1282. PMLR, 2017.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2018. URL https //openreview. wtt.:t//eooooomm/dd///of
Jean Ogier du Terrail, Samy-Safvan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, Boris Muzellec, Constantin Philippenko, Santiago Silva, Maria Teleúczuk, Shadi Albarqoumi, Salman Avestimehr, Aurélien Bellet, Aymeric Dieuleveut, Martin Jaggi, Sai Praneeth Karimireddy, Marco Lorenzi, Giovanni Neglia, Mare Tommasi, and Mathieu Andreux. Flamby: Datasets and benchmarks for cross-silo federated learning in rcalistic healthcare settings. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 5315 5334. Curran Associates, Inc., 2022.
Lloyd Shapley et al. value for n-person games. Princeton University Press Princeton, 1953.
Zhuan Shi, Lan Zhang, Zhenyu Yao, Lingjuan Lyu, Cen Chen, Li Wang, Junhao Wang, and Xiang-Yang Li. FodFAIM: A model performanco-basee fair incentive mechanism for federated learning. IEEE Transactions Big Data, 2022.
Nurbek Tastan and Karthik Nandakumar. CaPriDe Learning: Confidential and private decentralized learning based encryption-friennll distillation loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8084-8092, June 2023.
Nurbek Tastan, Samar Fares, Toluwani Aremu, Samuel Horváth, and Karthik Nandakumar. Redefining contributions: Shapley-driven federated learning. In Kate Larson (ed.), Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 5009 5017. International Joint
Conferences on Artificial Intelligence Organization, 8 2024. doi: 10.24963 htt/://aaa2////5/.. 2024/554. URL https:
//doi 10tgg/10.000
1000ooor//1....963333..oo 21w.2/24c3......22. 2024/554 Main Track.
Tianhao Wang, Johannes Rausch, Ce Zhang, Ruoxi Jia, and Dawn Song. A principled approach to data valuation for federated learning. Federated Learning: Privacy and Incentive, pp. 153 167, 2020.
Lin Xiao, Stephen Boyd, and Sanjay Lall. A scheme for robust distributed sensor fusion based average consensus. In IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005., pp. 63 70. IEEE, 2005.
Xinyi Xu and Lingjuan Lyu. A reputation mechanism all you need: Collaborative fairness and adversarial rosustuess in federated learning. In International Workshop on Federuted Learning for User Privacy and Data Confidentiality in Conjunction with ICML 2021 (FL-ICML. 21), 2021.
Xinyi Xu, Lingjuan Lyu, Xingiun Ma, Chenglin Miao, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Gradient driven rewards guarantee fairness in collaborative machine learning. Advances in Neural Information Processing Systems, 14:11044-66111 2021.
Hao Yang, Shiyu Shen, Wangchen Dai, Lu Zhou, Zhe Liu, and Yunlei Zhao Phantom: A cuda-accelerated word-wise homomorphic encryption library. IEEE Transactions on Dependable and Secure Computing, 21 (5):4895-444 4906, 2024.
Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, and Wotao Yin. Exponential graph is provably efficient for decentralized deep training. Advances in Neural Information Processing Systems, 34: 13975-13987, 2021.
Jingwen Zhang, Yuezhou Wu, and Rong Pan. Incentive mechanism for horizontal federated learning based on reputation and reverse auction. In Proceedings of the Web Conference 2021, pp. 147-956, 2021.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint aXXiv:2011.06610, 2020.
Zirui Zhou, Lingyang Chu, Changxin Liu, Lanjum Wang, Jian Pei, and Yong Zhang. Towards fair federated learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery 8 Data Mining, pp. 4100-4101, 2021.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
16

---

Contents
Introduction
1
2
Related Work and Background
2
2.1
Preliminaries
3
Towards a Better Fairness Metric
5
Proposed CYCle Protocol
6
Reputation Scoring
6
4.2
Adaptive Sharing
4.3
Theoretical Comparison with FedAvg via Mean Estimation
8
4.4
Extension to Gossip-SGD)
5
Experiments
9
5.1
Experimental Sctup
Experimental Results
Conclusion
12
Mean Estimation under CYCle Protocol
18
B Numerical Experiments
22
B.1 Varying Degrees of Heterogeneity
22
B.2 Imbalanced Data
C
Extension Gossip-SGD
23
C.1 Gossip-SGD)
23
C.2 Integrating CYCle with Gossip SGD
25
D Notation Summary
26
E
Computational and Communication Complexity
E.1
Computational Complexity
27
E.2
Communication Complexity
28
F
Experimental Setup. Implementation Details
28
G
Additional Experimental Results
29
G.1
Reputation Score Visualization
29
................................
G.2
Remaining Results of Table 2
30
................................
G.3
2SC-199C2019 experiments
30
G.4
Experiments on CIFAR-100
33
G.5
Imbalanced Data Study
33
G.6
Free Rider Study
34
G.7
Scalability to Larger Client Populations
34
G.8
Tradc-off between MCG and CGS
36
G.9
Hyperparameter Sensitivity
36
10
Additional Results
37
17

---

Published in Transactions on Machine Learning Research (08/2025)
A Mean Estimation under CYCle Protocol
Standard FL. The vanilla FedAvg algorithm, a standard FL approach, does not ensure an optimal final model for all participants in a federated setting. Consider a simple scenario involving two clients, N = 2, each aiming to estimate the mean of their data distribution by minimizing the loss function fx(w) = (w - 0x)². However, these clients can't compute the true mean directly due having only N& samples from their distributions, denoted ek. ~ [O v²), Vj [N+]. Consequently, they minimize the empirical loss function (w(w)) (wo - )²)² 2 + (0g- 0z)², where 0z N₂ Σ Ck.j, leading to Tx = 0g as the minimizer of (w)
1
Additionally, the following quantities are defined:
22
01
02 2
2
2 =
N
Note that the distribution of the empirical means themselves is distributed normally, due to the linear additivity property of independent normal random variables. This sctting is similar to CCo et al. 2022 2024,, albeit with few modifications.
0 N(O,,
02 ~ 72)
(10)
Lemma A.1. Cho et al. 2022 The probability (likelihood) that the model obtained from collaborative
training outperforms the standalone training model is upper-bourded by exx ( framework.
Ta
2
in standard FL
5
Proof. We analyze the probability that the global model obtained using FodAvg algorithm is superior to one trained independently (standalone). The model in the standard FL approach is defined as:
0
02
W
(11)
2
Then, the usefulness of the federated model can be mcasured in the following way:
P P(( - 011) S (0) - -11)2)
01 + 0g
2
01
s (0) - -112
(12)
2
0
+ 02
2
0
(0) 0, <0
(13)
2
02 01
2
02 0
P
2(1
01)
(14)
2
0g 01
2
02 0
n -0 - 81))
+ 2(0 01)
<
2
PP
02
01
+ +2 - 01)
02 1 0
s <0
20 n (0 s o1))
(15)
2
02 01
2(
01 00
50=00 50
n (02 (00=01)
2
02
0
O2 -1) ) s0 n pg sa))
$0
P
+
2
01)
(16)
2
18

---

Published in Transactions on Machine Learning Research (08/2025)
2
01 2
IP
+ +2(0 01) -0)==) + +P(02
- 81 s0)
(17)
12 2 0) + 22 50)
where Z1 N (1)..22))
5 " .2 ~ NN(T0:22773))
(18)
2
S exp
10
+ exp
20
S 2 2exp
26
(19)
5-2
2/2
572
where 15
uses P(A) = P(An B) + ( BC) )
uses
P(An B) S P(A), 18 uses the linear additivity
property of independent normal random variables and (10 (19
uses Chernoff bound.
CYCle Algorithm. The CYCle algorithm uniquely incorporates the reputations of cach client during the aggregation of their computed means. Considering the scenario from the perspective of Client 1, who possesses an estimate 0 and receives 0g from Client 2, the aggregation is expressed as:
12, 0 11.20 02 w1 = (1 -1-11211
(20)
2
As CYCle operates on a decentralized principle, each client develops its final model based on individual reputations denoted by r. For example, for Client 2, the model is:
U2 = (1
22 02
22 0 2
+
(21)
Notably, wy and w2 are not necessarily the same due to the distinct reputation scores. The following discussion will simplify the notation by dropping indices and focusing on the aggregation process from the perspective of Client 1.
Reputation Scoring. In the CYCle algorithm, each participant calculates the reputation of their collabo-
rators by assessing the distance between their shared means, denoted as d d= (0-00 01 used to determine their reputations as follows:
This distance is then
1.0
if 1
r
2-d
d
if < 2
(22)
0.0
if
Here, d represents the empirical heterogeneity parameter, referred to as 96. The rationale for the assigned
values is based on the degree of heterogeneity 22
If 22 < 1, it indicates minimal heterogeneity, suggesting that participants can benefit significantly from collaboration.
If 2/ 2, it signals substantial heterogeneity, advising participants to avoid collaboration.
Thus, the reputation scores are directly influenced by the empirical heterogeneity observed between the participants.
Cost Function. To evaluate the CYCle algorithm against the standard, contralized FL algorithm, we consider the empirical losses each participant ends up solving. In FedAvg, all clients achieve the common mean:
0
0
w
(23)
2
19

---

Published in Transactions on Machine Learning Research (08/2025)
and the empirical loss for Client 1 is:
Fi Fi(w)= Fi ( (6/ + 02
70 01)
(24)
This formulation shows that each client encounters non-reducible loss terms in the FedAvg algorithm. For the CYCle algorithm, the model aggregation is Erputation-weighted:
(1-2) 2 -
2)=² _ 02
(25)
yielding the empirical loss:
Fi(w) = 12 + (01 - 01)²
(26)
In CYCle, the empirical heterogeneity parameter 98 is adjusted using the reputation scores r, allowing each client to potentially achieve final model that is as good as or better than their standalone model. Conversely, in FedAvg all clients share the same final model. As the heterogeneity between clients increases (18), the efficacy (usefulness) of the global model significantly diminishes, as highlighted in Lemma A.1 This comparison illustrates the advantage of CYCle in managing heterogeneity among the participants to enhance individual outcomes.
Minimizing the CYCle Objective: Ensuring Positive Outcomes. Unlike standard FL where increasing true heterogeneity 16 drastically reduces the usefulness of the global model (as noted in Lemma A. CYCle modulates the aggregation phase using reputation scores which are derived from the empirical heterogeneity parameter 38 22
Theorem A.1 (Restated Theorem
The probabilily (likelihood) that the model oblained through collabo- ration (w) surpasses standalone-trained model (0) is lower bounded by exp (- - in CYCle algorithm..
Proof. We assess the performance (usefulness) of the collaborative model for client i using:
P( (w: 0.) s (0.-0,))
(27)
This analysis includes both of the scenarios: (i) 0, 02 which directly implies probability of 1.0, and (ii)
significant difference between 02 and 02 (32 > 0), leading to no collaboration, also resulting in a probability of 1.0. The CYCle algorithm is not interested in obtaining final global model and each client is self-interested in minimizing their own objective functions.
Case 1: 88 1
P (wn) - -11)2 S (0) - 01)2) 01)2
(28)
= IP (w)) - 0,)² + 21111111))))))11 - -1) s0)
01)
(29)
(0 0 2
02 0
(0) - 01) =0)=
(30)
2
02 01
2
0g
0
0 0) -0 0 (02 81) 01)
TP
+
2
2
02
0
2
0
0
( 01) s0) n 18g s 281
+P
(31)
2
(1 + 2001 01)
S 00 n 1020000 71)
(32)
-P((0 - 10 <
- n (02 > 01))
1
(33)
2
20

---

Published in Transactions on Machine Learning Research (08/2025)
PPO < 02) P(O 01 -
2 10 < 02)
(34)
2 P(O < 0)) P(O - 01 S
(35)
2
P( < 82) P(Z
> 27
where Z N(
(36)
N exp
(37)
4
where 30
uses the reputation score r when 9/2 < 1 (refer to 22
31 uses P(A) = P(ANB)+P(AN
32 uses the definition of 98 < 1 and the second part of the expression can be dropped since to can take
values from -1 to 0, it can be assigned zero probability, 355 uses P(A B) 2 P(A), (36) uses 01 -1 ~ (((,7²)
37
P(0 02) 2 2 and P(Z 2 x) 2 exp( exp(-x²2 where Z N(O, 1). The similar bound can be found for the
second client.
Case 2: 1 < 3 2
In this case, we can follow the similar approach above and we get:
P P(( (wn 0,)2 s (0: - -01)²)
(38)
-P((I) - -0,)² 3 2(w1 - 0,)(0 01) 20)
(39)
((5600 - 01)) + +r - 0,)(0, - 01) 0)
(40)
P (r((
02 - 2
02 0
2 + +
2 (0) 01) s0)
(41)
PP (1 + 2(0
01) <0]
n 0.))
(42)
1
IV
g
exp
4,2
(43)
The reason behind getting the same bound as in Case 1 is that the expression in 41 is maximized when r r= (2 - --8)== = 1, which necessitates 3a R I, when 0₁ < 02.
Case 3: 382 >
P (w) - =,,)²=000 -1))) = (00) - 0,) (0, - 61)3) =
(44)
where wi = (1 r
-1- - -0 -, 0, since r 0 for -/6 > 2 (Eqs.
25
22
As such, CYCle effectively adapts to various degrees of heterogeneity, ensuring that collaboration invariably enhances or matches the standalone training performance. This is quantitatively supported by a lower bound of exp (- 432 1/p) demonstrating robustness against heterogeneity effects.
Scope and Limitations. The result in Theorem 4.1 is derived under a simplified two-client mean estimation setting. While this toy model is useful to illustrate how collaboration may fail under heterogeneity and how CYCle can mitigate that through similarity-based reputation, it not intended as a universal lower bound for all learning socnarios. In particular, the lower bound on success probability does not generalize trivially to more complex settings (e.g., neural networks, high-dimensional tasks, or N > clients).
Extending the analysis to larger populations introduces new technical challenges. Specifically, when N > 2, (i) the reputation becomes a full matrix rather than a scalar, (ii) truc-mean gaps form a vector rather than single yo, and (iii) the same noisy estimate 0, appears in multiple clients' aggregates, introducing
21

---

Published in Transactions on Machine Learning Research (08/2025)
cross-correlations.
The bias term then depends the worst-case alignment across all pairs, while the variance term includes complex cross-terms from covariance expansion. Bounding these quantities is non-trivial and remains an open problem.
We include this example primarily to explain the mechanism by which data heterogencity undermines fairness in standard FT. approaches, and to motivate our design choices in CYCle. Developing more general guarantees is an exciting direction for future work.
B Numerical Experiments
B.1 Varying Degrees of Heterogeneity
For the first experiment, we adopt the scenario outlined in previous studies (Cho ct al. 2022 2024). For the simulation involving two clients, we define the true means of the clients 0, 0, Ta where Yo [0,5]] 5). The empirical means, 01 and 02, are sampled from the distribution NV 1) and M/20, 1) respectively, with equal sample sizes (N₁ N22 N). The average usefulness is calculated over 10000 runs.
The results, depicted in Figure 8 highlight the differing impacts of heterogeneity on the final model's usefulness. Under the FedAvg protocol, as heterogeneity increases, the global model's usefulness drops drastically, polentially disadvantaging each client by diverging from their optimal solutions (empirical means). Conversely, the CYCle approach exhibits an increase in the usefulness of the resultant models (w,) irrespective of the heterogeneity level. This indicates an effective mechanism within CYCle for clients to discern and select good collaborators based on the proximity their updates. In scenarios of pronounced heterogeneity, resulting in low reputation scores, clients are incentivized to rely more heavily on their own updates, effectively mitigating collaboration when it proves detrimental.
B.2 Imbalanced Data
In this experiment, we consider scenario where both clients share the same true mean 0, = 02 = 0, effectively setting the true heterogeneity parameter 28 = 0. Despite this, the clients differ in their number of
1.0
1.0
0.8
Usefulness of wi
Usefulness of W;
0.6
FedAvg
0.6
FedAvg
CYCle (Client 1)
CYCle (Client 1)
0.4
CYCle (Client 2)
0.4
CYCle (Client
0.2
0.2
0.0
0.0
0
10
15
20
25
5
10
15
20
25
Data Heterogeneity (vž)
Data Heterogeneity (v²)
(a) Over 1000 runs
(b) Over 10000 runs
Figure 8: Numerical study on the usefulness of the model obtained after collaboration (P((w, 0,)2 < (0))),)²) in two client mean estimation. Tn FedAvg, a single global model is used, whereas in CY each client. computes their model (wx) independently.
22

---

Published in Transactions on Machine Learning Research (08/2025)
Estimated params of Wi
1.0
0.8
0.5
Usefulness of
0.0
0.6
00.5
0,4
-100
5
10
15
20
25
5
10
15
20
25
Ratio of data points (N,/N2)
Ratio of data points (Nz/N2)
FedAvg
CYCle (Client
CYCle (Client
Figure 9: Numerical study on the impact of data imbalance on model usefulness (IP(ww; 0,)² S (0: 0,)9))
and the depiction of 0; estimates for N((0,, in two client mean estimation across varying degrees of data ratios.
a² N
samples, influencing their empirical means. Specifically, 0 and 0g are drawn from distributions N(O,, and (0, T² N2
respectively, with o² to
The results, presented Figure g illustrate the differential impacts of collaboration based on sample size. Collaboration tends benefit Client more than Client with Client being the higher-contributin client in terms of data. The variance of Client. 1 decreases their data ralio increases, suggesting that Client would benefit from limiting trust in Client 2's contributions under significant data imbalances. In contrast, the variance for Client 2 remains relatively the same regardless of the data ratio.
Figure 9 (right side) demonstrates the varying utilities collaboration for each client. For Client 1, the usefulness collaboration decreases as the data ratio becomes more skewed, whereas remains constant for Client 2, implying that the collaboration useful for Client Notably, the FodAvg approach generally results in worse outcomes compared to both clients in the CYCle scenario, especially as imbalances in data contribution increase.
C Extension to Gos
This appendix provides detailed background on decentralized optimization and describes how the proposed CYCle protocol is integrated with Gossip-GGD. We include full derivation of the update rules, communication graph structures, and Algorithm 3 This section supports the main discussion in Section 4.4
which focuses on federated distillation settings, decentralized algorithms typically aim solve the average consensus problem. This problem is formalized
Unlike the distillation- based approach described in Equations D and 2
D;
(45)
where E Rd are local vectors distributed across N nodes.
C.1 SoD
Description of Gossip-SGD. Classic decentralized algorithms for solving the average consensus problem
are often based on gossip-based algorithms (Xiao et al. 2005 Boyd et al. 2006 Koloskova et al. 2019
2020
Ying et al. 2021 that generate sequences (x) 1:0 every node E [N] through iterative updates of the
23

---

Published in Transactions on Machine Learning Research (08/2025)
Participant Pi
Participant Pa
Participant Px
D
D2
DN
Mæ
D1 (i)
Mæ
D2 (22,
May DN (KN,
SGD step. 22 (1+1/2)
SGD step: (1+1/2) 2)
SGD step: 2: (1+1/2)
(4)
(8)
(9)
(
2
22
'19
$19
Gossip step: (+1) Σ 16
Mixing matrix, row 1
(1+1/2)
W1
W11, 212..... WIN]
Figure 10: Illustration of the Gossip-SGD algorithm for N participants, focusing on the learning process of participant P1. As depicted in the figure, each participant performs SGD updates locally, followed by gossip step where they share and exchange with other participants.
form:
N
(1+1)
(8)
+
(46)
where 7 E (0, is a stepsize, Wij E [0, 1] are the weights, and A Rd is the vector (e.g. gradients) sent from node j to node i at iteration t.
Communication occurs only if Wij > 0, and no communicttio nceded when Wij = 0. If the weights are symmetric (wi, = Wyee,, they define a communication graph G= = [[N], E) with edges {i.j} E if way > and self-loops (1) E E for i e [N]. The matrix W RNNN known as the miring matrix or gossip matrix, specifies the connectivity of the network, with (W);g = Wij-
The objective in decentralized optimization is to minimize the global function:
N
min F(x),
where F(x) = Σ f((x,),
(47)
N Nd
1=1
where fa represents the local objective at node
To solve the optimization problem in Equation 47 the Gossip-SGD algorithm alternates between the following
two steps:
(1+1/2) xz.) -
(e) E
(t) 9.
= V (z(11) (2()))
b SGD stop
(48)
;
Wi 1++1/2)
Gossip step
(49)
Combining both steps, the Gossip-SGD update in matrix form becomes:
z (+1) W (z!10) - () -))()))))
where (())
(0 92 . 9N )0)
(6) (0)
E Nxd
(50)
24

---

fully connected graph, ring graph and Published in Transactions on Machine Learning Research (08/2025)
P1
P1
Po 6
P6 6
P2
Ps
P2
Ps
Ps
P3
Ps
P3
Ps
Pa
Ps
Fully Connected
Ring
Exponential
Figure 11: Different types of communication graphs/topologies: exponential graph.
Communication topologies. Communication graphs play a critical role in decentralized learning, where these graphs define the communication topology, specifying which nodes (clients) can exchange model updates during training. Unlike standard federated learning (FI.), where updates are aggregated by a central server, decentralized FL relies on peer-to-peer exchanges guided by the graph structure. Common topologies, depicted in Figure include
complete graphs, where all nodes communicate directly;
ring graphs, where nodes only communicate with their immediate neighbors; and
exponential graphs, which optimize connectivity while minimizing commuication overhead.
The mixing matrix W is constructed using uniform weights based on the graph topology:
+1' if (i.5) E i i=
[W
[(0++ 0.
(51)
otherwise.
where [W(i) denotes the number of neighbors of node i.
In our work, we initialize the communication graph as fully connected graph but incorporate two key mechanisms to make the topology dynamic: (i) reputation scoring and (ii) adaptive sharing (detailed in Section 4.. These mechanisms allow the communication graph to evolve during training, enabling improved fairness. We detail the specifics in Section C.2
C.2 Integrating CYCle with Gos
The algorithm illustrates how extend the Gossip-SGD algorithm within the framework of our proposed CYCle protocol. The central modification involves the use of a dynamic mixing matrix, which is updated every R communication rounds. This update is guided by a reputation scoring mechanism based on gradient alignment and an adaptive sharing strategy. In this version of CYCle, we adopt a different mapping function than Equation] 7 emphasizing the flexibility to use any mapping function, provided it preserves the probabilistic nature (i.e., maps values to the range [0, 1]). Specifically, we use softmax function with a hyperparameter B. The parameter 3 controls the trade-off between utility and fairness: higher values of B penalize smaller cosine similarities, thereby promoting fairness, while lower values prioritize equity in updates, effectively emulating fully-connected graph topology we stochastically sample a binary matrix S from the mixing matrix W using Bernoulli sampling (or coin toss). The resulting binary mask determines the communication partners for that round, where updates are exchanged only between nodes corresponding to the entries of 1.0 in S. This adaptive sharing mechanism enhances efficiency by facilitating productive collaborations and minimizing unproductive exchanges. Unlike PDL-based methods, which conduct multiple communications within a single FL round and sample updates for distillation, our approach in Gossip-SGD performs the sampling over the communication rounds T. We
special case. During each communication round,
25

---

M Published in Transactions on Machine Learning Research (08/2025)
Algorithm Gossip-SGD algorithm based on CYCle Protocol
Input: for each node i [M] initialize x!!) RR number of communication rounds T, stepsize 7, and mixing matrix W/O)
1: for round / = 0,1, , T do
2:
for each participant i [M] in parallel do
Compute 9i (6) Vfc a; (t)
for each neighbor j { E and mod == 0) do
Sij + 1 cos 9 9j
(8) (1)
/2
6:
Tij - arbitrary mapping function(sts)
softmax function
7:
wij + awj + (1 -1--a a)rij with a 0.5
(t)
end for (1+1/2) x;
(0) xi 1g (1)
stochastic gradient updates
10: end for
11:
Construct W, (W)s wij
12:
Generate stochastic binary mask S + Bernoull((W)
13:
Construct W W @ S and normalize each row to sum up
14:
(1+1) = W/2+1/2 where 111122 = 22) ,..., EN
1+1/2
1+1/2)
gossip averaging
15:
end for
evaluated this approach in Section G.3 on the Fed-ISIC2019 dataset. This dataset, being naturally partitioned and inherently non-ITD, serves as challenging and realistic benchmark for assessing federated learning algorithms.
Notation Summary
To improve clarity and readability throughout the paper, we provide Table 5 as a reference for the main notation. This table summarizes the symbols introduced in Section 2.1 and used consistently throughout. the paper. By consolidating key mathematical symbols and definitions in place, the table is intended to serve quick reference resource for readers, especially in later sections where several variables reappear in extended formulations. This addition also responds to reviewer feedback requesting improved accessibility of the notation throughout the manuscript.
Table 5: Summary of key notation used throughout the paper.
Symbol
Description
P Participants and Datasets
N
Total number of participants/chee /clintsss/ssesssssseeeess Index set of participants
N (1,2...,,N)
Pr
Participant/clien indexed by n E / Local training dataset of participant n
D.
D.
Local validation (hold-out) datasct of participant n
Input and Output Spaces
2 E X
Input sample from input space X' C Rd
y E y d
M) with M classes Dimensionality of input features
Ground-truth label; y = ,1,
Number oulput classes
C R
Logits space (output of Me)
26

---

Published in Transactions on Machine Learning Research (08/2025)
Table 5: continued from previous page
Symbol
Description
Models and Predictions
Ma: X +
Supervised classification model with parameters 0 Logits-generating model mapping inputs to Z C R- Parameters of participant n's model Ma,
Mo: : x + 2
On
2
Logit vector output by Ma(œ)
oT
Temperature-scaled softmax function: Ør(z) = softmax(z/T) Predicted class probabilities: p - oT(2)
p
T
Temperature parameter controlling softmax sharpness
Losses
LCE.
Cross-cntropy loss of participant Pairwise distillation loss from & to n
D.
DDL.
Total distillation loss for participant n
Ln
Total training loss of participant Weight on distillation loss in L.,
An
A(n,k)
Weight assigned to collaborator k's contribution in DL for n Kullback-Lcibler divergence used for distillation
KL(-I-)
Evaluation Metrics
Bn
Standalone accuracy of Ma, on D,, Post-collaboration accuracy of Me; on Dr Collaborative gain (CG) participant n
An
In An -Bn
E
Computational and Communication Complexity
This section expands on the computational and communication implications of the proposed CYCle protocol, particularly in comparison to baseline methods such as CaPriDe learning (Tastan Nandakumar 2023 and Gos (Boyd et al.) 2006). We quantify overhead associated with encrypled operations, gradient alignment, and communication costs introduced by our protocol, and clarify where CYCle improves upon or matches the efficiency of existing techniques.
E.1 Computational Complexity
The most significant computational component introduced CYCle[PDL) stems from its reliance on encrypted distillation loss computations, inherited directly from CaPriDe learning (Tastan & Nandakumar 2023) As detailed in Table 2 of their work, the per-sample inference time under FHE-based encryplion a commodity CPU is approximately 110 seconds, which represents the cryptographic bottleneck of such secure collaborative schernes. Though recent advancements in the cryptographic field have led to improvements by using GPU-
accelerated CKKS libraries, reducing the timing by more than 10 times Kim et al. 2024 Yang et al. 2024 Jin al.] ).24).
However, unlike CaPriDe learning, where encrypted predictions are computed and transmitted at every training round regardless of peer behavior, CYCle strategically reduces this cost by avoiding encrypted inference in rounds where peer reputation does not warrant response when they do not get sampled. This modification reduces the frequency of FHE operations without compromising convergence, leading to substantially lower per-round oryptographic overhead. In practice, this enables CYCle to remain computationally viable at scale, especially in large or heterogeneous client networks.
Another computational addition in CYCle the use of gradient alignment to guide selective sharing. This is implemented via cosine similarity computations between local gradients obtained with respect to cross-entropy
27

---

Published in Transactions on Machine Learning Research (08/2025)
loss and received peer gradients obtained with respect to distillation losses, which scale linearly with model size. These operations are performed once every R rounds (where R = 5 in experiments), and in aggregate, introduce less than a 5% overhead per local step. The periodic nature of these operations, coupled with modest cost, ensures they do not become a computational bottlencek. Similarly, sharing decisions are computed locally and stochastically, requiring no inter-client coordination or additional encryption.
Thus, compared CaPriDe learning, CYCle significantly reduces encrypted inference load, and compared to Gossip-SGD, it introduces only marginal overhead through lightweight, periodic gradient comparisons.
E.2 Communication Complexity
In terms of communication, CYCle adheres closely to the decentralized communication patterns of CaPriDe learning and Gossip-SGD but incorporates reputation-aware mechanisms that reduce message exchanges without altering the underlying protocol structure.
CYCIe(PDL). Similar to CaPriDe learning, CYCle requires clients to share encrypted predictions (logits) with selected clients for participant distillation. However, unlike CaPriDe learning where every client responds to every query, CYCle introduces a reputation-aware mechanism to selectively respond, thus reducing the number of encrypted transmissions. This strategy directly reduces the volume of encrypted data exchanged per round. Empirically, we observe 35 50% reduction in encrypted communication relative to CaPriDe learning across our experimental settings. Each encrypled prediction, typically represented an FHE ciphertext, incurs payload size of 1-2KB, depending the number of classes M, since the pairwise distillation loss is an M-dimensional vector.
CY CYCle (GGs When integrated with Gossip-SGD, CYCle does not incur any additional com- munication overhead beyond what is already required by the baseline. Clients continue to exchange model updates every round, as in standard Gossip protocols. Crucially, the snnutaation-eighhee mixing matrix is never transmitted; each client computes it locally based on observed peer behavior. Additionally, clients may selectively omit update transmissions to cow-contributing clients, thereby reducing communication frequency in practice. This sclective sharing leads to lower average communication volume per round, especially in heterogeneous or adversarial environments where client contribution quality varies significantly.
F
Experimental Setup. Implementation Details
We keep the same training hyperparameters for all experiments. Cross-entropy loss is used for local model training in all methods. The PDL algorithms (VPDL and CYCle) use Kullback-Leibler (KL) divergence based predictions as the distillation loss. The optimizer is stochastic gradient descent (SGD) with momentum. The learning rate is initially set to 0.1 and is updated every 25 rounds using a scheduler with a learning rate decay 0.1. For FL algorithms, we set the number of collaborating rounds to tma = 100 and the local update epoch to 1. For the decentralized algorithms, we use epochs of local training before collaboration, followed by 75 rounds of collaboration. The batch size is set 128 for CIFAR-10 and CIFAR-100 and 32 for the Fed-ISTC2019 dataset. Furthermore, a 0.5, Ao = 50, Topt 0.25, and Tmaz 0.75. In the experiments involving Gossip-SGD, we set B = 15 and R = 5. We conduct our experiments on NVIDIA A A100-PCIE-40GB GPUs on an internal cluster server, with each run utilizing a single GPU. The execution time for each run averages around 1.35 hours.
28

---

Published in Transactions on Machine Learning Research (08/2025)
G
Additional Experimental Results
This section is organized in the following manner:
Heatmap visualizations of reputation scores for the remaining data splitting scenarios, which include homogeneous, Dirichlet (0.5), and imbalanced (0.6, 1) cases.
Remaining results of Table 2
3. Details of the Fed-ISIC2019 data distribution and experimental results of CYCle when combined with Gossip-SGD.
Experimental results on the CIFAR-100 dataset.
5. More analysis of the imbalanced data splitting scenario.
More analysis of the free rider sccnario.
7. Scalability to larger client populations.
Trade-off between MCG and CGS.
9. Hyperparameter sensitivity.
G.1 Reputation Score Visualization
In Figure 12 present the reputation scores calculated the CIFAR-10 dataset for group of five participants (N 5). The figure illustrates the results at rounds = 0, 25, 75, under different data partitioning schemes: (i) homogeneous, (ii) heterogeneous (Dirichlet (ō = 0.5)), (iii) imbalanced (w = 0.6, m = 1). In the homogeneous setting, participants have similar reputation scores that correlate with the sizes of the datasets they hold. Over time, these scores trend downwards as the cross-entropy loss gains prominence over the total distillation loss (Eq. []. The helerogeneous and imbalanced (0.6, 1) scenarios exhibit a similar pattern, where the reputation scores align with the size of the data they possess. In the latter scenario, since P1 holds
Round
Round 25
Round 75
Round
Round 25
Round 75
1.0
1.0
1.00
1.00
1.00
0.57
0.61
0.64
0.64
38
0.34
0.34
0.34
0.80
0.91
0.86 0.80
0:37
0.65
0.55 0.61
0.32 0.39 0.41
1.00
1.00
1.00
1.00
0.61
0.59
0.60
0.59
0.33
0.30
0.31
0.32
0.99
0.00
1.000
0.75
0.98
0.77
0.87
0.55
0:77
0.69
0.65
1.00
0.68
0.34
0.72
0.40
0.320.43
0.6
1.00
1.00
0.67
0.35
0.59
0.67
0.62
0.39
0.33
.000
1.00
1.00
1.00
0.66
0.69
0.62
0.64
38
0.33
0.34
0.91
0.79
0.83
0.97
0.64
0.40
0.77
0.55
0.74
0.07
0.35
0.4
0.50
Homogeneous setting
(b) Dirichlet (0.5) setting
Round 0
Round 25
Round 75
0.63
0.67 1..00.77..00
0.63
0.66
0.30 20
31 0.26
20 0.
0.200
20
.000
1.00
1.00
1.00
1.00
0.97
0.88
.00
0.63
0.6.0.5 58
0.97
0.88
0.700
0.69
0.40
1.00
1.00
1.00
1.00
1.00
0..9
62
7.666.663
1.00
0.90
0.92
0.90
0.88
1.00
0.000.70000
0.
0.66
1.00
0.58
0.447
0.43
(c) Imbalanced (0.6, 1) setting
Figure 12: Heatmap visualization of reputation scores for the given number of participants N and data partition schemes: (a) Homogeneous, (b) Dirichlet (0.5) and (c) Imbalanced (0.6, 1).
29

---

Published in Transactions on Machine Learning Research (08/2025)
60% of the data, its reputation remains consistent, prompting other participants to engage more actively in collaboration to receive updates from this major contributor.
It must be noted that even when dataset sizes are identical among participants, as seen in homogeneous (Pi to Ps) and imbalanced (0.6, 1) (P2 to Ps) data splitting scenarios, there may be stochastic variations data quality resulting in higher reputation scores for some participants compared to others. For instance, Fig. 12c Pa is assigned marginally higher reputation score by the other participants, despite having an identical dataset size 22, P4, and Ps-
G.2 Remaining Results of Table 2
This section extends our analysis of collaboration gain and spread (MCG, CGS) for the CIFAR-10 and CIFAR-100 datasets using our proposed algorithm. While Table 2 focused on scenarios with N (5, 10} participants, the additional results presented in Table + cover the scenario with N = 2 participants. This study provides further insight into the performance dynamics under a smaller number of participants and complements the study shown Table
Table 6: Collaboration gain and spread (MCG, CGS) on CIFAR-10 and CIFAR-100 datasets and the given partition strategies for N = 2 using our proposed algorithm. This table supplements Table 2 which covers
= 10).
Datasct
CIFAR-10
CIFAR-IO6
Dataset
CIFAR-10
CIFAR-100
Split
Method
MVA
MCG
CGS
MVA
MCG
CGS
Method
MVA
MCG
CGS
MVA
MCG
CGS
VPDL
91.99
0.41
0.08
71.52
4.87
0.55
DSGD
86.26
3.55
0.27
59.52
7.59
0.16
Homogenoous
CYCle
92.58
1.00
0.01
72.03
5.38
0.22
CYCle
86.36
3.64
0.21
59.53
7.61
0.12
Dirichiet (0.5)
VPDI.
84.90
1.45
1.88
64.99
8.95
0.31
DSGD
84.03
16.92
12.51
55.95
10.73
8.13
CYCle
66.37
10.33
0.25
CYCle
14.52
VPDL
70.37
5.52
0.66
DSGD
13.59
Dirichlet (2.0)
CYCle
91.00
1.16
0.15
72.09
7.04
0.18
CYCle
86.53
5.73
0.99
57.81
13.50
0.52
VPDL
90.36
0.24
0.08
71.45
4.95
0.89
DSGD
85.92
3.77
0.91
58.93
9.83
0.33
Dirichlet (5.0)
CYCle
91.29
1.17
0.03
72.10 12
5.62
0.36
CYCle
86.15
4.00
0.58
58.82
9.72
0.11
G.3 Fed-ISIC2019 experiments
Fed-ISIC2019.
We evaluate our framework on the real-world non-llD Fed-ISIC2019 dataset. Note that for easy visualization, the participants this experiment are sorted in the docrcasing order of their dataset size.
Figure [4] illustrates the probability that each participant shared updates with their neighbors over T 100 communication rounds. The reported values represent averages; for example, value of 0.51 indicates that Pi shared updates with P2 during 51 out of 100 communccation rounds. We observe that our framework captures distribution shift, as evidenced by the fact that the data held by the first client does not overlap
with that of other clients (as shown in the UMAP visualization in Figure 1f of Ogier du Terrail et al. 2022
Data Distribution. Figure 13 displays a heatmap plot illustrating the distribution of data by class among participants. In Figure 14 it is evident that despite P, having the bulk of the data, it has noticeable distribution shift compared to other par- ticipants. This indicates that other participants are not allocated certain classes due to the specific distri- bution settings employed in (Ogier du Terrail et al.) 2022). It is noteworthy that initially, P, assigns lower reputation score P4. However, over time, Pa's 's reputation improves, change attributable to the exclusive knowledge of class 8 shared only by Pr and P4-
Training Set
Test Set
0.37
0.85 0.47 0.37
0.08
0.49
0.8
farticpants
N40000 0:33 0.30
0.0% 0:2 0.
0.00 0.000.0..00
0.13 6:17 0.00
0.16 0.160.007
0.03 0.19
0.52 0.02
0.17 0.16
0.05 0.01
0.20
0.20
0.000.35 0.00
e +00.88 0.07 0.08 0.12 0.23 0.13 0.00
0.30
0.08
0.07
0.11
0.15
0.150.18
0.18
0.18 0.20
0.4
10.0.05 0.04 0.00 0.00 0.07 0.00 0.00 0.00
0.06 0.03
0.00
0.00
0.10
0.00
0.00 0.00
0.2
0000.02 0.03 0.00 0.00 0.00 0.02 0.00 0.00
0.01
0.03
0.00
0.00 0.02 0.00
0.02 0.00
0.0
2 Classes
Classes
Figure 13: Fed-ISIC2019 data distribution.
30

---

Published in Transactions on Machine Learning Research (08/2025)
Participant
Participant
Participant
Participant
Participant
Participant 6
0.37
0.51
0.24
0.54
0.33
0.69
0.93
0.03
n50
0.77
0.1
0.03
0.52
0.0
0.28
0.20
Figure 14: Illustration of communication graph over T communication rounds, representing the percentage of
times each participant shared updates with its neighbors.
Participant 1
Participant 2
Participant
Balanced Accuracy
0.4
0.2
Participant
Participant
Participant
Balanced Accuracy
0.2
100
20
Comm. Rounds
Comm. Rounds
Comm. Rounds
SA
Gossip SGD (Complete)
Gessip SGD (Ring)
Gossip SGD (Exponential)
CYCle
Figure 15: Per-participant performance plots on the Fed-ISTC2019 datasel comparing different methods: standalone training, Gossip-SGD with various topologies (fully connected, ring, exponential), and the proposod CYCle protocol. SA stands for standalone training.
Experimental Results. Figure 15 illustrates the performance of the following methods: (i) standalone training, (ii) Gossip-SGD with a complete graph topology, (iii) Gossip-SGD with a ring topology, (iv) Gossip-SGD with an exponential graph topology, and (v) our proposed CYCle algorithm integrated with Gossip-SGD. The experiments span 100 communication rounds, and we report balanced accuracy as the primary metric due to the highly imbalanced nature of the classes. The results are derived from global test set created by pooling the test sets of all participants to ensure a fair comparison.
As shown in the plots, participants 5 and 6 struggle signifi- cantly, achieving performance below 20%. Interestingly, par- ticipants and achieve the best performance despite having smaller datasets compared to participants and 2. Among the methods, Gossip-SGD with complete graph topology achieves the highest overall performance. However, in terms of fairness (as shown in Figure 16a), it fails to provide equitable outcomes (Pearson's 's corr. coefficient of -0.2649), as all par- ticipants converge to a uniform final performance regardless of their contributions.
Table 7: MCG and CGS values (Figure 15
Method
MCG
CGS
Gossip-SGD (Complete)
30.26
15.64
Gossip-SGD (Ring)
27.72
14.80
Gossip-SGI (Exponential)
29.45
14.24
CYCle
23.79
10.94
In contrast, our CYCle algorithm mitigates this issue by suppressing the collaboration gains of participants contributing less providing low-quality updates. This results in high correlation coefficient of 0.9577,
31

---

Published in Transactions on Machine Learning Research (08/2025)
Pearson Corr.: 0.3727, p-val: 0.4667
3-77.p-val:
Balances Accurac
0.6
Bolanced Accuracy
Accuracy
0.6
0.4
0.2
0.2
0.2
Complete
5A
Ring
0.0
P2
P3
P4
P5
P6
0.0
P4
Participants
Participants
Participants
(a) Gossip-SGD with Complete Graph.
(b) Gussip-SGD with Ring Graph.
(c) CYCle with Dynamic Topology.
Figure 16: Per-par Per-participant accuracy and Pearson's correlation coefficient of the Gossip-SGD algorithm with fully connected and ring topologies, alongside the performance of our proposed CYCle algorithm. SA stands for standalone training.
with a small p-value of 0.0026, successfully rejecting the null hypothesis (that there is no correlation) at significance level of 0.05 (see Figure [6cp. These results highlight the efficacy of our approach in balancing both performance and fairnoss across participants. We also report the same plot for Gossip-SGD with a ring topology in Figure [6b] which fails to deliver fair outcomes for all participants. For the results on the MCG and CGS metrics, we refer the reader to Table
Fooding
CYCo
P2
P
P
Homegeeneuus
Heterogeneeus (-Dir(0.5))
Imbalanced (0.8,
Imbalanced (0.6. 1)
Imbalanced (0.35. 2)
Figure 17: Per-partin Per-participant performance comparison CIFAR-100 dataset: Validation accuracy evaluated with N = 5 participants.
Table 8: Per-participant performance comparison on CIFAR-100 dataset:
Validation accuracy evaluated with
N =5 participants.
Partition
No.
SA
FedAvg
VPDL
CYCle
Partition
No.
SA
FedAvg
VPDL
CYCle
48.85
70.25
57.05
56.10
74.50
71.80
70.05
75.30
2
47.65
70.10
56.00
57.05
20.80
73.45
27.40
29.85
Homogeneous
3
48.45
69.55
57.30
57.10
Imbalanced (0.8,
21.25
72.10
28.40
29.70
50.25
69.85
57.25
58.55
22.25
72.10
28.50
30.20
50.60 5
70.35
56.10
59.00
20.35
73.55
29.70
30.40
39.40
69.20
48.15
50.35
61.445
70.00
62.10
67.15
2
31.30
69.25
46.80
44.15
59.30
70.60
59.35
65.95
Dirichlet (0.5)
3
38.00
67.5
47,00
49.80
Iuballancd (0.35, 2)
29.65
70.55
40.15
39.80
4
39.50
67.70
50.10
5/2.00
28. 10
70.95
41.05
40.30
5
31.05
67.60
39.75
43.60
32.00
69.55
39.75
41.95
Partition
No.
SA
Aed
VPDI.
CYCle
68.25
70.80
65.80
70.8()
29.90
70.90
39.30
39.00
Iubalauced (0.6, 1)
30.15
71.80
35.70
40.
28.50
71.35
39.45
38.05
29.40
69.15
39.80
39.55
32

---

Published in Transactions on Machine Learning Research (08/2025)
G.4 Experiments CIFAR-100
We perform the same experiment outlined in Table 1 and Figure 3 on the CIFAR-100 dataset. Our goal is to assess the effectiveness and fairness of CYCle under increased label granularity and distributional heterogeneity. Figure I7 and Table X depict the performance of each participant within the N = 5 group across the various data partitioning strategies detailed in Section 5.1 Similar to the observations in Figure in the imbalanced (0.8, 1) scenario, the FedAvg algorithm results in a decline in accuracy for Pı, indicative of negative collaboration gain. Following this, we compile the mean validation accuracy (MVA), mean collaboration gain (MCG), and collaboration gain spread (CGS) for each method and data splitting scenario in Table 9
As in CIFAR-10 experiments, we observe that FedAvg often achieves the highest mcan validation accuracy (MVA), particularly in homogeneous and moderately in imbalanced settings. However, this comes at a significant fairness cost. VPDL, while privacy-preservin and decentralized, underperforms in both MVA and MCG across all secnarios. Although avoids extreme gain incquality (moderate CGS), fails to leverage meaningful collaboration in dala-hheeroggeeouu environments, largely due to its uniform treatment of client contributions.
Table 9: Performance comparison CIFAR-100 dataset: Validation accuracy evaluated with N = 5 participants. The table presents the performance of our proposed framework compared to the FedAvg algorithm. We employ MVA (t), MCG (t) and CGS (J) as evaluation metrics.
Setting
Homogeneous
Dirichlet (0.5)
Imbalanced (0.8, 1)
Imbalanced (0.35,
Imbalanced (0.6,
Metric
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
FodAvg
70.02
20.86
1.07
32.46
3.98
72.60
40.77
21.77
70.33
15.06
70.80
33.56
15.54
VPDL
56.74
7.58
1.20
46.36
10.51
2.59
36.81
4.98
4.84
48.48
6.37
5.19
44.61
7.37
4.98
CYCle
57.56
8.40
0.69
47.98
12.13
0.68
39.09
7.26
3.30
51.03
8.92
2.40
45.68
8.44
3.00
G.5 Imbalanced Data Study
In this study, we explore range of values for the parameters x and m in imbalanced settings, involving five participants (N = 5), as detailed in Section 5.1 The purpose of this variation is to demonstrate the
Table 10: Performance comparison on CIFAR-10 dataset under imbalanced data: Validation accuracy evaluated with N- = participants. We employ MVA (t), MCG (t) and CGS (L) as evaluation metrics.
Setting
Imbalanced (0.4,
Imbalanced (0.12, 1)
Imhalanced (0.41,
Imbalanced (0.23,
Imbalaneed (0.14, 2)
Metric
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
MVA
MCG
CGS
FedAvg
91.04
10.38
4.93
8.02
4.00
90.17
17.48
13.99
91.23
7.73
90.69
8.42
4.94
VPDL
83.36
2.69
3.04
86.18
3.22
2.74
74.90
2.22
4.17
85.82
2.33
2.09
86.18
3.70
3.67
CYCle
85.02
4.35
2.26
86.78
3.82
2.41
76.71
4.02
2.27
86.48
2.98
1.17
86.92
4.44
3.47
Standalone
Fooling
CYCe
P2
P1
7% Imbalanced (0.4. 1)
P Imbalanced (0.12, 1)
Imbalanced (0.41, 2)
Imbelancee (0.23, 21
Imbalanced (0.14. 2)
Figure 18: Per-participant performance comparison CIFAR-10 dataset. using the given imbalanced splitting
scenarios.
33

---

Published in Transactions on Machine Learning Research (08/2025)
effectiveness of our proposed CYCle method in ensuring positive collaboration gains for each participant, showcasing higher degree of fairness in comparison to FedAvg and VPDI. For a comprehensive view of these results, please see Figure 18 and Table TO Table 10 reports performance across five different imbalance settings using three metrics: MVA, MCG, and CGS. FedAvg suffers from a higher CGS and, in some scenarios, a large variance in individual participant outcomes, indicating lairness issues. VPDL offers stronger privacy guarantees but significantly underperforms in both accuracy and fairness. CYCle, contrast, achieves a compelling balance. It consistently improves fairness over both baselines, evidenced by its lower CGS across all five settings. This suggests that CYCle more equitably distributes collaborative benefits.
These findings confirm that CYCle not only preserves collaboration benefits but also effectively mitigates the challenges of unequal data distributions, an essential feature for practical decentralized learning algorithms.
G.6 Free Rider Study
In Table 1 and Figure 19 examine the outcomes using CYCle approach under the varying degrees of label flipping (ranging from 0.0 to 1.0) when two participants (P4 and PP)) are free riders. The findings show that the honest participants (P1, P2, and P3) maintain positive collaboration gains, despite the presence of malicious users / free riders. It's observed that when the flip rale is greater than 0, the collaboration gain decreases relative to scenarios where the flip rate is O. This decline linked to the fact that these three participants collectively hold only 60% of the total datasct. this experiment, we use the CIFAR-10 datasot and a homogeneous data splitting approach, with other parameters remaining consistent with those detailed
in Section 5.1
Table 11: Validation accuracies of participants (P1 Ps)
Flip rate 0.2
Flip rate 0.5
Flip rate 110
1.0
0.08 0.01
0.81
0.00 0.00
0.70
with P4 and Ps having varying rates of label flipping. The last column refers to the average accuracy of honest participants (PI P2).
0.62
0.020.00
0.00
0.00
0.6
0.51 0.512.62.68
0.41
0.64
0.63
0.67
0.39
0.00
0.00
0.00
0.00
0.4
0.745.40.84
0.61
0.54
0.65
6.57
9.33
0.00
0.00
0.00
0.00
Setting
P1
Pa
Pa
P4
Ps
avK
Standalone ((,)
53.42
83.67
83.15
$3.13
83.63
83.41
0.18
0:
0.08
0.05
0.00
2
3
5
Flip 0.0
86.12
87.03
86.33
85.62
56.10
86.49
Flip rase 0.2
84.25
84.20
84.
63. 10
65:40
84.18
Flip 0.5
84.30
84.00
81.95
36.95
35.75
84.08
Figure 19: Heatmap visualization of reputation
Flip race 1.0
84.15
84.20
83.45
3.40
3.10
83.93
scores with corrupted labels at P4 and Ps.
G.7 Scalability to Larger Client Populations
To evaluate the applicability of CYCle beyond cross-silo federated learning, we conducted additional experi- ments on CIFAR-10 and CIFAR-100 with N 20 clients. Data was partitioned using a Dirichlet distribution with a 0.5 to simulate non-ii.d. hetcrogeneity across clients. We compare three methods: standalone (no collaboration), Gossip-SGD (with full connectivity), and CYCle.
Figure 20 displays the per-participant accuracies on CIFAR-10 and 100 datasets. In particular, clients 2, 7, and 15 exhibit significantly lower standalone performance (Figure 20a indicating limited data quality or quantity. CYCle is able to identify such low-contributing clients through a reputation mechanism and adjusts collaboration accordingly. This ensures that while these clients benefit from collaboration, they do not disproportionately influence others, thus maintaining fairness and robustness.
These results demonstrate that CYCle maintains robust fairness and stable performance when scaled to larger client populations beyond the cross-silo setting. In contrast to Gossip-SGD, which exhibits a degradation of fairness (indicated by a negative correlation in Table 12 CYCle achieves a significantly more equitable distribution of collaborative gains.
For CIFAR-10, both methods achieve similar MCG values (28.59 for Gossip-SGD vs. 27.87 for CYCIe), indicating comparable average collaborative benefits. However, the collaboration gain spread (CGS) is considerably lower for CYCle (6.06 vs. 7.70), showing that it reduces the inequality in the results of
34

---

Published in Transactions on Machine Learning Research (08/2025)
CIFAR-10 Per-Participant
CIFAR-100 Per-Participant
Standalone
Standalone
Accuracy
Accuracy
Gossip-SGD
Gossip-SGD
CYCle
CYCle
8
9
9
10
2
10
30
11
1
11
12
20
12
20
13
19
13
14
18
14
18
15
15
16
16
(a) CIFAR-10
(b) CIFAR-100
Figure 20: Per-participant performance radar plots for N 20 clients on (a) CTFAR- and (b) CIFAR-100 datascts using three methods: standalone (no collaboration), Gossip-SGD (with a complete graph), and CYCle.
collaboration. Morcover, the strong positive corrclation (p= 0.9353) under CYCle implics that participants with more useful valuable data tend to receive proportionally higher rewards, signaling fairness-aligned reward mechanism. In contrast, Gossip-SGD shows negative correlation (p -0.4637, which implies misalignment between client contribution and received gains.
A similar trend holds for CIFAR-100. The MCG values are again comparable, but CYCle achieves a markedly lower CGS, indicating greater consistency in how benefits are distributed. Pearson's correlation improves dramatically from = -0. 7297 (Gossip-SGD) to = 0.8164 (CYCle), further supporting that CYCle not only promotes fairness, but aligns collaboration rewards more effectively with actual client contribution.
We also note that CYCle be naturally extended to scenarios with significantly larger client populations and partial participation. In such cases, subset of clients can be sampled in each round and CYCle can be applied by constructing subgraph over the active clients. This involves computing sub-mixing matrix using only the participating clients and running the CYCle protocol locally on this induced set of clients. Reputation and gradient alignment remain local and round-specific, requiring no global state. This design makes CYCle compatible with the standard partial participation strategies commonly used in cross-device FL. However, since our focus is on cross-silo settings with full participation, we leave empirical evaluation under partial participation for future work.
Datasct
Method
MCG
CGS
Pearson corr. (p)
Gossip-SGD CYCle
28.59
7.70
-0.4637
CIFAR-10
27.87
6.06
0.9353
Gossip-SGD
22.68
2.12
-0.7297
CIFAR-100
CYCle
22.88
1.25
0.8164
Table 12: Comparison of collaborative fairness metrics for N 20 clients.
35

---

Published in Transactions on Machine Learning Research (08/2025)
In summary, CYCle preserves average performance while significantly improving fairness and reward alignment between participants. These gains are partieularly important in federated settings with data heterogeneity and varying client quality, as they help prevent both under- and over-compennation in collaborative learning.
G.8 Trade-off between MCG and CGS
Figure 21 presents plot depicting the MCG and CGS values for 10 different data-split scenarios, which are specified in Table | and Table 10] Each point in the plot corresponds to the average performance of a method under one scenario, with the r-axis representing the mean collaboration gain (MCG) and the y-axis representing the collaboration gain spread (CGS). The bottom right region of the plot, indicating high MCG and low CGS, represents the ideal operational regime, where all participants benefit meaningfully and fairly from collaboration. It must. be emphasized that the results of our CYCle approach fall predominantly towards the bottom-right corner of the plot. On average, CYCle achieves the highest MCG and the lowest CGS in the 10 sccnarios. While bascline methods such as CFFL, RFFL, and CGSV may outperform CYCle in one of the two metrics under certain scenarios, they exhibit a clear trade-off: improvements in collaborative gain often come at the cost of high inequality (larger CGS) and vice versa. For example, CGSV may occasionally achieve a higher MCG, but its CGS is substantially worse, indicating that a few participants benefit disproportionatel; while others gain little or even suffer negative outcomes. In contrast, methods such as RFFL may better control CGS but fail to deliver sufficient gain, as reflected in lower MCG values.
The directionality of the arrows overlaid on the figure further emphasizes the optimization goal: to move toward the bottom right.
OFFL
RFFL
CGSV
CYCle
Beller
10-
8-
2-
Better
14
MCG
Homogeneous Dirichlet (0.5)
Imbalanced (0.35, 2]
Imbalanced (0 (2.12. 1)
Imbalanced (0.14, 2)
Imbalanced (0.6, 1)
Imbalanced (0 (0.41.2 2)
Average
Imbalanced (0.8, 1)
imbalanced (0.4, 1}
Imbalanced (0 (0.23, 2)
Figure 21: Plot of MCG and CGS values for 10 distinct data splits, ranging from homogencous to spectrum of imbalanced scenarios.
G.9 Hyperparameter Sensitivity
CYCle (PDL). We have defined only two hyperparameters in our model, Topt and Tmaz, which have geometric interpretation (degree of gradient alignment required for beneficial collaboration). In our experiments, set Topt 0.25 and Tmaz 0.75. Figure 22a shows the sensitivity analysis for these hyperparameters, where it is evident that increasing the values of Tiaa and Topt correlates with improved MCG in a homogeneous setting. This is expected as higher values of these parameters bring our algorithm closer to the VPDL. framework. However, the MCG and CGS (Figure (2b) values for different hyperparameter settings are comparable, indicating that the reported results are robust.
These parameters play a well-defined and interpretable role in our framework through the mapping function (Equation 7. The parameters Topt and Fmaz define soft acceptance region over the similarity space (e-g.,
36

---

Published in Transactions on Machine Learning Research (08/2025)
Homogeneous
Imbalanced (0.8, 1)
Homogeneous
Imbalanced (0.8,
0.2 0.15
2.62
2.81
2.91
2.98
3.28
3.25 2.8
3.78 4.10
1.10..77
3.75
1..49
1.03 0.330.79
0.66
3.10
1.4
2.50
3.80
2.54
3.44
3.50
0.3 0.25 0.2
0.97
0.490.49..000
0.74
1.11
1.78
3.05
2.63
1.2
Topt
0.35 0.3 0.25
2.71
2.69
3.73
3.08
3.13
1.11
3.77
3.25
Topt
1.45
1.05
1.632.05
1.86
1.0
-2.69
3.45
2.70
8.38
3.54
4.37
2.772.2.
2.21
2.37
3.00
1.31
1.01 1.18
0.50
0.81
2.78
2.332...3
2.75
2.85
2.65
3.95
3.91 3.78
4.01
3.14
3.47
11.00
0.00
0.77
0.76
2.23
2.03
3.03
2.56
1.75
-0.6
2.50
0.65
0.7
0.75
0,85
0.65 0.7
0.75
0.8
0.85
0.65
0.7
0.75
0.8 0.85
0.65
0.7
0.75
0.8
0.85
Tmax
Tmax
Tmax
Tyax
(a) MCG (1)
(b) CGS (4)
Figure 22: Sensitivity analysis to Topt and Tmar parameters.
give full credit to highly aligned clients h(s) = 1, while blocking collaboration with poorly aligned ones h(s) = 0). As shown in Figure 22 we explored a range of reasonable values and found that going below Topt = 0.25 results degraded collaboration fairness and utility, as unqualified updates are included too permissively. The selected values are therefore informed by both the empirical similarity statistics and the thcoretical geometry of the collaboration space. For these reasons, we did not conduct exhaustive tuning and instead used principled, task-informed choices.
CYCle (Gossip-SGD). The only hyperparameter in the gossip-basod variant of protocol is 3, which controls the sharpness of the softmax function used to derive the communication probabilities from gradient alignment scores. Higher 3 values assign greater weight to stronger alignments, enforcing more selective and fairness-oriented peer interactions, while lower values yield more uniform communication, resembling a fully connected topology. Figure 23a shows the performance across participants under varying 3 E (5, 15, 35, 50, 100). Although participant-leve performance varies slightly, all configurations exhibit. reasonable fairness, validating the robustness of CYCle under different settings. Figure 23b further illustrates the trade-off between utility (MCG) and fairness (Pearson correlation of pro- and post-collaboration accuracies). We observe that intermediale values of B (e.g., 35-50) strike favorable balance, delivering high collaboration gain without sacrificing fairness.
Gossip-SGD with CYCle (8 parameter study)
Utility vs. Fairness
Balanced Accuracy
0.6
0.9
0.8
22
Pearson
20
0.2
(15)
(50)
CYCle
CYCle (35)
CrCle (100)
18
0.6
0.0
16
P3
100
Participants
(a) Performance of CYCle under different parameters E {5, 15, 35, 50, 100). (b) MCG vs. correlation coefficient.
Figure 23: Study on the impact of parameter 3 on performance and analysis of the utility versus fairness trade-off using the CYCle algorithm.
G.10 Additional Results
The accuracy metrics for the individual participants for the experiments on the CIFAR-10 datasel (corre- sponding Figures and are summarized in Table 13 for greater clarity. These results clearly demonstrate the negative gain experienced by the participant P. in the imbalanced settings under most existing FL algorithms and how the proposed CYCle approach overcomes this problem.
37

---

Published in Transactions on Machine Learning Research (08/2025)
Table 13: Per- Per-participant performance comparison on CIFAR-10 dataset: Validation accuracy evaluated with N = participants. The left table presents the performance of our proposed framework compared to the FedAvg algorithm. The right table compares collaboration gain and fairness our proposed CYCle algorithm with existing works (columns 4-6). We employ MVA (t), MCG (f) and CGS (4) evaluation metrics.
Split:
No./M Mot/Metric
Fed Avg
VPDL
CYCle
Split No./Metric
SA
CFFL
RFFL
CGSV
CYCle
83.42
91.02
$5.82
86.12
1
60.98
62.05
61.65
63.32
72.42
Honnogoeoou
90.32
84.02
87.03
Homoggeeons
62.32
63.67
3
83.15
90.63
84.70
86.33
59.20
61.58
61.07
63.08
71.48
4
83,13
90.82
85.27
85.62
4
60.50
62.50
60.80
61 88
72.55
5
83.63
90.23
85.12
86.10
5
61.67
63.53
61.65
64.42
72.43
MVA
83.40
90.60
84.98
86.24
MVA
60.44
62.66
61.50
63.27
71.95
MCG
0.00
7.20
1.58
2.84
MCG
0.00
2.21
1.05
2.83
11.51
CGS
0.48
CGS
0.00
0.87
79.48
48.45
55.30
Dirichles (0.5)
65.25
88.48
74.17
73.55
Dirichlet (0.5)
48.72
49.28
50.07
53.10
53.20
3
73.92
88.35
77.17
79.87
49.15
31.15
50.70
53.60
53.82
4
67.55
88.68
74.17
78.70
44.17
49.30
45.72
54.70
47.95
5
61.62
89.35
71.40
73.05
42.57
47.82
47.65
55.57
49.28
MVA
67.36
88.76
74.27
76.93
MVA
46.55
49.04
48.52
54.45
51.21
MCG
MCG
2.49
CGS
0.00
4.31
2.31
2.13
CGS
0.00
2.35
1.63
3.40
1.1.
Imbalanced (0.S, 1)
1
92.77
90.23
89.87
93.80
1
74.32
69.38
71.47
56.55
77.15
2
56.85
90.33
60.77
63.02
2
52.95
56,08
52.80
56.92
57.10
Imbelanced (0.8,
3
53.82
90.07
62.08
62.82
3
48.95
56.52
52. 13
54.77
58.73
58.00
90.15
61.52
63.35
48.40
54.97
52.40
55.33
57.93
90.07
61.68
63.30
51
57.35
52.53
56.23
MVA
67.18
69.26
MVA
56.27
55.96
MCG
0.00
26.35
3.36
5.44
MCG
0.00
3.48
1.08
0.78
6.62
CGS
0.00
14.51
3.58
2.56
CGS
0.00
4.39
2.45
9.32
2.79
Imbulanood (0:35, 2)
1
89.30
90.45
$5.50
89.90
Imbalameod (0.35, 2)
1
69.43
66.38
641.22
63.10
74.13
2
85.98
90.53
84.00
89.73
2
69.70
68.52
63.55
62. 15
74.07
3
69.07
89.83
73. 73.97
75.35
3
45.17
62.18
53.70
60.58
64.93
69.23
90.40
74.53
75.58
61.53
63.93
69.53
90.48
73.55
75.02
5
47.12
64.37
56.98
62.66
64.23
MVA
77.22
90.34
78.71
81.12
MVA
55.09
65.08
58.51
61.99
68.26
MCG
0.00
13.12
1.49
3.89
MOG
0.00
9.99
3.43
6.91
13.
CGS
0.00
9.61
4.01
2.65
CGS
0.00
9.95
7.46
11.34
7.12
Imbalaneco (0-8, 1]
1
92.27
90.38
89.25
92.80 02.80
Imbalanced (0.8, 1)
1
74.23
69.30
61.58
62 33
76.22
2
68.65
90.73
71.58
71.50
2
45.08
65.70
49.05
43
64.17
71.58
72.65
47.15
60.40
68.78
72.85
73.45
43.80
63.97
48.67
60.53
59.10
5
70.60
90.43
71.95
72.87
5
45.05
64.98
50.30
61. 31.98
63.30
MVA
73.88
90.16
75.43
76.72
MVA
50.85
65.20
51.35
61.34
64.03
MCG
0.00
16.28 25
1.55
2.83
MCG
0.00
14.35
0.50
10.49
14.08
CGS
0.00
9.11
2.45
1.38
CGS
0.00
9.79
6.74
11.23
6.22
38


Reviews:
[{"review_id": "yMn0aj4oNk", "reviewer": "Reviewer_5Hrc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strenghts\n1. **Well motivated research problem and rationale:** The problem tackled in this paper is important, as authors very clearly show by examples in section 3. The rationale behind their proposed approach is conceptually simple and appears as a clean problem reformulation w.r.t the classical global risk minimization common in general decentralized optimization.\n2. **Nice theoretical foundations:** authors include a lower bound that proves an advatage w.r.t FedAvg in learning in a fair manner. Even though the settings is somewhat synthetic and of limited general utility (see questions on that), is provides solid grounding to the proposed methodology.\n3. **Good empirical results:** besides the \"goodness\" of the results per se, the experiments directly validate the claims on the proposed approach, specifically the fact that the algorithm automatically \"discovers\" which clients are useful for their own objective and which are not. \n\n\n## Weaknesses\n1. **Placement of the contents:** in my opinion, this paper contains more than 12 pages of \"actual\" main content. I believe some choices regarding the placement of contents between main paper and appendix do not fully expose the full contribution, as I required reviewing significant portion of the appendix, while some sections of the main paper could go to the appendix. Addressing this concern is very important to ensure the clarity of the final manuscript. Please have a look to the requested changes for some suggestions.\n2. **Unclear motivation behind metric clipping and momentum in score calculation:** misalignment metric clipping is motivated by experimental evidence, but it is not evident which is the role of the involved hyperparameters on the fairness objective. Similarly, momentum in score calculation is used as a way to smooth it down: it is not clear to me why we would expect large stochastic variations of the reputation score, especially in later stages of training. I would like the authors to discuss these aspects, and if possible suggest more straighforward altenatives.\n3. **It is unclear if the approach is limited to cross-silo FL settings:** the experiment only consider cross-silo settings with very low number of clients. A motivation for that is the burden required for PDL, which restricts the feasibility to cases in which the number of clients is small. However, it is not clear to me is the approach per se would work equally well on settings with more clients, and possibly partial participation. I understand this is somewhat outside the scope, but it would be nice if authors could discuss this aspect and possibly provide some experiments with larger number of clients.\n4. **Unclear extent of theoretical claims:** it is unclear to me wether the theoretical part, which is appreciated, has some predictive value for real-world cases outside the toy theoretical setting with 2 clients. This doubt arise from the fact that, while lower bounds are usually on minimum errors (_i.e._ proving that there exist at least one problem/family of problems in which a given measure of error is $\\geq \\epsilon > 0$), in this work is used to provide the minimum probability of __success__ only for the learning problems of the form of theorem A.1. In that sense, while lower bounds are typically strong, in this case the result is limited to a very specific setting. The claim in the paper is correct (_i.e._ it is reported that the guarantees hold for that specific setting), but I think that the lack of discussion could mislead readers into thinking that the result have more general validity. I believe that moving section A into the main paper (see requested critical changes), would give opportunity to remark the extent of the theoretical claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission studies the problem of collaborative fairness in decentralized learning, i.e. designing a learning system that fairly rewards each involved party from the perspective of their perfomance gains derived by participating to the collaboration. This is opposed to just maximizing the expected accuracy gain, which can be not fair for each individual party. To this end, they propose novel fairness objective that maximize the \"mean collaboration gain\" (MCG) while minimizing the \"collaborative gain spread\" (CGS), and propose the CYCle protocol for private decentralized learning (PDL). Authors also show that their protocol can be adapted to work with gossip-based decentralized algorithms. The paper includes theoretical results lower bounding the probability of achieving better results than FedAvg on a simple problem with 2 clients, and presents wide experimentation on CIFAR-10/100 and Fed-ISIC, in both homogeneous and heterogeneous settings. Overall, the paper proves quite a strong point in motivating the need of a different collaborative metric, and results clearly support the claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical changes\n1. **Revise the placement of contents:** in my opinion, the theoretical part and the extension to Gossip-SGD are important contribution and should appear in the main paper. In particular, section A (theoretical contribution) should be moved back in the main paper in section 4.3 and section C to section 4.4. At the same time, the implementation details of section 5.1 (which occupy about a page) can be moved to the appendix. Some sections (e.g. sec. 3) could me more concise. If space is an hard constraint (e.g. if the paper has to fit 12 pages of main content) I believe figure 4 could be moved to the appendix and figures 2-7 reduced in sized or moved as well. Results on FedIsic (section D.3) are explicitly claimed and should be presented in the main paper.\n2. **Discusss more in detail the limitation of theoretical results:**  If my intuition is correct, the guarantees of theorem A.1 only hold for the specific problem considered, and does not trivially extend to other cases. Can authors confirm if this is the correct interpretation? If yes, I think it would be beneficial to discuss the current limitation, such that future work may improve. For example, would the findings generalize with arbitrarily as many clients? If yes, do you believe it is possible to provide a more general proof? If not, could you indicate which are the difficulties in proving a more general claim? \n\n## Suggested changes\n1. **Include details about communication cost:** while it is not the focus of the paper, I think reporting how much data has been exchanged during the training process can be useful for future research on how to make the learning process more communication efficient.\n\n## Minor\n1. **Typos:** There could be some typos in the proof of Lemma A.1: I believe that second line of eq. (17) misses the application of similar passages applied to the first line, and reference to eq. (14) should probably refer to eq. (15) instead.\n2. **Notations:** the notation (sec 2.1) is consistent and mostly clear, but I found some difficulties recalling the meaning of each symbol in later sections of the paper. I invite the authors to consider simplifying it for improving clarity.\n\n## Other suggestions / Questions\n1. I would briefly restate the advantages of the proposed approach w.r.t. previous ones just before section 2.1\n2. **Suggestion on experimental setting:** I would have considered datasets with severe domain shift for the experimentation (e.g. DomainNet): in those cases, if my understanding is correct, the algorithm would streghten the reputation scores of clients holding examples from the same domain. Probably, the algorithms would have worked even better in those cases (see next point for rationale).\n3. **Connection to PFL:** If we interpret the matrix of reputations scores as the adiancency matrix of a graph, the algorithm could be used to find cliques of clients with similar objectives, and allow more general forms of personalization than letting each client have a (potentially) different model than others. Did you think of possible extensions in this sense?\n4. **Intuitions about extension to other tasks?** Can you share some intuitions of what you think we can expect applying the proposed scheme in NLP?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GyO8UlmtgT", "reviewer": "Reviewer_xUX9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper addresses the important yet underexplored problem of fairness in decentralized collaborative learning, particularly without centralized coordination—relevant to privacy-sensitive domains such as healthcare and finance.The authors identify limitations of correlation-based fairness metrics and propose a principled alternative based on maximizing Mean Collaboration Gain while minimizing its standard deviation. This is well-justified both theoretically and empirically. The method is comprehensively evaluated on CIFAR-10, CIFAR-100, and Fed-ISIC2019 under various data heterogeneity and imbalance scenarios, demonstrating consistently positive collaboration gains and superior fairness compared to baselines. \n\nWeaknesses:\n\nThe theoretical analysis is confined to a simple two-party mean estimation example. A more general or deeper theoretical understanding for complex models (e.g., neural networks) is lacking.\n    \nThe protocol introduces non-negligible computation (e.g., gradient alignment and periodic sharing decisions). The impact on overall convergence speed and computational/communication overhead is not sufficiently analyzed.\n    \nThe choice of baselines is somewhat inappropriate. While the comparison to FL algorithms is appreciated, fairness comparisons with more recent or competitive decentralized learning frameworks (e.g., Cronus) are missing.\n\nThere are minor issues in the writing. For example, the definition of $\\nu$ and $\\mu$ in the bound of $P(G_n < 0)$ is not clearly stated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents CYCle, a novel protocol designed to enhance collaborative fairness in privacy-preserving decentralized learning. The method introduces a gradient alignment-based reputation mechanism to dynamically adjust knowledge transfer between participants and promotes fairer outcomes across heterogeneous clients.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qbwYy2DOfA", "reviewer": "Reviewer_XGxa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. **Methodological Innovation**: Introduced gradient alignment into reputation scoring for the first time, combining distillation loss to dynamically adjust knowledge sharing weights, providing a new perspective on decentralized collaborative fairness (see Section 4.1).\n2. **Theoretical Rigor**: Through theoretical analysis of the mean estimation problem, demonstrated the superiority of CYCle under data heterogeneity (Section 4.3 and Appendix A), enhancing the credibility of the method.\n3. **Experimental Comprehensiveness**: Covered multiple data partitioning scenarios including homogeneous, heterogeneous (Dirichlet distribution), and imbalanced data, and validated robustness across datasets (Fed-ISIC2019) and architectures (ResNet18 vs. custom CNN) (Section 5.2).\n4. **Reproducibility**: Complete hyperparameter configurations (Section 5.1) provide an important benchmark for subsequent research.\n\n**Weaknesses**\n1. **Insufficient Baseline Comparison**: Did not compare with the latest SOTA methods (such as FedFair [1], FairVFL [2]), which weakens the universality of the conclusions (Tables 1 and 2).\n2. **Lack of Complexity Analysis**: Did not quantify computational costs (such as the time for computing encrypted distillation loss) and communication costs (such as the dynamic mixing matrix update frequency in Gossip-SGD extensions), affecting the practical assessment (Section 5.1).\n3. **Insufficient Adversarial Robustness**: Only simulated label-flipping attacks (Section 5.2) and did not consider more complex Byzantine attacks (such as gradient tampering), limiting the method's application potential in open environments.\n\n[1] Che, X., Hu, J., Zhou, Z., Zhang, Y., and Chu, L., “Training Fair Models in Federated Learning without Data Privacy Infringement”, <i>arXiv e-prints</i>, Art. no. arXiv:2109.05662, 2021. doi:10.48550/arXiv.2109.05662.\n\n[2] Qi, T., “FairVFL: A Fair Vertical Federated Learning Framework with Contrastive Adversarial Learning”, <i>arXiv e-prints</i>, Art. no. arXiv:2206.03200, 2022. doi:10.48550/arXiv.2206.03200.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article presents the **CYCle protocol**, a reputation scoring mechanism based on gradient alignment, aimed at improving collaborative fairness in private decentralized learning (PDL). The core contributions include: \n1. Highlighting the shortcomings of traditional collaborative fairness metrics (based on the correlation between contribution and final accuracy) in scenarios of negative collaboration gain, a new paradigm for quantifying fairness is proposed, focusing on maximizing mean collaboration gain (MCG) and minimizing collaboration gain spread (CGS). \n2. A dynamic reputation scoring mechanism is designed, which assesses participant contribution through gradient alignment of cross-entropy loss and distillation loss, and based on this, an adaptive knowledge sharing strategy is implemented to ensure that all participants achieve positive collaboration gains. \n3. The CYCle protocol is extended to decentralized algorithms such as Gossip-SGD, achieving fair collaboration through a dynamic mixing matrix, and proving its superiority to FedAvg in heterogeneous data scenarios. \n4. The effectiveness of the method is validated on datasets such as CIFAR-10, CIFAR-100, and Fed-ISIC2019, especially demonstrating a significant reduction in CGS in highly non-independent and identically distributed (Non-IID) data scenarios, while maintaining high MCG.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Key Modifications** \n1. **Supplement Comparative Experiments**: Include comparisons with recent SOTA methods (such as FedFair [1], FairVFL [2]) to validate the superiority of CYCle in a broader range of scenarios, which impacting the validity of conclusions. \n2. **Add Complexity Analysis**: Supplement time complexity and communication overhead data in Tables or the appendix (such as single-round training time, encryption operation time). \n\n**Improvement Suggestions** \n1. **Expand Adversarial Experiments**: Increase robustness testing against attacks such as gradient tampering and model poisoning (Section 5.2). \n2. **Parameter Sensitivity Analysis**: Provide a quantitative analysis of the impact of τ_opt and τ_max on MCG/CGS (currently only partially shown in Figure 20).\n\n[1] Che, X., Hu, J., Zhou, Z., Zhang, Y., and Chu, L., “Training Fair Models in Federated Learning without Data Privacy Infringement”, <i>arXiv e-prints</i>, Art. no. arXiv:2109.05662, 2021. doi:10.48550/arXiv.2109.05662.\n\n[2] Qi, T., “FairVFL: A Fair Vertical Federated Learning Framework with Contrastive Adversarial Learning”, <i>arXiv e-prints</i>, Art. no. arXiv:2206.03200, 2022. doi:10.48550/arXiv.2206.03200.", "broader_impact_concerns": "1. **Privacy Protection Measures**: Although CaPriDe uses fully homomorphic encryption, it is necessary to clarify the specific protection mechanisms for sensitive information in the training data (e.g., medical images from Fed-ISIC2019). \n2. **Abuse Risk Analysis**: Discuss the potential risks of the model being misused for unfair competition (e.g., high-contribution participants monopolizing knowledge sharing) and mitigation strategies.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)