DFML: Decentralized Federated Mutual Learning
Authors: Yasser H. Khalil
Amir Hossein Estiri
Mahdi Beitollahi
Nader Asadi
Sobhan Hemati
Xu Li
Guojun Zhang
Xi Chen
Venue: Accepted by TMLR ()

Abstract: In the realm of real-world devices, centralized servers in Federated Learning (FL) present challenges including communication bottlenecks and susceptibility to a single point of failure. Additionally, contemporary devices inherently exhibit model and data heterogeneity. Existing work lacks a Decentralized FL (DFL) framework capable of accommodating such heterogeneity without imposing architectural restrictions or assuming the availability of additional data. To address these issues, we propose a Decentralized Federated Mutual Learning (DFML) framework that is serverless, supports nonrestrictive heterogeneous models, and avoids reliance on additional data. DFML effectively handles model and data heterogeneity through mutual learning, which distills knowledge between clients, and cyclically varying the amount of supervision and distillation signals. Extensive experimental results demonstrate consistent effectiveness of DFML in both convergence speed and global accuracy, outperforming prevalent baselines under various conditions. For example, with the CIFAR-100 dataset and 50 clients, DFML achieves a substantial increase of +17.20% and +19.95% in global accuracy under Independent and Identically Distributed (IID) and non-IID data shifts, respectively.

Published in Transactions on Machine Learning Research (08/2024)
DFML: Decentralized Federated Mutual Learning
Yasser H. Khalil
yasser. khakil @huaweii com
Huowei Nooh's 's Ark Lab, Montreal, Canada.
Amir H. Estiri
Huawei Noah's Ark Lab, Montreal, Canada.
Mahdi Beitollahi
mahdi, www..beeeeelaaii@uueee...cc
Huawei Noah's Ark Lab, Montreal, Canada.
Nader Asadi
nader, madeer.asali@lhwwwee.ccmm
Huowei Noah's Ark Lab, Montreal, Conada.
Sobhan Hemati
wobhan.hemaai@@ruuuec...omm
Huowei Noah's Ark Lab, Montreal, Conada.
Xu Huowei Technologies Canada Inc., Ottawa, Conada.
wu. ww.lica@hhawwi..com com
Guojun Zhang
Huowei Nooh Ark Lab, Montreal, Canada.
Xi Chen Huowei Noeh's 's Ark Lab, Montreal, Canada.
di.chen, Cluuwwii.com com
Reviewed on OpenReview: https: // openreview. net/ net/forum? forum? id= 19Huz.bbubh
Abstract
In the realm of real-world devices, contralized servers in Foderated Learning (FL) present challenges including communication bottlenecks and susceptibility a single point of fail- ure. Additionally, contemporary devices inherently exhibit model and data heterogeneity. Existing work lacks a Decentralized FL (DFL) framework capable of accommodating such heterogeneity without imposing architectural restrictions assuming the availability of additionall data. To address these issues, we propose Decentralized Federated Mutual Learning (DFML) framework that is serverless, supports nonrestrictive heterogencous mod- els, and avoids reliance on additional data. DFML effectively handles model and data heterogeneity through mutual learning, which distills knowledge between clients, and cycli- cally varying the amount of supervision and distillation signals. Extensive experimental results demonstrate consistent effectiveness of DFML in both convergence speed and global accuracy, outperforming prevalent baselines under various conditions. For example, with the CIFAR-100 1-0 datasst and 50 clients, DFML achieves substantial increase of +17.20% and +19955% in global accuracy under Independent and Identically Distributed (IID) and non-IID data shifts, respectively.
1 Introduction
Federated Learning (FL) stands as a promising paradigm machine learning that enables decentralized learning without sharing raw data, thereby cnhancing data privacy. Although, Contralized FL (CFL) has been predominant in the literature (McMahan et al. 2017] Alam et al. [022 Diao al., 2020 Horvath al.,
2021 Caldas et al.] 2018), it relies central Communication with server be bottleneck,

---

Published in Transactions on Machine Learning Research (08/2024)
especially when numerous dispersed devices exist, and server is vulnerable to a single point of failure. To avoid these challenges, Decentralized FL (DFL.) serves an alternative, facilitating knowledge sharing among clients without the need of a central server (Beltrán et al. 2023 Yuan et al. 2023 Giuseppi et al, 2022 Li et al. 2021). DFL also offers computational and energy efficiency, as resources are distributed across clients instead of being concentrated in a centralized source. Furthermore, the distribution of compulational loads across clients allows DFL offer larger scalability, enabling the involvement of a larger number of clients and even the support of larger-scale models without overburdening the central server.
Federated Averaging (FedAvg) [McMahan et al. 2017) is widely adopted FL approach that dis- perses knowledge by averaging the parameters of models. Despite its advantages, FedAvg faces a sig- nificant limitation: the lack of support for model heterogeneity. This limitation becomes impracti- cal in real-world scenarios where devices inherently possess diverse architectures. In DFL system
50
Model Heteregeneity
Global Accuracy (8%
40
30
Data Heterogenelty
20
Homogeneous Models Data Heterogeneous Models Data Heterogeneous Models
with model heterogeneity, FodAvg confines param- eter averaging to models with the same architec- tures, thereby hindering knowledge sharing among clients. This problem exacerbates with the pres- ence of heterogencity in data between clients, affect- ing the preservation of global knowledge (Ye et al.) 2023 Figure [ demonstrates the adverse effects of model and data heterogeneity using decentralized FedAvg. This entails need for a novel framework that better supports model and data heterogeneity in DFL. In this paper, we quantify global knowledge using global accuracy which is measured based on a global test dataset.
10
0
0
500
1000
1500
2000
Communication Round
Figure 1: Demonstrating the adverse impact of model and data hetcrogencity global accuracy using de- centralized FedAvg. The experiment uses CIFAR-100 dataset with 50 clients. Homogeneous models and IID data signify clients with identical model archi- lectures and data distributions. In contrast, hetero- geneous models and non-ITD data indicate variations in both model architectures and data distributions among clients. Additional experimental details can be found in Section 4.4.1
Researchers have extended FedAvg to support model heterogeneity, but these extensions often impose
constraints on model architectures (Diao et al.] 2020; Alam et al. 2022 Shen et al. 2020). Another approach that addresses model heterogencity in FL involves mutual learning, where models collabora- tively learn by teaching each other a task (Zhang et al.) 2018 Li et al. 2021 Mutual learning en- ables knowledge transfer as models mimic each other's class probabilities. In this process, cach client acts as both student and teacher. The objective function comprises supervision loss compo- nent and another responsible for distilling knowledge from experts (Hinton et al.) 2015). However, ex- isting works utilizing knowledge distillation require a server or additional data (Lin et al. 2020 Li & Wang, 2019; Li ct al.) 2020a). Reliance on public data can be impractical, especially in sensitive do- mains such as health. Other works rely on generating synthetic data to perform knowledge trans- fer (Zhang et al.) 2022bb a Li et al.) 2020b) Heinbaugh et al.] 2023) Dai et al.) 2024). which intro- duces an extra privacy concern. Thus, a solution that avoids using additional data more desirable.
To this end, we propose Decentralized Federated Mutual Learning (DFML) frame- work that is 1) serverless, 2) supports model heterogeneity without imposing any architec- tural constraints, and 3) does not require ad- ditional public data. We define decentral- ized or serverless systems as lacking a dedi- cated, centralized client responsible for man- aging knowledge transfer. Table highlights the advantages of our proposed DFML over prior arts. As will be shown in the results sec-
Table 1: Comparison between DFML and other FL methods.
No Server
Nonrestrictive Heterogeneous
No Additional Data
Framework
FedAvg
X
HeteroFLL
x
FedRolex
X
X
FML
X
X
FedDF
X
FedMD
X
/
DT
V
X
DFML (Ours)
2

---

Published in Transactions on Machine Learning Research (08/2024)
Client 1 (Aggregator)
Client 3
Client N
Local Train
W
D3
W3
D
WN
Mutual Learning
Wi
W
W2
W
W
Client 4 (Sender)
Client (Sender)
W2
att
Local Train
Local Train
((
D2
W₂
Wi
W
W1
D
Our proposed DFML framework. In each communccation round t, randomly selected clients (senders) send their locally trained models Wn. to another randomly chosen client (aggregator). Mutual learning takes place at the aggregator using (*), The updated models W and a are then transmitted back to the senders. (1)) controls the impact of the loss components the objective function (see Section 3 and is computed based on a scheduler function. t denotes the current communication round. Different shapes and sizes signify model and data heterogeneity. In this example, clients 2 and act as senders, while client serves as the aggregator.
Figure 2:
tion, DFML outperforms other baselines in addressing the model and data heterogeneity problems in DFL. Figure 2 depicts proposed DFML framework. In each communication round, multiple clients (senders) transmit their models to another client (aggregator) for mutual learning, effectively handling model hetero- geneily. DFML leverages the aggregator's data for knowledge distillation, eliminating the need for extra data. Additionally, DFML addresses data heterogeneity by employing re- Weighted SoftMax (WSM) cross- cross-entroyy Legate al. 2023), which prevents models from drifting toward local objectives.
Furthermore, we observed varying performance levels when using different fixed values to balance the ratio between the supervision and distillation loss components in the objective function. In response, propose eyclic variation (Loshchilov & Hutter 2016 Smith 2017) of the ratio between these loss components. The cyclical approach iteratively directs the model's objective between the aggregator's data and the reliance on experts to distill knowledge using that data. Gradually, as the distillation signal becomes dominant, global knowledge increases and eventually reaches a peak. This cyclical knowledge distillation further enhances global knowledge compared to using a fixed ratio between the supervision and distillation loss components.
In this paper, we empirically demonstrate the superior performance of DFML over state-of-the-art baselines in terms of convergence speed and global accuracy. The main contributions of this paper are summarized follows:
We propose novel mutual learning framework that operates in DFL, supports nonrestrictive het- erogeneous models and does not rely on additional data.
We propose cyclically varying the ratio between the supervision and distillation signals in the ob- jective function to enhance global accuracy.
2
Related Work
Due to the vastness of the FL literature, restrict review to works most relevant to our research. This includes studies that support homogeneous and heterogeneous architectures, operate in DFL, address catastrophic forgetting, and adapt knowledge distillation.
1 Refer Section 5.2
3

---

Published in Transactions on Machine Learning Research (08/2024)
2.1 Homogeneous and Restrictive Heterogeneous Support
FL. aims to learn global knowledge from distributed clients without sharing their private data. FedArg (McMahan et al. 2017) uses server to perform parameter averaging on locally updated models, but its support is limited to homogencous architectures due to the nature of parameter averaging. In contrast, decentralized FedAvg (Li et al. 2021 Roy et al. 2019 Giuseppi et al.) 2022 Savazzi et. al. 2020 depends on client-to-client communication, with aggregation occurring on any participating clients. While other methods support heterogeneous models through parameter averaging, however, their support is constrained.
Methods like Federated Dropout (Caldas al.) 2018), HeteroFL (Diao al., 2020), and FedRolex (Alam) et al. 2022) employ partial training with random, static, and rolling technique for sub-model extractions, respectively. Lastly, FML (Shen et al. 2020) conducts mutual learning between personalized (heterogeneous) models, and global (homogeneous) model. However, FML assumes the existence of a global model, and all the global knowledge resides in that model instead of the clients' heterogeneous models. This differs from our goal of transferring global knowledge to each of the clients' heterogenoous models.
2.2 Nonrestrictive Heterogeneous Support
Works that support model heterogeneity without imposing constraints exist, however, they require assistance from a server and the availability public data (Li & Wang 2019 Lin et al. 2020). FedMD (Li Wang, 2019) assumes the availability of public data on all clients. In FedMD, clients gencrate predictions using public data which are then communicated are communicated back to the clients and are used to update the heterogeneous models using knowledge distillation. FedDF (Lin et al. 2020) communicates heterogencous models to server, where prototypes are assumed to exist. The server facilitates parameter averaging of models with same architectures, and knowledge distillation using unlabelled public data. The reliance on a server and additional data limits the applicability of these methods.
the server for averaging. Then, the averaged predictions
2.3 Decentralized FL
Def-KT (Li et al. 2021) operates within a DFL framework, where clients communicate models to other clients (aggregators) for mutual learning. Despite its serverless nature, Def-KT only supports homogeneous architectures it replaces the aggregator's model with the incoming model. This hinders its effectiveness in scenarios with model heterogeneity.
2.4
Catastrophic Forgetting
Catastrophic forgetting focuses acquiring new tasks without forgetting previous knowledge (Wang et al. De Lange et al.] 2021]. In the context of FL among clients with non-IID data distributions, catas- trophic forgetting occurs, preventing models from reaching optimal global accuracy. To mitigate this issue, rescarchers aim to shield the learned representations from drastic adaptation. Asymmetric cross-entropy (ACE) is applied to the supervision signal of the objective function to address the representation drift problem (Caccia et al.] (021).. ACE uses masked softmax on the classes that do not exist in the current data. Another approach, proposed by Legate et al. (2023), modifies the loss function of each client using re- Weighted SoftMax (WSM) cross-entropy, with re-weighting based on each client's 's class distribution.
2023
2.5
Adaptive Knowledge Distillation
In the literature, existing works have explored scaling the loss function in knowledge distillation (Zhou et al. 2021 Clark et al.] 2019); however, the advancements in this area have been minimal. WSL (Zhou et al. 2021 handles sample-wise bias-variance trade-off during distillation, while ANL-KD (Clark et al. 2019) gradually transitions the model from distillation to supervised learning. Early in training, the model primarily distills knowledge to leverage a useful training signal, and towards the end of the training, it relics more on labels to surpass its teachers. Additionally, although Fed Yogi (Reddi et al. 2020) proposes adaptive optimization, but its reliance server limits its applicability.
4

---

Published in Transactions on Machine Learning Research (08/2024)
3 Proposed Approach
To begin with, outline the DFL system within which our proposed DFML operates. Then, we provide a detailed explanation of DFML without delving into the discussion of varying the balance between the loss components. Lastly, we explain cyclical knowledge distillation and peak models.
3.1 System Setup
In our DEL setup, there are N clients, each client is equipped with local training data Dn E (DI,D2.... Dr regular model Wn. E {W1, W2, ... Wx), peak model Wn (2,WW,... Wx), and local weight alpha an E {ai, a2, j. In each communication round, client is chosen at random to act as the aggregator a. The choice of the aggregator is determined by the previous aggregator from the preceding round. In the initial round, the client with the lowest number is assigned the role of the aggregator. Additionally, several clients are randomly chosen as senders 8 send their models the aggregator.
3.2 DFML Formulation
The goal of DFML is to enable the exchange of lo- cal information among clients without the need to share raw data. DFML facilitates knowledge shar- ing across multiple communication rounds T until convergence. In each round T, set of par- ticipants (P S U {a}) is randomly selected. For each selected clients n E P, their regular models W,, undergo local training on their private data D,. This ensures that W,, retains its local knowlodge, allowing knowledge transfer to other parlicipants during the aggregation process. The process of dis- tilling knowledge other models is referred aggregation. Subscquently, all locally trained mod- els are sent to the aggregator for the aggregation process. When multiple clients send their mod- els to the aggregator, multiple experts contribute during aggregation, enhancing the accuracy of the global knowledge transfer.
Algorithm DFML
Initialize N clients, each client n has data Dn and two models: regular W,, and peak Wn. The local weight alpha of each client On O. for communication round t 2,2,.... ...,'' do Randomly select one aggregator E {I,..,, N} Randomly select senders S C a Participants p SU
Input:
\\ Client Side for all n E P do for all batch X, local data D, do
Wn < locally train Wn using Equation 2 Send locally updated models W. for all s e to a
1) Aggregator Side
(() + scheduler(.)
for l: = 1, 2, ..., do for all batch X, E local data Da do do 2n logits(Wn,,
for all
DFML employs weighted mutual learning for ag- gregation to allow models to collaboratively learn from each other using the aggregator's 's data. This technique ensures that larger models contribute more significantly to knowledge transfer compared to smaller models, leveraging their finer knowledge. DFML conducts the aggregation process K times to maximize knowledge transfer without impacting the communication cost per round. Following this, all updated models Wi+ are transmitted back to their respective senders. Subsequently, each par- ticipant n E P replaces its model W,, with the updated version W++. This entire process re- peated until global knowledge has been dissemi- nated across the entire network. Cyclical knowl- edge distillation and the peak models are explained in Section 3.2.1 Algorithm describes our proposed DFML framework. The objective function of DFML is as follows:
for all P do Update Wn using (1)) and logits according to Equation [
Send back updated models W and (t) for all
8 SES
11 Client Side
for all n E P do
Wn t W \1 Update regular model
if (t) 2 an then
W < W \\ Update peak model Set local weight on +- (1)
(1 LWSM aLKI.
(1)
5

---

Published in Transactions on Machine Learning Research (08/2024)
where LWSM represents the supervision loss signal computed using re-weighted softmax crtssyy (WSM) [Legate et al. 2023), and LKI. represents the distillation loss signal computed by Ballback-Leibble Divergence (KL). The hyperparameter a controls the balance between these two loss components. LWSM is defined as follows:
Σ (e )
CWSM = Σ (((())))))) - log xE
(2)
'EC
where Xa represents a data batch drawn from the distribution Da of aggregator a. in denotes the logits of data sample x with model weights W,, of client n, y(x) is the data label, 3, is a vector representing the proportion of label с present in the aggregator's dataset, and C' the set of classes in the entire dataset. During the aggregation process, LWSM has access only the aggregator's data and computed for cach W, available at the aggregator. However, during the local training stage, i.e. before the models are sent to the aggregator, LWSM is also used at each client n, exploiting its private data to undergo local training.
The distillation loss component LKL is defined as follows:
P
B P uutn
P
(L(p(()) |l pn()))
(3)
Σ
ZEX, 77
where Xa corresponds to a data batch drawn from the distribution Da of aggregator a. P denotes the set of participants in mutual learning, including the senders and the aggregator. & represents the model size based on the number of trainable parameters. Finally, P²(x) and (x) are the teacher q and student n predictions of the data sample z with model weights Wg and Wa, respectively. u a dummy variable indexing all teachers.
The use of WSM in both local training and mutual learning serves as protective measure against catas- trophic forgetting, which arises from shifts in data distribution between clients. WSM ensures that the modols update parameters considering the proportion of available labels. This strategy prevents models from altering their accumulated knowledge on labels that are not present in the current data distribution, thereby safeguarding against catastrophic forgetting,
3.2.1 Cyclic knowledge distillation
Cyclical knowledge distillation is manifested by periodically adjusting the value of a in the objective function during each communicatio round. Inspired by (Loshchilov & Hutter] 2016 [Smthh] 2017 Izmailov et al. 2018), use the cyclical behavior to vary a. Cosine annealing scheduler, defined in Equation 4 is used to generate the cyclical behavior. This dynamic variation in a occurs with each new aggregator selection, leading to mutual learning with distinct value al each round. The cyclical process, influencing the balance between the supervision and distillation loss components, contributes to an overall increase in global knowledge. The global knowledge is measured by global accuracy throughout training.
a (t) = amin
2 (Omaz - Ommnn (1 + cos( +)));
(4)
where (t)) is the value at the current communication round 6, T is the maximum communication round, while amin and amaz are the ranges of the a values.
Figure 3 depicts the impact of cyclical a on global accuracy. When the supervision signal dominates, each Wn exclusively learns from D2 without collaborating with other models (experts). This focus on the supervision signal directs the model's objective toward Da: causing W,, lose previously acquired global knowledge from earlier rounds. As the distillation signal gains prominence, W,, begins reacquire global knowledge, facilitating simultaneous knowledge distillation among all models. With dominant distillation signal, each model exclusively learns from other experts, maximizing global knowledge in each one.
Therefore, the peak in global accuracy is reached when the distillation signal is dominant (a is maximum), and the lowest attained when the supervision signal is dominant (a is minimum). We observed that cyclically changing between the two signals leads to a higher global accuracy compared to using either one exclusively or linear combination of them. This cyclical adjustment of a is crucial for the continuous growth in global accuracy throughout training, albeit with fluctuations in global accuracy.
6

---

Published in Transactions on Machine Learning Research (08/2024)
C (1 ax) (Supervision signal) + (Distilaatonn signal)
Peak models
update
Distillation signal only
Supervision
signal only
Increase in global accuracy as distillation signal becomes dominant
Decrease global accuracy supervision signal becomes dominant
Figure 3: Illustrating the impact of cyclically varying a on global accuracy. Peak models are updated up the first a maximum and every subsequent time reaches its maximum limit. In this example, is varied using& a cosine annealing scheduler.
3.2.2 Peak models
To counteract the undesirable fluctuations in global accuracy resulting from the cyclical process, we introduce an additional model for each client, termed the peak model Wn- The primary role of Wn to retain the best global parameters of Wx. Specifically, each W,, updated whenever W,, is aggregated with a dominant distillation signal. Wn are detached from the training process and are kept in frozen state, preserving the maximum global accuracy achieved so far. Also, the peak models continuously updated from the initial communication round up to the first maximum, allowing them to quickly reach the first global accuracy peak. Thus, the peak models act as stabilizing mechanism, retaining the optimal global knowledge attained.
4 Experiments
4.1 Dataset
We evaluate our proposed DFML against prevalent baselines using five datasets including CIFAR-10/100, FMNIST, Caltech101, Oxford Pets, and Stanford Cars. The evaluation covers experiments on two data distribution shifts: Independent and Identically Distributed (IID) and non-IID. The non-IID distribution involves a heavy label shift based on the Dirichlet distribution with B = 0.1. The dataset is distributed among clients by dividing the train set into N splits, either evenly for IID or utilizing Dirichlet distribution for non-IID. Fach split is further segmented into training and validation sets following Caltech101, samples are first split 80:20, where the 20% represents the global test set, and the remaining samples follows the defined splitting strategy above. Validation sets are employed to assess local performance, while the entire test set evaluates the global accuracy of the clients. The global accuracy of DFML is evaluated by examining the peak models unless stated otherwisc. The data partitions for all clients are available in Appendix A.1
80:20 ratio. For
4.2 Implementations
In our experiments, we utilize CNN, ResNet, VIT, and EfficientNet our model architectures. The evalu- ation of DFML encompasses three architecture modes: homogeneous, restrictive heterogeneous, and nonre- strictive heterogeneous. We name these three modes: HO, H1, and H2, respectively. Details of these modes and the associated model architoctures are outlined in Table To ensure a fair comparison, all experiments including the baselines are run with WSM instead of Cross- Entsopyy (CE), unless specified otherwise. Further implementation details are available in Appendix A.2

---

Published in Transactions on Machine Learning Research (08/2024)
Table 2: Different model specifications supported by the baselines and DFML in the network.
Different Model Types
Different
Different
Framework
# of Layers
Width layers
Dec. FodAvg
X
Dec. FedProx
x
Def-KT
X
Dec. HeteroFL
X
Dec. FedRolex
X
DFML (Ours)
4.3 Baselines
To address the absence baselines tailored for DFL settings, we derive baselines by decentralizing some state- of-the-art CFL algorithms, adapting them function within our DFL system. We begin our experiments by constraining model architectures to enable comparison with existing basclines. The initial experiments focus homogeneous architectures for direct comparison with baselines like decentralized FedAvg, FedProx Li et al.) 2020ch and Def-KTT, Our derived decentralized version of FedAvg and FedProx are referred to as decentralized FodAvg and FedProx, respectively. Def-KTT intrinsically operates in a DFL framework.
Subsequently, we conduct experiments with restrictive heterogeneous architectures, where models have the same number of layers but different widths. This allows comparison of DFML with our derived decentralized versions of partial training methods (HeteroFL and FedRolex) alongside FedAvg. Further details on the derived decentralized baselines are provided in Appendix A.3
Following this, we demonstrate the full capabilitios of DFML by conducting experiments with nonrestrictive heterogeneous architectures, which are incompatible with partial training algorithms. The only baseline available for this set of experiments is decentralized FedAvg. In this paper. we omit the comparison of
DFML with baselines that require additional data such as (Lin al. 2020 Li & Wang 2019) fairness.
ensure
Table 2 summarizes the model features supported by the baselines and our proposed DFML in the network. Decentralized FedAvg, decentralized FedProx and Def-KT baselines only accommodate homogeneous models, requiring all models to be of same type, with the same number of layers, and each layer have the same dimensions. Decentralized HeteroFI. and FedRolex baselines support models of the same type, with the same number of layers, but they allow layers to have different widths. contrast, our proposed DFML can support different model types, models with different number of layers, and varying widths.
4.4 Results
We evaluate DFML against other state-of-the-arl baselines. We first demonstrate the effectiveness of DFML in handling model and data heterogeneity. Second, we prove that DFML outperforms baselines terms of final convergence speed and final global accuracy across three architecture modes: homogeneous, restrictive heterogeneous, and nonrestrictive heterogeneous architectures. Next, we demonstrate the performance of each cluster of architectures in experiment with nonrestrictive heterogeneous architectures. Finally, we present the scalability of DFML under a significant model heterogeneity scenario.
4.4.1 Model and Data Heterogeneity
Figure demonstrates that our proposed DFML under model and data helerogeneity, mitigates the impact on global accuracy more effectively than decentralized FedAvg and HeteroFL. To ensure fair comparison between homogeneous and hetcrogeneous experiments, ters in each case. This was achieved by selecting the median model of the five different architectures from the heterogeneous experiment as the model used in the homogeneous experiment.
maintained the same average number of parame-
4.4.2 Homogeneous Architectures
Table demonstrates that our DFML outperforms decentralized FedAvg, decentralized FedProx, and Def- KT in terms of global accuracy across datasets and under two data distributions. Larger improvements are
8

---

Published in Transactions on Machine Learning Research (08/2024)
Model Helerogeneily
40
Accuracy
30
Heterogeneity
Dec. FedAvg
Homa. Models and IID Data
Dec. HeteroFL
Hetero. Models IID Data
(Ours)
Hetero. non-IID Data
500
1000
1500
2000
Communication Round
Figure 4: Demonstrating the global accuracy gain DFML achieves in comparison with decentralized FedAvg and HeteroFL under model and data heterogeneity. CIFAR-100 dataset is used with 50 clients and CNN architectures.
Table 3: Global accuracy comparison using homogeneous CNN architectures with 50 clients and 25 senders. For Def-KT, 25 aggregators are selected, and 1 aggregator for the other methods.
CIFAR-10
CIFAR-100
Method
IID
non-III
IID
non-IID
Dec. FodAvg
80.00 am
74.68 nm
48.32 u1
43.17 a.s
Dec. FedProx
79.93 : a.14
74.55 = 0.0
47.97 2m
0.4
Def-KT
79.29
72.59
48.43
43.84
DFML (Ours)
80.96
76.27
50.47
46.41
Table 4: Global accuracy comparison using restrictive heterogeneous architectures with 50 clients and 25 senders. Architectures used are CNN, ResNet, and VIT.
ResNet
CIFAR-10
CIFAR-10
CIFAR-100
Method
IID
mon-IID
son-1ID
non-1ID
IID
1ID
nos-IID
Fodlex
50.86
$3.50
79.84
26.50
52.76
HeteroPP
kmer
Reditooo
cm
27.65
81.70
53.06
DFML (Ouns)
79.02 1..
75.87 km.
18.60
44.20
71.24 1000
53.52
46.27 1m
68.38 1..
54.50
28.56
Table 5: Global accuracy comparison using restrictive heterogeneous architectures. conducted using ResNet architectures with 50 clients and 25 senders.
The experiments are
FMNIST
Caltech101
Oxford Pets
Stanford Cars
Method
non-IID
IID
non-IID
IID
non-IID
HID
non-IID
Dec. FedAvg
84072 100
73.53z
36.7 36.74g
22.832
10:20gon
9.212on
2.83g
3.44
Dec. HeteroFL
91.061 am
85.19: cm
19.05. ea:
38 38.041 w.s
23.35. wat
17.08. wa
5.56. w..
6.25 use
Dec. FedRoiex
91.26.
88.06.
49.14.
37.75.
22.67-
18.39-
5,88
6.09.
DFML (Ours)
92.01 0.e
87.31: wae
60.05: oos
50.07: uw
40.42: uxe
31.25 was
41.12: o.se
25.31: cav
recorded under non-IID data shifts. To align the number of communications per round for all baselines, adjustments were made for Def-KT by sctting the number of aggregators to 25. This is necessary as in Def- KT each sending model should be received by a different aggregator. Decentralized FedProx is not explored further in the remaining experiments as it yielded approximately the same performance as decentralized FedAvg under homogeneous architectures. Centralized FedProx requires the availability of a global model at each participating client to restrict local updates to stay close to the global model. However, in a decentralized setting, a single global model does not exist, and each client treats its local model, before local training, as a global model. This limitation explains why FedProx did not outperform FedAvg in the decentralized setting.
4.4.3 Heterogeneous Architectures with Restrictions
In this set of experiments, some restrictions are applied to heterogeneous architectures. Tables and 5 demonstrate that DFML consistently outperforms all baselines terms global accuracy across different architectures, datasets, and both IID and non-IID data distributions. Another observation that the partial training baselines outperformed decentralized FedAvg, which is expected as more knowledge is shared by averaging overlapping parameters of the models rather than only averaging models with the same architec-
9

---

Published in Transactions on Machine Learning Research (08/2024)
Table 6: Global accuracy comparison using nonrestrictive hetcrogencous architectures. The experiments are conducted using CNN architectures. Different numbers of clients N are used, with 50% x.N as senders.
10
:0
100
CIFAR-10
CIFAR-100 100
CIPAR-10
CIFAR-100
CIFAR-10
CIFAR-100 100
Method
IID
Bon-IID
IID
Bon-IID
IID
non-IID
11D
no0 HID
IID
nop- IID
IID
nom- IID
Dee: Feelbeg
71.19
1575 am
33.58
7501 1.m
71.51 71.51 A.
10:05 or
33.19 amm
70.27 ..
71 00
57.70 ...
32-30
25.0 ...
DEML (Ours)
60.87
74,30
54.03
49.07
51.74 804
76.51
50.39
46.22
79.94 coa
71.75
47.00
42.84
Table 7: Global accuracy comparison using nonrestrictive heterogeneous architectures. The experiments are conducted using EfficientNet architectures with 10 clients and 5 senders.
Caltech101
Oxford Pets
Stanford Cars
Method
IDD
non-IID
IID
non-III
IID
non-IID
Dec. FedAvg
54.75
30.92:
18.34
15.34. on
6.07:
11.83 a1
DFML (Ours)
83.42
65.70.
68.90.
42.941
70.29. 0.39
50.63.
Table 8: Overall communication rounds DFML requires to attain the samn accuracy as decentralized FedAvg achieves at communication rounds 100 and 500. Both methods have the same commuication cost per round. Different numbers of clients N are used.
N:10
N 50
N 100
Dec. FedAvg
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Communication Round
Method
IID
non-III
IID
non-III
IID
non-IID
HID
non-IID
IID
non-III
IID
non-III
DFML (Ours)
100
10
20
20
30
30
40
50
300
20
90
150
130
130
Table 9: Global accuracy comparison between DFML and decentralized FedAvg with different supervision signals and cyclically varying a. The dataset used CIFAR-100, with 50 clients and 25 senders.
CNN
ResNet
ViT
Method
CE / WSM
Cyclical a
IID
non-lID
IID
non-lID
IID
non-IID
CE
X
47.44 .7.44
29.16
49.13 oao
18.78
35.66
14.02
DFML (Ours)
CE
V
48.44 a
41.47
55.29
41.07
36.62
22.34
WSM
X
47.57
43.59
49.77
30.91 2.00
34.04
25.04
WSM
V
48.60 0.0x
44.26
53.32
46.27
36.48
28.56
w.e
tures. Furthermore, we implemented decentralized Federated Dropout; however, the results are not reported as it did not work. The poor performance is attributed to local models being assigned random parameters from the global model generated at the aggregator.
4.4.4 Heterogeneous Architectures without Restrictions
After confirming that our proposed DFML competes with the prevalent baselines in DFL, we showcase the strength of DFML in knowledge transfer using nonrestrictive heterogeneous models. Additionally, we evaluate DFML under different N clients. From Tables 0 and 7 it is evident that across various N clients, datasets, and data distributions, DFML consistently achieves superior global accuracy compared to the baseline. Table shows that DFML requires fewer communication rounds to reach the same accuracy as decentralized FedAvg does at specific rounds. The communication cost per round for all experiments is 50, involving the transmission of 25 models to and from the aggregator. Appendix A.4 provides further experimental analysis on DFML. A visual illustration of the convergence speedup achieved by DFML compared to the baselines is presented in Appendix A.4.4 Moreover, comparison between DFML and a decentralized version of FML is provided in Appendix A.4.1I
4.4.5 Performance per Cluster of Architectures
Figure 5 presents the performance of each architecture cluster in DFML compared to decentralized FedAvg, In this experiment, five different CNN architectures are distributed among 50 clients. The figure demonstrates that the bigger the architecture size, the higher the attained global accuracy. Moreover, all clusters in DFML surpasses their corresponding clusters in decentralized Fed Avg.
10

---

Published in Transactions on Machine Learning Research (08/2024)
H2. CNN, CIFAR-100, HID, N:50, 15|25
H2. CNN, CIFAR-100, non-lID, N:50, [51:25
50
50
40
CO
C3
CD
C3
Dec. FedAvg
Cl
C4
Dec. FedAvg
C1
CA
DFML
C2
DFML
C2
500
1000
1500
2000
500
1000
1500
2000
Figure 5: Performance comparison between the different architecture clusters in both DFML and recentral--
ized FedAvg. In this experiment, five nonrestrictive heterogeneous architectures are distributed among 50 clients. CO, C1, C2, C3, and C4 represent the global accuracy average of all models with CNN architectures [32, 64, 128, 256], [32, 64, 128], [32, 64], [16, 32, 64], and [8, 16, 32, 64], respectively.
VIT ResNet CNN, CIFAR 100, HI,, N:50, |5|:25
VIT + ResNet + CNN, CIFAR 100, non IID, N:50, [5]:25 55
50
Global Accuracy (%)
40
40
Global Accuracy
30
30
20
20
10
Dec. FedAvg
Dec. FedAvg
1000
2000
1000
1500
2000
Communication Round
Communication Round
Figure 6: Comparison between DFML and decentralized FedAvg under significant model heterogeneity, Ten
different architectures are distributed among 50 clients including: VIT, 4 ResNet, and CNN architectures.
4.4.6 High Model Heterogeneity
To showcase the scalability of DFML with model heterogeneity, we conduct an experiment involving 50 clients with significant model heterogeneity. We compare the results obtained by DFML. to decentralized FedAvg. In this experiment, ten different architectures are deployed and selected from Table I0 the two largest ViT architectures, the four largest ResNet architectures, and the four largest CNN architectures under the H2 category. Figure T demonstrates that DFML. performs effectively under heavy model heterogeneity conditions and greatly outperforms decentralized FodAvg.
5 Analysis
In this section, we first demonstrate the effect of using different fixed a values versus cyclically adjusting it. Second, we present the impact of cyclical a on global accuracy by evaluating the regular models. Sub- sequently, we examine the behavior exhibited by both regular and peak models as a changes, and highlight the ability of the peak models capture the peaks in global accuracy. Last, we conduct an analysis understand the effects of different supervision signals and cyclical a on global accuracy.
5.1 Regular vs Peak Models
Figure 7 shows the fluctuations in global accuracy as a is cyclically varied. When a = 0, representing only the supervision signal in the objective function, global accuracy at its lowest. This is attributed to models being optimized solely toward the aggregator's local data. However, as a increases and eventually reaches its maximum defined value, models gain knowledge from each other through knowledge distillation, resulting
11

---

Published in Transactions on Machine Learning Research (08/2024)
H2, CNN, CIAA--10, non-ID, N:50, 151:25
H2. CNN, CIFAR-100, non-lID, N:50, [5|:25
1.0
30
1.0
Global Accuracy (8
25
0.8
0.8
20
Global Accuracy
20
0.6
0.6
15
15
0..
-0.4
10
5
DFML Full- Cycle Oscillation
5
DFML Half Cycle Oscillation
100
150
200
250
100
150
200
250
Communication Round
Communication Round
Figure 7: Performance comparison of the regular models against cyclically oscillating a with full and half
cycles, respectively. The supervision signal used is CE.
CNN, non II,, N:50, [5|:25
Global (96
30
20
10
Regular Models
DFML Peak Models
0
500
1000
1500
2000
Communication Round
Figure 8: Performance comparison between DFML regular models, that are communicated and updated in each communication round, and peak models that are updated only when a peak occurs. The supervision signal used is CE.
in a peak global accuracy. The decline in global accuracy starts when the distillation signal diminishes, and the supervisory signal takes over. To maintain global accuracy at the peaks, the peak models are used. Initially, the peak models are updated regularly with every regular model update until the first maximum a value is reached. After that, the peak models are only updated when the regular models are aggregated using the highest value of a. Figure depicts the global accuracy of both regular and peak models, highlighting the stability achieved by the peak models throughout training.
5.2 Fixed vs Cyclical
As shown in Figure 7 the highest global accuracy is consistently achieved when reaches its maximum defined value. Now, we address the impact of using different fixed values for a and whether setting a = throughout training yields similar performance as cyclically changing a. Figure | demonstrates that using different fixed values of a, under different distribution shifts, results in varied performance levels, Furthermore, fixing
leads to the worst global accuracy because when only the distillation signal is present throughout training without any supervision, noise will be propagated. This results in experts leaching each other incorrect information.
5.3 Supervision Signal and Cyclical
The impact of different supervision signals (CE and WSM) and cyclical a on DFML is presented in Table g Results indicate that the use of WSM primarily enhances the global accuracy non-lID data distribution shifts. Furthermore, the addition of cyclical on top of CE and WSM further improves global accuracy. The best outcomes are mostly reported when both WSM and cyclical are applied. Additional experimental analysis cyclical knowledge distillation be found in Appendix A.5
12

---

Published in Transactions on Machine Learning Research (08/2024)
H1, CNN, CIFAR-100, IID, N:50, 18:25
H2. CNN, CIFAR-100, non-lID, N:50, [5]:25
50
50
DFNLL a:0.3
DFML a:0.5
Global Arcuracy (1)
Global Accuracy (56)
DFML
30
30
DFML 0:0.3
DFML 10ML--::000 0:0.5
20
DFML a:0.9
20
DFML 0:1.0
DFML Cyclical
DFML Cyclical
500
1000
1500
2000
500
1000
1500
2000
Communication Round
Communication Round
Figure 9: Performance comparison between different fixed values and cyclically varying it, under IID and non-IID data distributions. The supervision signal used is CE.
6 Limitations
While DFML presents significant advancements over existing baselines, it does have certain limitations that need to be acknowledged. One notable limitation is its computational expense. DFML roquires aggregators to possess sulficient memory to receive multiple models and enough compulational power to perform mutual learning, involving both forward and backward passes on all models at the aggregator. Another limitation is the lack convergence analysis within in this paper. While proving the convergence properties of DFML is beyond the current scope, proving the convergence properties of DFML is left for future work.
7 Conclusion
We proposed DFML, a framework that supports a decentralized knowledge transfer among heterogeneous models, without architectural constraints or reliance on additional data. DFML overcomes common central- ization issues such as communication bottlenecks and single points of failure, making it robust alternative for real-world applications. DFML outperformed state-of-the-arl baselines in addressing model and dala heterogeneity in DFL, showcasing better convergence speed and global accuracy.
References
Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Model-heterogennous federated learning with rolling sub-model extraction. Advances in Neural Information Processing Systems, 35:29677-29690, 2022.
Enrique Tomás Martíncz Beltrán, Mario Quiles Pérez, Pedro Miguel Sánchez Sánchez, Sergio López Bernal, Gérôme Bovet, Manuel Gil Pérez, Gregorio Martínez Pérez, and Alberto Huertas Celdrán. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges. TEFE Communi- cations Surveys & Tutorials, 2023.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05022 2021.
Scbastian Caldas, Jakub Konečny, Brendan McMahan, and Amoct Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXio preprint arXiw:1812.07210 2018.
Kevin Clark, Minh-Thang Luong, Urvashi Khandclwal, Christopher D Manning, and Quoc V Le. Bam! born-again multi-task networks for natural language understanding. arXio preprint arXiv:1907.0453 2019.
Rong Dai, Yonggang Zhang, Ang Li, Tongliang Liu, Xun Yang, and Bo Han. Enhancing one-shot federated learning through data and ensemble co-boosting. Xiv preprint arXio:2402. 15070, 2024.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE trensactions on pottern analysis and machine intelligence, 44(7):3366 3385, 2021.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogencous clients. arXiv preprint arXiv:2010.01: 0124, 2020.
Alessandro Giuseppi, Sabato Manfredi, Danilo Menegatti, Antonio Pietrabissa, and Cecilia Poli. Decen- tralized federated learning for nonintrusive load monitoring in smart energy communities. In 2022 30th Mediterranean Conference on Control and Automation (MED), pp. 312-317. IEEE, 2022.
Clare Elizabeth Heinbaugh, Emilio Luz-Ricca, and Huajie Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, Distilling the knowledge in a neural network, arXiv preprint WxXiu1503.0255) 2015.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane, Fjord: Fair and accurate federated learning under heterogencous targets with ordered dropout. Advances in Neural Information Processing Systems, 14:12876-12888 2021.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 054007 2018
Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets. arXiv preprint arXi::2122. 13492, 2021.
Gwen Legate, Lucas Caccia, and Eugene Belilovsky. Re-weighted softmax cross-entropy to control forgetting in federated learning. arXiv preprint arXiv:2304.05260, 2023.
Chengxi Li, Gang Li, and Pramod K Varshney. Decentralized foderated learning via mutual knowledge transfer. IEEF Internet of Things Journal, 9(2):1136-114 2021.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.0830 03008881, 2019.
Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. arXiv preprint 2ii:22000.01017, 2020a.
Qinbin Li, Bingsheng Hc, and Dawn Song. Practical one-shot federated learning for cross-silo setting. arXiv preprint arXtv:2010.01017, 2020b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020c.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXio preprint S.Xi:1668.0095. 2016.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiienn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273 1282. PMLR, 2017.
Sashank Reddi, Zachary Charles, Manzil Zaheor, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXio preprint orXiv:2003.0029. 2020.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Abhijit Guha Roy, Shayan Siddiqui, Sebastian Pölsterl, Nassir Navab, and Christian Wachinger. Braintor- rent: A peer-to-peer environment. for decentralized federated learning. arxiv preprint 2019.
Stefano Savazzi, Monica Nicoli, Vittorio Rampa, and Sanaz Kianoush. Federated learning with mutually cooperating devices: A consensus approach towards server-less model optimization. In ICASSP 2020-2020 IEEE International Conference on Acouslics, Speech and Signal Processing (ICASSP), pp. 3937 3941. IEEE, 2020.
Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning. arXiv preprint arXi::206.1167 2020.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464-472. IEEE, 2017.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.0044 2023.
Mang Ye, Xiuwen Fang, Bo Du, Pong C Yucn, and Dacheng Tao. Heterogencous federated learning: State- of-the-art and research challenges. ACM Compuling Surveys, 56(3):1 44, 2023.
Liangqi Yuan, Lichao Sun, Philip Yu, and Ziran Wang. Decentralized federated learning: A survey and perspective. arXiv preprint orxi022306. 01603, 2023.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Data-free one-shot federated learning. Advances in Neurol Information Processing Systems, 35: 21414 21428, 2022a.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yo Yu Duan. Fine-tuning global model via data- free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVE CVF conference on computer vision and pattern recognition, pp. 10174, 2022b.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 4320-4328, 2018.
Helong Zhou, Liangehen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Re- thinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. aiv preprint arXiv:2102.00650 2021.
15

---

Published in Transactions on Machine Learning Research (08/2024)
CIFAR-10, IID, N:10
CIFAR 10, IID, N:50
CIFAR 10, IID, N:100
8
8
Label
5
3
6
10
11
16
21
26 Client
31
36
41
46
50
91 100
Client
Client
CIFAR-100, IID, A:10
CIFAR-100, IID, N:50
CIFAR-100, N:100
90
80
80
80
Label
60
60
60
40
40
40
30
20
20
10
10
1
2
4
5
7
6
10
6
21
16
21
26
31
36
41
46
50
12
21
31
41
51
61
71
81
91
100
Client
CIFAR N:10
CIFAR: 10, IID, N:50
CIFAR IID, N:100
Label
3
6
10
11
16
21
26
31
36
41
46
50
11
21
31
41
51 Client
81
91 100
Client
Client
non-IID,
CIFAR-
99
99
90
90
90
70
70
60
60
50
50
40
40
30
20
20
20
0
0
1
7
10
6
16
26 Client
36
46
50
21
31
41 51 61 71 Client
81
900
Client
{10, 50, 100} for both CIFAR-10 and CIFAR-100 datascts. The size of the red circle represents the magnitude of data samples for each class label in each client.
Figure 10: Data partitions based on IID and non-lII distribution with clients
A Appendix
A.1 Dataset Distributions
The data partitions for both IID and non-IID distributions of CIFAR-10 and CIFAR-100 datasets are illus-
trated in Figure 10
16

---

Published in Transactions on Machine Learning Research (08/2024)
1.0
0.8
Alpha (a)
0.6
0.4
0.2
0.0
0
100
200
300
400
500
Communiation Round
Figure 11: Cyclic oscillation of a with incremental period increase. or ranges from 0 - 1.
A.2 Implementation Details
In our setup, assume star topology where all clients send and receive models from any other client. Each communication round involves randomly selecting 50% of the clients as senders S, with an additional client randomly chosen as the aggregator a (unless specified otherwise). We utilize SGD optimizer for each client with momentum 0.9 and weight decay 5e-4. The learning rate is selected from {0.1, 0.01, 0.001). The batch size set to for the EfficientNet experiments, 16 for the ResNet experiments using Caltech101, Oxford Pets, and StanfordCars datascts, and batch size of 64 is used for all other experiments, For the cyclic scheduler, increased after each completion. is oscillated from 0 to maximum value selected between {0.8, 0.9, 1.0). Figure [1 illustrates an example of the behavior of throughout training. The number of mutual learning epochs K, performed at the aggregator, is set to 10. Moreover, the temperature is configured to 1. All experiments are repeated for 3 trials with random seeds. With the existence of an a and period schedulers, aggregators need to aware of communication round t, and the round when the period was last updated compute the period and a value. To achieve this, each aggregator needs communicate these two values the next aggregator.
apply cosine anmealing. The initial oscillating period set to 10 and is incrementally
The architectures employed in our experiments are presented in Table 10 Modes HO, H1, and H2 refer to homogeneous, restrictive helerogeneous, and nonrestrictive heterogeneous architectures, respectively. In mode H1, the model rates for different model types are: [2", 12,22-1,2 2-,,2-2,2 2,23,,2 3-3,._'.. Here, smaller models are scaled versions the largest model in terms of width. Mode H2 designates heterogencous architectures with no constraints, allowing each client to have a different number of layers and hidden channel sizes. In our HI and H2 experiments, the models are evenly distributed among clients.
For CNNs, the values inside the array represent the number of channels in each layer, and the array's length corresponds to the number of layers. Each CNN layer has 5 kernels followed by the ReLU activation function, 2x2 2 max pooling, and layer normalization. In ResNets, the largest model is pre-activated ResNet18, while others are scaled versions of 18 The values inside the array represent the number of channels per layer, and each layer consists two blocks. Similarly, for ViTs, the largest model is comprised with 512 channels each; the others are scaled versions the largest VIT. Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA) (Lee et al. 2021) are used our ViT architoctures solve the lack of locality inductive bias and enable us to use non-pretrained ViTs on small datasets. Additionally, for ViTs, the patch size is set to 4 x 4, head dimensions to 64, depth to 2, dropout to 0.1, and embedding dropout to 0.1.
layers
A.3 Baselines
A.3.1
Decentralized FedAvg
When senders send their heterogeneous models to the aggregator, docentralized FodAvg performs parameter averaging exclusively among models with identical architectures. In the homogencous scenario, averaging encompasses all available models. In contrast, in heterogeneous scenarios, clusters of global models
17

---

Published in Transactions on Machine Learning Research (08/2024)
Table 10: Different architecture modes.
Mode
Model
Name
Heterogeneity
Type
Architectures
HO
Homogeneous
CNN
[32, 64]
128, 256]
[64, 128]
CNN
[32, 64
[16, 32]
[8, 16]
64, 128, 256, 512]
[32, 64, 128, 256)
Restrictive
H1
ResNet
[16, 32, 64, 128]
Heterogencous
8, 16, 32, 64)
[4,8, 16, 32]
[512, 512]
[256, 256
VIT
[128, 128
664, 64
332, 32]
[32, 64, 128, 256
[32, 64, 128]
CNN
[32, 64]
[16, 32, 64)
Nonrestrictive
[8 16, 32, 64]
H2
Heterogeneous
B0
B1
EfficientNet
B2
B3
B4
formed identical models in cach group are averaged together. The resulting global models are then communicated back to the clients with the same model architecture. During parameler averaging, weights are assigned based on the number of data samples in each client. Algorithm 2 provides a detailed explanation of decentralized AvdAvg. Model parameters FedAvg are aggregated as follows:
W ΣnEp dn
Σ dnW,..
DEP
where W is the global model and W,, is the model of client The weight d, is based on the number of data samples in the client.
A.3.2 Decentralized FedProx
Decentralized FedProx similar decentralized FedAvg in that subset of clients send their locally trained models to the aggregator for parameter averaging to form global model, which is then sent back to the senders. However, FedProx adds proximal term to the local training objective, which helps improve the method's stability by restricting the local updates to be closer to the initial (global) model. The proximal term necessitates keeping a copy of the global model while performing local training. In decentralized setting with partial client participation, multiple versions of global models exist throughout the network. Decentralizing FedProx requires each client to treat model as the global model before performing local training. Algorithm
2 provides detailed explanation of decentralized FodProx. The hyperparameter μ is selected from (0.5,1,5 1,2).
18

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 2 Decentralized FedAvg and FedProx
Input:
Initialize N clients, each client has model Wn and data Dn. All models with the same architectures have the same initialization. for communication round t= 1,,,.... T do
Randomly select one aggregator a E .I, N} Randomly select senders 5 C (I,..., N). a.000 $ Participants SU(()
1\ Client Side for all E P do
W < W
for all batch X, E local data D, do
Wn < locally train Wn using Equation
Wn - locally train Wa using Equation 2 + W
Wn
Send locally updated models W, for all E S to a
11 Aggregator Side
Each cluster for all E U do
U contains group of models of same architectures.
W + Aggregate homogeneous models, for all W, E u, according to Equation
for all u E 1 do for all W. u do
Wn < W \\ Fork W into local models
Send back updated models W, for all
A.3.3 Decentralized Partial Training
Similar to DFML and decentralized FedAvg, in decentralized partial training methods each client owns local model, and in each communication round several clients are randomly selected. One client is designated as the aggregator, while the others act senders. The senders transmit their models to the aggregator after training their models locally. At the aggregator, the largest available model serves as the global model W,. A Algorithm 3 provides a detailed description of decentralized Federated Dropout, HeteroFL, and FedRolex. Model parameters in partial training methods are aggregated as follows:
W W.B = EnEp dn
Σ d da W.,.s)))
(6)
PEP
where W(..)) is the jth parameter at layer i of global model W,, while WRitss is the jth parameter at layer of client n. The client weight is equal for all clients, dn P/
Decentralized Federated Dropout In each communication round within Federated Dropout (Caldas et al. 2018), sub-models (local models) are extracted from the centralized W,, based a random selection pro- cess. The parameters representing the sub-models that are randomly chosen through this selection are then transmitted to clients for local training. After local training, the updated parameters are sent back to W for aggregation. The random extraction schome for layer of sub-model W,, for client n is extracted from W, as follows:
X,, {ja integer ja [0, for 1 s < [rn]]],
(7)
where X is the parameter indices of layer i extracted from W,- denotes W,, rate relative to W J denotes the total number of parameters in layer of W,. A total of [raJa] is randomly selected from layer of W, for W..
Our derived decentralized version of Federated Dropout involves generating random indices (Equation 7 at the aggregator, guided by the largest available model (W,). Subsequently, each model is assigned random set of indices equivalent to its size. The aggregation process then carried out using these randomly selected
19

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 3 Decentralized Federated Dropout
HeteroFL, and FedRolex
Input: Initialize N clients, cach client has model W,, and data Dn, Each model has rate Tuy which is the model's size rate compared to the largest model in the network. Models with the samne rate have the same initialization.
for communication round t = 2, ...,'' I' do Randomly select one aggregator a E fl.... N) Randomly select senders S c {1,... N), a a Participants U
\\ Client Side for all P do for all batch X, E local data D, do
W < locally train Wn using Equation
Send locally updated models W, for all E S to 11 Aggregator Side
Set W, to be like the largest available model
Local models are assigned indices A for all i and n E P, where Xx is from Equation or Equation g
Equation 9 for all n E P do
Wn < W... for all \\ Split Wg into local models
Send back updated models W, for all e
indices. Once the aggregation is finalized, sub-models are created from Wg using the same set of indices. Finally, these sub-models are transmitted back the respective participating clients.
Decentralized HeteroFL. Unlike Federated Dropout, HeteroFL (Diao et al. 2020 consistently extracts sub- models from a predefined sect ion of Wg- Specifically, HeteroFL. ext excts sub-models from W, starting from index up to the maximum layer size W,, The extraction scheme is defined as follows:
Xn,s (0, (0. - 1}.
(8)
In the decentralized HetcroFL approach, the parameters of cach layer all sub-models share the same starting point (index 0). Consequently, al the aggregator, parameter averaging takes place with overlapping indices from the available models. After aggregation, the updated parameters of each layer from all models, spanning from index up to the maximum size, are communicated back to their respective clients.
Decentralized FedRolex In FedRolex (Alam et al. 2022). local clients are initially generated from the global model beginning from index and extending up to the local models' capacity. In the first communication round, the sub-models generated similarly to HetcroFL. However, in cach subsequent communication round, the starting point of the indices shifts to the right. The sub-model extraction in FedRolex is defined as follows:
if [rnJk] s J., X (f.++1..... 111 1 1,.,,, [naJal - J,} otherwise,
(1,8+1,-., 1+ [nnck] 1)
(9)
where 8=t mod Jr. t is the current communication round and J; is the size of layer of W
In decentralized FedRolex, the size of W, is determined by the largest available model, and as a result, the rightward shift in indices computed based on the current communication round and J/ of the selected W,- The indices are calculated using X from Equation g These indices are utilized for aggregation and extract the updated local models after aggregation. In decentralized FodRolex, since aggregators must be aware of & to compute the indices, each aggregator needs to communicate i to the next aggregator.
20

---

Published in Transactions on Machine Learning Research (08/2024)
HO, CNN, CIFAR-10O, BD, N:50, 15|/25
HO, CNN. CIFAR-100, non-IID, N:50, [5]:25
55
55
Global Accuracy (99)
45
Global Accuracy (86)
45
25
25
Cen, FedAvg
Cen. FedAvg
15
Dec. FedAvg
15
Dec. FedAvg
DFML
DFML
500
1000
1500
2000
500
1000
1500
2000
Communication Round
Communication Round
Figure 12: Performance comparison between centralized (with server) and decentralized (serverless) settings.
com
A.3.4 Decentralized FML
FML (Shen et al. 2020) is centralized framework that relies on central server. In FML, each client owns a local model, which is not transmitted during training. In each communication round, the global model forks its model into all participating clients. Shen et al. 2020 named the forked models: meme models. Subsequently, at cach client, the local (hetcrogencous) model engages in mutual learning with the meme (homogeneous) model using their respective local data. After mutual learning is complete, the meme models are communicated back to the server for aggregation.
In decentralized FML, two models are dedicated to cach client: the first is the heterogencous model, and the second is the homogeneous (meme) model. In each communication round, several clients perform mutual learning between their heterogeneous and homogeneous models using their local data. Next, the homogeneous models from all participating clients are transmitted to the aggregator for aggregation. After aggregation is complete, the aggregated model transmitted back to all participating clients. This process repeats for the remaining communication rounds.
We decentralized FML for comparison with our proposed DFML. It is crucial to emphasize that de- centralized FML and our proposed DFML are two distinct frameworks. The key differences are as follows: 1) DFMT. uses one heterogeneous model per client for training, while decentralized FMI. uses two models per client for training; 2) DFML aims to transfer global knowledge to hetcrogencous models, whereas FML treats the heterogeneous models as personalized models and the homogeneous models to hold the global knowledge; and 3) FML requires a server, and we derived the decentralized version of FML facilitate comparison with our proposed DFML.
A.4 DFML: Further Analysis
A.4.1 Centralized vs Decentralized
We highlight a challenge of a serverless setting compared to having a centralized server in the network. For simplicity, we illustrate the difference using FedAvg. In centralized FedAvg, a global model exists at the server, and each communication round, the global model is distributed to randomly selected clients. Even when partial client participation, there is always one version of the global model at the server. In a decentralized setting with partial participation, clients send their model to another client for aggregation and receive back the aggregated model, resulting in a version of the global model that differs from the previous round especially with different clients selected for participation. The existence of multiple versions of global models among clients in a serverless network affects the convergence speed compared to a centralized server where the latest version of the global model is transmitted to clients in every round. Figure [2 illustrates the convergence speed drop between a centralized and docentralized network using FedAvg and homogencous architectures. Additionally, our proposed DFML challenges the centralized FedAvg. As mentioned in Section there are other advantages of serverless network compared to having centralized server.
21

---

Published in Transactions on Machine Learning Research (08/2024)
#2, CNN, CIFAR-10O, BD, N:50, 15|:25
H2, CNN. CIFAR-100, non-lID, N:50, [5]:25
55
55
50
50
Global Accuracy (%9
40
Global Accuracy (86)
40
DFML- Random Aggregator
DFML Fixed Aggregator
DFML Random Aggregator
DFML- Fixed Aggregator
500
1000
1500
2000
500
1000
1500
2000
Communication Round
Communication Round
Figure 13: Performance comparison between using the same client (client 0) and randomly selecting client in each communication round to perform the aggregation.
H2, CNN, CIFAR 100, HID, N:50, |S|:25
/2. CNN, CIFAR 100, non IID, N:50, [5]:25
50
40
Gobbll Accuracy 1%)
Global Global Accuracy 1%)
40
30
30
20
DFML
DFML
10
DFML without KLD
DFML without KLD
10
10a
10a
10
10a
10a
10a
10
10c
Communication Cost (MB)
Communication Cost (MB)
Figure 14: Performance comparison between using both distillation and supervision signals and only using the supervision signal at the aggregator to perform knowledge transfer.
A.4.2 Fixed vs Random Aggregator
Figure 13 shows that using fixed client to perform the aggregation significantly limits performance since the same local data is used in the knowledge transfer between models. In this experiment, the senders are randomly selected in each communication round. while the aggregator is always client 0.
A.4.3 Effect of Knowledge Distillation
We explore the effect of utilizing a distillation signal in the objective function compared to just having a supervision signal at the aggregator (Equation Figure 14 illustrates the communication cost difference between having both supervision and distillation signals the aggregator's objective function versus just having the supervision signal. The figure shows that without the distillation signal, the computation cost is approximately order of magnitude higher than with the addition of the distillation signal to the supervision signal at the aggregator, as described in Equation 1
A.4.4 Convergence Speedup
We illustrate in Figures under three heterogencity settings: homogeneous architectures, restrictive heterogencous architectures, and nonrestrictive heterogeneous architectures; respectively.
15
16 and 17 the convergence speedup achieved by DFML compared to the baselines
A.4.5 Local Accuracy
Our proposed DFML not only surpasses the baselines in global accuracy but also achieves competitive re- sults in local accuracy. As shown in Figure 18 the local accuracy attained by DFML generally exceeds
22

---

Published in Transactions on Machine Learning Research (08/2024)
CNN, CIFAR-15, 10, M50
CNN, OFAR-100, II., M5
CFA5-10, moo1D.
CANL OIFR-116, rovilD,
80-
Dosat Aoc.urey
Goos
Fedlegg
500
1300
1520
2000
500
2003
1530
2005
500
1020
1500
2010
500
1000
1500
2302
Commuricationoooo Sound
Commun outin Round
Communibatton Sound
Commun cation Round
Figure 15: Comparison between DFML, decentralized Fed,vg and Def-KT using homogeneous architectures.
N1, 11 M500, 19125
CIAM-100, N/50,
HL. CAN, CIFA-10 not-40, (S50,
HL. CMN, C C-RA--100 no5-1
Oloba Acconiiy (30
Chow Accurrry
60
60
Doc Fedlug Hentroll
Dec: Replirg
Dec. FedMag
Relleg
20
Dec HeteroF
-00
20
Herenf
Dec Pecffclix
0
500
1000
15:0
2000
500
1003
1920
2000
500
10000
1500
2000
500
1000
1500
2005
Communicaaionnnoo Round]
Commaniation Round
Communicatoos found
Commun Calin Round
Figure 16: Comparison between DFML, decentralized partial training algorithms, and decentralized FedAvg
using restrictive heterogeneous architectures.
X2. CKN, CPA4-10 ID, 1815
H2. CMBL CIFA-113, S0RED, N110. 1915
12,CMN, CFAA-100, CE..11, I1,, A10 1915
H0, CNN, OIFA-150, roe-It,, 6.10,
72
Clasel AeT.UNc
60-
10
Clonal
12
50g
1000
1500
2000
300
1000
15:0
2000
500
1000
1500
2000
300
1000
1500
2002
Commusication
Commmnisazon Round
Communication
N2, CMN,, CIFAR-10, (1), W150, 1925
J2.CW.C.RRB-1. NeHilO, A150 19/25
19:25
12, 7MN, CITAR-105, ID, (M5A, 19125
#2. CWW. CIFAR-100, m01-1, W/55, (9125
-2
Cotea Aecurarry [16)
22
15
Gidb
22
530
1050
35.0
2000
3003
1530
2000
3020
2000
1.000
1500
2003
H2: CVL, OPAN-40, II.. A0300, 18:50
h2. CMb, CP48 non-8D, R:100. 19:30
15
H2. CWA.CPAA-10....... ID, MA10, 51550
H2. CMI, CPAN-100, N0100. (3):0
150
0:dba Aoc. merr 152
Techon
Dec. Teclvs
FedFeg
DTNL
DENL
1300
2003
2000
1000
1500
2002
Figure 17: Comparison between DFML and docentralized FedAvg for different numbers of clients and data distributions using non-restrictive heterogeneous architectures.
that of decentralized FodAvg. Although DFML exhibits slightly lower local accuracy compared to decen- tralized FedAvg in the CIFAR-10 non-IID experiment with 10 clients, it remains competitive. Moreover, the corresponding global accuracy achieved by DFMT. in that experiment surpasses decentralized FedAvg.
23

---

Published in Transactions on Machine Learning Research (08/2024)
70
Local Accroo (00
70
Arcuenry (54)
Feclus
FedWwr)
Feclus
Fedfag
DENL
500
1O00
15:0
2000
500
2003
1539
2000
500
1000
1500
2010
500
1000
1506
2302
Commutication Round
Commun oaton Rouns
Comminication Sound
Commun cation Round
CIRA--10, M550,
C.FR-15,, nonil,, 650.
40, CMS, CIFAR-165,
H2,CMW, C CIFAR-10O, no1-8,
(00)
L00 Recoursry (16)
15
Lacal
55
Fuckin
5.
Poldeg
Fookys
PodAag
500
1300
35:00
2000
100
1003
1500
2000
500
3000
3500
2000
500
1000
1500
2005
Communicaton Round
Commanaation Round
Communicatoo Sound
Commun Calion Round
DVL OPAA-42, (12, A1200.
82. CMI,, CI140-10. 42, A1120. 13:30
OVV CIAN-100. ID, MACo, 51932
N2. 2M., CPAN-ADD, S4HHD. N:002
15
Local Accuocy 176)
-5
Fullin
Polleg
12
Fooky
Fodlag
0
530
1000
35:00
2000
100
2003
1500
2000
500
3000
1500
2000
500
1000
1500
2005
Commutication
Commun catton Round
Communicatton
Cammun calion Bound
Figure 18: Local accuracy comparison between DFML and decentralized FedAvg for different numbers of clients and data distributions using nestrestrietive heterogeneous architectures.
A.4.6 Different Number of Participants
We investigate the performance of DFML with varying number of senders S in each commuication round. Figure 19 compares DFML. and decentralized FedAvg with 50%, 20%, and 10% senders. DFML exhibits effective learning even with fewer participants compared to docentralized FedAvg. Additionally, both methods with a reduced number of participants demonstrate slower convergence speed compared to the 50% scenario, which is expected.
Due to the low participation rate, increase the number of peak updates. With limited number of participating models, updating them only when the maximum is reached results in slower convergence speeds. Therefore, we allow multiple updates instead of updating the models solely at the maximum a. We estimate that an appropriate number of peak updates is [ST
N for |S] < 50% the clients. Consequently, with |S| 10%, updates are applied in the largest five values. If cyclical is not added to DFML, adjusting the number of peak updates is unnecessary, as peak models will no longer be needed and only the updated models must be communicated back to senders.
A.4.7 Weighted vs Vanilla Average of KL Divergences (KLs)
In Equation we use a weighted average of KL divergences (KLs) ween all leacher models and the student. The weighting is determined based on the number of trainable parameters in each teacher model. Figure 20 demonstrates that the weighted average leads to better convergence speed and global accuracy compared vanilla averaging. The reason is that larger models tend have a higher probability of possessing finer knowledge, thus giving them more weight during knowledge distillation results in better knowledge transfer.
24

---

Published in Transactions on Machine Learning Research (08/2024)
CNN,
//2. CNN. CIFAR-100, non-lID, N:50, 18:25
f1, CNN, CIFAR-100, non-IID, N:50, [s:22
50
40
40
Global Accuracy
40
Global Accurccy
30
DFML (Vanilla Learning!
DFML (Mutual Learning)
20
20
20
10
Dec. FodAvg
151:50%
151:20%
DFML- Weighted Average
DFML- Vanilla Average KLS
10
DFML
15::00%
20
500
500
1000
1500
2000
Communication Round
Communication Round
Figure 19:
Evaluating DFML.
Figure 20: Weighted vs vanilla av- erage of KL divergences (KLs) in the distillation signal of the objec- Live function.
Figure Mutual vs vanilla knowledge transfer.
against decentralized FedAvg with fewer number of senders S.
H2, CNN. CIFAR-100, non-IID, N:50,
H2, CNN. CIFAR-10D, non-lID, N:50, [s]:25
50
50
Global G Accuracy (%)
40
Global Accuracy (%5
40
30
Dec, FedAug
20
FedAvg
DFML K:
DFML-
DFML K:
DFML K:
10
DFML K: 10
10
DFML K: 10
DF ML Ki 20
DFML 20
500
1000
1500
2000
10
1012
10
10t4
1015
Communication Round
Computation cost (FLOPs)
Figure 22: Effect of increasing mutual learning epochs K on global accuracy with respect to (Left) commu- nication rounds and (Right) computation cost.
A.4.8 Mutual vs Vanilla Knowledge Transfer
With vanilla knowledge transfer, only the aggregator's model is updated, with the other models act teachers. Conversely, with mutual learning, all models updated. Figure 21 demonstrates the significant improvement in global accuracy and convergence speed achieved through mutual learning compared to vanilla knowledge transfer.
A.4.9 Effect of Increasing Mutual Learning Epochs K
Figure 22 shows that increasing the number of mutual learning epochs K at each aggregator contributes to faster convergence speed. A higher number of K enables more knowledge to be distilled from the experts, leading to improved convergence speed. This, in turn, results in a more efficient communication cost throughout training. On the other hand, the computational cost will increase significantly. From the figure we can see that incrcasing the K boyond 10 does not result in significant enhancement in convergence speed at the expense adding more communications between clients. Further, selecting K lower than 10 will save computation but will increase the communication cost. Thus, there is trade-off between computation and communication cost for convergence. Last, Even though decentralize FedAvg is more computation efficient than DFML, but it does not reach to the same global accuracy level DFML.
A.4.10 Different Topology
In all previous experiments, we used a mesh topology, where all clients can reach each other. However, this subsection, we explore a different network topology, illustrated in Figure 23 In this new topology, clients are divided into two groups, each connected in mesh configuration, with single link connecting
25

---

Published in Transactions on Machine Learning Research (08/2024)
P:
Wi
FFs
ClantD
Wy
Clantz
Clesni
Ciant?
Dy
Z
W
Figure 23: Different. Topology.
Table 11: Global accuracy comparison using restrictive heterogencous architectures with 50 clients and 25 senders. The experiments are conducted using ResNet architectures using the topology illustrated in Figure
CIFAR-10
CIFAR-100
Method
IID
non-IID
IID
non-III>
Dec. FodAvg
44.71 0
22.11 9.00
10.36 0.38
7.51 0.14
Dec. HeteroFL
58.91 = u37
43.45 0m2
18.2 0.38
14.81 = u22
Dec. FedRolex
58.91 0.00
43.54 0.10
18.20 6.08
15.10 0.11
DFML (Ours)
78.87
68.52
45.02
40.31
Table 12: Global accuracy comparison using nonrestrictive hetcrogencous architoctures with 50 clients and 25 senders. The experiments are conducted using CNN architectures using the topology illustrated in Figure
23
CIFAR-10
CIFAR-100
Method
IID
non-IID
IID
non-IID
Dec. FedAvg
51.45 oar
25.03
27.10 eu
20. oa
DFML (Ours)
78.87
68.52
45.01
40.30
the two groups. The first group contains clients with odd addresses (1.3,5,.... N 1] and the second group contains clients with even addresses [2, 2,4,6,... 6,...,,1 .J. The clients connecting group 1 and group 2 have the "median" address from the list of addresses in its group in their respective groups.
We conducted experiments using both restrictive and nonrestrictive heterogeneous architectures. The results, provided in Tables 11 and 12 demonstrates that even with this different topology, DFML continues to surpass the baselines.
A.4.11 DFML vs Decentralized FML
In decentralized FML, the clients' heterogeneous models are the same as the models used in our proposed DFML, which based on CNN architectures and mode H2 from Table [0] The homogeneous model size used is [32, 64], which is the median of models of the five different architectures. Figure 24 compares our DFML and docentralized FML. We include the global accuracy of the local hetcrogencous and the homogeneous meme models. We observe that the homogeneous meme models have a better convergence speed than our DFML, and some cases, lead to better final accuracy. However, the global knowledge performance of heterogencous models in decentralized FMI. much worse than DFML and is even lower than decentralized FedAvg. As our goal to transfer global knowledge clients' heterogeneous models, we consider that DFML significantly outperforms the decentralized FML framework.
26

---

Published in Transactions on Machine Learning Research (08/2024)
70-
Gooul Accuraor (5)
Foskeg
Dosst www.ororryyy
40
15
ruacal
Dec. PodAeg
Dec Fooles
Goodl
Dec. Pedfag
Onc 3.ccs
DIM..
FHI Boocl ModeeO
Dect FMt (lacal Mtdees
Dec: (Neme Model)
FNI (Weme Modee)
Dec: FM, (Heme Modelo
500
1OC0
1520
2000
500
2003
1830
2005
500
1000
1500
2010
500
1000
1500
2309
Commuricationoo.o Sound
Commun oatio Roure
Communicatios Sound
Commun cation Round
Figure 24: Comparison between our proposed DFML and decentralized FML.
A2, CNN, CIAR--100, non-HD, N:50, 15):25
H2. CNN, CIFAR-100, non-HID, N:50, |S 15:25
H2. CNN. CIFAR-10O, non ID. N:50, (9)25
50
45
40
40
30-
Global Accurscy
30
20
20
(G + 3aal
20
Global
DFML d.0.5
DFMI- (yCos
DFML (Coy + ade))
10
Dec. FedAvg
ACE
DFML Cyclical
DFML AM- KD
10
DFML- ace)
DFML- Cyl Gal
DFML
0
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Communication Round
Cammunication Round
Figure 25: Comparison between different supervision signals.
Comparing various adaptive techniques with our pro- posod DFML. The supervision sig- nal used is CE.
Figure 26:
Figure 27: Different schedulers as- signed to the loss components in the objective function.
A.5 Cyclical Knowledge Distillation: Further Analysis
A.5.1 Different Supervision Signals
As previously mentioned, mitigating catastrophic forgetting is crucial in applications such as FL where data distribution shift cxists across clients. Training models on one dataset can lead to optimizing its objective towards that specific local data, causing it forget tasks learned previously from other clients. ACE (Caccia et al. 2021) and WSM (Legate et al., 2023) are two approaches to mitigate catastrophic forgetting. Figure 25 illustrates the improvement in global accuracy when the supervision signal is changed from CE ACE supervision signal leads to the highest global accuracy, as it takes into account the number of samples for each class label in the clients. The maximum range of a oscillation is tuned for each supervision signal to achieve the best final accuracy. Tuning the maximum a range is important, particularly in scenarios where the supervisory signal is non-noisy, such as in IID distribution shift, or when using ACE or WSM in non-lID settings. Completely diminishing the supervisory signal (equivalent to setting a = 1) such cases would lead to a performance decline. Therefore, in situation where the supervision signal is not noisy, the maximum value is better be set 0.8 or 0.9. For instance, in non-IID cases with CE the supervision signal, where the signal is very noisy, setting yields the best performance.
WSM. Results show that WSM
A.5.2 Different Adaptive Techniques
In Figure 26 we compare the performance of different adaptive techniques, including WSL (Zhou et al. 2021) and ANL-KD (Clark et al. 2019), with our proposed cyclical DFML framework. The results indicate that WSL and ANL.KD show negligible improvement compared to DFML with fixed a = 0.5.
27

---

Published in Transactions on Machine Learning Research (08/2024)
H2, CNN, CIFAR-100, non-lID, N:50, |s|:25
45
Global Accuracy (%)
40
30
DFML (CE) Fixed Period
DFML (CE) Incremental Period
500
1000
1500
2000
Communication Round
Figure 28: Comparison between fixing and increasing the period throughout training. sct to 10.
The initial period is
A.5.3 Different Schedulers for Loss Components
In Equation ! the objective function consists of two loss components: supervision and distillation loss signals. In Equation [0] we cxamine the impact of independently varying each loss component global accuracy. Specifically, we compare additional scenarios where one scheduler is assigned to Log alone, another scenario where scheduler is assigned to LKL. alone, and a third scenario where one scheduler scales both loss components with the same factor. For this experiment, we use CE as the supervision signal, as the improvements are more notable under the non-IID distribution shifts. Figure 27 illustrates the comparison of assigning independent schedulers to loss components. In this experiment, 7 oscillates from 1 + 0, and a oscillates from 0 1.
C ==>Le + aLKI
(10)
Scaling the distillation signal alone and leaving the scale of Lce untouched does not yield any advancement in performance compared to keeping fixed for both loss components. This indicates that the supervision signal has a more significant impact than the distillation signal. Performance gains are observed when the Cee signal is reduced, allowing more influence on the CKL. signal. In contrast, when Le oscillates while the LKL scale is kept fixed, it results in the same performance as when both LCE and LKL signals are scaled in opposite directions (Equation [ This is because the Lee signal is dominant without any scaling, and it diminishes it allows the LKI. signal signal reaches 0, and the LKL signal's scale is 1. Finally, scaling both Lee and LKL signals with common scheduler leads to inferior performance compared to scaling LCE alone or scaling both signals in opposite directions. The poor performance of oscillating LKL signal alone is attributed to the continuous dominance of LCE signal during mutual learning, cansing the models to drift toward the aggregator's local data.
take precedence. The peak value is attained when the LCE
A.5.4 Fixed vs Increasing Period
Figure 28 demonstrates that increasing the period over lime results in better convergence speed, higher global accuracy, and enhanced stability. The period is initially set to 10. In the fixed period experiment, the period is kept constant, while in the increasing period experiment, the period is incremented. Starting with a small period is crucial for more frequent peak updates, which accelcrates convergence speed. However, over time, increasing the period proves beneficial, allowing models to transition from the supervision to the distillation signal more slowly. This extended time in the distillation-dominae instance, if all clients are participating and the period is initially set at 100, better accuracy achieved after 100 communication rounds compared to constant period of 10. However, the convergence speed is notably affected, as a period of 100, results in one peak at communication round 100, while a period of 10 leads to 10 peaks. Further, in scenarios with partial participation, then after 100 rounds only the participating clients will be updated. Whereas a smaller initial period ensures that, on average, all clients are updated several times within the first 100 rounds. Therefore, to reap the benefits of high convergence speed and improved final accuracy, we set the period to be initially small and increment it gradually.
region enhances global accuracy. For
28


Reviews:
[{"review_id": "KRbqZzF0TR", "reviewer": "Reviewer_TnFy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- DFML supports nonrestrictive heterogeneous models and does not rely on additional data, making it more practical for real-world scenarios.\n- Extensive experiments demonstrate that DFML consistently outperforms state-of-the-art baselines in terms of convergence speed and global accuracy under various conditions.\n\nWeaknesses\n- The impact of different network topologies on the performance of DFML is not investigated. The star topology is symmetric and has good stability. I think you should evaluate DFML on more Irregular topologies.\n- The experiments do not cover a wide range of real-world datasets and applications, which could help validate the generalizability of the proposed framework.\n- The impact of the number of mutual learning epochs K on the performance of DFML is not thoroughly investigated. The paper mentions that increasing K contributes to faster convergence but does not provide a detailed analysis of the trade-off between computational cost and performance improvement.\n- Although the framework does not rely on additional data, the privacy risks associated with sharing model parameters and the potential for privacy leaks during the mutual learning process are not addressed. Nowadays, there exsit some powful model inversion attack methods that may cause privacy leakage.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel Decentralized Federated Mutual Learning (DFML) framework that is serverless, supports heterogeneous models without architectural constraints, and does not rely on additional data. DFML handles model and data heterogeneity through mutual learning that distills knowledge between clients, and by cyclically varying the amount of supervision and distillation signals. Extensive experiments demonstrate DFML consistently outperforms prevalent baselines in convergence speed and global accuracy under various conditions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I think this paper is well-written and has significant reasearch value. However, there are some issues to be addressed.\n\n- (major) Investigate the impact of different network topologies on the performance of DFML. \n- (major) Conduct an ablation study to evaluate the individual contributions of the key components of DFML, such as mutual learning, cyclical variation of α, and the use of peak models. \n- (minor) Expand the experiments to cover a wider range of real-world datasets and applications.\n- (minor) Include a theoretical analysis of the convergence properties and limitations of DFML.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CcbG4o3ZFl", "reviewer": "Reviewer_Pqap", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nHandling heterogeneity in Decentralized Federated Learning setting is well-motivated.\n\nThe experiments are extensive and easy to follow.\n\nThe results show the proposed method is effective.\n\nWeaknesses:\n\nAlthough the communication cost is not impacted, the computation cost might be high for the selected aggregator (client). \n\nIf a less powerful edge device is selected as the aggregator and it cannot run the model sent from other clients, how can the method still work? For example, if an edge device can only deploy a CNN model, how can it aggregate a ViT model sent from other clients? \n\nWill the learned model bias toward the data on the last aggregator at the end of training?\n\nIn the global accuracy comparison using homogeneous CNN architectures, can the authors compare DFML with a decentralized version of an FL method designed for data heterogeneity?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Decentralized Federated Mutual Learning (DFML) to address heterogeneity in a Decentralized Federated Learning (DFL) setting. Specifically, DFML transmits models to a randomly selected aggregator to handle model heterogeneity and uses re-weighted SoftMax to address data heterogeneity. The aggregation is performed by distilling knowledge on the aggregator client. A cyclical approach is used to balance two loss functions. The experimental results show the proposed method is effective in addressing the non-IID challenge.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.\n\nPlease consider using \\citep{} if the authors are not part of the sentence.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q7D8EeRRRl", "reviewer": "Reviewer_LN2c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths ]\n\n1 The studied problem is promising.\n\n2 Introducing accuracy on IID data is convincing.\n\n3 Multiple architectures are leveraged for evaluation.\n\n\n[Weaknesses]\n\n1 The claimed \"no server\"  (or serverless) is confusing and requires careful clarification. Specifically, it seems like changing the server in FedAvg to the \"no server\" setting claimed in this work induces no challenge. Namely, we can perform aggregation on any client, as claimed in this work.\n\n2 The claimed architectural constraints are widely studied in one-shot FL, where limited or no constraints are leveraged.\n\n3 The proposed cyclic knowledge distillation would introduce numerous costs as the FL system is scaling up, e.g., with more data or clients.\n\n4 The experimental results are not convincing, as merely CIFAR10 and CIFAR100 are considered in the evaluation.\n\n5 The proposed method would approach the centralized training setting and performance if the proposed learning paradigm does not fix the client for knowledge distillation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims at serverless and relaxed-constraint FL. To this end, the authors propose a novel learning framework in which models are uploaded to certain clients and fine-tuned using their data. Some experiments are conducted to verify the effectiveness of the proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1 The motivation/significance of the highlighted contributions should be carefully clarified, see Weaknesses 1, 2, and 5.\n\n2 More experiments are required.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a new idea for handling heterogeneity in decentralized federated learning systems and proposes a new method of knowledge distillation for clients. 

Before rebuttal, reviewers raised several concerns regarding the extensive computational demands of the method, potential biases inherent in the model, the need for more rigorous ablation studies, and a perceived lack of sufficient empirical evidence supporting the method's efficacy. After rebuttal, reviewers acknowledged that major concerns have been addressed.

The limitation of this paper acknowledged by most reviewers is its demand on computational cost, which should not be a major concern for rejecting this paper. The authors have also clearly acknowledged these limitations in their manuscript.
Decision: Accept (poster)