Homogenizing Non-IID Datasets via In-Distribution Knowledge Distillation for Decentralized Learning
Authors: Deepak Ravikumar
Gobinda Saha
Sai Aparna Aketi
Kaushik Roy
Venue: Accepted by TMLR ()

Abstract: Decentralized learning enables serverless training of deep neural networks (DNNs) in a distributed manner on multiple nodes. One of the key challenges with decentralized learning is heterogeneity in the data distribution across the nodes. Data heterogeneity results in slow and unstable global convergence and therefore poor generalization performance. In this paper, we propose In-Distribution Knowledge Distillation (IDKD) to address the challenge of heterogeneous data distribution. The goal of IDKD is to homogenize the data distribution across the nodes. While such data homogenization can be achieved by exchanging data among the nodes sacrificing privacy, IDKD achieves the same objective using a common public dataset across nodes without breaking the privacy constraint. This public dataset is different from the training dataset and is used to distill the knowledge from each node and communicate it to its neighbors through the generated labels. With traditional knowledge distillation, the generalization of the distilled model is reduced due to misalignment between the private and public data distribution. Thus, we introduce an Out-of-Distribution (OoD) detector at each node to label a subset of the public dataset that maps close to the local training data distribution. Our experiments on multiple image classification datasets and graph topologies show that the proposed IDKD scheme is more effective than traditional knowledge distillation and achieves state-of-the-art generalization performance on heterogeneously distributed data with minimal communication overhead.

Published in Transactions on Machine Learning Research (07/2024)
Homogenizing Non-IID Datasets via In-Distribution Knowl- edge Distillation for Decentralized Learning
Deepak Ravikumar
dravikum dravskum@purdue.cmm edu
Elmore Family School of Electrical and Computer Engineering, Purdue Universily, West Lafayelte, 47907, USA
Gobinda Saha
gsaha Daprduee.. edu
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA
Sai Aparna Aketi
saketi@purdue. edu
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA
Kaushik Roy
kaushik Buurduc edu
Elmore Family School of Electrical and Computer Engineering, Purdue Universily, West Lafayelle, IN 47907, USA
Reviewed on OpenReview: https: // openreview. net/ forum? id= Cuy IVa
Abstract
Decentralized learning enables server-less training of deep neural networks (DNNs) in a dis- tributed manner on multiple nodes. One of the key challenges with decentralized learning heterogeneity in the data distribution across the nodes. Data heterogeneity results in slow and unstable global convergence and therefore poor generalization performance. In this pa- per, we propose In-Distribution Knowledge Distillation (IDKD) to address the challenge of heterogencous data distribution. The goal of IDKD is to homogenize the data distribution across the nodes. While such data homogenization can be achieved by exchanging data among the nodes sacrificing privacy, IDKD achieves the same objective using a common public dataset across nodes without breaking the privacy constraint. This public dataset different from the training dataset and is used to distill the knowledge from each node and communicate to its neighbors through generated labels. With traditional knowledge distillation, the generalization of the distilled model is reduced due misalignment between (OoD)
the private and public data distribution. Thus, we introduce an delector at each node to label a subset of the public datasel that maps close to the local training data distribution. Our experiments on multiple image classification datasets and graph topologies show that the proposed IDKD scheme more effective than traditional knowledge distillation and achieves state-of- the-art generalization performance on heteroge- neously distributed data with minimal communication overhead
1 Introduction
There has been an explosion of Internet of Things (IoTT) devices and smartphones around the world (Lim) al. 2020 These edge devices are being equipped with advanced sensors, computing, and communication sapabilities The variely and the wealth of data collected by these devices is opening new possibilities in
medical applications (Pryss et al. 2015). air quality sensing (aant et al. 2011), and more. As more data
"ode available

---

Published in Transactions on Machine Learning Research (07/2024)
(a) Pre-IDKD
(b) Public Data Latent Space
(c) Exchange High Conf. Subset, More Homogeneous Data
(d) Post IDKD, more data, Better boundaries
Node I Private
Node 2 Private
Public Set
Low Confidence
Public
Decition Boundary Low Cont)
Node
Exchange High Confidence Labels
Node 2
Deciun Brwndary fLow Cord)
Figure A conceptual overview of IDKD homogenization. We visualize an exaggerated latent space of 2-node decentralized setup with non-IID data having 4 classes. (a) Two similar networks with small differences in decision boundaries due to data skew. (b) Public data visualized in the latent space (c) Exchange IDKD identified subset using labels only (d) More homogeneous data distribution resulting in better decision boundaries post IDKD training.
is collected at the edge there is a need to not only learn at the edge for efficiency reasons but also maintain privacy (Lim et al.
2020: Wang et al. 2020
Decentralized deep learning aims address both these issues by distributing the training process among many compute nodes (Kempe et al. 2003 Tsitsikiis 1984p. Each node works on subset of the data allowing the use of large models and huge datasets. Those local datasets are called private sets they not shared with the other nodes. During training, each node communicates gradients model parameters with its neighbors. The choice of information (gradient, model parameter, etc.) to communicate is dependent on the algorithm used, In general, each node aims to reach a global consensus model using its own local model and updates from its neighbors.
Training DNNs a decentralized setup gives rise a number of challenges. The main challenges come manner across
from the slow spread of information and when data is distributed in a heterogeneous/nong
the nodes (Lin et al.
2021). This results in slow and unstable global convergence and poor generalization
performance [Lin et al. 2021 To address these issues many meth (Dandi et al. 2022 Duchi et al. 2011
Lian et al.) 2017 2018 Lin et al. 2021 Neglia et al. 2020 Tang et al. 2018 Vogels et al. 2021) 1b have been
proposed. Techniques have been proposed to improve the properties of the mixing matrix (Assran et al.
2019 Duchi et al. 2011 Nedić et al. 2018 Neglia et al.
to address the slow spread of information, while most. other approaches have primarily focused on the design of decentralized optimization algorithm
2020
to handle heterogeneous data distribution (Dandi al. 2022 Lian al. 2017 2018 Lin et al. 2021 Tang
et al. 2018 Vogels et al.) 2021). In contrast previous techniques,
aim to homogenize heterogeneous
data distribution without sharing data among nodes to leverage the full potential of existing decentralized algorithms. To achieve this we leverage distillation (Hinton et al. 2015 Distillation has been used to address non-IID distribution in federated learning (FL). However, FL techniques such as DS-FL (Taahara
et al. 2021), FedGen (Zhu et al.) 2021b), and FedDF (Lin et al. 2020 rely on a central server and thus cannot be readily adopted in a decentralized setting. In Def-KTT (Li al. 2021 the authors proposed a peer- peer knowledge transfer technique however, their algorithm implicitly assumes complete graph connectivity and thus cannot be applied in all decentralized setups. As far as know work is the only work that
2

---

Published in Transactions on Machine Learning Research (07/2024)
addresses non-IID distribution from the data perspective in a decentralized setting. Table summarizes previous approaches addressing non-IID data distribution in centralized and decentralized learning.
Technique
Centralized Decentralized
Knowledge Distillation
FedGen Zhu et al.
2021b
DS-FL
Itahara et al.
2021
FedDF Lin et al.
2020
Def-KT
Li et al.
2021
QG-DSGDm-N N
Lin et al.
2021
DSGD Lian et al. 2017
D2
Tang et al.
2018
QG-IDKD (Ours)
Table 1: Prior works have proposed distillation to address non-IID data distribution in federated learning, however as far we know the proposed work is the only approach in distributed decentralized learning leveraging distillation.
In this paper, propose In-Distribution Knowledge Distillation (IDKD), new technique to improve decentralized training over heterogeneous data to obtain global solution (i.e. same model across the nodes). IDKD aims to distill heterogencous datasets to make them more IID. Distillation (Hinton et al., 2015) a process in which a teacher network transfers knowledge to a student network. This is often done by having the teacher network label a corpus of unlabelled public data (this is separate from the training set). The student network trained to match the teacher labels on this corpus of data. In our problem setting, the teacher networks are other nodes in the graph and the student is the local network. By using unlabelled public dataset, distill the knowledge from each node and communicate it to its neighbors. However, with distillation, the alignment of the public and private sets is crucial for generalization performance (Hinton et al.] 2015). Thus we propose the use of an Out-of-Distribution (OoD) detector during the distillation process. This improves distillation performance and model generalization while reducing communication overhead.
A conceptual overview of IDKD visualized in Figure D We visualize toy latent space of neighboring nodes in decentralized setup with non-lID data. These two nodes will learn different decision boundaries due to data skew (see Figure D ()). Our goal is to encourage these nodes to learn similar decision boundaries that generalize better via knowledge distillation. In the traditional knowledge distillation methods, entire public data is used regardless of its alignment with the privale dataset. However, in the decentralized setting, each node generates a different set of labels for the public data due to the model/data variation. This gives rise to the issue of label conflict (sce public samples with black dotted outlines in Figure ( To address the issues of data alignment and label conflict, we propose excluding low-confidence samples and communicating the labels of high-confidence samples (shown in red arrow in Figure [c)). This label exchange results in more homogeneous data distribution, which in turn results in better decision boundaries and improved generalization performance (as seen in Figure [d).
To summarize our contributions,
We propose new distillation- based approach to handle non-IID data distribution in a decentralized setting to obtain global solution (i.e. same model across the nodes). We focus on the dataset, rather than algorithm design.
We show that public-private dataset misalignment reduces distillation performance. To address this we propose using an Out-of-Distribation (OoD) detector.
2In DefKTT the authors assume complete graph connectivity and thus equivalent centralized setup.
3

---

Published in Transactions on Machine Learning Research (07/2024)
We show that the proposed IDKD framework is superior (2 13%) to vanilla knowledge distillation in a decentralized setting and achieves up to 4 - 8% improvement over the state-of-the-art in gen- eralization performance on heterogeneously distributed data with minimal communication overhead (~> 2%).
2 Related Work
Knowledge distillation was proposed (Hinton et al. 2015 to reduce the computational complexity of large models. It was used to distill a larger teacher model into a smaller student model while retaining similar performance. However, distillation assumes that the training dataset Dr and the public (a.k.a student) dataset Dp are similar (Ahn et al. 2019 Hinton et al.) 2015 Tian et al. 2020p. Hence, any misalignment between the training set and the distillation set reduces generalization performance. To address this issue, in our work, we propose using an OoD detector to distill on subset that is aligned with the local dataset improving distillation performance.
Decentralized Learning is a branch of distributed learning that collaboratively trains a DNN model across multiple nodes holding local data, without exchanging it. The key aspect being the peer-to-peer exchange of information in the form of model parameters or gradients (no central server). Rescarchers proposod Decentralized Stochastic Gradient Descent (DSGD) (Lian et al.] 2017 2018) by combining SGD with gossip algorithms (Xiao & Boyd] 2004) enable decentralized deep learning. To improve performance researchers have since proposed versions of DSGD with local momentum (DSGDm) (Assran et al. 2019 Kong et al. 2021 Koloskova et al. 2020 Yuan et al. 2021). The above-mentioned works have demonstrated that decentralized algorithrs can perform comparably to centralized algorithms on benchmark vision datasets. However, these techniques assume the data to be Independent. and Identically Distributed (IID). Training DNN models in a decentralized fashion with non-lID data still remains a major challenge. There have been several efforts in the literature address this challenge of non IID data in a decentralized setup (Aketi et al.) 2022 Esfandiari
2021 Tang al.) 2018). Several approaches (Aketi et al. 2021 have attempted to improve non-llD performance by manipulating the local gradients at the cost of communication overhead over DSGD. The authors of D (Tang et al., 2018)) proposed a form of bias correction technique and theoretically show that it eliminates the inffuence of data heterogeneity. The authors of quasi-global momentum (QG-DSGDm-N (Lin et al. 2021) claim that their approach stabilizes training and they provide empirical evidence to show its ellectiveness.
et al. 2021 Koloskova et al., 2021 Lin et al.,
2022
Esfandiari et al. 2021 Koloskova et al.
Previously discussed methods focus on decentralized optimization algorithm design. Another orthogonal direction to improve the performance with non-IID data is to explore the field of knowledge distillation (KD) to reach a better consensus across the nodes. KD has been well explored in federated learning (FL)
setups with centralized server for non-IID data (Ttahara et al. 2021 Jeong et al. 2018 Li & Wang. 2019:
Lin et al.) 2020 Zhu et al. 2021a (). Prior works in FL Eltahara et al. 2021 Jeong et al. 2018 have leveraged KD to reduce communication. Researchers (Zhu et al.) (0211)) have proposed data-free knowledge distillation to deal with non-IID data to learn a generator and avoid using a public dataset. The authors of Federated Distillation Fusion (FedDF) (Lin et al.] 2020) propose distillation framework for federated model fusion which allows clients to have heterogeneous models/date and train the server model with less communication. However, most of the KD approaches proposed for non-lID data in FL setups leverage the central server and thus, are intractable in decentralized setup. In this paper, we explore the KD paradigm for decentralized setups. Unlike previous research, we leverage KD to approach the non-lID problem through the lens of dataset homogenization.
3 Method
The proposed In-Distribution Knowledge Distillation (IDKD) aims to homogenize non-lID data across the nodes to make full use of existing decentralized training schemes. IDKD decentralized training makes use of two datasots, a private training dataset D local to cach node and a public datasct Dp (common across nodes). Using the unlabelled public dataset Dp, we distill the knowledge from each node and communicate it to other nodes in the graph. In our problem setting, the teacher networks are the other nodes in the
4

---

Published in Transactions on Machine Learning Research (07/2024)
graph and the student is the local network. The proposed IDKD framework utilizes 5-step process:
Private Set
(i)
Private Set
Node 1 Private
Node Decision Boundary
Node 2 Private
Public Set
Low Confidence
Epoch
Public Set
Choice of Local or Decentralized Training
Epoch K, Node Convergence
Node Decision Boundary
Public Set
Public
Generate Labels
M
OoD
(iii) + (iv)
M
OoD Detector
(b) Visualizing decision boundaries of 2 nodes.
Label Exchange
OoD Detector
Public In-Dst.
Public Subset In-iitt.
Deep Neural Net
Softmax
Confidence
OoD
Identiffy an
Yess
Private Set
Private
Threshold
Private
Update f = f
Calibration
Dataset
Fine tuning epochs
Dataset
OoD Threshold Calibration
Epoch K
(a) Overview proposed IDKD framework decen- tralized training.
(c) The OoD detector used by the IDKD framework.
Figure 2: (a) Overview of the proposed IDKD framework for docentralized training. (1) Each node trains the private datasel Dj till convergence for the public dataset are generated (ii) OoD detector is calibrated (iv) OoD detector is used to extract subset dataset DjD that similar to the private dataset. (v) Soft labels corresponding to DiD are exchanged between the neighbors and the models are fine-tuned on the private and public dataset subset. (b) Visualizing decision boundaries of nodes. For improved KD we include ID-like data from the public set while excluding low-confidence conflicting examples. (c) The OoD detector used by the IDKD framework. A calibration dataset and the privale dataset are used as OoD data and ID data respectively. This used to identify the optimal threshold.
decentralized training algorithm. (ii) Next, soft labels
Initial training, (ii) Generating soft labels at each node, (iii) OoD detector calibration, (iv) In-Distribution (ID) subset generation, and (v) Label exchange and fine-tuning. The pseudocode for the IDKD framework is described by Algorithm and an overview of different steps is given in Figure Next, we provide details for each of these steps.
Initial Training Each node trained on the private dataset D till convergence using decentralized training scheme. This corresponds to Line in Algorithm The IDKD framework is general and sup- ports any user-specific decentralized training scheme. In our work, we choose QG-DSGDm-N (Lin et al. 2021 which is one of the state-of-the-art decentralized training schemes targeting non-lID data without communication overhead over DSGD.
Local Soft Label Generation After the initial training process, we forward propagate the public dataset Dp on each node gather the corresponding soft labels (shown in line in Algorithm 1 A soft label sp is vector that has probability caaaabiiiittykkellhaiaie /likelihood score for each output class.
5

---

Published in Transactions on Machine Learning Research (07/2024)
OoD Detector In vanilla distillation, the labels generated by the teacher model on the public dataset are used to train the student model. The key to effective distillation is the aligmment of the public and private datasets. Further, specific to decentralized KD, we observe that the low-confidence public data samples (seen in Figure b) as dotted samples) usually fall on different sides of the decision boundary for different nodes T.e., have conflicting labels. This property can be attributed to the differences in the decision boundary across the nodes arising from data skew. For example Figure 2 b), the sample with the red dotted outline is Class Hexagon according to Node while is Class Square according Node 2. This creates label conflicts when communicating the labels to the neighbors. Specifically, two cases occur (a) both nodes have low confidence in sample, in which case the sample should be ignored (b) one node is confident while the other is not. In this case, the high confidence label is retained which is potentially labeled from model with more samples (for that class). Hence, to address both issues (datase aset alignment and labeling conflict), we use an offooii-iibuttionn (OoD) detector to identify a subset of Dp that aligns with the private training set D and excludes low-confidence samples.
The OoD detector identifies the samples in the public set Dp that look like In-Distribution (ID) data to the local model on each node. Among different Ool detectors proposed in literature [Hendrycks & Gimpel
2017 Liu et al. 2020b; Ravikumar & Roy 2022 Ravikumar et al. 2023 Yang et al. 2023). we choose the
maximum softmax probability (MSP) detector [Hendrycks & Gimpel 2017) because of its simplicity and
low computational overhead. Figure ((c) visualizes the MSP detector. The samples with confidence greater
than a threshold & are classified as ID by the MSP detector
The hyperparameter t (threshold) is tuned based on the True Positive Rate (TPR) of the OoD detector. The ideal threshold tops is the point at which the TPR is maximized while minimizing the false positive rate (FPR) (Fawcett, 2006). To calculate TPR and FPR, we use calibration dataset De (we use the public dataset but another dataset may also be used) as OoD data and the private dataset D as ID data. Subsequently, the threshold was adjusted iteratively to maximize TPR while minimizing FPR to identify the optimal value topt- This corresponds to Line in Algorithm and is visualized in Figure cc..
Algorithm 1 IDKD framework al each node
Input: Public dataset Dp, local training dataset Dí, local validation dataset D number of nodes N, OoD calibration dataset Dc, Decentralized training algorithm A, OoD Detector OoD
Output: IDKD trained model M
1: Díy, + Dr.
2: for e E {1,2, total epochs} do
3: M + Train(AA, Dir)
4: if local convergence & e mod k = then
KDp + f
topt + Optimal(Dc, Di, OoD)
Dip {(p,s)) E DDDp max( > topt
8:
DiD =0
9:
for j E Neighbors(i) do
10:
# Only labels are sent DID - Send Send Receive( ():
11:
DND + DN U (DID)
12:
13:
end for
14: DID + Lverrgge
15: Dj, DID u Dr
16: end
17:
end for
18: return M
In-Distribution (ID) Subset Generation In this step, we use the calibrated OoD detector on each node to identify subset of the public dataset Dp that aligns with the private training set DJ. On each node, the generated soft labels from the public dataset are processed by the MSP OoD detector. The samples from the public data that are classified In-Distribution (ID) by the OoD detector are added to the local

---

Published in Transactions on Machine Learning Research (07/2024)
subset Din- The samples from the public dataset with maximum softmax probability greater than topt are classified as ID samples. The pseudocode for ID subset generation is shown in line 7 in Algorithm [
Label Exchange and Fine Tuning Once the In-Distribution subset (DiD C Dp) is identified for each node, the corresponding soft labels are communicated to the neighbors of each node. Each node now has access to the soft. labels of the In-Distribution subsets corresponding to its neighbors
the graph (Lines) The soft labels are averaged to obtain the final training labels on the distilled ID datasets (Line [4] This process of label exchange with the neighbors is repeated every k epochs and hence the information flows across the graph. Every exchange the training set is updated Dír DID U Dr- (refer line 15 in Algorithm
13..
Where Dr is the original private set at the beginning of training. This process provides desired data homogenization at each local node.
4 Experiments
4.1 Setup
In a decentralized training setup, each node has a local private dataset. This dataset not shared between the nodes. The nodes are connected in fixed or time-varying graph topology (such as a ring, chain, etc.). The aim of such sctup is to converge to a global model while only being able to communicate with immediate neighbors. Prior research (Dandi et al. 2022] has shown that the specifics of such network (i.e. network configuration) play an important role in convergence. In our work, we analyze our method on 2 different graph topologies Ring and Social Network. The ring topology is an important consideration because it is a topology where information traversal between the nodes is among the slowest. Social graphs are of interest because lets us consider networks with a lower spectral gap (i.e. better connectivity) than a ring network. Thus, making our analysis more complete. For the social network, we use the Florentine families graph (Breiger & Pattison 1986). To show the scalability of the proposed method, experiments with varying graph sizes (8 to 32 nodes). Further, to show the effect of data heterogeneity, we run experiments with the Dirichlet paramcter a set to 1.1.1 0.05. The a value of 0.1 is considered significantly helerogeneous.
present the
Datasets and models: For our experiments, we use ResNet He et al. 2016) architecture on CIFAR-10 10, CIFAR-100 (Krizhevsky et al.] 2009), and Imagenette (Howard, 2018) datasots. We choose TinyImageNet (Li al., 2015), LSUN (Yu et al. 2015), and Uniform-Noise as the distillation dalasets (public dataset). We use NesNet20 with EvoNorm Liu et al. 2020a). ie. the Batch Norm layers of ResNet20 are replaced with Evo as Batch Norm [Ioffe & Szegedy 2015) is shown to fail in decentralized setting with heterogeneous data distribution (Andreux et al.) 2020; Hsich et al. 2020). We report the test accuracy of the
/consensus model. All our experiments are conducted for three random seeds and the mean and standard deviation results are reported. For CIFAR datasets, we train the models for 300 epochs with a mini-batch size of 32 per node. When using the IDKD framework, the label exchange done epoch 240. The learning rate is decayed by 10 when the number of training epochs reaches 60% and 80% of the total number of epochs.
Non-IID data distribution: We sample from the Dirichlet distribution to partition the data (non- overlapping) across nodes in a non-lID fashion. The Dirichlet distribution is a multivariate generalization of the beta distribution. Once the dataset is partitioned, the data is never shuffled across the nodes. The IIDness of the data partition is controlled by the Dirichlet parameter a. Larger the value of a, the more IID the data partition and vice versa. When a small, it more likely that each node has data samples from only one class. The details of implementation, compute resources used and hyperparameters are provided along with the source code in the supplementary material.
4.2 Results
We evaluate the performance of the proposed IDKD methodology on various datascts and graph topologies and compare against the current state-of-the-ae decentralized learning algorithm. We chose QG-DSGDm- N [Lin et al. 2021) our primary baseline as it achieves fhate-of-theeart, performance heterogeneous data

---

Published in Transactions on Machine Learning Research (07/2024)
Ring = 16)
Ring (a m 32)
Dataset
Method
0.1
0.05
0.1
0.05
SGD-Ceutte Centralized
Goyal et al. 2017
90.94 0.03
90.85 0.17
DSGD Lim of al.
2017
86.61
1: 0.22
75.40 3:28
51.96 1: 5.36
83.29 1: 0.66
39.42 7.13
47.67 1: 7.97
CIPAR- 10
Relay-SGD Vogels et al.
2021
86.45
#: 0.15
79.11 1.17
68.15 4: 2.01
65.74 4 0.48
76.39 +: 2.68
67.11 4: 7.54
QG-DSGDm-N Lin al.
2021
88.98
0.11
82.94
2.10
73.41 + 4.12
89.31 + 0.11
82.23
1.40
72:41 + 3.45
QG-DSGDm-N + KD Li al.
2021
88.53
1. 0.52
85.34 0.77
77.18 1. 2.00
87.86
0.42
83.16
0.28
77.66
2.53
QG-IDKD (Ouns)
89.53 + 0.29
86.47 0.44
81.70 + 1.74
88.67 + 0.19
85.55
0.23
79.43 + 0.72
SGD Centrelizdd
Goyal
al.
65.96 0.15
65.62
0.51
DSGD
Lim of al.
2017
39.17 +: 0.25
54.00 0.46
48.20 + 1.07
50.61 d: 0.81
49:20
0.74
46.57 :: 0.70
CIFAR--10 100
Rolay-SGD Vogels al.
2021
61.77 + 1.01
54.11 0.68
50.15 + 0.76
57.38 + 0.79
52.75 1.35
47.73 + 1.14
QG-DSGDm-N Lin
2021
63.11
53.36 1.84
17.09 + 2.21
61.75 0.23
56,96 0.51
50.00 2.06
QG-DSGDD-N + KD Lá et al.
2021
41.80 + 0.23
48.53 1.18
42.60 + 1.54
48.46 + 1.13
47.10 2.00
41.21 2.11
QG-IDKD (Ours)
62.12 0.93 56.65 0.80
52.92 + 1.06
61.04 + 0.33
57.44 0.69
54.16 + 1.49
Evaluating the performance of the proposed framework against existing decentralized training schemes on CIFAR-10 and CIEAR-100 datasets using Please note that for the SGD-Centralized, we report the results for a random IID data distribution. For QG-IDKD (ours) TinyImageNe was used the public dataset.
Table 2:
on two different network sizes.
without incurring communication overhead over DSGD (Lian et al. 2017). For exhaustive analysis, we present other baselines that incur no communication overhead such (QG-DSGDDm) with vanilla KD (Goyal et al.
(Li et al. 2021) and Relay-SGD (Vogels et al. 2021). Further. we include 2017) as a reference to show the upper bound of performance. Note that for the SGD-Centralized that data is randomly distributed across the nodes (IID data). All the methods are trained on identical initial seeds, rrper-parametery and data distributions. Note that the key hyper-parameter in KD methods the distillation temperature and it needs to be tuned appropriately. We vary the distillation temperature from 1000 and report the best-performing distillation model (temperature 10). The performance results are presented in Table 2 From Table 2] we observe that the proposed IDKD (TinyImageNet used public dataset) performs significantly better than existing schemes, especially as skew increases (i.e. a decreases). On CIFAR datasets with a skew of 0.05, IDKD improves the accuracy by 4-8% compared to QG-DSGDm-N and 2 - 13% compared to vanilla KD. We draw two main conclusions from Table 2 First, the proposed IDKD method performs the best and purperoomss vanilla knowledge distillation. Second, the use of the OoD detector the key to the observed performance boost.
Graph Topologics: In this section, explore two different graph topologies for the decentralized setup i... ring and social network. To add variety to the datasets used, we use the ImageNette dataset (Howard, 2018). Similar to the previous section we train a ResNeet20FFvoormm architecture and use TinyImageNet the public dataset. The results presented in 3show that the trends for the ImageNette dataset are the same as the CIFAR datasets. The proposed technique performs better than QG-DSGDm-N, The exception being Ring of 8 with a = 0.1. Thus, the takeaway from this experiment that IDKD performs better on various graph topologies.
Node Failures (Time-Varee Varying Graphs): In earlier discussions, we focused on static graphs. To demon- strate the real-world relevance of this method, examine node failures, which lead to time-varying graphs, Previous basclines lack convergence guarantees under time-varying graphs. Therefore, this section presents findings using DSGD and SGP (Assran et al.) 2019). both of which are known to converge under time-varying graphs. We implement IDKD with DGSD to highlight improvements achieved through data homogenization. For simulating node failures, we introduced random communication failures at rates 1% and 2% (a failure is triggered if a uniformly generated random number between 0 1 falls below the specified failure rate). The results shown in Table validate that. IDDD enhances DSGD's effectiveness via data homogenization
8

---

Published in Transactions on Machine Learning Research (07/2024)
Ring (n = 8)
Social Network (n 15)
Dataset
Method
Q' 0.1
a 0.05
0.1
a 0.05
DSGD
76.12 5.29
62.99 + 3.14
76.81 + 1.68
69.42 4.29
Relay-SGD
77.18 + 1.92
62.67 + 5.95
75.99 + 1.82
69.95 1.19
CIFARIO
QG-DSGDm-N
77.79 + 1.19
65.54 + 3.37
77.79 + 1.19
68.29 6.71
QG-DSGDm-N + KD
81.23 + 0.14
76.87 +: 1.05
84.08 + 0.22
75.03 = 0.18
QG-IDKD (Ours)
84.58 + 1.40
77.83 = 1.98
84.72 = 0.75
80.33 + 2.50
DSGD
48.67 + 2.65
43.15 + 3.47
55.11 + 1.91
50.41 2.36
Relay-SGD
52.44 0.98
44.38 1 2.75
54.75 1 0.10
50.00 0.77
CIFAR100
QG-DSGDm-N
49.06 + 1.30
44.80 + 1.39
52.14 + 2.54
46.31 2.21
QG-DSGDm-N + KD
52.21 + 0.54
49.67 + 0.36
51.60 + 2.09
18.97 + 0.83
QG-IDKD (Ours)
55.01 + 0.78
50.38 = 1.53
56.02 + 1.81
51.51 + 2.45
DSGD
48.60 + 8.89
48.09 + 11.45
42.78 + 5.51
48.51 5.57
MGD
72.88 + 4.29
70.95 + 5.22
78.08 + 2.79
76.80 3.79
ImageNette
QG-DSGDm-N
78.30 1.33
73.31 5.39
77.09 + 2.37
75.39 2.80
QG-DSGDm-NN KD
49.67 + 0.36
52.21 1 0.54
51.60 1 2.09
67.74 1.13
QG-IDKD (Ours)
74.18 + 3.01
74.60 + 0.45
78.24 + 4.49
76.51 + 2.96
Table 3: Accuracy of ResNee20-EvoNorm trained on different graph topologies. For QG-IDKD (ours) and QG-DSGDm-N + KD, TinylmageNet was used as the public dataset.
Datasct
Failure Rate
Method
CIFAR-10
CIFAR-100
ImageNette
DSGD
67.61 + 3.82
42.21
3.24
55.05 2.10
1%
SGP
57.52 + 0.97
34.04 + 5.52
64.91 + 5.41
DSGD-IDKD (Ours)
78.78 + 1.86
43.46 + 2.25
62.30 + 3.29
DSGD
64.18 6.44
44.40
1.88
64.23
4.85
2%
SGP
60.18 2.09
34.38
4.77
61.02
6.65
DSGD-IDKD (Ours)
79.96 + 0.28
45.06 + 2.47
64.499
1.26
Table 4: Accuracy of ResNoe20-ExxNorm trained on ring (8 nodes, a = 0.05) topology with CIFAR-10, CIFAR-100 and ImageNette datasets with 1% and 2% communication failure.
even under node failures. Interstingly, we observed that at the specified communication failure rates, such disruptions act as form of regularization, boosting the performance of DSGD (see Table 3 vs. Table
The performance improvements achieved by knowledge distillation methods are highly dependent on the selection of a suitable public dataset. To reduce this dependence, the proposed IDKD framework utilizes an OoD detector. To evaluate the impact of public datasets, we run the IDKD framework with various public datasets such as LSUN (Yu et al. 2015) and Uniform Noise. Table 3 compares the vanilla KD with the proposed IDKD framework on these public datasets. Note that we use a subset of the LSUN datasct with 300,000 samples for distillation. The performance trends of the proposod method are the same observed when using ImageNet the distillation dataset. Thus, IKKD is able to choose subset of the distillation dataset that aligns with the private dataset improving the KD performance. Another aspect is the amount of public data used, we found that more data generally helps upto a limit (for details of this experiment see the Appendix).
Impact of Public Dataset Selection:
Visualizing effect of IDKD: To visualize the effect IDKD on data heterogeneity, we compare the class distribution with and without IDKD. In particular, we compute the normalized number of samples per class on a single node with and without IDKD, To remove other inffuences, we collect the class distributions from the same run. That is, for the without IDKD sample distribution (pre-IDKD), we save the initial distribution
9

---

Published in Transactions on Machine Learning Research (07 (07/2024) 2024)
0.5
Post-IDKD
80
3.5
Normalized Class Count
Pre-IDKD
0.4
70
3.0
60
0.3
Accuracy
2.5
Test Test
50
QG-DSGDM-N
2.0
0.2
40
Accuracy (solid)
(dashed)
1.5
30
0.1
20
10
0.5
0.0
2
4
6
8
0
50
100
150
200
250
300
Class
Epochs
(a) Visualizing normalized class distribution pre- IDKD and post-IDKD from single node of 16- nodes ring graph. CIFAR-10 with a = 0.1 distilled with TinylmageNet.
(b) accuracy, test loss (mean std) vs epochs of the proposed IDKD framework and QG-DSGDm-N for training CIFAR-10 dataset on Res.Net20- ResNet20-EvoNorm. Ring 16 with skew of 0.05.
Figure 3: Visualizing the results of the proposed IDKD method (a) on the data distribution pre and post IDKD (b) comparing the convergence of TDKD vs DSGDm-N.
sampled from the Dirichlet distribution. For the distribution with IDKD (post-IDKD), we collect the soft labels along with the initial distribution. Since we use soft labels for the public ID dataset Din, the soft labels are counted for all the classes where the value is non-zero. For example, if soft label is [0.1, 0.2, 0.7, O, OJ, we add 0.1 to the sample count for class 1, 0.2 to the sample count for class 2, and so on. This is repeated for all the samples and the count rounded the nearest whole number. This count is then normalized. Figure 3a visualizes the normalized samples per class with and without IDKD while training CIFAR-10 on 16-ring with skew of 0.1. We clearly samples for classes 0, 1, 3, 4, 5, and 8. We observe that the samples are more evenly distributed post-IDKD.
reduction in data skew (more IID) after performing IDKD.
Pre-IDKD
see that there
To evaluate the convergence performance of QG-IDKD, we plot test accuracy and test loss error vs epochs (Figure 3b). Figure 3b plots the average accuracy and standard deviation for all nodes in the graph. Since we use the same sceds and random states QG-IDKD and QG-DSGDm-N perform similarly until the homogenization step epoch 240, after which QG-IDKD outperforms QG-DSGDm-N, The results were plotted for 20-NNeeO--wwoorm ring 16 with skew of 0.05 for the CIFAR-10 dataset. Figure 3b also plots the results for the test loss for the same configuration and we see that similar to test accuracy there is significant reduction in test loss after the homogenization step by IDKD.
Convergence:
Accuracy on Distillation Dataset
Dataset
Method
TinyImageNe
LSUN
Uniform-Noise
QG-DSGDm-N + KD
77.18 + 2.00
75.48 + 1.90
74.15 + 3.05
CIFAR-10
QG-IDKD (Ours)
81.70 + 1.74
78.56 + 2.02
77.45 + 3.77
QG-DSGDm-N KD
42.69 = 1.54
46.43 + 3.22
45.80 + 1.44
CIFAR-100
QG-IDKD (Ours)
52.92 + 1.06
51.62 + 1.15
48.78 + 2.02
QG-DSGDm-N KD
62.29 +: 7.66
57.55 + 6.94
60.87 + 3.66
ImageNette
QG-IDKD (Ours)
74.60 + 0.45
73.63 + 0.86
75.96 + 0.97
Table 5: distilled with subset of the LSUN, TinylmageNet, and Uniform-Noise. We the Ring topology with 16 nodes for CIFAR-10 and CIFAR-100 and 8 nodes for ImageNette. Dirichlet parameter 0.05.
10

---

Published in Transactions on Machine Learning Research (07/2024)
Ring 16)
Ring (n 32)
Dutaset
Method
a=
= 0.05
0
0.1
Cr 0.05
SGD-Centralized
90.94 + 0.03
90.85 +: 0.17
QG-DSGDm-N
88.98 + 0.11
82.94 + 2.10
73.41 + 4.12
89.31 + 0.11
82.23 + 1.40
72.41 + 3.45
CIFAR-10
QG-DSGDtn-N + KD
88.85 + 0.52
85.34 + 0.77
77.18 + 2.00
87.86 + 0.42
83.16 +: 0.28
77.66
+ 2.53
QG-IDKD (Ours)
89.02
0.28
86.84 + 0.57
81.69 + 2.19
88.89 + 0.39
84.96
0.24
79.81 + 0.95
SGD-Centralized
65.96 0.15
65.62 + 0.51
QG-DSGDm-NN
63.11 0.13
53.36 1.84
47.09 + 2.21
61.75 +: 0.23
56.96 0.54
50.00 2.06
CIFAR-100
QG-DSGDm-N + KD
41.89 0.23
48.53 1.18
42.09 + 1.54
48.46 + 1.13
47.10 2.00
41.21 2.11
QG-IDKD (Ours)
61.84 + 1.12
56.56 + 0.82
52.83 + 0.59
60.53 + 0.16
57.02 + 0.74
53.97 + 0.82
Table 6: Performance of the proposed framework against existing docentralized training schemes on CIFAR- 10 and CIFAR-100 datasets using ResNNe220-EvoNorm on two different network sizes under iso-ileration setting. Thus, the compute overhead is O. Please note that for the SGD-Centralized, for random IID data distribution. For QG-IDKD (ours) TinylmageNet was used the public dataset. The results are almost identical to the ones presented in Table which are under iso-epoch setting.
report the results
Here we analyze the compute cost of the proposed method. For compute analysis we consider iso-itcration performance results. Before we analyze the results, we briefly discuss iso-iteration. Each iteration of the algorithm uses a minibatch of data during backpropagation to compute the gradients. Thus under iso-iterations (i.e. the same number of iterations) and fixed Mani-batch size (was set to 32 for these experiments), the algorithms under consideration have the same compute cost. Thus, for the results presented in Table 6 the compute overhead of our method between 1.5 % (best case) 15% (worst case), average case Rs 4.5% over the baseline algorithm (depends on how many samples from the public dataset were used during training; see the Appendix for details), and the communication overhead remains the same as discussed before. Further, it is interesting to note that the results of iso-iteration IDKD (Table ( show similar performance uplift as iso-epoch (Table 2.. Further, to add a more varied dataset result we also trained FaShionMNIST (a dataset dissimilar to CIFAR10/100 or ImageNette) with iso-iterations until QG-DSGDm-N convergence. For this this experiment the mini-batch size was set to 32, on 16 node-ring with a 0.05, for 35400 iterations. QG-DSGDm-N obtained 77.04 + 0.02 while QG-IDKD achieved 80.71 + 0.06 when using TinyImagenet as public dataset. Clearly, from the FashionMNIST results and Table we see that IDKD improves classification performance while having minimal compute overhead.
Computation Overhead:
Communication Cost: It is common to report bytes transmitted per iteration (Koloskova et al. 2020
to measure communication cost. Note that training epoch contains multiple iterations. We report the communication cost per itcration for CIFAR-10 and CIFAR-100 datasets at iso-iteration. We compare QG- DSGDm-N and the proposed IDKD framework. We use a ResNet 20-EvoNorm for these experiments and report MiB (MebiBytes, MiB 1021² bytes) per iteration in Table Table 7 the mean and stan- dard deviation from three different seeds. The variation in the number of MiB per iteration for the proposed technique is due to the fact that different seeds/runs have different data distribution, thus the mumber of labels that are transmitted between nodes varies. From Table 7 we see that the proposed method has very minimal overhead in terms of communication cost (~ 2% on average). This is because the communication cost of label exchange (very small) is amortized across all iterations. Further, in terms of cumulative com- munication cost, IDKD) will add more communication due to the increased number of iterations arising from the larger training set. However, with QG-IDKD we need fewer epochs which compensates for more data. This is seen in Figure 3b. where maximum performance is achieved within 10 epochs (i.e. by epoch 250) of data homogenization.
Ablation Study: An ablation study was performed to assess the effects of the components of the pro- posed IDKD method, utilizing the baseline algorithm QG-DSGDm-N, QG-DSGDm-N with KD (knowledge distillation), and QG-IDDD (which incorporates KD and an OoD detector). The results are shown in Fig- ure g which compares the performance between the bascline QG-DSGDm-N, QG-DSGDm-N with KD, and QG-IDKD. Furthermore, we examined the impact of initializing with a pretrained model using the public Ciiaset-TinyImageeee denoted as 'Pretrained QG-DSGDm-N in blue. The experiments were conducted
11

---

Published in Transactions on Machine Learning Research (07 /222)
Ablation Study
MiB per iter
81.7
Dataset
Method
77.18
73.44
Ring (n 16)
Ring (n = 32)
Accuracy (in %6)
60
63.08
QG-DSGDm-N
3.13
3.13
52.92
CF-10
45.17
47.09
42.69
QG-IDKD (Ours)
3.33 = 0.05
3.18 = 0.09
Pretrained QG-DSGDmuNN
Baseline (QG-DSGDm-N)
20
Baseline KD
QG-DSGDm-N
3.19
3.19
Baseline KD - IDKD
CF-100
0
QG-IDKD (Ours)
3.20 + 0.01
3.21 + 0.02
Dutasets
Table 7: Comparing communication cost in MiB per iteration for the proposed method and QG-DSGDm-N (a = 0.1).
Figure 4: Ablation study results on CIFAR-10 and CIFAR-100 when using a = 0.05 and a 16 node ring network topology.
Graph
Dataset
Method
0.1
0.05
Ring (n = 64)
QG-DSGDm-N
70.65 + 1.77
66.37 = 2.89
CIFAR-10
QG-IDKD (Ours)
79.74 +: 0.25
76.66 + 1.21
QG-DSGDm-N
53.26 + 0.55
48.76 + 1.09
CIFAR-100
QG-IDKD (Ours) 56.91 + 0.60
54.07 + 0.38
Table 8: Performance of IDKD against. QG-DSGDm-N on CIFAR-10 and CIFAR-100 datasets using ResNet20-EvvNorm under iso-iterations. TinylmageNet was used as the public dataset.
with a = 0.05 on a 16-node ring topology. From Figure we see that solely using KD does not consis- tently enhance performance compared the baseline, but incorporating an OoD detector consistently does, Interestingly, pre-training appears to reduce performance. Additional results also be found in Table 2
Scalability: Here we analyze the scalability of IDKD the number of nodes increases. In addition to 8, 16, and 32-node results in Tables 2 and 6 we provide 64-node iso-iteration results in Table S From Table 0 and Table g we see that as the number of nodes increases, all methods lose performance on ring networks (also observed in other related work on generic graphs, this is due to decreased speed of information travel as graph size increases). However, we observe that IDKD consistently outperforms other techniques in all cases, thus IDKD scales well the number of nodes increase.
This paper introduces the IDKD framework, which is designed for non-ITD decentralized learning. However, it may also be suitable for federated learning. It should be noted that federated systems are a subset of decentralized systems. For instance, a decentralized system configured in a star lopology without data at the central node mirrors a federated learning framework. Results from applying IDKD to model sharing approaches are presented in Tables 2 and 4 In addition to applying IDKD on QG-DSGDm-N, Table shows the results of applying IDKD on DSGD. This shows that IDKD can be readily adapted to federated learning scenarios aimed at. minimizing communication rounds.
IDKD and Federated Learning:
5 Conclusion
Decentralized learning methods achieve state-of-the-at performance on various vision benchmarks when the data distributed in an IID fashion. However, they fail to achieve the same when the data distribution among the nodes is heterogeneous. In this paper, we propose In- In-strribution Knowledge Distillation (IDKD), novel approach to deal with non-IID data through knowledge distillation. The proposed method aims to
12

---

Published in Transactions on Machine Learning Research (07 (07/2024)
homogenize the data distribution on each node through the aggregation of the distilled labels subset of the public dataset. In particular, each node identifies and labels a subset of the public data that is proxy to its local (private) dataset with the help of its local model and an OoD detector. The distilled versions of the local datasets are aggregated in a pecr-to-peer fashion and the local models are then finc-tuned on the combined corpus of the distilled datasets along with the local dataset. We show that the proposed IDKD framework is superior to vanilla knowledge distillation (2 13%) in docentralized setting. Our experiments on various datasets and graph topologies show that the proposed IDKD framework achieves up to 8% improvement in performance over the seate-of-the-an techniques non-IlD data with minimal communication overhead (~ 2%).
Acknowledgments
This work was supported in part by, the Center for the Co-Design of Cognitive Systems (CoCoSys), a DARPA-sponsored JUMP 2.0 center, the Semiconductor Research Corporation (SRC), and the National Science Foundation.
References
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 163--177, 2019.
Sai Aparna Aketi, Sangamesh Kodge, and Kaushik Roy. Neighborhood gradient clustering: An efficient decentralized learning method for non-iid data distributions. arXiv preprint vrXhv2209.143900 2022.
Mathieu Andreux, Jean Ogier du Terrail, Constance Beguier, and Eric W Tramel. Siloed federated learning for multi-centric histopathology datasets. In Domain Adaptation and Representation Transfer, and Dis- tributed and Collaborative Learning: Second MICCAI Workshop, DART 2020, and First MICCAI Work- shop, DCL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings 2. pp. 129 139. Springer, 2020.
Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning, pp. 344 353. PMLR, 2019.
Ronald L Breiger and Philippa E Pattison. Cumulated social roles: The duality of persons and their algebras. Social networks, 8(3)2115 226,, 1986.
Yatin Dandi, Anastasia Koloskova, Martin Jaggi, and Sebastian U Stich. Data- hetrroheerr aware mixing for decentralized learning. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
John C Duchi, Alekh Agarwal, and Martin J Wainwright. Dual averaging for distributed optimization: Convergence analysis and network scaling, IEEE Transactions on Automatic control, 57(3):592-66) 606, 2011.
Yasaman Esfandiari, Sin Yong Tan, Zhanhong Jiang, Aditya Balu, Ethan Herron, Chinmay Hegde, and Soumik Sarkar. Cross-gradient aggregation for decentralized learning from non-iid data. In International Conference Muchine Learning, pp. 3036 3046. PMLR, 2021.
Tom Fawcett, An introduction to roc analysis. Pattern recognition letters, 27(8):861-874, 2006.
Raghu Ganti, Fan Ye, and Hui Tei. Mobile crowdsensing: current state and future challenges. TEEE communications Magazine, 49(1):33 39, 2011.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint or X 1706. 02677, 2017.
13

---

Published in Transactions on Machine Learning Research (07/2024)
W Gropp, Ewing Lusk, R Ross, and Rajeev Thakur. Using mpi-2: Advanced features of the message passing interface. In 2003 Proceedings IEFE International Conference on Cluster Compuling, pp. xix-xix TREE Computer Society, 2003.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image rocognition. Proceedings of the IEEE conference on computer vision and paltern recognilion, pp. 770 778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. In International Conference Learning Represenlations, 2017. URI. https:// httpr//wwnee//ooomiid-tiiiit
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a noural network. In NIPS 02531
Deep Learning and Represenlation Learning Workshop, 2015. URL http:htttp::/ttt httpaaawwwv...aaaaaaa
Jeremy Howard. Imagenette a subset of 10 easily classified classes from the imagenet dataset. https:
//github. hatab//...c.sstaa/aanenttttt 2018.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of de- centralized machine learning. In International Conference on Machine Learning, pp. 4387 4398. PMLR, 2020.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. pmlr, 2015.
Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-basel semi-supervised federated learning for commuicattinn-elliienn collaborative training with non-iid private data. IEEE Transactions on Mobile Computing, 22(1):191 200,, 2021.
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Scong-Lyun Kim. Communictiion-efficeet on-device machine learning: Federated distillation and augmentation under non- iid private dala. arxiv preprint
1818.11188 1. 11479, 2018.
David Kempe, Alin Dobra, and Johannes Gehrke Gossip-based computation of aggregate information. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings., pp. 482-491. IEEE, 2003.
Anastasia Koloskova, Tao Lin, Sebastian U Stich, and Martin Jaggi. Decentralized deep learning with arbitrary communication compression. Tn International Conference on Learning Representations, 2020. URL httpp:://opaarevier.eeee
Anastasiia Koloskova, Tao Lin, and Sebastian U Stich. An improved analysis of gradient tracking for
decentralized machine learning. Advances in Neural Information Processing Systems, 34:11422 11435, 2021.
Lingjing Kong, Tao Lin, Anastasia Koloskova, Martin Jaggi, and Sebastian Stich. Consensus control for docentralized doep learning. In International Conference on Machine Learning, pp. 5686-5696. PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Chengxi Li, Gang Li, and Pramod K Varshney. Decentralized federated learning via mutual knowledge transfer. IEEE Internet of Things Journal, 0(2):1136-1144 2021.
Daliang Li and Jumpu Wang. Fedmd: Heterogenous foderated learning via model distillation. arXiv preprint arXiv:1910.0885 2019.
Fei-Fei Li, Andrej Karpathy, and Justin Johnson. Tiny imagenet visual recognition challenge. cs231n. cs23In.stanford. waw.afordddduttttt-nnnnaeeeeeeee 2200.zzzp 2015.
http://
14

---

Published in Transactions on Machine Learning Research (07/2024)
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning, pp. 3043 3052. PMLR, 2018.
Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Bing-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 22(3):203112000 2020.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351 2363, 2020.
Tao Lin, Sai Pranceth Karimiroddy, Scbastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerat- ing decentralized deep learning on heterogeneous data, In International Conference on Machine Learning, pp. 6654-6665. PMLR, 2021.
Hanxiao Liu, Andy Brock, Karen Simonyan, and Quoe Le. Evolving nornalizationnaalivvtiio layers. Ad- vances in Neural Information Processing Systems, 33:13539 13550, 2020a.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection.
In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advonces in Neural Infor-
malion Processing Systems, volume 33, pp. 21464 21475. Curran Associates, Inc., 2020b. URL https:
Angelia Nedić, Alex Olshevsky, and Michael G Rabbat. Network topology and communicationn.ccmpttaaaiiii tradeoffs decentralized optimization. Proceedings of the IEEE, 106(5):953 976, 2018.
Giovanni Neglia, Chuan Xu, Don Towsley, and Gianmarco Calbi. Decentralized gradient methods: does topology matter? In International Conference on Artificial Intelligecce and Statistics, pp. 2348 2358. PMLR, 2020.
Adam Passke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, ZZeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, pirh-peeformance deep learning library. Advances in neural information processing systems, 32, 2019.
Rüdiger Pryss, Manfred Reichert, Jochen Herrmann, Berthold Langguth, and Winfried Schlee. Mobile crowd sensing in clinical and psychological trials-a a case study. In 2015 IEEE 28th international symposium on computer- based medical systems, pp. 23-24. IEEE, 2015.
Deepak Ravikumar and Kaushik Roy. Norm-scaling for out-of-distribution detection.
ar preprint
arXiv:2205.033 034933, 2022.
Deepak Ravikumar, Sangamesh Kodge, Isha Garg, and Kaushik Roy. Intra-class mixup for out-of-distribution detection. IEEE Access, 11:25968 25981, 2023.
Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. D2: Decentralized training over decentralized data. In International Conference on Machine Learning, pp. 1848-4856 PMLR, 2018.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020. URL httppppr/pppppeeeeeesee////rrrrp
John N Tsitsiklis. Problems in decentralized decision making and computation. Technical report, Mas- sachusetts Inst of Toch Cambridge Lab for Information and Decision Systems, 1984.
Thijs Vogels, Lie Hc, Anastasiaa Koloskova, Sai Pranccth Karimireddy, Tao Lin, Scbastian U Stich, and Martin Jaggi. Relaysum for decentralized deep learning on heterogeneous data. Advances in Neural 2021.
Information Processing Systems,
15

---

Published in Transactions on Machine Learning Research (07/2024)
Xiaofei Wang, Yiwen Han, Victor CM Leung, Dusit Niyato, Xueqiang Yan, and Xu Chen. Convergence of edge computing and deep learning: A comprehensive survey. IEEE Communications Surveys & Tulorials, 22(2):869 904, 2020.
Lin Xiao and Stephen Boyd. Fast linear iterations for distributed averaging. Systems 8 Control Letters, 53 (1):65-78, 2004.
Taocun Yang, Yaping Huang, Yanlin Xie, Junbo Liu, and Shengchun Wang. Mixood: Improving out-of- distribution detection with enhanced data mixup. ACM Transactions on Multimedia Computing. Com- municalions and Applications, 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.0880 2015.
Kun Yuan, Yiming Chen, Xinmeng Huang, Yingya Zhang, Pan Pan, Yinghui Xu, and Wotao Yin. Decentlam: Decentralized momentum sgd for large-batch deep training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3029 3039,,00,,--000 2021.
Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. Newrocomputing, 465:371-390 2021a.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou, Data-free knowledge distillation for heterogeneous federated learning. In International Conference Machine Learning, pp. 12878 177881 PMLR, 2021b.
16

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Appendix
Here we present additional details about the experiments. To improve reproducibility, the source code is provided the IDKD Lrpp__Sorce__Caee Code directory of the accompanying zip file.
Dataset Statistics
To evaluate the performance of the proposed IDKD framework, we used various image classification bench- marks. To improve reproducibility we provide more details on the datasets and split used for our experimental evaluation. For all the experiments, used 10% train-validation split to identify the hyperparameters. Once the hyperparameters were tuned on the validation set, we re-ran the experiment on the complete train- ing set with the identified hyperparameters. Details of the dataset sizes are reported in Table g for the LSUN Yu et al., 2015) dataset we used the first 300,000 samples subset for our experiments. This was done because LSUN a very large dataset and would require significant compute resources to use the complete training datasct. For training the neural nets the image size for CIFAR-10 and CIFAR-100 were 32 x 32 (C H W) and for ImageNette was 64 x 64. During distillation, the distilling dataset cropped to the same size as the training dataset.
Please note
Dataset
Train Set Size
Validation Set Size
Test Set Size
CIFAR-10
45,000 (90%)
5,000 (10%)
10,000
CIFAR-100
45,000 (90%)
5,000 (10%)
10,000
ImageNette
8,522 (90%)
947 (10%)
3925
TinyImageNet*
100,000 (100%)
LSUN
300,000 (100%)
Uniform Noise
150,000 (100%)
Table 9: Training, Validation and Test set sizes for the datasets used. Datasets with were used as public dataset for distillation hence all the images from the training set were used without partitioning to a validation set.
Dataset
Method
LR
BS
N
Topology
Weight Decay
7
QG-DSGDm-N DSGD
QG-IDKD
0.5
0.9
32
16, 32
Ring
1.00E-04
N/A
CIFAR-10
0.1
0.9
32
16, 32
Ring
5.00E-04
1
Relay-SGD
0.1
0.9
32
16, 32
Chain
5.00F-04
QG-DSGDm-N
QG-IDKD
0.5
16,
Ring
1.00E-04
N/A
CIFAR-100
DSGD
0.1
0.9
32
16, 32
Ring
5.00E-04
L
Relay-SGD
0.1
0.9
32
16, 32
Chain
5.00E-04
1
QG-DSGDm-N
QG-IDKD
0.5
0.9
32
8, 15
Ring, Social
1.00E-04
N/A
ImageNette
DSGD
0.1
0.9
32
8, 15
Ring, Social
5.00E-04
1
Relay-SGD
0.1
16,
Chain
5.00E-04
Table 10: Hyper parameters used for training the models on various datasets and decentralized algorithms. Dirichlet parameter o, Momentum B, Batch Size B.S, consensus step size y, N is the number of Nodes in the graph.
17

---

Published in Transactions on Machine Learning Research (07 2024)
Compute Resources and Reproducibility
For all the experiments we used cluster of 8 nodes. 4 of these nodes were equipped with Int) Xeon(R) Silver 4114 CPU with 93 GB of usable system memory and 3 NVIDIA GeForce GTX 1080 Ti. The other 4 nodes were equipped with an Inte Xeon(R) Silver 4114 CPU with 187GB GB of usable main memory and NVIDIA GeForce GTX 2080 Ti. All the nodes ran CentOS Limux release 7.9.2009 (Core). Regarding packages and versions used for implenentation, a detailed requirements file provided with the source code. In brief, for communication, we use MPI (Gropp al. (003) implementation from mpich3.2, for parallel decentralized learning. We used Pylorch framework (Paszke et al. 2019) for automatic differentiation of deep learning models. The source code includes README file with details on how to reproduce the results. The results reported in the paper we run on seeds 4, 34, and 5. We have tried our best to make the exact runs reproducible, the only requirement being the need to run on the same underlying Nvidia hardware i.e 1080Ti and 2080Ti. This requirement due to the application stack details. However, we maintain that when the code run on different hardware statistically similar results will be obtained.
Hyper Parameters
The hyperparameters used for training the models are presented in Table 10 The hyperparameters were luned using a validation set (10% of the train set). The final accuracy was reported by using these hyper- parameters trained on the complete train set. The learning rate denoted by LR, momentum by B, batch size by BS, number of nodes in the graph by N and consensus stop size 7
Dataset
No. of Public Images Exchanged with Neighbor
CIFARIO
15,892
CIFAR100
2,559
ImageNette
21,295
Table 11: Average (over 3 seeds and 8 nodes each) number of public images detected as ID (and exchanged) by the OoD detector when the public dataset is Neteee,,, using an 8 node ring configuration a = 0.05
Computing IDKD overhead
To calculate IDKD overhead, we consider training of a 8-node ring on CIFAR10. Assuming a batch size of 32 and the public dataset has 100,000 images (Tiny TinylmageNet). On average, the iso iteration epoch Rs 262 for CIFAR100. That is, QG-IDKD at 262 epochs is the same as QG-DGSDm-N 300 epochs (for iso-iteration results, refer to Table 6 Thus if we consider a forward pass compute to be ix the, backward pass is 2x hence iteration is 3x forward pass. Thus to train model using QG-DGSDm-N (baseline) for 300 epochs is:
costbaseline X num iterations
(1)
num train samples
3 X num epochs num nodes x batch size
(2)
3 x 300 X 50,000 000
(3)
8 x 32
175, 781.25 forward passes
(4)
To calculate the overhead we use that IDKD is performed every 5 epochs (which what is used for the results). For the average iso-iteration epochs this results in 3 IDKD exchanges (i.e. 3 num idkd ex.), thus the overhead is from the forward pass the public data:
costoverhead num idkd ex. x (forward pass on all public images forward pass on all selected public images)
18

---

100000 Published in Transactions on Machine Learning Research (07 (07/2024)
This is because run for iso-iteration for both IDKD and baseline, hence the mumber of backward passes are the same, the overhead is only from the forward pass the public dataset. However, forward pass on all public dataset is not wasted since the images identified by the OoD detector can be used in backward pass. This is the term subtracted from Equation From Table for CIFAR10 we see its 15,892. Plugging this into Equation we get:
costoverhead = mum idkd ex. (num of public image batches - num selected public image batches)
mum idkd mum of public images mum of public images
batch size
100, 000 15, 892
3 X
32
84, 108
7885.125 forward passes
Thus the average overhead for CIFAR10 is
7885,125
X 100
175, 781.25
4.48%
The best case is 1 IDKD exchange for which the overhead is 1.49% the worst is 10 exchanges for which the overhead 14.95%.
Length of Public Dataset
To investigate the size of the public dataset required for training in CIFAR-10 and CIFAR-100 using IDKD, uniform noise was used as the public dataset. Different sizes of this dataset were tested, including 10000, 20000, 50000, and 100000. These data sets were used to run the QG-IDKD algorithm. The accuracy results of these models were collected and shown in the graph in Figure 5 In this study, an 8-node ring topology with a 0.05 was used. The results shown in Figure 5 suggest that the amount of public dataset needed varies depending on the training set. Generally, both datasets (CIFAR-10 and CIFAR-100) experience improvements with increased data, although gains fade off beyond 50,000.
Length of Public Set vs. Accuracy
80
70
77.29
78.36
80.16
60
Accuracy (in
50
40
48.08
49.00
48.15
49.42
30
20
CIFAR-10
10
CIFAR-100
10000
20000
50000
Public Dataset Length
Figure 5: The effect of length of public dataset on the accuracy of QG-IDKD for and node ring topology
with a = 0.05 for CIFAR-10 and CIFAR-100 datasots.
19


Reviews:
[{"review_id": "uGAwkTE32Y", "reviewer": "Reviewer_7rgJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Effective Handling of Data Heterogeneity: The paper successfully tackles the issue of heterogeneous data distributions across nodes—a common obstacle in decentralized systems.\n2. Minimal Communication Overhead: Another significant strength of the paper is its achievement in reducing the communication overhead typically associated with decentralized learning.\n\nWeaknesses\n1. Lack of Ablation Study: One significant weakness in the paper is the absence of an ablation study.\n2. Lack of Computation Cost Analysis: The paper does not provide a detailed analysis of the computational costs associated with the proposed IDKD method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces In-Distribution Knowledge Distillation (IDKD), a novel method designed to improve decentralized learning on non-IID datasets. It addresses data heterogeneity across nodes by using a public dataset for knowledge distillation without compromising privacy. By incorporating an Out-of-Distribution (OoD) detector, IDKD selectively distills knowledge based on data alignment, thus overcoming the common pitfalls of traditional knowledge distillation methods. The experimental results show that IDKD outperforms existing methods in terms of generalization performance on diverse image classification tasks and network configurations, achieving this with minimal communication overhead. This approach provides an efficient solution for homogenizing data distribution in decentralized environments while maintaining privacy.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. To better understand the efficacy and contribution of each individual component of your proposed IDKD method, an ablation study is essential. This study should detail how each component (e.g., the OOD detector and the specific distillation strategy) contributes to the overall performance improvements.\n2. Your manuscript currently lacks a detailed analysis of the computational costs associated with implementing IDKD in decentralized settings.", "broader_impact_concerns": "Not involved.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tQJvd805yt", "reviewer": "Reviewer_tEFU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written and well-illustrated.\n\nFig. 2 is very helpful to communicate the high-level idea & motivate the need for using an OOD detector to find the most aligned public subset for each private dataset.\n\nThe literature is sufficiently well-covered.\n\nThe high-level idea on data homogenization is interesting.\n\nWeakness\n\nI have the following concerns:\n\nFirst, the paper is not super clear about whether this is about learning a common solution given heterogeneous clients or learning multiple personalized solutions (one per client). Depending on the specific setting, the evaluation method might need to change.\n\nCan the authors provide further clarifications on this?\n\nSecond, I think one main motivation for decentralized learning is to remove a choke point of computation and communication. That is, if a node fails, the entire system will still work robustly. However, the paper does not seem to have this evaluation.\n\nNote that, in practical scenarios, even with a server-coordinated federated learning system, only a subset of nodes will participate in the aggregation step. The authors should demonstrate that their data homogenization scheme is robust / resilient against node failure (e.g., setting a drop-out probability for each node)\n\nThird, have the authors considered comparing with a much more simpler, pure engineering solution such as having one node fitting a model on the public dataset and distributing it across the network (sending the model to its neighbors, and their neighbors and so on). \n\nEach client can then fine-tune the public model on its private data. I suspect that this is the might be the most effective way if the setting is to generate personalized solution per node. For example, see https://openreview.net/forum?id=hDDV1lsRV8\n\nOtherwise, if the goal is instead to learn a common solution for all nodes, each client can also distribute its fine-tuned model to its neighbors and hence, to the entire network. This is probably not more expensive than the authors' proposed distillation mechanism. \n\nFourth, I wonder if the base distributed algorithm (in decentralized system) that the authors leverage will be suitable for federated learning. In my view, one key distinction between distributed & federated learning is that (in addition to data privacy) the latter also wants to minimize the no. of communication rounds, which is achieved via model sharing (rather than gradient sharing). \n\nIs this also the case here? This is not really a show-stopper but it does deserve some discussion (preferably in the main text). \n\nLast, I think the experiment is a bit light. The network size seems to be a bit small (32). \nCould the authors run an experiment in larger scale, e.g., more clients (> 100), and with certain probability for node failure (to demonstrate the resilience aspect)?\n\nFurthermore, the experiments in both Tables are not complete. In Table 2, there is no results reported for ImageNette. In Table 3, there is a lack of comparison with most baselines. There is also no result reported for CIFAR datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a data-homogenization approach to a decentralized federated learning with non-IID client data. \n\nThe main (high-level) contribution here is the data-homogenization component (via knowledge distillation) which can be added to any existing decentralized deep learning algorithm.\n\nThis contribution is motivated from the fact that existing decentralized learning has not considered distilling from an existing public dataset to align and aggregate information across heterogeneous client better.\n\nThus, the main technical innovation (to substantiate the above high-level contribution) is an OOD-guided knowledge distillation algorithm. Once the decentralized learning algorithm finishes: \n\n1. Each client will use the same public dataset to distill knowledge from its local model, resulting in a (public samples, soft labels) dataset\n2. An OOD detector is utilized to determine a subset of the distilled dataset to communicate. \n\nThe 2nd step is to ensure that the distillation is focused on the part of the public dataset that is most aligned with the private data.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggest the following addition:\n\n1. Complete the experiment in Tables 2 & 3. Add more experiment with larger network & more clients. \n\n2. Demonstrate the resilience of the proposed scheme when each node has certain failure probability (see weakness #2) -- will we still have better performance than existing decentralized learning algorithm that this method builds on?\n\n3. Discussing the points I raised in weaknesses #1 and #4\n\n4. Compare with pure engineering solutions to this problem along the direction that I outlined in weakness #3.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "saQWG9tYcq", "reviewer": "Reviewer_F2Mo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n1. The authors propose a method called In-Distribution Knowledge Distillation (IDKD) to address this challenge by homogenizing the data distribution across nodes without sacrificing privacy\n\n2. To overcome the misalignment between private and public data distribution, an Out-of-Distribution (OoD) detector is introduced at each node to label a subset of the public dataset that maps close to the local training data distribution\n\n3. The experiments conducted on multiple image classification datasets and graph topologies demonstrate that the proposed IDKD scheme outperforms traditional knowledge distillation and achieves state-of-the-art generalization performance on heterogeneously distributed data with minimal communication overhead\n\nWeakness: \n\n1. In the problem setting, it needs to assume there is a public dataset that across every node. I think this is a very strong assumption. The authors need more justification that why this assumption is reasonable and should provide some real-world scenarios. \n\n2. The experimental settings are very unclear. For example, (1) the paper consider different public data size without any analysis such as how many data will be sufficient. (2) The non-i.i.d. setting is unclear. \n\n3. The improvement compared to previous results is very limited while the method in the paper needs a very strong assumption on public data, such as Table 2 and Table 5. And it could be worse than the previous methods. \n\n4. Although the framework is simple, it is only a combination of different approaches directly. I think there should be some novel parts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on the challenge of heterogeneity in data distribution in decentralized learning, which leads to slow and unstable global convergence and poor generalization performance. The authors propose a method called In-Distribution Knowledge Distillation (IDKD) to address this challenge by homogenizing the data distribution across nodes without sacrificing privacy. The experiments conducted on multiple image classification datasets and graph topologies demonstrate that the proposed IDKD scheme outperforms traditional knowledge distillation and achieves state-of-the-art generalization performance on heterogeneously distributed data with minimal communication overhead.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "See my weakness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the work should be accepted, and basically seem to be in agreement that while there are experiments, settings, and comparisons that could strengthen the work (e.g. larger scales, more challenging communication settings) and that the reliance on a public dataset is potentially a drawback (though as the authors note, their method works even when initializing the public dataset to be purely noisy), the paper gives enough evidence towards its stated accomplishments that it warrants acceptance.

Reviewers specifically highlighted the breadth of experimental comparisons (to other methods, to other settings, in various topologies and on various datasets) as a strength that outweighed any reservations they had. Thus, I recommend acceptance as is.
Decision: Accept (poster)