Gossip Distillation: Decentralized Deep Learning
Transmitting neither Training Data nor Models
Taisuke Moriwaki

Kazuyuki Shudo

Tokyo Institute of Technology

Kyoto University

Abstract—While deep learning requires a large amount oftraining data to obtain a highly accurate model, it is notalways possible to collect the data in one place for privacy
and other reasons. Furthermore, eliminating centralized servers
and allowing all nodes to communicate in a decentralized way,
improves fault tolerance and eliminates the unfairness of servers
getting learned models first.
In existing methods, a node transmits a learned model to othernodes. Our proposal, Gossip Distillation is to apply Knowledge
Distillation to communication between nodes. A node transmits
inference results on common data, not the model itself. The
method reduces the amount of communication and makes it
possible to combine multiple different models for each node.
For CINIC-10, only 5.18 MiB of the inference results aretransmitted between nodes though existing methods requires
49.03 MiB of ResNet-18 as the main model to be transmitted.
In addition, the proposed method allows different sub models tobe trained in parallel with the main model. Achieved accuracies
are comparable with existing centralized methods.
Index Terms—Decentralized deep learning, gossip, Knowledge
Distillation

I. I NTRODUCTION
Training deep neural networks typically requires large anddiverse data sets. It hinders learning with data that are distributed to multiple organizations and cannot be shared withothers. Federated Learning [1] have enabled deep learning
without sharing data among learning nodes.
But Federated Learning comes with the constraint that users
must trust a centralized server and the organization operating
it. To address this issue, decentralized distributed learning
has become increasingly important in modern society, withproposals such as Oguni et al. [10], [11].
In such decentralized learning, a node transmits a modelto other nodes as well as Federated Learning. Our proposal,
Gossip Distillation is to apply Knowledge Distillation to such
communication between nodes. A node transmits inference results on common data, not the model itself. It reduces the sizeof data transmitted between nodes by several times as shownin Table IV and Figure 1. For CINIC-10, only 5.18 MiB of
the inference results are transmitted though existing methods
requires 49.03 MiB of ResNet-18 as the main model to betransmitted. In addition, the proposed method allows different
sub models, DenseNet-121 and MobileNet V3 to be traindin parallel with the main model. The trained models achieve
comparable accuracies with existing centralized methods.
This paper is organized as follows. Related work is shown in
Section II. Section III presents our proposed method. Section

IV shows experimental setups and results, and Section V
summarizes our contribution.
II. R ELATED WORK
A. Federated Learning
Federated Learning uses a central server and numerous
connected devices, including mobile devices and IoT devices.
In contrast to traditional centralized learning, where data issent to the central server for training, Federated Learning sends
model gradient information instead. The node devices perform
training using this information, and then the model gradientsare sent back to the server to integrate them into a new model.
This process is repeated iteratively. As a result, data does nothave to be shared with others, making it possible to maintainprivacy.
B. Knowledge Distillation
Knowledge Distillation is a method for achieving highaccuracy even with lightweight and simple models. This is to
use the results of training a model with a large and complex
structure for training a lightweight model.
This improves accuracy because more detailed data can beused for training than the usual training data given by 0–1.
C. Decentralized Learning via Adaptive Distillation (DLAD)
Decentralized Learning via Adaptive Distillation (DLAD)[6] is a centralized method that incorporates the conceptof Knowledge Distillation into Federated Learning. While
Federated Learning sends the gradient of the learned model tothe server, DLAD infers the distillation dataset in addition to
the learned model on the private dataset. The inference results
are sent to the central server, which adaptively integrates themaccording to the contents of the private dataset, and the servertrains a new model that can output the integrated results.
Contrary to its name, DLAD is a centralized method involving the central server. In contrast, our proposed method isdecentralized.
D. Distributed machine learning with gossip
Gossip [7] is a method to multicast messages over a network
and is often used in a distributed system. Gossip learning [8]is a decentralized distributed learning method using gossip.
It is for classification problems with linear models, not for
deep learning. This method is a privacy-conscious learning
method that does not move data from one node to another

Proc. IEEE Virtual Conference on Communications(IEEE VCC 2023), November 2023

and uses gossip as a communication method. Takahashi et al.
[9] improved gossip learning by manipulating the selection
probability according to the order of the nodes. Oguni etal. applied gossip-based communication to deep learning toachieve decentralized deep learning [10], [11]. They delat with
the issue of heterogeneity of learning nodes [10] and resolved
congestion by changing the communication frequency [11].
Hu et al. [12] tried to combine gossip into Federated Learning.
Specifically, the weights of models owned by multiple clientsare combined using the gossip technique. Therefore, gossip
has been considered as a promising technique for decentralized
learning.
III. G OSSIP D ISTILLATIONIn this chapter, we describe the proposed method, Gossip
Distillation.

Algorithm 1 Gossip Distillation (local phase)
Input: Initial Weight W0 , Each model Mi , Each Private Data
P Di , Common Data Xc , number of Nodes N
Output: Each Inference List ILi , Trained Weight Wi
function L OCAL P HASE(W0 , P Di , DRS)
for all each client number doWi ← U pdate(W0 , P Di )ILi ← Inf erence(Wi , Xc )end forend function

times data has been transmitted so far, Ctgt . The receiving
node, Nget , which has its own Xc , receives Ltgt and Ctgt , andcreates its own list Lget and count Cget . The lists are thenintegrated by a weighted average as follows:

A. Problem Formulation(i)

Let x ∈ X be an input sample (e.g. image or sound), andy ∈ Y be the label for the input sample. In this research,
we will focus on N -class classification problems where Y ={1, 2, . . . , N }.
To formulate the problem in classification learning, we
consider the existence of multiple clients.
Assuming that there are N nodes N1 , N2 , . . . , NN , eachpossessing a labeled dataset Di = (Xi , Yi ) for i ∈ 1, 2, . . . , N
(i)
(i)
(i)
where Xi = {xj } and Yi = {yj }, also yj is the label(i)annotated to xj .
The set Di can only be viewed by node Ni , and cannot beviewed by any other nodes {Nj |j ̸= i} in the network.
Hereafter, we reffer Di to as Private Data (PD).
Each node has its own model Mi : X → Yi learned using
PD.
Since it can be fully assumed that real-world data is NonIID, PD is also Non-IID, Therefore, even if the same image
is inferred using Mi and Mj (i ̸= j), it is not necessarily
guaranteed to yield the same result.
Our goal is to infer the output results of both Mi and Mj
correctly and identically by utilizing each other’s classification
ability.
B. Distillation
To transfer the classification ability of a node’s Mi to othermodels, we use the idea of Knowledge Distillation [5] in thisstudy. In addition to PD, there exists a label-free dataset thatcan be viewed by any node, called the Distillation Dataset(DD). To create DD, a separate set of data, the Common Data
X c (CD), is prepared apart from PD.
By using DD, the classification ability that is only possessedby the model owned by the node trained on PD can bepropagated to other models. However, it becomes difficult
to simultaneously send and receive knowledge from multiple
nodes, so the gossip (pull) method is used to communicate
one-on-one repeatedly.
Specifically, when a node Ntgt , which owns the model
Mtgt , infers Xc , it creates a list Ltgt , and the number ofLmix(i) =(i)

Ltgt × Ctgt + Lget × Cgeti ∈ {1, 2, . . . , n} (1)Ctgt + Cget

where n is the total number of nodes in the CD.
Then, we create a label list Yc using the maximum elementof the list generated by Equation 1, i.e., Yc(i) = max Lmix(i) .
After ward we create DD, Ddist = (X c, Yc) with images as
X c and labels as Yc. DD is repeatedly regenerated every time
another inference list is incorporated through communication,and then the model Mi is trained again using the regenerated
DD. Ultimately, it is expected that each node’s Mi will acquire
the classification ability of all data in PD.
C. Proposed method
1) Local phase: First, training the model Mi on the P Di
owned by each node.
Then, after inferring the Distillation Dataset on Mi , saving
the inference results for each class. Specifically, we followed
the algorithm 1.
2) Gossip phase: In this section, we update the model Mi
by exchanging the ILi generated in section III-C1 between
nodes, creating a DD, and training it. We repeat this process
for a number of rounds that has been set in advance.
1) Specify any node other than oneself.
2) Receive the inference list of that client and the numberof times the loop is executed for the specified client.
3) Create DD according to equation 1.
4) Learn the model owned by oneself using DD.
5) Update the inference list for transmission by inferring
the image of DD using the learned model.
Specifically, as shown in Algorithm2.
IV. E XPERIMENTS
In this chapter, we will conduct experiments using the
proposed Gossip Distillation method. We will compare theproposed method with DLAD (Section II-C) [6], which uses
Knowledge Distillation in centralized distributed learning, asa previous study.

Proc. IEEE Virtual Conference on Communications(IEEE VCC 2023), November 2023

Algorithm 2 Gossip Distillation (gossip phase)
Input: Each Trained Weight Wi , Round count R, Target Node Ntgt , User
Node Nusr
Each Node has set N = (L, C)
function G OSSIP P HASE(R, Nusr )for all Rounds r = 1, 2, . . . , R do
tgt ← Randomly selected node id
download Ntgt = (Ltgt , Ctgt )function MAKE D ISTILLATION DATASET(Ntgt , Nusr )image. . . Images for Distillation Datasetlabel. . . make as follows
1. Integrate like Equation 1 by using (Ntgt , Nusr )2. The highest-valued index of each element is the label.
return DDi = (image, label)end function
Wi ← U pdate(Wi , DDi )ILi ← Inf erence(Wi , DRS)
end forend function

A. Datasets
In this section, we describe the datasets used in the experiment. We use MNIST, CIFAR-10, and CINIC-10 [13] inaccordance with DLAD.CINIC-10 is a huge dataset consistingof 270,000 images extracted from CIFAR-10 and ImageNet,making it a challenging dataset.
In all datasets, we use 20% of the training dataset as private
data and the remaining 80% as distillation data, simulating
real-world situations where there are more unlabeled data thanlabeled data.
Regarding the dataset used for private data, we randomly
select it but fix the random seed to ensure that it has thesame distribution during the experimental stage. In addition,
we split the PD into n clients, where each node Ni has an/idataset Di = (Xj , Yj )j=1 .
In this experiment, we split 20% of the data, which is 60,000
for MNIST and 50,000 for CIFAR-10, without duplicates.
Although DLAD allows duplicates, we do not permit themin this experiment. As a result, the size of the PD owned byeach client is about 2% of the training dataset.
For CINIC-10, 90,000 samples are prepared for both thetraining dataset and the validation dataset. Therefore, we canuse the validation dataset as distillation data, but we chose to
split the training dataset into 90,000 parts, similar to the ratioof MNIST and CIFAR, to create the aforementioned situation.
We tested using five different data distributions, as shownin Table I. In addition to the same distribution as DLAD,
we added one distribution where the data distribution varies
among clients.
• IID All clients have data with probability pi=[0.1, 0.1, 0.1, . . . , 0.1].
• Non-IID #1 Each client has data for 2 out of 10 classes.
Specifically, p5k+1 = [0.5, 0.5, 0, 0, . . . , 0], p5k+2 =[0, 0, 0.5, 0.5, 0, . . . , 0], . . . such that each client has different classes.
• Non-IID #2 All clients have data for classes 0–4, but
only one client has data for classes 5–9. Specifically,

TABLE IDATA VARIANCE USED IN THE EXPERIMENT.

IID
Non-IID #1
Non-IID #2
Non-IID #3
Non-IID #4

U5n+1

U5n+2

U5n+3

U5n+4

U5n+5

0−9
0, 1
0 − 4, 50, 1, 2, 3

0−9
2, 3
0 − 4, 60, 4, 5, 6

0−9
4, 5
0 − 4, 7
1, 4, 7, 8i (Ui )

0−96, 7
0 − 4, 8
2, 5, 7, 9

0−9
8, 9
0 − 4, 9
3, 6, 8, 9

TABLE IIN EURAL NETWORK MODELS USED IN EXPERIMENTS . ( SIZE IS CAPACITY
OF PARAMETERS IN THE MODEL )Network

Number of parameters

Size (MiB)

ResNet-18
DenseNet-121
MobileNet V3 (small)11, 689, 512
7, 978, 856
2, 542, 856

49.03
33.47
10.66

p5k+1 = [0.1, 0.1, . . . , 0.1, 0.5, 0, . . . , 0], p5k+2 =[0.1, 0.1, . . . , 0.1, 0, 0.5, 0, . . . , 0], . . . , p5k+5=[0.1, 0.1, . . . , 0.1, 0, . . . , 0, 0.5] such that each client
has different classes.
• Non-IID #3 Each client has data for 4 classes, and eachclass is owned by two different client groups i.e.,(
0.25 (class included in private client)pk =0(otherwise)
The classes owned by each client node are shown in TableI.
• Non-IID #4 Each client has data for only one
class, assuming a stricter situation than previous studies. Specifically, p10k+1 = [1, 0, . . . , 0], p10k+2 =[0, 1, 0, . . . , 0], . . . , p10k+10 = [0, . . . , 0, 1] such that eachclient owns a different class.
The theoretical values of the accuracy of the models heldby each node trained using only PD are expected to be(IID, Non-IID #1, Non-IID #2, Non-IID #3, Non-IID #4) =(1, 0.2, 0.6, 0.4, 0.1).
B. Models
We use the Deep Residual Network (ResNet) [14] as themain client model for this experiment. For experiments using
different models, we utilize the Densely-connected Convolutional Networks (DenseNet) [15] and MobileNet V3 (small)[16]. The former is used for comparison with DLAD, while thelatter is used to investigate the impact of using a lightweightand simple model in conjunction with ResNet-18. The sizes of
the three models used in this experiment are shown in TableII.
In each model, pre-trained weights were employed, followed
by 200 epochs with a learning rate of 5 × 10−6 using SGD
Optimizer, and a mini-batch of size 40 for the local round.
In addition, only one epoch of learning is performed at eachstep in the gossip phase. The learning rate and mini-batch size

Proc. IEEE Virtual Conference on Communications(IEEE VCC 2023), November 2023

Data Size

TABLE IIID ETAILS OF THE MACHINE USED IN EXPERIMENTS .

50

Machine specification

Model of ResNet-18
Model of DenseNet-121
Model ofMobileNet v3(small)

」

10.66

10

Size (MiB)MNISTCIFAR-10
CINIC-10

3.46
2.88
5.18

were set to the same conditions as those used for training the
local model.
C. Experiment setup
The specification of the machine used in all the following
experiments is shown in Table III. A computer network andcommunication over it are simulated on the machine, but deeplearning actually took place on the machine.
D. Experimemt details
In this experiment, we will first observe the accuracy under
the same conditions as DLAD and verify the differences. Next,
as explained above, we will conduct experiments combining
ResNet-18, DenseNet-121, and MobileNet V3 (small), respectively, to compare and verify the accuracy. Consequently, we
will also measure the size of ILi transmitted in the gossip
phase. This will allow us to compare the size of the ILitransmitted with methods such as Federated Learning, whichtransmits the weights of the model.
E. Results
The proposed method reduced the size of data transmitted
between nodes. Table IV shows the sizes of Distillation
Datasets, that is transmitted between nodes at each step in thegossip phase. They are much smaller than the sizes of models
transmitted in the existing techniques shown in Table VI. For
CINIC-10, only 5.18 MiB of inference results are transmitted
between nodes in the proposed method though the existing
methods require 49.03 MiB of ResNet-18 as the main model
to be transmitted. Figure 1 summarizes the sizes of Distillation
Datasets and the sizes of models.
The proposed method achieved comparable accuracy withexisting centralized methods. Table V and Figure 2 show
the classification accuracy when training ResNet-18 on eachdataset and method. The values represent the median accuracy
on 10 nodes. The higher values in DLAD and Gossip Distillation are bolded. First, the overall accuracy of each model inthe gossip phase is higher than the accuracy of each model inthe local phase in all experiments, indicating that distillation
makes sense. Comparing the accuracy in the gossip phase with゜

Dataset

33.47

0
2

TABLE IVS IZE OF D ISTILLATION DATASET TRANSMITTED BETWEEN NODES AT
EACH STEP IN THE GOSSIP PHASE ( TO BE COMPARED WITH TABLE II).

——

49.03

0
3

Ubuntu 20.04.2 LTSIntel Xeon Platinum 8368 Processor (38 core, 2.4 GHz) × 2
NVIDIA Tesla A100-PCIE-40GBea
s e:i.
8三 ）az,
(

OS
CPU
GPU

40

I一

Distillation Dataset forMNISTDistillation Dataset for CIFAR-10
Distillation Dataset for CINIC-10

3.46

2.88

5.18

Fig. 1. Size of data transmitted between nodes at each step in the gossip
phase.

that of DLAD and the case where all data are in one place(labeled), we found that the accuracy in the gossip phase isalmost equal to, and sometimes exceeds, that of DLAD, andcan even compete well with labeled.
In the local phase, the accuracy was low, due to the small
number of data owned by the PDs, but the accuracy of eachmodel with respect to the PDs was high regardless of IID or
non-IID (>99% in MNIST and >75% in CINIC-10). This fact
may be the reason for the significant improvement in accuracy
for the test data in the gossip phase, soon after the start ofthe information transfer round. In addition, a comparison of
the accuracy of the Non-IID4 pattern in the same data set
shows that the results are almost the same. From this result, it
can be inferred that all knowledge is spread over the rounds,
regardless of the PD owned by each user. Figure 3 plots theaccuracy at the end of the local phase and the accuracy in thegossip phase for the IID, Non-IID #1∼3 of CIFAR-10 in Table
V. The graphs show that the accuracy increases in the earlyrounds, indicating the significance of Gossip Distillation.
The proposed method enabled combination of multiple
different models. Table VI shows show achieved accuracy
in case combining multiple different models. In this case,
ResNet-18 and other neural networks were trained with 5
nodes each. Table VI shows the median accuracy of each of thefive nodes. Other datasets also show similar or slightly higher
accuracy than ResNet-18 alone. The results using MobileNet
V3 (small) also show the same level of accuracy as thoseobtained using ResNet-18 alone. The results with MobileNetalso show that Gossip Distillation is also accurate, although
the accuracy is lower than that of ResNet. The results of theexperiment using the same conditions, except that 10 models
were replaced by 10 MobileNets, showed an accuracy of about
0.47, indicating that the same level of accuracy can be achievedby combining multiple models. This indicates that Gossip
Distillation can be performed by combining different models.
V. C ONCLUSION
In this study, we applied Knowledge Discovery to communication in decentralized deep learning. The proposed method
reduced the size of data transmitted between nodes by several

Proc. IEEE Virtual Conference on Communications(IEEE VCC 2023), November 2023

TABLE V
ACCURACY. ( MODELS ARE ALL R ES N ET-18, n = 10)
DatasetDistribution

IID

NIID1

MNIST
NIID2
NIID3

NIID4

IID

NIID1

CIFAR-10
NIID2
NIID3

NIID4

IID

NIID1

CINIC-10
NIID2
NIID3

DLAD [6]labeled [6]0.9821

0.98200.98280.98400.73140.66570.68470.7027

0.63230.6266

0.5666

0.59340.9836

0.98680.98450.98570.71150.81270.75760.77550.6256

0.68800.61830.6574

Gossip Distillation(local phase)

0.70890.18610.46980.35040.38700.16690.26930.27980.3596

0.15920.25360.2609Gossip Distillation(gossip phase)

0.96440.96470.96690.96800.71870.71590.71810.7226

0.59350.6096

0.59730.6064

0.97150.7280NIID4

0.6157

Fig. 2. Accuracy achieved at the end of local and gossip phases, compared with DLAD and the case where all data are in one place (labeled).

TABLE VI
ACCURACY FOR THE CASE WHERE MULTIPLE MODELS COMBINED (n = 10)
Main Model
Sub Model
DatasetNon-IID # 1
Res/other

ResNet-18
MNIST

DLAD [6]Gossip Distillation−/−
0.9709 / 0.9781

DenseNet-121
CIFAR-100.6657 / 0.6642
0.7279 / 0.7403

CINIC-10

MNIST

MobileNet V3(small)
CIFAR-10

CINIC-10−/−0.6096 / 0.6180−/−
0.9718 / 0.9540−/−
0.7265 / 0.5550−/−
0.6077 / 0.4796

Proc. IEEE Virtual Conference on Communications(IEEE VCC 2023), November 2023

Fig. 3. Accuracy for each round in the gossip phase.

times. It also enabled combination of various models depending the performance of the learning devices. Experimental
results showed that the proposed method can achieve accuracy
comparable to that of existing centralized methods.
Future work includes experiments on more realistic networks and large number of devices. Enabling trustless distributed learning is a possible direction because it is difficult
to establish trust with such a large number of devices.work
ACKNOWLEDGMENTS
We would like to express our gratitude to Dr. Ryo Yonetani,for discussion.
R EFERENCES[1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B.A. y Arcas,“Communication-efficient learning of deep networks from decentralized
data,” Proc. AISTATS, Fort Lauderdale, Florida, USA, April 2017.
[2] Rauniyar, A., “Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directions,”
arXiv e-prints, 2022, doi:10.48550/arXiv.2208.03392.
[3] D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li
and H. Vincent Poor,“Federated Learning for Internet of Things:
A Comprehensive Survey,” in IEEE Communications Surveys &Tutorials, vol. 23, no. 3, pp. 1622-1658, thirdquarter 2021, doi:
10.1109/COMST.2021.3075439.
[4] Y. Qu et al., “Decentralized Privacy Using Blockchain-Enabled
Federated Learning in Fog Computing,” in IEEE Internet ofThings Journal, vol. 7, no. 6, pp. 5171–5183, June 2020, doi:
10.1109/JIOT.2020.2977383.
[5] Hinton, G., Vinyals, O., and Dean, J., “Distilling the Knowledge in a
Neural Network,” in NIPS Deep Learning and Representation Learning
Workshop, 2015.

[6] J. Ma, R. Yonetani and Z. Iqbal, “Adaptive Distillation for Decentralized
Learning from Heterogeneous Clients,” in 2020 25th International
Conference on Pattern Recognition (ICPR), Milan, Italy, 2021 pp. 74867492.
[7] S. Maarten van and T. Andrew S., “Distributed Systems Third edition,”
2020.
[8] O. Róbert , H. István and J. Márk, “Gossip Learning with Linear Models
on Fully Distributed Data,” Concurrency and Computation Practice andExperience, 2013, 25, 10.1002/cpe.2858.
[9] Y. Takahashi and K. Shudo, “Unbiased machine learning methods fordata on P2P networks,” DEIM Forum 2017, 2017.
[10] H. Oguni and K. Shudo, “Addressing the Heterogeneity of A Wide Area
Network for DNNs,” Proc. IEEE CCNC 2021, Las Vegas, NV, USA,2021, pp. 1-6, doi: 10.1109/CCNC49032.2021.9369585.
[11] H. Oguni and K. Shudo, “Communication Scheduling for Gossip SGD
in a Wide Area Network,” IEEE Access, Vol.9, pp.77873–77881, doi:
10.1109/ACCESS.2021.3083639.
[12] Hu, C., Jiang, J., and Wang, Z., “Decentralized Federated
Learning: A Segmented Gossip Approach”, arXiv e-prints, 2019.
doi:10.48550/arXiv.1908.07782.
[13] Darlow, L. N., Crowley, E. J., Antoniou, A., and Storkey, A. J.,“CINIC-10 is not ImageNet or CIFAR-10”, arXiv e-prints, 2018.
doi:10.48550/arXiv.1810.03505.
[14] K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image
Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), Las Vegas, NV, USA, 2016 pp. 770-778.
[15] G. Huang, Z. Liu, L. Van Der Maaten and K. Q. Weinberger, “Densely
Connected Convolutional Networks,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017,
pp. 2261-2269, doi: 10.1109/CVPR.2017.243.
[16] A. Howard et al., “Searching for MobileNetV3,” 2019 IEEE/CVF
International Conference on Computer Vision (ICCV), Seoul, Korea(South), 2019, pp. 1314-1324, doi: 10.1109/ICCV.2019.00140.

Proc. IEEE Virtual Conference on Communications(IEEE VCC 2023), November 2023