CoDeC: Communication-Efficient Decentralized Continual Learning
Authors: Sakshi Choudhary
Sai Aparna Aketi
Gobinda Saha
Kaushik Roy
Venue: Accepted by TMLR ()

Abstract: Training at the edge utilizes continuously evolving data generated at different locations. Privacy concerns prohibit the co-location of this spatially as well as temporally distributed data, deeming it crucial to design training algorithms that enable efficient continual learning
over decentralized private data. Decentralized learning allows serverless training with spatially distributed data. A fundamental barrier in such setups is the high bandwidth cost of communicating model updates between agents. Moreover, existing works under this training paradigm are not inherently suitable for learning a temporal sequence of tasks while retaining the previously acquired knowledge. In this work, we propose CoDeC, a novel communication-efficient decentralized continual learning algorithm that addresses these challenges. We mitigate catastrophic forgetting while learning a distributed task sequence by incorporating orthogonal gradient projection within a gossip-based decentralized learning algorithm. Further, CoDeC includes a novel lossless communication compression scheme based on the gradient subspaces. We theoretically analyze the convergence rate for our algorithm and demonstrate through an extensive set of experiments that CoDeC successfully learns distributed continual tasks with minimal forgetting. The proposed compression scheme results in up to 4.8× reduction in communication costs without any loss in performance.

Published in Transactions on Machine Learning Research (03/2024)
CoDeC: Communication-Efiiien Decentralized d Continual Learning
Sakshi Choudhary
choudh238purdue.
Department of Electrical and Computer Engineering
Purdue Universily
Sai Aparna Aketi
saketi edu
Department Electrical and Computer Engineering Purdue University
Gobinda Saha
gsaha@purdue.. edu
Department of Electrical and Computer Engineering Purdue University
Kaushik Roy
kaushikk Spurdue edu
Department of Electrical and Computer Engineering
Purdue Universily
Reviewed on OpenReview: https: // openreview. net/ form?? id= N050mOC1BA
Abstract
Training at the edge utilizes continuously evolving data generated at different locations. Privacy concerns prohibit the co-location of this spatially as well as temporally distributed data, deeming it crucial to design training algorithms that enable efficient continual learn- ing over decentralized private data. Decentralized learning allows serverless training with spatially distributed data. A fundamental barrier such sctups the high bandwidth cost of communicating model updates between agents. Moreover, existing works under this training paradigm not inherently suitable for learning a temporal sequence of tasks while retaining the previously acquired knowledge. In this work, propose CoDeC, novel communictiion-effiien decentralized continual learning algorithm that addresses these chal- lenges. We mitigate catastrophic forgetting while learning a distributed task sequence by incorporating orthogonal gradient projection within a gossip-based decentralized learning algorithm. Further, CoDeC includes a novel lossless communication compression scheme based on the gradient subspaces. We theoretically analyze the convergence rate for our algorithm and demonstrate through extensive set of experiments that CoDeC success- fully learns distribuled continual tasks with minimal forgetting. The proposed compression scheme results in to 4.8x reduction in communication costs without any loss in perfor- mance.
1 Introduction
Deep neural networks have demonstrated exceptional performance for many visual recognition tasks over the past decade. This has been fueled by the explosive growth of available training data. and powerful comput- ing resources. Edge devices such smartphones, drones, and Internet-of-Thine Things (IoT) sensors contribute towards generating this massive amount of data (Sh et al.] 2020). Interestingly, this data is spatially dis- tributed, while continuously evolving over time. Large-scale deep neural network training has traditionally relied upon the availability of humongous amount of data at a central server. This mainly poses three
The Torch implementation can found httpe://ggttubb..cm//aaaaat

---

Published in Transactions on Machine Learning Research (03/2024)
Data
Data
Data
Agent 1
1
GPM
B. RR
GPM1
RGS
Forth 9t
Agent 9,
CGS
GPM2
CoDeC
Agent
godd
GPM
GPM 2
GPM
Gold
Time
doo
Task 1
Task 2
T
Agent 2
Agent 3
(a)
(b)
(c)
An overview of CoDeC. (a) Data for each incoming task is independently and identically distributed (IID) over the decentralized agents. Each agent has a GPM (Gradient Projection Memory) which updatod after learning each (ask. (b) Based the sparse graph topology, the agents communicate coelficients associated with the model updates at each training iteration. (c) GPM partitions each layer's subspace into two orthogonal subspaces.
Figure 1:
challenges: (1) high network bandwidth requirements to collect this dispersed data from numerous learning agents, (2) data privacy concerns for locally-generatee data accessed by the central server and (3) adapting to changing data distributions without expensive training from scratch. This motivates the need for learning algorithms to enable efficient distributed training by utilizing spatially and temporally distributed data.
Centralized distributed learning (i.e. federated learning) has emerged to train models over spatially dis- tributed data without compromising on user privacy Konečný et al. (2016). This approach relies upon a central parameter server to collect local model updates, process, and send the global updates back to the agents. However, the central server may lead to a single point of failure and network bandwidth issues (As- sran et al.] 2019). To address these concerns, several docentralized learning algorithms have been developed BBancchi et al. 2013 Lan et al.) 2017 Lian et al., 2017 Assran et al. 2019). Decentralized learning is a pecr-to-pecr learning paradigm, where agents communicate only with their neighbors without the need for central server. Each agent learns global generalized model by aggregating locally computed model updates shared by neighbors. However, decentralized learning algorithms are not inherently equipped to thrive in dynamic learning environments with a temporal sequence of changing data distributions.
Gradient-based optimization methods like plain SGD and DPSGD (Lian et al. 2017) inherently update model parameters by minimizing the loss function with respect to the current data distribution. This results in overwriting of parameters learned for the previous task(s), leading to catastrophic forgetting (Mecloskey
& Cohen 1989 Ratcliff 1990). Hence, continual learning techniques focus on learning consecutive tasks
without lorgetting the past acquired knowledge (Lee el. al.
2017 Lopez- Paz & Ranzato,
2017 Saha et al.
2021a Mallya et al. 2018 Kirkpatrick et al. 2017 Farajtabar et al. 2020 Wang et al. 2021 Saha et al.
2021
Table summmrrzes previous works that address the challenges of learning with spatially and/or temporally
distributed data. In this paper, we propose CoDeC to enable serverless training with data distributed across space as well time. To the best of our knowledge, this is the first work that demonstrates such decentralized continual learning setup. Our algorithm has three components: (1) SGD combined with gossip averaging (iiao & Boyd 2003) shown in (Lian et al. 2017) to learn with spatially distributed private (2)a, Gradient Projection Memory (GPM) (Saha et al. 2021 to continually learn a temporal task sequence with minimal forgetting and (3) a novel lossless communication compression scheme to reduce the bandwidth requirements during training. Our setup illustrated figure
GPM partitions each layer's gradient space into two orthogonal subspaces: Core Gradient Space (CGS) and Residual Gradient Space (RGS) shown in [c). Important gradient directions (CGS) for previous tasks
2

---

Published in Transactions on Machine Learning Research (03/2024)
Table 1: Comparison with prior works Continual Learning (CL), Federated Learning (FL) and Decentral- ized Learning (DL) paradigm. Note that FL techniques require server, while DL techniques are serverless. (t) denotes decentralized agents independently learning different tasks given time, and communicating via fully-connected topology.
Technique
CL
FL
DL
FedOpt (Končnný et al. 2016
DPSGD LLan et al. 2017
EWC
Kirkkatrickk ct a.. 2017
GPM Saha et al. 2021
FodWeI1
Yoon et al. 2021
FLwF-2T
Usmanova
2021
DCIL
Zhang et al. 2022
SKILL [Ge et al. CoueC(ouur)
2023
are stored in gradient projection memory (GPM), and gradient updates for the new tasks are taken along RGS to minimize interference. We find the basis vectors that span RGS and represent model updates as a linear combination these vectors. We communicate the coefficients associated with these basis vectors instead of the model updates and achieve lossless communication compression. Further, theoretical insights into the convergence rate of CoDeC prove that it is possible to achieve similar rates the stato-of-the- art decentralized learning approaches such as DPSGD (Lian et al. 2017). We provide empirical evidence by performing experiments over various standard image-classification datasets, networks, graph sizes, and topologies. We also extend our analysis by designing and evaluating a decentralized continual learning benchmark MedMNIST-5 using biomedical imago-clasiffiatton datasets from MedMNIST-v2 (Yang et al. 2021). This imitates a practical real-world application where multiple health organizations aim learn global generalized model without sharing the locally accessible patients' data. These models need to be updated with the emergence of variants of disease, new diseases, or new diagnostic methods in continual manner.
Contributions: We summarize our contributions as follows:
We propose CoDeC, a commuicatioon-fficcatt docentralized continual lcarning algorithm that ad- dresses a challenging problem: leveraging spatially and temporally distributed data optimize a global model while preserving data privacy.
We introduce novel lossless communication compression scheme based gradient subspaces.
We theoretically show that our algorithm converges at the rate of O(1/VNK), where N is the number of agents and K is the number of training iterations.
Experiments over a variety of image-classffcattion datasets, networks, graph sizes, and topolo- gics demonstrate minimal forgetting and up to 4.8x reduction in commuicatiin costs with iso- performance relative to the full communication baseline.
2
Related Work
2.1 Decentralized Learning
Several works exist in the docentralized learning paradigm which enable distributing training without utilizing a central server (Bianchi et al.) 2013 Lan el al.) 2017 Lian et al. 20177 Assran et al. 2019 Balu el al. 2021). DPSGD [Lian et al.) 2017) provides theoretical analysis for the convergence rate of decentralized learning algorithms, proving it be similar to their centralized counterpart (Dean al. 2012)) CoDeC, we utilize DPSGD (Lian al. 2017) and modify it to send model updates instead of model parameters. Note, these existing works are not. equipped to learn temporal task sequence without forgetting past knowledge.
3

---

Published in Transactions on Machine Learning Research (03/2024)
To reduce the communication overhead for decentralized learning, several compression techniques
Koloskova
et al.
2019 Tang et al. 2019: Aketi et. al. 2021 have been explored. DeepSqueeze (Tang et al. 2019] in-
troduced error- coorencomeee communication compression to decentralized training. Choco-SGD Koloskova
ct al.
2019) communicates compressed model updates rather than paramcters and achieves better perfor- mance than Tang et al. (201)). However, it is orthogonal to the compression scheme we present and can be
used synergy with our approach. Moreover, all of the above-mentioned compression techniques are lossy and require additional hyperparameter tuning.
2.2 Continual Learning
The majority of continual learning works fall into three categories (Wickramasinghe et al., 2023): network expansion, replay and regularization- based methods. Network expansion based methods (Rusu et al. 2016; Lee et al. 2017) overcome catastrophic forgetting by dedicating different model parameters to each task. Replay-based methods store training samples from the past tasks in the memory or synthesize old data from generative models for rehearsal (Chaudhry et al., 2019 Rebuffi et al. 2017 Shin et al. (017)) Regularization-
based methods penalize changes to parameters KKrrkatrrick et al.) 2017 Zenke et al., 2017). or constrain gradient directions (Saha et. al. 2021 Wang et al.) 2021 Saha & Roy 2023) important for previous tasks. These methods rely on the availability of centrally located training data and hence fail to be directly ap- plicable to a distributed learning scenario. Network expansion based methods in a decentralized continual learning setup may give rise to model heterogeneity across agents over time, while replay-based methods can lead to privacy concerns. Thus, we explore regularization based methods like GPM (Saha et al., 2021), SGP (Saha & Roy 2023), EWC (Kirkpatrick et al.] 2017) and SI (Zenke et al. 2017) in this work. We utilize GPM in CoDeC and show superior performance than D-EWC and D-SI, decentralized continual learning baselines we implemented with EWC and S1 respectively. We further extend CoDeC to incorporate scaled gradient updates as shown SGP.
2.3 Distributed Continual Learning
FedWelT YYon al. 2021 tackled the problem of federated continual learning through the decomposition of model parameters at cach client into global and local task-adaptive parameters. FLwF-2T (Usmanoval eet al. 2021) developed a distillation-based method for class-incremental federated continual learning. Unlike serverless training setup, these works utilize central server to aggregate and send global updates to the agents. CoLLA (Rostami et al.] 2017] focused on multi-agent distributed lifelong learning and proposed a distributed optimization algorithm for a network of synhhroouus learning agents. Note that it uses parametric models and is not directly applicable to modern deep neural networks. SKILL (Ge et al., 2023) proposes a distributed lifelong learning mechanism, where each agent uses a common pre-trained backbone and learns a task-specific head module. After training, these task-specific heads are shared among agents via a fully connected graph. However, each SKILI agent is independently learning different task al a given time, lacking the concept of collaborative learning demonstrated in CoDeC.
3 Methodology
3.1 Problem Formulation
In this work, we optimize a DNN model to learn from spatially and temporally distributod data. The communication topology is modeled as graph G ([N], W), where N is the number of learning agents and W is the mixing matrix indicating the graph's connectivity. wy encodes the effect of agent j agent i, and wij- = 0 implics there is no direct communication link between the two agents.
Consider learning scenario where T tasks are learned sequentially. Now, for any task T E {1,.., the corresponding dataset D, is independently and identically distributed (II) across the N agents as (Dr,1, D Dr.2: Dr.3....D,.N' For each r, we aim to minimize the global loss function Fx) given equation] 1 Here, Fr, (dr,,, x) is the local loss function per task at agent and fx) is the expected value of (r,(dr,,, over the dataset Dr..-
4

---

Published in Transactions on Machine Learning Research (03/2024)
1 N
min Fr(x): - N ==1 xER
Σ (x)(
(1)
where fT,(x)= = Ed (Fr, Vi
Decentralized optimization of this global loss function (x) is based on the current dataset D,. crucial challenge is to optimize (x) such that the past information acquired from tasks 2, (r-1) retained.
Inspired by Saha et al. (2021), we define a subspace that contains important gradient directions associated with all the past tasks and modify the local gradient updates of the current task to be orthogonal to this subspace L.e, to lie in RGS. This ensures minimal catastrophic forgetting.
Typically, decentralized agents communicate the model parameters with their neighbors in each training iteration (Lian et al.) 2017]. Note that in the proposed algorithm the model updates lie in RGS, which is a smaller vector subspace compared to the entire gradient space. To utilize this property for enabling lossless communication compression, we communicate model updates with neighbors similar to Koloskova et al. 2019) rather than the model parameters.
3.2 Approach
We demonstrate the flow of CoDeC in Algorithm I All hyperparameters are synchronized between the agents at the beginning of the training. Fach agent. i computes the gradient update g' (Vf=x(d,;xx)) with respect to model parameters x", evaluated on mini-batch dz, We obtain g', the orthogonal projection of the local gradients using GPM memory M (line 6, algorithm D. The parameters of each agent are updated using this g' which ensures minimal forgetting. Then, each agent performs a gossip averaging step using x' and X (line 8, algorithm D X represent the copics of x² maintained by all the neighbors of agent j and in general x = x5. The computed model updates (denoted by q²) lie in the RGS subepace spanned by the basis vectors contained in O'. Therefore, we express them as a linear combination of these basis vectors and find the associated coefficients, c' to communicate with the neighbors (line 10, algorithm ( Upon receiving these coofficients, the agents reconstruct the neighbors' updates without any loss in information (line 13, algorithm 1p. Communicating the coefficients (ct) leads to lossless compression, which we elaborate upon in section 3.3 The local copy & is updated using the reconstructed model updates q² (line 14, algorithm Note that our algorithm requires each agent to only store the sum of neighbors' models EjeN(e) Wiy resulting in 0(1) memory overhead, independent of the number of neighbors.
At the end of each task, important gradient directions are obtained using a Singular Value Decomposition (SVD) representation of the input activations of each layer (Saha et al.J (2021). These gradient directions are added as basis vectors to the CGS matrix A and subsequently removed from the RGS Matrix O. SVD is caleulated using a subset of training data at any randomly chosen agent and communicated to other agents iteratively using the communication graph.
3.3 Lossless Compression
Stochastic Gradient Descent (SGD) updates lie in the span of input data points ZZhng et al. 2017 Lever- aging this fact, GPM (Saha et al. 2021) performs SVD on a representation matrix R, and finds basis vectors corresponding to the important gradient directions for previous tasks. R is constructed by performing forward pass of n, samples from the training dataset for task through the network and concatenating the input activations for each layer I (equation 2). Subsequently, the SVD of representation, Rt in equation 2 is used to obtain the matrix U containing a set of orthonormal basis vectors which span the entire gradient space.
= R = [2 22, 2
(2)
SV SVD(R)) = UVE(V
The threshold hyperparameter 4h determines the number of basis vectors chosen from U represent important gradient directions for any particular task. These vectors span a subspace in the gradient space which we define the Core Gradient Space (CGS). They added to the GPM matrix M = [M))
5

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm crtmmmuiication-ffciieee Decentralized Continual Learning (CoDeC)
Input: Each agent i E N] initializes model parameters x6, step size 7/ mixing matrix W = [wa]oyepLN)) 8/0) - x
M [] and O' [T] for all layers 1 = 1, .,.,, GPM Memory M ((M)).) RGS Matrix 0
(1): neighbors agent (including itself), T: total tasks, K: training iterations
Each agent simultaneously implements the TRAIN() procedure
1. procedure TRAIN()
2. for r = ,,... T do
3. for 0, ,,..., K 1
4.
dz D
5.
gi Vfaa(dr:;i
6.
E ge - -(M' -MM' (MM)) # for each layer
7.
x }) =x - n8k
8.
x x -ESex(o wis(x} x()
9.
q x -x²
c = (0')" qi
11.
for each E (i) do
Send ci and receive oj
13.
q (0 c
14.
8 (k+1) qe x
15.
end
16.
end
# GPM Update
17.
random 1.2....N) if = p do
18.
19.
Update M', 0' for each layer I E L
20.
Update M {(M'
21.
Update O {(0')==}}
Send M, to all agents
23.
end
end
25.
return
For task 7 > 1, we ensure that the CGS vectors being added to the GPM matrix in round 7 are orthogonal to all the CGS vectors stored in M. Before performing SVD on the representation matrix R for each layer we perform the following projection step:
= -M'(M)) R?
SVD is then performed on RC and new ort hogonal basis vectors are added to M. This ensures that the newly added basis vectors for task + are unique and orthogonal to the vectors already present in M.
The following update rule used to obtain orthogonal gradient update g for the later tasks:
g' = --M''M''M'))
(4)
Here, g' is the original local gradient update at agent i at layer I, and the projection of g' on CGS is (M'M' Jg Let the input space for layer be of dimension ny. This implies that U contains na orthonormal basis vectors. Now based esh, after every task, set of ri basis vectors corresponding the top r singular values are stored in M. Hence, g lies in (m -r)) dimensional orthogonal subspace denoted as the Residual Gradient Space (RGS).
The basis vectors which span RGS are the remaining (n (ra--1) vectors contained in U.. We store them in the RGS Matrix O (())) ny r n, and r, incrcases as the task sequence progresses. We note that the gradient updates tend to lie in smaller subspace (i.c. RGS) whose dimensionality decreases based on fsh and the number of tasks.
6

---

Published in Transactions on Machine Learning Research (03/2024)
In algorithm model updates q are computed at every training iteration k. Since all the local gradients Bi lie in RGS, the updates qi also lie in RGS (refer to A.6 for the proof). Therefore, we express layer-wise qi as a linear combination of the basis vectors in O' and find the associated coefficients cj. The neighbors of agent i reconstruct the updates qi from the received c This encoding and decoding of qj require two additional matrix multiplications (lines 10 and 13 in algorithm D. Our approach ensures that all agents have the same M and O matrices so that the reconstruction is exact. Hence, we achieve lossless communication compression by the virtue of taking orthogonal gradient updates avoid catastrophic forgetting.
4 Convergence Rate Analysis
In this section, provide convergence analysis for our algorithm. In particular, provide an upper bound for [ 11 where VF (xx) is the average gradient achieved by the averaged model across all agents, Since our claims are valid for each task, the task subscript is dropped for the following analysis. We make the following assumptions:
Assumption Lipschitz Gradients: Each function fi(x) is T-smooth.
Assumption 2 Bounded Variance: The variance of the stochastic gradients is assumed to be bounded. There exist constants and ō such that
EdDD, [VF d) Vx((()]| < o²
(5)
N Vf((x) - VF(())I < Vi,I
(6)
Assumption 3 - Doubly Stochastic Mixing Matrix: W is real doubly stochastic matrix with X (W) and max IA s vp < 1, where (W) is the 8th largest cigenvalue of W and p constant.
The above assumptions are commonly used in most decentralized learning works (Lian et al. 2017 Tang
et al.] 2019; Esfandiari al. 2021 Since we modify the original gradient update g', we introduce an additional assumption:
Assumption 4 Bounded Orthogonal Updates: For each agent have:
g S Algg'1|
(7)
where μ (0, 1] signifies how constrained the gradient space is. In particular, μ encapsulates the average impact of the dimension of RGS subspace during training.
To ensure that the gradient update after projection is in the descent direction, we provide the following lemma:
Lemma 1. Given the original gradient update -g' in the descent direction, the orthogonal gradient update -g' is also in the descent direction.
Please refer to Appendix A.1 for the proof. Theorem 2 presents the convergence of CoDeC (proof in Appendix
A.3
Theorem 2.
Given assumptions 1-4, let step size satisfy the following condition:
1
V(I - 2 + 12 - (1 vP)
(8)
T, <
6142
For all K 2 1, we have
Σ E [VFF
1
(xF
(Xo) -
CKK
(9)
2 C₂ N
+ Ca 2211 ( (1- vp)²
(1-
30
382
+
(1
3p)²
where C = - t), C2 Tnp 201, C3 1.21/221.
7

---

Published in Transactions on Machine Learning Research (03/2024)
The result of theorem 2 shows that the norm of the average gradient achieved by the consensus model is upper-bounded by the suboptimality gap (F (Xn) F) the sampling variance (o), the gradient variations (6), and the constraint on the gradient space (n). The suboptimality gap signifies how good the model initialization is. a indicates the variation in gradients due to stochasticity, while is related to gradient variations across the agents. From equation g we observe that μ appears in the last term and effectively scales a and A detailed explanation of the constraints on step size 1 presented in Appendix A.4. We present corollary show the convergence rate of CoDeC in terms of the training iterations. Note that we denote an Ob,,) if on S cb,, where > 0 is a constant.
Corollary 3. Suppose that the step size satisfies 1 0 B (x)))) so 5O(VRR + R K
For sufficiently large K and some constant N
C>0, 0,
K K
(10)
The proof for Corollary 3 is detailed in Appendix A.5. It indicates that CoDeC achieves a convergence rate of O VNK for each task. This rate is similar to the well-known best result in decentralized SGD algorithms [Lian et al. 2017). Since n² appears only in the higher order term Ks it does not affect the order of the convergence rate.
5 Experimental Setup
For cach task, the data distribution is IID across the agents. The agents communicate with their neighbors after every mini-batch update. We present results for different graph topologies and sizes: directed ring with 4/8/16 agents and undirected torus with 8/16 agents. In the directed ring topology, each agent has only 1 neighbor. Meanwhile, the torus topology has higher connectivity, with 3 and 4 neighbors for graph sizes of 8 and 16 agents respectively. We evaluate CoDeC on three well-known continual learning benchmark datasets: 10-Split 100 (Krizhevsky) 2009), 20-Split. MinilmageNet (Vinyals et al. 2016) and a sequence of 5-Datasets (Ebrahimi et | 2000)). 10-Split CIFAR-100 is constructed by splitting CIFAR-100 into 10 tasks, where each task comprises of 10 classes. We use a 5-layer AlexNet for experiments with Split CIFAR-100. 20-Split minilmageNNet has 20 sequential tasks, where each task comprises classes. The sequence of 5-Datasets includes CTFAR-10, MNIST, SVHN (Netzer et al. 2011) notMNIST (Bulatov 2011) and Fashion MNIST (Xiao et al. 2017). For Split minilmageNet and 5-Datascts, we use reduced ResNet18 architect ure similar to LopLopezPPe Paz& & Ranzato (2117). We design MedMNIST-5, a biomedical decentralized continual learning benchmark based on the datasets in MedMNIST-v2 (Yang et al. 2021). McdMNIST-5 consists of the following sequential classification tasks: TissucMNIST, OrganAMNIST, OCTMNIST, PathMNIST, BloodMNIST. ResNet-18 architecture used to evaluate the performance. For all experiments, batch normalization parameters are learned for the first task and frozen for subsequent tasks. We use 'multi-head' setting, where each task has a separate final classifier with no constraints on gradient updates. Please refer to Appendix A.7 training hyperparameterss respectively.
Implementation details:
9 for details related to architectures, dataset statistics, and
Baselines: To the best of our knowledge, our proposed setup is unique and hence there are no directly comparable baselines. Therefore, for a fair comparison we implement two baselines D-EWC and D-SI based on well known continual learning techniques Elastic Weight Consolidation (EWC) (Kirkpatrick al. 2017) and Synaptic Intelligence (ST) (Zenke et al. 2017) respectively. Please refer algorithm 2 and 3 for the detailed implementation. At the end of each task, these methods compute statistics to penalize the pa- rameter updates to mitigate forgetting. D-EWC computes Fisher information matrix F,, while D-SI utilizes parameter specific contribution to changes in the total loss to compute importance measure Ω. To provide an upper bound on performance, we add two baselines: STL and D-STL. Single Task Learning (STL) rep- resents a setting where all the tasks are learned sequentially in a centralized setup without any constraints. Decentralized Single Task Learning (D-STL) baseline extends STL to a decentralized setting.
Performance Metrics: We mainly focus on these metries: (1) Average Accuracy (ACC): measures the average test. classification accuracy of all tasks (2) Backward Transfer (BWT): indicates the impact on
8

---

Published in Transactions on Machine Learning Research (03/2024)
the past knowledge after learning new tasks where negative BWT implics forgetting (3) Communication Compression (CC): measures the relative reduction in the communication cost achieved through our lossless compression scheme. ACC and BWT can be formally defined as:
T E
ACC = T
AT.: BWT = T Σ AT - Ai,i
(11)
=
Here, T is the total number of tasks and AT, the accuracy of the model on 7th task after learning T tasks sequentially.
6 Results and Discussions
Continual Learning Statistics Aggregation: We implement two versions for each technique based on how continual learning statistics are computed and consolidated at the end of each task: broadcast and all- gather. In broadcast, an agent is randomly chosen and the corresponding statistics are calculated and sent all the other agents. In all all--gaher, all agents compute these statistics using their local data, and the global average of these statistics is utilized to mitigate forgetting. Note that all-gather incurs more computational cost as compared to broadcast. D-SI and D-EWC give sub-optimal performance with broadcast, while CoDeC gives similar performance for both versions as shown in Table 2 (refer to Table 11 in Appendix for additional results). Hence, for results in Table broadcast for CoDeC.
and we choose all-gather for D-EWC and D-SI, and
Table 2: Impact of broadcast and all- all-gather for continual learning for Split minilmageNet over 8 agent directed ring.
Setup
ACC(%)
BWT(%)
D-ST (broadcast)
36.14 + 1.61
-12.90 + 1.15
D-S1 (all-gather)
45.58 1.21
-3.67 = 1.27
D-EWC (broadcast.)
37.20 1 0.58
0.27 0.15
D-EWC (all-gather)
46.39 + 1.54
-1.64 = 1.11
CoDeC(broadcst)
53.22 1.82
0.08
CoDeC (all-gather)
53:25 + 1.48
0.50 0.21
Performance Comparison: We present two versions of our approach: CoDeC, which uses the lossless compression scheme, and CfDeC(ff) an implementation with full communication. As shown in Table 3 for Split CIFAR-100 we obtain 3-4% better ACC than D-EWC with a similar order of BWT. CoDeC achieves 12-13% better ACC than D-SI, with marginally better BWT. Our proposed compression technique results in 1.86x reduction the communication cost on average without any performance degradation.
Table 4 demonstrates learning a longer task sequence Split MiniImageNet, and we outperform both D- EWC and D-SI by 6-11% in terms of ACC with better BWT in some cases. We achieve 1.43x reduction in communication cost on average over range of graph sizes. Learning dynamics of task 1 after learning each task in Split minilmageNet are shown in figure 2 Note that for D-SI, the training converges only for a lower learning rate as compared to D-EWC and CoDeC (details in Appendix A. Results on 5-Datasets demonstrate learning across diverse datasets. As shown in Table 5 although we report better BWT for D- EWC and D-SI in most cases, we achieve 0.5-5% better accuracy and upto 2.2x reduced communication cost. We believe that final average accuracy (ACC) and backward transfer (BWT) are both equally important in continual learning scenarios. It is possible to achieve zero BWT by freezing the model weights after learning the first task. However, that leads to sub-optimal ACC due to lack of ability to learn the subsequent tasks effectively. In our setup, we strive to achieve a balance between these two metrics by tuning the threshold hyperparameter 6th accordingly. It. is possible to use a higher 6x and achieve a lower BWT, but at the cost of lower ACC. Hence, Eth, can be tuned to achieve lower BWT or higher ACC as required.
Additionally, we present results on MedMNIST-5, decentralized continual learning benchmark we propose to imitate scenario where healthcare organizations aim to optimize global model while maintaining
9

---

Published in Transactions on Machine Learning Research (03/2024)
Table 3: Split CIFAR-100 over AlexNet using directed ring and torus. CL sctup and provide an upper-bound on the performance.
(*): methods that don't adhere to
Directed Ring
Torus
Agents
Setup
ACC(%)
BWT(%)
CC
ACC(%)
BWT(%)
CC
STL"
70.56 0.20
D-STL"
69.22
D-S1
44.90 = 0.10
-0.83
+ 0.64
1x
D-EWC
53.12
0.24
0.18
1x
4
CoDeC(f)
57.54 + 0.25
-1.22
1 0.22
1x
CoDeC
57.83
-0.95
0.05
1.86x
D-STL"
64.99 0.41
65.17 1 0.44
D-SI
39.54 0.16
-1.08 1: 0.85
1x
39.36 i 0.40
-1.25
1: 0.65
D-EWC
50.52 0.58
0.51 0.09
1x
49.41 0.88
0.29
0.27
1x
8
CoDeC(f)
53.57 = 0.38
-0.65 + 0.52
1x
53.54 + 0.35
-1.15 + 0.41
1x
CoDeC
53.63
-0.43 + 1.85x
53.62
-0.64
1.86x
D-STL.
58.31 = 0.49
59.29 + 0.12
D-SI
34.66 1.15
0.4
1x
34.86 0.68
-1.66
1x
D-EWC
45.52 0.00
0.22 = 0.34
1x
44.53 = 0.77
-0.20 + 0.56
1x
16
CoDeC(I)
48.05 0.45
-0.38
1 0.12
1x
48.19 1 0.27
-0.29
0.11
1x
CoDeC
48.16 0.33
-0.18 1 0.28
1.84x
48.36 0.04
-0.26
0.31
1.84x
Table 4: Split Minilmagenet over ResNet-11 using directed ring and torus. (*): methods that don't adhere
to CL setup.
Directed Ring
Torus
Agents
Setup
ACC(%)
BWT(%)
CC
ACC(%)
BWT(%)
CC
STL"
70.18 + 2.75
D-STL
69.36 + 0.78
D-S1
51.01 + 0.86
-4.00
1x
D-EWC
52.81 1: 2.80
-1.07 2.03
1x
CoDeC(I)
60.03 1. 0.75
0.36 1.11
1x
CoDeC
59.00 1: 2.56
-0.79
0.27
1.51x
D-STL"
63.13 + 0.86
66.27 + 1.47
D-S1
45.58 + 1.24
-3,67 1.27
1x
46.00 : 0.73
-3.21 + 0.51
1x
D-EWC
46.39 + 1.54
-1.64 1.11
1x
48.23 4 3.14
-1.02 + 1.16
8
CoDeC(f)
53.22 1: 1.82
0.08 1 0.45
1x
59.90 1 0.48
0.37
1 0.24
1x
CoDeC
53.30 + 1.25
-0.46 0.48
1.37x
59.97 + 0.87
-0.19
0.98
1.53x
D-STL"
57.09 1 1.55
63.51
D-SI
39.55 + 0.87
-2.03 0.69
1x
39.96 + 0.47
-1.74 + 0.96
1x
D-EWC
39.67 + 1.37
-1.32 1.18
45.14 + 0.18
-0.64
1x
16
CoDeC(I)
45.29 + 3.58
-0.99 1.40
1x
51.03 + 2.51
-0.01 + 0.67
1x
CoDeC
45.68 + 0.77
0.61 = 0.79
1.42x
51.32 1.05
0.26 + 0.56
1.39x
patient's privacy. Results in Table 6 show that CoDeC outperforms D-EWC and D-SI by 7% and 1% respectively in terms of ACC while incurring minimum BWT. The lossless compression scheme results 2x reduction in the communication cost. In all our experiments, we observe that ACC decreases we increase the graph size, while BWT remains of the similar order.
We also compare CoDeC with SKILL, distributed lifelong learning algorithm (results in Table 12 Fur- thermore, we present results on non-lID data distribution across the agents in Table B
Task-wise Communication Costs: The reduction in communication cost is a reflection of the constraints on the direction of gradient updates. As the gradient updates are not constrained for the first task, they occupy the entire gradient space. However, gradient updates after learning task constrained the RGS subspace, whose dimensionality decreases the task sequence progresses. This implies increase
10

---

Published in Transactions on Machine Learning Research (03/2024)
Table 5: 5-Datasets over ResNet-18 using directed ring and torus. (*): methods that don't adhere to CL
setup.
Directed Ring
Torus
Agents
Setup
ACC(%)
(WT(%)
CC
ACC(%)
BWT(%)
CC
STL"
92.79 0.08
D-STL"
92.51
D-S1
82.44 = 0.29
-1.52 + 0.13
1x
D-EWC
86.82
-3.37 + 0.80
1x
4
(f
87.24 0.23
-4.05 1 0.05
1x
CoDeC
87.41
-1.03
2.13x
D-STL"
92.31 0.06
92.32 1 0.15
D-SI
80.36 0.15
-2.64 4: 0.07
1x
79.55 1 0.33
-3.07
0.13
D-EWC
85.69 0.19
-0.92 + 0.14
1x
82.99 3.25
-2.10
1.60
1x
8
(fDeC(f)
86.54 0.04
-4.37 + 0.17
1x
85.92 + 0.18
-5.10
+ 0.17
1x
CoDeC
86,23
-4.61 +
2.17x
86.15
-4.85
2.19x
D-STL.
92.16 0.16
91.76 1 0.09
D-S1
78.53 0.62
-5.2
1x
77.00 0.11
-5.76
0.22
D-EWC
82.19 0.45
-0.18 + 0.05
1x
81.48 = 0.12
-0.56 + 0.14
1x
16
CoDeC(I)
86.36 0.15
-4.36 1. 0.19
1x
84.91 1 0:20
-5.48
0.22
CoDeC
86.41 0.16
-4.37 +: 0.24
2.16x
85.00 1 0.55
-5.52
0.35
2.23x
Table 6: MedMNIST-5 over ResNet-18 using 16 agent directed ring topology.
Setup
ACC(%)
BWT(%)
CC
D-STT.
76.55 +
D-ST
61.53 + 0.49
-7.51 + 0.72
1x
D-EWC
55.62 + 1.23
-0.72 = 0.25
CoDeC(f)
62.51 + 0.46
-0.37 = 0.27
1x
CoDeC
62.47 1 0.13
-0.07 1 0.44
2.03x
in compression ratios, which clearly reflected in our results highlighting task-wise CC in figure 3 In essence, as the gradient space becomes more constrained, it suffices for agents to communicate less with their neighbors. Hence, we achiove CC of 2.1x for task 2, with this increasing up 4.8x for task 5.
Scope of ACC-CC Trade-off: It. is possible to get better ACC with CoDeC, albeit at the cost of reduced CC. This is achieved by relaxing the orthogonal gradient constraint and allowing gradient updates along some CGS basis vectors, demonstrated in Scaled Gradient Projection (SGP) (Saha & Roy 2023 The gradient update along each CGS basis vector is scaled according to its importance À for the past tasks.
4
Legent aggats agents
55
Commiciiction Compersson
Accurayy
CoDeC
50-
D.EWO
85
40
35
Tusks
Tooks
Figure Evolution of task 1 accuracy over the course of 20 tasks from Split MinilmageNet 4 agent ring topology
Figure 3: Task-wiss ResNet-11
CC for 5-Datasets over
11

---

Published in Transactions on Machine Learning Research (03/2024)
Table Split CIFAR-100 over AlexNet using directed ring and torus topologies with scaled gradient updates.
Directed Ring
Torus
Agents
Setup
ACC(%)
BWT((%)
CC
ACC(%)
(WT(%)
CC
CfDeC(f)
59.07 = 0.51
-2.19 = 0.25
1x
CoDeC
59.20 = 0.6
-1.95 = 0.26
1.42x
CoDeC(f)
55.33 0.21
-1.02 = 0.59
1x
56.02 + 0.4
-0.88 + 0.35
1x
CoDeC
55.68 0.36
-1.10 = 0.27
1.42x
55.69 = 0.27
-1.01
1.42x
CoDeC(f)
49.68
0.51
-0.51
+: 0.37
1x
50.15
0.12
-0.09
0.19
1x
16
CoDeC
49.56 = 0.90
-0.57 +: 0.30
1.42x
50.00 + 0.49
-0.88 + 0.23
1.42x
The lower the A, the lesser the importance of the responding basis direction. >=1 implies no gradient steps can be taken along that basis direction. The following update rule is used to oblain the scaled gradient update g¹:
Commincatioo Compresio
(53.77,00
1.6
(05,00.1)
(51.31-0.)
1.5
(..15
g'=g' ('''-(M'(MM)) gg'
(12)
65.42.012)
Here, is a diagonal matrix containing À in its di- agonal. Note that A I corresponds to the GPM update rule in equation The computation of À in- volves a scale coefficient a. À tends to for large a values, and SGP converges to GPM. To incorporate gradient scaling in CoDeC, the basis vectors in M with Ả < 1 are appended to O the end of each task. This provides us with a knob achieve an ACC-CC trade-off, shown in figure | We find that ACC increases
10
90
100
1000
Scale Coeffeiest [a] for SCP
Figure 4: Impact of on CC for Split CIFAR-100 over an 8 agent ring. The data labels denote (ACC, BWT).
decreases, albeit at the cost. of lower CC. This is the result of the gradient updates lying in larger subspace for smaller values of For a=1000, the results are similar to the ones presented in Table 3 where allow only orthogonal gradient updates. The best performance in terms of ACC is obtained at a=10 with CC=1.42x, and we report these results in Table
Avg. Consensus Error
Arg. Consensus Error
10
10
10-3
10-1
Task DPSGD
Task DPSGD
4
Task CoDeC
Task CoDeC
1
Task DPSGD
Task DPSGD
Task 9 CoDeC
Task CoDeC
10-3
25
50
75
100
Epachs
Epochs
Figure 5: Average consensus error (CE) for Split CIFAR-100 (left) and Split MinilmageNNet (right) over an 8 agent ring topology. Task '7' DPSGD (CoDeC) denotes CE when 7t task learned without (with) orthogonal gradient constraints.
We also investigate the effect of taking orthogonal gradient updates upon the average consensus error, which we formally define as:
Consensus Error:
CE
&
x211
(13)
N
Here, 5t represents the global average of the model parameters x at any given iteration k. We provide upper bound for the consensus error in Appendix A.2 CE is measure of the effectiveness of gossip
12

---

Published in Transactions on Machine Learning Research (03/2024)
averaging in the decentralized learning scenario. In particular, lower CE implics that the agents are closer to achieving a global consensus. In figure 5 we show CE with and without orthogonal updates for task 2 and 9 after each training epoch for Split CIFAR-100 and Split minilmageNett. As the training progresses, CE consistently reduces as expected. We observe that the rate of achieving consensus is similar for the two cases. In other words, CoDeC enables decentralized continual learning without hindering the gossip averaging mechanism.
7
Conclusion
This work proposes CoDeC, a novel commuication--ffiientt docentralized continual learning algorithm. CoDeC enables serverless training with spatially and temporally distributed private data and mitigates catastrophic forgetting by taking gradient steps orthogonal to the gradient directions important for previous tasks. These orthogonal gradient updates, and hence the model updates, lie in lower dimensional gradient subspace. We exploit this fact to achieve lossless communication compression without requiring any addi- tional hyperparameters. Further, we provide theoretical insights into the convergence rate of our algorithm. Our results demonstrate that CoDeC is very effective in learning distributed continual tasks with minimal backward transfer and up to 4.8x reduced communication overhead during training.
8 Acknowledgments
This work was supported in part by, the Center for the Co-Design of Cognitive Systems (COCOSYS), a DARPA-sponsreed JUMP center, the Semiconductor Research Corporation (SRC), the National Science Foundation, and DARPA ShELL.
13

---

Published in Transactions on Machine Learning Research (03/2024)
References
Sai Aparna Aketi, Amandeep Singh, and Jan M. Rabaey. Sparse-push: Commuicatioon & energy-efficient decentralized distributed learning over directed & time-varying graphs with non-iid datascts. CoRR. abs/2102.05575,, 2021. URL
Sai Aparna Aketi, Sangamesh Kodge, and Kaushik Roy. Neighborhood gradient mean: An efficient decen- tralized learning method for non-iid data. Transactions on Machine Learning Research, 2023.
Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed doep learning. In International Conference on Machine Learning, pp. 344-353. PMLR, 2019.
Aditya Balu, Zhanhong Jiang, Sin Yong Tan, Chinmay Hedge, Young M Lee, and Soumik Sarkar. De- centralized deep learning using momentum-accelenatee consensus. In ICASSP 2021 2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
2021. doi:
10.1109/ICASSPP9922 9ttp://CAASS199722222.99995155 2021.4415644
Pascal Bianchi et al. Performance of distributed stochastic approximation algorithm. IEEE Transactions on Information Theory, 59(11):7400 7418, 2013. doi: 10.109//111:001:2255111.
Yaroslav Bulatov. Notmmist dataset. Google (Books/OCR). Tech. Rep..Online], 2011. URL http:
//yaroslavw..llggeana htt.://www...app..ooo/ooo/o/oea.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference Learning Representations, 2019.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc'aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoe Le, and Andrew Ng. Large scale distributed deep networks. In F. Percira, C.J. Burges, L. Bottou, and K.Q. Weinberger (ods.), Advances Newral Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL http://///ooceeeg..eeeeeeraeeeee
pdf
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. In The European Conference on Computer Vision (ECCV), 2020.
Yasaman Esfandiari, Sin Yong Tan, Zhanhong Jiang, Aditya Balu, Ethan Herron, Chinmay Hegde, and Soumik Sarkar. Cross-gradien aggregation for decentralized learning from non-iid data. CoRR, 0bs//2103.00511. 2021. URL https::// http:///aaxi.../aab////////
Mehrdad Farajtabar, Navid Avizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762 3773. PMLR, 2020.
Yunhao Ge, Yuecheng Li, Di Wu, Ao Xu, Adam M. Jones, Amanda Sofie Rios, lordanis Fostiropou- los, shixian wen, Po-Hsuan Huang, Zachary William Murdock, Gozde Sahin, Shuo Ni, Kiran Lekkala, Sumedh Anand Sontakke, and Laurent Itti. Lightweight learner for shared knowledge lifelong learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:///penreviee..ee forumPid-
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons. The non-iid data quagmire of decentralized machine learning. CoRR, abs/190 hts/1910.00189, 2019. URL http:://arxiv..ogabba/////e/ 00189
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwintn Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114:3521 3526, 2017.
Anastasia Koloskova, Tao Lin, Scbastian U. Stich, and Martin Jaggi. Decentralized deep learning with arbitrary communication compression, 2019. URL https:://taaat wwtps/iww.rggaa//////oooaa 199bbb./9/000009666
14

---

Published in Transactions on Machine Learning Research (03/2024)
Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence, 2016. URL httts::///axtiiii org/ httgg/bb//161000 02527
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Guanghui Lan et al. Communication-effiiiettt algorithms for decentralized and stochastic optimization, 2017.
URL
Jeongtae Lee, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Lifelong learning with dynamically expand- able networks. CoRR, abs/1708.01550 0154477 2017. URL http: wwtp:///////............
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? case study for decentralized parallel stochastic gradient descent, 2017. URL
Tao Lin, Sai Praneeth Karimireddy, Sebastian U. Stich, and Martin Jaggi. Quasi-global momentum: Accelcrating decentralized deep learning on heterogencous data. CoRR, abs/2102.04761, 2021. URL
https://arxiv http:://aarav..orr//a//21...... 04761
David Lopez-Pas Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
Arun Mallya et al. Packnet: Adding multiple tasks to a single network by iterative pruning. 2018 TEFE/ IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 7765 7773, 2018.
Michael Mecloskey and Neil J. Cohen. Catastrophic interference in connectionist networks: The soquential learning problem. The Psychology of Learning and Motivation, 24:104-169, 1989.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Roger Rateliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97 2:285-308, 1990.
Sylvestre-Alvis Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 5533 5542, 2017.
Mohammad Rostami, Soheil Kolouri, Kyungnam Kim, and Eric Faton. Multi-agent. distributed lifelong 2017.
learning for collective knowledge acquisition. arXiv preprint
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascamu, and Raia Hadsoll. Progressive neural networks. ArXiv, abs/ 0bs/1606.04661, 2016.
Gobinda Saha and Kaushik Roy. Continual learning with scaled gradient projection. Proceedings of the AAAI Conference Artificial Intelligence, 37(8):9677 Jun. 2023. doi: 10.1609 10.1609/aaii.v2/ v37.8.2617 URL https://ojs aaai. org/ hts///ndexx,./e http:///w....aatttcc////////////
Gobinda Saha, Isha Garg, Aayush Ankit, and Kaushik Roy. Space: Structured compression and sharing of representational space for continual learning. IEFE Access, 9:150480-150494, 2021a. doi: 10.1109/ ACCESS, 200.SS..2.2.11160211..
Gobinda Saha et al. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021. URL
Joan SerrA, DÄdac SurAs, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4548 4557. PMLR, 10-15 Jul 2018.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Yuanming Shi, Kai Yang, Tao Jiang, Jun Zhang, and Khaled B Letaief. Communication-cffcientt edge ai: Algorithms and systems. IEEE Communications Surveys 8 Tulorials, 22(4):2167-212 2020.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Contimual learning with deep generative replay. CoRR, abs/1705.08690, 2017. URL
Hanlin Tang, Xiangru Lian, Shuang Qiu, Lei Yuan, Ce Zhang, Tong Zhang, and Ji Liu. Deepsqueeze: Parallel stochastic gradient descent with double-pass error- cropensmmeeneteed compression. CoRR, abs/ 1907.0737,7 2019. URL btttp://arxiv.org/abb....
073446
Anastasiaa Usmanova, François Portet, Philippe Lalanda, and German Vega. A distillation-based ap- proach integrating continual learning and federated learning for pervasive services. arXiv preprinl arXiv:2109. 1.X1:2209..0499,, 2021.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning, In Advances in Newral Information Processing Systems, volume 29, 2016.
Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 184 193, June 2021.
Buddhi Wickramasinghe, Gobinda Saha, and Kaushik Roy. Continual learning: review of techniques, challenges and future directions. IEEE Transactions on Artificial Intelligence, pp. 1-21, 2023. doi: 10. 1109/TAA.2023.3333
Han Xiao et al. Fashion- mnist: a novel image dataset for benchmarking machine learning algorithms. ArXiv, abs/1708.075 077747 2017.
Lin Xiao and S. Boyd. Fast linear iterations for distributed averaging. In 42md IEEE International Conference on Decision and Control (IEEE Cot. No.03CH37475), volume 5, pp. 4997 5002 Vol.5, 2003. doi: 10.1109/
EDCC2000.11224222
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale lightweight. benchmark for 2d and 3d biomedical image classification. CoRR, abs/ abs/2110.144995 2021. URL https://arxiy: wwt.eoor//abba/aeeee 2912999101.47995
Jachong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, 2021. URL
https://arxivv.rr//aet www.oogg//bbbboeeeeoo 2000. .31996
Hao Yu, Rong Jin, and Sen Yang. On the lincar speedup analysis of communication efficient momentun sgd for distributed non-convex optimization, 2019. URL httpppp://aaziw.orr//aaaaaa//
03817
Friedemann Zenke el al. Continual learning through synaplic intelligence. In Doina. Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3087-3995, International Convention Centre, Sydncy, Australia, 06-11
Aug 2017. PMLR. CRL http::/pprocedding mlr 77.r
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Xiaohan Zhang, Songlin Dong, Jinjie Chen, Qi Tian, Yihong Gong, and Xiaopeng Hong. Deep class- incremental learning from decentralized data. JEEE Transactions on Neural Networks and Learning Sys- tems, pp. 1-14, 2022. doi: N0.1009/NNNLS..022...3
16

---

Published in Transactions on Machine Learning Research (03/2024)
A Appendix
Proofs for the lemma, theorem and corollary presented in the main paper are detailed in A.1
A.2
3.
A.4
and A.5 sections. Details related to the network architectures and datascts used in our experiments are presented in A.7 and A.8 respectively. We list all our training hyperparameterss A.9 We also provide details about implementation of our basclines D-EWC and D-ST in A.10 Some additional results are available
in AA.II We perform our experiments on single machine with 4 NVIDIA GeForce GTX 1080 Ti GPUs. All the agents in our experiments are distributed evenly over these 4 GPUs. For instance, in the case of 16-agent ring/tor /orrus topology, each GPU is utilized by 4 agents.
A.1 Proof of Lemma 1
The orthogonal projection g' of the original gradient update g' with respect to GPM is obtained as:
g = g" - (MM(MM) 'g¹
(14)
From the above equation we can write:
g' g = + (M'(M') 'g*
(15)
We have:
(g'. = (g' + (M' (M)))))))))
)g'gg"> = (g'.g") + ((M'(M"))) >)"'g)))))))))
(16)
Since g' and M'M)) gg are orthogonal to each other:
(M'(M)) ) =
(17)
Substituting equation 17 into equation 16
(g' (g. (g'gg)
|
(18)
We make sure |gg' A by using eth. 1. From the above equation, we see that the dot product is greater than O. This implies that if -g' is in the descent direction, -g¹ is also in the descent direction.
A.2 Bounds Consensus Error
This section provides upper bound on the consensus error. We follow the same approach as Esfandiari
ct al. 2021b. The update rule for our algorithm is as follows:
N
R = = " N
8k-1
(19)
2 denotes the averaged model across all the agents a given iteration k. For the rest of the analysis, the initial value will be directly set to O. From equation 19
we have:
N
1
X
X4
g
(20)
N
We introduce some key notations and properties:
1
11T
N
G
(21)
X
X
G B gk:
gh N
H
Ví (x
17

---

Published in Transactions on Machine Learning Research (03/2024)
For all the above matrices, | . ||a,l? obtain:
where a, is the i-th column of the matrix A. Thus, we
X (I - Q)1 Σ xx x
(22)
For each doubly stochastic matrix W, the following properties hold true
QW WQ:
(I- (I-Q)-= W(I-Q);
For any integer & 2 1, W] S (VP)*, where || l|e is the spectrum norm of matrix.
For N arbitrary real square matrices i E [1,
N
N |
(23)
1=
We are now ready provide bound on the consensus error. Since X - W W we have:
= -
(24)
Applying the above equation times we have:
& (I - Q) = XI W Σ
h =-
GI
Σ -I-Q
(25)
T=1
12 Q)
G Q) 27
X
1/
(26)
81
We find the upper bound for term 7.
G. Q)W
E
C, Q
- -
G|
[| Σ
4 2 (h-
(h
Ell |
7=0
T=0
(27)
k-1
4 p1 -1--1-
1 BII | + BI |
T=0 0
k-1k-
2
2² (2
Elll G. |2
Σ
r -
G+
vp)
-=0
(a) follows from equation 23
(b) follows from assumption
18

---

Published in Transactions on Machine Learning Research (03/2024)
(c) follows from the incquality ry S 2 (x² + y²) for any two real numbers 2, y.
(d) is derived from En of 1- =2"= & 2
I-VB
We proceed with finding the bounds for ||
E B||G, 11 B H H H.
N
(28)
I 3 || G
H
+ 3E | H, (I Q) 13 + H|
E 3No + 3N50 + 3E N
v |
(a) holds because
H| 1
E1l Σn [((
Substituting
28 in
27
E
Σ -
1
112
k-1 1
3No + 3N5 + 3
Σ
N
vx(()))1)
N
0
(29)
3N (02 + +0)
NN
+
vp)
Substituting 29 into the main inoquality
(66
2
3No2
3N82
3/
1
E
XI 2)
S p²
(1 (1-vp)² +
+ (1 vp)
Σ
[x(())111
(30)
vp)
(1 vp}²
N
Summing over & E {1,.... K - 1} and noting that E
Xo(I Q)
0:
X (I - 2) 1
3N K K-1k-1
N V fi 11
Σ E
< CCK+ +
N 1=1
k= T=0
CK + (1- vp)
u2 2
K-1
Σ
pl )ー
Elll 1
[x(())) < CK+
3/
Σ
El|
1
Σ
V(((i) (31)
1 Vp
N
vp)
N
k=0
k=
13N
3N5 (1 vp)²
where 12
Dividing both sides by N:
Σ X 1
302
382
312 K-1
N
STpIP (1 vp)²
K+ (1 -p)
Σ
E|l N
( (32)
(1- vp)2 (1-vp)
This directly implies:
Σ 1
- X
11
$ 1222 (1 2p)² + (1 - v
30²
35
K +
3r1²2²² 22
Σ
K-1 1
Ell
N vs(( (33)
N
(1-vp)²
(1 vp)
k=
ル=
19

---

Published in Transactions on Machine Learning Research (03/2024)
A.3 Proof for Theorem 2
When F is I-smooth, we have:
B[ + X xx,) + 1[|||x+++ XR++ - x=
(34)
We proceed by analysing /:
B (VVF(()) - - -17
Σ
(35)
1 N
1 Σ Sk ) (V-E(((F(())) -1
EVFF(X (xt), -17
Σ Bi g
+ gi
N i=1
B[(VF
(xx); -1
Σ
gi
gd
EV+
(x*),
-7
Σ
N
11
111
(36)
We first analyse 11:
1 Σ
1
1
BVB(FF())))
B2 giM
BBI
Σ
(gl
(37)
2L
N
This holds as (a, b) Analysing III:
|||||| / b1|
E VF (FF(X))) (x++) -17
1
Σ
8k ||
-nn
(F(())
fx(xx)))
(38)
N
i=1
1=1
With the aid of the equity (a, b) 1/2 |1/aa122
||b|2
||a
b/l3]
we have
N v (xx) 1 IVF 2| +
1 N N
(VF (xx).
V
-
vi(xx)11
(39)
N
1=1
Analysing *
1 N
[VF(x)
f
v((((x))
N Σ
1
(x(x)))
(40)
< N
(xx)) v((X))
Σ L [ - xt11
N
Substituting 40 back into
39 we have:
1
N
|IVF(S)))2 + N Σ v=((2))
N
N
(VF (8t)
V
1 N
Σ
X xx,11
(41)
N
2
20

---

Published in Transactions on Machine Learning Research (03/2024)
Substituting 37 and
41
into
36
and
36
into
135
11
Lnp
1
(((VF(*)) xx+1
EI
xx)] _
Ell
gg
gi 1
2L
2
N
(42)
N
"
E
Σ
fx
L2B
| - x3(191
2
N
N
i=1
From equation 20 we have:
N
x x&|1?] = 17 2B
Σ
1
(43)
N7
Substituting 42 and
34
!
L
1
N
E E[F(x++)) 1)1 S [xx((x)] +
|3 +
B11l
(g -
2T.
2
N
(44)
1 N
N
?L
N
L
V 11 +
E
x
11
11
2
2
t=1
Rearranging the terms and dividing by C1 (3 g - 21.
to find the bound for B V] 113)
N
1 Ci
[[F(x))] - B[F(xx+1)] + C2
Eil
Σ
g
D
+
E1I
M
N
N
(45)
N + C3 Ej
Σ
| x1.1111 C. B[ll Σ
i=1
where C2 = Lnd //0, C33 = L 2 C4 = 1/200.
We first analyze *:
N
N
N E|| Σ
N
El
Σ
g
E[|
8k 1
gg
BiI++ +
1
N
N2
i=1
i=1
N
8
B||
TM
1 +1|
(I
MM gg))11 A
MM T
(g I|[I MM
(g))l
N
N2
1=1
v(((+)) (1)
E(|
+E
N
(46)
(a) follows from the fact that g is an orthogonal projection gi and it defined by the GPM matrix M.
(b) follows from all agents having the same GPM matrix M
(c) is the conclusion of Lemma 1 in Yu et al. 2019
21

---

Published in Transactions on Machine Learning Research (03/2024)
Substituting 46
into
45p
and summing over & E f0, K - 1}:
K-1
K-1
0²
| (1)
E
|FF
$ Ci
E [F(((x)) -
F
+
C2
Σ
+
+
N
k==
(47)
N
K-1
- C3
Σ
Σ
x xill?
CC
- C4
Σ E
H (
K=
Dividing both sides by K:
K-1
K-1
1 N
(T)
B
||'
1 CiK
1
(xF -F1) (F))
+ C2
+C2
Σ
E
Σ
K
N
K
N
0=
k=
i=1
(48)
K-1
N
K-1
N
27
E
X
x ||]
- C4
Σ
E
(
K
N
K
N
1=1
k=
Using equation
33 in the above equation, we have:
K-
E
VF
X
]||
1
([F(xx)) - FF + +C2
0²
+C2
K-1
Σ K
E
1 N
Σ v
11)
K
C1K
N
N
K=
k=0
1=1
C3 K
302
382
377² 31ppx p2
K-1
N
+
1f
112
+ (1
K +
E1I
(49)
vD)
vpj²
(1 vp)
k=
1=1
K-1
C.
Σ
1
E
(x
K
N
K=
Rearranging the terms:
1 K-1 EE(INE(SN))1
S CiK
1
[ (Xo) -F1)
+C2 o²
+ x2z
302
+
382
K
N
(1- vD)²
(1
vp)²
k=
(50)
3Csr1²p²
-ca)
(
N
1))
+ C2 +
Σ
Vli(tt)
(1 vp)
When (C2 +
3Car2u² [I-vP]
C.) s 0, we have:
K-1
E [x*))
S C₁K (E(F(x))-F))
1
-F1) + +C2 d²
N +Cs
30²
+ (1- /
352
(51)
K
(p-vp)
km
A.4 Discussion on the Step Size
Recall the condition C1 >>0 0. This implies 17 > +,
The condition for equation (51 to be truc is C22 +
3Cnr2
- C.)) s 0. Therefore, we have:
(I-VP)
3L3
(1 vp) + 19 150
(52)
Solving this inequality, combining the fact that 17 > 0, we have then the specific form of 1t":
1 = V(I 2p)² + 12p - (1 - vp)
(53)
6lp²
22

---

Published in Transactions on Machine Learning Research (03/2024)
Hence, the step size is defined as
1 L <<ク S v(1 - vp) + 12p (1 vP)
(54)
6Lμ²²
A.5 Proof for Corollary 3
According to equation 51 the right hand side, there are three terms with different coefficients with respect to the step size We separately investigate each term:
77 o
implies C1 ===0
Therefore for the first term:
F(X0) - =0
(55)
C1K
VNK
For the second term:
C₂
N
N
=
N
K
= O
VNK
(56)
For the third term:
N
C'Cs O
(57)
K
By omitting N in non-dominant terms, there exists constant C > 0 such that the overall convergence rate is follows:
( 1
K B S C VNK K
(58)
which suggests when N is fixed and K is sufficiently large, CoDeC enables the convergence rate of O [NN
A.6 Proof for qf Lying in RGS
Here, we prove that qf from line 9 in algorithm I lies in RGS.
Line 8 (Algorithm 1 x x(x+)) + Ejer() wx/( x()
Line 9 (Algorithm q x x
After simplifying :
q x(x+)) + Σ
wis win(Xf xt) x
(59)
JEN(1)
From line 7 in Algorithm I we have, x (k+
x
nga
Substituting xt ()+ in equation 59 q - EjeNro (ig(X} x))
Now, we know that øt is the orthogonal gradient component of g and hence it lies in RGS subspace. If the gossip update component i.c. ΣjeN(a) Why xf) also lics in RGS, we can conclude that qe lics in RGS by linearity.
We prove this using the linearity property of the vector spaces and induction.
Say a = E, Why (+) -x/) Σ3 Way( - zt). af lies in RGS as záj = zá (synchronized initialization). Now, ai+1 = E, wiy(ay - n - g.
GPM update ensures that gradients z and oi lie in RGS and from induction, we have af lie in RGS. From linearity, conclude that af lies in RGS for any agent. i. Hence, q also lies in RGS.
23

---

Published in Transactions on Machine Learning Research (03/2024)
A.7 Network Architecture
AlexNet-like architecture: For our experiments, we scale the output channels each layer of the architecture used in SerrA et al. (2018). The network consists of 3 convolutional layers of 16, 32, and 64 filters with 4 x 4, 3, and 2 2 kernel sizes, respectively and 2 fully connected layers of 512 units each. A 2 x 2 max- maxx-pooling layer follows the convolutional layers. Rectified linear units are used as activations. Dropout of 0.2 is used for the first two layers and 0.5 for the rest of the layers.
Reduced ResNet18 architecture: This similar to the architecture used by Lopez- Pappe--Pzz & Ran- zato (2017). We replace the 4 average- pooling layer with 2 layer. For experiments with minilmageNet, we use convolution with stride 2 in the first layer.
All the networks use ReLU in the hidden units and softmax with cross entropy loss in the final layer.
A.8 Datasets
Table and 9 provide the details related to the datasots used our experiments. For McdMNIST-5 datasct statistics, please refer to table 2 in Yang et al. (2021). The training samples/Laske are independently and identically distributed (IID) across agents without any data overlap. For instance, for graph size of 4 agents, each agent has 5000/4 = 1250 training samples for particular task in Split CIFAR-100.
Table 8: Dataset Statistics for Split CIFAR-100 and Split-minilmageNet
Split. CIFAR-100
Split minilmageNNet
num. of tasks
10
20
input size
3 x 32 x 32
84 x 84
# Classes/task
10
5
# Training samples/ ttakks
5,000
2,500
# Test samples/ stasks
1,000
500
Table 5-Datasets Statistics
CIFAR-10
MNIST
SVHN
Fashion MNIST
notMNIST
Classes
10
10
10
10
10
# Training samples/tass stakkss
50,000
60,000
73,257
60,000
16,853
# Test samples/tass tasks
10,000
10,000
26,032
10,000
1,873
A.9 Hyperparameters
All our experiments were run for three randomly chosen seeds. We decay the learning rate by a factor of after 50% and 75% of the training, unless mentioned otherwise. For Split CIFAR-1100, we use a mini-batch size of 22 per agent, and run all our experiments for total of 100 epochs for each task. For Split MinilmageNeet, we use mini-batch size of 10 per agent and 10 epochs for each task. For 5-Datasets and McdMNIST-5, we use a mini-batch size of 32 per agent, and 50 epochs for each task. We list additional Iyperparameters in Table 10. For D-SI, in case of Split CIFAR-100 we use a lower learning rale of 0.0005 for the last 2 tasks when number of agents is 8 or 16. Similarly for 5-Datasets, we lower the learning rate to 0.001 after the 1st task. This tuning is required for the training to converge in case of D-SI. The average consensus error plots shown in figure in the main paper were obtained with a cosine annealing based learning rate scheduling instead of the step decay mentioned earlier.
24

---

Published in Transactions on Machine Learning Research (03/2024)
Table 10: List of hyperparameters for the baselines and our approach. 'Ir' represents initial learning rate. 'incre' represents how the 6/ is incremented for each new task.
Dataset
Setup
Hyperparameters
D-SI
Ir: 0.001, 0.1
Split CIFAR-100
D-EWC
Ir: 0.05, A: 5000
CoDeC
Ir: 0.01, Erp 0.97, incre: 0.003
D-ST
Ir: 0.001, e: 0.3
Split MiniImageNet
D-EWC
Ir: 0.03, A: 5000
CoDeC Ir: 0.1, 6nx: 0.985, incre: 0.0003
D-SI
Ir: 0.01, c: 0.1
G-Datasets
D-EWC
Ir: 0.03, A: 5000
CoDeC
Ir: 0.1, ExA: 0.965, incre:
D-SI
Ir: 0.001, c: 0.5
MedMNIST-5
D-EWC
Ir: 0.001, A: 5000
CoDeC
Ir: 0.001, cas: 0.99, incre: 0
A.10 Baseline Implementation
Algorithm 2 and 3 demonstrate the flow of D-EWC and D-SI respoctively, the basclines which extend EWC [Kirkpatrick et al. 2017) and SI (Zeuke et al. 2017) to ralizedd setting, The loss function minimized in D-EWC is of the form (r, 4, xxx)) shown in line 5, algorithm 2 Here, À is regularization coefficient
which signifies the importance given to the past tasks. x.. and x represent model parameters for a particular layer I. Unlike CoDeC, here we generate the Fisher matrix F at each agent and then do a global averaging step before utilizing it for continually learning the next task. We do so because D-EWC performs best when entire training data is used to generate the Fisher matrix.
SI aims to minimize the loss function fr,(dd x of the form shown in line 5, algorithm 3 Here, is dimensionless strength parameter which signifies the importance given to the past tasks. SI computes w in an online manner, which is used to update Ω at the end of each task. Ω is an importance estimate, which scales the per-parameter regularization strength. Ω generated at each agent and then globally averaged before utilizing it for the next task.
STL and D-STL provide upper bounds on performance. These are not continual learning techniques and may not be feasible in resource-consstranee environments they requires excessive number of model parameters.
A.11 Additional Results
Continual Learning Statistics Aggregation: Additional results in Table 11 demonstrate the impact of broadcast and all-gather for continual learning statistics aggregation.
Task-wise CC: We present additional results for task-wise CC, similar to figure 3 in the paper. Figure
a) demonstrates that task-wise CC ranges from 1.2x to 1.8x for Split minilmageNet. Figure 3 shows fask-wise CC ranging from 1.2x to 4.45x for Split CIFAR-100.
Training Loss vs Epochs with and without compression: We present some results to emphasize the lossless nature of our proposed communication compression scheme. Figure Figure/showss shows training loss after each epoch for a particular agent for task and 9 in Split CIFAR-100 sequence with and without compression. The convergence rate of the training loss not affected by applying the proposed compression scheme.
25

---

Published in Transactions on Machine Learning Research (03/2024)
Table 11: Impact all-gather and broadcast technique for continual learning over agent directed ring.
Dataset
Setup
ACC
BWT
D-SI (broadcast)
27.2
-14.14
Split CIFAR-10M
D-ST (all-gather)
39.36
-1.98
D-EWC (broadcast)
46.66
-0.06
D-EWC (all-gather)
50.46
0.42
CoDeC(broudeust)
53.13
-1.00
CoDeC (all-gather)
52.7
-0.39
D-SI (broadcast)
37.57
-43.65
3-Datasets
D-ST (all-gather)
80.23
-2.70
D-EWC (broadcast)
71.54
-0.12
D-EWC (all-gather)
85.9
-0.92
CoDeC(broadcast)
86.56
-4.38
CoDeC (all-gather)
86.46
-4.00
1.70
2.30
Oknnuskkkt Caurosin
3,00
0:5
000
Tuke
Tadk
(a) Split MiniImageNet.
(b) Split CIFAR-100
Figure 6: Task-wise CC for (a) Split MiniImageNet and (b) Split CIFAR- 100 over directed ring topology.
CoDaC)))
Task CaOeet
ColleC
22
Traning
1.0
1.6
WWW
102
100
Fpoche
Fpochs
(a) Task 2
(b) Task 9
Figure 7: Training loss epochs for (a) task 2 and (b) task 9 in Split CIFAR-100 100 sequence with CoDeC(f) and CoDeC using AlexNet over a directed ring with 8 agents
CoDeC vs SKILL: Each agent in SKILL Ge et al. 2023) uses a common pre-trained frozen backbone built-in at initialization so that only the last layer (or head) and a unique set of bias parameters are learned for each task. After training, these bias paramcters and task-specific heads are shared among agents via a fully connected graph. However, each SKILL agent is independently learning different task at a given time, lacking the concept of collaborative learning retenstrated in CoDeC. The data for each task is spatially
26

---

Published in Transactions on Machine Learning Research (03/2024)
distributed in our setup, while SKILL assumes access to the entire data available in centralized manner single agent. Hence, in CoDeC it becornes essential to communicate with peers during training. Moreover, we do not assume access to pre-trained model at initialization, and update the weights for each task while mitigating forgetting. Unlike SKILL where the number of learned parameters increases with each task, we update the same set. of weights for the entire lask sequence. Hence, SKILL and CoDeC target two different scenarios even though both utilize decentralized agents while continually learning task sequence.
To compare SKILL with CoDeC, employ SKILL in scenario where each task's data is distributed across the agents. Similar to CoDeC, SKILL agents aim to learn global generalized model with spatially and temporally distributed data. For iso-comparison, both SKILL and CoDeC use ResNet-18 pre-trained on ImageNet and perfect task oracle at test time. In SKILL, the agents do not communicate with their peers during training and share the bias parameters and heads only at the end of the training. Meanwhile, the agents in CoDeC communicate with their peers during training by utilizing the gossip averaging mechanism. Results in Table 12 show the importance of collaborative lcarning in scenarios where each task's data is spa- tially distributed. CoDeC achieves about 5.6% better accuracy than SKILL by the virtue of communicating while learning each task. As the number of agents increases from 8 to 16, SKILL performs much worse than CoDeC because the number of available training samples at each agent also reduces.
Table 12: Comparison between SKILL and CoDeC for Split minilmageNee over ResNet-18 using directed ring topology
Agents
Setup
ACC (%)
BWT (%)
8
SKILL
79.21
0.00
CoDeC
83.81
1.15
16
SKILL
74.33
0.00
CoDeC
80.94
0.77
Table 13: Results for non-IID data with 0.5 skew for Split CIFAR-100 and Split minilmageNee over a directed ring topology with agents. D-STL is not continual learning baseline and serves as an upper bound on performance.
Dataset
Setup
ACC
BWT
CC
D-STL
62.40 + 0.50
Split CIFAR-100
D-SI
38.81 0.72
-0.79
0.54
1x
D-EWC
49.20 + 0.35
-0.13
0.21
1x
CoDeC(f)
50.71 +: 0.47
-0.46 4 0.23
1x
CoDeC
50.60 + 0.38
-0.41
0.39
2.53x
D-STL
60.89 + 0.61
Split miniImageNet
D-S1
45.64 + 0.51
-1.97 + 0.32
1x
D-EWC
46.57 + 2.07
-1.15 + 0.64
1x
CoDeC(f)
51.28 + 0.23
-0.08 0.76
1x
CoDeC
50.93 + 0.16
-0.88
1.32x
Rosults on Non-IID Data: We conducted experiments with 50% label-wise non- across the agents (i.e. s05) (Hsieh et al. 2019). CoDeC utilizes a variant of DPSGD (Lian et al. (017) as the base decentralized algorithm for IID data across the agents. However, DPSGD has been shown to perform poorly with non-IID data. Techniques like Quasi-Global Momentum (QGM) (Lin et al. 2021 and Neighborhood
Gradient Mean (NGM) (Aketi et al.) 2023)) were introduced to improve decentralized learning performance in the presence of data. heterogeneity across the agents.
27

---

Published in Transactions on Machine Learning Research (03/2024)
For our experiments, employ QGM for learning with non-IID data as it improves performance without introducing extra communication overhead. Note that NGM performs better than QGM but at the cost of 2x communication.
Our results in Table 13 show that CoDeC outperforms D-EWC and D-SI by 1.5% and 11.8% respectively for Split CIFAR-100. CoDeC achieves minimal backward transfer while achieving a CC of 2.53x. For Split minilmageNNet, CoDeC achieves about 4-5% better accuracy than D-EWC and D-SI while incurring low BWT and CC of 1.32x.
Runtime Comparison: We present relative training runtimes in Table 14 For Split CIFAR-100 and Split minilmageNet, (f), D-EWC and D-SI have similar runtimes, For 5-Datasets, D-EWC and D-S1 have about 30% higher runtime than CoDeC(().
CoDoC generally has the highest time due the two extra matrix multiplications required for communication compression. Specifically, line 10 algorithm encodes model updates into coefficients which are commu- nicated to the neighbors by each agent. Line 11 in algorithm decodes the received coeffieients back into model updates. Therefore, CoDeC achieves the highest accuracy and lossless communication compression at the cost of increased runtime.
Training time (relative) for Split CIFAR-100, Split minilmageNet and 5-Datasets over a directed ring topology with 8 agents
Table 14:
Dataset
Setup
Training time
CoDeC(f)
Split CIFAR-100
CoDeC
1.27
D-EWC
1.09
D-ST
0.98
CoDeC(f)
1
Split minilmageNet
CoDeC
1.68
D-EWC
1
D-S1
0.92
CoDeC(()
1
5- 5-Datasets
CoDeC
1.61
D-EWC
1.30
D-SI
1.31
28

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm 3 Decentralized Synaptic Intelligence (D-ST)
Algorithm 2 Decentralized Elastic Weight Con- solidation (D-EWC)
Input: Each agent [] initializes model parameters x step size ? mixing matrix
W [utyk,je[1,,)) Ro) 0, nt = [[] for all layers
Input: Each agent E N] initializes model parameters x/. step size n, mixing matrix
I = 1,2,,.., importance measure S2' {(222) J running importance estimate w= []. damping parameter & = 10-3, old model parameters x(o) 0, Miii: neighbors of agent i (including itself), T: total tasks, K: number of training iterations
W [uig]_JE(.,N, (0) 0, F' [] for all layers = L, Fisher Matrix F = ((F)) old model parameters X(o) = O, M(): neighbors of agent i (including itself), T: total tasks, K: number of training iterations
Each agent simultaneously implements the TRAIN( ) procedure
Each agent simultancously implements the TRAIN(
) procedure
procedure TRAIN(
1. procedure TRAIN()
2. for r = I,.... T do
2. for r= l,.... T do
3. for = 0,1, 0.1....,
K
do
3. for k 0....... K
do
d7, ~ Di (r.((d,xi))) = fT.i x
4.
d,,, Drs fr.s(r.s:ij) = (d
5.
ΣE= 1== Ot (x x;_1)²
Σi=0 F (xs x(_1
6.
8 = (fr.(d x
6.
g V (d,,
w wt gi (x) x
7.
x ((++) x
188
x (k+ = x - ng
8.
x = xx + ΣjEN() why (8/ x()
x x (x+ + EjEN(o why (8) xx.
9.
q = x - -x
10.
for each j E (t) do
10.
q =xx1 -xi
11.
Send q and receive q
11.
for cach E Nii do
12.
(k+1) q +
12.
Send qi and receive qó
13.
end end
13.
* q + +
14.
end end
15.
Save x
15.
16.
# EWC Update
16.
Save x²
17.
Update F' for each layer
17.
# ST Update
Update F (F")
18.
Ω =
(x"" w
(for each layer
1.4
19.
p randomm1,,22..
1)
20.
if i== do
19.
Update Ω = ((22 )
21.
Gather F from all agents F= =ug(FF, (2,,,,,2...... FN. Send F to all agents
20.
p = random ....N
22.
21.
if i p do
23.
22.
Gather L2 from all agents Ω= aug ( Ω2 .2.....2 Send Ω to all agents
24.
end
23.
end
24.
return
25.
end
26.
end
27.
return
29


Reviews:
[{"review_id": "G0nRrUsJSR", "reviewer": "Reviewer_Gvsm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The writing is generally clear - I have no significant issues with the presentation from a mechanical perspective and any copyediting mistakes do not affect overall clarity.\n2. As far as I am aware, this use of GPM in decentralized setting is unique, and the problem of continual learning in decentralized settings is important.\n3. The empirical results which are presented are generally convincing, and -  combined with the results in the appendix - thorough. \n## Weaknesses\n1. The greatest weakness in my opinion is that the proposed approach is not fit for the intended use. The paper claims enabling decentralized continual learning and the abstract even describes a typical decentralized setup with private data at different sites. However, this setup is very difficult to imagine as having identically distributed data. In fact, the majority of the cases where private datasets are virtually pooled to obtain a stronger model are motivated by enriching the overall data distribution by combining quite diverse parts of the overall joint spread across the sites. The functioning and assumptions of the algorithm and, I assume, the proof will not apply to realistic situations, where data across sites has different distributions and covers different regions of the domain. Additionally, it is not reasonable to expect tasks to perfectly align across the sites in the order that they will be presented to the model. Moreover, there is no reason to expect the bases computed via an SVD at each site to be the same or even similar. This situation is not clearly explained in the paper (see my questions in the corresponding section).\n2. A possible explanation of the weakness number 1, is that the method is for decentralized training on a local server across multiple nodes. In this case, the users will have full control of reshuffling the data and splitting it across the nodes according to any assumptions needed by the algorith. However, in this case the complete omission of runtime analysis becomes the major concern. Although I do acknowledge that communication limitations are important in an of themselves, it is difficult to argue for the benefits of communication compression alone without a comparable analysis of runtime. It would be straightforward to provide both a theoretical and empirical analysis of runtime, and I think this is a necessary inclusion. Even if the runtime analysis results in slower runtime than baselines, the authors are responsible for highlighting this limitation (or strength if the method is faster) and arguing with respect to these findings.\n   - Note, runtime analysis and comparison with simpler pre-exisiting baselines is an important omission in either the cases of training on decentralized private data or the data deliberately distributed across nodes in a single server.\n3. A related, but somewhat  lesser, exclusion is that no theoretical analysis of the communication is provided (as far as I have found).  It is standard in this field to provide a theoretical analysis of the communication complexity, and this would help to augment the discussion in which the authors discuss their compression ratio.\n4. A further weakness is that the authors omit a number of important implementation details, such as the hardware information both for individual agents (CPU, GPU, etc) and the network (connection bandwidth, etc). These factors are usually most important when runtime is discussed; however, network bandwidth can also be important for arguing for the benefits of compression (there are arguments that compression is not always needed in high-bandwidth scenarios, etc).\n## Questions\n1. My main question regards the first distributed communication step (algoritm 1, line 8 ) in which parameters are communicated across the network.  It seems to me that this step could be skipped entirely if we start with the same model parameters at each agent. This is a common practice in distributed SGD for example, where we assume that models start with the same set of parameters, and so we only require gradient updates in order to arrive at a common model. Why is it desirable to assume differently initialized parameters across agents? Couldn't we avoid this unnecessary overhead by assuming a common starting seed?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors present CoDeC, a method for decentralized continual learning. The method utilizes orthogonal gradient projection memory, in which each layer of the network is partitioned into orthogonal subspaces with the aim being to mitigate interference between previous tasks and new updates.  Decentralized learning is able to be performed by a gossip-based averaging  in which model parameters are first communicated to neighbors, and averaged locally. Neighboring model updates are then computed by only sending coefficients of the bases spanning the RGS subspace from the orthogonal gradient projection.\n\nThe authors provide theoretical and empirical analysis of convergence, as well as empirical comparisons of the performance of their method with baselines which are reasonable given the unique problem setting and focus on continual learning.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide runtime analysis and comparisons with other methods.\n2. Provide technical specifications for the implementation details.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KZKugLOqxG", "reviewer": "Reviewer_m7Vb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper combines gossip averaging and GPM to propose a novel algorithm for decentralized continual learning. This appears to be the first work that has looked at continual learning in the decentralized (non-federated) setting and as such it addresses an important research question. The algorithm is also backed by both theoretical analysis and empirical improvements in accuracy over baselines.\n\n2. The authors also provide a novel approach for communication compression in this setting by communicating residual coefficients whose dimensionality is much smaller than that of the gradient. This provides upto 4.8x reduction in communication cost in experiments.\n\n3. The paper also introduces a new dataset MedMNIST-5 which can serve as a benchmark biomedical dataset for future distributed continual learning works.\n\nWeaknesses:\n\n1. The paper makes some strong assumptions like IID data at each node and presentation of tasks at the same order at all nodes, especially because these assumptions are more likely to be violated in continual learning settings where data at each node may have its own spatial/temporal distribution and it's not necessary that the data for the same task is collected at all nodes at the same time.\n\n2. Some parts of the algorithm are not clearly explained. Please see requested changes below for suggestions for improvement.\n\n3. SKILL, which according to Section 2.3 is the only other related work studying decentralized continual learning for neural networks, is not included as a baseline. \n\n4. The performance of CoDeC w.r.t the BWT metric appears to be mixed in Table 3 and 4. Since the main goal of continual learning is to prevent forgetting it appears that BWT is a more important metric than ACC and so these mixed results seem to suggest that CoDeC isn't really meeting its main goal for some settings at least.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel algorithm for decentralized continual learning called CoDeC which uses Gradient Projection Memory (GPM) to prevent catastrophic forgetting in decentralized learning settings where data from different tasks is presented sequentially. The algorithm also reduces the communication cost by sending GPM coefficients of the model updates instead of the updates themselves. Experiments show accuracy improvements over baselines and upto 4.8x reduction in communication costs for a range of datasets and network topologies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It appears from Section 2.3 that SKILL is the closest related work to CoDeC and as such I feel it should be included as a baseline if possible. Even if it learns the tasks independently and not collaboratively it should be possible to compare the final task level accuracy of SKILL and CoDeC as well as the communication cost of the two appraoches right? If it is not possible please clarify why that is the case.\n\n2. It is not clear from Lines 8 and 9 of Algorithm 1 why $\\mathbf{q}_{i}^{k}$ lies in the RGS subspace. Please clarify.\n\n3. From line 14 it seems like $\\hat{x_{0}}^{j}$ will need to be communicated to all nodes since we don't have any coefficients at the beginning to reconstruct it, unless $\\hat{x}_{0}^{j}$ is the same for all $j$. Please clarify if that is the case.\n\n 4. It is not clear to me how you ensure that the CGS vectors being added to the GPM matrix in round $\\tau$ are orthogonal to the CGS vectors added in previous rounds. Please explain this more clearly.\n\n5. Do you have any intuition/explanations for the mixed BWT results in Table 3 and 4? If yes, please include a discussion of the same in the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KMacWAsQ0S", "reviewer": "Reviewer_kjKk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1.\tThe considered problem, communication-efficient decentralized continual learning, is very important and well-motivated in the distributed learning and continual learning literatures.\n2.\tThe authors provide theoretical guarantees for the convergence rate of the proposed algorithm.\n3.\tThe authors conduct experiments on a wide range of datasets and evaluate the proposed algorithm under various performance metrics, including accuracy, backward transfer and communication compression.\n\n**Weaknesses:**\n\n1.\tThe provided theoretical results seem to only provide the convergence rate to the stationary point? Does the proposed decentralized continual learning scheme converge to the global optima? The authors should give more discussion on the convergence guarantee of algorithm CoDeC.\n2.\tHow can we see the performance of algorithm CoDeC in terms of communication efficiency from the provided theoretical results? Is there any theoretical guarantee for communication costs?\n3.\tHow do the provided theoretical results reflect that CoDeC works well in avoiding catastrophic forgetting? It would be better if the authors can give more discussion on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes CoDeC, a novel communication-efficient decentralized continual learning algorithm that addresses these challenges. CoDeC mitigates catastrophic forgetting while learning a distributed task sequence by incorporating orthogonal gradient projection within a gossip-based decentralized learning algorithm. In addition, CoDeC includes a novel lossless communication compression scheme based on the gradient subspaces. The authors theoretically analyze the convergence rate for their algorithm and demonstrate through an extensive set of experiments that CoDeC successfully learns distributed continual tasks with minimal forgetting. The proposed compression scheme results in up to 4.8× reduction in communication costs without any loss in performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the review on weaknesses above.", "broader_impact_concerns": "I do not have ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main concern raised by the reviewers is the unrealistic IID assumption, which will not be verified in practice. The authors discussed this in the revised version and show encouraging new experimental results, but the concern remains. While I share the concern that this work might not be effective in practice, the proposed method is sound and algorithms appear correct. In line with TMLR's goals, I am thus in favour of accepting the work.
Decision: Accept (poster)