SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Agents (n)	Method	ResNet-20
α = 1	α = 0.1	α = 0.01
16	DSGDm (IID)	89.75 ± 0.29
DSGD [15]	84.17 ± 0.32	72.21 ± 2.37	54.66 ± 4.74
GUT (ours)	84.72 ± 0.20	81.86 ± 1.99	70.16 ± 4.94
QG-DSGDm [16]	88.23 ± 0.51	84.21 ± 2.12	79.85 ± 2.11
QG-GUTm (ours)	88.22 ± 0.36	86.44 ± 0.36	81.04 ± 1.66
32	DSGDm (IID)	88.52 ± 0.23
DSGD [15]	78.25 ± 0.42	62.97 ± 1.90	42.58 ± 1.84
GUT (ours)	79.24 ± 0.33	76.07 ± 0.23	60.72 ± 1.03
QG-DSGDm [16]	87.15 ± 0.33	83.50 ± 1.04	69.99 ± 0.60
QG-GUTm (ours)	87.48 ± 0.33	84.94 ± 0.60	72.04 ± 3.18
Agents (n)	Method	VGG-11
α = 1	α = 0.1	α = 0.01
16	DSGDm (IID)	85.76 ± 0.28
DSGD [15]	81.78 ± 0.29	76.20 ± 0.81	68.93 ± 1.23
GUT (ours)	82.12 ± 0.09	81.24 ± 0.95	76.62 ± 1.37
QG-DSGDm [16]	84.23 ± 0.47	81.70 ± 0.79	77.08 ± 3.19
QG-GUTm (ours)	84.46 ± 0.33	83.05 ± 0.48	78.32 ± 1.03
32	DSGDm (IID)	84.75 ± 0.30
DSGD [15]	79.75 ± 0.56	73.37 ± 1.02	59.93 ± 1.60
GUT (ours)	80.37 ± 0.33	79.55 ± 1.00	73.59 ± 1.26
QG-DSGDm [16]	83.67 ± 0.28	80.82 ± 0.19	74.25 ± 2.02
QG-GUTm (ours)	84.32 ± 0.11	83.39 ± 0.38	77.41 ± 3.44
Table 2: Average test accuracy of different decentralized algorithms evaluated on CIFAR-10 dataset trained on ResNet-20 over various graph topologies
Method	Dyck Graph (32 agents)	Torus (32 agents)
α = 0.1	α = 0.01	α = 0.1	α = 0.01
DSGDm [15]	77.31 ± 1.72	51.27 ± 2.51	75.37 ± 3.11	49.32 ± 2.88
QG-DSGDm [16]	86.49 ± 0.81	81.32 ± 1.50	86.88 ± 0.30	85.20 ± 0.56
QG-GUTm (ours)	86.93 ± 0.53	84.80 ± 0.47	87.75 ± 0.42	86.20 ± 0.82
Table 3: Average test accuracy of different decentralized algorithms evaluated on various datasets, distributed with different degrees of heterogeneity over 16 agents ring topology
Method	Fashion MNIST (LeNet-5)	CIFAR-100 (ResNet-20)	Imagenette (MobileNet-V2)
α = 0.1	α = 0.01	α = 0.1	α = 0.01	α = 0.1	α = 0.01
DSGdm [15]	86.59 ± 0.92	77.00 ± 3.53	47.93 ± 1.69	42.56 ± 2.71	66.02 ± 4.59	38.69 ± 11.8
QG-DSGdm [16]	89.94 ± 0.44	83.43 ± 0.94	53.19 ± 1.68	44.17 ± 3.64	63.60 ± 4.50	39.49 ± 4.57
QG-GUTm	90.11 ± 0.02	84.60 ± 1.00	53.40 ± 1.23	50.45 ± 1.30	66.52 ± 3.68	43.85 ± 8.24
We present the experimental results on various graph topologies and datasets to demonstrate the scalability and generalizability of  $QG-GUTm$ . We train the CIFAR-10 dataset on ResNet-20 over the Dyck graph and Torus graph to exemplify the impact of connectivity on the proposed technique. As shown in Table. 2, we obtain  $0.5 - 3.5\%$  performance gains with varying connectivity (or spectral gap).
Further, we evaluate  $QG$ -GUTm on various image datasets such as Fashion MNIST, and Imagenette and on challenging datasets such as CIFAR-100. Table 3 shows that  $QG$ -GUTm outperforms QGDSDm by  $0.2 - 6.2\%$  across various datasets. Therefore, in a decentralized deep learning setup, the proposed GUT and  $QG$ -GUTm algorithms are more robust to heterogeneity in the data distribution and can outperform all the comparison methods with an average improvement of  $2\%$ .
First, we analyze different ways of utilizing or tracking model update information as shown in Table. 4. We present two different update rules apart from GUT and DSGD[15] and also compare them with gradient tracking [19]. Rule-a applies the proposed tracking mechanism on model updates but does not change the reference of tracking variable  $y_{j}$  received from the neighbors to itself (refer sec. 3 for details on changing the reference). In the case of Rule-b, each agent computes the difference between the averaged neighborhood model update  $(W(X^{t} - X^{t - 1}))$  along with its own model update  $(X^{t} - X^{t - 1})$  and adds the difference between the two as a bias correction. Table. 4 shows that such naive ways of tracking or bias correction update rules (rule-a,b) do not improve the performance of decentralized learning on heterogeneous data. This confirms our findings that the GUT technique is an effective and provable way to track the consensus model and can outperform the gradient tracking mechanism without any communication overhead.
Table 4: Analyzing the different variations of model updates. Evaluating test accuracy on CIFAR-10 dataset trained on ResNet-20 over a 16 agent ring topology with  $\alpha = 0.1$
Method	Update Rule	Communication Parameters (Cost)	Test accuracy
DSGD	Xt+1=Xt-ηYt	Xt(1×)	72.21 ± 2.37
Yt=Gt-1/η(W-I)Xt
Rule-a	Xt+1=Xt-ηYt	Yt(1×)	72.78 ± 0.80
Yt=Gt-1/η(W-I)Xt+μ[WT-1-(Gt-1-1/η(W-I)Xt-1)]
Rule-b	Xt+1=Xt-ηYt	Yt(1×)	72.62 ± 1.16
Yt=Gt-1/η(W-I)Xt+μ[-1/η(W-I)(Xt-Xt-1)]
GUT	Xt+1=Xt-ηYt	Yt(1×)	81.86 ± 1.99
Yt=Gt-1/η(W-I)Xt+μ[WT-1-Gt-1-1/η(W-I)(Xt-Xt-1)]
Gradient Tracking	Xt+1=Xt-η[Yt-1/η(W-I)Xt]	Xt,Yt(2×)	80.61 ± 2.41
Yt=Gt+WYt-1-Gt-1
Table 5: Evaluating Global Update Tracking (GUT) with various versions of momentum using CIFAR-10 dataset trained on ResNet-20 architecture over 16 agents ring topology
Method	Local Momentum	Nesterov	Quasi-Global Momentum	Global Update Tracking	Test Accuracy α = 0.1
DSGD	x	x	x	x	72.21 ± 2.37
DSGdm	✓	x	x	x	79.87 ± 1.73
DSGDm-N	✓	✓	x	x	81.31 ± 0.51
QG-DSGdm	x	x	✓	x	84.21 ± 2.12
QG-DSGDm-N	x	✓	✓	x	85.12 ± 1.11
GUT	x	x	x	✓	81.86 ± 1.99
GUTm	✓	x	x	✓	79.95 ± 1.67
GUTm-N	✓	✓	x	✓	82.08 ± 1.74
QG-GUTm	x	x	✓	✓	86.44 ± 0.36
QG-GUTm-N	x	✓	✓	✓	86.55 ± 0.49
We then proceed to investigate the effect of different variants of momentum with GUT. From Table. 5 (refer to Appendix D for more results), we can conclude that the quasi-global variant of Global Update Tracking always surpasses the other methods. This indicates that the proposed GUT algorithm accelerates decentralized optimization and can be used in synergy with quasi-global momentum to achieve maximal performance gains.
Furthermore, Figure 3(a) illustrates the effect of scaling  $\mu$  on the test accuracy with QG-GUTm and note that  $\mu = 0$  shows the test accuracy for QG-DSGDm. Figure 3(b), 3(c) showcase the scalability
of  $QG$ -GUTm on different graph sizes and model sizes.  $QG$ -GUTm outperforms QG-DSGDm by  $\sim 1.7\%$  over different graph sizes and  $\sim 1.4\%$  over different model sizes.
(a)  $n = 16$  , ResNet-20
(b)  $\alpha = 0.1$  , ResNet-20
Figure 3: Ablation study on the hyper-parameter  $\mu$ , number of agents  $n$  and model size. The test accuracy is reported for the CIFAR-10 dataset trained on ResNet architecture over ring topology.
(c)  $\alpha = 0.1, n = 16$
We demonstrated the superiority of the Global Update Tracking (GUT) algorithm through an elaborate set of experiments and ablation studies. In our experiments, we focused on doubly-stochastic and symmetric graph structures. The proposed GUT algorithm can be easily extended to directed and time-varying graphs by combining it with stochastic gradient push (SGP) [2]. Further, the additional terms added by GUT can also be interpreted as a bias correction mechanism where the added bias pushes the local model towards the consensus (averaged) model. The matrix representation of this interpretation of GUT is given by (13). and analyzed in Lemma 1.
$$
X ^ {t + 1} = W X ^ {t} - \eta \left(G ^ {t} + \mu B ^ {t}\right); \quad B ^ {t + 1} = - \frac {1}{\eta} \left[ \left(2 W - I\right) \left(X ^ {t + 1} - X ^ {t}\right) + \eta G ^ {t} \right]. \tag {13}
$$
Lemma 1. Given assumptions 3, we define  $\bar{b}^t = B^t\frac{1}{n}\mathbb{1}\mathbb{1}^T$ , where  $\mathbb{1}$  is a vector of all ones. For all  $t$ , we have:  $\bar{b}^t = \mu \bar{b}^{t - 1}$ .
A complete proof for Lemma 1 can be found in Appendix A.2. Lemma 1 highlights that the average bias added by  $GUT$  is zero as  $\bar{b}^0$  is zero. Hence, the  $GUT$  algorithm crucially preserves the average value of the decentralized system. A feature we leverage to establish Theorem 1.
There are two potential limitations of the GUT algorithm - a) memory overhead and b) introduction of an additional hyper-parameter. GUT requires the agents to keep a copy of the averaged model parameters of their neighbors which adds an extra memory buffer of the size of model parameters. The storage of the tracking variable also adds to the memory overhead, requiring additional memory equivalent to the size of model parameters. We also introduce a new hyper-parameter  $\mu$  which has to be tuned similarly to the learning rate or momentum coefficient tuning. Besides, the theoretical analysis presented for the GUT algorithm does not consider momentum and assumes the communication to be synchronous. We leave the theoretical analysis of  $QG-GUTm$  and formulation of the asynchronous version of GUT as a future research direction.
Decentralized learning on heterogeneous data is the key to launching ML training on edge devices and thereby efficiently leveraging the humongous amounts of user-generated private data. In this paper, we propose Global Update Tracking (GUT), a novel decentralized algorithm designed to improve learning over heterogeneous data distributions. The convergence analysis presented in the paper shows that the proposed algorithm matches the best-known rate for decentralized algorithms. Additionally, the paper introduces a quasi-global momentum version of the algorithm, QG-GUTm, to further enhance the performance gains. The empirical evidence from experiments on different model architectures, datasets, and topologies demonstrates the superior performance of both algorithms. In summary, the proposed algorithm and its quasi-global momentum version have the potential to facilitate more scalable and efficient decentralized learning on edge devices.
This work was supported in part by, the Center for the Co-Design of Cognitive Systems (COCOSYS), a DARPA-sponsored JUMP center, the Semiconductor Research Corporation (SRC), and the National Science Foundation.
in Neural Information Processing Systems, volume 33, pages 13539-13550. Curran Associates, Inc., 2020.
[18] Angela Nedic. Distributed gradient methods for convex machine learning problems in networks: Distributed optimization. IEEE Signal Processing Magazine, 37(3):92-101, 2020.
[19] Shi Pu and Angela Nedic. Distributed stochastic gradient tracking methods. Mathematical Programming, 187:409-457, 2021.
[20] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[21] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[22] Yuki Takezawa, Han Bao, Kenta Niwa, Ryoma Sato, and Makoto Yamada. Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data. arXiv preprint arXiv:2209.15505, 2022.
[23] Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu.  $d^2$ : Decentralized training over decentralized data. In International Conference on Machine Learning, pages 4848-4856. PMLR, 2018.
[24] Thijs Vogels, Lie He, Anastasiia Koloskova, Sai Praneeth Karimireddy, Tao Lin, Sebastian U Stich, and Martin Jaggi. Relaysum for decentralized deep learning on heterogeneous data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 28004–28015. Curran Associates, Inc., 2021.
[25] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
[26] Lin Xiao and Stephen Boyd. Fast linear iterations for distributed averaging. Systems & Control Letters, 53(1):65-78, 2004.
A Convergence Rate Proof
In this work, we solve the optimization problem of minimizing global loss function  $f(x)$  distributed across  $n$  agents as given below. Note that  $F_{i}$  is a local loss function (for example, cross-entropy loss) defined in terms of the data sampled  $(d_{i})$  from the local dataset  $D_{i}$  at agent  $i$ .
$$
\min  _ {x \in \mathbb {R} ^ {d}} f (x) = \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (x),
$$
$$
a n d f _ {i} (x) = \mathbb {E} _ {d _ {i} \in D _ {i}} \left[ F _ {i} (x; d _ {i}) \right] \forall i.
$$
We reiterate the update scheme of GUT presented in Algorithm. 1 in a matrix form:
$$
X ^ {t + 1} = X ^ {t} - \eta Y ^ {t}
$$
$$
Y ^ {t} = \Delta^ {t} + \mu \left[ W Y ^ {t - 1} - \frac {1}{\eta} (W - I) X ^ {t} - \Delta^ {t - 1} \right] \tag {14}
$$
$$
\Delta^ {t} = G ^ {t} - \frac {1}{\eta} (W - I) X ^ {t},
$$
where  $W$  is the mixing matrix,  $I$  is the identity matrix,  $X = [x_{1}, x_{2}, \ldots, x_{n}] \in \mathbb{R}^{d \times n}$  is the matrix containing model parameters,  $x_{i} \in \mathbb{R}^{d}$  is model parameters of agent  $i$ ,  $Y = [y_{1}, y_{2}, \ldots, y_{n}] \in \mathbb{R}^{d \times n}$  is the matrix containing tracking variables,  $G = [g_{1}, g_{2}, \ldots, g_{n}] \in \mathbb{R}^{d \times n}$  is the matrix containing local gradients,  $\mu$  is the GUT scaling factor,  $\eta$  is the learning rate. Now, we rewrite the above equation in the form of a bias correction update,
$$
X ^ {t + 1} = W X ^ {t} - \eta \left(G ^ {t} + \mu B ^ {t}\right)
$$
$$
B ^ {t} = - \frac {1}{\eta} \left[ \left(2 W - I\right) \left(X ^ {t} - X ^ {t - 1}\right) + \eta G ^ {t - 1} \right]. \tag {15}
$$
A.1 Assumptions
We assume that the following statements hold:
Assumption 1 - Lipschitz Gradients: Each function  $f_{i}(x)$  is L-smooth i.e.,  $||\nabla f_{i}(y) - \nabla f_{i}(x)|| \leq L||y - x||$ . Equivalently,
$$
f _ {i} (y) \leq f _ {i} (x) + \left\langle \nabla f _ {i} (x), y - x \right\rangle + \frac {L}{2} \left\| y - x \right\| ^ {2} \tag {16}
$$
Assumption 2 - Bounded Variance: The variance of the stochastic gradients is assumed to be bounded.
$$
\mathbb {E} _ {d \sim D _ {i}} | | \nabla F _ {i} (x; d) - \nabla f _ {i} (x) | | ^ {2} \leq \sigma^ {2} \forall i \in [ 1, n ]
$$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} | | \nabla f _ {i} (x) - \nabla f (x) | | ^ {2} \leq \zeta^ {2}
$$
Assumption 3 - Doubly Stochastic Mixing Matrix: The mixing matrix  $W$  is a real doubly stochastic matrix with  $\lambda_1(W) = 1$  and
$$
\max  \left\{\left| \lambda_ {2} (W) \right|, \left| \lambda_ {n} (W) \right| \right\} \leq 1 - \rho <   1
$$
where  $\lambda_{i}(W)$  is the  $i^{th}$  largest eigenvalue of W and  $\rho$  is the spectral gap. The mixing matrix satisfies  $\mathbb{E}_W||ZW - \bar{Z} ||_F^2\leq (1 - \rho)||ZW - \bar{Z} ||_F^2$ , where  $\bar{Z} = Z\frac{1}{n}\mathbb{1}\mathbb{1}^T$ . We also have  $W\mathbb{1} = \mathbb{1}$  and  $W^{T}\mathbb{1} = \mathbb{1}$ .
Further, we define the average gradients  $\bar{g}^t = \frac{1}{n}\sum_{i=1}^{n}\nabla F_i(x_i^t,d_i^t)$  where  $d_i^t$  is sampled mini-batch of data on node  $i$
Lemma 1: Given assumptions 3, we define  $\bar{b}^t = B^t\frac{1}{n}\mathbb{1}\mathbb{1}^T$ , where  $\mathbb{1}$  is a vector of all ones. For all  $t$ , we have:  $\bar{b}^t = \mu \bar{b}^{t - 1}$ .
Proof. Starting from the definition of  $B^t$
$$
B ^ {t} = - \frac {1}{\eta} [ (2 W - I) (X ^ {t} - X ^ {t - 1}) + \eta G ^ {t - 1} ]
$$
multiply  $\frac{1}{n}\mathbb{1}\mathbb{1}^T$  on both sides
$$
\bar {b} ^ {t} = - \frac {1}{\eta} \left[ \bar {x} ^ {t} - \bar {x} ^ {t - 1} + \eta \bar {g} ^ {t - 1} \right] \quad (\because (2 W - I) \mathbb {1} = \mathbb {1})
$$
now, multiplying  $\frac{1}{n}\mathbb{1}\mathbb{1}^T$  to  $X^{t + 1} = WX^t -\eta (G^t +\mu B^t)$
$$
\begin{array}{l} \bar {x} ^ {t + 1} = \bar {x} ^ {t} - \eta \bar {g} ^ {t} - \eta \mu \bar {b} ^ {t} \Longrightarrow \bar {x} ^ {t} - \bar {x} ^ {t - 1} + \eta \bar {g} ^ {t - 1} = - \eta \mu \bar {b} ^ {t - 1} \\ \Longrightarrow \bar {b} ^ {t} = \mu \bar {b} ^ {t - 1} \\ \end{array}
$$
Given that  $\bar{b}^0 = 0$ , the average bias is zero at each iteration. This indicates that the proposed algorithm  $GUT$  preserves the average of the system in an average consensus task.
This section presents the detailed proof of the convergence bounds of GUT algorithm given by Theorem 1. Firstly, we analyze the one-step progress of the averaged model parameters  $\bar{x}$ . Note that,  $\bar{X}^t = [\bar{x}^t,\bar{x}^t,\dots ,\bar{x}^t ]\in \mathbb{R}^{d\times n}$  and  $\bar{x}^t = \frac{1}{n}\sum_{i = 1}^n x_i^t$
Lemma 2. Given assumptions 1-3 and  $\eta \leq \frac{1}{4L}$ , we have
$$
\mathbb {E} f (\bar {x} ^ {t + 1}) \leq \mathbb {E} f (\bar {x} ^ {t}) - \frac {\eta}{4} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} - \frac {\eta}{4} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f (x _ {i} ^ {t}) | | ^ {2} + \frac {L \eta^ {2} \sigma^ {2}}{n} + \frac {3 L \eta^ {2}}{n} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2}.
$$
Proof. From the definition of  $X^{t + 1}$ , we have
$$
\begin{array}{l} X ^ {t + 1} = W X ^ {t} - \eta [ G ^ {t} + \mu B ^ {t} ] \\ \Longrightarrow \bar {x} ^ {t + 1} = \bar {x} ^ {t} - \eta \bar {g} ^ {t} \quad (\because \bar {b} ^ {t} = 0 \text {f r o m L e m m a 1}) \\ \end{array}
$$
using L-smoothness assumption given by (16)
$$
\mathbb {E} f (\bar {x} ^ {t + 1}) \leq \mathbb {E} f (\bar {x} ^ {t}) + \mathbb {E} \langle \nabla f (\bar {x} ^ {t}), \bar {x} ^ {t + 1} - \bar {x} ^ {t} \rangle + \frac {L}{2} \mathbb {E} | | \bar {x} ^ {t + 1} - \bar {x} ^ {t} | | ^ {2}
$$
$$
\begin{array}{l} = \mathbb {E} f (\bar {x} ^ {t}) + \mathbb {E} \langle \nabla f (\bar {x} ^ {t}), - \eta \bar {g} ^ {t} \rangle + \frac {L \eta^ {2}}{2} \mathbb {E} | | \bar {g} ^ {t} | | ^ {2} \\ = \mathbb {E} f (\bar {x} ^ {t}) - \eta \mathbb {E} \langle \nabla f (\bar {x} ^ {t}), \mathbb {E} [ \bar {g} ^ {t} ] \rangle + \frac {L \eta^ {2}}{2} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla F _ {i} (x _ {i} ^ {t}) | | ^ {2} \\ = \mathbb {E} f (\bar {x} ^ {t}) - \eta \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \langle \nabla f (\bar {x} ^ {t}), \nabla f _ {i} \left(x _ {i} ^ {t}\right) \rangle + \frac {L \eta^ {2}}{2} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\nabla F _ {i} \left(x _ {i} ^ {t}\right) \pm \nabla f _ {i} \left(x _ {i} ^ {t}\right)\right) | | ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \mathbb {E} f (\bar {x} ^ {t}) - \eta \mathbb {E} \langle \nabla f (\bar {x} ^ {t}), \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} (x _ {i} ^ {t}) \rangle + \frac {L \eta^ {2}}{2} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} (x _ {i} ^ {t}) | | ^ {2} + \frac {L \eta^ {2} \sigma^ {2}}{n} \\ \stackrel {(b)} {=} \mathbb {E} f (\bar {x} ^ {t}) + \frac {L \eta^ {2} \sigma^ {2}}{n} + \frac {L \eta^ {2}}{2} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} (x _ {i} ^ {t}) | | ^ {2} - \frac {\eta}{2} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} \\ - \frac {\eta}{2} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} \left(x _ {i} ^ {t}\right) | | ^ {2} + \frac {\eta}{2} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\nabla f _ {i} \left(x _ {i} ^ {t}\right) - \nabla f (\bar {x} ^ {t})\right) | | ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(c)} {\leq} \mathbb {E} f (\bar {x} ^ {t}) + \frac {L \eta^ {2} \sigma^ {2}}{n} + \left(\frac {L \eta^ {2}}{2} - \frac {\eta}{2}\right) \mathbb {E} \| \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} (x _ {i} ^ {t}) \| ^ {2} - \frac {\eta}{2} \mathbb {E} \| \nabla f (\bar {x} ^ {t}) \| ^ {2} \\ + \frac {\eta}{2 n} \sum_ {i = 1} ^ {n} \mathbb {E} | | \nabla f _ {i} \left(x _ {i} ^ {t}\right) - \nabla f (\tilde {x} ^ {t}) | | ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(d)} {\leq} \mathbb {E} f (\bar {x} ^ {t}) + \frac {L \eta^ {2} \sigma^ {2}}{n} + (\frac {L \eta^ {2}}{2} - \frac {\eta}{2}) \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} (x _ {i} ^ {t}) | | ^ {2} - \frac {\eta}{2} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} \\ + \frac {L ^ {2} \eta}{2 n} \sum_ {i = 1} ^ {n} \mathbb {E} | | x _ {i} ^ {t} - \bar {x} ^ {t} | | ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(e)} {\leq} \mathbb {E} f (\bar {x} ^ {t}) - \frac {\eta}{4} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} (x _ {i} ^ {t}) | | ^ {2} - \frac {\eta}{4} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} + \frac {L \eta^ {2} \sigma^ {2}}{n} \\ + \frac {3 L ^ {2} \eta}{n} \sum_ {i = 1} ^ {n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} \\ \end{array}
$$
(a) uses assumption-2 ((6)). (b) uses the fact that  $-2\langle a,b\rangle = -||a||^2 -||b||^2 +||a - b||^2$  (c) uses Jensen's inequality. (d) uses L-smoothness condition. (e) follows from the assumption that  $\eta \leq \frac{1}{4L}$ .
Now, we proceed to bound the consensus error through Lemma 3.
Lemma 3. Given assumptions 1-3 and  $\eta \leq \frac{\rho}{7L}$ , we have
$$
\frac {1}{n} \mathbb {E} | | X ^ {t + 1} - \bar {X} ^ {t + 1} | | _ {F} ^ {2} \leq \frac {1 - \rho / 4}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + \frac {1 2 \eta^ {2} \zeta^ {2}}{\rho} + 4 \eta^ {2} \sigma^ {2} + \frac {6 \eta^ {2} \mu^ {2}}{\rho n} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2}.
$$
Proof. Starting from the update step 15
$$
\begin{array}{l} \frac {1}{n} \mathbb {E} | | X ^ {t + 1} - \bar {X} ^ {t + 1} | | _ {F} ^ {2} = \frac {1}{n} \mathbb {E} | | W X ^ {t} - \eta [ G ^ {t} + \mu B ^ {t} ] - (\bar {X} ^ {t} - \eta \bar {G} ^ {t}) | | _ {F} ^ {2} \\ = \frac {1}{n} \mathbb {E} | | W X ^ {t} - \bar {X} ^ {t} - \eta (G ^ {t} - \bar {G} ^ {t}) - \eta \mu B ^ {t} | | _ {F} ^ {2} \\ \leq \frac {1}{n} \mathbb {E} | | W X ^ {t} - \bar {X} ^ {t} - \eta (\mathbb {E} [ G ^ {t} ] - \mathbb {E} [ \bar {G} ^ {t} ]) - \eta \mu B ^ {t} | | _ {F} ^ {2} + 4 \eta^ {2} \sigma^ {2} \\ \stackrel {(a)} {\leq} \frac {1 + \rho / 2}{n} \mathbb {E} | | W X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + \frac {\eta^ {2} (1 + 2 / \rho)}{n} \mathbb {E} | | \mathbb {E} [ G ^ {t} ] - \mathbb {E} [ \bar {G} ^ {t} ] - \mu B ^ {t} | | _ {F} ^ {2} \\ + 4 \eta^ {2} \sigma^ {2} \\ \stackrel {(b)} {\leq} \frac {(1 - \rho) (1 + \rho / 2)}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + \frac {3 \eta^ {2}}{n \rho} \mathbb {E} | | \mathbb {E} [ G ^ {t} ] - \mathbb {E} [ \bar {G} ^ {t} ] - \mu B ^ {t} | | _ {F} ^ {2} \\ + 4 \eta^ {2} \sigma^ {2} \\ \leq \frac {(1 - \rho) (1 + \rho / 2)}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + 4 \eta^ {2} \sigma^ {2} + \frac {6 \eta^ {2}}{n \rho} \mathbb {E} | | \mathbb {E} [ G ^ {t} ] - \mathbb {E} [ \bar {G} ^ {t} ] | | _ {F} ^ {2} \\ + \frac {6 \eta^ {2} \mu^ {2}}{n \rho} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} \\ \leq \frac {(1 - \rho / 2)}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + 4 \eta^ {2} \sigma^ {2} + \frac {6 \eta^ {2}}{n \rho} \mathbb {E} | | \mathbb {E} [ G ^ {t} ] - \nabla f (\bar {x} ^ {t}) | | _ {F} ^ {2} \\ + \frac {6 \eta^ {2} \mu^ {2}}{n \rho} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} \\ \leq \frac {(1 - \rho / 2)}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + 4 \eta^ {2} \sigma^ {2} + \frac {6 \eta^ {2} \mu^ {2}}{n \rho} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} \\ + \frac {6 \eta^ {2}}{n \rho} \sum_ {i = 1} ^ {n} \mathbb {E} | | \nabla f _ {i} \left(x _ {i} ^ {t}\right) \pm \nabla f _ {i} (\bar {x} ^ {t}) - \nabla f (\bar {x} ^ {t}) | | _ {F} ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(c)} {\leq} \frac {(1 - \rho / 2)}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + 4 \eta^ {2} \sigma^ {2} + \frac {1 2 \eta^ {2} \zeta^ {2}}{\rho} + \frac {6 \eta^ {2} \mu^ {2}}{n \rho} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} \\ + \frac {1 2 \eta^ {2}}{n \rho} \sum_ {i = 1} ^ {n} \mathbb {E} | | \nabla f _ {i} \left(x _ {i} ^ {t}\right) - \nabla f _ {i} (\bar {x} ^ {t}) | | _ {F} ^ {2} \\ \stackrel {(d)} {\leq} \frac {(1 - \rho / 2)}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + 4 \eta^ {2} \sigma^ {2} + \frac {1 2 \eta^ {2} \zeta^ {2}}{\rho} + \frac {6 \eta^ {2} \mu^ {2}}{n \rho} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} \\ + \frac {1 2 \eta^ {2} L ^ {2}}{n \rho} \sum_ {i = 1} ^ {n} \mathbb {E} | | x _ {i} ^ {t} - \bar {x} ^ {t} | | _ {F} ^ {2} \\ = \left(\frac {1 - \rho / 2}{n} + \frac {1 2 \eta^ {2} L ^ {2}}{n \rho}\right) \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + 4 \eta^ {2} \sigma^ {2} + \frac {1 2 \eta^ {2} \zeta^ {2}}{\rho} \\ + \frac {6 \eta^ {2} \mu^ {2}}{n \rho} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} \\ \stackrel {(e)} {\leq} \frac {1 - \rho / 4}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + 4 \eta^ {2} \sigma^ {2} + \frac {1 2 \eta^ {2} \zeta^ {2}}{\rho} + \frac {6 \eta^ {2} \mu^ {2}}{n \rho} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} \\ \end{array}
$$