SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) follows from the fact that  $||a + b||^2 \leq (1 + \alpha)||a||^2 + (1 + \frac{1}{\alpha})||b||^2 \forall \alpha > 0$  and let  $\alpha = \frac{\rho}{2}$ . (b) uses  $\mathbb{E}_W||ZW - \bar{Z}||_F^2 \leq (1 - \rho)||ZW - \bar{Z}||_F^2$  and  $1 + \frac{2}{\rho} \leq \frac{3}{\rho}$ . (c) uses assumption-2 (7). (d) uses L-smoothness condition. (e) follows from the assumption that  $\eta \leq \frac{\rho}{7L}$
The next step is to find an upper bound for the bias term  $\mathbb{E}||B^t ||_F^2$
Lemma 4. Given assumptions 1-3 and  $\frac{\mu}{1 - \mu} \leq \frac{\rho}{42}$ , we have
$$
\frac {6 \eta^ {2} \mu^ {2}}{\rho n (1 - \mu)} \mathbb {E} | | B ^ {t + 1} | | _ {F} ^ {2} \leq \left(\frac {6 \eta^ {2} \mu^ {2}}{\rho n (1 - \mu)} - \frac {6 \eta^ {2} \mu^ {2}}{\rho n}\right) \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + \frac {\rho}{8 n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + \frac {\eta^ {2} \zeta^ {2} \rho}{8} + \frac {\eta^ {2} \sigma^ {2} \rho (1 - \mu)}{8}.
$$
Proof. starting from the update step 15
$$
\begin{array}{l} B ^ {t + 1} = - \frac {1}{\eta} \left[ (2 W - I) \left(X ^ {t + 1} - X ^ {t}\right) + \eta G ^ {t} \right] \\ = - \frac {1}{\eta} \left[ (2 W - I) \left(W X ^ {t} - \eta G ^ {t} - \eta \mu B ^ {t} - X ^ {t}\right) + \eta G ^ {t} \right] \\ = - \frac {1}{\eta} [ W (2 W - I) - I ] X ^ {t} + 2 (W - I) G ^ {t} + \mu (2 W - I) B ^ {t}. \\ \end{array}
$$
$$
\frac {1}{n} \mathbb {E} | | B ^ {t + 1} | | _ {F} ^ {2} = \frac {1}{n} \mathbb {E} | | - \frac {1}{\eta} (W (2 W - I) - I) X ^ {t} + 2 (W - I) G ^ {t} + \mu (2 W - I) B ^ {t} | | _ {F} ^ {2}
$$
$$
\begin{array}{l} = \frac {1}{n} \mathbb {E} | | - \frac {1}{\eta} (W (2 W - I) - I) X ^ {t} + 2 (W - I) \left(G ^ {t} - \bar {G} ^ {t}\right) + \mu (2 W - I) B ^ {t} | | _ {F} ^ {2} \\ = \frac {1}{n} \mathbb {E} | | - \frac {1}{\eta} (W (2 W - I) - I) X ^ {t} + 2 (W - I) \mathbb {E} [ G ^ {t} - \bar {G} ^ {t} ] + \mu (2 W - I) B ^ {t} | | _ {F} ^ {2} \\ + \frac {1}{n} \mathbb {E} | | 2 (W - I) \left(G ^ {t} - \mathbb {E} [ G ^ {t} ] - \left(\bar {G} ^ {t} - \mathbb {E} [ \bar {G} ^ {t} ]\right)\right) | | _ {F} ^ {2} \\ \leq \frac {1}{n} \mathbb {E} | | \frac {1}{\eta} (I - W (2 W - I)) X ^ {t} + 2 (W - I) \mathbb {E} [ G ^ {t} - \bar {G} ^ {t} ] + \mu (2 W - I) B ^ {t} | | _ {F} ^ {2} \\ + 8 \sigma^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \frac {1}{n} \left(1 + \frac {1 - \mu}{\mu}\right) \mathbb {E} | | \mu (2 W - I) B ^ {t} | | _ {F} ^ {2} + 8 \sigma^ {2} \\ + \frac {1}{n} \left(1 + \frac {\mu}{1 - \mu}\right) \mathbb {E} | | \frac {1}{\eta} (I - W (2 W - I)) X ^ {t} + 2 (W - I) \mathbb {E} [ G ^ {t} - \bar {G} ^ {t} ] | | _ {F} ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \leq \frac {\mu}{n} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + 8 \sigma^ {2} + \frac {2}{n (1 - \mu)} \mathbb {E} | | \mathbb {E} [ G ^ {t} - \bar {G} ^ {t} ] | | _ {F} ^ {2} \\ + \frac {2}{n \eta^ {2} (1 - \mu)} \mathbb {E} | | (I - W (2 W - I)) X ^ {t} | | _ {F} ^ {2} \\ = \frac {1 - (1 - \mu)}{n} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + 8 \sigma^ {2} + \frac {2}{n (1 - \mu)} \mathbb {E} | | \mathbb {E} [ G ^ {t} - \bar {G} ^ {t} ] | | _ {F} ^ {2} \\ + \frac {2}{n \eta^ {2} (1 - \mu)} \mathbb {E} | | (2 W + I) (I - W) X ^ {t} | | _ {F} ^ {2} \\ \stackrel {(b)} {\leq} \frac {1 - (1 - \mu)}{n} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + 8 \sigma^ {2} + \frac {2}{n (1 - \mu)} \mathbb {E} | | \mathbb {E} [ G ^ {t} - \bar {G} ^ {t} ] | | _ {F} ^ {2} \\ + \frac {1 8}{n \eta^ {2} (1 - \mu)} \mathbb {E} | | (I - W) X ^ {t} | | _ {F} ^ {2} \\ = \frac {1 - (1 - \mu)}{n} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + 8 \sigma^ {2} + \frac {2}{n (1 - \mu)} \mathbb {E} | | \mathbb {E} [ G ^ {t} - \bar {G} ^ {t} ] | | _ {F} ^ {2} \\ + \frac {1 8}{n \eta^ {2} (1 - \mu)} \mathbb {E} | | (I - W) (X ^ {t} - \bar {X} ^ {t}) | | _ {F} ^ {2} \\ \leq \frac {1 - (1 - \mu)}{n} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + 8 \sigma^ {2} + \frac {3 6}{n \eta^ {2} (1 - \mu)} \mathbb {E} | | X ^ {t} - \bar {X}) | | _ {F} ^ {2} \\ + \frac {2}{n (1 - \mu)} \mathbb {E} | | \mathbb {E} [ G ^ {t} ] \pm \nabla f (\bar {x} ^ {t}) - \mathbb {E} [ \bar {G} ^ {t} ] | | _ {F} ^ {2} \\ \leq \frac {1 - (1 - \mu)}{n} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + 8 \sigma^ {2} + \frac {3 6}{n \eta^ {2} (1 - \mu)} \mathbb {E} | | X ^ {t} - \bar {X}) | | _ {F} ^ {2} \\ + \frac {8 \zeta^ {2}}{1 - \mu} + \frac {4 L ^ {2}}{n (1 - \mu)} \mathbb {E} | | X ^ {t} - \bar {X}) | | _ {F} ^ {2} \\ = \frac {1 - (1 - \mu)}{n} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + 8 \sigma^ {2} + \frac {4 (9 + \eta^ {2} L ^ {2})}{n \eta^ {2} (1 - \mu)} \mathbb {E} | | X ^ {t} - \bar {X}) | | _ {F} ^ {2} + \frac {8 \zeta^ {2}}{1 - \mu} \\ \end{array}
$$
Multiplying both sides with  $\frac{6\eta^2\mu^2}{\rho(1 - \mu)}$
$$
\begin{array}{l} \frac {6 \eta^ {2} \mu^ {2}}{n \rho (1 - \mu)} \mathbb {E} | | B ^ {t + 1} | | _ {F} ^ {2} \leq \left(\frac {6 \eta^ {2} \mu^ {2}}{n \rho (1 - \mu)} - \frac {6 \eta^ {2} \mu^ {2}}{n \rho}\right) \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + \frac {2 4 (9 + \eta^ {2} L ^ {2}) \mu^ {2}}{n \rho (1 - \mu) ^ {2}} \mathbb {E} | | X ^ {t} - \bar {X}) | | _ {F} ^ {2} \\ + \frac {4 8 \eta^ {2} \mu^ {2} \sigma^ {2}}{\rho (1 - \mu)} + \frac {4 8 \eta^ {2} \mu^ {2} \zeta^ {2}}{\rho (1 - \mu) ^ {2}} \\ \stackrel {(c)} {\leq} \left(\frac {6 \eta^ {2} \mu^ {2}}{n \rho (1 - \mu)} - \frac {6 \eta^ {2} \mu^ {2}}{n \rho}\right) \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + \frac {\rho}{8 n} \mathbb {E} | | X ^ {t} - \bar {X}) | | _ {F} ^ {2} \\ + \frac {\eta^ {2} \rho \sigma^ {2} (1 - \mu)}{8} + \frac {\eta^ {2} \rho \zeta^ {2}}{8} \\ \end{array}
$$
Note that  $W - I < I$ ,  $I - W < 2I$ ,  $(W - I)\bar{X}^t = 0$  and  $(W - I)\bar{G}^t = 0$ . (a) follows from the fact that  $\| a + b\|^2 \leq (1 + \alpha)\| a\|^2 + (1 + \frac{1}{\alpha})\| b\|^2 \forall \alpha > 0$  and let  $\alpha = \frac{1 - \mu}{\mu}$ . (b) uses the fact that  $\| AB\|_F^2 \leq \sigma_{max}^2(A)\| B\|_F^2$  where  $A = 2W + I$ ,  $B = (I - W)X^t$  and  $\sigma_{max}^2(A) = 9$ . (c) uses the assumption  $\frac{\mu}{1 - \mu} \leq \frac{\rho}{42}$  and  $\eta \leq \frac{\rho}{7L}$ . This implies that  $\frac{24(9 + \eta^2L^2)\mu^2}{\rho(1 - \mu)^2} \leq \frac{\rho}{8}$  and  $\frac{48\mu^2}{\rho(1 - \mu)^2} \leq \frac{\rho}{8}$
We present the proof for Theorem 1 using Lemmas. 2, 3, and 4. Adding Lemmas. 3, and 4 and simplifying, we get
$$
\begin{array}{l} \frac {2 4 L ^ {2} \eta}{n} \mathbb {E} | | X ^ {t + 1} - \bar {X} ^ {t + 1} | | _ {F} ^ {2} + \frac {1 4 4 L ^ {2} \eta^ {3} \mu^ {2}}{n \rho^ {2} (1 - \mu)} \mathbb {E} | | B ^ {t + 1} | | _ {F} ^ {2} \leq \left(\frac {2 4 L ^ {2} \eta}{n} - \frac {3 L ^ {2} \eta}{n}\right) \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} \tag {17} \\ + \frac {1 4 4 L ^ {2} \eta^ {3} \mu^ {2}}{n \rho^ {2} (1 - \mu)} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + \frac {3 1 2 L ^ {2} \eta^ {3}}{\rho^ {2}} (\zeta^ {2} + \sigma^ {2} (2 - \mu)) \\ \end{array}
$$
Finally, define a function  $\Phi^t$  as shown below
$$
\Phi^ {t} = \frac {2 4 L ^ {2} \eta}{n} \mathbb {E} | | X ^ {t} - \bar {X} ^ {t} | | _ {F} ^ {2} + \frac {1 4 4 L ^ {2} \eta^ {3} \mu^ {2}}{n \rho^ {2} (1 - \mu)} \mathbb {E} | | B ^ {t} | | _ {F} ^ {2} + \mathbb {E} [ f (\bar {x} ^ {t}) - f ^ {*} ] \tag {18}
$$
Now adding Lemma 2 and (17), we have the following
$$
\begin{array}{l} \Phi^ {t + 1} \leq \Phi^ {t} - \frac {\eta}{4} \mathbb {E} | | \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} \left(x _ {i} ^ {t}\right) | | ^ {2} - \frac {\eta}{4} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} + \frac {L \eta^ {2} \sigma^ {2}}{n} + \frac {3 1 2 L ^ {2} \eta^ {3}}{\rho^ {2}} \left(\zeta^ {2} + \sigma^ {2} (2 - \mu)\right) \\ \leq \Phi^ {t} - \frac {\eta}{4} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} + \frac {L \eta^ {2} \sigma^ {2}}{n} + \frac {3 1 2 L ^ {2} \eta^ {3}}{\rho^ {2}} (\zeta^ {2} + \sigma^ {2} (2 - \mu)) \\ \Longrightarrow \frac {\eta}{4} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} \leq \left(\Phi^ {t} - \Phi^ {t + 1}\right) + \frac {L \eta^ {2} \sigma^ {2}}{n} + \frac {3 1 2 L ^ {2} \eta^ {3}}{\rho^ {2}} \left(\zeta^ {2} + \sigma^ {2} (2 - \mu)\right) \\ \end{array}
$$
Summing over t
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} \leq \frac {4}{\eta T} \left(f \left(\bar {x} ^ {0} - f ^ {*}\right) + \eta \frac {4 L \sigma^ {2}}{n} + \eta^ {2} \frac {1 2 4 8 L ^ {2}}{\rho^ {2}} \left(\zeta^ {2} + \sigma^ {2} (2 - \mu)\right). \right. \tag {19}
$$
This concludes the proof of the Theorem 1.
A.4 Proof of Corollary. 1
In the proof of Theorem 1, we assumed the following the following constraints on learning rate  $\eta$  and scaling factor  $\mu$ :
(i)  $\eta \leq \min \left\{\frac{1}{4L},\frac{\rho}{7L}\right\}$
(ii)  $\frac{\mu}{1 - \mu} \leq \frac{\rho}{42}$ .
We assume that the step size  $\eta$  is  $\mathcal{O}(\sqrt{\frac{n}{T}})$ , where  $n$  is the total number of agents and  $T$  is the number of iterations. Given this assumption, we have the following order of each term in (19) of Theorem 1.
$$
\frac {4}{\eta T} (f (\bar {x} ^ {0} - f ^ {*}) = \mathcal {O} \Big (\frac {1}{\sqrt {n T}} \Big).
$$
For the remaining terms we have,
$$
\eta \frac {4 L \sigma^ {2}}{n} = \mathcal {O} \Big (\frac {1}{\sqrt {n T}} \Big), \quad \eta^ {2} \frac {1 2 4 8 L ^ {2}}{\rho^ {2}} (\zeta^ {2} + \sigma^ {2} (2 - \mu)) = \mathcal {O} \Big (\frac {n}{T} \Big).
$$
Therefore, by omitting the constant  $n$  in this context of higher order terms, there exists a constant  $C > 0$  such that the overall convergence rate is as follows:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} \leq C \left(\frac {1}{\sqrt {n T}} + \frac {1}{T}\right),
$$
which suggests when  $T$  is sufficiently large, GUT enables the convergence rate of  $\mathcal{O}\left(\frac{1}{\sqrt{nT}}\right)$ .
B Algorithmic details
In this section, we present the pseudo-code for  $QG-GUTm$  which combines the proposed GUT algorithm with quasi-global momentum as Algorithm 2 and its PyTorch implementation version is shown in Algorithm 3. We also summarize the memory-efficient implementation of GUT in Algorithm 4.
Algorithm 2 Global Update Tracking with momentum (QG-GUTm)
Input: Each agent  $i \in [1, n]$  initializes model weights  $x_{i}^{(0)}$  and neighbors' copy  $\hat{x}_{j}^{(0)}$ , momentum buffer  $m_{i}^{(0)}$ , step size  $\eta$ , momentum coefficient  $\beta$ , mixing matrix  $W = [w_{ij}]_{i,j \in [1,n]}$ , GUT scaling factor  $\mu$ ,  $I_{ij}$  are elements of  $n \times n$  identity matrix,  $\mathcal{N}(i)$  represents neighbors of  $i$  including itself, and note  $\hat{x}_{i}^{t} = x_{i}^{t}$ .
1. procedure TRAIN()
2. for  $t = 0,1,\dots ,T - 1$  do
3.  $d_{i}^{t}\sim D_{i}$
4.  $g_{i}^{t} = \nabla_{x}f_{i}(d_{i}^{t};\sum_{j\in \mathcal{N}(i)}w_{ij}\ast \hat{x}_{j}^{t})$
5.  $\delta_i^t = g_i^t -\frac{1}{\eta}\sum_{j\in \mathcal{N}(i)}(w_{ij} - I_{ij})*\hat{x}_j^t$
6.  $y_{i}^{t} = \delta_{i}^{t} + \mu \big[\sum_{j\in \mathcal{N}(i)}w_{ij}(m_{j}^{t - 1} - \frac{1}{\eta} (\hat{x}_{j}^{t} - x_{i}^{t})) - \delta_{i}^{t - 1}\big]$
7.  $m_{i}^{t} = \beta m_{i}^{t - 1} + (1 - \beta)y_{i}^{t}$
8.  $\mathsf{SENDRECEIVE}(m_{ij}^{t})$
9.  $x_{i}^{t + 1} = x_{i}^{t} - \eta m_{i}^{t}$
10.  $\hat{x}_j^{t + 1} = \hat{x}_j^t -\eta m_j^t\forall j\in N(i)\backslash i$
11. end
12. return
Algorithm 3 Global Update Tracking with momentum (QG-GUTm) - Pytorch Implementation
Each agent simultaneously implements the TRAIN() procedure
2. for  $t = 0,1,\dots ,T - 1$  do
3.  $d_i^t\sim D_i$
4.  $g_{i}^{t} = \nabla_{x}f_{i}(d_{i}^{t};\sum_{j\in \mathcal{N}(i)}w_{ij}\ast \hat{x}_{j}^{t})$
5.  $\delta_{i}^{t} = g_{i}^{t} - \frac{1}{\eta}\sum_{j\in \mathcal{N}(i)}(w_{ij} - I_{ij})*\hat{x}_{j}^{t}$
6.  $y_{i}^{t} = \delta_{i}^{t} + \mu \big[\sum_{j\in \mathcal{N}(i)}w_{ij}(m_{j}^{t - 1} - \frac{1 + \beta}{\eta} (\hat{x}_{j}^{t} - x_{i}^{t})) - \delta_{i}^{t - 1}\big]$
7.  $m_{i}^{t} = \beta m_{i}^{t - 1} + y_{i}^{t}$
8.  $\mathbf{S}\mathbf{E}\mathbf{N}\mathbf{D}\mathbf{R}\mathbf{E}\mathbf{C}\mathbf{E}\mathbf{I}\mathbf{V}(\boldsymbol {m}_i^t)$
9.  $x_{i}^{t + 1} = x_{i}^{t} - \eta m_{i}^{t}$
10.  $\hat{x}_j^{t + 1} = \hat{x}_j^t -\eta m_j^t\forall j\in N(i)\backslash i$
11. end
12. return
Algorithm 4 Global Update Tracking (Memory Efficient Implementation)
Input: Each agent  $i \in [1, n]$  initializes model parameters  $x_i^0$  and weighted model parameters of neighborhood  $s_i^0$ , step size  $\eta$ , GUT scaling factor  $\mu$ , mixing matrix  $W = [w_{ij}]_{i,j \in [1,n]}$ ,  $\mathcal{N}(i)$  represents neighbors of  $i$  including itself.
1. procedure TRAIN()
2. for  $t = 0,1,\dots ,T - 1$  do
3.  $d_{i}^{t}\sim D_{i}$
4.  $g_{i}^{t} = \nabla_{x}F_{i}(s_{i}^{t};d_{i}^{t})$
5.  $\delta_i^t = g_i^t -\frac{1}{\eta} (s_i^t -x_i^t)$
6.  $y_{i}^{t} = \delta_{i}^{t} + \mu \Big[\sum_{j\in \mathcal{N}(i)}w_{ij}y_{j}^{t - 1} - \frac{1}{\eta} (s_{i}^{t} - x_{i}^{t}) - \delta_{i}^{t - 1}\Big]$
7. SENDRECEIVE  $(y_{i}^{t})$
8.  $x_{i}^{t + 1} = x_{i}^{t} - \eta y_{i}^{t}$
9.  $s_i^{t + 1} = s_i^t -\eta \sum_{j\in \mathcal{N}(i)}w_{ij}y_j$
10. end
11. return  $\frac{1}{n}\sum_{i = 1}^{n}x_i^T$
C Decentralized Learning Setup
For the decentralized setup, we use an undirected ring, undirected Dyck graph, and undirected torus graph topologies with a uniform mixing matrix. The undirected ring topology for any graph size has 3 peers per agent including itself and each edge has a weight of  $\frac{1}{3}$ . The undirected Dyck topology with 32 agents has 4 peers per agent including itself and each edge has a weight of  $\frac{1}{4}$ . The undirected torus topology with 32 agents has 5 peers per agent including itself and each edge has a weight of  $\frac{1}{5}$ . All our experiments were conducted on a system with Nvidia GTX 1080ti card with 4 GPUs except for ImageNette simulations. We used NVIDIA A40 card with 4 GPUs for ImageNette simulations.
In this section, we give a brief description of the datasets used in our experiments. We use a diverse set of datasets each originating from a different distribution of images to show the generalizability of the proposed techniques.
CIFAR-10: CIFAR-10 [13] is an image classification dataset with 10 classes. The image samples are colored (3 input channels) and have a resolution of  $32 \times 32$ . There are 50,000 training samples with 5000 samples per class and 10,000 test samples with 1000 samples per class.
CIFAR-100: CIFAR-100 [13] is an image classification dataset with 100 classes. The image samples are colored (3 input channels) and have a resolution of  $32 \times 32$ . There are 50,000 training samples with 500 samples per class and 10,000 test samples with 100 samples per class. CIFAR-100 classification is a harder task compared to CIFAR-10 as it has 100 classes with very few samples per class to learn from.
Fashion MNIST: Fashion MNIST [25] is an image classification dataset with 10 classes. The image samples are in greyscale (1 input channel) and have a resolution of  $28 \times 28$ . There are 60,000 training samples with 6000 samples per class and 10,000 test samples with 1000 samples per class.
Imagenette: Imagenette [10] is a 10-class subset of the ImageNet dataset. The image samples are colored (3 input channels) and have a resolution of  $224 \times 224$ . There are 9469 training samples with roughly 950 samples per class and 3925 test samples.
C.2 Network Architecture
We replace  $\mathrm{ReLU + }$  BatchNorm layers of all the model architectures with EvoNorm-S0 as it was shown to be better suited for decentralized learning over non-IID distributions.
VGG-11: We modify the standard VGG-11 [21] architecture by reducing the number of filters in each convolutional layer by  $4\times$  and using only one dense layer with 128 units. Each convolutional layer is followed by EvoNorm-S0 as the activation-normalization layer. VGG-11 has  $0.58M$  trainable parameters.
ResNet-20: For ResNet-20 [8], we use the standard architecture with  $0.27M$  trainable parameters except that BatchNorm+ReLU layers are replaced by EvoNorm-S0.
LeNet-5: For LeNet-5 [14], we use the standard architecture with 61, 706 trainable parameters.
MobileNet-V2: We use the standard MobileNet-V2 [20] architecture used for CIFAR dataset with  $2.3M$  parameters except that BatchNorm+ReLU layers are replaced by EvoNorm-S0.
C.3 Hyper-parameters
This section presents a detailed description of the hyper-parameters used in our experiments. All the experiments were run for three randomly chosen seeds. We decay the step size by  $10\mathrm{x}$  after  $50\%$  and  $75\%$  of the training, unless mentioned otherwise. The hyper-parameter  $\mu$  is set to 0.9 for all the experiments using GUT optimizer. We used grid search to choose the hyper-parameter  $\mu$  for QG-GUTm.
Hyper-parameters for experiments in Table 1: All the experiments have the stopping criteria set to 200 epochs. The initial learning rate is set to 0.1. We decay the step size by  $10 \times$  in multiple steps at  $100^{th}$  and  $150^{th}$  epoch. Table 6 presents values of the scaling factor  $\mu$  used in the experiments. For all the experiments, we use a mini-batch size of 32 per agent. The stopping criteria is a fixed number of epochs. We have used a momentum of 0.9 for all QG-DSDm and QG-GUTm experiments.
Table 6: The value of scaling factor  $\mu$  used for training CIFAR-10 with non-IID data using ResNet-20 and VGG-11 model architectures presented in Table 1
Agents (n)	Method	ResNet-20
α = 1	α = 0.1	α = 0.01
16	DSGD	0.0	0.0	0.0
GUT(ours)	0.9	0.9	0.9
QG-DSGDm	0.0	0.0	0.0
QG-GUTm (ours)	0.04	0.06	0.04
32	DSGD	0.0	0.0	0.0
GUT(ours)	0.9	0.9	0.9
QG-DSGDm	0.0	0.0	0.0
QG-GUTm (ours)	0.04	0.04	0.04
Agents (n)	Method	VGG-11
α = 1	α = 0.1	α = 0.01
16	DSGD	0.0	0.0	0.0
GUT(ours)	0.9	0.9	0.9
QG-DSGDm	0.0	0.0	0.0
QG-GUTm (ears)	0.06	0.08	0.09
32	DSGD	0.0	0.0	0.0
GUT(ours)	0.9	0.9	0.9
QG-DSGDm	0.0	0.0	0.0
QG-GUTm (ears)	0.08	0.08	0.08
Hyper-parameters for experiments in Table 2: All the experiments have the stopping criteria set to 200 epochs. The initial learning rate is set to 0.1. We decay the step size by  $10 \times$  in multiple steps at  $100^{th}$  and  $150^{th}$  epoch. Table 7 presents values of the scaling factor  $\mu$  used in the experiments. For all the experiments, we use a mini-batch size of 32 per agent. The stopping criteria is a fixed number of epochs. For all QG-DSDm and QG-GUTm experiments, we have used a momentum of 0.9 and Nesterov is set to False. We did not use any regularization in our experiments on non-IID data i.e., weight decay is set to zero. We set the weight decay to be  $1e^{-4}$  for experiments on IID data (DSGDm case in Table. 1).
Hyper-parameters for experiments in Table 3: All the experiments with Fashion-MNIST and Imagenette datasets have the stopping criteria set to 100 epochs where as CIFAR-100 experiments
Table 7: The value of scaling factor  $\mu$  used for training CIFAR-10 with non-IID data using ResNet-20 model architecture over varies graph topologies presented in Table 2
Method	Dyck Graph (32 agents)	Torus (32 agents)
α = 0.1	α = 0.01	α = 0.1	α = 0.01
QG-DSGDm	0.0	0.0	0.0	0.0
QG-GUTm	0.05	0.05	0.05	0.05
have the stopping criteria as 200 epochs. The initial learning rate is set to 0.1 for experiments on CIFAR-100 and Fashion MNIST datasets. The initial learning rate is set to 0.01 for experiments on the Imagenette dataset. We decay the step size by  $10\times$  in multiple steps at  $50^{th}$  and  $75^{th}$  epoch. Table 8 presents values of the scaling factor  $\mu$  used in the experiments. For all the experiments, we use a mini-batch size of 32 per agent. The stopping criteria is a fixed number of epochs. We have used a momentum of 0.9 for all QG-DSDm and QG-GUTm experiments.
Table 8: The value of scaling factor  $\mu$  used for training different datasets over 16 agents ring topology presented in Table 3
Method	Fashion MNIST (LeNet-5)	CIFAR-100 (ResNet-20)	Imagenette (MobileNet-V2)
α = 0.1	α = 0.01	α = 0.1	α = 0.01	α = 0.1	α = 0.01
QG-DSGDm	0.0	0.0	0.0	0.0	0.0	0.0
QG-GUTm	0.01	0.005	0.005	0.005	0.03	0.04
Table. 9 shows that the quasi-global variant of Global Update Tracking surpasses the other existing methods even for a higher degree of heterogeneity i.e.,  $\alpha = 0.01$ . We also observe that the Nesterov momentum hurts the performance when the heterogeneity in the data distribution is high. Table. 10 compares the DSGDm baseline with the proposed GUT and QG-GUTm. It shows that for a higher degree of heterogeneity, GUT outperforms even the momentum version of DSGD. Table. 11 evaluates the proposed method on various datasets trained on same model architecture (ResNet-20).
Table 9: Evaluating Global Update Tracking (GUT) with various versions of momentum using CIFAR-10 dataset trained on ResNet-20 architecture over 16 agents ring topology for  $\alpha = 0.01$ .
Method	Local Momentum	Nesterov	Quasi-Global Momentum	Global Update Tracking	Test Accuracy α = 0.01
DSGD	x	x	x	x	54.66 ± 4.74
DSGdm	✓	x	x	x	65.62 ± 4.95
DSGDm-N	✓	✓	x	x	63.66 ± 4.44
QG-DSGdm	x	x	✓	x	79.85 ± 2.13
QG-DSGDm-N	x	✓	✓	x	78.64 ± 2.14
GUT	x	x	x	✓	70.16 ± 4.94
GUTm	✓	x	x	✓	64.25 ± 5.31
GUTm-N	✓	✓	x	✓	63.42 ± 2.74
QG-GUTm	x	x	✓	✓	81.04 ± 1.66
QG-GUTm-N	x	✓	✓	✓	80.09 ± 3.82
E Hardware Efficiency
In this section, we present the quantitative results on communication, memory, and compute overheads of the various decentralized algorithms. The communication cost incurred by the proposed GUT
Table 10: Average test accuracy of different decentralized algorithms evaluated on CIFAR-10, distributed with different degrees of heterogeneity (non-IID) for various models over ring topologies.
Agents (n)	Method	ResNet-20
α = 1	α = 0.1	α = 0.01
16	GUT(ours)	84.72 ± 0.20	81.86 ± 1.99	70.16 ± 4.94
DSGDm [15]	86.60 ± 0.54	79.87 ± 1.73	65.62 ± 4.95
QG-GUTm (ours)	88.22 ± 0.36	86.44 ± 0.36	81.04 ± 1.66
32	GUT(ours)	79.24 ± 0.33	76.07 ± 0.23	60.72 ± 1.03
DSGDm [15]	86.12 ± 0.32	77.43 ± 1.71	52.82 ± 4.02
QG-GUTm (ours)	87.48 ± 0.33	84.94 ± 0.60	72.04 ± 3.18
Agents (n)	Method	VGG-11
α = 1	α = 0.1	α = 0.01
16	GUT(ours)	82.12 ± 0.09	81.24 ± 0.95	76.62 ± 1.37
DSGDm [15]	81.77 ± 0.38	74.20 ± 1.89	58.44 ± 14.58
QG-GUTm (ours)	84.46 ± 0.33	83.05 ± 0.48	78.32 ± 1.03
32	GUT(ours)	80.37 ± 0.33	79.55 ± 1.00	73.59 ± 1.26
DSGDm [15]	81.89 ± 0.29	74.73 ± 0.73	61.60 ± 2.80
QG-GUTm (ours)	84.32 ± 0.11	83.39 ± 0.38	77.41 ± 3.44
Table 11: Average test accuracy of different decentralized algorithms evaluated with various datasets trained on ResNet-20 architecture, distributed with different degrees of heterogeneity over 16 agents ring topology.
Method	Fashion MNIST	CIFAR-100	Imagenette
α = 0.1	α = 0.01	α = 0.1	α = 0.01	α = 0.1	α = 0.01
DSGDm [15]	87.89 ± 2.34	79.41 ± 3.29	47.93 ± 1.69	42.57 ± 2.71	66.89 ± 3.12	47.87 ± 4.03
QG-DSGDm [16]	92.21 ± 0.01	90.59 ± 0.92	53.19 ± 1.68	44.17 ± 3.64	73.93 ± 2.01	56.30 ± 5.43
QG-GUTm	92.55 ± 0.16	91.70 ± 0.36	53.40 ± 1.23	50.45 ± 1.30	75.44 ± 2.22	57.47 ± 5.33
and  $QG-GUTm$  methodologies is the same as the DSGD or QG-DSGDm techniques. All the above-mentioned algorithms communicate a vector with the size same as the model parameters. However, gradient tracking incurs  $2 \times$  communication cost in terms of model size. The communication cost and memory requirements for all the experiments are reported in Table. 12 and 13.
We report the numbers for communication, memory, and compute overheads for the memory-efficient implementation of GUT Algorithm. 4 in Table. 14. Memory overhead is reported as the percentage of additional memory required per agent during training with a batch size of 32.
$$
\text {M e m o r y} = \frac {\text {A d d i t i o n a l m e m o r y d u e t o G U T}}{\text {T o t a l M e m o r y}}
$$
The total memory includes the memory required to store model parameters, activations, gradients, gossip buffer, tracking variable  $(y_{i})$ , and weighted neighbors' parameters  $(s_i)$ . We observe that for compact models such as ResNet and MobileNet, the memory overhead is less than  $2\%$ . However, for larger models such as VGG-11, the memory overhead shoots up to  $14\%$ . The computational overhead is reported as the percentage of additional FLOPs required per sample per agent during training.