SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sai Aparna Aketi Abolfazl Hashemi Kaushik Roy
West Lafayette, IN 47906
{saketi, abolfazl, kaushik}@purdue.edu
Decentralized learning enables training of deep learning models over large distributed datasets generated at different locations, without the need for a central server. However, in practical scenarios, the data distribution across these devices can be significantly different, leading to a degradation in model performance. In this paper, we focus on designing a decentralized learning algorithm that is less susceptible to variations in data distribution across devices. We propose Global Update Tracking (GUT), a novel tracking-based method that aims to mitigate the impact of heterogeneous data in decentralized learning without introducing any communication overhead. We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette), model architectures, and network topologies. Our experiments show that the proposed method achieves state-of-the-art performance for decentralized learning on heterogeneous data via a  $1 - 6\%$  improvement in test accuracy compared to other existing techniques.
Decentralized learning is a branch of distributed optimization which focuses on learning from data distributed across multiple agents without a central server. It offers many advantages over the traditional centralized approach in core aspects such as data privacy, fault tolerance, and scalability [18]. It has been demonstrated that decentralized learning algorithms [15] can perform comparable to centralized algorithms on benchmark vision datasets. Decentralized Parallel Stochastic Gradient Descent (DSGD) presented in [15] combines SGD with a gossip averaging algorithm [26]. Further, the authors analytically show that the convergence rate of DSGD is similar to its centralized counterpart [5]. A momentum version of DSGD referred to as Decentralized Momentum Stochastic Gradient Descent (DSGd m) was proposed in [3]. The authors in [2] introduce Stochastic Gradient Push (SGP) which extends DSGD to directed and time-varying graphs. Recently, a unified framework for analyzing gossip-based decentralized SGD methods and the best-known convergence guarantees was presented in [11].
One of the key assumptions to achieve state-of-the-art performance by all the above-mentioned decentralized algorithms is that the data is independently and identically distributed (IID) across the agents. In particular, the data is assumed to be distributed in a uniform and random manner across the agents. This assumption does not hold in most real-world settings where the data distributions across the agents are significantly different (non-IID/heterogeneous) [9]. The effect of heterogeneous data in a peer-to-peer decentralized setup is a relatively under-studied problem and an active area of research.
Recently, there have been few efforts to bridge the performance gap between IID and non-IID data for a decentralized setup [16, 23, 19, 7, 1, 24]. Cross Gradient Aggregation [7] and Neighborhood
Gradient Clustering [1] algorithms utilize the concept of cross-gradients to reduce the impact of heterogeneous data and show significant improvement in performance (test accuracy). However, these techniques incur  $2 \times$  communication cost than the standard decentralized algorithms such as DSGD.  $D^{2}$  algorithm proposed in [23] is shown to be agnostic to data heterogeneity and can be employed in deep learning tasks. One of the major limitations of  $D^{2}$  is that its convergence requires mixing topologies with negative eigenvalue bounded from below by  $-\frac{1}{3}$ . Additionally, it has been shown that  $D^{2}$  performs worse than DSGD in some cases [16].
Tracking mechanisms such as Gradient Tracking (GT) [6, 19] and Momentum Tracking (MT) [22] have been proposed to tackle heterogeneous data in decentralized settings. But these algorithms improve the performance at the cost of  $2 \times$  communication overhead. The authors in [16] introduce Quasi-Global Momentum (QGM), a decentralized learning method that mimics the global synchronization of momentum buffer to mitigate the difficulties of decentralized learning on heterogeneous data. Recently, RelaySGD was presented in [24] that replaces the gossip averaging step with RelaySum. Since RelaySGD deals with the gossip averaging step, it is orthogonal to the aforementioned algorithms and can be used in synergy with them. QG-DSGDm [16] which incorporates QGM into DSGDm sets the current state-of-the-art for decentralized learning on heterogeneous data without increasing the communication cost. This work investigates the following question: Can we improve decentralized learning on heterogeneous data through a tracking mechanism without any communication overhead?
To that effect, we present Global Update Tracking (GUT), a novel decentralized learning algorithm designed to improve performance under heterogeneous data distribution. Motivated by, yet distinct from, the gradient tracking mechanism, we propose to track the consensus model  $(\bar{x}^t)$  by tracking global/average model updates, where  $x_{i}^{t}$  is the model parameters on agent  $i$  at time step  $t$  and  $\bar{x}$  is the averaged model parameters. In the traditional tracking-based methods [19, 22] that track average gradients, each agent communicates both model parameters  $x_{i}^{t}$  and the tracking variable  $y_{i}^{t}$  with its neighbors resulting in  $2\times$  communication overhead. The proposed GUT algorithm overcomes this bottleneck by allowing agents to store a copy of their neighbors' model parameters and then tracking the model updates instead of the gradients. This results in communicating only the tracking variable  $y_{i}^{t}$  that yields the model update  $(x_{i}^{t} - x_{i}^{t - 1})$ . We demonstrate the effectiveness of the proposed algorithm through an exhaustive set of experiments on various datasets, model architectures, and graph topologies. We also provide a detailed convergence analysis showing that the convergence rate of GUT algorithm is consistent with the state-of-the-art decentralized learning algorithms. Further, we show that QG-GUTm - Global Update Tracking with Quasi-Global momentum beats the current state-of-the-art decentralized learning algorithm (i.e., QG-DSGDm) on heterogeneous data under iso-communication cost.
In summary, we make the following contributions.
- We propose Global Update Tracking (GUT), a novel tracking-based decentralized learning algorithm to mitigate the impact of heterogeneous data distribution.
- We theoretically establish the non-asymptotic convergence rate of the proposed algorithm to a first-order solution.
- Through an exhaustive set of experiments on various datasets, model architectures, and graph topologies, we establish that the proposed Global Update Tracking with Quasi-Global momentum (QG-GUTm) outperforms the current state-of-the-art decentralized learning algorithm on a spectrum of heterogeneous data.
In this section, we provide the background on the decentralized setup with peer-to-peer connections.
The main goal of decentralized machine learning is to learn a global model using the knowledge extracted from the locally stored data samples across  $n$  agents while maintaining privacy constraints. In particular, we solve the optimization problem of minimizing the global loss function  $f(x)$  distributed across  $n$  agents as given in (1). Note that  $F_{i}$  is a local loss function (for example, cross-entropy loss)
defined in terms of the data  $(d_i)$  sampled from the local dataset  $D_{i}$  at agent  $i$
$$
\min  _ {x \in \mathbb {R} ^ {d}} f (x) = \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (x), \tag {1}
$$
where  $f_{i}(x) = \mathbb{E}_{d_{i}\sim D_{i}}[F_{i}(x;d_{i})]$ , for all  $i$ .
The optimization problem is typically solved by combining stochastic gradient descent [4] with global consensus-based gossip averaging [26]. The communication topology is modeled as a graph  $G = ([N], E)$  with edges  $\{i, j\} \in E$  if and only if agents  $i$  and  $j$  are connected by a communication link exchanging the messages directly. We represent  $\mathcal{N}(i)$  as the neighbors of agent  $i$  including itself. It is assumed that the graph  $G$  is strongly connected with self-loops i.e., there is a path from every agent to every other agent. The adjacency matrix of the graph  $G$  is referred to as a mixing matrix  $W$  where  $w_{ij}$  is the weight associated with the edge  $\{i, j\}$ . Note that, weight 0 indicates the absence of a direct edge between the agents, and the elements of the Identity matrix are represented by  $I_{ij}$ . Similar to the majority of previous works in decentralized learning, the mixing matrix is assumed to be doubly stochastic. Further, the initial models and all the hyperparameters are synchronized at the beginning of the training. The communication among the agents is assumed to be synchronous.
Traditional decentralized algorithms such as DSGD [15] assume the data across the agents to be Independent and Identically Distributed (IID). In DSGD, each agent  $i$  maintains local parameters  $x_{i}^{t}\in \mathbb{R}^{d}$  and updates them as follows.
$$
\text {D S G D :} x _ {i} ^ {t + 1} = \sum_ {j \in \mathcal {N} (i)} w _ {i j} \left(x _ {j} ^ {t} - \eta g _ {j} ^ {t}\right); \quad g _ {i} ^ {t} = \nabla F _ {j} \left(x _ {i} ^ {t}, d _ {i} ^ {t}\right). \tag {2}
$$
We focus on a decentralized setup with non-IID/heterogeneous data. In particular, the heterogeneity in the data distribution comes in the form of skewed label partition similar to [9]. Decentralized learning with the DSGD algorithm on heterogeneous data distribution results in performance degradation due to huge variations in the local gradients across the agents. To tackle this, authors in [16] propose a momentum-based optimization technique (QG-DSGDm) introducing Quasi-Global momentum as shown in (3).
$$
\text {Q G - D S G D m :} x _ {i} ^ {t + 1} = \sum_ {j \in \mathcal {N} (i)} w _ {i j} \left[ x _ {j} ^ {t} - \eta \left(g _ {j} ^ {t} + \beta m _ {j} ^ {t - 1}\right) \right]; \quad m _ {i} ^ {t} = \mu m _ {i} ^ {t - 1} + (1 - \mu) \frac {x _ {i} ^ {t - 1} - x _ {i} ^ {t}}{\eta}. \tag {3}
$$
QG-DSGDm improves the performance of decentralized learning on heterogeneous data without any communication overhead and is used as a baseline for comparison in this work.
Gradient Tracking (GT) mechanisms [19, 22] are also known to improve decentralized learning on heterogeneous data by reducing the variance between the local gradient and the averaged (global) gradient. To achieve this, the gradient tracking algorithm introduces a tracking variable  $y_{i}^{t}$  that approximates the total gradient and is used to update the local parameters  $x_{i}^{t}$  (refer to (4)).
$$
\text {G T}: x _ {i} ^ {t + 1} = \sum_ {j \in \mathcal {N} (i)} w _ {i j} \left(x _ {j} ^ {t} - \eta y _ {j} ^ {t}\right); \quad y _ {i} ^ {t} = \sum_ {j \in \mathcal {N} (i)} w _ {i j} y _ {j} ^ {t - 1} - g _ {i} ^ {t - 1} + g _ {i} ^ {t}. \tag {4}
$$
The update rule of tracking variable is such that it recursively adds a correction term  $\left(\sum_{j\in \mathcal{N}(i)}w_{ij}y_j^{t - 1} - g_i^{t - 1}\right)$  to the local gradient  $g_{i}^{t}$ , pushing  $y_{i}^{t}$  to be closer to the global gradients  $\left(\frac{1}{n}\sum_{j = 1}^{n}g_{j}^{t}\right)$ . This requires each agent  $i$  to communicate two sets of parameters  $x_{i}^{t}$  and  $y_{i}^{t}$  with its neighbors. Thus, the gradient tracking algorithm improves the decentralized learning on non-IID data at the cost of  $2\times$  communication overhead.
3 Global Update Tracking
We present Global Update Tracking (GUT), a novel algorithm for decentralized deep learning on non-IID data distribution. GUT is a communication-free tracking mechanism that aims to mitigate the difficulties of decentralized training when the data distributed across the agents is heterogeneous.
In order to attain the benefits of gradient tracking without communication overhead, we propose to apply the tracking mechanism with respect to the model updates  $x_{i}^{t} - x_{i}^{t - 1}$  instead of the gradients
Algorithm 1 Global Update Tracking (GUT)
Input: Each agent  $i \in [1, n]$  initializes model parameters  $x_{i}^{0}$  and neighbors' copy  $\hat{x}_{j}^{0}$ , step size  $\eta$ , GUT scaling factor  $\mu$ , mixing matrix  $W = [w_{ij}]_{i,j \in [1,n]}$ ,  $\mathcal{N}(i)$  represents neighbors of  $i$  including itself, and note  $\hat{x}_{i}^{t} = x_{i}^{t}$ .
Each agent simultaneously implements the  $\mathrm{TRAIN}()$  procedure
1. procedure TRAIN()
2. for  $t = 0,1,\dots ,T - 1$  do
3.  $d_{i}^{t}\sim D_{i}$
4.  $g_{i}^{t} = \nabla_{x}F_{i}(\sum_{j\in \mathcal{N}(i)}w_{ij}\hat{x}_{j}^{t};d_{i}^{t})$
5.  $\delta_i^t = g_i^t -\frac{1}{\eta}\sum_{j\in \mathcal{N}(i)}w_{ij}(\hat{x}_j^t -x_i^t)$
6.  $y_{i}^{t} = \delta_{i}^{t} + \mu \Big[\sum_{j\in \mathcal{N}(i)}w_{ij}(y_{j}^{t - 1} - \frac{1}{\eta} (\hat{x}_{j}^{t} - x_{i}^{t})) - \delta_{i}^{t - 1}\Big]$
7.  $\mathbf{SEND}\mathbf{RECEIVE}(y_i^t)$
8.  $x_{i}^{t + 1} = x_{i}^{t} - \eta y_{i}^{t}$
9.  $\hat{x}_j^{t + 1} = \hat{x}_j^t -\eta y_j^t\forall j\in \mathcal{N}(i)\backslash i$
10. end
11. return  $\frac{1}{n}\sum_{i = 1}^{n}x_i^T$
$g_{i}^{t}$ . Firstly, to design a tracking mechanism without additional communication cost, each agent  $i$  communicates model updates instead of model parameters to its neighbors. An agent  $i$  stores a copy of its neighbor's parameters as  $\hat{x}_j$  and updates it using the received model updates to retrieve the current version of the neighbor's parameters as shown in line-9 of Algorithm 1. A memory-efficient implementation of the algorithm (Algorithm 4 in Appendix B) requires each agent to store  $s_i = \sum_{j\in \mathcal{N}(i)}w_{ij}\hat{x}_j$  instead of storing each neighbor's copy separately requiring only  $\mathcal{O}(1)$  additional memory [12].
Now, we define a variable  $\delta_i^t$  on each agent  $i$  that accumulates the local gradient update  $g_i^t$  and the gossip averaging update  $\sum_{j}(w_{ij} - I_{ij})\hat{x}_j^t$  as shown in line-5 of Algorithm 1. Note that we can recover the DSGD update defined in (2) by using  $\delta_i^t$  in the update rule i.e.,  $x_i^{t + 1} = x_i^t -\eta \delta_i^t$ . We then proceed to compute the tracking variable  $y_i^t$ , as described in line-6 of Algorithm 1, using the combined model update (local gradient part and gossip averaging part) reflected by  $\delta_i^t$ . The gossip averaging part of the update for each agent  $i$  i.e.,  $\sum_{j}w_{ij}(\hat{x}_j^t -x_i^t)$  is computed with respect to its own model weights. To account for this in the computation of tracking variable  $y_i^t$ , the agents have to adjust the information received from the neighbors (i.e.,  $y_j^{t}$ 's) to change the reference to itself. This is reflected as an additional term  $\frac{1}{\eta} (\hat{x}_j^t -x_i^t)$  in the update rule given by line-6 of Algorithm 1. Further, we scale the correction term of the tracking variable by a factor  $\mu$ , a hyper-parameter, which is tuned to extract the maximum benefits of the proposed algorithm.
In summary, the update scheme of GUT can be re-formulated in the following matrix form where  $X = [x_{1},\ldots ,x_{n}]\in \mathbb{R}^{d\times n}$  are the model parameters and  $G = [g_1,\dots,g_n]\in \mathbb{R}^{d\times n}$  are stochastic gradients.
$$
\begin{array}{l} X ^ {t + 1} = X ^ {t} - \eta Y ^ {t}, \\ Y ^ {t + 1} = G ^ {t + 1} - \frac {1}{\eta} (W - I) X ^ {t + 1} + \mu \left[ W Y ^ {t} - G ^ {t} - \frac {1}{\eta} (W - I) \left(X ^ {t + 1} - X ^ {t}\right) \right]. \tag {5} \\ \end{array}
$$
Finally, we show that integrating the proposed GUT algorithm with Quasi-Global Momentum improves the current state-of-the-art significantly without any communication overhead. The pseudocode for the momentum version of our algorithm (QG-GUTm) is presented in Appendix B.
This section provides the convergence analysis for the proposed GUT Algorithm. We assume that the following standard assumptions hold:
Assumption 1 (Lipschitz Gradients). Each function  $f_{i}(x)$  is  $L$ -smooth i.e.,  $||\nabla f_{i}(y) - \nabla f_{i}(x)|| \leq L||y - x||$ .
Assumption 2 (Bounded Variance). The stochastic gradients are unbiased and their variance is assumed to be bounded.
$$
\mathbb {E} _ {d \sim D _ {i}} | | \nabla F _ {i} (x; d) - \nabla f _ {i} (x) \| ^ {2} \leq \sigma^ {2} \forall i \in [ 1, n ], \tag {6}
$$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} | | \nabla f _ {i} (x) - \nabla f (x) | | ^ {2} \leq \zeta^ {2}. \tag {7}
$$
Assumption 3 (Doubly Stochastic Mixing Matrix). The mixing matrix  $W$  is a real doubly stochastic matrix with  $\lambda_1(W) = 1$  and
$$
\max  \left\{\left| \lambda_ {2} (W) \right|, \left| \lambda_ {n} (W) \right| \right\} \leq 1 - \rho <   1, \tag {8}
$$
where  $\lambda_{i}(W)$  is the  $i^{th}$  largest eigenvalue of  $W$  and  $\rho$  is the spectral gap.
The above assumptions are commonly used in most decentralized learning setups. Theorem 1 presents the convergence of the proposed GUT algorithm and the proof is detailed in Appendix A.
Theorem 1. (Convergence of GUT algorithm) Given Assumptions 1, 2, and 3 let step size  $\eta \leq \frac{\rho}{7L}$  and the scaling factor  $\frac{\mu}{1 - \mu} \leq \frac{\rho}{42}$ . For all  $T \geq 1$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} \leq \frac {4}{\eta T} \left(f \left(\bar {x} ^ {0}\right) - f ^ {*}\right) + \eta \frac {4 L \sigma^ {2}}{n} + \eta^ {2} \frac {1 2 4 8 L ^ {2}}{\rho^ {2}} \left(\zeta^ {2} + \sigma^ {2} (2 - \mu)\right), \tag {9}
$$
where  $f(\bar{x}^0) - f^*$  is the sub-optimality gap,  $\bar{x}$  is the average/consensus model parameters.
The result of the Theorem 1 shows that the averaged gradient of the averaged model is upper-bounded by the sub-optimality gap (the difference between the initial objective function value and the optimal value), the sampling variance  $(\sigma)$ , and gradient variations across the agents representing data heterogeneity  $(\zeta)$ . Further, we present a corollary to show the convergence rate of GUT in terms of the number of iterations.
Corollary 1. Suppose that the step size satisfies  $\eta = \mathcal{O}\left(\sqrt{\frac{n}{T}}\right)$  For a sufficiently large  $T$  we have,
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} | | \nabla f (\bar {x} ^ {t}) | | ^ {2} \leq \mathcal {O} \left(\frac {1}{\sqrt {n T}} + \frac {1}{T}\right). \tag {10}
$$
Corollary 1 indicates that the GUT algorithm achieves linear speedup with a convergence rate of  $\mathcal{O}\left(\frac{1}{\sqrt{nT}}\right)$  when  $T$  is sufficiently large and is independent of communication topology. In other words, the communication complexity to find an  $\epsilon$ -first order solution, i.e.,  $\mathbb{E}\|\nabla f(\bar{x})\|^2 \leq \epsilon$  is  $\mathcal{O}\left(\frac{\sigma^2}{n\epsilon^2}\right)$ . This convergence rate is similar to the well-known best result for decentralized SGD algorithms [15] in the literature.
In this section, we analyze the performance of the proposed GUT and QG-GUTm techniques and compare them with the baseline DSGD algorithm [15] and the current state-of-the-art QG-DSGDm [16] respectively. The source code is available at https://github.com/aparna-aketi/global_update_tracking
The efficiency of the proposed method is demonstrated through our experiments on a diverse set of datasets, model architectures, graph topologies, and graph sizes. We present the analysis on (a) Datasets: CIFAR-10, CIFAR-100, Fashion MNIST, and Imagenette. (b) Model architectures: VGG-11, ResNet-20, LeNet-5 and, MobileNet-V2. All the models use Evonorm [17] as the activation-normalization layer as it is shown to be better suited for decentralized learning on heterogeneous data.
Figure 1: Ring Graph (left), Dyck Graph (center), and Torus Graph (right).
(c) Graph topologies: Ring graph with 2 peers per agent, Dyck graph with 3 peers per agent, and Torus graph with 4 peers per agent (refer Figure 1). (d) Number of agents: 16-40 agents. We use the Dirichlet distribution to generate disjoint non-IID data across the agents. The created data partition across the agents is fixed, non-overlapping, and never shuffled across agents during the training. The degree of heterogeneity is regulated by the value of  $\alpha$  – the smaller the  $\alpha$  the larger the non-IIDness across the agents. We report the test accuracy of the consensus model averaged over three randomly chosen seeds. The details of the decentralized setup and hyperparameters for all the experiments are presented in Appendix C.
5.2 Average Consensus Task
We first consider an average consensus task that is isolated from the learning through stochastic gradient descent. Here the aim is that all the agents should reach a consensus which is the average value of the initial information each agent holds. The following equations show the simplified version of GUT (11) and QG-GUTm (12) after removing the gradient update part.
$$
X ^ {t + 1} = X ^ {t} + Y ^ {t}; \quad Y ^ {t} = (W - I) X ^ {t} + \mu \left[ W Y ^ {t - 1} - (W - I) \left(X ^ {t - 1} - X ^ {t}\right) \right], \tag {11}
$$
$$
X ^ {t + 1} = X ^ {t} + \hat {M} ^ {t}; \quad M ^ {t} = \beta M ^ {t - 1} + (1 - \beta) \left(X ^ {t} - X ^ {t - 1}\right) \tag {12}
$$
$$
\hat {M} ^ {t} = \beta M ^ {t} + (1 - \beta) [ (W - I) X ^ {t} + \mu (W \hat {M} ^ {t - 1} - (W - I) (X ^ {t - 1} - X ^ {t})) ].
$$
Note that setting the hyper-parameter  $\mu$  as 0 in the (11), 12 gives simple gossip[26] and quasi-global gossip [16] respectively and all the agents communicate  $X^{t} - X^{t - 1}$  at iteration  $t$  with their neighbors.
Figure.2 shows the average consensus error i.e.,  $\frac{1}{n} ||X^t -\bar{X} ||_F^2$  over time for the average consensus task on the Ring topology with respect to various algorithms. We observe that the gossip averaging with GUT converges faster than simple gossip averaging. Figure.2(c) illustrates that for graphs with a smaller spectral gap (which corresponds to more agents), the proposed QG-GUTm can converge faster than quasi-global gossip (gossip with QGM) resulting in better decentralized optimization.
(a) 64 agents
(b) 128 agents
(c) 256 agents
Figure 2: Decentralized average consensus problem on an undirected ring topology
5.3 Decentralized Deep Learning Results
We evaluate the efficiency of GUT and its quasi-global momentum version  $QG-GUTm$  with the help of an exhaustive set of experiments. We compare GUT with DSGD and the momentum version  $QG-GUTm$  with QG-DSGDm to show that the proposed method outperforms the current state-of-the-art.
Table. 1 shows the average test accuracy for training ResNet-20 and VGG-11 models on the CIFAR-10 dataset with varying degrees of non-IIIDness over ring topology of 16 and 32 agents. We observe that GUT consistently outperforms DSGD for all models, graph sizes, and degree of heterogeneity with a significant performance gain varying from  $1 - 18\%$ . The quasi-global momentum version of our algorithm, QG-GUTm, beats QG-DSGDm with  $1 - 3.5\%$  improvement in the case of the CIFAR-10 dataset partitioned with a higher degree of heterogeneity ( $\alpha = 0.1, 0.01$ ).
Table 1: Test accuracy of different decentralized algorithms evaluated on CIFAR-10, distributed with different degrees of heterogeneity (non-IID) for various models over ring topologies. The results are averaged over three seeds where the standard deviation is indicated. We also include the results of the IID baseline as DSGDm (IID) where the local data is randomly partitioned independent of  $\alpha$ .