## 3 The Proposed Method: SHAT
### Update Strategy for Model Convergence  
By updating the local model of each worker using the global model, the local models of all workers could be mixed indirectly through the global model. Thus, we aim to reduce the difference between the global model and the local model of each worker by effectively mixing the local models of workers. To this end, inspired by [8; 11], we define the generalized update rule for local model of each worker \(w^{i}\) in asynchronous distributed training, based on the problem formulation represented in Equation (1). By taking the gradient descent with respect to \(w^{i}\) on Equation (1), we get:  
\[w^{i}=w^{i}-\eta\cdot(g^{i}+\rho(w^{i}-\bar{w})) \tag{2}\]  
Denote \(\alpha=\eta\cdot\rho\) and \(w^{i}=w^{i}-\eta\cdot g^{i}\), which can be considered equal to the local parameter update of worker \(i\). Then, we get the following update rule for \(w^{i}\):  
\[w^{i} =w^{i}-\alpha(w^{i}-\bar{w})\] \[=(1-\alpha)w^{i}+\alpha\bar{w} \tag{3}\]  
Equation (3) implies that a worker updates its local model by taking the weighted average between its local model \(w^{i}\) and the global model \(\bar{x}\) where \(\alpha\) is an oscillating weight factor between local and global models. We note that this formulation can generalize diverse parameter update rules in distributed training. For example, if \(\alpha=0\), it is the same as local training only (i.e., ensemble), while, if \(\alpha=1\), it degenerates to the update method  
Figure 2: The difference between synchronous and asynchronous distributed training; (**a**) synchronous distributed training; (**b**) asynchronous distributed training. in existing asynchronous distributed training (e.g., Hogwild [6]), where the local model of each worker is completely replaced by the global model. With the generalized update rule represented in Equation (3), we can transform the problem of asynchronous distributed training to the problem of setting the oscillating weight factor \(\alpha\). Now, let us describe the update strategy for the local model of each worker (i.e., how to set \(\alpha\)) in SHAT. The proposed update strategy is based on the following intuition: the less the amount of the gradients of a worker is applied to the global model in PS, the larger the difference between the local model of the worker and the global model becomes. By this intuition, the difference may increase more as the number of workers gets larger (i.e., the larger scale of distributed training) and workers have more different training speeds (i.e., the larger heterogeneity among workers). Then, we take into account (1) the scale of distributed training and (2) the heterogeneity among workers in updating the local model of each worker, for effectively reducing the difference. Given the number of workers \(n\) and the degree of the staleness of each worker \(s^{i}\), we set the weight of each worker \(\alpha^{i}\) as follows. \[\alpha^{i}=1-\frac{s^{i}}{\log n},\quad s^{i}=\frac{n}{c^{i}}. \tag{4}\]  
Here, we define the degree of the staleness of each worker as \(s^{i}=\frac{n}{c^{i}}\), where \(c^{i}\) is the number of updates by other workers in the global model since the previous update by worker \(i\) has been performed (i.e., \(s^{i}\approx 1,c^{i}\approx n\) in a homogeneous cluster, where all workers have almost same training speed). Thus, in SHAT, the weight of the global model for updating the local model of each worker, \(\alpha^{i}\) is determined by the scale of distributed training and the heterogeneity among workers. The weight of the global model increases with a more number of workers and a less degree of heterogeneity among workers, helping (1) the local models of workers to be combined more effectively and (2) the local model of each worker to catch up quickly the learning direction of the global model. We note that \(\alpha^{i}\) is not a hand-tuned hyperparmeter but automatically decided by the number of workers \(n\) and the degree of the staleness of each worker \(s^{i}\).