Query: asynchronization synchronization distributed Machine Learning

title: Divide and Shuffle Synchronization for Distributed Machine Learning
abstract: Distributed Machine Learning suffers from the bottleneck of synchronization to all reduce workers' updates. Previous works mainly consider better network topology, gradient compression, or stale updates to speed up communication and relieve the bottleneck. However, all these works ignore the importance of reducing the scale of synchronized elements and inevitable serial executed operators. To address the problem, our work proposes the Divide and Shuffle Synchronization(DS Sync) which divides workers into several parallel groups and shuffles group members. DS Sync only synchronizes the workers in the same group so that the scale of a group is much smaller. The shuffle of workers maintains the algorithm's convergence speed, which is interpreted in theory. Comprehensive experiments also show the significant improvements in the latest and popular models like Bert, WideResnet, and DeepFM on challenging datasets.

title: HiPS: Hierarchical Parameter Synchronization in Large Scale Distributed Machine Learning
abstract: In large scale distributed machine learning (DML) system, parameter (gradient) synchronization among machines plays an important role in improving the DML performance. State of the art DML synchronization algorithms, either the parameter server (PS) based algorithm or the ring allreduce algorithm, work in a flat way and suffer when the network size is large. In this work, we propose HiPS, a hierarchical parameter (gradient) synchronization framework in large scale DML. In HiPS, server centric network topology is used to better embrace RDMA/RoCE transport between machines, and the parameters (gradients) are synchronized in a hierarchical and hybrid way. Our evaluation in BCube and Torus network demonstrates that HiPS can better match server centric networks. Compared with the flat algorithms (PS based and ring based) HiPS reduces the synchronization time by 73% and 75% respectively.

title: Stay Fresh: Speculative Synchronization for Fast Distributed Machine Learning
abstract: Large machine learning models are typically trained in parallel and distributed environments. The model parameters are iteratively refined by multiple worker nodes in parallel, each processing a subset of the training data. In practice, the training is usually conducted in an asynchronous parallel manner, where workers can proceed to the next iteration before receiving the latest model parameters. While this maximizes the rate of updates, the price paid is compromised training quality as the computation is usually performed using stale model parameters. To address this problem, we propose a new scheme, termed speculative synchronization. Our scheme allows workers to speculate about the recent parameter updates from others on the fly, and if necessary, the workers abort the ongoing computation, pull fresher parameters, and start over to improve the quality of training. We design an effective heuristic algorithm to judiciously determine when to restart training iterations with fresher parameters by quantifying the gain and loss. We implement our scheme in MXNet a popular machine learning framework and demonstrate its effectiveness through cluster deployment atop Amazon EC2. Experimental results show that speculative synchronization achieves up to 3x speedup over the asynchronous parallel scheme in many machine learning applications, with little additional communication overhead.

title: Distributed Machine Learning through Heterogeneous Edge Systems
abstract: Many emerging AI applications request distributed machine learning (ML) among edge systems (e.g. IoT devices and PCs at the edge of the Internet) where data cannot be uploaded to a central venue for model training, due to their large volumes and/or security/privacy concerns. Edge devices are intrinsically heterogeneous in computing capacity, posing significant challenges to parameter synchronization for parallel training with the parameter server (PS) architecture. This paper proposes ADSP, a parameter synchronization scheme for distributed machine learning (ML) with heterogeneous edge systems. Eliminating the significant waiting time occurring with existing parameter synchronization models, the core idea of ADSP is to let faster edge devices continue training, while committing their model updates at strategically decided intervals. We design algorithms that decide time points for each worker to commit its model update, and ensure not only global model convergence but also faster convergence. Our testbed implementation and experiments show that ADSP outperforms existing parameter synchronization models significantly in terms of ML model convergence time, scalability and adaptability to large heterogeneity.

title: Distributed machine learning load balancing strategy in cloud computing services
abstract: Mobile service computing is a new cloud computing model that provides various cloud services for mobile intelligent terminal users through mobile internet access. The quality of service is an essential problem faced by mobile service computing. In this paper, we demonstrate a series of research studies on how to accelerate the training of a distributed machine learning (ML) model based on cloud service. Distributed ML has become the mainstream way of today's ML models training. In traditional distributed ML based on bulk synchronous parallel, the temporary slowdown of any node in the cluster will delay the calculation of other nodes because of the frequent occurrence of synchronous barriers, resulting in overall performance degradation. Our paper proposes a load balancing strategy named adaptive fast reassignment (AdaptFR) Based on this, we built a distributed parallel computing model called adaptive dynamic synchronous parallel (A DSP) A DSP uses a more relaxed synchronization model to reduce the performance consumption caused by synchronous operations while ensuring the consistency of the model. At the same time, A DSP also implements the AdaptFR load balancing strategy, which addresses the straggler problem caused by the performance difference between nodes under the premise of ensuring the accuracy of the model. The experiments show that A DSP can effectively improve the training speed while ensuring the accuracy of the model in the distributed ML model training.

title: Job scheduling for distributed machine learning in optical WAN
abstract: Large companies operate tens of data centers (DCs) across the globe to serve their customers and store data. On the other hand, many machine learning applications need a global view of such global data to pursue high model accuracy. However, for this Geo distributed machine learning (Geo DML) it is infeasible to move all data together over wide area networks (WANs) due to scarce WAN bandwidth, privacy concerns and data sovereignty laws. Therefore, most Geo DML systems leverage geo distributed approaches to train models, where global model synchronization is required between DCs over WAN. With the rapid increase of training data and the model sizes, it is challenging to efficiently utilize scarce and heterogeneous WAN bandwidth to synchronize models. With the advancement of optical technology, network topology becomes reconfigurable in optical WAN, which brings a new opportunity for Geo DML training over WAN. We propose to optimize Geo DML training with centralized joint control of the network and reconfigurable optical layers. We respectively prove the intra job and inter job scheduling problems are NP hard and strongly NP hard. For intra job scheduling, RoWAN based on deterministic rounding algorithm, is presented to dynamically change the topology by reconfiguring the optical devices, and allocate path and rate for each flow. For inter job scheduling, delayed SWRT is provided to schedule multiple jobs according to their priorities. The simulations in real topologies show that RoWAN reduces global model synchronization communication time of single iteration by up to 15.54% 48.2% on average in comparison with the traditional solutions. Compared to other three inter job scheduling approaches, delayed SWRT can reduce the weighted job completion time (WJCT) by about 60% 44.8% and 28.76%

title: DSANA: A distributed machine learning acceleration solution based on dynamic scheduling and network acceleration
abstract: Distributed machine learning(DML) has become a feasible solution to deal with the growing training data and models. Reviewing the existing architecture of DML, Parametric server(PS) architecture stands out in iterative convergence algorithms and widely deployed in practice, thanks to flexible expansion and so on. Under this architecture, the parameter synchronization mode based on Bulk Synchronous Parallel(BSP) has become one of the research hotspots. As for the BSP mode, each iteration efficiency is determined by the slowest node in the cluster, therefore, the straggler problem becomes the main reason for reducing the efficiency of DML training, which is even more prominent in the heterogeneous cloud services. Existing works mainly focus on the straggler problem, and the importance of communication is usually ignored. However, inefficient communication is also one of the reasons for the inefficiency of DML iterations. In this paper, we propose DSANA, which first alleviates certain straggler problems by dynamically scheduling computation tasks. Secondly, DSANA improves the overlap of computation/communication by dividing larger transmission parameters, thus further improving the iteration efficiency of DML training. We conduct comparison experiments with the classic iterative algorithm PageRank on four different scale data sets in two cloud service scenarios. The experimental results show that DSANA can improve the training efficiency to 36.6%\sim$ 56.4% compared with the baseline solution.

title: 2D HRA: Two Dimensional Hierarchical Ring Based All Reduce Algorithm in Large Scale Distributed Machine Learning
abstract: Gradient synchronization, a process of communication among machines in large scale distributed machine learning (DML) plays a crucial role in improving DML performance. Since the scale of distributed clusters is continuously expanding, state of the art DML synchronization algorithms suffer from latency for thousands of GPUs. In this article, we propose 2D HRA, a two dimensional hierarchical ring based all reduce algorithm in large scale DML. 2D HRA combines the ring with more latency optimal hierarchical methods, and synchronizes parameters on two dimensions to make full use of the bandwidth. Simulation results show that 2D HRA can efficiently alleviate the high latency and accelerate the synchronization process in large scale clusters. Compared with traditional algorithms (ring based) 2D HRA achieves up to 76.9% reduction in gradient synchronization time in clusters of different scale.

title: A Scalable, High Performance, and Fault Tolerant Network Architecture for Distributed Machine Learning
abstract: In large scale distributed machine learning (DML) the network performance between machines significantly impacts the speed of iterative training. In this paper we propose BML, a scalable, high performance and fault tolerant DML network architecture on top of Ethernet and commodity devices. BML builds on BCube topology, and runs a fully distributed gradient synchronization algorithm. Compared to a Fat Tree network with the same size, a BML network is expected to take much less time for gradient synchronization, for both low theoretical synchronization time and its benefit to RDMA transport. With server/link failures, the performance of BML degrades in a graceful way. Experiments of MNIST and VGG 19 benchmarks on a testbed with 9 dual GPU servers show that, BML reduces the job completion time of DML training by up to 56.4% compared with Fat Tree running state of the art gradient synchronization algorithm.

title: Lotus: A New Topology for Large scale Distributed Machine Learning
abstract: Machine learning is at the heart of many services provided by data centers. To improve the performance of machine learning, several parameter (gradient) synchronization methods have been proposed in the literature. These synchronization algorithms have different communication characteristics and accordingly place different demands on the network architecture. However, traditional data center networks cannot easily meet these demands. Therefore, we analyze the communication profiles associated with several common synchronization algorithms and propose a machine learning oriented network architecture to match their characteristics. The proposed design, named Lotus, because it looks like a lotus flower, is a hybrid optical/electrical architecture based on arrayed waveguide grating routers (AWGRs) In Lotus, a complete bipartite graph is used within the group to improve bisection bandwidth and scalability. Each pair of groups is connected by an optical link, and AWGRs between adjacent groups enhance path diversity and network reliability. We also present an efficient routing algorithm to make full use of the path diversity of Lotus, which leads to a further increase in network performance. Simulation results show that the network performance of Lotus is better than Dragonfly and 3D Torus under realistic traffic patterns for different synchronization algorithms.