Distributed Deep Learning In Open Collaborations

1 Introduction . The deep learning community is becoming increasingly more reliant on transfer learning . In computer vision , pretraining convolutional networks on large image collections such as ImageNet [ 1 ] is the de facto standard for a wide range of applications ranging from object detection [ 2 ] and semantic segmentation [ 3 ] to image classification [ 4 ] and even learning perceptual similarity [ 5 ] . A growing number of natural language processing systems capitalize on language models with billions of ∗Equal contribution . Correspondence to mryabinin0 @ gmail.com Detailed author contributions are listed at the end of the work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . parameters [ 6 , 7 , 8 , 9 , 10 , 11 ] trained on vast unlabeled corpora . Similar trends have emerged in areas such as speech processing [ 12 ] , reinforcement learning [ 13 ] , and computational biology [ 14 , 15 ] . Training these models is a notoriously time-consuming and challenging task : it often requires hundreds of high-end GPU servers [ 10 , 16 ] and would take multiple years on a single device [ 17 ] . Most academic and independent researchers simply can not afford to train state-of-the-art models from scratch , which slows down scientific progress and practical adoption of deep learning . Historically , the deep learning community has addressed this problem via “ model hubs ” or “ model zoos ” — public repositories for pretrained model checkpoints [ 18 , 19 , 20 , 21 ] . These repositories have played a significant role in the democratization of deep learning , allowing everyone to reap the benefits of large-scale training runs conducted by corporations and universities with sufficient resources . However , model hubs are limited to a narrow subset of datasets and tasks that match the interests of model creators . For instance , in natural language processing , it is often difficult to find up-to-date models for more than a handful of languages [ 22 ] . In turn , computer vision hubs rarely feature models trained on drawings , satellite images , 3D renders , microscopy , or any other data that does not resemble ImageNet . As a result , many researchers in these areas can only work on problems for which there are available pretrained models rather than the problems that most need solving . However , there might be an alternative way to obtain pretrained models : to train these models collaboratively . This approach , known as volunteer ( or grid ) computing , allows many independent parties to combine their computational resources and collectively perform large-scale experiments [ 23 , 24 , 25 ] . The raw compute performance of such collaborations often exceeds that of the fastest supercomputers [ 26 ] ; however , fully utilizing it can be challenging due to several reasons . First , devices that contribute to collaborative experiments can range from GPU servers and high-end workstations to consumer-grade computers and even smartphones [ 27 ] . Second , most of these devices use household internet connection with limited bandwidth and low reliability . Third , participants in such projects often donate their hardware part-time , joining and leaving the experiment at will . While it is theoretically possible to train neural networks on this kind of infrastructure , modern distributed training strategies are only efficient in a narrow range of conditions . For instance , training with Ring All-Reduce [ 28 ] works well for identical servers but suffers significant performance penalties from network latency or bandwidth variation [ 29 ] . Another technique known as Parameter Server can handle heterogeneous devices at the cost of being less scalable [ 30 ] . Applying any of these strategies outside their preferred conditions may significantly reduce the training throughput [ 31 ] , which makes them difficult to apply in the volatile infrastructure of volunteer computing . This issue is further complicated by the unique limitations of volunteer devices , such as network address translation ( NAT ) , regional access restrictions , or variations in performance . In this study , we carefully analyze the above challenges and come up with a practical solution for Distributed Deep Learning in Open Collaborations ( DeDLOC ) . DeDLOC is based on a novel algorithm that adapts to the available hardware in order to maximize the training throughput . Depending on the infrastructure , DeDLOC can recover parameter servers [ 30 ] , All-Reduce SGD [ 32 ] , decentralized SGD [ 33 ] , BytePS [ 34 ] , or an intermediate strategy that combines all of them . Using this algorithm , we propose a system for collaborative training designed to accommodate a large number of heterogeneous devices with uneven compute , bandwidth , reliability , and network capabilities . The contributions of our work can be summarized as follows : • We analyze the unique challenges of distributed training in open collaborations and propose a practical recipe for training in these conditions . • We formulate a novel distributed training algorithm that interpolates between traditional strategies to directly maximize the training performance for the available hardware . • We verify the effectiveness of the proposed algorithm and system design for unsupervised pretraining of ALBERT-Large and SwAV under realistic conditions . • We run collaborative training with actual volunteers , achieving competitive results to models trained on hundreds of data center GPUs . We also report insights on the collaborator activity and share the codebase for running similar experiments in the future2 . 2Code and training configurations are available at github.com/yandex-research/DeDLOC 2 Related work . 2.1 Distributed training . In this work , we focus on distributed data-parallel training , where each device runs forward and backward pass of the entire model on a subset of training examples . While there are many alternative techniques [ 35 , 36 , 37 ] , data-parallel is still the most popular strategy . Even the model-parallel approaches for extremely large models rely on data parallelism at the top level [ 37 , 16 , 38 ] . Training on multiple nodes was first implemented with parameter server ( PS ) [ 30 ] . This training strategy relies on a dedicated node that stores model parameters and executes optimization steps using the gradients sent by workers . In turn , worker nodes iteratively download the latest version of model parameters from the server , compute gradients and submit them back to the PS . This strategy is easy to implement and use , but it has an unavoidable bottleneck : the entire system performance is limited by the network throughput of a single server . Since then , the scientific community proposed numerous extensions to PS that alleviate the bottleneck by reducing the communication load [ 39 , 40 , 41 , 42 , 43 ] , introducing asynchronous updates [ 44 , 45 ] or training with multiple servers [ 46 , 34 ] . The issue of uneven communication load has also inspired the development and widespread adoption of another group of methods that rely on All-Reduce for gradient averaging [ 47 , 48 , 49 ] . All-Reduce is a family of collective operations that allow nodes to efficiently aggregate ( e.g . average ) their local vectors and distribute the result across all devices [ 28 , 50 , 51 ] . Unlike parameter servers , All-Reduce assigns equal roles to all devices , making it easier to scale to a large number of homogeneous workers . The popularity of AR-SGD sparked many practical applications for different scenarios . One particularly relevant application is elastic training [ 52 , 53 ] , which allows the user to add or remove workers at any point without interrupting the training run . While this bears a lot of similarity with collaborative training , we have found that elastic training systems are designed around global state synchronization , which makes them highly dependent on the homogeneity of the workers and their network connectivity . The overall efficiency is bounded by the performance of the lowest-performing node ; as a result , introducing even a single low-bandwidth participant to such systems reduces the training speed by orders of magnitude . Seeking to avoid the need for synchronization and centralized orchestration , the research community has developed decentralized training algorithms . These algorithms can be broadly divided into two categories : directly passing updates between peers [ 54 , 55 ] or running All-Reduce in small alternating groups [ 56 , 29 ] . Compared to PS and All-Reduce , both categories provide a greater degree of fault tolerance but often require more steps to converge due to delayed updates [ 33 , 29 ] . Most practical use cases of the above techniques take place in HPC or cloud conditions , but there is one notable exception . In Federated Learning , multiple parties train a shared model on decentralized privacy-sensitive data that can not be shared between devices [ 57 ] . For that reason , federated learning algorithms prioritize data privacy over training efficiency , often leaving most of the compute resources unused [ 58 , 59 ] . For a more detailed overview of Federated Learning , refer to Appendix A . 2.2 Volunteer Computing . Volunteer computing ( VC ) is a paradigm of distributed computing where people donate the idle time of their desktops , smartphones , and other personal devices to solve a computationally hard problem collectively . This approach has seen successful applications in bioinformatics , physics and other scientific areas [ 60 , 61 , 62 , 23 , 63 , 64 , 65 ] . In all these applications , volunteer computing allows researchers to access vast computational resources . In Folding @ home , over 700,000 volunteers have collectively contributed 2.43 exaFLOPs of compute to COVID-19 research in April of 2020 [ 26 ] . Another project named BOINC ( Berkeley Open Infrastructure for Network Computing ) brings together 41.548 petaFLOPs from over 790,000 active computers as of 17 March 2020 [ 25 ] . Volunteer computing systems were also the first “ supercomputers ” to reach 1 petaFLOP and 1 exaFLOP barriers [ 26 , 66 ] . These results became possible due to the contributions of a broad range of devices from high-end workstations to smartphones and even gaming consoles [ 67 ] . Unfortunately , this compute diversity is also the main limitation of VC . Any volunteer computing system should be able to run on a wide range of available hardware and maintain integrity even if some participants disconnect . Furthermore , the resources available to a project can vary over time , as most volunteers are only sharing their hardware when it is unused . Finally , volunteer devices are interconnected with a shared high latency network at typical home internet connection speeds . As a result , there were only a few successful attempts to apply volunteer computing to machine learning workloads . One such project is MLC @ Home [ 68 ] , which relies on volunteers to train many small independent models . This specific problem can be solved with no direct communication between participants . By contrast , distributed training of a single model requires significantly more communication and does not allow a natural way to “ restart ” failed jobs . When it comes to distributed training of neural networks , most volunteer computing projects rely on parameter server architectures [ 69 , 70 , 71 ] . As a result , these systems are bounded by the throughput of parameter servers and the memory available on the weakest GPU . The only notable exception is Learning @ home [ 72 ] , which uses expert parallelism to train larger models spanning multiple computers ; however , this approach has only been tested in simulated conditions .

Summary: This paper attempts to deploy distributed training of deep neural networks over an open collaborative distributed environment, where participators can dynamically join the computational task.  The approach is mainly based on modification of data parallel training to overcome the technique challenges introduced by the collaborative environment.  Both standard and real-world pre-train benchmarks are included to evaluate the proposed system.