Distributed Deep Learning In Open Collaborations

1 Introduction . The deep learning community is becoming increasingly more reliant on transfer learning . In computer vision , pretraining convolutional networks on large image collections such as ImageNet [ 1 ] is the de facto standard for a wide range of applications ranging from object detection [ 2 ] and semantic segmentation [ 3 ] to image classification [ 4 ] and even learning perceptual similarity [ 5 ] . A growing number of natural language processing systems capitalize on language models with billions of ∗Equal contribution . Correspondence to mryabinin0 @ gmail.com Detailed author contributions are listed at the end of the work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . parameters [ 6 , 7 , 8 , 9 , 10 , 11 ] trained on vast unlabeled corpora . Similar trends have emerged in areas such as speech processing [ 12 ] , reinforcement learning [ 13 ] , and computational biology [ 14 , 15 ] . Training these models is a notoriously time-consuming and challenging task : it often requires hundreds of high-end GPU servers [ 10 , 16 ] and would take multiple years on a single device [ 17 ] . Most academic and independent researchers simply can not afford to train state-of-the-art models from scratch , which slows down scientific progress and practical adoption of deep learning . Historically , the deep learning community has addressed this problem via “ model hubs ” or “ model zoos ” — public repositories for pretrained model checkpoints [ 18 , 19 , 20 , 21 ] . These repositories have played a significant role in the democratization of deep learning , allowing everyone to reap the benefits of large-scale training runs conducted by corporations and universities with sufficient resources . However , model hubs are limited to a narrow subset of datasets and tasks that match the interests of model creators . For instance , in natural language processing , it is often difficult to find up-to-date models for more than a handful of languages [ 22 ] . In turn , computer vision hubs rarely feature models trained on drawings , satellite images , 3D renders , microscopy , or any other data that does not resemble ImageNet . As a result , many researchers in these areas can only work on problems for which there are available pretrained models rather than the problems that most need solving . However , there might be an alternative way to obtain pretrained models : to train these models collaboratively . This approach , known as volunteer ( or grid ) computing , allows many independent parties to combine their computational resources and collectively perform large-scale experiments [ 23 , 24 , 25 ] . The raw compute performance of such collaborations often exceeds that of the fastest supercomputers [ 26 ] ; however , fully utilizing it can be challenging due to several reasons . First , devices that contribute to collaborative experiments can range from GPU servers and high-end workstations to consumer-grade computers and even smartphones [ 27 ] . Second , most of these devices use household internet connection with limited bandwidth and low reliability . Third , participants in such projects often donate their hardware part-time , joining and leaving the experiment at will . While it is theoretically possible to train neural networks on this kind of infrastructure , modern distributed training strategies are only efficient in a narrow range of conditions . For instance , training with Ring All-Reduce [ 28 ] works well for identical servers but suffers significant performance penalties from network latency or bandwidth variation [ 29 ] . Another technique known as Parameter Server can handle heterogeneous devices at the cost of being less scalable [ 30 ] . Applying any of these strategies outside their preferred conditions may significantly reduce the training throughput [ 31 ] , which makes them difficult to apply in the volatile infrastructure of volunteer computing . This issue is further complicated by the unique limitations of volunteer devices , such as network address translation ( NAT ) , regional access restrictions , or variations in performance . In this study , we carefully analyze the above challenges and come up with a practical solution for Distributed Deep Learning in Open Collaborations ( DeDLOC ) . DeDLOC is based on a novel algorithm that adapts to the available hardware in order to maximize the training throughput . Depending on the infrastructure , DeDLOC can recover parameter servers [ 30 ] , All-Reduce SGD [ 32 ] , decentralized SGD [ 33 ] , BytePS [ 34 ] , or an intermediate strategy that combines all of them . Using this algorithm , we propose a system for collaborative training designed to accommodate a large number of heterogeneous devices with uneven compute , bandwidth , reliability , and network capabilities . The contributions of our work can be summarized as follows : • We analyze the unique challenges of distributed training in open collaborations and propose a practical recipe for training in these conditions . • We formulate a novel distributed training algorithm that interpolates between traditional strategies to directly maximize the training performance for the available hardware . • We verify the effectiveness of the proposed algorithm and system design for unsupervised pretraining of ALBERT-Large and SwAV under realistic conditions . • We run collaborative training with actual volunteers , achieving competitive results to models trained on hundreds of data center GPUs . We also report insights on the collaborator activity and share the codebase for running similar experiments in the future2 . 2Code and training configurations are available at github.com/yandex-research/DeDLOC 2 Related work . 2.1 Distributed training . In this work , we focus on distributed data-parallel training , where each device runs forward and backward pass of the entire model on a subset of training examples . While there are many alternative techniques [ 35 , 36 , 37 ] , data-parallel is still the most popular strategy . Even the model-parallel approaches for extremely large models rely on data parallelism at the top level [ 37 , 16 , 38 ] . Training on multiple nodes was first implemented with parameter server ( PS ) [ 30 ] . This training strategy relies on a dedicated node that stores model parameters and executes optimization steps using the gradients sent by workers . In turn , worker nodes iteratively download the latest version of model parameters from the server , compute gradients and submit them back to the PS . This strategy is easy to implement and use , but it has an unavoidable bottleneck : the entire system performance is limited by the network throughput of a single server . Since then , the scientific community proposed numerous extensions to PS that alleviate the bottleneck by reducing the communication load [ 39 , 40 , 41 , 42 , 43 ] , introducing asynchronous updates [ 44 , 45 ] or training with multiple servers [ 46 , 34 ] . The issue of uneven communication load has also inspired the development and widespread adoption of another group of methods that rely on All-Reduce for gradient averaging [ 47 , 48 , 49 ] . All-Reduce is a family of collective operations that allow nodes to efficiently aggregate ( e.g . average ) their local vectors and distribute the result across all devices [ 28 , 50 , 51 ] . Unlike parameter servers , All-Reduce assigns equal roles to all devices , making it easier to scale to a large number of homogeneous workers . The popularity of AR-SGD sparked many practical applications for different scenarios . One particularly relevant application is elastic training [ 52 , 53 ] , which allows the user to add or remove workers at any point without interrupting the training run . While this bears a lot of similarity with collaborative training , we have found that elastic training systems are designed around global state synchronization , which makes them highly dependent on the homogeneity of the workers and their network connectivity . The overall efficiency is bounded by the performance of the lowest-performing node ; as a result , introducing even a single low-bandwidth participant to such systems reduces the training speed by orders of magnitude . Seeking to avoid the need for synchronization and centralized orchestration , the research community has developed decentralized training algorithms . These algorithms can be broadly divided into two categories : directly passing updates between peers [ 54 , 55 ] or running All-Reduce in small alternating groups [ 56 , 29 ] . Compared to PS and All-Reduce , both categories provide a greater degree of fault tolerance but often require more steps to converge due to delayed updates [ 33 , 29 ] . Most practical use cases of the above techniques take place in HPC or cloud conditions , but there is one notable exception . In Federated Learning , multiple parties train a shared model on decentralized privacy-sensitive data that can not be shared between devices [ 57 ] . For that reason , federated learning algorithms prioritize data privacy over training efficiency , often leaving most of the compute resources unused [ 58 , 59 ] . For a more detailed overview of Federated Learning , refer to Appendix A . 2.2 Volunteer Computing . Volunteer computing ( VC ) is a paradigm of distributed computing where people donate the idle time of their desktops , smartphones , and other personal devices to solve a computationally hard problem collectively . This approach has seen successful applications in bioinformatics , physics and other scientific areas [ 60 , 61 , 62 , 23 , 63 , 64 , 65 ] . In all these applications , volunteer computing allows researchers to access vast computational resources . In Folding @ home , over 700,000 volunteers have collectively contributed 2.43 exaFLOPs of compute to COVID-19 research in April of 2020 [ 26 ] . Another project named BOINC ( Berkeley Open Infrastructure for Network Computing ) brings together 41.548 petaFLOPs from over 790,000 active computers as of 17 March 2020 [ 25 ] . Volunteer computing systems were also the first “ supercomputers ” to reach 1 petaFLOP and 1 exaFLOP barriers [ 26 , 66 ] . These results became possible due to the contributions of a broad range of devices from high-end workstations to smartphones and even gaming consoles [ 67 ] . Unfortunately , this compute diversity is also the main limitation of VC . Any volunteer computing system should be able to run on a wide range of available hardware and maintain integrity even if some participants disconnect . Furthermore , the resources available to a project can vary over time , as most volunteers are only sharing their hardware when it is unused . Finally , volunteer devices are interconnected with a shared high latency network at typical home internet connection speeds . As a result , there were only a few successful attempts to apply volunteer computing to machine learning workloads . One such project is MLC @ Home [ 68 ] , which relies on volunteers to train many small independent models . This specific problem can be solved with no direct communication between participants . By contrast , distributed training of a single model requires significantly more communication and does not allow a natural way to “ restart ” failed jobs . When it comes to distributed training of neural networks , most volunteer computing projects rely on parameter server architectures [ 69 , 70 , 71 ] . As a result , these systems are bounded by the throughput of parameter servers and the memory available on the weakest GPU . The only notable exception is Learning @ home [ 72 ] , which uses expert parallelism to train larger models spanning multiple computers ; however , this approach has only been tested in simulated conditions .

Summary: The paper discusses a distributed training setup where multiple small institutes/groups pool computational resources together for training ML models collaboratively. The paper focuses on two key problems: (1) maintaining consistent training outcomes under dynamic composition of participants. (2) determining the communication strategy adaptively based on dynamic participants & network conditions.